Partnering with the Federal Railroad Administration, New Jersey Transit and New Jersey Department of Transportation (NJDOT), a research team at Rutgers University is using artificial intelligence (AI) techniques to analyze rail crossing safety issues. Utilizing closed-circuit television (CCTV) cameras installed at rail crossings, a team of Rutgers researchers, Asim Zaman, Xiang Liu, Zhipeng Zhang, and Jinxuan Xu, have developed and refined an AI-aided framework for detection of railroad trespassing events to identify the behavior of trespassers and capture video of infractions.  The system uses an object detection algorithm to efficiently observe and process video data into a single dataset.

Rail trespassing is a significant safety concern resulting in injuries and deaths throughout the country, with the number of such incidents increasing over the past decade. Following passage of the 2015 Fixing America’s Surface Transportation (FAST) Act that mandated the installation of cameras along passenger rail lines, transportation agencies have installed CCTV cameras at rail crossings across the country.  Historically, only through recorded injuries and fatalities were railroads and transportation agencies able to identify crossings with trespassing issues. This analysis did not integrate information on near misses or live conditions at the crossing. Cameras could record this data, but reviewing the video would be a laborious task that required a significant resource commitment and could lead to missed trespassing events due to observer fatigue.

Zaman, Liu, Zhang, and Xu saw this problem as an opportunity to put AI techniques to work and make effective use of the available video and automate the observational process in a more systematic way. After utilizing AI for basic video analysis in a prior study, the researchers theorized that they could train an AI and deep learning to analyze the videos from these crossings and identify all trespassing events.

Working with NJDOT and NJ TRANSIT, they gained access to video footage from a crossing in Ramsey, NJ.  Using a deep learning-based detection method named You Only Look Once or YOLO, their AI-framework detected trespassings, differentiated the types of violators, and generated clips to review. The tool identified a trespass only when the signal lights and crossing gates were active and tracked objects that changed from image to image in the defined space of the right-of-way. Figure 1 depicts the key steps in the process for application of AI in the analysis of live video stream or archived surveillance video.

The researchers applied AI review to 1,632 hours of video and 68 days of monitoring. They discovered 3,004 instances of trespassing, an average of 44 per day and nearly twice an hour. The researchers were able to demonstrate how the captured incidents could be used to formulate a demographic profile of trespassers (Figure 2) and better examine the environmental context leading to trespassing events to inform the selection and design of safety countermeasures (Figure 3).

A significant innovation from this research has been the production of the video clip that shows when and how the trespass event occurred; the ability to visually review the precise moment reduces overall data storage and the time needed performing labor-intensive reviews. (Zhang, Zaman, Xu, & Liu, 2022)

With the efficient assembly and analysis of video big data through AI techniques, agencies have an opportunity, as never before, to observe the patterns of trespassing. Extending this AI research method to multiple locations holds promise for perfecting the efficiency and accuracy in application of AI techniques in various lighting, weather and other environmental conditions and, more generally, to building a deeper understanding of the environmental context contributing to trespassing behaviors.

In fact, the success of this AI-aided Railroad Trespassing Tool has led to new opportunities to demonstrate its use. The researchers have already expanded their research to more crossings in New Jersey and into North Carolina and Virginia. (Bruno, 2022) The Federal Railroad Administration has also awarded the research team a $582,859 Consolidated Rail Infrastructure and Safety Improvements Grant to support the technology’s deployment at five at-grade crossings in New Jersey, Connecticut, Massachusetts, and Louisiana. (U.S. DOT, Federal Railroad Administration, 2021) Rutgers University and Amtrak have provided a 42 percent match of the funding.

The program’s expansion in more places may lead to further improvements in the precision and quality of the AI detection data and methods.  The researchers speculate that this technology could integrate with Positive Train Control (PTC) systems and highway Intelligent Transportation Systems (ITS). (Zhang, Zaman, Xu, & Liu, 2022) This merging of technologies could revolutionize railroad safety. To read more about this study and methodology, see this April 2022 Accident Analysis & Prevention article.