Newswise — New York City’s thousands of traffic cameras capture endless footage every day, but analyzing that footage to identify safety issues and implement improvements requires resources that most transit agencies typically don’t have.
Now, researchers at New York University’s Tandon School of Engineering have developed an artificial intelligence system that can automatically identify collisions and near misses in existing traffic videos by combining verbal reasoning and visual intelligence. This could change the way cities improve road safety without major new investments.
The study, published in the journal Accident Analysis and Prevention, won New York City’s Vision Zero Research Award, which annually recognizes research that aligns with New York City’s traffic safety priorities and provides actionable insights. Professor Khan Ozbey, senior author of the paper, presented the research at the 8th Annual Road Research Symposium on 19 November.
The study exemplifies a cross-disciplinary collaboration between computer vision experts at New York University’s new Center for Robotics and Embodied Intelligence and traffic safety researchers at NYU Tandon’s C2SMART Center, where Ozbay is director.
By automatically identifying when and where collisions and near misses occur, the team’s system, called SeeUnsafe, can help transit agencies pinpoint dangerous intersections and road conditions that require intervention before more serious incidents occur. It leverages a pre-trained AI model that can understand both images and text, and represents one of the first applications of multimodal large-scale language models for analyzing long-duration traffic videos.
“There are 1,000 cameras in New York City running 24 hours a day, seven days a week. It’s impossible for people to go through and analyze all the footage manually,” Osbay said. “SeeUnsafe offers city governments a highly effective way to make the most of their existing investments.”
“Government agencies don’t need to be computer vision experts to use this technology without having to collect and label their own data to train AI-based video analytics models,” added NYU Tandon Associate Professor Chen Feng, co-founder director of the Center for Robotics and Embodied Intelligence and co-author of the paper.
When tested on the Toyota Woven Traffic Safety dataset, SeeUnsafe outperformed other models, correctly classifying videos as crashes, near misses, or normal traffic 76.71% of the time. The system can also identify which specific road users are involved in critical events, with a success rate of up to 87.5%.
Traditionally, road safety measures have only been implemented after an accident has occurred. By analyzing near-miss patterns, such as vehicles following too closely to pedestrians or performing unsafe maneuvers at intersections, agencies can proactively identify danger zones. This approach makes it possible to take preventive measures before serious accidents occur, such as improving signage, optimizing signal timing, and redesigning road layouts.
This system generates a “road safety report”. This is a natural language explanation for the decision that describes factors such as weather conditions, traffic, and the specific movements that led to the near miss or collision.
Although the system has limitations, including sensitivity to object tracking accuracy and challenges in low-light conditions, it establishes the foundation for using AI to “understand” road safety situations from vast amounts of traffic footage. The researchers suggest that this approach could be extended to in-vehicle dashcams to enable real-time risk assessment from the driver’s perspective.
This research adds to the growing number of C2SMART efforts that can improve New York City’s transportation system. Recent projects include studying the strain large electric trucks put on the city’s roads and bridges, analyzing how speed cameras change driver behavior in different neighborhoods, developing a “digital twin” that can find smarter routes to reduce FDNY response times, and a multi-year collaboration with the city to monitor overweight vehicles that cause damage on the Brooklyn-Queens Expressway.
In addition to Ozbey and Feng, the paper’s authors include first author Dr. Ruixuan Zhang. Student studying transportation engineering at New York University Tandon School. Beichen Wang and Juexiao Zhang are both graduate students at New York University’s Courant Institute for Mathematical Sciences. and Zilin Bian, who recently received his Ph.D. from New York University Tandon. He graduated and is currently an assistant professor at Rochester Institute of Technology.
Funding for this research came from the National Science Foundation and the U.S. Department of Transportation’s University Transportation Center Program.
