This is good news for sports broadcasters and fans who want a human touch. AI doesn’t know the ball.
A new study by researchers at the University of North Carolina at Chapel Hill and Northeastern University finds that top AI models are terrible at analyzing professional sports. The study, which has not yet been peer-reviewed, sought to analyze how capable the most popular AI models are in the areas of perception, reasoning, simulation, and agency, four characteristics that are difficult to assess using existing testing methods.
To explore how AI might work in these areas, researchers looked to the wider world of sports and created a new type of AI test. The new test, called Strategic Video Intelligence, or “SVI Bench,” consisted of 35,000 hours of sports footage, including basketball, soccer, and hockey, 15 million annotated plays, 15,000 hours of expert analysis, 23,000 postgame reports, and 103,000 statistical records.
The AI performed best in perception, identifying which players performed which actions at specific points in the match. But even there they struggled. These models, including ChatGPT, Google’s Gemini, and the open source model Qwen, were able to monitor which players were doing what about 74 percent of the time. These odds are enough to get even a volunteer Little League announcer fired.
AI models were far worse at inferring and explaining causal relationships. why Some plays ended in failure, and the success rate dropped to an average of nearly 40%. For example, when researchers asked the models to identify what was unusual about Cody Martin’s 3-point shot that bounced off the backboard before landing in the bucket, ChatGPT responded that it was “the first game in which he made a 3-point shot.”
Simulations, asking the AI to find evidence to predict things like where the player would physically go based on their trajectory, were also disastrous. During these tests, the best-performing model was functionally tossing a coin to guess the player’s next step, but performance deteriorated further when the model was asked to plot long moves toward the goal or basket.
Lorenzo Torresani, a computer science researcher at Northeastern University and co-author of the study, said in a university press blurb that AI “cannot tell you why things happen or what will happen next.”
Researchers looked at the models’ agencies and found they were asked to perform complex post-game analysis of statistics and trends, essentially the same way a human announcer would, but their accuracy dropped to just 5 percent.
“A good sportscaster doesn’t just explain what’s on the screen; they explain why plays worked, predict what’s going to happen next, and decide which moments are important,” Torresani said. “Our research shows that AI is already pretty good at the descriptive part, but falls short in the rest.”
While sportscasters may certainly breathe a sigh of relief, the findings are also good news for other knowledge workers, amid persistent fears that AI automation will upend the job market.
“The same gap appears in any work where the value is not in explaining what you see, but in understanding why events unfold, predicting what will happen next, determining what is important, and recommending what to do about it,” Torresani concluded.
More about AI in sports: New York Jets say they will switch to AI, shocking fans
