Bad artificial intelligence doesn't just miss the mark. It can lead to real consequences for people and serious risks for businesses.
It was a message from Professor Arvind Narayanan of Princeton University in his recent talk at MIT about his new book, “AI Snake Oil: How Artificial Intelligence Can, Can't, How to Tell The Differences.” This book was co-authored by Sayash Kapoor.
Their core argument: Some AI tools used today, especially in employment, lending and criminal justice, are not only poorly performed. They simply don't work. Others work as claimed, but are used in the wrong way or for harmful purposes. Also, in both scenarios, AI tools can produce inaccurate, biased, or dangerous outcomes when applied at large scale.
For business leaders, the challenge is knowing what to trust and where to draw the lines. Narayanan provided the following insights:
Find weak links in predictive AI
Predictive AI systems are designed to predict human behavior and support decision-making, but are increasingly used in areas such as employment and criminal justice. According to Narayanan, many of these tools do not meet their claims.
One example he quoted: software that analyzes 30-second job seeker videos and evaluates personality traits based on audio and body language. These videos often do not focus on candidate qualifications – sometimes just by hobbies – are used to generate scores that claim to predict job fit.
Narayanan called this approach “a generator of elaborate random numbers,” pointing to experiments testing software reliability. Minor visual changes, such as adding bookshelves to the background and removing pairs of glasses, led to “radially different scores” even when the underlying video was the same.
These concerns range from the criminal justice system where algorithms are used to guide decisions about whether someone should be detained before trial. The performance of these tools is often weak, Narayanan said – they have a prediction accuracy of less than 70% at best.
“We make decisions about someone else's freedom based on something slightly more accurate than the flip of the coin,” he said.
Leading an AI-led organization
Directly on MIT Sloan
Sign up now
Focus on what generate AI works
Not all AI is snake oil. Generation AI – a tool that can generate text, images and code – has already proven valuable.
“Generating AI is useful for essentially every knowledge worker, that is, those who make a living,” Narayanan said.
He provided a personal example of using an AI-powered app generator to help his daughter understand fragments. On the spot, the model helped him build an interactive tool that turns learning into a game.
“We played about 15 minutes on this and it really helped her,” he said. “I couldn't imagine doing this a few years ago.”
Still, even the most impressive system is not perfect. Hallucinations that occur when the model generates information that sounds accurate but is not, remain known risks. Given that the generated AI is inherently accompanied by randomness, they are not easily modified.
Takeout: These tools are only useful when combined with proper checks and accountability.
Address hidden risks
As generator AI tools become more widely used, so is the risk of misuse. Many examples are already being played in the real world.
Narayanan pointed to an AI-generated foraging guide that provides inaccurate and potentially dangerous advice for users about which mushrooms are safe. He also noted the rapid spread of the Deepfake Pornography app. This means that you can create unconsensually express images using the uploaded photos.
Additionally, even if an AI system is used for its intended purpose, its effects can still be damaging.
For example, facial recognition technology has reached a high level of technical accuracy, but does not rule out concerns about misuse. “Loady surveillance using facial recognition…it works really, really well now,” Narayanan said. “And that's actually part of why it's harmful when used without proper guardrails.”
This is the lesson. Just because a tool works doesn't mean it's safe. Leaders need to weigh both their value and their potential misuse.
Ask the right question
Narayanan suggested asking two questions to avoid costly missteps. Does the tool work as it is claimed? And even so, can it cause harm?
He pointed to AI-based fraud detectors as a warning story that fails tests. These tools often flag the wrong students, especially non-native English speakers. “They don't work. …It feels like snake oil to me,” Narayanan said.
Generating AI has its own pitfalls. Narayanan said that AI-based agents can do things like navigating websites, shopping online, and more, but there is no credibility that users expect. Software products intended to complete these tasks are “severely dead upon arrival,” he said.
His company guidance: Keep grounded. Focus on a narrow, well-defined problem. And don't mistake hype for preparation.
Treat AI as infrastructure, not magic
AI should not be seen as magic, it should be seen as infrastructure, Narayanan said. “One day, a lot of what we call AI today will disappear into the background,” he said.
That's not the reason for retreat, that's why we keep it sharp. “We need to know which applications are inherently harmful or exaggerated,” he said. “Even if it makes sense to deploy AI applications, you still need GuardRails.”
Check out Arvind Narayanan's talk about Ai Snake Oil
