While AI can come up with convincing answers, the bigger risk is when people stop asking questions, the professor says.
Lucy Gill-Simmen, associate dean of education and student experience at Royal Holloway, University of London, said the technology could discourage people from questioning, verifying and researching information themselves.
“Hallucinations are a visible problem because we notice them when the answer is wrong,” she told Business Insider about AI’s well-documented tendency to fabricate false information. “The more serious risk is that people stop asking how they know if the answer is correct.”
Gil Simmen said knowledge has traditionally required effort: comparing sources, testing assumptions, and overcoming uncertainty. In contrast, AI can generate convincing answers without requiring the user to go through the process.
“My concern is not just misinformation,” she said. “It’s that people are less inclined to examine, question, and investigate for themselves. In education, learning often happens in struggle. AI removes a lot of that struggle.”
AI as a replacement for thinking
Gil-Simmen’s concerns reflect a growing body of research on what Wharton researchers call “cognitive surrender,” or the tendency to accept answers generated by AI rather than actively evaluate them.
Stephen Shaw, a postdoctoral fellow in marketing at Wharton, warned about this in a report earlier this year, telling Business Insider that people risk becoming “passive followers of ideas they haven’t thought about” by adopting ideas generated by AI without fully processing them.
In three experiments in which 1,372 participants completed 9,593 reasoning tasks, they found that participants chose to consult an AI assistant on more than half of the tasks, and once consulted, they accepted the suggestion about 92.7% of the time when it was correct and 79.8% of the time even when it was intentionally wrong.
Similar concerns arise in education. Kimberley Hardcastle, a business marketing professor at Northumbria University in the UK, said AI could erode people’s ability to independently verify, challenge and build on knowledge without algorithms.
Gil Simmen has described this phenomenon as “epistemological atrophy,” or “the gradual weakening of the habit of acquiring knowledge.”
“When people work through problems on their own, they develop mental models, strengthen their reasoning skills, and gain confidence in their own thinking,” she says.
“The problem is not that AI will replace memory; it is that there is a risk that AI will replace thinking.”
“The illusion of understanding”
Gil Simen sees this firsthand in her She said students sometimes experience an “illusion of understanding.”
AI-generated explanations may sound convincing, but when students are asked to reason on their own or apply ideas to new contexts, the depth of understanding is often much weaker, she says.
Wharton University professor Ethan Mollick said AI works best as a form of “collaborative intelligence” that helps people explore ideas and challenge assumptions, as long as humans remain responsible for evaluating the AI’s output.
In a paper he co-authored with his wife, learning scientist Lilac Molik, he wrote that students should “critically evaluate and question AI artifacts, rather than passively accept them,” and ensure that AI “acts as a tool to support their work, rather than a replacement.”
“In some ways, ironically, the more capable the AI becomes, the more important human capabilities become,” Gil-Simmen said.
“In a world filled with fluent output produced by AI, success will depend less on the generation of information and more on judicious evaluation of it,” she added.