Using AI to “see” what we see

When we see the world, our brains do not only recognize objects such as “dogs” and “cars,” but we also understand the broader meanings, such as what is happening, where it is happening, how everything fits. But for years, scientists have not had a good way to measure its rich and complex understanding.

Now, in a new study published today in Nature Machine Intelligence, Ian Sharast, an associate professor of psychology at the University of Montreal, explains how to grasp it using a large-scale language model (LLM) with colleagues at the University of Minnesota, the University of German University Osnabrück and Frey University Berlin.

“By supplying these LLMs with descriptions of natural scenes, the same kind of AI behind tools like ChatGpt, we created something like “language-based fingerprints” of the meaning of the scene,” said Charest, a holder of Udem's basic neuroscience and a member of the Mila -Quebec Ai Institute.

“Amazing,” he said, “These fingerprints closely matched the brain activity patterns recorded while people were watching the same scene on the MRI scanner,” including groups of children and big city skylines.

“For example,” said Charest. LLMS allows you to decode visual scenes that a person has recognized in a statement. You can also use LLM-encoded representations to accurately predict how the brain will respond to scenes in scenes that include food, locations, and human faces. ”

Researchers went even further. They trained artificial neural networks to incorporate images to predict these LLM fingerprints, and found that these networks did a better job in conforming brain responses than many of the most advanced AI visual models available today.

And this despite this despite the fact that these available models are trained with much less data.

These concepts of “artificial neural networks” were supported by Professor Tim Kietmann, a professor of machine learning at Osnabrück University, and his team. The first author of this study was Professor Adrian Drigg of the University of Berlin.

“What we've learned suggests that the human brain may represent complex visual scenes, which is surprisingly similar to the way modern language models understand text,” said Charest, who continues to study the subject.

“Our research,” he continues.

“These new technologies could one day help develop visual prosthetics for people with visual impairments. But ultimately, this is a step forward in understanding how the human brain understands meaning from the visual world.”