Neuroscientists use AI to simulate how the brain makes sense of the visual world

A research team at the Wu Tsai Neuroscience Institute at Stanford University has made great strides in using AI to recreate how the brain processes sensory information and makes sense of the world, breaking new ground in virtual neuroscience.

When we watch the second hand of a clock tick, in the visual areas of the brain, groups of neighboring angle-selective neurons fire in succession as the hand circles the clock face. These cells form beautiful “pinwheel” maps, with each segment representing the visual perception of a different angle. Other visual areas of the brain contain maps of more complex and abstract visual features, such as distinguishing between familiar images of faces and places, and activate distinct neural “neighborhoods.”

Such functional maps can be found throughout the brain, delighting and perplexing neuroscientists, who have long wondered why the brain evolved into the map-like layout that only modern science can observe.

To answer this question, a team of researchers at Stanford University developed a new kind of AI algorithm, a Topographic Deep Artificial Neural Network (TDANN), that uses just two rules: natural sensory inputs and spatial constraints on connections, and found that it can successfully predict both the sensory responses and spatial organization of multiple parts of the human brain's visual system.

After seven years of extensive research, their findings were published May 10 in a new paper in the journal Neuroscience, “A unifying framework for the functional organization of early and higher ventral visual cortex.” Neuron.

The research team was led by Dan Yamins, assistant professor and researcher of psychology and computer science in the Wu Tsai Neuroscience Institute, and Calanit Grill-Spector, professor of psychology at the institute.

Unlike traditional neural networks, TDANNs incorporate spatial constraints, arranging virtual neurons on a two-dimensional “cortical sheet” and requiring nearby neurons to share similar responses to sensory input.

As the model learned to process the images, this topographical structure formed a spatial map that mimicked the way neurons in the brain self-organize in response to visual stimuli. Specifically, the model reproduced complex patterns such as the pinwheel structure of the primary visual cortex (V1) and clusters of neurons in the higher ventral temporal cortex (VTC) that respond to categories such as faces and places.

Eshed Margalit, lead author of the study and a PhD candidate alongside Yamins and Grill-Spector, said the team used a self-supervised learning approach to improve the accuracy of their training models that simulate the brain.

“This is probably similar to how babies learn about their visual world,” Margalit says. “I don't think we expected initially that this would have such a big impact on the accuracy of the trained model, but to have a good model of the brain, we need to get the training task of the network right.”

This fully trainable model could help neuroscientists better understand the rules of how the brain self-organizes, whether in the case of vision, as in this study, or other sensory systems such as hearing.

“When the brain is trying to learn something about the world, say when it looks at two snapshots of a person, it places neurons that respond similarly close together in the brain and forms a map,” said Grill Spector, the Susan S. and William H. Hindle Professor in the School of Humanities and Sciences. “We think this principle should be applicable to other systems.”

This innovative approach has important implications for both neuroscience and artificial intelligence. For neuroscientists, TDANNs offer new perspectives for studying how the visual cortex develops and functions, potentially transforming the treatment of neurological diseases. For AI, insights gained from the brain's organization could lead to more sophisticated visual processing systems, as well as teaching computers to “see” like humans.

The discovery may also help explain why the human brain works so energy efficiently: For example, it can perform trillions of calculations using just 20 watts of power, whereas a supercomputer requires a million times more energy to perform the same calculations.

The new findings highlight that neuronal maps, and the spatial or topographical constraints that drive them, likely play a role in keeping the wiring that connects the brain's 100 billion neurons as simple as possible. These insights could be key to designing more efficient artificial systems inspired by the brain's elegance.

“AI is power constrained,” Yamins said. “In the long term, figuring out how to run artificial systems on much lower power consumption could help advances in AI.”

Energy-efficient AI could help advance virtual neuroscience, where experiments can be performed faster and at scale. In this study, as a proof of principle, the researchers demonstrated that a topographical deep artificial neural network replicates brain-like responses to a range of natural visual stimuli. This suggests that in the future, such systems could be used as a fast, inexpensive playground for prototyping neuroscience experiments and quickly identifying hypotheses for future testing.

Virtual neuroscience experiments could also advance human medicine. For example, better training artificial vision systems in the same way that babies visually learn about the world could help AI see the world like humans do, with the center of gaze sharper than the rest of the field of vision. Other applications could help develop visual prosthetics and accurately simulate the effects of disease or injury on different parts of the brain.

“It would be really exciting if we could make predictions that would help us develop prosthetic devices for people who have lost their vision,” Grill-Spector said.