Listening device: EEG electrodes implanted in the brain record neuronal chatter while a person talks.
Dialogue is a semi-choreographed dance. That is, a series of interactions in which each person must first understand the words of the other person and then plan and execute a response. These steps—language comprehension, speech production, and the transitions between—have distinct patterns of brain activity, according to a new preprint. This preprint is the first study to use implanted electrodes to record brain activity while people have natural conversations.
“Our main contribution to the field is that these conversations are natural conversations,” says researcher Jin Kai, a lecturer in neurosurgery at Harvard Medical School.
Several studies have investigated what happens in the brain when a person is listening to speech and, to a lesser extent, speaking, but during unrestricted natural conversation. What happens in the brain is almost completely unmapped. The main reason is that the tools to track it have not yet been developed. “New possibilities are emerging,” said Sydney Cash, associate professor of neurology at Harvard Medical School and co-leader of the study. “The ability to do this is becoming more prevalent.”
Studies have also shown that the way the brain processes verbal information is strikingly similar to how natural language processing (NLP) models compute the same verbal exchanges. An NLP model is a type of artificial intelligence model used in chatbots such as ChatGPT to determine the intended meaning of text and generate a response.
The results suggest that combining NLP with brain data analysis could provide a powerful tool for language learning, said Christian Helf, assistant professor of neuroscience at Maastricht University in the Netherlands. (not involved in this study). Furthermore, “the inclusion of semantic information about speech in these algorithms may also improve accuracy.”
IIn the study, which was posted to bioRxiv in March, researchers recorded electroencephalography (EEG) electrodes implanted in the brains of six patients with severe epilepsy who were monitored before undergoing surgery to treat their symptoms. was analyzed. The researchers acquired neural recordings from a total of 855 electrodes implanted in 33 brain regions while the participants engaged in open-ended conversations with the experimenter for 20 to 92 minutes.
To focus on slices of brain activity that reflect the semantic aspects of speech rather than the acoustic or motor components of speech, the researchers also ran the same conversational texts in an NLP called GPT2. . NLP is pre-trained to process language through a layered structure, and a layer is essentially a series of processors. Similar to previous research, different layers were found to process different aspects of information. The lower layers focused on word-level aspects of the language, while the upper layers focused on contextual and thematic aspects.
Across different brain regions, about 10% of the activity captured by the electrodes during speech was strongly correlated with the activity of artificial neurons or nodes in the layer-by-layer organized NLP model. Of this subset of brain signals, those corresponding to speech comprehension and planning correlated with the upper layers of NLP, and those corresponding to individual word utterances correlated with the lower layers. “Our results showed noticeable differences in his NLP layer, which controls speech generation and speech planning and understanding,” says Cai.
“We are not suggesting this particular thing. [NLP] The model is a replication of neural activity,” he says. “Rather, the general form of processing appears to have similarities in both humans and machines.”
When the researchers analyzed brain activity, they found that different electrodes across many brain regions were activated depending on whether the participants were speaking or listening. “Overall, we found very few electrodes that show selectivity for both speaking and hearing. They separate fairly well,” says Kai.
EEG recordings also captured differences in brain activity between these two modes of speech. Signals tended toward low frequencies when participants were trying to speak, but tended toward mid frequencies when participants were listening.
As previous studies have suggested, low frequencies are associated with top-down processing (in other words, beginning with more conceptual information and spilling over to sensory and motor functions), while high frequencies are associated with bottom-up processing. says Kai. “Our hypothesis is that in order to speak, we know what we want to say and transmit that signal to the motor areas and throughout the brain, whereas in order to hear we need the senses. is. [input]”
R.Researchers are increasingly using NLP models to investigate language representations in the brain. In 2022, a team used functional MRI (fMRI) to capture the brain activity of participants listening to a podcast. Running both the audio and brain signals on her NLP revealed strong similarities between how the brain and NLP calculated intelligibility.
Last month, another team trained NLP on fMRI brain scans of people listening to podcasts. The resulting machine partially deciphered thoughts into words, triggering the dread of mind-reading.
But so far, these and other efforts have only involved language understanding, not production, Herb explains. This is because the facial movements that people make when speaking generally distort recordings. An invasive approach piggybacked on clinical needs, like the one used by Cash and his colleagues, “I think it’s the most ethical approach,” Helf says.
Emily Meyers, professor of speech, language, and hearing sciences at the University of Connecticut at Storrs, says there’s no question that two people in conversation will change and synchronize with each other’s brain activity. Cash and colleagues’ approach could monitor these changes. “There are really interesting questions about how these dynamics work together and synchronize with each other,” she says.
Cash said he hopes the group can build on the techniques they’ve used to explore some of the other dimensions beyond the semantic aspects of language. For example, the visual or emotional context of a conversation, facial expressions and tones that occur during a conversation all contribute to meaning. Understanding their interactions in conversation could indicate intervention for people with autism and other communication problems, he says.
