summary: A new artificial intelligence system, the semantic decoder, can convert brain activity into continuous text. The system could revolutionize communication for people who are unable to speak due to illnesses such as stroke.
This non-invasive approach uses data from fMRI scanners to convert thoughts into text without the need for surgical implants. Although not perfect, this AI system succeeds in capturing the essence of human thought half the time.
Important facts:
- Semantic Decoder AI was developed by researchers at the University of Texas at Austin.
- It works on a Transformer model similar to what powers Open AI’s ChatGPT and Google’s Bard.
- The system could potentially be used with more portable brain imaging systems such as functional near-infrared spectroscopy (fNIRS).
sauce: Austin, Utah
A new artificial intelligence system called a semantic decoder can convert a person’s brain activity while listening to a story or silently imagining telling a story into a continuous text stream.
A system developed by researchers at the University of Texas at Austin could allow conscious but physically unable to speak, such as those debilitated by stroke, to communicate clearly again.
Research published in journals natural neurosciencewas led by computer science PhD student Jerry Tan and Alex Huth, assistant professor of neuroscience and computer science at UT Austin.
This work relies in part on a Transformer model similar to what powers Open AI’s ChatGPT and Google’s Bard.
Unlike other language decoding systems in development, this system does not require the subject to have surgical implants, making the process non-invasive. Also, participants are not required to use only words from a given list.
Brain activity is measured using an fMRI scanner after extensive training of the decoder. In this case, individuals listen to podcasts on their scanners for hours.
Afterwards, if the participants were willing to have their thoughts deciphered, they could listen to new stories or imagine telling stories so that the machine would respond to texts from brain activity alone. can be generated.
“As a non-invasive method, this is a big step forward compared to what has been done before (usually in one word or short sentence),” Huth said. increase. “We are using complex ideas to obtain models that decode long-term continuous languages.”
The result is not a verbatim transcript. Instead, the researchers designed it to capture the gist of what was being said or thought, albeit imperfectly. If the decoder is trained to monitor participants’ brain activity, about half the time, the machine produces text that closely (and sometimes exactly) matches the intended meaning of the original word. To do.
For example, in an experiment, participants hearing the speaker’s statement, “I don’t have a driver’s license yet,” interpreted the thought as, “She hasn’t even started practicing driving yet.” Hearing those words, “I didn’t know if I should scream, cry, or run away. Instead, I said, ‘Leave me alone!'” It started and she just said, ‘Leave me alone.'”
Starting with an early version of the paper published as a preprint online, the researchers addressed questions about potential abuses of the technology. This paper describes how only cooperative participants who actively participate in decoder training can decode.
Results for non-decoder-trained individuals were incomprehensible, and if decoder-trained participants later resisted (e.g., thinking differently), the results were equally unusable.
“We take concerns that it could be used for bad purposes very seriously and have tried to avoid it,” Tan said. “We want to ensure that people only use these kinds of technologies when they want to, and that they are useful.”
In addition to asking participants to listen and think about stories, the researchers asked subjects to watch four short silent videos while inside the scanner. A semantic decoder was able to use brain activity to accurately describe specific events from the video.
This system relies on the time required of the fMRI machine, making it impractical for use outside the laboratory at this time. However, the researchers believe the work could be applied to other, more portable brain-imaging systems, such as functional near-infrared spectroscopy (fNIRS).
“fNIRS measures where the blood flow is more or less in the brain at different times, which turned out to be exactly the same kind of signal that fMRI is measuring,” says Huis. said Mr. “Therefore, our approach should just be applied to fNIRS,” he noted, although the resolution of fNIRS would be lower.
This research was supported by the Whitehall Foundation, the Alfred P. Sloan Foundation, and the Burroughs Wellcome Fund.
Other co-authors on the study are Amanda Revell, a former research assistant in the Huth lab, and Shiley Jain, a computer science graduate student at the University of Texas at Austin.
Alexander Huth and Jerry Tang have filed PCT patent applications related to this work.
About this AI research news
author: Mark Earhart
sauce: Austin, Utah
contact: Mark Earhart – UT Austin
image: Image credited to Neuroscience News
Original research: closed access.
“Semantic reconstruction of continuous language from non-invasive brain recordings.” Jerry Tang et al. natural neuroscience
overview
Semantic reconstruction of continuous language from noninvasive brain recordings
A brain-computer interface that decodes serial language from non-invasive recordings would have many scientific and practical applications. However, currently non-invasive language decoders are only able to identify stimuli among a small set of words or phrases. Here, we present a noninvasive decoder that reconstructs continuous language from cortical semantic representations recorded using functional magnetic resonance imaging (fMRI). Given the new brain recordings, this decoder produces comprehensible word sequences that restore the meaning of perceived speech, imaginary speech, and even silent video, making it possible for a single decoder to be applied to a wide variety of tasks. It shows what you can do. We tested the decoder across the cortex and found that continuous language could be decoded independently from multiple regions. Since brain-computer interfaces must respect mental privacy, we tested whether subject cooperation was required for successful decoding and found that both training and application of the decoder required subject cooperation. It turns out there is. Our findings demonstrate the feasibility of a non-invasive verbal brain-computer interface.
