Science and Technology Daily, Beijing, May 22 (Reporter Liu Xia) American scientists have developed the first bilingual brain reading device. This brain implant can decode the words spoken in Spanish or English in the user's brain in real time, and help bilingual people who cannot speak to communicate with others in two languages. This achievement provides new insights into how the human brain processes language, and is expected to bring a long-term device to restore multilingual communication for people who are unable to communicate verbally. Relevant papers were published in Nature Biomedical Engineering on the 20th.
Panqiao, a volunteer who participated in the study, suffered a stroke when he was 20 years old, which caused most of his body to be paralyzed. He could only make groans and grunts, and could not speak clear words and sentences. When Panqiao was in his 30s, Edward Zhang, a neurosurgeon at the University of California, San Francisco, cooperated with him to study the lasting effects of stroke on the brain. In a groundbreaking study published in 2021, Zhang's team surgically implanted electrodes into Panqiao's cerebral cortex to record neural activity and translate them into words on the screen.
The first sentence "My home is outside" that Panqiao "said" was translated in English, but Panqiao's mother tongue is Spanish, and he learned English after a stroke. In order to enable Pan Qiao to communicate bilingually, Zhang's team developed an AI system and trained the system with nearly 200 words in Pan Qiao's mind. Panqiao speaks each word in his brain, forming a unique neural pattern, and records it through electrodes.
Then, when Pancho said the phrase out loud in his head, the research team applied the AI system to the phrase. The system has a Spanish module and an English module. For the first word in the phrase, the Spanish module selects the Spanish word that best matches the neural pattern; The same is true for the English part, just choose from English vocabulary. The results show that these modules can distinguish English and Spanish with 88% accuracy based on the first word, and decode the correct sentence with 75% accuracy.
Sergey Stavisky, a neuroscientist at the University of California, Davis, said that this research is an important contribution to the emerging field of speech restoration neural prosthesis.
