Breakthrough AI Translates Brainwaves into Speech for Paralysed Man

A guy with paralysis may now have discussions in real time without the typical voice lag thanks to a brain-computer interface.

Using a brain-controlled synthetic voice, a guy who lost his ability to talk can now converse in real time and even sing.

Through electrodes inserted into the man’s brain, the brain-computer interface detects his neural activity and instantly produces speech sounds that match his desired emphasis, intonation, and pitch.

According to Sergey Stavisky of the University of California, Davis, “this is kind of the first of its kind for instantaneous voice synthesis – within 25 milliseconds.”

Maitreyee Wairagkar, who is also at UC Davis, adds that in order to make the speech easier to understand, the technology needs to be enhanced. According to Wairagkar, the man, who has amyotrophic lateral sclerosis and lost the capacity to speak, nevertheless claims that it makes him “happy” and that it sounds like his own voice.

There are already brain-computer interface-based speech neuroprostheses available, but they typically take a few seconds to translate brain activity into sounds. According to Stavisky, this hinders spontaneous conversation because people are unable to interrupt, clarify, or reply in real time. “It’s like having a phone conversation with a bad connection.”

Wairagkar, Stavisky, and his associates inserted 256 electrodes into the areas of the man’s brain that aid in controlling the speaking muscles in his face in order to more realistically synthesise speech. Subsequently, during several sessions, the researchers recorded his brain activity while displaying thousands of sentences on a screen and asking him to try pronouncing them out loud, occasionally with particular intonations.

“For instance, you could say, ‘How are you doing today?’ or ‘How are you doing today?’ and that would change the sentence’s semantics,” explains Stavisky. “Compared to earlier systems, that makes for a much richer, more natural exchange—and a big step forward.”

An artificial intelligence model was then trained to link particular patterns of brain activity to the phrases and inflections the individual was attempting to convey using the data that the team had provided. After that, the system used the brain impulses to make speech, creating a voice that reflected his intended words as well as his preferred delivery.

Using voice-cloning technologies to make the artificial speech seem like the man’s own, the researchers even trained the AI on audio recordings from before the man’s illness worsened.

He was asked to attempt singing simple tunes with various pitches as part of another experiment. Real-time decoding of his desired pitch allowed their model to modify the singing voice it generated.

According to Wairagkar, he also used the system to talk without being asked and to make sounds like “hmm,” “eww,” or invented words.

Teammate David Brandman, who is also a UC Davis student, describes him as “a very articulate and intelligent man.” “From being immobile and unable to communicate, he is now able to work full-time and engage in meaningful conversations.”