Researchers at Stanford University have created an artificial neural network that converts what you imagine writing with a pen into text using signals from the brain. This device has achieved a typing speed of 90 characters per minute, which is more than double that of the old record for hands-free typing devices.
Previously, these hands-free devices allowed you to move the cursor with your head or eyes to type out messages using a head or eye tracking system. Movements unrelated to the task would disrupt the process, so participants can’t look up, look around or do anything else while typing. In practice, this limited the usability of these devices for both comfort and practical reasons
To overcome this limitation, the researchers decided to implant two small sensor arrays under the surface of the brain of a 65-year old man who was paralysed after sustaining a spinal cord injury to directly collect signals from neurons. Each sensor was able to detect signals from 100 neurons. The sensors do not target specific neurons. Although thousands of millions of neurons are involved in coordinating hand movements for writing, the signals from these 200 neurons provide sufficient information to build up a reliable neural network for interpreting the signals. As the man imagined writing letters with a pen, signals from the sensors were fed into an artificial neural network.
Training a neural network often requires thousands of pieces of data. This is feasible when large data sets already exist or can be easily generated. In this case, the data would take the form of brain signals while writing a certain letter, so it is not practical to generate a data set that large by asking the man to think about writing thousands of letters. Instead, the researchers took some samples this way and generated more data by adding random noise to these samples, creating a large synthetic data set.
Two tiny arrays of implanted electrodes relayed information from the brain area that controls the hands and arms to an algorithm, which translated it into letters that appeared on a screen. Credit: F. Willett et al./Nature 2021/Erika Woodrum
Since the neural network is trained using data from the man with the sensors implanted in an unpredictable location, it cannot directly translate to another person. Sensors will need to be implanted and the neural net retrained for each individual using the device.
The result is pleasing. The man was able to type at 90 characters per minute using this system, which is close to the average typing speed of 115 characters per minute for people his age on a smartphone. The typed text was 94.1% accurate, which was boosted to 99% when an autocorrect tool was used. Previously, brain-computer interfaces were only able to interpret large movements and struggled to pick up fine movements such as handwriting.
The next step for the team is to build on this work and create a speech decoder for people who are unable to speak but still have the neural pathways for doing so.
Sparkes, Matthew 2021, AI lets man with paralysis type by just thinking about handwriting, 12nd May 2021, https://www.newscientist.com/article/2277272-ai-lets-man-with-paralysis-type-by-just-thinking-about-handwriting/