Part III: Approaching the goal of “human-computer integration”

Zhejiang University has always been in the vanguard of research into the brain-computer interface thanks to multi-disciplinary cooperation in neuroscience, information engineering technology and medicine.

Researchers at Zhejiang University took the lead in applying the brain-computer interface to animal models in China, including the brain-computer interface commanding an external manipulator to complete different gestures such as “hooking, grabbing, pinching and holding” and the brain mind controlling a rat to walk through the maze.

In 2014, researchers used the human mind to command a manipulator to make such complex hand movements as “rock-paper-scissors”. In 2016, the “brain-computer interface” research team won the first prize of the Wu Wenjun AI Science & Technology Award”.

In this study, the research team has made a series of new breakthroughs by implanting electrodes in the brain of an elderly paraplegic patient and realizing the three-dimensional movement of a robotic arm and a manipulator via mind control.

The challenge began with how to accurately implant electrodes into a patient’s brain with minimal damage. Neurons in the cerebral cortex have 6 layers, and electrodes should be implanted into the 5th layer. “This is a really formidable task. If the implantation position is too shallow, it won’t achieve any effect. If it is too deep, it will damage other nerves. This is a completely brand-new operation for us,” said ZHANG Jianmin. The research team used a surgical robot to accurately feed the two electrodes to the given positions with an error of less than 0.5 mm.

The next key question was how to realize “mind control” successfully. In previous studies, subjects in experiments were relatively young and middle-aged adults. Their EEG signaling quality was far superior to that of Mr. ZHANG, thus rendering easier analysis. “Initially, we attempted to duplicate several sets of linear algorithms from abroad, but the results were far from satisfactory,” said WANG Yueming.

“The grabbing of the coke bottle by the robotic arm is actually the outcome of a series of smooth signals. Mr. ZHANG is too old to concentrate for a long time and signals are relatively unstable,” said WANG Yueming, “A minor misinterpretation of a subtle movement may lead to the failure in grabbing the coke bottle. We encountered colossal difficulties in the very beginning.” Later, researchers introduced the non-linear neural network algorithm and developed a solution for elderly patients.

The research team adopted a step-by-step training method. Mr. ZHANG was first asked to control the mouse on the computer screen to track and click the ball in motion. Then, he practiced commanding the robotic arm to complete the movements in 9 directions. Finally, he simulated the movements of shaking hands, drinking water and eating. In this way, they approached the goal of “human-computer integration”.