More than music: the "He Xuan" band and Zhejiang University's quest to build next-gen robots

Recently, a band named Hexuan performed a stunning show at Hangzhou West Railway Station, and they’ve already received numerous invitations from other bands. Their repertoire is growing, ranging from "Ode to my homeland" to "The sea gives a laugh", "Blue and White Porcelain", and "Lonely Warrior". But they are no ordinary band—they are a robotic band formed by Professor WEI Song from Zhejiang University.

WEI Song and ZHU Qiuguo, the founder of DEEP Robotics, were undergraduate classmates and are among the earliest pioneers in the field of humanoid robots in China. WEI Song is a researcher at Zhejiang University's Robotics Institute, the deputy director of the Hangzhou Haichuang Humanoid Robot Innovation Center, and holds a PhD in Mechanical and Electronic Engineering from the class of 2013.

Haichuang took a rather romantic approach to its work: playing the piano. But it's not just about playing music—it's about providing robots with an ever-evolving, complex scenario to improve upon.

One of the most impressive results from this team so far is the "Hexuan Band", which includes robots that can play the piano, yangqin (dulcimer), drums, and even wind instruments and stringless guitars that are still in training. Their skills are continually expanding. The piano-playing robots have five fingers on each "hand" and can hit black and white keys with high precision, swiftly switching between them. Their feet also tap the pedals in rhythm. With the addition of the yangqin and drum-playing robots, Hexuan has already debuted in numerous performances across tech exhibitions, communities, and shopping malls. These robots never miss a beat or strike a wrong note—just like seasoned musicians.

Despite this impressive performance, there are still fundamental differences between robots and human musicians. Currently, these robots rely on intricate coordination between dexterous hands and mechanical arms. Humans program the robots, feed them sheet music, and then use algorithms to convert every note into specific finger movements. However, the robots still cannot respond in real-time or improvise. Every performance involves a digital simulation of the robot's actions before taking the stage, and there are constant refinements to improve their play.

Why create piano-playing robots? WEI Song's team has a clear rationale. First, compared to simpler tasks like pouring a glass of water, playing music is a complex scenario that tests the capabilities of both "hands" simultaneously. In contrast, tasks like pouring water mostly require the robot to focus on object detection and visual recognition. Playing the piano, on the other hand, involves understanding timing and anticipating future movements—requiring foresight. Additionally, it's not just about the hands but also the arms working together in harmony, just like humans, to maintain coordinated movement while playing.

Similar to how humans are graded in piano exams, as the difficulty of the music increases, the challenge for the robots also grows. Currently, WEI Song and his team rate the robots based on their playing techniques and the complexity of the music they can perform. These ratings can be quantified through measures such as finger span and arm speed.

As the robots improve in their playing skills, their intelligence and precision also advance. The team envisions a future where tasks like cleaning the house, washing dishes, or folding clothes become easy for robots once their musical capabilities have been perfected. Therefore, before robots enter the home for simpler tasks, they need to be trained in a high-precision, complex environment—piano playing is the perfect training ground for this.

In WEI Song's vision, robotic bands could become creative partners for human musicians. "The hardware needs to be more stable, and the algorithms need to be smarter", he explains. In his ideal collaboration, a human’s mere glance would be enough for the robot to know what to play. If the musical style shifts suddenly, the robot would instantly adapt to the change.

WEI Song's journey with robots began 22 years ago. In 2003, he entered Zhejiang University from Shandong. "As a boy, I loved watching mecha anime and had a deep love for robots", he recalls with a smile. "So, I chose to major in Mechanical and Electronic Engineering". Now, he hopes to seize the opportunity and delve deeply into AI, as he often tells his team members: "If we miss this wave of humanoid robotics, we might have to wait another 20 years. But if we catch it, everything will take off."

Adapted and translated from the article written by Zhejiang University Alumni Association

Translator: MING Xuen Amelia（'26, Journalism）

Editor: HAN Xiao