Echoes of ancient China: ZJU’s team visualizes refrain and music in Shijing

Interested in the beautiful melodies of merry old ballads? Curious about how would a Chinese ballad sound like, or even look like? The team led by Prof. CHEN Xiaojiao and Prof. WANG Xiaosong at Zhejiang University School of Art and Archeology made it possible by reviving the ancient Chinese ballads collection Shijing with the assistance of visualization technology. The project, “Echoes of Ancient China: Visualizing Refrain and Music in the Shijing”, was recently granted the Best Poster Award in the 15th IEEE Pacific Visualization Symposium. The style of ancient Chinese ballads was expressively shown in the conference, charming the audience with distinct oriental beauty.

IEEE PacificVis2022 Best Poster Award 

A ballad collection awaiting resonance

As the first classic collection of ballads in China, Shijing was composed as early as the sixth century B.C., which has been circulated throughout ages for its fine rhythm, poetic expressions and abundant sentiments it invokes. “In the past, by chanting a particular poem, people could express their feelings and emotions. Is there any similar way existing for appreciation?” That is the initial puzzle of the project.

To unravel this problem, the team turns the spotlight on the rhythm data in Shijing. The ballads in Shijing generally deploys a unique sentence pattern called refrain, wherein ballads repeat a line or part of a line, but with an alternation that helps advance the narrative. This pattern constitutes a dynamic tide of sentence, and resonating with it is the tide of listeners’ emotion, thus giving rise to the unique metrical charm of oriental ballads.

“Rhythm data must submit to certain rules, so we want to probe the fantastic database of Shijing, and to explore the visual presentation of text data as well,”introduced Prof. CHEN Xiaojiao, co-leader of this project. By converting the repetitive structure and the rhythm into some visual data, a new look of Shijing is likely to be presented.

Artistic visualization bridging poetry, painting and music

To evoke multiple senses and produce a synesthetic experience, the team proposes three novel visualization methods for refraining in Shijing, namely, visual analysis, music visualization and physical visualization. 

Take musical visualization for example. The team categorizes Chinese characters in Shijing into three groups—normal words, repeated words and replaced words (the latter two are refrain words)—and assigns them respectively white, blue and green for further visual coding. In its presentation, refrains are displayed with the metaphor of green landscapes, as if each poem floats on the rolling waves, with a blue mountain lying between the repeated words and a green one between the replaced words.

At the same time, the poems are accompanied by moving melodies out of three traditional Chinese instruments, Guqin(古琴), Xiao(萧), and Drum(鼓), with each instrument exclusively played on each one of the three categorized characters.

(a) Metaphor of green landscapes and (b)Music Visualization.

Moreover, the seven tones of ancient Chinese music are also applied to correspond to the seven colors extracted from the A Panorama of Rivers and Mountains, a masterpiece of ancient Chinese painting. In this way, poems in Shijing, accompanied by ancient rhythm and the hierarchical colors in green landscape, are performed in a harmonious fusion of poetry, paintings and music, which innovatively provides the audience with a synesthetic experience of arts.

Video clip of the project’s musical visualization

Visualization as impetus animating traditional arts 

The ancient classics of Chinese arts is a marvelous reservoir to be fathomed, and the characteristics of visualization technology fits well to animate traditional arts. For the team, this study on Shijing is proved to be an inspiring start: not only as an innovative achievement in the interdisciplinary study of digital humanities, but also an attempt to fuse arts with visualization technology.

Speaking of the intention of this project, Prof. Chen expressed the team’s aspiration to facilitate the venture of cultural exchange with visualization. “We hope this attempt will serve as a feasible solution for the promotion of poetic rhyme. By visualizing the beautiful rhythm in Shijing, we attempt to arouse a fresh aesthetic experience on the appreciation of ancient classics, which, furthermore, will provide momentum for the promotion of traditional Chinese arts.”

Source: School of Art and Archeology, Zhejiang University

Translator: ZHANG Jinmei

Editor: TIAN Minjie

Photo and video credit: the team led by Prof. CHEN Xiaojiao and Prof. WANG Xiaosong