Reviving degraded colors: AI-powered restoration of traditional Chinese paintings

Ancient paintings shouldn't always appear dull and gray.

In this digital age, we are faced with more and more challenges in preserving cultural heritage, and restoring colors in ancient paintings is one of them.

At the exhibition on A Comprehensive Collection of Ancient Chinese Paintings unveiled at the National Museum of China last year, TANG Tan, a researcher at the Zhejiang University School of Art and Archaeology, encountered a cute little girl. The girl asked her mother, “Mom, why do the paintings here look so old? Were they originally this color?”

This made TANG Tan realize that even with state-of-the-art preservation techniques, degraded colors cannot survive during the long history. This challenge motivates him to propose novel artificial intelligence (AI) approaches to recover the “lost beauty.”

“Instead of physically restoring ancient paintings, our basic idea is to propose a collaborative design framework that leverages the computational power of AI models,” said TANG Tan. “Our project is called ‘Visualizing and Re-coloring Ancient Chinese Paintings with Congruent Poem-Painting Context .’ We aim to explore the fusion of art and technology, thereby breathing new life into traditional Chinese paintings using AI models.”

On TANG Tan’s personal homepage, there is a streak of simple resume. He studied mathematics and applied mathematics for his undergraduate degree and completed his Ph.D. in computer science at Zhejiang University. This year, he joined the School of Art and Archaeology as a ZJU 100 Young Professor.

Why a computer scientist ventured into the field of art?

TANG Tan is keen on calligraphy and painting. He learnt sketching when he was young. After becoming a faculty member in Zhejiang University, he has been heavily involved in the project of "A Comprehensive Collection of Ancient Chinese Paintings". This massive project was officially approved in 2005 by XI Jinping,then Party secretary of Zhejiang Province, and is compiled and published by Zhejiang University and the Zhejiang Provincial Cultural Relics Bureau. It is a large-scale cultural project that spans time and space.

TANG Tan is introducing the project of "A Comprehensive Collection of Ancient Chinese Paintings".

In May 2021, the “Classics in the Flourishing Age: A Comprehensive Collection of Ancient Chinese Paintings” exhibition was staged nationwide. The fusion of traditional ink paintings and modern digital technology was a showcase for not only the Chinese characteristics, charm, and style of Chinese paintings but also the profoundness of historical paintings and the broad prospects of cultural preservation, inheritance, and utilization. Waves of audiences feel the enduring Chinese cultural heritage intertwined in the river of history and the rhythm of modernity.

The exhibition did not settle for the mere presentation of high-definition printed images; rather, it called for a seamless integration of art and technology to bring ancient paintings to life and offer audiences more immersive experiences. Impressed by this exhibition, TANG Tan joined the curation team and brought into full play his expertise in exploring the possibilities of artistic design with the help of AI. “I was wondering whether it’s possible to bring back the degraded colors of ancient paintings in our exhibition.”

The congruent poetry-painting context for color restoration

At the dawn of this year, this young man, with his unique “literary and artistic” style, embarked on his exploration.

“To complete this task, we need a power—one that can transcend time and space, retain authenticity, and provide a basis for color restoration. But where can we find this basis?” TANG Tan found the answer at this year’s 2nd National Conference on Reading.

At one exhibition at Zhejiang Exhibition Hall, a young boy recited a poem inscribed on WU Changshuo’s painting Autumn Chrysanthemum. His melodious recitation made TANG Tan realize that words possess the power to transcend time and space. “Paintings may degrade, but poems won’t, as poems can convey sentiments through words, create empathy, and build a bridge between the past and the present in the river of history.”

TANG Tan (left) guides students in adopting AI tools to restore ancient paintings.

This power opened the door to discovering the colors of ancient paintings.

Poetry is the art of language, and painting is a form of visual art. The relationship between poetry and painting has sparked intense debates in the West. However, SU Shi, a prominent ancient Chinese poet, holds that as two different art forms, their “oneness” is inevitably a “profound” connection. He stresses that painting and poetry bear close resemblances in creation, appreciation, purpose and aesthetic beauty. To portray this profound connection, TANG Tan and his team introduced a cross-modal deep learning model to learn the mapping relationship between poetry and painting.

TANG Tan used DONG Yuan’s “Residents on the Outskirts of Dragon Abode” to explain the “wisdom” of the AI model in a popular way.

This silk painting, created during the Five Dynasties (AD 907-979), is a representative artwork of blue-green landscape painting. However, the image of the existing genuine work has become somewhat dim with the passage of time.

“How can we retrieve the colors of the mountains? Our AI model emphasizes the ‘harmony of poetry and painting’. If we rely solely on paintings to infer other paintings, we will obtain limited information. Therefore, we also need to find a poem,” said TANG Tan. There is a long postscript to this awe-inspiring masterpiece by Emperor Qianlong. Using this as the core, AI5 first composed a seven-character poem. When TANG Tan clicked the painting, hundreds of works in different dynasties would be displayed on the left side of the page. However, when he selected the first two lines depicting the scenery of the mountains, several works and a heat map would appear on the right side of the page. “These works are what AI considers closest to the quintessence of the painting and the poem. For example, ‘View of the Riverside at Dusk’ is also one of the representative works of blue-green landscape painting,” he explained. “In general, the works selected by AI are intimately related to the original painting. They could be works by the disciples of the original painter and even the same painter, or works from the same period or of similar style.

When the restoration area was selected, the mountain top in “Residents on the Outskirts of Dragon Abode” would be rendered a highly atmospheric blue-green color, taking only a couple of seconds to a dozen seconds.

AI in ancient painting restoration: challenges and prospects

So far, the AI model developed by TANG Tan and his team has been able to explore the colors of ancient paintings efficiently. But can this technology truly be applied to the restoration process of ancient paintings?

“Sure!” TANG Tan responded confidently. Although there is still controversy revolving around the concept of using AI to restore ancient paintings, most people agree that the AI tool is instrumental to saving time and improving efficiency.

The conventional process of restoring ancient paintings involves several steps, including washing, uncovering, patching, and completion. This task demands that restoration experts be equipped with profound literary knowledge and professional artistic skills. It often takes several months or even years to restore a single painting. For instance, it took nine months for the artist to restore and duplicate The Painting of Bodhisattvas Burning Incense as well as a significant amount of time on literature research.

TANG Tan is testing the digital exhibition equipment.

“Digitalization provides more favorable restoration conditions. Thanks to our large model, the data collection process, which used to take at least one week, can now be shortened to a day or two, thereby considerably reducing trial and error costs,” said TANG Tan. “Modern artists can use various digital design tools to duplicate ancient paintings. This is also helpful for cultivating future restoration experts.”

“We restore ancient paintings in the way that incorporates human creativity, making them more humanistic rather than cold mathematical symbols.”

TANG Tan’s journey of exploration has not yet stop.

“Painting accentuates color and imagery context, and imagery context is the most elusive. We are also questioning the lack of poetic text. How do strike a balance when using AI to restore ancient paintings?” In TANG Tan’s view, imagery context is the mental image that a person forms when reading a poem. TANG Tan and his team have long been committed to investigate the congruence between poems and paintings. “Restoring traditional painting colors is not just a technical task but also a responsibility for preserving cultural heritage. Our vision is to integrate the restoration of traditional painting colors with advanced technology to protect and preserve cultural heritage.”

Translated from the article written by LU Yao, YAO Yingkang, KE Yineng in the CHAO News (潮新闻)

Photo credit: YAO Yingkang from the CHAO News (潮新闻), TANG Tan