Virtual restoration of the colored paintings on weathered beams in the Forbidden City using multiple deep learning algorithms

The virtual inpainting of artworks provides a nondestructive mode of hypothesis visualization, and it is especially attractive when physical restoration raises too many methodological and ethical concerns. At the same time, in Cultural Heritage applications, the level of details in virtual reconstruction and their accuracy are crucial. We propose an inpainting algorithm that is based on generative adversarial network, with two generators: one for edges and another one for colors. The color generator rebalances chromatically the result by enforcing a loss in the discretized gamut space of the dataset. This way, our method follows the modus operandi of an artist: edges first, then color palette, and, at last, color tones. Moreover, we simulate the stochasticity of the lacunae in artworks with morphological variations of a random walk mask that recreate various degradations, including craquelure. We showcase the performance of our model on a dataset of digital images of wall paintings from the Dunhuang UNESCO heritage site. Our proposals of restored images are visually satisfactory and they are quantitatively comparable to state-of-the-art approaches.

Colour-Balanced Edge-Guided Digital Inpainting: Applications on Artworks.

Restoration of damaged artworks based on a generative adversarial network

Abstract The Northern Wei Dynasty (386–534 CE) murals of Cave 254 in the Mogao Grottoes, China, have been extensively affected by pigment color changes and fading. These issues severely hinder efforts to correctly understand the value and painting process of murals from this historic period. The virtual reconstruction of the faded murals reflects the analysis results of the pigments applied during the painting process and provides a new direction for conservation and art research. However, simple virtual reconstructions may not be accurate owing to deficiencies in our understanding of the color-changed pigments and fading of image lines. In this study, multi-band imaging (MSI) was performed to obtain infrared-reflected false color, ultraviolet-reflected false color, and ultraviolet luminescence images of the mural. A portable X-ray fluorescence spectrometer, portable digital microscope, scanning electron microscope, energy dispersive X-ray spectrometer, liquid chromatography–mass spectrometer, and confocal Raman microscope were used. Non-invasive and minimally invasive analyses of local portions of different color areas were performed to identify their pigments and map the strata, and a hue–saturation–brightness palette was constructed for these pigments. Finally, the pigment analysis results were combined with the multi-band image features to determine the pigment distribution of the mural, which was then used to virtually reconstruct the original color and appearance of the faded mural under ideal conditions. This study is the first to use a virtual reconstruction based on objective analyses to simulate the original color, painting processes, and pigment stratigraphy of a mural from the Northern Wei Dynasty. A preliminary discussion of the relationship between the painting processes of the mural and color changes in its pigments was also performed. The findings of this study will provide new perspectives on the study of Northern Wei Dynasty murals. 

/pdf/virtual-reconstruction-of-the-painting-process-and-original-3moso8js.pdf

Virtual reconstruction of the painting process and original colors of a color-changed Northern Wei Dynasty mural in Cave 254 of the Mogao Grottoes

https://iopscience.iop.org/article/10.1088/1742-6596/1335/1/012024/pdf

Image Stitching and Blending of Dunhuang Murals Based on Image Pyramid

As time goes by, the art pieces inside Dunhuang Grottoes have suffered from tremendous damage such as mural deterioration, and they are usually difficult to be repaired. Although we can achieve digital preservation by modeling the caves and preserving mural as textures in virtual environment, we still cannot have a glimpse of how the grottoes look like without damage. In this work, we propose a systematic restoration framework, which is based on Generative Adversarial Network (GAN) technique, for these high-resolution but deteriorated mural textures. The main idea is to make the machine learn the transformation between deteriorated mural textures and restored mural textures. However, the resolution of training texture images (i.e. 8192×8192) is too high to be applied with GAN technology directly due to GPU RAM limitation. Instead, our method restores a set of high-resolution yet color-inconsistent textures patch-by-patch and a set of low-resolution but color-consistent full textures, and then combines them to get the final high-resolution and color-consistent result.

XinYi Lin

Papers

Dunhuang mural restoration using deep learning