Talking face generation aims at generating photo-realistic video portraits of a target person driven by input audio. Due to its nature of one-to-many mapping from the input audio to the output video (e.g., one speech content may have multiple feasible visual appearances), learning a deterministic mapping like previous works brings ambiguity during training, and thus causes inferior visual results. Although this one-to-many mapping could be alleviated in part by a two-stage framework (i.e., an audio-to-expression model followed by a neural-rendering model), it is still insufficient since the prediction is produced without enough information (e.g., emotions, wrinkles, etc.). In this paper, we propose MemFace to complement the missing information with an implicit memory and an explicit memory that follow the sense of the two stages respectively. More specifically, the implicit memory is employed in the audio-to-expression model to capture high-level semantics in the audio-expression shared space, while the explicit memory is employed in the neural-rendering model to help synthesize pixel-level details. Our experimental results show that our proposed MemFace surpasses all the state-of-the-art results across multiple scenarios consistently and significantly.

Memories are One-to-Many Mapping Alleviators in Talking Face Generation

Machine learning methods for conditional data generation usually build a mapping from source conditional data X to target data Y. The target Y (e.g., text, speech, music, image, video) is usually high-dimensional and complex, and contains information that does not exist in source data, which hinders effective and efficient learning on the source-target mapping. In this paper, we present a learning paradigm called regeneration learning for data generation, which first generates Y' (an abstraction/representation of Y) from X and then generates Y from Y'. During training, Y' is obtained from Y through either handcrafted rules or self-supervised learning and is used to learn X-->Y' and Y'-->Y. Regeneration learning extends the concept of representation learning to data generation tasks, and can be regarded as a counterpart of traditional representation learning, since 1) regeneration learning handles the abstraction (Y') of the target data Y for data generation while traditional representation learning handles the abstraction (X') of source data X for data understanding; 2) both the processes of Y'-->Y in regeneration learning and X-->X' in representation learning can be learned in a self-supervised way (e.g., pre-training); 3) both the mappings from X to Y' in regeneration learning and from X' to Y in representation learning are simpler than the direct mapping from X to Y. We show that regeneration learning can be a widely-used paradigm for data generation (e.g., text generation, speech recognition, speech synthesis, music composition, image generation, and video generation) and can provide valuable insights into developing data generation methods.

Regeneration Learning: A Learning Paradigm for Data Generation

Generating talking person portraits with arbitrary speech audio is a crucial problem in the field of digital human and metaverse. A modern talking face generation method is expected to achieve the goals of generalized audio-lip synchronization, good video quality, and high system efficiency. Recently, neural radiance field (NeRF) has become a popular rendering technique in this field since it could achieve high-fidelity and 3D-consistent talking face generation with a few-minute-long training video. However, there still exist several challenges for NeRF-based methods: 1) as for the lip synchronization, it is hard to generate a long facial motion sequence of high temporal consistency and audio-lip accuracy; 2) as for the video quality, due to the limited data used to train the renderer, it is vulnerable to out-of-domain input condition and produce bad rendering results occasionally; 3) as for the system efficiency, the slow training and inference speed of the vanilla NeRF severely obstruct its usage in real-world applications. In this paper, we propose GeneFace++ to handle these challenges by 1) utilizing the pitch contour as an auxiliary feature and introducing a temporal loss in the facial motion prediction process; 2) proposing a landmark locally linear embedding method to regulate the outliers in the predicted motion sequence to avoid robustness issues; 3) designing a computationally efficient NeRF-based motion-to-video renderer to achieves fast training and real-time inference. With these settings, GeneFace++ becomes the first NeRF-based method that achieves stable and real-time talking face generation with generalized audio-lip synchronization. Extensive experiments show that our method outperforms state-of-the-art baselines in terms of subjective and objective evaluation. Video samples are available at https://genefaceplusplus.github.io .

GeneFace++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generation

Dental CBCT and panoramic images are important imaging modalities used in dental diagnosis and treatment planning. In order to acquire a panoramic image without an additional panoramic scan, in this study, we proposed a method of reconstructing a panoramic image by extracting panoramic projection data from dental CBCT projection data. After specifying the patient's dental arch from the patient's CBCT image, panoramic projection data are extracted from the CBCT projection data along the appropriate panoramic scan trajectory that fits the dental arch. A total of 40 clinical human datasets and one head phantom dataset were used to test the proposed method. The clinical human dataset used in this study includes cases in which it is difficult to reconstruct panoramic images from CBCT images, such as data with severe metal artifacts or data without teeth. As a result of applying the panoramic image reconstruction method proposed in this study, we were able to successfully acquire panoramic images from the CBCT projection data of various patients. The proposed method acquires a universally applicable panoramic image that is less affected by CBCT image quality and metal artifacts by extracting panoramic projection data from dental CBCT data and reconstructing a panoramic image. 

/pdf/panoramic-dental-tomosynthesis-imaging-by-use-of-cbct-8l968oxa.pdf

Panoramic dental tomosynthesis imaging by use of CBCT projection data

We present READ Avatars, a 3D-based approach for generating 2D avatars that are driven by audio input with direct and granular control over the emotion. Previous methods are unable to achieve realistic animation due to the many-to-many nature of audio to expression mappings. We alleviate this issue by introducing an adversarial loss in the audio-to-expression generation process. This removes the smoothing effect of regression-based models and helps to improve the realism and expressiveness of the generated avatars. We note furthermore, that audio should be directly utilized when generating mouth interiors and that other 3D-based methods do not attempt this. We address this with audio-conditioned neural textures, which are resolution-independent. To evaluate the performance of our method, we perform quantitative and qualitative experiments, including a user study. We also propose a new metric for comparing how well an actor's emotion is reconstructed in the generated avatar. Our results show that our approach outperforms state of the art audio-driven avatar generation methods across several metrics. A demo video can be found at \url{https://youtu.be/QSyMl3vV0pA}

READ Avatars: Realistic Emotion-controllable Audio Driven Avatars

Panoramic images have been widely used in the diagnosis of dental diseases. In the process of panoramic image reconstruction, the position of the dental arch curve usually affects the quality of display content, especially the completion level of the panoramic image. In addition, the metal implants in the patient’s mouth often lead the contrast of the panoramic image to decrease. This paper describes a method to automatically synthesize panoramic images from dental cone beam computed tomography (CBCT) data. The proposed method has two essential features: the first feature is that the method can detect the dental arch curve through axial maximum intensity projection images over different ranges, and the second feature is that our method is able to adjust the intensity distribution of the implant in critical areas, to reduce the impact of the implant on the contrast of the panoramic image. The proposed method was tested on 50 CBCT datasets; the panoramic images generated by this method were compared with images attained from three other commonly used approaches and then subjectively scored by three experienced dentists. In the comprehensive image contrast score, the method in this paper has the highest score of 11.16 ± 2.64 points. The results show that the panoramic images generated by this method have better image contrast.

/pdf/a-fast-automatic-reconstruction-method-for-panoramic-images-4qgoj9wx.pdf

A Fast Automatic Reconstruction Method for Panoramic Images Based on Cone Beam Computed Tomography

While previous speech-driven talking face generation methods have made significant progress in improving the visual quality and lip-sync quality of the synthesized videos, they pay less attention to lip motion jitters which greatly undermine the realness of talking face videos. What causes motion jitters, and how to mitigate the problem? In this paper, we conduct systematic analyses on the motion jittering problem based on a state-of-the-art pipeline that uses 3D face representations to bridge the input audio and output video, and improve the motion stability with a series of effective designs. We find that several issues can lead to jitters in synthesized talking face video: 1) jitters from the input 3D face representations; 2) training-inference mismatch; 3) lack of dependency modeling among video frames. Accordingly, we propose three effective solutions to address this issue: 1) we propose a gaussian-based adaptive smoothing module to smooth the 3D face representations to eliminate jitters in the input; 2) we add augmented erosions on the input data of the neural renderer in training to simulate the distortion in inference to reduce mismatch; 3) we develop an audio-fused transformer generator to model dependency among video frames. Besides, considering there is no off-the-shelf metric for measuring motion jitters in talking face video, we devise an objective metric (Motion Stability Index, MSI), to quantitatively measure the motion jitters by calculating the reciprocal of variance acceleration. Extensive experimental results show the superiority of our method on motion-stable face video generation, with better quality than previous systems.

StableFace: Analyzing and Improving Motion Stability for Talking Face Generation

During the automatic reconstruction of panoramic images, the effect of dental arch curve fitting will affect the integrity of the content of the panoramic image. Metal implants in the patient's mouth usually lead to a decrease in the contrast of the panoramic image, which affects the doctor's diagnosis. In this paper, an automatic oral panoramic image reconstruction method was proposed. By calculating key image areas and image extraction fusion algorithms, the dental arch curve could be automatically detected and adjusted on a small number of images, and the intensity distribution of teeth, bone tissue and metal implants on the image could be adjusted to reduce the impact of metal on other tissues, to generate high-quality panoramic images. The method was tested on 50 cases of cone beam computed tomography (CBCT) data with good results, which can effectively improve the quality of panoramic images.

Liang Song

Papers

Memories are One-to-Many Mapping Alleviators in Talking Face Generation

A Fast Automatic Reconstruction Method for Panoramic Images Based on Cone Beam Computed Tomography

StableFace: Analyzing and Improving Motion Stability for Talking Face Generation

[Research on panoramic image reconstruction based on oral cone beam computed tomography].