https://doi.org/10.1016/j.cag.2022.03.009

Sympathy for the digital: Influence of synthetic voice on affinity, social presence and empathy for photorealistic virtual humans

When we see our interlocutor, our brain seamlessly extracts visual cues from their face and processes them along with the sound of their voice, making speech an intrinsically multimodal signal. Visual cues are especially important in noisy environments, when the auditory signal is less reliable. Neuronal oscillations might be involved in the cortical processing of audiovisual speech by selecting which sensory channel contributes more to perception. To test this, we designed computer-generated naturalistic audiovisual speech stimuli where one mismatched phoneme-viseme pair in a key word of sentences created bistable perception. Neurophysiological recordings (high-density scalp and intracranial electroencephalography) revealed that the precise phase angle of theta-band oscillations in posterior temporal and occipital cortex of the right hemisphere was crucial to select whether the auditory or the visual speech cue drove perception. We demonstrate that the phase of cortical oscillations acts as an instrument for sensory selection in audiovisual speech processing.

/pdf/the-phase-of-cortical-oscillations-determines-the-perceptual-n9zoj14cac.pdf

The phase of cortical oscillations determines the perceptual fate of visual cues in naturalistic audiovisual speech

Humans communicate with each other in a multimodal way. Even with several technologies mediating remote communication, face-to-face contact is still our main and most natural way to exchange information. Despite continuous advances in interaction modalities, such as speech interaction, much can be done to improve its naturalness and efficiency, particularly by considering the visual cues transmitted by facial expressions through audiovisual speech synthesis (AVS). To this effect, several approaches have been proposed, in the literature, mostly based in data-driven methods. These, while presenting very good results, rely on models that work as black boxes without a direct relation with the actual process of producing speech and, hence, do not contribute much to our understanding of the underpinnings of the synergies between the audio and visual outputs. In this context, the authors proposed a first proof of concept for an articulatory-based approach to AVS, supported on the articulatory phonology framework, and argued that this research needs to be challenged and informed by fast methods to translate it to interactive applications. In this article, we describe further evolutions of the pronunciation module of the AVS core system along with the proposal of a set of interaction modalities to enable its integration in applications to enable a faster translation into real scenarios. The proposed modalities are designed in line with the W3C recommendations for multimodal interaction architectures making it easy to integrate with any applications that consider it.

Designing and Deploying an Interaction Modality for Articulatory-Based Audiovisual Speech Synthesis

Since the concept of artificial intelligence was introduced in 1956, AI technology has been gradually applied in various fields, including journalism. This paper focuses on research related to AI news anchors, and two correlated experiments are applied to examine audiences’ perceived attractiveness of AI news anchors from a psychological dimension. Study 1 focuses on the different variables that influence the behavioral willingness of AI news anchor viewers, while Study 2 focuses on the mediating and moderating variables that influence audiences’ psychological changes. The results indicate that non-humanoid female AI news anchors who use anthropomorphic voices to broadcast news obtain the highest perceived attractiveness among audiences. Additionally, the mediating effect of perceived attractiveness and the negative moderating effect on the inherent impression of traditional news anchors are both verified in the study. Based on the research findings, the implications and suggestions are addressed accordingly.

/pdf/what-do-you-think-of-ai-research-on-the-influence-of-ai-news-29fapkps.pdf

What Do You Think of AI? Research on the Influence of AI News Anchor Image on Watching Intention

Should We Believe Our Eyes or Our Ears? Processing Incongruent Audiovisual Stimuli by Russian Listeners

Natural speech is processed in the brain as a mixture of auditory and visual features. An example of the importance of visual speech is the McGurk effect and related perceptual illusions that result from mismatching auditory and visual syllables. Although the McGurk effect has widely been applied to the exploration of audio-visual speech processing, it relies on isolated syllables, which severely limits the conclusions that can be drawn from the paradigm. In addition, the extreme variability and the quality of the stimuli usually employed prevents comparability across studies. To overcome these limitations, we present an innovative methodology using 3D virtual characters with realistic lip movements synchronized on computer-synthesized speech. We used commercially accessible and affordable tools to facilitate reproducibility and comparability, and the set-up was validated on 24 participants performing a perception task. Within complete and meaningful French sentences, we paired a labiodental fricative viseme (i.e. /v/) with a bilabial occlusive phoneme (i.e. /b/). This audiovisual mismatch is known to induce the illusion of hearing /v/ in a proportion of trials. We tested the rate of the illusion while varying the magnitude of background noise and audiovisual lag. Overall, the effect was observed in 40% of trials. The proportion rose to about 50% with added background noise and up to 66% when controlling for phonetic features. Our results conclusively demonstrate that computer-generated speech stimuli are judicious, and that they can supplement natural speech with higher control over stimulus timing and content.

/pdf/animated-virtual-characters-to-explore-audio-visual-speech-p6urvp3knv.pdf

Antoine Provost

Papers

Animated virtual characters to explore audio-visual speech in controlled and naturalistic environments.