Using Best’s (1995) perceptual assimilation model (PAM), we investigated auditory–visual (AV), auditory-only (AO), and visual-only (VO) perception of Thai tones. Mandarin and Cantonese (tone-language) speakers were asked to categorize Thai tones according to their own native tone categories, and Australian English (non-tone-language) speakers to categorize Thai tones into their native intonation categories—for instance, question or statement. As comparisons, Thai participants completed a straightforward identification task, and another Australian English group identified the Thai tones using simple symbols. All of the groups also completed an AX discrimination task. Both the Mandarin and Cantonese groups categorized AO and AV Thai falling tones as their native level tones, and Thai rising tones as their native rising tones, although the Mandarin participants found it easier to categorize Thai level tones than did the Cantonese participants. VO information led to very poor categorization for all groups, and AO and AV information also led to very poor categorizations for the English intonation categorization group. PAM’s predictions regarding tone discriminability based on these category assimilation patterns were borne out for the Mandarin group’s AO and AV discriminations, providing support for the applicability of the PAM to lexical tones. For the Cantonese group, however, PAM was unable to account for one specific discrimination pattern—namely, their relatively good performance on the Thai high–rising contrast in the auditory conditions—and no predictions could be derived for the English groups. A full account of tone assimilation will likely need to incorporate considerations of phonetic, and even acoustic, similarity and overlap between nonnative and native tone categories.

/pdf/perceptual-assimilation-of-lexical-tone-the-roles-of-234ldfktw4.pdf

Perceptual assimilation of lexical tone: The roles of language experience and visual information

Head motion naturally occurs in synchrony with speech and may carry paralinguistic information, such as intention, attitude and emotion, in dialogue communication. With the aim of verifying the relationship between head motion and the dialogue acts carried by speech, analyses were conducted on motion-captured data for several speakers during natural dialogues. The analysis results first confirmed the trends of our previous work, showing that regardless of the speaker, nods frequently occur during speech utterances, not only for expressing dialogue acts such as agreement and affirmation, but also appearing at the last syllable of the phrase, in strong phrase boundaries, especially when the speaker is talking confidently, or expressing interest in the interlocutor's talk. Inter-speaker variability indicated that the frequency of head motion may vary according to the speaker's age or status, while intra-speaker variability indicated that the frequency of head motion also differs depending on the inter-personal relationship with the interlocutor. A simple model for generating nods based on rules inferred from the analysis results was proposed and evaluated in two types of humanoid robots. Subjective scores showed that the proposed model could generate head motions with naturalness comparable to the original motions.

Head motions during dialogue speech and nod timing control in humanoid robots

We propose a model to automatically generate whole body motions accompanying utterances at appropriate times, similar to humans, by using various types of natural-language-analysis information obtained from spoken language. Specifically, we focus on the co-occurrence relationship between various types of natural-language-analysis information such as words included in the spoken language, parts of speech, a thesaurus, word positions, dialogue acts of the spoken language, and human motions. Our model automatically generates nods, head postures, facial expressions, hand gestures, and upper-body posture using such information. We first recorded a two-person dialogue and constructed a multimodal corpus including utterance and whole body motion information. Next, using the constructed corpus, we constructed our model for generating a motion for each phrase unit using machine learning and using words, parts of speech, a thesaurus, word positions, and speech acts of the entire spoken language as inputs. These types of natural-language-analysis information were useful for motion generation. The effectiveness of our model was verified through a subjective experiment using a virtual conversational agent. As a result, the agent's body motions and impressions regarding naturalness of motion, degree of coincidence between utterance and motion, humanness of the agent, and likability of the agent improved with our model.

Generating Body Motions using Spoken Language in Dialogue

With the aim of automatically generating head motions during speech utterances, analyses are conducted for verifying the relations between head motions and linguistic and paralinguistic information carried by speech utterances. Motion captured data are recorded during natural dialogue, and the rotation angles are estimated from the head marker data. Analysis results showed that nods frequently occur during speech utterances, not only for expressing specific dialog acts such as agreement and affirmation, but also as indicative of syntactic or semantic units, appearing at the last syllable of the phrases, in strong phrase boundaries. Analyses are also conducted on the dependence on linguistic, prosodic and voice quality information of other head motions, like shakes and tilts, and discuss about the potentiality for their use in automatic generation of head motions. The paper also proposes a method for controlling the head actuators of an android based on the rotation angles, and evaluates the mapping from the human head motions.

Analysis of head motions and speech, and head motion control in an android

This chapter surveys the role of visual cues in Chinese lexical tone production and perception, addressing the extent to which visual information involves either linguistically relevant cues to signal tonal category distinctions or is attention-grabbing in general. Specifically, the survey summarizes research findings on which visual facial cues are relevant for tone production, whether these cues are adopted in native and non-native audio-visual tone perception, and whether visual hand gestures also affect tone perception. Production findings demonstrate that head, jaw, eyebrow, and lip movements are aligned with specific spatial and temporal pitch movement trajectories of different tones, suggesting linguistically meaningful associations of these visual cues to tone articulation. Perception findings consistently show that specific facial and hand gestures corresponding to pitch movements for individual tones do benefit tone intelligibility, and these benefits can be augmented by linguistic experience. Together, these findings suggest language-specific mechanisms in cross-modal tone production and perception.

Wilbers

Papers

Analysis of head motions and speech, and head motion control in an android