http://www.ece.uvic.ca/~bctill/papers/facerec/HaxHofGob2000.pdf

The distributed human neural system for face perception.

From the Publisher:
This book takes an empirical approach to language processing, based on applying statistical and other machine-learning algorithms to large corpora.Methodology boxes are included in each chapter. Each chapter is built around one or more worked examples to demonstrate the main idea of the chapter. Covers the fundamental algorithms of various fields, whether originally proposed for spoken or written language to demonstrate how the same algorithm can be used for speech recognition and word-sense disambiguation. Emphasis on web and other practical applications. Emphasis on scientific evaluation. Useful as a reference for professionals in any of the areas of speech and language processing.

/pdf/speech-and-language-processing-an-introduction-to-natural-atzaecqfem.pdf

Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition

The aim of this paper is to develop a theoretical model and a set of terms for understanding and discussing how we recognize familiar faces, and the relationship between recognition and other aspects of face processing. It is suggested that there are seven distinct types of information that we derive from seen faces; these are labelled pictorial, structural, visually derived semantic, identity-specific semantic, name, expression and facial speech codes. A functional model is proposed in which structural encoding processes provide descriptions suitable for the analysis of facial speech, for analysis of expression and for face recognition units. Recognition of familiar faces involves a match between the products of structural encoding and previously stored structural codes describing the appearance of familiar faces, held in face recognition units. Identity-specific semantic codes are then accessed from person identity nodes, and subsequently name codes are retrieved. It is also proposed that the cognitive system plays an active role in deciding whether or not the initial match is sufficiently close to indicate true recognition or merely a ‘resemblance’; several factors are seen as influencing such decisions.

This functional model is used to draw together data from diverse sources including laboratory experiments, studies of everyday errors, and studies of patients with different types of cerebral injury. It is also used to clarify similarities and differences between processes responsible for object, word and face recognition.

Understanding face recognition

We argue that an understanding of the faculty of language requires substantial interdisciplinary cooperation. We suggest how current developments in linguistics can be profitably wedded to work in evolutionary biology, anthropology, psychology, and neuroscience. We submit that a distinction should be made between the faculty of language in the broad sense (FLB)and in the narrow sense (FLN) . FLB includes a sensory-motor system, a conceptual-intentional system, and the computational mechanisms for recursion, providing the capacity to generate an infinite range of expressions from a finite set of elements. We hypothesize that FLN only includes recursion and is the only uniquely human component of the faculty of language. We further argue that FLN may have evolved for reasons other than language, hence comparative studies might look for evidence of such computations outside of the domain of communication (for example, number, navigation, and social relations).

/pdf/the-faculty-of-language-what-is-it-who-has-it-and-how-did-it-2ls6go98wi.pdf

The faculty of language: what is it, who has it, and how did it evolve?

Deep networks have been successfully applied to unsupervised feature learning for single modalities (e.g., text, images or audio). In this work, we propose a novel application of deep networks to learn features over multiple modalities. We present a series of tasks for multimodal learning and show how to train deep networks that learn features to address these tasks. In particular, we demonstrate cross modality feature learning, where better features for one modality (e.g., video) can be learned if multiple modalities (e.g., audio and video) are present at feature learning time. Furthermore, we show how to learn a shared representation between modalities and evaluate it on a unique task, where the classifier is trained with audio-only data but tested with video-only data and vice-versa. Our models are validated on the CUAVE and AVLetters datasets on audio-visual speech classification, demonstrating best published visual speech classification on AVLetters and effective shared representation learning.

/pdf/multimodal-deep-learning-2qi0e2kef0.pdf

Multimodal Deep Learning

MOST verbal communication occurs in contexts where the listener can see the speaker as well as hear him. However, speech perception is normally regarded as a purely auditory process. The study reported here demonstrates a previously unrecognised influence of vision upon speech perception. It stems from an observation that, on being shown a film of a young woman's talking head, in which repeated utterances of the syllable [ba] had been dubbed on to lip movements for [ga], normal adults reported hearing [da]. With the reverse dubbing process, a majority reported hearing [bagba] or [gaba]. When these subjects listened to the soundtrack from the film, without visual input, or when they watched untreated film, they reported the syllables accurately as repetitions of [ba] or [ga]. Subsequent replications confirm the reliability of these findings; they have important implications for the understanding of speech perception.

Hearing lips and seeing voices

An experiment is reported, the results of which confirm and extend an earlier observation that visual information for the speaker’s lip movements profoundly modifies the auditorv perception of natural speech by normally hearing subjects. The effect is most pronounced when there is auditory information for a bilabial utterance combined with visual information for a nonlabial utterance. However, the effect is also obtained with the reverse combination, although to a lesser extent. These findings are considered for their relevance to auditory theories of speech perception.

/pdf/visual-influences-on-speech-perception-processes-5ga40pws31.pdf

Visual influences on speech perception processes.

Two competing hypotheses concerning the nature of inter-modal development are outlined. The first sees the development proceeding from a state of discrete, independent sensory systems towards integration and synthesis between modalities. This is contrasted with a second viewpoint in which the early responsiveness of the organism is seen as a-modal and the developmental sequence as one of increasing sensory differentiation. A number of studies which have investigated auditory-visual co-ordination in young human infants, are reviewed. It is concluded that the data support the notion of ontogenetic development being a process of integration between sensory systems that are initially relatively independent.

Auditory-Visual Coordination in the First Year of Life.

A system is described involving an audio-video tape recorder, a control device, and a delay timer that dubs speech sounds onto prerecorded lip movements on videotape. Accuracy of synchrony or measured asynchrony between the sound and lip movements is a few milliseconds.

/pdf/a-simple-procedure-for-accurately-manipulating-face-voice-1edadusykz.pdf

John Macdonald

Papers

Hearing lips and seeing voices

Visual influences on speech perception processes.

Auditory-Visual Coordination in the First Year of Life.

A simple procedure for accurately manipulating face-voice synchrony when dubbing speech onto videotape