Portion of a computer screen with an icon image

A system and method are disclosed for facilitating speech recognition through the processing of visual speech cues. These speech cues may include the position of the lips, tongue and/or teeth during speech. In one embodiment, upon capture of a frame of data by an image capture device, the system identifies a speaker and a location of the speaker. The system then focuses in on the speaker to get a clear image of the speaker's mouth. The system includes a visual speech cues engine which operates to recognize and distinguish sounds based on the captured position of the speaker's lips, tongue and/or teeth. The visual speech cues data may be synchronized with the audio data to ensure the visual speech cues engine is processing image data which corresponds to the correct audio data.

Rgb/depth camera for improving speech recognition

For generating a live video signal (51), a voice signal (41) is received from a remote viewer (2) via a telecommunications network (4). Determined from the voice signal (41) is corresponding lip movement data. Furthermore, in response to the voice signal (41), body motion data is generated. The body motion data is generated based on animation data representative of body motions performed by a human actor (7) in response to the voice signal (41) and captured by a motion capturing system. An animated graphical avatar (97) is generated in real-time and assigned to the remote viewer (2), lip movements of the avatar (97) being controlled by the lip movement data, and body motions of the avatar (97) being controlled by the body motion data. A live video signal (51) is generated for broadcasting to remote display devices (3), e.g. a television set, the live video signal (51) including an animated video scene (90) with at least the animated graphical avatar (97) and an audio component comprising the voice signal (41). The remote viewer (2) can be integrated dynamically and effectively in a live television broadcast as an animated avatar (97), moving lips and body in synch and in accordance with his voice.

System and method for generating a video signal

Display screen with emoticon

A computer numerical processing method for representing audio information for use in conjunction with human hearing is described. The method comprises approximating an eigenfunction equation representing a model of human hearing, calculating the approximation to each of a plurality of eigenfunctions from at least one aspect of the eigenfunction equation, and storing the approximation to each of a plurality of eigenfunctions for use at a later time. The approximation to each of a plurality of eigenfunctions represents audio information. The model of human hearing includes a bandpass operation with a bandwidth having the frequency range of human hearing and a time-limiting operation approximating the time duration correlation window of human hearing.

Auditory eigenfunction systems and methods

A technique for extracting visemes includes receiving successive frames of digitized analog speech information obtained from the speech signal at a fixed rate (210), filtering each of the successive frames of digitized analog speech information to synchronously generate time domain frame classification vectors at the fixed rate (215, 220, 225, 230, 235, 240), and analyzing each of the time domain classification vectors (250) to synchronously generate a set of visemes corresponding to each of the successive frames of digitized speech information at the fixed rate. Each of the time domain frame classification vectors is derived from one of the successive frames of digitized analog speech information. N multi-taper discrete prolate spheroid sequence basis (MTDPSSB) functions (220) that are factors of a Fredholm integral of the first kind may be used for the filtering, and the analyzing may use a spatial classification function (250). The latency is less than 100 milliseconds.

Eric R. Buhrke

Papers

Speech receiving device and viseme extraction method and apparatus