Speech enhancement and speech separation are two related tasks, whose purpose is to extract either one or more target speech signals, respectively, from a mixture of sounds generated by several sources. Traditionally, these tasks have been tackled using signal processing and machine learning techniques applied to the available acoustic signals. Since the visual aspect of speech is essentially unaffected by the acoustic environment, visual information from the target speakers, such as lip movements and facial expressions, has also been used for speech enhancement and speech separation systems. In order to efficiently fuse acoustic and visual information, researchers have exploited the flexibility of data-driven approaches, specifically deep learning, achieving strong performance. The ceaseless proposal of a large number of techniques to extract features and fuse multimodal information has highlighted the need for an overview that comprehensively describes and discusses audio-visual speech enhancement and separation based on deep learning. In this paper, we provide a systematic survey of this research topic, focusing on the main elements that characterise the systems in the literature: acoustic features; visual features; deep learning methods; fusion techniques; training targets and objective functions. In addition, we review deep-learning-based methods for speech reconstruction from silent videos and audio-visual sound source separation for non-speech signals, since these methods can be more or less directly applied to audio-visual speech enhancement and separation. Finally, we survey commonly employed audio-visual speech datasets, given their central role in the development of data-driven approaches, and evaluation methods, because they are generally used to compare different systems and determine their performance.

An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation

This work presents a scalable solution to open-vocabulary visual speech recognition To achieve this, we constructed the largest existing visual speech recognition dataset, consisting of pairs of text and video clips of faces speaking (3,886 hours of video) In tandem, we designed and trained an integrated lipreading system, consisting of a video processing pipeline that maps raw video to stable videos of lips and sequences of phonemes, a scalable deep neural network that maps the lip videos to sequences of phoneme distributions, and a production-level speech decoder that outputs sequences of words The proposed system achieves a word error rate (WER) of 409% as measured on a held-out set In comparison, professional lipreaders achieve either 864% or 929% WER on the same dataset when having access to additional types of contextual information Our approach significantly improves on other lipreading approaches, including variants of LipNet and of Watch, Attend, and Spell (WAS), which are only capable of 898% and 768% WER respectively

Large-Scale Visual Speech Recognition

/pdf/large-scale-visual-speech-recognition-3is4d0a8b8.pdf

Humans involuntarily tend to infer parts of the conversation from lip movements when the speech is absent or corrupted by external noise. In this work, we explore the task of lip to speech synthesis, i.e., learning to generate natural speech given only the lip movements of a speaker. Acknowledging the importance of contextual and speaker-specific cues for accurate lip-reading, we take a different path from existing works. We focus on learning accurate lip sequences to speech mappings for individual speakers in unconstrained, large vocabulary settings. To this end, we collect and release a large-scale benchmark dataset, the first of its kind, specifically to train and evaluate the single-speaker lip to speech task in natural settings. We propose a novel approach with key design choices to achieve accurate, natural lip to speech synthesis in such unconstrained scenarios for the first time. Extensive evaluation using quantitative, qualitative metrics and human evaluation shows that our method is four times more intelligible than previous works in this space.

/pdf/learning-individual-speaking-styles-for-accurate-lip-to-4cze26gl0r.pdf

Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis

Humans involuntarily tend to infer parts of the conversation from lip movements when the speech is absent or corrupted by external noise. In this work, we explore the task of lip to speech synthesis, i.e., learning to generate natural speech given only the lip movements of a speaker. Acknowledging the importance of contextual and speaker-specific cues for accurate lip-reading, we take a different path from existing works. We focus on learning accurate lip sequences to speech mappings for individual speakers in unconstrained, large vocabulary settings. To this end, we collect and release a large-scale benchmark dataset, the first of its kind, specifically to train and evaluate the single-speaker lip to speech task in natural settings. We propose a novel approach with key design choices to achieve accurate, natural lip to speech synthesis in such unconstrained scenarios for the first time. Extensive evaluation using quantitative, qualitative metrics and human evaluation shows that our method is four times more intelligible than previous works in this space. Please check out our demo video for a quick overview of the paper, method, and qualitative results. this https URL

In this study, we propose a deep neural network for reconstructing intelligible speech from silent lip movement videos. We use auditory spectrogram as spectral representation of speech and its corresponding sound generation method resulting in a more natural sounding reconstructed speech. Our proposed network consists of an autoencoder to extract bottleneck features from the auditory spectrogram which is then used as target to our main lip reading network comprising of CNN, LSTM and fully connected layers. Our experiments show that the autoencoder is able to reconstruct the original auditory spectrogram with a 98% correlation and also improves the quality of reconstructed speech from the main lip reading network. Our model, trained jointly on different speakers is able to extract individual speaker characteristics and gives promising results of reconstructing intelligible speech with superior word recognition accuracy.

Lip2AudSpec: Speech reconstruction from silent lip movements video

Lip2Audspec: Speech Reconstruction from Silent Lip Movements Video

Gesture recognition helps in the development of a more natural and intuitive human computer interaction. It has several applications in virtual reality and can be used to control robots as well as home appliances. In this paper, the design and working of a compact handheld device that works with a computer to recognize hand gestures has been presented. A single 3-D accelerometer has been used for sensing the motion and Support Vector Machine has been employed for recognizing the gesture. The data processing has been implemented in MATLAB and a graphical user interface has also been developed to make the application user friendly. All digits from 0-9 have been recognized with a high accuracy for both user dependent and independent gesture recognition. In contrast to the previous models for digit recognition, a simpler approach that uses frame based temporal features has been presented to give a high recognition rate.. General Terms Gesture recognition, Accelerometer, Support vector machine.

/pdf/3d-accelerometer-based-gesture-device-for-the-recognition-of-fgtcsekv8u.pdf

3D Accelerometer based Gesture Device for the Recognition of Digits

Finding the camera pose is an important step in many egocentric video applications. It has been widely reported that, state of the art SIAM algorithms fail on egocentric videos [1, 2, 3, 4]. In this paper, we propose a robust method for camera pose estimation, designed specifically for egocentric videos. In an egocentric video, the camera views the same scene point multiple times as the wearer's head sweeps back andforth. We use this specific motion profile to perform. Short loop closures aligned with wearer'sfootsteps. For egocentric videos, depth estimation is usually noisy. In an important departure, we use 2D computations for rotation averaging which do not rely upon depth estimates. The two modification results in much more stable algorithm as is evidentfrom our experiments on various egocentric video datasets for different egocentric applications. The proposed algorithm resolves a long standing problem in egocentric vision and unlocks new usage scenarios for future applications.

Computing Egomotion with Local Loop Closures for Egocentric Videos

stents are becoming widely used as percutaneous interventions of pulmonary artery are progressively replacing conventional open heart surgeries. Presently, X-ray images of pulmonary artery i.e. angiograms are visually inspected by physician to determine the proper stent size. The size estimation of stent done preoperative which has not been assessed so far, is studied in this paper. A reliable method to determine the size of stent using a deformation model before performing angioplasty is proposed. A cylindrical affine transformation model is used to predict the expected deformation of the stent to fit it as closely as possible into artery. The method was evaluated on artery data set of 5 cases in which the patient was suffering from stenosis of pulmonary artery to obtain satisfactory results. Keywordsinterventions, Stent, Fitting, Pulmonary artery,

/pdf/stent-fitting-using-cylindrical-affine-transformation-model-4y6p0qrlzd.pdf

Himani Arora

Papers

Lip2AudSpec: Speech reconstruction from silent lip movements video

Lip2Audspec: Speech Reconstruction from Silent Lip Movements Video

3D Accelerometer based Gesture Device for the Recognition of Digits

Computing Egomotion with Local Loop Closures for Egocentric Videos

Stent Fitting using Cylindrical Affine Transformation Model for Pulmonary Artery