In this paper, a neural network-based lip reading system is proposed. The system is lexicon-free and uses purely visual cues. With only a limited number of visemes as classes to recognise, the system is designed to lip read sentences covering a wide range of vocabulary and to recognise words that may not be included in system training. The system has been testified on the challenging BBC Lip Reading Sentences 2 (LRS2) benchmark dataset. Compared with the state-of-the-art works in lip reading sentences, the system has achieved a significantly improved performance with 15% lower word error rate. In addition, experiments with videos of varying illumination have shown that the proposed model has a good robustness to varying levels of lighting. The main contributions of this paper are: 1) The classification of visemes in continuous speech using a specially designed transformer with a unique topology; 2) The use of visemes as a classification schema for lip reading sentences; and 3) The conversion of visemes to words using perplexity analysis. All the contributions serve to enhance the accuracy of lip reading sentences. The paper also provides an essential survey of the research area.

/pdf/lip-reading-sentences-using-deep-learning-with-only-visual-3cbkyb14rw.pdf

Lip Reading Sentences Using Deep Learning With Only Visual Cues

The problem of Lip-reading has become an important research challenge in recent years. The goal is to recognise speech from lip movements. Most of the Lip-reading technologies developed so far are camera-based, which require video recording of the target. However, these technologies have well-known limitations of occlusion and ambient lighting with serious privacy concerns. Furthermore, vision-based technologies are not useful for multi-modal hearing aids in the coronavirus (COVID-19) environment, where face masks have become a norm. This paper aims to solve the fundamental limitations of camera-based systems by proposing a radio frequency (RF) based Lip-reading framework, having an ability to read lips under face masks. The framework employs Wi-Fi and radar technologies as enablers of RF sensing based Lip-reading. A dataset comprising of vowels A, E, I, O, U and empty (static/closed lips) is collected using both technologies, with a face mask. The collected data is used to train machine learning (ML) and deep learning (DL) models. A high classification accuracy of 95% is achieved on the Wi-Fi data utilising neural network (NN) models. Moreover, similar accuracy is achieved by VGG16 deep learning model on the collected radar-based dataset. 

/pdf/pushing-the-limits-of-remote-rf-sensing-by-reading-lips-28448s40.pdf

Pushing the limits of remote RF sensing by reading lips under the face mask

Deaf students can only use their visual to acquire their first language namely sign language. Mostly hearing people do not understand sign language therefore deaf students use lip-reading to communicate with them. However, lip-reading especially in foreign language not always works because some phonemes are not visible in the lips. This study investigates the communication method of deaf students in learning English as foreign language. It observes the classroom interaction in an Indonesian special school for deaf students. The result shows that deaf students use four communication methods such as sign language, lip-reading, finger-spelling, and Tadoma. The result also shows that deaf students understand words or utterances which are familiar for them. However, deaf students find difficulties in pronouncing words or utterances. They tend to pronounce in Bahasa Indonesia. Therefore, it is recommended that teacher and hearing parents of deaf child must update with the latest technology or material for deaf students learning.

/pdf/the-communication-methods-in-english-classroom-for-2okp5uzx71.pdf

The Communication Methods in English Classroom for Indonesian Deaf Students

The idea of lip reading as a visual technique which people may use to translate lip movement into phrases without relying on speech itself is fascinating. There are numerous application areas in which lip reading could provide full assistance. Although there may be a downside to using the lip reading system, whether it may range from problems such as time constraint to minor word recognition mistakes, further development of the system should be an active cycle. Ongoing research into improvement of the lip reading system performance by way of suitable choices of feature extraction and classifiers is essential to track the developing trend in both fields of technology as well as in pattern recognition. This paper discusses a short review on the existing methods and classifiers which have been used in previous work in the field of lip reading.

A review on methods and classifiers in lip reading

The performance of automated lip reading using visemes as a classification schema has achieved less success compared with the use of ASCII characters and words largely due to the problem of different words sharing identical visemes. The Generative Pre-Training transformer is an effective autoregressive language model used for many tasks in Natural Language Processing, including sentence prediction and text classification. 
This paper proposes a new application for this model and applies it in the context of lip reading, where it serves as a language model to convert visual speech in the form of visemes, to language in the form of words and sentences. The network uses the search for optimal perplexity to perform the viseme-to-word mapping and is thus a solution to the one-to-many mapping problem that exists whereby various words that sound different when spoken look identical. This paper proposes a method to tackle the one-to-many mapping problem when performing automated lip reading using solely visual cues in two separate scenarios: the first scenario is where the word boundary, that is, the beginning and the ending of a word, is unknown; and the second scenario is where the boundary is known. 
Sentences from the benchmark BBC dataset "Lip Reading Sentences in the Wild"(LRS2), are classified with a character error rate of 10.7% and a word error rate of 18.0%. The main contribution of this paper is to propose a method of predicting words through the use of perplexity analysis when only visual cues are present, using an autoregressive language model.

Disentangling Homophemes in Lip Reading using Perplexity Analysis.

Deaf and hard of hearing people often have problems being able to understand and lip read other people. Usually deaf and hard of hearing people feel left out of conversation and sometimes they are actually ignored by other people. There are a variety of ways hearing-impaired person can communicate and gain accsss to the information. Communication support includes both technical and human aids. Human aids include interpreters, lip-readers and note-takers. Interpreters translate the Sign Language and must therefore be qualified. In this paper, vision system is used to track movements of the lip. In the experiment, the proposed system succesfully can differentiate 11 type of phonemes and then classified it to the respective viseme group. By using the proposed system the hearing-impaired persons could practise pronaunciations by themselve without support from the instructor.

https://iopscience.iop.org/article/10.1088/1757-899X/53/1/012016/pdf

S. K. Za’ba

Papers

Speech Analysis Based On Image Information from Lip Movement