This paper describes a client-server system developed to enable hearing impaired persons to participate in lectures by providing real-time displayed transcripts. The core of this system is formed by an ASR module running on a recognition server and processing the input audio-video stream. This engine utilizes a large lexicon, topic-specific language models mixed properly from various sources (e.g. transcripts of spontaneous utterances, theses, web discussions) and unsupervised incremental speaker adaptation methods to cope with spontaneous lecture speech in highly inflective Czech language. The raw output of the ASR module is converted into a more readable form using a developed post-processing module based on finite state transducers. The resulting formated text (i.e. containing punctuation marks, digit strings, etc) is then displayed on the screen of each client device (e.g. notebook or tablet) in the lecture room.

Real-Time Lecture Transcription using ASR for Czech Hearing Impaired or Deaf Students.

Efficient error correction of recognition output is a major barrier in the adoption of speech interfaces. This thesis addresses this problem through a novel correction framework and user interface. The system uses constraints provided by the user to enhance re-recognition, correcting errors with minimal user effort and time. In our web interface, users listen to the recognized utterance, marking incorrect words as they hear them. After they have finished marking errors, they submit the edits back to the speech recognizer where it is merged with previous edits and then converted into a finite state transducer. This FST, modeling the regions of correct and incorrect words in the recognition output, is then composed with the recognizer's language model and the utterance is re-recognized. We explored the use of our error correction technique in both the lecture and restaurant domain, evaluating the types of errors and the correction performance in each domain. With our system, we have found significant improvements over other error correction techniques such as n-best lists, re-speaking or verbal corrections, and retyping in terms of actions per correction step, corrected output rate, and ease of use. Thesis Supervisor: James Glass Title: Principal Research Scientist Thesis Co-Supervisor: Lee Hetherington Title: Research Scientist

Efficient error correction for speech systems using constrained re-recognition

The learning within lectures of hearing-impaired students can be hindered by errors in captions generated by speech recognition. My research intends to address this problem by investigating ways of correcting these captions. I summarise approaches to automatic error correction and describe the preliminary studies that have been conducted. These studies show that human editors set a tough benchmark for automatic correction to meet and indicate that automatic correction is feasible. Finally, I summarise my intention to develop a correction framework that will permit quantitative and qualitative testing of correction methods.

Enhancing accessibility through correction of speech recognition errors

It is part of ATLAS.ti User Conference 2015 – qualitative data analysis and beyond / Editor: Susanne Friese and Thomas Ringmayr; Universitatsverlag der TU Berlin, 2016; ISBN 978-3-7983-2822-8 (composite publication) DOI 10.14279/depositonce-5079 [http://dx.doi.org/10.14279/depositonce-5079]

/pdf/transcribing-with-atlas-ti-4e6islf9iv.pdf

Transcribing with ATLAS.ti

Automatic speech recognition (ASR) in the educational environment could be a solution to address the problem of gaining access to the spoken words of a lecture for many students who find lectures hard to understand, such as those whose mother tongue is not English or who have a hearing impairment. In such an environment, it is difficult for ASR to provide transcripts with Word Error Rates (WER) less than 25% for the wide range of speakers. Reducing the WER reduces the time and therefore cost of correcting errors in the transcripts. To deal with the variation of acoustic features between speakers, ASR systems implement automatic vocal tract normalisation (VTN) that warps the formants (resonant frequencies) of the speaker to better match the formants of the speakers in the training set. The ASR also implements automatic dynamic time warping (DTW) to deal with variation in the speaker’s rate of speaking, by aligning the time series of the new spoken words with the time series of the matching spoken words of the training set. This research investigates whether the ASR’s automatic estimation of VTN and DTW can be enhanced through pre-processing the recording by manually warping the formants and speaking rate of the recordings using sound processing libraries (Rubber Band and SoundTouch) before transcribing the pre-processed recordings using ASR. An initial experiment, performed with the recordings of two male and two female speakers, showed that pre-processing the recording could improve the WER by an average of 39.5% for male speakers and 36.2% for female speakers. However the selection of the best warp factors was achieved through an iterative ‘trial and error’ approach that involved many hours calculating the word error rate for each warp factor setting. Finding a more efficient approach for selecting the warp factors for pre-processing was then investigated. The second experiment investigated the development of a modification function using, as its training set, the best warp factors from the ‘trial and error’ approach to estimate the modification percentage required to improve the WER of a recording. A modification function was found that on average improved the WER by 16% for female speakers and 7% for male speakers.

Improving automatic speech recognition transcription through signal processing

Lectures can be digitally recorded and replayed to provide multimedia revision material for students who attended the class and a substitute learning experience for students unable to attend. Deaf and hard of hearing people can find it difficult to follow speech through hearing alone or to take notes while they are lip-reading or watching a sign-language interpreter. Synchronising the speech with text captions can ensure deaf students are not disadvantaged and assist all learners to search for relevant specific parts of the multimedia recording by means of the synchronised text. Automatic Speech Recognition has been used to provide real-time captioning directly from lecturers’ speech in classrooms but it has proved difficult to obtain accuracy comparable to stenography. This paper describes the development, testing and evaluation of a system that enables editors to correct errors in the captions as they are created by Automatic Speech Recognition and makes suggestions for future possible improvements.

J Gerrard

Papers

Correcting Automatic Speech Recognition Errors in Real Time