Digital processing of speech signals

Dysarthria is a disorder that affects an individual’s speech intelligibility due to the paralysis of muscles and organs involved in the articulation process. As the condition is often associated with physically debilitating disabilities, not only do such individuals face communication problems, but also interactions with digital devices can become a burden. For these individuals, automatic speech recognition (ASR) technologies can make a significant difference in their lives as computing and portable digital devices can become an interaction medium, enabling them to communicate with others and computers. However, ASR technologies have performed poorly in recognizing dysarthric speech, especially for severe dysarthria, due to multiple challenges facing dysarthric ASR systems. We identified these challenges are due to the alternation and inaccuracy of dysarthric phonemes, the scarcity of dysarthric speech data, and the phoneme labeling imprecision. This paper reports on our second dysarthric-specific ASR system, called Speech Vision (SV) that tackles these challenges by adopting a novel approach towards dysarthric ASR in which speech features are extracted visually, then SV learns to see the shape of the words pronounced by dysarthric individuals. This visual acoustic modeling feature of SV eliminates phoneme-related challenges. To address the data scarcity problem, SV adopts visual data augmentation techniques, generates synthetic dysarthric acoustic visuals, and leverages transfer learning. Benchmarking with other state-of-the-art dysarthric ASR considered in this study, SV outperformed them by improving recognition accuracies for 67% of UA-Speech speakers, where the biggest improvements were achieved for severe dysarthria.

/pdf/speech-vision-an-end-to-end-deep-learning-based-dysarthric-32sot4vaia.pdf

Speech Vision: An End-to-End Deep Learning-Based Dysarthric Automatic Speech Recognition System

Despite the rapid progress of automatic speech recognition (ASR) technologies in the past few decades, recognition of disordered speech remains a highly challenging task to date. Disordered speech presents a wide spectrum of challenges to current data intensive deep neural networks (DNNs) based ASR technologies that predominantly target normal speech. This paper presents recent research efforts at the Chinese University of Hong Kong (CUHK) to improve the performance of disordered speech recognition systems on the largest publicly available UASpeech dysarthric speech corpus. A set of novel modelling techniques including neural architectural search, data augmentation using spectra-temporal perturbation, model based speaker adaptation and cross-domain generation of visual features within an audio-visual speech recognition (AVSR) system framework were employed to address the above challenges. The combination of these techniques produced the lowest published word error rate (WER) of 25.21% on the UASpeech test set 16 dysarthric speakers, and an overall WER reduction of 5.4% absolute (17.6% relative) over the CUHK 2018 dysarthric speech recognition system featuring a 6-way DNN system combination and cross adaptation of out-of-domain normal speech data trained systems. Bayesian model adaptation further allows rapid adaptation to individual dysarthric speakers to be performed using as little as 3.06 seconds of speech. The efficacy of these techniques were further demonstrated on a CUDYS Cantonese dysarthric speech recognition task.

Recent Progress in the CUHK Dysarthric Speech Recognition System

Dysarthria is a condition where people experience a reduction in speech intelligibility due to a neuromotor disorder. Previous works in dysarthric speech recognition have focused on accurate recognition of words encountered in training data. Due to the rarity of dysarthria in the general population, a relatively small amount of publicly-available training data exists for dysarthric speech. The number of unique words in these datasets is small, so ASR systems trained with existing dysarthric speech data are limited to recognition of those words. In this paper, we propose a data augmentation method using voice conversion that allows dysarthric ASR systems to accurately recognize words outside of the training set vocabulary. We demonstrate that a small amount of dysarthric speech data can be used to capture the relevant vocal characteristics of a speaker with dysarthria through a parallel voice conversion system. We show that it’s possible to synthesize utterances of new words that were never recorded by speakers with dysarthria, and that these synthesized utterances can be used to train a dysarthric ASR system.

Synthesis of New Words for Improved Dysarthric Speech Recognition on an Expanded Vocabulary

Dysarthria is a speech-motor disorder that affects the articulatory systems inhibiting their speech communication efforts. To handle their communication problems, a speech recognition-based augmentative and alternative communication aid is used as an attractive alternative. However, successful development of an automatic speech recognition (ASR)-based aid depends on the availability of sufficient speech data for training. Building an ASR system for dysarthric speakers is difficult due to limited amount of training data and large inter-and-intra speaker variabilities. Using normal speaker's speech data for data augmentation or adaptation for low intelligible dysarthric speakers would be extremely challenging due to huge variation in acoustic characteristics between these two category of speakers. In the current article, a two-level data augmentation is performed on dysarthric speech based on virtual linear microphone array-based synthesis followed by multi-resolution feature extraction. With the augmented speech data, an isolated word hybrid DNN-HMM-based ASR system is trained using UA speech corpus and Tamil dysarthric speech corpus developed by the authors. Performance of the ASR system shows a reduced WER of up to 32.79%, 35.75% for low and very low intelligible speakers with dysarthria compared to recent works on data augmentation reported for dysarthric speech recognition.

Data Augmentation Using Virtual Microphone Array Synthesis and Multi-Resolution Feature Extraction for Isolated Word Dysarthric Speech Recognition

An augmentative and alternative speech communication (AASC) aid comprises a speech recognition system and a speech synthesis system. The main challenge in developing such an aid for dysarthric speakers lies in handling errors in the text derived from the recognition system. These errors (substitution, deletion, and insertion) may be due to inability of a dysarthric speaker to utter certain phones (articulatory error) or due to inaccuracy of the models trained (modeling error). Most existing AASC approaches only focus on the articulatory errors and the ones that do address both errors, and do not differentiate between them. However, this paper performs a three-level cascaded analysis to identify and distinguish between these errors, as differentiating these errors will aid in appropriately handling them. Furthermore, analyses in the paper are independent of the syntax of utterances. Based on these analyses, weighted phone confusion transducers are formulated and used to correct erroneous text from the recognition system. The corrected text is finally synthesized by a text-to-speech synthesis system. The proposed AASC is observed to significantly reduce a word error rate of severe dysarthric speakers from 100% to 41.52%, moderate from 61.85% to 18.08%, and mild from 12.23% to 8.55%.

A Weighted Speaker-Specific Confusion Transducer-Based Augmentative and Alternative Speech Communication Aid for Dysarthric Speakers

Dysarthria is a motor speech disorder that causes inability to control and coordinate one or more articulators. This makes it difficult for a dysarthric speaker to utter certain speech sound units, thereby producing poorly articulated, slurred, and unintelligible speech. Hence, a speech supportive system needs to be developed to support them in their social difficulties. The current work aims at developing a speech supportive system, the objectives of which are threefold, namely (i) identifying the articulatory errors of each dysarthric speaker, (ii) developing a speech recognition system that corrects the errors in dysarthric speech by incorporating the findings from the first fold using a speaker-specific dictionary and (iii) developing an HMM-based speaker-adaptive speech synthesis system that synthesizes the error-corrected text for each dysarthric speaker retaining their identity. In the current work, the articulatory errors are analysed and identified, for 10 dysarthric speakers from the Nemours dysarthric speech corpus, using isolated-style phoneme recognition system trained with TIMIT speech corpus, followed by product of likelihood Gaussian-based analysis. The estimated articulatory errors are incorporated into a phoneme recognition system using speaker-specific dictionary and bigram language model. The error-corrected text is then synthesized as speech. The synthesized speech is evaluated to check its intelligibility and naturalness using mean opinion score. To further improve the intelligibility, speech rate of the synthesized speech is modified using time-domain pitch synchronous overlap add (TDPSOLA) technique. The results are quite encouraging, and this system is expected to be developed as a speech assistive device for a large vocabulary, in the near future, in a hand-held device.

Speech-Input Speech-Output Communication for Dysarthric Speakers Using HMM-Based Speech Recognition and Adaptive Synthesis System

Dysarthria is a traumatic neuromotor disorder that affects the physical production of speech. It reduces the function of primary articulators that are involved in speech. Recent research has presented that the use of articulatory data provide a better assessment towards identifying the speech intelligibility of dysarthric speakers over acoustic models that are modelled based on the biological perception of speech through Mel scale than directly involving the anatomy of speech production. Articulatory data of speech, include positional data of the articulators obtained from 12 Electromagnetic Articulograph (EMA) sensor channels with each sensor channel containing 6 parameters namely x, y, z, Φ, θ, and rms. While identifying the phone intelligibility deficit for the dysarthric speakers, articulatory data of all 12 sensor combinations each with 6 parameters may create turbulence in the intelligibility assessment. Hence, an appropriate mapping of the sound units with the sensor combinations is vital. The current work, focuses on mapping the phones with the EMA sensor channels based on their place of articulation. The parameters are also grouped based on the 3D-spherical co-ordinate distribution of the articulator as (x, y, θ) and (y, z, Φ). The mapping of the EMA sensor channels with the sound units are further validated through HMM-based acoustic only models and FDA scores. The phones are trained through their corresponding articulatory sensor combination information using a 5-class support vector machine and tested through a 10-fold cross validation technique for both the parameter groups. The 5 classes include mild, moderate, moderate-to-severe, severe and normal. Each phone can be classified to any of the 5 class in both the parameter groups based on their speech intelligibility. The results show that the EMA sensor combinations with (y, z, Φ) parameter group for each phone coincide well with the acoustic and FDA scores.

T. A. Mariya Celin

Papers

Data Augmentation Using Virtual Microphone Array Synthesis and Multi-Resolution Feature Extraction for Isolated Word Dysarthric Speech Recognition

A Weighted Speaker-Specific Confusion Transducer-Based Augmentative and Alternative Speech Communication Aid for Dysarthric Speakers

Speech-Input Speech-Output Communication for Dysarthric Speakers Using HMM-Based Speech Recognition and Adaptive Synthesis System

Electromagnetic articulograph sensor-to-sound unit mapping-based intelligibility assessment of dysarthric speech

Data Augmentation Techniques for Transfer Learning-Based Continuous Dysarthric Speech Recognition