We present an overview of the ASR challenge for non-native children’s speech organized for a special session at Interspeech 2020. The data for the challenge was obtained in the context of a spoken language proficiency assessment administered at Italian schools for students between the ages of 9 and 16 who were studying English and German as a foreign language. The corpus distributed for the challenge was a subset of the English recordings. Participating teams competed either in a closed track, in which they could use only the training data released by the organizers of the challenge, or in an open track, in which they were allowed to use additional training data. The closed track received 9 entries and the open track received 7 entries, with the best scoring systems achieving substantial improvements over a state-of-the-art baseline system. This paper describes the corpus of non-native children’s speech that was used for the challenge, analyzes the results, and discusses some points that should be considered for subsequent challenges in this domain in the future.

Overview of the Interspeech TLT2020 Shared Task on ASR for Non-Native Children's Speech.

Unlike traditional (hybrid) Automatic Speech Recognition (ASR), end-to-end ASR systems simplify the training procedure by directly mapping acoustic features to sequences of graphemes or characters, thereby eliminating the need for specialized acoustic, language, or pronunciation models. However, one drawback of end-to-end ASR systems is that they require more training data than conventional ASR systems to achieve similar word error rate (WER). This makes it difficult to develop ASR systems for tasks where transcribed target data is limited such as developing ASR for Second Language (L2) speakers of Swedish. Nonetheless, recent advancements in selfsupervised acoustic learning, manifested in wav2vec models [1, 2, 3], leverage the available untranscribed speech data to provide compact acoustic representation that can achieve low WER when incorporated in end-to-end systems. To this end, we experiment with several monolingual and cross-lingual selfsupervised acoustic models to develop end-to-end ASR system for L2 Swedish. Even though our test is very small, it indicates that these systems are competitive in performance with traditional ASR pipeline. Our best model seems to reduce the WER by 7% relative to our traditional ASR baseline trained on the same target data.

Self-supervised end-to-end ASR for low resource L2 Swedish

The phonological representation of the Korean lateral has been widely discussed while less attention has been paid to the phonetic realizations of the sound. In the present study, we examine the static and dynamic properties of the Korean lateral from a production study and we measure the duration, the three formants and the time-normalized trajectories. Based on the results of a production study, it was found that (i) the duration of the lateral approximant in Korean is constant irrespective of the preceding vowel contexts, that (ii) the three formants of the lateral are severely influenced by the preceding contexts, and that (iii) the initial part of the lateral articulation share similar acoustic properties with the preceding vowels, but as the articulation continues, it becomes unaffected by the vowels.

/pdf/an-acoustic-study-of-the-word-final-lateral-approximant-in-105qcvp3rf.pdf

An Acoustic Study of the Word-final Lateral Approximant in Korea

Automatic Speech Recognition and Pronunciation Error Detection of Dutch Non-native Speech: cumulating speech resources in a pluricentric language

Automatic speech recognition (ASR) in children is a rapidly evolving field, as children become more accustomed to interacting with virtual assistants, such as Amazon Echo, Cortana, and other smart speakers, and it has advanced the human–computer interaction in recent generations. Furthermore, non-native children are observed to exhibit a diverse range of reading errors during second language (L2) acquisition, such as lexical disfluency, hesitations, intra-word switching, and word repetitions, which are not yet addressed, resulting in ASR’s struggle to recognize non-native children’s speech. The main objective of this study is to develop a non-native children’s speech recognition system on top of feature-space discriminative models, such as feature-space maximum mutual information (fMMI) and boosted feature-space maximum mutual information (fbMMI). Harnessing the collaborative power of speed perturbation-based data augmentation on the original children’s speech corpora yields an effective performance. The corpus focuses on different speaking styles of children, together with read speech and spontaneous speech, in order to investigate the impact of non-native children’s L2 speaking proficiency on speech recognition systems. The experiments revealed that feature-space MMI models with steadily increasing speed perturbation factors outperform traditional ASR baseline models.

/pdf/audio-augmentation-for-non-native-childrens-speech-wfukiwol.pdf

Audio Augmentation for Non-Native Children’s Speech Recognition through Discriminative Learning

This study investigates differences in sentence and story production between native and non-native speakers of English for use with a system of Automatic Speech Recognition (ASR). Previous studies have shown that production errors by non-native speakers of English include misproduced segments (Flege, 1995), longer pause duration (Anderson-Hsieh and Venkatagiri, 1994), abnormal pause location within clauses (Kang, 2010), and non-reduction of function words (Jang, 2009). The present study uses phonemically balanced sentences from TIMIT (Garofolo et al., 1993) and a story to provide an additional comparison of the differences in production by native and non-native speakers of English. Consistent with previous research, preliminary results suggest that non-native speakers of English fail to produce flaps and reduced vowels, insert or delete segments, engage in more self-correction, and place pauses in different locations from native speakers. Non-native English speakers furthermore produce different patterns of intonation from native speakers and produce errors indicative of transfer from their L1 phonology, such as coda deletion and vowel epenthesis. Native speaker productions also contained errors, the majority of which were content-related. These results indicate that difficulties posed by English ASR systems in recognizing non-native speech are due largely to the heterogeneity of non-native production.

A comparison between native and non-native speech for automatic speech recognition

This paper investigates acoustic characteristics of English liquids produced by Korean EFL learners. Features of liquids such as F2, F3 and duration were extracted from target sounds produced by Koreans and native speakers. Then, normalization processes were applied to the features in order to eliminate minor factors such as speaker variability and speaking rate. Through statistical analyses, it was found that the way Korean EFL learners utilize major features to pronounce English liquids is different from native speakers. Also, it was confirmed that the learners’ L1 interferes with their pronunciation of English liquids. For instance, when EFL learners produced the English lateral incorrectly, acoustic characteristics of Korean [l], which are significantly different from those of English [l], were included in their production. Based on the analysis, a discussion follows as to whether the current results are in line with traditional theoretical models of nonnative pronunciation acquisition.

Acoustic characteristics of English liquids produced by Korean learners of English

Despite the absence of clear and reliable word boundary markers, listeners recognize words in spoken sentences. A previous study tested a spoken-word recognition model, Shortlist, by asking speakers of British English to identify real words within nonwords (McQueen, Norris & Cutler, 1994). Some words were embedded within the onsets of longer words with either a weak-strong (WS) or strong-weak (SW) stress pattern (e.g., “mess” in /dəmɛs/, the onset of “domestic,” “sack” in /saekɹɪf/, the onset of “sacrifice"), and other words were embedded without a real word onset with either a WS or SW pattern (e.g., /nəmɛs/ or /mɛstəm/ for “mess” and /saekɹək/ or /kləsaek/ for “sack”). The original study reported both competition effects (e.g. competition from “domestic” hindered recognition of “mess") and prosodic effects (e.g. the stress in “mess” facilitated segmenting it from the preceding context). This current study aimed to replicate these results in American-English. Despite the different listener population and dialect, pilot results for the current study confirm both types of effect. This data will be used to test a new American English version of the Shortlist-B model of spoken word recognition (Norris & McQueen, 2008).

A replication of competition and prosodic effects on spoken word recognition

Speech contains vast variability, more than we often expect based on phonetics courses. Spontaneous, conversational speech in particular is frequently realized with deletions and alterations to many of the expected sounds. We refer to this as reduced speech. Reduced speech in context is quite perceptible to native listeners, but less so to non-native learners. Since L2 learners will encounter reduced speech when they leave the language classroom, spontaneous speech presents an opportunity for teaching perception of more natural speech to L2 learners. Furthermore, spontaneous speech can be quite intriguing for phonetics and linguistics students, so it also presents an opportunity for engaging students at all levels with language and phonetics.

Spontaneous speech in the teaching of phonetics and speech perception

In deployment, systems that use speech as input must make use of automated transcriptions. Yet, typically when these systems are evaluated, gold transcriptions are assumed. We explicitly examine the impact of transcription errors on the downstream performance of a multi-modal system on three related tasks from three datasets: emotion, sarcasm, and personality detection. We include three separate transcription tools and show that while all automated transcriptions propagate errors that substantially impact downstream performance, the open-source tools fair worse than the paid tool, though not always straightforwardly, and word error rates do not correlate well with downstream performance. We further find that the inclusion of audio features partially mitigates transcription errors, but that a naive usage of a multi-task setup does not.

Seongjin Park

Papers

A comparison between native and non-native speech for automatic speech recognition

Acoustic characteristics of English liquids produced by Korean learners of English

A replication of competition and prosodic effects on spoken word recognition

Spontaneous speech in the teaching of phonetics and speech perception

Me, myself, and ire: Effects of automatic transcription quality on emotion, sarcasm, and personality detection