An audio-visual corpus has been collected to support the use of common material in speech perception and automatic speech recognition studies. The corpus consists of high-quality audio and video recordings of 1000 sentences spoken by each of 34 talkers. Sentences are simple, syntactically identical phrases such as "place green at B 4 now". Intelligibility tests using the audio signals suggest that the material is easily identifiable in quiet and low levels of stationary noise. The annotated corpus is available on the web for research use.

An audio-visual corpus for speech perception and automatic speech recognition

Speech recognition was measured as a function of spectral resolution (number of spectral channels) and speech-to-noise ratio in normal-hearing (NH) and cochlear-implant(CI) listeners. Vowel, consonant, word, and sentence recognition were measured in five normal-hearing listeners, ten listeners with the Nucleus-22 cochlear implant, and nine listeners with the Advanced Bionics Clarion cochlear implant. Recognition was measured as a function of the number of spectral channels (noise bands or electrodes) at signal-to-noise ratios of +15, +10, +5, 0 dB, and in quiet. Performance with three different speech processing strategies (SPEAK, CIS, and SAS) was similar across all conditions, and improved as the number of electrodes increased (up to seven or eight) for all conditions. For all noise levels, vowel and consonant recognition with the SPEAK speech processor did not improve with more than seven electrodes, while for normal-hearing listeners, performance continued to increase up to at least 20 channels. Speech recognition on more difficult speech materials (word and sentence recognition) showed a marginally significant increase in Nucleus-22 listeners from seven to ten electrodes. The average implant score on all processing strategies was poorer than scores of NH listeners with similar processing. However, the best CI scores were similar to the normal-hearing scores for that condition (up to seven channels). CI listeners with the highest performance level increased in performance as the number of electrodes increased up to seven, while CI listeners with low levels of speech recognition did not increase in performance as the number of electrodes was increased beyond four. These results quantify the effect of number of spectral channels on speech recognition in noise and demonstrate that most CI subjects are not able to fully utilize the spectral information provided by the number of electrodes used in their implant.

/pdf/speech-recognition-in-noise-as-a-function-of-the-number-of-2e0j71t121.pdf

Speech recognition in noise as a function of the number of spectral channels: Comparison of acoustic hearing and cochlear implants

Do listeners process noisy speech by taking advantage of "glimpses"-spectrotemporal regions in which the target signal is least affected by the background? This study used an automatic speech recognition system, adapted for use with partially specified inputs, to identify consonants in noise. Twelve masking conditions were chosen to create a range of glimpse sizes. Several different glimpsing models were employed, differing in the local signal-to-noise ratio (SNR) used for detection, the minimum glimpse size, and the use of information in the masked regions. Recognition results were compared with behavioral data. A quantitative analysis demonstrated that the proportion of the time-frequency plane glimpsed is a good predictor of intelligibility. Recognition scores in each noise condition confirmed that sufficient information exists in glimpses to support consonant identification. Close fits to listeners' performance were obtained at two local SNR thresholds: one at around 8 dB and another in the range -5 to -2 dB. A transmitted information analysis revealed that cues to voicing are degraded more in the model than in human auditory processing.

/pdf/a-glimpsing-model-of-speech-perception-in-noise-35tohj1zpa.pdf

A glimpsing model of speech perception in noise.

http://web.mit.edu/hst.723/www/ThemePapers/Implants/WilsonDormanReview2008.pdf

Cochlear implants: a remarkable past and a brilliant future

Objectives/Hypothesis: This study documents the importance of preserving residual low-frequency acoustic hearing as those with more residual hearing are selected for cochlear implantation. Surgical strategies used for hearing preservation with a short hybrid cochlear implant are outlined. The benefits of preserved residual low-frequency hearing, improved word understanding in noise, and music appreciation are described. Study Design: Multicenter, prospective, single-subject design. Methods: Records were reviewed of 21 individuals participating in an Food and Drug Administration (FDA) feasibility clinical trial who have received an Iowa/Nucleus 10 mm electrode. A second group of subjects receiving implants at the University of Iowa that have used the 10 mm device between 2 years and 6 months were also reviewed. Outcome measures included standardized tests of monosyllabic word understanding, spondees in noise, and common melody recognition. Results: Lowfrequency hearing was maintained in all individuals immediately postoperative. One subject lost hearing at 2.5 months postoperative after a viral infection. The group has averaged a loss of 9 dB low-frequency acoustic hearing between 125 and 1,000 Hz. Monosyllabic word understanding scores at 6 months for a group being followed for an FDA clinical trial using the implant plus hearing aids was 69% correct. For the long-term group receiving implants at Iowa, monosyllabic word understanding in those who have used the device between 6 months and 2 years is 79%. Other important findings include improved recognition of speech in noise (9 dB improvement) as compared with standard cochlear implant recipients who were matched for speech recognition in quiet and near normal recognition of common melodies. Conclusion: The surgical strategies outlined have been successful in preservation of low-frequency hearing in 96% of individuals. Combined electrical and acoustical speech processing has enabled this group of volunteers to gain improved word understanding as compared with their preoperative hearing with bilateral hearing aids and a group of individuals receiving a standard cochlear implant with similar experience with their device. The improvement of speech in noise and melody recognition is attributed to the ability to distinguish fine pitch differences as the result of preserved residual low-frequency acoustic hearing. Preservation of low-frequency acoustic hearing is important for improving speech in noise and music appreciation for the hearing impaired, both of which are important in real-life situations. Key Words: Hearing Preservation, cochlear implant, hybrid cochlear implant, hearing in noise. Laryngoscope, 115:796–802, 2005

Preservation of Hearing in Cochlear Implant Surgery: Advantages of Combined Electrical and Acoustical Speech Processing

Current multichannel cochlear implant devices provide high levels of speech performance in quiet. However, performance deteriorates rapidly with increasing levels of background noise. The goal of this study was to investigate whether the noise susceptibility of cochlear implant users is primarily due to the loss of fine spectral information. Recognition of vowels and consonants was measured as a function of signal-to-noise ratio in four normal-hearing listeners in conditions simulating cochlear implants with both CIS and SPEAK-like strategies. Six conditions were evaluated: 3-, 4-, 8-, and 16-band processors (CIS-like), a 6/20 band processor (SPEAK-like), and unprocessed speech. Recognition scores for vowels and consonants decreased as the S/N level worsened in all conditions, as expected. Phoneme recognition threshold (PRT) was defined as the S/N at which the recognition score fell to 50% of its level in quiet. The unprocessed speech had the best PRT, which worsened as the number of bands decreased. Recognition of vowels and consonants was further measured in three Nucleus-22 cochlear implant users using either their normal SPEAK speech processor or a custom processor with a four-channel CIS strategy. The best cochlear implant user showed similar performance with the CIS strategy in quiet and in noise to that of normal-hearing listeners when listening to correspondingly spectrally degraded speech. These findings suggest that the noise susceptibility of cochlear implant users is at least partly due to the loss of spectral resolution. Efforts to improve the effective number of spectral information channels should improve implant performance in noise.

Effects of noise and spectral resolution on vowel and consonant recognition: acoustic and electric hearing.

Initial and medial consonants were recorded in three vowel contexts for use in speech recognition experiments. Five male and five female talkers were recorded producing the twenty-five consonants /b,d,g,p,t,k,m,n,ŋ,l,r,f,v,θ,ð,s,z,∫,t∫,dȝ,ȝ,j,w,ʍ,h/ in medial (v/C/v) and initial (C/v) positions using vowels /a/ (“hod”), /i/ (“heed”), and /u/ (“who’d”). The sampling rate for these recordings was 44.1 kHz. Representative tokens of each consonant were amplitude normalized to the steady-state portion of the vowel. Listening tests were conducted with normal-hearing listeners on a subset of twenty consonants in all three vowel contexts and in initial and medial positions. The results showed that the consonants were clearly recognized with only a few minor confusions, primarily between /v/ and /ð/. The full set of recordings is available for research use.

Consonant recordings for speech testing.

Learning electrically stimulated speech patterns can be a new and difficult experience for many cochlear implant users. In the present study, ten cochlear implant patients participated in an auditory training program using speech stimuli. Training was conducted at home using a personal computer for 1 hour per day, 5 days per week, for a period of 1 month or longer. Results showed a significant improvement in all patients’ speech perception performance. These results suggest that moderate auditory training using a computer-based auditory rehabilitation tool can be an effective approach for improving the speech perception performance of cochlear implant patients.

https://www.tigerspeech.com/2005-ARLO-Moderate-Training.pdf

Moderate auditory training can improve speech performance of adult cochlear implant patients

Currently there are few standardized speech testing materials for Mandarin-speaking cochlear implant (CI) listeners. In this study, Mandarin speech perception (MSP) sentence test materials were developed and validated in normal-hearing subjects listening to acoustic simulations of CI processing. Percent distribution of vowels, consonants, and tones within each MSP sentence list was similar to that observed across commonly used Chinese characters. There was no significant difference in sentence recognition across sentence lists. Given the phonetic balancing within lists and the validation with spectrally degraded speech, the present MSP test materials may be useful for assessing speech performance of Mandarin-speaking CI listeners.

/pdf/development-and-validation-of-the-mandarin-speech-perception-88xg99joxi.pdf

Xiaosong Wang

Papers

Speech recognition in noise as a function of the number of spectral channels: Comparison of acoustic hearing and cochlear implants

Effects of noise and spectral resolution on vowel and consonant recognition: acoustic and electric hearing.

Consonant recordings for speech testing.

Moderate auditory training can improve speech performance of adult cochlear implant patients

Development and validation of the Mandarin speech perception test