The environmental robustness of DNN-based acoustic models can be significantly improved by using multi-condition training data. However, as data collection is a costly proposition, simulation of the desired conditions is a frequently adopted strategy. In this paper we detail a data augmentation approach for far-field ASR. We examine the impact of using simulated room impulse responses (RIRs), as real RIRs can be difficult to acquire, and also the effect of adding point-source noises. We find that the performance gap between using simulated and real RIRs can be eliminated when point-source noises are added. Further we show that the trained acoustic models not only perform well in the distant-talking scenario but also provide better results in the close-talking scenario. We evaluate our approach on several LVCSR tasks which can adequately represent both scenarios.

A study on data augmentation of reverberant speech for robust speech recognition

Speech enhancement and separation are core problems in audio signal processing, with commercial applications in devices as diverse as mobile phones, conference call systems, hands-free systems, or hearing aids. In addition, they are crucial preprocessing steps for noise-robust automatic speech and speaker recognition. Many devices now have two to eight microphones. The enhancement and separation capabilities offered by these multichannel interfaces are usually greater than those of single-channel interfaces. Research in speech enhancement and separation has followed two convergent paths, starting with microphone array processing and blind source separation, respectively. These communities are now strongly interrelated and routinely borrow ideas from each other. Yet, a comprehensive overview of the common foundations and the differences between these approaches is lacking at present. In this paper, we propose to fill this gap by analyzing a large number of established and recent techniques according to four transverse axes: 1 the acoustic impulse response model, 2 the spatial filter design criterion, 3 the parameter estimation algorithm, and 4 optional postfiltering. We conclude this overview paper by providing a list of software and data resources and by discussing perspectives and future trends in the field.

/pdf/a-consolidated-perspective-on-multimicrophone-speech-qnbaqxmtvq.pdf

A Consolidated Perspective on Multimicrophone Speech Enhancement and Source Separation

Time delay estimation has been a research topic of significant practical importance in many fields (radar, sonar, seismology, geophysics, ultrasonics, hands-free communications, etc.). It is a first stage that feeds into subsequent processing blocks for identifying, localizing, and tracking radiating sources. This area has made remarkable advances in the past few decades, and is continuing to progress, with an aim to create processors that are tolerant to both noise and reverberation. This paper presents a systematic overview of the state-of-the-art of time-delay-estimation algorithms ranging from the simple cross-correlation method to the advanced blind channel identification based techniques. We discuss the pros and cons of each individual algorithm, and outline their inherent relationships. We also provide experimental results to illustrate their performance differences in room acoustic environments where reverberation and noise are commonly encountered.

/pdf/time-delay-estimation-in-room-acoustic-environments-an-2y9s0s5kf2.pdf

Time delay estimation in room acoustic environments: an overview

This paper presents stable and fast update rules for independent vector analysis (IVA) based on auxiliary function technique. The algorithm consists of two alternative updates: 1) weighted covariance matrix updates and 2) demixing matrix updates, which include no tuning parameters such as step size. The monotonic decrease of the objective function at each update is guaranteed. The experimental evaluation shows that the derived update rules yield faster convergence and better results than natural gradient updates.

Stable and fast update rules for independent vector analysis based on auxiliary function technique

This paper addresses the determined blind source separation problem and proposes a new effective method unifying independent vector analysis (IVA) and nonnegative matrix factorization (NMF). IVA is a state-of-the-art technique that utilizes the statistical independence between sources in a mixture signal, and an efficient optimization scheme has been proposed for IVA. However, since the source model in IVA is based on a spherical multivariate distribution, IVA cannot utilize specific spectral structures such as the harmonic structures of pitched instrumental sounds. To solve this problem, we introduce NMF decomposition as the source model in IVA to capture the spectral structures. The formulation of the proposed method is derived from conventional multichannel NMF (MNMF), which reveals the relationship between MNMF and IVA. The proposed method can be optimized by the update rules of IVA and single-channel NMF. Experimental results show the efficacy of the proposed method compared with IVA and MNMF in terms of separation accuracy and convergence speed.

Determined blind source separation unifying independent vector analysis and nonnegative matrix factorization

LREC2000: the 2nd International Conference on Language Resources and Evaluation, May 31 - June 2, 2000, Athens, Greece.

/pdf/acoustical-sound-database-in-real-environments-for-sound-1qza849hb6.pdf

Acoustical Sound Database in Real Environments for Sound Scene Understanding and Hands-Free Speech Recognition

Accurate localization of multiple sound sources is indispensable for the microphone array-based high quality sound capture. For single sound source localization, the CSP (cross-power spectrum phase analysis) method has been proposed. The CSP method localizes a sound source as a crossing point of sound directions estimated using different microphone pairs. However, when localizing multiple sound sources, the CSP method has a problem that the localization accuracy is degraded due to cross-correlation among different sound sources. To solve this problem, this paper proposes a new method which suppresses the undesired cross-correlation by synchronous addition of CSP coefficients derived from multiple microphone pairs. Experiment results in a real room showed that the proposed method improves the localization accuracy when increasing the number of the synchronous addition.

/pdf/localization-of-multiple-sound-sources-based-on-a-csp-21u3f3mepn.pdf

Localization of multiple sound sources based on a CSP analysis with a microphone array

It is very important for multi-lingual tele-conferencing through speech-to-speech translation to capture distant-talking speech with high quality. In addition, the speaker image is also needed to realize a natural communication in a multi-lingual tele-conference. A microphone array is an ideal candidate as an effective method for capturing distant-talking speech. Uttered speech can be enhanced and speaker image can be captured by steering a microphone array and a video camera in the speaker direction. However, to realize automatic steering of the microphone array and the video camera, it is necessary to localize the talker. To overcome this problem, we propose steering the microphone array and the video camera automatically toward a multilingual tele-conference through speech-to-speech translation. To realize the proposed system, we use the CSP coefficient addition method for speaker localization and the ATR-MATRIX for speechto-speech translation. We conducted experiments in a real room. Direction of Arrival (DOA) estimation rate (i.e., speaker image capturing rate) was 97.7%, speech recognition rate was 90.0%, and TOEIC score was 530 540 points, subject to locating the speaker at 2 meters distance from the microphone array. We also confirmed that the translated speech and the speaker image can be shown immediately after accurately steering the microphone array and the video camera in the speaker direction and translating the speech beamformed by a microphone array in real-time.

Automatic steering of microphone array and video camera toward multi-lingual tele-conference through speech-to-speech translation

It is very important for multilingual teleconferencing through speech-to-speech translation to capture distant-talking speech with high quality. In addition, the speaker image is also needed to realize a natural communication in such a conference. A microphone array is an ideal candidate for capturing distant-talking speech. Uttered speech can be enhanced and speaker images can be captured by steering a microphone array and a video camera in the speaker direction. However, to realize automatic steering, it is necessary to localize the talker. To overcome this problem, we propose collaborative steering of the microphone array and the video camera in real-time for a multilingual teleconference through speech-to-speech translation. We conducted experiments in a real room environment. The speaker localization rate (i.e., speaker image capturing rate) was 97.7%, speech recognition rate was 90.0%, and TOEIC score was 530/spl sim/540 points, subject to locating the speaker at a 2.0 meter distance from the microphone array.

Collaborative steering of microphone array and video camera toward multi-lingual tele-conference through speech-to-speech translation

In real environments, the presence of room reverberations seriously degrades the quality in sound capture. To solve this problem, multiple beamforming, which forms directivity not only in the direction of the desired sound source but also in the direction of reflection images, was proposed by Flanagan et al. (1993). However, it is difficult to apply this method practically in real environments, since this application requires that the distortion of reflection sound signals by wall impedances be equalized. The paper proposes a multiple beamforming algorithm that equalizes the amplitude-spectrum and phase-spectrum of reflection signals by a cross-spectrum method. Evaluation experiments are conducted in real environment. In a SDR (signal to distortion ratio) evaluation, the proposed multiple beamformer achieves signal distortion reduction more effectively than the conventional single beamformer and the conventional multiple beamformer without equalization. In addition, in an automatic speech recognition evaluation, the equalized multiple beamformer achieves a higher recognition performance than those of the above conventional beamformers.

T. Nishiura

Papers

Acoustical Sound Database in Real Environments for Sound Scene Understanding and Hands-Free Speech Recognition

Localization of multiple sound sources based on a CSP analysis with a microphone array

Automatic steering of microphone array and video camera toward multi-lingual tele-conference through speech-to-speech translation

Collaborative steering of microphone array and video camera toward multi-lingual tele-conference through speech-to-speech translation

Speech enhancement by multiple beamforming with reflection signal equalization