This paper presents an easily-configurable robot audition system using the Histogram-based Recursive Level Estimation (HRLE) method. In order to achieve natural human-robot interaction, a robot should recognize human speeches even if there are some noises and reverberations. Since the precision of automatic speech recognizers (ASR) have been degraded by such interference, many systems applying speech enhancement processes have been reported. However, performance of most reported systems suffer from acoustical environmental changes. For example, an enhancement process optimized for steady-state noise, such as fan noise, yields low performance when the process is used for non-steady-state noises, such as background music. The primary reason is mismatches of parameters because the appropriate parameters change according to the acoustical environments. To solve this problem, we propose a robot audition system that optimizes parameters adaptively and automatically. Our system applies linear and non-linear enhancement sub-processes. For the linear sub-process, we used Geometric Source Separation with the Adaptive Step-size method (GSS-AS). This adjusts the parameters adaptively and does not have any manual parameters. For the non-linear sub-process, we applied a spectral subtraction-based enhancement method with the HRLE method that is newly introduced in this paper. Since HRLE controls the threshold level parameter implicitly based on the statistical characteristics of noise and speech levels, our system has high robustness against acoustical environmental changes. For robot audition systems, all processes should be performed in real-time. We also propose implementation techniques to make HRLE run in real-time and show the effectiveness. We evaluate performance of our system and compare it to conventional systems based on the Minima Controlled Recursive Average (MCRA) method and Minimum Mean Square Error (MMSE) method. The experimental results show that our system achieves better performance than the conventional systems.

http://www.jp.honda-ri.com/upload/document/entry/20110324/Nakajima_2010_635_2320.pdf

An easily-configurable robot audition system using Histogram-based Recursive Level Estimation

Accurate detection, localization and tracking of multiple moving speakers permits a wide spectrum of applications Techniques are required that are versatile, robust to environmental variations, and not constraining for non-technical end-users Based on distant recording of spontaneous multiparty conversations, this thesis focuses on the use of microphone arrays to address the question Who spoke where and when? The speed, the versatility and the robustness of the proposed techniques are tested on a variety of real indoor recordings, including multiple moving speakers as well as seated speakers in meetings Optimized implementations are provided in most cases We propose to discretize the physical space into a few sectors, and for each time frame, to determine which sectors contain active acoustic sources (Where? When?) A topological interpretation of beamforming is proposed, which permits both to evaluate the average acoustic energy in a sector for a negligible cost, and to locate precisely a speaker within an active sector One additional contribution that goes beyond the eld of microphone arrays is a generic, automatic threshold selection method, which does not require any training data On the speaker detection task, the new approach is dramatically superior to the more classical approach where a threshold is set on training data We use the new approach into an integrated system for multispeaker detection-localization Another generic contribution is a principled, threshold-free, framework for short-term clustering of multispeaker location estimates, which also permits to detect where and when multiple trajectories intersect On multi-party meeting recordings, using distant microphones only, short-term clustering yields a speaker segmentation performance similar to that of close-talking microphones The resulting short speech segments are then grouped into speaker clusters (Who?), through an extension of the Bayesian Information Criterion to merge multiple modalities On meeting recordings, the speaker clustering performance is signicantly improved by merging the classical mel-cepstrum information with the short-term speaker location information Finally, a close analysis of the speaker clustering results suggests that future research should investigate the effect of human acoustic radiation characteristics on the overall transmission channel, when a speaker is a few meters away from a microphone

http://publications.idiap.ch/downloads/reports/2006/rr-06-77.pdf

Spatio-Temporal Analysis of Spontaneous Speech with Microphone Arrays

Acoustic sensing from a multi-rotor drone is heavily degraded by the strong ego-noise produced by the rotating motors and propellers. To address this problem, we propose a blind source separation (BSS) framework that extracts a target sound from noisy multi-channel signals captured by a microphone array mounted on a drone. The proposed method addresses the challenging problem of permutation alignment, in extremely low signal-to-noise-ratio scenarios (e.g. SNR $ −15 dB), by performing clustering on the time activities of the separated signals across frequencies. Since initialization plays an important role to the success of clustering, we propose a pre-processing algorithm which uses time-frequency spatial filtering (TFS) to generate a reference to pre-align the permutation. The pre-alignment not only improves the performance of clustering and permutation alignment, but also solves the target-channel selection problem for BSS. The proposed method integrates the advantages of both TFS and BSS. Experimental results with real-recorded data show that the proposed method is capable of processing the audio stream continuously in a blockwise manner and also remarkably outperforms the state-of-the-art.

/pdf/a-blind-source-separation-framework-for-ego-noise-reduction-o5u3k5v0ld.pdf

A Blind Source Separation Framework for Ego-Noise Reduction on Multi-Rotor Drones

A noise power estimation system for estimating noise power of each frequency spectral component includes a cumulative histogram generating section for generating a cumulative histogram for each frequency spectral component of a time series signal, in which the horizontal axis indicates index of power level and the vertical axis indicates cumulative frequency and which is weighted by exponential moving average; and a noise power estimation section for determining an estimated value of noise power for each frequency spectral component of the time series signal based on the cumulative histogram.

Noise power estimation system, noise power estimating method, speech recognition system and speech recognizing method

We propose a new type of noise reduction method suitable for autonomous mobile robots that require small and light-weighted devices and low-computational-cost algorithms. In the proposed method, four-line omni-directional micro electro mechanical system (MEMS) microphones are mounted on the robot head. The proposed method can reduce various kinds of noises simultaneously: directional noises are reduced by time-frequency masking using null beamformers and subtractive beamformers, and diffuse noises are reduced by multi-channel Wiener filtering using coherences. The effectiveness of the proposed method was shown in terms of speech recognition accuracies and speech qualities: the word error rate was reduced by 45% and 0.16 points of PESQ-MOS were improved compared with conventional time-frequency masking.

Ears of the robot: Noise reduction using four-line ultra-micro omni-directional microphones mounted on a robot head

A sound source separation method using four-line directivity microphones mounted on a head of a robot is proposed and applied to speech recognition under existence of two disturbances of speech. Sound source separation methods using microphones mounted on robot heads generally used strict head-related transfer functions(HRTF). We propose a robust sound source separation that does not require an estimate of a strict HRTF. Our method takes advantage of a sound pressure difference with the robot head acting as a sound barrier. The enhancement of the difference in the target speech is performed by signal processing of three layers:two-line SAFIA, twoline Spectral Subtraction and their integration. The experimental results of three simultaneous utterance recognition with vocabulary of 20K show that the proposed method is effective in achieving 71% error reduction.

Recognition of three simultaneous utterance of speech by four-line directivity microphone mounted on head of robot

This paper presents a method for solving the permutation problem. This is a problem specific to frequency domain blind source separation within the framework of independent component analysis. Towards this problem, we propose a method which uses reference signals. For each frequency bin, the permutation alignment is fixed by calculating correlation coefficients between the reference signal and the separated signal. Reference signals are obtained as signals corresponding to each individual original sources. The reference signals are chosen or obtained subjectively, and do not need to be separated well. For example, the conventional beamforming technique gives suitable reference signals. To show the effectiveness of this method, we conducted a experiment of continuous speech recognition in a real room. The experimental results of double talk recognition with 20K vocabulary show that the proposed method is effective to achieve 20% error reduction rate compared with the established DOA-based approach.

A method for solving the permutation problem of frequency-domain BSS using reference signal

Although end-to-end text-to-speech (TTS) models can generate natural speech, challenges still remain when it comes to estimating sentence-level phonetic and prosodic information from raw text in Japanese TTS systems. In this paper, we propose a method for polyphone disambiguation (PD) and accent prediction (AP). The proposed method incorporates explicit features extracted from morphological analysis and implicit features extracted from pre-trained language models (PLMs). We use BERT and Flair embeddings as implicit features and examine how to combine them with explicit features. Our objective evaluation results showed that the proposed method improved the accuracy by 5.7 points in PD and 6.0 points in AP. Moreover, the perceptual listening test results confirmed that a TTS system employing our proposed model as a front-end achieved a mean opinion score close to that of synthesized speech with ground-truth pronunciation and accent in terms of naturalness.

/pdf/polyphone-disambiguation-and-accent-prediction-using-pre-3k3y9821.pdf

Polyphone Disambiguation and Accent Prediction Using Pre-Trained Language Models in Japanese TTS Front-End

A novel speech segregation method using a microphone array with multiple directivities is proposed and applied to speech recognition under existence of disturbance speech. Conventional microphone array techniques use only single directivity of their own. It is very difficult for this kind of array technique to remove the influence of the disturbance. In our method, redundant simultaneous equations of the amplitudes of sound sources are generated by using these multiple directivities. The solution of these equations gives good estimates of disturbances. The spectral subtraction is applied with these estimates of disturbances, and the perfect enhancement of target speech is performed. The experimental results of double talk recognition with 20 K vocabulary show that the proposed enhancement technique is effective to achieve 45 % error reduction.

Speech enhancement based on multiple directivity patterns using a microphone array

NLP models are susceptible to learning spurious biases (i.e., bugs) that work on some datasets but do not properly reflect the underlying task. Explanation-based model debugging aims to resolve spurious biases by showing human users explanations of model behavior, asking users to give feedback on the behavior, thenusing the feedback to update the model. While existing model debugging methods have shown promise, their prototype-level implementations provide limited practical utility. Thus, we propose XMD: the first open-source, end-to-end framework for explanation-based model debugging. Given task- or instance-level explanations,users can flexibly provide various forms of feedback via an intuitive, web-based UI. After receiving user feedback, XMD automatically updates the model in real time, by regularizing the model so that its explanationsalign with the user feedback. The new model can then be easily deployed into real-world applications via Hugging Face. Using XMD, we can improve the model’s OOD performance on text classification tasks by up to 18%.

Toshiyuki Sekiya

Papers

Recognition of three simultaneous utterance of speech by four-line directivity microphone mounted on head of robot

A method for solving the permutation problem of frequency-domain BSS using reference signal

Polyphone Disambiguation and Accent Prediction Using Pre-Trained Language Models in Japanese TTS Front-End

Speech enhancement based on multiple directivity patterns using a microphone array

XMD: An End-to-End Framework for Interactive Explanation-Based Debugging of NLP Models