Most existing datasets for sound event recognition (SER) are relatively small and/or domain-specific, with the exception of AudioSet, based on a massive amount of audio tracks from YouTube videos and encompassing over 500 classes of everyday sounds. However, AudioSet is not an open dataset---its release consists of pre-computed audio features (instead of waveforms), which limits the adoption of some SER methods. Downloading the original audio tracks is also problematic due to constituent YouTube videos gradually disappearing and usage rights issues, which casts doubts over the suitability of this resource for systems' benchmarking. To provide an alternative benchmark dataset and thus foster SER research, we introduce FSD50K, an open dataset containing over 51k audio clips totalling over 100h of audio manually labeled using 200 classes drawn from the AudioSet Ontology. The audio clips are licensed under Creative Commons licenses, making the dataset freely distributable (including waveforms). We provide a detailed description of the FSD50K creation process, tailored to the particularities of Freesound data, including challenges encountered and solutions adopted. We include a comprehensive dataset characterization along with discussion of limitations and key factors to allow its audio-informed usage. Finally, we conduct sound event classification experiments to provide baseline systems as well as insight on the main factors to consider when splitting Freesound audio data for SER. Our goal is to develop a dataset to be widely adopted by the community as a new open benchmark for SER research.

FSD50K: an Open Dataset of Human-Labeled Sound Events

Endowing machines with sensing capabilities similar to those of humans is a prevalent quest in engineering and computer science. In the pursuit of making computers sense their surroundings, a huge effort has been conducted to allow machines and computers to acquire, process, analyze and understand their environment in a human-like way. Focusing on the sense of hearing, the ability of computers to sense their acoustic environment as humans do goes by the name of machine hearing. To achieve this ambitious aim, the representation of the audio signal is of paramount importance. In this paper, we present an up-to-date review of the most relevant audio feature extraction techniques developed to analyze the most usual audio signals: speech, music and environmental sounds. Besides revisiting classic approaches for completeness, we include the latest advances in the field based on new domains of analysis together with novel bio-inspired proposals. These approaches are described following a taxonomy that organizes them according to their physical or perceptual basis, being subsequently divided depending on the domain of computation (time, frequency, wavelet, image-based, cepstral, or other domains). The description of the approaches is accompanied with recent examples of their application to machine hearing related problems.

https://www.mdpi.com/2076-3417/6/5/143/pdf

A Review of Physical and Perceptual Feature Extraction Techniques for Speech, Music and Environmental Sounds

http://publications.idiap.ch/downloads/papers/2008/Tommasi_PRL_2008.pdf

Discriminative cue integration for medical image annotation

The spectro-temporal receptive field (STRF) of an auditory neuron represents those characteristics of the sound stimulus in both the time and frequency domain that affect the firing probability of the neuron. The STRF is determined under stationary stimulus conditions for Gaussian wide-band noise. It has been demonstrated that for some neurons the response to that noise could to a considerable extent be derived from the STRF. In the present study the usefulness of the STRF is tested to predict responses to other stimuli such as noise with different frequency content and to species-specific vocalisations. It appears that the predicted response to vocalisations is at best in qualitative agreement with the actual response.

Prediction of the responses of auditory neurons in the midbrain of the grass frog based on the spectro-temporal receptive field

During language acquisition, infants have the benefit of visual cues to ground spoken language. Robots similarly have access to audio and visual sensors. Recent work has shown that images and spoken captions can be mapped into a meaningful common space, allowing images to be retrieved using speech and vice versa. In this setting of images paired with untranscribed spoken captions, we consider whether computer vision systems can be used to obtain textual labels for the speech. Concretely, we use an image-to-words multi-label visual classifier to tag images with soft textual labels, and then train a neural network to map from the speech to these soft targets. We show that the resulting speech system is able to predict which words occur in an utterance---acting as a spoken bag-of-words classifier---without seeing any parallel speech and text. We find that the model often confuses semantically related words, e.g. "man" and "person", making it even more effective as a semantic keyword spotter.

Visually grounded learning of keyword prediction from untranscribed speech

Robust speech detection in real acoustic backgrounds with perceptually motivated features

Robust detection of speech embedded in real acoustic background noise is considered using an approach based on subband amplitude modulation spectral (AMS) features and trained discriminative classifiers. Performance is evaluated in particular for situations in which speech is embedded in acoustic backgrounds not presented during classifier training, and for signal-to-noise ratios (SNR) from −10 dB to 20 dB. The results show that (1) Generalization to novel background classes with AMS features yields better performance in 84% of investigated situations, corresponding to an SNR benefit of about 10 dB compared to mel-frequency cepstral coefficient (MFCC) features. (2) On known backgrounds, AMS and MFCCs achieve similar performance, with a small advantage for AMS in negative SNR regimes. (3) Standard voice activity detection (ITU G729.B) performs significantly worse than the classification-based approach.

Modulation-based detection of speech in real background noise: Generalization to novel background classes

A classification method is presented that detects the presence of speech embedded in a real acoustic background of non-speech sounds. Features used for classification are modulation components extracted by computation of the amplitude modulation spectrogram. Feature selection techniques and support vector classification are employed to identify modulation components most salient for the classification task and therefore considered as highly characteristic for speech. Results show that reliable detection of speech can be performed with less than 10 optimally selected modulation features, the most important ones are located in the modulation frequency range below 10 Hz. Detection of speech in a background of non-speech signals is performed with about 90% test-data accuracy at a signal-to-noise level of 0 dB. Compared to standard ITU G729.B voice activity detection, the proposed method results in increased true positive and reduced false positive rates induced by a real acoustic background.

Detection of speech embedded in real acoustic background based on amplitude modulation spectrogram features.

Categorization is one of the fundamental building blocks of cognitive systems. Object categorization has traditionally been addressed in the vision domain, even though cognitive agents are intrinsically multimodal. Indeed, biological systems combine several modalities in order to achieve robust categorization. In this paper we propose a multimodal approach to object category detection, using audio and visual information. The auditory channel is modeled on biologically motivated spectral features via a discriminative classifier. The visual channel is modeled by a state of the art part based model. Multimodality is achieved using two fusion schemes, one high level and the other low level. Experiments on six different object categories, under increasingly difficult conditions, show strengths and weaknesses of the two approaches, and clearly underline the open challenges for multimodal category detection.

/pdf/object-category-detection-using-audio-visual-cues-28fbvbbxva.pdf

Object category detection using audio-visual cues

In this study, a new generic framework for the detection and interpretation of disagreement (“incongruence”) between different classifiers [1] is applied to the problem of detecting novel acoustic objects in an office environment. Using a general model that detects generic acoustic objects (standing out from a stationary background) and specific models tuned to particular sounds expected in the office, a novel object is detected as an incongruence between the models: the general model detects it as a generic object, but the specific models can not identify it as any of the known office-related sources. The detectors are realized using amplitude modulation spectrogram and RASTA-PLP features with support vector machine classification. Data considered are speech and non-speech sounds embedded in real office background at signal-to-noise ratios (SNR) from +20 dB to −20 dB. Our approach yields approximately 90% hit rate for novel events at 20 dB SNR, 75% at 0 dB and reaches chance level below −10 dB.

Jörg-Hendrik Bach

Papers

Robust speech detection in real acoustic backgrounds with perceptually motivated features

Modulation-based detection of speech in real background noise: Generalization to novel background classes

Detection of speech embedded in real acoustic background based on amplitude modulation spectrogram features.

Object category detection using audio-visual cues

Detecting novel objects in acoustic scenes through classifier incongruence.