Audio event recognition, the human-like ability to identify and relate sounds from audio, is a nascent problem in machine perception. Comparable problems such as object detection in images have reaped enormous benefits from comprehensive datasets - principally ImageNet. This paper describes the creation of Audio Set, a large-scale dataset of manually-annotated audio events that endeavors to bridge the gap in data availability between image and audio research. Using a carefully structured hierarchical ontology of 632 audio classes guided by the literature and manual curation, we collect data from human labelers to probe the presence of specific audio classes in 10 second segments of YouTube videos. Segments are proposed for labeling using searches based on metadata, context (e.g., links), and content analysis. The result is a dataset of unprecedented breadth and size that will, we hope, substantially stimulate the development of high-performance audio event recognizers.

Audio Set: An ontology and human-labeled dataset for audio events

The mismatch negativity (MMN) in basic research of central auditory processing: a review.

In the present article, the basic research using the mismatch negativity (MMN) and analogous results obtained by using the magnetoencephalography (MEG) and other brain-imaging technologies is reviewed. This response is elicited by any discriminable change in auditory stimulation but recent studies extended the notion of the MMN even to higher-order cognitive processes such as those involving grammar and semantic meaning. Moreover, MMN data also show the presence of automatic intelligent processes such as stimulus anticipation at the level of auditory cortex. In addition, the MMN enables one to establish the brain processes underlying the initiation of attention switch to, conscious perception of, sound change in an unattended stimulus stream. 2007 International Federation of Clinical Neurophysiology. Published by Elsevier Ireland Ltd. All rights reserved.

/pdf/invited-review-the-mismatch-negativity-mmn-in-basic-research-4qrhy54ph5.pdf

Invited review The mismatch negativity (MMN) in basic research of central auditory processing: A review

The purpose of this review is to formulate a revised model of information processing that takes into account recent research on memory storage, selective attention, effortful versus automatic processing, and the mutual constraints that these areas place on one another. One distinctive aspect of the proposed model is the inclusion of two phases of sensory storage in each modality. The first phase extends sensation for several hundred milliseconds, whereas the second phase is a vivid recollection of sensation. The mechanism of at least the longer phase is the activation of features in long-term memory, comparable to the mechanism of non-sensory, short-term storage. Another distinctive aspect of the model is that habituation/dishabituation and central executive processes together are assumed to determine the focus of attention, without the need for either an early or a late attentional filter. Research issues that contribute to a comparison of models are discussed.

Evolving conceptions of memory storage, selective attention, and their mutual constraints within the human information-processing system.

The dynamics of attending: How people track time-varying events.

User analysis is a crucial aspect of user-centered systems design, yet Human–Computer Interaction (HCI) has yet to formulate reliable and valid characterizations of users beyond gross distinctions based on task and experience. Individual differences research from mainstream psychology has identified a stable set of characteristics that would appear to offer potential application in the HCI arena. Furthermore, in its evolution over the last 100 years, research on individual differences has faced many of the problems of theoretical status and applicability that are common to HCI. In the present paper, the relationship between work in cognitive and differential psychology and current analyses of users in HCI is examined. It is concluded that HCI could gain significant predictive power if individual differences research was related to the analysis of users in contemporary systems design.

User analysis in HCI—the historical lessons from individual differences research

Auditory discrimination abilities of professional musicians were compared with those of nonmusicians. The stimuli for the frequency‐discrimination tasks were 300‐msec sinusoidal tones, 300‐msec square waves, and tone patterns consisting of ten 40‐msec tones played sequentially. The musicians’ difference thresholds for single tones were between Δf /f=0.001 and 0.0045. One‐half of the nonmusicians attained thresholds almost as low; the rest attained larger thresholds, up to Δf /f=0.017. The results for the pattern stimuli show a clearer separation between the nonmusicians and musicians, whose median difference thresholds were about three times smaller. However, nonmusician listeners who had previously trained with patterns not in the test set had different thresholds, substantially smaller than those obtained by the musicians. The appropriateness of preferential recruitment of musicians for psychoacoustic research is discussed. The responses to a musical background survey and correlations between the survey...

Performance on frequency‐discrimination tasks by musicians and nonmusicians

Four experiments investigated the acoustical correlates of similarity and categorization judgments of environmental sounds. In Experiment 1, similarity ratings were obtained from pairwise comparisons of recordings of 50 environmental sounds. A three-dimensional multidimensional scaling (MDS) solution showed three distinct clusterings of the sounds, which included harmonic sounds, discrete impact sounds, and continuous sounds. Furthermore, sounds from similar sources tended to be in close proximity to each other in the MDS space. The orderings of the sounds on the individual dimensions of the solution were well predicted by linear combinations of acoustic variables, such as harmonicity, amount of silence, and modulation depth. The orderings of sounds also correlated significantly with MDS solutions for similarity ratings of imagined sounds and for imagined sources of sounds, obtained in Experiments 2 and 3—as was the case for free categorization of the 50 sounds (Experiment 4)—although the categorization data were less well predicted by acoustic features than were the similarity data.

/pdf/similarity-and-categorization-of-environmental-sounds-36us69z3mx.pdf

Similarity and categorization of environmental sounds.

Three experiments tested listeners’ ability to identify 70 diverse environmental sounds using limited spectral information. Experiment 1 employed low- and high-pass filtered sounds with filter cutoffs ranging from 300 to 8000 Hz. Listeners were quite good (>50% correct) at identifying the sounds even when severely filtered; for the high-pass filters, performance was never below 70%. Experiment 2 used octave-wide bandpass filtered sounds with center frequencies from 212 to 6788 Hz and found that performance with the higher bandpass filters was from 70%–80% correct, whereas with the lower filters listeners achieved 30%–50% correct. To examine the contribution of temporal factors, in experiment 3 vocoder methods were used to create event-modulated noises (EMN) which had extremely limited spectral information. About half of the 70 EMN were identifiable on the basis of the temporal patterning. Multiple regression analysis suggested that some acoustic features listeners may use to identify EMN include envelope shape, periodicity, and the consistency of temporal changes across frequency channels. Identification performance with high- and low-pass filtered environmental sounds varied in a manner similar to that of speech sounds, except that there seemed to be somewhat more information in the higher frequencies for the environmental sounds used in this experiment.

Spectral-temporal factors in the identification of environmental sounds.

Thresholds for formant‐frequency discrimination were obtained for ten synthetic English vowels patterned after a female talker. To estimate the resolution of the auditory system for these stimuli, thresholds were measured using well‐trained subjects under minimal‐stimulus‐ uncertainty procedures. Thresholds were estimated for both increments and decrements in formant frequency for the first and second formants. Reliable measurements of threshold were obtained for most formants tested, the exceptions occurring when a harmonic of the fundamental was aligned with the center frequency of the test formant. In these cases, unusually high thresholds were obtained from some subjects and asymmetrical thresholds were measured for increments versus decrements in formant frequency. Excluding those cases, thresholds for formant frequency, ΔF, are best described as a piecewise‐linear function of frequency which is constant at about 14 Hz in the F1 frequency region (<800 Hz), and increases linearly in the F2 region. In the F2 region, the resolution for formant frequency is approximately 1.5%. The present thresholds are similar to previous estimates in the F1 region, but about a factor of three lower than those in the F2 region. Comparisons of these results to those for pure tones and for complex, nonspeech stimuli are discussed.

Charles S. Watson

Papers

User analysis in HCI—the historical lessons from individual differences research

Performance on frequency‐discrimination tasks by musicians and nonmusicians

Similarity and categorization of environmental sounds.

Spectral-temporal factors in the identification of environmental sounds.

Formant‐frequency discrimination for isolated English vowels