Spatial ventriloquism refers to the phenomenon that a visual stimulus such as a flash can attract the perceived location of a spatially discordant but temporally synchronous sound. An analogous example of mutual attraction between audition and vision has been found in the temporal domain, where temporal aspects of a visual event, such as its onset, frequency, or duration, can be biased by a slightly asynchronous sound. In this review, we examine various manifestations of spatial and temporal attraction between the senses (both direct effects and aftereffects), and we discuss important constraints on the occurrence of these effects. Factors that potentially modulate ventriloquism—such as attention, synesthetic correspondence, and other cognitive factors—are described. We trace theories and models of spatial and temporal ventriloquism, from the traditional unity assumption and modality appropriateness hypothesis to more recent Bayesian and neural network approaches. Finally, we summarize recent evidence probing the underlying neural mechanisms of spatial and temporal ventriloquism.

/pdf/intersensory-binding-across-space-and-time-a-tutorial-review-45zwjm3vav.pdf

Intersensory binding across space and time: A tutorial review

The auditory system derives locations of sound sources from spatial cues provided by the interaction of sound with the head and external ears. Those cues are analyzed in specific brainstem pathways and then integrated as cortical representation of locations. The principal cues for horizontal localization are interaural time differences (ITDs) and interaural differences in sound level (ILDs). Vertical and front/back localization rely on spectral-shape cues derived from direction-dependent filtering properties of the external ears. The likely first sites of analysis of these cues are the medial superior olive (MSO) for ITDs, lateral superior olive (LSO) for ILDs, and dorsal cochlear nucleus (DCN) for spectral-shape cues. Localization in distance is much less accurate than that in horizontal and vertical dimensions, and interpretation of the basic cues is influenced by additional factors, including acoustics of the surroundings and familiarity of source spectra and levels. Listeners are quite sensitive to sound motion, but it remains unclear whether that reflects specific motion detection mechanisms or simply detection of changes in static location. Intact auditory cortex is essential for normal sound localization. Cortical representation of sound locations is highly distributed, with no evidence for point-to-point topography. Spatial representation is strictly contralateral in laboratory animals that have been studied, whereas humans show a prominent right-hemisphere dominance.

Chapter 6 - Sound localization

Animals, including humans, use interaural time differences (ITDs) that arise from different sound path lengths to the two ears as a cue of horizontal sound source location. The nature of the neural code for ITD is still controversial. Current models differentiate between two population codes: either a map-like rate-place code of ITD along an array of neurons, consistent with a large body of data in the barn owl, or a population rate code, consistent with data from small mammals. Recently, it was proposed that these different codes reflect optimal coding strategies that depend on head size and sound frequency. The chicken makes an excellent test case of this proposal because its physical prerequisites are similar to small mammals, yet it shares a more recent common ancestry with the owl. We show here that, like in the barn owl, the brainstem nucleus laminaris in mature chickens displayed the major features of a place code of ITD. ITD was topographically represented in the maximal responses of neurons along each isofrequency band, covering approximately the contralateral acoustic hemisphere. Furthermore, the represented ITD range appeared to change with frequency, consistent with a pressure gradient receiver mechanism in the avian middle ear. At very low frequencies, below 400 Hz, maximal neural responses were symmetrically distributed around zero ITD and it remained unclear whether there was a topographic representation. These findings do not agree with the above predictions for optimal coding and thus revive the discussion as to what determines the neural coding strategies for ITDs.

Maps of interaural time difference in the chicken’s brainstem nucleus laminaris

On the role of eye and head position in spatial localisation behaviour

s and posTers Van Barneveld D.C.P.B.M., Van Wanrooij M.M. and Van Opstal A.J. (2011) The reference frame of the ventriloquist effect TU/e Dag van de Perceptie 2011 Van Opstal A.J. and Van Barneveld D.C.P.B.M. (2011) Audio-vestibular interactions in human orienting, Vergadering Werkgemeenschap Auditief Systeem Van Barneveld D.C.P.B.M., Kiemeneij A.C.M. and Van Opstal (2010) Free-field sound localization during passive whole-body rotation Association for Research in Otolaryngology Midwinter Meeting 2010, 174 Kiemeneij A.C.M., Van Barneveld D.C.P.B.M. and Van Opstal A.J. (2010) Saccades to brief visual flashes presented during passive vestibular whole-body rotation FENS abstract, vol. 5, 084.8 Van Barneveld D.C.P.B.M., Binkhorst F. and Van Opstal A.J. (2010) Failure to compensate for vestibularly-evoked passive head rotations in human sound localization FENS abstract, vol. 5, 083.11 Van Barneveld D.C.P.B.M. and Van Opstal A.J. (2008) Influence of eye position on audiovestibular interaction during whole-body rotation Neuroscience Meeting Planner. Washington, DC: Society for Neuroscience 367.11 Van Barneveld D.C.P.B.M., Meijerink, E. and Van Opstal A.J. (2008) Influence of eye position on audio-vestibular interactions during whole-body rotation TNO Soesterberg Dag van de Perceptie 2008 Cpter 11 Eplogue CurriCulum ViTae Denise van Barneveld werd op 1 juli 1983 geboren in Geleen. Hier ging zij naar basisschool De Driesprong en vervolgens naar de Albert Schweitzer Scholengemeenschap (later gefuseerd tot het Graaf Huijn College), waar ze in 2001 cum laude haar VWO diploma behaalde. Hierna verhuisde zij naar Nijmegen om daar de opleiding Natuurwetenschappen te volgen aan de Radboud Universiteit. Als snel werd duidelijk dat haar hart lag bij de biofysica, met name de werking van de hersenen. Een logisch vervolg op de bachelor Natuurwetenschappen was dan ook de research master Cognitive Neuroscience, waaraan zij in 2004 begon. Tegelijkertijd heeft zij de master Natuurwetenschappen afgerond. Na het beëindigen van deze twee opleidingen, is zij in 2007 als promovenda aan de slag gegaan bij de afdeling Biofysica aan de Radboud Universiteit, onder begeleiding van Prof. Dr. A. John van Opstal. Deze afdeling is in 2008 opgegaan in het Donders Institute for Brain, Cognition and Behaviour. De resultaten van het onderzoek dat zij daar gedaan heeft, staan beschreven in dit proefschrift. Sinds 1 november 2011 werkt zij aan het Leids Universitair Medisch Centrum, waar ze bij de afdeling KNO onderzoek doet naar de verbetering van spraakverstaaan in een ruizige omgeving van patiënten met een cochleair implantaat.

Integration of exteroceptive and interoceptive cues in spatial localization

Interaural time differences are an important cue for azimuthal sound localization. It is still unclear whether the same neuronal mechanisms underlie the representation in the brain of interaural time difference in different vertebrates and whether these mechanisms are driven by common constraints, such as optimal coding. Current sound localization models may be discriminated by studying the spectral distribution of response peaks in tuning curves that measure the sensitivity to interaural time difference. The sound localization system of the barn owl has been studied intensively, but data that would allow discrimination between currently discussed models are missing from this animal. We have therefore obtained extracellular recordings from the time-sensitive subnuclei of the barn owl's inferior colliculus. Response peaks were broadly scattered over the physiological range of interaural time differences. A change in the representation of the interaural phase differences with frequency was not observed. In some neurons, response peaks fell outside the physiological range of interaural time differences. For a considerable number of neurons, the peak closest to zero interaural time difference was not the behaviorally relevant peak. The data are in best accordance with models suggesting that a place code underlies the representation of interaural time difference. The data from the high-frequency range, but not from the low-frequency range, are consistent with predictions of optimal coding. We speculate that the deviation of the representation of interaural time difference from optimal-coding models in the low-frequency range is attributable to the diminished importance of low frequencies for catching prey in this species.

https://www.jneurosci.org/content/jneuro/27/15/4191.full.pdf

Distribution of Interaural Time Difference in the Barn Owl's Inferior Colliculus in the Low- and High-Frequency Ranges

To program a goal-directed orienting response toward a sound source embedded in an acoustic scene, the audiomotor system should detect and select the target against a background. Here, we focus on whether the system can segregate synchronous sounds in the midsagittal plane (elevation), a task requiring the auditory system to dissociate the pinna-induced spectral localization cues. Human listeners made rapid head-orienting responses toward either a single sound source (broadband buzzer or Gaussian noise) or toward two simultaneously presented sounds (buzzer and noise) at a wide variety of locations in the midsagittal plane. In the latter case, listeners had to orient to the buzzer (target) and ignore the noise (nontarget). In the single-sound condition, localization was accurate. However, in the double-sound condition, response endpoints depended on relative sound level and spatial disparity. The loudest sound dominated the responses, regardless of whether it was the target or the nontarget. When the sounds had about equal intensities and their spatial disparity was sufficiently small, endpoint distributions were well described by weighted averaging. However, when spatial disparities exceeded ∼45°, response endpoint distributions became bimodal. Similar response behavior has been reported for visuomotor experiments, for which averaging and bimodal endpoint distributions are thought to arise from neural interactions within retinotopically organized visuomotor maps. We show, however, that the auditory-evoked responses can be well explained by the idiosyncratic acoustics of the pinnae. Hence basic principles of target representation and selection for audition and vision appear to differ profoundly.

https://www.jneurosci.org/content/jneuro/30/1/194.full.pdf

Pinna Cues Determine Orienting Response Modes to Synchronous Sounds in Elevation

Orienting responses to audiovisual events in the environment can benefit markedly by the integration of visual and auditory spatial information. However, logically, audiovisual integration would only be considered successful for stimuli that are spatially and temporally aligned, as these would be emitted by a single object in space-time. As humans do not have prior knowledge about whether novel auditory and visual events do indeed emanate from the same object, such information needs to be extracted from a variety of sources. For example, expectation about alignment or misalignment could modulate the strength of multisensory integration. If evidence from previous trials would repeatedly favour aligned audiovisual inputs, the internal state might also assume alignment for the next trial, and hence react to a new audiovisual event as if it were aligned. To test for such a strategy, subjects oriented a head-fixed pointer as fast as possible to a visual flash that was consistently paired, though not always spatially aligned, with a co-occurring broadband sound. We varied the probability of audiovisual alignment between experiments. Reaction times were consistently lower in blocks containing only aligned audiovisual stimuli than in blocks also containing pseudorandomly presented spatially disparate stimuli. Results demonstrate dynamic updating of the subject's prior expectation of audiovisual congruency. We discuss a model of prior probability estimation to explain the results.

Acquired prior knowledge modulates audiovisual integration.

We studied the influence of frequency on sound localization in free-flying barn owls by quantifying aspects of their target-approaching behavior to a distant sound source during ongoing auditory stimulation. In the baseline condition with a stimulus covering most of the owls hearing range (1-10 kHz), all owls landed within a radius of 20 cm from the loudspeaker in more than 80% of the cases and localization along the azimuth was more accurate than localization in elevation. When the stimulus contained only high frequencies (>5 kHz) no changes in striking behavior were observed. But when only frequencies from 1 to 5 kHz were presented, localization accuracy and precision decreased. In a second step we tested whether a further border exists at 2.5 kHz as suggested by optimality models. When we compared striking behavior for a stimulus having energy from 2.5 to 5 kHz with a stimulus having energy between 1 and 2.5 kHz, no consistent differences in striking behavior were observed. It was further found that pre-takeoff latency was longer for the latter stimulus than for baseline and that center frequency was a better predictor for landing precision than stimulus bandwidth. These data fit well with what is known from head-turning studies and from neurophysiology.

Target-approaching behavior of barn owls (Tyto alba): influence of sound frequency

This study compares the performance of a newly developed gaze (eye-in-space) measurement technique based on double magnetic induction (DMI) by a custom-made gold-plated copper ring on the eye with the classical scleral search coil (SSC) technique to record two-dimensional (2D) head-unrestrained gaze shifts. We tested both systems simultaneously during head-free saccades toward light-emitting diodes (LEDs) within the entire oculomotor range (+/-35 deg). The absence of irritating lead wires in the case of the DMI method leads to a higher guarantee of success (no coil breakage) and to less irritation on the subject's eye, which results in a longer and more comfortable measurement time. Correlations between DMI and SSC signals for horizontal and vertical eye position, velocity, and acceleration were close to 1.0. The difference between the SSC signal and the DMI signal remains within a few degrees. In our current setup the resolution was about 0.3 deg for the DMI method versus 0.2 deg for the SSC technique. The DMI method is an especially good alternative in the case of patient and laboratory animal gaze control studies where breakage of the SSC lead wires is particularly cumbersome.

/pdf/applying-double-magnetic-induction-to-measure-two-12rhtbrug4.pdf

Peter Bremen

Papers

Distribution of Interaural Time Difference in the Barn Owl's Inferior Colliculus in the Low- and High-Frequency Ranges

Pinna Cues Determine Orienting Response Modes to Synchronous Sounds in Elevation

Acquired prior knowledge modulates audiovisual integration.

Target-approaching behavior of barn owls (Tyto alba): influence of sound frequency

Applying double magnetic induction to measure two-dimensional head-unrestrained gaze shifts in human subjects.