Navigation within a closed environment requires analysis of a variety of acoustic cues, a task that is well developed in many visually impaired individuals, and for which sighted individuals rely almost entirely on visual information. For blind people, the act of creating cognitive maps for spaces, such as home or office buildings, can be a long process, for which the individual may repeat various paths numerous times. While this action is typically performed by the individual on-site, it is of some interest to investigate at which point this task can be performed off-site, at the individual's discretion. In short, is it possible for an individual to learn an architectural environment without being physically present? If so, such a system could prove beneficial for navigation preparation in new and unknown environments. The main goal of the present research can therefore be summarized as investigating the possibilities of assisting blind individuals in learning a spatial environment configuration through the listening of audio events and their interactions with these events within a virtual reality experience. A comparison of two types of learning through auditory exploration has been performed: in situ real displacement and active navigation in a virtual architecture. The virtual navigation rendered only acoustic information. Results for two groups of five participants showed that interactive exploration of virtual acoustic room simulations can provide sufficient information for the construction of coherent spatial mental maps, although some variations were found between the two environments tested in the experiments. Furthermore, the mental representation of the virtually navigated environments preserved topological and metric properties, as was found through actual navigation.

/pdf/exploration-of-architectural-spaces-by-blind-people-using-bu7swrqtz2.pdf

Exploration of architectural spaces by blind people using auditory virtual reality for the construction of spatial knowledge

Binaural audio provides a listener with 3D sound sensation, allowing a rich perceptual experience of the scene. However, binaural recordings are scarcely available and require nontrivial expertise and equipment to obtain. We propose to convert common monaural audio into binaural audio by leveraging video. The key idea is that visual frames reveal significant spatial cues that, while explicitly lacking in the accompanying single-channel audio, are strongly linked to it. Our multi-modal approach recovers this link from unlabeled video. We devise a deep convolutional neural network that learns to decode the monaural (single-channel) soundtrack into its binaural counterpart by injecting visual information about object and scene configurations. We call the resulting output 2.5D visual sound---the visual stream helps "lift" the flat single channel audio into spatialized sound. In addition to sound generation, we show the self-supervised representation learned by our network benefits audio-visual source separation. Our video results: http://vision.cs.utexas.edu/projects/2.5D_visual_sound/

2.5D Visual Sound

Method and apparatus for producing, combining, and customizing virtual sound environments. A binaural sound system (400) includes a transceiver (492) configured for receiving a signal (600) containing at least a first type of information and a second type of information. The first type of information includes enunciated data (602). The enunciated data specifies certain information intended to be audibly enunciated to a user. The second type of information comprises first type of metadata (604-1) and a second type of metadata (604-2). The first type of metadata includes information which identifies a characteristic of the enunciated data exclusive of spatial position information. The second type of metadata identifies a spatial position information associated with the enunciated data.

Method and apparatus for creating a multi-dimensional communication space for use in a binaural audio system

We describe an experiment that explores the contribution of auditory and other features to the illusion of plausibility in a virtual environment that depicts the performance of a string quartet. ‘Plausibility’ refers to the component of presence that is the illusion that the perceived events in the virtual environment are really happening. The features studied were: Gaze (the musicians ignored the participant, the musicians sometimes looked towards and followed the participant's movements), Sound Spatialization (Mono, Stereo, Spatial), Auralization (no sound reflections, reflections corresponding to a room larger than the one perceived, reflections that exactly matched the virtual room), and Environment (no sound from outside of the room, birdsong and wind corresponding to the outside scene). We adopted the methodology based on color matching theory, where 20 participants were first able to assess their feeling of plausibility in the environment with each of the four features at their highest setting. Then five times participants started from a low setting on all features and were able to make transitions from one system configuration to another until they matched their original feeling of plausibility. From these transitions a Markov transition matrix was constructed, and also probabilities of a match conditional on feature configuration. The results show that Environment and Gaze were individually the most important factors influencing the level of plausibility. The highest probability transitions were to improve Environment and Gaze, and then Auralization and Spatialization. We present this work as both a contribution to the methodology of assessing presence without questionnaires, and showing how various aspects of a musical performance can influence plausibility.

/pdf/the-plausibility-of-a-string-quartet-performance-in-virtual-521mzkgksj.pdf

The Plausibility of a String Quartet Performance in Virtual Reality

Higher-order Ambisonics HOA is a representation of spatial sound fields that facilitates capturing, manipulating, recording, transmission and playback of complex audio scenes with superior spatial resolution, both in 2D and 3D. The sound field is approximated at and around a reference point in space by a Fourier-Bessel series. The invention uses space warping (12, 13, 14; 16) for modifying the spatial content and/or the reproduction of sound-field information that has been captured or produced as a higher-order Ambisonics representation. Different warping characteristics are feasible for 2D and 3D sound fields. The warping is performed in space domain without performing scene analysis or decomposition. Input HOA coefficients with a given order are decoded to the weights or input signals of regularly positioned (virtual) loudspeakers.

Method and apparatus for changing the relative positions of sound objects contained within a higher-order ambisonics representation

A computationally efficient 3D real time rendering engine for binaural sound reproduction via headphones is presented Binaural sound reproduction requires to filter the virtual sound source signals with head related transfer functions (HRTFs) To improve humans localization capabilities head tracking as well as room simulation have to be incorporated This yields the problem of high-quality, time-varying interpolation between different HRTFs To overcome this problem a virtual ambisonic approach is used that results in a bank of time-invariant HRTF filter

/pdf/a-3d-ambisonic-based-binaural-sound-reproduction-system-3ko9jl0qm3.pdf

A 3D Ambisonic Based Binaural Sound Reproduction System

Convincing binaural sound reproduction via headphones requires filtering the virtual sound source signals with head related transfer functions (HRTFs). Furthermore, humans are able to improve their localization capabilities by small unconscious head movements. Therefore it is important to incorporate head-tracking. This yields the problem of high-quality, time-varying interpolation between different HRTFs. A further improvement of human localization accuracy can be done by considering room simulation yielding a huge amount of virtual sound sources. To increase the computational efficiency of the proposed system, a virtual ambisonic approach is used, that result in a bank of time-invariant HRTF filter independent of the number of sources to encode.

/pdf/3d-binaural-sound-reproduction-using-a-virtual-ambisonic-1h1pxfobmt.pdf

3D binaural sound reproduction using a virtual ambisonic approach

A method of computationally efficient 3D sound reproduction via headphones is presented using a virtual Ambisonic approach. Previous studies have shown that incorporating head tracking as well as room simulation is important to improve sound source localization capabilities. The simulation of virtual acoustic space requires to filter the stimuli with head related transfer functions (HRTFs). In time-varying systems this yields the problem of high quality interpolation between different HRTFs. The proposed model states that encoding signals into Ambisonic domain results in time-invariant HRTF filters. The proposed system is implemented on a usual notebook using Pure Data (PD), a graphically based open source real time computer music software.

/pdf/a-3d-real-time-rendering-engine-for-binaural-sound-4yzp1iusci.pdf

A 3d real time rendering engine for binaural sound reproduction

This paper presents a library for programming 3D binaural sound reproduction systems using Pure Data (pd), an open source computer music programming language. The theory that forms the basis of the proposed library is an improved virtual Ambisonics approach. Using this approach provides computationally efficient implementation of multiple moving sound sources, room simulation, head tracking and time varying listener positions in virtual space. Furthermore, simple GUI objects for controlling parameters are implemented to provide easy to use environment.

A library for realtime 3d binaural sound reproduction in pure data (pd)

Topic of this paper is an interactive sound reproduction system to create virtual sonic environments to visual spaces, called Virtual Audio Reproduction Engine for Spatial Environments (VARESE). Using VARESE, Edgard Varse’ s Pome Olectronique was recreated within a simulated Philips Pavilion (a construction originally designed by Le Corbusier and Iannis Xenakis for the 1958 World Fair in Brussels, then dismantled after its closing). The system draws on binaural sound reproduction principles including spatialization techniques based on the Ambisonics theory. Using headphones and a head-tracker, listeners can enjoy a preset reproduction of the Pome Olectronique from their individual standpoint as they freely move through the virtual architectural space. While VARESE was developed specically for its use in reconstructing Poem Olectronique, it is e xible enough to function as a standard interpreter of sounds and visual objects, enabling users to design their own spatializations. The system runs on a standard laptop PC with a modern graphics card, using Miller Puckette’s computer music software Pure Data (pd), plus an extended graphic interface (running equally on Linux and MS Windows operating systems and featuring a maximum of ve sound sources using Ambisonics).

https://smartech.gatech.edu/bitstream/1853/50082/1/ZmoelnigLorenz2005.pdf

Thomas Musil

Papers

A 3D Ambisonic Based Binaural Sound Reproduction System

3D binaural sound reproduction using a virtual ambisonic approach

A 3d real time rendering engine for binaural sound reproduction

A library for realtime 3d binaural sound reproduction in pure data (pd)

Hearing Varese's Poeme Electronique inside a virtual philips pavilion