This article is a survey of deep learning methods for single and multiple sound source localization, with a focus on sound source localization in indoor environments, where reverberation and diffuse noise are present. We provide an extensive topography of the neural network-based sound source localization literature in this context, organized according to the neural network architecture, the type of input features, the output strategy (classification or regression), the types of data used for model training and evaluation, and the model training strategy. Tables summarizing the literature survey are provided at the end of the paper, allowing a quick search of methods with a given set of target characteristics. 

https://asa.scitation.org/doi/pdf/10.1121/10.0011809

A survey of sound source localization with deep learning methods

This article is a survey on deep learning methods for single and multiple sound source localization. We are particularly interested in sound source localization in indoor/domestic environment, where reverberation and diffuse noise are present. We provide an exhaustive topography of the neural-based localization literature in this context, organized according to several aspects: the neural network architecture, the type of input features, the output strategy (classification or regression), the types of data used for model training and evaluation, and the model training strategy. This way, an interested reader can easily comprehend the vast panorama of the deep learning-based sound source localization methods. Tables summarizing the literature survey are provided at the end of the paper for a quick search of methods with a given set of target characteristics.

A Survey of Sound Source Localization with Deep Learning Methods.

Convolutional Neural Networks (CNNs) are a popular choice for estimating Direction of Arrival (DoA) without explicitly estimating delays between multiple microphones. The CNN method first optimises unknown filter weights (of a CNN) by using observations and ground-truth directional information. This trained CNN is then used to predict incident directions given test observations. Most existing methods train using spectrally-flat random signals and test using speech. In this paper, which focuses on single source DoA estimation, we find that training with speech or music signals produces a relative improvement in DoA accuracy for a variety of audio classes across 16 acoustic conditions and 9 DoAs, amounting to an average improvement of around 17% and 19% respectively when compared to training with spectrally flat random signals. This improvement is also observed in scenarios in which the speech and music signals are synthesised using, for example, a Generative Adversarial Network (GAN). When the acoustic environments during test and training are similar and reverberant, training a CNN with speech outperforms Generalized Cross Correlation (GCC) methods by about 125%. When the test conditions are different, a CNN performs comparably. This paper takes a step towards answering open questions in the literature regarding the nature of the signals used during training, as well as the amount of data required for estimating DoA using CNNs.

/pdf/on-improved-training-of-cnn-for-acoustic-source-localisation-4gkumwr5zo.pdf

On Improved Training of CNN for Acoustic Source Localisation

Visual SLAM for underwater vehicles: A survey

A method for multiple acoustic source localization using a tetrahedral microphone array and a convolutional neural network (CNN) is presented. Our method presents a novel approach for the estimation of acoustic source direction of arrival (DoA), both azimuth and elevation, utilizing a non-coplanar microphone array. In our approach, we use the phase component of the short-time Fourier transform (STFT) of the microphone array’s signals as the input feature for the CNN and a DoA probability density map as the training target. Our findings imply that our method outperforms the currently available methods for multiple sound source DoA estimation in both accuracy and speed.

Estimation of Azimuth and Elevation for Multiple Acoustic Sources Using Tetrahedral Microphone Arrays and Convolutional Neural Networks

In this paper, we propose an approach for robust visual Simultaneous Localisation and Mapping (SLAM) in underwater environments leveraging acoustic, inertial and altimeter/depth sensors. Underwater visual SLAM is challenging due to factors including poor visibility caused by suspended particles in water, a lack of light and insufficient texture in the scene. Because of this, many state-of-the-art approaches rely on acoustic sensing instead of vision for underwater navigation.Building on the sparse visual SLAM system ORB-SLAM2, this paper proposes to improve the robustness of camera pose estimation in underwater environments by leveraging acoustic odometry, which derives a drifting estimate of the 6-DoF robot pose from fusion of a Doppler Velocity Log (DVL), a gyroscope and an altimeter or depth sensor. Acoustic odometry estimates are used as motion priors and we formulate pose residuals that are integrated within the camera pose tracking, local and global bundle adjustment procedures of ORB-SLAM2.The original design of ORB-SLAM2 supports a single map and it enters relocalisation when tracking is lost. This is a significant problem for scenarios where a robot does a continuous scanning motion without returning to a previously visited location. One of our main contributions is to enable the system to create a new map whenever it encounters a new scene where visual odometry can work. This new map is connected with its predecessor in a common graph using estimates from the proposed acoustic odometry. Experimental results on two underwater vehicles demonstrate the increased robustness of our approach compared to baseline ORB-SLAM2 in both controlled, uncontrolled and field environments.

Robust Underwater Visual SLAM Fusing Acoustic Sensing

Acoustic source localization (ASL) is an important problem. Despite much attention over the past few decades, rapid and robust ASL still remains elusive. A popular approach is to use a circular array of microphones to record the acoustic signal followed by some form of optimization to deduce the most likely location of the source. In this paper, we study the impact of the configuration of microphones on the accuracy of localization. We perform experiments using simulation as well as real measurements using a 72-microphone acoustic camera which confirm that circular configurations lead to higher localization error than spiral and wheel configurations when considering large regions of space. Moreover, the configuration of choice is intricately tied to the optimization scheme. We show that direct optimization of well known formulations for ASL yield errors similar to the state of the art (steered response power) with 6 × less computation.

/pdf/impact-of-microphone-array-configurations-on-robust-indirect-2uuzyqjse9.pdf

Impact of Microphone Array Configurations on Robust Indirect 3d Acoustic Source Localization

Accurate estimation of Time-Difference of Arrivals (TDOAs) is necessary to perform accurate sound source localization. The problem has traditionally been solved by using methods such as Generalized Cross-Correlation, which uses the entire signal to accurately estimate TDOAs. However, this could pose a problem in distributed sensor networks in which the amount of data that can be transmitted from each sensor to a fusion center is limited, such as in underwater scenarios or other challenging environments. Inspired by approaches from computer vision, in this paper we identify Scale-Invariant Feature Transform (SIFT) keypoints in the signal spectrogram. We perform crosscorrelation on the signal using only the information available at those extracted keypoints. We test our algorithm in scenarios featuring different noise and reverberation conditions, and using different speech signals and source locations. We show that our algorithm can estimate Time-Difference of Arrivals (TDOAs) and the source location within an acceptable error range at a compression ratio of 40: 1.

/pdf/a-compressed-encoding-scheme-for-approximate-tdoa-estimation-53q7tbq073.pdf

A Compressed Encoding Scheme for Approximate Tdoa Estimation

The stereo vision problem gives rise to two sub-problems: the correspondence and the stereo calibration. Once the sub-problems have been solved, depth is estimated by triangulation. Depth estimation, using stereo images, is based on disparity – the relative displacement between corresponding points – and the camera geometry. Small errors in disparity may produce large errors in depth estimates due to the ill-posedness nature of the stereo vision problem. Moreover, if the camera geometry is unknown, it is estimated using corresponding points. A solution of stereo vision problem may be improved by identifying corresponding points which are inaccurately estimated. In this paper, the classification of a set of estimated corresponding points uses Delaunay triangulation by restricting it to a given subset of estimated corresponding points. Delaunay edges among estimated corresponding points are used to build undirected graphs. Classification criteria based on adjacencies are defined in order to decide whether or not a corresponding point is correctly estimated. Experimental evaluation, using ground truth image sets for quantitative analysis, shown values of specificity around 70% while sensitivity up to 96%.

Elizabeth Vargas

Papers

Robust Underwater Visual SLAM Fusing Acoustic Sensing

On Improved Training of CNN for Acoustic Source Localisation

Impact of Microphone Array Configurations on Robust Indirect 3d Acoustic Source Localization

A Compressed Encoding Scheme for Approximate Tdoa Estimation

Classifying Estimated Stereo Correspondences Based on Delaunay Triangulation