Audio-visual tracking of multiple speakers requires to estimate the state (e.g. velocity and location) of each speaker by leveraging the information of both audio and visual modalities. Estimating the number of speakers and their states jointly remains a challenging problem. We propose an Audio-Visual Possion Multi-Bernoulli Mixture Filter (AV-PMBM) that can not only predict the number of speakers but also give accurate estimation of their states. We also propose a novel sound source localization technique based on DOA information and a deep learning based object detector to provide reliable audio measurements for the AV tracker. To our knowledge, this represents the first attempt using PMBM for multi-speaker tracking with audio visual modalities. Experiments on the AV16.3 dataset demonstrate that AV-PMBM achieves state-of-the-art performance in optimal sub-pattern assignment (OSPA).

/pdf/audio-visual-tracking-of-multiple-speakers-via-a-pmbm-filter-2jolpkis.pdf

Audio-Visual Tracking of Multiple Speakers Via a PMBM Filter

Robot perception plays a crucial role in human-robot interaction (HRI). The
perception system provides the robot with information of the surroundings
and enables it to interact with people. In a conversational scenario, a group
of people may chat in front of the robot and move freely. In such situations,
robots are expected to understand where the people are, who is speaking, or
what they are talking about. This thesis concentrates on answering the first
two questions, namely speaker tracking and diarization. To that end, we use
different modalities of the robot’s perception system. Similar to seeing and
hearing for humans, audio and visual information are critical cues for robots
in a conversational scenario. Advancements in computer vision and audio
processing in the last decade revolutionized robot perception abilities and enabled
us to build joint audio-visual applications. In this thesis, we present the
following contributions: we first develop a variational Bayesian framework
for tracking multiple objects. The variational Bayesian framework provides
closed-form tractable problem solutions, enabling an efficient tracking process.
The framework is first applied to visual multiple-person tracking. The
birth and death processes are built jointly to deal with the varying number
of people in the scene. We then augment the framework by exploiting the
complementarity of vision and robot motor information. On the one hand, the
robot’s active motion can be integrated into the visual tracking system to stabilize
the tracking. On the other hand, visual information can be used to perform
motor servoing. As a next step we combine audio and visual information in the
framework and exploit the association between the acoustic feature frequency
bins with tracked people, to estimate the smooth trajectories of people, and to
infer their acoustic status (i.e. speaking or silent). To adapt the framework to
applications with no vision information, we employ it to acoustic-only speaker
localization and tracking. Online dereverberation techniques are first applied
then followed by the tracking system. Finally, we propose a variant of the
acoustic-only tracking model based on the von-Mises distribution, which is
specifically adapted to directional data. All proposed methods are validated
on datasets both qualitatively and quantitatively.

Audio-Visual Multiple-Speaker Tracking for Robot Perception

Estrategias de predicao de curto prazo sao uma importante ferramenta usada para planejamento e operacao de sistemas eletricos, bem como fundamentais para o processo de suporte a decisao para compra e venda de energia eletrica no mercado futuro. Particularmente, em se tratando de mercado de energia, uma componente importante para predicao de consumo sao os dias especiais (feriados ou dias atipicos, por exemplo). Tratar-se a predicao de tais componentes pode ser uma tarefa complexa, dado seu comportamento atipico, quando comparado a predicao de consumo em dias comuns. Em adicao, via de regra, o numero reduzido de amostras dificulta o treino e validacao adequados dos algoritmos de predicao de consumo em dias especiais. Este trabalho propoe um modelo para predicao de consumo de curto prazo que utiliza a tecnica Information Theoretic Learning Mean-Shift para clusterizacao e densificacao dos valores de consumo em dias especiais, e algoritmos de Redes Neurais Artificiais e Regressao Linear Multipla para predicao. O modelo foi aplicado em um problema de predicao de consumo da concessionaria de energia eletrica da regiao norte do Brasil, o que proporcionou uma melhoria na acuracia dos resultados ja obtido pelos metodos utilizados pela concessionaria.

/pdf/estrategia-para-predicao-de-consumo-de-energia-eletrica-de-4tb7tqjqix.pdf

Estratégia para predição de consumo de energia elétrica de curto prazo: uma abordagem baseada em densificação com MEAN SHIFT para tratamento de dias especiais

Audio-visual signals can be used jointly for robotic perception as they complement each other. Such multi-modal sensory fusion has a clear advantage, especially under noisy acoustic conditions. Speaker localization, as an essential robotic function, was traditionally solved as a signal processing problem that now increasingly finds deep learning solutions. The question is how to fuse audio-visual signals in an effective way. Speaker tracking is not only more desirable, but also potentially more accurate than speaker localization because it explores the speaker's temporal motion dynamics for smoothed trajectory estimation. However, due to the lack of large annotated dataset, speaker tracking is not well studied as speaker localization. In this paper, we study robotic speaker Direction of Arrival (DoA) estimation with a focus on audio-visual fusion and tracking methodology. We propose a Cross-Modal Attentive Fusion (CMAF) mechanism, which explores self-attention to learn intra-modal temporal dependencies, and cross-attention mechanism for inter-modal alignment. We also collect a realistic dataset on a robotic platform to support the study. The experimental results demonstrate that our proposed network outperforms the state-of-the-art audio-visual localization and tracking methods under noisy conditions, with an improved accuracy of 5.82% and 3.62% at SNR = −20 dB, respectively.

https://ieeexplore.ieee.org/ielx7/6570655/6633080/09968308.pdf

Audio-Visual Cross-Attention Network for Robotic Speaker Tracking

Audio and visual signals can be used jointly to provide comple-mentary information for multi-speaker tracking. Face detectors and color histogram can provide visual measurements while Di-rection of Arrival (DOA) lines and global coherence field (GCF) maps can provide audio measurements. GCF, as a traditional sound source localization method, has been widely used to provide audio measurements in audio-visual speaker tracking by estimating the positions of speakers. However, GCF cannot di-rectly deal with the scenarios of multiple speakers due to the emergence of spurious peaks on the GCF map, making it diffi-cult to find the non-dominant speakers. To overcome this limitation, we propose a phase-aware VoiceFilter and a separation-before-localization method, which enables the audio mixture to be separated into individual speech sources while retaining their phases. This allows us to calculate the GCF map for multiple speakers, thereby their positions accurately and concurrently. Based on this method, we design an adaptive audio measurement likelihood for audio-visual multiple speaker tracking using Poisson multi-Bernoulli mixture (PMBM) filter. The experiments demonstrate that our proposed tracker achieves state-of-the-art results on the AV16.3 dataset.

Audio Visual Multi-Speaker Tracking with Improved GCF and PMBM Filter

The probability hypothesis density (PHD) filter based on sequential Monte Carlo (SMC) approximation (also known as SMC-PHD filter) has proven to be a promising algorithm for multispeaker tracking. However, it has a heavy computational cost as surviving, spawned, and born particles need to be distributed in each frame to model the state of the speakers and to estimate jointly the variable number of speakers with their states. In particular, the computational cost is mostly caused by the born particles as they need to be propagated over the entire image in every frame to detect the new speaker presence in the view of the visual tracker. In this paper, we propose to use the audio data to improve the visual SMC-PHD (V-SMC-PHD) filter by using the direction of arrival angles of the audio sources to determine when to propagate the born particles and reallocate the surviving and spawned particles. The tracking accuracy of the audio-visual SMC-PHD (AV-SMC-PHD) algorithm is further improved by using a modified mean-shift algorithm to search and climb density gradients iteratively to find the peak of the probability distribution, and the extra computational complexity introduced by mean-shift is controlled with a sparse sampling technique. These improved algorithms, named as AVMS-SMC-PHD and sparse-AVMS-SMC-PHD, respectively, are compared systematically with AV-SMC-PHD and V-SMC-PHD based on the AV16.3, AMI, and CLEAR datasets.

Kilic

Papers

Data and Codes for reproducing the results in "Mean-Shift and Sparse Sampling Based SMC-PHD Filtering for Audio Informed Visual Speaker Tracking"