In this paper, we investigate the audio-visual event localization problem. This task is to localize a visible and audible event in a video. Previous methods first divide a video into short segments, and then fuse visual and acoustic features at the segment level. The duration of these segments is usually short, making the visual and acoustic feature of each segment possibly not well aligned. Direct concatenation of the two features at the segment level can be vulnerable to a minor temporal misalignment of the two signals. We propose a Dual Attention Matching (DAM) module to cover a longer video duration for better high-level event information modeling, while the local temporal information is attained by the global cross-check mechanism. Our premise is that one should watch the whole video to understand the high-level event, while shorter segments should be checked in detail for localization. Specifically, the global feature of one modality queries the local feature in the other modality in a bi-directional way. With temporal co-occurrence encoded between auditory and visual signals, DAM can be readily applied in various audio-visual event localization tasks, e.g., cross-modality localization, supervised event localization. Experiments on the AVE dataset show our method outperforms the state-of-the-art by a large margin.

/pdf/dual-attention-matching-for-audio-visual-event-localization-3bjc0pgvhf.pdf

Dual Attention Matching for Audio-Visual Event Localization

We investigate the weakly-supervised audio-visual video parsing task, which aims to parse a video into temporal event segments and predict the audible or visible event categories. The task is challenging since there only exist video-level event labels for training, without indicating the temporal boundaries and modalities. Previous works take the overall event labels to supervise both audio and visual model predictions. However, we argue that such overall labels harm the model training due to the audio-visual asynchrony. For example, commentators speak in a basketball video, but we cannot visually find the speakers. In this paper, we tackle this issue by leveraging the cross-modal correspondence of audio and visual signals. We generate reliable event labels individually for each modality by swapping audio and visual tracks with other unrelated videos. If the original visual/audio data contain event clues, the event prediction from the newly assembled data would still be highly confident. In this way, we could protect our models from being misled by ambiguous event labels. In addition, we propose the cross-modal audio-visual contrastive learning to induce temporal difference on attention models within videos, i.e., urging the model to pick the current temporal segment from all context candidates. Experiments show we outperform state-of-the-art methods by a large margin.

/pdf/exploring-heterogeneous-clues-for-weakly-supervised-audio-kp0bok8c23.pdf

Exploring Heterogeneous Clues for Weakly-Supervised Audio-Visual Video Parsing

In this paper, we introduce a new problem, named audio-visual video parsing, which aims to parse a video into temporal event segments and label them as either audible, visible, or both. Such a problem is essential for a complete understanding of the scene depicted inside a video. To facilitate exploration, we collect a Look, Listen, and Parse (LLP) dataset to investigate audio-visual video parsing in a weakly-supervised manner. This task can be naturally formulated as a Multimodal Multiple Instance Learning (MMIL) problem. Concretely, we propose a novel hybrid attention network to explore unimodal and cross-modal temporal contexts simultaneously. We develop an attentive MMIL pooling method to adaptively explore useful audio and visual content from different temporal extent and modalities. Furthermore, we discover and mitigate modality bias and noisy label issues with an individual-guided learning mechanism and label smoothing technique, respectively. Experimental results show that the challenging audio-visual video parsing can be achieved even with only video-level weak labels. Our proposed framework can effectively leverage unimodal and cross-modal temporal contexts and alleviate modality bias and noisy labels problems.

Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing

For every event occurring in the real world, most often a sound is associated with the corresponding visual scene. Humans possess an inherent ability to automatically map the audio content with visual scenes leading to an effortless and enhanced understanding of the underlying event. This triggers an interesting question: Can this natural correspondence between video and audio, which has been diminutively explored so far, be learned by a machine and modeled jointly to localize the sound source in a visual scene? In this paper, we propose a novel algorithm that addresses the problem of localizing sound source in unconstrained videos, which uses efficient fusion and attention mechanisms. Two novel blocks namely, Audio Visual Fusion Block (AVFB) and Segment-Wise Attention Block (SWAB) have been developed for this purpose. Quantitative and qualitative evaluations show that it is feasible to use the same algorithm with minor modifications to serve the purpose of sound localization using three different types of learning: supervised, weakly supervised and unsupervised. A novel Audio Visual Triplet Gram Matrix Loss (AVTGML) has been proposed as a loss function to learn the localization in an unsupervised way. Our empirical evaluations demonstrate a significant increase in performance over the existing state-of-the-art methods, serving as a testimony to the superiority of our proposed approach.

/pdf/see-the-sound-hear-the-pixels-3c1m5c1jez.pdf

See the Sound, Hear the Pixels

There are rich synchronized audio and visual events in our daily life. Inside the events, audio scenes are associated with the corresponding visual objects; meanwhile, sounding objects can indicate and help to separate their individual sounds in the audio track. Based on this observation, in this paper, we propose a cyclic co-learning (CCoL) paradigm that can jointly learn sounding object visual grounding and audio-visual sound separation in a unified framework. Concretely, we can leverage grounded object-sound relations to improve the results of sound separation. Meanwhile, benefiting from discriminative information from separated sounds, we improve training example sampling for sounding object grounding, which builds a co-learning cycle for the two tasks and makes them mutually beneficial. Extensive experiments show that the proposed framework outperforms the compared recent approaches on both tasks, and they can benefit from each other with our cyclic co-learning. The source code and pre-trained models are released in https://github.com/YapengTian/CCOL-CVPR21.

/pdf/cyclic-co-learning-of-sounding-object-visual-grounding-and-1emj2h8x08.pdf

Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separation

Automatically generating a natural language sentence to describe the content of an input video is a very challenging problem. It is an essential multimodal task in which auditory and visual contents are equally important. Although audio information has been exploited to improve video captioning in previous works, it is usually regarded as an additional feature fed into a black box fusion machine. How are the words in the generated sentences associated with the auditory and visual modalities? The problem is still not investigated. In this paper, we make the first attempt to design an interpretable audio-visual video captioning network to discover the association between words in sentences and audio-visual sequences. To achieve this, we propose a multimodal convolutional neural network-based audio-visual video captioning framework and introduce a modality-aware module for exploring modality selection during sentence generation. Besides, we collect new audio captioning and visual captioning datasets for further exploring the interactions between auditory and visual modalities for high-level video understanding. Extensive experiments demonstrate that the modality-aware module makes our model interpretable on modality selection during sentence generation. Even with the added interpretability, our video captioning network can still achieve comparable performance with recent state-of-the-art methods.

An Attempt towards Interpretable Audio-Visual Video Captioning

Video captioning [8] aims to automatically generate a natural language sentence to describe the dynamic, potentially complex multimodal scene inside of a video. Most of the previous works [8, 10] focus on exploring better visionand-language modelings and put less emphasis on the multimodal aspect of video captioning, where audio often reveals important in-scene and out-of-scene information and contributes to the language generation in its unique ways. For example, it adds the difficulty to describe the singing event by watching the audio-mute video in Fig. 1. Although this is not new to the multimedia community, many works over there aim to optimize the video captioning metrics with an uninterpretable fusion strategy. The basic questions as to what extent different modalities (auditory and visual) contribute to a particular sentence, and furthermore, to individual words in a sentence remain underexplored. It is our belief that unfolding these questions is valuable to making fundamental progress on video captioning. At first glance, it is seemingly impossible to answer the above questions. The first challenge is that there is no annotation denoting the individual contributions of the auditory or visual modality made to texts in any of the existing video captioning datasets—such a process is difficult to quantify without breakthroughs in Neurophysiology and Psychophysics. Instead, we study these questions from a computational perspective, where we mine signals from audio and video and compete their associations to text. The second challenge lies on the computational framework. Recurrent neural networks (RNNs) are widely used as decoders for video captioning. Despite the success in modeling sequential dependencies, RNN decoder-based architectures have inherent limitations to perform modalityinterpretable video captioning. When generating a word, besides using current provided/attended features and previous words, these models always exploit hidden states of the RNN decoder. The latter contains memorized information from different modalities, which makes the models impossible to disentangle the contributions from individual modalities for predicting the words. In this paper, we aim to disentangle the interplay of the two modalities and make the first attempt to interpretable audio-visual video captioning. Concretely, we propose a novel multimodal convolutional neural networkHumans: (1) people singing and dancing. (2) a group of people singing and dancing.

https://openaccess.thecvf.com/content_CVPRW_2019/papers/Sight and Sound/Yapeng_Tian_Audio-Visual_Interpretable_and_Controllable_Video_Captioning_CVPRW_2019_paper.pdf

Justin Goodman

Papers

An Attempt towards Interpretable Audio-Visual Video Captioning

Audio-Visual Interpretable and Controllable Video Captioning