Embodiments herein describe an audio forwarding regularizer and an information bottleneck that are used when training a machine learning (ML) system. The audio forwarding regularizer receives audio training data and identifies visually irrelevant and relevant sounds in the training data. By controlling the information bottleneck, the audio forwarding regularizer forwards data to a generator that is primarily related to the visually irrelevant sounds, while filtering out the visually relevant sounds. The generator also receives data regarding visual objects from a visual encoder derived from visual training data. Thus, when being trained, the generator receives data regarding the visual objects and data regarding the visually irrelevant sounds (but little to no data regarding the visually relevant sounds). Thus, during an execution stage, the generator can generate sounds that are relevant to the visual objects while not adding visually irrelevant sounds to the videos.

Generating synchronized sound from videos

The invention provides a method for locating a sound source from a video, and belongs to the field of cross-modal learning. According to the method, a training sample video is acquired in a training stage and preprocessed, a neural network composed of a full connection layer and a sound source positioning neural network composed of a positioning network are constructed, the sound source positioning neural network is trained by using a preprocessed training sample, and the trained sound source positioning neural network is obtained. In the test stage, a test video is obtained and preprocessed,then the test video is input into the trained sound source positioning neural network, the similarity is calculated, synchronization of sound and video images and sound source positioning after synchronization are further carried out through the similarity, and therefore the problem of sound source positioning of the asynchronous video is solved. The method can automatically find the correspondingrelation between each object and the sound in the video picture, is high in positioning accuracy, is high in position precision, and is very high in application value.

Sun Fuchun

Papers

Method of locating sound source from video