Open AccessPosted Content
Sound-Guided Semantic Image Manipulation
TLDR
In this paper, a framework that directly encodes sound into the multi-modal (image-text) embedding space and manipulates an image from the space is proposed.Abstract:
The recent success of the generative model shows that leveraging the multi-modal embedding space can manipulate an image using text information. However, manipulating an image with other sources rather than text, such as sound, is not easy due to the dynamic characteristics of the sources. Especially, sound can convey vivid emotions and dynamic expressions of the real world. Here, we propose a framework that directly encodes sound into the multi-modal (image-text) embedding space and manipulates an image from the space. Our audio encoder is trained to produce a latent representation from an audio input, which is forced to be aligned with image and text representations in the multi-modal embedding space. We use a direct latent optimization method based on aligned embeddings for sound-guided image manipulation. We also show that our method can mix text and audio modalities, which enrich the variety of the image modification. We verify the effectiveness of our sound-guided image manipulation quantitatively and qualitatively. We also show that our method can mix different modalities, i.e., text and audio, which enrich the variety of the image modification. The experiments on zero-shot audio classification and semantic-level image classification show that our proposed model outperforms other text and sound-guided state-of-the-art methods.read more
Citations
More filters
Journal ArticleDOI
GAN Inversion: A Survey
TL;DR: GAN inversion aims to invert a given image back into the latent space of a pretrained GAN model so that the image can be faithfully reconstructed from the inverted code by the generator as discussed by the authors .
Journal ArticleDOI
A review on multimodal zero‐shot learning
TL;DR: Multimodal zero-shot learning (MZSL) as mentioned in this paper is a general solution for incorporating prior knowledge into data-driven models and achieving accurate class identification, which can fully exploit the advantages of both technologies and is expected to produce models with greater generalization ability.
References
More filters
Proceedings ArticleDOI
A Style-Based Generator Architecture for Generative Adversarial Networks
TL;DR: This paper proposed an alternative generator architecture for GANs, borrowing from style transfer literature, which leads to an automatically learned, unsupervised separation of high-level attributes (e.g., pose and identity when trained on human faces) and stochastic variation in the generated images.
Posted Content
Representation Learning with Contrastive Predictive Coding
TL;DR: This work proposes a universal unsupervised learning approach to extract useful representations from high-dimensional data, which it calls Contrastive Predictive Coding, and demonstrates that the approach is able to learn useful representations achieving strong performance on four distinct domains: speech, images, text and reinforcement learning in 3D environments.
Proceedings ArticleDOI
Image Style Transfer Using Convolutional Neural Networks
TL;DR: A Neural Algorithm of Artistic Style is introduced that can separate and recombine the image content and style of natural images and provide new insights into the deep image representations learned by Convolutional Neural Networks and demonstrate their potential for high level image synthesis and manipulation.
Posted Content
UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
TL;DR: This work introduces UCF101 which is currently the largest dataset of human actions and provides baseline action recognition results on this new dataset using standard bag of words approach with overall performance of 44.5%.
Proceedings ArticleDOI
ArcFace: Additive Angular Margin Loss for Deep Face Recognition
TL;DR: This paper presents arguably the most extensive experimental evaluation against all recent state-of-the-art face recognition methods on ten face recognition benchmarks, and shows that ArcFace consistently outperforms the state of the art and can be easily implemented with negligible computational overhead.