scispace - formally typeset
Open AccessProceedings Article

Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization.

Reads0
Chats0
TLDR
In this article, adaptive instance normalization (AdaIN) is proposed to align the mean and variance of the content features with those of the style features, which enables arbitrary style transfer in real-time.
Abstract
Gatys et al. recently introduced a neural algorithm that renders a content image in the style of another image, achieving so-called style transfer. However, their framework requires a slow iterative optimization process, which limits its practical application. Fast approximations with feed-forward neural networks have been proposed to speed up neural style transfer. Unfortunately, the speed improvement comes at a cost: the network is usually tied to a fixed set of styles and cannot adapt to arbitrary new styles. In this paper, we present a simple yet effective approach that for the first time enables arbitrary style transfer in real-time. At the heart of our method is a novel adaptive instance normalization (AdaIN) layer that aligns the mean and variance of the content features with those of the style features. Our method achieves speed comparable to the fastest existing approach, without the restriction to a pre-defined set of styles. In addition, our approach allows flexible user controls such as content-style trade-off, style interpolation, color & spatial controls, all using a single feed-forward neural network.

read more

Content maybe subject to copyright    Report

Citations
More filters
Posted Content

Image-to-Image Translation via Group-wise Deep Whitening-and-Coloring Transformation

TL;DR: In this paper, the authors proposed an end-to-end approach tailored for image translation that efficiently approximates this transformation with their novel regularization methods and further extend their approach to a group-wise form for memory and time efficiency as well as image quality.
Proceedings ArticleDOI

Semantically Multi-Modal Image Synthesis

TL;DR: A novel Group Decreasing Network (GroupDNet) is proposed that leverages group convolution in the generator and progressively decreases the group numbers of the convolutions in the decoder, which is armed with much more controllability on translating semantic labels to natural images and has plausible high-quality yields for datasets with many classes.
Posted Content

Unrestricted Adversarial Examples via Semantic Manipulation

TL;DR: This paper introduces "unrestricted" perturbations that manipulate semantically meaningful image-based visual descriptors -- color and texture -- in order to generate effective and photorealistic adversarial examples.
Proceedings ArticleDOI

Reference-Based Sketch Image Colorization Using Augmented-Self Reference and Dense Semantic Correspondence

TL;DR: In this paper, the ground truth for sketch image colorization is obtained by using the identical image with geometric distortion as a virtual reference, which is used in the internal attention mechanism for color transfer from reference to sketch input.
Journal ArticleDOI

Texture and art with deep neural networks.

TL;DR: Recent advances in texture synthesis that were motivated by visual neuroscience have led to a substantial advance in image synthesis and manipulation in computer vision using convolutional neural networks, and how these can in turn inspire new research in visual perception and computational neuroscience is discussed.
References
More filters
Proceedings Article

Adam: A Method for Stochastic Optimization

TL;DR: This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework.
Proceedings Article

Very Deep Convolutional Networks for Large-Scale Image Recognition

TL;DR: In this paper, the authors investigated the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting and showed that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 layers.
Journal ArticleDOI

Generative Adversarial Nets

TL;DR: A new framework for estimating generative models via an adversarial process, in which two models are simultaneously train: a generative model G that captures the data distribution and a discriminative model D that estimates the probability that a sample came from the training data rather than G.
Proceedings Article

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

TL;DR: Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin.
Book ChapterDOI

Microsoft COCO: Common Objects in Context

TL;DR: A new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding by gathering images of complex everyday scenes containing common objects in their natural context.
Related Papers (5)