Semantic Image Inpainting with Deep Generative Models
Raymond A. Yeh,Chen Chen,Teck Yian Lim,Alexander G. Schwing,Alexander G. Schwing,Mark Hasegawa-Johnson,Minh N. Do +6 more
- pp 6882-6890
Reads0
Chats0
TLDR
A novel method for semantic image inpainting, which generates the missing content by conditioning on the available data, and successfully predicts information in large missing regions and achieves pixel-level photorealism, significantly outperforming the state-of-the-art methods.Abstract:
Semantic image inpainting is a challenging task where large missing regions have to be filled based on the available visual data. Existing methods which extract information from only a single image generally produce unsatisfactory results due to the lack of high level context. In this paper, we propose a novel method for semantic image inpainting, which generates the missing content by conditioning on the available data. Given a trained generative model, we search for the closest encoding of the corrupted image in the latent image manifold using our context and prior losses. This encoding is then passed through the generative model to infer the missing content. In our method, inference is possible irrespective of how the missing content is structured, while the state-of-the-art learning based method requires specific information about the holes in the training phase. Experiments on three datasets show that our method successfully predicts information in large missing regions and achieves pixel-level photorealism, significantly outperforming the state-of-the-art methods.read more
Citations
More filters
Journal ArticleDOI
Image Inpainting: A Review
TL;DR: The work in this paper was made by NPRP grant # NPRP8-140-2-065 from the Qatar National Research Fund (a member of the Qatar Foundation).
Proceedings ArticleDOI
COCO-GAN: Generation by Parts via Conditional Coordinating
TL;DR: COnditional COordinate GAN (COCO-GAN) of which the generator generates images by parts based on their spatial coordinates as the condition and the discriminator learns to justify realism across multiple assembled patches by global coherence, local appearance, and edge-crossing continuity is proposed.
Journal ArticleDOI
Scaffolding protein functional sites using deep learning
Jue Wang,Sidney Lisanza,David Juergens,Doug Tischer,Joseph L. Watson,Karla M Castro,Robert J. Ragotte,Amijai Saragovi,Lukas F. Milles,Minkyung Baek,Ivan Anishchenko,Wei Yang,Derrick R. Hicks,Marc Expòsit,Thomas Schlichthaerle,Jung Ho Chun,Justas Dauparas,N. Bennett,Basile I. M. Wicky,Andrew G. Muenks,Frank DiMaio,Bruno E. Correia,Sergey Ovchinnikov,David Baker +23 more
TL;DR: Wang et al. as mentioned in this paper proposed two deep learning methods to design proteins that contain prespecified functional sites, which can enable the scaffolding of desired functional residues within a well-folded designed protein.
Proceedings ArticleDOI
PEPSI : Fast Image Inpainting With Parallel Decoding Network
TL;DR: A novel network structure, called PEPSI: parallel extended-decoder path for semantic inpainting that reduces the number of convolution operations almost by half as compared to the conventional coarse-to-fine networks and exhibits superior performance to other models in terms of testing time and qualitative scores.
Journal ArticleDOI
Semantic Photo Manipulation with a Generative Image Prior
David Bau,Hendrik Strobelt,William Peebles,Jonas Wulff,Bolei Zhou,Jun-Yan Zhu,Antonio Torralba +6 more
TL;DR: The authors adapts the image prior learned by GANs to image statistics of an individual image, which can accurately reconstruct the input image and synthesize new content consistent with the appearance of the original image.
References
More filters
Proceedings Article
Adam: A Method for Stochastic Optimization
Diederik P. Kingma,Jimmy Ba +1 more
TL;DR: This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework.
Journal ArticleDOI
Image quality assessment: from error visibility to structural similarity
TL;DR: In this article, a structural similarity index is proposed for image quality assessment based on the degradation of structural information, which can be applied to both subjective ratings and objective methods on a database of images compressed with JPEG and JPEG2000.
Journal ArticleDOI
Generative Adversarial Nets
Ian Goodfellow,Jean Pouget-Abadie,Mehdi Mirza,Bing Xu,David Warde-Farley,Sherjil Ozair,Aaron Courville,Yoshua Bengio +7 more
TL;DR: A new framework for estimating generative models via an adversarial process, in which two models are simultaneously train: a generative model G that captures the data distribution and a discriminative model D that estimates the probability that a sample came from the training data rather than G.
Journal Article
Visualizing Data using t-SNE
TL;DR: A new technique called t-SNE that visualizes high-dimensional data by giving each datapoint a location in a two or three-dimensional map, a variation of Stochastic Neighbor Embedding that is much easier to optimize, and produces significantly better visualizations by reducing the tendency to crowd points together in the center of the map.
Proceedings Article
Auto-Encoding Variational Bayes
Diederik P. Kingma,Max Welling +1 more
TL;DR: A stochastic variational inference and learning algorithm that scales to large datasets and, under some mild differentiability conditions, even works in the intractable case is introduced.