Generating the Future with Adversarial Transformers
Carl Vondrick,Antonio Torralba +1 more
- pp 2992-3000
TLDR
This work presents a model that generates the future by transforming pixels in the past, and explicitly disentangles the models memory from the prediction, which helps the model learn desirable invariances.Abstract:
We learn models to generate the immediate future in video. This problem has two main challenges. Firstly, since the future is uncertain, models should be multi-modal, which can be difficult to learn. Secondly, since the future is similar to the past, models store low-level details, which complicates learning of high-level semantics. We propose a framework to tackle both of these challenges. We present a model that generates the future by transforming pixels in the past. Our approach explicitly disentangles the models memory from the prediction, which helps the model learn desirable invariances. Experiments suggest that this model can generate short videos of plausible futures. We believe predictive models have many applications in robotics, health-care, and video understanding.read more
Citations
More filters
Proceedings ArticleDOI
Future Frame Prediction for Anomaly Detection - A New Baseline
TL;DR: In this article, Liu et al. propose to detect abnormal events by enforcing the optical flow between predicted frames and ground truth frames to be consistent, and this is the first work that introduces a temporal constraint into the video prediction task.
Posted Content
Stochastic Adversarial Video Prediction
TL;DR: This work shows that latent variational variable models that explicitly model underlying stochasticity and adversarially-trained models that aim to produce naturalistic images are in fact complementary and combines the two to produce predictions that look more realistic to human raters and better cover the range of possible futures.
Posted Content
Video-to-Video Synthesis
TL;DR: In this article, a video-to-video synthesis approach under the generative adversarial learning framework is proposed, which achieves high-resolution, photorealistic, temporally coherent video results on a diverse set of input formats.
Proceedings Article
Stochastic Variational Video Prediction
TL;DR: In this paper, a stochastic variational video prediction (SV2P) method is proposed to predict a different possible future for each sample of its latent variables for real-world video.
Book ChapterDOI
R2P2: A ReparameteRized Pushforward Policy for Diverse, Precise Generative Path Forecasting
TL;DR: A method to forecast a vehicle’s ego-motion as a distribution over spatiotemporal paths, conditioned on features embedded in an overhead map, and obtains expressions for the cross-entropy metrics that can be efficiently evaluated and differentiated, enabling stochastic-gradient optimization.
References
More filters
Proceedings Article
Adam: A Method for Stochastic Optimization
Diederik P. Kingma,Jimmy Ba +1 more
TL;DR: This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework.
Journal ArticleDOI
Generative Adversarial Nets
Ian Goodfellow,Jean Pouget-Abadie,Mehdi Mirza,Bing Xu,David Warde-Farley,Sherjil Ozair,Aaron Courville,Yoshua Bengio +7 more
TL;DR: A new framework for estimating generative models via an adversarial process, in which two models are simultaneously train: a generative model G that captures the data distribution and a discriminative model D that estimates the probability that a sample came from the training data rather than G.
Journal ArticleDOI
The Pascal Visual Object Classes (VOC) Challenge
TL;DR: The state-of-the-art in evaluated methods for both classification and detection are reviewed, whether the methods are statistically different, what they are learning from the images, and what the methods find easy or confuse.
Posted Content
Conditional Generative Adversarial Nets
Mehdi Mirza,Simon Osindero +1 more
TL;DR: The conditional version of generative adversarial nets is introduced, which can be constructed by simply feeding the data, y, to the generator and discriminator, and it is shown that this model can generate MNIST digits conditioned on class labels.
Posted Content
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
TL;DR: This work introduces a class of CNNs called deep convolutional generative adversarial networks (DCGANs), that have certain architectural constraints, and demonstrates that they are a strong candidate for unsupervised learning.