Video Frame Interpolation via Adaptive Convolution
Simon Niklaus,Long Mai,Feng Liu +2 more
- pp 2270-2279
Reads0
Chats0
TLDR
In this paper, a deep fully convolutional neural network is proposed to estimate a spatially-adaptive convolution kernel for each pixel, which captures both the local motion between the input frames and the coefficients for pixel synthesis.Abstract:
Video frame interpolation typically involves two steps: motion estimation and pixel synthesis. Such a two-step approach heavily depends on the quality of motion estimation. This paper presents a robust video frame interpolation method that combines these two steps into a single process. Specifically, our method considers pixel synthesis for the interpolated frame as local convolution over two input frames. The convolution kernel captures both the local motion between the input frames and the coefficients for pixel synthesis. Our method employs a deep fully convolutional neural network to estimate a spatially-adaptive convolution kernel for each pixel. This deep neural network can be directly trained end to end using widely available video data without any difficult-to-obtain ground-truth data like optical flow. Our experiments show that the formulation of video interpolation as a single convolution process allows our method to gracefully handle challenges like occlusion, blur, and abrupt brightness change and enables high-quality video frame interpolation.read more
Citations
More filters
Proceedings ArticleDOI
Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation
TL;DR: In this paper, an end-to-end convolutional neural network is proposed for variable-length multi-frame video interpolation, where the motion interpretation and occlusion reasoning are jointly modeled.
Proceedings ArticleDOI
Video Frame Interpolation via Adaptive Separable Convolution
Simon Niklaus,Long Mai,Feng Liu +2 more
TL;DR: In this article, a deep fully convolutional neural network is proposed to estimate pairs of 1D kernels for all pixels simultaneously, which allows for the incorporation of perceptual loss to train the network to produce visually pleasing frames.
Journal ArticleDOI
Video Enhancement with Task-Oriented Flow
TL;DR: Task-Oriented Flow (TOFlow) as mentioned in this paper is a self-supervised, task-specific representation for low-level video processing, which is trained in a supervised manner.
Journal ArticleDOI
Video Enhancement with Task-Oriented Flow
TL;DR: T task-oriented flow (TOFlow), a motion representation learned in a self-supervised, task-specific manner, is proposed, which outperforms traditional optical flow on standard benchmarks as well as the Vimeo-90K dataset in three video processing tasks.
Proceedings ArticleDOI
Burst Denoising with Kernel Prediction Networks
TL;DR: In this paper, a convolutional neural network architecture is proposed for predicting spatially varying kernels that can both align and denoise frames, and a synthetic data generation approach based on a realistic noise formation model, and an optimization guided by an annealed loss function to avoid undesirable local minima.
References
More filters
Proceedings ArticleDOI
Deep Residual Learning for Image Recognition
TL;DR: In this article, the authors proposed a residual learning framework to ease the training of networks that are substantially deeper than those used previously, which won the 1st place on the ILSVRC 2015 classification task.
Proceedings Article
ImageNet Classification with Deep Convolutional Neural Networks
TL;DR: The state-of-the-art performance of CNNs was achieved by Deep Convolutional Neural Networks (DCNNs) as discussed by the authors, which consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax.
Proceedings Article
Very Deep Convolutional Networks for Large-Scale Image Recognition
Karen Simonyan,Andrew Zisserman +1 more
TL;DR: This work investigates the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting using an architecture with very small convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers.
Proceedings Article
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Sergey Ioffe,Christian Szegedy +1 more
TL;DR: Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin.
Proceedings ArticleDOI
Fully convolutional networks for semantic segmentation
TL;DR: The key insight is to build “fully convolutional” networks that take input of arbitrary size and produce correspondingly-sized output with efficient inference and learning.