Space-Time Distillation for Video Super-Resolution
Zeyu Xiao,Xueyang Fu,Jie Huang,Zhen Cheng,Zhiwei Xiong +4 more
- pp 2113-2122
Reads0
Chats0
TLDR
Wang et al. as mentioned in this paper proposed a space-time distillation (STD) scheme to exploit both spatial and temporal knowledge in the VSR task, which can be easily incorporated into any network without changing the original network architecture.Abstract:Â
Compact video super-resolution (VSR) networks can be easily deployed on resource-limited devices, e.g., smartphones and wearable devices, but have considerable performance gaps compared with complicated VSR networks that require a large amount of computing resources. In this paper, we aim to improve the performance of compact VSR networks without changing their original architectures, through a knowledge distillation approach that transfers knowledge from a complicated VSR network to a compact one. Specifically, we propose a space-time distillation (STD) scheme to exploit both spatial and temporal knowledge in the VSR task. For space distillation, we extract spatial attention maps that hint the high-frequency video content from both networks, which are further used for transferring spatial modeling capabilities. For time distillation, we narrow the performance gap between compact models and complicated models by distilling the feature similarity of the temporal memory cells, which are encoded from the sequence of feature maps generated in the training clips using ConvLSTM. During the training process, STD can be easily incorporated into any network without changing the original network architecture. Experimental results on standard benchmarks demonstrate that, in resource-constrained situations, the proposed method notably improves the performance of existing VSR networks without increasing the inference time.read more
Citations
More filters
Proceedings ArticleDOI
EDPN: Enhanced Deep Pyramid Network for Blurry Image Restoration
TL;DR: Zhou et al. as discussed by the authors proposed an enhanced deep pyramid network (EDPN) for blurry image restoration from multiple degradations, by fully exploiting the self and cross-scale similarities in the degraded image.
Journal ArticleDOI
Video super-resolution based on deep learning: a comprehensive survey
TL;DR: In this paper , a survey of state-of-the-art video super-resolution methods based on deep learning is presented, where the authors propose a taxonomy and classify the methods into seven sub-categories according to the ways of utilizing inter-frame information.
Proceedings ArticleDOI
Joint Video Summarization and Moment Localization by Cross-Task Sample Transfer
TL;DR: This work explores a new solution for video summarization by transferring samples from a correlated task equipped with abundant training data, and proposes an importance Propagation based collaborative Teaching Network (iPTNet), which significantly outperforms previous state-of-the-art video summarizing methods.
Proceedings ArticleDOI
Stereo Video Super-Resolution via Exploiting View-Temporal Correlations
TL;DR: Wang et al. as discussed by the authors proposed a view-temporal attention module (VTAM) to integrate the information of cross-time-cross-view for constructing high-resolution stereo videos.
Book ChapterDOI
MuLUT: Cooperating Multiple Look-Up Tables for Efficient Image Super-Resolution
TL;DR: MuLUT as mentioned in this paper extends SR-LUT by enabling the cooperation of multiple LUTs, termed MuLUT, which achieves a significant improvement up to 1.1 dB PSNR, while preserving its efficiency.
References
More filters
Posted Content
Distilling the Knowledge in a Neural Network
TL;DR: This work shows that it can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model and introduces a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse.
Proceedings Article
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Adam Paszke,Sam Gross,Francisco Massa,Adam Lerer,James Bradbury,Gregory Chanan,Trevor Killeen,Zeming Lin,Natalia Gimelshein,Luca Antiga,Alban Desmaison,Andreas Kopf,Edward Z. Yang,Zachary DeVito,Martin Raison,Alykhan Tejani,Sasank Chilamkurthy,Benoit Steiner,Lu Fang,Junjie Bai,Soumith Chintala +20 more
TL;DR: This paper details the principles that drove the implementation of PyTorch and how they are reflected in its architecture, and explains how the careful and pragmatic implementation of the key components of its runtime enables them to work together to achieve compelling performance.
Journal ArticleDOI
Image Super-Resolution Via Sparse Representation
TL;DR: This paper presents a new approach to single-image superresolution, based upon sparse signal representation, which generates high-resolution images that are competitive or even superior in quality to images produced by other similar SR methods.
Proceedings ArticleDOI
Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network
Wenzhe Shi,Jose Caballero,Ferenc Huszar,Johannes Totz,Andrew Peter Aitken,Rob Bishop,Daniel Rueckert,Zehan Wang +7 more
TL;DR: This paper presents the first convolutional neural network capable of real-time SR of 1080p videos on a single K2 GPU and introduces an efficient sub-pixel convolution layer which learns an array of upscaling filters to upscale the final LR feature maps into the HR output.
Book ChapterDOI
Learning a Deep Convolutional Network for Image Super-Resolution
TL;DR: This work proposes a deep learning method for single image super-resolution (SR) that directly learns an end-to-end mapping between the low/high-resolution images and shows that traditional sparse-coding-based SR methods can also be viewed as a deep convolutional network.
Related Papers (5)
Residual Feature Distillation Network for Lightweight Image Super-Resolution
Jie Liu,Jie Tang,Gangshan Wu +2 more