Journal ArticleDOI
MicroISP: Processing 32MP Photos on Mobile Devices with Deep Learning
Andrey Ignatov,Anastasia Sycheva,Radu Timofte,Yu Hua Nicole Tseng,Yu-Syuan Xu,Po-Hsiang Yu,Cheng-Ming Chiang,Hsien-Kai Kuo,Min-Hung Chen,Chia-Ming Cheng,Luc Van Gool +10 more
- pp 729-746
TLDR
In this paper , the authors presented a novel micro-ISP model designed specifically for edge devices, taking into account their computational and memory limitations, which is capable of processing up to 32MP photos on recent smartphones using the standard mobile ML libraries and requiring less than 1 second to perform the inference.Abstract:
While neural networks-based photo processing solutions can provide a better image quality compared to the traditional ISP systems, their application to mobile devices is still very limited due to their very high computational complexity. In this paper, we present a novel MicroISP model designed specifically for edge devices, taking into account their computational and memory limitations. The proposed solution is capable of processing up to 32MP photos on recent smartphones using the standard mobile ML libraries and requiring less than 1 second to perform the inference, while for FullHD images it achieves real-time performance. The architecture of the model is flexible, allowing to adjust its complexity to devices of different computational power. To evaluate the performance of the model, we collected a novel Fujifilm UltraISP dataset consisting of thousands of paired photos captured with a normal mobile camera sensor and a professional 102MP medium-format FujiFilm GFX100 camera. The experiments demonstrated that, despite its compact size, the MicroISP model is able to provide comparable or better visual results than the traditional mobile ISP systems, while outperforming the previously proposed efficient deep learning based solutions. Finally, this model is also compatible with the latest mobile AI accelerators, achieving good runtime and low power consumption on smartphone NPUs and APUs. The code, dataset and pre-trained models are available on the project website: https://people.ee.ethz.ch/~ihnatova/microisp.htmlread more
Citations
More filters
Journal ArticleDOI
Efficient and Accurate Quantized Image Super-Resolution on Mobile NPUs, Mobile AI & AIM 2022 challenge: Report
Andrey Ignatov,Radu Timofte,Ma. Cristine Faye J. Denna,Abdelbadie Younes,G. Gankhuyag,Jin Huh,Myeong Kyun Kim,Kihwan Yoon,Hyeongjun Moon,Seungho Lee,Yoonsik Choe,Jinwoo Jeong,Sungjei Kim,M Smyl,Tomasz Latkowski,Pawel Kubik,Michał Sokolski,Yu Ma,Jiahao Chao,Zhou Zhou,Hong-Xin Gao,Zhen Yang,Zhenbing Zeng,Zhen-bing Zhuge,Chenghua Li,Dan Zhu,Mengdi Sun,Ran Duan,Yanping Gao,Lingshun Kong,Long Sun,Xing Jian Zhang,Jiawei Zhang,Yaqi Wu,Jinshan Pan,Gao-Xiang Yu,Jin Zhang,Feng Zhang,Zhe Ma,Hongbin Wang,Hojin Cho,Steve Kim,Hua Li,Yan Ma,Ziwei Luo,Youwei Li,Lei Yu,Zhihong Wen,Qi Wu,Haoqiang Fan,Shuaicheng Liu,Lize Zhang,Zhikai Zong,J. Kwon,Junxi Zhang,Meng-Ying Li,N Fu,Guanchen Ding,Han Zhu,Zhen Chen,Gen Li,Li Sun,Dafeng Zhang,Neo Karl Yang,Jerry X. Zhao,Mustafa Ayazoglu,Bahri Batuhan Bilecen,Shota Hirose,Kasidis Arunruangsirilert,Luo Ao,Ho Chun Leung,Andrew Wei,Jie Liu,Qiang Li,Dahai Yu,Ao Li,Lei Luo,Ce Zhu,Seongmin Hong,Dong-Chun Park,Joonhee Lee,Byeong-Hyun Lee,Seunggyu Lee,Sengsub Chun,Ruiyuan He,Xuhao Jiang,Haihang Ruan,Xinjian Zhang,Jing Liu,Garas Gendy,Nabil Sabor,Jin-Long Hou,Guanghui He +92 more
TL;DR: In this article , the authors proposed an efficient quantized image super-resolution solution that can demonstrate a real-time performance on mobile NPUs, which is fully compatible with the above NPU, demonstrating an up to 60 FPS rate when reconstructing Full HD resolution images.
Journal ArticleDOI
Efficient Single-Image Depth Estimation on Mobile Devices, Mobile AI & AIM 2022 Challenge: Report
Andrey Ignatov,Grigory Malivenko,Radu Timofte,Lukasz Treszczotko,Xin-ke Chang,Piotr Ksiazek,Michal Lopuszynski,Maciej Pioro,Rafal Rudnicki,M Smyl,Yujie Ma,Zhenyu Li,Zehui Chen,Jialei Xu,Xianming Liu,Junjun Jiang,Xu Shi,Di Xu,Yanan Li,Xiaotao Wang,Lei Lei,Ziyu Zhang,Yicheng Wang,Zilong Huang,Guozhong Luo,Gang Yu,Bin Fu,Jiaqi Li,Yiran Wang,Zihao Huang,Zhiguo Cao,Marcos V. Conde,D.N. Sapozhnikov,Byeong-Hyun Lee,Dong-Chun Park,Seongmin Hong,Joonhee Lee,Seunggyu Lee,Sengsub Chun +38 more
TL;DR: In this paper , the authors used a large-scale RGB-to-depth dataset that was collected with the ZED stereo camera capable to generate depth maps for objects located at up to 50 meters.
Proceedings ArticleDOI
PyNet-V2 Mobile: Efficient On-Device Photo Processing With Neural Networks
Andrey Ignatov,Grigory Malivenko,Radu Timofte,Yu Hua Nicole Tseng,Yu-Syuan Xu,Po-Hsiang Yu,Cheng-Ming Chiang,Hsien-Kai Kuo,Min-Hung Chen,Chia-Ming Cheng,Luc Van Gool +10 more
TL;DR: Gmalivenko et al. as mentioned in this paper proposed a novel PyNET-V2 Mobile CNN architecture designed specifically for edge devices, being able to process RAW 12MP photos directly on mobile phones under 1.5 seconds and producing high perceptual photo quality.
Journal ArticleDOI
Learned Smartphone ISP on Mobile GPUs with Deep Learning, Mobile AI & AIM 2022 Challenge: Report
Andrey Ignatov,Radu Timofte,Shuai Li,Chaoyu Feng,Furui Bai,Xiaotao Wang,Lei Lei,Ziyao Yi,Yan Xiang,Zibin Liu,Sha Li,Ke Ming Shi,Dehui Kong,Ke Xu,Minsu Kwon,Yaqi Wu,Jiesi Zheng,Zhihao Fan,Xun Wu,Feng Zhang,Albert No,Minhyeok Cho,Zewen Chen,Xiaze Zhang,Ran Li,Juan Wang,Zhiming Wang,Marcos V. Conde,Ui-Jin Choi,Georgy Perevozchikov,Egor I. Ershov,Zheng Hui,Mengchuan Dong,Xin Lou,Wei Zhou,Cong Pang,Haina Qin,Mingxuan Cai +37 more
TL;DR: In this article , the authors developed an end-to-end AI-based image signal processing (ISP) pipeline replacing the standard mobile ISPs that can run on modern smartphone GPUs using TensorFlow Lite.
Journal ArticleDOI
Power Efficient Video Super-Resolution on Mobile NPUs with Deep Learning, Mobile AI & AIM 2022 challenge: Report
Andrey Ignatov,Radu Timofte,Cheng-Ming Chiang,Hsien-Kai Kuo,Yu-Syuan Xu,Man-Yu Lee,A. Lu,Chia-Ming Cheng,Chih-Cheng Chen,Jia-Ying Yong,Hong-Han Shuai,Wen-Huang Cheng,Zhuang Jia,Tianyu Xu,Yijian Zhang,Longnan Bao,Heng Sun,Di Zhang,Sihan Gao,Shaoli Lin,Biao Wu,Xiaofeng Zhang,Cheng-yong Zheng,Kaidi Lu,Ning Wang,Xiaoqing Sun,Hao Chung Wu,Xuncheng Liu,Weizhan Zhang,Caixia Yan,Haipeng Du,Qinghua Zheng,Qi Rong Wang,Wan-Ci Chen,Ran Duan,Mengdi Sun,Dan Zhu,Guannan Chen,Hojin Cho,Steve Kim,Shijie Yue,Chenghua Li,Zhen-bing Zhuge,Wei-Wei Chen,Wenxu Wang,Yufeng Zhou,X. Cai,Hengxing Cai,Kele Xu,Li Liu,Zehua Cheng,Wenyi Lian,W. Lian +52 more
TL;DR: In this paper , the authors proposed an end-to-end real-time video super-resolution solution for mobile NPUs optimized for low energy consumption, using the REDS training dataset containing video sequences for a 4X video upscaling task.
References
More filters
Proceedings Article
Adam: A Method for Stochastic Optimization
Diederik P. Kingma,Jimmy Ba +1 more
TL;DR: This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework.
Journal ArticleDOI
Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
TL;DR: Zhang et al. as mentioned in this paper proposed a feed-forward denoising convolutional neural networks (DnCNNs) to handle Gaussian denobling with unknown noise level.
Posted Content
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
TL;DR: This work considers image transformation problems, and proposes the use of perceptual loss functions for training feed-forward networks for image transformation tasks, and shows results on image style transfer, where aFeed-forward network is trained to solve the optimization problem proposed by Gatys et al. in real-time.
Posted Content
Accurate Image Super-Resolution Using Very Deep Convolutional Networks
TL;DR: This work presents a highly accurate single-image superresolution (SR) method using a very deep convolutional network inspired by VGG-net used for ImageNet classification and uses extremely high learning rates enabled by adjustable gradient clipping.
Book ChapterDOI
ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks
TL;DR: ESRGAN as mentioned in this paper improves the perceptual loss by using the features before activation, which could provide stronger supervision for brightness consistency and texture recovery, and won the first place in the PIRM2018-SR Challenge (region 3).