Divide and Conquer for Full-Resolution Light Field Deblurring

doi:10.1109/CVPR.2018.00672

Home
/
Papers
/
Divide and Conquer for Full-Resolution Light Field Deblurring

Proceedings Article•DOI•

Divide and Conquer for Full-Resolution Light Field Deblurring

M R Mahesh Mohan¹, A. N. Rajagopalan¹•Institutions (1)

Indian Institute of Technology Madras¹

01 Jun 2018-pp 6421-6429

TL;DR: A new blind motion deblurring strategy for LFs which alleviates limitations significantly and is CPU-efficient computationally and can effectively deblur full-resolution LFs.

read less

Abstract: The increasing popularity of computational light field (LF) cameras has necessitated the need for tackling motion blur which is a ubiquitous phenomenon in hand-held photography. The state-of-the-art method for blind deblurring of LFs of general 3D scenes is limited to handling only downsampled LF, both in spatial and angular resolution. This is due to the computational overhead involved in processing data-hungry full-resolution 4D LF altogether. Moreover, the method warrants high-end GPUs for optimization and is ineffective for wide-angle settings and irregular camera motion. In this paper, we introduce a new blind motion deblurring strategy for LFs which alleviates these limitations significantly. Our model achieves this by isolating 4D LF motion blur across the 2D subaperture images, thus paving the way for independent deblurring of these subaperture images. Furthermore, our model accommodates common camera motion parameterization across the subaperture images. Consequently, blind deblurring of any single subaperture image elegantly paves the way for cost-effective non-blind deblurring of the other subaperture images. Our approach is CPU-efficient computationally and can effectively deblur full-resolution LFs.

...read moreread less

Content maybe subject to copyright Report

Citations

PDF

Open Access

More filters

Journal Article•DOI•

A multi-aperture optical flow estimation method for an artificial compound eye

[...]

Sidong Wu¹, Gexiang Zhang¹, Ferrante Neri², Ming Zhu³, Tao Jiang³, Klaus-Dieter Kuhnert⁴ - Show less +2 more•Institutions (4)

Southwest Jiaotong University¹, De Montfort University², Chengdu University of Information Technology³, University of Siegen⁴

01 Jan 2019-Integrated Computer-aided Engineering

13 citations

Journal Article•DOI•

Fast and Full-Resolution Light Field Deblurring Using a Deep Neural Network

[...]

Jonathan Samuel Lumentut¹, Tae Hyun Kim, Ravi Ramamoorthi², In Kyu Park¹•Institutions (2)

Inha University¹, University of California, San Diego²

31 Mar 2019-arXiv: Computer Vision and Pattern Recognition

TL;DR: This work generates a complex blurry light field dataset and proposes a learning-based deblurring approach that is about 16K times faster than Srinivasan et.

...read moreread less

Abstract: Restoring a sharp light field image from its blurry input has become essential due to the increasing popularity of parallax-based image processing. State-of-the-art blind light field deblurring methods suffer from several issues such as slow processing, reduced spatial size, and a limited motion blur model. In this work, we address these challenging problems by generating a complex blurry light field dataset and proposing a learning-based deblurring approach. In particular, we model the full 6-degree of freedom (6-DOF) light field camera motion, which is used to create the blurry dataset using a combination of real light fields captured with a Lytro Illum camera, and synthetic light field renderings of 3D scenes. Furthermore, we propose a light field deblurring network that is built with the capability of large receptive fields. We also introduce a simple strategy of angular sampling to train on the large-scale blurry light field effectively. We evaluate our method through both quantitative and qualitative measurements and demonstrate superior performance compared to the state-of-the-art method with a massive speedup in execution time. Our method is about 16K times faster than Srinivasan et. al. [22] and can deblur a full-resolution light field in less than 2 seconds.

...read moreread less

12 citations

Cites background or methods from "Divide and Conquer for Full-Resolut..."

...These problems were solved partially by recent LF deblurring works [14, 16] but are still inapplicable on any LF camera as the post-capture processing, due to their slow execution time (∼30 minutes)....
[...]
...Although their model was able to produce a better result than the state-of-the-art [22], the MDF model did not include out-of-plane translation (z-axis translation)....
[...]
...This problem is solved by Mahesh Mohan and Rajagopalan [16] who implemented 2-DOF in-plane translation and 1-DOF z-axis rotation model (3-DOF) following the model of motion density function (MDF) [6]....
[...]
...Our method is designed to address the limitation of previous works that assume 3-DOF translational [22] and 3-DOF motion density function (MDF) [16]....
[...]
...The blur model is designed within 6-DOF motion as opposed to the 3-DOF model from previous approaches [16, 22]....
[...]

Proceedings Article•DOI•

Unconstrained Motion Deblurring for Dual-Lens Cameras

[...]

Mahesh Mohan M R¹, Sharath Girish¹, Rajagopalan Ambasamudram¹•Institutions (1)

Indian Institute of Technology Madras¹

01 Oct 2019

TL;DR: A generalized blur model is proposed that elegantly explains the intrinsically coupled image formation model for dual-lens set-up, which are by far most predominant in smartphones and reveals an intriguing challenge that stems from an inherent ambiguity unique to this problem which naturally disrupts this coherence.

...read moreread less

Abstract: Recently, there has been a renewed interest in leveraging multiple cameras, but under unconstrained settings. They have been quite successfully deployed in smartphones, which have become de facto choice for many photographic applications. However, akin to normal cameras, the functionality of multi-camera systems can be marred by motion blur which is a ubiquitous phenomenon in hand-held cameras. Despite the far-reaching potential of unconstrained camera arrays, there is not a single deblurring method for such systems. In this paper, we propose a generalized blur model that elegantly explains the intrinsically coupled image formation model for dual-lens set-up, which are by far most predominant in smartphones. While image aesthetics is the main objective in normal camera deblurring, any method conceived for our problem is additionally tasked with ascertaining consistent scene-depth in the deblurred images. We reveal an intriguing challenge that stems from an inherent ambiguity unique to this problem which naturally disrupts this coherence. We address this issue by devising a judicious prior, and based on our model and prior propose a practical blind deblurring method for dual-lens cameras, that achieves state-of-the-art performance.

...read moreread less

8 citations

Cites background or methods from "Divide and Conquer for Full-Resolut..."

...Second, any method for DL-BMD must ensure scene-consistent disparities in the deblurred imagepair (akin to angular coherence in light fields [23, 40]), which also incidentally opens up many potential applications [14, 29, 37, 24]....
[...]
...Following [23, 26, 42, 52, 48], we consider a blurred image as the integration of rotation-induced projections of world over the exposure time, the rotations being caused by camera shake, but do not constrain the COR to be only at the optical center....
[...]
...For the case of light field cameras, existing methods constrain all multi-view images to share identical camera settings and ego-motions [18, 5, 23, 40]....
[...]
...Also, the imaging principle of light field is quite different due to the lens effect [5, 23]....
[...]
...For computational cameras, we considered state-of-the-art stereo BMD [51] and light field BMD [23]....
[...]

Journal Article•DOI•

6-DOF motion blur synthesis and performance evaluation of light field deblurring

[...]

Jonathan Samuel Lumentut¹, Williem², In Kyu Park¹•Institutions (2)

Inha University¹, Binus University²

01 Dec 2019-Multimedia Tools and Applications

TL;DR: The experiment results show that the proposed blur model can maintain the parallax information (depth-dependent blur) in a light field image and produce a synthetic blurry light field dataset based on the 6-DOF model.

...read moreread less

Abstract: Motion deblurring is essential for reconstructing sharp images from given a blurry input caused by the camera motion. The complexity of this problem increases in a light field due to its depth-dependent blur constraint. A method of generating synthetic 3 degree-of-freedom (3-DOF) translation blur on a light field image without camera rotation has been introduced. In this study, we generate a camera translation and rotation (6-DOF) motion blur model that preserves the consistency of the light field image. Our experiment results show that the proposed blur model can maintain the parallax information (depth-dependent blur) in a light field image. Furthermore, we produce a synthetic blurry light field dataset based on the 6-DOF model. Finally, to validate the usability of the synthetic dataset, we conduct extensive benchmarking using state-of-the-art motion deblurring algorithms.

...read moreread less

3 citations

Additional excerpts

...To the best of our knowledge, only few previous studies that work on light field deblurring [14, 18, 22] and surprisingly, no previous studies that work on generating blurry light field dataset....
[...]

Journal Article•DOI•

Learning a Degradation-Adaptive Network for Light Field Image Super-Resolution

[...]

Yingqian Wang, Zhengyu Liang, Longguang Wang, Jungang Yang, Wei An, Yulan Guo - Show less +2 more

13 Jun 2022-arXiv.org

TL;DR: Compared with existing state-of- the-art single and LF image SR methods, the proposed LF-DAnet method achieves superior SR performance under a wide range of degradations, and generalizes better to real LF images.

...read moreread less

Abstract: —Recent years have witnessed the great advances of deep neural networks (DNNs) in light ﬁeld (LF) image super-resolution (SR). However, existing DNN-based LF image SR methods are developed on a single ﬁxed degradation (e.g., bicubic downsampling), and thus cannot be applied to super-resolve real LF images with diverse degradations. In this paper, we propose the ﬁrst method to handle LF image SR with multiple degradations. In our method, a practical LF degradation model that considers blur and noise is developed to approximate the degradation process of real LF images. Then, a degradation-adaptive network (LF-DAnet) is designed to incorporate the degradation prior into the SR process. By training on LF images with multiple synthetic degradations, our method can learn to adapt to different degradations while incorporating the spatial and angular information. Extensive experiments on both synthetically degraded and real-world LFs demonstrate the effectiveness of our method. Compared with existing state-of- the-art single and LF image SR methods, our method achieves superior SR performance under a wide range of degradations, and generalizes better to real LF images. Codes and models are available at https://github.com/YingqianWang/LF-DAnet.

...read moreread less

2 citations

References

PDF

Open Access

More filters

Proceedings Article•DOI•

Image and depth from a conventional camera with a coded aperture

[...]

Anat Levin, Rob Fergus, Frédo Durand, William T. Freeman

29 Jul 2007

TL;DR: A simple modification to a conventional camera is proposed to insert a patterned occluder within the aperture of the camera lens, creating a coded aperture, and introduces a criterion for depth discriminability which is used to design the preferred aperture pattern.

...read moreread less

Abstract: A conventional camera captures blurred versions of scene information away from the plane of focus. Camera systems have been proposed that allow for recording all-focus images, or for extracting depth, but to record both simultaneously has required more extensive hardware and reduced spatial resolution. We propose a simple modification to a conventional camera that allows for the simultaneous recovery of both (a) high resolution image information and (b) depth information adequate for semi-automatic extraction of a layered depth representation of the image. Our modification is to insert a patterned occluder within the aperture of the camera lens, creating a coded aperture. We introduce a criterion for depth discriminability which we use to design the preferred aperture pattern. Using a statistical model of images, we can recover both depth information and an all-focus image from single photographs taken with the modified camera. A layered depth map is then extracted, requiring user-drawn strokes to clarify layer assignments in some cases. The resulting sharp image and layered depth map can be combined for various photographic applications, including automatic scene segmentation, post-exposure refocusing, or re-rendering of the scene from an alternate viewpoint.

...read moreread less

1,489 citations

Journal Article•DOI•

An information fidelity criterion for image quality assessment using natural scene statistics

[...]

Hamid R. Sheikh¹, Alan C. Bovik¹, G. de Veciana¹•Institutions (1)

University of Texas at Austin¹

01 Dec 2005-IEEE Transactions on Image Processing

TL;DR: This paper proposes a novel information fidelity criterion that is based on natural scene statistics and derives a novel QA algorithm that provides clear advantages over the traditional approaches and outperforms current methods in testing.

...read moreread less

Abstract: Measurement of visual quality is of fundamental importance to numerous image and video processing applications. The goal of quality assessment (QA) research is to design algorithms that can automatically assess the quality of images or videos in a perceptually consistent manner. Traditionally, image QA algorithms interpret image quality as fidelity or similarity with a "reference" or "perfect" image in some perceptual space. Such "full-reference" QA methods attempt to achieve consistency in quality prediction by modeling salient physiological and psychovisual features of the human visual system (HVS), or by arbitrary signal fidelity criteria. In this paper, we approach the problem of image QA by proposing a novel information fidelity criterion that is based on natural scene statistics. QA systems are invariably involved with judging the visual quality of images and videos that are meant for "human consumption". Researchers have developed sophisticated models to capture the statistics of natural signals, that is, pictures and videos of the visual environment. Using these statistical models in an information-theoretic setting, we derive a novel QA algorithm that provides clear advantages over the traditional approaches. In particular, it is parameterless and outperforms current methods in our testing. We validate the performance of our algorithm with an extensive subjective study involving 779 images. We also show that, although our approach distinctly departs from traditional HVS-based methods, it is functionally similar to them under certain conditions, yet it outperforms them due to improved modeling. The code and the data from the subjective study are available at [1].

...read moreread less

1,334 citations

"Divide and Conquer for Full-Resolut..." refers background or methods in this paper

...Using IFC/VIF, Figs....
[...]
...Quantitative Evaluation: We introduce an LF-version of information fidelity criterion (IFC) [19] and visual information fidelity (VIF) [18], which are shown to be the best metrics for BMD evaluation in [13], by averaging these metric over subaperture images....
[...]
...(a) LF-version of IFC [19] (b) LF-version of VIF [18]...
[...]

Journal Article•DOI•

Single lens stereo with a plenoptic camera

[...]

Edward H. Adelson¹, J.Y.A. Wang¹•Institutions (1)

Massachusetts Institute of Technology¹

01 Feb 1992-IEEE Transactions on Pattern Analysis and Machine Intelligence

TL;DR: The authors describe a camera for performing single lens stereo analysis, which incorporates a single main lens along with a lenticular array placed at the sensor plane and extracts information about both horizontal and vertical parallax, which improves the reliability of the depth estimates.

...read moreread less

Abstract: Ordinary cameras gather light across the area of their lens aperture, and the light striking a given subregion of the aperture is structured somewhat differently than the light striking an adjacent subregion. By analyzing this optical structure, one can infer the depths of the objects in the scene, i.e. one can achieve single lens stereo. The authors describe a camera for performing this analysis. It incorporates a single main lens along with a lenticular array placed at the sensor plane. The resulting plenoptic camera provides information about how the scene would look when viewed from a continuum of possible viewpoints bounded by the main lens aperture. Deriving depth information is simpler than in a binocular stereo system because the correspondence problem is minimized. The camera extracts information about both horizontal and vertical parallax, which improves the reliability of the depth estimates. >

...read moreread less

1,229 citations

"Divide and Conquer for Full-Resolut..." refers background in this paper

...The increase in popularity of LFCs can be attributed to their attractive features over conventional cameras (CCs), including post-capture refocusing, f-stopping, depth sensing [22, 1, 16], etc....
[...]
...LFCs achieve this by capturing multiple (subaperture) images instead of a single CC image by segregating the light reaching the CC-sensor into multiple angular components; and synthesize these images post-capture to form an image of desired CC setting [16, 1]....
[...]

Proceedings Article•

Fast Image Deconvolution using Hyper-Laplacian Priors

[...]

Dilip Krishnan¹, Rob Fergus¹•Institutions (1)

New York University¹

07 Dec 2009

TL;DR: This paper describes a deconvolution approach that is several orders of magnitude faster than existing techniques that use hyper-Laplacian priors and is able to deconvolve a 1 megapixel image in less than ~3 seconds, achieving comparable quality to existing methods that take ~20 minutes.

...read moreread less

Abstract: The heavy-tailed distribution of gradients in natural scenes have proven effective priors for a range of problems such as denoising, deblurring and super-resolution. These distributions are well modeled by a hyper-Laplacian (p(x) ∝ e-k|x|α ), typically with 0.5 ≤ α ≤ 0.8. However, the use of sparse distributions makes the problem non-convex and impractically slow to solve for multi-megapixel images. In this paper we describe a deconvolution approach that is several orders of magnitude faster than existing techniques that use hyper-Laplacian priors. We adopt an alternating minimization scheme where one of the two phases is a non-convex problem that is separable over pixels. This per-pixel sub-problem may be solved with a lookup table (LUT). Alternatively, for two specific values of α, 1/2 and 2/3 an analytic solution can be found, by finding the roots of a cubic and quartic polynomial, respectively. Our approach (using either LUTs or analytic formulae) is able to deconvolve a 1 megapixel image in less than ~3 seconds, achieving comparable quality to existing methods such as iteratively reweighted least squares (IRLS) that take ~20 minutes. Furthermore, our method is quite general and can easily be extended to related image processing problems, beyond the deconvolution application demonstrated.

...read moreread less

1,226 citations

"Divide and Conquer for Full-Resolut..." refers methods in this paper

...Deconvolution method Direct (Gaussian) [5] (Fast hyperLaplacian) [6] (0....
[...]
...(c) Direct approach using Gaussian prior, (d) Fast MAP estimation with hyper-Laplacian prior using lookup table [5], (e) MAP estimation with heavy-tailed prior (α = 0....
[...]
...8) which is solved using a lookup table [5], (c) A heavy-tailed prior (α = 0....
[...]
...(d) [5] (Hyper-Laplacian prior) (e) [6] (0....
[...]

Proceedings Article•DOI•

Blind deconvolution using a normalized sparsity measure

[...]

Dilip Krishnan¹, Terence Tay, Rob Fergus¹•Institutions (1)

New York University¹

20 Jun 2011

TL;DR: A new type of image regularization which gives lowest cost for the true sharp image is introduced, which allows a very simple cost formulation to be used for the blind deconvolution model, obviating the need for additional methods.

...read moreread less

Abstract: Blind image deconvolution is an ill-posed problem that requires regularization to solve. However, many common forms of image prior used in this setting have a major drawback in that the minimum of the resulting cost function does not correspond to the true sharp solution. Accordingly, a range of additional methods are needed to yield good results (Bayesian methods, adaptive cost functions, alpha-matte extraction and edge localization). In this paper we introduce a new type of image regularization which gives lowest cost for the true sharp image. This allows a very simple cost formulation to be used for the blind deconvolution model, obviating the need for additional methods. Due to its simplicity the algorithm is fast and very robust. We demonstrate our method on real images with both spatially invariant and spatially varying blur.

...read moreread less

1,054 citations

"Divide and Conquer for Full-Resolut..." refers background or methods in this paper

...As the effect of translation is negligible for normal camera-shakes [23], 3D rotation-only approximation is commonly employed in CCs....
[...]
...• Our work bridges the gap between the well-studied CC-BMD and emerging LFC-BMD, and facilitates mapping of analogous techniques (such as MDF formulation, efficient filter flow framework, and scalespace strategy) developed for the former to the later....
[...]
...State-of-the-art CC-BMD methods [17, 26, 21] are based on the motion density function (MDF) [5] which allows both narrow- and wide-angle systems as well as nonparametric camera motion, have a homography-based filter flow framework for computational efficiency [8], and employ a scale-space approach to accommodate large blurs....
[...]
...We now briefly digress to discuss the MDF model employed in the state-of-the-art CC-BMD [17, 26, 23]....
[...]
...State-of-the-art CC-BMD methods proceed by alternating minimization of ωλ and I in a scalespace manner to accommodate large blurs (i.e., MDF estimation starts with a downsampled blurred image where the motion blur is less prominent, and proceeds to finer scale MDF-estimation using the previous estimate)....
[...]