scispace - formally typeset
Search or ask a question
Journal ArticleDOI

Deep Recurrent Network for Fast and Full-Resolution Light Field Deblurring

TL;DR: In this paper, a light field recurrent deblurring network was proposed to recover sharp light field from its blurry input. But the deblurability of the deblated light field was not improved.
Abstract: The popularity of parallax-based image processing is increasing while in contrast early works on recovering sharp light field from its blurry input (deblurring) remain stagnant. State-of-the-art blind light field deblurring methods suffer from several problems such as slow processing, reduced spatial size, and simplified motion blur model. In this paper, we solve these challenging problems by proposing a novel light field recurrent deblurring network that is trained under 6 degree-of-freedom camera motion-blur model. By combining the real light field captured using Lytro Illum and synthetic light field rendering of 3D scenes from UnrealCV, we provide a large-scale blurry light field dataset to train the network. The proposed method outperforms the state-of-the-art methods in terms of deblurring quality, the capability of handling full-resolution, and a fast runtime.
Citations
More filters
Journal ArticleDOI
TL;DR: A comprehensive and timely survey of recently published deep-learning based image deblurring approaches can be found in this article , where the authors discuss common causes of image blur, introduce benchmark datasets and performance metrics, and summarize different problem formulations.
Abstract: Image deblurring is a classic problem in low-level computer vision with the aim to recover a sharp image from a blurred input image. Advances in deep learning have led to significant progress in solving this problem, and a large number of deblurring networks have been proposed. This paper presents a comprehensive and timely survey of recently published deep-learning based image deblurring approaches, aiming to serve the community as a useful literature review. We start by discussing common causes of image blur, introduce benchmark datasets and performance metrics, and summarize different problem formulations. Next, we present a taxonomy of methods using convolutional neural networks (CNN) based on architecture, loss function, and application, offering a detailed review and comparison. In addition, we discuss some domain-specific deblurring applications including face images, text, and stereo image pairs. We conclude by discussing key challenges and future research directions.

65 citations

Journal ArticleDOI
TL;DR: In this article, an on-the-fly blurry data augmenter that can be run during training and test stages is proposed. And the proposed deblurring module is also equipped with hand-crafted prior extracted using the state-of-theart human body statistical model.
Abstract: Present-day deep learning-based motion deblurring methods utilize the pair of synthetic blur and sharp data to regress any particular framework. This task is designed for directly translating a blurry image input into its restored version as output. The aforementioned approach relies heavily on the quality of the synthetic blurry data, which are only available before the training stage. Handling this issue by providing a large amount of data is expensive for common usage. We answer this challenge by providing an on- the-fly blurry data augmenter that can be run during training and test stages. To fully utilize it, we incorporate an unorthodox scheme of deblurring framework that employs the sequence of blur-deblur-reblur-deblur steps. The reblur step is assisted by a reblurring module (synthesizer) that provides the reblurred version (pseudo-blur) of its sharp or deblurred counterpart. The proposed module is also equipped with hand-crafted prior extracted using the state-of-the-art human body statistical model. This prior is employed to map human and non-human regions during adversarial learning to fully perceive the characteristics of human-articulated and scene motion blurs. By engaging this approach, our deblurring module becomes adaptive and achieves superior outcomes compared to recent state-of-the-art deblurring algorithms.

2 citations

Book ChapterDOI
30 Nov 2020
TL;DR: This work constructs a localized adversarial framework that solves both human-articulated and camera motion blurs and generates a novel dataset that simulates realistic blurry human motion while maintaining the presence of camera motion.
Abstract: In recent decades, the skinned multi-person linear model (SMPL) is widely exploited in the image-based 3D body reconstruction. This model, however, depends fully on the quality of the input image. Degraded image case, such as the motion-blurred issue, downgrades the quality of the reconstructed 3D body. This issue becomes severe as recent motion deblurring methods mainly focused on solving the camera motion case while ignoring the blur caused by human-articulated motion. In this work, we construct a localized adversarial framework that solves both human-articulated and camera motion blurs. To achieve this, we utilize the result of the restored image in a 3D body reconstruction module and produces a localized map. The map is employed to guide the adversarial modules on learning both the human body and scene regions. Nevertheless, training these modules straight-away is impractical since the recent blurry dataset is not supported by the 3D body predictor module. To settle this issue, we generate a novel dataset that simulates realistic blurry human motion while maintaining the presence of camera motion. By engaging this dataset and the proposed framework, we show that our deblurring results are superior among the state-of-the-art algorithms in both quantitative and qualitative performances.

2 citations

Journal ArticleDOI
13 Oct 2022
TL;DR: The idempotent constraint is introduced into the deblurring framework and a deep idem Potent network is presented to achieve improved blind non-uniform deblurred performance with stable re-deblurring.
Abstract: Single image blind deblurring is highly ill-posed as neither the latent sharp image nor the blur kernel is known. Even though considerable progress has been made, several major difficulties remain for blind deblurring, including the trade-off between high-performance deblurring and real-time processing. Besides, we observe that current single image blind deblurring networks cannot further improve or stabilize the performance but significantly degrades the performance when re-deblurring is repeatedly applied. This implies the limitation of these networks in modeling an ideal deblurring process. In this work, we make two contributions to tackle the above difficulties: (1) We introduce the idempotent constraint into the deblurring framework and present a deep idempotent network to achieve improved blind non-uniform deblurring performance with stable re-deblurring. (2) We propose a simple yet efficient deblurring network with lightweight encoder-decoder units and a recurrent structure that can deblur images in a progressive residual fashion. Extensive experiments on synthetic and realistic datasets prove the superiority of our proposed framework. Remarkably, our proposed network is nearly $6.5\times $ smaller and $6.4\times $ faster than the state-of-the-art while achieving comparable high performance.

1 citations

Journal ArticleDOI
TL;DR: Zhang et al. as discussed by the authors proposed a light field occlusion removal network (LFORNet), which consists of three key sub-networks: the foreground occlusions location (FOL), the background content recovery (BCR), and the refinement sub-network.
Abstract: Occlusion removal in an image can aid in facilitating the robustness of numerous computer vision tasks, e . g ., detection and tracking in surveillance. However, the invisible property of contents behind the occlusions limits occlusion removal from the single view. Recently, the emerging light field (LF) data, which contains rich multi-view perception of the scene, provides potential solution for this challenge. To better exploit the capability of occlusion location and occluded contents recovery from LF data, in this paper, we propose a LF occlusion removal network (LFORNet), which consists of three key sub-networks: the foreground occlusion location (FOL) sub-network, the background content recovery (BCR) sub-network, and the refinement sub-network. Specifically, both FOL sub-network and BCR sub-network explore the multi-view information of LF data, and thus they are constructed with the same network structure to estimate the occlusion mask and the coarse occluded contents map, respectively. The refinement sub-network aggregates the above two outputs to obtain the refined occlusion removal. Meanwhile, we use multi-angle view stacks as the input of the network, which can make full use of the inherent information among the LF views. Experimental results show that our method is suitable for different sizes of occlusions, and surpasses the state-of-the-art approaches in both synthetic and real-world scenes. • A light field de-occlusion network was proposed. • The network breaks the main task into several sub-tasks. • This network can extract occlusion mask while removing occlusion task. • Experimental results show that our method achieves superior performances. counterparts.

1 citations

References
More filters
Posted Content
TL;DR: A small change in the stylization architecture results in a significant qualitative improvement in the generated images, and can be used to train high-performance architectures for real-time image generation.
Abstract: It this paper we revisit the fast stylization method introduced in Ulyanov et. al. (2016). We show how a small change in the stylization architecture results in a significant qualitative improvement in the generated images. The change is limited to swapping batch normalization with instance normalization, and to apply the latter both at training and testing times. The resulting method can be used to train high-performance architectures for real-time image generation. The code will is made available on github at this https URL. Full paper can be found at arXiv:1701.02096.

3,118 citations

Book ChapterDOI
05 Sep 2010
TL;DR: It is found that strong edges do not always profit kernel estimation, but instead under certain circumstance degrade it, which leads to a new metric to measure the usefulness of image edges in motion deblurring and a gradient selection process to mitigate their possible adverse effect.
Abstract: We discuss a few new motion deblurring problems that are significant to kernel estimation and non-blind deconvolution. We found that strong edges do not always profit kernel estimation, but instead under certain circumstance degrade it. This finding leads to a new metric to measure the usefulness of image edges in motion deblurring and a gradient selection process to mitigate their possible adverse effect. We also propose an efficient and high-quality kernel estimation method based on using the spatial prior and the iterative support detection (ISD) kernel refinement, which avoids hard threshold of the kernel elements to enforce sparsity. We employ the TV-l1 deconvolution model, solved with a new variable substitution scheme to robustly suppress noise.

1,056 citations

Proceedings ArticleDOI
20 Jun 2011
TL;DR: A new type of image regularization which gives lowest cost for the true sharp image is introduced, which allows a very simple cost formulation to be used for the blind deconvolution model, obviating the need for additional methods.
Abstract: Blind image deconvolution is an ill-posed problem that requires regularization to solve. However, many common forms of image prior used in this setting have a major drawback in that the minimum of the resulting cost function does not correspond to the true sharp solution. Accordingly, a range of additional methods are needed to yield good results (Bayesian methods, adaptive cost functions, alpha-matte extraction and edge localization). In this paper we introduce a new type of image regularization which gives lowest cost for the true sharp image. This allows a very simple cost formulation to be used for the blind deconvolution model, obviating the need for additional methods. Due to its simplicity the algorithm is fast and very robust. We demonstrate our method on real images with both spatially invariant and spatially varying blur.

1,054 citations

Journal ArticleDOI
Ren Ng1
01 Jul 2005
TL;DR: A theorem that, in the Fourier domain, a photograph formed by a full lens aperture is a 2D slice in the 4D light field is demonstrated, which yields a Fourier-domain algorithm for digital refocusing.
Abstract: This paper contributes to the theory of photograph formation from light fields. The main result is a theorem that, in the Fourier domain, a photograph formed by a full lens aperture is a 2D slice in the 4D light field. Photographs focused at different depths correspond to slices at different trajectories in the 4D space. The paper demonstrates the utility of this theorem in two different ways. First, the theorem is used to analyze the performance of digital refocusing, where one computes photographs focused at different depths from a single light field. The analysis shows in closed form that the sharpness of refocused photographs increases linearly with directional resolution. Second, the theorem yields a Fourier-domain algorithm for digital refocusing, where we extract the appropriate 2D slice of the light field's Fourier transform, and perform an inverse 2D Fourier transform. This method is faster than previous approaches.

563 citations