Joint Bi-layer Optimization for Single-Image Rain Streak Removal
01 Oct 2017-pp 2545-2553
TL;DR: A novel method for removing rain streaks from a single input image by decomposing it into a rain-free background layer B and aRain-streak layer R, which outperforms the state-of-the-art.
Abstract: We present a novel method for removing rain streaks from a single input image by decomposing it into a rain-free background layer B and a rain-streak layer R. A joint optimization process is used that alternates between removing rain-streak details from B and removing non-streak details from R. The process is assisted by three novel image priors. Observing that rain streaks typically span a narrow range of directions, we first analyze the local gradient statistics in the rain image to identify image regions that are dominated by rain streaks. From these regions, we estimate the dominant rain streak direction and extract a collection of rain-dominated patches. Next, we define two priors on the background layer B, one based on a centralized sparse representation and another based on the estimated rain direction. A third prior is defined on the rain-streak layer R, based on similarity of patches to the extracted rain patches. Both visual and quantitative comparisons demonstrate that our method outperforms the state-of-the-art.
Citations
More filters
TL;DR: This work attempts to leverage powerful generative modeling capabilities of the recently introduced conditional generative adversarial networks (CGAN) by enforcing an additional constraint that the de-rained image must be indistinguishable from its corresponding ground truth clean image.
Abstract: Severe weather conditions, such as rain and snow, adversely affect the visual quality of images captured under such conditions, thus rendering them useless for further usage and sharing. In addition, such degraded images drastically affect the performance of vision systems. Hence, it is important to address the problem of single image de-raining. However, the inherent ill-posed nature of the problem presents several challenges. We attempt to leverage powerful generative modeling capabilities of the recently introduced conditional generative adversarial networks (CGAN) by enforcing an additional constraint that the de-rained image must be indistinguishable from its corresponding ground truth clean image. The adversarial loss from GAN provides additional regularization and helps to achieve superior results. In addition to presenting a new approach to de-rain images, we introduce a new refined loss function and architectural novelties in the generator–discriminator pair for achieving improved results. The loss function is aimed at reducing artifacts introduced by GANs and ensure better visual quality. The generator sub-network is constructed using the recently introduced densely connected networks, whereas the discriminator is designed to leverage global and local information to decide if an image is real/fake. Based on this, we propose a novel single image de-raining method called image de-raining conditional generative adversarial network (ID-CGAN) that considers quantitative, visual, and also discriminative performance into the objective function. The experiments evaluated on synthetic and real images show that the proposed method outperforms many recent state-of-the-art single image de-raining methods in terms of quantitative and visual performances. Furthermore, the experimental results evaluated on object detection datasets using the Faster-RCNN also demonstrate the effectiveness of proposed method in improving the detection performance on images degraded by rain.
747 citations
Cites methods from "Joint Bi-layer Optimization for Sin..."
...In such cases, previous works have designed appropriate prior in solving (1) such as sparsity prior [10]– [13], Gaussian Mixture Model (GMM) prior [14] and patchrank prior [15]....
[...]
08 Sep 2018
TL;DR: A novel deep network architecture based on deep convolutional and recurrent neural networks for single image deraining based on contextual information is proposed and outperforms the state-of-the-art approaches under all evaluation metrics.
Abstract: Rain streaks can severely degrade the visibility, which causes many current computer vision algorithms fail to work. So it is necessary to remove the rain from images. We propose a novel deep network architecture based on deep convolutional and recurrent neural networks for single image deraining. As contextual information is very important for rain removal, we first adopt the dilated convolutional neural network to acquire large receptive field. To better fit the rain removal task, we also modify the network. In heavy rain, rain streaks have various directions and shapes, which can be regarded as the accumulation of multiple rain streak layers. We assign different alpha-values to various rain streak layers according to the intensity and transparency by incorporating the squeeze-and-excitation block. Since rain streak layers overlap with each other, it is not easy to remove the rain in one stage. So we further decompose the rain removal into multiple stages. Recurrent neural network is incorporated to preserve the useful information in previous stages and benefit the rain removal in later stages. We conduct extensive experiments on both synthetic and real-world datasets. Our proposed method outperforms the state-of-the-art approaches under all evaluation metrics. Codes and supplementary material are available at our project webpage: https://xialipku.github.io/RESCAN.
539 citations
Cites background from "Joint Bi-layer Optimization for Sin..."
...[24] combine three different kinds of image priors....
[...]
18 Jun 2018
TL;DR: A novel density-aware multi-stream densely connected convolutional neural network-based algorithm, called DID-MDN, for joint rain density estimation and de-raining, which achieves significant improvements over the recent state-of-the-art methods.
Abstract: Single image rain streak removal is an extremely challenging problem due to the presence of non-uniform rain densities in images. We present a novel density-aware multi-stream densely connected convolutional neural network-based algorithm, called DID-MDN, for joint rain density estimation and de-raining. The proposed method enables the network itself to automatically determine the rain-density information and then efficiently remove the corresponding rain-streaks guided by the estimated rain-density label. To better characterize rain-streaks with different scales and shapes, a multi-stream densely connected de-raining network is proposed which efficiently leverages features from different scales. Furthermore, a new dataset containing images with rain-density labels is created and used to train the proposed density-aware network. Extensive experiments on synthetic and real datasets demonstrate that the proposed method achieves significant improvements over the recent state-of-the-art methods. In addition, an ablation study is performed to demonstrate the improvements obtained by different modules in the proposed method. The code can be downloaded at https://github.com/hezhangsprinter/DID-MDN
535 citations
Cites background or methods from "Joint Bi-layer Optimization for Sin..."
...Input DSC [21] (ICCV’15) GMM [19] (CVPR’16) CNN [6] (TIP’17) JORDER [36] (CVPR’17) DDN [7] (CVPR’17) JBO [47] (ICCV’17) DID-MDN Test1 0.7781/21.15 0.7896/21.44 0.8352/22.75 0.8422/22.07 0.8622/24.32 0.8978/ 27.33 0.8522/23.05 0.9087/ 27.95 Test2 0.7695/19.31 0.7825/20.08 0.8105/20.66 0.8289/19.73 0.8405/22.26 0.8851/25.63 0.8356/22.45 0.9092/ 26.0745 Heavy Medium Light Figure 5: Samples synthetic images in three different conditions....
[...]
...The proposed DID-MDN method is compared with the following recent state-of-the-art methods: (a) Discriminative sparse codingbased method (DSC) [21] (ICCV’15), (b) Gaussian mixture model (GMM) based method [19] (CVPR’16), (c) CNN method (CNN) [6] (TIP’17), (d) Joint Rain Detection and Removal (JORDER) method [36] (CVPR’17), (e) Deep detailed Network method (DDN) [7] (CVPR’17), and (f) Joint Bi-layer Optimization (JBO) method [47] (ICCV’17)....
[...]
...These include sparse coding-based methods [16, 11, 47], lowrank representation-based methods [3, 39] and GMM-based (gaussian mixture model) methods [19]....
[...]
...Similar results are also observed from [47]....
[...]
...Input DSC [21] (ICCV’15) GMM [19] (CVPR’16) CNN [6] (TIP’17) JORDER [36] (CVPR’17) DDN [7] (CVPR’17) JBO [47] (ICCV’17) DID-MDN...
[...]
01 Jun 2019
TL;DR: A semi-automatic method that incorporates temporal priors and human supervision to generate a high-quality clean image from each input sequence of real rain images is proposed, and a novel SPatial Attentive Network (SPANet) is proposed to remove rain streaks in a local-to-global manner.
Abstract: Removing rain streaks from a single image has been drawing considerable attention as rain streaks can severely degrade the image quality and affect the performance of existing outdoor vision tasks. While recent CNN-based derainers have reported promising performances, deraining remains an open problem for two reasons. First, existing synthesized rain datasets have only limited realism, in terms of modeling real rain characteristics such as rain shape, direction and intensity. Second, there are no public benchmarks for quantitative comparisons on real rain images, which makes the current evaluation less objective. The core challenge is that real world rain/clean image pairs cannot be captured at the same time. In this paper, we address the single image rain removal problem in two ways. First, we propose a semi-automatic method that incorporates temporal priors and human supervision to generate a high-quality clean image from each input sequence of real rain images. Using this method, we construct a large-scale dataset of ∼29.5K rain/rain-free image pairs that covers a wide range of natural rain scenes. Second, to better cover the stochastic distribution of real rain streaks, we propose a novel SPatial Attentive Network (SPANet) to remove rain streaks in a local-to-global manner. Extensive experiments demonstrate that our network performs favorably against the state-of-the-art deraining methods.
387 citations
Cites methods from "Joint Bi-layer Optimization for Sin..."
...Methods Input DSC [29] LP [26] Clear[10] JORDER [40] DDN [11] JBO[47] DID-MDN[42] Our SPANet...
[...]
...[47] exploit rain streak directions to first determine the rain-dominant regions, which are used to guide the process of separating rain streaks from background details based on rain-dominant patch statistics....
[...]
...In the last decade, we have witnessed a continuous progress on rain removal research with many methods proposed [20, 29, 26, 5, 47, 9], through carefully modeling...
[...]
Posted Content•
TL;DR: A better and simpler baseline deraining network by considering network architecture, input and output, and loss functions is provided and is expected to serve as a suitable baseline in future deraining research.
Abstract: Along with the deraining performance improvement of deep networks, their structures and learning become more and more complicated and diverse, making it difficult to analyze the contribution of various network modules when developing new deraining networks. To handle this issue, this paper provides a better and simpler baseline deraining network by considering network architecture, input and output, and loss functions. Specifically, by repeatedly unfolding a shallow ResNet, progressive ResNet (PRN) is proposed to take advantage of recursive computation. A recurrent layer is further introduced to exploit the dependencies of deep features across stages, forming our progressive recurrent network (PReNet). Furthermore, intra-stage recursive computation of ResNet can be adopted in PRN and PReNet to notably reduce network parameters with graceful degradation in deraining performance. For network input and output, we take both stage-wise result and original rainy image as input to each ResNet and finally output the prediction of {residual image}. As for loss functions, single MSE or negative SSIM losses are sufficient to train PRN and PReNet. Experiments show that PRN and PReNet perform favorably on both synthetic and real rainy images. Considering its simplicity, efficiency and effectiveness, our models are expected to serve as a suitable baseline in future deraining research. The source codes are available at this https URL.
344 citations
Cites methods from "Joint Bi-layer Optimization for Sin..."
...Among these methods, Gaussian mixture model (GMM) [21], sparse representation [35], and low rank representation [1] have been adopted for modeling background image or rain layers....
[...]
...On the one hand, linear summation is usually adopted as the composition model [1, 21, 35]....
[...]
References
More filters
TL;DR: In this article, a structural similarity index is proposed for image quality assessment based on the degradation of structural information, which can be applied to both subjective ratings and objective methods on a database of images compressed with JPEG and JPEG2000.
Abstract: Objective methods for assessing perceptual image quality traditionally attempted to quantify the visibility of errors (differences) between a distorted image and a reference image using a variety of known properties of the human visual system. Under the assumption that human visual perception is highly adapted for extracting structural information from a scene, we introduce an alternative complementary framework for quality assessment based on the degradation of structural information. As a specific example of this concept, we develop a structural similarity index and demonstrate its promise through a set of intuitive examples, as well as comparison to both subjective ratings and state-of-the-art objective methods on a database of images compressed with JPEG and JPEG2000. A MATLAB implementation of the proposed algorithm is available online at http://www.cns.nyu.edu//spl sim/lcv/ssim/.
40,609 citations
Book•
23 May 2011TL;DR: It is argued that the alternating direction method of multipliers is well suited to distributed convex optimization, and in particular to large-scale problems arising in statistics, machine learning, and related areas.
Abstract: Many problems of recent interest in statistics and machine learning can be posed in the framework of convex optimization. Due to the explosion in size and complexity of modern datasets, it is increasingly important to be able to solve problems with a very large number of features or training examples. As a result, both the decentralized collection or storage of these datasets as well as accompanying distributed solution methods are either necessary or at least highly desirable. In this review, we argue that the alternating direction method of multipliers is well suited to distributed convex optimization, and in particular to large-scale problems arising in statistics, machine learning, and related areas. The method was developed in the 1970s, with roots in the 1950s, and is equivalent or closely related to many other algorithms, such as dual decomposition, the method of multipliers, Douglas–Rachford splitting, Spingarn's method of partial inverses, Dykstra's alternating projections, Bregman iterative algorithms for l1 problems, proximal methods, and others. After briefly surveying the theory and history of the algorithm, we discuss applications to a wide variety of statistical and machine learning problems of recent interest, including the lasso, sparse logistic regression, basis pursuit, covariance selection, support vector machines, and many others. We also discuss general distributed optimization, extensions to the nonconvex setting, and efficient implementation, including some details on distributed MPI and Hadoop MapReduce implementations.
17,433 citations
"Joint Bi-layer Optimization for Sin..." refers methods in this paper
...Hence, we efficiently solve it by adopting the alternating direction method of multipliers (ADMM) technique [3] by alternatively updating B and α in the following two subproblems (with T iterations):...
[...]
23 Jun 2013
TL;DR: This work tackles saliency detection from a scale point of view and proposes a multi-layer approach to analyze saliency cues, by finding saliency values optimally in a tree model.
Abstract: When dealing with objects with complex structures, saliency detection confronts a critical problem - namely that detection accuracy could be adversely affected if salient foreground or background in an image contains small-scale high-contrast patterns. This issue is common in natural images and forms a fundamental challenge for prior methods. We tackle it from a scale point of view and propose a multi-layer approach to analyze saliency cues. The final saliency map is produced in a hierarchical model. Different from varying patch sizes or downsizing images, our scale-based region handling is by finding saliency values optimally in a tree model. Our approach improves saliency detection on many images that cannot be handled well traditionally. A new dataset is also constructed.
1,624 citations
"Joint Bi-layer Optimization for Sin..." refers methods in this paper
...Second, we explored saliency detection with and without rain using [31]....
[...]
...Here, we randomly selected 30 images from the CSSD dataset [31], added rain to each of them, and then applied our method to remove the rain....
[...]
01 Aug 2008
TL;DR: This paper advocates the use of an alternative edge-preserving smoothing operator, based on the weighted least squares optimization framework, which is particularly well suited for progressive coarsening of images and for multi-scale detail extraction.
Abstract: Many recent computational photography techniques decompose an image into a piecewise smooth base layer, containing large scale variations in intensity, and a residual detail layer capturing the smaller scale details in the image. In many of these applications, it is important to control the spatial scale of the extracted details, and it is often desirable to manipulate details at multiple scales, while avoiding visual artifacts.In this paper we introduce a new way to construct edge-preserving multi-scale image decompositions. We show that current basedetail decomposition techniques, based on the bilateral filter, are limited in their ability to extract detail at arbitrary scales. Instead, we advocate the use of an alternative edge-preserving smoothing operator, based on the weighted least squares optimization framework, which is particularly well suited for progressive coarsening of images and for multi-scale detail extraction. After describing this operator, we show how to use it to construct edge-preserving multi-scale decompositions, and compare it to the bilateral filter, as well as to other schemes. Finally, we demonstrate the effectiveness of our edge-preserving decompositions in the context of LDR and HDR tone mapping, detail enhancement, and other applications.
1,381 citations
"Joint Bi-layer Optimization for Sin..." refers methods in this paper
...Inspired by [9], we use a weighted Laplacian term to formulate Ω(R) with spatially-varying smoothing capability:...
[...]
22 Apr 2010
TL;DR: This paper surveys the various options such training has to offer, up to the most recent contributions and structures of the MOD, the K-SVD, the Generalized PCA and others.
Abstract: Sparse and redundant representation modeling of data assumes an ability to describe signals as linear combinations of a few atoms from a pre-specified dictionary. As such, the choice of the dictionary that sparsifies the signals is crucial for the success of this model. In general, the choice of a proper dictionary can be done using one of two ways: i) building a sparsifying dictionary based on a mathematical model of the data, or ii) learning a dictionary to perform best on a training set. In this paper we describe the evolution of these two paradigms. As manifestations of the first approach, we cover topics such as wavelets, wavelet packets, contourlets, and curvelets, all aiming to exploit 1-D and 2-D mathematical models for constructing effective dictionaries for signals and images. Dictionary learning takes a different route, attaching the dictionary to a set of examples it is supposed to serve. From the seminal work of Field and Olshausen, through the MOD, the K-SVD, the Generalized PCA and others, this paper surveys the various options such training has to offer, up to the most recent contributions and structures.
1,345 citations
"Joint Bi-layer Optimization for Sin..." refers background or methods in this paper
...Dictionary-based sparse prior [23] describes an image patch as a linear combination of a few atoms from a pre-specified dictionary....
[...]
...While a dictionary-based sparse prior has been previously used for rain streak removal [23, 21], Dong et al....
[...]