Due to the poor lighting condition and limited dynamic range of digital imaging devices, the recorded images are often under-/over-exposed and with low contrast. Most of previous single image contrast enhancement (SICE) methods adjust the tone curve to correct the contrast of an input image. Those methods, however, often fail in revealing image details because of the limited information in a single image. On the other hand, the SICE task can be better accomplished if we can learn extra information from appropriately collected training data. In this paper, we propose to use the convolutional neural network (CNN) to train a SICE enhancer. One key issue is how to construct a training data set of low-contrast and high-contrast image pairs for end-to-end CNN learning. To this end, we build a large-scale multi-exposure image data set, which contains 589 elaborately selected high-resolution multi-exposure sequences with 4,413 images. Thirteen representative multi-exposure image fusion and stack-based high dynamic range imaging algorithms are employed to generate the contrast enhanced images for each sequence, and subjective experiments are conducted to screen the best quality one as the reference image of each scene. With the constructed data set, a CNN can be easily trained as the SICE enhancer to improve the contrast of an under-/over-exposure image. Experimental results demonstrate the advantages of our method over existing SICE methods with a significant margin.

Learning a Deep Single Image Contrast Enhancer from Multi-Exposure Images

It is known that local filtering-based edge preserving smoothing techniques suffer from halo artifacts. In this paper, a weighted guided image filter (WGIF) is introduced by incorporating an edge-aware weighting into an existing guided image filter (GIF) to address the problem. The WGIF inherits advantages of both global and local smoothing filters in the sense that: 1) the complexity of the WGIF is O(N) for an image with N pixels, which is same as the GIF and 2) the WGIF can avoid halo artifacts like the existing global smoothing filters. The WGIF is applied for single image detail enhancement, single image haze removal, and fusion of differently exposed images. Experimental results show that the resultant algorithms produce images with better visual quality and at the same time halo artifacts can be reduced/avoided from appearing in the final images with negligible increment on running times.

Weighted guided image filtering.

Producing a high dynamic range (HDR) image from a set of images with different exposures is a challenging process for dynamic scenes. A category of existing techniques first register the input images to a reference image and then merge the aligned images into an HDR image. However, the artifacts of the registration usually appear as ghosting and tearing in the final HDR images. In this paper, we propose a learning-based approach to address this problem for dynamic scenes. We use a convolutional neural network (CNN) as our learning model and present and compare three different system architectures to model the HDR merge process. Furthermore, we create a large dataset of input LDR images and their corresponding ground truth HDR images to train our system. We demonstrate the performance of our system by producing high-quality HDR images from a set of three LDR images. Experimental results show that our method consistently produces better results than several state-of-the-art approaches on challenging scenes.

Deep high dynamic range imaging of dynamic scenes

We propose a simple yet effective structural patch decomposition approach for multi-exposure image fusion (MEF) that is robust to ghosting effect. We decompose an image patch into three conceptually independent components: signal strength, signal structure, and mean intensity. Upon fusing these three components separately, we reconstruct a desired patch and place it back into the fused image. This novel patch decomposition approach benefits MEF in many aspects. First, as opposed to most pixel-wise MEF methods, the proposed algorithm does not require post-processing steps to improve visual quality or to reduce spatial artifacts. Second, it handles RGB color channels jointly, and thus produces fused images with more vivid color appearance. Third and most importantly, the direction of the signal structure component in the patch vector space provides ideal information for ghost removal. It allows us to reliably and efficiently reject inconsistent object motions with respect to a chosen reference image without performing computationally expensive motion estimation. We compare the proposed algorithm with 12 MEF methods on 21 static scenes and 12 deghosting schemes on 19 dynamic scenes (with camera and object motion). Extensive experimental results demonstrate that the proposed algorithm not only outperforms previous MEF algorithms on static scenes but also consistently produces high quality fused images with little ghosting artifacts for dynamic scenes. Moreover, it maintains a lower computational cost compared with the state-of-the-art deghosting schemes. 1 1 The MATLAB code of the proposed algorithm will be made available online. Preliminary results of Section III-A [1] were presented at the IEEE International Conference on Image Processing, Canada, 2015.

Robust Multi-Exposure Image Fusion: A Structural Patch Decomposition Approach

Guided image filter (GIF) is a well-known local filter for its edge-preserving property and low computational complexity. Unfortunately, the GIF may suffer from halo artifacts, because the local linear model used in the GIF cannot represent the image well near some edges. In this paper, a gradient domain GIF is proposed by incorporating an explicit first-order edge-aware constraint. The edge-aware constraint makes edges be preserved better. To illustrate the efficiency of the proposed filter, the proposed gradient domain GIF is applied for single-image detail enhancement, tone mapping of high dynamic range images and image saliency detection. Both theoretical analysis and experimental results prove that the proposed gradient domain GIF can produce better resultant images, especially near the edges, where halos appear in the original GIF.

Gradient Domain Guided Image Filtering

It is very challenging to synthesize a high dynamic range (HDR) image from multiple differently exposed low dynamic range images when there are moving objects in the images. This is due to the fact that the moving objects will cause ghosting artifacts to appear in the synthesized HDR image. To prevent such artifacts, a patching algorithm is required to correct motion regions such that all the moving objects are synchronized in the differently exposed images. In this paper, a new optimization problem is formulated to correct the motion regions of the multiple differently exposed images by considering both spatial and temporal consistencies. The resultant scheme is a hybrid patching scheme composed of a correction method which is an intensity mapping function at pixel level, and a hole-filling method that uses block-level template matching. The proposed patching scheme is not only robust to large intensity changes in these input images, but also at regions that are over-or under-exposed. Experimental results show that the proposed method is able to prevent ghosting artifacts from appearing in the final synthesized HDR image.

Hybrid Patching for a Sequence of Differently Exposed Images With Moving Objects

In this paper, we propose an intensity mapping function (IMF) based scheme to detect moving objects in a set of low dynamic range (LDR) images with different known exposure times. The objective is to remove ghosting artifacts from the eventual high dynamic range (HDR) image. Our contributions include a bidirectional similarity detection method, an adaptive threshold for movement detection, and an IMF based method for the synthesis of pixels to fill in the regions of moving objects. Experimental results show that the proposed scheme outperforms existing schemes.

Movement detection for the synthesis of high dynamic range images

Embodiments provide a method for processing an input image having a plurality of pixels, wherein each pixel has a pixel position. The method may include determining, for each pixel position, a vector based on the input image; and determining, for each pixel position, a detail value for the pixel position based on the determined vectors at a plurality of neighboring pixel positions within a predetermined neighboring block of the pixel position.

Method and device for image processing

This letter presents a novel exposure-robust method to align differently exposed images. First, a directional mapping approach is introduced to normalize differently exposed images so as to alleviate the effect of saturation. Then, a non-parametric local binary pattern (LBP) is employed to represent intensity-invariant features of these images. An efficient two-stage alignment is proposed for motion estimation. Experiments on a variety of synthesized and real image sequences demonstrate that the proposed method is less sensitive to the reference image, and robust to 12 exposure values (EV) increments, which is superior to existing methods.

Zijian Zhu

Papers

Weighted guided image filtering.

Hybrid Patching for a Sequence of Differently Exposed Images With Moving Objects

Movement detection for the synthesis of high dynamic range images

Method and device for image processing

Exposure-Robust Alignment of Differently Exposed Images