Robust Multi-Exposure Image Fusion: A Structural Patch Decomposition Approach
TL;DR: The proposed algorithm not only outperforms previous MEF algorithms on static scenes but also consistently produces high quality fused images with little ghosting artifacts for dynamic scenes and maintains a lower computational cost compared with the state-of-the-art deghosting schemes.
Abstract: We propose a simple yet effective structural patch decomposition approach for multi-exposure image fusion (MEF) that is robust to ghosting effect. We decompose an image patch into three conceptually independent components: signal strength, signal structure, and mean intensity. Upon fusing these three components separately, we reconstruct a desired patch and place it back into the fused image. This novel patch decomposition approach benefits MEF in many aspects. First, as opposed to most pixel-wise MEF methods, the proposed algorithm does not require post-processing steps to improve visual quality or to reduce spatial artifacts. Second, it handles RGB color channels jointly, and thus produces fused images with more vivid color appearance. Third and most importantly, the direction of the signal structure component in the patch vector space provides ideal information for ghost removal. It allows us to reliably and efficiently reject inconsistent object motions with respect to a chosen reference image without performing computationally expensive motion estimation. We compare the proposed algorithm with 12 MEF methods on 21 static scenes and 12 deghosting schemes on 19 dynamic scenes (with camera and object motion). Extensive experimental results demonstrate that the proposed algorithm not only outperforms previous MEF algorithms on static scenes but also consistently produces high quality fused images with little ghosting artifacts for dynamic scenes. Moreover, it maintains a lower computational cost compared with the state-of-the-art deghosting schemes. 1 1 The MATLAB code of the proposed algorithm will be made available online. Preliminary results of Section III-A [1] were presented at the IEEE International Conference on Image Processing, Canada, 2015.
Citations
More filters
TL;DR: This paper proposes to use the convolutional neural network (CNN) to train a SICE enhancer, and builds a large-scale multi-exposure image data set, which contains 589 elaborately selected high-resolution multi-Exposure sequences with 4,413 images.
Abstract: Due to the poor lighting condition and limited dynamic range of digital imaging devices, the recorded images are often under-/over-exposed and with low contrast. Most of previous single image contrast enhancement (SICE) methods adjust the tone curve to correct the contrast of an input image. Those methods, however, often fail in revealing image details because of the limited information in a single image. On the other hand, the SICE task can be better accomplished if we can learn extra information from appropriately collected training data. In this paper, we propose to use the convolutional neural network (CNN) to train a SICE enhancer. One key issue is how to construct a training data set of low-contrast and high-contrast image pairs for end-to-end CNN learning. To this end, we build a large-scale multi-exposure image data set, which contains 589 elaborately selected high-resolution multi-exposure sequences with 4,413 images. Thirteen representative multi-exposure image fusion and stack-based high dynamic range imaging algorithms are employed to generate the contrast enhanced images for each sequence, and subjective experiments are conducted to screen the best quality one as the reference image of each scene. With the constructed data set, a CNN can be easily trained as the SICE enhancer to improve the contrast of an under-/over-exposure image. Experimental results demonstrate the advantages of our method over existing SICE methods with a significant margin.
632 citations
Cites background or methods from "Robust Multi-Exposure Image Fusion:..."
...(b) The enhanced image by a deghosting MEF method [2]....
[...]
...13 state-of-the-art MEF and HDR algorithms are employed in this process, including 8 MEF methods: Mertens09 [13], Raman09 [40], Shen11 [41], Zhang12 [42], Li13 [12], Shen14 [43], Ma17 [2], Kou17 [44], and 5 stack-based HDR methods: Sen12 [14], Hu13 [32], Bruce14 [45], Oh15 [15], Photomatix [46]....
[...]
...With the sequence, multi-exposure image fusion (MEF) [2], [12], [13] and stack-based high dynamic range (HDR) imaging (with a following tone mapping operator) methods [14], [15]...
[...]
...13 state-of-the-art MEF and HDR algorithms are employed in this process, including 8 MEF methods: Mertens09 [13], Raman09 [39], Shen11 [40], Zhang12 [41], Li13 [12], Shen14 [42], Ma17 [2], Kou17 [43], and 5 stackbased HDR methods: Sen12 [14], Hu13 [32], Bruce14 [44], Oh15 [15], Photomatix [45]....
[...]
...In the last decade, researchers have spent much efforts to design de-ghosting algorithms [2], [14], [32] and learning-based methods like [33] proposed to map the multi-exposure image sequences to an HDR image....
[...]
03 Apr 2020
TL;DR: This paper unify the image fusion problem into the texture and intensity proportional maintenance problem of the source images, and defines a uniform form of loss function based on these two kinds of information, which can adapt to different fusion tasks.
Abstract: In this paper, we propose a fast unified image fusion network based on proportional maintenance of gradient and intensity (PMGI), which can end-to-end realize a variety of image fusion tasks, including infrared and visible image fusion, multi-exposure image fusion, medical image fusion, multi-focus image fusion and pan-sharpening. We unify the image fusion problem into the texture and intensity proportional maintenance problem of the source images. On the one hand, the network is divided into gradient path and intensity path for information extraction. We perform feature reuse in the same path to avoid loss of information due to convolution. At the same time, we introduce the pathwise transfer block to exchange information between different paths, which can not only pre-fuse the gradient information and intensity information, but also enhance the information to be processed later. On the other hand, we define a uniform form of loss function based on these two kinds of information, which can adapt to different fusion tasks. Experiments on publicly available datasets demonstrate the superiority of our PMGI over the state-of-the-art in terms of both visual effect and quantitative metric in a variety of fusion tasks. In addition, our method is faster compared with the state-of-the-art.
222 citations
Cites methods from "Robust Multi-Exposure Image Fusion:..."
...They are used in target detection, HDTV, medical diagnosis and other fields (Ma, Ma, and Li 2019; Ma et al. 2017; Xing et al. 2018)....
[...]
TL;DR: A novel multi-modality medical image fusion method based on phase congruency and local Laplacian energy that achieves competitive performance in both the image quantity and computational costs is presented.
Abstract: Multi-modality image fusion provides more comprehensive and sophisticated information in modern medical diagnosis, remote sensing, video surveillance, and so on. This paper presents a novel multi-modality medical image fusion method based on phase congruency and local Laplacian energy. In the proposed method, the non-subsampled contourlet transform is performed on medical image pairs to decompose the source images into high-pass and low-pass subbands. The high-pass subbands are integrated by a phase congruency-based fusion rule that can enhance the detailed features of the fused image for medical diagnosis. A local Laplacian energy-based fusion rule is proposed for low-pass subbands. The local Laplacian energy consists of weighted local energy and the weighted sum of Laplacian coefficients that describe the structured information and the detailed features of source image pairs, respectively. Thus, the proposed fusion rule can simultaneously integrate two key components for the fusion of low-pass subbands. The fused high-pass and low-pass subbands are inversely transformed to obtain the fused image. In the comparative experiments, three categories of multi-modality medical image pairs are used to verify the effectiveness of the proposed method. The experiment results show that the proposed method achieves competitive performance in both the image quantity and computational costs.
220 citations
Cites background from "Robust Multi-Exposure Image Fusion:..."
...Since spatial domain methods can completely preserve the spatial information of source images, they have state-of-theart performance in multi-focus [15], [17], [48] and multiexposure [27], [28] image fusion....
[...]
TL;DR: The obtained results indicate that the fusion of artificially under-exposed images can effectively remove the effect of haze, even in challenging situations where other current image dehazing techniques fail to produce good-quality results.
173 citations
TL;DR: A gradient ascent-based algorithm, which starts from any initial point in the space of all possible images and iteratively moves towards the direction that improves MEF-SSIM until convergence, and the final high quality fused image appears to have little dependence on the initial image.
Abstract: We propose a multi-exposure image fusion (MEF) algorithm by optimizing a novel objective quality measure, namely the color MEF structural similarity (MEF-SSIM $_c$ ) index. The design philosophy we introduce here is substantially different from existing ones. Instead of pre-defining a systematic computational structure for MEF ( e.g. , multiresolution transformation and transform domain fusion followed by image reconstruction), we directly operate in the space of all images, searching for the image that optimizes MEF-SSIM $_c$ . Specifically, we first construct the MEF-SSIM $_c$ index by improving upon and expanding the application scope of the existing MEF-SSIM algorithm. We then describe a gradient ascent-based algorithm, which starts from any initial point in the space of all possible images and iteratively moves towards the direction that improves MEF-SSIM $_c$ until convergence. Numerical and subjective experiments demonstrate that the proposed algorithm consistently produces better quality fused images both visually and in terms of MEF-SSIM $_c$ . The final high quality fused image appears to have little dependence on the initial image. The proposed optimization framework is readily extensible to construct better MEF algorithms when better objective quality models for MEF are available.
151 citations
Cites background from "Robust Multi-Exposure Image Fusion:..."
...2786138 In recent years, many MEF algorithms have been proposed [3]–[21]....
[...]
References
More filters
TL;DR: This paper presents a method for extracting distinctive invariant features from images that can be used to perform reliable matching between different views of an object or scene and can robustly identify objects among clutter and occlusion while achieving near real-time performance.
Abstract: This paper presents a method for extracting distinctive invariant features from images that can be used to perform reliable matching between different views of an object or scene. The features are invariant to image scale and rotation, and are shown to provide robust matching across a substantial range of affine distortion, change in 3D viewpoint, addition of noise, and change in illumination. The features are highly distinctive, in the sense that a single feature can be correctly matched with high probability against a large database of features from many images. This paper also describes an approach to using these features for object recognition. The recognition proceeds by matching individual features to a database of features from known objects using a fast nearest-neighbor algorithm, followed by a Hough transform to identify clusters belonging to a single object, and finally performing verification through least-squares solution for consistent pose parameters. This approach to recognition can robustly identify objects among clutter and occlusion while achieving near real-time performance.
46,906 citations
TL;DR: In this article, a structural similarity index is proposed for image quality assessment based on the degradation of structural information, which can be applied to both subjective ratings and objective methods on a database of images compressed with JPEG and JPEG2000.
Abstract: Objective methods for assessing perceptual image quality traditionally attempted to quantify the visibility of errors (differences) between a distorted image and a reference image using a variety of known properties of the human visual system. Under the assumption that human visual perception is highly adapted for extracting structural information from a scene, we introduce an alternative complementary framework for quality assessment based on the degradation of structural information. As a specific example of this concept, we develop a structural similarity index and demonstrate its promise through a set of intuitive examples, as well as comparison to both subjective ratings and state-of-the-art objective methods on a database of images compressed with JPEG and JPEG2000. A MATLAB implementation of the proposed algorithm is available online at http://www.cns.nyu.edu//spl sim/lcv/ssim/.
40,609 citations
04 Jan 1998
TL;DR: In contrast with filters that operate on the three bands of a color image separately, a bilateral filter can enforce the perceptual metric underlying the CIE-Lab color space, and smooth colors and preserve edges in a way that is tuned to human perception.
Abstract: Bilateral filtering smooths images while preserving edges, by means of a nonlinear combination of nearby image values. The method is noniterative, local, and simple. It combines gray levels or colors based on both their geometric closeness and their photometric similarity, and prefers near values to distant values in both domain and range. In contrast with filters that operate on the three bands of a color image separately, a bilateral filter can enforce the perceptual metric underlying the CIE-Lab color space, and smooth colors and preserve edges in a way that is tuned to human perception. Also, in contrast with standard filtering, bilateral filtering produces no phantom colors along edges in color images, and reduces phantom colors where they appear in the original image.
8,738 citations
"Robust Multi-Exposure Image Fusion:..." refers background in this paper
...H. Yong is with the Department of Computing, The Hong Kong Polytechnic University, Hong Kong, and also with the Institute for Information and System Sciences and Ministry of Education Key Laboratory of Intelligent Networks and Network Security, Xi’an Jiaotong University, Xi’an 710049, China…...
[...]
TL;DR: A technique for image encoding in which local operators of many scales but identical shape serve as the basis functions, which tends to enhance salient image features and is well suited for many image analysis tasks as well as for image compression.
Abstract: We describe a technique for image encoding in which local operators of many scales but identical shape serve as the basis functions. The representation differs from established techniques in that the code elements are localized in spatial frequency as well as in space. Pixel-to-pixel correlations are first removed by subtracting a lowpass filtered copy of the image from the image itself. The result is a net data compression since the difference, or error, image has low variance and entropy, and the low-pass filtered image may represented at reduced sample density. Further data compression is achieved by quantizing the difference image. These steps are then repeated to compress the low-pass image. Iteration of the process at appropriately expanded scales generates a pyramid data structure. The encoding process is equivalent to sampling the image with Laplacian operators of many scales. Thus, the code tends to enhance salient image features. A further advantage of the present code is that it is well suited for many image analysis tasks as well as for image compression. Fast algorithms are described for coding and decoding.
6,975 citations
TL;DR: A review of recent as well as classic image registration methods to provide a comprehensive reference source for the researchers involved in image registration, regardless of particular application areas.
6,842 citations