NEAR-INFRARED GUIDED COLOR IMAGE DEHAZING
Chen Feng, Shaojie Zhuo, Xiaopeng Zhang, Liang Shen
Qualcomm Canada Inc.
Sabine Süsstrunk
EPFL
ABSTRACT
Near-infrared (NIR) light has stronger penetration capability
than visible light due to its long wavelength, thus being less
scattered by particles in the air. This makes it desirable
for image dehazing to unveil details of distant objects in
landscape photographs. In this paper, we propose an improved
image dehazing scheme using a pair of color and NIR
images, which effectively estimates the airlight color and
transfers details from the NIR. A two-stage dehazing method
is proposed by exploiting the dissimilarity between RGB and
NIR for airlight color estimation, followed by a dehazing
procedure through an optimization framework. Experiments
on captured haze images show that our method can achieve
substantial improvements on the detail recovery and the color
distribution over the existing image dehazing algorithms.
Keywords- Image dehazing, RGB, NIR, Detail transfer,
Optimization, Airlight color
1. INTRODUCTION
Haze and mist significantly reduce the visibility in landscape
photographs, which impact visual quality and bring difficulties
for many computer vision applications [1]. Accurately
estimating the airlight color and recovering the lost details
in the color image is a fundamental and challenging problem
in image processing.
Optically, the haze effect is due to the presence of particles
in the atmosphere, with comparable size to the wavelength in
the visible band (haze ∼ 0.1um, mist ∼ 1um), that absorb
and scatter light. Reflected light from distant objects is
attenuated and diffused by the particles. As the particle
density increases, both color and details of distant scene
will fade away. The advantage of deep penetration of the
near-infrared (NIR) due to its long wavelength (∼ 1um)
makes it possible to unveil the details, which could be
completely lost in the visible band.
Image dehazing in general involves two tasks, removing
the airlight color effect and recovering the lost details.
State-of-the-art image dehazing algorithms [2][3][4], remove
the haze based on a single RGB image. The core idea in these
studies is to estimate the airlight color and the transmission
map under certain assumption, such as dark channel prior,
and then reconstruct the haze-free image based on the haze
model. However, these algorithms suffer from an inherent
problem of the single image input, which may not contain
any scene details. Once the information is lost in the input
image, it is very difficult for these algorithms to unveil the
ground truth. In general, the recovered images at dense haze
regions tend to be noisy and lack texture details.
Removing the airlight color is fundamental to image
dehazing. Inaccurate estimation of the airlight color could
result in unwanted color shift issues. Most current literatures
[2][4] simply approximate the airlight color from the brightest
region in the scene by assuming such regions are usually at
infinity and have the most haze. However, this approach often
fails by mistakenly selecting white objects (e.g., clouds) to
estimate the airlight color. Essentially, it is hard to use single
RGB image to differentiate the object with haze-like color
(i.e., cloud) from the object under actual haze.
There are some other approaches removing the haze
by exploiting the difference of two or more images of the
same scene that have different properties. Nayar et al. use
two images with the same scene captured with significant
different mediums [1], which is impractical in reality and
hard to deliver immediate results. Shwartz et al. use two
images captured with different degrees of polarization by
rotating a polarizing filter attached to the camera [5], which
cannot handle dynamic scenes for most outdoor landscape
pictures where objects like trees and clouds moving quickly.
The near-infrared spectrum can be easily acquired by
using off-the-shelf digital cameras with minor modifications
[6], or potentially through a single RGBN camera, in which
multiple images with different properties can be captured
simultaneously [7]. In [8], L. Schaul et al. first proposed
to fuse the NIR detail information in the color image under
the Weighted Least Squares (WLS) framework [9] without
airlight color detection. The fundamental issue of this
approach is that the texture details in the non-haze regions
will also be boosted in luminance channel, thus resulting in
color shifting artifacts.
Most previous works suffer from either detail lost or
color shifting. Enlightened by the general haze model [3],
however, we try to recover the color and transfer the details in
one shot, aiming to provide a practical and complete solution.
By exploiting dissimilarity of NIR and other color bands, we
refined airlight color esitmation in a much meaningful way.
By formulating image de-hazing as an optimization problem
and introducing NIR gradient constraint, we succesfully
removed haze effect by revealing accurate details and color,
while leaving non-haze regions with minimum impact.
In this paper, we propose a novel image dehazing approach,
using the images captured in both visible (400 − 700nm) and
near-infrared (700 − 1100nm) bands. Our main contribution
are in two folds:
• Propose an optimization framework to resolve image
de-hazing problem guided with NIR gradient constraints.
• Refine airlight color estimation by exploiting the differences
between NIR and RGB channels.
2. PROBLEM FORMULATION
Consider a general haze model [3],
I(x) = t(x)J(x) + (1 − t(x))A, (1)
where for each pixel x, I(x) stands for the observed image;
J(x) is the haze-free image; A represents for the global
airlight color, which is a 3 × 1 vector in typical outdoor
landscape photographs; t(x) is the medium transmission
describing the portion of the light that is not scattered and
reaches the camera. Therefore, by removing the haze, it is
required to recover J given the color image I
RGB
and the
near-infrared image I
NIR
. For the brevity of discussion in
this paper, we assume the pair of RGB and NIR images are
well registered.
Airlight Color
Estimation
Image Dehazing
Optimization
RGB-NIR
Image Pair
(well aligned)
Haze-free Color
Image
INPUT IMAGE DEHAZING OUTPUT
Fig. 1. The overview of the proposed image dehazing scheme.
We propose a two-stage dehazing scheme: an airlight
color estimation stage by exploiting the dissimilarity between
RGB and NIR; and an image dehazing stage by enforcing the
NIR gradient constraint through an optimization framework.
Fig. 1 shows the overall workflow of the proposed scheme.
3. AIRLIGHT ESTIMATION
In order to recover J as formulated in (1), the first step
of our approach is to estimate the global airlight color A.
A commonly used idea in literatures is to approximate the
airlight color from the most hazed region in the scene, where
the transmission tends to be zero. However, this approach
has limitation on scenarios with light haze. Inspired by
[3], we consider that the transmission t depends on the
scene depth and the density of the haze, while the intrinsic
color J depends on the illumination of the scene and the
surface reflectance. Therefore, it is reasonable to assume
that t and J are uncorrelated within a local patch. The idea
we proposed here to estimate the airlight color consists of
finding a local patch Ω with pixels having large similarities,
followed by searching an airlight color that leads to the
smallest correlation between t and J.
One of our key innovations is to use NIR to help finding
such a good local patch for the airlight color estimation. A
good local patch should meet two major criteria: i) pixels
within the patch should have intermediate level of haze so
that both J and A contribute to the observed intensity value
I in equation (1); ii) pixels within the patch should have
similar properties (i.e., surface reflectance). Therefore, we
first generate a haze map to find pixels that satisfy i), and then
adopt RGB-NIR relationship to meet the criteria ii).
Consider particle scattering, we can make two observations.
Firstly, haze increases the intensity value over all R, G and B
channels and thus reduces the image contrast. In other words,
the smallest intensity value over the three color channels
infers the density of the haze. We refer to this observation
as the haze prior. Secondly, since blue light has the shortest
wavelength in the visible band compared with NIR, it is
scattered more and thus changes quickly as depth increases.
Therefore, the difference between the blue channel and the
NIR channel can be used to refine the density of the haze. By
considering the two observations, we define a haze map H,
as shown in (2), to indicate the density of the haze.
H = min{ min
k∈{R,G,B}
(I
k
), D}, D = N {|I
B
− I
NIR
|}, (2)
where N {·} represents the max-min normalization. Therefore,
large value in H infers small transmission.
Criterion i) requires the patch contains an intermediate
level of haze. Therefore, we calculate the histogram of H,
and use the first valley h as a reference to select a coarse patch
region Ω
c
. Pixels within Ω
c
should have similar haze density,
and the similarity is defined by δ,
Ω
c
= {x : |H(x) − h| ≤ δ}. (3)
Within the coarse patch region, a refined local patch Ω
is obtained by searching those connected pixels with similar
color properties. Particulary, pixels that have similar NIR
value and blue to NIR ratios are retained.
By using the pair of RGB and NIR images, we are able
to infer the density of haze and thus, pixels that have large
similarities are selected to generate a local patch, within
which we search for an airlight color A that leads to the
smallest correlation, measured by Pearson’s correlation
coefficient C, between t and J. Please refer to [3] for
the detailed definition of C. Specifically, we update A’s
components using the steepest decent method by minimizing
the Equation (4). Fig. 2 illustrates the haze map H, its
Fig. 2. Airlight color estimation. Top-left shows the original RGB
image and the pixels selected to esitmate airlight color (makred with
red cross). The pixels are selected based on first valley in histogram
(bottom-right) of haze map H (bottom-left).
histogram and the selected patches for airlight color estimation.
A = arg min
∀(x)∈Ω
C(J, t)
2
(4)
4. IMAGE DEHAZING
The core to the algorithm is to formulate the whole image
dehazing process as an optimization problem. This process
includes estimating the haze-free color image and transferring
details from the NIR to the color image.
4.1. Initialization
Divided by the estimated airlight color A at both sides in
equation (1) together with the haze prior, it is easy to derive
t
0
= 1 − min
k∈{R,G,B}
(I
k
/A). (5)
Since the haze prior may not lead min
k∈{R,G,B}
(J
k
/A)
to be exactly equal to zero for each pixel, the transmission
map t
0
calculated from (5) may tend to be smaller than the
actual value. In addition, if we remove the haze thoroughly,
the image may look unnatural. To compensate for this, we
optionally keep a small amount of haze by introducing a
weight ω, which is large for distant objects and small for
close objects, namely,
ω =
1
1 + e
−10(N {H}−0.5)
∗ (a − b) + a, (6)
where a and b are pre-determined parameters (a = 0.6,b =
0.4 in our experiments).
In order to get a good initialization, we use the RGB
input image as a guide to refine the transmission map t
0
for
better edge alignment through guided filter [10]. Finally, the
initialized J
0
is calculated by,
J
0
=
I − A
max(t, ε)
+ A, (7)
where ε equals to 0.1 in our experiments.
4.2. Optimization Framework
Recall our problem formulation in (1), removing the haze
basically requires the recovery of J given I
RGB
and I
NIR
.
Statistically, this can be reformulated by finding the largest
joint probability of (J, t) given I
RGB
and I
NIR
. Based on
Bayes’ theorem, we further derive
P (J, t|I
RGB
, I
N IR
) =
P (I
RGB
, I
N IR
|J, t)P (J, t)
P (I
RGB
, I
N IR
)
∝ P (I
RGB
|J, t)P (I
N IR
|J, t)P (J)P (t). (8)
Equation (8) contains four terms, in which P (I
RGB
|J, t)
corresponds to the haze model as defined in (1); P(I
NIR
|J, t)
represents the relationship between the color and the NIR
images, which we refer to as the NIR constraint ; P (J) and
P (t) represent the color image prior and the transmission
prior, respectively.
Based on the statistical analysis in (8), we find the optimal
solution for J and t by solving the following optimization
problem,
(
ˆ
J,
ˆ
t) = arg min
(J,t)
k tJ + (1 − t)A − I
RGB
k
2
+ λ
1
w|∇J − ∇I
NIR
|
α
+ λ
2
|∇J|
β
+ λ
3
k ∇t k
2
,
(9)
where α, β ∈ (0, 1). The first term in (9) comes from our haze
model, which would produce the least noise in the haze-free
image. The fact that NIR penetrates further than the visible
band due to its long wavelength allows us to transfer the
details from the NIR image to the color image. Therefore,
we add a weighted gradient constraint in our second term.
The last two terms are the smoothness priors for the natural
image and the transmission map. In the weighted gradient
constraint, we add large weights on distant objects and small
weights on close objects. Thus, weight w is defined as
w =
1
1 + e
−10(0.5−t)
. (10)
In equation (9), λ
2
and λ
3
are pre-determined parameters with
small values (0.01 in our experiments), and λ
1
controls the
level of detail transfer that comes from the NIR image.
By solving the optimization problem stated in (9) using
Iteratively Reweighted Least Squares (IRLS) with initialized
J
0
and t
0
derived from (7) and (5), we can recover the
haze-free image J and the transmission map t simultaneously.
Fig. 3. The comparison with state-of-the-art image dehazing algorithms. Columns-wise from left to right: input RGB-NIR image pairs,
single image dehazing results by [2], multiple image dehazing results by [8], our results and the zoomed-in comparison. The estimated
airlight color of each method is indicated with values. Red rectangle areas are further zoomed-in and compared in the last column.
Different from traditional single image dehazing algorithms,
our approach transfers detail information from the NIR
image by enforcing the gradient constraint. By using the
optimization framework, it handles well on the noise issue
that comes from the division term in (7) when the transmission
value t is small. In addition, the transmission map t can be
different for each color channel, providing more accurate
recovered results.
After the airlight veil is removed in
ˆ
J, the overall tone of
the haze-free image could become darker than the input RGB
image. In our final results, a local tone mapping is applied
as a post-processing procedure to enhance the perceptual
experience.
5. EXPERIMENTAL RESULTS
This section provides results of the proposed dehazing method
applied on real captured image pairs using a modified commercial
camera. Specifically, the visible band ranges from 400
to 700nm, and the NIR filter allows IR ranging from 700
to 1100nm to pass. NIR and visible images of the same
scene are obtained by placing alternatively a NIR or visible
blocking filter on the lens. Potentially the pair of images can
be captured simultaneously through a single RGBN camera
in the future design [7].
Fig. 3 compares our dehazing result with the most recent
state-of-the-art single/multiple image dehazing algorithms
[2][8], where the first column shows the input image pairs. It
is clear to see that NIR retains more details on distant objects
than the corresponding color image. Compared with He’s
single image dehazing method [2], our result produces more
details on distant objects. This is simply due to the fact that
NIR can penetrate further than the visible band in the haze.
Meanwhile, our result produces a haze free image with much
less noise on distant objects. This benefit comes from the
first term in our optimization framework (9). Furthermore,
our result preserves the original illumination of the scene.
Comparing to Schaul et al.’s work [8], since our approach
adopts the haze model, we can not only selectively enhance
the details based on the transmission map, but also well
recover the original color of the scene.
From the experiments, it takes around 60 seconds to
process a 1 megapixel color image using matlab on a laptop
with 2.6GHz processor and 4G RAM.
6. CONCLUSION
In this paper, we have proposed an image dehazing method
using a pair of RGB and NIR images. The intuition behind
this technique is that NIR has deep penetration due to its long
wavelength and thus according to the Rayleigh’s scattering
law, the details of distant objects can be well preserved in
the NIR image. We have estimated the airlight color by
exploiting the dissimilarity between the RGB and the NIR,
and developed an image dehazing procedure to enforce the
NIR gradient constraint through an optimization framework.
Experimental results on real captured images demonstrate
that the proposed image dehazing method unveils the details
of the scene with less noise and better color distribution.
7. REFERENCES
[1] S K. Nayar and S G. Narasimhan, “Vision in bad
weather,” IEEE International Conference on Comput
Vision (ICCV), pp. 820–827, 1999.
[2] K M. He, J. Sun, and X O. Tang, “Single image haze
removal using dark channel prior,” IEEE Conference
on Computer Vision and Pattern Recognition (CVPR),
2009.
[3] R. Fattal, “Single image dehazing,” ACM Transactions
on Graphics (Proc. ACM SIGGRAPH), 2008.
[4] C O. Ancuti, C. Ancuti, C. Hermans, and P. Bekaert,
“A fast semi-inverse approach to detect and remove the
haze from a single image,” 10th Asian Conference on
Computer Vision (ACCV), pp. 501–514, 2010.
[5] S. Shwartz, E. Namer, and Y. Schechner, “Blind haze
separation,” IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), 2006.
[6] N. Salamati, A. Germain, and S. Süsstrunk, “Removing
shadows from images using color and near-infrared,”
Proc. IEEE International Conference on Image
Processing (ICIP), 2011.
[7] Y M. Lu, C. Fredembach, M. Vetterli, and S. Süsstrunk,
“Designing color filter arrays for the joint capture
of visible and near-infrared images,” Proc. IEEE
International Conference on Image Processing (ICIP),
2009.
[8] L. Schaul, C. Fredembach, and S. Süsstrunk, “Color
image dehazing using the near-infrared,” Proc. IEEE
International Conference on Image Processing (ICIP),
2009.
[9] Z. Farbman, R. Fattal, D. Lischinski, and R. Szeliski,
“Edge preserving decompositions for multi-scale tone
and detail manipulation,” International Conference on
Computer Graphics and Interactive Techniques, 2008.
[10] K M. He, J. Sun, and X O. Tang, “Guided image
filtering,” The 11th European Conference on Computer
Vision (ECCV), 2010.
![Fig. 3. The comparison with state-of-the-art image dehazing algorithms. Columns-wise from left to right: input RGB-NIR image pairs, single image dehazing results by [2], multiple image dehazing results by [8], our results and the zoomed-in comparison. The estimated airlight color of each method is indicated with values. Red rectangle areas are further zoomed-in and compared in the last column.](/figures/fig-3-the-comparison-with-state-of-the-art-image-dehazing-1gb5tui6.webp)