Automatic Estimation and Removal of
Noise from a Single Image
Ce Liu, Student Member, IEEE, Richard Szeliski, Fellow, IEEE, Sing Bing Kang, Senior Member, IEEE,
C. Lawrence Zitnick, Member, IEEE, and William T. Freeman, Senior Member, IEEE
Abstract—Image denoising algorithms often assume an additive white Gaussian noise (AWGN) process that is independent of the
actual RGB values. Such approaches cannot effectively remove color noise produced by today’s CCD digital camera. In this paper, we
propose a unified framework for two tasks: automatic estimation and removal of color noise from a single image using piecewise smooth
image models. We introduce the noise level function (NLF), which is a continuous function describing the noise level as a function of
image brightness. We then estimate an upper bound of the real NLF by fitting a lower envelope to the standard deviations of per-segment
image variances. For denoising, the chrominance of color noise is significantly removed by projecting pixel values onto a line fit to the
RGB values in each segment. Then, a Gaussian conditional random field (GCRF) is constructed to obtain the underlying clean image
from the noisy input. Extensive experiments are conducted to test the proposed algorithm, which is shown to outperform state-of-the-art
denoising algorithms.
Index Terms—Image denoising, piecewise smooth image model, segmentation-based computer vision algorithms, noise estimation,
Gaussian conditional random field, automatic vision system.
Ç
1INTRODUCTION
I
MAGE denoising has been studied for decades in computer
vision, image processing and statistical signal processing.
This problem not only provides a good platform to examine
natural image models and signal separation algorithms, but
it is also important part of image enhancement for digital
image acquisition systems. These two directions are both
important and will be explored in this paper.
Most of the existing image denoising work assumes
additive white Gaussian noise (AWGN) and removes the
noise independently of the RGB image data. However, the
type and level of the noise generated by digital cameras are
unknown if the series and brand of the camera, as well as the
camera settings (ISO, shutter speed, aperture, and flash on/
off), are unknown. For instance, the exchangeable image file
format (EXIF) metadata attached with each picture can be
lost in image format conversion and image file transferring.
Meanwhile, the statistics of the color noise is not indepen-
dent of the RGB channels because of the demosaic process
embedded in cameras. Therefore, most current denoising
approaches are not truly automatic and cannot effectively
remove color noise. This fact prevents the noise removal
techniques from being practically applied to digital image
denoising and enhancing applications.
In some image denoising software, the user is required to
specify a number of smooth image regions to estimate the
noise level. This motivated us to adopt a segmentation-based
approach to automatically estimate the noise level from a
single image. Since the noise level is dependent on the image
brightness, we propose to estimate an upper bound of the
noise level function (NLF) from the image. The image is
partitioned into piecewise smooth regions in which the mean
is the estimate of brightness, and the standard deviation is an
overestimate of noise level. The prior probability of the noise
level functions is learned by simulating the digital camera
imaging process and are used to help estimate the curve
correctly where there is missing data.
Since separating signal and noise from a single input is
underconstrained, it is in theory impossible to completely
recover the original image from the noise contaminated
observation. The goal of image denoising is to preserve image
features as much as possible while eliminating noise. There
are a number of goals we want to meet in designing image
denoising algorithms.
1. The perceptually flat regions should be as smooth as
possible. Noise should be completely removed from
these regions.
2. Image boundaries should be well preserved. This
means that the boundary should not be blurred or
sharpened.
3. Texture detail should not be lost. This is one of the
hardest criteria to match. Since image denoising
algorithms tend to smooth the image, it is easy to
lose texture detail during smoothing.
4. The global contrast should be preserved (that is, the
low frequencies of the denoised and input images
should be identical).
5. No artifacts should appear in the denoised image.
The global co ntrast is probably the easiest to meet,
whereas some of the other principles are nearly incompa-
tible. For instance, goals 1 and 3 are difficult to adjust
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 30, NO. 2, FEBRUARY 2008 299
. C. Liu and W.T. Freeman are with the Computer Science and Artificial
Intelligence Laboratory, Massachusetts Institute of Technology, 32 Vassar
Street, Cambridge, MA 02139. E-mail: {celiu, billf}@mit.edu.
. R. Szeliski, S.B. Kang, and C.L. Zitnick are with Microsoft Research, One
Microsoft Way, Redmond, WA 98052-6399.
E-mail: {szeliski, SingBing.Kang, larryz}@microsoft.com.
Manuscript received 30 Sept. 2006; revised 13 Mar. 2007; accepted 22 Mar.
2007; published online 15 May 2007.
Recommended for acceptance by J. Luo.
For information on obtaining reprints of this article, please send e-mail to:
tpami@computer.org, and reference IEEECS Log Number TPAMI-0695-0906.
Digital Object Identifier no. 10.1109/TPAMI.2007.1176.
0162-8828/08/$25.00 ß 2008 IEEE Published by the IEEE Computer Society
together since most denoising algorithms cannot distin-
guish flat from textured regions within a single input
image. Satisfying goal 5 is important but not always easy,
for example, wavelet-based denoising algorithms tend to
generate ringing artifacts.
Ideally, the same image model should be used for both
noise estimation and denoising. We found that a segmenta-
tion-based approach is well suited to both tasks. After a
natural image is oversegmented into piecewise smooth
regions, the pixel values within each segment approximately
lie on a 1D line in RGB space due to the physics of image
formation [26], [24], [20]. This important fact can help to
significantly reduce color noise. We further improve the
results by constructing a Gaussian conditional random field
(GCRF) to estimate the clean image (signal) from the noisy
image.
Experiments are conducted, with both quantitatively
convincing and visually pleasing results to demonstrate that
our segmentation-based denoising algorithm outperforms
the state of the art. Our approach is distinctively automatic
since the noise level is automatically estimated. Automati-
cally estimating the noise level can benefit other computer
vision algorithms as well. For example, the parameters of
stereo, motion estimation, edge detection, and super resolu-
tion algorithms can be set as a function of the noise level so
that we can avoid tweaking the parameters for different noise
levels [30].
The paper is organized as follows: After reviewing
relevant work in Section 2, we introduce our piecewise
smooth image model in Section 3. In Section 4, we propose
the method for noise estimation from a single image. Our
segmentation-based image denoising algorithm is p re-
sented in detail in Section 5, with results shown in Section 6.
We discuss issues of color noise, modeling, and automation
in Section 7 and provide concluding remarks in Section 8.
2RELATED WORK
In this section, we briefly review previous work on image
denoising and noise estimation. Image denoising techni-
ques differ in the choice of image prior models, and many
noise estimation techniques assume white Gaussian noise
(AWGN).
2.1 Image Denoising
In the past three decades, a variety of denoising methods
have been developed in the image processing and computer
vision communities. Although seemingly very different,
they all share the same property: to keep the meaningful
edges and remove less meaningful ones. We categorize the
existing image denoising work by their different natural
image prior models and the corresponding representation
of natural image statistics.
Wavelets. When a natural image is decomposed into
multiscale-oriented subbands [31], we observe highly
kurtotic marginal distributions [16]. To enforce the marginal
distribution to have high kurtosis, we can simply suppress
low-amplitude v alues while retaining high-amplitude
values, a technique known as coring [40], [44].
In [43], the joint distribution of wavelets were found to
be dependent. A joint coring technique is developed to infer
the wavelet coefficients in a small neighborhood across
different orientation and scale subbands simultaneously.
The typical joint distribution for denoising is a Gaussian
scale mixture (GSM) model [38]. In addition, wavelet-
domain hidden Markov models have been applied to image
denoising with promising results [9], [14].
Although the wavelet-based method is popular and
dominant in denoising, it is hard to remove the ringing
artifacts of wavelet reconstruction. In other words, wavelet-
based meth ods tend to introduce additional edges or
structures in the denoised image.
Anisotropic diffusion. The simplest method for noise
removal is Gaussian filtering, which is equivalent to solving
anisotropicheat diffusion equation [47], a second-order linear
PDE. To keep sharp edges, anisotropic diffusion can be
performed using I
t
¼ divðcðx; y; tÞrIÞ[35], where cðx; y; tÞ¼
gðkrIðx; y; tÞkÞ, and g is a monotonically decreasing function.
As a result, for high gradient pixels, cðx; y; tÞ is small and
therefore gets less diffused. For low gradient pixels, cðx; y; tÞ
has a higher value, and these pixels get blurred with
neighboring pixels. A more sophisticated way of choosing
gðÞis discussed in [4]. Compared to simple Gaussian filtering,
anisotropic diffusion smooths out noise while keeping edges.
However, it tends to overblur the image and sharpen the
boundary with many texture details lost.
More advanced partial differential equations (PDEs)
have been developed so that a specific regulariza tion
process is designed for a given (user-defined) underlying
local smoothing geometry [53], preserving more texture
details than the classical anisotropic diffusion methods.
FRAME and FOE. As an alternative to measuring
marginal or joint distributions on wavelet coefficients, a
complete prior model over the whole image can be learned
from marginal distributions [19], [56]. Thus, it is natural to
use a Bayesian inference for denoising or restoration [55],
[41]. The processed image I typically takes the iterative form
I
t
¼ I
t1
þ
X
n
i¼1
F
1
i
0
i
ðF
i
I
t1
Þþ
1
2
ðI
obs
I
t1
Þ
2
"#
;
where fF
i
g are linear filters (F
1
i
is the filter obtained by
mirroring F
i
around its center pixel), f
i
g are t he
corresponding Gibbs potential functions,
2
is the variance
of noise, and t is the index of iteration. Because the
derivative
0
i
typically has high values close to zero and low
values at high amplitude, the above PDE is very similar to
anisotropic diffusion if the F
i
s are regarded as derivative
filters at different directions [55].
Learning a Gibbs distribution using MCMC can be
inefficient. Furthermore, these methods can have the same
drawbacks as anisotropic diffusion: over smoothing and
edge sharpening.
Bilateral filtering. An alternative way of adapting
Gaussian filtering to preserve edges is bilateral filtering
[52], where both space and range distances are taken into
account. The essential relationship between bilateral filter-
ing and anisotropic diffusion is derived in [2]. A fast
bilateral filtering algorithm is also proposed in [11], [34].
Bilateral filtering has been widely adopted as a simple
algorithm for denoising, for example, video denoising in
[3]. However, it cannot handle speckle noise, and it also has
the tendency to oversmooth and to sharpen edges.
Nonlocal methods. If both the scene and camera are static,
we can simply take multiple pictures and use the mean to
remove the noise. This method is impractical for a single
300 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 30, NO. 2, FEBRUARY 2008
image, but a temporal mean can be computed from a spatial
mean—as long as there are enough similar patterns in the
single image. We can find the similar patterns to a query patch
and take the mean or other statistics to estimate the true pixel
value, for example, in [1], [6]. A more rigorous formulation of
this approach is through sparse coding of the noisy input [12].
Nonlocal methods are an exciting innovation and work
well for texture-like images conta ining many repeated
patterns. However, compared to other denoising algorithms
that have Oðn
2
Þ complexity, where n is the image width,
these algorithms have Oðn
4
Þ time complexity, which is
prohibitive for real-world applications.
Conditional random fields (CRFs). Recently, CRFs [27]
have been a promising model for statistical inference.
Without an explicit prior model on the signal, CRFs are
flexible at modeling short and long range constraints and
statistics. Since the noisy input and the clean image are well
aligned at image features, CRFs, in particular, GCRFs can be
well applied to image denoising. Preliminary success has
been shown in the denoising work in [48]. Learning GCRFs
is also addressed in [50].
2.2 Noise Estimation
Image-dependent noise can be estimated from multiple
images or a single image. Estimation from multiple images
is an overconstrained problem and was addressed in [25].
Estimation from a single imag e, however, is an under-
constrained problem and further assumptions have to be
made for the noise. In the image denoising literature, noise is
often assumed to be additive white Gaussian noise (AWGN).
A widely used estimation method is based on the mean
absolute deviation [10]. In [17], the noise level is estimated
from the gradient of smooth or small textured regions, and the
signal-dependent noise level is estimated for each intensity
interval. In [46], Stefano et al. proposed three methods to
estimate noise levels based on training samples and the
statistics (Laplacian) of natural images. In [37], a generalized
expectation maximization algorithm is proposed to estimate
the spectral features of a noise source corrupting an observed
image.
Techniques for noise estimation followed by noise reduc-
tion have been proposed, but they tend to be heuristic. For
example, in [22], a set of statistics of film grain noise are used
to estimate and remove the noise produced from scanning the
photographic element under uniform exposures. In [45],
signal-dependent noise is estimated from the smooth regions
of the image by segmenting the image gradient with an
adaptive threshold. The estimated signal-dependent noise is
applied to the whole image for noise reduction. This work
was further extended in [21] by associating a default film-
related noise model to the image based on its source
identification tag. The noise model is then adjusted using
the image statistics. In certain commercially available image
enhancement software, such as Neat Image
TM
,
1
the noise level
can be semiautomatically estimated by specifying featureless
areas to profile noise. Neat Image
TM
also provides calibration
tools to estimate the amount of noise for a specific camera and
camera setting; precalibrated noise profiles for various
cameras are also available to directly denoise images.
By comparison, our technique avoids the tedious noise
measurement process for each camera used. Furthermore,
our technique provides a principled way for estimating a
continuous NLF from a single image under the Bayesian
inference framework.
3PIECEWISE SMOOTH IMAGE MODEL
The piecewise smooth image model was used by Terzo-
poulos [51] to account for the regularization of the
reconstructed image. The concept of piecewise smooth (or
continuous) was elaborated by Blake and Zisserman [5]. In
this section, we discuss the reconstruction of piecewise
smooth image model from an image and some important
properties of this model.
3.1 Image Segmentation
Image segmentation algorithms are designed based on
piecewise smooth image prior to partition pixels into regions
with both similar spatial coordinates and RGB pixel values.
There are a variety of segmentation algorithms, including
mean shift [8] and graph-based methods [15]. Since the focus
of this paper is not on segmentation, we choose a simple
K-Means clustering method for grouping pixels into regions,
as described in [57]. Each segment is represented by a mean
color and spatial extent. The spatial extent is computed so that
the shape of the segment is biased toward convex shapes and
that all segments have similar size.
3.2 Segment Statistics and Affine Reconstruction
Let the image lattice be L. It is completely partitioned to a
number of regions f
i
g, where L¼
S
i
i
and
i
\
j
¼;for
i 6¼ j. Let v 2 IR
2
be the coordinate variable, and IðvÞ2IR
3
be
the RGB value of the pixel. Since in this section we focus on the
statistics within each segment, we shall use to represent a
segment and v 2 to index pixels in segment .
We can fit an affine model in segment to minimize the
squared error:
A
¼ arg min
A
X
v2
IðvÞA½v
T
1
T
2
; ð1Þ
where A 2 IR
33
is the affine matrix. We call the reconstruc-
tion fðvÞ¼A
½v
T
1
T
the affine reconstruction of segment .
The residual is rðvÞ¼IðvÞfðvÞ.
We assume that the residual consists of two parts: subtle
texture variation hðvÞ, which is also part of signal, and noise
nðvÞ, that is, rðvÞ¼hðvÞþnðvÞ. In other words, the observed
image can be decomposed into IðvÞ¼fðvÞþhðvÞþnðvÞ. The
underlying clean image or signal is thus sðvÞ¼fðvÞþhðvÞ,
which is to be estimated from the noisy input. sðvÞ, hðvÞ, and
nðvÞ are all 3D vectors in RGB space.
Let the covariance matrices of IðvÞ, sðv
Þ, hðvÞ, and nðvÞ be
I
,
s
,
h
, and
n
, respectively. We assume that fðvÞ is a
nonrandom process, and rðvÞ and nðvÞ are random
variables. Therefore,
s
¼
h
. Suppose signal sðvÞ, and
noise nðvÞ are independent, we have
r
¼
s
þ
n
; ð2Þ
which leads to
r
n
ð3Þ
that is, the variance of the residual bounds the noise
variance.
LIU ET AL.: AUTOMATIC ESTIMATION AND REMOVAL OF NOISE FROM A SINGLE IMAGE 301
1. http://www.neatimage.com.
3.3 Boundary Blur Estimation
If we merely use per-segment affine reconstruction, the
reconstructed image has artificial boundaries, and the
original boundaries would be artificially sharpened. To
avoid that, we estimate the blur from the original image in
the following way. For each hypothesized blur b from
b
min
ð¼ 0Þ to b
max
ð¼ 2:5Þ in steps of bð¼ 0:25Þ, we compute
the blurred image f
blur
ðv; bÞ¼fðvÞGðu; bÞ, where Gðu; bÞ
is a Gaussian kernel with sigma b. We then compute the
error image I
err
such that I
err
ðv; bÞ¼kIðvÞf
blur
ðv; bÞk
2
.
We dilate each boundary curve C
ij
five times into regions
i
and
j
to obtain a mask
ij
. The best blur b
ij
for C
ij
corresponds to the minimum aggregate error I
err
ðv; bÞ over
ij
or b
ij
¼ arg min
b
P
v2
ij
I
err
ðv; bÞ.
To reinstate the blur in the transition region
ij
,we
simply replace fðvÞ with f
blur
ðv; b
ij
Þ. Note that this assumes
that the amount of blur in
i
and
j
is the same, which is
strictly not true in general. However, we found that this
approximation generates satisfactory results. After this
process is done for every pair of region s, we obtain
boundary blurred piecewise affine reconstruction f
blur
ðvÞ.
The piecewise smooth image model is illustrated in
Figs. 1a, 1b, 1c, and 1d. The example image (Fig. 1a) taken
from Berkeley image segmentation database [32] is parti-
tioned to piecewise smooth regions (Fig. 1b) by the segmenta-
tion algorithm. The per-segment affine reconstruction is
shown in Fig. 1c, where we can see artificial boundaries
between regions and the true boundaries are sharpened. After
blur estimation and reinstatement in Fig. 1d, the boundaries
become much smoother.
3.4 Important Properties of the Piecewise Smooth
Image Model
There are three important properties of our piecewise
smooth image model that led us to choose it as the model
for both noise estimation and removal. They are
1. the piecewise smooth image model is consistent with
a sparse image prior;
2. the color distribution per each segment can be well
approximated by a line segment, due to the physics
of image formation [26], [24], [20]; and
3. the standard deviation of residual per each segment
is the upper bound of the noise level in that segment.
The last property follows from (3). For the first two
properties, we again use the example image in Fig. 1a to
examine them. For the reconstructed image (Fig. 1d), we
compute the log histograms of the horizontal and vertical
derivatives and plotted them in Fig. 2. The long tails clearly
show that the piecewise smooth reconstruction match the
high-kurtosis statistics of natural images [33]. This image
model also shares some similarity with the so-called dead
leaves model [29].
For the second property, we compute the eigenvalues and
eigenvectors of the RGB values f IðvÞg in each region. The
eigenvalues are sorted in descending order and displayed in
Fig. 1e. Obviously, the red channel accounts for the majority
of the RGB channels, a fact that proves the first eigenvalue of
each segment is significantly larger than the second eigenva-
lue. Therefore, when we project the pixel values onto the first
302 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 30, NO. 2, FEBRUARY 2008
Fig. 1. Illustration of piecewise smooth image model. (a) Original image. (b) Segmentation. (c) Per-segment affine reconstruction. (d) Affine
reconstruction plus boundary blur. (e) The sorted eigenvalues in each segment. (f) RGB values projected onto the largest eigenvector.
Fig. 2. The log histograms of the (a) horizontal and (b) vertical derivative
filter responses of the reconstruction in Fig. 1d.
eigenvalue while ignoring the other two, we get an almost
perfect reconstruction in Fig. 1f. The mean square error (MSE)
between the original image (Fig. 1a) and projected (Fig. 1f) is
5:31 10
18
or a peak signal to noise ratio (PSNR) of 35.12 dB.
These numbers demonstrate that the RGB values in each
segment lie in a line.
Having demonstrated these properties of our piecewise
smooth image model, we are ready to develop models for
both noise estimation and removal.
4NOISE ESTIMATION FROM A SINGLE IMAGE
Although the standard deviation of each segment of the
image is the upper bound of noise, as shown in (3), it is not
guaranteed that the means of the segments cover the full
range of image intensities and may not give a bound on the
noise for all image intensities. Besides, the estimate of
standard deviation itself is also a random variable, which
has variance as well. Therefore, a rigorous statistical frame-
work is needed for the inference. In this section, we introduce
the noise level functions (NLFs) and a simulation approach to
learn the priors. A Bayesian approach is proposed to infer the
upper bound of the noise level function from a single input.
4.1 Learning a Prior for Noise Level Functions
The noise standard deviation as function of brightness is
called the noise level function (NLF). For a particular brand of
camera and a fixed parameter setting, the NLF can be
estimated by fixing the camera on a tripod, taking multiple
shots toward a static scene and then computing the mean as
the estimate of the brightness and the standard deviation as
the noise level for each pixel of every RGB channel. The
standard deviation as a function of the mean intensity is the
desired NLF. We shall use such an experimentally measured
NLF as the reference method to test our algorithm, although it
is expensive and time consuming.
As an alternative, we propose a simulation-based ap-
proach to obtain NLFs. We build a model for the noise level
functions of (CCD) cameras. We introduce the terms of our
camera noise model, showing the dependence of the NLF on
the camera response function (CR function, the image
brightness as function of scene irradiance). Given a CR
function, we can synthesize realistic camera noise. Thus, from
a parameterized set of CR functions, we derive the set of
possible NLFs. This restricted class of NLFs allows us to
accurately estimate the NLF for i mage intensities not
observed in the image.
4.1.1 Noise Model of a CCD Camera
A CCD digital camera converts the irradiance, the photons
coming into the imaging sensor, to electrons and, finally, to
bits. See Fig. 3 for the imaging pipeline of a CCD camera.
There are five primary noise sources as stated in [25],
namely, fixed pattern noise, dark current noise, shot noise,
amplifier noise, and quantization noise. These noise terms are
simplified in [54]. Following the imaging equation in [54],
we propose the following noise model of a CCD camera:
I ¼ fðL þ n
s
þ n
c
Þþn
q
; ð4Þ
where I is the observed image brightness, fðÞ is CRF, n
s
accounts for all the noise components that are dependent
on irradiance L, n
c
accounts for the independent noise
before gamma correction, and n
q
is additional quantization
and amplification noise. Since n
q
is the minimum noise
attached to the camera, and most cameras can achieve very
low noise, n
q
will be ignored in our model. We assume
noise statistics Eðn
s
Þ¼0, Varðn
s
Þ¼L
2
s
, and Eðn
c
Þ¼0,
Varðn
c
Þ¼
2
c
. Note the linear dependence of the variance
of n
s
on the irradiance L [54].
4.1.2 Camera Response Function
The camera response function models the nonlinear pro-
cesses in a CCD camera that perform tonescale (gamma) and
white balance correction [42]. There are many ways to
estimate CR functions given a set of images taken under
different exposures. To explore the common properties of
many different CR functions, we downloaded 201 real-world
response functions from http://www.cs.c olumbia.e du/
CAVE [23]. Note that we chose only 190 saturated CR
functions since the unsaturated curves are mostly synthetic.
Each CR function is a 1,024-dimensional vector that repre-
sents the dis cretized ½0; 1!½0; 1 function, where both
irradiance L and brightness I are normalized to be in the
range [0, 1]. We use the notation crf(i) to represent the ith
function in the database.
4.1.3 Synthetic CCD Noise
In principle, we could set up optical experiments to measure
precisely for each camera how the noise level changes with
image brightness. However, this would be time consuming
LIU ET AL.: AUTOMATIC ESTIMATION AND REMOVAL OF NOISE FROM A SINGLE IMAGE 303
Fig. 3. CCD camera imaging pipeline, redrawn from [54].
![Fig. 12. Seventeen images are selected from Berkeley image segmentation database [32] to evaluate the proposed algorithm. The file names (numbers) are shown beneath each picture.](/figures/fig-12-seventeen-images-are-selected-from-berkeley-image-2t5p8pef.webp)
