HAL Id: hal-01053888
https://hal.inria.fr/hal-01053888
Submitted on 17 Sep 2014
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entic research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diusion de documents
scientiques de niveau recherche, publiés ou non,
émanant des établissements d’enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
Deblurring Shaken and Partially Saturated Images
Oliver Whyte, Josef Sivic, Andrew Zisserman
To cite this version:
Oliver Whyte, Josef Sivic, Andrew Zisserman. Deblurring Shaken and Partially Saturated Images.
International Journal of Computer Vision, Springer Verlag, 2014, �10.1007/s11263-014-0727-3�. �hal-
01053888�
IJCV manuscript No.
(will be inserted by the editor)
Deblurring Shaken and Partially Saturated Images
Oliver Whyte · Josef Sivic · Andrew Zisserman
Received: date / Accepted: date
Abstract We address the problem of deblurring images de-
graded by camera shake blur and saturated (over-exposed)
pixels. Saturated pixels violate the common assumption that
the image-formation process is linear, and often cause ring-
ing in deblurred outputs. We provide an analysis of ringing
in general, and show that in order to prevent ringing, it is
insufficient to simply discard saturated pixels. We show that
even when saturated pixels are removed, ringing is caused
by attempting to estimate the values of latent pixels that
are brighter than the sensor’s maximum output. Estimating
these latent pixels is likely to cause large errors, and these
errors propagate across the rest of the image in the form of
ringing. We propose a new deblurring algorithm that locates
these error-prone bright pixels in the latent sharp image, and
by decoupling them from the remainder of the latent image,
greatly reduces ringing. In addition, we propose an approx-
imate forward model for saturated images, which allows us
to estimate these error-prone pixels separately without caus-
ing artefacts. Results are shown for non-blind deblurring of
Parts of this work were previously published in the IEEE Workshop on
Color and Photometry in Computer Vision, with ICCV 2011 (
Whyte
et al. 2011
)
O. Whyte
Microsoft Corporation
Redmond, WA, USA
E-mail: oliverw@microsoft.com
J. Sivic
INRIA - Willow Project
Laboratoire d’Informatique de l’Ecole Normale Supérieure
(CNRS/ENS/INRIA UMR 8548), Paris, France
E-mail: josef.sivic@ens.fr
A. Zisserman
Visual Geometry Group
Department of Engineering Science
University of Oxford
Oxford, UK
E-mail: az@robots.ox.ac.uk
real photographs containing saturated regions, demonstrat-
ing improved deblurred image quality compared to previous
work.
Keywords Non-Blind Deblurring · Saturation · Ringing ·
Outliers
1 Introduction
The task of deblurring “shaken” images has received con-
siderable attention recently (
Fergus et al. 2006; Cho and Lee
2009
; Gupta et al. 2010; Joshi et al. 2010; Whyte et al. 2010;
Shan et al. 2008). Significant progress has been made to-
wards reliably estimating the point spread function (PSF)
for a given blurred image, which describes how the image
was blurred. Likewise, when the PSF for an image is known,
many authors have proposed methods to invert the blur pro-
cess and recover a high quality sharp image (referred to as
“non-blind deblurring”).
One problematic feature of blurred images, and in par-
ticular “shaken” images, which has received relatively little
attention is the presence of saturated (over-exposed) pixels.
These arise when the radiance of the scene exceeds the range
of the camera’s sensor, leaving bright highlights clipped at
the maximum output value (e.g. 255 for an image with 8
bits per pixel). To anyone who has attempted to take hand-
held photographs at night, such pixels should be familiar as
the conspicuous bright streaks left by electric lights, such as
in Figure
1a. These bright pixels, with their clipped values,
violate the assumption made by most deblurring algorithms
that the image formation process is linear, and as a result can
cause obtrusive artefacts in the deblurred images. This can
be seen in the deblurred images in Figures
1b and 1c.
In this paper we address the problem of deblurring im-
ages containing saturated pixels, offering an analysis of the
artefacts caused by existing algorithms, and a new algorithm
2 Oliver Whyte et al.
which avoids such artefacts by explicitly handling saturated
pixels. Our method is applicable for all causes of blur, how-
ever in this work we focus on blur caused by camera shake
(motion of the camera during the exposure).
The process of deblurring an image typically involves
two steps. First, the PSF is estimated, either using a blind
deblurring algorithm (
Fergus et al. 2006; Yuan et al. 2007;
Shan et al. 2008; Cho and Lee 2009) to estimate the PSF
from the blurred image itself, or by using additional hard-
ware attached to the camera (
Joshi et al. 2010; Tai et al.
2008
). Second, a non-blind deblurring algorithm is applied
to estimate the sharp image, given the PSF. In this work we
address the second of these two steps for the case of satu-
rated images, and assume that the PSF is known or has been
estimated already. Unless otherwise stated, all the results in
this work use the algorithm of
Whyte et al. (2011) to esti-
mate a spatially-variant PSF. The algorithm is based on the
method of Cho and Lee (2009), and estimates the PSF di-
rectly from the blurred image itself. Figure
1d shows the
output of the proposed algorithm, which contains far fewer
artefacts than the two existing algorithms shown for com-
parison.
1.1 Related Work
Saturation has not received wide attention in the literature,
although several authors have cited it as the cause of arte-
facts in deblurred images (
Fergus et al. 2006; Cho and Lee
2009
; Tai et al. 2011). Harmeling et al. (2010b) address
the issue in the setting of multi-frame blind deblurring by
thresholding the blurred image to detect saturated pixels,
and ignoring these in the deblurring process. When multiple
blurred images of the same scene are available, these pixels
can be safely discarded, since there will generally remain
unsaturated pixels covering the same area in other images.
Recently,
Cho et al. (2011) have also considered satu-
rated pixels, in the more general context of non-blind deblur-
ring with outliers, and propose an expectation-maximisation
algorithm to iteratively identify and exclude outliers in the
blurred image. Saturated pixels are detected by blurring the
current estimate of the sharp image and finding places where
the result exceeds the range of the camera. Those blurred
pixels detected as saturated are ignored in the subsequent it-
erations of the deblurring algorithm. In Section
4 we discuss
why simply ignoring saturated pixels is, in general, not suf-
ficient to prevent artefacts from appearing in single-image
deblurring.
In an alternative line of work, several authors have pro-
posed algorithms for non-blind deblurring that are robust
against various types of modeling errors, without directly
addressing the sources of those errors.
Yuan et al. (2008)
propose a non-blind deblurring algorithm capable of sup-
pressing “ringing” artefacts during deblurring, using multi-
scale regularisation. Yang et al. (2009) and Xu and Jia (2010)
also consider non-blind deblurring with robust data-fidelity
terms, to handle non-Gaussian impulse noise, however their
formulations do not handle arbitrarily large deviations from
the linear model, such as can be caused by saturation.
Many algorithms exist for non-blind deblurring in the
linear (non-saturated) case, perhaps most famously the
Wiener filter (
Wiener 1949) and the Richardson-Lucy algo-
rithm (
Richardson 1972; Lucy 1974). Recently, many au-
thors have focused on the use of regularisation, derived from
natural image statistics, to suppress noise in the output while
encouraging sharp edges to appear (
Krishnan and Fergus
2009
; Levin et al. 2007; Joshi et al. 2010; Afonso et al. 2010;
Tai et al. 2011; Zoran and Weiss 2011).
For the problem of “blind” deblurring, where the PSF
is unknown, single-image blind PSF estimation for camera
shake has been widely studied using variational and max-
imum a posteriori (MAP) algorithms (
Fergus et al. 2006;
Shan et al. 2008; Cho and Lee 2009; Cai et al. 2009; Xu and
Jia 2010
; Levin et al. 2011; Krishnan et al. 2011). Levin et al.
(2009) review several approaches and provide a ground-truth
dataset for comparison on spatially-invariant blur. While
most work has focused on spatially-invariant blur, several
approaches have also been proposed for spatially-varying
blur (Whyte et al. 2010; Gupta et al. 2010; Harmeling et al.
2010a
; Joshi et al. 2010; Tai et al. 2011).
The remainder of this paper proceeds as follows: We be-
gin in Section
2 by providing some background on non-blind
deblurring and saturation in cameras. In Section
3 we anal-
yse some of the properties and causes of “ringing” artefacts
(which are common when deblurring saturated images), and
discuss the implications of this analysis in Section
4. Based
on this discussion, in Section
5 we describe our proposed
approach. We present deblurring results and comparison to
related work in Section
6.
2 Background
In most existing work on deblurring, the observed image
produced by a camera is modelled as a linear blur operation
applied to a sharp image, followed by a random noise pro-
cess. Under this model, an observed blurred image g (writ-
ten as an N × 1 vector, where N is the number of pixels in
the image) can be written in terms of a (latent) sharp image
f (also an N × 1 vector) as
g
∗
= Af (1)
g = g
∗
+ ε, (2)
where A is an N × N matrix representing the discrete PSF,
g
∗
represents the “noiseless” blurred image, and ε is some
random noise affecting the image. Typically, the noise ε is
Deblurring Shaken and Partially Saturated Images 3
(a) Blurred image with saturated pixels (b) Deblurred with the Richardson-Lucy algorithm
(c) Deblurred with the method of Krishnan and Fergus
(2009)
(d) Deblurred with the
proposed approach
(a)
(b)
(c)
(d)
Fig. 1. Deblurring in the presence of saturation. Existing deblurring methods, such as those in (
b) and (c), do not take account of saturated
pixels. This leads to large and unsightly artefacts in the results, such as the “ringing” around the bright lights in the zoomed section. Using the
proposed method (d), the ringing is greatly reduced and the quality of the deblurred image is improved. The PSF for this 1024 × 768 pixel image
causes a blur of about 35 pixels in width, and was estimated directly from the blurred image using the algorithm described by
Whyte et al. (2012).
modelled as following either a Poisson or Gaussian distribu-
tion, independent at each pixel.
For many causes of blur, the matrix A can be parame-
terised by a small set of weights w, often referred to as a
blur kernel, such that
A =
X
k
w
k
T
k
, (3)
where each N ×N matrix T
k
applies some geometric trans-
formation to the sharp image f. Classically, T
k
have been
chosen to model translations of the sharp image, allowing
Equation (
1) to be written as a 2D convolution of f with w.
For blur caused by camera shake (motion of the camera dur-
ing exposure), recent work (
Gupta et al. 2010; Joshi et al.
2010
; Whyte et al. 2010; Tai et al. 2011) has shown that us-
ing projective transformations for T
k
is more appropriate,
and leads to more accurate modeling of the spatially-variant
blur caused by camera shake. The remainder of this work is
agnostic to the form of A, and thus can be applied equally
to spatially-variant and spatially-invariant blur.
Non-blind deblurring (where A is known) is generally
performed by attempting to solve a minimisation problem
of the form
min
f
L(g, Af ) + αφ(f), (4)
where the data-fidelity term L penalises sharp images that
do not closely fit the observed data (i.e. L is a measure of
“distance” between g and Af ), and the regularisation term
φ penalises sharp images that do not adhere to some prior
model of sharp images. The scalar weight α balances the
contributions of the two terms in the optimisation.
In a probabilistic setting, where the random noise ε is as-
sumed to follow a known distribution, the data-fidelity term
can be derived from the negative log-likelihood:
L(g, Af ) = − log p(g|Af ), (5)
where p(g|Af) denotes the probability of observing the
blurry image g, given a sharp image f and PSF A (often re-
ferred to as the likelihood). If the noise follows pixel-
independent Gaussian distributions with uniform variance,
4 Oliver Whyte et al.
the appropriate data-fidelity term is
L
G
(g, Af ) =
X
i
g
i
− (Af )
i
2
, (6)
where (Af)
i
indicates the i
th
element of the vector Af . With
Gaussian noise, Equation (
4) is typically solved using stan-
dard linear least-squares algorithms, such as conjugate gra-
dient descent (Levin et al. 2007). For the special case of
spatially-invariant blur, and provided that the regularisation
term φ can also be written as a quadratic function of f, Equa-
tion (
4) has a closed-form solution in the frequency domain,
which can be computed efficiently using the fast Fourier
transform (
Wiener 1949; Gamelin 2001).
If the noise follows a Poisson distribution, the appropri-
ate data-fidelity term is
L
P
(g, Af ) = −
X
i
g
i
log(Af )
i
− (Af )
i
. (7)
The classic algorithm for deblurring images with Poisson
noise is the Richardson-Lucy algorithm (
Richardson 1972;
Lucy 1974), an iterative algorithm described by a simple
multiplicative update equation. The incorporation of regu-
larisation terms into this algorithm has been addressed by
Tai et al. (2011) and Welk (2010). We discuss this algorithm
further in Section
5.
A third data-fidelity term that is more robust to outliers
than the two mentioned above, and which has been applied
for image deblurring with impulse noise is the ℓ
1
norm (
Bar
et al. 2006
; Yang et al. 2009) (corresponding to noise with a
Laplacian distribution):
L
L
(g, Af ) =
X
i
g
i
− (Af )
i
. (8)
Although this data-fidelity term is more robust against noise
values ε
i
with large magnitudes, compared to the Gaussian
or Poisson data-fidelity terms, it still produces artefacts in
the presence of saturation (
Cho et al. 2011).
For clarity, in the remainder of the paper we denote the
data-fidelity term L(g, Af) simply as L(f ), since we con-
sider the blurred image g and the PSF matrix A to be fixed.
2.1 Sensor Saturation
Sensor saturation occurs when the radiance of the scene
within a pixel exceeds the camera sensor’s range, at which
point the sensor ceases to integrate the incident light, and
produces an output that is clamped to the largest output value.
This introduces a non-linearity into the image formation pro-
cess that is not modelled by Equations (
1) and (2). To cor-
rectly describe this effect, our model must include a non-
linear function R, which reflects the sensor’s non-linear re-
sponse to incident light. This function is applied to each
❞
(a) 0.05s
No saturation
(b) 0.2s
Some saturation
(c) 0.8s
Heavy saturation
Three different exposures of a scene containing bright lights
0 0.5 1 1.5 2
0
0.5
1
(d) Intensities from the 0.2s
exposure (
b) plotted on the
vertical axis, against the
0.05s exposure (a) on the
horizontal axis
0 0.5 1 1.5 2
0
0.5
1
(e) Intensities from the 0.8s
exposure (
c) plotted on the
vertical axis, against the
0.05s exposure (a) on the
horizontal axis
Fig. 2. Saturated and unsaturated photos of the same scene. (
a)–(c)
3 different exposure times for the same scene, with bright regions that
saturate in the longer exposures. A small window has been extracted,
which is unsaturated at the shortest exposure, and increasingly satu-
rated in the longer two. (
d) Scatter plot of the intensities in the small
window in (
b) against those in the window in (a), normalised by expo-
sure time. (
e) Scatter plot of the intensities in the window in (c) against
the window in (
a), normalised by exposure time. The scatter plots in
(
d) and (e) clearly show the clipped linear relationship expected.
pixel of the image before it is output by the sensor, i.e.
g
i
= R
g
∗
i
+ ε
i
, (9)
where ε
i
represents the random noise on pixel i.
For the purpose of describing saturation, we model the
non-linear response function R as a truncated linear func-
tion, i.e. R(x) = min(x, 1), for intensities scaled to lie in
the range [0, 1]. This model is supported by the data in Fig-
ure
2, which shows the relationship between pixel intensi-
ties in three different exposures of a bright light source. The
pixel values in the short exposure (with no saturation) and
the longer exposures (with saturation) clearly exhibit this
clipped linear relationship. As the length of the exposure in-
creases, more pixels saturate.
Due to the non-linearity in the image formation process,
simply applying existing methods for non-blind deblurring
(which assume a linear image formation model) to images
affected by saturation, produces deblurred images which ex-
hibit severe artefacts in the form of “ringing” – medium
frequency ripples that spread across the image, e.g. in Fig-
ures
1b and 1c. Clearly, the saturation must be taken into
![Fig. 7. Power spectra of blur and gradient filters. The power spectrum of the 1D blur kernel in Figure 4 (solid blue line), and the power spectrum of the derivative filter [1,−1] (dashed red line), often used for regularisation in non-blind deblurring. The blur kernel has minima in its power spectrum, which correspond to poorly-constrained frequencies in the deblurred solution. At the low-frequency minima (e.g. near 5 cycles/image), the gradient filter has also lost most of its power, and so ringing at these frequencies is unlikely to be suppressed by gradientbased regularisation](/figures/fig-7-power-spectra-of-blur-and-gradient-filters-the-power-8r2k4czk.webp)
![Fig. 6. Synthetic example of deblurring with saturated pixels. (a) A sharp image f∗ containing some bright pixels with intensity twice the maximum image value. (b) The sharp image f is convolved with w and clipped at 1 to simulate saturation. No noise is added in this example. (c) The mask of pixels in g that are not saturated, i.e. mi = 1 (white) if (f∗ ∗ w)i < 1, and mi = 0 (black) otherwise. (d) The mask of pixels in g that are not influenced by the bright pixels in f∗, i.e. mi = 1 if ( (f∗ ≥ 1) ∗ w ) i = 0. See Section 5.1 for further explanation. The following rows show the results of deblurring g with 1000 iterations of the Richardson-Lucy algorithm, starting from three different initialisations. (1st row) initialised with blurred image. (2nd row) initialised with random values in [0, 1]. (3rd row) initialised with true sharp image. As can be seen, in (f), ringing appears regardless of the initialisation, indicating that L is encouraging this to occur. In (g), ringing appears with some initialisations but not others, indicating that L does not encourage it, but does not suppress it either. In (h), ringing is almost gone, since we have removed the most destabilising data. Counterintuitively, although we have discarded more data, we end up with less ringing. Note that by removing all the blurred pixels shown in (d), we have no information about the bright pixels in f , and they simply retain their initial values. Note also that the deblurred images f̂ may contain pixel values greater than 1, however they are clamped to 1 before writing the image to file](/figures/fig-6-synthetic-example-of-deblurring-with-saturated-pixels-1yadullu.webp){kind=tracking}
