![Fig. 1. Some examples of saliency detection. (a) Input images. (b) Results from one local contrast method [5]. (c) Results from one global contrast method [15]. (d) Results from the background prior based method [18]. (e) Results from the proposed method. (f) Ground truth salient object masks.](/figures/fig-1-some-examples-of-saliency-detection-a-input-images-b-2pfovkbb.webp)
1051-8215 (c) 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TCSVT.2014.2381471, IEEE Transactions on Circuits and Systems for Video Technology
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
1
Copyright (c) 2014 IEEE. Personal use of this material is
permitted.
However, permission to use this material for any other purposes
must be obtained from the IEEE by sending an email to
pubs-permissions@ieee.org
1051-8215 (c) 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TCSVT.2014.2381471, IEEE Transactions on Circuits and Systems for Video Technology
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
2
Abstract—Detection of salient objects from images is gaining
increasing research interest in recent years as it can substantially
facilitate a wide range of content-based multimedia applications.
Based on the assumption that foreground salient regions are
distinctive within a certain context, most conventional approaches
rely on a number of hand designed features and their
distinctiveness measured using local or global contrast. Although
these approaches have shown effective in dealing with simple
images, their limited capability may cause difficulties when
dealing with more complicated images. This paper proposes a
novel framework for saliency detection by first modeling the
background and then separating salient objects from the
background. We develop stacked denoising autoencoders with
deep learning architectures to model the background where latent
patterns are explored and more powerful representations of data
are learnt in an unsupervised and bottom up manner. Afterwards,
we formulate the separation of salient objects from the
background as a problem of measuring reconstruction residuals
of deep autoencoders. Comprehensive evaluations on three
benchmark datasets and comparisons with 9 state-of-the-art
algorithms demonstrate the superiority of the proposed work.
Index Terms—salient object detection, stacked denoising
autoencoder, background prior, deep reconstruction residual.
I. I
NTRODUCTION
ALIENT
object detection aiming to discover the most
important and informative parts in an image is gaining
intensive research attention recently as it can serve as a base for
a large number of multimedia applications such as image
resizing, image montage, action analysis and visual recognition
[1-4]. Based on the underlying hypothesis that the salient
stimulus is distinct from its contextual stimuli, most existing
saliency detection models need to solve two key problems: i)
extract effective features to represent the image and, ii) develop
an optimal mechanism to measure the distinctiveness over the
extracted features.
The performance of saliency detection models heavily relies
Manuscript received on April 14, 2014. This work was partially supported
by the National Science Foundation of China under Grant 61103061, 91120005,
and 61473231.
Junwei Han, Dingwen Zhang, Xintao Hu, and Lei Guo are with School of
Automation, Northwestern Polytechnical University, Xi’an China. (phone and
fax: 86-29-88431318; e-mail: junweihan2010@gmail.com).
Jinchang Ren is with the Department of Electronic and Electrical
Engineering, University of Strathclyde, UK.
Feng Wu is with School of Information Science, University of Science and
Technology of China.
on the features (data representations) being used. In the last 15
years, a variety of features have been proposed for the task of
image saliency detection. The earliest saliency computation
model by Itti et al. [5] proposed three biological plausible
features including color, intensity, and orientation. In Judd et al.
[6], besides Itti's three features, several new features were
introduced to characterize image content, which include the
local energy of the steerable pyramid filters, subband pyramids
based features, 3D color histogram, and horizon line detector.
As visual attention could be directed by specific objects, some
detectors of face, car, and person were treated as features for
detecting saliency [6, 7]. All these feature representations are
hand-designed and require significant amounts of domain
knowledge. However, hand-designed features in general suffer
poor generalization capability for different images, especially
due to the lack of thorough understanding of the biological
mechanisms and principles of human visual attention as well as
weak human intuition involved. A few recent approaches tried
to learn better representations from natural scenes for saliency
detection by using independent component analysis (ICA) [8],
sparse coding [9, 10], and low-rank matrix recovery [11].
Nevertheless, due to the shallow-structured architectures used
these methods still have limited representational power and are
insufficient to capture high-level information and latent
patterns of complex image data. To overcome such drawbacks,
in this paper, we investigate the feasibility of learning more
powerful representation directly from the raw image data itself
in an unsupervised way for the task of saliency detection.
The saliency or distinctiveness is typically measured by
image contrast computation over features, where various
contrast measures have been presented. Depending on the
extent of context in which the contrast is calculated, these
approaches can be classified into local-contrast based methods
and global-contrast based methods. Local-contrast based
methods estimate the saliency of an image pixel or an image
patch by calculating the contrast against its local neighborhood,
and some representative local methods include the
center-surround difference [5, 6, 12, 13], incremental coding
length [10], and self-resemblance [14]. Global-contrast based
methods characterize the saliency of an image region as the
uniqueness in the entire image. Previous literatures have
proposed a variety of approaches to model the global contrast
from different perspectives. To be specific, in [15] and [16] the
global contrast is derived in the frequency domain with the
hypothesis that salient regions are normally less frequent. Han
et al. [9] and Zhang et al. [8] utilized the Gaussian models to
Background Prior Based Salient Object
Detection via Deep Reconstruction Residual
Junwei Han, Dingwen Zhang, Xintao Hu, Lei Guo, Jinchang Ren, and Feng Wu
S
1051-8215 (c) 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TCSVT.2014.2381471, IEEE Transactions on Circuits and Systems for Video Technology
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
3
calculate the global contrast. Cheng et al. [17] proposed to
model the global contrast on the region level where each
region's contrast is generated by a weighted summation of the
differences between itself and all other regions. Shen et al. [11]
represented a whole image as a low-rank matrix with sparse
noises where sparse noises denote the salient regions.
In spite of extensive efforts, local and global contrast based
approaches still suffer from some drawbacks. First, these
approaches normally can only highlight object boundaries but
fail to detect the whole target region uniformly as shown in the
examples given in Fig. 1. This problem may be alleviated in
some global-contrast based methods while the results yielded
are still unsatisfactory. Second, although the salient objects
often present high contrast, the inverse might unnecessarily be
true [11]. In many complex images (as shown in the third
example of Fig. 1), the background contains small-scale
high-contrast patterns which may lead to previous
contrast-based methods fail in such cases.
Essentially, the true aim of salient object detection is to find
objects that are distinctive from the image background. It needs
to calculate the contrast between the objects and the image
background and then select those with high contrast as the
salient objects. However, the local and global contrast-based
methods do not identify which regions form the image
background. They blindly assume the neighboring regions or
the entire image to be the background and then calculate the
contrast between each location and the assumed background.
As their assumed background may not be the real one, the
determined contrast also becomes incorrect, which in turn
reduces the performance of saliency detection. To overcome
these problems, a few emerging methods [18, 19] using
background priors were proposed based on the idea of
modeling the property of background first and thereby
separating salient objects from the background. Specially, Wei
et al. [18] exploited the boundary and connectivity priors about
the background in natural images and detected saliency based
on the geodesic distance. Considering that the salient object
may be partially cropped on the boundary, this work adopts an
existing saliency detection method [33] to compute the saliency
of boundary patches and generates weights for the virtual
background nodes. However, in some challenging images
where the work [33] could not calculate the saliency of
boundary patches precisely, the method of [18] is difficult to
obtain satisfactory results. Yang et al. [19] modeled saliency
detection as a manifold ranking problem and proposed a
two-stage scheme for graph labelling. They represent the image
as a close-loop graph with superpixels as nodes. In saliency
detection, they first use the nodes on the image boundary as
background seeds to rank other nodes in the graph. Then, in the
second stage, they select the salient nodes from the detection
results of the first stage and use them to refine the saliency of
other nodes in the graph. On the assumption that the image
boundary is mostly background, these methods result in a
background template. As a result, the contrast between salient
object and background can be precisely obtained. By
incorporating background priors into traditional contrast-based
methods, they show improved results in saliency detection.
However, existing background prior based methods still
have certain limitations. Typically, there are four scenarios
where performing background prior based saliency detection as
summarized below.
1) The entire image boundary is a large and smoothly
connected region (see the first row of Fig. 1);
2) The regions defined within the image boundary look
different whereas they may share certain latent pattern (see the
second row of Fig. 1);
3) The background is complex (for example, containing
small-scale high-contrast patterns) and regions of image
boundary are different as shown in the third row of Fig. 1;
4) Salient objects significantly touch the image boundary and
parts of them are wrongly considered as background as shown
in the fourth row of Fig. 1.
As can be seen in Fig. 1, existing background prior based
approaches [18] are effective for the first scenario and
moderately effective for the second scenario. However,
unsatisfactory results are produced in dealing with the last two
scenarios. In this paper, we propose a novel background prior
based saliency detection framework using stacked denoising
autoencoder (SDAE) with deep learning architectures. In the
proposed work, SDAE is used to model image background.
Rather than adopting hand-designed features as used in
previous works [18, 19], the deep-structured SDAE is
employed to learn more powerful representation directly from
the raw image data in an unsupervised way, which also enables
to capture the latent pattern of the input data hierarchically. It
thus helps to deal with the second scenario (shown in the
second row of Fig. 1) where the background regions share
latent patterns. Then, the measure of contrast between salient
objects and the background is formulated as the reconstruction
residuals in the deep-structured SDAE. Different from the
previous works [18, 19] which mainly focused on the way to
calculate the similarity or distinctiveness between a certain
image patch and the image boundary, the proposed work pays
more attention to modeling the background regions.
Fig. 1.
Some examples of saliency detection. (a) Input images. (b) Results
from one local contrast method [5]. (c) Results from one global contrast
method [15]. (d) Results from the background prior based method [18]. (e)
Results from the proposed method. (f) Ground truth salient object masks.
1051-8215 (c) 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TCSVT.2014.2381471, IEEE Transactions on Circuits and Systems for Video Technology
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
4
Specifically, the sparsity is considered when training SDAE
models, which is helpful to suppress the saliency of the
background regions. Therefore, it is robust in handling the third
scenario (shown in the third row of Fig. 1) where the most
challenging task is to avoid mis-highlighting the small-scale
high-contrast background regions in the saliency maps. In
addition, the learning process of SDAE with the usage of
stochastic corruption criteria is helpful to train a deep model for
better robustness and feature representation. Thus, the trained
robust SDAE shows promising performance in these scenarios.
Fig. 2 illustrates the workflow of the proposed framework.
First, we down sample the original image to multiple scales to
generate the multi-scale inputs. Afterwards, we explore the
background prior via SDAE and detect salient regions by deep
reconstruction residuals which can reflect the distinctness
between the background and salient regions. Finally, post
processes are applied to integrate the salient object detection
results for each scale of input and generate the final saliency
map by image organization refinement and region smoothing.
The rest of the paper is organized as follows. Section II
introduces the proposed approach in details. Section III
presents experimental results with quantitative evaluation in
comparison with a group of state-of-the-art approaches. Finally,
several concluding remarks are drawn in Section IV.
II. T
HE
P
ROPOSED
A
PPROACH
In this section, we discuss the proposed method for salient
object detection in details. It includes three subsections, which
in turn introduce SDAE, the proposed salient detection
framework, and two useful post-processing steps, respectively.
A. Stacked Denoising Autoencoder (SDAE)
Autoencoders are simple learning neural networks which aim
to transform inputs into outputs with the least possible amount
of distortion for learning latent patterns of the given data. While
conceptually simple, they play an important role in machine
learning and feature representation. More recently,
autoencoders have taken center stage again in the “deep
architecture” approaches [20-23], where autoencoders are
stacked and pre-trained in an unsupervised fashion. These deep
architectures have been shown to lead to state-of-the-art results
on a number of classification and regression problems [24].
As a form of neural network, the classical autoencoder [24] is
an unsupervised learning algorithm that applies
back-propagation and sets the target values of the network
outputs to be equal to the inputs. Specifically, it includes an
encoding process and a decoding process. The encoding
process uses an encoding function
( )f ,
θ
i f
x
to take a
nonlinear mapping from the visible input vector
i
x
to a hidden
representation vector
i
y
by using an affine transformation with
a projection matrix
W
and a bias
b
. Normally, the sigmoid
function
1 (1 ( ))sigm / exp
η η
= + −
is used as the
deterministic mapping as follows:
( ) ( )
f
f , sigm
θ
= = +W
i i i
y x x b
(1)
A decoding function
( )g ,
θ
i g
y
is adopted to map the hidden
representation
i
y
back to a reconstruction representation
i
z
through a similar transformation:
( ) ( )
g
g , sigm ' '
θ
= = +W
i i i
z y y b
(2)
After the decoding process, the obtained reconstruction is
taken as a prediction of input
i
x
. The training of an
autoencoder is to optimize the parameters
={ , }
f
θ
W b
and
={ , }' '
θ
W
g
b
by minimizing the mean-squared reconstruction
error between the training data and their reconstructed data via:
arg ,
f g
,
min L
θ θ
X Z
(3)
2
2
1
1
,
2
X Z
m
i i
i
L || ||
=
= −
∑
x z
(4)
where
={ }, ={ } [1, ]
i i
i m
∈X Z
x z
denote all the training and
reconstructed data, respectively.
Stacked autoencoder (SAE) is a deep learning architecture of
the classical autoencoders, which is built by stacking additional
unsupervised feature learning layers, and can be trained using
greedy methods for each additional layer. Specifically, once the
first layer is trained, the hidden representation of the first layer
can be treated as the input of the second layer. As a result, any
number of the K layers in this deep architecture can be trained
effectively. This deep architecture allows SAE to learn more
complex mapping from the input to hidden representations and
capture the latent patterns which reflects the most homogametic
property shared among the training data.
Fig. 2. The workflow of the proposed framework.
1051-8215 (c) 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TCSVT.2014.2381471, IEEE Transactions on Circuits and Systems for Video Technology
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
5
The stacked denoising autoencoder (SDAE) [25] is an
extension of the SAE. It builds a deep architecture by stacking
multiple layers of the denoising autoencoder (DAE) which
reconstructs the input into a corrupted and partially destroyed
version. By introducing stochastic corruption to the training
samples, SDAE can avoid over-fitting and achieve better learnt
features, where non-trivial features are robust to input noise and
useful for the further tasks. For a two-layered SDAE, it is done
by first corrupting the initial input
∈
X
i
x
into
ɶ
i
x
by using a
stochastic mapping
= ( )
qD |
ɶ ɶ
i i i
x x x
. According to [24, 25],
= ( )
qD |
ɶ ɶ
i i i
x x x
is implemented by randomly selecting a
fraction (10% in this paper) of the input data and forcing them
to be zero. In the bottom layer, corrupted input
ɶ
i
x
is then
mapped to a hidden representation
1 1 1
( )
f
f ,
θ
=
ɶ
i i
y x
from
which we reconstruct a
1 1 1 1
( )
g
g ,
θ
=
i i
z y
.
Once the bottom layer is trained, the hidden representation of
the bottom layer
1
i
y
is henceforth used as the input of the
second layer
2
i
x
to train a new denoising autoencoder as
follows:
( )
qD |=
ɶ ɶ
2 2 2
i i i
x x x
(5)
2
( )
f
f ,
θ
=
ɶ
2 2 2
i i
y x
(6)
2 2
( )
g
g ,
θ
=
ɶ
2 2
i i
z y
(7)
Note that SDAE still minimizes the reconstruction loss
between a clean input
X
and its reconstruction representation
Z
. It thus forces the learning of a far more clever mapping than
the identity, e.g. extracting useful features for denoising [25].
Motivated by the physiological evidence that describing
patterns with less active neurons minimizes the probability of
destructive cross talk, a regularization term that penalizes a
deviation of the expected activation of the hidden units
(representation vector) from a fixed (low) level
ρ
is applied to
constrain the sparsity to the target activation function [26]. By
taking a single layer autoencoder for example, the target
activation function with sparsity constraint can be written as:
arg
f g
sparsity
j
,
ˆ
min L , , ,
θ θ
ρ ρ
X Z
(8)
1
+ KL( || )
X Z X Z
N
j j
j
sparsity
ˆ
ˆ
L , , , L ,
ρ ρ β ρ ρ
=
=
∑
(9)
1
KL( || ) log (1 )log
1
j
j j
ˆ
ˆ
ˆ
ρ ρ
ρ ρ ρ ρ
ρ ρ
−
= + −
−
(10)
where
β
is the weight of the sparsity penalty,
N
is the
number of features in the weight matrix,
ρ
is the target
average activation of the hidden units, and
1
= [ ]
m
j i= j i
ˆ
m
ρ
∑
y
is the average activation of the
j th
hidden unit
j
y
over the
m
training data. The Kullback-Leibler divergence
KL( )
⋅
provides the sparsity constraint. As in sparse coding, a
non-redundant over-complete feature set is learned when
ρ
is
small. Here we set
=0 05
.
ρ
as suggested in [26]. Usually,
training a DAE is straightforward, where the back-propagation
algorithm can be used to compute the gradient of the objective
function [26, 27], and the same target activation function can be
used in all the layers when training SDAE. As the labels of the
input data are not needed in the training process above, the
layer-wise training step is actually unsupervised.
B. Saliency Detection via Deep Reconstruction Residual
As we mentioned in Section I, local and global
contrast-based methods lack the ability to precisely compute
the contrast between foreground objects and the background.
Inspired by the success of [18], this paper develops the
framework along the pipeline of modeling the background and
thereby separating salient objects from the background. We
follow the basic rule of photographic composition and assume
that the image boundary is mostly background. Then, the
contrast between salient object and the background can be more
precisely obtained. Specifically, we separately define four
boundaries for each image as shown in side-specific SDAE
training of Fig. 2. The height of two horizontal boundaries is
then percent of the image height and their width is the image
width. Similarly, the width of two vertical boundaries is then
percent of the image width and their height is the image height.
To valid the assumption that the image boundary is mostly
background, we compute the percentage of foreground pixels
(labeled in the ground truth) within the defined image
boundaries in two widely used databases (the SOD database [40]
and the SED dataset [50]). The statistic result shows that, for
most images, only less than 10% of pixels in the image
boundary are foreground pixels, which demonstrates that our
assumption is reasonable. For the small number of foreground
patches, the learning process of SDAE could decrease their
influence by minimizing the objective function with the
reconstruction error term when modeling the background.
As shown in Fig. 2, the proposed framework mainly consists
of three components: multi-scale inputs generation, salient
region detection via deep reconstruction residual, and post
processing. According to [28, 29], scale is an important factor
for identifying objects of different sizes. Similar to [28], we use
five scales as
1 1 1 1 1
{ , }
2 3 4 5 6
, , ,
of the original image size to
generate multi-scale inputs. It is more sensitive to small objects
at the large scale whereas it is more likely to highlight the inner
regions of large objects at the small scale.
Afterwards, we model the background using SDAEs
described in last subsection and then detect saliency by deep
reconstruction residuals for each scale. Specifically, we
construct four deep residual maps based on four boundaries
(Side-specific deep reconstruction residual maps shown in Fig.
2) and integrate them for the final map, which is referred to as
the separation/combination (SC) approach [19]. Specifically,
each image boundary is divided into patches of
6 6
×
pixels
with an overlapping of 2 pixels in each direction. Afterwards,
we establish the SDAE model with a visible (input) layer with
6 6 3=108
× ×
visible units and two hidden layers. According to