Saliency Detection via Absorbing Markov Chain
Bowen Jiang
1
, Lihe Zhang
1
, Huchuan Lu
1
, Chuan Yang
1
, and Ming-Hsuan Yang
2
1
Dalian University of Technology
2
University of California at Merced
Abstract
In this paper, we formulate saliency detection via ab-
sorbing Markov chain on an image graph model. We joint-
ly consider the appearance divergence and spatial distri-
bution of salient objects and the background. The virtual
boundary nodes are chosen as the absorbing nodes in a
Markov chain and the absorbed time from each transient
node to boundary absorbing nodes is computed. The ab-
sorbed time of transient node measures its global similar-
ity with all absorbing nodes, and thus salient objects can
be consistently separated from the background when the
absorbed time is used as a metric. Since the time from
transient node to absorbing nodes relies on the weights on
the path and their spatial distance, the background region
on the center of image may be salient. We further exploit
the equilibrium distribution in an ergodic Markov chain to
reduce the absorbed time in the long-range smooth back-
ground regions. Extensive experiments on four benchmark
datasets demonstrate robustness and efο¬ciency of the pro-
posed method against the state-of-the-art methods.
1. Introduction
Saliency detection in computer vision aims to ο¬nd the
most informative and interesting region in a scene. It has
been effectively applied to numerous computer vision tasks
such as content based image retrieval [32] , image segmen-
tation [30], object recognition [24] and image adaptation
[21]. Existing methods are developed with bottom-up visu-
al cues [19, 10, 26, 34] or top-down models [4, 36].
All bottom-up saliency methods rely on some prior
knowledge about salient objects and backgrounds, such
as contrast, compactness, etc. Different saliency method-
s characterize the prior knowledge from different perspec-
tives. Itti et al. [16] extract center-surround contrast at mul-
tiple spatial scales to ο¬nd the prominent region. Bruce et al.
[6] exploit Shannons self-information measure in local con-
text to compute saliency. However, the local contrast does
not consider the global inο¬uence and only stands out at ob-
ject boundaries. Region contrast based methods [8, 17] ο¬rst
segment the image and then compute the global contrast of
those segments as saliency, which can usually highlight the
entire object. Fourier spectrum analysis has also been used
to detect visual saliency [15, 13]. Recently, Perazzi et al.
[25] unify the contrast and saliency computation into a s-
ingle high-dimensional Gaussian ο¬ltering framework. Wei
et al. [33] exploit background priors and geodesic distance
for saliency detection. Yang et al. [35] cast saliency detec-
tion into a graph-based ranking problem, which performs
label propagation on a sparsely connected graph to char-
acterize the overall differences between salient object and
background.
In this work, we reconsider the properties of Markov ran-
dom walks and their relationship with saliency detection.
Existing random walk based methods consistently use the
equilibrium distribution in an ergodic Markov chain [9, 14]
or its extensions, e.g. the site entropy rate [31] and the
hitting time [11], to compute saliency, and have achieved
success in their own aspects. However, these models stil-
l have some certain limitations. Typically, saliency mea-
sure using the hitting time often highlights some particu-
lar small regions in objects or backgrounds. In addition,
equilibrium distribution based saliency models only high-
light the boundaries of salient object while object interior
still has low saliency value. To address these issues, we in-
vestigate the properties of absorbing Markov chains in this
work. Given an image graph as Markov chain and some
absorbing nodes, we compute the expected time to absorp-
tion (i.e. the absorbed time) for each transient node. The
nodes which have similar appearance (i.e. large transition
probabilities) and small spatial distance to absorbing nodes
can be absorbed faster. As salient objects seldom occupy all
four image boundaries [33, 5] and the background regions
often have appearance connectivity with image boundaries,
when we use the boundary nodes as absorbing nodes, the
random walk starting in background nodes can easily reach
the absorbing nodes. While object regions often have great
contrast to the image background, it is difο¬cult for a ran-
dom walk from object nodes to reach these absorbing nodes
(represented by boundary nodes). Hence, the absorbed time
starting from object nodes is longer than that from back-
ground nodes. In addition, in a long run, the absorbed time
with similar starting nodes is roughly the same. Inspired
1
Figure 1. The time property of absorbing Markov chain and ergod-
ic Markov chain. From left to right: input image with superpixels
as nodes; the minimum hitting time of each node to all boundary
nodes in ergodic Markov chain; the absorbed time of each node
into all boundary nodes in absorbing Markov chain. Each kind of
time is normalized as a saliency map respectively.
by these observations, we formulate saliency detection as a
random walk problem in the absorbing Markov chain.
The absorbed time is not always effective especially
when there are long-range smooth background regions near
the i mage center. We further explore the effect of the equi-
librium probability in saliency detection, and exploit it to
regulate the absorbed time, thereby suppressing the salien-
cy of this kind of regions.
2. Related Work
Previous works that simulate saliency detection in ran-
dom walk model include [9, 14, 11, 31]. Costa et al. [9]
identify the saliency region based on the frequency of visits
to each node at the equilibrium of the random walk. Harel
et al. [14] extend the above method by deο¬ning a dissimi-
lar measure to model the transition probability between two
nodes. In [31], Wang et al. introduce the entropy rate and
incorporate the equilibrium distribution to measure the av-
erage information transmitted from a node to the others at
one step, which is used to predict visual attention. A ma-
jor problem using the equilibrium distribution is that this
approach often only highlights the texture and boundary re-
gions rather than the entire object, as the equilibrium prob-
ability in the cluttered region is larger than in homogeneous
region when using the dissimilarity of two nodes to rep-
resent their transition probability. Furthermore, the main
objectives in [9, 14, 31] are to predict human ο¬xations on
natural images as opposed to identifying salient regions that
correspond to objects, as illustrated in this paper.
The approach most related to ours is Gopalakrishnan et
al. [11], which exploits the hitting time on the fully con-
nected graph and the s parsely connected graph to ο¬nd the
most salient seed, based on which some background seed-
s are determined again. They then use the difference of the
hitting times to the two kinds of seeds to compute the salien-
cy for each node. While they alleviate the problem of using
the equilibrium distribution to measure saliency, the iden-
tiο¬cation of the salient seed is difο¬cult, especially for the
scenes with complex salient objects. More importantly, the
hitting time based saliency measure prefers to highlight the
global rare regions and does not suppress the backgrounds
very well, thereby decreasing the overall saliency of object-
s (See Figure 1). This can be explained as follows. The
hitting time is the expected time taken to reach a node if
the Markov chain is started in another node. The ergodic
Markov chain doesnβt have a mechanism that can synthet-
ically consider the relationships between a node and mul-
tiple speciο¬c nodes (e.g. seed nodes). In [11], to describe
the relevance of a node to background seeds, they use the
minimum hitting time to take all the background seeds into
account. The minimum time itself is sensitive to some noise
regions in the image.
Different from the above methods, we consider the ab-
sorbing Markov random walk, which includes two kinds of
nodes (i.e. absorbing nodes and transient nodes), to mea-
sure saliency. For an absorbing chain started in a transien-
t node, the probability of absorption in an absorbing node
indicates the relationship between the two nodes, and the
absorption time therefore implicates the selective relation-
ships between this transient node and all the absorbing n-
odes. Since the boundary nodes usually contain the global
characteristics of the image background, by using them as
absorbing nodes, the absorbed time of each transient node
can reο¬ect its overall similarity with the background, which
helps to distinguish salient nodes from background nodes.
Moreover, as the absorbed time is the expected time to all
the absorbing nodes, it covers the effect of all the bound-
ary nodes, which can alleviate the inο¬uence of particular re-
gions and encourage the similar nodes in a local context to
have the similar saliency, thereby overcoming the defects of
using the equilibrium distribution [9, 14, 11, 31]. Different
from [9, 14] which directly use the equilibrium distribution
to simulate human attention, we exploit it to weigh the ab-
sorbed time, thereby suppressing the saliency of long-range
background regions with homogeneous appearance.
3. Principle of Markov Chain
Given a set of states π = {π
1
,π
2
,...,π
π
},aMarkov
chain can be completely speciο¬ed by the π Γ π transition
matrix P, in which π
ππ
is the probability of moving from
state π
π
to state π
π
. This probability does not depend up-
on which state the chain is in before the current state. The
chain starts in some state and move from one state to anoth-
er successively.
3.1. Absorbing Markov Chain
The state π
π
is absorbing when π
ππ
=1, which means
π
ππ
=0for all π β= π. A Markov chain is absorbing if it has
at least one absorbing state. It is possible to go from every
transient state to some absorbing state, not necessarily in
one step. Considering an absorbing chain with π absorbing
2
states and π‘ transient states, renumber the states so that the
transient states come ο¬rst, then the transition matrix P has
the following canonical form,
P β
ξ
QR
0I
ξ
, (1)
where the ο¬rst π‘ states are transient and the last π states are
absorbing. Q β [0, 1]
π‘Γπ‘
contains the transition probabili-
ties between any pair of transient states, while R β [0, 1]
π‘Γπ
contains the probabilities of moving from any transient state
to any absorbing state. 0 is the π Γ π‘ zero matrix and I is the
π Γ π identity matrix.
For an absorbing chain, we can derive its fundamental
matrix N =(I β Q)
β1
, where π
ππ
can be interestingly
interpreted as the expected number of times that the chain
spends in the transient state π given that the chain starts in
the transient state π , and the sum
ξ
π
π
ππ
reveals the expect-
ed number of times before absorption (into any absorbing
state). Thus, we can compute the absorbed time for each
transient state, that is,
y = N Γ c, (2)
where c is a π‘ dimensional column vector all of whose ele-
ments are 1.
3.2. Ergodic Markov Chain
An ergodic Markov chain is one in which it is possi-
ble to go from every state to every state, not necessarily
in one step. An ergodic chain with any starting state always
reaches equilibrium after a certain time, and the equilibri-
um state is characterized by the equilibrium distribution π,
which satisο¬es the equation
πP = π, (3)
where P is the ergodic transition matrix. π is a strictly
positive probability vector, where π
π
describes the expected
probability of the chain staying in state π
π
in equilibrium.
When the chain starts in state π
π
, the mean recurrent time β
π
(i.e., the expected number of times to return to state π
π
) can
be derived from the equilibrium distribution π. That is,
β
π
=
1
π
π
, (4)
where π indexes all the states in the ergodic Markov chain.
The more states there are similar to state π
π
nearby, the less
β
π
is. The derivation details and proofs can be found in [12].
3.3. Saliency Measure
Given an input image represented as a Markov chain
and some background absorbing states, the saliency of each
transient state is deο¬ned as the expected number of times
Figure 2. Illustration of the absorbing nodes. The superpixels out-
side the yellow bounding box are the duplicated boundary super-
pixels, which are used as the absorbing nodes.
before being absorbed into all absorbing nodes by Eq 2. In
this work, the transition matrix is constructed on a sparse-
ly connected graph, where each node corresponds to a s-
tate. Because we compute the full resolution saliency map,
some virtual nodes are added to the graph as absorbing s-
tates, which is detailed in the next section.
In the conventional absorbing Markov chain problems,
the absorbing nodes are manually labelled with the ground-
truth. However, as absorbing nodes for saliency detection
are selected by the proposed algorithm, some of them may
be incorrect. They have insigniο¬cant effects on the ο¬nal
results, which are explained in the following sections.
4. Graph Construction
We construct a single layer graph πΊ(π,πΈ) with super-
pixels [3] as nodes π and the links between pairs of nodes
as edges πΈ. Because the salient objects seldom occupy all
image borders [33], we duplicate the boundary superpixels
around the image borders as the virtual background absorb-
ing nodes, as shown in Figure 2. On this graph, each node
(transient or absorbing) is connected to the transient nodes
which neighbour it or share common boundaries with its
neighbouring nodes. That means that any pair of absorb-
ing nodes are unconnected. In addition, we enforce that all
the transient nodes around the image borders (i.e., bound-
ary nodes) are fully connected with each other, which can
reduce the geodesic distance between similar superpixels.
The weights of the edges encode nodal afο¬nity such that n-
odes connected by an edge with high weight are considered
to be strongly connected and edges with low weights repre-
sent nearly disconnected nodes. In this work, the weight π€
ππ
of the edge π
ππ
between adjacent nodes π and π is deο¬ned as
π€
ππ
= π
β
β₯π₯
π
βπ₯
π
β₯
π
2
, (5)
where π₯
π
and π₯
π
are the mean of two nodes in the CIE LAB
color space, and π is a constant that controls the strength of
3
the weight. We ο¬rst renumber the nodes so that the ο¬rst π‘
nodes are transient nodes and the last π nodes are absorbing
nodes, then deο¬ne the afο¬nity matrix A which represents
the reverence of nodes as
π
ππ
=
ξ
ξ
ξ
π€
ππ
π β π(π), 1 β€ π β€ π‘
1 if π = π
0 otherwise
(6)
where π (π) denotes the nodes connected to node π.The
degree matrix that records the sum of the weights connected
to each node is written as
D = diag(
ξ
π
π
ππ
). (7)
Finally, the transition matrix P on the sparsely connected
graph is given as
P = D
β1
Γ A, (8)
which is actually the r aw normalized A. As the nodes are
locally connected, P is a sparse matrix with a small number
of nonzero elements.
The sparsely connected graph restricts the random walk
to only move within a local region in each step, hence the
expected time spent to move from transient node π£
π‘
to ab-
sorbing node π£
π
is determined by two major factors. One
is the spatial distance between the two nodes. Their dis-
tance is larger, and the expected time is longer. The other is
the transition probabilities of the nodes along the different
paths from π£
π‘
to π£
π
. Large probabilities are able to shorten
the expected time to absorption. Given starting node π£
π‘
,the
shorter the time is, the larger the probability of absorption
in node π£
π
is in a long run.
5. Saliency Detection
Given the transition matrix P by Eq. 8, we can easily
extract the matrix Q by Eq. 1, based on which the funda-
mental matrix N is computed. Then, we obtain the saliency
map S by normalizing the absorbed time y computed by
Eq. 2 to the range between 0 and 1, that is
S(π)=
y(π) π =1, 2,...,π‘, (9)
where π indexes the transient nodes on graph, and
y denotes
the normalized absorbed time vector.
Most saliency maps generated by the normalized ab-
sorbed time
y are effective, but some background nodes
near the image center may not be adequately suppressed
when they are in long-range homogeneous region, as shown
in Figure 3. That can be explained as follows. Most n-
odes in this kind of background regions have large transi-
tion probabilities, which means that the random walk may
transfer many times among these nodes before reaching the
Figure 3. Examples showing the beneο¬ts of the update process-
ing. From left to right, input images, results without and with the
update processing.
absorbing nodes. The sparse connectivity of the graph re-
sults that the background nodes near the image center have
longer absorbed time than the similar nodes near the im-
age boundaries. Consequently, the background regions n-
ear the image center possibly present comparative saliency
with salient objects, thereby decreasing the contrast of ob-
jects and backgrounds in the resulted saliency maps. To
alleviate this problem, we update the saliency map by using
a weighted absorbed time y
w
, which can be denoted as:
y
w
= N Γ u, (10)
where u is the weighting column vector. In this work, we
use the normalized recurrent time of an ergodic Markov
chain, of which the transition matrix is the row normalized
Q, as the weight u.
The equilibrium distribution π for the ergodic Markov
chain can be computed from the afο¬nity matrix A as
π
π
=
ξ
π
π
ππ
ξ
ππ
π
ππ
, (11)
where π, π index all the transient nodes. Since we deο¬ne the
edge weight π€
ππ
as the similarity between two nodes, the
nodes within the homogeneous region have large weighted
sum
ξ
π
π
ππ
. This means the recurrent time in this kind of
region is small as shown in Figure 3. For this reason, we use
the average recurrent time β
π
of each node π to weight the
corresponding element π
ππ
(i.e., the expected time spending
in node π before absorption given starting node π ) in each
row of the fundamental matrix N. Precisely, given the e-
quilibrium distribution π, β
π
is computed by Eq. 4 and the
weighting vector u is computed as:
π’
π
=
β
π
ξ
π
β
π
, (12)
where π and π index all the transient nodes on graph.
By the update processing, the saliency of the long-range
homogeneous regions near the image center can be sup-
pressed as Figure 3 illustrates. However, if the kind of re-
gion belongs to salient object, its saliency will be also in-
correctly suppressed. Therefore, we deο¬ne a principle to
4
Figure 4. Examples in which the salient objects appear at the image
boundaries. From top to down: input images, our saliency maps.
decide which maps need to be further updated. We ο¬nd that
object regions have great global contrast to background re-
gions in good saliency maps, while it is not the case in the
defective maps as the examples in Figure 3, which consis-
tently contain a number of regions with mid-level saliency.
Hence, given a saliency map, we ο¬rst calculate its gray his-
togram g with ten bins, and then deο¬ne a metric π ππππ to
characterize this kind of tendency as follows:
π ππππ =
10
ξ
π=1
π(π) Γ min(π, (11 β π)), (13)
where π indexes all the bins. The larger π ππππ means that
there are longer-range regions with mid-level saliency in the
saliency map.
It should be noted that the absorbing nodes may in-
clude object nodes when the salient objects touch the im-
age boundaries, as shown in Figure 4. These imprecise
background absorbing nodes may result that the object re-
gions close to t he boundary are suppressed. However, the
absorbed time considers the effect of all boundary nodes
and depends on two factors: the edge weights on the path
and the spatial distance, so the parts of object which are far
from or different from the boundary absorbing nodes can be
highlighted correctly. The main procedure of the proposed
method is summarized in Algorithm 1.
Algorithm 1 Saliency detection based on Markov random walk
Input: An image and required parameters.
1. Construct a graph πΊ with superpixels as nodes, and use bound-
ary nodes as absorbing nodes;
2. Compute the afο¬nity matrix A by Eq. 6 and the transition ma-
trix P by Eq. 8;
3. Extract the matrix Q from P by Eq. 1, and compute the funda-
mental matrix N =(I β Q)
β1
and the map S by Eq. 9;
4. Compute the π ππππ by Eq. 13, if π ππππ < πΎ , output S and
return;
5. Compute the recurrent time h by Eq. 11 and 4, and the weight
u by Eq. 12, then compute the saliency map S by Eq. 10 and 9;
Output: the full resolution saliency map.
6. Experimental Results
We evaluate the proposed method on four benchmark
datasets. The ο¬rst one is the MSRA dataset [18] which con-
tains 5,000 images with the ground truth marked by bound-
ing boxes. The second one is the ASD dataset, a subset of
the MSRA dataset, which contains 1,000 images with accu-
rate human-labelled ground truth provided by [2]. The third
one is the SED dataset [28], which contains: the single ob-
ject sub-dataset SED1 and two objects sub-dataset SED2.
Each sub-dataset contains 100 images and have accurate
human-labelled ground truth. The fourth one is the most
challenging SOD dataset which contains 300 images from
the Berkeley segmentation dataset [22]. This dataset is ο¬rst
used for salient object segmentation evaluation [23], where
seven subjects are asked to label the foreground salient ob-
ject masks. For each object mask of each subject, a consis-
tency score is computed based on the labels of the other six
subjects. We select and combine the object masks whose
consistency scores are higher than 0.7 as the ο¬nal ground
truth as done in [33]. We compare our method with ο¬fteen
state-of-the-art saliency detection algorithms: the IT [16],
MZ [20], LC [37], GB [14], SR [15], AC [1], FT [2], S-
ER [31], CA [27], RC [8], CB [17], SVO [7], SF [25], L-
R [29] and GS [33] methods.
Experimental Setup: We set the number of superpixel n-
odes π = 250 in all the experiments. There are two param-
eters in the proposed algorithm: the edge weight π in Eq. 5
to controls the strength of weight between a pair of nodes
and the threshold πΎ of π ππππ in Eq. 13 to indicate the quality
of the saliency map. These two parameters are empirically
chosen, π
2
=0.1 and πΎ =2for all the test images in the
experiments.
Evaluation Metrics: We evaluate all methods by precision,
recall and F-measure. The precision is deο¬ned as the ratio
of salient pixels correctly assigned to all the pixels of ex-
tracted regions. The recall is deο¬ned as the ratio of detected
salient pixels to the ground-truth number. The F-measure
is the overall performance measurement computed by the
weighted harmonic of precision and recall:
πΉ
π½
=
(1 + π½
2
)π πππππ πππ Γ π
πππππ
π½
2
π πππππ πππ + π
πππππ
. (14)
We set π½
2
=0.3 to stress precision more than recall, the
same to [2, 8, 25]. Similar as previous works, two eval-
uation criteria are used in our experiments. First, we bi-
segment the saliency map using every threshold in the range
[0 : 0.05 : 1], and compute precision and recall at each val-
ue of the threshold to plot the precision-recall curve. Sec-
ond, we compute the precision, recall and F-measure with
an adaptive threshold proposed in [2], which is deο¬ned as
twice the mean saliency of the image.
5
