HAL Id: hal-01264072
https://hal.archives-ouvertes.fr/hal-01264072
Submitted on 16 Feb 2016
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entic research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diusion de documents
scientiques de niveau recherche, publiés ou non,
émanant des établissements d’enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
Simultaneous Registration and Change Detection in
Multitemporal, Very High Resolution Remote Sensing
Data
Maria Vakalopoulou, Konstantinos Karatzalos, Nikos Komodakis, Nikos
Paragios
To cite this version:
Maria Vakalopoulou, Konstantinos Karatzalos, Nikos Komodakis, Nikos Paragios. Simultaneous Reg-
istration and Change Detection in Multitemporal, Very High Resolution Remote Sensing Data. 2015
IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Jun 2015,
Boston, United States. pp.61-69, �10.1109/CVPRW.2015.7301384�. �hal-01264072�
Simultaneous Registration and Change Detection in Multitemporal, Very High
Resolution Remote Sensing Data
Maria Vakalopoulou
1,2
, Konstantinos Karatzalos
1
, Nikos Komodakis
2
, Nikos Paragios
3
1
Remote Sensing Laboratory, Natio-
2
Ecole des Ponts ParisTech,
3
Center for Visual Computing
nal Technical University of Athens University Paris-Est Ecole Centrale de Paris
mariavak@mail.ntua.gr nikos.komodakis@enpc.fr nikos.paragios@ecp.fr
karank@central.ntua.gr
Abstract
In order to exploit the currently continuous streams of
massive, multi-temporal, high-resolution remote sensing
datasets there is an emerging need to address efficiently the
image registration and change detection challenges. To this
end, in this paper we propose a modular, scalable, metric
free single shot change detection/registration method. The
approach exploits a decomposed interconnected graphical
model formulation where registration similarity constraints
are relaxed in the presence of change detection. The defor-
mation space is discretized, while efficient linear program-
ming and duality principles are used to optimize a joint so-
lution space where local consistency is imposed on the de-
formation and the detection space. Promising results on
large scale experiments demonstrate the extreme potentials
of our method.
1. Introduction
The current generation of space-borne and airborne sen-
sors are generating nearly continuous streams of massive
multi-temporal high resolution remote sensing data. How-
ever, in order to efficiently exploit these datasets their ac-
curate co-registration is the first indispensable processing
step. Despite the fact that image registration is among the
most studied problems in computer vision, most remote
sensing and GIS software still employ semi-automated reg-
istration procedures when it comes to very large, multispec-
tral, very high resolution satellite data [17, 7]. This is, how-
ever, far from a cost-effective solution especially if we con-
sider huge multi-temporal datasets that require accurate co-
registration [13, 12].
In addition, the primary goal of the analysis of multi-
temporal datasets is the detection of changes between differ-
ent land cover types [3, 11]. In particular, change detection
Unregistered multi-temporal satellite data
July 2006 July 2011
Registration & Change Detection Framework
E = E
reg
+ E
chng
Registration & Change Detection Framework
E
reg,ch
= V
reg,ch
+ V
pg,ch
+ V
pg,reg
Registered Data Detected Changes
Figure 1. The developed framework addresses simultaneous the
registration and change detection tasks.
of man-made objects is still an emerging challenge due to
the significant importance for various engineering and en-
vironmental applications [18, 5, 10, 25, 3]. Apart from Na-
tional and local government applications like the update of
cadastral and other GIS databases, companies like Google
and Microsoft are seeking to include extensively up-to-date
2D and 3D urban models in their products (e.g., Microsoft
Virtual Earth and Google Earth).
1
Change detection, however, from multi-temporal earth
observation data is not a trivial task and still remains a chal-
lenge. Among the various recently proposed methods, those
based on Markov Random Fields [8, 24, 1], kernels [2, 26]
and neural networks [22, 23] have gained important atten-
tion. Focusing on man-made object change detection [4, 20]
in urban and peri-urban regions, several approaches have
been proposed based on very high resolution optical and
radar data [19, 23, 6, 20]. However, these change detec-
tion techniques assume and require accurately co-registered
data in order to perform pixel-by-pixel or region-by-region
multi-temporal data fusion, correlation or any change anal-
ysis.
In this paper, we propose a simultaneous registration and
change detection approach motivated by the fact that on one
hand, the registration of very high resolution data seems to
be optimally addressed through deformation grids and pow-
erful discrete optimization [12], while on the other hand, the
desired changes are located in the regions for which corre-
spondences between the unregistered multi-temporal data
can not be established (Figure 1).
To this end, we have designed, developed and evaluated
a modular, scalable, metric-free single shot change detec-
tion/registration method. The approach exploits a decom-
posed interconnected graphical model formulation where
registration similarity constraints are relaxed in the pres-
ence of change detection. We employ a discretized, grid-
based deformation space. State-of-the-art linear program-
ming and duality principles have been employed to opti-
mize the joint solution space where local consistency is im-
posed on the deformation and the detection space. The un-
supervised framework has been designed to handle and pro-
cess large very high resolution multispectral remote sens-
ing data, while optimized for man-made object change de-
tection in urban and peri-urban regions. The developed
method has been validated through large scale experiments
on several multi-temporal very high resolution optical satel-
lite datasets.
The main contributions of the developed method are (i)
the novel, single and modular joint registration and change
detection framework, (ii) the metric-free formulation which
allows numerous and change-specific implementations, (iii)
the low computational complexity which allows near real-
time performance with gpu implementation. It should be
mentioned that the detected changes can not be directly em-
ployed for the update of, e.g., a geospatial database, since
the developed unsupervised framework does not include
any prior information about the geometry of the man-made
objects.
G
reg
G
ch
V
reg,ch
qp
Figure 2. Each graph contains a smoothness term which imposes
the necessary homogeneity within the graph. The interaction be-
tween the two graphs is performed by the similarity cost which
connect the registration with the change detection terms.
2. Methodology
2.1. MRF formulation
We have designed and built an MRF model over two dif-
ferent graphs of the same dimensions. The first deformable
graph corresponds to the registration term (G
reg
) and the
second one to the change detection term (G
ch
). Each graph
contains a smoothness term which impose the necessary ho-
mogeneity within the graph. The interaction between the
two graphs is performed by the similarity cost which con-
nect the registration with the change detection terms.
Each graph is superimposed on the image [9] and there-
fore every node of the graph acts and depends on a subset of
pixels in the vicinity (depending on the chosen interpolation
strategy). With such a manner every pixel can participate
through a certain weight to the graph related to its distance
from the nodes. The computational complexity is therefore
lower as graph’s dimensions are smaller than the unregis-
tered raw images. In particular, the dimensions of the graph
are related to the image dimensions forming a trade off be-
tween accuracy and computational complexity. In such a
setting the deformation of a pixel is defined through an in-
terpolation of the displacement of the proximal graph nodes
as follows:
T (x) = x +
X
p∈G
η(||x − p||)d
p
(1)
where d
p
is a displacement value, η(.) is the projection
function, p is a control point and x is a pixel in the image.
After the optimization, the optimal labels will be pro-
jected to the image pixels using the same projection func-
tion η(.). Once the similarity criterion has been defined, the
next step consists of imposing certain continuity on the de-
formation space which is discussed in the next subsection.
That way, we formulate an energy function E
reg,ch
=
E
reg
+ E
ch
and we couple the two different graphs to one.
The label for each node p belonging to the graph G, will
be l
p
= [l
c
, l
reg
] where l
c
are the labels for the change de-
tection, l
c
∈ {0, 1} and l
reg
are the labels for the registra-
tion l
reg
∈ ∆ where ∆ = [d
1
, . . . , d
n
] corresponding to all
possible displacements. Concluding the label space can be
summarized as L = {0, 1} × ∆.
2.2. The Registration Energy Term
Let us denote a pair of images where A is the source
image and V is the target image defined on a domain Ω. The
goal of image registration is to define a transformation map
T which will project the source image to the target image.
V (x) = A ◦ T (x) (2)
Let us consider a discrete set of labels L
reg
= [1, . . . , n],
and a set of discrete displacements ∆ = [d
1
, . . . , d
n
]. We
seek to assign a label l
reg
p
to each grid node p, where each
label corresponds to a discrete displacement d
l
reg
p
∈ ∆.
The energy formulation for the registration comprises of
a similarity cost (that seeks to satisfy the equation 2) and a
smoothness penalty on the deformation domain. The sim-
ilarity cost depends on the presence of changes and will
be subsequently defined. The smoothness term penalises
neighbouring nodes that have different displacement labels,
depending on the distance of the labelled displacements.
V
pq ,reg
(l
reg
p
, l
reg
q
) = ||d
l
reg
p
− d
l
reg
q
|| (3)
where p and q are neighbouring nodes.
2.3. The Change Detection Energy Term
The goal of the change detection term is to estimate the
changed and unchanged image regions. We employ two dif-
ferent labels in order to address the change detection prob-
lem l
c
p
∈ [0, 1]. The energy formulation for the change de-
tection corresponds to a smoothness term which penalizes
neighbouring nodes with different change labels.
V
pq ,ch
(l
c
p
, l
c
q
) = ||l
c
p
− l
c
q
|| (4)
2.4. Coupling the Energy Terms
The coupling between change detection and registration
is achieved through the interconnection between the two
graphs. Assuming a pair of corresponding nodes of the two
graphs, one would expect that in the absence of change the
similarity cost should be satisfied and in that case the poten-
tial will be:
V
reg,ch
(l
reg
p
, l
c
p
) = (1 − l
c
p
) ·
Z
Ω
ˆη(||x − p||)
ρ(V (x), A(x + d
l
reg
p
))dx + l
c
p
· C
(5)
where we simply take all pixels in the vicinity of the graph
node and project them back to the grid node with a weight
that is proportional to the distance. In the presence of
change, we use a fixed cost C. These two terms are in-
tegrated as in (equation 5) which simply uses a fixed cost
in the presence of changes and the image matching cost in
their absence.
With a slight abuse of notation we consider a node with
an index p ∈ G (we recall that the two graphs are identical)
corresponding to the same node throughout the two graphs
(G
reg
, G
ch
).We can now integrate all terms to a single en-
ergy which detect changes, establishes correspondences and
impose smoothness in the change detection and the defor-
mation map as follows:
E
reg,ch
(l
c
, l
reg
) = w
1
·
X
p∈G
V
reg,ch
(l
reg
p
, l
c
p
)+
w
2
·
X
p∈G
reg
X
q ∈N (p)
V
pq ,reg
(l
reg
p
, l
reg
q
)+
w
3
·
X
p∈G
ch
X
q ∈N (p)
V
pq ,ch
(l
c
p
, l
c
q
)
(6)
where V
reg,ch
(l
reg
p
, l
c
p
) represents the coupling term for
each node at each label, V
pq ,reg
(l
reg
p
, l
reg
q
) the pairwise or
binary term for the registration and V
pq ,ch
(l
c
p
, l
c
q
) the pair-
wise for the change detection.
In particular, a similarity function ρ(.) is used in order
to compare the two images, while a constant value C is
used in order to define the changes. In such a setting, opti-
mizing an objective function seeking similarity correspon-
dences is not meaningful and deformation vectors should be
the outcome of the smoothness constraint on the displace-
ment space. However, the areas of change are unknown and
is one of the objective of the optimization process. Without
loss of generality we can assume that the matching cost cor-
responding to change can correspond to a value that can be
determined from the distribution of these costs on the en-
tire domain (it is metric dependent). Let us consider that
this value is known and that it is independent from the im-
age displacements, so we can distinguish the regions that
have been changed. The advantage of the methodology is
that by solving the two problems simultaneously first we
have less false changes caused by the misregistration of the
images and then the changed regions do not affect the en-
tire displacement map, as we do not calculate the displace-
ment there and their final displacement is caused by the un-
changed neighbour regions.
Finally, the pairwise costs for both terms have been de-
scribed in equation 3 and 4.
2.5. Optimization
There are several techniques for the minimization of an
MRF model which can be generally summarised into those
based on the message passing and those on graph cut meth-
ods. The first category is related to the linear programming
relaxation [14]. The optimization of the implementation is
performed by FastPD which is based on the dual theorem of
linear programming [15, 16].
3. Implementation
The minimization of the MRF energy is performed by a
multi-scale framework. Concerning the image, iteratively
different levels of Gaussian image pyramids are used. Con-
cerning the grid, in a similar way we consider different lev-
els of it, beginning with a sparser grid. Having very large re-
mote sensing imagery the multi-scale approach diminishes
the computational complexity without losing in terms of ac-
curacy. The different levels of the images and the grid with
the consistency of nodes in the grid are defined by the user.
In all our experiments, 2 image and 3 grid levels were found
adequate for the very high resolution satellite data.
Regarding the label space, a search for possible displace-
ments along 8 directions (x, y and diagonal axes) is per-
formed, while the change labels are always two and corre-
spond to change or no change description. The number of
the registration labels is the same at each level. Depending
on the parameter label factor the values of registration la-
bels change towards the optimal ones. The source image is
deformed according to the optimal labels and it is updated
for the next level. The value 0.8 was employed for updat-
ing the registration labels. Last but not least, the maximum
displacement was set smaller than 0.4 times the distance of
two consecutive nodes in order to preserve the right dis-
placement of every node.
In addition, several methods block matching methods
can be employed. Semantic changes in multitemporal im-
agery affect the local intensities and also change the struc-
ture of the region. One of the problems in traditional change
detection techniques, is that change in intensities does not
directly mean semantic change. This was crucial since the
focus, here, was on urban and peri-urban regions and man-
made object changes. The optimal displacement using the
SADG function are calculated using the weighted sum of
the difference between the pair and the gradient inner prod-
uct. On the other hand any other similarity measure as mu-
tual information, normalized cross correlation, ratio corre-
lation can be used. In Section 4 we have tested different
similarity functions. For the SADG metric and focused on
man-made object changes, the fixed cost C was set to 100.
In particular, higher C values result to less changes. The
parameter is not so sensitive since values between 90 to 120
lead to comparable results.
Last but not least, the number of iterations per level was
set to 10, the regularization parameter for the registration
task to 35 and for the change detection one to 3.5. The
function used for the projection from pixels to nodes and
reverse was the Cubic B-splines.
4. Experimental Results and Evaluation
4.1. Dataset
The developed framework was applied to several pairs
of multispectal VHR images from different satellite sen-
sors (i.e., Quickbird and WorldView-2). The multi-temporal
dataset covers approximately a 9 km
2
region in the East
Prefecture of Attica in Greece. All datasets were acquired
between the years of 2006 and 2011. The dataset is quite
challenging both due to its size and the pictured complex-
ity derived from the different acquisition angles. For the
quantitative evaluation the ground truth was manually col-
lected and annotated after an attentive and laborious photo-
interpretation done by an expert.
4.2. Experimental Results
Extensive experiments were performed over several im-
ages pairs and based on several similarity metrics namely
the Sum of Absolute Differences (SAD), the Sum of
Square Differences (SSD), the Normalized Cross Correla-
tion (NCC), the Normalized Mutual Information (NMI), the
Correlation Ratio (CR), the Sum of Gradient Inner Products
(GRAD), the Normalized Correlation Coefficient plus Sum
of Gradient Inner Products (CCGIP), the Hellinger Distance
(HD), the Jensen-Renyi Divergence (JRD), the Mutual In-
formation (MI) and the Sum Absolute of Differences plus
Gradient Inner Products (SADG). The experimental results
were evaluated both qualitative and quantitative for the reg-
istration and the change detection tasks.
Moreover, in order to evaluate quantitatively the devel-
oped algorithm, the standard quality metrics of Complete-
ness, Correctness and Quality were calculated on the de-
tected object level. The True Positives (TP), False Nega-
tives (FN) and False Positives (FP) were calculated in all
cases.
Completeness =
T P
T P + F N
(7)
Correctness =
T P
T P + F P
(8)
Quality =
T P
T P + F P + F N
(9)
where TP is the number of correctly detected changes, FN
is the number of changes that haven’t been detected by the
algorithm and FP is the number of false alarms.
Regarding the evaluation of the registration, a number
of ground control points (GCPs) were manually collected
in both unregistered and registered data. In particular, the
GCPs contained several points on building roof tops which
usually have the largest displacements. The displacement
