![Fig. 4. Overview of the ISPRS Potsdam data set. The aerial images shown are those provided by the ISPRS benchmark [54].](/figures/fig-4-overview-of-the-isprs-potsdam-data-set-the-aerial-2vq8u7lm.webp)
![Fig. 2. Conceptual illustration of the data flow through our variant of an FCN, which is used for the semantic segmentation of aerial images. Three skip connections are highlighted by pale red, pale green, and pale blue, respectively. Note that we added a third (pale red) skip connection in addition to the original ones (pale green and pale blue) of [5].](/figures/fig-2-conceptual-illustration-of-the-data-flow-through-our-jfmq7d3n.webp)
![Fig. 3. Our FCN architecture, which adds one more skip connection (after Pool_2, shown red) to the original model of [5]. Neurons form a 3-D structure per layer: dimensions are written in brackets, where the first number indicates the amount of feature channels, and second and third represent spatial dimensions.](/figures/fig-3-our-fcn-architecture-which-adds-one-more-skip-2ra8m4n8.webp)
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING 1
Learning Aerial Image Segmentation
From Online Maps
Pascal Kaiser, Jan Dirk Wegner, Aurélien Lucchi, Martin Jaggi, Thomas Hofmann,
and Konrad Schindler, Senior Member, IEEE
Abstract—This paper deals with semantic segmentation of
high-resolution (aerial) images where a semantic class label is
assigned to each pixel via supervised classification as a basis for
automatic map generation. Recently, deep convolutional neural
networks (CNNs) have shown impressive performance and have
quickly become the de-facto standard for semantic segmentation,
with the added benefit that task-specific feature design is no
longer necessary. However, a major downside of deep learning
methods is that they are extremely data hungry, thus aggravating
the perennial bottleneck of supervised classification, to obtain
enough annotated training data. On the other hand, it has
been observed that they are rather robust against noise in the
training labels. This opens up the intriguing possibility to avoid
annotating huge amounts of training data, and instead train the
classifier from existing legacy data or crowd-sourced maps that
can exhibit high levels of noise. The question addressed in this
paper is: can training with large-scale publicly available labels
replace a substantial part of the manual labeling effort and
still achieve sufficient performance? Such data will inevitably
contain a significant portion of errors, but in return virtually
unlimited quantities of it are available in larger parts of the
world. We adapt a state-of-the-art CNN architecture for semantic
segmentation of buildings and roads in aerial images, and
compare its performance when using different training data sets,
ranging from manually labeled pixel-accurate ground truth of the
same city to automatic training data derived from OpenStreetMap
data from distant locations. We report our results that indicate
that satisfying performance can be obtained with significantly less
manual annotation effort, by exploiting noisy large-scale training
data.
Index Terms— Crowdsourcing, image classification, machine
learning, neural networks, supervised learning, terrain mapping,
urban areas.
I. INTRODUCTION
H
UGE volumes of optical overhead imagery are captured
every day with airborne or spaceborne platforms, and
that volume is still growing. This “data deluge” makes manual
interpretation prohibitive, and hence machine vision must be
employed if we want to make any use of the available data.
Perhaps the fundamental step of automatic mapping is to
assign a semantic class to each pixel, i.e., convert the raw
data to a semantically meaningful raster map (which can then
be further processed as appropriate with, e.g., vectorization or
map generalization techniques). The most popular tool for that
Manuscript received January 30, 2017; revised April 11, 2017
and May 29, 2017; accepted June 19, 2017. (Corresponding author:
Jan Dirk Wegner.)
The authors are with ETH Zürich, 8093 Zürich, Switzerland (e-mail:
jan.wegner@geod.baug.ethz.ch).
Color versions of one or more of the figures in this paper are available
online at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TGRS.2017.2719738
task is supervised machine learning. Supervision with human-
annotated training data is necessary to inject the task-specific
class definitions into the generic statistical analysis. In most
cases, reference data for classifier training are generated man-
ually for each new project, which is a time-consuming and
costly process. Manual annotation must be repeated every time
the task, the geographic location, the sensor characteristics, or
the imaging conditions change, and hence the process scales
poorly. In this paper, we explore the tradeoff between the
following:
1) pixel-accurate but small-scale ground truth available;
2) less accurate reference data that are readily available in
arbitrary quantities, at no cost.
For our study, we make use of online map data from Open-
StreetMap [1]–[3] (OSM, http://www.openstreetmap.org) to
automatically derive weakly labeled training data for three
classes, buildings, roads,andbackground (i.e., all others).
These data are typically collected using two main sources.
1) Volunteers collect OSM data either in situ with GPS
trackers or by manually digitizing very high resolution
(VHR) aerial or satellite images that have been donated.
2) National mapping agencies donate their data to OSM to
make it available to a wider public.
Since OSM is generated by volunteers, our approach can
be seen as a form of crowd-sourced data annotation; but other
existing map databases, e.g., legacy data within a mapping
agency, could also be used.
As image data for our study, we employ high-resolution
RGB orthophotographs from Google Maps,
1
since we could
not easily get access to comparable amounts of other high-
resolution imagery [> 100 km
2
at ≈ 10-cm ground sampling
distance (GSD)].
Clearly, these types of training data will be less accurate.
Sources of errors include coregistration errors, e.g., in our
case, OSM polygons and Google images were independently
geo-referenced; limitations of the data format, e.g., OSM only
has road centerlines and category, but no road boundaries;
temporal changes not depicted in outdated map or image data;
or simply sloppy annotations, not only because of a lack of
training or motivation, but also because the use cases of most
OSM users require not even meter-level accuracy.
Our study is driven by the following hypotheses.
1) The sheer volume of training data can possibly compen-
sate for the lower accuracy (if used with an appropriate
robust learning method).
1
specifications of Google Maps data can be found at
https://support.google.com/mapcontentpartners/answer/144284?hl=en
0196-2892 © 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
2 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING
2) The large variety present in very large training sets
(e.g., spanning multiple different cities) could potentially
improve the classifier’s ability to generalize to new
unseen locations.
3) Even if high-quality training data are available, the
large volume of additional training data could potentially
improve the classification.
4) If low-accuracy large-scale training data help, then it
may also allow one to substitute a large portion of the
manually annotated high-quality data.
We investigate these hypotheses when using deep convolu-
tional neural networks (CNNs). Deep networks are at present
the top-performing method for high-resolution semantic label-
ing and are therefore the most appropriate choice for our
study.
2
At the same time, they also fulfill the other require-
ments for our study: they are data hungry and robust to label
noise [4]. And they make manual feature design somewhat
obsolete: once training data are available, retraining for differ-
ent sensor types or imaging conditions is fully automatic, with-
out scene-specific user interaction such as feature definition or
preprocessing. We adopt a variant of the fully convolutional
network (FCN) [5], and explore the potential of combining
end-to-end trained deep networks with massive amounts of
noisy OSM labels. We evaluate the extreme variant of our
approach, without any manual labeling, on three major cities
(Chicago, Paris, and Zurich) with different urban structures.
Since quantitative evaluations on these large data sets are
limited by the inaccuracy of the labels, which is also present
in the test sets, we also perform experiments for a smaller
data set from the city of Potsdam. There, high-precision
manually annotated ground truth is available, which allows us
to compare different levels of project-specific input, including
the baseline where only manually labeled training data are
used, the extreme case of only automatically generated training
labels, and variants in between. We also assess the mod-
els’ capabilities regarding generalization and transfer learning
between unseen geographic locations.
We find in this paper that training on noisy labels does
work well, but only with substantially larger training sets.
Whereas with small training sets (≈ 2km
2
), it does not reach
the performance of hand-labeled pixel-accurate training data.
Moreover, even in the presence of high-quality training data,
massive OSM labels further improve the classifier, and hence
can be used to significantly reduce the manual labeling efforts.
According to our experiments, the differences are really due to
the training labels, since segmentation performance of OSM
labels is stable across different image sets of the same scene.
For practical reasons, our study is limited to buildings
and roads, which are available from OSM, and to RGB
images from Google Maps, subject to unknown radiometric
manipulations. We hope that similar studies will also be
performed with the vast archives of proprietary image and
map data held by state mapping authorities and commercial
2
All top-performing methods on big benchmarks are CNN
variants, both in generic computer vision, e.g., the Pascal
VOC Challenge, http://host.robots.ox.ac.uk/pascal/VOC/, and in
remote sensing, e.g., the ISPRS semantic labeling challenge,
http://www2.isprs.org/commissions/comm3/wg4/semantic-labeling.html
satellite providers. Finally, this is a step in a journey that will
ultimately bring us closer to the utopian vision that a whole
range of mapping tasks no longer need user input, but can be
completely automated by the world wide Web.
II. R
ELATED WORK
There is a huge literature about semantic segmentation in
remote sensing. A large part deals with rather low-resolution
satellite images, whereas our work in this paper deals with
VHR aerial images (see [6] for an overview).
Aerial data with a ground sampling distance GSD ≤ 20 cm
contains rich details about urban objects such as roads, build-
ings, trees, and cars, and is a standard source for urban
mapping projects. Since urban environments are designed by
humans according to relatively stable design constraints, early
work attempted to construct object descriptors via sets of rules,
most prominently for building detection in 2-D [7], [8] or in
3-D [9]–[11], and for road extraction [12]–[14]. A general
limitation of hierarchical rule systems, be they top-down or
bottom-up, is poor generalization across different city layouts.
Hard thresholds at early stages tend to delete information
that can hardly be recovered later, and hard-coded expert
knowledge often misses important evidence that is less obvious
to the human observer.
Machine learning thus aims to learn classification rules
directly from the data. As local evidence, conventional classi-
fiers are fed with raw pixel intensities, simple arithmetic com-
binations such as vegetation indices, and different statistics or
filter responses that describe the local image texture [15]–[17].
An alternative is to precompute a large redundant set of
local features for training and let a discriminative classi-
fier (e.g., boosting and random forest) select the optimal
subset [18]–[21] for the task.
More global object knowledge that cannot be learned from
local pixel features can be introduced via probabilistic priors.
Two related probabilistic frameworks have been successfully
applied to this task, marked point processes (MPPs) and
graphical models. For example, [22] and [23] formulate MPPs
that explicitly model road network topologies, while [24] use
a similar approach to extract building footprints. MPPs rely
on object primitives like lines or rectangles that are matched
to the image data by sampling. Even if data driven [25], such
Monte Carlo sampling has high computational cost and does
not always find good configurations. Graphical models provide
similar modeling flexibility, but in general also lead to hard
optimization problems. For restricted cases (e.g., submodular
objective functions), efficient optimizers exist. Although there
is a large body of literature that aims to tailor conditional ran-
dom fields for object extraction in computer vision and remote
sensing, relatively few authors tackle semantic segmentation
in urban scenes (see [26]–[30]).
Given the difficulty of modeling high-level correlations,
much effort has gone into improving the local evidence by
finding more discriminative object features [21], [31], [32].
The resulting feature vectors are fed to a standard classifier
(e.g., decision trees or support vector machines) to infer
probabilities per object category. Some authors invest a lot
of efforts to reduce the dimension of the feature space to
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
KAISER et al.: LEARNING AERIAL IMAGE SEGMENTATION FROM ONLINE MAPS 3
a maximally discriminative subset (see [33]–[36]), although
this seems to have only limited effect—at least with modern
discriminative classifiers.
Deep neural networks do not require a separate feature
definition step, but instead learn the most discriminative
feature set for a given data set and task directly from raw
images. They go back to [37] and [38], but at the time were
limited by a lack of computing power and training data. After
their comeback in the 2012 ImageNet challenge [39], [40],
deep learning approaches, and in particular deep CNNs, have
achieved impressive results for diverse image analysis tasks.
State-of-the-art network architectures (see [41]) have many
(often 10–20, but up to >100) layers of local filters and
thus large receptive fields in the deep layers, which makes
it possible to learn complex local-to-global (nonlinear) object
representations and long-range contextual relations directly
from raw image data. An important property of deep CNNs
is that both training and inference are easily parallelizable,
especially on GPUs, and thus scale to millions of training and
testing images.
Quickly, CNNs were also applied to semantic segmenta-
tion of images [42]. Our approach in this paper is based
on the FCN architecture of [5], which returns a structured
spatially explicit label image (rather than a global image label).
While spatial aggregation is nevertheless required to represent
context, FCNs also include in-network upsampling back to
the resolution of the original image. They have already been
successfully applied to semantic segmentation of aerial images
(see [43]–[45]). In fact, the top performers on the ISPRS
semantic segmentation benchmark all use CNNs. We note that
(nonconvolutional) deep networks in conjunction with OSM
labels have also been applied for patch-based road extraction
in overhead images of ≈ 1 m GSD at large scale [46], [47].
More recently, Máttyus et al. [48] combine OSM data with
aerial images to augment maps with additional information
from imagery like road widths. They design a sophisticated
random field to probabilistically combine various sources of
road evidence, for instance, cars, to estimate road widths at
global scale using OSM and aerial images.
To the best of our knowledge, only two works have made
attempts to investigate how results of CNNs trained on large-
scale OSM labels can be fine-tuned to achieve more accurate
results for labeling remote sensing images [49], [50]. However,
we are not aware of any large-scale, systematic, comparative,
and quantitative study that investigates using large-scale train-
ing labels from inaccurate map data for semantic segmentation
of aerial images.
III. M
ETHODS
We first describe our straightforward approach to generate
training data automatically from OSM, and then give technical
details about the employed FCN architecture and the training
procedure used to train our model.
A. Generation of Training Data
We use a simple automatic approach to generate data sets of
VHR aerial images in RGB format and corresponding labels
for classes building, road,andbackground.Aerialimagesare
downloaded from Google Maps, and geographic coordinates
of buildings and roads are downloaded from OSM. We prefer
to use OSM maps instead of Google Maps, because the
latter can only be downloaded as raster images.
3
OSM data
can be accessed and manipulated in vector format, and each
object type comes with meta data and identifiers that allow
straightforward filtering. Regarding coregistration, we find that
OSM and Google Maps align relatively well, even though
they have been acquired and processed separately.
4
Most
local misalignments are caused by facades of high buildings
that overlap with roads or background due to perspective
effects. It is apparent that in our test areas Google provides
orthophotographs rectified with respect to a bare earth digital
terrain model (DTM), not “true” orthophotographs rectified
with a digital surface model (DSM). According to our own
measurements on a subset of the data, this effect is relatively
mild, generally < 10 pixels displacement. We found that this
does not introduce major errors as long as there are no high-
rise buildings. It may be more problematic for extreme scenes
such as Singapore or Manhattan.
To generate pixel-wise label maps, the geographic coor-
dinates of OSM building corners and road center lines are
transformed to pixel coordinates. For each building, a polygon
through the corner points is plotted at the corresponding image
location. For roads, the situation is slightly more complex.
OSM provides only coordinates of road center lines, but
no precise road widths. There is, however, a road category
label (“highway tag”) for most roads. We determined an
average road width for each category on a small subset of
the data, and validated it on a larger subset (manually, one-
off). This simple strategy works reasonably well, with a
mean error of ≈11 pixels for the road boundary, compared
with ≈100 pixels of road width.
5
In (very rare) cases where
the ad hoc procedure produced label collisions, pixels claimed
by both building and road were assigned to buildings. Pixels
neither labeled building nor road form the background class.
Examples of images overlaid with automatically generated
OSM labels are shown in Fig. 1.
B. Neural Network Architecture
We use a variant of FCNs in this paper (see Fig. 2). Fol-
lowing the standard neural network concept, transformations
are ordered in sequential layers that gradually transform the
pixel values to label probabilities. Most layers implement
learned convolution filters, where each neuron at level l takes
its input values only from a fixed-size spatially localized
window W in the previous layer (l−1), and outputs a vector of
differently weighted sums of those values, c
l
=
i∈W
w
i
c
l−1
i
.
Weights w
i
are shared across all neurons of a layer, which
reflects the shift invariance of the image signal and drastically
3
Note that some national mapping agencies also provide publicly
available map and other geo-data, e.g., the USGS national map pro-
gram: https://nationalmap.gov/
4
Note that it is technically possible to obtain world coordinates of objects
in Google Maps and enter those into OSM, and this might in practice also be
done to some extent. However, OSM explicitly asks users not to do that.
5
Average deviation based on ten random samples of Potsdam, Chicago,
Paris, and Zurich.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
4 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING
Fig. 1. Example of OSM labels overlaid with Google Maps images for (a) Zurich and (b) Paris. (Left) Aerial image and a magnified detail. (Right) Same
images overlaid with building (red) and road (blue) labels. Background is transparent in the label map.
Fig. 2. Conceptual illustration of the data flow through our variant of an FCN, which is used for the semantic segmentation of aerial images. Three skip
connections are highlighted by pale red, pale green, and pale blue, respectively. Note that we added a third (pale red) skip connection in addition to the
original ones (pale green and pale blue) of [5].
reduces the number of parameters. Each convolutional layer is
followed by a rectified linear unit (ReLU) c
l
rec
= max(0, c
l
),
which simply truncates all negative values to 0 and leaves
positive values unchanged [51].
6
Convolutional layers are
interspersed with max-pooling layers that downsample the
image and retain only the maximum value inside a (2 × 2)
neighborhood. The downsampling increases the receptive field
of subsequent convolutions, and lets the network learn corre-
lations over a larger spatial context. Moreover, max-pooling
achieves local translation invariance at object level. The out-
puts of the last convolutional layers (which are very big to
capture global context, equivalent to a fully connected layer
of standard CNNs) is converted to a vector of scores for the
6
Other nonlinearities are sometimes used, but ReLU has been shown to
facilitate training (backpropagation) and has become the de-facto standard.
three target classes. These score maps are of low resolution,
and hence they are gradually upsampled again with convo-
lutional layers using a stride of only (12) pixel.
7
Repeated
downsampling causes a loss of high-frequency content, which
leads to blurry boundaries that are undesirable for pixel-wise
semantic segmentation. To counter this effect, feature maps
at intermediate layers are merged back in during upsampling
(the so-called “skip connections,” see Fig. 2). The final full-
resolution score maps are then converted to label probabilities
with the sof tmax function.
7
This operation is done by layers that are usually called “deconvolution
layers” in [5] (and also in Fig. 3) although the use of this terminology has been
criticized since most implementations do not perform a real deconvolution but
rather a transposed convolution.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
KAISER et al.: LEARNING AERIAL IMAGE SEGMENTATION FROM ONLINE MAPS 5
Fig. 3. Our FCN architecture, which adds one more skip connection (after Pool_2, shown red) to the original model of [5]. Neurons form a 3-D structure per
layer: dimensions are written in brackets, where the first number indicates the amount of feature channels, and second and third represent spatial dimensions.
C. Implementation Details
The FCN we use is an adaptation of the architecture
proposed in [5], which itself is largely based on the VGG-16
network architecture [41]. In our implementation, we slightly
modify the original FCN and introduce a third skip connection
(marked red in Fig. 2), to preserve even finer image details.
We found that the original architecture, which has two skip
connections after Pool_3 and Pool_4 (see Fig. 3), was still
not delivering sufficiently sharp edges. The additional higher
resolution skip connection consistently improved the results
for our data (see Section IV-B). Note that adding the third skip
connection does not increase the total number of parameters
but, on the contrary, slightly reduces it ( [5]: 134
277
737, ours:
134
276
540; the small difference is due to the decomposition
of the final upsampling kernel into two smaller ones).
D. Training
All model parameters are learned by minimizing a multino-
mial logistic loss, summed over the entire 500 × 500 pixel
patch that serves as input to the FCN. Prior to train-
ing/inference, intensity distributions are centered indepen-
dently per patch by subtracting the mean, separately for each
channel (RGB).
All models are trained with stochastic gradient descent
with a momentum of 0.9, and minibatch size of one image.
Learning rates always start from 5 × 10
−9
and are reduced
by a factor of ten twice when the loss and average F
1
scores stopped improving. The learning rates for biases of
convolutional layers were doubled with respect to learning
rates of the filter weights. Weight decay was set to 5 ×10
−4
,
and dropout probability for neurons in layers ReLU_6 and
ReLU_7wasalways0.5.
Training was run until the average F
1
-score on the validation
data set stopped improving, which took between 45 000 and
140 000 iterations (3.5–6.5 epochs). Weights were initialized
as in [52], except for experiments with pretrained weights.
It is a common practice in deep learning to publish pre-
trained models together with source code and paper, to ease
repeatability of results and to help others avoid training from
scratch. Starting from pretrained models, even if these have
been trained on a completely different image data set, often
improves performance, because low-level features like contrast
edges and blobs learned in early network layers are very
similar across different kinds of images.
We will use two different forms of pretraining. Either
we rely on weights previously learned on the Pascal VOC
benchmark [53] (made available by Long et al. [5]), or we
pretrain ourselves with OSM data. In Section IV, it is always
specified whether we use VOC, OSM, or no pretraining at all.
IV. E
XPERIMENTS
We present extensive experiments on four large data sets of
different cities to explore the following scenarios.
1) Complete Substitution: Can semantic segmentation be
learned without any manual labeling? What performance