![Figure 1. Sample video frames from each of the 6 categories in the new dataset available at www.changedetection.net and typical detection results obtained using basic background subtraction [4] reported in the last row of Table 1.](/figures/figure-1-sample-video-frames-from-each-of-the-6-categories-22sllk9t.webp)
changedetection.net: A New Change Detection Benchmark Dataset
Nil Goyette
1
Pierre-Marc Jodoin
1
Fatih Porikli
2
Janusz Konrad
3
Prakash Ishwar
3
1
Universit
´
e de Sherbrooke
2
Mitsubishi Electric Research Laboratories
3
Boston University
MOIVRE Imaging Group ECE Department
Sherbrooke, J1K 2R1, Canada Cambridge MA, 02139, USA Boston MA, 02215, USA
Abstract
Change detection is one of the most commonly encoun-
tered low-level tasks in computer vision and video process-
ing. A plethora of algorithms have been developed to date,
yet no widely accepted, realistic, large-scale video dataset
exists for benchmarking different methods. Presented here
is a unique change detection benchmark dataset consisting
of nearly 90,000 frames in 31 video sequences representing
6 categories selected to cover a wide range of challenges
in 2 modalities (color and thermal IR). A distinguishing
characteristic of this dataset is that each frame is meticu-
lously annotated for ground-truth foreground, background,
and shadow area boundaries – an effort that goes much be-
yond a simple binary label denoting the presence of change.
This enables objective and precise quantitative comparison
and ranking of change detection algorithms. This paper
presents and discusses various aspects of the new dataset,
quantitative performance metrics used, and comparative re-
sults for over a dozen previous and new change detection
algorithms. The dataset, evaluation tools, and algorithm
rankings are available to the public on a website
1
and will
be updated with feedback from academia and industry in
the future.
1. Introduction
Detection of change, and in particular motion, is a fun-
damental low-level task in many computer vision and video
processing applications. Examples include visual surveil-
lance (people counting, crowd monitoring, action recogni-
tion, anomaly detection, forensic retrieval, etc.), smart en-
vironments (occupancy analysis, parking lot management,
etc.), and content retrieval (video annotation, event detec-
tion, object tracking). Change detection is closely coupled
with higher level inference tasks such as detection, local-
ization, tracking, and classification of moving objects, and
is often considered to be critical preprocessing step. Its im-
portance can be gauged by the large number of algorithms
that have been developed to-date and the even larger num-
ber of articles that have been published on this topic. A
1
www.changedetection.net
quick search for ‘motion detection’ on IEEE Xplore
c
re-
turns over 4,400 papers.
Among the many variants of change detection algo-
rithms, there seems to be no single algorithm that compe-
tently addresses all of the inherent real-life challenges in-
cluding sudden illumination variations, background move-
ments, shadows, camouflage effects (photometric similarit y
of object and background) and ghosting artifacts (delayed
detection of a moving object after it has moved away). Fur-
thermore, due to the tremendous effort required to generate
a benchmark dataset that contains pixel precision ground-
truth labels and provides a balanced coverage of the range
of challenges representative of the real world, no compre-
hensive large-scale evaluation of change detection has been
conducted to date.
The lack of a comprehensive dataset has a number of
negative implications. Firstly, it makes it difficult to as-
certain with confidence which algorithms would perform
robustly when the assumptions they are built upon are vi-
olated. Moreover, many algorithms tend to overfit specific
scenarios. For example, a method may be tuned to be robust
to shadows but may not be as robust to background motion.
A dataset that includes many different scenarios and uses a
variety of performance measures would go a long way to-
wards providing an objective assessment. Secondly, not all
authors are willing to (or have the resources to) compare
their methods against the most advanced and promising ap-
proaches. As a consequence, an overwhelming importance
has been accorded to a small subset of easily implementable
methods such as [23, 9, 26] that were developed in the late
1990’s. The more recent and advanced methods have been
marginalized as a result. Besides, the implementation of the
same method varies significantly from one research group
to another in the choice of parameters and the use of other
pre- and post-processing steps. Thirdly, the fact that authors
often use their own data (that are not widely available to ev-
eryone) makes a fair comparison much more problematic if
not impossible.
Recognizing the importance of change detection to
the computer vision and video processing communities,
we have prepared a unique change detection benchmark
dataset: changedetection.net (CDnet) that consists of nearly
978-1-4673-1612-5/12/$31.00 ©2012 IEEE 1
90,000 frames in 31 video sequences representing 6 video
categories (including thermal). This new dataset is the foun-
dation of the 2012 IEEE Change Detection Workshop [1].
CDnet contains diverse motion and change detection chal-
lenges in addition to typical indoor and outdoor scenes that
are encountered in most surveillance, smart environments,
and video analytics applications. A distinguishing feature
of CDnet is the fact that each image is meticulously anno-
tated for ground-truth foreground, background, and shadow
region boundaries; an effort that goes much beyond a sim-
ple binary label denoting the presence of the change. The
existence of ground-truth masks permits a precise compar-
ison and ranking of change detection algorithms. CDnet
also supplies a selection of evaluation tools in MATLAB and
Python for quantitatively assessing the performance of dif-
ferent methods according to 7 distinct metrics.
The overarching objectives of CDnet and its associated
workshop can be listed as:
1. To provide the research community with a rigorous and
comprehensive scientific benchmarking facility, a rich
dataset of videos, a set of utilities, and an access to
author-approved algorithm implementations for testing
and ranking of existing and new algorithms for motion
and change detection. The already extensive dataset
will be regularly revised and expanded with feedback
from the academia and industry.
2. To establish, maintain, and update a rank list of the
most accurate motion and change detection algorithms
in the various categories for years to come.
3. To help identify the remaining challenges in order to
provide focus for future research.
Next, we provide an overview of the existing datasets
and then present the details of CDnet including its cate-
gories, ground-truth annotations, performance metrics, and
a summary of the comparative rankings of the methods that
we tested at the IEEE Change Detection Workshop held in
conjunction with CVPR 2012.
2. Overview of Prior Efforts
Several datasets and survey papers have been presented
for the evaluation of change detection algorithms in the past.
2.1. Previous Datasets
Without aiming to be exhaustive, we list below a few key
datasets and describe their characteristics:
• Wallflower [25]: This is a fairly well-known dataset
that continues to be used today. It contains 7 videos,
each representing a specific challenge such as illu-
mination change, background motion, etc. Only one
frame per video has been labeled.
• PETS [27]: The Performance Evaluation of Tracking
and Surveillance (PETS) program was launched with
the goal of evaluating visual tracking and surveillance
algorithms. The program has been collecting videos
for the scientific community since the year 2000 and
now contains several dozen videos. Many of these
videos have been manually labeled by bounding boxes
with the goal of evaluating the performance of tracking
algorithms.
• CAVIAR
2
: This dataset contains more than 80 staged
indoor videos representing all kinds of human behav-
ior such as walking, browsing, shopping, fighting, etc.
Like the PETS dataset, a bounding box is associated
with each moving character.
• i-LIDS
3
: This dataset contains 4 scenarios (parked ve-
hicle, abandoned object, people walking in a restricted
area, doorway). Due to the size of the videos (more
than 24 hours of footage) the videos are not fully la-
beled.
• ETISEO
4
: This dataset contains more than 80 video
clips of various indoor and outdoor scenes. Since the
ground truth consists mainly of high-level information
such as the bounding box, object class, event type, etc.,
this dataset is more suitable for tracking, classification
and event recognition than change detection.
• VSSN 2006
5
: This dataset contains 9 semi-synthetic
videos composed of a real background and artificially-
moving objects. The videos contain animated back-
ground, illumination changes and shadows, however
include no frames void of activity.
• IBM
6
: This dataset contains 15 indoor and outdoor
videos taken from PETS 2001 plus additional videos.
For each video, 1 frame out of 30 is labeled with a
bounding box around each foreground moving object.
Further details about these datasets, and many others,
can be found on a web page of the European CANTATA
project
7
. With the exception of the Wallflower and VSSN
2006 datasets, all others have ground-truth information rep-
resented in terms of the bounding box for each foreground
object. Furthermore, the focus in the above datasets is
more on tracking as well as human behavior and interac-
tion recognition than change detection. As such, the above
datasets do not contain the diversity of video categories
present in the new dataset.
2
http://homepages.inf.ed.ac.uk/rbf/CAVIARDATA1
3
http://www.homeoffice.gov.uk/science-research/hosdb/i-lids
4
http://www-sop.inria.fr/orion/ETISEO
5
http://mmc36.informatik.uni-augsburg.de/VSSN06 OSAC
6
http://www.research.ibm.com/peoplevision/performanceevaluation.html
7
http://www.hitech-projects.com/euprojects/cantata/datasets cantata/
2
2.2. Survey Papers
Below, we list key survey papers that are devoted to the
comparison and ranking of change and motion detection al-
gorithms. Each paper uses its own dataset.
• Benezeth et al., 2010 [4] use a collection of 29 videos
(15 camera-captured, 10 semi-synthetic, and 4 syn-
thetic) taken from PETS 2001, the IBM dataset, and
the VSSN 2006 dataset. The authors also use semi-
synthetic videos composed of synthetic foreground ob-
jects (people and cars) moving over a camera-captured
background.
• Bouwmans et al., 2008 [5] survey only GMM methods
and use the Wallflower dataset.
• Nascimento and Marques, 2006 [16] use a single PETS
2011 video sequence which they manually label at
pixel resolution using a graphical editor.
• Brutzer et al., 2011 [6] use a synthetic (computer-
generated) dataset produced from only one 3D scene
representing a street corner. The sequences include
illumination changes, dynamic background, shadows
and noise, while lacking frames with no activity.
• Prati et al., 2001 [19] use indoor sequences containing
one moving person that are manually segmented into
foreground (human), shadow, and background areas.
Only 112 frames have ground-truth labels.
• Rosin and Ioannidis, 2003 [21]usealabelingpro-
gram that automatically locates moving objects based
on their position in space and properties such as color,
size, shape, etc. These properties were not used by
the change detection algorithms tested. However, the
videos used are not realistic as they are limited to lab
scenes with balls rolling on the floor.
• Bashir and Porikli, 2006 [3] conduct a performance
evaluation of tracking algorithms using the PETS 2001
dataset by comparing the detected bounding box loca-
tions with the ground-truth.
At a high level, the existing surveys suffer from three
main limitations. First, the statistics reported in these papers
have not been computed on a well-balanced dataset com-
posed of real (camera-captured) videos. Typically, synthetic
videos, real videos with synthetic moving objects pasted in,
or real videos out of which only 1 frame has been manually
segmented for ground truth are used. Furthermore, very few
datasets contain more than 10 videos. Secondly, none of
the papers are accompanied by a fully-operational web site
that allows users to upload their results and compare them
against those of others. Thirdly, the survey papers often re-
port common, fairly simple motion detection methods, and
do not report the performance of more complex methods.
3. New Dataset: CDnet
CDnet, presented at the IEEE Change Detection Work-
shop [1], consists of 31 videos depicting indoor and out-
door scenes with boats, cars, trucks, and pedestrians that
have been captured in different scenarios and contain a
range of challenges. The videos have been obtained with
different cameras ranging from low-resolution IP cameras,
through mid-resolution camcorders and PTZ cameras, to
thermal cameras. As a consequence, spatial resolutions of
the videos in CDnet vary from 320 × 240 to 720 × 576.
Also, due to diverse lighting conditions present and com-
pression parameters used, the level of noise and compres-
sion artifacts varies from one video to another. The length
of the videos also varies from 1,000 to 8,000 frames and
the videos shot by low-end IP cameras suffer from notice-
able radial distortion. Different cameras may have differ-
ent hue bias (due to different white balancing algorithms
employed) and some cameras apply automatic exposure ad-
justment resulting in global brightness fluctuations in time.
We believe that the fact that our videos have been captured
under a range of settings will help prevent this dataset from
favoring a certain family of change detection methods over
others.
The videos are grouped into six categories according to
the type of challenge each represents. We selected videos so
that the challenge in one category is unique to that category.
For example, only videos in the “Shadows” category con-
tain strong shadows and only those in the “Dynamic Back-
ground” category contain strong parasitic background mo-
tion. Such a grouping is essential for a clear identification of
the strengths and weaknesses of different change detection
methods. With the exception of one video in the “Baseline”
category, that comes from the PETS 2006 dataset, all the
videos have been captured by the authors.
3.1. Video Categories
31 videos totaling nearly 90,000 frames are grouped into
the following 6 categories (Fig. 1) that have been selected
to cover a wide range of change detection challenges that
are representative of typical visual data captured today in
surveillance, smart environment, and video analytics appli-
cations:
1. Baseline: This category contains four videos, two in-
door and two outdoor. These videos represent a mix-
ture of mild challenges typical of the next 4 categories.
Some videos have subtle background motion, others
have isolated shadows, some have an abandoned object
and others have pedestrians that stop for a short while
and then move away. These videos are fairly easy, but
not trivial, to process, and are provided mainly as ref-
erence.
3
“Baseline” “Dynamic Background” “Camera Jitter” “Shadows” “Interm. Object Motion” “Thermal”
Figure 1. Sample video frames from each of the 6 categories in the new dataset available at www.changedetection.net and typical
detection results obtained using basic background subtraction [4] reported in the last row of Table 1.
2. Dynamic Background: There are six videos in this
category depicting outdoor scenes with strong (para-
sitic) background motion. Two videos represent boats
on shimmering water, two videos show cars passing
next to a fountain, and the last two depict pedestrians,
cars and trucks passing in front of a tree shaken by the
wind (second column in Fig. 1).
3. Camera Jitter: This category contains one indoor and
three outdoor videos captured by unstable (e.g., vibrat-
ing) cameras. The jitter magnitude varies from one
video to another.
4. Shadows: This category consists of two indoor and
four outdoor videos exhibiting strong as well as faint
shadows. Some shadows are fairly narrow while others
occupy most of the scene. Also, some shadows are cast
by moving objects while others are cast by trees and
buildings.
5. Intermittent Object Motion: This category contains
six videos with scenarios known for causing “ghost-
ing” artifacts in the detected motion, i.e., objects move,
then stop for a short while, after which they start
moving again. Some videos include still objects that
suddenly start moving, e.g., a parked vehicle driving
away, and also abandoned objects. This category is
intended for testing how various algorithms adapt to
background changes. One example of such a challenge
is shown in the 5-th column of Fig. 1 where new ob-
jects are added to or existing objects are removed from
the scene.
6. Thermal: In this category, five videos (three outdoor
and two indoor) have been captured by far-infrared
cameras. These videos contain typical thermal arti-
facts such as heat stamps (e.g., bright spots left on a
seat after a person gets up and leaves), heat reflection
on floors and windows (see the last column of Fig. 1),
and camouflage effects, when a moving object has the
same temperature as the surrounding regions.
We would like to mention that although camou-
flage, caused by moving objects that have very similar
color/texture to the background, is among the most glaring
change detection issues, we have not created a camouflage
category. This is partially because almost every real video
sequence contains some level of camouflage. It is difficult
to create a dataset in which there is a category exclusively
with camouflage challenges while other categories are void
of it.
3.2. Ground-Truth Labels
As mentioned in Section 2, the current online datasets
have been designed mainly for testing tracking and scene
understanding algorithms, and thus the ground truth is pro-
vided in the form of bounding boxes. Although this can be
used to validate change detection methods, a precise vali-
dation requires ground truth at pixel resolution. Therefore,
ideally, videos should be labeled a number of times by dif-
ferent persons and the results averaged out. This, however,
is impractical due to resource and time constraints. Fur-
thermore, it is very difficult for a person to produce uncon-
troversial binary ground-truth images for camera-captured
videos. This is particularly difficult near moving object
boundaries and in semi-transparent areas. Due to motion
blur and partially-opaque objects (e.g., sparse bushes, di rty
windows, fountains), pixels in these areas may contain both
the moving object and background. As a consequence, one
cannot reliably classify such pixels as belonging to either
Static or Moving class. Since these areas carry a certain
level of uncertainty, evaluation metrics should not be com-
puted for pixels in these areas. Therefore, we decided to
produce ground-truth images with the following labels:
Static: assigned grayscale value of 0,
Shadow: assigned grayscale value of 50,
4
Non-ROI
8
: assigned grayscale value of 85,
Unknown: assigned grayscale value of 170,
Moving assigned grayscale value of 255.
The Static and Moving classes are associated with pixels
for which the motion status is obvious. The Shadow label
is associated with hard and well-defined moving shadows
such as the one in Fig. 2. Hard shadows are among the most
difficult artifacts to cope with and we believe that adding
this extra information improves the richness and utility of
the dataset. Please note that evaluation metrics discussed in
Section 3.3 consider the Shadow pixels as Static pixels. The
Unknown label is assigned to pixels that are half-occluded
and those corrupted by motion blur. All pixels located close
to moving-object boundaries are automatically labeled as
Unknown (Fig. 2). This prevents evaluation metrics from
being corrupted by pixels whose status is unclear.
The Non-ROI (not in region of interest) label serves two
purposes. Firstly, since most change detection methods in-
cur a delay before their background model stabilizes, we la-
beled the first few hundred frames of each video sequence as
Non-ROI. This prevents the corruption of evaluation metrics
due to errors during initialization. Secondly, the Non-ROI
label prevents the metrics from being corrupted by activities
unrelated to the category considered. An example of this sit-
uation is shown in the second row of Fig. 2, which illustrates
a sequence of cars that arrive, stop at a street light and then
move away. The goal of the video is to measure how well
a change detection method can handle intermittent motion.
However, since the scene is cluttered with unrelated activi-
ties (cars on the highway) the Non-ROI label puts the focus
on street-light activities. Similarly, the top row in Fig. 2 il-
lustrates the Shadow category; the Non-ROI label is used to
prevent the metrics from corruption by trees moving in the
background.
3.3. Evaluation Metrics
Finding the right metric to accurately measure the ability
of a method to detect motion or change without producing
excessive false positives and false negatives is not trivial.
For instance, recall favors methods with a low False Nega-
tive Rate. On the contrary, specificity favors methods with a
low False Positive Rate. Having the entire precision-recall
tradeoff curve or the ROC curve would be ideal, but not
all methods have the flexibility to sweep through the com-
plete gamut of tradeoffs. In addition, one cannot, in general,
rank-order methods based on a curve. We deal with these
difficulties by reporting the average performance of each
method for each video category with respect to 7 different
performance metrics each of which has been well-studied
in the literature. Specifically, for each method, each video
category, and each metric, we report the performance (as
8
ROI stands for Region of Interest.
Motion
Static
Outside ROI
hard shadow
Unknown
Static
Motion
Outside ROI
Unknown
Figure 2. Sample video frames from the Bungalows and Street
light sequences and corresponding 5-class ground-truth label
fields.
measured by the value of the metric) of the method aver-
aged across all the videos of the category.
Let TP = number of true positives, TN = number of
true negatives, FN = number of false negatives, and FP =
number of false positives. The 7 metrics that we use are:
1. Recall (Re): TP/(TP + FN)
2. Specificity (Sp): TN/(TN + FP)
3. False Positive Rate (FPR): FP/(FP + TN)
4. False Negative Rate (FNR): FN/(TN + FP)
5. Percentage of Wrong Classifications (PWC):
100(FN + FP)/(TP + FN + FP + TN)
6. Precision (Pr): TP/(TP + FP)
7. F -measure: 2
Pr·Re
Pr+Re
For the Shadow category, we also provide an average
False Positive Rate that is confined to the hard-shadow areas
(FPR-S).
For each method, the above metrics are first computed
for each video in each category. For example, the recall
metric for a particular video v in a category c is computed
as follows:
Re
v,c
= TP
v,c
/(TP
v,c
+ FN
v,c
).
Then, a category-average metric for each category is com-
puted from the values of the metric for all videos in a single
category. For example, the average recall metric of category
c is given by
Re
c
=
1
|N
c
|
v
Re
v,c
where |N
c
| is the number of videos in category c. We also
report an overall-average metric which is the simple average
5