© 2003 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in
any current or future media, including reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of
this work in other works.
DETECTING MOVING OBJECTS, GHOSTS AND SHADOWS IN VIDEO STREAMS 1
Detecting Moving Objects, Ghosts and Shadows in
Video Streams
Rita Cucchiara
∗ 1
,Costantino Grana
1
, Massimo Piccardi
2
, Andrea Prati
1
Abstract
Background subtraction methods are widely exploited for moving object detection in videos in many applications, such as
traffic monitoring, human motion capture and video surveillance. How to correctly and efficiently model and update the background
model and how to deal with shadows are two of the most distinguishing and challenging aspects of such approaches. This work
proposes a general-purpose method which combines statistical assumptions with the object-level knowledge of moving objects,
apparent objects (ghosts) and shadows acquired in the processing of the previous frames. Pixels belonging to moving objects,
ghosts and shadows are processed differently in order to supply an object-based selective update. The proposed approach exploits
color information for both background subtraction and shadow detection to improve object segmentation and background update.
The approach proves fast, flexible and precise in terms of both pixel accuracy and reactivity to background changes.
Keywords
Background modeling, color segmentation, reactivity to changes, shadow detection, video surveillance, object-level knowledge
I. INTRODUCTION
D
ETECTION of moving objects in video streams is the first relevant step of information extraction in
many computer vision applications, including video surveillance, people tracking, traffic monitoring
and semantic annotation of videos. In these applications, robust tracking of objects in the scene calls for a
reliable and effective moving object detection that should be characterized by some important features: high
precision, with the two meanings of accuracy in shape detection and reactivity to changes in time; flexibility
in different scenarios (indoor, outdoor) or different light conditions; and efficiency, in order for detection to
be provided in real-time. In particular, while the fast execution and flexibility in different scenarios should
∗
Corresponding author.
1
Dipartimento di Ingegneria dell’Informazione, Universit
`
a di Modena e Reggio Emilia, Via Vignolese, 905 - 41100 Modena - Italy - phone:
+39-059-2056136 - fax: +39-059-2056126 - e-mail:{rita.cucchiara/andrea.prati}@unimo.it, grana@dsi.unimo.it
2
Department of Computer Systems, Faculty of IT, University of Technology, Sydney - Broadway NSW 2007 - Australia - phone: +61-2-9514-
7942 - fax: +61-2-9514-1807 - e-mail: massimo@it.uts.edu.au
2 DETECTING MOVING OBJECTS, GHOSTS AND SHADOWS IN VIDEO STREAMS
be considered basic requirements to be met, precision is another important goal. In fact, a precise moving
object detection makes tracking more reliable (the same object can be identified more reliably from frame to
frame if its shape and position are accurately detected) and faster (multiple hypotheses on the object’s identity
during time can be pruned more rapidly). In addition, if object classification is required by the application,
precise detection substantially supports correct classification.
In this work, we assume that the models of the target objects and their motion are unknown, so as to achieve
maximum application independence. In the absence of any a priori knowledge about target and environment,
the most widely adopted approach for moving object detection with fixed camera is based on background
subtraction [1][2][3][4][5][6][7][8][9]. An estimate of the background (often called a background model) is
computed and evolved frame by frame: moving objects in the scene are detected by the difference between
the current frame and the current background model. It is well known that background subtraction carries two
problems for the precision of moving object detection. The first problem is that the model should reflect the
real background as accurately as possible, to allow the system accurate shape detection of moving objects.
The detection accuracy can be measured in terms of correctly and incorrectly classified pixels during normal
conditions of the object’s motion (i.e. the “stationary background” case). The second problem is that the
background model should immediately reflect sudden scene changes such as the start or stop of objects, so as
to allow detection of only the actual moving objects with high reactivity (the “transient background” case).
If the background model is neither accurate nor reactive, background subtraction causes the detection of
false objects, often referred to as “ghosts” [1][3]. In addition, moving object segmentation with background
suppression is affected by the problem of shadows [4][10]. Indeed, we would like the moving object detection
to not classify shadows as belonging to foreground objects, since the appearance and geometrical properties
of the object can be distorted, which in turn affects many subsequent tasks such as object classification and
the assessment of moving object position (normally considered to be the shape centroid). Moreover, the
probability of object undersegmentation (where more than one object is detected as a single object) increases
due to connectivity via shadows between different objects.
CUCCHIARA, GRANA, PICCARDI AND PRATI 3
Feature Systems
Statistics • Minimum and maximum values [1]
• Median [11][12], *
• Single Gaussian [5][4][13]
• Multiple Gaussians [14][10][3]
• Eigenbackground approximation [15][6]
• Minimization of Gaussian differences [7]
Adaptivity [1][6][5][8][16][2], *
Selectivity [10][2][8][1], *
Shadow [4][10], *
Ghost [1][3], *
High-frequency • Temporal filtering [14][15][6]
illumination changes • Size filtering *
Sudden global [1], *
illumination changes
TABLE I
COMPARED BACKGROUND SUBTRACTION APPROACHES. OUR APPROACH IS REFERRED WITH *.
Many works have been proposed in the literature as a solution to an efficient and reliable background
subtraction. Table I is a classification of the most relevant papers based on the features used. Most of the
approaches use a statistical combination of frames to compute the background model (see Table I). Some of
these approaches propose to combine the current frame and previous models with recursive filtering (adaptiv-
ity in Table I) to update the background model. Moreover, many authors propose to use pixel selectivity by
excluding from the background update process those pixels detected as in motion. Finally, problems carried
by shadows have been addressed [4][10][17]. In this paper we propose a novel simple method that exploits
all these features, combining them so as to efficiently provide detection of moving objects, ghosts and shad-
ows. The main contribution of this proposal is the integration of knowledge of detected objects, shadows
and ghosts in the segmentation process to enhance both object segmentation and background update. The
resulting method proves to be accurate and reactive, and at the same time fast and flexible in the applications.
II. DETECTING MOVING OBJECTS, GHOSTS AND SHADOWS
The first aim of our proposal is to detect real moving objects with high accuracy, limiting false negatives
(object’s pixels that are not detected) as much as possible. The second aim is to extract pixels of moving
4 DETECTING MOVING OBJECTS, GHOSTS AND SHADOWS IN VIDEO STREAMS
objects with the maximum responsiveness possible, avoiding detection of transient spurious objects, such as
cast shadows, static objects or noise.
To accomplish these aims, we propose a taxonomy of the objects of interest in the scene, using the following
definitions (see also Fig. 1):
• Moving visual object (MVO): set of connected points belonging to object characterized by non-null mo-
tion.
• Uncovered Background: the set of visible scene points currently not in motion.
• Background (B): is the computed model of the background.
• Ghost (G): a set of connected points detected as in motion by means of background subtraction, but not
corresponding to any real moving object.
• Shadow: a set of connected background points modified by a shadow cast over them by a moving object.
Shadows can be further classified as MVO shadow (MVO
SH
), that is, a shadow connected with an MVO and
hence sharing its motion, and ghost shadow (G
SH
), being a shadow not connected with any real MVO.
Static cast shadows are neither detected nor considered since they do not affect moving object segmentation
if background subtraction is used: in fact, static shadows are included in the background model. A ghost
shadow can be a shadow cast either by a ghost or an MVO: the shape and/or position of the MVO with
respect to the light source can lead to the shadow not being connected to the object that generates it.
Our proposal makes use of the the explicit knowledge of all the above five categories for a precise segmen-
tation and an effective background model update. We call our approach Sakbot (Statistical And Knowledge-
Based ObjecT detection) since it exploits statistics and knowledge of the segmented objects to improve both
background modeling and moving object detection. Sakbot is depicted in Fig. 1, reporting the aforementioned
taxonomy. Sakbot’s processing is the first step for different further processes, such as object classification,
tracking, video annotation and so on.
Let us call p a point of the video frame at time t (I
t
). I
t
(p) is the value of point p in the color space. Since
images are acquired by standard color cameras or decompressed from videos with standard formats, the basic