THÈSE
Pour obtenir le grade de
DOCTEUR DE L’UNIVERSITÉ DE GRENOBLE
Spécialité : Mathématique, Informatique (Robotique)
Arrêté ministérial :
Présentée par
Ricardo Omar CHAVEZ GARCIA
Thèse dirigée par Olivier AYCARD
préparée au sein Laboratoire d’Informatique de Grenoble
et de Mathématiques, Sciences et Technologies de l’Information, Infor-
matique
Multiple Sensor Fusion for Detec-
tion, Classification and Tracking of
Moving Objects in Driving Environ-
ments
Thèse soutenue publiquement le 25 septembre 2014,
devant le jury composé de :
M., Michel DEVY
LAAS-CNRS, Rapporteur
M., François CHARPILLET
INRIA Nancy, Rapporteur
Mme., Michelle ROMBAUT
Gipsa-Lab, Président, Examinatrice
M., Yassine RUICHEK
Université de Technologie de Belfort-Montbéliard, Examinateur
M., Olivier AYCARD
Université de Grenoble1, Directeur de thèse
Multiple Sensor Fusion for Detection,
Classification and Tracking of Moving
Objects in Driving Environments
ii
To:
my family
and my friends.
iii
Abstract
Advanced driver assistance systems (ADAS) help drivers to perform complex driving
tasks and to avoid or mitigate dangerous situations. The vehicle senses the external
world using sensors and then builds and updates an internal model of the environment
configuration. Vehicle perception consists of establishing the spatial and temporal rela-
tionships between the vehicle and the static and moving obstacles in the environment.
Vehicle perception is composed of two main tasks: simultaneous localization and map-
ping (SLAM) deals with modelling static parts; and detection and tracking moving
objects (DATMO) is responsible for modelling moving parts of the environment. The
perception output is used to reason and decide which driving actions are the best for
specific driving situations. In order to perform a good reasoning and control, the sys-
tem has to correctly model the surrounding environment. The accurate detection and
classification of moving objects is a critical aspect of a moving object tracking system.
Therefore, many sensors are part of a common intelligent vehicle system.
Multiple sensor fusion has been a topic of research since long; the reason is the
need to combine information from different views of the environment to obtain a more
accurate model. This is achieved by combining redundant and complementary mea-
surements of the environment. Fusion can be performed at different levels inside the
perception task.
Classification of moving objects is needed to determine the possible behaviour of
the objects surrounding the vehicle, and it is usually performed at tracking level. Knowl-
edge about the class of moving objects at detection level can help to improve their
tracking, reason about their behaviour, and decide what to do according to their na-
ture. Most of the current perception solutions consider classification information only
as aggregate information for the final perception output. Also, the management of in-
complete information is an important issue in these perception systems. Incomplete in-
formation can be originated from sensor-related reasons, such as calibration issues and
hardware malfunctions; or from scene perturbations, like occlusions, weather issues
and object shifting. It is important to manage these situations by taking into account
iv
the degree of imprecision and uncertainty into the perception process.
The main contributions in this dissertation focus on the DATMO stage of the per-
ception problem. Precisely, we believe that including the object’s class as a key element
of the object’s representation and managing the uncertainty from multiple sensors de-
tections, we can improve the results of the perception task, i.e., a more reliable list of
moving objects of interest represented by their dynamic state and appearance informa-
tion. Therefore, we address the problems of sensor data association, and sensor fusion
for object detection, classification, and tracking at different levels within the DATMO
stage. We believe that a richer list of tracked objects can improve future stages of an
ADAS and enhance its final results.
Although we focus on a set of three main sensors: radar, lidar, and camera, we pro-
pose a modifiable architecture to include other type or number of sensors. First, we
define a composite object representation to include class information as a part of the
object state from early stages to the final output of the perception task. Second, we
propose, implement, and compare two different perception architectures to solve the
DATMO problem according to the level where object association, fusion, and classifica-
tion information is included and performed. Our data fusion approaches are based on
the evidential framework, which is used to manage and include the uncertainty from
sensor detections and object classifications. Third, we propose an evidential data as-
sociation approach to establish a relationship between two sources of evidence from
object detections. We apply this approach at tracking level to fuse information from
two track representations, and at detection level to find the relations between observa-
tions and to fuse their representations. We observe how the class information improves
the final result of the DATMO component. Fourth, we integrate the proposed fusion
approaches as a part of a real-time vehicle application. This integration has been per-
formed in a real vehicle demonstrator from the interactIVe European project.
Finally, we analysed and experimentally evaluated the performance of the pro-
posed methods. We compared our evidential fusion approaches against each other
and against a state-of-the-art method using real data from different driving scenarios
and focusing on the detection, classification and tracking of different moving objects:
pedestrian, bike, car and truck. We obtained promising results from our proposed ap-
proaches and empirically showed how our composite representation can improve the
final result when included at different stages of the perception task.
Key Words: Multiple sensor fusion, intelligent vehicles, perception, DATMO, clas-
sification, multiple object detection & tracking, Dempster-Shafer theory
v