This is a repository copy of Pedestrian Models for Autonomous Driving Part II: High-Level
Models of Human Behaviour.
White Rose Research Online URL for this paper:
https://eprints.whiterose.ac.uk/162669/
Version: Accepted Version
Article:
Camara, F orcid.org/0000-0002-2655-1228, Bellotto, N, Cosar, S et al. (11 more authors)
(2021) Pedestrian Models for Autonomous Driving Part II: High-Level Models of Human
Behaviour. IEEE Transactions on Intelligent Transportation Systems, 22 (9). pp. 5453-
5472. ISSN 1524-9050
https://doi.org/10.1109/TITS.2020.3006767
© 2020 IEEE. All rights reserved. Personal use of this material is permitted. Permission
from IEEE must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes, creating new
collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted
component of this work in other works.
eprints@whiterose.ac.uk
https://eprints.whiterose.ac.uk/
Reuse
Items deposited in White Rose Research Online are protected by copyright, with all rights reserved unless
indicated otherwise. They may be downloaded and/or printed for private study, or other acts as permitted by
national copyright laws. The publisher or other rights holders may allow further reproduction and re-use of
the full text version. This is indicated by the licence information on the White Rose Research Online record
for the item.
Takedown
If you consider content in White Rose Research Online to be in breach of UK law, please notify us by
emailing eprints@whiterose.ac.uk including the URL of the record and the reason for the withdrawal request.
IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS 1
Pedestrian Models for Autonomous Driving
Part II: High-Level Models of Human Behaviour
Fanta Camara
1,2
, Nicola Bellotto
2
, Serhan Cosar
3
, Florian Weber
4
, Dimitris Nathanael
5
, Matthias Althoff
6
,
Jingyuan Wu
7
, Johannes Ruenz
7
, Andr
´
e Dietrich
8
, Gustav Markkula
1
, Anna Schieben
9
, Fabio Tango
10
,
Natasha Merat
1
and Charles Fox
1,2,11
Abstract—Autonomous vehicles (AVs) must share space with
pedestrians, both in carriageway cases such as cars at pedestrian
crossings and off-carriageway cases such as delivery vehicles
navigating through crowds on pedestrianized high-streets. Unlike
static obstacles, pedestrians are active agents with complex, inter-
active motions. Planning AV actions in the presence of pedestrians
thus requires modelling of their probable future behaviour as
well as detecting and tracking them. This narrative review article
is Part II of a pair, together surveying the current technology
stack involved in this process, organising recent research into
a hierarchical taxonomy ranging from low-level image detection
to high-level psychological models, from the perspective of an
AV designer. This self-contained Part II covers the higher levels
of this stack, consisting of models of pedestrian behaviour, from
prediction of individual pedestrians’ likely destinations and paths,
to game-theoretic models of interactions between pedestrians and
autonomous vehicles. This survey clearly shows that, although
there are good models for optimal walking behaviour, high-level
psychological and social modelling of pedestrian behaviour still
remains an open research question that requires many conceptual
issues to be clarified. Early work has been done on descriptive and
qualitative models of behaviour, but much work is still needed
to translate them into quantitative algorithms for practical AV
control.
Index Terms—Review, survey, pedestrians, autonomous vehi-
cles, sensing, detection, tracking, trajectory prediction, pedestrian
interaction, microscopic and macroscopic behaviour models,
game-theoretic models, signalling models, eHMI, datasets.
I. INTRODUCTION
To operate successfully in the presence of pedestrians,
autonomous vehicles require input from a huge variety of
models that have to work seamlessly together. These models
range from simple visual models for detection of pedestrians,
to predicting their future movements using psychological and
sociological methods. Part I of this two-part survey [33] cov-
ered models for sensing, detection, recognition, and tracking
of pedestrians. Part II here reviews models for pedestrian
This project has received funding from EU H2020 interACT (723395).
1
Institute for Transport Studies (ITS), University of Leeds, UK
2
Lincoln Centre for Autonomous Systems, University of Lincoln, UK
3
Institute of Engineering Sciences, De Montfort University, UK
4
Bayerische Motoren Werke Aktiengesellschaft (BMW), Germany
5
School of Mechanical Engineering, Nat. Tech. University of Athens
6
Department of Computer Science, Technische Universit
¨
at M
¨
unchen
7
Robert Bosch GmbH, Germany
8
Chair of Ergonomics, Technische Universit
¨
at M
¨
unchen (TUM), Germany
9
DLR (German Aerospace Center), Germany
10
Centro Ricerche Fiat (CRF), Italy
11
Ibex Automation Ltd, UK
Manuscript received 2019-03-11; Revisions: 2019-10-21, 2020-03-26.
Sensing
Detection
Recognition
Tracking
Prediction
Interaction
Game
Theory
Signalling
Level 0: No Automation
Level 1: ‘Hands on’
Level 2: ‘Hands off’
Level 3: ‘Eyes off’
Level 5: Full Automation
Level 4: ‘Mind off’
Fig. 1. Main structure of the review.
trajectory prediction, interaction of pedestrians, and behavioral
modelling of pedestrians, and also experimental resources to
validate all the types of models. Interacting with pedestrians
is a particular type of social intelligence. Autonomous vehi-
cles will need to utilize many different levels of models of
pedestrians, each addressing different aspects of perception
and action. Each of these models can be based on empirical
science results or obtained via machine learning. In contrast to
the models of Part I, Part II requires models from higher levels
of the technology stack, as researched by psychologists and
taught in advanced driver training programmes. For instance,
drivers often try to infer the personality of other humans,
predict their likely behaviours, and interact with them to
communicate mutual intentions [102]. Between the high level
surveyed in this Part II and the low levels of Part I, researchers
infer psychological information from perceptual information.
As an example, researchers build systems to recognize the
body language, gestures, and demographics information of
pedestrians to better predict their likely goals and behaviours.
Despite the importance of bridging the research between the
higher and lower levels, their connection is still thin, both
conceptually and in terms of actual implementations.
While prediction of likely future pedestrian trajectories is
becoming increasingly well understood, models for actively
controlling pedestrian interactions – including game-theoretic
models – are still in their infancy. Active control here means
that the vehicle’s own future actions are taken into account
in predicting how the pedestrian will respond, and vice versa.
One reason is that sufficient data to rigorously study interac-
tion between pedestrians has only recently become available
IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS 2
TABLE I
PROPOSE D MA PPING FROM SAE LEVELS TO PEDESTRIAN MODEL REQUIREMENTS.
SAE LEVEL DESCRIPTION MODEL REQUIREMENTS SECTION
0 No Automation. Automated system issues warnings and may mo-
mentarily intervene, but has no sustained vehicle control.
Sensing Part I [33] Sec. II
1 Hands on. The driver and the automated system share control of
the vehicle. For example, adaptive cruise control (ACC), where the
driver controls steering and the automated system controls speed.
The driver must be ready to resume full control when needed.
+Detection Part I [33] Sec. III
2 Hands off. The automated system takes full control of the vehicle
(steering and speed). The driver must monitor and be prepared to
intervene immediately. Occasional contact between hand and wheel
is often mandatory to confirm that the driver is ready to intervene.
+Recognition
+Tracking
Part I [33] Sec. IV
Part I [33] Sec. V
3 Eyes off. Driver can safely turn attention away from the driving
tasks, e.g. use a phone or watch a movie. Vehicle will handle
situations that call for an immediate response, like emergency
braking. The driver must still be prepared to intervene within some
limited time.
+Unobstructed Walking Models, Known Goals
+Behaviour Prediction, Known Goals
+Behaviour Prediction, Unknown Goals
Sec. II-A
Sec. II-B
Sec. II-C
4 Mind off. No driver attention is required for safety, except in limited
spatial areas (geofenced) or under special circumstances, like traffic
jams.Outside of these areas or circumstances, the vehicle must be
able to safely abort or transfer control to the human.
+Event/Activity Models
+Effects of Pedestrian Class on Trajectory
+Pedestrian Interaction Models
+Game Theoretic and Signalling Models
Sec. II-D
Sec. II-E
Sec. III
Sec. IV
5 Full automation. No human intervention is required at all, fully
automated driving.
+Extreme Robustness and Reliability
Note: ‘+X’ means that ‘X’ is required in addition to the requirements of the previous level.
as presented in Sec. V on experimental resources. Another
reason is that one first has to be able to reliably sense, detect,
recognize, and track pedestrians in order to gather enough data
for modelling interaction and game-theoretic models. A third
reason is that interaction and game-theoretic models are only
relevant in crowded environments, while many situations do
not require much interaction. However, crowded environments
are those that are typically most relevant for autonomous
driving. Fig. 1 shows the review structure.
To assess the maturity of the methods presented, the level
of autonomy is used, as defined by the Society of Automotive
Engineers (SAE) – the same measure has already been used in
Part I [33]. For the convenience of the reader, the five SAE lev-
els are briefly presented, ranging from simple driver assistance
tools to full self-driving [183]. Requirements for pedestrian
modelling increase with each level, with lower levels typically
requiring lower and more mature levels of pedestrian models,
such as detection and tracking, while higher levels require
models for psychological and social understanding to fully
interact with pedestrians in a human-like way [30]. Table I
gives an overview of SAE levels and requirements mappings.
While many papers propose pedestrian models at various
levels, no unifying theory has yet been produced which would
make it possible to easily transfer results across all levels
from detection to prediction. This review uncovers bottlenecks
in transferring results to facilitate closing existing research
gaps. Also, many existing studies only consider results from
empirical science or those obtained via machine learning. This
survey provides an overview considering both possibilities.
While machine learning results work particularly well for
detection and recognition, they are not yet performing so
well for prediction. Some reasons are that prediction is a
more high-dimensional problem, with dimensions including
goals, obstacles, various state variables of pedestrians, and
road geometry. A further reason is that less labelled data is
available for training prediction models. A promising future
direction is to combine empirical science results with machine
learning to better safeguard techniques using machine learning
and to avoid over-fitting.
While similar concepts apply to modelling human drivers
and their vehicles for interactions with AVs, this article
presents a review of the state of the art specifically in
modelling human pedestrians for social decision-making. In
some cases it goes beyond modelling aspects to also cover
more conceptual aspects or empirical psychological findings,
when the studies in question are judged to have very direct
applicability to mathematical models. Results from human
driving cannot be directly translated to pedestrians due to the
variability in locomotion, the differences in shape, the changes
in postures and the less-structured environment.
Pedestrians are defined as humans moving on and near
public highways including roads and pedestrianised areas, who
walk using their own locomotive power. This excludes, for
example, humans moving on cycles, wheelchairs and other
mobility devices, skates and skateboards, or those transported
by other humans. This review does not cover interactions of
traffic participants without pedestrians: a survey on trajectory
prediction of on-road vehicles is provided in [123] and a
survey on vision-based trajectory learning is provided in [146].
This Part II is organized as shown in Fig. 2. In Sec. II,
methods for predicting the movements of pedestrians are
reviewed. In particular, we consider models and methods for
unstructured environments, for prediction around obstacles, to
estimate destinations, and for the prediction of events such as
IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS 3
Fig. 2. Structure of the paper.
crossing the road. These methods are enhanced in Sec. III
for groups of pedestrians interacting with each other. This
section considers the complete variety of researched models
from macroscopic models only considering flow of people
to microscopic models that consider individual pedestrians.
In many situations, interaction models do not require game
theory, because pedestrians often have different goals. How-
ever, there are also many situations, where pedestrians have
competing goals, e.g., when several pedestrians have to pass a
narrow passage. In such situations, the game theoretic models
presented in Sec. IV can be very useful. Finally, Sec. V
surveys available resources: datasets and simulators, both for
pedestrians and vehicles.
II. BEHAVIOUR MODELS WITHOUT INTERACTION
The tracking models reviewed in Part I are kinematic in
that they assume that pedestrians move in physical and/or pose
space in motion described by kinematic models. This is a very
basic assumption – human drivers typically have much more
complex understandings and hence predictions of pedestrian
behavior which they use to drive safely in their presence
[102]. These range from slightly more advanced kinematic
understandings such as edestrians tend to walk in straight lines
to models of how they are likely to interact with static objects
in their environment, and predictions of pedestrians’ likely
destinations from reading the street scene.
This section reviews such models starting from simple
unobstructed path models to uncertain destination models and
more advanced event/activity models. These models do not yet
consider interaction with other agents. Figure 3 summarizes
the classes of models presented in this section. A previous
review was proposed by Ridel et al. [172], which mainly con-
sidered pedestrian crossing intent and offered a restricted view
of the different models developed for trajectory prediction.
A. Unobstructed Walking Models with Known Goals
Given a start location and orientation, and a goal location,
humans do not typically turn towards the goal on the spot
(which would waste time) and then walk in a straight line,
but rather set off walking in their initial heading and adjust
their orientation gradually as they walk, resulting in smooth,
curved trajectories from origin to destination [72]. Models
from optimal control theory as also used in robotics [50] define
cost functions for travel time, speed, and accelerations, to
reproduce these characteristic curved trajectories. The model
in [72] instead achieves curved trajectories by modelling the
rate of turning of the pedestrian as a function of the visual
angle and distance to the goal. A simple kinematic model
Trajectory
&
Interaction
Models
Signalling Interaction Models
Game Theoretic Models
Macroscopic Models: Crowd Interactions
Microscopic Models: Group Interactions
Microscopic Models: Two Agents’ Interactions
Event/Activity Models
Behaviour Prediction with Unknown Goals
Behaviour Prediction with Known Goals
Unobstructed Walking Models with Known Goals
Fig. 3. Pedestrian behaviour prediction and interaction models.
consists in considering human locomotion as a nonholonomic
motion [161], using the unicycle model (1) where the pedes-
trian walking trajectory is represented by the trajectory of their
center of gravity, 2D coordinates (x, y) and by the angle θ,
˙x = u
1
cos θ
˙y = u
1
sin θ
˙
θ = u
2
(1)
where u
1
is the forward velocity and u
2
is the angular velocity.
Assuming known origin and destination with inverse optimal
control, one can reliably predict human walking paths using
this model [9] [155].
B. Behaviour Prediction with Known Goals
Here, the likely behaviour of a pedestrian in a static en-
vironment is considered, given a map. Pedestrians are likely
to route around obstacles, and to stop at the edges of roads
before crossing. This section does not consider social effects
of other agents – this is presented later in Sec. III.
1) Dynamic Graphical Models: Dynamic Graphical Mod-
els (DGM) are Graphical Models of a particular topology,
containing some Markovian sequence of variables over time.
DGMs include simple Markov and Hidden Markov Models
and also more complex models. The method in [145] used
tracking in a DGM based on particle filter approximation to
infer beliefs over future pedestrian trajectories and combined
this with a GNSS (Global Navigation Satellite System) mod-
ule that provides information about the hazardous areas and
people.
2) Gaussian Process Methods: Habibi et al. [88] proposed
a context-based approach to pedestrian trajectory prediction
using Gaussian Processes [166]. This model incorporates
context features such as the pedestrian’s distance to the traffic
IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS 4
light, the distance to the curbside, and the curbside orientation
in the transition learning phase to improve the prediction.
A context-based augmented semi non-negative sparse coding
(CASNSC) algorithm is used to predict pedestrian trajectories.
3) Deep Learning Methods: Bock et al. [24] developed a
Recurrent Neural Network (LSTM) model to learn pedestrian
behaviour patterns at intelligent intersections using camera
data from the onboard vehicle and the infrastructure. The
model can predict trajectories for a horizon of 5s.
4) Other Methods: Kruse et al. [121] was one of the first
attempts to statistically infer human motion patterns from data
and incorporate them in a robot motion planner for obstacle
avoidance. Garz
´
on et al. [77] presented a pedestrian trajectory
prediction model based on two path planning algorithms that
require a set of possible goals, a map and the initial position. It
then computes similarities between the obtained and observed
trajectories into probabilities. This model is run along with a
pedestrian detector and tracker. Tamura et al. [198] proposed
a pedestrian behaviour model that is based on social forces
and takes into account the intention of the pedestrian in the
trajectory prediction by defining a set of subgoals. In [170]
the uncertain goals are used as latent variables to guide the
motion prediction of pedestrians. Their positions are predicted
by combining forward propagation of a physical model with
local a priori information (e.g., obstacles and different road
types) from the start position, and by planning the trajectory
from a goal position. The distribution over the destinations is
modeled with a particle filter.
In [209], Vasishta et al. presented a model based on the
principle of natural vision that incorporates contextual in-
formation extracted from the environment to the pedestrian
behavior and it especially tries to predict hazardous behavior
such as crossing in non authorized areas. The aforementioned
model in [72] considers goals and obstacles as distance-
dependent attractors and repellers in heading angle space. The
contributions from the goal and obstacles are linearly com-
bined, yielding a momentary rate of acceleration of heading,
which results in human-like trajectories for simultaneous goal-
seeking and obstacle avoidance. In [57], Dias et al. developed
a model simulating pedestrian behaviour around corners, using
minimum jerk theory and one-thirds power law concept. Their
model uses Monte Carlo simulation to generate pedestrian
trajectories with turning maneuvers, which were comparable
to empirical trajectories.
C. Behaviour Prediction with Unknown Goals
Many of the above models assume known probable des-
tinations for pedestrians, which enable routing to act not
just around local obstacles, but to predict entire long-term
trajectories, such as for pedestrians intending to cross the road.
However, in reality a pedestrian’s destination is rarely given.
1) Dynamic Graphical Models: Ziebart et al. [233] pre-
sented a pedestrian trajectory prediction model that takes into
account hindrance due to robot motion, as is required in off-
carriageway interactions such as last mile AVs in pedestri-
anized areas. A maximum entropy inverse optimal control
technique, introduced in [232], is used and is equivalent to
a soft-maximum version of Markov decision process (MDP)
that accounts for decision uncertainty into the trajectories
distribution. The cost function is a linear combination of the
features (e.g obstacles) in the environment. People’s motion
can be modeled by an MDP and by choosing a certain path,
there is an immediate reward. The model is conditioned on
a known destination location but the model reasons about all
possible destinations and the real destination is not known at
the prediction time. The destination is inferred in a Bayesian
way, by computing the prior distributions over destinations
using previous observed trajectories. When there is no previous
data, features (door, chair etc.) in the environment are used
to model the destination. In [113], Kitani et al. extended
[232], [233] by incorporating visual features to forecast future
activities and destinations. The observations provided by the
vision system (e.g. tracking algorithm) are assumed to be noisy
and uncertain therefore they used a hidden variable Markov de-
cision process (hMDP) where the agent knows its own states,
action and reward but observes only noisy measurements.
Negative Log-Loss (NLL) is used as a probabilistic metric and
Modified Hausdorff Distance (MHD) as a physical measure of
the distance between two trajectories. Vasquez [210] extends
the work of Ziebart [233] and Kitani [113] while reducing
computational costs.
Bennewitz et al. [18], [17] proposed a learning method for
human motion recognition using the expectation maximization
(EM) and a hidden Markov model (HMM) for clustering and
predicting human trajectories and incorporating them into a
robot path planner. In [221], Wu et al. presented a model that
uses Markov chains for pedestrian motion prediction (able to
deal with non-Gaussian distribution and several constraints).
A heuristic method is proposed to automatically infer the
positions of several potential goals on a generic semantic map.
It also incorporates policies to predict the pedestrian motion
direction and takes into account other traffic participants by
incorporating a collision checking approach. Borgers et al.
[29] presented a model that predicts pedestrians’ route choice
based on Markov chains. Similarly, Bai et al. [11] presented a
real-time approximate POMDP (Partially Observable Markov
Decision Process) controller, DESPOT, for use in high-street
type environments. The method is intention-aware in the sense
of inferring pedestrian destinations and route plans from their
observed motion over time, and accounting for the value of
this information against the value of making progress while
planning a robot’s own route around them. Karasev et al.
[110] presented a long-term prediction model that incorporates
environmental constraints with the intent modeled by a policy
in a MDP framework. The pedestrian state is estimated using
a Rao-Blackwellized filter and pedestrian intent by planning
according to a stochastic policy. This model assumes that
pedestrians behave rationally.
2) Deep Learning Methods: Hug et al. [98] proposed a
LSTM-MDL model combined with a particle filter method
for multi-modal trajectory prediction, and tested on Stanford
Drone Dataset (SDD) [176]. Rehder et al. [171] proposed
a method to infer pedestrian destinations. The trajectory
prediction is computed as a goal-oriented motion planning.
The whole system is based on deep-learning and trained
