With the development of deep representation learning, the domain of reinforcement learning (RL) has become a powerful learning framework now capable of learning complex policies in high dimensional environments. This review summarises deep reinforcement learning (DRL) algorithms and provides a taxonomy of automated driving tasks where (D)RL methods have been employed, while addressing key computational challenges in real world deployment of autonomous driving agents. It also delineates adjacent domains such as behavior cloning, imitation learning, inverse reinforcement learning that are related but are not classical RL algorithms. The role of simulators in training agents, methods to validate, test and robustify existing solutions in RL are discussed.

Deep Reinforcement Learning for Autonomous Driving: A Survey

With the development of deep representation learning, the domain of reinforcement learning (RL) has become a powerful learning framework now capable of learning complex policies in high dimensional environments. This review summarises deep reinforcement learning (DRL) algorithms and provides a taxonomy of automated driving tasks where (D)RL methods have been employed, while addressing key computational challenges in real world deployment of autonomous driving agents. It also delineates adjacent domains such as behavior cloning, imitation learning, inverse reinforcement learning that are related but are not classical RL algorithms. The role of simulators in training agents, methods to validate, test and robustify existing solutions in RL are discussed. 

/pdf/deep-reinforcement-learning-for-autonomous-driving-a-survey-2f5i21xk.pdf

A digital twin smart city for citizen feedback

Traffic signal control is an emerging application scenario for reinforcement learning. Besides being as an important problem that affects people's daily life in commuting, traffic signal control poses its unique challenges for reinforcement learning in terms of adapting to dynamic traffic environment and coordinating thousands of agents including vehicles and pedestrians. A key factor in the success of modern reinforcement learning relies on a good simulator to generate a large number of data samples for learning. The most commonly used open-source traffic simulator SUMO is, however, not scalable to large road network and large traffic flow, which hinders the study of reinforcement learning on traffic scenarios. This motivates us to create a new traffic simulator CityFlow with fundamentally optimized data structures and efficient algorithms. CityFlow can support flexible definitions for road network and traffic flow based on synthetic and real-world data. It also provides user-friendly interface for reinforcement learning. Most importantly, CityFlow is more than twenty times faster than SUMO and is capable of supporting city-wide traffic simulation with an interactive render for monitoring. Besides traffic signal control, CityFlow could serve as the base for other transportation studies and can create new possibilities to test machine learning methods in the intelligent transportation domain.

CityFlow: A Multi-Agent Reinforcement Learning Environment for Large Scale City Traffic Scenario

Simulation has the potential to massively scale evaluation of self-driving systems, enabling rapid development as well as safe deployment. Bridging the gap between simulation and the real world requires realistic multi-agent behaviors. Existing simulation environments rely on heuristic-based models that directly encode traffic rules, which cannot capture irregular maneuvers (e.g., nudging, U-turns) and complex interactions (e.g., yielding, merging). In contrast, we leverage real-world data to learn directly from human demonstration, and thus capture more naturalistic driving behaviors. To this end, we propose TrafficSim, a multi-agent behavior model for realistic traffic simulation. In particular, we parameterize the policy with an implicit la-tent variable model that generates socially-consistent plans for all actors in the scene jointly. To learn a robust policy amenable for long horizon simulation, we unroll the policy in training and optimize through the fully differentiable simulation across time. Our learning objective incorporates both human demonstrations as well as common sense. We show TrafficSim generates significantly more realistic traffic scenarios as compared to a diverse set of baselines. Notably, we can exploit trajectories generated by TrafficSim as effective data augmentation for training better motion planner.

/pdf/trafficsim-learning-to-simulate-realistic-multi-agent-11k8e9ng1h.pdf

TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors

Microscopic traffic simulation is an invaluable tool for traffic research. In recent years, both the scope of research and the capabilities of the tools have been extended considerably. This article presents the latest developments concerning intermodal traffic solutions, simulator coupling and model development and validation on the example of the open source traffic simulator SUMO.

/pdf/microscopic-traffic-simulation-using-sumo-14sgzmhde7.pdf

Microscopic Traffic Simulation using SUMO

For emergency vehicle drivers it is an important task to reach the incident location as fast as possible. Therefore a self-organizing green wave could help emergency vehicles
to accomplish this goal. This study presents an approach how emergency vehicle can be prioritized at traffic lights and simulates the possible benefit for the emergency vehicle.
Traffic data from vehicular communication can be used to find the optimal timing for the traffic light to modify the existing traffic phases and reduce the possible negative impact
on other traffic participants.

/pdf/modelling-green-waves-for-emergency-vehicles-using-connected-g21hk0rtd9.pdf

Modelling green waves for emergency vehicles using connected traffic data.

This paper presents how emergency vehicles can be modeled and simulated in the microscopic traffic simulation SUMO (Simulation of Urban MObility). The special rights of emergency vehicles are implemented in the SUMO framework and can be switched off and on in the simulation with a blue light device. The surrounding traffic reacts accordingly to the emergency vehicle and form an emergency lane. In addition real world data from emergency vehicles are used to evaluate the driving behavior of emergency vehicles and compare it to real world and simulated vehicle characteristics. The evaluation results show that the simulated vehicles pass an intersection generally faster than in real world. For emergency vehicle a time saving of in average one second at a single intersection could be measured for right turning vehicles.

/pdf/analysis-of-the-traffic-behavior-of-emergency-vehicles-in-a-4z9kb1mkui.pdf

Analysis of the traffic behavior of emergency vehicles in a microscopic traffic simulation

Traffic simulations can help to investigate new traffic and transportation management solutions for overcoming
problems like traffic jams, accidents or environmental
pollution For this a valid simulation model is
needed This paper provides an overview of the open
source traffic simulation framework SUMO (Simulation
of Urban MObility) and evaluates the implemented carfollowing
models at controlled intersections with regard
to vehicle positions and speeds Particularly intersections
can be bottlenecks for high traffic volumes and
have a higher risk for accidents Therefore this study
focuses on traffic behavior at urban intersections

Evaluation of car-following-models at controlled intersections

Timing issue is very critical for emergency service. Due to the increasing traffic it has become more and more challenging for emergency vehicles to reach their destinations within a given service period in cities, especially during rush hours. Therefore, there is a need to model routing strategies to support emergency vehicles. Microscopic traffic simulation can therefore be an adequate decision-support tool for route strategies analysis. The GPS positions of 24 emergency vehicles in Brunswick have been tracked over a time period of 5 years.In this paper, the route choice behavior of emergency vehicles is analyzed with the collected trajectory data in Brunswick, Germany. In addition, microscopic traffic simulation is also used to understand the deviation between simulated and real route choices. For this purpose an emergency vehicle model was implemented in the traffic simulation SUMO. The emergency vehicle model allows vehicle in the simulation to violate red traffic lights and force other traffic participants to give way etc.Emergency vehicle drivers have time pressure when they are on duty. Furthermore, the likelihood for emergency vehicle drivers being involved in a traffic accident is significant higher than for private vehicles. Therefore, their concern on route choice may be different from that of other normal road users. Besides travel time the other essential route choice factors are also investigated and derived for developing the respective route choice model so that the costs of the simulated routes of emergency vehicles are close to those in the reality. It is expected that the developed model can be transferable and applied in other cities.

Laura Bieker-Walz

Papers

Microscopic Traffic Simulation using SUMO

Modelling green waves for emergency vehicles using connected traffic data.

Analysis of the traffic behavior of emergency vehicles in a microscopic traffic simulation

Evaluation of car-following-models at controlled intersections

Modelling of Emergency Vehicles Route Choice with Use of Trajectory Data