Reproductive Success, T.H. Clutton-Brock (Ed.). The University of Chicago Press, Chicago (1988), ix

A model used for velocity control during car following is proposed based on reinforcement learning (RL). To optimize driving performance, a reward function is developed by referencing human driving data and combining driving features related to safety, efficiency, and comfort. With the developed reward function, the RL agent learns to control vehicle speed in a fashion that maximizes cumulative rewards, through trials and errors in the simulation environment. To avoid potential unsafe actions, the proposed RL model is incorporated with a collision avoidance strategy for safety checks. The safety check strategy is used during both model training and testing phases, which results in faster convergence and zero collisions. A total of 1,341 car-following events extracted from the Next Generation Simulation (NGSIM) dataset are used to train and test the proposed model. The performance of the proposed model is evaluated by the comparison with empirical NGSIM data and with adaptive cruise control (ACC) algorithm implemented through model predictive control (MPC). The experimental results show that the proposed model demonstrates the capability of safe, efficient, and comfortable velocity control and outperforms human drivers in that it 1) has larger TTC values than those of human drivers, 2) can maintain efficient and safe headways around 1.2s, and 3) can follow the lead vehicle comfortably with smooth acceleration (jerk value is only a third of that of human drivers). Compared with the MPC-based ACC algorithm, the proposed model has better performance in terms of safety, comfort, and especially running speed during testing (more than 200 times faster). The results indicate that the proposed approach could contribute to the development of better autonomous driving systems. Source code of this paper can be found at https://github.com/MeixinZhu/Velocity_control .

Safe, efficient, and comfortable velocity control based on reinforcement learning for autonomous driving

Autonomous vehicle (AV) technology can provide a safe and convenient transportation solution for the public, but the complex and various environments in the real world make it difficult to operate safely and reliably. A connected autonomous vehicle (CAV) is an AV with vehicle connectivity capability, which enhances the situational awareness of the AV and enables the cooperation between AVs. Hence, CAV technology can enhance the capabilities and robustness of AV to be a promising transportation solution in the future. This paper introduces a representative architecture of CAVs and surveys the latest research advances, methods, and algorithms for sensing, perception, planning, and control of CAVs. It reviews the state-of-the-art and state-of-the-practice (when applicable) of a multi-layer Perception-Planning-Control architecture including on-board sensors and vehicular communications, the methods of sensor fusion and localization and mapping in the perception layer, the algorithms of decision making and trajectory planning in the planning layer, and the control strategies of trajectory tracking in the control layer. Furthermore, the implementations and impact of vehicle connectivity and the corresponding consequential challenges of cooperative perception, complex connected decision making, and multi-vehicle controls are summarized and their significant research issues enumerated. Most importantly, the critical review in this paper provides a list and discussion of the remaining challenges and unsolved problems of CAVs in each Section which would be helpful to researchers in the field. The comprehensive coverage of this paper makes it particularly useful to academic researchers, practitioners, and students alike.

Research Advances and Challenges of Autonomous and Connected Ground Vehicles

This study compares Deep Reinforcement Learning (DRL) and Model Predictive Control (MPC) for Adaptive Cruise Control (ACC) design in car-following scenarios. A first-order system is used as the Control-Oriented Model (COM) to approximate the acceleration command dynamics of a vehicle. Based on the equations of the control system and the multi-objective cost function, we train a DRL policy using Deep Deterministic Policy Gradient (DDPG) and solve the MPC problem via Interior-Point Optimization (IPO). Simulation results for the episode costs show that, when there are no modeling errors and the testing inputs are within the training data range, the DRL solution is equivalent to MPC with a sufficiently long prediction horizon. Particularly, the DRL episode cost is only 5.8% higher than the benchmark optimal control solution provided by optimizing the entire episode via IPO. The DRL control performance degrades when the testing inputs are outside the training data range, indicating inadequate machine learning generalization. When there are modeling errors due to control delay, disturbances, and/or testing with a High-Fidelity Model (HFM) of the vehicle, the DRL-trained policy performs better when the modeling errors are large while having similar performances as MPC when the modeling errors are small.

Comparison of Deep Reinforcement Learning and Model Predictive Control for Adaptive Cruise Control

Cochlodinium Type '78 Yatsushiro (Fig. 1), occurs almost exclusively in Yatsushiro Sea, southwest of Kyushu Island, and is incriminated in mass mortality of cultured fish such as yellowtail and red sea bream in this region.1) The mechanism by which fish are killed is uncertain, but our preliminary research results indicate that fish are presumably affected by some toxic substances released during the bloom of this dinoflagellate.

日本水産學會誌 = Bulletin of the Japanese Society of Scientific Fisheries

Cooperative adaptive cruise control (CACC) communicates the relevant preceding vehicle state data to the follower (ego) vehicle to improve the vehicle following capabilities. In general, the CACC utilizes the preceding vehicle's desired acceleration from wireless communication as a feedforward term in the controller of the ego vehicle, which dominantly determines the total control input. However, communication loss would degrade CACC to adaptive cruise control (ACC), where the lack of the feedforward term during communication loss would increase the inter-vehicular distance or, otherwise, may lead to collision during vehicle emergency braking. This paper proposes a control algorithm with an adaptive Kalman filter estimating the acceleration of a preceding vehicle, and the estimated acceleration is implemented as a feedforward signal in the ego-vehicle CACC controller in case of communication loss. The proposed control algorithm is evaluated by the experiments using mobile robots that emulate driving. In addition, the simulations of real vehicles are also conducted. The results of simulations and robot experiments show that the performance of implementing the adaptive Kalman filter during communication loss is better than fallback to ACC and the normal Kalman filter based on the Singer model.

/pdf/cooperative-adaptive-cruise-control-with-adaptive-kalman-4dzfzmyjo5.pdf

Cooperative Adaptive Cruise Control With Adaptive Kalman Filter Subject to Temporary Communication Loss

Deep Reinforcement Learning (DRL) is used here for decentralized decision-making and longitudinal control for high-speed on-ramp merging. The DRL environment state includes the states of five vehicles: the merging vehicle, along with two preceding and two following vehicles when the merging vehicle is or is projected on the main road. The control action is the acceleration of the merging vehicle. Deep Deterministic Policy Gradient (DDPG) is the DRL algorithm for training to output continuous control actions. We investigated the relationship between collision avoidance for safety and jerk minimization for passenger comfort in the multi-objective reward function by obtaining the Pareto front. We found that, with a small jerk penalty in the multi-objective reward function, the vehicle jerk could be reduced by 73% compared with no jerk penalty while the collision rate was maintained at zero. Regardless of the jerk penalty, the merging vehicle exhibited decision-making strategies such as merging ahead or behind a main-road vehicle.

Anti-Jerk On-Ramp Merging Using Deep Reinforcement Learning

The majority of current studies on autonomous vehicle control via deep reinforcement learning (DRL) utilize point-mass kinematic models, neglecting vehicle dynamics which includes acceleration delay and acceleration command dynamics. The acceleration delay, which results from sensing and actuation delays, results in delayed execution of the control inputs. The acceleration command dynamics dictates that the actual vehicle acceleration does not rise up to the desired command acceleration instantaneously due to dynamics. In this work, we investigate the feasibility of applying DRL controllers trained using vehicle kinematic models to more realistic driving control with vehicle dynamics. We consider a particular longitudinal car-following control, i.e., Adaptive Cruise Control (ACC), problem solved via DRL using a point-mass kinematic model. When such a controller is applied to car following with vehicle dynamics, we observe significantly degraded car-following performance. Therefore, we redesign the DRL framework to accommodate the acceleration delay and acceleration command dynamics by adding the delayed control inputs and the actual vehicle acceleration to the reinforcement learning environment state, respectively. The training results show that the redesigned DRL controller results in near-optimal control performance of car following with vehicle dynamics considered when compared with dynamic programming solutions.

Yuan Lin

Papers

Comparison of Deep Reinforcement Learning and Model Predictive Control for Adaptive Cruise Control

Cooperative Adaptive Cruise Control With Adaptive Kalman Filter Subject to Temporary Communication Loss

Anti-Jerk On-Ramp Merging Using Deep Reinforcement Learning

Longitudinal Dynamic versus Kinematic Models for Car-Following Control Using Deep Reinforcement Learning

Longitudinal Dynamic versus Kinematic Models for Car-Following Control Using Deep Reinforcement Learning