Proceedings ArticleDOI
Online actor critic algorithm to solve the continuous-time infinite horizon optimal control problem
Kyriakos G. Vamvoudakis,Frank L. Lewis +1 more
- pp 58-65
Reads0
Chats0
TLDR
This paper presents an online adaptive algorithm implemented as an actor/critic structure which involves simultaneous continuous-time adaptation of both actor and critic neural networks, and calls this ‘synchronous’ policy iteration.Abstract:
In this paper we discuss an online algorithm based on policy iteration for learning the continuous-time (CT) optimal control solution with infinite horizon cost for nonlinear systems with known dynamics. We present an online adaptive algorithm implemented as an actor/critic structure which involves simultaneous continuous-time adaptation of both actor and critic neural networks. We call this ‘synchronous’ policy iteration. A persistence of excitation condition is shown to guarantee convergence of the critic to the actual optimal value function. Novel tuning algorithms are given for both critic and actor networks, with extra terms in the actor tuning law being required to guarantee closed-loop dynamical stability. The convergence to the optimal controller is proven, and stability of the system is also guaranteed. Simulation examples show the effectiveness of the new algorithm.read more
Citations
More filters
Journal ArticleDOI
Reinforcement Learning and Feedback Control: Using Natural Decision Methods to Design Optimal Adaptive Controllers
TL;DR: In this article, the authors describe the use of reinforcement learning to design feedback controllers for discrete and continuous-time dynamical systems that combine features of adaptive control and optimal control, which are not usually designed to be optimal in the sense of minimizing user-prescribed performance functions.
Journal ArticleDOI
Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics
Yu Jiang,Zhong-Ping Jiang +1 more
TL;DR: This paper presents a novel policy iteration approach for finding online adaptive optimal controllers for continuous-time linear systems with completely unknown system dynamics, using the approximate/adaptive dynamic programming technique to iteratively solve the algebraic Riccati equation using the online information of state and input.
Journal ArticleDOI
Optimal and Autonomous Control Using Reinforcement Learning: A Survey
TL;DR: Q-learning and the integral RL algorithm as core algorithms for discrete time (DT) and continuous-time (CT) systems, respectively are discussed, and a new direction of off-policy RL for both CT and DT systems is discussed.
Journal ArticleDOI
Data-Driven Robust Approximate Optimal Tracking Control for Unknown General Nonlinear Systems Using Adaptive Dynamic Programming Method
TL;DR: A novel data-driven robust approximate optimal tracking control scheme is proposed for unknown general nonlinear systems by using the adaptive dynamic programming (ADP) method and a robustifying term is developed to compensate for the NN approximation errors introduced by implementing the ADP method.
Journal ArticleDOI
A novel actor-critic-identifier architecture for approximate optimal control of uncertain nonlinear systems
Shubhendu Bhasin,Rushikesh Kamalapurkar,Marcus Johnson,Kyriakos G. Vamvoudakis,Frank L. Lewis,Warren E. Dixon +5 more
TL;DR: An online adaptive reinforcement learning-based solution is developed for the infinite-horizon optimal control problem for continuous-time uncertain nonlinear systems using a novel actor-critic-identifier (ACI) architecture to approximate the Hamilton-Jacobi-Bellman equation.
References
More filters
Book
Reinforcement Learning: An Introduction
TL;DR: This book provides a clear and simple account of the key ideas and algorithms of reinforcement learning, which ranges from the history of the field's intellectual foundations to the most recent developments and applications.
Book
Introduction to Reinforcement Learning
TL;DR: In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning.
Neuro-Dynamic Programming.
TL;DR: In this article, the authors present the first textbook that fully explains the neuro-dynamic programming/reinforcement learning methodology, which is a recent breakthrough in the practical application of neural networks and dynamic programming to complex problems of planning, optimal decision making, and intelligent control.
Book
Neuro-dynamic programming
TL;DR: This is the first textbook that fully explains the neuro-dynamic programming/reinforcement learning methodology, which is a recent breakthrough in the practical application of neural networks and dynamic programming to complex problems of planning, optimal decision making, and intelligent control.