Primal-Dual Q-Learning Framework for LQR Design
Donghwan Lee,Jianghai Hu +1 more
TLDR
In this article, a new optimization formulation of the linear quadratic regulator (LQR) problem via the Lagrangian duality theories was proposed to lay theoretical foundations of potentially effective RL algorithms.Abstract:
Recently, reinforcement learning (RL) is receiving more and more attentions due to its successful demonstrations outperforming human performance in certain challenging tasks. The goal of this paper is to study a new optimization formulation of the linear quadratic regulator (LQR) problem via the Lagrangian duality theories in order to lay theoretical foundations of potentially effective RL algorithms. The new optimization problem includes the Q-function parameters so that it can be directly used to develop Q-learning algorithms, known to be one of the most popular RL algorithms. We prove relations between saddle-points of the Lagrangian function and the optimal solutions of the Bellman equation. As an example of its applications, we propose a model-free primal-dual Q-learning algorithm to solve the LQR problem and demonstrate its validity through examples.read more
Citations
More filters
Posted Content
LQR through the Lens of First Order Methods: Discrete-time Case.
TL;DR: It is shown that this cost function of the Linear-Quadratic-Regulator is smooth and coercive, and an alternate means of noting its gradient dominated property is provided, and it is proved that these flows are exponentially stable in the sense of Lyapunov.
Discrete Time Linear Systems Theory And Design With Applications
TL;DR: Thank you for reading discrete time linear systems theory and design with applications, and maybe you have knowledge that, people have look numerous times for their favorite novels, but end up in harmful downloads.
Posted Content
Policy Gradient-based Algorithms for Continuous-time Linear Quadratic Control
TL;DR: This work considers the continuous-time Linear-Quadratic-Regulator (LQR) problem in terms of optimizing a real-valued matrix function over the set of feedback gains, and develops the necessary formalism and insights for projected gradient descent, allowing for a sublinear rate of convergence to a first-order stationary point.
Posted Content
Event-triggered Learning for Linear Quadratic Control
TL;DR: A structured approach is obtained that decides when model learning is beneficial, by analyzing the probability distribution of the linear quadratic cost and designing a learning trigger that leverages Chernoff bounds.
Journal ArticleDOI
On the Optimization Landscape of Dynamic Output Feedback Linear Quadratic Control
TL;DR: It is shown that the dLQR cost varies with similarity transformations, and an explicit form of the optimal similarity transformation for a given observable stabilizing controller is derived, which provides an optimality certificate for policy gradient methods under mild assumptions.
References
More filters
Book
Reinforcement Learning: An Introduction
TL;DR: This book provides a clear and simple account of the key ideas and algorithms of reinforcement learning, which ranges from the history of the field's intellectual foundations to the most recent developments and applications.
Book
Convex Optimization
Stephen Boyd,Lieven Vandenberghe +1 more
TL;DR: In this article, the focus is on recognizing convex optimization problems and then finding the most appropriate technique for solving them, and a comprehensive introduction to the subject is given. But the focus of this book is not on the optimization problem itself, but on the problem of finding the appropriate technique to solve it.
Book
Dynamic Programming and Optimal Control
TL;DR: The leading and most up-to-date textbook on the far-ranging algorithmic methododogy of Dynamic Programming, which can be used for optimal control, Markovian decision problems, planning and sequential decision making under uncertainty, and discrete/combinatorial optimization.