Adaptive dynamic programming (ADP) and reinforcement learning are quite relevant to each other when performing intelligent optimization. They are both regarded as promising methods involving important components of evaluation and improvement, at the background of information technology, such as artificial intelligence, big data, and deep learning. Although great progresses have been achieved and surveyed when addressing nonlinear optimal control problems, the research on robustness of ADP-based control strategies under uncertain environment has not been fully summarized. Hence, this survey reviews the recent main results of adaptive-critic-based robust control design of continuous-time nonlinear systems. The ADP-based nonlinear optimal regulation is reviewed, followed by robust stabilization of nonlinear systems with matched uncertainties, guaranteed cost control design of unmatched plants, and decentralized stabilization of interconnected systems. Additionally, further comprehensive discussions are presented, including event-based robust control design, improvement of the critic learning rule, nonlinear $ {H_{\infty }}$ control design, and several notes on future perspectives. By applying the ADP-based optimal and robust control methods to a practical power system and an overhead crane plant, two typical examples are provided to verify the effectiveness of theoretical results. Overall, this survey is beneficial to promote the development of adaptive critic control methods with robustness guarantee and the construction of higher level intelligent systems.

Adaptive Critic Nonlinear Robust Control: A Survey

To synthesize fixed-final-time control-constrained optimal controllers for discrete-time nonlinear control-affine systems, a single neural network (NN)-based controller called the Finite-horizon Single Network Adaptive Critic is developed in this paper. Inputs to the NN are the current system states and the time-to-go, and the network outputs are the costates that are used to compute optimal feedback control. Control constraints are handled through a nonquadratic cost function. Convergence proofs of: 1) the reinforcement learning-based training method to the optimal solution; 2) the training error; and 3) the network weights are provided. The resulting controller is shown to solve the associated time-varying Hamilton-Jacobi-Bellman equation and provide the fixed-final-time optimal solution. Performance of the new synthesis technique is demonstrated through different examples including an attitude control problem wherein a rigid spacecraft performs a finite-time attitude maneuver subject to control bounds. The new formulation has great potential for implementation since it consists of only one NN with single set of weights and it provides comprehensive feedback solutions online, though it is trained offline.

Finite-Horizon Control-Constrained Nonlinear Optimal Control Using Single Network Adaptive Critics

In this paper, the infinite horizon optimal robust guaranteed cost control of continuous-time uncertain nonlinear systems is investigated using neural-network-based online solution of Hamilton–Jacobi–Bellman (HJB) equation. By establishing an appropriate bounded function and defining a modified cost function, the optimal robust guaranteed cost control problem is transformed into an optimal control problem. It can be observed that the optimal cost function of the nominal system is nothing but the optimal guaranteed cost of the original uncertain system. A critic neural network is constructed to facilitate the solution of the modified HJB equation corresponding to the nominal system. More importantly, an additional stabilizing term is introduced for helping to verify the stability, which reinforces the updating process of the weight vector and reduces the requirement of an initial stabilizing control. The uniform ultimate boundedness of the closed-loop system is analyzed by using the Lyapunov approach as well. Two simulation examples are provided to verify the effectiveness of the present control approach.

Neural-Network-Based Online HJB Solution for Optimal Robust Guaranteed Cost Control of Continuous-Time Uncertain Nonlinear Systems

In this paper, we develop and analyze an optimal control method for a class of discrete-time nonlinear Markov jump systems (MJSs) with unknown system dynamics. Specifically, an identifier is established for the unknown systems to approximate system states, and an optimal control approach for nonlinear MJSs is developed to solve the Hamilton-Jacobi-Bellman equation based on the adaptive dynamic programming technique. We also develop detailed stability analysis of the control approach, including the convergence of the performance index function for nonlinear MJSs and the existence of the corresponding admissible control. Neural network techniques are used to approximate the proposed performance index function and the control law. To demonstrate the effectiveness of our approach, three simulation studies, one linear case, one nonlinear case, and one single link robot arm case, are used to validate the performance of the proposed optimal control method.

Optimal control for unknown discrete-time nonlinear Markov jump systems using adaptive dynamic programming.

This paper presents a novel robust regulation method for a class of continuous-time nonlinear systems subject to unmatched perturbations. To begin with, the robust regulation problem is transformed into an optimal regulation problem by constructing a value function for the auxiliary system. Then, a simultaneous policy iteration (SPI) algorithm is developed to solve the optimal regulation problem within the framework of adaptive dynamic programming. To implement the SPI algorithm, actor and critic networks are employed to approximate the optimal control and the optimal value function, respectively, and the Monte Carlo integration method is applied to obtain the unknown weight parameters. Finally, two examples, including a power system, are provided to demonstrate the applicability of the developed approach.

Adaptive Dynamic Programming for Robust Regulation and Its Application to Power Systems

In this study, an optimal control algorithm based on Hamilton-Jacobi-Bellman (HJB) equation, for the bounded robust controller design for finite-time-horizon nonlinear systems, is proposed. The HJB equation formulated using a suitable nonquadratic term in the performance functional to take care of magnitude constraints on the control input. Utilising the direct method of Lyapunov stability, we have proved the optimality of the controller with respect to a cost functional, that includes penalty on the control effort and the maximum bound on system uncertainty. The bounded controller requires the knowledge of the upper bound of system uncertainty. In the proposed algorithm, neural network is used to approximate the time-varying solution of HJB equation using least squares method. Proposed algorithm has been applied on the nonlinear system with matched and unmatched system uncertainties. Necessary theoretical and simulation results are presented to validate proposed algorithm.

Fixed final time optimal control approach for bounded robust controller design using Hamilton-Jacobi-Bellman solution

In this paper, constrained controller design is proposed for a class of nonlinear discrete-time uncertain system having matched type system uncertainties, using the solution of HJB (Hamilton-Jaccobi-Bellman) equation. The discrete-time HJB equation is formulated using a suitable non-quadratic term in the performance functional to tackle constraints on the control input. Based on the non-quadratic functional, a greedy HDP algorithm is used to obtain the constrained robust-optimal controller. The constrained robust controller requires knowledge of the upper bound of system uncertainties. For facilitating the implementation of the iterative algorithm, two neural networks are used to approximate the value function and to compute the optimal control policy, respectively. Their weights have been tuned using least squares method. Proposed algorithm has been applied on a nonlinear discrete-time system with matched uncertainties.

/pdf/constrained-controller-design-for-a-class-of-nonlinear-28kpgbml06.pdf

Constrained controller design for a class of nonlinear discrete-time uncertain systems

In this paper, a new algorithm is proposed for the constrained control of weakly coupled nonlinear systems. The controller design problem is solved by solving Hamilton-Jacobi-Bellman(HJB) equation with modified cost to tackle constraints on the control input and unknown coupling. In the proposed controller design framework, coupling terms have been formulated as model uncertainties. The bounded controller requires the knowledge of the upper bound of the uncertainty. In the proposed algorithm, Neural Network (NN) is used to approximate the solution of HJB equation using least squares method. Necessary theoretical and simulation results are presented to validate proposed algorithm.

/pdf/constrained-control-of-weakly-coupled-nonlinear-systems-3dwiawaf6u.pdf

M. Gopal

Papers

Fixed final time optimal control approach for bounded robust controller design using Hamilton-Jacobi-Bellman solution

Constrained controller design for a class of nonlinear discrete-time uncertain systems

Constrained Control of Weakly Coupled Nonlinear Systems Using Neural Network