This paper deals with the design of an $ {H}_{ {\infty }}$ tracking controller for nonlinear continuous-time systems with completely unknown dynamics. A general bounded $L_{2} $ -gain tracking problem with a discounted performance function is introduced for the $ {H}_{ {\infty }}$ tracking. A tracking Hamilton–Jacobi–Isaac (HJI) equation is then developed that gives a Nash equilibrium solution to the associated min–max optimization problem. A rigorous analysis of bounded $L_{2}$ -gain and stability of the control solution obtained by solving the tracking HJI equation is provided. An upper-bound is found for the discount factor to assure local asymptotic stability of the tracking error dynamics. An off-policy reinforcement learning algorithm is used to learn the solution to the tracking HJI equation online without requiring any knowledge of the system dynamics. Convergence of the proposed algorithm to the solution to the tracking HJI equation is shown. Simulation examples are provided to verify the effectiveness of the proposed method.

$ {H}_{ {\infty }}$ Tracking Control of Completely Unknown Continuous-Time Systems via Off-Policy Reinforcement Learning

In this paper, a value iteration adaptive dynamic programming (ADP) algorithm is developed to solve infinite horizon undiscounted optimal control problems for discrete-time nonlinear systems. The present value iteration ADP algorithm permits an arbitrary positive semi-definite function to initialize the algorithm. A novel convergence analysis is developed to guarantee that the iterative value function converges to the optimal performance index function. Initialized by different initial functions, it is proven that the iterative value function will be monotonically nonincreasing, monotonically nondecreasing, or nonmonotonic and will converge to the optimum. In this paper, for the first time, the admissibility properties of the iterative control laws are developed for value iteration algorithms. It is emphasized that new termination criteria are established to guarantee the effectiveness of the iterative control laws. Neural networks are used to approximate the iterative value function and compute the iterative control law, respectively, for facilitating the implementation of the iterative ADP algorithm. Finally, two simulation examples are given to illustrate the performance of the present method.

Value Iteration Adaptive Dynamic Programming for Optimal Control of Discrete-Time Nonlinear Systems

In this paper, the problem of adaptive fuzzy decentralized optimal control is investigated for a class of nonlinear large-scale systems in strict-feedback form. The considered nonlinear large-scale systems contain the unknown nonlinear functions and unmeasured states. By utilizing the fuzzy logic systems to approximate the unknown nonlinear functions and cost functions, a fuzzy state observer is established to estimate the unmeasured states. The control design is divided into two phases. First, by using the state observer and the backstepping design technique, a feedforward decentralized controller with parameters adaptive laws is designed, by which the original controlled strict-feedback nonlinear large-scale system is transformed into an equivalent affine nonlinear large-scale system. Second, by using adaptive dynamic programming theory, a feedback decentralized optimal controller is developed for the equivalent affine nonlinear system. The whole adaptive fuzzy decentralized optimal control scheme consists of a feedforward decentralized controller and a feedback decentralized optimal controller. It is shown that the proposed adaptive fuzzy decentralized optimal control approach can guarantee that all the signals in the closed-loop system are bounded, and the tracking errors converge to a small neighborhood of zero. In addition, the proposed control approach can guarantee that the cost functions are minimized. Simulation results are given to demonstrate the effectiveness of the proposed control approach.

Observer-Based Adaptive Fuzzy Decentralized Optimal Control Design for Strict-Feedback Nonlinear Large-Scale Systems

Adaptive dynamic programming (ADP) and reinforcement learning are quite relevant to each other when performing intelligent optimization. They are both regarded as promising methods involving important components of evaluation and improvement, at the background of information technology, such as artificial intelligence, big data, and deep learning. Although great progresses have been achieved and surveyed when addressing nonlinear optimal control problems, the research on robustness of ADP-based control strategies under uncertain environment has not been fully summarized. Hence, this survey reviews the recent main results of adaptive-critic-based robust control design of continuous-time nonlinear systems. The ADP-based nonlinear optimal regulation is reviewed, followed by robust stabilization of nonlinear systems with matched uncertainties, guaranteed cost control design of unmatched plants, and decentralized stabilization of interconnected systems. Additionally, further comprehensive discussions are presented, including event-based robust control design, improvement of the critic learning rule, nonlinear $ {H_{\infty }}$ control design, and several notes on future perspectives. By applying the ADP-based optimal and robust control methods to a practical power system and an overhead crane plant, two typical examples are provided to verify the effectiveness of theoretical results. Overall, this survey is beneficial to promote the development of adaptive critic control methods with robustness guarantee and the construction of higher level intelligent systems.

Adaptive Critic Nonlinear Robust Control: A Survey

In this paper, we propose a data-driven supplementary control approach with adaptive learning capability for air-breathing hypersonic vehicle tracking control based on action-dependent heuristic dynamic programming (ADHDP). The control action is generated by the combination of sliding mode control (SMC) and the ADHDP controller to track the desired velocity and the desired altitude. In particular, the ADHDP controller observes the differences between the actual velocity/altitude and the desired velocity/altitude, and then provides a supplementary control action accordingly. The ADHDP controller does not rely on the accurate mathematical model function and is data driven. Meanwhile, it is capable to adjust its parameters online over time under various working conditions, which is very suitable for hypersonic vehicle system with parameter uncertainties and disturbances. We verify the adaptive supplementary control approach versus the traditional SMC in the cruising flight, and provide three simulation studies to illustrate the improved performance with the proposed approach.

Air-Breathing Hypersonic Vehicle Tracking Control Based on Adaptive Dynamic Programming

In this brief, an optimal control scheme based on adaptive dynamic programming (ADP) is developed to solve infinite-horizon optimal control problems of continuous-time complex-valued nonlinear systems. A new performance index function is established on the basis of complex-valued state and control. Using system transformations, the complex-valued system is transformed into a real-valued one, which overcomes Cauchy-Riemann conditions effectively. With the transformed system and the performance index function, a new ADP method is developed to obtain the optimal control law by using neural networks. A compensation controller is developed to compensate the approximation errors of neural networks. Stability properties of the nonlinear system are analyzed and convergence properties of the weights for neural networks are presented. Finally, simulation results demonstrate the performance of the developed optimal control scheme for complex-valued nonlinear systems.

Adaptive Dynamic Programming for a Class of Complex-Valued Nonlinear Systems

A novel self-learning optimal control method for a class of discrete-time nonlinear systems is proposed based on iteration adaptive dynamic programming (ADP) algorithm. It is proven that the iteration costate functions converge to the optimal one, and a detailed convergence analysis of the iteration ADP algorithm is given. Furthermore, echo state network (ESN) architecture is used as the approximator of the costate function for each iteration. To ensure the reliability of the ESN approximator, the ESN mean square training error is constrained in the satisfactory range. Two simulation examples are given to demonstrate that the proposed control method has a fast response speed due to the special structure and the fast training process.

A new self-learning optimal control laws for a class of discrete-time nonlinear systems based on ESN architecture

This paper proposes a novel finite-time optimal control method based on input---output data for unknown nonlinear systems using adaptive dynamic programming (ADP) algorithm. In this method, the single-hidden layer feed-forward network (SLFN) with extreme learning machine (ELM) is used to construct the data-based identifier of the unknown system dynamics. Based on the data-based identifier, the finite-time optimal control method is established by ADP algorithm. Two other SLFNs with ELM are used in ADP method to facilitate the implementation of the iterative algorithm, which aim to approximate the performance index function and the optimal control law at each iteration, respectively. A simulation example is provided to demonstrate the effectiveness of the proposed control scheme.

Neural-network-based approach to finite-time optimal control for a class of unknown nonlinear systems

Optimal Tracking Control for a Class of Unknown Discrete-time Systems with Actuator Saturation via Data-based ADP Algorithm

Optimal Tracking Control for a Class of Unknown Discrete-time Systems with Actuator Saturation via Data-based ADP Algorithm: Optimal Tracking Control for a Class of Unknown Discrete-time Systems with Actuator Saturation via Data-based ADP Algorithm

Chang-Yin Sun

Papers

Adaptive Dynamic Programming for a Class of Complex-Valued Nonlinear Systems

A new self-learning optimal control laws for a class of discrete-time nonlinear systems based on ESN architecture

Neural-network-based approach to finite-time optimal control for a class of unknown nonlinear systems

Optimal Tracking Control for a Class of Unknown Discrete-time Systems with Actuator Saturation via Data-based ADP Algorithm

Optimal Tracking Control for a Class of Unknown Discrete-time Systems with Actuator Saturation via Data-based ADP Algorithm: Optimal Tracking Control for a Class of Unknown Discrete-time Systems with Actuator Saturation via Data-based ADP Algorithm