Survey/review study
Adaptive Dynamic Programming for Networked Control Systems under Communication Constraints: A Survey of Trends and Techniques

Xueli Wang 1, Ying Sun 1,*, and Derui Ding 2


1 Department of Control Science and Engineering, University of Shanghai for Science and Technology, Shanghai 200093, China
2 School of Software and Electrical Engineering, Swinburne University of Technology, Melbourne 3122, Australia
* Correspondence: yingsun1991@163.com
 
 
Received: 22 September 2022
Accepted: 28 November 2022
Published: 22 December 2022
 

Abstract: The adaptive dynamic programming (ADP) technology has been widely used benefiting from its recursive structure in forward and the prospective conception of reinforcement learning. Furthermore, ADP-based control issues with communication constraints arouse ever-increasing research consideration in theoretical analysis and engineering applications due mainly to the wide participation of digital communications in industrial systems. The latest development of ADP-based optimal control with communication constraints is systematically surveyed in this paper. To this end, the development of ADP-based dominant methods is first investigated from their structures and implementation. Then, technical challenges and corresponding approaches are comprehensively and thoroughly discussed and the existing results are reviewed according to the constraint types. Furthermore, some applications of the ADP method in practical systems are summarized. Finally, future topics are lighted on ADP-based control issues.

/pdf/adaptive-dynamic-programming-for-networked-control-systems-1wkgo9ru.pdf

Adaptive Dynamic Programming for Networked Control Systems Under Communication Constraints: A Survey of Trends and Techniques

This paper proposes a novel distributed fault-tolerant consensus tracking control design for multi-agent systems with abrupt and incipient actuator faults under fixed and switching topologies. The fault and state information of each individual agent is estimated by merging unknown input observer in the decentralized fault estimation hierarchy. Then, two kinds of distributed fault-tolerant consensus tracking control schemes with average dwelling time technique are developed to guarantee the mean-square exponential consensus convergence of multi-agent systems, respectively, on the basis of the relative neighboring output information as well as the estimated information in fault estimation. Simulation results demonstrate the effectiveness of the proposed fault-tolerant consensus tracking control algorithm.

Distributed Fault-Tolerant Consensus Tracking Control of Multi-Agent Systems Under Fixed and Switching Topologies

The adaptive practical optimal time-varying formation tracking problems of the disturbed high-order multi-agent systems with a noncooperative leader are considered. Different from the former achievements, the effects of the leader’s unknown control input and followers’ external disturbances are both considered in the optimal time-varying formation tracking issues. Firstly, an adaptive practical optimal time-varying formation tracking protocol is proposed. The extended state observers and adaptive neural networks are introduced to estimate the integrated uncertainty and value function for the adaptive protocol, respectively. Then, an algorithm is presented to determine the control parameters for the adaptive optimal protocol and neural networks weights update laws. Thirdly, the stability and the optimal formation tracking property are analyzed for the closed loop disturbed high-order multi-agent system. Finally, the numerical simulation results are presented for revealing the effectiveness of the obtained theoretical methods.

Adaptive Practical Optimal Time-Varying Formation Tracking Control for Disturbed High-Order Multi-Agent Systems

In this paper, an event-triggered optimal tracking control of discrete-time multi-agent systems is addressed by using reinforcement learning. In contrast to traditional reinforcement learning-based methods for optimal coordination and control of multi-agent systems with a time-triggered control mechanism, an event-triggered mechanism is proposed to update the controller only when the designed events are triggered, which reduces the computational burden and transmission load. The stability analysis of the closed-loop multi-agent systems with event-triggered controller is described. Further, to implement the proposed scheme, an actor-critic neural network learning structure is proposed to approximate performance indices and to on-line learn the event-triggered optimal control. During the training process, event-triggered weight tuning law has been designed, wherein the weight parameters of the actor neural networks are adjusted only during triggering instances compared with traditional methods with fixed updating periods. Further, a convergence analysis of the actor-critic neural network is provided via Lyapunov method. Finally, two simulation examples show the effectiveness and performance of the obtained event-triggered reinforcement learning controller. 

Distributed Optimal Tracking Control of Discrete-Time Multiagent Systems via Event-Triggered Reinforcement Learning

Active resilient control for two‐dimensional systems under denial‐of‐service attacks

In this paper, the decentralized optimal control problem is addressed for a class of large-scale systems subject to injection attacks. All subsystem matrices are considered to be unavailable to the designer. A model-free decentralized sliding mode control (SMC) scheme for each subsystem is designed via just utilizing its own state information and the known bounds of the interconnections and the injection attacks. Moreover, the adaptive dynamic programming (ADP) approach is incorporated to deal with the infinite horizon optimal control problem for the sliding mode dynamics, which is equivalent to the solution of a set of parallel algebraic Riccati equations. Furthermore, a novel parallel policy iteration algorithm is developed to implement the proposed decentralized SMC scheme without using all subsystems dynamics matrices. Specifically, it is shown that during the whole policy iteration steps, the reachability of each sliding variable and the stability of each sliding mode dynamics are guaranteed simultaneously by the online updating decentralized SMC scheme. Finally, the applicability of the proposed novel ADP-based decentralized SMC strategy is illustrated by a two-machine power system subject to three different injection attacks.

Long-Yang Huang

Papers

ADP-Based Security Decentralized Sliding Mode Control for Partially Unknown Large-Scale Systems Under Injection Attacks

Partial model-free sliding mode control design for a class of disturbed systems via computational learning algorithm