This paper presents a comprehensive literature review on applications of deep reinforcement learning (DRL) in communications and networking. Modern networks, e.g., Internet of Things (IoT) and unmanned aerial vehicle (UAV) networks, become more decentralized and autonomous. In such networks, network entities need to make decisions locally to maximize the network performance under uncertainty of network environment. Reinforcement learning has been efficiently used to enable the network entities to obtain the optimal policy including, e.g., decisions or actions, given their states when the state and action spaces are small. However, in complex and large-scale networks, the state and action spaces are usually large, and the reinforcement learning may not be able to find the optimal policy in reasonable time. Therefore, DRL, a combination of reinforcement learning with deep learning, has been developed to overcome the shortcomings. In this survey, we first give a tutorial of DRL from fundamental concepts to advanced models. Then, we review DRL approaches proposed to address emerging issues in communications and networking. The issues include dynamic network access, data rate control, wireless caching, data offloading, network security, and connectivity preservation which are all important to next generation networks, such as 5G and beyond. Furthermore, we present applications of DRL for traffic routing, resource sharing, and data collection. Finally, we highlight important challenges, open issues, and future research directions of applying DRL.

Applications of Deep Reinforcement Learning in Communications and Networking: A Survey

This paper presents a comprehensive literature review on applications of deep reinforcement learning in communications and networking. Modern networks, e.g., Internet of Things (IoT) and Unmanned Aerial Vehicle (UAV) networks, become more decentralized and autonomous. In such networks, network entities need to make decisions locally to maximize the network performance under uncertainty of network environment. Reinforcement learning has been efficiently used to enable the network entities to obtain the optimal policy including, e.g., decisions or actions, given their states when the state and action spaces are small. However, in complex and large-scale networks, the state and action spaces are usually large, and the reinforcement learning may not be able to find the optimal policy in reasonable time. Therefore, deep reinforcement learning, a combination of reinforcement learning with deep learning, has been developed to overcome the shortcomings. In this survey, we first give a tutorial of deep reinforcement learning from fundamental concepts to advanced models. Then, we review deep reinforcement learning approaches proposed to address emerging issues in communications and networking. The issues include dynamic network access, data rate control, wireless caching, data offloading, network security, and connectivity preservation which are all important to next generation networks such as 5G and beyond. Furthermore, we present applications of deep reinforcement learning for traffic routing, resource sharing, and data collection. Finally, we highlight important challenges, open issues, and future research directions of applying deep reinforcement learning.

This article systematically studies consensus of linear multiagent systems (MASs) on directed graphs through adaptive event-triggered control. It presents innovative adaptive event-triggered state-feedback protocols with novel composite event-triggering conditions. Two specific designs in terms of different event-triggering conditions and laws of adaption are first discussed for linear MASs on strongly connected directed graphs, which are then extended to general directed graphs that contain a spanning tree. Moreover, another adaptive event-triggered protocol is proposed for solving leader–follower consensus that tracks a leader of a bounded control input. The protocols inherit the merits of both adaptive control and event-triggered control: the protocols can be implemented in a fully distributed way, since the Laplacian is avoided in design, and each agent only needs to know the relative information between neighbors at discrete instants determined by event-triggering conditions. Compared with the existing related results, the proposed protocols are applicable for linear MASs on general directed graphs, and moreover, the time-dependent term in the event-triggering conditions is allowed to be a class of positive $L_1$ functions. Two numerical examples clearly verify the effectiveness of the proposed protocols.

Adaptive Event-Triggered Consensus of Multiagent Systems on Directed Graphs

This paper proposes a fully distributed control protocol that achieves time-varying group formation tracking for linear multiagent systems connected via a directed graph. The group formation tracking often leads to sub-formations especially when the leaders are placed far apart or they have separate control inputs. In the proposed approach, the followers are distributed into several subgroups and each subgroup attains the predefined subformation along with encompassing the leaders. Each subgroup can be assigned multiple leaders, contrary to the single-leader case considered in most existing literature, which makes the current problem nontrivial. When multiple leaders exist in a subgroup, the subformation attained by that subgroup keeps tracking a convex combination of the states of the leaders. A distributed adaptive control protocol has been introduced in this paper which uses only relative state information and, thus, avoids direct computation of the graph Laplacian matrix. Due to the virtue of this, the proposed scheme remains effective even when some of the agents get disconnected from the network due to sudden communication failure. An algorithm is provided to outline the steps to design the control law to attain time-varying group formation tracking with multiple leaders. Toward the end, a case study on multitarget surveillance operation is taken up to show an important application of the proposed adaptive control technique.

/pdf/distributed-adaptive-time-varying-group-formation-tracking-25j6l5l2wz.pdf

Distributed Adaptive Time-Varying Group Formation Tracking for Multiagent Systems With Multiple Leaders on Directed Graphs

It has been witnessed that swarm systems are superior to individual agents in performing complicated tasks. In recent years, new results in some branches of control for swarm systems have developed and investigated with respect to various objectives and scenarios. This survey is to take a glimpse into some newly developed control techniques for swarm systems, especially those presented after 2013. The covered topics include some up-to-date progress in the areas of consensus, formation, flocking, containment, optimal coverage/mission planning, and sensor networks. Contributions and connections of the mentioned references are discussed briefly. Based on the new results in control of swarm systems, some possible new future research topics are suggested.

A survey on recent progress in control of swarm systems

The preservation of connectivity in mobile robot networks is critical to the success of most existing algorithms designed to achieve various goals. The most basic method to preserve connectivity is to have each agent preserve its set of neighbors for all time. More advanced methods preserve a (minimum) spanning tree in the network. Other methods are based on increasing the algebraic graph connectivity, which is given by the second smallest eigenvalue $\lambda_{2}({\cal L})$ of the graph Laplacian ${\cal L} $ that represents the network. These methods typically result in a monotonic increase in connectivity until the network is completely connected. In previous work by the authors, a continuous feedback control method had been proposed which allows the connectivity to decrease, that is, edges in the network may be broken. This method requires agents to have knowledge of the entire network. In this paper, we modify the controller to use only local information. The connectivity controller is based on maximization of $\lambda_{2}({\cal L} ) $ and artificial potential functions and can be used in conjunction with artificial potential-based formation controllers. The controllers are extended for implementation on nonholonomic-wheeled mobile robots, and the performance is demonstrated in an experiment on a team of wheeled mobile robots.

Collision-Free Formation Control with Decentralized Connectivity Preservation for Nonholonomic-Wheeled Mobile Robots

The preservation of connectivity in mobile robot networks is critical to the success of existing algorithms designed to achieve various goals. The available connectivity control algorithms mainly work through preservation of existing edges in the network. A link may be deleted if distributed decision making determines that the edge is not a cut-bridge. A controller is presented which allows edges to be broken in a continuous manner without higher-level decision making. The controller is based on maximization of the second smallest eigenvalue of the graph Laplacian. The controllers are designed for holonomic robots, and are extended for implementation on non-holonomic wheeled mobile robots. Finally, the performance of the extended controllers are demonstrated experimentally.

http://vigir.missouri.edu/~gdesouza/Research/Conference_CDs/IEEE_IROS_2013/media/files/1050.pdf

Hazen Eckert

Papers

Collision-Free Formation Control with Decentralized Connectivity Preservation for Nonholonomic-Wheeled Mobile Robots

Connectivity preserving formation control with collision avoidance for nonholonomic wheeled mobile robots