This paper presents a comprehensive literature review on applications of deep reinforcement learning (DRL) in communications and networking. Modern networks, e.g., Internet of Things (IoT) and unmanned aerial vehicle (UAV) networks, become more decentralized and autonomous. In such networks, network entities need to make decisions locally to maximize the network performance under uncertainty of network environment. Reinforcement learning has been efficiently used to enable the network entities to obtain the optimal policy including, e.g., decisions or actions, given their states when the state and action spaces are small. However, in complex and large-scale networks, the state and action spaces are usually large, and the reinforcement learning may not be able to find the optimal policy in reasonable time. Therefore, DRL, a combination of reinforcement learning with deep learning, has been developed to overcome the shortcomings. In this survey, we first give a tutorial of DRL from fundamental concepts to advanced models. Then, we review DRL approaches proposed to address emerging issues in communications and networking. The issues include dynamic network access, data rate control, wireless caching, data offloading, network security, and connectivity preservation which are all important to next generation networks, such as 5G and beyond. Furthermore, we present applications of DRL for traffic routing, resource sharing, and data collection. Finally, we highlight important challenges, open issues, and future research directions of applying DRL.

Applications of Deep Reinforcement Learning in Communications and Networking: A Survey

Along with the rapid developments in communication technologies and the surge in the use of mobile devices, a brand-new computation paradigm, Edge Computing, is surging in popularity. Meanwhile, Artificial Intelligence (AI) applications are thriving with the breakthroughs in deep learning and the many improvements in hardware architectures. Billions of data bytes, generated at the network edge, put massive demands on data processing and structural optimization. Thus, there exists a strong demand to integrate Edge Computing and AI, which gives birth to Edge Intelligence. In this paper, we divide Edge Intelligence into AI for edge (Intelligence-enabled Edge Computing) and AI on edge (Artificial Intelligence on Edge). The former focuses on providing more optimal solutions to key problems in Edge Computing with the help of popular and effective AI technologies while the latter studies how to carry out the entire process of building AI models, i.e., model training and inference, on the edge. This paper provides insights into this new inter-disciplinary field from a broader perspective. It discusses the core concepts and the research road-map, which should provide the necessary background for potential future research initiatives in Edge Intelligence.

Edge Intelligence: The Confluence of Edge Computing and Artificial Intelligence

Along with the rapid developments in communication technologies and the surge in the use of mobile devices, a brand-new computation paradigm, edge computing, is surging in popularity. Meanwhile, the artificial intelligence (AI) applications are thriving with the breakthroughs in deep learning and the many improvements in hardware architectures. Billions of data bytes, generated at the network edge, put massive demands on data processing and structural optimization. Thus, there exists a strong demand to integrate edge computing and AI, which gives birth to edge intelligence. In this article, we divide edge intelligence into AI for edge (intelligence-enabled edge computing) and AI on edge (artificial intelligence on edge). The former focuses on providing more optimal solutions to key problems in edge computing with the help of popular and effective AI technologies while the latter studies how to carry out the entire process of building AI models, i.e., model training and inference, on the edge. This article provides insights into this new interdisciplinary field from a broader perspective. It discusses the core concepts and the research roadmap, which should provide the necessary background for potential future research initiatives in edge intelligence.

This paper presents a comprehensive literature review on applications of deep reinforcement learning in communications and networking. Modern networks, e.g., Internet of Things (IoT) and Unmanned Aerial Vehicle (UAV) networks, become more decentralized and autonomous. In such networks, network entities need to make decisions locally to maximize the network performance under uncertainty of network environment. Reinforcement learning has been efficiently used to enable the network entities to obtain the optimal policy including, e.g., decisions or actions, given their states when the state and action spaces are small. However, in complex and large-scale networks, the state and action spaces are usually large, and the reinforcement learning may not be able to find the optimal policy in reasonable time. Therefore, deep reinforcement learning, a combination of reinforcement learning with deep learning, has been developed to overcome the shortcomings. In this survey, we first give a tutorial of deep reinforcement learning from fundamental concepts to advanced models. Then, we review deep reinforcement learning approaches proposed to address emerging issues in communications and networking. The issues include dynamic network access, data rate control, wireless caching, data offloading, network security, and connectivity preservation which are all important to next generation networks such as 5G and beyond. Furthermore, we present applications of deep reinforcement learning for traffic routing, resource sharing, and data collection. Finally, we highlight important challenges, open issues, and future research directions of applying deep reinforcement learning.

The Internet of Things (IoT) is penetrating many facets of our daily life with the proliferation of intelligent services and applications empowered by artificial intelligence (AI). Traditionally, AI techniques require centralized data collection and processing that may not be feasible in realistic application scenarios due to the high scalability of modern IoT networks and growing data privacy concerns. Federated Learning (FL) has emerged as a distributed collaborative AI approach that can enable many intelligent IoT applications, by allowing for AI training at distributed IoT devices without the need for data sharing. In this article, we provide a comprehensive survey of the emerging applications of FL in IoT networks, beginning from an introduction to the recent advances in FL and IoT to a discussion of their integration. Particularly, we explore and analyze the potential of FL for enabling a wide range of IoT services, including IoT data sharing, data offloading and caching, attack detection, localization, mobile crowdsensing, and IoT privacy and security. We then provide an extensive survey of the use of FL in various key IoT applications such as smart healthcare, smart transportation, Unmanned Aerial Vehicles (UAVs), smart cities, and smart industry. The important lessons learned from this review of the FL-IoT services and applications are also highlighted. We complete this survey by highlighting the current challenges and possible directions for future research in this booming area.

/pdf/federated-learning-for-internet-of-things-a-comprehensive-3ttswrbjdg.pdf

Federated Learning for Internet of Things: A Comprehensive Survey

This paper studies the impact of connection admission control (CAC) on the congestion management practices and the revenue of a monopoly access point (AP). The AP provides congestion-indication signals that suggest users to choose their access probabilities in response to network loading conditions. A Stackelberg leader-follower game is then formulated to analyze the interaction between the AP and the users. In particular, the AP first estimates the probable utility degradation of existing users consequent upon the admission of an incoming user. Second, the AP decides whether the connection of the incoming user should be admitted or not. The proposed CAC policy is completely distributed and can be implemented by individual APs using only local information. Simulation results show that the proposed algorithm achieves higher saturation throughput as well as greater revenue gain when compared with an existing algorithm.

Distributed connection admission control integrated with pricing for QoS provisioning and revenue maximization in wireless random access networks

Video caching is one of the most important research issues in Content-Centric Network (CCN) and greatly affects its overall performance. The computational complexity of state-of-the-art optimal caching schemes is high, due to the arbitrary network topologies. In this paper, the popularity-aware video caching in topology-known CCN is studied. The complex arbitrary network typology is mapped into a virtual cascade network topology and a caching scheme is designed in accordance with the transformed virtual network rather than the original network. This scheme is proved optimal, and is with polynomial computational complexity. Simulations are conducted and the results show that the proposed scheme outperforms the existing schemes.

Topology Mapping for Popularity-Aware Video Caching in Content-Centric Network

The encode-and-forward mechanism of network coding (NC) system, not only could provide increased network throughput, but also might get seriously vulnerable to pollution attacks. It has been an interesting and challenging topic how to design a secure, efficient and publicly verifiable homomorphic NC scheme. The existing cryptography-based NC schemes are grouped in either public key cryptosystem (PKC), or symmetric key cryptosystem (SKC). NC schemes in PKC naturally have public verifiability, but imply much more computation cost and longer operation delay. NC schemes in SKC have cheaper computations cost, but are dilemma about how to share the secret key to those intermediate nodes who might be malicious. Therefore, in this paper, we provide a new NC scheme that based on null-space HMAC with hierarchically sharing keys. The inner sharing keys are for the destination nodes to verify the integrity of the messages; the outer sharing keys are for the intermediate nodes to verify the integrity of the received packets. Our scheme shows a way how to balance the computation efficiency and the public verifiability for the NC system with a SKC scheme.

A publicly verifiable network coding scheme with null-space HMAC

/pdf/progress-in-data-acquisition-of-wearable-sensors-3pw9x9xp.pdf

Progress in Data Acquisition of Wearable Sensors

In this paper, a utility-based load distribution approach is introduced for relieving congestion at certain popular spaces within the network. Congestion-indication signals are provided for suggesting users to select APs in response to network load conditions. The heavier traffic load an AP has, the lower utility each associated user gains. Hence, the users incentively adapt themselves to associate to less congested APs. The effectiveness of the algorithm on improving the degree of load balance is evaluated using simulations. Simulation results show that the proposed algorithm achieves greater balance and higher resource utilization when compared with the best existing algorithms.

Cheng Zhang

Papers

Distributed connection admission control integrated with pricing for QoS provisioning and revenue maximization in wireless random access networks

Topology Mapping for Popularity-Aware Video Caching in Content-Centric Network

A publicly verifiable network coding scheme with null-space HMAC

Progress in Data Acquisition of Wearable Sensors

Utility-based load distribution for QoS provisioning and utility maximization in wireless random access networks