This paper presents a comprehensive literature review on applications of deep reinforcement learning (DRL) in communications and networking. Modern networks, e.g., Internet of Things (IoT) and unmanned aerial vehicle (UAV) networks, become more decentralized and autonomous. In such networks, network entities need to make decisions locally to maximize the network performance under uncertainty of network environment. Reinforcement learning has been efficiently used to enable the network entities to obtain the optimal policy including, e.g., decisions or actions, given their states when the state and action spaces are small. However, in complex and large-scale networks, the state and action spaces are usually large, and the reinforcement learning may not be able to find the optimal policy in reasonable time. Therefore, DRL, a combination of reinforcement learning with deep learning, has been developed to overcome the shortcomings. In this survey, we first give a tutorial of DRL from fundamental concepts to advanced models. Then, we review DRL approaches proposed to address emerging issues in communications and networking. The issues include dynamic network access, data rate control, wireless caching, data offloading, network security, and connectivity preservation which are all important to next generation networks, such as 5G and beyond. Furthermore, we present applications of DRL for traffic routing, resource sharing, and data collection. Finally, we highlight important challenges, open issues, and future research directions of applying DRL.

Applications of Deep Reinforcement Learning in Communications and Networking: A Survey

The rapid uptake of mobile devices and the rising popularity of mobile applications and services pose unprecedented demands on mobile and wireless networking infrastructure. Upcoming 5G systems are evolving to support exploding mobile traffic volumes, real-time extraction of fine-grained analytics, and agile management of network resources, so as to maximize user experience. Fulfilling these tasks is challenging, as mobile environments are increasingly complex, heterogeneous, and evolving. One potential solution is to resort to advanced machine learning techniques, in order to help manage the rise in data volumes and algorithm-driven applications. The recent success of deep learning underpins new and powerful tools that tackle problems in this space. In this paper, we bridge the gap between deep learning and mobile and wireless networking research, by presenting a comprehensive survey of the crossovers between the two areas. We first briefly introduce essential background and state-of-the-art in deep learning techniques with potential applications to networking. We then discuss several techniques and platforms that facilitate the efficient deployment of deep learning onto mobile systems. Subsequently, we provide an encyclopedic review of mobile and wireless networking research based on deep learning, which we categorize by different domains. Drawing from our experience, we discuss how to tailor deep learning to mobile environments. We complete this survey by pinpointing current challenges and open future directions for research.

/pdf/deep-learning-in-mobile-and-wireless-networking-a-survey-ct8cy2kuqi.pdf

Deep Learning in Mobile and Wireless Networking: A Survey

Automatic modulation recognition (AMR) is an essential and challenging topic in the development of the cognitive radio (CR), and it is a cornerstone of CR adaptive modulation and demodulation capabilities to sense and learn environments and make corresponding adjustments. AMR is essentially a classification problem, and deep learning achieves outstanding performances in various classification tasks. So, this paper proposes a deep learning-based method, combined with two convolutional neural networks (CNNs) trained on different datasets, to achieve higher accuracy AMR. A CNN is trained on samples composed of in-phase and quadrature component signals, otherwise known as in-phase and quadrature samples, to distinguish modulation modes, that are relatively easy to identify. We adopt dropout instead of pooling operation to achieve higher recognition accuracy. A CNN based on constellation diagrams is also designed to recognize modulation modes that are difficult to distinguish in the former CNN, such as 16 quadratic-amplitude modulation (QAM) and 64 QAM, demonstrating the ability to classify QAM signals even in scenarios with a low signal-to-noise ratio.

Data-Driven Deep Learning for Automatic Modulation Recognition in Cognitive Radios

As of today, the fifth generation (5G) mobile communication system has been rolled out in many countries and the number of 5G subscribers already reaches a very large scale. It is time for academia and industry to shift their attention towards the next generation. At this crossroad, an overview of the current state of the art and a vision of future communications are definitely of interest. This article thus aims to provide a comprehensive survey to draw a picture of the sixth generation (6G) system in terms of drivers, use cases, usage scenarios, requirements, key performance indicators (KPIs), architecture, and enabling technologies. First, we attempt to answer the question of "Is there any need for 6G?" by shedding light on its key driving factors, in which we predict the explosive growth of mobile traffic until 2030, and envision potential use cases and usage scenarios. Second, the technical requirements of 6G are discussed and compared with those of 5G with respect to a set of KPIs in a quantitative manner. Third, the state-of-the-art 6G research efforts and activities from representative institutions and countries are summarized, and a tentative roadmap of definition, specification, standardization, and regulation is projected. Then, we identify a dozen of potential technologies and introduce their principles, advantages, challenges, and open research issues. Finally, the conclusions are drawn to paint a picture of "What 6G may look like?". This survey is intended to serve as an enlightening guideline to spur interests and further investigations for subsequent research and development of 6G communications systems.

The Road Towards 6G: A Comprehensive Survey

The innovations provided by sixth generation wireless communication (6G) as compared to fifth generation (5G) are considered in this article based on analysis of related works. With the aim of achieving diverse performance improvements for the various 6G requirements, five 6G core services are identified. Two centricities and eight key performance indices (KPIs) are detailed to describe these services, then enabling technologies to fulfill the KPIs are discussed. A 6G architecture is proposed as an integrated system of the enabling technologies and is then illustrated using four typical urban application scenarios. Potential challenges in the development of 6G technology are then discussed and possible solutions are proposed. Finally, opportunities for exploring 6G are analyzed in order to guide future research.

6G: Opening New Horizons for Integration of Comfort, Security, and Intelligence

The recent concept of massive multiple-input multiple-output (MIMO) can significantly improve the capacity of the communication network, and it has been regarded as a promising technology for the next-generation wireless communications. However, the fundamental challenge of existing massive MIMO systems is that high computational complexity and complicated spatial structures bring great difficulties to exploit the characteristics of the channel and sparsity of these multi-antennas systems. To address this problem, in this paper, we focus on channel estimation and direction-of-arrival (DOA) estimation, and a novel framework that integrates the massive MIMO into deep learning is proposed. To realize end-to-end performance, a deep neural network (DNN) is employed to conduct offline learning and online learning procedures, which is effective to learn the statistics of the wireless channel and the spatial structures in the angle domain. Concretely, the DNN is first trained by simulated data in different channel conditions with the aids of the offline learning, and then corresponding output data can be obtained based on current input data during online learning process. In order to realize super-resolution channel estimation and DOA estimation, two algorithms based on the deep learning are developed, in which the DOA can be estimated in the angle domain without additional complexity directly. Furthermore, simulation results corroborate that the proposed deep learning based scheme can achieve better performance in terms of the DOA estimation and the channel estimation compared with conventional methods, and the proposed scheme is well investigated by extensive simulation in various cases for testing its robustness.

Deep Learning for Super-Resolution Channel Estimation and DOA Estimation Based Massive MIMO System

Millimeter wave (mmWave) massive multiple-input multiple-output (MIMO) has been regarded to be an emerging solution for the next generation of communications, in which hybrid analog and digital precoding is an important method for reducing the hardware complexity and energy consumption associated with mixed signal components. However, the fundamental limitations of the existing hybrid precoding schemes is that they have high computational complexity and fail to fully exploit the spatial information. To overcome these limitations, this paper proposes, a deep-learning-enabled mmWave massive MIMO framework for effective hybrid precoding, in which each selection of the precoders for obtaining the optimized decoder is regarded as a mapping relation in the deep neural network (DNN). Specifically, the hybrid precoder is selected through training based on the DNN for optimizing precoding process of the mmWave massive MIMO. Additionally, we present extensive simulation results to validate the excellent performance of the proposed scheme. The results exhibit that the DNN-based approach is capable ofminimizing the bit error ratio (BER) and enhancing spectrum efficiency of the mmWave massive MIMO, which achieves better performance in hybrid precoding compared with conventional schemes while substantially reducing the required computational complexity.

Deep-Learning-based Millimeter-Wave Massive MIMO for Hybrid Precoding

Beamforming is considered as one of the most important techniques for designing advanced multiple-input and multiple-output (MIMO) systems. Among existing design criterions, sum rate maximization (SRM) under a total power constraint is a challenge due to its nonconvexity. Existing techniques for the SRM problem only obtain local optimal solutions but require huge amount of computation due to their complex matrix operations and iterations. Unlike these conventional methods, we propose a deep learning based fast beamforming design method without complex operations and iterations. Specifically, we first derive a heuristic solution structure of the downlink beamforming through the virtual equivalent uplink channel based on optimum MMSE receiver which separates the problem into power allocation and virtual uplink beamforming (VUB) design. Next, beamforming prediction network (BPNet) is designed to perform the joint optimization of power allocation and VUB design. Moreover, the BPNet is trained offline using two-step training strategy. Simulation results demonstrate that our proposed method is fast while obtains the comparable performance to the state-of-the-art method.

Fast Beamforming Design via Deep Learning

Millimeter wave (mmWave) massive multiple-input multiple-output (MIMO) has been regarded to be an emerging solution for the next generation of communications, in which hybrid analog and digital precoding is an important method for reducing the hardware complexity and energy consumption associated with mixed signal components. However, the fundamental limitations of the existing hybrid precoding schemes are that they have high-computational complexity and fail to fully exploit the spatial information. To overcome these limitations, this paper proposes a deep-learning-enabled mmWave massive MIMO framework for effective hybrid precoding, in which each selection of the precoders for obtaining the optimized decoder is regarded as a mapping relation in the deep neural network (DNN). Specifically, the hybrid precoder is selected through training based on the DNN for optimizing precoding process of the mmWave massive MIMO. Additionally, we present extensive simulation results to validate the excellent performance of the proposed scheme. The results exhibit that the DNN-based approach is capable of minimizing the bit error ratio and enhancing the spectrum efficiency of the mmWave massive MIMO, which achieves better performance in hybrid precoding compared with conventional schemes while substantially reducing the required computational complexity.

Jie Yang

Papers

Deep Learning for Super-Resolution Channel Estimation and DOA Estimation Based Massive MIMO System

Data-Driven Deep Learning for Automatic Modulation Recognition in Cognitive Radios

Deep-Learning-based Millimeter-Wave Massive MIMO for Hybrid Precoding

Fast Beamforming Design via Deep Learning

Deep-Learning-Based Millimeter-Wave Massive MIMO for Hybrid Precoding