This paper presents a comprehensive literature review on applications of deep reinforcement learning (DRL) in communications and networking. Modern networks, e.g., Internet of Things (IoT) and unmanned aerial vehicle (UAV) networks, become more decentralized and autonomous. In such networks, network entities need to make decisions locally to maximize the network performance under uncertainty of network environment. Reinforcement learning has been efficiently used to enable the network entities to obtain the optimal policy including, e.g., decisions or actions, given their states when the state and action spaces are small. However, in complex and large-scale networks, the state and action spaces are usually large, and the reinforcement learning may not be able to find the optimal policy in reasonable time. Therefore, DRL, a combination of reinforcement learning with deep learning, has been developed to overcome the shortcomings. In this survey, we first give a tutorial of DRL from fundamental concepts to advanced models. Then, we review DRL approaches proposed to address emerging issues in communications and networking. The issues include dynamic network access, data rate control, wireless caching, data offloading, network security, and connectivity preservation which are all important to next generation networks, such as 5G and beyond. Furthermore, we present applications of DRL for traffic routing, resource sharing, and data collection. Finally, we highlight important challenges, open issues, and future research directions of applying DRL.

Applications of Deep Reinforcement Learning in Communications and Networking: A Survey

Dimensions of Reinforcement Learning

This paper presents a comprehensive literature review on applications of deep reinforcement learning in communications and networking. Modern networks, e.g., Internet of Things (IoT) and Unmanned Aerial Vehicle (UAV) networks, become more decentralized and autonomous. In such networks, network entities need to make decisions locally to maximize the network performance under uncertainty of network environment. Reinforcement learning has been efficiently used to enable the network entities to obtain the optimal policy including, e.g., decisions or actions, given their states when the state and action spaces are small. However, in complex and large-scale networks, the state and action spaces are usually large, and the reinforcement learning may not be able to find the optimal policy in reasonable time. Therefore, deep reinforcement learning, a combination of reinforcement learning with deep learning, has been developed to overcome the shortcomings. In this survey, we first give a tutorial of deep reinforcement learning from fundamental concepts to advanced models. Then, we review deep reinforcement learning approaches proposed to address emerging issues in communications and networking. The issues include dynamic network access, data rate control, wireless caching, data offloading, network security, and connectivity preservation which are all important to next generation networks such as 5G and beyond. Furthermore, we present applications of deep reinforcement learning for traffic routing, resource sharing, and data collection. Finally, we highlight important challenges, open issues, and future research directions of applying deep reinforcement learning.

In modern wireless communication systems, wireless channel modeling has always been a fundamental task in system design and performance optimization. Traditional channel modeling methods, such as ray-tracing and geometry- based stochastic channel models, require in-depth domain-specific knowledge and technical expertise in radio signal propagations across electromagnetic fields. To avoid these difficulties and complexities, a novel generative adversarial network (GAN) framework is proposed for the first time to address the problem of autonomous wireless channel modeling without complex theoretical analysis or data processing. Specifically, the GAN is trained by raw measurement data to reach the Nash equilibrium of a MinMax game between a channel data generator and a channel data discriminator. Once this process converges, the resulting channel data generator is extracted as the target channel model for a specific application scenario. To demonstrate, the distribution of a typical additive white Gaussian noise channel is successfully approximated by using the proposed GAN-based channel modeling framework, thus verifying its good performance and effectiveness.

Generative-Adversarial-Network-Based Wireless Channel Modeling: Challenges and Opportunities

In this paper, we propose a novel framework of low-cost link adaptation for spatial modulation multiple-input multiple-output (SM-MIMO) systems-based upon the machine learning paradigm. Specifically, we first convert the problems of transmit antenna selection (TAS) and power allocation (PA) in SM-MIMO to ones-based upon data-driven prediction rather than conventional optimization-driven decisions. Then, supervised-learning classifiers (SLC), such as the K -nearest neighbors (KNN) and support vector machine (SVM) algorithms, are developed to obtain their statistically-consistent solutions. Moreover, for further comparison we integrate deep neural networks (DNN) with these adaptive SM-MIMO schemes, and propose a novel DNN-based multi-label classifier for TAS and PA parameter evaluation. Furthermore, we investigate the design of feature vectors for the SLC and DNN approaches and propose a novel feature vector generator to match the specific transmission mode of SM. As a further advance, our proposed approaches are extended to other adaptive index modulation (IM) schemes, e.g., adaptive modulation (AM) aided orthogonal frequency division multiplexing with IM (OFDM-IM). Our simulation results show that the SLC and DNN-based adaptive SM-MIMO systems outperform many conventional optimization-driven designs and are capable of achieving a near-optimal performance with a significantly lower complexity.

/pdf/adaptive-spatial-modulation-mimo-based-on-machine-learning-m279ty0vhd.pdf

Adaptive Spatial Modulation MIMO Based on Machine Learning

System capacity and service coverage are the most critical performance metrics in cellular wireless communication networks. Usually, system capacity enhancements are at the expense of service coverage degradations, and vice versa. This capacity-coverage tradeoff and the associated joint optimization problem becomes very challenging in massive multiple-input multiple-output (MIMO) wireless systems, due to a large amount of antenna tilt values to be configured and very sophisticated inter-cell interference conditions, under massive antenna scenarios. This paper proposes a novel approach, namely group alignment of user signal strength (GAUSS), to efficiently support the user scheduling for the massive MIMO system, and thus serve as an effective parameter for the coverage and capacity optimization (CCO) problem. Together with a unified threshold of Quality of Service, i.e. the minimum signal-to-interference-plus-noise ratio ( $\textit {SINR}_{min}$ ) for user satisfaction, GAUSS can effectively control the variance of signal strengths of multiple users in the neighborhood. Moreover, an intelligent and efficient deep-learning enabled coverage and capacity optimization (DECCO) algorithm is proposed and evaluated, which adopts a pre-trained deep policy gradient-based neural network to dynamically derive GAUSS and $\textit {SINR}_{min}$ during CCO. Furthermore, an inter-cell interference coordination (ICIC) is proposed to enhance the CCO performance. Analytical and simulation results show that the proposed DECCO algorithm can effectively achieve a much better performance balance between system capacity and service coverage than traditional fixed optimization (FO) and proportional fair optimization (PFO) algorithms. Specifically, DECCO significantly increases the overall spectrum efficiency by 24% and 40%, respectively, than FO and PFO in a typical massive MIMO system.

/pdf/decco-deep-learning-enabled-coverage-and-capacity-wj6r0u9vp4.pdf

DECCO: Deep-Learning Enabled Coverage and Capacity Optimization for Massive MIMO Systems

The integration of Internet of things (IoT) and intelligent transportation system (ITS) is expected to improve the traffic efficiency and enhance the driving experience. However, due to the dynamic traffic environment and various types of vehicles, it is a challenge to perform vehicle classification and speed estimation with a single magnetic sensor. In this paper, based on a single low-cost magnetic sensor, a scheme is proposed to achieve vehicle classification and speed interval estimation by designing a two-dimensional convolutional neural network (CNN). Specifically, we extract the magnetic field data of each vehicle and then convert the collected data into a two-dimensional grayscale image. In this way, the images of vehicle signals with different types and driving speeds can be used as the input data to train the designed CNN model. With the designed CNN model, we classify the vehicles into 7 types and estimate the speed interval of each vehicle, where the speeds in the range of 10km/h-70km/h are divided into 6 intervals of size 10km/h. The performance of the proposed vehicle classification and speed estimation scheme is evaluated by experiments, where the experimental results show that the accuracy of vehicle classification and the accuracy of speed interval estimation are 97.83% and 96.85%, respectively.

/pdf/vehicle-classification-and-speed-estimation-based-on-a-jeooav2n1l.pdf

Vehicle Classification and Speed Estimation Based on a Single Magnetic Sensor

As the number of antennas goes large in the massive multiple-input multiple-output (MIMO) scenario, the large-scale fading plays an increasingly important role in the system performance. In this paper, we investigate the uplink sum rate of a massive distributed MIMO system over composite fading channels portrayed by the Rayleigh–Lognormal (RLN) model, where both zero forcing (ZF) and ZF decision-feedback receivers are considered. By exploring a novel and generalized method based on the integral and limit theories, the sum rate is expressed as a series with regard to the negative power of the transmit antennas, which is followed by a brief discussion on the asymptotic behavior when the number of transmit antennas goes infinite. Numeral experiments testify our theoretical analysis and prove the effectiveness even when the accuracy parameter is small. The influence of the number and location of the radio ports is also discussed.

Rui Chen

Papers

DECCO: Deep-Learning Enabled Coverage and Capacity Optimization for Massive MIMO Systems

Vehicle Classification and Speed Estimation Based on a Single Magnetic Sensor

Uplink Sum Rate Analysis of Massive Distributed MIMO Systems Over Composite Fading Channels