This paper presents a comprehensive literature review on applications of deep reinforcement learning (DRL) in communications and networking. Modern networks, e.g., Internet of Things (IoT) and unmanned aerial vehicle (UAV) networks, become more decentralized and autonomous. In such networks, network entities need to make decisions locally to maximize the network performance under uncertainty of network environment. Reinforcement learning has been efficiently used to enable the network entities to obtain the optimal policy including, e.g., decisions or actions, given their states when the state and action spaces are small. However, in complex and large-scale networks, the state and action spaces are usually large, and the reinforcement learning may not be able to find the optimal policy in reasonable time. Therefore, DRL, a combination of reinforcement learning with deep learning, has been developed to overcome the shortcomings. In this survey, we first give a tutorial of DRL from fundamental concepts to advanced models. Then, we review DRL approaches proposed to address emerging issues in communications and networking. The issues include dynamic network access, data rate control, wireless caching, data offloading, network security, and connectivity preservation which are all important to next generation networks, such as 5G and beyond. Furthermore, we present applications of DRL for traffic routing, resource sharing, and data collection. Finally, we highlight important challenges, open issues, and future research directions of applying DRL.

Applications of Deep Reinforcement Learning in Communications and Networking: A Survey

Modern chip multiprocessor (CMP) systems employ multiple memory controllers to control access to main memory. The scheduling algorithm employed by these memory controllers has a significant effect on system throughput, so choosing an efficient scheduling algorithm is important. The scheduling algorithm also needs to be scalable — as the number of cores increases, the number of memory controllers shared by the cores should also increase to provide sufficient bandwidth to feed the cores. Unfortunately, previous memory scheduling algorithms are inefficient with respect to system throughput and/or are designed for a single memory controller and do not scale well to multiple memory controllers, requiring significant finegrained coordination among controllers. This paper proposes ATLAS (Adaptive per-Thread Least-Attained-Service memory scheduling), a fundamentally new memory scheduling technique that improves system throughput without requiring significant coordination among memory controllers. The key idea is to periodically order threads based on the service they have attained from the memory controllers so far, and prioritize those threads that have attained the least service over others in each period. The idea of favoring threads with least-attained-service is borrowed from the queueing theory literature, where, in the context of a single-server queue it is known that least-attained-service optimally schedules jobs, assuming a Pareto (or any decreasing hazard rate) workload distribution. After verifying that our workloads have this characteristic, we show that our implementation of least-attained-service thread prioritization reduces the time the cores spend stalling and significantly improves system throughput. Furthermore, since the periods over which we accumulate the attained service are long, the controllers coordinate very infrequently to form the ordering of threads, thereby making ATLAS scalable to many controllers. We evaluate ATLAS on a wide variety of multiprogrammed SPEC 2006 workloads and systems with 4–32 cores and 1–16 memory controllers, and compare its performance to five previously proposed scheduling algorithms. Averaged over 32 workloads on a 24-core system with 4 controllers, ATLAS improves instruction throughput by 10.8%, and system throughput by 8.4%, compared to PAR-BS, the best previous CMP memory scheduling algorithm. ATLAS's performance benefit increases as the number of cores increases.

/pdf/atlas-a-scalable-and-high-performance-scheduling-algorithm-586cczzqud.pdf

ATLAS: A scalable and high-performance scheduling algorithm for multiple memory controllers

This paper presents a comprehensive literature review on applications of deep reinforcement learning in communications and networking. Modern networks, e.g., Internet of Things (IoT) and Unmanned Aerial Vehicle (UAV) networks, become more decentralized and autonomous. In such networks, network entities need to make decisions locally to maximize the network performance under uncertainty of network environment. Reinforcement learning has been efficiently used to enable the network entities to obtain the optimal policy including, e.g., decisions or actions, given their states when the state and action spaces are small. However, in complex and large-scale networks, the state and action spaces are usually large, and the reinforcement learning may not be able to find the optimal policy in reasonable time. Therefore, deep reinforcement learning, a combination of reinforcement learning with deep learning, has been developed to overcome the shortcomings. In this survey, we first give a tutorial of deep reinforcement learning from fundamental concepts to advanced models. Then, we review deep reinforcement learning approaches proposed to address emerging issues in communications and networking. The issues include dynamic network access, data rate control, wireless caching, data offloading, network security, and connectivity preservation which are all important to next generation networks such as 5G and beyond. Furthermore, we present applications of deep reinforcement learning for traffic routing, resource sharing, and data collection. Finally, we highlight important challenges, open issues, and future research directions of applying deep reinforcement learning.

One method for wirelessly determining a position of a mobile station includes measuring a round trip time (RTT) to a plurality of wireless access points, estimating a first distance to each wireless access point based upon the round trip time delay and an initial processing time associated with each wireless access point, estimating a second distance to each wireless access point based upon supplemental information, combining the first and second distance estimates to each wireless access point, and calculating the position based upon the combined distance estimates. Another method includes measuring a distance to each wireless access point based upon a wireless signal model, calculating a position of the mobile station based upon the measured distance, determining a computed distance to each wireless access point based upon the calculated position of the mobile station, updating the wireless signal model, and determining whether the wireless signal model has converged.

Wireless position determination using adjusted round trip time measurements

We present a novel and complete tool-set for evaluating the delivery quality of MPEG video transmissions in simulations of a network environment. This tool-set is based on the EvalVid framework. We extend the connecting interfaces of EvalVid to replace its simple error simulation model by a more general network simulator like NS2. With this combination, researchers and practitioners in general can analyze through simulation the performance of real video streams, i.e. taking into account the video semantics, under a large range of network scenarios. To demonstrate the usefulness of our new tool-set, we point out that it enables the investigation of the relationship between two popular objective metrics for Quality of Service (QoS) assessment of video quality delivery: the PSNR (Peak Signal to Noise Ratio) and the fraction of decodable frames. The results show that the fraction of decodable frames reflects well the behavior of the PSNR metric, while being less time-consuming. Therefore, the fraction of decodable frames can be an alternative metric to objectively assess through simulations the delivery quality of transmission in a network of publicly available video trace files.

An Evaluation Framework for More Realistic Simulations of MPEG Video Transmission

With EvalVid we present a complete framework and tool-set for evaluation of the quality of video transmitted over a real or simulated communication network. Besides measuring QoS parameters of the underlying network, like loss rates, delays, and jitter, we support also a subjective video quality evaluation of the received video based on the frame-by-frame PSNR calculation. The tool-set has a modular construction, making it possible to exchange both the network and the codec. We present here its application for MPEG-4 as example. EvalVid is targeted for researchers who want to evaluate their network designs or setups in terms of user perceived video quality. The tool-set is publicly available [11].

/pdf/evalvid-a-framework-for-video-transmission-and-quality-3lx7icxz5y.pdf

EvalVid – A Framework for Video Transmission and Quality Evaluation

The road vehicle communication system is an infrastructure network to be deployed along the roads for future intelligent transportation systems. Recently, in order to provide high bandwidth data traffic a radio over fiber based network at millimeter-wave bands has been proposed. In this network a control station is connected to functionally simple base stations via optical fibers. Due to millimeter-wave properties the cell size is very small; therefore, the system is characterized by small cell and high mobility. As a result, it is obvious that mobility management becomes very significant. In this paper we propose a medium access control scheme featuring fast handover and dynamic bandwidth allocation using the ability of centralized control of radio over fiber networks.

A radio over fiber network architecture for road vehicle communication systems

Systems and methods for distance measurement in wireless networks are disclosed. A method for measuring distance between nodes or devices in a wireless network comprises estimating a distance between network devices based on a first distance measurement method, estimating the distance based on a second distance measurement method, and processing the first and second distance estimates to determine convergence and to generate an enhanced distance measurement. Additional distance estimates may be combined to further improve accuracy. Convergence information may be provided indicating whether two or more distance estimates converge. In some implementations, the first distance estimate may be generated by a transmission time estimation method, the second distance estimate may be generated by a received signal strength distance estimate, the two estimates may be combined by averaging to generate an enhanced distance estimate, and the difference between the estimates may be compared to a threshold to determine whether the estimates have sufficiently converged to within a desired convergence range.

Systems and methods for distance measurement in wireless networks

If highly compressed multimedia streams are transported over packet networks, losses of individual packets can impair the perceptual quality of the received stream in different degrees, depending on the content and context of the lost packet. In this paper, we investigate the impact of individual packet loss on the perceptual speech quality in Voice-over-IP systems using three popular coding types and receiver-side loss concealment algorithms. We set up a testing environment to measure the impairment of individual packet losses and define an appropriate quality metric. We evaluate published algorithms on packet loss quality prediction (DTX, Source-Driven Packet Marking and SPB-DiffMark) and identify their strengths and weaknesses. The quality of a VoIP telephone call can be enhanced significant, if a precise packet-loss quality model decides for each VoIP packet, how it should be forwarded throughout the network.

/pdf/on-the-importance-of-a-voip-packet-29vw9c7ws2.pdf

On the Importance of a VoIP Packet

This paper proposes a framework for interference mitigation in multi-BSS infrastructure 802.11 WLANs. Our interference mitigation approach is based on Access Point (AP) Coordination. With this approach, interfering BSSs negotiate and switch from the 802.11 CSMA/CA to a time slotted mechanism if users' QoS is observed to be degraded, diagnoses conclude that the cause is high interference, and the switch to the time slotted modus is expected to be useful and feasible. The proposed algorithms within the framework are driven by measurements. We utilize the wireless bandwidth and improve the fairness among WLAN users. We present results of detailed simulation experiments as well as real implementation.

/pdf/a-framework-for-interference-mitigation-in-multi-bss-802-11-5fxvuq2rvi.pdf

Berthold Rathke

Papers

EvalVid – A Framework for Video Transmission and Quality Evaluation

A radio over fiber network architecture for road vehicle communication systems

Systems and methods for distance measurement in wireless networks

On the Importance of a VoIP Packet

A framework for interference mitigation in multi-BSS 802.11 wireless LANs