This paper presents a comprehensive literature review on applications of deep reinforcement learning (DRL) in communications and networking. Modern networks, e.g., Internet of Things (IoT) and unmanned aerial vehicle (UAV) networks, become more decentralized and autonomous. In such networks, network entities need to make decisions locally to maximize the network performance under uncertainty of network environment. Reinforcement learning has been efficiently used to enable the network entities to obtain the optimal policy including, e.g., decisions or actions, given their states when the state and action spaces are small. However, in complex and large-scale networks, the state and action spaces are usually large, and the reinforcement learning may not be able to find the optimal policy in reasonable time. Therefore, DRL, a combination of reinforcement learning with deep learning, has been developed to overcome the shortcomings. In this survey, we first give a tutorial of DRL from fundamental concepts to advanced models. Then, we review DRL approaches proposed to address emerging issues in communications and networking. The issues include dynamic network access, data rate control, wireless caching, data offloading, network security, and connectivity preservation which are all important to next generation networks, such as 5G and beyond. Furthermore, we present applications of DRL for traffic routing, resource sharing, and data collection. Finally, we highlight important challenges, open issues, and future research directions of applying DRL.

Applications of Deep Reinforcement Learning in Communications and Networking: A Survey

Client-side video players employ adaptive bitrate (ABR) algorithms to optimize user quality of experience (QoE). Despite the abundance of recently proposed schemes, state-of-the-art ABR algorithms suffer from a key limitation: they use fixed control rules based on simplified or inaccurate models of the deployment environment. As a result, existing schemes inevitably fail to achieve optimal performance across a broad set of network conditions and QoE objectives.We propose Pensieve, a system that generates ABR algorithms using reinforcement learning (RL). Pensieve trains a neural network model that selects bitrates for future video chunks based on observations collected by client video players. Pensieve does not rely on pre-programmed models or assumptions about the environment. Instead, it learns to make ABR decisions solely through observations of the resulting performance of past decisions. As a result, Pensieve automatically learns ABR algorithms that adapt to a wide range of environments and QoE metrics. We compare Pensieve to state-of-the-art ABR algorithms using trace-driven and real world experiments spanning a wide variety of network conditions, QoE metrics, and video properties. In all considered scenarios, Pensieve outperforms the best state-of-the-art scheme, with improvements in average QoE of 12%--25%. Pensieve also generalizes well, outperforming existing schemes even on networks for which it was not explicitly trained.

/pdf/neural-adaptive-video-streaming-with-pensieve-3y10qvjqj1.pdf

Neural Adaptive Video Streaming with Pensieve

Machine Learning (ML) has been enjoying an unprecedented surge in applications that solve problems and enable automation in diverse domains. Primarily, this is due to the explosion in the availability of data, significant improvements in ML techniques, and advancement in computing capabilities. Undoubtedly, ML has been applied to various mundane and complex problems arising in network operation and management. There are various surveys on ML for specific areas in networking or for specific network technologies. This survey is original, since it jointly presents the application of diverse ML techniques in various key areas of networking across different network technologies. In this way, readers will benefit from a comprehensive discussion on the different learning paradigms and ML techniques applied to fundamental problems in networking, including traffic prediction, routing and classification, congestion control, resource and fault management, QoS and QoE management, and network security. Furthermore, this survey delineates the limitations, give insights, research challenges and future opportunities to advance ML in networking. Therefore, this is a timely contribution of the implications of ML for networking, that is pushing the barriers of autonomic network operation and management.

/pdf/a-comprehensive-survey-on-machine-learning-for-networking-3t5h3kbpe8.pdf

A comprehensive survey on machine learning for networking: evolution, applications and research opportunities

Modern video players employ complex algorithms to adapt the bitrate of the video that is shown to the user. Bitrate adaptation requires a tradeoff between reducing the probability that the video freezes and enhancing the quality of the video shown to the user. A bitrate that is too high leads to frequent video freezes (i.e., rebuffering), while a bitrate that is too low leads to poor video quality. Video providers segment the video into short chunks and encode each chunk at multiple bitrates. The video player adaptively chooses the bitrate of each chunk that is downloaded, possibly choosing different bitrates for successive chunks. While bitrate adaptation holds the key to a good quality of experience for the user, current video players use ad-hoc algorithms that are poorly understood. We formulate bitrate adaptation as a utility maximization problem and devise an online control algorithm called BOLA that uses Lyapunov optimization techniques to minimize rebuffering and maximize video quality. We prove that BOLA achieves a time-average utility that is within an additive term O(1/V) of the optimal value, for a control parameter V related to the video buffer size. Further, unlike prior work, our algorithm does not require any prediction of available network bandwidth. We empirically validate our algorithm in a simulated network environment using an extensive collection of network traces. We show that our algorithm achieves near-optimal utility and in many cases significantly higher utility than current state-of-the-art algorithms. Our work has immediate impact on real-world video players and BOLA is part of the reference player implementation for the evolving DASH standard for video transmission.

BOLA: Near-optimal bitrate adaptation for online videos

Dynamic Adaptation Streaming over HTTP (DASH) enhances the Quality of Experience (QoE) for users by automatically switching quality levels according to network conditions. Various adaptation schemes have been proposed to select the most suitable quality level during video playback. Adaptation schemes are currently based on the measured TCP throughput received by the video player. Although video buffer can mitigate throughput fluctuations, it does not take into account the effect of the transition of quality levels on the QoE.In this paper, we propose a QoE-aware DASH system (or QDASH) to improve the user-perceived quality of video watching. We integrate available bandwidth measurement into the video data probes with a measurement proxy architecture. We have found that our available bandwidth measurement method facilitates the selection of video quality levels. Moreover, we assess the QoE of the quality transitions by carrying out subjective experiments. Our results show that users prefer a gradual quality change between the best and worst quality levels, instead of an abrupt switching. Hence, we propose a QoE-aware quality adaptation algorithm for DASH based on our findings. Finally, we integrate both network measurement and the QoE-aware quality adaptation into a comprehensive DASH system.

/pdf/qdash-a-qoe-aware-dash-system-3bp4gt7ege.pdf

QDASH: a QoE-aware DASH system

In this dataset paper, we present and make available real-world measurements of the throughput that was achieved at the application layer when adaptive HTTP streaming was performed over 3G networks using mobile devices. For the streaming sessions, we used popular commute routes in and around Oslo (Norway) traveling with different types of public transportation (metro, tram, train, bus and ferry). We also have a few logs using a car. Each log provides a times-tamp, GPS coordinates and the measured number of bytes downloaded for approximately every second of the route. The dataset can be used in several ways, but the most obvious application is to emulate the same network bandwidth behavior (on specific geographical positions) for repeated experiments.

Commute path bandwidth traces from 3G networks: analysis and applications

A lot of people around the world commute using public transportation and would like to spend this time viewing streamed video content such as news or sports updates. However, mobile wireless networks typically suffer from severe bandwidth fluctuations, and the networks are often completely unresponsive for several seconds, sometimes minutes. Today, there are several ways of adapting the video bitrate and thus the video quality to such fluctuations, for example, using scalable video codecs or segmented adaptive HTTP streaming that switches between nonscalable video streams encoded in different bitrates. Still, for a better long-term video playout experience that avoids disruptions and frequent quality changes while using existing video adaptation technology, it is desirable to perform bandwidth prediction and planned quality adaptation.This article describes a video streaming system for receivers equipped with a GPS. A receiver's download rate is constantly monitored, and periodically reported back to a central database along with associated GPS positional data. Thus, based on the current location, a streaming device can use a GPS-based bandwidth-lookup service in order to better predict the near-future bandwidth availability and create a schedule for the video playout that takes likely future availability into account. To create a prototype and perform initial tests, we conducted several field trials while commuting using public transportation. We show how our database has been used to predict bandwidth fluctuations and network outages, and how this information helps maintain uninterrupted playback with less compromise on video quality than possible without prediction.

Video streaming using a location-based bandwidth-lookup service for bitrate planning

There are many available commercial streaming solutions that perform quality adaption. An important issue with respect to users' perceived quality is how the system schedules the quality levels to match the available network resources. In this study, we compare several adaptive media players on the market to see how they perform in challenging streaming scenarios on a mobile 3G network. Bandwidth data collected in real-world field trials is used in all tests. We investigate how the media players respond to fluctuating bandwidth and outages, and how this affects the quality levels used, the bandwidth utilization, and the number and duration of buffer underruns. We found significant differences in performance and optimization goals between the different players' schedulers. We conclude that the quality scheduler is an important factor in providing a satisfying quality of experience when using an adaptive media player.

A comparison of quality scheduling in commercial adaptive HTTP streaming solutions on a 3G network

A well known challenge with mobile video streaming is fluctuating bandwidth. As the client devices move in and out of network coverage areas, the users may experience varying signal strengths, competition for the available resources and periods of network outage. These conditions have a significant effect on video quality. In this paper, we present a video streaming solution for roaming clients that is able to compensate for the effects of oscillating bandwidth through bandwidth prediction and video quality scheduling. We combine our existing adaptive segmented HTTP streaming system with 1) an application layer framework for creating transparent multi-link applications, and 2) a location based QoS information system containing GPS coordinates and accompanying bandwidth measurements, populated through crowd-sourcing. Additionally, we use real-time traffic information to improve the prediction by, for example, estimating the length of a commute route. To evaluate our prototype, we performed real-world experiments using a popular tram route in Oslo, Norway. The client connected to multiple networks, and the results show that our solution increases the perceived video quality significantly. Also, we used simulations to evaluate the potential of aggregating bandwidth along the route.

Mobile video streaming using location-based network prediction and transparent handover

One of the main challenges when streaming video to mobile devices is to handle fluctuating bandwidth and frequent network outages as the device is brought in and out of areas with network coverage. In this paper, we propose bitrate and video quality planning algorithms for mobile streaming scenarios using a GPS-based bandwidth-lookup service in order to reduce frequently changing video quality and the number of playout interruptions. Our real world experiments using an adaptive segmented HTTP video streaming system while travelling on popular commute routes indicate that the users' quality of experience is greatly increased.

Paul Vigmostad

Papers

Commute path bandwidth traces from 3G networks: analysis and applications

Video streaming using a location-based bandwidth-lookup service for bitrate planning

A comparison of quality scheduling in commercial adaptive HTTP streaming solutions on a 3G network

Mobile video streaming using location-based network prediction and transparent handover

Bitrate and video quality planning for mobile streaming scenarios using a GPS-based bandwidth lookup service