This paper presents a comprehensive literature review on applications of deep reinforcement learning (DRL) in communications and networking. Modern networks, e.g., Internet of Things (IoT) and unmanned aerial vehicle (UAV) networks, become more decentralized and autonomous. In such networks, network entities need to make decisions locally to maximize the network performance under uncertainty of network environment. Reinforcement learning has been efficiently used to enable the network entities to obtain the optimal policy including, e.g., decisions or actions, given their states when the state and action spaces are small. However, in complex and large-scale networks, the state and action spaces are usually large, and the reinforcement learning may not be able to find the optimal policy in reasonable time. Therefore, DRL, a combination of reinforcement learning with deep learning, has been developed to overcome the shortcomings. In this survey, we first give a tutorial of DRL from fundamental concepts to advanced models. Then, we review DRL approaches proposed to address emerging issues in communications and networking. The issues include dynamic network access, data rate control, wireless caching, data offloading, network security, and connectivity preservation which are all important to next generation networks, such as 5G and beyond. Furthermore, we present applications of DRL for traffic routing, resource sharing, and data collection. Finally, we highlight important challenges, open issues, and future research directions of applying DRL.

Applications of Deep Reinforcement Learning in Communications and Networking: A Survey

Resource management problems in systems and networking often manifest as difficult online decision making tasks where appropriate solutions depend on understanding the workload and environment. Inspired by recent advances in deep reinforcement learning for AI problems, we consider building systems that learn to manage resources directly from experience. We present DeepRM, an example solution that translates the problem of packing tasks with multiple resource demands into a learning problem. Our initial results show that DeepRM performs comparably to state-of-the-art heuristics, adapts to different conditions, converges quickly, and learns strategies that are sensible in hindsight.

/pdf/resource-management-with-deep-reinforcement-learning-3u31aw7drv.pdf

Resource Management with Deep Reinforcement Learning

Client-side video players employ adaptive bitrate (ABR) algorithms to optimize user quality of experience (QoE). Despite the abundance of recently proposed schemes, state-of-the-art ABR algorithms suffer from a key limitation: they use fixed control rules based on simplified or inaccurate models of the deployment environment. As a result, existing schemes inevitably fail to achieve optimal performance across a broad set of network conditions and QoE objectives.We propose Pensieve, a system that generates ABR algorithms using reinforcement learning (RL). Pensieve trains a neural network model that selects bitrates for future video chunks based on observations collected by client video players. Pensieve does not rely on pre-programmed models or assumptions about the environment. Instead, it learns to make ABR decisions solely through observations of the resulting performance of past decisions. As a result, Pensieve automatically learns ABR algorithms that adapt to a wide range of environments and QoE metrics. We compare Pensieve to state-of-the-art ABR algorithms using trace-driven and real world experiments spanning a wide variety of network conditions, QoE metrics, and video properties. In all considered scenarios, Pensieve outperforms the best state-of-the-art scheme, with improvements in average QoE of 12%--25%. Pensieve also generalizes well, outperforming existing schemes even on networks for which it was not explicitly trained.

/pdf/neural-adaptive-video-streaming-with-pensieve-3y10qvjqj1.pdf

Neural Adaptive Video Streaming with Pensieve

Digital Control Of Dynamic Systems This well-respected, market-leading text discusses the use of digital computers in the real-time control of dynamic systems. The emphasis is on the design of digital controls that achieve good dynamic response and small errors while using signals that are sampled in time and quantized in amplitude. Digital Control of Dynamic Systems (3rd Edition): Franklin ... This well-respected, market-leading text discusses the use of digital computers in the real-time control of dynamic systems. The emphasis is on the design of digital controls that achieve good dynamic response and small errors while using signals that are sampled in time and quantized in amplitude. Digital Control of Dynamic Systems: Gene F. Franklin ... Digital Control of Dynamic Systems, 2nd Edition. Gene F. Franklin, Stanford University. J. David Powell, Stanford University Digital Control of Dynamic Systems, 2nd Edition Pearson This well-respected work discusses the use of digital computers in the real-time control of dynamic systems. The emphasis is on the design of digital controls that achieve good dynamic response and small errors while using signals that are sampled in time and quantized in amplitude. MATLAB statements and problems are thoroughly and carefully integrated throughout the book to offer readers a complete design picture. Digital Control of Dynamic Systems, 3rd Edition ... Digital control of dynamic systems | Gene F. Franklin, J. David Powell, Michael L. Workman | download | B–OK. Download books for free. Find books Digital control of dynamic systems | Gene F. Franklin, J ... Abstract This well-respected work discusses the use of digital computers in the real-time control of dynamic systems. The emphasis is on the design of digital controls that achieve good dynamic... (PDF) Digital Control of Dynamic Systems Digital Control of Dynamic Systems, Addison.pdf There is document Digital Control of Dynamic Systems, Addison.pdfavailable here for reading and downloading. Use the download button below or simple online reader. The file extension PDFand ranks to the Documentscategory. Digital Control of Dynamic Systems, Addison.pdf Download ... Automatic control is the science that develops techniques to steer, guide, control dynamic systems. These systems are built by humans and must perform a specific task. Examples of such dynamic systems are found in biology, physics, robotics, finance, etc. Digital Control means that the control laws are implemented in a digital device, such as a microcontroller or a microprocessor. Introduction to Digital Control of Dynamic Systems And ... The discussions are clear, nomenclature is not hard to follow and there are plenty of worked examples. The book covers discretization effects and design by emulation (i.e. design of continuous-time control system followed by discretization before implementation) which are not to be found on every book on digital control. Amazon.com: Customer reviews: Digital Control of Dynamic ... Find helpful customer reviews and review ratings for Digital Control of Dynamic Systems (3rd Edition) at Amazon.com. Read honest and unbiased product reviews from our users. Amazon.com: Customer reviews: Digital Control of Dynamic ... 1.1.2 Digital control Digital control systems employ a computer as a fundamental component in the controller. The computer typically receives a measurement of the controlled variable, also often receives the reference input, and produces its output using an algorithm. Introduction to Applied Digital Control From the Back Cover This well-respected, marketleading text discusses the use of digital computers in the real-time control of dynamic systems. The emphasis is on the design of digital controls that achieve good dynamic response and small errors while using signals that are sampled in time and quantized in amplitude. Digital Control of Dynamic Systems (3rd Edition) Test Bank `Among the advantages of digital logic for control are the increased flexibility `of the control programs and the decision-making or logic capability of digital `systems, which can be combined with the dynamic control function to meet `other system requirements. `The digital controls studied in this book are for closed-loop (feedback) Every day, eBookDaily adds three new free Kindle books to several different genres, such as Nonfiction, Business & Investing, Mystery & Thriller, Romance, Teens & Young Adult, Children's Books, and others.

http://ieeexplore.ieee.org/iel6/29/26151/01163502.pdf

Digital control of dynamic systems

User-perceived quality-of-experience (QoE) is critical in Internet video applications as it impacts revenues for content providers and delivery systems. Given that there is little support in the network for optimizing such measures, bottlenecks could occur anywhere in the delivery system. Consequently, a robust bitrate adaptation algorithm in client-side players is critical to ensure good user experience. Previous studies have shown key limitations of state-of-art commercial solutions and proposed a range of heuristic fixes. Despite the emergence of several proposals, there is still a distinct lack of consensus on: (1) How best to design this client-side bitrate adaptation logic (e.g., use rate estimates vs. buffer occupancy); (2) How well specific classes of approaches will perform under diverse operating regimes (e.g., high throughput variability); or (3) How do they actually balance different QoE objectives (e.g., startup delay vs. rebuffering). To this end, this paper makes three key technical contributions. First, to bring some rigor to this space, we develop a principled control-theoretic model to reason about a broad spectrum of strategies. Second, we propose a novel model predictive control algorithm that can optimally combine throughput and buffer occupancy information to outperform traditional approaches. Third, we present a practical implementation in a reference video player to validate our approach using realistic trace-driven emulations.

/pdf/a-control-theoretic-approach-for-dynamic-adaptive-video-1mmix9lrz0.pdf

A Control-Theoretic Approach for Dynamic Adaptive Video Streaming over HTTP

Bitrate adaptation is critical in ensuring good users’ quality-of-experience (QoE) in Internet video delivery system. Several efforts have argued that accurate throughput prediction can dramatically improve (1) initial bitrate selection for low startup delay and high initial resolution; (2) midstream bitrate adaptation for high QoE. However, prior ef- forts did not systematically quantify real-world throughput predictability or develop good prediction algorithms. To bridge this gap, this paper makes three key technical contributions: First, we analyze the throughput characteristics in a dataset with 20M+ sessions. We find: (a) Sessions sharing similar key features (e.g., ISP, region) present similar initial values and dynamical patterns; (b) There is a natural “stateful” dynamical behavior within a given session. Second, building on these insights, we develop CS2P, a better throughput prediction system. CS2P leverages data-driven approach to learn (a) clusters of similar sessions, (b) an initial throughput predictor, and (c) a Hidden-Markov-Model based midstream predictor modeling the stateful evolution of throughput. Third, we develop a prototype system and show by trace-driven simulation and real-world experiments that CS2P outperforms state-of-art by 40% and 50% median pre- diction error respectively for initial and midstream through- put and improves QoE by 14% over buffer-based adaptation algorithm.

https://dl.acm.org/doi/pdf/10.1145/2934872.2934898

CS2P: Improving Video Bitrate Selection and Adaptation with Data-Driven Throughput Prediction

Client-side bitrate adaptation algorithms play a critical role in delivering a good quality of experience for Internet video. Many studies have shown that current solutions perform suboptimally, and despite the proliferation of several proposals in this space, both from commercial providers and researchers, there is still a distinct lack of clarity and consensus w.r.t. several natural questions: (1) What objectives does/should such an algorithm optimize? (2) What environment signals such as buffer occupancy or throughput estimates should an algorithm use in its control loop? (3) How sensitive is an algorithm to operating conditions (e.g., bandwidth stability, buffer size, available bitrates)? This work attempts to bring clarity to this discussion by casting adaptive bitrate streaming as a model-based predictive control problem. We demonstrate the initial promise of shedding light on these questions using this control-theoretic abstraction.

/pdf/toward-a-principled-framework-to-design-dynamic-adaptive-2z6omxey2t.pdf

Toward a Principled Framework to Design Dynamic Adaptive Streaming Algorithms over HTTP

We present Panoptic Neural Fields (PNF), an object-aware neural scene representation that decomposes a scene into a set of objects (things) and background (stuff). Each object is represented by an oriented 3D bounding box and a multi-layer perceptron (MLP) that takes position, direction, and time and outputs density and radiance. The background stuff is represented by a similar MLP that additionally outputs semantic labels. Each object MLPs are instance-specific and thus can be smaller and faster than previous object-aware approaches, while still leveraging category-specific priors incorporated via meta-learned initialization. Our model builds a panoptic radiance field representation of any scene from just color images. We use off-the-shelf algorithms to predict camera poses, object tracks, and 2D image semantic segmentations. Then we jointly optimize the MLP weights and bounding box parameters using analysis-by-synthesis with self-supervision from color images and pseudo-supervision from predicted semantic segmentations. During experiments with real-world dynamic scenes, we find that our model can be used effectively for several tasks like novel view synthesis, 2D panoptic segmentation, 3D scene editing, and multiview depth prediction.

Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation

Semantic segmentation of 3D meshes is an important problem for 3D scene understanding. In this paper we revisit the classic multiview representation of 3D meshes and study several techniques that make them effective for 3D semantic segmentation of meshes. Given a 3D mesh reconstructed from RGBD sensors, our method effectively chooses different virtual views of the 3D mesh and renders multiple 2D channels for training an effective 2D semantic segmentation model. Features from multiple per view predictions are finally fused on 3D mesh vertices to predict mesh semantic segmentation labels. Using the large scale indoor 3D semantic segmentation benchmark of ScanNet, we show that our virtual views enable more effective training of 2D semantic segmentation networks than previous multiview approaches. When the 2D per pixel predictions are aggregated on 3D surfaces, our virtual multiview fusion method is able to achieve significantly better 3D semantic segmentation results compared to all prior multiview approaches and competitive with recent 3D convolution approaches.

Xiaoqi Yin

Papers

A Control-Theoretic Approach for Dynamic Adaptive Video Streaming over HTTP

CS2P: Improving Video Bitrate Selection and Adaptation with Data-Driven Throughput Prediction

Toward a Principled Framework to Design Dynamic Adaptive Streaming Algorithms over HTTP

Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation

Virtual Multi-view Fusion for 3D Semantic Segmentation