Machine Learning is the study of methods for programming computers to learn. Computers are applied to a wide range of tasks, and for most of these it is relatively easy for programmers to design and implement the necessary software. However, there are many tasks for which this is difficult or impossible. These can be divided into four general categories. First, there are problems for which there exist no human experts. For example, in modern automated manufacturing facilities, there is a need to predict machine failures before they occur by analyzing sensor readings. Because the machines are new, there are no human experts who can be interviewed by a programmer to provide the knowledge necessary to build a computer system. A machine learning system can study recorded data and subsequent machine failures and learn prediction rules. Second, there are problems where human experts exist, but where they are unable to explain their expertise. This is the case in many perceptual tasks, such as speech recognition, hand-writing recognition, and natural language understanding. Virtually all humans exhibit expert-level abilities on these tasks, but none of them can describe the detailed steps that they follow as they perform them. Fortunately, humans can provide machines with examples of the inputs and correct outputs for these tasks, so machine learning algorithms can learn to map the inputs to the outputs. Third, there are problems where phenomena are changing rapidly. In finance, for example, people would like to predict the future behavior of the stock market, of consumer purchases, or of exchange rates. These behaviors change frequently, so that even if a programmer could construct a good predictive computer program, it would need to be rewritten frequently. A learning program can relieve the programmer of this burden by constantly modifying and tuning a set of learned prediction rules. Fourth, there are applications that need to be customized for each computer user separately. Consider, for example, a program to filter unwanted electronic mail messages. Different users will need different filters. It is unreasonable to expect each user to program his or her own rules, and it is infeasible to provide every user with a software engineer to keep the rules up-to-date. A machine learning system can learn which mail messages the user rejects and maintain the filtering rules automatically. Machine learning addresses many of the same research questions as the fields of statistics, data mining, and psychology, but with differences of emphasis. Statistics focuses on understanding the phenomena that have generated the data, often with the goal of testing different hypotheses about those phenomena. Data mining seeks to find patterns in the data that are understandable by people. Psychological studies of human learning aspire to understand the mechanisms underlying the various learning behaviors exhibited by people (concept learning, skill acquisition, strategy change, etc.).

Machine learning

Statistics for Spatial Data.

This paper presents a comprehensive literature review on applications of deep reinforcement learning (DRL) in communications and networking. Modern networks, e.g., Internet of Things (IoT) and unmanned aerial vehicle (UAV) networks, become more decentralized and autonomous. In such networks, network entities need to make decisions locally to maximize the network performance under uncertainty of network environment. Reinforcement learning has been efficiently used to enable the network entities to obtain the optimal policy including, e.g., decisions or actions, given their states when the state and action spaces are small. However, in complex and large-scale networks, the state and action spaces are usually large, and the reinforcement learning may not be able to find the optimal policy in reasonable time. Therefore, DRL, a combination of reinforcement learning with deep learning, has been developed to overcome the shortcomings. In this survey, we first give a tutorial of DRL from fundamental concepts to advanced models. Then, we review DRL approaches proposed to address emerging issues in communications and networking. The issues include dynamic network access, data rate control, wireless caching, data offloading, network security, and connectivity preservation which are all important to next generation networks, such as 5G and beyond. Furthermore, we present applications of DRL for traffic routing, resource sharing, and data collection. Finally, we highlight important challenges, open issues, and future research directions of applying DRL.

Applications of Deep Reinforcement Learning in Communications and Networking: A Survey

In this chapter, you will see how the simplex method simplifies when it is applied to a class of optimization problems that are known as “network flow models.” You will also see that if a network flow model has “integer-valued data,” the simplex method finds an optimal solution that is integer-valued.

Flows in Networks

In this chapter we deal with the following nonlinear fractional programming problem: 

$$P:\mathop{{\max }}\limits_{{x \in s}} q(x) = (f(x) + \alpha )/((x) + \beta )$$

where f, g: R n → R, α, β ∈ R, S ⊆ R n . To simplify things, and without restricting the generality of the problem, it is usually assumed that, g(x) + β + 0 for all x ∈ S,S is non-empty and that the objective function has a finite optimal value.

Nonlinear Fractional Programming

Efficient data dissemination in Vehicular Ad hoc NETworks (VANETs) has been a challenging issue due to vehicle movements, limited wireless resources and lossy characteristics of wireless communication. In this paper, we use dynamically generated backbone vehicles to disseminate information. The proposed protocol selects the backbone vehicles by considering vehicle movement dynamics and link quality between vehicles. The proposed approach can significantly reduce the MAC layer contention time at each node while maintaining a high packet dissemination ratio. We show the effectiveness of the proposed protocol by using theoretical analysis and computer simulations.

Data Dissemination with Dynamic Backbone Selection in Vehicular Ad Hoc Networks

In vehicular ad hoc networks (VANETs), multi-hop broadcast communications are required for many applications including driver assistance systems. However, providing a low end-to-end latency has been very challenging. In this paper, we propose a joint MAC network layer multi-hop broadcast protocol. For the network layer, the proposed protocol reduces the number of sender nodes by using common forwarder nodes for different traffic flows. At the MAC layer, the proposed protocol uses a Q-Learning algorithm to adjust the contention window size. By interacting with the environment, the protocol can find the best contention window size to transmit data packets and therefore can provide a high packet dissemination ratio and low end-to-end delay for various scenarios. The simulation results demonstrate the advantage of the proposed protocol over other existing alternatives.

Joint MAC and Network Layer Control for VANET Broadcast Communications Considering End-to-End Latency

This paper addresses energy-aware traffic offloading in stochastic heterogeneous cellular networks (HCNs). The objective is to minimize energy consumption of the HCN while maintaining Quality-of-Service experienced by the mobile users. For each cell, the energy consumption depends on its associated system load, which is coupled with system loads in other cells due to the sharing over a common spectrum band. Such a traffic offloading problem is modeled by a discrete-time Markov decision process (DTMDP). Based on the traffic observations and the traffic offloading operations, the network controller learns to solve the optimal traffic offloading strategy with no prior knowledge of the DTMDP statistics. To deal with the curse of dimensionality, we design a centralized Q-learning with compact state representation algorithm, which is named as QC-learning. Moreover, a decentralized QC-learning algorithm is developed such that the macro-cell base stations (BSs) can independently manage the operations of small-cell BSs by making use of the network information obtained from the network controller. Simulations validate the proposed studies.

A learning approach for traffic offloading in stochastic heterogeneous cellular networks

With the increase of the number of wireless sensing or metering devices, the collection of sensing data using wireless communication becomes an important part of many applications. Cognitive radio technology can be used to facilitate the deployment of a bandwidth efficient sensor system. In this paper, we propose a data collection and dissemination framework for cognitive radio sensor systems to fully utilize wireless resources while maintaining a reliably connected and efficient topology for each channel. In the proposed framework, each sensor node selects a channel considering primary user channel utilization and network connectivity. In this way, the data collection and dissemination can be performed with a high reliability and short delay while avoiding a harmful effect on primary users. We use computer simulations to evaluate the proposed framework.

Dynamic channel assignment and routing for cognitive sensor networks

In dense small cell networks (DSCNs), small cells are heterogeneous due to the unplanned deployment and various traffic loads, resulting in different energy efficiency (EE) preferences. In this paper, taking into account the heterogeneity of small cells, energy saving (ES) and EE are jointly optimized through subchannel allocation, subframe configuration and power allocation. In order to quantify the effects of small cells’ heterogeneous information on EE, an EE preference function is first defined. Then, the joint ES and EE optimization problem is formulated as a multi-objective optimization problem. Due to the coupling of ES and EE, obtaining the solution is non-trivial. Therefore, we propose a heterogeneity-aware ES and EE (HESEE) optimization algorithm, where subchannel allocation is optimized to ensure the fairness of active subframes required by the users in the same small cell base stations (SBSs). Then subframe configuration is conducted via group formation sleep mechanism. Particularly, address the non-concave sum-of-ratios optimization for system EE, the concave-convex procedure (CCCP) method is adopted. Simulation results show that the proposed HESEE algorithm can optimize the SBSs’ EEs according to their EE preferences. In addition, the HESEE algorithm can achieve good performance in reducing energy consumption as well as improving the system EE.

/pdf/heterogeneity-aware-energy-saving-and-energy-efficiency-9igvjab2si.pdf

Celimuge Wu

Papers

Data Dissemination with Dynamic Backbone Selection in Vehicular Ad Hoc Networks

Joint MAC and Network Layer Control for VANET Broadcast Communications Considering End-to-End Latency

A learning approach for traffic offloading in stochastic heterogeneous cellular networks

Dynamic channel assignment and routing for cognitive sensor networks

Heterogeneity-Aware Energy Saving and Energy Efficiency Optimization in Dense Small Cell Networks