Machine Learning is the study of methods for programming computers to learn. Computers are applied to a wide range of tasks, and for most of these it is relatively easy for programmers to design and implement the necessary software. However, there are many tasks for which this is difficult or impossible. These can be divided into four general categories. First, there are problems for which there exist no human experts. For example, in modern automated manufacturing facilities, there is a need to predict machine failures before they occur by analyzing sensor readings. Because the machines are new, there are no human experts who can be interviewed by a programmer to provide the knowledge necessary to build a computer system. A machine learning system can study recorded data and subsequent machine failures and learn prediction rules. Second, there are problems where human experts exist, but where they are unable to explain their expertise. This is the case in many perceptual tasks, such as speech recognition, hand-writing recognition, and natural language understanding. Virtually all humans exhibit expert-level abilities on these tasks, but none of them can describe the detailed steps that they follow as they perform them. Fortunately, humans can provide machines with examples of the inputs and correct outputs for these tasks, so machine learning algorithms can learn to map the inputs to the outputs. Third, there are problems where phenomena are changing rapidly. In finance, for example, people would like to predict the future behavior of the stock market, of consumer purchases, or of exchange rates. These behaviors change frequently, so that even if a programmer could construct a good predictive computer program, it would need to be rewritten frequently. A learning program can relieve the programmer of this burden by constantly modifying and tuning a set of learned prediction rules. Fourth, there are applications that need to be customized for each computer user separately. Consider, for example, a program to filter unwanted electronic mail messages. Different users will need different filters. It is unreasonable to expect each user to program his or her own rules, and it is infeasible to provide every user with a software engineer to keep the rules up-to-date. A machine learning system can learn which mail messages the user rejects and maintain the filtering rules automatically. Machine learning addresses many of the same research questions as the fields of statistics, data mining, and psychology, but with differences of emphasis. Statistics focuses on understanding the phenomena that have generated the data, often with the goal of testing different hypotheses about those phenomena. Data mining seeks to find patterns in the data that are understandable by people. Psychological studies of human learning aspire to understand the mechanisms underlying the various learning behaviors exhibited by people (concept learning, skill acquisition, strategy change, etc.).

Machine learning

Statistics for Spatial Data.

This paper presents a comprehensive literature review on applications of deep reinforcement learning (DRL) in communications and networking. Modern networks, e.g., Internet of Things (IoT) and unmanned aerial vehicle (UAV) networks, become more decentralized and autonomous. In such networks, network entities need to make decisions locally to maximize the network performance under uncertainty of network environment. Reinforcement learning has been efficiently used to enable the network entities to obtain the optimal policy including, e.g., decisions or actions, given their states when the state and action spaces are small. However, in complex and large-scale networks, the state and action spaces are usually large, and the reinforcement learning may not be able to find the optimal policy in reasonable time. Therefore, DRL, a combination of reinforcement learning with deep learning, has been developed to overcome the shortcomings. In this survey, we first give a tutorial of DRL from fundamental concepts to advanced models. Then, we review DRL approaches proposed to address emerging issues in communications and networking. The issues include dynamic network access, data rate control, wireless caching, data offloading, network security, and connectivity preservation which are all important to next generation networks, such as 5G and beyond. Furthermore, we present applications of DRL for traffic routing, resource sharing, and data collection. Finally, we highlight important challenges, open issues, and future research directions of applying DRL.

Applications of Deep Reinforcement Learning in Communications and Networking: A Survey

In this chapter, you will see how the simplex method simplifies when it is applied to a class of optimization problems that are known as “network flow models.” You will also see that if a network flow model has “integer-valued data,” the simplex method finds an optimal solution that is integer-valued.

Flows in Networks

In this chapter we deal with the following nonlinear fractional programming problem: 

$$P:\mathop{{\max }}\limits_{{x \in s}} q(x) = (f(x) + \alpha )/((x) + \beta )$$

where f, g: R n → R, α, β ∈ R, S ⊆ R n . To simplify things, and without restricting the generality of the problem, it is usually assumed that, g(x) + β + 0 for all x ∈ S,S is non-empty and that the objective function has a finite optimal value.

Nonlinear Fractional Programming

Existing multi-hop routing protocols for vehicular ad hoc networks (VANETs) do not consider the packet payload size and some important MAC layer issues for the route selection. In this paper, we first solve the performance anomaly problem by providing the same transmission time for different nodes which have different channel qualities using an adaptive frame aggregation mechanism. Next, we propose a packet size-aware routing protocol where a communication route is determined by taking into account the payload size of data packets. We consider multiple metrics for the route selection specifically vehicle mobility, frame aggregation efficiency, and link quality. The proposed protocol is evaluated using real-world experiments as well as computer simulations.

A routing protocol for VANETs with adaptive frame aggregation and packet size awareness

The Tor network is a widespread overlay anonymous network that is used by millions of users today. Tor focuses on improving the privacy of its users by routing their traffics through a series of onion routers that are located around the world. Since Tor operations mainly depend on volunteers who donated onion routers in the network, there are variances of capacities amongst different onion routers, which degrade the performances of Tor. In an effort to reduce the congestion problems in Tor, first we implemented control metrics to detect congestion on the entry OR. Second, we performed dynamic estimation of circuit congestion to select the best path in Tor. Third, we addressed the unfair distribution of bandwidth between the bulk and light traffics in Tor. We distributed the light and bulk traffics in Tor by having almost the same throughputs compared to default Tor multiplexing circuit approach. We use RTTs, circuit congestion and OR capacities as the main metrics to select the path. From our experimental analysis, we showed that our circuit switching method is effective to reduce the congestion problems in Tor.

Evaluation of dynamic circuit switching to reduce congestion in Tor

Accurate vehicle speed prediction is one of the important issues for achieving advanced intelligent transportation systems (ITS). There are two main methods for vehicle speed prediction, specifically, the statistical prediction and neural network-based approach. However, most existing neural network-based approaches use recurrent neural networks, which can not address the spatial characteristics of traffic flows. In this paper, we propose a convolutional neural network-based approach for a better estimation of vehicle traffics.

Vehicle Speed Prediction with Convolutional Neural Networks for ITS

This paper demonstrates a route selection mechanism on a testbed with heterogeneous device-to-device (D2D) wireless communication for a 5G network scenario. The source node receives information about the primary users’ (PUs’) (or licensed users’) activities and available routes from the macrocell base station (or a central controller) and makes a decision to select a multihop route to the destination node. The source node from small cells can either choose: (a) a route with direct communication with the macrocell base station to improve the route performance; or (b) a route with D2D communication among nodes in the small cells to offload traffic from the macrocell to improve spectrum efficiency. The selected D2D route has the least PUs’ activities. The route selection mechanism is investigated on our testbed that helps to improve the accuracy of network performance measurement. In traditional testbeds, each node (e.g., Universal Software Radio Peripheral (USRP) that serves as the front-end communication block) is connected to a single processing unit (e.g., a personal computer) via a switch using cables. In our testbed, each USRP node is connected to a separate processing unit, i.e., raspberry Pi3 B+ (or RP3), which offers three main advantages: (a) control messages and data packets are exchanged via the wireless medium; (b) separate processing units make decisions in a distributed and heterogeneous manner; and (c) the nodes are placed further apart from one another. Therefore, in the investigation of our route selection scheme, the response delay of control message exchange and the packet loss caused by the operating environment (e.g., ambient noise) are implied in our end-to-end delay and packet delivery ratio measurement. Our results show an increase of end-to-end delay and a decrease of packet delivery ratio due to the transmission of control messages and data packets in the wireless medium in the presence of the dynamic PUs’ activities. Furthermore, D2D communication can offload 25% to 75% traffic from macrocell base station to small cells.

https://www.mdpi.com/2079-9292/10/4/387/pdf

An Experimental Study on D2D Route Selection Mechanism in 5G Scenarios

This paper puts forwards an on-line reinforcement learning framework for the problem of traffic offloading in a stochastic Markovian heterogeneous cellular network (HCN), where the time-varying traffic demand of mobile terminals (MTs) can be offloaded from macrocells to small-cells. Our aim is to minimize the average energy consumption of the HCN while maintaining the Quality-of-Service (QoS) experienced by MTs. For each cell (i.e., a macrocell or a small-cell), the energy consumption is determined by its system load which is coupled with the system loads served in other cells due to the sharing over a common frequency band. We model the energy-aware traffic offloading in such HCNs as a constrained Markov decision process (C-MDP). The statistics of the C-MDP depends on a selected traffic offloading strategy and thus, the actions performed by a network controller have a long-term impact on the network state evolution. Based on the traffic demand observations and the traffic offloading operations, the controller gradually optimizes the strategy with no prior knowledge of the process statistics. Numerical experiments are conducted to show the effectiveness of the proposed learning framework in balancing the tradeoff between energy saving and QoS satisfaction.

Celimuge Wu

Papers

A routing protocol for VANETs with adaptive frame aggregation and packet size awareness

Evaluation of dynamic circuit switching to reduce congestion in Tor

Vehicle Speed Prediction with Convolutional Neural Networks for ITS

An Experimental Study on D2D Route Selection Mechanism in 5G Scenarios

Greenly offloading traffic in stochastic heterogeneous cellular networks