Machine Learning is the study of methods for programming computers to learn. Computers are applied to a wide range of tasks, and for most of these it is relatively easy for programmers to design and implement the necessary software. However, there are many tasks for which this is difficult or impossible. These can be divided into four general categories. First, there are problems for which there exist no human experts. For example, in modern automated manufacturing facilities, there is a need to predict machine failures before they occur by analyzing sensor readings. Because the machines are new, there are no human experts who can be interviewed by a programmer to provide the knowledge necessary to build a computer system. A machine learning system can study recorded data and subsequent machine failures and learn prediction rules. Second, there are problems where human experts exist, but where they are unable to explain their expertise. This is the case in many perceptual tasks, such as speech recognition, hand-writing recognition, and natural language understanding. Virtually all humans exhibit expert-level abilities on these tasks, but none of them can describe the detailed steps that they follow as they perform them. Fortunately, humans can provide machines with examples of the inputs and correct outputs for these tasks, so machine learning algorithms can learn to map the inputs to the outputs. Third, there are problems where phenomena are changing rapidly. In finance, for example, people would like to predict the future behavior of the stock market, of consumer purchases, or of exchange rates. These behaviors change frequently, so that even if a programmer could construct a good predictive computer program, it would need to be rewritten frequently. A learning program can relieve the programmer of this burden by constantly modifying and tuning a set of learned prediction rules. Fourth, there are applications that need to be customized for each computer user separately. Consider, for example, a program to filter unwanted electronic mail messages. Different users will need different filters. It is unreasonable to expect each user to program his or her own rules, and it is infeasible to provide every user with a software engineer to keep the rules up-to-date. A machine learning system can learn which mail messages the user rejects and maintain the filtering rules automatically. Machine learning addresses many of the same research questions as the fields of statistics, data mining, and psychology, but with differences of emphasis. Statistics focuses on understanding the phenomena that have generated the data, often with the goal of testing different hypotheses about those phenomena. Data mining seeks to find patterns in the data that are understandable by people. Psychological studies of human learning aspire to understand the mechanisms underlying the various learning behaviors exhibited by people (concept learning, skill acquisition, strategy change, etc.).

Machine learning

Statistics for Spatial Data.

This paper presents a comprehensive literature review on applications of deep reinforcement learning (DRL) in communications and networking. Modern networks, e.g., Internet of Things (IoT) and unmanned aerial vehicle (UAV) networks, become more decentralized and autonomous. In such networks, network entities need to make decisions locally to maximize the network performance under uncertainty of network environment. Reinforcement learning has been efficiently used to enable the network entities to obtain the optimal policy including, e.g., decisions or actions, given their states when the state and action spaces are small. However, in complex and large-scale networks, the state and action spaces are usually large, and the reinforcement learning may not be able to find the optimal policy in reasonable time. Therefore, DRL, a combination of reinforcement learning with deep learning, has been developed to overcome the shortcomings. In this survey, we first give a tutorial of DRL from fundamental concepts to advanced models. Then, we review DRL approaches proposed to address emerging issues in communications and networking. The issues include dynamic network access, data rate control, wireless caching, data offloading, network security, and connectivity preservation which are all important to next generation networks, such as 5G and beyond. Furthermore, we present applications of DRL for traffic routing, resource sharing, and data collection. Finally, we highlight important challenges, open issues, and future research directions of applying DRL.

Applications of Deep Reinforcement Learning in Communications and Networking: A Survey

In this chapter, you will see how the simplex method simplifies when it is applied to a class of optimization problems that are known as “network flow models.” You will also see that if a network flow model has “integer-valued data,” the simplex method finds an optimal solution that is integer-valued.

Flows in Networks

In this chapter we deal with the following nonlinear fractional programming problem: 

$$P:\mathop{{\max }}\limits_{{x \in s}} q(x) = (f(x) + \alpha )/((x) + \beta )$$

where f, g: R n → R, α, β ∈ R, S ⊆ R n . To simplify things, and without restricting the generality of the problem, it is usually assumed that, g(x) + β + 0 for all x ∈ S,S is non-empty and that the objective function has a finite optimal value.

Nonlinear Fractional Programming

With the development of advanced information and communication technology, the traditional centralized service model alone no longer meets the increasing demand of data exchange in intelligent transportation systems (ITS). While Internet of Vehicles (IoV) technology has been introduced to achieve more advanced ITS, there are still some unsettled issues such as flexibility and fault tolerance. The conventional centralized approach for ITS is vulnerable to the single point of failure, and lack of flexibility due to its dependence on a trusted third party (TTP). The emergence of blockchain technology provides a potential direction to address these problems. However, due to varying vehicle densities, it is challenging to select the best blockchain parameters to satisfy the application requirements. In this paper, we propose a multi-channel blockchain scheme that can use the best parameters in accordance with the vehicle density. The proposed scheme first defines multiple blockchain channels where each channel is optimized for a certain vehicle density level. Then, the system selects the best channel according to the vehicle density, and the application requirements on the transaction throughput and latency. We use extensive simulations to show that the proposed blockchain scheme achieves a significantly better performance as compared with existing baselines.

/pdf/multi-channel-blockchain-scheme-for-internet-of-vehicles-15d2o2adqe.pdf

Multi-Channel Blockchain Scheme for Internet of Vehicles

QoS of applications in an edge node of mobile edge computing (MEC) could degrade due to congestion of wireless access or short of mobile edge computing resource. In order to improve QoS such as TCP throughput, we propose a VM migration method, which takes a VM from congested node to another node in a mobile edge. Users can choose a far but less-congested node, instead of a near but congested node. The choice is made based on an expected TCP throughput. We analyze how much the migration can improve TCP throughput.

Mobile edge computing based VM migration for QoS improvement

Packet delivery ratio and end-to-end delay are the two most important metrics for vehicular ad hoc network applications. In this paper, we propose a MAC layer protocol which can provide a high packet delivery ratio, low end-to-end delay, and high fairness for various scenarios. The proposed protocol uses a Q-Learning algorithm to adjust the contention window size in order to provide an efficient channel access scheme for various network situations. The simulation results demonstrate the advantage of the proposed protocol over other alternatives.

A MAC protocol for delay-sensitive VANET applications with self-learning contention scheme.

The communication infrastructure is most likely to be damaged after a major disaster occurred, which would lead to further chaos in the disaster stricken area. Modern rescue activities heavily rely on the wireless communications, such as safety status report, disrupted area monitoring, evacuation instruction, rescue coordination, etc. Large amount of data generated from victims, sensors and responders must be delivered and processed in a fast and reliable way, even when the normal communication infrastructure is degraded or destroyed. To this end, reconstructing the post-disaster network by deploying MDRU (Movable and Deployable Resource Unit) and relay unit at edge is a very promising solution. However, the optimal wireless access control in this heterogeneous hastily formed network is extremely challenging, due to the frequent varying environment and the lack of statistics information in advance in post-disaster scenarios. In this paper, we propose a learning based wireless access control approach for edge-aided disaster response network. More specifically, we model the wireless access control procedure as a discrete-time single agent Markov decision process, and solve the problem by exploiting deep reinforcement learning technique. By extensive simulation results, we show that the proposed mechanism significantly outperforms the baseline schemes in terms of delay and packet drop rate.

/pdf/wireless-access-control-in-edge-aided-disaster-response-a-20clalktpy.pdf

Wireless Access Control in Edge-Aided Disaster Response: A Deep Reinforcement Learning-Based Approach

Vehicular Internet-of-Things applications require an efficient Vehicle-to-Everything (V2X) communication scheme. However, it is particularly challenging to achieve a high throughput and low latency with limited wireless resources in highly dynamic vehicular networks. In this article, we propose a scheme that enhances V2V communications through integration of vehicle edge-based forwarding and learning-based edge selection policy optimization. The proposed scheme has three main characteristics. First, the Hierarchical edge-based preemptive route creation is introduced to create hierarchical edges and conduct efficient packet forwarding as well as route aggregation. Second, Two-stage learning is introduced to select efficient edge nodes using big data driven traffic prediction and reinforcement learning-based edge node selection. Third, Context-aware edge selection is employed to improve the performance of edge-based forwarding in various contexts. We use real traffic big data and realistic vehicular network simulations to evaluate the performance of the proposed scheme and show the advantage over other baseline approaches.

/pdf/edge-based-v2x-communications-with-big-data-intelligence-16e2uh6amc.pdf

Celimuge Wu

Papers

Multi-Channel Blockchain Scheme for Internet of Vehicles

Mobile edge computing based VM migration for QoS improvement

A MAC protocol for delay-sensitive VANET applications with self-learning contention scheme.

Wireless Access Control in Edge-Aided Disaster Response: A Deep Reinforcement Learning-Based Approach

Edge-Based V2X Communications With Big Data Intelligence