This article surveys the literature over the period of the last decade on the emerging field of self organisation as applied to wireless cellular communication networks. Self organisation has been extensively studied and applied in adhoc networks, wireless sensor networks and autonomic computer networks; however in the context of wireless cellular networks, this is the first attempt to put in perspective the various efforts in form of a tutorial/survey. We provide a comprehensive survey of the existing literature, projects and standards in self organising cellular networks. Additionally, we also aim to present a clear understanding of this active research area, identifying a clear taxonomy and guidelines for design of self organising mechanisms. We compare strength and weakness of existing solutions and highlight the key research areas for further development. This paper serves as a guide and a starting point for anyone willing to delve into research on self organisation in wireless cellular communication networks.

A Survey of Self Organisation in Future Cellular Networks

Future wireless networks have a substantial potential in terms of supporting a broad range of complex compelling applications both in military and civilian fields, where the users are able to enjoy high-rate, low-latency, low-cost and reliable information services. Achieving this ambitious goal requires new radio techniques for adaptive learning and intelligent decision making because of the complex heterogeneous nature of the network structures and wireless services. Machine learning (ML) algorithms have great success in supporting big data analytics, efficient parameter estimation and interactive decision making. Hence, in this article, we review the thirty-year history of ML by elaborating on supervised learning, unsupervised learning, reinforcement learning and deep learning. Furthermore, we investigate their employment in the compelling applications of wireless networks, including heterogeneous networks (HetNets), cognitive radios (CR), Internet of Things (IoT), machine to machine networks (M2M), and so on. This article aims for assisting the readers in clarifying the motivation and methodology of the various ML algorithms, so as to invoke them for hitherto unexplored services as well as scenarios of future wireless networks.

/pdf/thirty-years-of-machine-learning-the-road-to-pareto-optimal-51ltzgc7ke.pdf

Thirty Years of Machine Learning: The Road to Pareto-Optimal Wireless Networks

Reinforcement learning (RL) is a powerful paradigm for sequential decision-making under uncertainties, and most RL algorithms aim to maximize some numerical value which represents only one long-term objective. However, multiple long-term objectives are exhibited in many real-world decision and control systems, so recently there has been growing interest in solving multiobjective reinforcement learning (MORL) problems where there are multiple conflicting objectives. The aim of this paper is to present a comprehensive overview of MORL. The basic architecture, research topics, and naive solutions of MORL are introduced at first. Then, several representative MORL approaches and some important directions of recent research are comprehensively reviewed. The relationships between MORL and other related research are also discussed, which include multiobjective optimization, hierarchical RL, and multiagent RL. Moreover, research challenges and open problems of MORL techniques are suggested.

Multiobjective Reinforcement Learning: A Comprehensive Overview

Reinforcement-learning (RL) algorithms can automatically learn optimal control strategies for nonlinear, possibly stochastic systems. A promising approach for RL control is experience replay (ER), which learns quickly from a limited amount of data, by repeatedly presenting these data to an underlying RL algorithm. Despite its benefits, ER RL has been studied only sporadically in the literature, and its applications have largely been confined to simulated systems. Therefore, in this paper, we evaluate ER RL on real-time control experiments that involve a pendulum swing-up problem and the vision-based control of a goalkeeper robot. These real-time experiments are complemented by simulation studies and comparisons with traditional RL. As a preliminary, we develop a general ER framework that can be combined with essentially any incremental RL technique, and instantiate this framework for the approximate Q-learning and SARSA algorithms. The successful real-time learning results that are presented here are highly encouraging for the applicability of ER RL in practice.

http://busoniu.net/files/papers/smcc11.pdf

Experience Replay for Real-Time Reinforcement Learning Control

By deploying additional low power nodes (LPNs) within the coverage area of traditional high power nodes (HPNs) and bringing them closer to users, underlay heterogeneous networks (HetNets) can significantly boost the overall spectral efficiency (SE) and energy efficiency (EE) through a full spatial resource reuse. Considering that the severe intra-tier interference among dense LPNs and inter-tier interference between LPNs and HPNs are challenging the successful rollout and commercial operations of underlay HetNets, a great emphasis is given towards advanced techniques that take interference control, radio resource allocation, and self-organization into account to enhance both SE and EE in this paper. The interference control techniques presented in this paper are classified as the spatial interference coordination at the transmitter and the interference cancelation at the receiver. For the radio resource allocation, the multi-dimensional optimization, cross-layer optimization, and cooperative radio resource management are comprehensively summarized. The self-configuration, self-optimization, and self-healing techniques for the self-organized underlay HetNets are surveyed. Furthermore, this paper outlines the potential open issues for underlay HetNets to improve SE and EE when combining with energy harvesting and cloud computing.

Recent Advances in Underlay Heterogeneous Networks: Interference Control, Resource Allocation, and Self-Organization

This paper presents a framework based on self-organization to jointly self-optimize the spectrum assignment and the transmission power in the context of downlink OFDMA femtocell deployments. It is a distributed framework where each femtocell acts as an autonomous entity. Femtocells perform spectrum and transmission power assignment based on users’ reported measurements, which are employed to sense intercell interference, including that from other femtocells or from macrocells in two-layer deployments. Results have been obtained for a realistic indoor femtocell deployment with and without macrocell interference. The paper shows that, thanks to self-organization, sensible performance improvements can be achieved in terms of spectral efficiency and power consumption reductions.

Self-Optimization of Spectrum Assignment and Transmission Power in OFDMA Femtocells

This paper presents a decentralized framework for dynamic spectrum assignment in multicell orthogonal frequency division multiple access (OFDMA) networks. The proposed framework allows each cell to autonomously decide the frequency resources it should use through a procedure that incorporates concepts from self-organization and machine learning in multiagent systems (MASs). Simulation results have been obtained for several scenarios, including both macrocells (MCs) and femtocells (FCs), revealing important improvements in terms of spectral efficiency and intercell interference mitigation over reference approaches, and close performance with the one obtained by a centralized strategy. Results also suggest that the framework would be practical for future FC cellular deployments where a high degree of independence of the network nodes is expected to reduce operational costs.

Intercell Interference Management in OFDMA Networks: A Decentralized Approach Based onReinforcement Learning

This paper presents a novel distributed framework to decide the spectrum assignment in a primary cellular radio access network. The distributed nature of the framework allows each cell to autonomously decide (by means of machine learning procedures) the best frequencies to use in order to maximize spectral efficiency, preserve quality-of-service, and generate spectrum gaps, so that secondary cognitive radio networks can improve overall spectrum usage. The proposed distributed framework has been validated over a downlink multicell OFDMA radio access network, showing comparable performance results with respect to its centralized counterpart and superior performance with respect to fixed frequency planning schemes.

/pdf/distributed-spectrum-management-based-on-reinforcement-51dgib6z29.pdf

Distributed spectrum management based on reinforcement learning

This paper proposes reinforcement learning as a foundational stone of a framework for efficient spectrum usage in the context of next-generation mobile cellular networks. The objective of the framework is to efficiently use the spectrum in a cellular orthogonal frequency-division multiple access network while unnecessary spectrum is released for secondary spectrum usage within a private commons spectrum access model. Numerical results show that the proposed framework obtains the best performance compared with other approaches for spectrum assignment. Moreover, the framework is relatively simple to implement in terms of computational requirements and signaling overhead.

/pdf/an-application-of-reinforcement-learning-for-efficient-1eio44lywv.pdf

An Application of Reinforcement Learning for Efficient Spectrum Usage in Next-Generation Mobile Cellular Networks

Next generation wireless networks will encompass a wide range of heterogeneous technologies in the radio access part. In such networks, the all-IP paradigm has been identified as a promising solution that will contribute benefits by providing IP-based transport through the radio and core network parts. However, this concept requires a precise management of the userpsilas mobility, especially in order to preserve userpsilas quality of service (QoS) throughout the sessionpsilas lifetime. The aim of this paper is to evaluate the quality of experience (QoE) that users perceive when the different QoS-aware mobility management strategies adopted in the AROMA project are applied. A real-time testbed that provides end-to-edge QoS in all-IP heterogeneous wireless access networks has been employed to obtain QoE results that hardly could be obtained by means of simulations.

/pdf/quality-of-experience-evaluation-under-qos-aware-mobility-3g2swx1nj1.pdf

Francisco Bernardo

Papers

Self-Optimization of Spectrum Assignment and Transmission Power in OFDMA Femtocells

Intercell Interference Management in OFDMA Networks: A Decentralized Approach Based onReinforcement Learning

Distributed spectrum management based on reinforcement learning

An Application of Reinforcement Learning for Efficient Spectrum Usage in Next-Generation Mobile Cellular Networks

Quality of experience evaluation under QoS-aware mobility mechanisms