We will review some of the major results in random graphs and some of the more challenging open problems. We will cover algorithmic and structural questions. We will touch on newer models, including those related to the WWW.

Random graphs

This document provides updates to IEEE Std 802.16's MIB for the MAC, PHY and asso- ciated management procedures in order to accommodate recent extensions to the standard.

IEEE Standard for Local and Metropolitan Area Networks Part 16: Air Interface for Fixed Broadband Wireless Access Systems Draft Amendment: Management Information Base Extensions

Cognitive radio (CR) is the enabling technology for supporting dynamic spectrum access: the policy that addresses the spectrum scarcity problem that is encountered in many countries. Thus, CR is widely regarded as one of the most promising technologies for future wireless communications. To make radios and wireless networks truly cognitive, however, is by no means a simple task, and it requires collaborative effort from various research communities, including communications theory, networking engineering, signal processing, game theory, software-hardware joint design, and reconfigurable antenna and radio-frequency design. In this paper, we provide a systematic overview on CR networking and communications by looking at the key functions of the physical (PHY), medium access control (MAC), and network layers involved in a CR design and how these layers are crossly related. In particular, for the PHY layer, we will address signal processing techniques for spectrum sensing, cooperative spectrum sensing, and transceiver design for cognitive spectrum access. For the MAC layer, we review sensing scheduling schemes, sensing-access tradeoff design, spectrum-aware access MAC, and CR MAC protocols. In the network layer, cognitive radio network (CRN) tomography, spectrum-aware routing, and quality-of-service (QoS) control will be addressed. Emerging CRNs that are actively developed by various standardization committees and spectrum-sharing economics will also be reviewed. Finally, we point out several open questions and challenges that are related to the CRN design.

Cognitive radio networking and communications: an overview

Recent years have witnessed significant advances in reinforcement learning (RL), which has registered tremendous success in solving various sequential decision-making problems in machine learning. Most of the successful RL applications, e.g., the games of Go and Poker, robotics, and autonomous driving, involve the participation of more than one single agent, which naturally fall into the realm of multi-agent RL (MARL), a domain with a relatively long history, and has recently re-emerged due to advances in single-agent RL techniques. Though empirically successful, theoretical foundations for MARL are relatively lacking in the literature. In this chapter, we provide a selective overview of MARL, with focus on algorithms backed by theoretical analysis. More specifically, we review the theoretical results of MARL algorithms mainly within two representative frameworks, Markov/stochastic games and extensive-form games, in accordance with the types of tasks they address, i.e., fully cooperative, fully competitive, and a mix of the two. We also introduce several significant but challenging applications of these algorithms. Orthogonal to the existing reviews on MARL, we highlight several new angles and taxonomies of MARL theory, including learning in extensive-form games, decentralized MARL with networked agents, MARL in the mean-field regime, (non-)convergence of policy-based methods for learning in games, etc. Some of the new angles extrapolate from our own research endeavors and interests. Our overall goal with this chapter is, beyond providing an assessment of the current state of the field on the mark, to identify fruitful future research directions on theoretical studies of MARL. We expect this chapter to serve as continuing stimulus for researchers interested in working on this exciting while challenging topic.

Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms

We propose an adaptive diffusion mechanism to optimize global cost functions in a distributed manner over a network of nodes. The cost function is assumed to consist of a collection of individual components. Diffusion adaptation allows the nodes to cooperate and diffuse information in real-time; it also helps alleviate the effects of stochastic gradient noise and measurement noise through a continuous learning process. We analyze the mean-square-error performance of the algorithm in some detail, including its transient and steady-state behavior. We also apply the diffusion algorithm to two problems: distributed estimation with sparse parameters and distributed localization. Compared to well-studied incremental methods, diffusion methods do not require the use of a cyclic path over the nodes and are robust to node and link failure. Diffusion methods also endow networks with adaptation abilities that enable the individual nodes to continue learning even when the cost function changes with time. Examples involving such dynamic cost functions with moving targets are common in the context of biological networks.

Diffusion Adaptation Strategies for Distributed Optimization and Learning Over Networks

Nature provides splendid examples of real-time learning and adaptation behavior that emerges from highly localized interactions among agents of limited capabilities. For example, schools of fish are remarkably apt at configuring their topologies almost instantly in the face of danger [1]: when a predator arrives, the entire school opens up to let the predator through and then coalesces again into a moving body to continue its schooling behavior. Likewise, in bee swarms, only a small fraction of the agents (about 5%) are informed, and these informed agents are able to guide the entire swarm of bees to their new hive [2]. It is an extraordinary property of biological networks that sophisticated behavior is able to emerge from simple interactions among lower-level agents [3].

Diffusion strategies for adaptation and learning over networks: an examination of distributed strategies and network behavior

Adaptive networks consist of a collection of nodes with adaptation and learning abilities. The nodes interact with each other on a local level and diffuse information across the network to solve estimation and inference tasks in a distributed manner. In this work, we compare the mean-square performance of two main strategies for distributed estimation over networks: consensus strategies and diffusion strategies. The analysis in the paper confirms that under constant step-sizes, diffusion strategies allow information to diffuse more thoroughly through the network and this property has a favorable effect on the evolution of the network: diffusion networks are shown to converge faster and reach lower mean-square deviation than consensus networks, and their mean-square stability is insensitive to the choice of the combination weights. In contrast, and surprisingly, it is shown that consensus networks can become unstable even if all the individual nodes are stable and able to solve the estimation task on their own. When this occurs, cooperation over the network leads to a catastrophic failure of the estimation task. This phenomenon does not occur for diffusion networks: we show that stability of the individual nodes always ensures stability of the diffusion network irrespective of the combination topology. Simulation results support the theoretical findings.

Diffusion Strategies Outperform Consensus Strategies for Distributed Estimation Over Adaptive Networks

In this paper, we investigate the self-organization and cognitive abilities of adaptive networks when the individual agents are allowed to move in pursuit of a target. The nodes act as adaptive entities with localized processing and are able to respond to stimuli in real-time. We apply adaptive diffusion techniques to guide the self-organization process, including harmonious motion and collision avoidance. We also provide stability and mean-square performance analysis of the proposed strategies, together with computer simulation to illustrate results.

Mobile Adaptive Networks

Adaptive networks rely on in-network and collaborative processing among distributed agents to deliver enhanced performance in estimation and inference tasks. Information is exchanged among the nodes, usually over noisy links. The combination weights that are used by the nodes to fuse information from their neighbors play a critical role in influencing the adaptation and tracking abilities of the network. This paper first investigates the mean-square performance of general adaptive diffusion algorithms in the presence of various sources of imperfect information exchanges, quantization errors, and model non-stationarities. Among other results, the analysis reveals that link noise over the regression data modifies the dynamics of the network evolution in a distinct way, and leads to biased estimates in steady-state. The analysis also reveals how the network mean-square performance is dependent on the combination weights. We use these observations to show how the combination weights can be optimized and adapted. Simulation results illustrate the theoretical findings and match well with theory.

Diffusion Adaptation Over Networks Under Imperfect Information Exchange and Non-Stationary Data

Adaptive networks, consisting of a collection of nodes with learning abilities, are well-suited to solve distributed inference problems and to model various types of self-organized behavior observed in nature. One important issue in designing adaptive networks is how to fuse the information collected from the neighbors, especially since the mean-square performance of the network depends on the choice of combination weights. We consider the problem of optimal selection of the combination weights and motivate one combination rule, along with an adaptive implementation. The rule is related to the inverse of the noise variances and is shown to be effective in simulations.

Sheng-Yuan Tu

Papers

Diffusion strategies for adaptation and learning over networks: an examination of distributed strategies and network behavior

Diffusion Strategies Outperform Consensus Strategies for Distributed Estimation Over Adaptive Networks

Mobile Adaptive Networks

Diffusion Adaptation Over Networks Under Imperfect Information Exchange and Non-Stationary Data

Optimal combination rules for adaptation and learning over networks