Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

There is, I think, something ethereal about i —the square root of minus one. I remember first hearing about it at school. It seemed an odd beast at that time—an intruder hovering on the edge of reality.

Usually familiarity dulls this sense of the bizarre, but in the case of i it was the reverse: over the years the sense of its surreal nature intensified. It seemed that it was impossible to write mathematics that described the real world in …

I and i

The advancement in wireless communications and electronics has enabled the development of low-cost sensor networks. The sensor networks can be used for various application areas (e.g., health, military, home). For different application areas, there are different technical issues that researchers are currently resolving. The current state of the art of sensor networks is captured in this article, where solutions are discussed under their related protocol stack layer sections. This article also points out the open research issues and intends to spark new interests and developments in this field.

/pdf/a-survey-on-sensor-networks-5beav0ez6k.pdf

A survey on sensor networks

We develop and analyze low-complexity cooperative diversity protocols that combat fading induced by multipath propagation in wireless networks. The underlying techniques exploit space diversity available through cooperating terminals' relaying signals for one another. We outline several strategies employed by the cooperating radios, including fixed relaying schemes such as amplify-and-forward and decode-and-forward, selection relaying schemes that adapt based upon channel measurements between the cooperating terminals, and incremental relaying schemes that adapt based upon limited feedback from the destination terminal. We develop performance characterizations in terms of outage events and associated outage probabilities, which measure robustness of the transmissions to fading, focusing on the high signal-to-noise ratio (SNR) regime. Except for fixed decode-and-forward, all of our cooperative diversity protocols are efficient in the sense that they achieve full diversity (i.e., second-order diversity in the case of two terminals), and, moreover, are close to optimum (within 1.5 dB) in certain regimes. Thus, using distributed antennas, we can provide the powerful benefits of space diversity without need for physical arrays, though at a loss of spectral efficiency due to half-duplex operation and possibly at the cost of additional receive hardware. Applicable to any wireless setting, including cellular or ad hoc networks-wherever space constraints preclude the use of physical arrays-the performance characterizations reveal that large power or energy savings result from the use of these protocols.

Cooperative diversity in wireless networks: Efficient protocols and outage behavior

Cognitive radio is viewed as a novel approach for improving the utilization of a precious natural resource: the radio electromagnetic spectrum. The cognitive radio, built on a software-defined radio, is defined as an intelligent wireless communication system that is aware of its environment and uses the methodology of understanding-by-building to learn from the environment and adapt to statistical variations in the input stimuli, with two primary objectives in mind: /spl middot/ highly reliable communication whenever and wherever needed; /spl middot/ efficient utilization of the radio spectrum. Following the discussion of interference temperature as a new metric for the quantification and management of interference, the paper addresses three fundamental cognitive tasks. 1) Radio-scene analysis. 2) Channel-state estimation and predictive modeling. 3) Transmit-power control and dynamic spectrum management. This work also discusses the emergent behavior of cognitive radio.

/pdf/cognitive-radio-brain-empowered-wireless-communications-9m6gu0vz1z.pdf

Cognitive radio: brain-empowered wireless communications

When n identical randomly located nodes, each capable of transmitting at W bits per second and using a fixed range, form a wireless network, the throughput /spl lambda/(n) obtainable by each node for a randomly chosen destination is /spl Theta/(W//spl radic/(nlogn)) bits per second under a noninterference protocol. If the nodes are optimally placed in a disk of unit area, traffic patterns are optimally assigned, and each transmission's range is optimally chosen, the bit-distance product that can be transported by the network per second is /spl Theta/(W/spl radic/An) bit-meters per second. Thus even under optimal circumstances, the throughput is only /spl Theta/(W//spl radic/n) bits per second for each node for a destination nonvanishingly far away. Similar results also hold under an alternate physical model where a required signal-to-interference ratio is specified for successful receptions. Fundamentally, it is the need for every node all over the domain to share whatever portion of the channel it is utilizing with nodes in its local neighborhood that is the reason for the constriction in capacity. Splitting the channel into several subchannels does not change any of the results. Some implications may be worth considering by designers. Since the throughput furnished to each user diminishes to zero as the number of users is increased, perhaps networks connecting smaller numbers of users, or featuring connections mostly with nearby neighbors, may be more likely to be find acceptance.

/pdf/the-capacity-of-wireless-networks-1oibgdldgk.pdf

The capacity of wireless networks

stochastic systems estimation identification and adaptive stochastic adaptive control eolss stochastic systems estimation identification and adaptive stochastic systems estimation identification and adaptive control of stochastic systems eolss stochastic systems estimation identification and adaptive stochastic systems: estimation, identification and adaptation in stochastic dynamic systems survey and new identification and stochastic adaptive control (systems identification and adaptive control methods for some robust stochastic adaptive control dspace@mit: home adaptation in stochastic dynamic systems survey and new chapter 1: introduction to adaptive control stochastic systems estimation identification and adaptive stochastic adaptive nash certainty equivalence control coefficient estimation in adaptive control systems maximum likelihood identification and realization of (size 44,85mb) download ebook stable adaptive systems optimal adaptive control of uncertain stochastic discrete 19,42mb file download system identification adaptive on-line identification and adaptive trajectory tracking ece686: filtering and control of stochastic linear systems robustness and convergence of least-squares identification 68,58mb file system identification adaptive control bahram adaptation in stochastic dynamic systems survey and new identification and system parameter estimation 1991 gbv stochastic adaptive control via consistent parameter adaptive control of stochastic sage pub stochastic delay estimation and adaptive control of eece 574 adaptive control basics of system identification robust identification of stochastic linear systems with robust adaptive els-qr algorithm for linear discrete time stochastic systems: the mathematics of filtering and ee/ise 556: stochastic systems fall 2013 usc search identification and system parameter estimation 1991 gbv

Stochastic systems : estimation, identification, and adaptive control

Theory and practice of recursive identification

Unlike wired networks, wireless networks do not come with links. Rather, links have to be fashioned out of the ether by nodes choosing neighbors to connect to. Moreover the location of the nodes may be random.The question that we resolve is: How many neighbors should each node be connected to in order that the overall network is connected in a multi-hop fashion? We show that in a network with n randomly placed nodes, each node should be connected to Θ(log n) nearest neighbors. If each node is connected to less than 0.074 log n nearest neighbors then the network is asymptotically disconnected with probability one as n increases, while if each node is connected to more than 5.1774 log n nearest neighbors then the network is asymptotically connected with probability approaching one as n increases. It appears that the critical constant may be close to one, but that remains an open problem.These results should be contrasted with some works in the 1970s and 1980s which suggested that the "magic number" of nearest neighbors should be six or eight.

The number of neighbors needed for connectivity of wireless networks

Recently, in an effort to improve the performance of wireless networks, there has been increased interest in protocols that rely on interactions between different layers. However, such cross-layer design can run at cross purposes with sound and longer-term architectural principles, and lead to various negative consequences. This motivates us to step back and reexamine holistically the issue of cross-layer design and its architectural ramifications. We contend that a good architectural design leads to proliferation and longevity, and illustrate this with some historical examples. Even though the wireless medium is fundamentally different from the wired one, and can offer undreamt of modalities of cooperation, we show that the conventional layered architecture is a reasonable way to operate wireless networks, and is in fact optimal up to an order. However the temptation and perhaps even the need to optimize by incorporating cross-layer adaptation cannot be ignored, so we examine the issues involved. We show that unintended cross-layer interactions can have undesirable consequences on overall system performance. We illustrate them by certain cross-layer schemes loosely based on recent proposals. We attempt to distill a few general principles for cross-layer design. Moreover, unbridled cross-layer design can lead to spaghetti design, which can stifle further innovation and be difficult to upkeep. At a critical time when wireless networks may be on the cusp of massive proliferation, the architectural considerations may be paramount. We argue that it behooves us to exercise caution while engaging in cross-layer design.

P. R. Kumar

Papers

The capacity of wireless networks

Stochastic systems : estimation, identification, and adaptive control

Theory and practice of recursive identification

The number of neighbors needed for connectivity of wireless networks

A cautionary perspective on cross-layer design