The leading and most up-to-date textbook on the far-ranging algorithmic methododogy of Dynamic Programming, which can be used for optimal control, Markovian decision problems, planning and sequential decision making under uncertainty, and discrete/combinatorial optimization. The treatment focuses on basic unifying themes, and conceptual foundations. It illustrates the versatility, power, and generality of the method with many examples and applications from engineering, operations research, and other fields. It also addresses extensively the practical application of the methodology, possibly through the use of approximations, and provides an extensive treatment of the far-reaching methodology of Neuro-Dynamic Programming/Reinforcement Learning.

/pdf/dynamic-programming-and-optimal-control-21dopm2oo5.pdf

Dynamic Programming and Optimal Control

In recent years there have been many successes of using deep representations in reinforcement learning. Still, many of these applications use conventional architectures, such as convolutional networks, LSTMs, or auto-encoders. In this paper, we present a new neural network architecture for model-free reinforcement learning. Our dueling network represents two separate estimators: one for the state value function and one for the state-dependent action advantage function. The main benefit of this factoring is to generalize learning across actions without imposing any change to the underlying reinforcement learning algorithm. Our results show that this architecture leads to better policy evaluation in the presence of many similar-valued actions. Moreover, the dueling architecture enables our RL agent to outperform the state-of-the-art on the Atari 2600 domain.

Dueling Network Architectures for Deep Reinforcement Learning

Deep reinforcement learning is poised to revolutionise the field of AI and represents a step towards building autonomous systems with a higher level understanding of the visual world. Currently, deep learning is enabling reinforcement learning to scale to problems that were previously intractable, such as learning to play video games directly from pixels. Deep reinforcement learning algorithms are also applied to robotics, allowing control policies for robots to be learned directly from camera inputs in the real world. In this survey, we begin with an introduction to the general field of reinforcement learning, then progress to the main streams of value-based and policy-based methods. Our survey will cover central algorithms in deep reinforcement learning, including the deep $Q$-network, trust region policy optimisation, and asynchronous advantage actor-critic. In parallel, we highlight the unique advantages of deep neural networks, focusing on visual understanding via reinforcement learning. To conclude, we describe several current areas of research within the field.

A brief survey of deep reinforcement learning

/pdf/dueling-network-architectures-for-deep-reinforcement-5f60efhtka.pdf

Dueling network architectures for deep reinforcement learning

A smart phone senses audio, imagery, and/or other stimulus from a user's environment, and acts autonomously to fulfill inferred or anticipated user desires. In one aspect, the detailed technology concerns phone-based cognition of a scene viewed by the phone's camera. The image processing tasks applied to the scene can be selected from among various alternatives by reference to resource costs, resource constraints, other stimulus information (e.g., audio), task substitutability, etc. The phone can apply more or less resources to an image processing task depending on how successfully the task is proceeding, or based on the user's apparent interest in the task. In some arrangements, data may be referred to the cloud for analysis, or for gleaning. Cognition, and identification of appropriate device response(s), can be aided by collateral information, such as context. A great number of other features and arrangements are also detailed.

Intuitive computing methods and systems

A device for providing access to a remote site is disclosed. Access to the device is gained through an authentication process during which a user password and biometrics are provided to the device. The device also includes a security feature such that only authorized users of the specific device can gain access to it. Once authenticated, the device authorizes access to a remote site (e.g., a web site or a server on a local area network). The communications from the device to the remote site is encrypted and further the hand-held device uses a computer generated password to gain access to the site. In this way, user generated passwords, which are typically simple and infrequently changed, are avoided in favor of a more complex and frequently changed computer generated passwords for site access.

Apparatus and method for authenticating access to a network resource

Means that allow multiple users to be authorized to authenticate through a single given mobile device are described. These means apply as well for the case that the number of users is so large the device does not store all of their authentication information in memory simultaneously. These means allow the authentication information to be securely transferred from a server to the device at the time that the user attempts to authenticate. The device utilizes means and methods that allow this information to be cached to speed up communication during periods when only a few users use the single device.

Apparatus and method for authenticating access to a network resource using multiple shared devices

In some embodiments, a method includes receiving, at a host device, a signal indicative of an authentication request for a client device to access a resource from a set of resources. A resource confidence value associated with the authentication request is calculated based at least in part on (1) a threat confidence vector associated with at least one risk mitigation score for each threat from a set of threats and (2) a set of resource vulnerability scores associated with the resource and each threat from the set of threats. The resource confidence value is compared to a resource confidence criterion associated with the resource from the set of resources. A signal indicative of a positive authentication is sent from the host device to the client device when the resource confidence value satisfies the resource confidence criterion such that the client device is granted access to the resource.

Devices and methods for threat-based authentication for access to computing resources

: A new algorithm advantage learning, is presented that improves on advantage updating by requiring that a single function be learned rather than two. Furthermore, advantage learning requires only a single type of update, the learning, while advantage updating requires two different types of updates, a learning update and a normalization update. The reinforcement learning system uses the residual form of advantage learning. An application of reinforcement learning to a Markov game is presented. The test-bed has continuous states and nonlinear dynamics. The advantage function is stored in a single-hidden-layer sigmoidal network. Speed of learning is increased by a new algorithm, Incremental Delta-Delta (IDD), which extends Jacob's (1988) Delta-Delta for use in incremental training, and differs from Sutton's Incremental Delta-Bar-Delta (1992) in that it does not require the use of a trace and is amenable for use with general function approximation systems. To our knowledge, this is the first time an approximate second order method has been used with residual algorithms. Empirical results are presented comparing convergence rates with and without the use of lDD for the reinforcement learning test-bed and for a supervised learning test-bed.

Multi-Agent Residual Advantage Learning with General Function Approximation.

Foveal vision features imagers with graded acuity coupled with context sensitive sensor gaze control, analogous to that prevalent throughout vertebrate vision. Foveal vision operates more efficiently than uniform acuity vision because resolution is treated as a dynamically allocatable resource, but requires a more refined visual attention mechanism. We demonstrate that reinforcement learning (RL) significantly improves the performance of foveal visual attention, and of the overall vision system, for the task of model based target recognition. A simulated foveal vision system is shown to classify targets with fewer fixations by learning strategies for the acquisition of visual information relevant to the task, and learning how to generalize these strategies in ambiguous and unexpected scenario conditions.

Mance E. Harmon

Papers

Apparatus and method for authenticating access to a network resource

Apparatus and method for authenticating access to a network resource using multiple shared devices

Devices and methods for threat-based authentication for access to computing resources

Multi-Agent Residual Advantage Learning with General Function Approximation.

Residual Q-Learning Applied to Visual Attention