Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

https://www2.econ.iastate.edu/tesfatsi/DeepLearningInNeuralNetworksOverview.JSchmidhuber2015.pdf

Deep learning in neural networks

Background on genetic algorithms, LISP, and genetic programming hierarchical problem-solving introduction to automatically-defined functions - the two-boxes problem problems that straddle the breakeven point for computational effort Boolean parity functions determining the architecture of the program the lawnmower problem the bumblebee problem the increasing benefits of ADFs as problems are scaled up finding an impulse response function artificial ant on the San Mateo trail obstacle-avoiding robot the minesweeper problem automatic discovery of detectors for letter recognition flushes and four-of-a-kinds in a pinochle deck introduction to biochemistry and molecular biology prediction of transmembrane domains in proteins prediction of omega loops in proteins lookahead version of the transmembrane problem evolutionary selection of the architecture of the program evolution of primitives and sufficiency evolutionary selection of terminals evolution of closure simultaneous evolution of architecture, primitive functions, terminals, sufficiency, and closure the role of representation and the lens effect Appendices: list of special symbols list of special functions list of type fonts default parameters computer implementation annotated bibliography of genetic programming electronic mailing list and public repository

Genetic Programming: On the Programming of Computers by Means of Natural Selection

Machine Learning is the study of methods for programming computers to learn. Computers are applied to a wide range of tasks, and for most of these it is relatively easy for programmers to design and implement the necessary software. However, there are many tasks for which this is difficult or impossible. These can be divided into four general categories. First, there are problems for which there exist no human experts. For example, in modern automated manufacturing facilities, there is a need to predict machine failures before they occur by analyzing sensor readings. Because the machines are new, there are no human experts who can be interviewed by a programmer to provide the knowledge necessary to build a computer system. A machine learning system can study recorded data and subsequent machine failures and learn prediction rules. Second, there are problems where human experts exist, but where they are unable to explain their expertise. This is the case in many perceptual tasks, such as speech recognition, hand-writing recognition, and natural language understanding. Virtually all humans exhibit expert-level abilities on these tasks, but none of them can describe the detailed steps that they follow as they perform them. Fortunately, humans can provide machines with examples of the inputs and correct outputs for these tasks, so machine learning algorithms can learn to map the inputs to the outputs. Third, there are problems where phenomena are changing rapidly. In finance, for example, people would like to predict the future behavior of the stock market, of consumer purchases, or of exchange rates. These behaviors change frequently, so that even if a programmer could construct a good predictive computer program, it would need to be rewritten frequently. A learning program can relieve the programmer of this burden by constantly modifying and tuning a set of learned prediction rules. Fourth, there are applications that need to be customized for each computer user separately. Consider, for example, a program to filter unwanted electronic mail messages. Different users will need different filters. It is unreasonable to expect each user to program his or her own rules, and it is infeasible to provide every user with a software engineer to keep the rules up-to-date. A machine learning system can learn which mail messages the user rejects and maintain the filtering rules automatically. Machine learning addresses many of the same research questions as the fields of statistics, data mining, and psychology, but with differences of emphasis. Statistics focuses on understanding the phenomena that have generated the data, often with the goal of testing different hypotheses about those phenomena. Data mining seeks to find patterns in the data that are understandable by people. Psychological studies of human learning aspire to understand the mechanisms underlying the various learning behaviors exhibited by people (concept learning, skill acquisition, strategy change, etc.).

Machine learning

Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

Pattern Recognition and Machine Learning

A common complaint about reinforcement learning (RL) is that it is too slow to learn a value function which gives good performance. This issue is exacerbated in continuous state spaces. This paper presents a straight-forward approach to speeding-up and even improving RL solutions by reusing features learned during a pre-training phase prior to Q-learning. During pre-training, the agent is taught to predict state change given a state/action pair. The effect of pre-training is examined using the model-free Q-learning approach but could readily be applied to a number of RL approaches including model-based RL. The analysis of the results provides ample evidence that the features learned during pre-training is the reason behind the improved RL performance.

Using supervised training signals of observable state dynamics to speed-up and improve reinforcement learning

Reinforcement learning (RL) agents learn by exploring the environment and then exploiting what they have learned. This frees the human trainers from having to know the preferred action or intrinsic value of each encountered state. The cost of this freedom is that RL is slower and more unstable than supervised learning. We explore the possibility that ensemble methods can remedy these shortcomings by investigating a novel technique which harnesses the wisdom of crowds by combining Q-function approximator estimates utilizing a simple combination scheme similar to the supervised learning approach known as bagging. Bagging approaches have not yet found widespread adoption in the RL literature nor has a comprehensive look at its performance been performed. Our results show that the proposed approach improves all three tasks and RL approaches attempted. The primary contribution of this work is a demonstration that the improvement is a direct result of the increased stability of the action portion of the state-action-value function. Subsequent experimentation demonstrates that the stability in learning allows an actor-critic method to find more efficient solutions. Finally we show that this approach can be used to decrease the amount of time necessary to solve problems which require a deep Q-learning (DQN) approach.

/pdf/the-wisdom-of-the-crowd-reliable-deep-reinforcement-learning-16wgbe5ie8.pdf

The Wisdom of the Crowd: Reliable Deep Reinforcement Learning Through Ensembles of Q-Functions.

A study was conducted to determine the ability of neural networks to extract high level control information from cylinder pressure data. Various experiments were performed using neural networks for pattern recognition on a series of data files consisting of cylinder pressure versus crank angle. The goal of these experiments was to estimate spark timing based on the cylinder pressure signature-all other engine parameters were held constant during the data collection process. Test results indicate that an approximate spark time value can be obtained using cylinder pressure data as the inputs to a neural network and spark timing as the output. >

Estimating ignition timing from engine cylinder pressure with neural networks

Modern automobiles have more than 70 electronic control units (ECUs) and 100 million lines of code to improve safety, fuel economy, performance, durability, user experience, and to reduce emissions. Automobiles are becoming increasingly interconnected with the outside world. Consequently, modern day automobiles are becoming more prone to cyber security attacks. Towards this end, we present an approach that uses machine learning to detect abnormal behavior, including malicious ones, on embedded networks in heavy vehicles. Our modular algorithm uses machine learning approaches on the internal network traffic in heavy vehicles to generate warning alarms in real-time. We tested our hypothesis on five separate data logs that characterize the operations of heavy vehicles having different specifications under varying driving conditions. We report a malicious detection rate of 98–99% and a mean accuracy rate of 96–99% across all experiments using five-fold cross-validation. Our analysis also shows that with a small subset of hand-crafted features, the complex dynamic behavior of heavy vehicle ECUs can be predicted and classified as normal or abnormal.

Using Machine Learning to Detect Anomalies in Embedded Networks in Heavy Vehicles

Electroencephalography (EEG) has been used extensively for classifying cognitive tasks. Many investigators have demonstrated classification accuracies well over 90% for some combinations of cognitive tasks, signal transformations, and classification methods. Unfortunately, EEG data is prone to significant interference from a wide variety of artifacts, particularly eye blinks. Most methods for classifying cognitive tasks with EEG data simply discard time windows containing eye blink artifacts. However, future applications of EEG-based cognitive task classification should not be hindered by eye blinks. The value of an EEG-controlled human-computer interface, for instance, would be severely diluted if it did not work in the presence of eye blinks. Fortunately, recent advances in blind signal separation algorithms and their applications to EEG data mitigate the artifact contamination issue. In this paper, we show how independent components analysis (ICA) and its extension for sub-Gaussian sources, extended ICA (eICA), can be applied to accurately classify cognitive tasks with eye blink contaminated EEG recordings.

Charles W. Anderson

Papers

Using supervised training signals of observable state dynamics to speed-up and improve reinforcement learning

The Wisdom of the Crowd: Reliable Deep Reinforcement Learning Through Ensembles of Q-Functions.

Estimating ignition timing from engine cylinder pressure with neural networks

Using Machine Learning to Detect Anomalies in Embedded Networks in Heavy Vehicles

EEG-based cognitive task classification with ICA and neural networks