Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

https://www2.econ.iastate.edu/tesfatsi/DeepLearningInNeuralNetworksOverview.JSchmidhuber2015.pdf

Deep learning in neural networks

Background on genetic algorithms, LISP, and genetic programming hierarchical problem-solving introduction to automatically-defined functions - the two-boxes problem problems that straddle the breakeven point for computational effort Boolean parity functions determining the architecture of the program the lawnmower problem the bumblebee problem the increasing benefits of ADFs as problems are scaled up finding an impulse response function artificial ant on the San Mateo trail obstacle-avoiding robot the minesweeper problem automatic discovery of detectors for letter recognition flushes and four-of-a-kinds in a pinochle deck introduction to biochemistry and molecular biology prediction of transmembrane domains in proteins prediction of omega loops in proteins lookahead version of the transmembrane problem evolutionary selection of the architecture of the program evolution of primitives and sufficiency evolutionary selection of terminals evolution of closure simultaneous evolution of architecture, primitive functions, terminals, sufficiency, and closure the role of representation and the lens effect Appendices: list of special symbols list of special functions list of type fonts default parameters computer implementation annotated bibliography of genetic programming electronic mailing list and public repository

Genetic Programming: On the Programming of Computers by Means of Natural Selection

Machine Learning is the study of methods for programming computers to learn. Computers are applied to a wide range of tasks, and for most of these it is relatively easy for programmers to design and implement the necessary software. However, there are many tasks for which this is difficult or impossible. These can be divided into four general categories. First, there are problems for which there exist no human experts. For example, in modern automated manufacturing facilities, there is a need to predict machine failures before they occur by analyzing sensor readings. Because the machines are new, there are no human experts who can be interviewed by a programmer to provide the knowledge necessary to build a computer system. A machine learning system can study recorded data and subsequent machine failures and learn prediction rules. Second, there are problems where human experts exist, but where they are unable to explain their expertise. This is the case in many perceptual tasks, such as speech recognition, hand-writing recognition, and natural language understanding. Virtually all humans exhibit expert-level abilities on these tasks, but none of them can describe the detailed steps that they follow as they perform them. Fortunately, humans can provide machines with examples of the inputs and correct outputs for these tasks, so machine learning algorithms can learn to map the inputs to the outputs. Third, there are problems where phenomena are changing rapidly. In finance, for example, people would like to predict the future behavior of the stock market, of consumer purchases, or of exchange rates. These behaviors change frequently, so that even if a programmer could construct a good predictive computer program, it would need to be rewritten frequently. A learning program can relieve the programmer of this burden by constantly modifying and tuning a set of learned prediction rules. Fourth, there are applications that need to be customized for each computer user separately. Consider, for example, a program to filter unwanted electronic mail messages. Different users will need different filters. It is unreasonable to expect each user to program his or her own rules, and it is infeasible to provide every user with a software engineer to keep the rules up-to-date. A machine learning system can learn which mail messages the user rejects and maintain the filtering rules automatically. Machine learning addresses many of the same research questions as the fields of statistics, data mining, and psychology, but with differences of emphasis. Statistics focuses on understanding the phenomena that have generated the data, often with the goal of testing different hypotheses about those phenomena. Data mining seeks to find patterns in the data that are understandable by people. Psychological studies of human learning aspire to understand the mechanisms underlying the various learning behaviors exhibited by people (concept learning, skill acquisition, strategy change, etc.).

Machine learning

Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

Pattern Recognition and Machine Learning

For many institutions of higher learning, the beginning of each semester is marked by a significant migration of young adults into the area. In the midst of the COVID19 pandemic, this presents an opportunity for active cases to be introduced into a community. Prior to the Fall 2020 semester, Colorado State University researchers combined student home locations with recent case counts compiled by the New York Times to assign a probability to each individual of arriving with COVID19. These probabilities were combined to estimate that there would be 7.8 new cases among the on-campus population. Comprehensive testing of arriving students revealed 7 new cases, which validated the approach. The procedure was repeated to explore what could happen if students had returned to campus after Fall break. The estimate of 48 cases corroborated the Universitys early decision to transition to fully remote learning after break.

/pdf/estimating-covid-19-cases-on-university-campuses-prior-to-1k3ft3cjm3.pdf

Estimating COVID-19 Cases on University Campuses Prior To Semester

Determining changes in global temperature and precipitation that may indicate climate change is complicated by annual variations. One approach for ﬁnding potential climate change indicators is to train a model that predicts the year from annual means of global temperatures and precipitations. Such data is available from the CMIP6 ensemble of simulations. Here a two-hidden-layer neural network trained on this data successfully predicts the year. Differences among temperature and precipitation patterns for which the model predicts speciﬁc years reveal changes through time. To ﬁnd these optimal patterns, a new way of interpreting what the neural network has learned is explored. Alopex, a stochastic correlative learning algorithm, is used to ﬁnd optimal temperature and precipitation maps that best predict a given year. These maps are compared over multiple years to show how temperature and precipitations patterns indicative of each year change over time.

An Interpretable Model of Climate Change Using Correlative Learning

Animal migrations are some of the most ubiquitous and one of the most threatened ecological processes. A wide range of migratory behaviours occur in nature, and this behaviour is not uniform between and within species, where even individuals in the same population can exhibit differentiation. While the environment largely drives migratory behaviour, it is necessary to understand the genetic mechanisms influencing migration to elucidate the adaptive potential of migratory species to cope with novel conditions and adapt to environmental change. In this study, we identified genes associated with migratory traits by undertaking pooled genome-wide scans on a natural population of a facultatively migrating ungulate. We identified genomic regions associated with variation in migratory direction in the study population, including FTM1, a gene linked to the formation of lipids, and DPPA3, a gene linked to epigenetic modifications of the maternal line. Such a genetic basis for migratory traits can have implications on heritability of behaviour and the flexibility of individuals to change their behaviour in the face of stochastic changes in their environment.

Genomic correlates for migratory direction in a free-ranging cervid

Reinforcement Learning (RL) is known for its high sample complexity, particularly in sparse reward settings. This is in part due to biased temporal-difference (TD) targets, particularly in the early stages of training. Our work proposes using state change predictions as an unbiased and non-sparse supplement for TD-targets. By training a forward model which shares a Q-network's initial layers, we allow transfer learning from model dynamics prediction to Q value function approximation. We discuss two variants, one doing this only in the initial steps of the training and another one using it throughout the training process. Both variants can be used as enhancements to state-of-the-art RL algorithms. Our results show that enhancing Double DQN (DDQN) and TD3 with this approach outperforms vanilla versions when applied to the Acrobot, MountainCar, and Cartpole RL benchmarks for DDQN and the HalfCheetah and Walker2D RL benchmarks for TD3. This result is particularly significant as it shows the power of transfer learning even without a priori knowledge.

Increased Reinforcement Learning Performance through Transfer of Representation Learned by State Prediction Model

Land managers in western North America often reverse succession by removing pinyon (Pinus spp.) and juniper (Juniperus spp.) trees to reduce fire risk and increase forage for wildlife and livestock. Because prescribed fire carries inherent risks, mechanical methods such as chaining, roller-chopping, and mastication are often used. Mechanical methods differ in cost and the size of woody debris produced, and may differentially impact plant and animal responses. We implemented a randomized, complete-block, split-plot experiment in December 2011 in the Piceance Basin, northwestern Colorado, USA, to compare mechanical methods and to explore seeding (subplot) interactions. We assessed vegetation 1-, 2-, 5-, and 6-years post-treatment, and mule deer (Odocoileus hemionus) response via GPS locations 3–8 years post-treatment. By 2016, treated plots had 3–5 times higher perennial grass cover and ~10 times higher cheatgrass (Bromus tectorum) cover than untreated control plots. Roller-chopped plots had both the highest non-native annual forb cover, and when seeded, the highest density of bitterbrush (Purshia tridentata), a nutritious shrub used by mule deer. Masticated plots had higher bitterbrush use during summer and fall, leaving less forage available for winter. Days of winter mule deer use from GPS point locations in chained and roller-chopped plots was ~70% higher than in control plots, while winter use in masticated plots was similar to control plots. Mule deer use appears related to a combination of hiding cover, resulting from residual woody debris, and winter forage availability. Roller-chopped plots provide the best combination of hiding cover and winter forage, but mastication or chaining, applied leaving dispersed security cover, may be better options at large scales or when invasive species concerns exist. 

https://onlinelibrary.wiley.com/doi/pdfdirect/10.1002/wsb.1421

Charles W. Anderson

Papers

Estimating COVID-19 Cases on University Campuses Prior To Semester

An Interpretable Model of Climate Change Using Correlative Learning

Genomic correlates for migratory direction in a free-ranging cervid

Increased Reinforcement Learning Performance through Transfer of Representation Learned by State Prediction Model

Plant and mule deer responses to pinyon‐juniper removal by three mechanical methods