Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

Computational properties of use of biological organisms or to the construction of computers can emerge as collective properties of systems having a large number of simple equivalent components (or neurons). The physical meaning of content-addressable memory is described by an appropriate phase space flow of the state of a system. A model of such a system is given, based on aspects of neurobiology but readily adapted to integrated circuits. The collective properties of this model produce a content-addressable memory which correctly yields an entire memory from any subpart of sufficient size. The algorithm for the time evolution of the state of the system is based on asynchronous parallel processing. Additional emergent collective properties include some capacity for generalization, familiarity recognition, categorization, error correction, and time sequence retention. The collective properties are only weakly sensitive to details of the modeling or the failure of individual devices.

https://www.pnas.org/content/pnas/79/8/2554.full.pdf

Neural networks and physical systems with emergent collective computational abilities

Many areas of science depend on exploratory data analysis and visualization. The need to analyze large amounts of multivariate data raises the fundamental problem of dimensionality reduction: how to discover compact representations of high-dimensional data. Here, we introduce locally linear embedding (LLE), an unsupervised learning algorithm that computes low-dimensional, neighborhood-preserving embeddings of high-dimensional inputs. Unlike clustering methods for local dimensionality reduction, LLE maps its inputs into a single global coordinate system of lower dimensionality, and its optimizations do not involve local minima. By exploiting the local symmetries of linear reconstructions, LLE is able to learn the global structure of nonlinear manifolds, such as those generated by images of faces or documents of text.

/pdf/nonlinear-dimensionality-reduction-by-locally-linear-vmiazg4oee.pdf

Nonlinear dimensionality reduction by locally linear embedding.

https://www2.econ.iastate.edu/tesfatsi/DeepLearningInNeuralNetworksOverview.JSchmidhuber2015.pdf

Deep learning in neural networks

Clustering is the unsupervised classification of patterns (observations, data items, or feature vectors) into groups (clusters). The clustering problem has been addressed in many contexts and by researchers in many disciplines; this reflects its broad appeal and usefulness as one of the steps in exploratory data analysis. However, clustering is a difficult problem combinatorially, and differences in assumptions and contexts in different communities has made the transfer of useful generic concepts and methodologies slow to occur. This paper presents an overview of pattern clustering methods from a statistical pattern recognition perspective, with a goal of providing useful advice and references to fundamental concepts accessible to the broad community of clustering practitioners. We present a taxonomy of clustering techniques, and identify cross-cutting themes and recent advances. We also describe some important applications of clustering algorithms such as image segmentation, object recognition, and information retrieval.

/pdf/data-clustering-a-review-4c452f5cab.pdf

Data clustering: a review

Statistical Aspects of the WEBSOM System in Organizing Document Collections

The Self-Organising Map (SOM) is an Artificial Neural Network (ANN) model consisting of a regular grid of processing units. A model of some multidimensional observation, e.g. a class of digital images, is associated with each unit. The map attempts to represent all the available observations using a restricted set of models. In unsupervised learning, the models become ordered on the grid so that similar models are close to each other. We review here the objective functions and learning rules related to the SOM, starting from vector coding based on a Euclidean metric and extending the theory of arbitrary metrics and to a subspace formalism, in which each SOM unit represents a subspace of the observation space. It is shown that this Adaptive-Subspace SOM (ASSOM) is able to create sets of wavelet- and Gabor-type filters when randomly displaced or moving input patterns are used as training data. No analytical functional form for these filters is thereby postulated. The same kind of adaptive system can create many other kinds of invariant visual filters, like rotation or scale-invariant filters, if there exist corresponding transformations in the training data. The ASSOM system can act as a learning feature-extraction stage for pattern recognisers, being able to adapt to arbitrary sensory environments. We then show that the invariant Gabor features can be effectively used in face recognition, whereby the sets of Gabor filter outputs are coded with the SOM and a face is represented by the histogram over the SOM units.

Visual feature analysis by the self-organising maps

Learning ″Semantotopic Maps″ from Context

In this paper we have applied the Learning Vector Quantiza-tion methods, including the latest developments 1, 3, 2, 4], to the task of Finnish speaker-dependent speech recognition. The main objective was to study the eeect of radically increasing the dimensionality of the context vectors. The high-dimensional feature vectors in our work represent the whole phoneme and they are formed by both averaging and concatenating short-time feature vectors within a time domain window. Excellent results are achieved in separate phoneme classiication of Finnish speech. Moreover, we also show how this method can be applied in combined labeling and segmentation of continuos speech. In this task we use an additional segmentation LVQ-codebook, and combine the information using HMMs.

LVQ-based speech recognition with high-dimensional context vectors.

Over the years, many divergent meanings have been associated with the term selforganization, e.g. automatic creation of structured systems and optimization of parameters in adaptive learning. In th...

Teuvo Kohonen

Papers

Statistical Aspects of the WEBSOM System in Organizing Document Collections

Visual feature analysis by the self-organising maps

Learning ″Semantotopic Maps″ from Context

LVQ-based speech recognition with high-dimensional context vectors.

Self-organized maps of sensory events.