Today's Web-enabled deluge of electronic data calls for automated methods of data analysis. Machine learning provides these, developing methods that can automatically detect patterns in data and then use the uncovered patterns to predict future data. This textbook offers a comprehensive and self-contained introduction to the field of machine learning, based on a unified, probabilistic approach. The coverage combines breadth and depth, offering necessary background material on such topics as probability, optimization, and linear algebra as well as discussion of recent developments in the field, including conditional random fields, L1 regularization, and deep learning. The book is written in an informal, accessible style, complete with pseudo-code for the most important algorithms. All topics are copiously illustrated with color images and worked examples drawn from such application domains as biology, text processing, computer vision, and robotics. Rather than providing a cookbook of different heuristic methods, the book stresses a principled model-based approach, often using the language of graphical models to specify models in a concise and intuitive way. Almost all the models described have been implemented in a MATLAB software package--PMTK (probabilistic modeling toolkit)--that is freely available online. The book is suitable for upper-level undergraduates with an introductory-level college math background and beginning graduate students.

Machine Learning : A Probabilistic Perspective

Conventionally, engineers have employed rigid materials to fabricate precise, predictable robotic systems, which are easily modelled as rigid members connected at discrete joints. Natural systems, however, often match or exceed the performance of robotic systems with deformable bodies. Cephalopods, for example, achieve amazing feats of manipulation and locomotion without a skeleton; even vertebrates such as humans achieve dynamic gaits by storing elastic energy in their compliant bones and soft tissues. Inspired by nature, engineers have begun to explore the design and control of soft-bodied robots composed of compliant materials. This Review discusses recent developments in the emerging field of soft robotics.

/pdf/design-fabrication-and-control-of-soft-robots-51z07seyl1.pdf

Design, fabrication and control of soft robots

There are many excellent toolkits which provide support for developing machine learning software in Python, R, Matlab, and similar environments. Dlib-ml is an open source library, targeted at both engineers and research scientists, which aims to provide a similarly rich environment for developing machine learning software in the C++ language. Towards this end, dlib-ml contains an extensible linear algebra toolkit with built in BLAS support. It also houses implementations of algorithms for performing inference in Bayesian networks and kernel-based methods for classification, regression, clustering, anomaly detection, and feature ranking. To enable easy use of these tools, the entire library has been developed with contract programming, which provides complete and precise documentation as well as powerful debugging tools.

/pdf/dlib-ml-a-machine-learning-toolkit-38ngf0r5xn.pdf

Dlib-ml: A Machine Learning Toolkit

In this paper we consider deterministic policy gradient algorithms for reinforcement learning with continuous actions. The deterministic policy gradient has a particularly appealing form: it is the expected gradient of the action-value function. This simple form means that the deterministic policy gradient can be estimated much more efficiently than the usual stochastic policy gradient. To ensure adequate exploration, we introduce an off-policy actor-critic algorithm that learns a deterministic target policy from an exploratory behaviour policy. We demonstrate that deterministic policy gradient algorithms can significantly outperform their stochastic counterparts in high-dimensional action spaces.

/pdf/deterministic-policy-gradient-algorithms-3p1p5nnfpn.pdf

Deterministic Policy Gradient Algorithms

A broad range of neural and behavioral data suggests that the brain contains multiple systems for behavioral choice, including one associated with prefrontal cortex and another with dorsolateral striatum. However, such a surfeit of control raises an additional choice problem: how to arbitrate between the systems when they disagree. Here, we consider dual-action choice systems from a normative perspective, using the computational theory of reinforcement learning. We identify a key trade-off pitting computational simplicity against the flexible and statistically efficient use of experience. The trade-off is realized in a competition between the dorsolateral striatal and prefrontal systems. We suggest a Bayesian principle of arbitration between them according to uncertainty, so each controller is deployed when it should be most accurate. This provides a unifying account of a wealth of experimental evidence about the factors favoring dominance by either system.

/pdf/uncertainty-based-competition-between-prefrontal-and-s5b6rfvmil.pdf

Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control

We present a nonlinear version of the recursive least squares (RLS) algorithm. Our algorithm performs linear regression in a high-dimensional feature space induced by a Mercer kernel and can therefore be used to recursively construct minimum mean-squared-error solutions to nonlinear least-squares problems that are frequently encountered in signal processing applications. In order to regularize solutions and keep the complexity of the algorithm bounded, we use a sequential sparsification process that admits into the kernel representation a new input sample only if its feature space image cannot be sufficiently well approximated by combining the images of previously admitted samples. This sparsification procedure allows the algorithm to operate online, often in real time. We analyze the behavior of the algorithm, compare its scaling properties to those of support vector machines, and demonstrate its utility in solving two signal processing problems-time-series prediction and channel equalization.

The kernel recursive least-squares algorithm

Gaussian Process Temporal Difference (GPTD) learning offers a Bayesian solution to the policy evaluation problem of reinforcement learning. In this paper we extend the GPTD framework by addressing two pressing issues, which were not adequately treated in the original GPTD paper (Engel et al., 2003). The first is the issue of stochasticity in the state transitions, and the second is concerned with action selection and policy improvement. We present a new generative model for the value function, deduced from its relation with the discounted return. We derive a corresponding on-line algorithm for learning the posterior moments of the value Gaussian process. We also present a SARSA based extension of GPTD, termed GPSARSA, that allows the selection of actions and the gradual improvement of policies without requiring a world-model.

Reinforcement learning with Gaussian processes

We present a novel Bayesian approach to the problem of value function estimation in continuous state spaces. We define a probabilistic generative model for the value function by imposing a Gaussian prior over value functions and assuming a Gaussian noise model. Due to the Gaussian nature of the random processes involved, the posterior distribution of the value function is also Gaussian and is therefore described entirely by its mean and covariance. We derive exact expressions for the posterior process moments, and utilizing an efficient sequential sparsification method, we describe an on-line algorithm for learning them. We demonstrate the operation of the algorithm on a 2-dimensional continuous spatial navigation domain.

/pdf/bayes-meets-bellman-the-gaussian-process-approach-to-504wehcpr8.pdf

Bayes meets bellman: the Gaussian process approach to temporal difference learning

The octopus arm requires special motor control schemes because it consists almost entirely of muscles and lacks a rigid skeletal support. Here we present a 2D dynamic model of the octopus arm to explore possible strategies of movement control in this muscular hydrostat. The arm is modeled as a multisegment structure, each segment containing longitudinal and transverse muscles and maintaining a constant volume, a prominent feature of muscular hydrostats. The input to the model is the degree of activation of each of its muscles. The model includes the external forces of gravity, buoyancy, and water drag forces (experimentally estimated here). It also includes the internal forces generated by the arm muscles and the forces responsible for maintaining a constant volume. Using this dynamic model to investigate the octopus reaching movement and to explore the mechanisms of bend propagation that characterize this movement, we found the following. 1) A simple command producing a wave of muscle activation moving at a constant velocity is sufficient to replicate the natural reaching movements with similar kinematic features. 2) The biomechanical mechanism that produces the reaching movement is a stiffening wave of muscle contraction that pushes a bend forward along the arm. 3) The perpendicular drag coefficient for an octopus arm is nearly 50 times larger than the tangential drag coefficient. During a reaching movement, only a small portion of the arm is oriented perpendicular to the direction of movement, thus minimizing the drag force.

https://www.physiology.org/doi/pdf/10.1152/jn.00684.2004

Dynamic Model of the Octopus Arm. I. Biomechanics of the Octopus Reaching Movement

We present a novel algorithm for sparse online greedy kernel-based nonlinear regression. This algorithm improves current approaches to kernel-based regression in two aspects. First, it operates online - at each time step it observes a single new input sample, performs an update and discards it. Second, the solution maintained is extremely sparse. This is achieved by an explicit greedy sparsification process that admits into the kernel representation a new input sample only if its feature space image is linearly independent of the images of previously admitted samples. We show that the algorithm implements a form of gradient ascent and demonstrate its scaling and noise tolerance properties on three benchmark regression problems.

Yaakov Engel

Papers

The kernel recursive least-squares algorithm

Reinforcement learning with Gaussian processes

Bayes meets bellman: the Gaussian process approach to temporal difference learning

Dynamic Model of the Octopus Arm. I. Biomechanics of the Octopus Reaching Movement

Sparse online greedy support vector regression