Today's Web-enabled deluge of electronic data calls for automated methods of data analysis. Machine learning provides these, developing methods that can automatically detect patterns in data and then use the uncovered patterns to predict future data. This textbook offers a comprehensive and self-contained introduction to the field of machine learning, based on a unified, probabilistic approach. The coverage combines breadth and depth, offering necessary background material on such topics as probability, optimization, and linear algebra as well as discussion of recent developments in the field, including conditional random fields, L1 regularization, and deep learning. The book is written in an informal, accessible style, complete with pseudo-code for the most important algorithms. All topics are copiously illustrated with color images and worked examples drawn from such application domains as biology, text processing, computer vision, and robotics. Rather than providing a cookbook of different heuristic methods, the book stresses a principled model-based approach, often using the language of graphical models to specify models in a concise and intuitive way. Almost all the models described have been implemented in a MATLAB software package--PMTK (probabilistic modeling toolkit)--that is freely available online. The book is suitable for upper-level undergraduates with an introductory-level college math background and beginning graduate students.

Machine Learning : A Probabilistic Perspective

A comprehensive review of spatiotemporal pattern formation in systems driven away from equilibrium is presented, with emphasis on comparisons between theory and quantitative experiments. Examples include patterns in hydrodynamic systems such as thermal convection in pure fluids and binary mixtures, Taylor-Couette flow, parametric-wave instabilities, as well as patterns in solidification fronts, nonlinear optics, oscillatory chemical reactions and excitable biological media. The theoretical starting point is usually a set of deterministic equations of motion, typically in the form of nonlinear partial differential equations. These are sometimes supplemented by stochastic terms representing thermal or instrumental noise, but for macroscopic systems and carefully designed experiments the stochastic forces are often negligible. An aim of theory is to describe solutions of the deterministic equations that are likely to be reached starting from typical initial conditions and to persist at long times. A unified description is developed, based on the linear instabilities of a homogeneous state, which leads naturally to a classification of patterns in terms of the characteristic wave vector q0 and frequency ω0 of the instability. Type Is systems (ω0=0, q0≠0) are stationary in time and periodic in space; type IIIo systems (ω0≠0, q0=0) are periodic in time and uniform in space; and type Io systems (ω0≠0, q0≠0) are periodic in both space and time. Near a continuous (or supercritical) instability, the dynamics may be accurately described via "amplitude equations," whose form is universal for each type of instability. The specifics of each system enter only through the nonuniversal coefficients. Far from the instability threshold a different universal description known as the "phase equation" may be derived, but it is restricted to slow distortions of an ideal pattern. For many systems appropriate starting equations are either not known or too complicated to analyze conveniently. It is thus useful to introduce phenomenological order-parameter models, which lead to the correct amplitude equations near threshold, and which may be solved analytically or numerically in the nonlinear regime away from the instability. The above theoretical methods are useful in analyzing "real pattern effects" such as the influence of external boundaries, or the formation and dynamics of defects in ideal structures. An important element in nonequilibrium systems is the appearance of deterministic chaos. A greal deal is known about systems with a small number of degrees of freedom displaying "temporal chaos," where the structure of the phase space can be analyzed in detail. For spatially extended systems with many degrees of freedom, on the other hand, one is dealing with spatiotemporal chaos and appropriate methods of analysis need to be developed. In addition to the general features of nonequilibrium pattern formation discussed above, detailed reviews of theoretical and experimental work on many specific systems are presented. These include Rayleigh-Benard convection in a pure fluid, convection in binary-fluid mixtures, electrohydrodynamic convection in nematic liquid crystals, Taylor-Couette flow between rotating cylinders, parametric surface waves, patterns in certain open flow systems, oscillatory chemical reactions, static and dynamic patterns in biological media, crystallization fronts, and patterns in nonlinear optics. A concluding section summarizes what has and has not been accomplished, and attempts to assess the prospects for the future.

/pdf/pattern-formation-outside-of-equilibrium-1601b9znum.pdf

Pattern formation outside of equilibrium

We present two classes of improved estimators for mutual information M(X,Y), from samples of random points distributed according to some joint probability density mu(x,y). In contrast to conventional estimators based on binnings, they are based on entropy estimates from k -nearest neighbor distances. This means that they are data efficient (with k=1 we resolve structures down to the smallest possible scales), adaptive (the resolution is higher where data are more numerous), and have minimal bias. Indeed, the bias of the underlying entropy estimates is mainly due to nonuniformity of the density at the smallest resolved scale, giving typically systematic errors which scale as functions of k/N for N points. Numerically, we find that both families become exact for independent distributions, i.e. the estimator M(X,Y) vanishes (up to statistical fluctuations) if mu(x,y)=mu(x)mu(y). This holds for all tested marginal distributions and for all dimensions of x and y. In addition, we give estimators for redundancies between more than two random variables. We compare our algorithms in detail with existing algorithms. Finally, we demonstrate the usefulness of our estimators for assessing the actual independence of components obtained from independent component analysis (ICA), for improving ICA, and for estimating the reliability of blind source separation.

Estimating mutual information.

Controlling chaos

A practical method for calculating largest Lyapunov exponents from small data sets

This article presents techniques for constructing classifiers that combine statistical information from training data with tangent approximations to known transformations; it demonstrates the techniques by applying them to a face recognition task. Our approach is to build Bayes classifiers with approximate class-conditional probability densities for measured data. The high dimension of the measurements in modern classification problems such as speech or image recognition makes inferring probability densities from feasibly sized training datasets difficult. We address the difficulty by imposing severely simplifying assumptions and exploiting a priori information about transformations to which classification should be invariant. For the face recognition task, we used a five-parameter group of such transformations consisting of rotation, shifts, and scalings. On the face recognition task, a classifier based on our techniques has an error rate that is 20% lower than that of the best algorithm in a reference s...

Classification Modulo Invariance, With Application to Face Recognition

Methods, apparatus and systems for detecting charged particles and obtaining tomography of a volume by measuring charged particles including measuring the momentum of a charged particle passing through a charged particle detector. Sets of position sensitive detectors measure scattering of the charged particle. The position sensitive detectors having sufficient mass to cause the charged particle passing through the position sensitive detectors to scatter in the position sensitive detectors. A controller can be adapted and arranged to receive scattering measurements of the charged particle from the charged particle detector, determine at least one trajectory of the charged particle from the measured scattering; and determine at least one momentum measurement of the charged particle from the at least one trajectory. The charged particle can be a cosmic ray-produced charged particle, such as a cosmic ray-produced muon. The position sensitive detectors can be drift cells, such as gas-filled drift tubes.

Measuring momentum for charged particle tomography

Increasing amounts of data obtained from high-throughput experiments in molecular biology require new analysis methods. One such high-throughput technology uses microarrays for the simultaneous measurement of expression levels of thousands of genes. Measuring the expression of many genes, even whole genomes, has proven useful for understanding the molecular basis of diseases and has allowed for a more systems level view of cellular processes. Besides the statistical analysis of such large amounts of data, another challenge is the biological interpretation of the analysis results. Biological experts trying to understand the biological meaning of the expression results are often overwhelmed by the amount of functional information in the literature and databases. 
We developed algorithms that address both of these challenges. The Clustering in SVD Subspace (CSS) algorithm identifies similarly regulated genes in time series gene expression data by identifying gene clusters in two-dimensional Singular Value Decomposition (SVD) subspaces. The MeSH Functional Theme Finder (MFTF) algorithm was developed for the discovery of biomedical functional themes associated in the literature with groups of genes or proteins, e.g. as obtained from high-throughput experiments. The CSS algorithm has been applied to two expression data sets, one obtained during the yeast cell-cycle and the second after herpes cytomegalovirus infection of human fibroblast cells. The algorithm successfully identified clusters of genes whose expression is similarly regulated during the respective cellular processes. The MFTF algorithm was applied to the gene clusters identified via the CSS algorithm in the herpes data set. The MFTF algorithm identified, in an automated way, the same main functional themes that were identified by a biological expert. In addition, the algorithm identified new relevant functional themes associated with the gene clusters. Finally, a large-scale validation of the MFTF algorithm is presented. The vector space model underlying the MFTF algorithm was used to correctly classify a large number of proteins into families of functionally related proteins, proving that keywords from literature can be used to capture functional relationships of proteins or genes.

Multivariate analysis of gene expression data and functional information: automated methods for functional genomics

We present a method for detecting a large number of moving targets, such as cars and people, in geographically
referenced video. The problem is difficult, due to the large and variable number of targets which enter and leave the field
of view, and due to imperfect geo-projection and registration. In our method, we assume feature extraction produces a
collection of candidate locations (points in 2D space) for each frame. Some of these locations are real objects, but many
are false alarms. Typical feature extraction might be frame differencing, or target recognition. For each candidate
location, and at each time step, our algorithm outputs a velocity estimate and confidence which can be thresholded to
detect objects with constant velocity. In this paper we derive the algorithm, investigate the free parameters, and compare
its performance to a multi-target tracking algorithm.

/pdf/a-recurrent-velocity-filter-for-detecting-large-numbers-of-1i07k3evaq.pdf

Andrew M. Fraser

Papers

Classification Modulo Invariance, With Application to Face Recognition

Measuring momentum for charged particle tomography

Multivariate analysis of gene expression data and functional information: automated methods for functional genomics

A recurrent velocity filter for detecting large numbers of moving objects

The F-UNCLE Project: Functional UNcertainty Constrained by Law and Experiment