Kernel methods provide a powerful and unified framework for pattern discovery, motivating algorithms that can act on general types of data (e.g. strings, vectors or text) and look for general types of relations (e.g. rankings, classifications, regressions, clusters). The application areas range from neural networks and pattern recognition to machine learning and data mining. This book, developed from lectures and tutorials, fulfils two major roles: firstly it provides practitioners with a large toolkit of algorithms, kernels and solutions ready to use for standard pattern discovery problems in fields such as bioinformatics, text analysis, image analysis. Secondly it provides an easy introduction for students and researchers to the growing field of kernel-based pattern analysis, demonstrating with examples how to handcraft an algorithm or a kernel for a new specific application, and covering all the necessary conceptual and mathematical tools to do so.

/pdf/kernel-methods-for-pattern-analysis-3ta60xrj0i.pdf

Kernel Methods for Pattern Analysis

Machine learning is one of the fastest growing areas of computer science, with far-reaching applications. The aim of this textbook is to introduce machine learning, and the algorithmic paradigms it offers, in a principled way. The book provides an extensive theoretical account of the fundamental ideas underlying machine learning and the mathematical derivations that transform these principles into practical algorithms. Following a presentation of the basics of the field, the book covers a wide array of central topics that have not been addressed by previous textbooks. These include a discussion of the computational complexity of learning and the concepts of convexity and stability; important algorithmic paradigms including stochastic gradient descent, neural networks, and structured output learning; and emerging theoretical concepts such as the PAC-Bayes approach and compression-based bounds. Designed for an advanced undergraduate or beginning graduate course, the text makes the fundamentals and algorithms of machine learning accessible to students and non-expert readers in statistics, computer science, mathematics, and engineering.

/pdf/understanding-machine-learning-from-theory-to-algorithms-5aeszqx33g.pdf

Understanding Machine Learning: From Theory To Algorithms

We investigate the use of certain data-dependent estimates of the complexity of a function class, called Rademacher and Gaussian complexities. In a decision theoretic setting, we prove general risk bounds in terms of these complexities. We consider function classes that can be expressed as combinations of functions from basis classes and show how the Rademacher and Gaussian complexities of such a function class can be bounded in terms of the complexity of the basis classes. We give examples of the application of these techniques in finding data-dependent risk bounds for decision trees, neural networks and support vector machines.

/pdf/rademacher-and-gaussian-complexities-risk-bounds-and-3iqch85980.pdf

Rademacher and gaussian complexities: risk bounds and structural results

This graduate-level textbook introduces fundamental concepts and methods in machine learning. It describes several important modern algorithms, provides the theoretical underpinnings of these algorithms, and illustrates key aspects for their application. The authors aim to present novel theoretical tools and concepts while giving concise proofs even for relatively advanced topics. Foundations of Machine Learning fills the need for a general textbook that also offers theoretical details and an emphasis on proofs. Certain topics that are often treated with insufficient attention are discussed in more detail here; for example, entire chapters are devoted to regression, multi-class classification, and ranking. The first three chapters lay the theoretical foundation for what follows, but each remaining chapter is mostly self-contained. The appendix offers a concise probability review, a short introduction to convex optimization, tools for concentration bounds, and several basic properties of matrices and norms used in the book. The book is intended for graduate students and researchers in machine learning, statistics, and related areas; it can be used either as a textbook or as a reference text for a research seminar.

Foundations of Machine Learning

Boosting is a general method for improving the accuracy of any given learning algorithm. Focusing primarily on the AdaBoost algorithm, this chapter overviews some of the recent work on boosting including analyses of AdaBoost’s training error and generalization error; boosting’s connection to game theory and linear programming; the relationship between boosting and logistic regression; extensions of AdaBoost for multiclass classification problems; methods of incorporating human knowledge into boosting; and experimental and applied work using boosting.

/pdf/the-boosting-approach-to-machine-learning-an-overview-1n76x70jzh.pdf

The Boosting Approach to Machine Learning An Overview

We construct data dependent upper bounds on the risk in function learning problems. The bounds are based on local norms of the Rademacher process indexed by the underlying function class, and they do not require prior knowledge about the distribution of training examples or any specific properties of the function class. Using Talagrand’s type concentration inequalities for empirical and Rademacher processes, we show that the bounds hold with high probability that decreases exponentially fast when the sample size grows. In typical situations that are frequently encountered in the theory of function learning, the bounds give nearly optimal rate of convergence of the risk to zero.

Rademacher Processes and Bounding the Risk of Function Learning

Let H :L 2(S,cal S,P)→L 2(S,cal S,P) be a compact integral operator with a symmetric kernel h. Let X i ,i∈N , be independent S-valued random variables with common probability law P. Consider the n×n matrix H ˜ n with entries n - 1 h(X i,X j),1≤i,j≤n (this is the matrix of an empirical version of the operator H with P replaced by the empirical measure Pn), and let Hn denote the modification of H˜ n , obtained by deleting its diagonal. It is proved that the l 2 distance between the ordered spectrum of Hn and the ordered spectrum of H tends to zero a.s. if and only if H is Hilbert-Schmidt. Rates of convergence and distributional limit theorems for the difference between the ordered spectra of the operators Hn (or H˜ n ) and H are also obtained under somewhat stronger conditions. These results apply in particular to the kernels of certain functions H =varphi(L) of partial differential operators L (heat kernels, Green functions).

/pdf/random-matrix-approximation-of-spectra-of-integral-operators-2nerhff16t.pdf

Random matrix approximation of spectra of integral operators

Probabilistic methods and statistical learning theory have been shown to provide approximate solutions to "difficult" control problems. Unfortunately, the number of samples required in order to guarantee stringent performance levels may be prohibitively large. This paper introduces bootstrap learning methods and the concept of stopping times to drastically reduce the bound on the number of samples required to achieve a performance level. We then apply these results to obtain more efficient algorithms which probabilistically guarantee stability and robustness levels when designing controllers for uncertain systems.

/pdf/improved-sample-complexity-estimates-for-statistical-3nfe4q6h2z.pdf

Improved sample complexity estimates for statistical learning control of uncertain systems

fMRI pattern classification using neuroanatomically constrained boosting

Given a probability space (S, S, P) and a symmetric measurable real valued kernel h on S × S, which defines a compact integral operator H from L 2(P) into L 2(P), we study the asymptotic behavior of spectral projections of the random n × n matrices H n with entries n −1h(X i,X j), 1 ≤ i,j ≤ n, and H n, obtained from H n by deleting of its diagonal. Here (X 1,… , X n) is a sample of independent random variables in (S, S) with common distribution P. We show that if H is a Hilbert-Schmidt operator, then the spectral projections of H n converge a.s. to the spectral projections of H (the convergence is understood in the sense of quadratic forms). Under slightly stronger assumptions, the convergence also holds for the spectral projections of H n. Moreover, under much stronger assumptions on the kernel h, we show that the fluctuations of the spectral projections of H n or H n are asymptotically Gaussian.

Vladimir Koltchinskii

Papers

Rademacher Processes and Bounding the Risk of Function Learning

Random matrix approximation of spectra of integral operators

Improved sample complexity estimates for statistical learning control of uncertain systems

fMRI pattern classification using neuroanatomically constrained boosting

Asymptotics of Spectral Projections of Some Random Matrices Approximating Integral Operators