Setting of the learning problem consistency of learning processes bounds on the rate of convergence of learning processes controlling the generalization ability of learning processes constructing learning algorithms what is important in learning theory?.

The Nature of Statistical Learning Theory

In the first part of the paper we consider the problem of dynamically apportioning resources among a set of options in a worst-case on-line framework. The model we study can be interpreted as a broad, abstract extension of the well-studied on-line prediction model to a general decision-theoretic setting. We show that the multiplicative weight-update Littlestone?Warmuth rule can be adapted to this model, yielding bounds that are slightly weaker in some cases, but applicable to a considerably more general class of learning problems. We show how the resulting learning algorithm can be applied to a variety of problems, including gambling, multiple-outcome prediction, repeated games, and prediction of points in Rn. In the second part of the paper we apply the multiplicative weight-update technique to derive a new boosting algorithm. This boosting algorithm does not require any prior knowledge about the performance of the weak learning algorithm. We also study generalizations of the new boosting algorithm to the problem of learning functions whose range, rather than being binary, is an arbitrary finite set or a bounded segment of the real line.

/pdf/a-decision-theoretic-generalization-of-on-line-learning-and-3ykfx1u7er.pdf

A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting

The use of randomness is now an accepted tool in Theoretical Computer Science but not everyone is aware of the underpinnings of this methodology in Combinatorics - particularly, in what is now called the probabilistic Method as developed primarily by Paul Erdoős over the past half century. Here I will explore a particular set of problems - all dealing with “good” colorings of an underlying set of points relative to a given family of sets. A central point will be the evolution of these problems from the purely existential proofs of Erdős to the algorithmic aspects of much interest to this audience.

/pdf/the-probabilistic-method-5g63k0itke.pdf

The Probabilistic Method

Convergence of Probability Measures. By P. Billingsley. Chichester, Sussex, Wiley, 1968. xii, 253 p. 9 1/4“. 117s.

Convergence of Probability Measures

This book is an introduction to probability theory covering laws of large numbers, central limit theorems, random walks, martingales, Markov chains, ergodic theorems, and Brownian motion. It is a comprehensive treatment concentrating on the results that are the most useful for applications. Its philosophy is that the best way to learn probability is to see it in action, so there are 200 examples and 450 problems.

Probability: Theory and Examples

This classic textbook offers a clear exposition of modern probability theory and of the interplay between the properties of metric spaces and probability measures. The first half of the book gives an exposition of real analysis: basic set theory, general topology, measure theory, integration, an introduction to functional analysis in Banach and Hilbert spaces, convex sets and functions and measure on topological spaces. The second half introduces probability based on measure theory, including laws of large numbers, ergodic theorems, the central limit theorem, conditional expectations and martingale's convergence. A chapter on stochastic processes introduces Brownian motion and the Brownian bridge. The edition has been made even more self-contained than before; it now includes a foundation of the real number system and the Stone-Weierstrass theorem on uniform approximation in algebras of functions. Several other sections have been revised and improved, and the comprehensive historical notes have been further amplified. A number of new exercises have been added, together with hints for solution.

/pdf/real-analysis-and-probability-4vmfl683ga.pdf

Real analysis and probability

Preface 1. Introduction: Donsker's theorem, metric entropy and inequalities 2. Gaussian measures and processes sample continuity 3. Foundations of uniform central limit theorems: Donsker classes 4. Vapnik-Cervonenkis combinatorics 5. Measurability 6. Limit theorems for Vapnik-Cervonenkis and related classes 7. Metric entropy, with inclusion and bracketing 8. Approximation of functions and sets 9. Sums in general Banach spaces and invariance principles 10. Universal and uniform central limit theorems 11. The two-sample case, the bootstrap, and confidence sets 12. Classes of sets or functions too large for central limit theorems Appendices Subject index Author index Index of notation.

/pdf/uniform-central-limit-theorems-2w96qbq58m.pdf

Uniform Central Limit Theorems

The Sizes of Compact Subsets of Hilbert Space and Continuity of Gaussian Processes

Let $(X, \mathscr{A}, P)$ be a probability space. Let $X_1, X_2,\cdots,$ be independent $X$-valued random variables with distribution $P$. Let $P_n := n^{-1}(\delta_{X_1} + \cdots + \delta_{X_n})$ be the empirical measure and let $\nu_n := n^\frac{1}{2}(P_n - P)$. Given a class $\mathscr{C} \subset \mathscr{a}$, we study the convergence in law of $\nu_n$, as a stochastic process indexed by $\mathscr{C}$, to a certain Gaussian process indexed by $\mathscr{C}$. If convergence holds with respect to the supremum norm $\sup_{C \in \mathscr{C}}|f(C)|$, in a suitable (usually nonseparable) function space, we call $\mathscr{C}$ a Donsker class. For measurability, $X$ may be a complete separable metric space, $\mathscr{a} =$ Borel sets, and $\mathscr{C}$ a suitable collection of closed sets or open sets. Then for the Donsker property it suffices that for some $m$, and every set $F \subset X$ with $m$ elements, $\mathscr{C}$ does not cut all subsets of $F$ (Vapnik-Cervonenkis classes). Another sufficient condition is based on metric entropy with inclusion. If $\mathscr{C}$ is a sequence $\{C_m\}$ independent for $P$, then $\mathscr{C}$ is a Donsker class if and only if for some $r, \sigma_m(P(C_m)(1 - P(C_m)))^r < \infty$.

https://projecteuclid.org/download/pdf_1/euclid.aop/1176995384

Richard M. Dudley

Papers

Real analysis and probability

Uniform Central Limit Theorems

The Sizes of Compact Subsets of Hilbert Space and Continuity of Gaussian Processes

Central Limit Theorems for Empirical Measures

A course on empirical processes