This paper demonstrates theoretically and empirically that a greedy algorithm called orthogonal matching pursuit (OMP) can reliably recover a signal with m nonzero entries in dimension d given O(m ln d) random linear measurements of that signal. This is a massive improvement over previous results, which require O(m2) measurements. The new results for OMP are comparable with recent results for another approach called basis pursuit (BP). In some settings, the OMP algorithm is faster and easier to implement, so it is an attractive alternative to BP for signal recovery problems.

/pdf/signal-recovery-from-random-measurements-via-orthogonal-5adhd8dwmq.pdf

Signal Recovery From Random Measurements Via Orthogonal Matching Pursuit

This report demonstrates theoretically and empirically that a greedy algorithm called
Orthogonal Matching Pursuit (OMP) can reliably recover a signal with m nonzero entries in dimension
d given O(mln d) random linear measurements of that signal. This is a massive improvement
over previous results, which require O(m2) measurements. The new results for OMP are comparable
with recent results for another approach called Basis Pursuit (BP). In some settings, the
OMP algorithm is faster and easier to implement, so it is an attractive alternative to BP for signal
recovery problems.

/pdf/signal-recovery-from-random-measurements-via-orthogonal-2uks7d8txo.pdf

Signal Recovery from Random Measurements Via Orthogonal Matching Pursuit: The Gaussian Case

Kernel methods provide a powerful and unified framework for pattern discovery, motivating algorithms that can act on general types of data (e.g. strings, vectors or text) and look for general types of relations (e.g. rankings, classifications, regressions, clusters). The application areas range from neural networks and pattern recognition to machine learning and data mining. This book, developed from lectures and tutorials, fulfils two major roles: firstly it provides practitioners with a large toolkit of algorithms, kernels and solutions ready to use for standard pattern discovery problems in fields such as bioinformatics, text analysis, image analysis. Secondly it provides an easy introduction for students and researchers to the growing field of kernel-based pattern analysis, demonstrating with examples how to handcraft an algorithm or a kernel for a new specific application, and covering all the necessary conceptual and mathematical tools to do so.

/pdf/kernel-methods-for-pattern-analysis-3ta60xrj0i.pdf

Kernel Methods for Pattern Analysis

Couplings and changes of variables.- Three examples of coupling techniques.- The founding fathers of optimal transport.- Qualitative description of optimal transport.- Basic properties.- Cyclical monotonicity and Kantorovich duality.- The Wasserstein distances.- Displacement interpolation.- The Monge-Mather shortening principle.- Solution of the Monge problem I: global approach.- Solution of the Monge problem II: Local approach.- The Jacobian equation.- Smoothness.- Qualitative picture.- Optimal transport and Riemannian geometry.- Ricci curvature.- Otto calculus.- Displacement convexity I.- Displacement convexity II.- Volume control.- Density control and local regularity.- Infinitesimal displacement convexity.- Isoperimetric-type inequalities.- Concentration inequalities.- Gradient flows I.- Gradient flows II: Qualitative properties.- Gradient flows III: Functional inequalities.- Synthetic treatment of Ricci curvature.- Analytic and synthetic points of view.- Convergence of metric-measure spaces.- Stability of optimal transport.- Weak Ricci curvature bounds I: Definition and Stability.- Weak Ricci curvature bounds II: Geometric and analytic properties.

Optimal Transport: Old and New

Introduction to support vector learning roadmap. Part 1 Theory: three remarks on the support vector method of function estimation, Vladimir Vapnik generalization performance of support vector machines and other pattern classifiers, Peter Bartlett and John Shawe-Taylor Bayesian voting schemes and large margin classifiers, Nello Cristianini and John Shawe-Taylor support vector machines, reproducing kernel Hilbert spaces, and randomized GACV, Grace Wahba geometry and invariance in kernel based methods, Christopher J.C. Burges on the annealed VC entropy for margin classifiers - a statistical mechanics study, Manfred Opper entropy numbers, operators and support vector kernels, Robert C. Williamson et al. Part 2 Implementations: solving the quadratic programming problem arising in support vector classification, Linda Kaufman making large-scale support vector machine learning practical, Thorsten Joachims fast training of support vector machines using sequential minimal optimization, John C. Platt. Part 3 Applications: support vector machines for dynamic reconstruction of a chaotic system, Davide Mattera and Simon Haykin using support vector machines for time series prediction, Klaus-Robert Muller et al pairwise classification and support vector machines, Ulrich Kressel. Part 4 Extensions of the algorithm: reducing the run-time complexity in support vector machines, Edgar E. Osuna and Federico Girosi support vector regression with ANOVA decomposition kernels, Mark O. Stitson et al support vector density estimation, Jason Weston et al combining support vector and mathematical programming methods for classification, Bernhard Scholkopf et al.

Advances in kernel methods: support vector learning

Notation.- 0. Isoperimetric Background and Generalities.- 1. Isoperimetric Inequalities and the Concentration of Measure Phenomenon.- 2. Generalities on Banach Space Valued Random Variables and Random Processes.- I. Banach Space Valued Random Variables and Their Strong Limiting Properties.- 3. Gaussian Random Variables.- 4. Rademacher Averages.- 5. Stable Random Variables.- 6 Sums of Independent Random Variables.- 7. The Strong Law of Large Numbers.- 8. The Law of the Iterated Logarithm.- II. Tightness of Vector Valued Random Variables and Regularity of Random Processes.- 9. Type and Cotype of Banach Spaces.- 10. The Central Limit Theorem.- 11. Regularity of Random Processes.- 12. Regularity of Gaussian and Stable Processes.- 13. Stationary Processes and Random Fourier Series.- 14. Empirical Process Methods in Probability in Banach Spaces.- 15. Applications to Banach Space Theory.- References.

Probability in Banach Spaces: Isoperimetry and Processes

Probability in Banach spaces

The concentration of measure phenomenon in product spaces roughly states that, if a set A in a product ΩN of probability spaces has measure at least one half, “most” of the points of Ωn are “close” to A. We proceed to a systematic exploration of this phenomenon. The meaning of the word “most” is made rigorous by isoperimetrictype inequalities that bound the measure of the exceptional sets. The meaning of the work “close” is defined in three main ways, each of them giving rise to related, but different inequalities. The inequalities are all proved through a common scheme of proof. Remarkably, this simple approach not only yields qualitatively optimal results, but, in many cases, captures near optimal numerical constants. A large number of applications are given, in particular to Percolation, Geometric Probability, Probability in Banach Spaces, to demonstrate in concrete situations the extremely wide range of application of the abstract tools.

/pdf/concentration-of-measure-and-isoperimetric-inequalities-in-yn056kcsc5.pdf

Concentration of measure and isoperimetric inequalities in product spaces

We introduce three new ways to measure the “distance” from a point to a subset of a product space and we prove corresponding concentration inequalities. Each of them allows to control the fluctuation of a new class of random variables.

New concentration inequalities in product spaces

Under natural conditions on a class $\mathscr{F}$ of functions on a probability space, near optimal bounds are given for the probabilities $P\big(\sup_{f\in\mathscr{F}}|\sum_{i\leq n} f(X_i) - nE(f)| \geq M\sqrt n\big)$. The method is a variation of this author's method to study the tail probability of the supremum of a Gaussian process.

Michel Talagrand

Papers

Probability in Banach Spaces: Isoperimetry and Processes

Probability in Banach spaces

Concentration of measure and isoperimetric inequalities in product spaces

New concentration inequalities in product spaces

Sharper Bounds for Gaussian and Empirical Processes