/pdf/semi-supervised-learning-literature-survey-4ccxy3rmmj.pdf

Semi-Supervised Learning Literature Survey

The goal of machine learning is to program computers to use example data or past experience to solve a given problem. Many successful applications of machine learning exist already, including systems that analyze past sales data to predict customer behavior, optimize robot behavior so that a task can be completed using minimum resources, and extract knowledge from bioinformatics data. Introduction to Machine Learning is a comprehensive textbook on the subject, covering a broad array of topics not usually included in introductory machine learning texts. In order to present a unified treatment of machine learning problems and solutions, it discusses many methods from different fields, including statistics, pattern recognition, neural networks, artificial intelligence, signal processing, control, and data mining. All learning algorithms are explained so that the student can easily move from the equations in the book to a computer program. The text covers such topics as supervised learning, Bayesian decision theory, parametric methods, multivariate methods, multilayer perceptrons, local models, hidden Markov models, assessing and comparing classification algorithms, and reinforcement learning. New to the second edition are chapters on kernel machines, graphical models, and Bayesian estimation; expanded coverage of statistical tests in a chapter on design and analysis of machine learning experiments; case studies available on the Web (with downloadable results for instructors); and many additional exercises. All chapters have been revised and updated. Introduction to Machine Learning can be used by advanced undergraduates and graduate students who have completed courses in computer programming, probability, calculus, and linear algebra. It will also be of interest to engineers in the field who are concerned with the application of machine learning methods. Adaptive Computation and Machine Learning series

Introduction to Machine Learning

In the field of machine learning, semi-supervised learning (SSL) occupies the middle ground, between supervised learning (in which all training examples are labeled) and unsupervised learning (in which no label data are given). Interest in SSL has increased in recent years, particularly because of application domains in which unlabeled data are plentiful, such as images, text, and bioinformatics. This first comprehensive overview of SSL presents state-of-the-art algorithms, a taxonomy of the field, selected applications, benchmark experiments, and perspectives on ongoing and future research. Semi-Supervised Learning first presents the key assumptions and ideas underlying the field: smoothness, cluster or low-density separation, manifold structure, and transduction. The core of the book is the presentation of SSL methods, organized according to algorithmic strategies. After an examination of generative models, the book describes algorithms that implement the low-density separation assumption, graph-based methods, and algorithms that perform two-step learning. The book then discusses SSL applications and offers guidelines for SSL practitioners by analyzing the results of extensive benchmark experiments. Finally, the book looks at interesting directions for SSL research. The book closes with a discussion of the relationship between semi-supervised learning and transduction. Adaptive Computation and Machine Learning series

/pdf/semi-supervised-learning-scg1exixez.pdf

Semi-Supervised Learning

On robust estimation of the location parameter

This comprehensive treatment of network information theory and its applications provides the first unified coverage of both classical and recent results. With an approach that balances the introduction of new models and new coding techniques, readers are guided through Shannon's point-to-point information theory, single-hop networks, multihop networks, and extensions to distributed computing, secrecy, wireless communication, and networking. Elementary mathematical tools and techniques are used throughout, requiring only basic knowledge of probability, whilst unified proofs of coding theorems are based on a few simple lemmas, making the text accessible to newcomers. Key topics covered include successive cancellation and superposition coding, MIMO wireless communication, network coding, and cooperative relaying. Also covered are feedback and interactive communication, capacity approximations and scaling laws, and asynchronous and random access channels. This book is ideal for use in the classroom, for self-study, and as a reference for researchers and engineers in industry and academia.

Network Information Theory

The problem of estimating an unknown discrete distribution from its samples is a fundamental tenet of statistical learning Over the past decade, it attracted significant research effort and has been solved for a variety of divergence measures Surprisingly, an equally important problem, estimating an unknown Markov chain from its samples, is still far from understood We consider two problems related to the min-max risk (expected loss) of estimating an unknown $k$-state Markov chain from its $n$ sequential samples: predicting the conditional distribution of the next sample with respect to the KL-divergence, and estimating the transition matrix with respect to a natural loss induced by KL or a more general $f$-divergence measure 
For the first measure, we determine the min-max prediction risk to within a linear factor in the alphabet size, showing it is $\Omega(k\log\log n\ / n)$ and $\mathcal{O}(k^2\log\log n\ / n)$ For the second, if the transition probabilities can be arbitrarily small, then only trivial uniform risk upper bounds can be derived We therefore consider transition probabilities that are bounded away from zero, and resolve the problem for essentially all sufficiently smooth $f$-divergences, including KL-, $L_2$-, Chi-squared, Hellinger, and Alpha-divergences

On Learning Markov Chains

In this paper, we reconsider the problem of distance to uniformity estimation of discrete distributions. As a fundamental problem in distribution property estimation, the problem with known alphabet size has been addressed in [1] [3] and is fairly well understood. In particular, let $k$ be the alphabet size and $\epsilon$ be the error tolerance parameter, people have shown that the corresponding $\epsilon$ -minimax sample complexity, i.e., the minimum sample size that is sufficient for achieving an estimation error of $\epsilon$ even in the worst case, is $\Theta(k/(\epsilon^{2}\log k))$ . Surprisingly, the natural setting where the distribution is over an alphabet of unknown size has not been studied. In this work, we propose and study the well-motivated yet unexplored problem of estimating the generalized distance to uniformity, i.e., the distance of an unknown distribution to the closest uniform distribution. We provide both upper and lower bounds for its $(S,\epsilon)$ -minimax sample complexity. Specifically, let $p$ be the underlying distribution and $S(p)$ be the support size of the closest uniform distribution to $p.\mathbf{For}\ \epsilon\in(4/\sqrt{\log S(p)},\ 1]$ , we present an estimator, that takes $\mathcal{O}(S(p)/(\epsilon^{3}\log\check{S}(p))$ independent samples from the underlying distribution, with probability 2/3, estimates its generalized distance to uniformity up to an additive error of $\epsilon$ without knowing the alphabet $\Omega$ or the support size $S(p)$ . In addition, the estimator can be computed in nearly linear time in the sample size. In the typical high precision regime where $\epsilon\in(0,0.15)$ , we show that the existence of an $\epsilon$ -adaptive estimator implies a lower bound of $\Omega_{\epsilon}(S/\log S)$ on the maximum $(S^{\prime},\epsilon)$ -minimax sample complexity over $[S/2,2S]$ .

Adaptive Estimation of Generalized Distance to Uniformity

Sample- and computationally-efficient distribution estimation is a fundamental tenet in statistics and machine learning. We present SURF, an algorithm for approximating distributions by piecewise polynomials. SURF is: simple, replacing prior complex optimization techniques by straight-forward {empirical probability} approximation of each potential polynomial piece {through simple empirical-probability interpolation}, and using plain divide-and-conquer to merge the pieces; universal, as well-known polynomial-approximation results imply that it accurately approximates a large class of common distributions; robust to distribution mis-specification as for any degree $d \le 8$, it estimates any distribution to an $\ell_1$ distance $< 3$ times that of the nearest degree-$d$ piecewise polynomial, improving known factor upper bounds of 3 for single polynomials and 15 for polynomials with arbitrarily many pieces; fast, using optimal sample complexity, running in near sample-linear time, and if given sorted samples it may be parallelized to run in sub-linear time. In experiments, SURF outperforms state-of-the art algorithms.

SURF: A Simple, Universal, Robust, Fast Distribution Learning Algorithm.

Standard redundancy measures the excess number of bits needed to compress sequences of a given length Instead, we consider relative redundancy that measures the excess number of bits for sequences of a given minimum description length Low relative redundancy implies that number of bits needed to compress any sequence is essentially the lowest possible We show that low relative redundancy implies low standard redundancy, that while block relative redundancy resembles block standard redundancy, sequential relative redundancy is twice its counterpart, and that common algorithms achieving standard redundancy have unbounded relative redundancy

Relative redundancy: a more stringent performance guarantee for universal compression

We consider the problem of estimating the distribution underlying an observed sample of data. Instead of maximum likelihood, which maximizes the probability of the ob served values, we propose a different estimate, the high-profile distribution, which maximizes the probability of the observed profile the number of symbols appearing any given number of times. We determine the high-profile distribution of several data samples, establish some of its general properties, and show that when the number of distinct symbols observed is small compared to the data size, the high-profile and maximum-likelihood distributions are roughly the same, but when the number of symbols is large, the distributions differ, and high-profile better explains the data.

Alon Orlitsky

Papers

On Learning Markov Chains

Adaptive Estimation of Generalized Distance to Uniformity

SURF: A Simple, Universal, Robust, Fast Distribution Learning Algorithm.

Relative redundancy: a more stringent performance guarantee for universal compression

On Modeling Profiles instead of Values