Statistical learning theory

About: Statistical learning theory is a research topic. Over the lifetime, 1618 publications have been published within this topic receiving 158033 citations.

Geostatistical Learning: Challenges and Opportunities

Abstract: Statistical learning theory provides the foundation to applied machine learning, and its various successful applications in computer vision, natural language processing and other scientific domains. The theory, however, does not take into account the unique challenges of performing statistical learning in geospatial settings. For instance, it is well known that model errors cannot be assumed to be independent and identically distributed in geospatial (a.k.a. regionalized) variables due to spatial correlation; and trends caused by geophysical processes lead to covariate shifts between the domain where the model was trained and the domain where it will be applied, which in turn harm the use of classical learning methodologies that rely on random samples of the data. In this work, we introduce the geostatistical (transfer) learning problem, and illustrate the challenges of learning from geospatial data by assessing widely-used methods for estimating generalization error of learning models, under covariate shift and spatial correlation. Experiments with synthetic Gaussian process data as well as with real data from geophysical surveys in New Zealand indicate that none of the methods are adequate for model selection in a geospatial context. We provide general guidelines regarding the choice of these methods in practice while new methods are being actively researched.

Support Vector Machine for Internet Traffic Identification

Abstract: In this paper a non linear system identification problem is addressed. A support vector regressor is used to solve the Internet traffic identification problem. We give a basic idea underlying support vector (SV) machine for regression, which is a novel type of learning machine based on statistical learning theory. Furthermore, we describe how SV regressor can be applied for non linear system identification. In our simulations results we present two type of kernel functions, the radial basis function (RBF), and the hyperbolic tangent, which are compared with the classical two-layer MLP (Multi-Layer- Perceptron) Neural Networks, trained to minimize a quadratic error objective with the back-propagation (BP) algorithm. The SV regressor outperforms the MLP and demonstrates its effectiveness for solving non linear system identification problems.

Learning Coefficients and Reproducing True Probability Functions in Learning Systems

Abstract: Recently, the widely applicable information criterion (WAIC) model selection method has been considered for reproducing and estimating a probability function from data in a learning system. The learning coefficient in Bayesian estimation serves to measure the learning efficiency in singular learning models, and has an important role in the WAIC method. Mathematically, the learning coefficient is the log canonical threshold of the relative entropy. In this paper, we consider the Vandermonde matrix-type singularity learning coefficients in statistical learning theory.

Learning-Based Probabilistic Design

Abstract: In Chap. 9 we provided an overview of some key results of statistical learning theory. In this chapter we discuss their specific application to the design of systems affected by uncertainty formulating a learning-based approach. In particular, we develop a randomized technique for solving a nonconvex semi-infinite optimization problem and compute the related sample complexity. A sequential algorithm which provides a solution to the same problem is also presented.

Learning GP-trees from Noisy Data

Abstract: We discuss the problem of model selection in Genetic Programming using the framework provided by Statistical Learning Theory, i.e. Vapnik-Chervonenkis theory (VC). We present empirical comparisons between classical statistical methods (AIC, BIC) for model selection and the Structural Risk Minimization method (based on VC-theory) for symbolic regression problems. Empirical comparisons of different methods for model selection suggest practical advantages of using VC-based model selection when using genetic training. keywords: Model selection, genetic programming, symbolic regression