Statistical learning theory

About: Statistical learning theory is a research topic. Over the lifetime, 1618 publications have been published within this topic receiving 158033 citations.

Regularized estimation of sums of exponentials in spaces generated by stable spline kernels

Abstract: A new nonparametric paradigm to model identification has been recently introduced in the literature. Instead of adopting finite-dimensional models of the system transfer function, the system impulse response is searched for within an infinite dimensional space using regularization theory. The method exploits the so called stable spline kernels which are associated with hypothesis spaces embedding information on both regularity and stability of the impulse response. In this paper, the potentiality of this approach is studied with respect to the reconstruction of sums of exponentials. In particular, first, we characterize the learning rates of our estimator in reconstructing such class of functions also exploiting recent advances in statistical learning theory. Then, we use Monte Carlo studies to illustrate the definite advantages of this new nonparametric approach over classical parametric prediction error methods in terms of accuracy in impulse responses reconstruction.

Support vector machines for nonlinear state space reconstruction: Application to the Great Salt Lake time series

Abstract: [1] The reconstruction of low-order nonlinear dynamics from the time series of a state variable has been an active area of research in the last decade. The 154 year long, biweekly time series of the Great Salt Lake volume has been analyzed by many researchers from this perspective. In this study, we present the application of a powerful state space reconstruction methodology using the method of support vector machines (SVM) to this data set. SVM are machine learning systems that use a hypothesis space of linear functions in a kernel-induced higher-dimensional feature space. SVM are optimized by minimizing a bound on a generalized error (risk) measure rather than just the mean square error over a training set. Under Mercer's conditions on the kernels the corresponding optimization problems are convex; hence global optimal solutions can be readily computed. The SVM-based reconstruction is used to develop time series forecasts for multiple lead times ranging from 2 weeks to several months. Unlike previously reported methodologies, SVM are able to extract the dynamics using only a few past observed data points out of the training examples. The reliability of the algorithm in learning and forecasting the dynamics is tested using split sample sensitivity analysis, with a particular interest in forecasting extreme states. Efforts are also made to assess variations in predictability as a function of initial conditions and as a function of the degree of extrapolation from the state space used for learning the model.

A PAC Approach to Application-Specific Algorithm Selection

Abstract: The best algorithm for a computational problem generally depends on the "relevant inputs," a concept that depends on the application domain and often defies formal articulation. While there is a large literature on empirical approaches to selecting the best algorithm for a given application domain, there has been surprisingly little theoretical analysis of the problem.This paper adapts concepts from statistical and online learning theory to reason about application-specific algorithm selection. Our models capture several state-of-the-art empirical and theoretical approaches to the problem, ranging from self-improving algorithms to empirical performance models, and our results identify conditions under which these approaches are guaranteed to perform well. We present one framework that models algorithm selection as a statistical learning problem, and our work here shows that dimension notions from statistical learning theory, historically used to measure the complexity of classes of binary- and real-valued functions, are relevant in a much broader algorithmic context. We also study the online version of the algorithm selection problem, and give possibility and impossibility results for the existence of no-regret learning algorithms.

On learning guarantees to unsupervised concept drift detection on data streams

Abstract: Motivated by the Statistical Learning Theory (SLT), which provides a theoretical framework to ensure when supervised learning algorithms generalize input data, this manuscript relies on the Algorithmic Stability framework to prove learning bounds for the unsupervised concept drift detection on data streams. Based on such proof, we also designed the Plover algorithm to detect drifts using different measure functions, such as Statistical Moments and the Power Spectrum. In this way, the criterion for issuing data changes can also be adapted to better address the target task. From synthetic and real-world scenarios, we observed that each data stream may require a different measure function to identify concept drifts, according to the underlying characteristics of the corresponding application domain. In addition, we discussed about the differences of our approach against others from literature, and showed illustrative results confirming the usefulness of our proposal.