Showing papers on "Statistical learning theory published in 2001"

On the mathematical foundations of learning

Abstract: (1) A main theme of this report is the relationship of approximation to learning and the primary role of sampling (inductive inference). We try to emphasize relations of the theory of learning to the mainstream of mathematics. In particular, there are large roles for probability theory, for algorithms such as least squares, and for tools and ideas from linear algebra and linear analysis. An advantage of doing this is that communication is facilitated and the power of core mathematics is more easily brought to bear. We illustrate what we mean by learning theory by giving some instances. (a) The understanding of language acquisition by children or the emergence of languages in early human cultures. (b) In Manufacturing Engineering, the design of a new wave of machines is anticipated which uses sensors to sample properties of objects before, during, and after treatment. The information gathered from these samples is to be analyzed by the machine to decide how to better deal with new input objects (see [43]). (c) Pattern recognition of objects ranging from handwritten letters of the alphabet to pictures of animals, to the human voice. Understanding the laws of learning plays a large role in disciplines such as (Cognitive) Psychology, Animal Behavior, Economic Decision Making, all branches of Engineering, Computer Science, and especially the study of human thought processes (how the brain works). Mathematics has already played a big role towards the goal of giving a universal foundation of studies in these disciplines. We mention as examples the theory of Neural Networks going back to McCulloch and Pitts [25] and Minsky and Papert [27], the PAC learning of Valiant [40], Statistical Learning Theory as developed by Vapnik [42], and the use of reproducing kernels as in [17] among many other mathematical developments. We are heavily indebted to these developments. Recent discussions with a number of mathematicians have also been helpful. In

Nonlinear modelling and support vector machines

Abstract: Neural networks such as multilayer perceptrons and radial basis function networks have been very successful in a wide range of problems. In this paper we give a short introduction to some new developments related to support vector machines (SVM), a new class of kernel based techniques introduced within statistical learning theory and structural risk minimization. This new approach lends to solving convex optimization problems and also the model complexity follows from this solution. We especially focus on a least squares support vector machine formulation (LS-SVM) which enables to solve highly nonlinear and noisy black-box modelling problems, even in very high dimensional input spaces. While standard SVMs have been basically only applied to static problems like classification and function estimation, LS-SVM models have been extended to recurrent models and use in optimal control problems. Moreover, using weighted least squares and special pruning techniques, LS-SVMs can be employed for robust nonlinear estimation and sparse approximation. Applications of (LS)-SVMs to a large variety of artificial and real-life data sets indicate the huge potential of these methods.

Statistical Learning and Kernel Methods

Abstract: We briefly describe the main ideas of statistical learning theory, support vector machines, and kernel feature spaces.

A Tutorial Introduction

Abstract: This chapter contains sections titled: Data Representation and Similarity, A Simple Pattern Recognition Algorithm, Some Insights From Statistical Learning Theory, Hyperplane Classifiers, Support Vector Classification, Support Vector Regression, Kernel Principal Component Analysis, Empirical Results and Implementations

Support vector regression for black-box system identification

Abstract: We demonstrate the use of support vector regression (SVR) techniques for black-box system identification. These methods derive from statistical learning theory, and are of great theoretical and practical interest. We describe the theory underpinning SVR, and compare support vector methods with other approaches using radial basis networks. Finally, we apply SVR to modeling the behaviour of a hydraulic robot arm, and show that SVR improves on previously published results.

Feature reduction and hierarchy of classifiers for fast object detection in video images

Abstract: We present a two-step method to speed-up object detection systems in computer vision that use Support Vector Machines (SVMs) as classifiers. In a first step we perform feature reduction by choosing relevant image features according to a measure derived from statistical learning theory. In a second step we build a hierarchy of classifiers. On the bottom level, a simple and fast classifier analyzes the whole image and rejects large parts of the background On the top level, a slower but more accurate classifier performs the final detection. Experiments with a face detection system show that combining feature reduction with hierarchical classification leads to a speed-up by a factor of 170 with similar classification performance.

Support vector machines for remote sensing image classification

Abstract: In the last decade, the application of statistical and neural network classifiers to remote-sensing images has been deeply investigated. Therefore, performances, characteristics, and pros and cons of such classifiers are quite well known, even from remote-sensing practitioners. In this paper, we present the application to remote-sensing image classification of a new pattern recognition technique recently introduced within the framework of the Statistical Learning Theory developed by V. Vapnik and his co-workers, namely, the Support Vector Machines (SVMs). In section 1, the main theoretical foundations of SVMs are presented. In section 2, experiments carried out on a data set of multisensor remote-sensing images are described, with particular emphasis on the design and training phase of a SVM. In section 3, the experimental results are reported, together with a comparison between the performances of SVMs, neural network, and k-NN classifiers.

Nonparametric regression estimation using penalized least squares

Abstract: We present multivariate penalized least squares regression estimates. We use Vapnik-Chervonenkis (see Statistical Learning Theory 1998) theory and bounds on the covering numbers to analyze convergence of the estimates. We show strong consistency of the truncated versions of the estimates without any conditions on the underlying distribution.

Algorithmic Luckiness

Abstract: In contrast to standard statistical learning theory which studies uniform bounds on the expected error we present a framework that exploits the specific learning algorithm used. Motivated by the luckiness framework [8] we are also able to exploit the serendipity of the training sample. The main difference to previous approaches lies in the complexity measure; rather than covering all hypotheses in a given hypothesis space it is only necessary to cover the functions which could have been learned using the fixed learning algorithm. We show how the resulting framework relates to the VC, luckiness and compression frameworks. Finally, we present an application of this framework to the maximum margin algorithm for linear classifiers which results in a bound that exploits both the margin and the distribution of the data in feature space.

Pattern Recognition and Density Estimation under the General i.i.d. Assumption

Abstract: Statistical learning theory considers three main problems, pattern recognition, regression and density estimation. This paper studies solvability of these problems (mainly concentrating on pattern recognition and density estimation) in the "high-dimensional" case, where the patterns in the training and test sets are never repeated. We show that, assuming an i.i.d. data source but without any further assumptions, the problems of pattern recognition and regression can often be solved (and there are practically useful algorithms to solve them). On the other hand, the problem of density estimation, as we formalize it, cannot be solved under the general i.i.d. assumption, and additional assumptions are required.

Tree-Structured Support Vector Machines for Multi-class Pattern Recognition

Abstract: Support vector machines (SVM) are learning algorithms derived from statistical learning theory. The SVM approach was originally developed for binary classification problems. In this paper SVM architectures for multi-class classification problems are discussed, in particular we consider binary trees of SVMs to solve the multi-class pattern recognition problem. Numerical results for different classifiers on a benchmark data set handwritten digits are presented.

Combining GMM's with Suport Vector Machines for Text-independent Speaker Verification

Abstract: Current best performing speaker recognition algorithms are based on Gaussian Mixture Models (GMM). Their results are not satisfactory for all experimental conditions, especially for the mismatched (train/test) conditions. Support Vector Machine is a new and very promissing technique in statistical learning theory. Recently, this technique produced very interesting results in image processing [2], [3], [4] and for the fusion of experts in biometric authentification [5]. In this paper we address the issue of using the Support Vector Learning technique in combination with the currently well performing GMM models, in order to improve speaker verification results.

Text-independent speaker recognition using support vector machine

Abstract: The support vector machine (SVM) is an important learning method of statistical learning theory, and is also a powerful tool for pattern recognition problems. This paper studies the speaker identification and verification problem using a support vector machine, and presents a SVM training method on large-scale samples according to the speech signal. A text-independent speaker recognition system based on SVM was implemented and the results show good performance.

Solving multi-class pattern recognition problems with tree-structured support vector machines

Abstract: Support vector machines (SVM) are learning algorithms derived from statistical learning theory. The SVM approach was originally developed for binary classification problems. In this paper SVM architectures for multi-class classification problems are discussed, in particular we consider binary trees of SVMs to solve the multi-class problem. Numerical results for different classifiers on a benchmark data set of handwritten digits are presented.

Parameter Estimation for Statistical Parsing Models: Theory and Practice of Distribution-Free Methods.

Abstract: A fundamental problem in statistical parsing is the choice of criteria and algo-algorithms used to estimate the parameters in a model. The predominant approach in computational linguistics has been to use a parametric model with some variant of maximum-likelihood estimation. The assumptions under which maximum-likelihood estimation is justified are arguably quite strong. This chapter discusses the statistical theory underlying various parameter-estimation methods, and gives algorithms which depend on alternatives to (smoothed) maximum-likelihood estimation. We first give an overview of results from statistical learning theory. We then show how important concepts from the classification literature - specifically, generalization results based on margins on training data - can be derived for parsing models. Finally, we describe parameter estimation algorithms which are motivated by these generalization bounds.

System identification: a learning theory approach

Abstract: The problem of system identification is formulated as a problem in statistical learning theory, because statistical learning theory is devoted to the derivation of finite time estimates. If system identification is to be combined with robust control theory to develop a sound theory of indirect adaptive control, it is essential to have finite time estimates of the sort provided by statistical learning theory. As an illustration of the approach, a result is derived showing that in the case of systems with fading memory, it is possible to combine standard result's in statistical learning theory (suitably modified to the present situation) with some fading memory arguments to obtain finite time estimates of the desired kind. In the case of linear systems, the results proved here are not overly conservative, but are more so in the case of nonlinear systems where the adjustable parameters enter linearly into the model description. Though the actual results derived here are rather preliminary in nature, it is hoped that future researchers will pursue the ideas presented here to extend the theory further.

Solving Multi-class Pattern Recognition Problems with Tree-Structured Support Vector Machines

Abstract: Support vector machines (SVM) are learning algorithms derived from statistical learning theory. The SVMapproac h was originally developed for binary classification problems. In this paper SVMarc hitectures for multi-class classification problems are discussed, in particular we consider binary trees of SVMs to solve the multi-class problem. Numerical results for different classifiers on a benchmark data set of handwritten digits are presented.

Appearance-based object recognition using support vector machines

Abstract: Support vector machines (SVM) are a class of algorithms derived from the statistical learning theory that are receiving growing interest by the computer vision community as they present some advantages over classical techniques. This work investigates the ability of SVM to perform appearance-based object recognition. Initial experiments indicated that this may be a promising approach to the problem.

Elements of Statistical Learning Theory

Abstract: This chapter contains sections titled: Introduction, The Law of Large Numbers, When Does LearningWork: the Question of Consistency, Uniform Convergence and Consistency, How to Derive a VC Bound, A Model Selection Example, Summary, Problems

Statistical-learning control of multiple-delay systems with application to ATM networks

Abstract: Congestion control in the ABR class of ATM network presents interesting challenges due to the presence of multiple uncertain delays. Recently, probabilistic methods and statistical learning theory have been shown to provide approximate solutions to challenging control problems. In this paper, using some recent results by the authors, an efficient statistical algorithm is used to design a robust, fixed-structure, controller for a high-speed communication network with multiple uncertain propagation delays.

Statistical learning theory and its application to pattern recognition

Abstract: The problem of pattern recognition is formulated as a classification in the statistic learning theory. Vapnik constructed a class of learning algorithms called support vector machine (SMV) to solve the problem. The algorithm not only has strong theoretical foundation but also provides a powerful tool for solving real-life problems. But it still has some drawbacks. Tow of them are 1) the computational complexity of finding the optimal separating hyperplane is quite high in the linearly separable case, and 2) in the linearly non-separable case, for any given sample set it's hard to choose a proper nonlinear mapping (kernel function) such that the sample set is linearly separable in the new space after the mapping. To overcome these drawbacks, we presented some new approaches. The main idea and some experimental results of the approaches are presented.© (2001) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.

Statistical learning and VC theory

Abstract: The article applies statistical learning theory to the supervised learning problem. Pattern recognition is covered, including Vapnik-Chervonenkis (VC) theory and the implications for support vector machines (SVMs), neural networks and decision trees. Real predictions are given for scale-sensitive dimensions. The article concludes by analysing large margin classification.

From neural networks to support vector machines(A)

Abstract: In the field of statistical pattern recognition, optimal classifiers may be designed theoretically based on the Bayesian decision rule, however, it is necessary for the implementation of the design to so1ve a more difficu1t problem of density estimation first The strategy adopted in HP neural networks is learning di-rectly front the measurement data( training samples), which is more efficient and effective Therefore the methodology of neural networks has been widely used in real life applications, but like other heuristic meth-ods, it lacks a solid theoretical foundation to direct engineering practice As the result of the breakthrough in the research of statistical inference, VC theory has been established and accepted as the modern statistical learning theory The behavior of neural networks may he explained by VC theory with mathematical rigor in addition, a more powerful learning method-the support vector machine has been constructed based on the theory and gained real life applications This paper is a tutorial in which the basic concepts of VC theory and the methodology of SVMs as applied to pattern recognition problems are reviewed

Unification of support vector machines and soft computing paradigms for pattern recognition

Abstract: This paper analyzes support vector machines (SVMs) and several commonly used soft computing paradigms for pattern recognition including neural and wavelet networks, and fuzzy systems. Bayesian classifiers, fuzzy partitions, etc and tries to outline the similarities and differences among them. Support vector machines provide a new approach to the problem of pattern recognition with clear connections to the underlying statistical learning theory. We try to bring SVMs into the framework of the unification paradigm called the weighted radial basis function paradigm. Unifying different classes of methods has enormous advantages, such as the ability to merge all such techniques within the same system. It is hoped that this paper would provide theoretical guides for the study and applications of support vector machine and soft computing paradigms.© (2001) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.

A new view on regularization theory

Abstract: The paper provides a new viewpoint on regularization theory from different perspectives. It is shown that the regularized solution can be derived from the Fourier transformation operator in the transformation domain and with equivalent form from the linear differential operator in the spatial domain. The state-of-the-art research in regularization is briefly reviewed with extended discussions on Occam's razor, minimum length description, Bayesian framework, pruning algorithms, statistical learning theory, and equivalent regularization.

Using statistical learning theory for modeling the uncertainty in business and engineering systems: a qualitative introduction

Abstract: Presents a qualitative introduction and justification of the application of statistical learning theory to uncertainty modeling in business and engineering systems. Using simple mathematical tools and metaphorical images, the main variables that govern the uncertainty in a physical system are defined. A general expression of uncertainty models is then obtained. The structure of this expression is the same as that of the uncertainty models that have been developed by rigorously applying the results of statistical learning theory.