Statistical learning theory

About: Statistical learning theory is a research topic. Over the lifetime, 1618 publications have been published within this topic receiving 158033 citations.

Modified cascade-correlation learning for classification

Abstract: The main advantages of cascade-correlation learning are the abilities to learn quickly and to determine the network size. However, recent studies have shown that in many problems the generalization performance of a cascade-correlation trained network may not be quite optimal. Moreover, to reach a certain performance level, a larger network may be required than with other training methods. Recent advances in statistical learning theory emphasize the importance of a learning method to be able to learn optimal hyperplanes. This has led to advanced learning methods, which have demonstrated substantial performance improvements. Based on these recent advances in statistical learning theory, we introduce modifications to the standard cascade-correlation learning that take into account the optimal hyperplane constraints. Experimental results demonstrate that with modified cascade correlation, considerable performance gains are obtained compared to the standard cascade-correlation learning. This includes better generalization, smaller network size, and faster learning.

Metric Learning for Prototype-based classification

Abstract: In this chapter, one of themost popular and intuitive prototype-based classification algorithms, learning vector quantization (LVQ), is revisited, and recent extensions towards automatic metric adaptation are introduced Metric adaptation schemes extend LVQ in two aspects: on the one hand a greater flexibility is achieved since the metric which is essential for the classification is adapted according to the given classification task at hand On the other hand a better interpretability of the results is gained since the metric parameters reveal the relevance of single dimensions as well as correlations which are important for the classification Thereby, the flexibility of the metric can be scaled from a simple diagonal term to full matrices attached locally to the single prototypes These choices result in a more complex form of the classification boundaries of the models, whereby the excellent inherent generalization ability of the classifier is maintained, as can be shown by means of statistical learning theory

Newton method for implicit Lagrangian twin support vector machines

Abstract: In this paper, we proposed an implicit Lagrangian twin support vector machine (TWSVM) classifiers by formulating a pair of unconstrained minimization problems (UMPs) in dual variables whose solutions will be obtained using finite Newton method. The advantage of considering the generalized Hessian approach for our modified UMPs reduces to solving just two systems of linear equations as opposed to solving two quadratic programming problems in TWSVM and TBSVM, which leads to extremely simple and fast algorithm. Unlike the classical TWSVM and least square TWSVM (LSTWSVM), the structural risk minimization principle is implemented by adding regularization term in the primal problems of our proposed algorithm. This embodies the essence of statistical learning theory. Computational comparisons of our proposed method against GEPSVM, TWSVM, STWSVM and LSTWSVM have been made on both synthetic and well-known real world benchmark datasets. Experimental results show that our method yields significantly better generalization performance in both computational time and classification accuracy.

Fast hyperparameter tuning using Bayesian optimization with directional derivatives

Abstract: In this paper we develop a Bayesian optimization based hyperparameter tuning framework inspired by statistical learning theory for classifiers. We utilize two key facts from PAC learning theory; the generalization bound will be higher for a small subset of data compared to the whole, and the highest accuracy for a small subset of data can be achieved with a simple model. We initially tune the hyperparameters on a small subset of training data using Bayesian optimization. While tuning the hyperparameters on the whole training data, we leverage the insights from the learning theory to seek more complex models. We realize this by using directional derivative signs strategically placed in the hyperparameter search space to seek a more complex model than the one obtained with small data. We demonstrate the performance of our method on the tasks of tuning the hyperparameters of several machine learning algorithms.

Classification of gaps at uncontrolled intersections and midblock crossings using support vector machines

Abstract: Gap acceptance predictions provide very important inputs for performance evaluation and safety analysis of uncontrolled intersections and pedestrian midblock crossings. The focus of this paper is on the application of support vector machines (SVMs) in understanding and classifying gaps at these facilities. The SVMs are supervised learning techniques originating from statistical learning theory and are widely used for classification and regression. In this paper, the feasibility of the SVM in analyzing gap acceptance is examined by comparing its results with existing statistical methods. To accomplish that objective, SVM and binary logit models (BLMs) were developed and compared by using data collected at three types of uncontrolled intersections. SVM performance was found to be comparable with that of the BLM in all cases and better in a few. Also, the categorical statistics and skill scores used for validating gap acceptance data revealed that the SVM performed reasonably well. Thus, the SVM technique can be used to classify and predict accepted and rejected gap values according to speed and distance of oncoming vehicles. This technique can be used in advance safety warning systems for vehicles and pedestrians waiting to cross major stream vehicles.