Statistical learning theory

About: Statistical learning theory is a research topic. Over the lifetime, 1618 publications have been published within this topic receiving 158033 citations.

Investigating Overparameterization for Non-Negative Matrix Factorization in Collaborative Filtering

Abstract: Overparameterization is one of the key techniques in modern machine learning, where a model with the higher complexity can generalize better on test data against the common knowledge of the bias-variance trade-off in classical statistical learning theory. In this paper, we empirically investigate the effect of overparameterization for matrix factorization-based models in collaborative filtering. Surprisingly, we firstly show that the performance of overparameterized non-negative matrix factorization (NMF) on test data gets better than that of the underparameterized NMF, which is commonly used to date, and is even competitive with the state-of-the-art collaborative filtering techniques. Moreover, we also show that the double descent phenomenon occurs when we increase the number of parameters of the NMF, where the test error decreases, increases, and decreases again as the model complexity grows, which has been recently reported in various machine learning methods such as deep learning models and kernel methods.

Linguistics and face recognition

Abstract: We describe in this paper a novel biometric methodology for face recognition suitable to address pose, illumination, and expression (PIE) image variability, temporal change, flexible matching, and last but not least occlusion and disguise that are usually referred to as denial and deception. The adverse conditions listed above affect the scope and performance of biometric analysis vis-a-vis both training and testing. The conceptual framework proposed here draws support from discriminative methods using likelihood ratios. At the conceptual level it links forensics and biometrics, while at the implementation level it links the Bayesian framework and statistical learning theory. As many of the concerns listed usually affect only parts of the face, a non-parametric recognition-by-part approach is advanced here for the purpose of reliable face recognition. Recognition-by-parts facilitates authentication because it does not seek for explicit invariance. Instead, it handles variability using component-based configurations that are flexible enough to compensate among others for limited pose changes, if any, and limited occlusion and disguise. The recognition-by-parts approach proposed here supports incremental and progressive processing. It is similar in nature to modern linguistics and practical intelligence with the emphasis on semantics and pragmatics. Layered categorization starts with face detection using implicit rather than explicit segmentation. It proceeds with face authentication that involves feature selection of local patch instances including dimensionality reduction, exemplar-based clustering of patches into parts, and data fusion for matching using boosting driven by parts that play the role of weak learners. The implementation, driven by transduction, employs proximity and typicality (ranking) realized using strangeness and random deficiency p-values, respectively. The feasibility and reliability of the proposed architecture has been validated using FERET and FRGC data. The paper concludes with suggestions for augmenting and enhancing the scope and utility of the recognition-by-parts architecture.

Combining Model for Regional GPS Height Conversion Based on Least Squares Support Vector Machines

Abstract: In geographical information engineering, height anomalies must be known in order to convert GPS ellipsoid heights into geodetic heights. There are many conversion models, such as polynomial, BP neural network and multi-quadrics fitting. Because the quasi-geoid is an irregular geometric object, every method has both advantages and disadvantages, and is appropriate to different conversion patterns. It is difficult to identify which conversion model is most suitable for a particular area. In order to obtain a more precise and reliable analytical result, the combined model based on least squares support vector machines (LSSVM) has been approached. Derived from statistical learning theory, LSSVM are learning systems that use a hypothesis space of linear function in a high dimensional feature space, trained with a learning algorithm from optimizations theory. As a result, the fitting ability of single models is significantly improved. The combined model still possesses the same important features as the single models. Examples are presented and the results are analyzed in detail to demonstrate the efficiency of the proposed methodology.

Statistical pattern recognition and damage detection in structural health monitoring of civil infrastructure systems

Abstract: The main problems of structural health monitoring (SHM) can be cast or regarded as problems in pattern recognition (PR) or machine learning. This approach makes use of the availability of appropriate data to learn the relationship between measured quantities and a diagnosis of state-of-health. This chapter describes how PR provides a natural framework for addressing questions of SHM from basic detection to damage location and quantification. The basic theory and practice are illustrated via the use of two experimental case studies; the first concerns the classification of acoustic emission data as part of a damage assessment of a bridge box girder and the second considers the problem of damage location within an aircraft wing. A number of PR algorithms are illustrated, based on both statistical and neural network approaches. A modern approach to machine learning based on concepts of statistical learning theory is also illustrated using the aircraft wing data.

An ordinal kernel trick for a computationally efficient support vector machine

Abstract: A principled approach to machine learning (ML) problems because of its mathematical foundations in statistical learning theory, support vector machines (SVM), a non-parametric method, require all the data to be available during the training phase. However, once the model parameters are identified, SVM relies only, for future prediction, on a subset of these training instances, called support vectors (SV). The SVM model is mathematically written as a weighted sum of these SV whose number, rather than the dimensionality of the input space, defines SVM's complexity. Since the final number of these SV can be up to half the size of the training dataset, SVM becomes challenged to run on energy aware computing platforms. We propose in this work Knee-Cut SVM (KCSVM) and Knee-Cut Ordinal Optimization inspired SVM (KCOOSVM) that use a soft trick of ordered kernel values and uniform subsampling to reduce SVM's prediction computational complexity while maintaining an acceptable impact on its generalization capability. When tested on several databases from UCL KCSVM and KCOOSVM produced promising results, comparable to similar published algorithms.