Statistical learning theory

About: Statistical learning theory is a research topic. Over the lifetime, 1618 publications have been published within this topic receiving 158033 citations.

Rademacher complexity and spin glasses: A link between the replica and statistical theories of learning

Abstract: Statistical learning theory provides bounds of the generalization gap, using in particular the Vapnik-Chervonenkis dimension and the Rademacher complexity. An alternative approach, mainly studied in the statistical physics literature, is the study of generalization in simple synthetic-data models. Here we discuss the connections between these approaches and focus on the link between the Rademacher complexity in statistical learning and the theories of generalization for typical-case synthetic models from statistical physics, involving quantities known as Gardner capacity and ground state energy. We show that in these models the Rademacher complexity is closely related to the ground state energy computed by replica theories. Using this connection, one may reinterpret many results of the literature as rigorous Rademacher bounds in a variety of models in the high-dimensional statistics limit. Somewhat surprisingly, we also show that statistical learning theory provides predictions for the behavior of the ground-state energies in some full replica symmetry breaking models.

Mapping Heavy Metal Content in Soils with Multi-Kernel SVR and LiDAR Derived Data

Abstract: Support vector regression (SVR) is a powerful machine learning technique in the framework of the statistical learning theory; while Kriging is a well-established prediction method traditionally used in the spatial statistics field. However, the two techniques share the same background of reproducing kernel Hilbert space (RKHS).

A Theoretical-Empirical Approach to Estimating Sample Complexity of DNNs

Abstract: This paper focuses on understanding how the generalization error scales with the amount of the training data for deep neural networks (DNNs). Existing techniques in statistical learning theory require a computation of capacity measures, such as VC dimension, to provably bound this error. It is however unclear how to extend these measures to DNNs and therefore the existing analyses are applicable to simple neural networks, which are not used in practice, e.g., linear or shallow (at most two-layer) ones or otherwise multi-layer perceptrons. Moreover many theoretical error bounds are not empirically verifiable. In this paper we derive estimates of the generalization error that hold for deep networks and do not rely on unattainable capacity measures. The enabling technique in our approach hinges on two major assumptions: i) the network achieves zero training error, ii) the probability of making an error on a test point is proportional to the distance between this point and its nearest training point in the feature space and at certain maximal distance (that we call radius) it saturates. Based on these assumptions we estimate the generalization error of DNNs. The obtained estimate scales as $\mathcal{O}\left( {\frac{1}{{\delta {N^{1/d}}}}} \right)$, where N is the size of the training data, and is parameterized by two quantities, the effective dimensionality of the data as perceived by the network (d) and the aforementioned radius (δ), both of which we find empirically. We show that our estimates match with the experimentally-obtained behavior of the error on multiple learning tasks using benchmark data-sets and realistic models. Estimating training data requirements is essential for deployment of safety critical applications such as autonomous driving, medical diagnostics etc. Furthermore, collecting and annotating training data requires a huge amount of financial, computational and human resources. Our empirical estimates will help to efficiently allocate resources.

Convex hull-based support vector machine rule extraction

Abstract: Support vector machine (SVM) is a machine learning method based on statistical learning theory, and it can avoid the disadvantages well, such as over-training, weak normalization capability, etc. However, the black-box characteristic of SVM has limited its application. In order to open the black-box, a new rule extraction algorithm based on convex hull theory is proposed in this paper. First, all the vectors were clustered to be some clusters on the decision hyper-plane; then, extracted the convex hull for every cluster; finally, the region of each convex hull covered were transferred to each interval-type rule. Rule extraction has been experimented on two public datasets of Iris and Breast-cancer, which results showed that the proposed method can improve the accuracy of rule covering and fidelity.

Multisensor system using support vector machines for water quality classification

Abstract: The field of monitoring drinking water acquires a particular importance in the last few years. The control of risks in the factories that produce and distribute water ensures the quality of this vital resource. Several methods and techniques were implemented in order to reduce these risks. We present here by a new technique called: support vector machines (SVMs). This method is developed from the statistical learning theory, which displays optimal training performances and generalization in several fields, among others the field of pattern recognition. The exposed technique ensures within a monitoring system, a direct and quasi permanent quality control of water. For a validation of the performances of this technique used as classification tool, a study in simulation of the training time, the recognition rate and the noise sensitivity, is carried out. With an aim of showing its functionality, an application test is presented.