Statistical learning theory

About: Statistical learning theory is a research topic. Over the lifetime, 1618 publications have been published within this topic receiving 158033 citations.

Research on Discharge Sound Recognition Based on Machine Learning and Convolutional Neural Network Training Algorithm

Abstract: Support Vector Machine (SVM) is a machine learning method based on statistical learning theory and structural risk minimization principle. The selection of many parameters directly affects the performance of SVM. SVM can be used as a classifier for estimating sound source position, and the anti-noise ability of the algorithm can be improved by selecting appropriate parameters. Convolutional neural network (CNN) can directly obtain effective information from the original image, omitting the processes of preprocessing, feature extraction and data reconstruction of the original image, and is highly invariant to displacement, scaling and other forms of distortion. By setting different solver parameters, network structure and the number of training samples, the results of defect recognition are compared and analyzed, and it is found that the improved Alexnet network has strong adaptive learning ability, which provides a new idea for pattern recognition in DC cable fault diagnosis.

Mathematical Models of Supervised Learning and Application to Medical Diagnosis

Abstract: Supervised learning models are applicable in many fields of science and technology, such as economics, engineering and medicine. Among supervised learning algorithms, there are the so-called Support Vector Machines (SVM), exhibiting accurate solutions and low training time. They are based on the statistical learning theory and provide the solution by minimizing a quadratic type cost function. SVM, in conjunction with the use of kernel methods, provide non-linear classification models, namely separations that cannot be expressed using inequalities on linear combinations of parameters. There are some issues that may reduce the effectiveness of these methods. For example, in multi-center clinical trials, experts from different institutions collect data on many patients. In this case, techniques currently in use determine the model considering all the available data. Although they are well suited to cases under consideration, they do not provide accurate answers in general. Therefore, it is necessary to identify a subset of the training set which contains all available information, providing a model that still generalizes to new testing data. It is also possible that the training sets vary over time, for example, because data are added and modified as a result of new tests or new knowledge. In this case, the current techniques are not able to capture the changes, but need to start the learning process from the beginning. The techniques, which extract only the new knowledge contained in the data and provide the learning model in an incremental way, have the advantage of taking into account only the experiments really useful and speed up the analysis. In this paper, we describe some solutions to these problems, with the support of numerical experiments on the discrimination among differ types of leukemia.

Reliable Estimation of KL Divergence using a Discriminator in Reproducing Kernel Hilbert Space

Abstract: Estimating Kullback Leibler (KL) divergence from samples of two distributions is essential in many machine learning problems. Variational methods using neural network discriminator have been proposed to achieve this task in a scalable manner. However, we noted that most of these methods using neural network discriminators suffer from high fluctuations (variance) in estimates and instability in training. In this paper, we look at this issue from statistical learning theory and function space complexity perspective to understand why this happens and how to solve it. We argue that the cause of these pathologies is lack of control over the complexity of the neural network discriminator function and could be mitigated by controlling it. To achieve this objective, we 1) present a novel construction of the discriminator in the Reproducing Kernel Hilbert Space (RKHS), 2) theoretically relate the error probability bound of the KL estimates to the complexity of the discriminator in the RKHS space, 3) present a scalable way to control the complexity (RKHS norm) of the discriminator for a reliable estimation of KL divergence, and 4) prove the consistency of the proposed estimator. In three different applications of KL divergence : estimation of KL, estimation of mutual information and Variational Bayes, we show that by controlling the complexity as developed in the theory, we are able to reduce the variance of KL estimates and stabilize the training

Binary classification with classical instances and quantum labels

Abstract: In classical statistical learning theory, one of the most well-studied problems is that of binary classification. The information-theoretic sample complexity of this task is tightly characterized by the Vapnik-Chervonenkis (VC) dimension. A quantum analog of this task, with training data given as a quantum state has also been intensely studied and is now known to have the same sample complexity as its classical counterpart. We propose a novel quantum version of the classical binary classification task by considering maps with classical input and quantum output and corresponding classical-quantum training data. We discuss learning strategies for the agnostic and for the realizable case and study their performance to obtain sample complexity upper bounds. Moreover, we provide sample complexity lower bounds which show that our upper bounds are essentially tight for pure output states. In particular, we see that the sample complexity is the same as in the classical binary classification task w.r.t. its dependence on accuracy, confidence and the VC-dimension.

Dimensionality reduction of RKHS model using Reduced Kernel Principal Component Analysis (RKPCA)

Abstract: This paper deals with the problem of complexity reduction of RKHS models developed on the Reproducing Kernel Hilbert Space (RKHS) using the statistical learning theory (SLT) devoted to supervised learning problems. However, the provided RKHS model suffers from the parameter number which equals the observations used in the learning phase. In this paper we propose a new way to reduce the number of parameters of RKHS model. The proposed method titled Reduced Kernel Principal Component Analysis (RKPCA) consists on approximating the retained principal components given by the KPCA method by a set of observation vectors which point to the directions of the largest variances with the retained principal components. The proposed method has been tested on a chemical reactor and the results were successful.