Showing papers on "Statistical learning theory published in 2022"
TL;DR: In this article , the authors proposed a new adaptive approach based on probabilistic support vector machine for reliability analysis (PSVM-RA), which adopts a new learning function that considers the wrong classification probability for each realization and maximizes the potential for new information offered by a candidate sample for the training set.
Abstract: Adaptive reliability analysis methods based on surrogate models, especially kriging, have been successfully implemented in many problems. However, the application of kriging is limited to low-dimensional problems with noncategorical performance data. Support vector machine (SVM), by contrast, addresses these limitations, but its application in reliability analysis faces several challenges with regard to robustness, accuracy, and efficiency. This study proposed a new adaptive approach based on probabilistic support vector machine for reliability analysis (PSVM-RA). Different from existing methods that only select training points in the margin of the SVM, the proposed method adopts a new learning function that considers the wrong classification probability for each realization and maximizes the potential for new information offered by a candidate sample for the training set. Moreover, the upper bound of the error that is introduced by the SVM in estimating the failure probability is derived based on a Poisson binomial distribution model considering the likelihood of wrong classification for all the points in the margin of the SVM. This upper bound of error was used in the proposed framework as a stopping criterion to guarantee the desired accuracy. Three numerical examples and an engineering application regarding the wind-reliability analysis of transmission towers were investigated to demonstrate the performance of the proposed method. It was demonstrated that PSVM-RA can provide robust estimates of failure probability when other state-of-the-art methods fail. Moreover, it offers a balance between efficiency and accuracy.
7 citations
TL;DR: A novel risk-averse support vector classifier machine (RA-SVCM), which can achieve a better generalization performance by considering the second order statistical information of loss function and is a general form of standard SVM, so it enriches the related studies of SVM.
3 citations
TL;DR: In this article , the performance of robust learning with Huber regression has been studied and a new comparison theorem is established, which characterizes the gap between the excess generalization error and the prediction error.
3 citations
TL;DR: In this article , the structural risk minimization principle is used for model selection in statistical learning for switch-linear hybrid systems, which leads to new theoretically sound bounds on the prediction error of switched models on the one hand, and a practical method for the estimation of the number of modes on the other hand.
3 citations
24 Jun 2022
TL;DR: The Support Vector Machine (SVM) as discussed by the authors is a binary classifier based on the linear classifier with the optimal margin in the feature space and thus the learning strategy is to maximize the margin, which can be transformed into a convex quadratic programming problem.
Abstract: The Support Vector methods was proposed by V.Vapnik in 1965, when he was trying to solve problems in pattern recognition. In 1971, Kimeldorf proposed a method of constructing kernel space based on support vectors. In 1990s, V.Vapnik formally introduced the Support Vector Machine (SVM) methods in Statistical Learning. Since then, SVM has been widely applied in pattern recognition, natural language process and so on. Informally, SVM is a binary classifier. The model is based on the linear classifier with the optimal margin in the feature space and thus the learning strategy is to maximize the margin, which can be transformed into a convex quadratic programming problem. It uses the principle of structural risk minimization instead of empirical risk minimization to fit small data samples. Kernel trick is used to transform non-linear sample space into linear space, decreasing the complexity of algorithm. Even though, it still has broader prospects for development.
1 citations
TL;DR: In this paper , a bagging ensemble of support vector machines (SVMs) is proposed for the lightning performance analysis of overhead distribution lines (OHLs), which facilitates a data-centrist and statistical view of the problem.
Abstract: This paper introduces a novel machine learning (ML) model for the lightning performance analysis of overhead distribution lines (OHLs), which facilitates a data-centrist and statistical view of the problem. The ML model is a bagging ensemble of support vector machines (SVMs), which introduces two significant features. Firstly, support vectors from the SVMs serve as a scaffolding, and at the same time give rise to the so-called curve of limiting parameters for the line. Secondly, the model itself serves as a foundation for the introduction of the statistical safety factor to the lightning performance analysis of OHLs. Both these aspects bolster an end-to-end statistical approach to the OHL insulation coordination and lightning flashover analysis. Furthermore, the ML paradigm brings the added benefit of learning from a large corpus of data amassed by the lightning location networks and fostering, in the process, a “big data” approach to this important engineering problem. Finally, a relationship between safety factor and risk is elucidated. THe benefits of the proposed approach are demonstrated on a typical medium-voltage OHL.
1 citations
TL;DR: In this paper , the authors provide two natural notions of learnability of a function class under a general stochastic process and show that both notions are in fact equivalent to online learnability.
Abstract: Statistical learning theory under independent and identically distributed (iid) sampling and online learning theory for worst case individual sequences are two of the best developed branches of learning theory. Statistical learning under general non-iid stochastic processes is less mature. We provide two natural notions of learnability of a function class under a general stochastic process. We show that both notions are in fact equivalent to online learnability. Our results hold for both binary classification and regression.
12 Sep 2022
TL;DR: In this article , a survey of recent nonasymptotic advances in statistical learning theory as relevant to control and system identification is presented, focusing on the linear quadratic regulator.
Abstract: This tutorial survey provides an overview of recent non-asymptotic advances in statistical learning theory as relevant to control and system identification. While there has been substantial progress across all areas of control, the theory is most well-developed when it comes to linear system identification and learning for the linear quadratic regulator, which are the focus of this manuscript. From a theoretical perspective, much of the labor underlying these advances has been in adapting tools from modern high-dimensional statistics and learning theory. While highly relevant to control theorists interested in integrating tools from machine learning, the foundational material has not always been easily accessible. To remedy this, we provide a self-contained presentation of the relevant material, outlining all the key ideas and the technical machinery that underpin recent results. We also present a number of open problems and future directions.
TL;DR: In this article , replica mean field theory is used to compute the generalization gap of machine learning models with quenched features, in the teacher-student scenario and for regression problems with quadratic loss function.
Abstract: Modern deep neural networks (DNNs) represent a formidable challenge for theorists: according to the commonly accepted probabilistic framework that describes their performance, these architectures should overfit due to the huge number of parameters to train, but in practice they do not. Here we employ results from replica mean field theory to compute the generalization gap of machine learning models with quenched features, in the teacher-student scenario and for regression problems with quadratic loss function. Notably, this framework includes the case of DNNs where the last layer is optimized given a specific realization of the remaining weights. We show how these results---combined with ideas from statistical learning theory---provide a stringent asymptotic upper bound on the generalization gap of fully trained DNN as a function of the size of the dataset $P$. In particular, in the limit of large $P$ and ${N}_{\text{out}}$ (where ${N}_{\text{out}}$ is the size of the last layer) and ${N}_{\text{out}}\ensuremath{\ll}P$, the generalization gap approaches zero faster than $2{N}_{\text{out}}/P$, for any choice of both architecture and teacher function. Notably, this result greatly improves existing bounds from statistical learning theory. We test our predictions on a broad range of architectures, from toy fully connected neural networks with few hidden layers to state-of-the-art deep convolutional neural networks.
18 Jul 2022
TL;DR: This paper explored the impact of structural complexity on the performance and statistical reliability of SVMs applied for text mining and found that the statistical reliability and performance reduce exponentially with the increase of the structural complexity of the SVMs.
Abstract: Support vector machines (SVM) are often applied in the context of machine learning analysis of various data. Given the nature of SVMs, these operate always in the sub-interpolation range as a machine learning method. Here we explore the impact of structural complexity on the performance and statistical reliability of SVMs applied for text mining. We set a theoretical framework for our analysis. We found experimentally that the statistical reliability and performance reduce exponentially with the increase of the structural complexity of the SVMs. This is an important result for the understanding of how the prediction error of SVM predictive data models behaves.
01 Jan 2022
TL;DR: The relevance vector machine (RM) as mentioned in this paper is a sparse probability model based on Bayesian theory that uses the idea of conditional distribution and maximum likelihood estimation to transform the nonlinear problem in low dimensional space into linear problem in high dimensional space through kernel function.
Abstract: Correlation vector machine (RM) is a sparse probability model based on Bayesian theory. It uses the idea of conditional distribution and maximum likelihood estimation to transform the nonlinear problem in low dimensional space into linear problem in high dimensional space through kernel function. It has the advantages of good learning ability, strong generalization ability, flexible kernel function selection and simple parameter setting. Because of its excellent learning performance, it has become a research hotspot in the field of machine learning. This paper introduces the classical relevance vector machine algorithm and its improved model, focuses on the ideas, methods and effects of using relevance vector machine algorithm to solve the classification and prediction problems in fault detection, pattern recognition, Cyberspace Security and other fields, summarizes the existing problems, and prospects the future research direction.
01 Jan 2022
TL;DR: Wang et al. as mentioned in this paper used support vector machine regression theory to predict economic risk not only enriches the existing risk prediction methods in theory, but also has important value in practical application.
Abstract: With the deepening of economic globalization, China's participation in global foreign trade activities is constantly diversified, and at the same time it faces more and more risks. China's long-term accumulated risks in the future may be released intensively, causing high incidence. Potential risks in macro-economy, business environment, sovereign credit, debt and other fields will bring certain losses to enterprises and even hinder their development. Support Vector Machine (SVM) based on statistical learning theory is a new machine learning algorithm, which can successfully deal with classification and regression problems. Because of the excellent learning performance of SVM, this technology has become a research hotspot in current academic circles. This paper expounds the basic theory of support vector machine in detail, constructs the basic framework of economic investment risk prediction model based on support vector machine, and gives the concrete steps to realize the model and the key problems to be solved. The algorithm is very practical. Using support vector machine regression theory to predict economic risk not only enriches the existing risk prediction methods in theory, but also has important value in practical application.
TL;DR: The active interface of statistical learning methods and quantitative finance models is surveyed in this paper . But the focus is on the use of statistical surrogates, also known as functional approximators, for learning input-output relationships relevant for financial tasks.
Abstract: We survey the active interface of statistical learning methods and quantitative finance models. Our focus is on the use of statistical surrogates, also known as functional approximators, for learning input–output relationships relevant for financial tasks. Given the disparate terminology used among statisticians and financial mathematicians, we begin by reviewing the main ingredients of surrogate construction and the motivating financial tasks. We then summarize the major surrogate types, including (deep) neural networks, Gaussian processes, gradient boosting machines, smoothing splines, and Chebyshev polynomials. The second half of the article dives deeper into the major applications of statistical learning in finance, covering ( a) parametric option pricing, ( b) learning the implied/local volatility surface, ( c) learning option sensitivities, ( d) American option pricing, and ( e) model calibration. We also briefly detail statistical learning for stochastic control and reinforcement learning, two areas of research exploding in popularity in quantitative finance. Expected final online publication date for the Annual Review of Statistics and Its Application, Volume 10 is March 2023. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.
31 May 2022
TL;DR: In this paper , the key concepts of statistical learning theory, such as generalization error, empirical risk minimization, bias-complexity tradeoff, and validation, are introduced.
Abstract: This first chapter introduces the key concepts of statistical learning theory, such as generalization error, empirical risk minimization, bias-complexity tradeoff, and validation. It also describes the probably approximately correct (PAC) framework and establishes that finite hypothesis classes are PAC-learnable.
TL;DR: This paper will introduce the basic theory of the Support vector machine, the basic idea of classification and the classification algorithm for the support vector machine that will be used, and solve the problem using an algorithm.
Abstract: Support vector machine is a part of machine learning approach based on statistical learning theory. Due to the higher accuracy of values, Support vector machines have become a focus for frequent machine learning users. This paper will introduce the basic theory of the Support vector machine, the basic idea of classification and the classification algorithm for the support vector machine that will be used. Solving the problem will use an algorithm, and prove the effectiveness of the algorithm on the data that has been used. In this study, the support vector machine has obtained very good accuracy results in its completion. The SVM classification uses kernel RBF with optimum parameters Cost = 5 and gamma = 2 is 88%.
TL;DR: Experiments on bearing and gear run-to-failure datasets are studied to show that the proposed BSWHI and its inherent statistical threshold can effectively detect early machine faults and simultaneously provide monotonic degradation assessment trends.
Abstract: Machine condition monitoring aims to evaluate machine health conditions by analyzing machine vibration signals, which is helpful to make timely maintenance decisions and prevent unexpected accidents. Currently, constructions of virtual and physical health indicators (HIs) are commonly used methods for machine condition monitoring. However, most classic physical and virtual HIs lack inherent thresholds, robustness, monotonicity, and interpretability for machine condition monitoring. In this paper, a statistical learning modeling based HI construction method for machine condition monitoring is proposed to solve these problems. Firstly, a statistical decision theory is suggested to clearly describe a machine condition monitoring objective, and subsequently shapes of square envelope spectra are robustly modeled by using a parametric statistical model called a penalized B-spline approximation. Further, an interpretable HI named B-spline weight HI (BSWHI) as well as an inherent statistical threshold is accordingly constructed based on the Mahalanobis distance between B-spline weights of testing samples and a healthy sample. Experiments on bearing and gear run-to-failure datasets are studied to show that the proposed BSWHI and its inherent statistical threshold can effectively detect early machine faults and simultaneously provide monotonic degradation assessment trends. The proposed interpretable BSWHI has achieved a substantial improvement over existing classic HIs.
TL;DR: Support vector machines regression (SVMR) is an important part of statistical learning theory as discussed by the authors and the main difference between SVMR and the classical least squares regression (LSR) is that SVMR uses the ϵ-insensitive loss rather than quadratic loss to measure the empirical error.
21 Jan 2022
TL;DR: In this paper , the authors used support vector machine (SVM) theory based on statistical learning theory to establish a calculation model for the in-situ stress field, which is the basic data for the excavation and support design of deep my roadways.
Abstract: The in-situ stress field is the basic data for the excavation and support design of deep my roadways. The article uses support vector machine (SVM) theory based on statistical learning theory to establish a calculation model. At the same time, the data of the horizontal maximum ground stress and pore pressure field measured at the well site were used to perform a fitting and the influence of the kernel function in the fitting calculation was compared and analyzed. Based on the principle of support vector machine, the mechanical parameters and initial stress field of the calculation area are inverted. The calculated results are basically consistent with the measured in-situ stress values and meet the accuracy requirements. This shows that the inversion result is consistent with the actual project, and the inversion method used is reasonable.