Showing papers on "Statistical learning theory published in 2014"

Adaptation, Learning, and Optimization Over Networks

Abstract: This work deals with the topic of information processing over graphs. The presentation is largely self-contained and covers results that relate to the analysis and design of multi-agent networks for the distributed solution of optimization, adaptation, and learning problems from streaming data through localized interactions among agents. The results derived in this work are useful in comparing network topologies against each other, and in comparing networked solutions against centralized or batch implementations. There are many good reasons for the peaked interest in distributed implementations, especially in this day and age when the word "network" has become commonplace whether one is referring to social networks, power networks, transportation networks, biological networks, or other types of networks. Some of these reasons have to do with the benefits of cooperation in terms of improved performance and improved resilience to failure. Other reasons deal with privacy and secrecy considerations where agents may not be comfortable sharing their data with remote fusion centers. In other situations, the data may already be available in dispersed locations, as happens with cloud computing. One may also be interested in learning through data mining from big data sets. Motivated by these considerations, this work examines the limits of performance of distributed solutions and discusses procedures that help bring forth their potential more fully. The presentation adopts a useful statistical framework and derives performance results that elucidate the mean-square stability, convergence, and steady-state behavior of the learning networks. At the same time, the work illustrates how distributed processing over graphs gives rise to some revealing phenomena due to the coupling effect among the agents. These phenomena are discussed in the context of adaptive networks, along with examples from a variety of areas including distributed sensing, intrusion detection, distributed estimation, online adaptation, network system theory, and machine learning.

Theory of Disagreement-Based Active Learning

Abstract: Active learning is a protocol for supervised machine learning, in which a learning algorithm sequentially requests the labels of selected data points from a large pool of unlabeled data. This contrasts with passive learning, where the labeled data are taken at random. The objective in active learning is to produce a highly-accurate classifier, ideally using fewer labels than the number of random labeled data sufficient for passive learning to achieve the same. This article describes recent advances in our understanding of the theoretical benefits of active learning, and implications for the design of effective active learning algorithms. Much of the article focuses on a particular technique, namely disagreement-based active learning, which by now has amassed a mature and coherent literature. It also briefly surveys several alternative approaches from the literature. The emphasis is on theorems regarding the performance of a few general algorithms, including rigorous proofs where appropriate. However, the presentation is intended to be pedagogical, focusing on results that illustrate fundamental ideas, rather than obtaining the strongest or most general known theorems. The intended audience includes researchers and advanced graduate students in machine learning and statistics, interested in gaining a deeper understanding of the recent and ongoing developments in the theory of active learning.

Deep Fisher Kernels -- End to End Learning of the Fisher Kernel GMM Parameters

Abstract: Fisher Kernels and Deep Learning were two developments with significant impact on large-scale object categorization in the last years. Both approaches were shown to achieve state-of-the-art results on large-scale object categorization datasets, such as ImageNet. Conceptually, however, they are perceived as very different and it is not uncommon for heated debates to spring up when advocates of both paradigms meet at conferences or workshops. In this work, we emphasize the similarities between both architectures rather than their differences and we argue that such a unified view allows us to transfer ideas from one domain to the other. As a concrete example we introduce a method for learning a support vector machine classifier with Fisher kernel at the same time as a task-specific data representation. We reinterpret the setting as a multi-layer feed forward network. Its final layer is the classifier, parameterized by a weight vector, and the two previous layers compute Fisher vectors, parameterized by the coefficients of a Gaussian mixture model. We introduce a gradient descent based learning algorithm that, in contrast to other feature learning techniques, is not just derived from intuition or biological analogy, but has a theoretical justification in the framework of statistical learning theory. Our experiments show that the new training procedure leads to significant improvements in classification accuracy while preserving the modularity and geometric interpretability of a support vector machine setup.

Empirical Inference: Festschrift in Honor of Vladimir N. Vapnik

Abstract: This book honours the outstanding contributions of Vladimir Vapnik, a rare example of a scientist for whom the following statements hold true simultaneously: his work led to the inception of a new field of research, the theory of statistical learning and empirical inference; he has lived to see the field blossom; and he is still as active as ever. He started analyzing learning algorithms in the 1960s and he invented the first version of the generalized portrait algorithm. He later developed one of the most successful methods in machine learning, the support vector machine (SVM) more than just an algorithm, this was a new approach to learning problems, pioneering the use of functional analysis and convex optimization in machine learning. Part I of this book contains three chapters describing and witnessing some of Vladimir Vapnik's contributions to science. In the first chapter, Lon Bottou discusses the seminal paper published in 1968 by Vapnik and Chervonenkis that lay the foundations of statistical learning theory, and the second chapter is an English-language translation of that original paper. In the third chapter, Alexey Chervonenkis presents a first-hand account of the early history of SVMs and valuable insights into the first steps in the development of the SVM in the framework of the generalised portrait method. The remaining chapters, by leading scientists in domains such as statistics, theoretical computer science, and mathematics, address substantial topics in the theory and practice of statistical learning theory, including SVMs and other kernel-based methods, boosting, PAC-Bayesian theory, online and transductive learning, loss functions, learnable function classes, notions of complexity for function classes, multitask learning, and hypothesis selection. These contributions include historical and context notes, short surveys, and comments on future research directions. This book will be of interest to researchers, engineers, and graduate students engaged with all aspects of statistical learning.

Robust Optimization using Machine Learning for Uncertainty Sets.

Abstract: Our goal is to build robust optimization problems that make decisions about the future, and where complex data from the past are used to model uncertainty. In robust optimization (RO) generally, the goal is to create a policy for decisionmaking that is robust to our uncertainty about the future. In particular, we want our policy to best handle the the worst possible situation that could arise, out of an uncertainty set of possible situations. Classically, the uncertainty set is simply chosen by the user, or it might be estimated in overly simplistic ways with strong assumptions; whereas in this work, we learn the uncertainty set from complex data from the past. The past data are drawn randomly from an (unknown) possibly complicated high-dimensional distribution. We propose a new uncertainty set design and show how tools from statistical learning theory can be employed to provide probabilistic guarantees on the robustness of the policy.

Image Understanding using Sparse Representations

Abstract: Image understanding has been playing an increasingly crucial role in several inverse problems and computer vision. Sparse models form an important component in image understanding, since they emulate the activity of neural receptors in the primary visual cortex of the human brain. Sparse methods have been utilized in several learning problems because of their ability to provide parsimonious, interpretable, and efficient models. Exploiting the sparsity of natural signals has led to advances in several application areas including image compression, denoising, inpainting, compressed sensing, blind source separation, super-resolution, and classification. The primary goal of this book is to present the theory and algorithmic considerations in using sparse models for image understanding and computer vision applications. To this end, algorithms for obtaining sparse representations and their performance guarantees are discussed in the initial chapters. Furthermore, approaches for designing overcomplete, data-adapted dictionaries to model natural images are described. The development of theory behind dictionary learning involves exploring its connection to unsupervised clustering and analyzing its generalization characteristics using principles from statistical learning theory. An exciting application area that has benefited extensively from the theory of sparse representations is compressed sensing of image and video data. Theory and algorithms pertinent to measurement design, recovery, and model-based compressed sensing are presented. The paradigm of sparse models, when suitably integrated with powerful machine learning frameworks, can lead to advances in computer vision applications such as object recognition, clustering, segmentation, and activity recognition. Frameworks that enhance the performance of sparse models in such applications by imposing constraints based on the prior discriminatory information and the underlying geometrical structure, and kernelizing the sparse coding and dictionary learning methods are presented. In addition to presenting theoretical fundamentals in sparse learning, this book provides a platform for interested readers to explore the vastly growing application domains of sparse representations.

A scenario approach to non-convex control design: Preliminary probabilistic guarantees

Abstract: Randomized optimization is a recently established tool for control design with modulated robustness. While for uncertain convex programs there exist randomized approaches with efficient sampling, this is not the case for non-convex problems. Approaches based on statistical learning theory are applicable for a certain class of non-convex problems, but they usually are conservative in terms of performance and are computationally demanding. In this paper, we present a novel scenario approach for a wide class of random non-convex programs. We provide a sample complexity similar to the one for uncertain convex programs, but valid for all feasible solutions inside a set of a-priori chosen complexity. Our scenario approach applies to many non-convex control-design problems, for instance control synthesis based on uncertain bilinear matrix inequalities.

Robust Optimization using Machine Learning for Uncertainty Sets

Abstract: Our goal is to build robust optimization problems for making decisions based on complex data from the past. In robust optimization (RO) generally, the goal is to create a policy for decision-making that is robust to our uncertainty about the future. In particular, we want our policy to best handle the the worst possible situation that could arise, out of an uncertainty set of possible situations. Classically, the uncertainty set is simply chosen by the user, or it might be estimated in overly simplistic ways with strong assumptions; whereas in this work, we learn the uncertainty set from data collected in the past. The past data are drawn randomly from an (unknown) possibly complicated high-dimensional distribution. We propose a new uncertainty set design and show how tools from statistical learning theory can be employed to provide probabilistic guarantees on the robustness of the policy.

Learning rates of lq coefficient regularization learning with gaussian kernel

Abstract: Regularization is a well-recognized powerful strategy to improve the performance of a learning machine and lq regularization schemes with are central in use. It is known that different q leads to different properties of the deduced estimators, say, l2 regularization leads to a smooth estimator, while l1 regularization leads to a sparse estimator. Then how the generalization capability of lq regularization learning varies with q is worthy of investigation. In this letter, we study this problem in the framework of statistical learning theory. Our main results show that implementing lq coefficient regularization schemes in the sample-dependent hypothesis space associated with a gaussian kernel can attain the same almost optimal learning rates for all . That is, the upper and lower bounds of learning rates for lq regularization learning are asymptotically identical for all . Our finding tentatively reveals that in some modeling contexts, the choice of q might not have a strong impact on the generalization capability. From this perspective, q can be arbitrarily specified, or specified merely by other nongeneralization criteria like smoothness, computational complexity or sparsity.

Piecewise function approximation and vertex partitioning schemes for multi-dividing ontology algorithm in AUC criterion setting (I)

Abstract: Ontology is a useful tool with wide applications in various fields and attracts widespread attention of scholars, and ontology concept similarity calculation is an essential problem in these application algorithms. An effective method to get similarity between vertices on ontology is based on a function, which maps ontology graph into a line and maps each vertex in graph into a real-value, and the similarity is measured by the difference of their corresponding scores. The area under the receiver operating characteristics curve (AUC) criterion multi-dividing method is suitable for ontology problem. In this paper, we present piecewise constant function approximation approach for AUC criterion multi-dividing ontology algorithm and focus on vertex partitioning schemes. Using the techniques of statistical learning theory, theoretical characteristics of the approximation algorithm are provided with partitioning schemes, and a splitting rule is designed for vertex partitioning.

A Combined Data Envelopment Analysis and Support Vector Regression for Efficiency Evaluation of Large Decision Making Units

Abstract: Data Envelopment Analysis (DEA) is a method for measuring efficiencies of Decision Making Units (DMUs). While it has been widely used in many industrial and economic applications, for large DMUs with many inputs and outputs, DEA would require huge computer resources in terms of memory and CPU time. Several studies have attempted to overcome this problem for large datasets. However, the approaches used in the prior researches have some drawbacks which include uncontrolled convergence and non-generalization. Support Vector Regression (SVR) as a generalization from Support Vector Machine (SVM) is a powerful technique based on statistical learning theory for solving many prediction problems in the real-world applications. Hence, in this paper, a new combination of DEA and SVR, DEA-SVR, method is proposed and evaluated for large scale data sets. We evaluate and compare the proposed method using five large datasets used in earlier research. Experimental results demonstrates that the proposed method outperforms the recent most promising combined method of DEA and back-propagation neural networks, DEA-NNs, in terms of accuracy in efficiency estimation.

Research on parameters optimisation of SVM based on swarm intelligence

Abstract: Support vector machine (SVM) is a new machine learning method based on statistical learning theory, which has become a hot research topic in the field of machine learning because of its excellent performance. However, the performance of SVM is very sensitive to its parameters. At present, swarm intelligence is the most common method to optimise the parameters of SVM. In this paper, the research on parameters optimisation of SVM based on swarm intelligence algorithms is reviewed. Firstly, we briefly introduce the theoretical basis of SVM. Secondly, we describe the latest progress of parameters optimisation of SVM based on swarm intelligence in recent years. Finally, we point out the research and development prospects of this kind of method.

An e---E-insensitive support vector regression machine

Abstract: According to the Statistical Learning Theory, the support vectors represent the most informative data points and compress the information contained in training set. However, a basic problem in the standard support vector machine is that when the data is noisy, there exists no guaranteed scheme in support vector machines' formulation to dissuade the machine from learning noise. Therefore, the noise which is typically presents in financial time series data may be taken into account as support vectors. In turn, noisy support vectors are modeled into the estimated function. As such, the inclusion of noise in support vectors may lead to an over-fitting and in turn to a poor generalization. The standard support vector regression (SVR) is reformulated in this article in such a way that the large errors which correspond to noise are restricted by a new parameter $$E$$ E . The simulation and real world experiments indicate that the novel SVR machine meaningfully performs better than the standard SVR in terms of accuracy and precision especially where the data is noisy, but in expense of a longer computation time.

New dimensionality reduction model (manifold learning) coupled with electronic tongue for green tea grade identification

Abstract: Multivariate data analysis methods play a key role in extracting effective features to denote original tea samples. The most commonly used multivariate data analysis methods are principle component analysis and linear discriminant analysis. These methods are based on statistical learning theory and complete in mathematics. However, there is correlation and redundancy among multiple sensors of electronic tongue, and it cannot guarantee that the tea samples are linearly separable in the original data space. The aim of this study is to conduct new dimensionality reduction methods: manifold learning algorithms, to extract effective features from the responses of electronic tongue sensors, and the algorithm which gives the highest recognition accuracy is considered to be the best for tea quality gradation. Experimental results show that supervised nonlinear manifold learning algorithms outperform other methods and achieve the highest recognition accuracy for green tea with four quality grades.

Contributions to statistical learning theory: estimator selection and change-point detection

Abstract: This report presents my main contributions to solving the estimator selection problem, in the statistical learning theory framework. First, a general methodology is presented, for building and analyzing estimator selection procedures. Then, it is applied to cross-validation and resampling procedures, to data-driven penalties based upon the concept of "minimal penalty", to the change-point detection problem and to a few other estimator selection problems.

An Obstacle Recognizing Mechanism for Autonomous Underwater Vehicles Powered by Fuzzy Domain Ontology and Support Vector Machine

Abstract: The autonomous underwater vehicle (AUV) and the problems associated with its safe navigation have been studied for the last two decades. The real-time underwater obstacle recognition procedure still has many complications associated with it and the issue becomes worse with vague sensor data. These problems can be coped with the merger of a robust classification mechanism and a domain knowledge acquisition technique. In this paper, we introduce a hybrid mechanism to recognize underwater obstacles for AUV based on fuzzy domain ontology and support vector machine (SVM). SVM is an efficient algorithm that was developed for recognizing 3D object in recent years and is a new generation learning system based on recent advances in statistical learning theory. The amalgamation of fuzzy domain ontology with SVM boosts the performance of the obstacle recognition module by providing the timely semantic domain information of the surrounding circumstances. Also the reasoning ability of the fuzzy domain ontology can expedite the obstacle avoidance process. In order to evaluate the performance of the system, we developed a prototype simulator based on OpenGL and VC

Vehicle tracking and positioning in GSM network using optimized SVM model

Abstract: Vehicle Tracking and positioning in GSM networkwith greater accuracy is one of the major popular research topics of Intelligence transportation System and it pass on with the evolution of techniques and methods which enable the data processor to learn and execute activities with the help of Machine learning. Support Vector Machine (SVM) is an isolated classifier which deals with both linear and nonlinear data from hyper-plane with the help of Supervised Learning Approach. Whereas the Statistical Learning Theory was unable to procure location information in a Mobile Computing because of functional dependencies of geographic coordinates from RSSI but SVM can predict the location fingerprint with regression estimation and linear operator inversion and realize the actual risk minimization by structural risk minimization. SVM can also deliver a good learning outcome in the face of less sample volume. The basic idea of SVM is for linearly separable samples, to find the optimal classification hyper-plane which can be described accurately and the samples are separated into two categories for the linearly non-separable problems; to transform the linear non-separable problems in the original space into the linearly separable problems in high-dimensional feature space by a nonlinearly transformation for the given samples of dataset. SVM gives a very low error rate when used for classification.

An ordinal kernel trick for a computationally efficient support vector machine

Abstract: A principled approach to machine learning (ML) problems because of its mathematical foundations in statistical learning theory, support vector machines (SVM), a non-parametric method, require all the data to be available during the training phase. However, once the model parameters are identified, SVM relies only, for future prediction, on a subset of these training instances, called support vectors (SV). The SVM model is mathematically written as a weighted sum of these SV whose number, rather than the dimensionality of the input space, defines SVM's complexity. Since the final number of these SV can be up to half the size of the training dataset, SVM becomes challenged to run on energy aware computing platforms. We propose in this work Knee-Cut SVM (KCSVM) and Knee-Cut Ordinal Optimization inspired SVM (KCOOSVM) that use a soft trick of ordered kernel values and uniform subsampling to reduce SVM's prediction computational complexity while maintaining an acceptable impact on its generalization capability. When tested on several databases from UCL KCSVM and KCOOSVM produced promising results, comparable to similar published algorithms.

An Effective Approximation Modeling Method for Ship Resistance in Multidisciplinary Ship Design Optimization

Abstract: Ship design is related to several disciplines such as hydrostatic, resistance, propulsion and economic. The traditional ship design process only involves independent design optimization with some regression formulas within each discipline. With such an approach, there is no guarantee to achieve the optimum design. At the same time, it is also crucial for modem ship design to improve the efficiency of ship optimization. Nowadays, Computational fluid dynamics (CFD) has brought into ship design optimization. However, there are still some problems such as modeling, calculation precision and time consumption even when CFD software is inlaid into the optimization procedure. Modeling is a far-ranging and all-around subject, and its precision directly affects the scientific decision in future. How to use an algorithm to establish a statistical approximation model instead of the CFD calculation will be the key problem. The Support Vector Machines (SVM), a new general machine learning method based on the frame of statistical learning theory, may solve the problems of non-linear classification and regression in sample space and be an effective method of processing the non-liner classification and regression. Recently, Support Vector Regression (SVR) has been introduced to solve regression and modeling problems and been used in wide fields. The classical SVR has two parameters to control the errors. A new algorithm of Support Vector Regression proposed in this article has only one parameter to control the errors, adds b2/2 to the item of confidence interval at the same time, and adopts the Laplace loss function. It is named Single-parameter Lagrangian Support Vector Regression (SPLSVR). This effective algorithm can improve the operation speed of program to a certain extent, and has better fitting precision. In practical design of ship, Design of Experiment (DOE) and the proposed support vector regression algorithm are applied to ship design optimization to construct statistical approximation model in this paper. The support vector regression algorithm approximates the optimization model and is updated during the optimization process to improve accuracy. The result indicates that SPLSVR method to establish approximate models can effectively solve complex engineering design optimization problem. Finally, some suggestions on the future improvements are proposed.

An Efficient and Simplified Model for Forecasting using SRM

Abstract: Learning form continuous financial systems play a vital role in enterprise operations. One of the most sophisticated non-parametric supervised learning classifiers, SVM (Support Vector Machines), provides robust and accurate results, however it may require intense computation and other resources. The heart of SLT (Statistical Learning Theory), SRM (Structural Risk Minimization) Principle can also be used for model selection. In this paper, we focus on comparing the performance of model estimation using SRM with SVR (Support Vector Regression) for forecasting the retail sales of consumer products. The potential benefits of an accurate sales forecasting technique in businesses are immense. Retail sales forecasting is an integral part of strategic business planning in areas such as sales planning, marketing research, pricing, production planning and scheduling. Performance comparison of support vector regression with model selection using SRM shows comparable results to SVR but in a computationally efficient manner. This research targeted the real life data to conclude the results after investigating the computer generated datasets for different types of model building.

Automatic Classification of Schizophrenia using Resting-state Functional Language Network via An Adaptive Learning Algorithm

Abstract: A reliable and precise classification of schizophrenia is significant for its diagnosis and treatment of schizophrenia. Functional magnetic resonance imaging (fMRI) is a novel tool increasingly used in schizophrenia research. Recent advances in statistical learning theory have led to applying pattern classification algorithms to access the diagnostic value of functional brain networks, discovered from resting state fMRI data. The aim of this study was to propose an adaptive learning algorithm to distinguish schizophrenia patients from normal controls using resting-state functional language network. Furthermore, here the classification of schizophrenia was regarded as a sample selection problem where a sparse subset of samples was chosen from the labele d training set. Using these selected samples, which we call informative vectors, a classifier for the clinic diagnosis of schizophrenia was established. We experimentally demonstrated that the proposed algorithm incorporating resting-state functional language network achieved 83.6% leave-one-out accuracy on resting-state fMRI data of 27 schizophrenia patients and 28 normal controls. In contrast with K-Nearest-Neighbor (KNN), Support Vector Machine (SVM) and l1-norm, our method yielded better classification performance. Moreover, our results suggested that a dysfunction of resting-state functional language network plays an important role in the clinic diagnosis of schizophrenia. Keywords: Schizophrenia, fMRI, resting-state, functional connectivity, language network, pattern classification, sample selection, informative vector

Emergency Classification Based on Support Vector Machine

Abstract: Natural disasters,and issues related to public health and social security have occurred frequently Emergency classification is a method to identify emergency events rapidly and accurately,and divide them into different levels The appropriate emergency measures are taken according to classification results At present,the actual classification decisions still rely mainly on intuition and experience of decision-makers In order to obtain the results of scientific classification,it needs to establish a classification model based on objective data and provides a scientific basis for decision-makers Support vector machine( SVM) is developed based on statistical learning theory It is an effective way to deal with classification problems This paper presents the application of support vector machine in order to solve emergency classification problemsSupport vector machine algorithm is originally designed for binary classification problem,and then used for multi-category classification problem There are two methods for extending SVM to multi-class SVM The first method is to construct a multi-class classifier by combing several binary classifiers The second method is to consider all data in an optimized formula for global optimization There are four ways to solve multi-class recognition problems through a combination of multiple binary classifiers: oneagainst-all,one-against-one,directed acidic graph and binary tree In this paper,one-against-one SVM is selected to solve emergency classification problemsThis paper describes the emergency classification process based on support vector machine The process consists of five steps:( 1) establish an index system based on emergency types and the analysis of relevant factors;( 2) collect historical data of the index system to constitute the SVM training sample set;( 3) use a training set to classify learning according to the SVM classification algorithm and obtain an inherent law;( 4) find support vector of the training set to construct SVM decision function; and( 5) enter the index data of classification object into the decision function and obtain the classification resultsThe classification of earthquake disasters is an example to verify the feasibility of the emergency classification method based on support vector machine According to an earthquake disaster database from the National Earthquake Science Data Sharing Center website,30 earthquake sample data from 1995 to 2004 were selected,with 24 of them as training data,and others as the testing dataSelected seismic data contain nine characteristic properties: Richter scale,epicenter intensity,VI degree area,the number of people affected,the number of deaths,the number of injuries,the number of houses destroyed,the number of total housing damage and direct economic losses According to the calculation result of LibSVM toolkit,experimental results show that support vector machine is effective to solve classification problems related to earthquake disastersIn the present paper,an emergency classification method based on support vector is introduced Earthquake classification is proposed as an example The one-against-one method of multi-class support vector machine is applied to the experiment Experimental results show that multi-class support vector machine is a feasible approach dealing with emergency classification problems

Structural risk minimization based weighted least squares power system state estimation method

Abstract: The invention discloses a structural risk minimization based weighted least squares power system state estimation method. According to the characteristic that the measurement number of power system state estimation is limited, a structural risk minimization based weighted least squares estimation model is provided through a statistical learning theory, so that the norm of a residue can be minimized, and meanwhile the confidence interval of a state variable can be minimized. Besides, a detailed solving process of the method is provided. The method accords with the structural risk minimization thought in the statistical learning theory, can obtain an estimation result more approximate to the true value of the state variable under limited measurement conditions, and has good engineering application prospects.