Showing papers on "Sequential minimal optimization published in 2007"

Support Vector Machine Solvers

Abstract: This chapter contains sections titled: Introduction, Support Vector Machines, Duality, Sparsity, Early SVM Algorithms, The Decomposition Method, A Case Study: LIBSVM, Conclusion and Outlook, Appendix

Support Vector Ordinal Regression

Abstract: In this letter, we propose two new support vector approaches for ordinal regression, which optimize multiple thresholds to define parallel discriminant hyperplanes for the ordinal scales. Both approaches guarantee that the thresholds are properly ordered at the optimal solution. The size of these optimization problems is linear in the number of training samples. The sequential minimal optimization algorithm is adapted for the resulting optimization problems; it is extremely easy to implement and scales efficiently as a quadratic function of the number of examples. The results of numerical experiments on some benchmark and real-world data sets, including applications of ordinal regression to information retrieval, verify the usefulness of these approaches.

Combination of independent component analysis and support vector machines for intelligent faults diagnosis of induction motors

Abstract: This paper studies the application of independent component analysis (ICA) and support vector machines (SVMs) to detect and diagnose of induction motor faults. The ICA is used for feature extraction and data reduction from original features. The principal components analysis is also applied in feature extraction process for comparison with ICA does. In this paper, the training of the SVMs is carried out using the sequential minimal optimization algorithm and the strategy of multi-class SVMs-based classification is applied to perform the faults identification. Also, the performance of classification process due to the choice of kernel function is presented to show the excellent of characteristic of kernel function. Various scenarios are examined using data sets of vibration and stator current signals from experiments, and the results are compared to get the best performance of classification process.

Robust support vector machines for classification and computational issues

Abstract: In this paper, we investigate the theoretical and numerical aspects of robust classification using support vector machines (SVMs) by providing second order cone programming and linear programming formulations. SVMs are learning algorithms introduced by Vapnik used either for classification or regression. They show good generalization properties and they are based on statistical learning theory. The resulting learning problems are convex optimization problems suitable for application of primal-dual interior points methods. We investigate the training of a SVM in the case where a bounded perturbation is added to the value of an input xi∈n. A robust SVM provides a decision function that is immune to data perturbations. We consider both cases where our training data are either linearly separable or non linearly separable respectively and provide computational results for real data sets.

Probabilistic support vector machines for multi-class alcohol identification

Abstract: In this work we address the use of support vector machines in multi-category problems. In our case, the objective is to classify eight different kinds of alcohols with just one SnO2 sensor using thermomodulation. The use of support vector machines in the field of sensors signals recognition is beginning to be used due to the ability to generalize in a binary classification problem with a small number of training samples. However, when a multi-class problem is presented, the outputs of the support vector machines are uncalibrated and should not be used to determine the category. In this work a step forward is added to the output of the binary classifiers to choose the category with a maximal a posteriori probability. Obtained results show that the ability of generalization provided by support vector machines improves the results obtained with other learning methods used in the electronic nose field and their use in multi-class problems can be addressed with the method proposed. To reduce the high dimensionality of the data we have benchmarked several feature extraction methods with probabilistic support vector machines.

Selecting Data for Fast Support Vector Machines Training

Abstract: In recent years, support vector machines (SVMs) have become a popular tool for pattern recognition and machine learning. Training a SVM involves solving a constrained quadratic programming problem, which requires large memory and enormous amounts of training time for largescale problems. In contrast, the SVM decision function is fully determined by a small subset of the training data, called support vectors. Therefore, it is desirable to remove from the training set the data that is irrelevant to the final decision function. In this paper we propose two new methods that select a subset of data for SVM training. Using real-world datasets, we compare the effectiveness of the proposed data selection strategies in terms of their ability to reduce the training set size while maintaining the generalization performance of the resulting SVM classifiers. Our experimental results show that a significant amount of training data can be removed by our proposed methods without degrading the performance of the resulting SVM classifiers.

An Improvement of One-Against-One Method for Multi-Class Support Vector Machine

Abstract: The support vector machine (SVM) has an excellent ability to solve binary classification problems. How to process multi-class problems with SVM is one of the present focuses. Among the existing multi-class SVM methods include one-against-one method, one-against-all method and some others. This paper presents an improved technique of one-against-one method that can largely reduce the number of the hyper-planes and speed up the predicting process. The experimental results show that the proposed method not only has promising accuracy and less training time, but also significantly improves the predicting speed in comparison with traditional one-against-one and one-against-all method.

Working Set Selection Using Functional Gain for LS-SVM

Abstract: The efficiency of sequential minimal optimization (SMO) depends strongly on the working set selection. This letter shows how the improvement of SMO in each iteration, named the functional gain (FG), is used to select the working set for least squares support vector machine (LS-SVM). We prove the convergence of the proposed method and give some theoretical support for its performance. Empirical comparisons demonstrate that our method is superior to the maximum violating pair (MVP) working set selection.

Precision and Recall in Rough Support Vector Machines

Abstract: Rough support vector machines (RSVMs) supplement conventional support vector machines (SVMs) by providing a better representation of the boundary region. Increasing interest has been paid to the theoretical development of RSVMs, which has already lead to a modification of existing SVM implementations as RSVMs. This paper shows how to extend the use of precision and recall from a SVM implementation to a RSVM implementation. Our approach is demonstrated in practice with the help of Gist, a popular SVM implementation.

An Approach for Incremental Semi-supervised SVM

Abstract: In this paper we propose an approach for incremental learning of semi-supervised SVM. The proposed approach makes use of the locality of radial basis function kernels to do local and incremental training of semi-supervised support vector machines. The algorithm introduces a se- quential minimal optimization based implementation of the branch and bound technique for training semi-supervised SVM problems. The novelty of our approach lies in the

Time Series Prediction Based on SVM and GA

Abstract: A new time series prediction method based on support vector machine (SVM) and genetic algorithm (GA) is proposed. At first, SVM is used to partition the whole input space into several disjointed regions. Secondly, GA is adopted to determine the parameter combination of the SVM corresponding to the partitioned region obtained above. At last, the different SVM in the different input-output spaces is constructed and used to predict time series. The simulation result shows that the multiple SVM achieve significant improvement in the generalization performance in comparison with the single SVM model.

SMO Algorithm Applied in Time Series Model Building and Forecast

Abstract: As a novel learning machine, the support vector machine (SVM) based on statistical learning theory can be used for regression: support vector regression (SVR). SVR has been applied successfully to time-series analysis, but its optimization algorithm is usually built up from certain quadratic programming (QP) packages. Therefore, for small datasets this is practical and QP routines are the best choice, but for large datasets, data processing runtimes become lengthy, which limits its application. Sequential minimal optimization (SMO) algorithm can improve operation speed and reduce this long runtime. In this paper, SVR that is based on the SMO algorithm is used to forecast two typical time series models: Wolfer sunspot number data and Box and Jenkins gas furnace data. The results of simulation prove that the operational speed of SVR using the SMO algorithm is improved in comparison to SVR employing QP optimization algorithm; moreover, the forecasting precision is better than that of neural network and SVR using QP optimization algorithm.

A genetic approach to training support vector data descriptors for background modeling in video data

Abstract: Detecting regions of interest in video sequences is one of the most important tasks in many high level video processing applications. In this paper a novel approach based on Support Vector Data Description (SVDD) is presented. The method detects foreground regions in videos with quasi-stationary backgrounds. The SVDD is a technique used in analytically describing the data from a set of population samples. The training of Support Vector Machines (SVM's) in general, and SVDD in particular requires a Lagrange optimization which is computationally intensive. We propose to use a genetic approach to solve the Lagrange optimization problem. The Genetic Algorithm (GA) starts with the initial guess and solves the optimization problem iteratively. Moreover, we expect to get accurate results with less cost than the Sequential Minimal Optimization (SMO) technique.

A Fast Learning Algorithm for One-Class Support Vector Machine

Abstract: Support vector machine (SVM) is a powerful tool to solve classification problems, this paper proposes a fast sequential minimal optimization (SMO) algorithm for training one-class support vector regression (OCSVM), firstly gives a analytical solution to the size two quadratic programming (QP) problem, then proposes a new heuristic method to select the working set which leads to algorithm's faster convergence. The simulation results indicate that the proposed SMO algorithm can reduce the training time of OCSVM, and the performance of proposed SMO algorithm is better than that of original SMO algorithm.

An Adaptive Internal Model Control Based on LS-SVM

Abstract: Based on least squares support vector machines regression algorithm, reverse model of system model is constructed, and adaptive internal model controller is developed in this paper. First, least squares support vector machine (LS-SVM) regression model and its training algorithm are introduced, provides SMO-based on pruning algorithms for LS-SVM. Then it is used in adaptive internal model control (IMC) for constructing internal model and designing the internal model controller. At last, LS-SVM regression based adaptive internal model control is used to control a benchmark nonlinear system. Simulation results show that the controller has simple structure, good control performance and robustness.

On Extending the SMO Algorithm Sub-Problem

Abstract: The support vector machine is a widely employed machine learning model due to its repeatedly demonstrated superior generalization performance. The sequential minimal optimization (SMO) algorithm is one of the most popular SVM training approaches. SMO is fast, as well as easy to implement; however, it has a limited working set size (2 points only). Faster training times can result if the working set size can be increased without significantly increasing the computational complexity. In this paper, we extend the 2-point SMO formulation to a 4-point formulation and address the theoretical issues associated with such an extension. We show that modifying the SMO algorithm to increase the working set size is beneficial in terms of the number of iterations required for convergence, and shows promise for reducing the overall training time.

Fast Classification Algorithm for Polynomial Kernel Support Vector Machines

Abstract: When the number of support vectors is large, the classification speed of a kernel function based on support vectors classifier is inevitably very slow in test phase, as it need to perform the computation between each support vector and the classified vector. To address this, a fast classification algorithm for polynomial kernel support vector machines is presented, which expands the decision function of SVM into polynomials, and classifies new patterns by calculating the polynomials’ value. The computational requirement of the algorithm is independent of the number of the support vectors, while the solution otherwise is unchanged. When the degree of the polynomial kernel or the dimension of the input space is small, the classification speed of this algorithm is much faster than the standard SVM classification method. The efficiency of this algorithm is also verified by the experiment result with real-world data set.

Sequential Minimal Optimization Algorithm Applied in Short-Term Load Forecasting

Abstract: Load forecasting is usually made by constructing models on relative information, such as climate and previous load demand data. Today more and more papers apply support vector machines in short-term load forecasting and get good results. In this paper we present a new load data qualification algorithm and combine it with improved sequential minimal optimization. The improved sequential minimal optimization algorithm can keep the length of the block data, which has remained unchanged, and it provides the excellent forecasting accuracy proved by the result of the experiment. The new load data qualification algorithm sorts the data with trade, and according to the electro-proportion of every trade does separate forecasting.

A New Orientation for Multi-Class SVM

Abstract: Combined by several binary-class SVMs, present multi-class SVMs are usually inefficient in training process. When there is large number of categories of data to classify, training it would be very difficult. Expanded from hyper-sphere one-class SVM (HS-SVM), hyper-sphere multi-class SIM (HSMC-SVM), which builds a HS-SVM for every category of data, is a direct classifier. Its training speed is faster than the combined multi-class classifiers. In order to fast train the HSMC-SVM, a training algorithm following the idea of SMO is proposed. For researching the generalization performance of HSMC-SVM, the theoretic upper bound of generalizing error of HSMC-SVM is analyzed too. As shown in the numeric experiments, the training speed of HSMC-SVM is faster than 1-v-r and 1-v-1, but the classification precision is lower than them. HSMC-SIM provides a new idea on researching fast directed multi-class classifiers in machine learning area.

A Novel Model of Working Set Selection for SMO Decomposition Methods

Abstract: In the process of training support vector machines (SVMs) by decomposition methods, working set selection is an important technique, and some exciting schemes were employed into this field. To improve working set selection, we propose a new model for working set selection in sequential minimal optimization (SMO) decomposition methods. In this model, it selects B as working set without reselection. Some properties are given by simple proof, and experiments demonstrate that the proposed method is in general faster than existing methods.

Candidate Vectors Selection for Training Support Vector Machines

Abstract: In this paper, a novel and concise method for the selection of candidate vectors (SCV) is proposed based on the structural information of two classes in the input space. First, the Euclidean distance of all samples to the boundary of the other classes is calculated. Then the relative distance is computed to reorder training samples ascendingly, and boundary samples will rank in front of others and have a higher probability to be candidate support vectors. A certain proportion of the foremost ranked samples are selected to form examples subset for training the SVM classification function by using the SMO. For linearly non-separable datasets with noise, an abnormal examples filtering (AEF) procedure is designed to find abnormal examples or outliers that may give rise to the distortion of structural information on the boundaries of two classes. Finally, two datasets are used to test the prediction accuracy of the SVM decision function estimated by the SMO and the AEF+SCV+SMO.

Small-shaped space target recognition based on wavelet decomposition and support vector machine

Abstract: A kind of method for small-shaped space target recognition was proposed in this paper based on feature extraction with wavelet decomposition and formative support vector machine (FSVM) with sequential minimal optimization (SMO) algorithm. Firstly, the significance and characteristics of space target recognition were discussed and a two-stage recognition strategy was designed. And then aiming at the characteristics of small-shaped space target recognition, a new method was implemented based on feature extraction with wavelet decomposition and FSVM with SMO algorithm. Simulation results show the good performance of the algorithm proposed in this paper: the correct rate is more than 97% within 1360 simulation samples of ten classes of small shaped space targets; meanwhile the algorithm is characterized with high speed of near real time in both implementation of training and testing.

P300 Detection Algorithm Based on Wavelet Decomposition and Support Vector Machines

Abstract: In order to overcome the shortcomings of support vector machines in terms of low training and detection speed for P300 detection,a new algorithm based on wavelet decomposition and support vector machines was proposed in this paper.Using wavelet decomposition and span estimation method,we implemented the feature extraction of EEG signals and the fast choice of the optimal parameters of support vector machines;and realized the detection of P300 component with support vector machines,which have a good classification performance.The algorithm was tested with a P300 dataset from the BCI competition 2003.The results showed that the algorithm designed in this paper was superior to the traditional support vector machines algorithms in terms of the training and detection speed.Using this algorithm,we achieved an accuracy of 100% in P300 detection within five repetitions.

Multi-fault diagnosis for turbo-pump based on mesh support vector machines

Abstract: Support vector machine(SVM) is a new general machine-learning tool based on structural risk minimization principle that exhibits good generalization even when fault samples are few. Fault diagnosis based on support vector ma- chines is discussed. Since basic support vector machines is originally designed for two-class classification, while most of fault diagnosis problems are multi-class cases, a new multi-class classification algorithm named mesh support vector machines is presented to solve the multi-class recognition prob- lems. It is a mesh classifier in which every class constructs two-class SVM classifiers with less than 4 other classes. It is simple and extensible, and has little repeated training amount, so the rate of training and recognition is expedited. The effec- tiveness of the method is verified by the application to the multi-fault diagnosis for turbo pump test bed.

Two sparsity-controlled schemes for 1-norm support vector classification

Abstract: Support vector machines (SVMs) are currently a very active research area for machine learning, data mining, etc. Sparsity control is an issue deserving further attention for the improvement of existing support vector machines techniques. This work presents two new sparsity control methods for 1- norm support vector classification. The first scheme, called SVC-sc1, is formulated by adding a penalty term in the objective function, whereas the second scheme, called SVC-sc2, is obtained by adding an extra inequality to the original optimization problem. The common goal is to reduce the number of retained support vectors. Besides mathematical formulation, we present test results on the benchmark Ripley data set. The experimental results indicate that both schemes outperform the conventional SVC, whereas SVC-sc2 has a still better performance than SVC-sc1.

Study on SVM On-Line Function Regression Method for Mass Data

Abstract: In order to overcome the problems that the SVM training time is too long for a large number of samples and that SVM cannot be trained online when the samples increase dynamically, a new approach of SVM online function regression for mass samples is put forward in this paper. And the validity of this method is proved by simulation experiment.

Fast Reduction for Large-Scale Training Data Set

Abstract: In order to cut down the time of training a large-scale data set by using SVM(support vector machine),a fast algorithm for reducing training sets was proposed based on class centroid.With this algorithm the most of non-support vectors are removed in the light of the geometrical distribution of samples.Experiments were made on several data sets at the level of 104 magnitude.The experimental results show that compared with the SMO(sequential minimal optimization) algorithm,the proposed algorithm decreases training time by 30% under the condition of ensuring the SVM's classification accuracy to greatly improve SVM's training speed.

Heuristics for Improving the Performance of Online SVM Training Algorithms

Abstract: Recently, the sequential minimal optimization algorithm (SMO) was introduced [1, 2] as an effective method for training support vector machines (SVMs) on classification tasks defined on sparse data sets. SMO differs from most SVM algorithms in that it does not require a quadratic programming (QP) solver. One problem with SMO is that its rate of convergence slows down dramatically when data is non-sparse and when there are many support vectors in the solution. This work addresses this issue by showing how the kernel function outputs can be effectively cached. Caching in SMO is non-trivial because SMO tends to randomly access data points in a training set. The proposed modifications can improve convergence time by orders of magnitude.

Generalized SMO-style decomposition algorithms

Abstract: Sequential Minimal Optimization (SMO) [14] is a major tool for solving convex quadratic optimization problems induced by Support Vector Machines (SVMs). It is based on the idea to iterativley solve subproblems of size two. In this work we will give a characterization of convex quadratic optimization problems, which can be solved with the SMO technique as well. In addition we will present an efficient 1/m- rate-certifying pair selection algorithm [8,13] leading to polynomial-time convergence rates for such problems.