Showing papers on "Sequential minimal optimization published in 2010"

A User's Guide to Support Vector Machines

Abstract: The Support Vector Machine (SVM) is a widely used classifier in bioinformatics. Obtaining the best results with SVMs requires an understanding of their workings and the various ways a user can influence their accuracy. We provide the user with a basic understanding of the theory behind SVMs and focus on their use in practice. We describe the effect of the SVM parameters on the resulting classifier, how to select good values for those parameters, data normalization, factors that affect training time, and software for training SVMs.

Multiple Kernel Learning and the SMO Algorithm

Abstract: Our objective is to train p-norm Multiple Kernel Learning (MKL) and, more generally, linear MKL regularised by the Bregman divergence, using the Sequential Minimal Optimization (SMO) algorithm. The SMO algorithm is simple, easy to implement and adapt, and efficiently scales to large problems. As a result, it has gained widespread acceptance and SVMs are routinely trained using SMO in diverse real world applications. Training using SMO has been a long standing goal in MKL for the very same reasons. Unfortunately, the standard MKL dual is not differentiable, and therefore can not be optimised using SMO style co-ordinate ascent. In this paper, we demonstrate that linear MKL regularised with the p-norm squared, or with certain Bregman divergences, can indeed be trained using SMO. The resulting algorithm retains both simplicity and efficiency and is significantly faster than state-of-the-art specialised p-norm MKL solvers. We show that we can train on a hundred thousand kernels in approximately seven minutes and on fifty thousand points in less than half an hour on a single core.

An ACO-based algorithm for parameter optimization of support vector machines

Abstract: One of the significant research problems in support vector machines (SVM) is the selection of optimal parameters that can establish an efficient SVM so as to attain desired output with an acceptable level of accuracy. The present study adopts ant colony optimization (ACO) algorithm to develop a novel ACO-SVM model to solve this problem. The proposed algorithm is applied on some real world benchmark datasets to validate the feasibility and efficiency, which shows that the new ACO-SVM model can yield promising results.

SMO-Style Algorithms for Learning Using Privileged Information.

Abstract: Recently Vapnik et al. [11, 12, 13] introduced a new learning model, called Learning Using Privileged Information (LUPI). In this model, along with standard training data, the teacher supplies the student with additional (privileged) information. In the optimistic case, the LUPI model can improve the bound for the probability of test error from O(1/ √ n) to O(1/n), where n is the number of training examples. Since semi-supervised learning model with n labeled and N unlabeled examples can only achieve the bound O(1/ √ n + N) in the optimistic case, the LUPI model can thus significantly outperform it. To implement LUPI model, Vapnik et al. [11, 12, 13] suggested to use an SVM-type algorithm called SVM+, which requires, however, to solve a more difficult optimization problem than the one that is traditionally used to solve SVM. In this paper we develop two new algorithms for solving the optimization problem of SVM+. Our algorithms have the structure similar to the empirically successful SMO algorithm for solving SVM. Our experiments show that in terms of the generalization error/running time tradeoff, one of our algorithms is superior over the widely used interior point optimizer.

Support vector machine for classification based on fuzzy training data

Abstract: Support vector machines (SVM) have been very successful in pattern recognition and function estimation problems, but in the support vector machines for classification, the training example is non-fuzzy input and output is y=+/-1; In this paper, we introduce the support vector machine which the training examples are fuzzy input, and give some solving procedure of the Support vector machine with fuzzy training data.

Adaptive pruning algorithm for least squares support vector machine classifier

Abstract: As a new version of support vector machine (SVM), least squares SVM (LS-SVM) involves equality instead of inequality constraints and works with a least squares cost function. A well-known drawback in the LS-SVM applications is that the sparseness is lost. In this paper, we develop an adaptive pruning algorithm based on the bottom-to-top strategy, which can deal with this drawback. In the proposed algorithm, the incremental and decremental learning procedures are used alternately and a small support vector set, which can cover most of the information in the training set, can be formed adaptively. Using this set, one can construct the final classifier. In general, the number of the elements in the support vector set is much smaller than that in the training set and a sparse solution is obtained. In order to test the efficiency of the proposed algorithm, we apply it to eight UCI datasets and one benchmarking dataset. The experimental results show that the presented algorithm can obtain adaptively the sparse solutions with losing a little generalization performance for the classification problems with no-noises or noises, and its training speed is much faster than sequential minimal optimization algorithm (SMO) for the large-scale classification problems with no-noises.

A Prediction Approach for Multichannel EEG Signals Modeling Using Local Wavelet SVM

Abstract: Accurate modeling of the multichannel electroencephalogram (EEG) signal is an important issue in clinical practice. In this paper, we propose a new local spatiotemporal prediction method based on support vector machines (SVMs). Combining with the local prediction method, the sequential minimal optimization (SMO) training algorithm, and the wavelet kernel function, a local SMO-wavelet SVM (WSVM) prediction model is developed to enhance the efficiency, effectiveness, and universal approximation capability of the prediction model. Both the spatiotemporal modeling from the measured time series and the details of the nonlinear modeling procedures are discussed. Simulations and experimental results with real EEG signals show that the proposed method is suitable for real signal processing and is effective in modeling the local spatiotemporal dynamics. This method greatly increases the computational speed and more effectively captures the local information of the signal.

Intelligent Optimization Methods for High-Dimensional Data Classification for Support Vector Machines

Abstract: Support vector machine (SVM) is a popular pattern classification method with many application areas. SVM shows its outstanding performance in high-dimensional data classification. In the process of classification, SVM kernel parameter setting during the SVM training procedure, along with the feature selection significantly influences the classification accuracy. This paper proposes two novel intelligent optimization methods, which simultaneously determines the parameter values while discovering a subset of features to increase SVM classification accuracy. The study focuses on two evolutionary computing approaches to optimize the parameters of SVM: particle swarm optimization (PSO) and genetic algorithm (GA). And we combine above the two intelligent optimization methods with SVM to choose appropriate subset features and SVM parameters, which are termed GA-FSSVM (Genetic Algorithm-Feature Selection Support Vector Machines) and PSO-FSSVM(Particle Swarm Optimization-Feature Selection Support Vector Machines) models. Experimental results demonstrate that the classification accuracy by our proposed methods outperforms traditional grid search approach and many other approaches. Moreover, the result indicates that PSO-FSSVM can obtain higher classification accuracy than GA-FSSVM classification for hyperspectral data.

A Modified Fixed-Threshold SMO for 1-Slack Structural SVMs

Abstract: In this paper, we describe a modified fixed-threshold sequential minimal optimization (FSMO) for 1-slack structural support vector machine (SVM) problems. Because the modified FSMO uses the fact that the formulation of 1-slack structural SVMs has no bias, it breaks down the quadratic programming (QP) problems of 1-slack structural SVMs into a series of smallest QP problems, each involving only one variable. For various test sets, the modified FSMO is as accurate as existing structural SVM implementations (n-slack and 1-slack SVM-struct) but is faster on large data sets.

A unified framework for Volatile Organic Compound classification and regression

Abstract: Motivated by the insect olfactory system, which resolves both the identity and the quantity of a nectar in parallel based on the same sensory cue, we address the problem of Volatile Organic Compound (VOC) classification and regression in a unified setting. We derive a maximum margin formulation for minimizing the empirical regression error and the classification error jointly, and then call the sequential minimal optimization procedure for solution. The solution yields a pool of support vectors that achieves both tasks almost equally accurately as individual performances of a support vector machine classifier and a support vector regressor designed independently. We investigate empirically the advantages and inconveniences of handling these two problems under a single formulation for odor identification and quantification. We demonstrate the method on an extensive dataset acquired by an array metal-oxide sensors for five VOC identities and a wide range of concentrations.

A distributed SVM for image annotation

Abstract: The popularity of SVMs has grown tremendously in the last few years for many different classification problems due to its generalization properties, however training SVMs require high computational power. Platt's SMO is one the fastest algorithm for training support vector machines, which takes the decomposition technique to the extreme by selecting a set of only two points as the working set then solving them analytically. However SMO becomes slow for large size training data set. In this paper we present a MapReduce based distributed implementation of SMO using Hadoop. The distributed SMO uses multiple core processors to process the training data. By partitioning the training data set into smaller subsets and allocating each of the partitioned subsets to a single Map task, each Map task optimizes the partition in parallel and finally the reducer combine the results. Experiments show the efficiency of the distributed SMO increases with the increase of the number of processors, the training speed of distributed SMO with 12 Map task is about 11times higher than standalone SMO. There is no significant difference in accuracy between distributed and standalone SMO.

A parallel and scalable digital architecture for training support vector machines

Abstract: To facilitate the application of support vector machines (SVMs) in embedded systems, we propose and test a parallel and scalable digital architecture based on the sequential minimal optimization (SMO) algorithm for training SVMs. By taking advantage of the mature and popular SMO algorithm, the numerical instability issues that may exist in traditional numerical algorithms are avoided. The error cache updating task, which dominates the computation time of the algorithm, is mapped into multiple processing units working in parallel. Experiment results show that using the proposed architecture, SVM training problems can be solved effectively with inexpensive fixed-point arithmetic and good scalability can be achieved. This architecture overcomes the drawbacks of the previously proposed SVM hardware that lacks the necessary flexibility for embedded applications, and thus is more suitable for embedded use, where scalability is an important concern.

A general-purpose dynamically reconfigurable SVM

Abstract: This paper presents an hardware implementation of the Sequential Minimal Optimization (SMO) for the Support Vector Machine (SVM) training phase. A general-purpose reconfigurable architecture, aimed to partial reconfiguration FPGAs, is developed, i.e., it supports different sizes of training sets, with wide-range number of samples and elements. The effects of fixed-point implementation are analyzed and data on area and frequency targeting the Xilinx Virtex-IV XC4VLX25 FPGA are provided. The architecture was able to perform the training in different learning benchmarks and the reconfigurable architecture was able to save 22.38% of FPGA's area.

Faster directions for second order SMO

Abstract: Second order SMO represents the state-of-the-art in SVM training for moderate size problems. In it, the solution is attained by solving a series of subproblems which are optimized w.r.t just a pair of multipliers. In this paper we will illustrate how SMO works in a two stage fashion, setting first the values of the bounded multipliers to the penalty factor C and proceeding then to adjust the non-bounded multipliers. Furthermore, during this second stage the selected pairs for update often appear repeatedly during the algorithm. Taking advantage of this, we shall propose a procedure to combine previously used descent directions that results in much fewer iterations in this second stage and that may also lead to noticeable savings in kernel operations.

A View of Support Vector Machines Algorithm on Classification Problems

Abstract: Support vector machine (SVM) algorithm has shown a good learning ability and generalization ability in classification, regression and forecasting. This paper mainly analyzes the the performance of support vector machine algorithm in the classification problem, including the algorithm in the kernel function selection, parameter optimization, and integration of other algorithms and to deal with multi-classification issues improvements. Concludes with a discussion of the SVM algorithm is the direction of further improvement.

Analysis of PCA based feature vectors for SVM posture classification

Abstract: Many classifiers have been employed to classify human posture classification; however, most of them only presents the average accuracy of the classification. Furthermore, the details of the measured parameters especially for SVM classifier are not measured. Therefore, the objective of this work is to analyse and classify human body posture using Support Vector Machine (SVM) techniques based on various two combinations of eigenpostures by considering two different solvers in the training process. The two solvers namely Sequential Minimal Optimization (SMO) and Matlab Quadratics Programming (QP) solvers have been studied and analyzed to perform the SVM training. The principal component analysis (PCA) method is applied to extract the features from human shape silhouettes. These extracted feature vectors are then used to perform human posture classification. Human posture evaluates which eigenpostures (feature vectors of the several eigenvalues) can be used to classify either human standing posture or human non-standing posture. Next, the solvers that produced the best performance in classifying human postures as well as the best combination of eigenpostures were selected. The results verified that the combination of second and fourth eigenpostures gives the superb performance with 100% correct classification and it is shown that the best solver in training process to classify human body posture classification is the SMO based on the shortest CPU time attained.

Dynamic fixed-point arithmetic design of embedded SVM-Based speaker identification system

Abstract: This work proposes a dynamic fixed-point arithmetic design for SVM-based speaker identification in embedded environment The whole speaker identification system includes LPCC extraction, SVM training with sequential minimal optimization (SMO), and SVM recognition The proposed dynamic fixed-point design is applied to each arithmetic procedure and fixed-point error analysis is also performed The fixed-point SVM-based speaker identification system have been implemented and evaluated on ARM9 DMA2400 The experimental results show that the speaker identification accuracy is slightly degraded with the proposed dynamic fixed-point technique.

An Efficient Support Vector Machine Algorithm for Solving Multi-class Pattern Recognition Problems

Abstract: In this paper, an efficient support vector machine (SVM) algorithm for solving multi-class pattern recognition problems is proposed. The samples in each class are trained by one-class SVM (OCSVM), respectively. And then several sets of support vectors (SVs) are obtained, which well express the distribution of the original training samples. These SVs finally are combined into a set of training samples and trained by one-versus-one (OVO) method. The experimental results show the proposed method can reduce the time of training procedure meanwhile the classification accuracy is not reduced. Furthermore, it generates less SVs than traditional way.

Semi-supervised facial expressions annotation using co-training with fast probabilistic tri-class SVMs

Abstract: Supervised learning requires a large amount of labeled data but the data labeling process can be expensive and time consuming, as it requires the efforts of human experts. Semi-supervised learning methods that can reduce the amount of required labeled data through exploiting the available unlabeled data to improve the classification accuracy. Here, we propose a learning framework to exploit the unlabeled data by decomposing multi-class problems into a set of binary problems and apply Co-Training to each binary problem. A probabilistic version of Tri-Class Support Vector Machine is proposed (SVM) that can discriminate between ignorance and uncertainty and an updated version of Sequential Minimal Optimization (SMO) algorithm is used for fast learning of Tri-Class SVMs. The proposed framework is applied to facial expressions recognition task. The results show that Co-Training can exploit effectively the independent views and the unlabeled data to improve the recognition accuracy of facial expressions.

Feature selection and fast training of subspace based support vector machines

Abstract: In this paper, we propose two methods for subspace based support vector machines (SS-SVMs) which are subspace based least squares support vector machines (SSLS-SVMs) and subspace based linear programming support vector machines (SSLP-SVMs): 1) optimum selection of the dictionaries of each class subspace from the standpoint of classification separability, and 2) speeding up training SS-SVMs. In method 1), for SSLS-SVMs, we select the dictionaries with optimized weights, and for SSLP-SVMs, we select the dictionaries without non-negative constraints. In method 2), the empirical feature space is obtained by using only the training data belonging to a class instead of using all the training data. Thus the dimension of the empirical feature space and training cost become lower. We demonstrate the effectiveness of the proposed methods over the conventional method for two-class bench mark datasets.

A sequential minimal optimization algorithm for the all-distances support vector machine

Abstract: The All-Distances SVM is a single-objective light extension of the binary µ-SVM for multi-category classification that is competitive against multi-objective SVMs, such as One-against-the-Rest SVMs and One-against-One SVMs. Although the model takes into account considerably less constraints than previous formulations, it lacks of an efficient training algorithm, making its use with medium and large problems impracticable. In this paper, a Sequential Minimal Optimization-like algorithm is proposed to train the All-Distances SVM, making large problems abordable. Experimental results with public benchmark data are presented to show the performance of the AD-SVM trained with this algorithm against other single-objective multi-category SVMs.

On the design of a hardware-software architecture for acceleration of SVM's training phase

Abstract: Support Vector Machines (SVM) is a new family of Machine Learning techniques that have been used in many areas showing remarkable results. Since training SVM scales quadratically (or worse) according of data size, it is worth to explore novel implementation approaches to speed up the execution of this type of algorithms. In this paper, a hardware-software architecture to accelerate the SVM training phase is proposed. The algorithm selected to implement the architecture is the Sequential Minimal Optimization (SMO) algorithm, which was partitioned so a General Purpose Processor (GPP) executes operations and control flow while the coprocessor executes tasks than can be performed in parallel. Experiments demonstrate that the proposed architecture can speed up SVM training phase 178.7 times compared against a software-only implementation of this algorithm.

Classification Based on Particle Swarm Optimization for Least Square Support Vector Machines Training

Abstract: Classification problem is an important and complex problem in machine learning. Support vector machine (SVM) has recently emerged as a powerful technique for solving problems in classification, but its performance mainly depends on the parameters selection of it. Parameters selection for SVM is very complex in nature and quite hard to solve by conventional optimization techniques such as least Squares algorithm, for it has some limitations associated with overfitting, local optimum problems. So in this paper, Particle Swarm Optimization algorithm is proposed to choose the parameters of support vector machine (SVM) automatically in Classification problem. This method has been applied in Iris classification problem, the results show that the proposed approach has a better generalization performance and is also more accurate and effective than LS-SVM.

Parameters Optimization of Support Vector Machine based on Improved Genetic Algorithm

Abstract: Support Vector Machines(SVM) is a promising artificial intelligence technique,but there is not a mature theoretic for choosing the parameters of SVM,which causes much discommodity to the appliance of SVM.Therefore,the Improved Genetic Algorithm was proposed utilizing the comprehensive searching ability to choose the parameters of SVM in this article in order to gain the classic parameters.Experimental results demonstrate an improvement of the generalization performance for support vector machines,which showed that this method is proved to be effectual.

Support Vector Machine Algorithm for Imbalanced Data Learning

Abstract: This paper proposes an improved SMO solving a quadratic optmization problem for class imbalanced learning. The SMO algorithm is aproporiate for solving the optimization problem of a support vector machine that assigns the different regularization values to the two classes, and the prosoposed SMO learning algorithm iterates the learning steps to find the current optimal solutions of only two Lagrange variables selected per class. The proposed algorithm is tested with the UCI benchmarking problems and compared to the experimental results of the SMO algorithm with the g-mean measure that considers class imbalanced distribution for gerneralization performance. In comparison to the SMO algorithm, the proposed algorithm is effective to improve the prediction rate of the minority class data and could shorthen the training time.

SVM in Predicting the Deformation of Deep Foundation Pit in Soft Soil Area

Abstract: Based on the measured deformation data series, future deformation value of deep foundation pit was predicted using Support Vector Machine (SVM) model in soft soil area Gauss kernel function, Sequential minimal optimization arithmetic, and the parameter value of C ande were determined by testing By using the method in example, results are shown to be in good agreement with measured data and laws reported in paper, and illustrates that SVM could perform well in solving fuzzy geotechnical engineering problem similar to deformation prediction As another act, the method and conclusion can be considered as reference for colleagues

A Novel Method for Training Large Scale E-Business SVM Models in a Grid Computing Environment

Abstract: The Support Vector Machines (SVM) become popular E-Business data mining tools recently, and the datasets of E-Business are usually large-scale. If Support Vector Machines are trained on large-scale datasets, the training time will be very long and the classifier’s accuracy will become lower too. As training a large-scale SVM is equated to solve a large-scale quadratic programming (QP) problem, so Path Following Interior Point Method (IPM) that can efficiently solve large scale QP problem in polynomial time is proposed to construct a new SVM learning algorithm on large-scale datasets. To improve the SVM learning efficiency, the dimensions of IPM direction equations are degraded first, then LDLT parallel decomposition method is used to solve the direction sub-equations efficiently, and the parallel algorithm is implemented in the ProActive grid-computing environment. The experiment results show that the new parallel SVM training algorithm is efficient and the SVM classifying accuracy is higher than libsvm.

A fast training method for OC-SVM based on the random sampling lemma

Abstract: Recently, One-class Support Vector Machine (OC-SVM) has been introduced to detect novel data or outliers. The key problem of training an OC-SVM is how to solve the constrained quadratic programming problem. The optimization process suffers from the problem of memory and time consuming. We present a new method to efficiently train the OC-SVM. Based on the random sampling lemma, the training dataset was firstly decomposed into subsets and each OC-SVM of subset was trained by Sequential Minimal Optimization (SMO). The combining lemmas of support vectors and outliers of OC-SVM were deduced. A new decision boundary was merged by decomposing and combining lemmas (DC). Experimental results demonstrate that the proposed method not only can handle larger scale data sets than standard SMO, but also outperforms SMO in time consumption.

Incremental Learning of Support Vector Machine Based on Hyperspheres

Abstract: Support vector machine (SVM) has been successfully applied to solving classification and regression problems. However,to solve the quadratic programming problem is needed for SVM training process,and the more the number of training samples,the longer the training process. A novel incremental learning method was therefore proposed combining the hypershpere approach with regressional SVM to reduce the number of training samples by using two concentric hypershperes,thus shortening the training time. Analysis results showed that the incremental learning method has lower computational complexity than the normal SVM training method. Experimental results demonstrated that the method proposed can dramatically save the training time with negaligible degradation of regression accuracy.