LIBSVM is a library for Support Vector Machines (SVMs). We have been actively developing this package since the year 2000. The goal is to help users to easily apply SVM to their applications. LIBSVM has gained wide popularity in machine learning and many other areas. In this article, we present all implementation details of LIBSVM. Issues such as solving SVM optimization problems theoretical convergence multiclass classification probability estimates and parameter selection are discussed in detail.

/pdf/libsvm-a-library-for-support-vector-machines-2gdxasg9zv.pdf

LIBSVM: A library for support vector machines

LIBLINEAR is an open source library for large-scale linear classification. It supports logistic regression and linear support vector machines. We provide easy-to-use command-line tools and library calls for users and developers. Comprehensive documents are available for both beginners and advanced users. Experiments demonstrate that LIBLINEAR is very efficient on large sparse data sets.

/pdf/liblinear-a-library-for-large-linear-classification-52qheag29c.pdf

LIBLINEAR: A Library for Large Linear Classification

Support vector machine (SVM) is a popular technique for classication. However, beginners who are not familiar with SVM often get unsatisfactory results since they miss some easy but signicant steps. In this guide, we propose a simple procedure, which usually gives reasonable results.

/pdf/a-practical-guide-to-support-vector-classication-5fsvikyh0n.pdf

A Practical Guide to Support Vector Classication

During the last decade, the data sizes have grown faster than the speed of processors. In this context, the capabilities of statistical machine learning methods is limited by the computing time rather than the sample size. A more precise analysis uncovers qualitatively different tradeoffs for the case of small-scale and large-scale learning problems. The large-scale case involves the computational complexity of the underlying optimization algorithm in non-trivial ways. Unlikely optimization algorithms such as stochastic gradient descent show amazing performance for large-scale problems. In particular, second order stochastic gradient and averaged stochastic gradient are asymptotically efficient after a single pass on the training set.

/pdf/large-scale-machine-learning-with-stochastic-gradient-4i7b83n87u.pdf

Large-Scale Machine Learning with Stochastic Gradient Descent

In this paper we propose a new framework for learning from large scale datasets based on iterative learning from small mini-batches. By adding the right amount of noise to a standard stochastic gradient optimization algorithm we show that the iterates will converge to samples from the true posterior distribution as we anneal the stepsize. This seamless transition between optimization and Bayesian posterior sampling provides an inbuilt protection against overfitting. We also propose a practical method for Monte Carlo estimates of posterior statistics which monitors a "sampling threshold" and collects samples after it has been surpassed. We apply the method to three models: a mixture of Gaussians, logistic regression and ICA with natural gradients.

/pdf/bayesian-learning-via-stochastic-gradient-langevin-dynamics-518ae00ef7.pdf

Bayesian Learning via Stochastic Gradient Langevin Dynamics

Pairwise coupling is a popular multi-class classification method that combines all comparisons for each pair of classes. This paper presents two approaches for obtaining class probabilities. Both methods can be reduced to linear systems and are easy to implement. We show conceptually and experimentally that the proposed approaches are more stable than the two existing popular methods: voting and the method by Hastie and Tibshirani (1998)

/pdf/probability-estimates-for-multi-class-classification-by-3n6kgzu1rn.pdf

Probability Estimates for Multi-class Classification by Pairwise Coupling

Platt's probabilistic outputs for Support Vector Machines (Platt, J. in Smola, A., et al. (eds.) Advances in large margin classifiers. Cambridge, 2000) has been popular for applications that require posterior class probabilities. In this note, we propose an improved algorithm that theoretically converges and avoids numerical difficulties. A simple and ready-to-use pseudo code is included.

/pdf/a-note-on-platt-s-probabilistic-outputs-for-support-vector-51kdazqpnw.pdf

A note on Platt's probabilistic outputs for support vector machines

Large-scale logistic regression arises in many applications such as document classification and natural language processing. In this paper, we apply a trust region Newton method to maximize the log-likelihood of the logistic regression model. The proposed method uses only approximate Newton steps in the beginning, but achieves fast convergence in the end. Experiments show that it is faster than the commonly used quasi Newton approach for logistic regression. We also extend the proposed method to large-scale L2-loss linear support vector machines (SVM).

/pdf/trust-region-newton-method-for-logistic-regression-nsw9ihonyb.pdf

Trust Region Newton Method for Logistic Regression

Large-scale logistic regression arises in many applications such as document classification and natural language processing. In this paper, we apply a trust region Newton method to maximize the log-likelihood of the logistic regression model. The proposed method uses only approximate Newton steps in the beginning, but achieves fast convergence in the end. Experiments show that it is faster than the commonly used quasi Newton approach for logistic regression. We also compare it with linear SVM implementations.

/pdf/trust-region-newton-methods-for-large-scale-logistic-donurz4gvt.pdf

Trust region Newton methods for large-scale logistic regression

The Bradley-Terry model for obtaining individual skill from paired comparisons has been popular in many areas. In machine learning, this model is related to multi-class probability estimates by coupling all pairwise classification results. Error correcting output codes (ECOC) are a general framework to decompose a multi-class problem to several binary problems. To obtain probability estimates under this framework, this paper introduces a generalized Bradley-Terry model in which paired individual comparisons are extended to paired team comparisons. We propose a simple algorithm with convergence proofs to solve the model and obtain individual skill. Experiments on synthetic and re al data demonstrate that the algorithm is useful for obtaining multi-class probability estimates. Moreover, we discuss four extensions of the proposed model: 1) weighted individual skill, 2) home-field advantage, 3) ties, and 4) comparisons with more than two teams.

/pdf/generalized-bradley-terry-models-and-multi-class-probability-1pkoj9hhrk.pdf

Ruby C. Weng

Papers

Probability Estimates for Multi-class Classification by Pairwise Coupling

A note on Platt's probabilistic outputs for support vector machines

Trust Region Newton Method for Logistic Regression

Trust region Newton methods for large-scale logistic regression

Generalized Bradley-Terry Models and Multi-Class Probability Estimates