We derive multiplicative updates for solving the nonnegative quadratic programming problem in support vector machines (SVMs). The updates have a simple closed form, and we prove that they converge monotonically to the solution of the maximum margin hyperplane. The updates optimize the traditionally proposed objective function for SVMs. They do not involve any heuristics such as choosing a learning rate or deciding which variables to update at each iteration. They can be used to adjust all the quadratic programming variables in parallel with a guarantee of improvement at each iteration. We analyze the asymptotic convergence of the updates and show that the coefficients of non-support vectors decay geometrically to zero at a rate that depends on their margins. In practice, the updates converge very rapidly to good classifiers.

methods

/pdf/multiplicative-updates-for-nonnegative-quadratic-programming-4phz7fpmzo.pdf

Multiplicative Updatings for Support Vector Machines

Multiplicative Updates for Nonnegative Quadratic Programming in Support Vector Machines

Non-negative matrix factorization (NMF) has previously been shown to be a useful decomposition tool for multivariate data. Non-negative bases allow strictly additive combinations which have been shown to be part-based as well as relatively sparse. We pursue a discriminative decomposition by coupling NMF objective with a maximum margin classifier, specifically a support vector machine (SVM). Conversely, we propose an NMF based regularizer for SVM. We formulate the joint update equations and propose a new method which identifies the decomposition as well as the classification parameters. We present classification results on synthetic as well as real datasets.

http://www.ifp.illinois.edu/~mdgupta/paper/NMF_SVM_CVPR_11.pdf

Non-negative matrix factorization as a feature selection tool for maximum margin classifiers

We discuss a simple sparse linear problem that is hard to learn with any algorithm that uses a linear combination of the training instances as its weight vector. The hardness holds even if we allow the learner to embed the instances into any higher dimensional feature space (and use a kernel function to define the dot product between the embedded instances). These algorithms are inherently limited by the fact that after seeing k instances only a weight space of dimension k can be spanned.

Our hardness result is surprising because the same problem can be efficiently learned using the exponentiated gradient (EG) algorithm: Now the component-wise logarithms of the weights are essentially a linear combination of the training instances and after seeing k instances. This algorithm enforces additional constraints on the weights (all must be non-negative and sum to one) and in some cases these constraints alone force the rank of the weight space to grow as fast as 2k.

Leaving the span

Various problems in nonnegative quadratic programming arise in the training of large margin classifiers. We derive multiplicative updates for these problems that converge monotonically to the desired solutions for hard and soft margin classifiers. The updates differ strikingly in form from other multiplicative updates used in machine learning. In this paper, we provide complete proofs of convergence for these updates and extend previous work to incorporate sum and box constraints in addition to nonnegativity.

/pdf/multiplicative-updates-for-large-margin-classifiers-31doq9a7z5.pdf

Multiplicative Updatings for Support Vector Machines

Citations

Multiplicative Updates for Nonnegative Quadratic Programming in Support Vector Machines

Non-negative matrix factorization as a feature selection tool for maximum margin classifiers

Leaving the span

Multiplicative Updates for Large Margin Classifiers

A multiplicative updating algorithm for training support vector machine.

Related Papers (3)

Multiplicative Updatings for Support Vector Learning

Generalized support vector machines.

Simplified Support Vector Machine Based on Reduced Vector Set Method