# Least Squares Support Vector Machine Classifiers

## Summary (1 min read)

### 1. Introduction

- Recently, support vector machines (Vapnik, 1995; Vapknik, 1998a; Vapnik, 1998b) have been introduced for solving pattern recognition problems.
- In this method one maps the data into a higher dimensional input space and one constructs an optimal separating hyperplane in this space.
- Later, the support vector method was extended for solving function estimation problems.
- In Section 3 the authors discuss the least squares support vector machine classifiers.
- In Section 4 examples are given to illustrate the support values and on a two-spiral benchmark problem.

### 2. Support Vector Machines for Classification

- In this Section the authors shortly review some basic work on support vector machines (SVM) for classification problems.
- XTk x+θ] (two layer neural SVM), whereσ, κ andθ are constants.
- Because the matrix associated with this quadratic programming problem is not indefinite, the solution to (11) will be global (Fletcher, 1987).

### 3. Least Squares Support Vector Machines

- N∑ k=1 αk{yk[wTϕ(xk)+ b] − 1+ ek}, (17) whereαk are Lagrange multipliers (which can be either positive or negative now due to the equality constraints as follows from the Kuhn-Tucker conditions (Fletcher, 1987)).
- (21) Hence, the classifier (1) is found by solving the linear set of Equations (20)–(21) instead of quadratic programming.
- The parameters of the kernels such asσ for the RBF kernel can be optimally chosen according to (12).
- The support valuesαk are proportional to the errors at the data points (18), while in the case of (14) most values are equal to zero.
- Hence, one could rather speak of a support value spectrum in the least squares case.

### 4. Examples

- The size of the circles indicated at the training data is chosen proportionally to the absolute values of the support values.
- This is different from SVM’s based on inequality constraints, where only points that are near the decision line have nonzero support values.
- The training data are shown on Figure 2 with two classes indicated by ’o’ and′∗′ (360 points with 180 for each class) in a two dimensional input space.
- The excellent generalization performance is clear from the decision boundaries shown on the figures.
- Other methods which have been applied to the two-spiral benchmark problem, such as the use of circular units (Ridella et al., 1997), have shown good performance as well.

### 5. Conclusions

- The authors discussed a least squares version of support vector machine classifiers.
- For a complicated two-spiral classification problem it is illustrated that a least squares SVM with RBF kernel is readily found with excellent generalization performance and low computational cost.

Did you find this useful? Give us your feedback

##### Citations

4,835 citations

### Cites background from "Least Squares Support Vector Machin..."

...Index Terms—Extreme learning machine (ELM), feature mapping, kernel, least square support vector machine (LS-SVM), proximal support vector machine (PSVM), regularization network....

[...]

...Cortes and Vapnik [1] study the relationship between SVM and multilayer feedforward neural networks and showed that SVM can be seen as a specific type of SLFNs, the so-called support vector networks....

[...]

3,672 citations

2,616 citations

### Cites methods from "Least Squares Support Vector Machin..."

...56. lssvmRadial t implements the least squares SVM (Suykens and Vandewalle, 1999), using the function lssvm in the kernlab package, with Gaussian kernel tuning the kernel spread with values 10−2..107....

[...]

1,841 citations

### Cites background from "Least Squares Support Vector Machin..."

...…the name “proximal support vector machines” (Fung and Mangasarian, 2001b,a), and Suykens et al., under the name “least-squares support vector machines” (Suykens and Vandewalle, 1999a,b, Suykens et al., 1999), both derive essentially the same algorithm (we view the presence or absence of a bias…...

[...]

1,767 citations

### Cites background from "Least Squares Support Vector Machin..."

...algorithms such as LS-SVM [49]....

[...]

...[39] further extended this study to generalized SLFNs with different type of hidden nodes (feature mappings) as well as kernels and showed that the simple unified algorithm of ELM can be obtained for regression, binary and multi-label classification cases which, however, have to be handled separately by SVMs and its variants [2, 45–49]....

[...]

##### References

^{1}

40,147 citations

### "Least Squares Support Vector Machin..." refers background in this paper

...Introduction Recently, support vector machines (Vapnik, 1995; Vapnik, 1998a; Vapnik, 1998b) have been introduced for solving pattern recognition problems....

[...]

...Being based onthe structural risk minimization principle and capacity concept with purecombinatorial definitions, the quality and complexity of the SVM solution does not depend directly on the dimensionality of the input space (Vapnik, 1995; Vapnik, 1998a; Vapnik, 1998b)....

[...]

...The functionφ(xk) in (9) is related then toψ(x, xk) by imposing φ(x) φ(xk) = ψ(x, xk), (10)...

[...]

...For all further details we refer to (Vapnik, 1995; Vapnik, 1998a; Vapnik, 1998b)....

[...]

...max αk Q1(αk;φ(xk)) = −12 N ∑ k,l=1 ykyl φ(xk) φ(xl) αkαl + N ∑ k=1 αk, (9)...

[...]

29,130 citations

26,531 citations

### "Least Squares Support Vector Machin..." refers background in this paper

...Introduction Recently, support vector machines (Vapnik, 1995; Vapnik, 1998a; Vapnik, 1998b) have been introduced for solving pattern recognition problems....

[...]

...Being based onthe structural risk minimization principle and capacity concept with purecombinatorial definitions, the quality and complexity of the SVM solution does not depend directly on the dimensionality of the input space (Vapnik, 1995; Vapnik, 1998a; Vapnik, 1998b)....

[...]

...The functionφ(xk) in (9) is related then toψ(x, xk) by imposing φ(x) φ(xk) = ψ(x, xk), (10)...

[...]

...For all further details we refer to (Vapnik, 1995; Vapnik, 1998a; Vapnik, 1998b)....

[...]

...Based on (10),Q2 can also be expressed in terms of ψ(xk, xl)....

[...]

^{1}

19,056 citations