For classification problems, the generalized eigenvalue proximal support vector machine (GEPSVM) and twin support vector machine (TWSVM) are regarded as milestones in the development of the powerful SVMs, as they use the nonparallel hyperplane classifiers. In this brief, we propose an improved version, named twin bounded support vector machines (TBSVM), based on TWSVM. The significant advantage of our TBSVM over TWSVM is that the structural risk minimization principle is implemented by introducing the regularization term. This embodies the marrow of statistical learning theory, so this modification can improve the performance of classification. In addition, the successive overrelaxation technique is used to solve the optimization problems to speed up the training procedure. Experimental results show the effectiveness of our method in both computation time and classification accuracy, and therefore confirm the above conclusion further.

Improvements on Twin Support Vector Machines

Feedforward deep neural networks that employ multiple hidden layers show high performance in many applications, but they demand complex hardware for implementation. The hardware complexity can be much lowered by minimizing the word-length of weights and signals, but direct quantization for fixed-point network design does not yield good results. We optimize the fixed-point design by employing backpropagation based retraining. The designed fixed-point networks with ternary weights (+1, 0, and −1) and 3-bit signal show only negligible performance loss when compared to the floating-point coun-terparts. The backpropagation for retraining uses quantized weights and fixed-point signal to compute the output, but utilizes high precision values for adapting the networks. A character recognition and a phoneme recognition examples are presented.

Fixed-point feedforward deep neural network design using weights +1, 0, and −1

In this paper we examine the effect of receptive field designs on classification accuracy in the commonly adopted pipeline of image classification. While existing algorithms usually use manually defined spatial regions for pooling, we show that learning more adaptive receptive fields increases performance even with a significantly smaller codebook size at the coding layer. To learn the optimal pooling parameters, we adopt the idea of over-completeness by starting with a large number of receptive field candidates, and train a classifier with structured sparsity to only use a sparse subset of all the features. An efficient algorithm based on incremental feature selection and retraining is proposed for fast learning. With this method, we achieve the best published performance on the CIFAR-10 dataset, using a much lower dimensional feature space than previous methods.

/pdf/beyond-spatial-pyramids-receptive-field-learning-for-pooled-4s351qvs9w.pdf

Beyond spatial pyramids: Receptive field learning for pooled image features

Under the framework of graph-based learning, the key to robust subspace clustering and subspace learning is to obtain a good similarity graph that eliminates the effects of errors and retains only connections between the data points from the same subspace (i.e., intrasubspace data points). Recent works achieve good performance by modeling errors into their objective functions to remove the errors from the inputs. However, these approaches face the limitations that the structure of errors should be known prior and a complex convex problem must be solved. In this paper, we present a novel method to eliminate the effects of the errors from the projection space (representation) rather than from the input space. We first prove that $\boldsymbol {\ell }_{\boldsymbol {1}}$ -, $\boldsymbol {\ell }_{\boldsymbol {2}}$ -, $\boldsymbol {\ell }_{\boldsymbol {\infty }}$ -, and nuclear-norm-based linear projection spaces share the property of intrasubspace projection dominance, i.e., the coefficients over intrasubspace data points are larger than those over intersubspace data points. Based on this property, we introduce a method to construct a sparse similarity graph, called L2-graph. The subspace clustering and subspace learning algorithms are developed upon L2-graph. We conduct comprehensive experiment on subspace learning, image clustering, and motion segmentation and consider several quantitative benchmarks classification/clustering accuracy, normalized mutual information, and running time. Results show that L2-graph outperforms many state-of-the-art methods in our experiments, including L1-graph, low rank representation (LRR), and latent LRR, least square regression, sparse subspace clustering, and locally linear representation.

Constructing the L2-Graph for Robust Subspace Learning and Subspace Clustering

Beyond sparsity: The role of L1-optimizer in pattern classification

In this brief paper, we propose a method of feature extraction for digit recognition that is inspired by vision research: a sparse-coding strategy and a local maximum operation. We show that our method, despite its simplicity, yields state-of-the-art classification results on a highly competitive digit-recognition benchmark. We first employ the unsupervised Sparsenet algorithm to learn a basis for representing patches of handwritten digit images. We then use this basis to extract local coefficients. In a second step, we apply a local maximum operation to implement local shift invariance. Finally, we train a support vector machine (SVM) on the resulting feature vectors and obtain state-of-the-art classification performance in the digit recognition task defined by the MNIST benchmark. We compare the different classification performances obtained with sparse coding, Gabor wavelets, and principal component analysis (PCA). We conclude that the learning of a sparse representation of local image patches combined with a local maximum operation for feature extraction can significantly improve recognition performance.

Simple Method for High-Performance Digit Recognition Based on Sparse Coding

https://www.inb.uni-luebeck.de/fileadmin/files/PUBPDFS/LaBaMa09.pdf

Sparse Coding Neural Gas: Learning of overcomplete data representations

Particular classes of signals, as for example natural images, can be encoded sparsely if appropriate dictionaries are used Finding such dictionaries based on data samples, however, is a difficult optimization task In this paper, it is shown that simple stochastic gradient descent, besides being much faster, leads to superior dictionaries compared to the Method of Optimal Directions (MOD) and the K-SVD algorithm The gain is most significant in the difficult but relevant case of highly overlapping subspaces, ie, when the data samples are jointly represented by a restricted set of dictionary elements Moreover, the so-called Bag of Pursuits method is introduced as an extension of Orthogonal Matching Pursuit, and it is shown that it provides an improved approximation of the optimal sparse coefficients and, therefore, significantly improves the performance of the here proposed gradient descent as well as of the MOD and K-SVD approaches Finally, it is shown how the Bag of Pursuits and a generalized version of the Neural Gas algorithm can be used to derive an even more powerful method for sparse coding Performance is analyzed based on both synthetic data and the practical problem of image deconvolution In the latter case, two different dictionaries are learned for sample images of buildings and flowers, respectively It is demonstrated that the learned dictionaries do indeed adapt to the image class and that they therefore yield superior reconstruction results

Robust and Fast Learning of Sparse Codes With Stochastic Gradient Descent

We consider the problem of learning an unknown (overcom- plete) basis from an unknown sparse linear combination. Introducing the "sparse coding neural gas" algorithm, we show how to employ a combina- tion of the original neural gas algorithm and Oja's rule in order to learn a simple sparse code that represents each training sample by a multiple of one basis vector. We generalise this algorithm using orthogonal matching pursuit in order to learn a sparse code where each training sample is rep- resented by a linear combination of k basis elements. We show that this method can be used to learn artificial sparse overcomplete codes.

Learning Data Representations with Sparse Coding Neural Gas

The well-known MinOver algorithm is a slight modification of the perceptron algorithm and provides the maximum-margin classifier without a bias in linearly separable two-class classification problems. DoubleMinOver as an extension of MinOver, which now includes a bias, is introduced. An O(t-1) convergence is shown, where t is the number of learning steps. The computational effort per step increases only linearly with the number of patterns. In its formulation with kernels, selected training patterns have to be stored. A drawback of MinOver and DoubleMinOver is that this set of patterns does not consist of support vectors only. DoubleMaxMinOver, as an extension of DoubleMinOver, overcomes this drawback by selectively forgetting all nonsupport vectors after a finite number of training steps. It is shown how this iterative procedure that is still very similar to the perceptron algorithm can be extended to classification with soft margins and be used for training least squares support vector machines (SVMs). On benchmarks, the SoftDoubleMaxMinOver algorithm achieves the same performance as standard SVM software.

Kai Labusch

Papers

Simple Method for High-Performance Digit Recognition Based on Sparse Coding

Sparse Coding Neural Gas: Learning of overcomplete data representations

Robust and Fast Learning of Sparse Codes With Stochastic Gradient Descent

Learning Data Representations with Sparse Coding Neural Gas

SoftDoubleMaxMinOver: Perceptron-Like Training of Support Vector Machines