We introduce a method of feature selection for Support Vector Machines. The method is based upon finding those features which minimize bounds on the leave-one-out error. This search can be efficiently performed via gradient descent. The resulting algorithms are shown to be superior to some standard feature selection algorithms on both toy data and real-life problems of face recognition, pedestrian detection and analyzing DNA microarray data.

/pdf/feature-selection-for-svms-2cjpx5wd2h.pdf

Feature Selection for SVMs

The Bayesian evidence framework has been successfully applied to the design of multilayer perceptrons (MLPs) in the work of MacKay. Nevertheless, the training of MLPs suffers from drawbacks like the nonconvex optimization problem and the choice of the number of hidden units. In support vector machines (SVMs) for classification, as introduced by Vapnik, a nonlinear decision boundary is obtained by mapping the input vector first in a nonlinear way to a high-dimensional kernel-induced feature space in which a linear large margin classifier is constructed. Practical expressions are formulated in the dual space in terms of the related kernel function, and the solution follows from a (convex) quadratic programming (QP) problem. In least-squares SVMs (LS-SVMs), the SVM problem formulation is modified by introducing a least-squares cost function and equality instead of inequality constraints, and the solution follows from a linear system in the dual space. Implicitly, the least-squares formulation corresponds to a regression formulation and is also related to kernel Fisher discriminant analysis. The least-squares regression formulation has advantages for deriving analytic expressions in a Bayesian evidence framework, in contrast to the classification formulations used, for example, in gaussian processes (GPs). The LS-SVM formulation has clear primal-dual interpretations, and without the bias term, one explicitly constructs a model that yields the same expressions as have been obtained with GPs for regression. In this article, the Bayesian evidence framework is combined with the LS-SVM classifier formulation. Starting from the feature space formulation, analytic expressions are obtained in the dual space on the different levels of Bayesian inference, while posterior class probabilities are obtained by marginalizing over the model parameters. Empirical results obtained on 10 public domain data sets show that the LS-SVM classifier designed within the bayesian evidence framework consistently yields good generalization performances.

Bayesian framework for least-squares support vector machine classifiers, Gaussian processes, and kernel fisher discriminant analysis

PART I: FUNDAMENTALS. 1 INTRODUCTION. 1.1 The World of Signals. 1.2 Digital Image Processing. 1.3 Elements of an Image Processing System. Appendix 1.A Selected List of Books on Image Processing and Computer Vision from Year 2000. References. 2 MATHEMATICAL PRELIMINARIES. 2.1 Laplace Transform. 2.2 Fourier Transform. 2.3 Z-Transform. 2.4 Cosine Transform. 2.5 Wavelet Transform. 3 IMAGE ENHANCEMENT. 3.1 Grayscale Transformation. 3.2 Piecewise Linear Transformation. 3.3 Bit Plane Slicing. 3.4 Histogram Equalization. 3.5 Histogram Specification. 3.6 Enhancement by Arithmetic Operations. 3.7 Smoothing Filter. 3.8 Sharpening Filter. 3.9 Image Blur Types and Quality Measures. 4 MATHEMATICAL MORPHOLOGY. 4.1 Binary Morphology. 4.2 Opening and Closing. 4.3 Hit-or-Miss Transform. 4.4 Grayscale Morphology. 4.5 Basic Morphological Algorithms. 4.6 Morphological Filters. 5 IMAGE SEGMENTATION. 5.1 Thresholding. 5.2 Object (Component) Labeling. 5.3 Locating Object Contours by the Snake Model. 5.4 Edge Operators. 5.5 Edge Linking by Adaptive Mathematical Morphology. 5.6 Automatic Seeded Region Growing. 5.7 A Top-Down Region Dividing Approach. 6 DISTANCE TRANSFORMATION AND SHORTEST PATH PLANNING. 6.1 General Concept. 6.2 Distance Transformation by Mathematical Morphology. 6.3 Approximation of Euclidean Distance. 6.4 Decomposition of Distance Structuring Element. 6.5 The 3D Euclidean Distance. 6.6 The Acquiring Approaches. 6.7 The Deriving Approaches. 6.8 The Shortest Path Planning. 6.9 Forward and Backward Chain Codes for Motion Planning. 6.10 A Few Examples. 7 IMAGE REPRESENTATION AND DESCRIPTION. 7.1 Run-Length Coding. 7.2 Binary Tree and Quadtree. 7.3 Contour Representation. 7.4 Skeletonization by Thinning. 7.5 Medial Axis Transformation. 7.6 Object Representation and Tolerance. 8 FEATURE EXTRACTION. 8.1 Fourier Descriptor and Moment Invariants. 8.2 Shape Number and Hierarchical Features. 8.3 Corner Detection. 8.4 Hough Transform. 8.5 Principal Component Analysis. 8.6 Linear Discriminate Analysis. 8.7 Feature Reduction in Input and Feature Spaces. 9 PATTERN RECOGNITION. 9.1 The Unsupervised Clustering Algorithm. 9.2 Bayes Classifier. 9.3 Support Vector Machine. 9.4 Neural Networks. 9.5 The Adaptive Resonance Theory Network. 9.6 Fuzzy Sets in Image Analysis. PART II: APPLICATIONS. 10 FACE IMAGE PROCESSING AND ANALYSIS. 10.1 Face and Facial Feature Extraction. 10.2 Extraction of Head and Face Boundaries and Facial Features. 10.3 Recognizing Facial Action Units. 10.4 Facial Expression Recognition in JAFFE Database. 11 DOCUMENT IMAGE PROCESSING AND CLASSIFICATION. 11.1 Block Segmentation and Classification. 11.2 Rule-Based Character Recognition System. 11.3 Logo Identification. 11.4 Fuzzy Typographical Analysis for Character Preclassification. 11.5 Fuzzy Model for Character Classification. 12 IMAGE WATERMARKING. 12.1 Watermarking Classification. 12.2 Spatial Domain Watermarking. 12.3 Frequency-Domain Watermarking. 12.4 Fragile Watermark. 12.5 Robust Watermark. 12.6 Combinational Domain Digital Watermarking. 13 IMAGE STEGANOGRAPHY. 13.1 Types of Steganography. 13.2 Applications of Steganography. 13.3 Embedding Security and Imperceptibility. 13.4 Examples of Steganography Software. 13.5 Genetic Algorithm-Based Steganography. 14 SOLAR IMAGE PROCESSING AND ANALYSIS. 14.1 Automatic Extraction of Filaments. 14.2 Solar Flare Detection. 14.3 Solar Corona Mass Ejection Detection. INDEX.

Image Processing and Pattern Recognition: Fundamentals and Techniques

http://www2.ece.ohio-state.edu/~zheng/publications/FS-SFS.pdf

FS_SFS: A novel feature selection method for support vector machines

http://www.bmva.org/bmvc/1999/papers/54.pdf

Support Vector Machines for Face Authentication

The success of a multimedia information system depends heavily on the way the data is represented. Although there are "natural" ways to represent numerical data, it is not clear what is a good way to represent multimedia data, such as images, video, or sound. We investigate various image representations where the quality of the representation is judged based on how well a system for searching through an image database can perform-although the same techniques and representations can be used for other types of object detection tasks or multimedia data analysis problems. The system is based on a machine learning method used to develop object detection models from example images that can subsequently be used for examples to detect-search-images of a particular object in an image database. As a base classifier for the detection task, we use support vector machines (SVM), a kernel based learning method. Within the framework of kernel classifiers, we investigate new image representations/kernels derived from probabilistic models of the class of images considered and present a new feature selection method which can be used to reduce the dimensionality of the image representation without significant losses in terms of the performance of the detection-search-system.

Image representations and feature selection for multimedia database search

This paper presents experimental comparisons of various image representations for object detection using kernel classifiers In particular it discusses the use of support vector machines (SVM) for object detection using as image representations raw pixel values, projections onto principal components, and Haar wavelets General linear transformations of the images through the choice of the kernel of the SVM are considered Experiments showing the effects of histogram equalization, a non-linear transformation, are presented Image representations derived from probabilistic models of the class of images considered, through the choice of the kernel of the SVM, are also evaluated Finally, we present a feature selection method using SVMs, and show experimental results

C. Papageorgiou

Papers

Image representations and feature selection for multimedia database search

Image representations for object detection using kernel classifiers