This chapter introduces the subject of statistical pattern recognition (SPR). It starts by considering how features are defined and emphasizes that the nearest neighbor algorithm achieves error rates comparable with those of an ideal Bayes’ classifier. The concepts of an optimal number of features, representativeness of the training data, and the need to avoid overfitting to the training data are stressed. The chapter shows that methods such as the support vector machine and artificial neural networks are subject to these same training limitations, although each has its advantages. For neural networks, the multilayer perceptron architecture and back-propagation algorithm are described. The chapter distinguishes between supervised and unsupervised learning, demonstrating the advantages of the latter and showing how methods such as clustering and principal components analysis fit into the SPR framework. The chapter also defines the receiver operating characteristic, which allows an optimum balance between false positives and false negatives to be achieved.

Statistical Pattern Recognition

We propose an independence criterion based on the eigen-spectrum of covariance operators in reproducing kernel Hilbert spaces (RKHSs), consisting of an empirical estimate of the Hilbert-Schmidt norm of the cross-covariance operator (we term this a Hilbert-Schmidt Independence Criterion, or HSIC). This approach has several advantages, compared with previous kernel-based independence criteria. First, the empirical estimate is simpler than any other kernel dependence test, and requires no user-defined regularisation. Second, there is a clearly defined population quantity which the empirical estimate approaches in the large sample limit, with exponential convergence guaranteed between the two: this ensures that independence tests based on HSIC do not suffer from slow learning rates. Finally, we show in the context of independent component analysis (ICA) that the performance of HSIC is competitive with that of previously published kernel-based criteria, and of other recently published ICA methods.

Measuring statistical dependence with Hilbert-Schmidt norms

A comprehensive survey on support vector machine classification: Applications, challenges and trends

Hyperspectral sensors record the reflectance from the Earth's surface over the full range of solar wavelengths with high spectral resolution. The resulting high-dimensional data contain rich information for a wide range of applications. However, for a specific application, not all the measurements are important and useful. The original feature space may not be the most effective space for representing the data. Feature mining, which includes feature generation, feature selection (FS), and feature extraction (FE), is a critical task for hyperspectral data classification. Significant research effort has focused on this issue since hyperspectral data became available in the late 1980s. The feature mining techniques which have been developed include supervised and unsupervised, parametric and nonparametric, linear and nonlinear methods, which all seek to identify the informative subspace. This paper provides an overview of both conventional and advanced feature reduction methods, with details on a few techniques that are commonly used for analysis of hyperspectral data. A general form that represents several linear and nonlinear FE methods is also presented. Experiments using two widely available hyperspectral data sets are included to illustrate selected FS and FE methods.

Feature Mining for Hyperspectral Image Classification

Hyperspectral imaging fully portrays materials through numerous and contiguous spectral bands. It is a very useful technique in various fields, including astronomy, medicine, food safety, forensics, and target detection. However, hyperspectral images include redundant measurements, and most classification studies encountered the Hughes phenomenon. Finding a small subset of effective features to model the characteristics of classes represented in the data for classification is a critical preprocessing step required to render a classifier effective in hyperspectral image classification. In our previous work, an automatic method for selecting the radial basis function (RBF) parameter (i.e., σ) for a support vector machine (SVM) was proposed. A criterion that contains the between-class and within-class information was proposed to measure the separability of the feature space with respect to the RBF kernel. Thereafter, the optimal RBF kernel parameter was obtained by optimizing the criterion. This study proposes a kernel-based feature selection method with a criterion that is an integration of the previous work and the linear combination of features. In this new method, two properties can be achieved according to the magnitudes of the coefficients being calculated: the small subset of features and the ranking of features. Experimental results on both one simulated dataset and two hyperspectral images (the Indian Pine Site dataset and the Pavia University dataset) show that the proposed method improves the classification performance of the SVM.

A Kernel-Based Feature Selection Method for SVM With RBF Kernel for Hyperspectral Image Classification

Soft-margin support vector machine (SVM) is one of the most powerful techniques for supervised classification. However, the performances of SVMs are based on choosing the proper kernel functions or proper parameters of a kernel function. It is extremely time consuming by applying the k-fold cross-validation (CV) to choose the almost best parameter. Nevertheless, the searching range and fineness of the grid method should be determined in advance. In this paper, an automatic method for selecting the parameter of the normalized kernel function is proposed. In the experimental results, it costs very little time than k-fold cross-validation for selecting the parameter by our proposed method. Moreover, the corresponding soft-margin SVMs can obtain more accurate or at least equal performance than the soft-margin SVMs by applying k-fold cross-validation to determine the parameters.

https://ir.nctu.edu.tw:443/bitstream/11536/15288/1/000299446100002.pdf

An Automatic Method for Selecting the Parameter of the Normalized Kernel Function to Support Vector Machines

Support vector machine (SVM) is one of the most powerful techniques for supervised classification. However, the performances of SVMs are based on choosing the proper kernel functions or proper parameters of a kernel function. It is extremely time consuming by applying the k-fold cross-validation (CV) to choose the almost best parameter. Nevertheless, the searching range and fineness of the grid method should be determined in advance. In this paper, an automatic method for selecting the parameter of the normalized kernel function is proposed. In the experimental results, it costs very little time than k-fold cross-validation for selecting the parameter by our proposed method. Moreover, the corresponding SVMs can obtain more accurate or at least equal performance than SVMs by applying k-fold cross-validation to determine the parameter.

Linear discriminant analysis (LDA) is a commonly used feature extraction method to resolve the Hughes phenomenon for classification. Moreover, many studies show that the spatia l information can greatly improve the classification perfor mance. Hence, for hyperspectral image classification, it is not only neces sary to use the available spectral information but also to ex ploit the spatial information. Recently, we proposed a fuzzy-based LDA (FLDA), an unsupervised feature extraction, and used it for clust ering problem. However, it is hard to apply in the image segmentation because the optimization problem is nonlinear and non-convex and the number of membership values, the product of the number of clusters and the number of pixels in the image, is too large. In this paper, a semi- supervised feature extraction method which is based on the scatter matrices of LDA and FLDA (FLDA) is proposed. The unknown samples and their membership values which are determined by the posteriors after applying the classifier are used to form the within- and between-cluster scatter matrices of FLDA. The experimental results on two hyperspectral images, the Washington DC Mall and the Indian Pine Site, show that the proposed method can yield a better classification performance than LDA in the small sam pling size problem.

/pdf/a-semi-supervised-feature-extraction-based-on-supervised-and-56x2qi0laf.pdf

A Semi-Supervised Feature Extraction based on Supervised and Fuzzy-based Linear Discriminant Analysis for Hyperspectral Image Classification

A k-nearest-neighbor classifier expects the class conditional probabilities to be locally constant. In this paper, we use the local separability based on NWFE criterion to establish an effective metric for computing a new neighborhood. For each test pattern, the modified neighborhood shrinks in the direction with high separability around this pattern and extends further in the other direction. This new neighborhood can often provide improvement in classification performance. Therefore, any neighborhood-based classifier can be employed by using the modified neighborhood. Then the class conditional probabilities tend to be more homogeneous in the modified neighborhood.

Hsin-Hua Ho

Papers

A Kernel-Based Feature Selection Method for SVM With RBF Kernel for Hyperspectral Image Classification

An Automatic Method for Selecting the Parameter of the Normalized Kernel Function to Support Vector Machines

An Automatic Method for Selecting the Parameter of the Normalized Kernel Function to Support Vector Machines

A Semi-Supervised Feature Extraction based on Supervised and Fuzzy-based Linear Discriminant Analysis for Hyperspectral Image Classification

A novel nearest neighbor classifier based on adaptive nonparametric separability