Variable and feature selection have become the focus of much research in areas of application for which datasets with tens or hundreds of thousands of variables are available. These areas include text processing of internet documents, gene expression array analysis, and combinatorial chemistry. The objective of variable selection is three-fold: improving the prediction performance of the predictors, providing faster and more cost-effective predictors, and providing a better understanding of the underlying process that generated the data. The contributions of this special issue cover a wide range of aspects of such problems: providing a better definition of the objective function, feature construction, feature ranking, multivariate feature selection, efficient search methods, and feature validity assessment methods.

/pdf/an-introduction-to-variable-and-feature-selection-22eu49r0zd.pdf

An introduction to variable and feature selection

http://romisatriawahono.net/lecture/rm/survey/machine%20learning/Chandrashekar%20-%20Feature%20Selection%20Methods%20-%202014.pdf

A survey on feature selection methods

Cross-validation (CV) is an effective method for estimating the prediction error of a classifier. Some recent articles have proposed methods for optimizing classifiers by choosing classifier parameter values that minimize the CV error estimate. We have evaluated the validity of using the CV error estimate of the optimized classifier as an estimate of the true error expected on independent data. We used CV to optimize the classification parameters for two kinds of classifiers; Shrunken Centroids and Support Vector Machines (SVM). Random training datasets were created, with no difference in the distribution of the features between the two classes. Using these "null" datasets, we selected classifier parameter values that minimized the CV error estimate. 10-fold CV was used for Shrunken Centroids while Leave-One-Out-CV (LOOCV) was used for the SVM. Independent test data was created to estimate the true error. With "null" and "non null" (with differential expression between the classes) data, we also tested a nested CV procedure, where an inner CV loop is used to perform the tuning of the parameters while an outer CV is used to compute an estimate of the error. The CV error estimate for the classifier with the optimal parameters was found to be a substantially biased estimate of the true error that the classifier would incur on independent data. Even though there is no real difference between the two classes for the "null" datasets, the CV error estimate for the Shrunken Centroid with the optimal parameters was less than 30% on 18.5% of simulated training data-sets. For SVM with optimal parameters the estimated error rate was less than 30% on 38% of "null" data-sets. Performance of the optimized classifiers on the independent test set was no better than chance. The nested CV procedure reduces the bias considerably and gives an estimate of the error that is very close to that obtained on the independent testing set for both Shrunken Centroids and SVM classifiers for "null" and "non-null" data distributions. We show that using CV to compute an error estimate for a classifier that has itself been tuned using CV gives a significantly biased estimate of the true error. Proper use of CV for estimating true error of a classifier developed using a well defined algorithm requires that all steps of the algorithm, including classifier parameter tuning, be repeated in each CV loop. A nested CV procedure provides an almost unbiased estimate of the true error.

/pdf/bias-in-error-estimation-when-using-cross-validation-for-shjbtfp9y5.pdf

Bias in error estimation when using cross-validation for model selection

Motivation: Cancer diagnosis is one of the most important emerging clinical applications of gene expression microarray technology. We are seeking to develop a computer system for powerful and reliable cancer diagnostic model creation based on microarray data. To keep a realistic perspective on clinical applications we focus on multicategory diagnosis. To equip the system with the optimum combination of classifier, gene selection and cross-validation methods, we performed a systematic and comprehensive evaluation of several major algorithms for multicategory classification, several gene selection methods, multiple ensemble classifier methods and two cross-validation designs using 11 datasets spanning 74 diagnostic categories and 41 cancer types and 12 normal tissue types.

Results: Multicategory support vector machines (MC-SVMs) are the most effective classifiers in performing accurate cancer diagnosis from gene expression data. The MC-SVM techniques by Crammer and Singer, Weston and Watkins and one-versus-rest were found to be the best methods in this domain. MC-SVMs outperform other popular machine learning algorithms, such as k-nearest neighbors, backpropagation and probabilistic neural networks, often to a remarkable degree. Gene selection techniques can significantly improve the classification performance of both MC-SVMs and other non-SVM learning algorithms. Ensemble classifiers do not generally improve performance of the best non-ensemble models. These results guided the construction of a software system GEMS (Gene Expression Model Selector) that automates high-quality model construction and enforces sound optimization and performance estimation procedures. This is the first such system to be informed by a rigorous comparative analysis of the available algorithms and datasets.

Availability: The software system GEMS is available for download from http://www.gems-system.org for non-commercial use.

Contact: alexander.statnikov@vanderbilt.edu

/pdf/a-comprehensive-evaluation-of-multicategory-classification-3vk1si319h.pdf

A comprehensive evaluation of multicategory classification methods for microarray gene expression cancer diagnosis

. Perhaps the most straightforward classiﬁer in the arsenal ormachine learning techniques is the Nearest Neighbour Classiﬁer – clas-siﬁcation is achieved by identifying the nearest neighbours to a queryexample and using those neighbours to determine the class of the query.This approach to classiﬁcation is of particular importance today becauseissues of poor run-time performance is not such a problem these dayswith the computational power that is available. This paper presents anoverview of techniques for Nearest Neighbour classiﬁcation focusing on;mechanisms for assessing similarity (distance), computational issues inidentifying nearest neighbours and mechanisms for reducing the dimen-sion of the data. 1 Introduction The intuition underlying Nearest Neighbour Classiﬁcation is quite straightfor-ward, examples are classiﬁed based on the class of their nearest neighbours. Itis often useful to take more than one neighbour into account so the technique ismore commonly referred to as k-Nearest Neighbour (k-NN) Classiﬁcation wherek nearest neighbours are used in determining the class. Since the training ex-amples are needed at run-time, i.e. they need to be in memory at run-time,it is sometimes also called Memory-Based Classiﬁcation. Because induction isdelayed to run time, it is considered a Lazy Learning technique. Because classiﬁ-cation is based directly on the training examples it is also called Example-BasedClassiﬁcation or Case-Based Classiﬁcation.The basic idea is as shown in Figure 1 which depicts a 3-Nearest NeighbourClassiﬁer on a two-class problem in a two-dimensional feature space. In thisexample the decision for q

https://researchrepository.ucd.ie/bitstream/10197/12360/1/UCD-CSI-2007-3.pdf

k-Nearest Neighbour Classifiers

This paper addresses a common methodological flaw in the comparison of variable selection methods. A practical approach to guide the search or the selection process is to compute cross-validation performance estimates of the different variable subsets. Used with computationally intensive search algorithms, these estimates may overfit and yield biased predictions. Therefore, they cannot be used reliably to compare two selection methods, as is shown by the empirical results of this paper. Instead, like in other instances of the model selection problem, independent test sets should be used for determining the final performance. The claims made in the literature about the superiority of more exhaustive search algorithms over simpler ones are also revisited, and some of them infirmed.

/pdf/overfitting-in-making-comparisons-between-variable-selection-3xvpima8zn.pdf

Overfitting in making comparisons between variable selection methods

Feature selection researchers often encounter a peaking phenomenon: a feature subset can be found that is smaller but still enables building a more accurate classifier than the full set of all the candidate features However, the present study shows that this peak may often be just an artifact due to the still too common mistake in pattern recognition — that of not using an independent test set

A Pitfall in Determining the Optimal Feature Subset Size

. In order to have a good estimate of the current forcing by anthropogenic aerosols, knowledge on past aerosol levels is needed. Aerosol optical depth (AOD) is a good measure for aerosol loading. However, dedicated measurements of AOD are only available from the 1990s onward. One option to lengthen the AOD time series beyond the 1990s is to retrieve AOD from surface solar radiation (SSR) measurements taken with pyranometers. In this work, we have evaluated several inversion methods designed for this task. We compared a look-up table method based on radiative transfer modelling, a non-linear regression method and four machine learning methods (Gaussian process, neural network, random forest and support vector machine) with AOD observations carried out with a sun photometer at an Aerosol Robotic Network (AERONET) site in Thessaloniki, Greece. Our results show that most of the machine learning methods produce AOD estimates comparable to the look-up table and non-linear regression methods. All of the applied methods produced AOD values that corresponded well to the AERONET observations with the lowest correlation coefficient value being 0.87 for the random forest method. While many of the methods tended to slightly overestimate low AODs and underestimate high AODs, neural network and support vector machine showed overall better correspondence for the whole AOD range. The differences in producing both ends of the AOD range seem to be caused by differences in the aerosol composition. High AODs were in most cases those with high water vapour content which might affect the aerosol single scattering albedo (SSA) through uptake of water into aerosols. Our study indicates that machine learning methods benefit from the fact that they do not constrain the aerosol SSA in the retrieval, whereas the LUT method assumes a constant value for it. This would also mean that machine learning methods could have potential in reproducing AOD from SSR even though SSA would have changed during the observation period.

/pdf/retrieval-of-aerosol-optical-depth-from-surface-solar-g920fmcg3i.pdf

Retrieval of aerosol optical depth from surface solar radiation measurementsusing machine learning algorithms, non-linear regression and a radiativetransfer-based look-up table

Cross-indexing is a method for selecting the optimal model complexity and estimating the corresponding performance. It aims to reduce the optimistic selection bias that may emerge when too many models are compared to each other using cross-validation as the performance estimator. In this paper, a generalization is introduced that covers the previously presented variations (cross-indexing A and B) as special cases. Originally, cross-indexing was suggested for decreasing the selection bias in a feature selection setting. In order to apply it to a generic model selection problem that presents a large number of candidate model structures and an infinite number of potential hyperparameter values, the method needs to be modified. This paper also describes one way of doing such modifications, and reports the promising results of using the consequent method in three open competitions.

Model Selection and Assessment Using Cross-indexing

In an exemplary method, repeated patterns are identified in a strip-like product. In the method the strip-like product is observed by at least one camera, and at least one digital image signal comprised of pixels is created for inspection. The image signal is searched for anomalies comprised of one or more pixels. A search image is created of any detected anomaly and its neighbourhood, and the search image is used to convolute the image signal being examined, creating a response image signal. The response image signal is used to determine image areas in the image signal being examined that are substantially similar to the search image.

Juha Reunanen

Papers

Overfitting in making comparisons between variable selection methods

A Pitfall in Determining the Optimal Feature Subset Size

Retrieval of aerosol optical depth from surface solar radiation measurementsusing machine learning algorithms, non-linear regression and a radiativetransfer-based look-up table

Model Selection and Assessment Using Cross-indexing

Method and apparatus for identifying repeated patterns