https://ai.stanford.edu/~ronnyk/wrappersPrint.pdf

Wrappers for feature subset selection

A central problem in machine learning is identifying a representative set of features from which to construct a classification model for a particular task. This thesis addresses the problem of feature selection for machine learning through a correlation based approach. The central hypothesis is that good feature sets contain features that are highly correlated with the class, yet uncorrelated with each other. A feature evaluation formula, based on ideas from test theory, provides an operational definition of this hypothesis. CFS (Correlation based Feature Selection) is an algorithm that couples this evaluation formula with an appropriate correlation measure and a heuristic search strategy. CFS was evaluated by experiments on artificial and natural datasets. Three machine learning algorithms were used: C4.5 (a decision tree learner), IB1 (an instance based learner), and naive Bayes. Experiments on artificial datasets showed that CFS quickly identifies and screens irrelevant, redundant, and noisy features, and identifies relevant features as long as their relevance does not strongly depend on other features. On natural domains, CFS typically eliminated well over half the features. In most cases, classification accuracy using the reduced feature set equaled or bettered accuracy using the complete feature set. Feature selection degraded machine learning performance in cases where some features were eliminated which were highly predictive of very small areas of the instance space. Further experiments compared CFS with a wrapper—a well known approach to feature selection that employs the target learning algorithm to evaluate feature sets. In many cases CFS gave comparable results to the wrapper, and in general, outperformed the wrapper on small datasets. CFS executes many times faster than the wrapper, which allows it to scale to larger datasets. Two methods of extending CFS to handle feature interaction are presented and experimentally evaluated. The first considers pairs of features and the second incorporates iii feature weights calculated by the RELIEF algorithm. Experiments on artificial domains showed that both methods were able to identify interacting features. On natural domains, the pairwise method gave more reliable results than using weights provided by RELIEF.

/pdf/correlation-based-feature-selection-for-machine-learning-578rirmvg5.pdf

Correlation-based Feature Selection for Machine Learning

Relief algorithms are general and successful attribute estimators. They are able to detect conditional dependencies between attributes and provide a unified view on the attribute estimation in regression and classification. In addition, their quality estimates have a natural interpretation. While they have commonly been viewed as feature subset selection methods that are applied in prepossessing step before a model is learned, they have actually been used successfully in a variety of settings, e.g., to select splits or to guide constructive induction in the building phase of decision or regression tree learning, as the attribute weighting method and also in the inductive logic programming.

A broad spectrum of successful uses calls for especially careful investigation of various features Relief algorithms have. In this paper we theoretically and empirically investigate and discuss how and why they work, their theoretical and practical properties, their parameters, what kind of dependencies they detect, how do they scale up to large number of examples and features, how to sample data for them, how robust are they regarding the noise, how irrelevant and redundant attributes influence their output and how different metrics influences them.

/pdf/theoretical-and-empirical-analysis-of-relieff-and-rrelieff-53x7gz1rmo.pdf

Theoretical and Empirical Analysis of ReliefF and RReliefF

We address the problem of finding a subset of features that allows a supervised induction algorithm to induce small high-accuracy concepts. We examine notions of relevance and irrelevance, and show that the definitions used in the machine learning literature do not adequately partition the features into useful categories of relevance. We present definitions for irrelevance and for two degrees of relevance. These definitions improve our understanding of the behavior of previous subset selection algorithms, and help define the subset of features that should be sought. The features selected should depend not only on the features and the target concept, but also on the induction algorithm. We describe a method for feature subset selection using cross-validation that is applicable to any induction algorithm, and discuss experiments conducted with ID3 and C4.5 on artificial and real datasets.

Irrelevant features and the subset selection problem

It is becoming increasingly apparent that, without some form of explanation capability, the full potential of trained artificial neural networks (ANNs) may not be realised. This survey gives an overview of techniques developed to redress this situation. Specifically, the survey focuses on mechanisms, procedures, and algorithms designed to insert knowledge into ANNs (knowledge initialisation), extract rules from trained ANNs (rule extraction), and utilise ANNs to refine existing rule bases (rule refinement). The survey also introduces a new taxonomy for classifying the various techniques, discusses their modus operandi, and delineates criteria for evaluating their efficacy.

Survey and critique of techniques for extracting rules from trained artificial neural networks

The MONK's problems: A Performance Comparison of Different Learning Algorithms

The ability to accurately identify and classify objects described in terms of feature vectors is highly dependent on the particular features chosen. In complex domains such as image understanding and machine learning it is frequently the case that there are many candidate features, but little insight as to which, if any, make up an effective set for identification and classification. In this chapter we describe a system that highly automates the process of constructing and selecting useful feature sets. A key element of the design of this system is the use of evolutionary computation techniques to manage the underlying search processes. Empirical results are presented which suggests that this approach is highly effective in complex, real world problem domains.

Evolutionary Feature Space Transformation

Concept acquisition is a form of inductive learning that induces general descriptions of concepts from specific instances of a given concept. AQ15 is a conceptual inductive learning program that uses feature-based examples of concepts to generate rules which describe the underlying concept. This program has been successfully applied to a wide variety of domains. We have been exploring the use of AQ15 on a difficult class of image processing problems, namely, to infer descriptions of textures from noisy examples. In this context, we find that AQ15 produces rules that are sub-optimal from two view points: 1) we need to minimize the number of features actually used for classification; and 2) we need to achieve high recognition rates with noisy data. Our multistrategy approach is to apply genetic algorithms to select the best feature subset to use in conjunction with AQ15 to achieve our goals. The proposed approach has been implemented and applied to an initial set of randomly selected texture data. Our results are encouraging and indicate significant advantages to a multistrategy approach in this domain.

/pdf/improving-the-performance-of-a-rule-induction-system-using-49kwkx3wqb.pdf

Haleh Vafaie

Papers

The MONK's problems: A Performance Comparison of Different Learning Algorithms

Evolutionary Feature Space Transformation

Improving the Performance of a Rule Induction System Using Genetic Algorithms

A Brief Review of AQ Learning Programs and Their Application to the MONKS Problems

Feafbre Space Mndormatlon