This paper investigates the application of the mutual information criterion to evaluate a set of candidate features and to select an informative subset to be used as input data for a neural network classifier. Because the mutual information measures arbitrary dependencies between random variables, it is suitable for assessing the "information content" of features in complex classification tasks, where methods bases on linear relations (like the correlation) are prone to mistakes. The fact that the mutual information is independent of the coordinates chosen permits a robust estimation. Nonetheless, the use of the mutual information for tasks characterized by high input dimensionality requires suitable approximations because of the prohibitive demands on computation and samples. An algorithm is proposed that is based on a "greedy" selection of the features and that takes both the mutual information with respect to the output class and with respect to the already-selected features into account. Finally the results of a series of experiments are discussed. >

/pdf/using-mutual-information-for-selecting-features-in-4ngyhldccw.pdf

Using mutual information for selecting features in supervised neural net learning

Introduction to neural networks.

The computational brain: Patricia S. Churchland and Terrence J. Sejnowski (MIT Press, Cambridge, MA, 1992); xi, 544 pages, $39.95

From the Publisher:
The concept of large margins is a unifying principle for the analysis of many different approaches to the classification of data from examples, including boosting, mathematical programming, neural networks, and support vector machines. The fact that it is the margin, or confidence level, of a classification--that is, a scale parameter--rather than a raw training error that matters has become a key tool for dealing with classifiers. This book shows how this idea applies to both the theoretical analysis and the design of algorithms.
The book provides an overview of recent developments in large margin classifiers, examines connections with other methods (e.g., Bayesian inference), and identifies strengths and weaknesses of the method, as well as directions for future research. Among the contributors are Manfred Opper, Vladimir Vapnik, and Grace Wahba.

Advances in Large Margin Classifiers

The author extends a previous review and focuses on feed-forward neural-net classifiers for static patterns with continuous-valued inputs. He provides a taxonomy of neural-net classifiers, examining probabilistic, hyperplane, kernel, and exemplar classifiers. He then discusses back-propagation and decision-tree classifiers; matching classifier complexity to training data; GMDH (generalized method of data handling) networks and high-order nets; K nearest-neighbor classifiers; the feature-map classifier; the learning vector quantizer; hypersphere classifiers; and radial-basis function classifiers. >

Pattern classification using neural networks

https://papers.cnl.salk.edu/PDFs/Analysis%20of%20Hidden%20Units%20in%20a%20Layered%20Network%20Trained%20to%20Classify%20Sonar%20Targets%201988-2996.pdf

R.Paul Gorman

Papers

Analysis of Hidden Units in a Layered Network Trained to Classify Sonar Targets