On the approximate realization of continuous mappings by neural networks

Sparse representation based classification has led to interesting image recognition results, while the dictionary used for sparse coding plays a key role in it. This paper presents a novel dictionary learning (DL) method to improve the pattern classification performance. Based on the Fisher discrimination criterion, a structured dictionary, whose dictionary atoms have correspondence to the class labels, is learned so that the reconstruction error after sparse coding can be used for pattern classification. Meanwhile, the Fisher discrimination criterion is imposed on the coding coefficients so that they have small within-class scatter but big between-class scatter. A new classification scheme associated with the proposed Fisher discrimination DL (FDDL) method is then presented by using both the discriminative information in the reconstruction error and sparse coding coefficients. The proposed FDDL is extensively evaluated on benchmark image databases in comparison with existing sparse representation and DL based classification methods.

/pdf/fisher-discrimination-dictionary-learning-for-sparse-4uyzo0y9li.pdf

Fisher Discrimination Dictionary Learning for sparse representation

This paper describes error-correction adjustment procedures for determining the weight vector of linear pattern classifiers under general pattern distribution. It is mainly aimed at clarifying theoretically the performance of adaptive pattern classifiers. In the case where the loss depends on the distance between a pattern vector and a decision boundary and where the average risk function is unimodal, it is proved that, by the procedures proposed here, the weight vector converges to the optimal one even under nonseparable pattern distributions. The speed and the accuracy of convergence are analyzed, and it is shown that there is an important tradeoff between speed and accuracy of convergence. Dynamical behaviors, when the probability distributions of patterns are changing, are also shown. The theory is generalized and made applicable to the case with general discriminant functions, including piecewise-linear discriminant functions.

A Theory of Adaptive Pattern Classifiers

Unlike a univariate decision tree, a multivariate decision tree is not restricted to splits of the instance space that are orthogonal to the features' axes. This article addresses several issues for constructing multivariate decision trees: representing a multivariate test, including symbolic and numeric features, learning the coefficients of a multivariate test, selecting the features to include in a test, and pruning of multivariate decision trees. We present several new methods for forming multivariate decision trees and compare them with several well-known methods. We compare the different methods across a variety of learning tasks, in order to assess each method's ability to find concise, accurate decision trees. The results demonstrate that some multivariate methods are in general more effective than others (in the context of our experimental assumptions). In addition, the experiments confirm that allowing multivariate tests generally improves the accuracy of the resulting decision tree over a univariate tree.

/pdf/multivariate-decision-trees-500ll42ehz.pdf

Multivariate Decision Trees

Articles, books, and technical reports on the theoretical and experimental estimation of probability of misclassification are listed for the case of correctly labeled or preclassified training data. By way of introduction, the problem of estimating the probability of misclassification is discussed in order to characterize the contributions of the literature.

Bibliography on estimation of misclassification

This paper describes iterative procedures for determining linear and piecewise linear discriminant functions for multicategory pattern classifiers. While classifiers with the same structure have often been proposed, it is less well known that their parameters can be efficiently determined by simple adjustment procedures. For linear discriminant functions, convergence proofs are given for procedures that are guaranteed to yield error-free solutions on design samples, provided only that such solutions exist. While no similar results are known for piecewise linear discriminant functions, simple procedures are given that have been effective in various experiments. The results of experiments with artificially generated multimodal data and with hand-printed alphanumeric characters are given to show that this approach compares favorably with other classification methods.

R. O. Duda

Papers

Pattern Classification by Iteratively Determined Linear and Piecewise Linear Discriminant Functions