In the typical nonparametric approach to classification in instance-based learning and data mining, random data (the training set of patterns) are collected and used to design a decision rule (classifier). One of the most well known such rules is the k-nearest-neighbor decision rule (also known as lazy learning) in which an unknown pattern is classified into the majority class among its k nearest neighbors in the training set. Several questions related to this rule have received considerable attention over the years. Such questions include the following. How can the storage of the training set be reduced without degrading the performance of the decision rule? How should the reduced training set be selected to represent the different classes? How large should k be? How should the value of k be chosen? Should all k neighbors be equally weighted when used to decide the class of an unknown pattern? If not, how should the weights be chosen? Should all the features (attributes) we weighted equally and if not how should the feature weights be chosen? What distance metric should be used? How can the rule be made robust to overlapping classes or noise present in the training data? How can the rule be made invariant to scaling of the measurements? How can the nearest neighbors of a new point be computed efficiently? What is the smallest neural network that can implement nearest neighbor decision rules? Geometric proximity graphs such as Voronoi diagrams and their many relatives provide elegant solutions to these problems, as well as other related data mining problems such as outlier detection. After a non-exhaustive review of some of the classical canonical approaches to these problems, the methods that use proximity graphs are discussed, some new observations are made, and open problems are listed.

Geometric proximity graphs for improving nearest neighbor methods in instance-based learning and data mining

This paper presents a relational framework for studying properties of labeled data points related to proximity and labeling information in order to improve the performance of the 1NN rule. Specifically, the class conditional nearest neighbor (ccnn) relation over pairs of points in a labeled training set is introduced. For a given class label c, this relation associates to each point a its nearest neighbor computed among only those points with class label c (excluded a). A characterization of ccnn in terms of two graphs is given. These graphs are used for defining a novel scoring function over instances by means of an information-theoretic divergence measure applied to the degree distributions of these graphs. The scoring function is employed to develop an effective large margin instance selection method, which is empirically demonstrated to improve storage and accuracy performance of the 1NN rule on artificial and real-life data sets.

/pdf/class-conditional-nearest-neighbor-for-large-margin-instance-3prngm40rz.pdf

Class Conditional Nearest Neighbor for Large Margin Instance Selection

This chapter discusses techniques inspired by Bayes decision theory. To the newcomer in the field of pattern recognition the chapter's algorithms and exercises are very important for developing a basic understanding and familiarity with some fundamental notions associated with classification. Most of the algorithms are simple in both structure and physical reasoning. The optimal Bayesian classifier is significantly simplified under the following assumptions: the classes are equiprobable; the data in all classes follow Gaussian distributions; the covariance matrix is the same for all classes; and the covariance matrix is diagonal and all elements across the diagonal are equal. Under these assumptions, it turns out that the optimal Bayesian classifier is equivalent to the minimum Euclidean distance classifier. The Euclidean classifier is often used, even if we know that the previously stated assumptions are not valid, because of its simplicity. It assigns a pattern to the class whose mean is closest to it with respect to the Euclidean norm. When the pdf that describes the data points in a class is not known, it has to be estimated prior to the application of the Bayesian classifier. In this section, we focus on a very popular method to model unknown probability density functions, known as mixture modeling. Nearest neighbor is one of the most popular classification rules, although it is an old technique.

Classifiers Based on Bayes Decision Theory

Publisher Summary
This chapter explores classifiers based on Bayes Decision Theory. The chapter primarily focuses on Bayesian classification and techniques for estimating unknown probability density functions based on the available experimental evidence. The chapter also deals with the design of the classifier in a pattern recognition system. While discussing the concept of minimizing the classification error probability, it is shown that the Bayesian classifier is optimal with respect to minimizing the classification error probability. Special focus is put on the Bayesian classification, the minimum distance (Euclidean and Mahalanobis), the nearest neighbor classifiers, and the naive Bayes classifier. The chapter approaches the classification problem via Bayesian probabilistic arguments with a goal to minimize the classification error probability or the risk. Examples are presented to show that not all problems are well suited to such approaches, and in those cases, it may be preferable to compute decision surfaces directly by means of alternative costs. The chapter also focuses on a particular family of decision surfaces associated with the Bayesian classification for the specific case of Gaussian density functions.

Chapter 2 – Classifiers Based on Bayes Decision Theory

Time-dependent, two-dimensional, electrothermal simulations based on random Voronoi networks have been developed to study the internal heating, current distributions and breakdown effects in ZnO varistors in response to high-voltage pulsing. The simulations allow for dynamic predictions of internal failures and to track the progression of hot-spots and thermal stresses. The focus is on internal grain-size variations and relative disorder including micropores. Our results predict that parameters such as the hold-off voltage, internal temperature, and average dissipated energy density would be higher with more uniform grains. This uniformity is also predicted to produce lower thermal stresses and to allow for the application of longer duration pulses. It is shown that the principal failure mechanism arises from internal localized melting, while thermal stresses are well below the thresholds for cracking. Finally, detrimental effects of micropores have been quantified and shown to be in agreement with experimental trends.

Electrothermal Model Evaluation of Grain Size and Disorder Effects on Pulsed Voltage Response of Microstructured ZnO Varistors

To reduce the memory requirements and the computation cost, many algorithms have been developed that perform nearest neighbor classification using only a small number of representative samples obtained from the training set. We call the classification model underlying all these algorithms as Voronoi networks (Vnets). We analyze the generalization capabilities of these networks by bounding the generalization error. The class of problems that can be solved by Vnets is characterized by the extent to which the set of points on the decision boundaries fill the feature space. We show that Vnets asymptotically converge to the Bayes classifier with arbitrarily high probability provided the number of representative samples grow slower than the square root of the number of training samples and also give the optimal growth rate of the number of representative samples. We redo the analysis for decision tree (DT) classifiers and compare them with Vnets. The bias/variance dilemma and the curse of dimensionality with respect to Vnets and DTs are also discussed.

/pdf/voronoi-networks-and-their-probability-of-misclassification-j4ov9zjwvb.pdf

M.A.L. Thathachar

Papers

Voronoi networks and their probability of misclassification