The simple Bayesian classifier is known to be optimal when attributes are independent given the class, but the question of whether other sufficient conditions for its optimality exist has so far not been explored. Empirical results showing that it performs surprisingly well in many domains containing clear attribute dependences suggest that the answer to this question may be positive. This article shows that, although the Bayesian classifier‘s probability estimates are only optimal under quadratic loss if the independence assumption holds, the classifier itself can be optimal under zero-one loss (misclassification rate) even when this assumption is violated by a wide margin. The region of quadratic-loss optimality of the Bayesian classifier is in fact a second-order infinitesimal fraction of the region of zero-one optimality. This implies that the Bayesian classifier has a much greater range of applicability than previously thought. For example, in this article it is shown to be optimal for learning conjunctions and disjunctions, even though they violate the independence assumption. Further, studies in artificial domains show that it will often outperform more powerful classifiers for common training set sizes and numbers of attributes, even if its bias is a priori much less appropriate to the domain. This article‘s results also imply that detecting attribute dependence is not necessarily the best way to extend the Bayesian classifier, and this is also verified empirically.

/pdf/on-the-optimality-of-the-simple-bayesian-classifier-under-z5uc9ciyb2.pdf

On the Optimality of the Simple Bayesian Classifier under Zero-One Loss

In this survey, we review work in machine learning on methods for handling data sets containing large amounts of irrelevant information. We focus on two key issues: the problem of selecting relevant features, and the problem of selecting relevant examples. We describe the advances that have been made on these topics in both empirical and theoretical work in machine learning, and we present a general framework that we use to compare different methods. We close with some challenges for future work in this area. @ 1997 Elsevier Science B.V.

/pdf/selection-of-relevant-features-and-examples-in-machine-5c8bbbwrgd.pdf

Selection of relevant features and examples in machine

Selection of relevant features and examples in machine learning

In this paper we present an average-case analysis of the Bayesian classifier, a simple induction algorithm that fares remarkably well on many learning tasks. Our analysis assumes a monotone conjunctive target concept, and independent, noise-free Boolean attributes. We calculate the probability that the algorithm will induce an arbitrary pair of concept descriptions and then use this to compute the probability of correct classification over the instance space. The analysis takes into account the number of training instances, the number of attributes, the distribution of these attributes, and the level of class noise. We also explore the behavioral implications of the analysis by presenting predicted learning curves for artificial domains, and give experimental results on these domains as a check on our reasoning.

An analysis of Bayesian classifiers

A Study of Thinking

We present an approach to modeling the average case behavior of learning algorithms. Our motivation is to predict the expected accuracy of learning algorithms as a function of the number of training examples. We apply this framework to a purely empirical learning algorithm, (the one-sided algorithm for pure conjunctive concepts), and to an algorithm that combines empirical and explanation-based learning. The model is used to gain insight into the behavior of these algorithms on a series of problems. Finally, we evaluate how well the average case model performs when the training examples violate the assumptions of the model.

/pdf/a-framework-for-average-case-analysis-of-conjunctive-3e53olk0z3.pdf

A Framework for Average Case Analysis of Conjunctive Learning Algorithms

We present an approach to modeling the average case behavior of learning algorithms. Our motivation is to predict the expected accuracy of learning algorithms as a function of the number of training examples. We apply this framework to a purely empirical learning algorithm, (the one-sided algorithm for pure conjunctive concepts), and to an algorithm that combines empirical and explanation-based learning. We evaluate the average-case models by comparing the accuracy predicted by the models to the actual accuracy obtained by running the learning algorithms.

Average case analysis of conjunctive learning algorithms

Publisher Summary This chapter discusses a framework for integrating empirical learning with explanation-based learning and presents an algorithm that does this with both pure conjunctive concepts and k-CNF concepts. The framework involves using an empirical and an explanation-based method to form separate hypotheses, and then combining the hypotheses from the separate sources to form a composite hypothesis. An additional important complication arises because the system is required to learn the domain theory (via an empirical method) while using the domain theory to support the explanation-based method. The hypotheses produced by explanation-based learning with a domain theory acquired with such a one-sided empirical learning method will also never be more general than the correct hypothesis. As both the empirical and explanation-based hypotheses are not more general than the correct hypothesis, they can be combined by finding the least general hypothesis consistent with both hypotheses. In this manner, the integrated hypothesis would be the least general hypothesis that is consistent with both the observed data and the domain knowledge.

/pdf/one-sided-algorithms-for-integrating-empirical-and-218oy1z290.pdf

One-sided algorithms for integrating empirical and explanation-based learning

Author(s): Sarrett, Wendy E.; Pazzani, Michael J. | Abstract: We present an approach to modeling the average case behavior of learning algorithms. Our motivation is to mathematically model the performance of learning algorithms in order to better understand the nature of their empirical behavior. We are interested in how differences in learning algorithms influence the expected accuracy of the concepts learned.We present the Average Case Learning Model and apply the model to three learning algorithms: a purely empirical algorithm (Bruner's Wholist), an algorithm which prefers analytical (explanation-based) learning over empirical learning (EBL-FIRST-TM) and an algorithm integrating both analytical and empirical learning (lOSC-TM). The Average Case Learning Model is unique in that it is able to accurately predict the expected behavior of learning algorithms. We compare average case analysis to Valiant's Probably Approximately Correct (PAC) learning model.

/pdf/average-case-analysis-of-empirical-and-explanation-based-5dckdsavis.pdf

Wendy Sarrett

Papers

A Framework for Average Case Analysis of Conjunctive Learning Algorithms

Average case analysis of conjunctive learning algorithms

One-sided algorithms for integrating empirical and explanation-based learning

Average case analysis of empirical and explanation-based learning algorithms