Self bounding learning algorithms

TLDR: A self-bounding learning algorithm is an algorithm which, in addition to the hypothesis that it outputs, outputs a reliable upper bound on the generalization error of this hypothesis.

Abstract: Most of the work which attempts to give bounds on the generalization error of the hypothesis generated by a learning algorithm is based on methods from the theory of uniform convergence. These bounds are a-priori bounds that hold for any distribution of examples and are calculated before any data is observed. In this paper we propose a different approach for bounding the generalization error after the data has been observed. A self-bounding learning algorithm is an algorithm which, in addition to the hypothesis that it outputs, outputs a reliable upper bound on the generalization error of this hypothesis. We first explore the idea in the statistical query learning framework of Keams [lo]. After that we give an explicit self bounding algorithm for learning algorithms that are based on local search. Permission to make digital or h,ard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for prolit or commercial advantage and that copies bear this notice and the full citation on the first page. To COPY otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific pemlission .and/or a fee. COLT 98 Madison WI USA Copyright ACM 1998 l-581 13-057--0/9X/ 7...$5.00