Learnability and the Vapnik-Chervonenkis dimension
TL;DR: This paper shows that the essential condition for distribution-free learnability is finiteness of the Vapnik-Chervonenkis dimension, a simple combinatorial parameter of the class of concepts to be learned.
Abstract: Valiant's learnability model is extended to learning classes of concepts defined by regions in Euclidean space En. The methods in this paper lead to a unified treatment of some of Valiant's results, along with previous results on distribution-free convergence of certain pattern recognition algorithms. It is shown that the essential condition for distribution-free learnability is finiteness of the Vapnik-Chervonenkis dimension, a simple combinatorial parameter of the class of concepts to be learned. Using this parameter, the complexity and closure properties of learnable classes are analyzed, and the necessary and sufficient conditions are provided for feasible learnability.
Citations
More filters
18 May 1993
TL;DR: Two new complexity classes between P and NP, very much in the spirit of MAXNP and MAXSNP are defined, and it is shown that computing the V-C dimension is complete for the more general class, whereas the other two problems are complete for a weaker class.
Abstract: The complexity of several natural computational problems in NP, which have been proposed but not categorized satisfactorily in the literature is characterized precisely. These problems can be solved in n/sup O(logn)/ time, and thus they are probably not NP-complete. Two new complexity classes between P and NP, very much in the spirit of MAXNP and MAXSNP, are defined. It is shown that computing the V-C dimension is complete for the more general class, whereas the other two problems are complete for the weaker class. >
156 citations
TL;DR: This paper applies a form of the competitive philosophy for the first time to the problem of prefetching to develop an optimal universal prefetcher in terms of fault rate, with particular applications to large-scale databases and hypertext systems.
Abstract: Caching and prefetching are important mechanisms for speeding up access time to data on secondary storage. Recent work in competitive online algorithms has uncovered several promising new algorithms for caching. In this paper, we apply a form of the competitive philosophy for the first time to the problem of prefetching to develop an optimal universal prefetcher in terms of fault rate, with particular applications to large-scale databases and hypertext systems. Our prediction algorithms with particular applications to large-scale databases and hypertext systems. Our prediction algorithms for prefetching are novel in that they are based on data compression techniques that are both theoretically optimal and good in practice. Intuitively, in order to compress data effectively, you have to be able to predict future data well, and thus good data compressors should be able to predict well for purposes of prefetching. We show for powerful models such as Markov sources and mthe order Markov sources that the page fault rate incurred by our prefetching algorithms are optimal in the limit for almost all sequences of page requests.
156 citations
TL;DR: A model of polynomial-time concept prediction is investigated which is a relaxation of the distribution-independent model of concept learning due to Valiant and prediction-preserving reductions are defined and are shown to be effective tools for comparing the relative difficulty of solving various prediction problems.
155 citations
TL;DR: It is shown that the problem of learning a probably almost optimal weight vector for a neuron is so difficult that the minimum error cannot even be approximated to within a constant factor in polynomial time (unless RP = NP); the same hardness result is obtained for several variants of this problem.
154 citations
Posted Content•
08 Mar 2017TL;DR: This work proves new upper and lower bounds on the VC-dimension of deep neural networks with the ReLU activation function, and proves a tight bound $\Theta(W U)$ on theVC-dimension.
Abstract: We prove new upper and lower bounds on the VC-dimension of deep neural
networks with the ReLU activation function. These bounds are tight for almost
the entire range of parameters. Letting $W$ be the number of weights and $L$ be
the number of layers, we prove that the VC-dimension is $O(W L \log(W))$ and
$\Omega( W L \log(W/L) )$. This improves both the previously known upper bounds
and lower bounds. In terms of the number $U$ of non-linear units, we prove a
tight bound $\Theta(W U)$ on the VC-dimension. All of these results generalize
to arbitrary piecewise linear activation functions.
154 citations
Cites background from "Learnability and the Vapnik-Chervon..."
...An established way to address this question is by considering VC-dimension, as it is well known that this asymptotically determines the sample complexity of PAC learning with such classifiers (Vapnik and Chervonenkis, 1971; Blumer et al., 1989)....
[...]
References
More filters
01 Jan 1979
42,654 citations
Book•
01 Jan 1979
TL;DR: The second edition of a quarterly column as discussed by the authors provides a continuing update to the list of problems (NP-complete and harder) presented by M. R. Garey and myself in our book "Computers and Intractability: A Guide to the Theory of NP-Completeness,” W. H. Freeman & Co., San Francisco, 1979.
Abstract: This is the second edition of a quarterly column the purpose of which is to provide a continuing update to the list of problems (NP-complete and harder) presented by M. R. Garey and myself in our book ‘‘Computers and Intractability: A Guide to the Theory of NP-Completeness,’’ W. H. Freeman & Co., San Francisco, 1979 (hereinafter referred to as ‘‘[G&J]’’; previous columns will be referred to by their dates). A background equivalent to that provided by [G&J] is assumed. Readers having results they would like mentioned (NP-hardness, PSPACE-hardness, polynomial-time-solvability, etc.), or open problems they would like publicized, should send them to David S. Johnson, Room 2C355, Bell Laboratories, Murray Hill, NJ 07974, including details, or at least sketches, of any new proofs (full papers are preferred). In the case of unpublished results, please state explicitly that you would like the results mentioned in the column. Comments and corrections are also welcome. For more details on the nature of the column and the form of desired submissions, see the December 1981 issue of this journal.
40,020 citations
Book•
01 Jan 1968
TL;DR: The arrangement of this invention provides a strong vibration free hold-down mechanism while avoiding a large pressure drop to the flow of coolant fluid.
Abstract: A fuel pin hold-down and spacing apparatus for use in nuclear reactors is disclosed. Fuel pins forming a hexagonal array are spaced apart from each other and held-down at their lower end, securely attached at two places along their length to one of a plurality of vertically disposed parallel plates arranged in horizontally spaced rows. These plates are in turn spaced apart from each other and held together by a combination of spacing and fastening means. The arrangement of this invention provides a strong vibration free hold-down mechanism while avoiding a large pressure drop to the flow of coolant fluid. This apparatus is particularly useful in connection with liquid cooled reactors such as liquid metal cooled fast breeder reactors.
17,939 citations
14,948 citations
Book•
01 Jan 1973
TL;DR: In this article, a unified, comprehensive and up-to-date treatment of both statistical and descriptive methods for pattern recognition is provided, including Bayesian decision theory, supervised and unsupervised learning, nonparametric techniques, discriminant analysis, clustering, preprosessing of pictorial data, spatial filtering, shape description techniques, perspective transformations, projective invariants, linguistic procedures, and artificial intelligence techniques for scene analysis.
Abstract: Provides a unified, comprehensive and up-to-date treatment of both statistical and descriptive methods for pattern recognition. The topics treated include Bayesian decision theory, supervised and unsupervised learning, nonparametric techniques, discriminant analysis, clustering, preprosessing of pictorial data, spatial filtering, shape description techniques, perspective transformations, projective invariants, linguistic procedures, and artificial intelligence techniques for scene analysis.
13,647 citations