Confidence Intervals for the Generalisation Error of Random Forests
TLDR
Out-of-bag error is commonly used as an estimate of generalisation error in ensemble-based learning models such as random forests and it is shown that these new confidence intervals have improved coverage properties over the näıve confidence interval, in real and simulated examples.Abstract:
Out-of-bag error is commonly used as an estimate of generalisation error in ensemble-based learning models such as random forests. We present confidence intervals for this quantity using the delta-method-after-bootstrap and the jackknife-after-bootstrap techniques. These methods do not require growing any additional trees. We show that these new confidence intervals have improved coverage properties over the naive confidence interval, in real and simulated examples.read more
Citations
References
More filters
Journal ArticleDOI
A new look at the statistical model identification
TL;DR: In this article, a new estimate minimum information theoretical criterion estimate (MAICE) is introduced for the purpose of statistical identification, which is free from the ambiguities inherent in the application of conventional hypothesis testing procedure.
Estimating the dimension of a model
TL;DR: In this paper, the problem of selecting one of a number of models of different dimensions is treated by finding its Bayes solution, and evaluating the leading terms of its asymptotic expansion.
Journal ArticleDOI
The Elements of Statistical Learning
TL;DR: Chapter 11 includes more case studies in other areas, ranging from manufacturing to marketing research, and a detailed comparison with other diagnostic tools, such as logistic regression and tree-based methods.
Journal ArticleDOI
Estimation of the Mean of a Multivariate Normal Distribution
TL;DR: In this article, an unbiased estimate of risk is obtained for an arbitrary estimate, and certain special classes of estimates are then discussed, such as smoothing by using moving averages and trimmed analogs of the James-Stein estimate.
Journal ArticleDOI
Estimating the Error Rate of a Prediction Rule: Improvement on Cross-Validation
TL;DR: In this paper, a prediction rule is constructed on the basis of some data, and then the error rate of this rule is estimated in classifying future observations using cross-validation.