We develop a Bayesian “sum-of-trees” model where each tree is constrained by a regularization prior to be a weak learner, and fitting and inference are accomplished via an iterative Bayesian backfitting MCMC algorithm that generates samples from a posterior. Effectively, BART is a nonparametric Bayesian regression approach which uses dimensionally adaptive random basis elements. Motivated by ensemble methods in general, and boosting algorithms in particular, BART is defined by a statistical model: a prior and a likelihood. This approach enables full posterior inference including point and interval estimates of the unknown regression function as well as the marginal effects of potential predictors. By keeping track of predictor inclusion frequencies, BART can also be used for model-free variable selection. BART’s many features are illustrated with a bake-off against competing methods on 42 different data sets, with a simulation experiment and on a drug discovery classification problem.

/pdf/bart-bayesian-additive-regression-trees-4a8lt041z2.pdf

BART: Bayesian additive regression trees

The Origins of Local Regression.- Local Regression Methods.- Fitting with LOCFIT.- Local Likelihood Estimation.- Density Estimation.- Flexible Local Regression.- Survival and Failure Time Analysis.- Discrimination and Classification.- Variance Estimation and Goodness of Fit.- Bandwidth Selection.- Adaptive Parameter Choice.- Computational Methods.- Optimizing Local Regression.

http://cyber.sibsutis.ru:82/Monarev/docs/nauka/PROBABILITY/MVsa_Statistics%20and%20applications/Loader%20C.%20Local%20Regression%20and%20Likelihood%20(Springer,%201999)(305s).pdf

Local Regression and Likelihood

Performance bounds for criteria for model selection are devel- oped using recent theory for sieves. The model selection criteria are based on an empirical loss or contrast function with an added penalty term moti- vated by empirical process theory and roughly proportional to the number of parameters needed to describe the model divided by the number of ob- servations. Most of our examples involve density or regression estimation settings and we focus on the problem of estimating the unknown density or regression function. We show that the quadratic risk of the minimum penal- ized empirical contrast estimatoris bounded by an index of the accuracy of the sieve. This accuracy index quanties the trade-off among the candidate models between the approximation error and parameter dimension relative to sample size. If we choose a list of models which exhibit good approximation prop- erties with respect to different classes of smoothness, the estimator can be simultaneously minimax rate optimal in each of those classes. This is what is usually called adaptation. The type of classes of smoothness in which one gets adaptation depends heavily on the list of models. If too many models are involved in order to get accurate approximation of many wide classes of functions simultaneously, it may happen that the estimator is only approx-

Risk bounds for model selection via penalization

Likelihood ratio theory has had tremendous success in parametric inference, due to the fundamental theory of Wilks. Yet, there is no general applicable approach for nonparametric inferences based on function estimation. Maximum likelihood ratio test statistics in general may not exist in nonparametric function estimation setting. Even if they exist, they are hard to find and can not; be optimal as shown in this paper. We introduce the generalized likelihood statistics to overcome the drawbacks of nonparametric maximum likelihood ratio statistics. A new S Wilks phenomenon is unveiled. We demonstrate that a class of the generalized likelihood statistics based on some appropriate nonparametric estimators are asymptotically distribution free and follow chi (2)-distributions under null hypotheses for a number of useful hypotheses and a variety of useful models including Gaussian white noise models, nonparametric regression models, varying coefficient models and generalized varying coefficient models. We further demonstrate that generalized likelihood ratio statistics are asymptotically optimal in the sense that they achieve optimal rates of convergence given by Ingster. They can even be adaptively optimal in the sense of Spokoiny by using a simple choice of adaptive smoothing parameter. Our work indicates that the generalized likelihood ratio statistics are indeed general and powerful for nonparametric testing problems based on function estimation.

/pdf/generalized-likelihood-ratio-statistics-and-wilks-phenomenon-4sltze6xgl.pdf

Generalized likelihood ratio statistics and Wilks phenomenon

1 Wavelets.- 1.1 What can wavelets offer?.- 1.2 General remarks.- 1.3 Data compression.- 1.4 Local adaptivity.- 1.5 Nonlinear smoothing properties.- 1.6 Synopsis.- 2 The Haar basis wavelet system.- 3 The idea of multiresolution analysis.- 3.1 Multiresolution analysis.- 3.2 Wavelet system construction.- 3.3 An example.- 4 Some facts from Fourier analysis.- 5 Basic relations of wavelet theory.- 5.1 When do we have a wavelet expansion?.- 5.2 How to construct mothers from a father.- 5.3 Additional remarks.- 6 Construction of wavelet bases.- 6.1 Construction starting from Riesz bases.- 6.2 Construction starting from m0.- 7 Compactly supported wavelets.- 7.1 Daubechies' construction.- 7.2 Coiflets.- 7.3 Symmlets.- 8 Wavelets and Approximation.- 8.1 Introduction.- 8.2 Sobolev Spaces.- 8.3 Approximation kernels.- 8.4 Approximation theorem in Sobolev spaces.- 8.5 Periodic kernels and projection operators.- 8.6 Moment condition for projection kernels.- 8.7 Moment condition in the wavelet case.- 9 Wavelets and Besov Spaces.- 9.1 Introduction.- 9.2 Besov spaces.- 9.3 Littlewood-Paley decomposition.- 9.4 Approximation theorem in Besov spaces.- 9.5 Wavelets and approximation in Besov spaces.- 10 Statistical estimation using wavelets.- 10.1 Introduction.- 10.2 Linear wavelet density estimation.- 10.3 Soft and hard thresholding.- 10.4 Linear versus nonlinear wavelet density estimation.- 10.5 Asymptotic properties of wavelet thresholding estimates.- 10.6 Some real data examples.- 10.7 Comparison with kernel estimates.- 10.8 Regression estimation.- 10.9 Other statistical models.- 11 Wavelet thresholding and adaptation.- 11.1 Introduction.- 11.2 Different forms of wavelet thresholding.- 11.3 Adaptivity properties of wavelet estimates.- 11.4 Thresholding in sequence space.- 11.5 Adaptive thresholding and Stein's principle.- 11.6 Oracle inequalities.- 11.7 Bibliographic remarks.- 12 Computational aspects and software.- 12.1 Introduction.- 12.2 The cascade algorithm.- 12.3 Discrete wavelet transform.- 12.4 Statistical implementation of the DWT.- 12.5 Translation invariant wavelet estimation.- 12.6 Main wavelet commands in XploRe.- A Tables.- A.1 Wavelet Coefficients.- A.2.- B Software Availability.- C Bernstein and Rosenthal inequalities.- D A Lemma on the Riesz basis.- Author Index.

Wavelets, Approximation, and Statistical Applications

A new variable bandwidth selector for kernel estimation is proposed The application of this bandwidth selector leads to kernel estimates that achieve optimal rates of convergence over Besov classes This implies that the procedure adapts to spatially inhomogeneous smoothness In particular, the estimates share optimality properties with wavelet estimates based on thresholding of empirical wavelet coefficients

/pdf/optimal-spatial-adaptation-to-inhomogeneous-smoothness-an-3demfhfmbe.pdf

Optimal spatial adaptation to inhomogeneous smoothness: an approach based on kernel estimates with variable bandwidth selectors

The problem of optimal adaptive estimation of a function at a given point from noisy data is considered. Two procedures are proved to be asymptotically optimal for different settings. First we study the problem of bandwidth selection for nonparametric pointwise kernel estimation with a given kernel. We propose a bandwidth selection procedure and prove its optimality in the asymptotic sense. Moreover, this optimality is stated not only among kernel estimators with a variable bandwidth. The resulting estimator is asymptotically optimal among all feasible estimators. The important feature of this procedure is that it is fully adaptive and it "works" for a very wide class of functions obeying a mild regularity restriction. With it the attainable accuracy of estimation depends on the function itself and is expressed in terms of the "ideal adaptive bandwidth" corresponding to this function and a given kernel. The second procedure can be considered as a specialization of the first one under the qualitative assumption that the function to be estimated belongs to some Holder class $\Sigma (\beta, L)$ with unknown parameters $\beta, L$. This assumption allows us to choose a family of kernels in an optimal way and the resulting procedure appears to be asymptotically optimal in the adaptive sense in any range of adaptation with $\beta \leq 2$.

/pdf/optimal-pointwise-adaptive-methods-in-nonparametric-1qb8976jsg.pdf

Optimal pointwise adaptive methods in nonparametric estimation

We address the problem of density estimation with $\mathbb{L}_s$-loss by selection of kernel estimators. We develop a selection procedure and derive corresponding $\mathbb{L}_s$-risk oracle inequalities. It is shown that the proposed selection rule leads to the estimator being minimax adaptive over a scale of the anisotropic Nikol'skii classes. The main technical tools used in our derivations are uniform bounds on the $\mathbb{L}_s$-norms of empirical processes developed recently by Goldenshluger and Lepski [Ann. Probab. (2011), to appear].

Bandwidth selection in kernel density estimation: oracle inequalities and adaptive minimax optimality

We address the problem of density estimation with "$\mathbb{L}_{s}$-loss by selection of kernel estimators. We develop a selection procedure and derive corresponding $\mathbb{L}_{s}$-risk oracle inequalities. It is shown that the proposed selection rule leads to the estimator being minimax adaptive over a scale of the anisotropic Nikol’skii classes. The main technical tools used in our derivations are uniform bounds on the $\mathbb{L}_{s}$-norms of empirical processes developed recently by Goldenshluger and Lepski [Ann. Probab. (2011), to appear].

/pdf/bandwidth-selection-in-kernel-density-estimation-oracle-ha25a7y2it.pdf

Bandwidth selection in kernel density estimation: Oracle inequalities and adaptive minimax optimality

The aim of this paper is to synthetically analyse the performances of thresholding and wavelet estimation methods. In this connection, it is useful to describe the maximal sets where these methods attain a special rate of convergence. We relate these “maxisets” to other problems naturally arising in the context of non parametric estimation, as approximation theory or information reduction. A second part of the paper is devoted to isolate two very special properties especially shared by wavelet bases, which allow them to behave almost as in an Hilbertian context even for Lp risks.

Oleg Lepski

Papers

Optimal spatial adaptation to inhomogeneous smoothness: an approach based on kernel estimates with variable bandwidth selectors

Optimal pointwise adaptive methods in nonparametric estimation

Bandwidth selection in kernel density estimation: oracle inequalities and adaptive minimax optimality

Bandwidth selection in kernel density estimation: Oracle inequalities and adaptive minimax optimality

Thresholding algorithms, maxisets and well-concentrated bases