Summary. Markov chain Monte Carlo and sequential Monte Carlo methods have emerged as the two main tools to sample from high dimensional probability distributions. Although asymptotic convergence of Markov chain Monte Carlo algorithms is ensured under weak assumptions, the performance of these algorithms is unreliable when the proposal distributions that are used to explore the space are poorly chosen and/or if highly correlated variables are updated independently. We show here how it is possible to build efficient high dimensional proposal distributions by using sequential Monte Carlo methods. This allows us not only to improve over standard Markov chain Monte Carlo schemes but also to make Bayesian inference feasible for a large class of statistical models where this was not previously so. We demonstrate these algorithms on a non-linear state space model and a Levy-driven stochastic volatility model.

https://rss.onlinelibrary.wiley.com/doi/pdf/10.1111/j.1467-9868.2009.00736.x

Particle Markov chain Monte Carlo methods

The random forest algorithm, proposed by L. Breiman in 2001, has been extremely successful as a general-purpose classification and regression method. The approach, which combines several randomized decision trees and aggregates their predictions by averaging, has shown excellent performance in settings where the number of variables is much larger than the number of observations. Moreover, it is versatile enough to be applied to large-scale problems, is easily adapted to various ad hoc learning tasks, and returns measures of variable importance. The present article reviews the most recent theoretical and methodological developments for random forests. Emphasis is placed on the mathematical forces driving the algorithm, with special attention given to the selection of parameters, the resampling mechanism, and variable importance measures. This review is intended to provide non-experts easy access to the main ideas.

/pdf/a-random-forest-guided-tour-3vnpr6ncpy.pdf

A random forest guided tour

Many scientific and engineering challenges—ranging from personalized medicine to customized marketing recommendations—require an understanding of treatment effect heterogeneity. In this paper, we develop a non-parametric causal forest for estimating heterogeneous treatment effects that extends Breiman's widely used random forest algorithm. In the potential outcomes framework with unconfoundedness, we show that causal forests are pointwise consistent for the true treatment effect, and have an asymptotically Gaussian and centered sampling distribution. We also discuss a practical method for constructing asymptotic confidence intervals for the true treatment effect that are centered at the causal forest estimates. Our theoretical results rely on a generic Gaussian theory for a large family of random forest algorithms. To our knowledge, this is the first set of results that allows any type of random forest, including classification and regression forests, to be used for provably valid statistical infe...

Estimation and Inference of Heterogeneous Treatment Effects using Random Forests

The random forest algorithm, proposed by L. Breiman in 2001, has been extremely successful as a general-purpose classification and regression method. The approach, which combines several randomized decision trees and aggregates their predictions by averaging, has shown excellent performance in settings where the number of variables is much larger than the number of observations. Moreover, it is versatile enough to be applied to large-scale problems, is easily adapted to various ad-hoc learning tasks, and returns measures of variable importance. The present article reviews the most recent theoretical and methodological developments for random forests. Emphasis is placed on the mathematical forces driving the algorithm, with special attention given to the selection of parameters, the resampling mechanism, and variable importance measures. This review is intended to provide non-experts easy access to the main ideas.

A Random Forest Guided Tour

Random forests are a scheme proposed by Leo Breiman in the 2000's for building a predictor ensemble with a set of decision trees that grow in randomly selected subspaces of data. Despite growing interest and practical use, there has been little exploration of the statistical properties of random forests, and little is known about the mathematical forces driving the algorithm. In this paper, we offer an in-depth analysis of a random forests model suggested by Breiman (2004), which is very close to the original algorithm. We show in particular that the procedure is consistent and adapts to sparsity, in the sense that its rate of convergence depends only on the number of strong features and not on how many noise variables are present.

/pdf/analysis-of-a-random-forests-model-3uegmd8tez.pdf

Analysis of a random forests model

This article introduces a new nonparametric method for estimating a univariate regression function of bounded variation. The method exploits the Jordan decomposition which states that a function of bounded variation can be decomposed as the sum of a non-decreasing function and a non-increasing function. This suggests combining the backfitting algorithm for estimating additive functions with isotonic regression for estimating monotone functions. The resulting iterative algorithm is called Iterative Isotonic Regression (I.I.R.). The main technical result in this paper is the consistency of the proposed estimator when the number of iterations $k_n$ grows appropriately with the sample size $n$. The proof requires two auxiliary results that are of interest in and by themselves: firstly, we generalize the well-known consistency property of isotonic regression to the framework of a non-monotone regression function, and secondly, we relate the backfitting algorithm to Von Neumann's algorithm in convex analysis.

/pdf/iterative-isotonic-regression-35zrah0x7v.pdf

Iterative isotonic regression

This article presents a variant of Fleming-Viot particle systems, which are a standard way to approximate the law of a Markov process with killing as well as related quantities. Classical Fleming-Viot particle systems proceed by simulating $N$ trajectories, or particles, according to the dynamics of the underlying process, until one of them is killed. At this killing time, the particle is instantaneously branched on one of the $(N-1)$ other ones, and so on until a fixed and finite final time $T$. In our variant, we propose to wait until $K$ particles are killed and then rebranch them independently on the $(N-K)$ alive ones. Specifically, we focus our attention on the large population limit and the regime where $K/N$ has a given limit when $N$ goes to infinity. In this context, we establish consistency and asymptotic normality results. The variant we propose is motivated by applications in rare event estimation problems.

On Synchronized Fleming-Viot Particle Systems.

Un outil de verification informatique, propre a traiter repetitivement une pluralite de jeux de donnees qui comprennent des donnees reparties selon une loi statistique L'outil comprend un estimateur (8), capable d'etablir pour un jeu de donnees une valeur caracterisant la reproduction, d'un critere relatif aux donnees qu'il contient, un pilote (6), agence pour appeler l'estimateur (8) avec une pluralite de jeux pour determiner une pluralite de valeurs, et pour etablir une nouvelle pluralite de jeux a partir de la pluralite de valeurs, et pour repeter l'appel de l'estimateur (8) avec une nouvelle pluralite de jeux etablie precedemment, jusqu'a verifier une condition qui fait intervenir un extremum de ladite pluralite de valeurs et/ou du nombre de repetitions L'outil comprend egalement un melangeur (10), capable d'etablir un nouveau jeu de donnees, sur la base d'un jeu de donnees existant, en maintenant la repartition selon la loi statistique, et le pilote (6) est agence pour appeler le melangeur (10) avec certains au moins des jeux de donnees en fonction d'une regle basee sur la pluralite de valeurs associee a ces jeux

Arnaud Guyader

Papers

Iterative isotonic regression

On Synchronized Fleming-Viot Particle Systems.

Outil de vérification informatique