We empirically show that Bayesian inference can be inconsistent under misspecification in simple linear regression problems, both in a model averaging/selection and in a Bayesian ridge regression setting. We use the standard linear model, which assumes homoskedasticity, whereas the data are heteroskedastic (though, significantly, there are no outliers). As sample size increases, the posterior puts its mass on worse and worse models of ever higher dimension. This is caused by hypercompression, the phenomenon that the posterior puts its mass on distributions that have much larger KL divergence from the ground truth than their average, i.e. the Bayes predictive distribution. To remedy the problem, we equip the likelihood in Bayes' theorem with an exponent called the learning rate, and we propose the SafeBayesian method to learn the learning rate from the data. SafeBayes tends to select small learning rates, and regularizes more, as soon as hypercompression takes place. Its results on our data are quite encouraging.

/pdf/inconsistency-of-bayesian-inference-for-misspecified-linear-1tp3y3ooeo.pdf

Inconsistency of Bayesian inference for misspecified linear models, and a proposal for repairing it

We explore an asymptotic justification for the widely used and empirically verified approach of assuming an asymmetric Laplace distribution (ALD) for the response in Bayesian Quantile Regression. Based on empirical findings, Yu and Moyeed (2001) argued that the use of ALD is satisfactory even if it is not the true underlying distribution. We provide a justification to this claim by establishing posterior consistency and deriving the rate of convergence under the ALD misspecification. Related literature on misspecified models focuses mostly on i.i.d. models which in the regression context amounts to considering i.i.d. random covariates with i.i.d. errors. We study the behavior of the posterior for the misspecified ALD model with independent but non identically distributed response in the presence of non-random covariates. Exploiting the specific form of ALD helps us derive conditions that are more intuitive and easily seen to be satisfied by a wide range of potential true underlying probability distributions for the response. Through simulations, we demonstrate our result and also find that the robustness of the posterior that holds for ALD fails for a Gaussian formulation, thus providing further support for the use of ALD models in quantile regression.

/pdf/posterior-consistency-of-bayesian-quantile-regression-based-3bzoh6ebmf.pdf

Posterior Consistency of Bayesian Quantile Regression Based on the Misspecified Asymmetric Laplace Density

In this paper a new nonparametric estimate of conditional quantiles is proposed, that avoids the problem of crossing quantile curves [calculated for various p 2 (0;1)]: The method uses an initial estimate of the conditional distribution function in a flrst step and solves the problem of inversion and monotonization with respect to p 2 (0;1) simultaneously. It is demonstrated that the new estimates are asymptotically normal distributed and asymptotically flrst order equivalent to quantile estimates obtained by local constant or local linear smoothing of the conditional distribution function. The performance of the new procedure is illustrated by means of a simulation study and some comparisons with the currently available procedures which are similar in spirit with the proposed method are presented.

Non-crossing nonparametric estimates of quantile curves

We propose a novel framework for fitting additive quantile regression models, which provides well-calibrated inference about the conditional quantiles and fast automatic estimation of the smoothing...

Fast Calibrated Additive Quantile Regression

Summary
The paper discusses the asymptotic validity of posterior inference of pseudo-Bayesian quantile regression methods with complete or censored data when an asymmetric Laplace likelihood is used. The asymmetric Laplace likelihood has a special place in the Bayesian quantile regression framework because the usual quantile regression estimator can be derived as the maximum likelihood estimator under such a model, and this working likelihood enables highly efficient Markov chain Monte Carlo algorithms for posterior sampling. However, it seems to be under-recognised that the stationary distribution for the resulting posterior does not provide valid posterior inference directly. We demonstrate that a simple adjustment to the covariance matrix of the posterior chain leads to asymptotically valid posterior inference. Our simulation results confirm that the posterior inference, when appropriately adjusted, is an attractive alternative to other asymptotic approximations in quantile regression, especially in the presence of censored data.

Posterior Inference in Bayesian Quantile Regression with Asymmetric Laplace Likelihood

We investigate the asymptotic behavior of Bayesian posterior distributions under independent and identically distributed ($i.i.d.$) misspecified models. More specifically, we study the concentration of the posterior distribution on neighborhoods of $f^{\star}$, the density that is closest in the Kullback--Leibler sense to the true model $f_0$. We note, through examples, the need for assumptions beyond the usual Kullback--Leibler support assumption. We then investigate consistency with respect to a general metric under three assumptions, each based on a notion of divergence measure, and then apply these to a weighted $L_1$-metric in convex models and non-convex models. Although a few results on this topic are available, we believe that these are somewhat inaccessible due, in part, to the technicalities and the subtle differences compared to the more familiar well-specified model case. One of our goals is to make some of the available results, especially that of , more accessible. Unlike their paper, our approach does not require construction of test sequences. We also discuss a preliminary extension of the $i.i.d.$ results to the independent but not identically distributed ($i.n.i.d.$) case.

On Posterior Concentration in Misspecified Models

A Sandwich Likelihood Correction for Bayesian Quantile Regression based on the Misspecified Asymmetric Laplace Density

We investigate the asymptotic behavior of Bayesian posterior distributions under model misspecification in the nonparametric context. Although a few results on this topic are available, we believe that that these are somewhat inaccessible due, in part, to the technicalities and the subtle differences compared to the more familiar well-specified model case. Our goal in this paper is to make some of the available results more accessible and transparent. In particular, we give a simple proof that sets which satisfy the separation condition in Kleijn and van der Vaart (2006) will have vanishing posterior mass. Unlike their approach, we do not require construction of test sequences and we get almost sure convergence as compared to mean convergence. Further, we relate the Kleijn–van der Vaart separation condition to a roughly equivalent condition involving the more familiar and more natural Kullback–Leibler contrast. We show by example that, the standard assumption in the misspecified case that the prior puts positive mass on Kullback–Leibler neighborhoods of the true model, does not necessarily ensure that the true model is in the support of the prior with respect to natural topologies (such as L1). This observation leads to a new condition involving the Kullback–Leibler contrast and, in turn, a posterior concentration result for weak neighborhoods in general and for certain L1 neighborhoods under a compactness condition.

/pdf/on-posterior-concentration-in-misspecified-models-5507xb852d.pdf

On posterior concentration in misspecified models

We provide a theoretical justification for the widely used and yet only empirically verified approach of using Asymmetric Laplace Density(ALD) in Bayesian Quantile Regression. We derive sufficient conditions for posterior consistency of the quantile regression parameters even if the true underlying likelihood is not ALD, by considering both the case of random as well as non-random covariates. While existing literature on misspecified models address more general models, our approach of exploiting the specific form of ALD allows for a more direct derivation. We verify that the conditions so derived are satisfied by a wide range of potential true underlying probability distributions. We also show that posterior consistency holds even in the case of improper priors as long as the posterior is well defined. We demonstrate the working of the method using simulations.

/pdf/posterior-consistency-of-bayesian-quantile-regression-under-55w1amcita.pdf

Karthik Sriram

Papers

Posterior Consistency of Bayesian Quantile Regression Based on the Misspecified Asymmetric Laplace Density

On Posterior Concentration in Misspecified Models

A Sandwich Likelihood Correction for Bayesian Quantile Regression based on the Misspecified Asymmetric Laplace Density

On posterior concentration in misspecified models

Posterior consistency of Bayesian quantile regression under a mis-specified likelihood based on asymmetric laplace density