Empirical Bayes vs. Fully Bayes Variable Selection

TLDR: In this article, the potential of alternative fully-Bayes methods, which instead margin out the hyperparameters with respect to prior distributions, is studied, and several structured prior formulations are considered for which fully Bayes selection and estimation methods are obtained.

About: This article is published in Journal of Statistical Planning and Inference.The article was published on 2008-04-01 and is currently open access. It has received 146 citations till now. The article focuses on the topics: g-prior & Bayes factor.

Figures

Fig. 2. FBU maximum log posteriors.

Table 1 Average losses over 1000 replications of the selection rules MML = CMML, CML = CCML, FB = CFB, FBU = CFBU, AIC and BIC, and of the posterior mean rules ̂ MML , ̂ CML ̂ FB and ̂ FBU

Fig. 1. FB maximum log posteriors.

Fig. 4. Average losses over 1000 replications of the posterior mean rules ̂ MML , ̂ CML , ̂ FB and ̂ FBU .

Fig. 3. Average losses over 1000 replications of the selection rules MML = CMML, CML = CCML, FB= CFB, FBU = CFBU, AIC and BIC.

Frequently asked questions

Q1. What are the contributions mentioned in the paper "Empirical bayes vs. fully bayes variable selection" ?

In this paper, the authors study the potential of alternative fully Bayes methods, which instead margin out the hyperparameters with respect to prior distributions. Analytical and simulation comparisons with empirical Bayes counterparts are studied. 

Q2. What is the common strategy under the variable selection problem?

Letting = 1, . . . , 2p index the subsets of X1, . . . , Xp, and letting X be the n × q design matrix corresponding to the th subset, this corresponds to uncertainty about which is the appropriate subset modelY = X + . (2) A common strategy under such variable selection uncertainty has been to select the th model which maximizes apenalized regression sum of squares criterion of the formSS /̂ 2 − F − q . 

Q3. What is the appropriate loss function for a selection rule?

From a decision theoretic point of view, the appropriate loss for such selection rules is the 0–1 loss function which is 0 if and only if the correct model is selected. 

Q4. What is the definition of the variable selection problem?

The variable selection problem arises when there is uncertainty about which, if any, of the explanatory variables should be dropped from the model. 

Q5. What is the posterior mode for c?

By suitable choices of c and , the posterior mode can be calibrated to correspond to traditional fixed penalty criteria such as AIC/Cp, BIC or RIC, respectively. 

Q6. What is the limiting distribution of wa,wb?

Note that U is the limiting distribution of wa,wb as wa = wb → ∞.Turning to c, the authors note that (1+c) serves as a scale parameter in the marginal distribution of f (y | c, ). 

Q7. What is the way to get the model right?

using such a loss function for simulation is problematic because getting the model exactly right is a very small probability event when so many models are being compared. 

Q8. What is the default prior for c?

For this purpose, the authors recommend setting b = 0 and using(c) = (1 + c)−(1+ ) for c ∈ (0, ∞) (19) with = 1 as the default prior on c. 

Q9. What is the way to limit attention to a manageable set of models?

In such cases, heuristics such as stepwise methods or stochastic search might be used to restrict attention to a manageable set of models.