Penalized Composite Quasi-Likelihood for Ultrahigh-Dimensional Variable Selection
01 Jun 2011-Journal of The Royal Statistical Society Series B-statistical Methodology (NIH Public Access)-Vol. 73, Iss: 3, pp 325-349
TL;DR: A data‐driven weighted linear combination of convex loss functions, together with weighted L1‐penalty is proposed and established a strong oracle property of the method proposed that has both the model selection consistency and estimation efficiency for the true non‐zero coefficients.
Abstract: In high-dimensional model selection problems, penalized least-square approaches have been extensively used. This paper addresses the question of both robustness and efficiency of penalized model selection methods, and proposes a data-driven weighted linear combination of convex loss functions, together with weighted L1-penalty. It is completely data-adaptive and does not require prior knowledge of the error distribution. The weighted L1-penalty is used both to ensure the convexity of the penalty term and to ameliorate the bias caused by the L1-penalty. In the setting with dimensionality much larger than the sample size, we establish a strong oracle property of the proposed method that possesses both the model selection consistency and estimation efficiency for the true non-zero coefficients. As specific examples, we introduce a robust method of composite L1-L2, and optimal composite quantile method and evaluate their performance in both simulated and real data examples.
Citations
More filters
Journal Article•
TL;DR: In this paper, a brief account of the recent developments of theory, methods, and implementations for high-dimensional variable selection is presented, with emphasis on independence screening and two-scale methods.
Abstract: High dimensional statistical problems arise from diverse fields of scientific research and technological development. Variable selection plays a pivotal role in contemporary statistical learning and scientific discoveries. The traditional idea of best subset selection methods, which can be regarded as a specific form of penalized likelihood, is computationally too expensive for many modern statistical applications. Other forms of penalized likelihood methods have been successfully developed over the last decade to cope with high dimensionality. They have been widely applied for simultaneously selecting important variables and estimating their effects in high dimensional statistical inference. In this article, we present a brief account of the recent developments of theory, methods, and implementations for high dimensional variable selection. What limits of the dimensionality such methods can handle, what the role of penalty functions is, and what the statistical properties are rapidly drive the advances of the field. The properties of non-concave penalized likelihood and its roles in high dimensional statistical modeling are emphasized. We also review some recent advances in ultra-high dimensional variable selection, with emphasis on independence screening and two-scale methods.
892 citations
TL;DR: This work proposes adaptive penalization methods for variable selection in the semiparametric varying-coefficient partially linear model and proves that the methods possess the oracle property.
Abstract: The complexity of semiparametric models poses new challenges to statistical inference and model selection that frequently arise from real applications In this work, we propose new estimation and variable selection procedures for the semiparametric varying-coefficient partially linear model We first study quantile regression estimates for the nonparametric varying-coefficient functions and the parametric regression coefficients To achieve nice efficiency properties, we further develop a semiparametric composite quantile regression procedure We establish the asymptotic normality of proposed estimators for both the parametric and nonparametric parts and show that the estimators achieve the best convergence rate Moreover, we show that the proposed method is much more efficient than the least-squares-based method for many non-normal errors and that it only loses a small amount of efficiency for normal errors In addition, it is shown that the loss in efficiency is at most 111% for estimating varying coefficient functions and is no greater than 136% for estimating parametric components To achieve sparsity with high-dimensional covariates, we propose adaptive penalization methods for variable selection in the semiparametric varying-coefficient partially linear model and prove that the methods possess the oracle property Extensive Monte Carlo simulation studies are conducted to examine the finite-sample performance of the proposed procedures Finally, we apply the new methods to analyze the plasma beta-carotene level data
265 citations
Cites methods from "Penalized Composite Quasi-Likelihoo..."
...Moreover, we show that the proposed method is much more efficient than the least-squares-based method for many non-normal errors and that it only loses a small amount of efficiency for normal errors....
[...]
TL;DR: This paper reviews the literature on sparse high dimensional models and discusses some applications in economics and finance, including variable selection methods that are proved to be effective in high dimensional sparse modeling.
Abstract: This article reviews the literature on sparse high-dimensional models and discusses some applications in economics and finance. Recent developments in theory, methods, and implementations in penalized least-squares and penalized likelihood methods are highlighted. These variable selection methods are effective in sparse high-dimensional modeling. The limits of dimensionality that regularization methods can handle, the role of penalty functions, and their statistical properties are detailed. Some recent advances in sparse ultra-high-dimensional modeling are also briefly discussed.
228 citations
TL;DR: In this article, a principal factor approximation (PFA) based method was proposed to solve the problem of false discovery control in large-scale multiple hypothesis testing, where a common threshold is used and a consistent estimate of realized FDP is provided.
Abstract: Multiple hypothesis testing is a fundamental problem in high-dimensional inference, with wide applications in many scientific fields. In genome-wide association studies, tens of thousands of tests are performed simultaneously to find if any single-nucleotide polymorphisms (SNPs) are associated with some traits and those tests are correlated. When test statistics are correlated, false discovery control becomes very challenging under arbitrary dependence. In this article, we propose a novel method—based on principal factor approximation—that successfully subtracts the common dependence and weakens significantly the correlation structure, to deal with an arbitrary dependence structure. We derive an approximate expression for false discovery proportion (FDP) in large-scale multiple testing when a common threshold is used and provide a consistent estimate of realized FDP. This result has important applications in controlling false discovery rate and FDP. Our estimate of realized FDP compares favorably with Efr...
199 citations
Posted Content•
TL;DR: An approximate expression for false discovery proportion (FDP) in large-scale multiple testing when a common threshold is used and a consistent estimate of realized FDP is provided, which has important applications in controlling false discovery rate and FDP.
Abstract: Multiple hypothesis testing is a fundamental problem in high dimensional inference, with wide applications in many scientific fields. In genome-wide association studies, tens of thousands of tests are performed simultaneously to find if any SNPs are associated with some traits and those tests are correlated. When test statistics are correlated, false discovery control becomes very challenging under arbitrary dependence. In the current paper, we propose a novel method based on principal factor approximation, which successfully subtracts the common dependence and weakens significantly the correlation structure, to deal with an arbitrary dependence structure. We derive an approximate expression for false discovery proportion (FDP) in large scale multiple testing when a common threshold is used and provide a consistent estimate of realized FDP. This result has important applications in controlling FDR and FDP. Our estimate of realized FDP compares favorably with Efron (2007)'s approach, as demonstrated in the simulated examples. Our approach is further illustrated by some real data applications. We also propose a dependence-adjusted procedure, which is more powerful than the fixed threshold procedure.
152 citations
References
More filters
102 citations
"Penalized Composite Quasi-Likelihoo..." refers background or methods in this paper
...(24) As shown in Koenker (1984) and Bickel (1973), when K → ∞, the optimally weighted CQR (WCQR) is as efficient as the maximum likelihood estimator, always more efficient than ECQR....
[...]
...The weighted composite quantile regression (CQR) was first studied by Koenker (1984) in classical statistical inference setting....
[...]
...2 Penalized Composite Quantile Regression The weighted composite quantile regression (CQR) was first studied by Koenker (1984) in classical statistical inference setting....
[...]
...This kind of ideas appeared already in traditional statistical inference with finite dimensionality (Koenker, 1984; Bai et al., 1992)....
[...]
...2 Penalized Composite Quantile Regression The weighted composite quantile regression (CQR) was first studied by Koenker (1984) in classical statistical inference setting. Zou and Yuan (2008) used equally weighted CQR...
[...]
TL;DR: In this article, the authors examined the large sample behavior of some estimates of the regression parameters and showed that the asymptotic efficiency of these procedures is independent of the design matrix.
Abstract: We consider the general linear model with independent symmetric errors. In this context we propose and examine the large sample behavior of some estimates of the regression parameters. For the location model these statistics are linear combinations of order statistics. In general they depend on a preliminary estimate and the ordered residuals based on it. The asymptotic efficiency of these procedures is independent of the design matrix. Specifically analogues of the median and trimmed and Winsorized means are proposed.
100 citations
TL;DR: It is shown that given an initial estimator, an estimator with a similar prediction loss but with a smaller number of non-zero coordinates can be found.
Abstract: We study a regression model with a huge number of interacting variables. We consider a specific approximation of the regression function under two assumptions: (i) there exists a sparse representation of the regression function in a suggested basis, (ii) there are no interactions outside of the set of the corresponding main effects. We suggest an hierarchical randomized search procedure for selection of variables and of their interactions. We show that given an initial estimator, an estimator with a similar prediction loss but with a smaller number of non-zero coordinates can be found.
48 citations
"Penalized Composite Quasi-Likelihoo..." refers background in this paper
...the L1-penalty (Tibshirani, 1996), SCAD (Fan and Li, 2001) or the hierarchical penalty (Bickel et al., 2008), resulting in the penalized composite quasi-likelihood problem:...
[...]
...…function such as Lp-penalty with 0 p 1 (Frank and Friedman, 1993), LASSO i.e. L1penalty (Tibshirani, 1996), SCAD (Fan and Li, 2001), hierarchical penalty (Bickel et al., 2008), resulting in the penalized composite quasi-likelihood problem: min β n∑ i=1 ρw(Yi −XTi β) + n p∑ j=1 pλ(|βj |)....
[...]
...It implies not only the model selection consistency and but also sign consistency (Zhao and Yu, 2006; Bickel et al., 2008, 2009): P (sgn(β̂ w ) = sgn(β∗)) = P (sgn(β̂ o ) = sgn(β∗)) → 1....
[...]
Posted Content•
TL;DR: This paper extends ISIS, without explicit definition of residuals, to a general pseudo-likelihood framework, which includes generalized linear models as a special case, and introduces a new technique to reduce the false discovery rate in the feature screening stage.
Abstract: Variable selection in high-dimensional space characterizes many contemporary prob- lems in scientific discovery and decision making. Many frequently-used techniques are based on independence screening; examples include correlation ranking or feature selection using a two- sample t-test in high-dimensional classification. Within the context of the linear model, Fan and Lv (2008) showed that this simple correlation ranking possesses a sure independence screen- ing property under certain conditions and that its revision, called iteratively sure independent screening (ISIS), is needed when the features are marginally unrelated but jointly related to the response variable. In this paper, we extend ISIS, without explicit definition of residuals, to a general pseudo-likelihood framework, which includes generalized linear models as a special case. Even in the least-squares setting, the new method improves ISIS by allowing variable deletion in the iterative process. Our technique allows us to select important features in high-dimensional classification where the popularly used two-sample t-method fails. A new technique is introduced to reduce the false discovery rate in the feature screening stage. Several simulated and two real data examples are presented to illustrate the methodology. Refreshments will be served at 3:30 PM in 0-112 Martin Hall.
43 citations
"Penalized Composite Quasi-Likelihoo..." refers methods in this paper
...The penalized composite quasi-likelihood method can also be used in sure independence screening (Fan and Lv, 2008; Fan and Song, 2010) or the iterated version (Fan et al., 2009), resulting in robust variable screening and selection....
[...]