Penalized Composite Quasi-Likelihood for Ultrahigh-Dimensional Variable Selection
01 Jun 2011-Journal of The Royal Statistical Society Series B-statistical Methodology (NIH Public Access)-Vol. 73, Iss: 3, pp 325-349
TL;DR: A data‐driven weighted linear combination of convex loss functions, together with weighted L1‐penalty is proposed and established a strong oracle property of the method proposed that has both the model selection consistency and estimation efficiency for the true non‐zero coefficients.
Abstract: In high-dimensional model selection problems, penalized least-square approaches have been extensively used. This paper addresses the question of both robustness and efficiency of penalized model selection methods, and proposes a data-driven weighted linear combination of convex loss functions, together with weighted L1-penalty. It is completely data-adaptive and does not require prior knowledge of the error distribution. The weighted L1-penalty is used both to ensure the convexity of the penalty term and to ameliorate the bias caused by the L1-penalty. In the setting with dimensionality much larger than the sample size, we establish a strong oracle property of the proposed method that possesses both the model selection consistency and estimation efficiency for the true non-zero coefficients. As specific examples, we introduce a robust method of composite L1-L2, and optimal composite quantile method and evaluate their performance in both simulated and real data examples.
Citations
More filters
TL;DR: In this paper, a kernel-based local polynomial methodology for nonparametric regression based on optimising a linear combination of several loss functions was developed, where the optimal weights for least squares and quantile loss functions can be chosen to provide maximum efficiency.
Abstract: We develop a new kernel-based local polynomial methodology for nonparametric regression based on optimising a linear combination of several loss functions. Optimal weights for least squares and quantile loss functions can be chosen to provide maximum efficiency and these optimal weights can be estimated from data. The resulting estimators are at least as efficient as those provided by existing procedures, but can be much more efficient for many distributions. The data-based weights adapt to the tails of the error distribution resulting in a procedure which is both robust and resistant. Furthermore, the assumption of homogeneous error variance is not required. To illustrate its practical use, we apply the proposed method to model the motorcycle data.
7 citations
Cites background or methods from "Penalized Composite Quasi-Likelihoo..."
...This idea is motivated by Bradic, Fan, and Wang (2011), which attempted to produce a robust and efficient estimator for high-dimensional linear regression by minimising composite loss functions simultaneously....
[...]
...However, different from a direct extension of Bradic et al. (2011) to nonparametric regression, we drop the finite second-moment assumption and combine the squared loss with multiple quantile loss functions for symmetric errors and pick weights to optimise the asymptotic efficiency....
[...]
TL;DR: It is shown that any accumulation point of the sequence generated by the iterative reweighted methods based on the new Lipschitz continuous ℓ1-ℓp minimization is a generalized first-order stationary point of this minimization.
Abstract: In this paper, we focus on the $$\ell _1-\ell _p$$
minimization problem with $$0
6 citations
TL;DR: The 2012 Frontiers of Quantile Regression Workshop as mentioned in this paper was the first workshop dedicated to quantile regression analysis, which was attended by 16 participants with broad geographic representation from all continents.
Abstract: Quantiles play an essential role in modern statistics, as emphasized by the fundamental work of Parzen (1978) and Tukey (1977). Quantile regression was introduced by Koenker and Bassett (1978) as a complement to least squares estimation (LSE) or maximum likelihood estimation (MLE) and leads to far-reaching extensions of ”classical” regression analysis by estimating families of conditional quantile surfaces, which describe the relation between a one-dimensional response y and a high dimensional predictor x. Since its introduction quantile regression has found great attraction in mathematical and applied statistics because of its natural interpretability and robustness, which yields attractive applications in such important areas as medicine, economics, engineering and environmental modeling. Although classical quantile regression theory is very well developed, the implicit definition of quantile regression still yields many new mathematical challenges such as multivariate, censored and longitudinal data, which were discussed during the workshop. Mathematics Subject Classification (2000): 62G10, 62G08, 62G30. Introduction by the Organisers The workshopFrontiers of quantile regression, organised by Victor Chernozhukov (Boston), Holger Dette (Bochum), Xuming He (Ann Arbor) and Roger Koenker (Champaign) was held 25 November – 1 December 2012. This meeting was well attended by 16 participants with broad geographic representation from all continents. During the workshop all mathematical aspects of the recent development in quantile regression analysis were discussed. A particular focus was on Multivariate quantile regression where several new concepts were presented by the 3340 Oberwolfach Report 56/2012 participants, including Bahadur representations, asymptotic normality and uniform convergence of the corresponding estimates. Other talks discussed quantile regression for longitudinal data and random effect models with applications in functional data analysis and biostatistics and the definition of new spectra of stationary time series via quantile regression methods. Several speakers presented their results on variable selection in high-dimensional quantile regression models, especially under the framework of “large p small n paradigm (here p refers to the dimension of the parameter to be estimated and n denotes the sample size). It was shown that useful model identification is possible when sparsity of the model is expected to hold. Two other speakers discussed quantile regression methods for censored data. Specifically, the following research fields in quantile regression were discussed during the workshop. (1) Multivariate quantile regression (2) Quantile regression for longitudinal data and random effect models (3) Bayesian analysis in quantile regression (4) Variable selection in high-dimensional quantile regression models (5) Quantile regression for censored data (6) Quantile regression in time series The workshop stimulated intensive discussions between all participants and new developments in various subfields of quantile regression analysis. For example, the problem of quantile regression for multivariate was discussed in three talks from different perspectives. Similarly, in the context of stationary time series a spectral theory will be developed, which avoids the existence of any moments.
6 citations
Cites background from "Penalized Composite Quasi-Likelihoo..."
...An intermediary step of the method required the estimation of a weighted least squares version of Lasso in which weights are estimated....
[...]
...Other very important work includes Chaudhuri [7], Chaudhuri, Doksum and Samarov [8], Härdle, Ritov, and Song [12], Cattaneo, Crump, and Jansson [6], and Kong, Linton, and Xia [17], among others, but this work focused on local, non-series, methods....
[...]
...The second step attempts to properly partial out the confounding factors z from the treatment estimating a suitable residual via heteroskedastic Lasso [8, 1]....
[...]
...For instance, Lasso can be substituted by Dantzig selector, SCAD, square-root Lasso, the associated post-model selection estimators or others....
[...]
...[8] Probal Chaudhuri, Kjell Doksum, and Alexander Samarov....
[...]
TL;DR: Composite quantile regression models have been shown to be effective techniques in improving the prediction accuracy [H. Zou and M. Yuan, Composite quantile regressions and the oracle model selectio... as discussed by the authors.
Abstract: Composite quantile regression models have been shown to be effective techniques in improving the prediction accuracy [H. Zou and M. Yuan, Composite quantile regression and the oracle model selectio...
6 citations
Cites methods from "Penalized Composite Quasi-Likelihoo..."
...[7] proposed a robust and efficient penalized composite quasi-likelihood method for ultrahigh dimensional variable selection....
[...]
TL;DR: In this paper, a penalized exponentially tilted (PET) likelihood for variable selection and parameter estimation for growing dimensional unconditional moment models in the presence of correlation among variables and model misspecification is presented.
5 citations
References
More filters
TL;DR: A new method for estimation in linear models called the lasso, which minimizes the residual sum of squares subject to the sum of the absolute value of the coefficients being less than a constant, is proposed.
Abstract: SUMMARY We propose a new method for estimation in linear models. The 'lasso' minimizes the residual sum of squares subject to the sum of the absolute value of the coefficients being less than a constant. Because of the nature of this constraint it tends to produce some coefficients that are exactly 0 and hence gives interpretable models. Our simulation studies suggest that the lasso enjoys some of the favourable properties of both subset selection and ridge regression. It produces interpretable models like subset selection and exhibits the stability of ridge regression. There is also an interesting relationship with recent work in adaptive function estimation by Donoho and Johnstone. The lasso idea is quite general and can be applied in a variety of statistical models: extensions to generalized regression models and tree-based models are briefly described.
40,785 citations
TL;DR: In this article, penalized likelihood approaches are proposed to handle variable selection problems, and it is shown that the newly proposed estimators perform as well as the oracle procedure in variable selection; namely, they work as well if the correct submodel were known.
Abstract: Variable selection is fundamental to high-dimensional statistical modeling, including nonparametric regression. Many approaches in use are stepwise selection procedures, which can be computationally expensive and ignore stochastic errors in the variable selection process. In this article, penalized likelihood approaches are proposed to handle these kinds of problems. The proposed methods select variables and estimate coefficients simultaneously. Hence they enable us to construct confidence intervals for estimated parameters. The proposed approaches are distinguished from others in that the penalty functions are symmetric, nonconcave on (0, ∞), and have singularities at the origin to produce sparse solutions. Furthermore, the penalty functions should be bounded by a constant to reduce bias and satisfy certain conditions to yield continuous solutions. A new algorithm is proposed for optimizing penalized likelihood functions. The proposed ideas are widely applicable. They are readily applied to a variety of ...
8,314 citations
Stanford University1, Cleveland Clinic2, University of Toronto3, Centre national de la recherche scientifique4, Université Paris-Saclay5, University of Paris-Sud6, Avaya7, Rutgers University8, RAND Corporation9, IBM10, University of Pennsylvania11, University of Western Australia12, University of Minnesota13
TL;DR: A publicly available algorithm that requires only the same order of magnitude of computational effort as ordinary least squares applied to the full set of covariates is described.
Abstract: The purpose of model selection algorithms such as All Subsets, Forward Selection and Backward Elimination is to choose a linear model on the basis of the same set of data to which the model will be applied. Typically we have available a large collection of possible covariates from which we hope to select a parsimonious set for the efficient prediction of a response variable. Least Angle Regression (LARS), a new model selection algorithm, is a useful and less greedy version of traditional forward selection methods. Three main properties are derived: (1) A simple modification of the LARS algorithm implements the Lasso, an attractive version of ordinary least squares that constrains the sum of the absolute regression coefficients; the LARS modification calculates all possible Lasso estimates for a given problem, using an order of magnitude less computer time than previous methods. (2) A different LARS modification efficiently implements Forward Stagewise linear regression, another promising new model selection method; this connection explains the similar numerical results previously observed for the Lasso and Stagewise, and helps us understand the properties of both methods, which are seen as constrained versions of the simpler LARS algorithm. (3) A simple approximation for the degrees of freedom of a LARS estimate is available, from which we derive a Cp estimate of prediction error; this allows a principled choice among the range of possible LARS estimates. LARS and its variants are computationally efficient: the paper describes a publicly available algorithm that requires only the same order of magnitude of computational effort as ordinary least squares applied to the full set of covariates.
7,828 citations
"Penalized Composite Quasi-Likelihoo..." refers background or methods in this paper
...…(16) can be recast as a penalized weighted least square regression argmin β n∑ i=1 w1∣∣∣Yi −XTi β̂ (0) ∣∣∣ + w2 ( Yi −XTi β )2 + n p∑ j=1 γλ(|β(0)j |)|βj | which can be efficiently solved by pathwise coordinate optimization (Friedman et al., 2008) or least angle regression (Efron et al., 2004)....
[...]
...) are all nonnegative. This class of problems can be solved with fast and efficient computational algorithms such as pathwise coordinate optimization (Friedman et al., 2008) and least angle regression (Efron et al., 2004). One particular example is the combination of L 1 and L 2 regressions, in which K= 2, ρ 1(t) = |t−b 0|andρ 2(t) = t2. Here b 0 denotes themedian of error distributionε. Iftheerror distribution is sym...
[...]
...i=1 w 1 Yi −XT i βˆ (0) +w 2 Yi −XT i β 2 +n Xp j=1 γλ(|β (0) j |)|βj| which can be efficiently solved by pathwise coordinate optimization (Friedman et al., 2008) or least angle regression (Efron et al., 2004). If b 0 6= 0, the penalized least-squares problem ( 16) is somewhat different from (5) since we have an additional parameter b 0. Using the same arguments, and treating b 0 as an additional parameter ...
[...]
...This class of problems can be solved with fast and efficient computational algorithms such as pathwise coordinate optimization (Friedman et al., 2008) and least angle regression (Efron et al., 2004)....
[...]
TL;DR: A new version of the lasso is proposed, called the adaptive lasso, where adaptive weights are used for penalizing different coefficients in the ℓ1 penalty, and the nonnegative garotte is shown to be consistent for variable selection.
Abstract: The lasso is a popular technique for simultaneous estimation and variable selection. Lasso variable selection has been shown to be consistent under certain conditions. In this work we derive a necessary condition for the lasso variable selection to be consistent. Consequently, there exist certain scenarios where the lasso is inconsistent for variable selection. We then propose a new version of the lasso, called the adaptive lasso, where adaptive weights are used for penalizing different coefficients in the l1 penalty. We show that the adaptive lasso enjoys the oracle properties; namely, it performs as well as if the true underlying model were given in advance. Similar to the lasso, the adaptive lasso is shown to be near-minimax optimal. Furthermore, the adaptive lasso can be solved by the same efficient algorithm for solving the lasso. We also discuss the extension of the adaptive lasso in generalized linear models and show that the oracle properties still hold under mild regularity conditions. As a bypro...
6,765 citations
TL;DR: In this article, a new approach toward a theory of robust estimation is presented, which treats in detail the asymptotic theory of estimating a location parameter for contaminated normal distributions, and exhibits estimators that are asyptotically most robust (in a sense to be specified) among all translation invariant estimators.
Abstract: This paper contains a new approach toward a theory of robust estimation; it treats in detail the asymptotic theory of estimating a location parameter for contaminated normal distributions, and exhibits estimators—intermediaries between sample mean and sample median—that are asymptotically most robust (in a sense to be specified) among all translation invariant estimators. For the general background, see Tukey (1960) (p. 448 ff.)
5,628 citations