Showing papers in "Journal of the American Statistical Association in 2009"

On Consistency and Sparsity for Principal Components Analysis in High Dimensions

Abstract: Principal components analysis (PCA) is a classic method for the reduction of dimensionality of data in the form of n observations (or cases) of a vector with p variables. Contemporary datasets often have p comparable with or even much larger than n. Our main assertions, in such settings, are (a) that some initial reduction in dimensionality is desirable before applying any PCA-type search for principal modes, and (b) the initial reduction in dimensionality is best achieved by working in a basis in which the signals have a sparse representation. We describe a simple asymptotic model in which the estimate of the leading principal component vector via standard PCA is consistent if and only if p(n)/n → 0. We provide a simple algorithm for selecting a subset of coordinates with largest sample variances, and show that if PCA is done on the selected subset, then consistency is recovered, even if p(n) ≫ n.

Partial Correlation Estimation by Joint Sparse Regression Models.

Abstract: This article features online supplementary material.In this article, we propose a computationally efficient approach—space (Sparse PArtial Correlation Estimation)—for selecting nonzero partial correlations under the high-dimension-low-sample-size setting. This method assumes the overall sparsity of the partial correlation matrix and employs sparse regression techniques for model fitting. We illustrate the performance of space by extensive simulation studies. It is shown that space performs well in both nonzero partial correlation selection and the identification of hub variables, and also outperforms two existing methods. We then apply space to a microarray breast cancer dataset and identify a set of hub genes that may provide important insights on genetic regulatory networks. Finally, we prove that, under a set of suitable assumptions, the proposed procedure is asymptotically consistent in terms of model selection and parameter estimation.

On the Concept of Depth for Functional Data

Abstract: The statistical analysis of functional data is a growing need in many research areas. In particular, a robust methodology is important to study curves, which are the output of many experiments in applied statistics. As a starting point for this robust analysis, we propose, analyze, and apply a new definition of depth for functional observations based on the graphic representation of the curves. Given a collection of functions, it establishes the “centrality” of an observation and provides a natural center-outward ordering of the sample curves. Robust statistics, such as the median function or a trimmed mean function, can be defined from this depth definition. Its finite-dimensional version provides a new depth for multivariate data that is computationally feasible and useful for studying high-dimensional observations. Thus, this new depth is also suitable for complex observations such as microarray data, images, and those arising in some recent marketing and financial studies. Natural properties of these ...

Generalized Thresholding of Large Covariance Matrices

Abstract: We propose a new class of generalized thresholding operators that combine thresholding with shrinkage, and study generalized thresholding of the sample covariance matrix in high dimensions. Generalized thresholding of the covariance matrix has good theoretical properties and carries almost no computational burden. We obtain an explicit convergence rate in the operator norm that shows the tradeoff between the sparsity of the true model, dimension, and the sample size, and shows that generalized thresholding is consistent over a large class of models as long as the dimension p and the sample size n satisfy log p/n → 0. In addition, we show that generalized thresholding has the “sparsistency” property, meaning it estimates true zeros as zeros with probability tending to 1, and, under an additional mild condition, is sign consistent for nonzero elements. We show that generalized thresholding covers, as special cases, hard and soft thresholding, smoothly clipped absolute deviation, and adaptive lasso, and comp...

p-Values for High-Dimensional Regression

Abstract: Assigning significance in high-dimensional regression is challenging. Most computationally efficient selection algorithms cannot guard against inclusion of noise variables. Asymptotically valid p-values are not available. An exception is a recent proposal by Wasserman and Roeder that splits the data into two parts. The number of variables is then reduced to a manageable size using the first split, while classical variable selection techniques can be applied to the remaining variables, using the data from the second split. This yields asymptotic error control under minimal conditions. This involves a one-time random split of the data, however. Results are sensitive to this arbitrary choice, which amounts to a “p-value lottery” and makes it difficult to reproduce results. Here we show that inference across multiple random splits can be aggregated while maintaining asymptotic control over the inclusion of noise variables. We show that the resulting p-values can be used for control of both family-wise error a...

Forward Regression for Ultra-High Dimensional Variable Screening

Abstract: Motivated by the seminal theory of Sure Independence Screening (Fan and Lv 2008, SIS), we investigate here another popular and classical variable screening method, namely, forward regression (FR). Our theoretical analysis reveals that FR can identify all relevant predictors consistently, even if the predictor dimension is substantially larger than the sample size. In particular, if the dimension of the true model is finite, FR can discover all relevant predictors within a finite number of steps. To practically select the “best” candidate from the models generated by FR, the recently proposed BIC criterion of Chen and Chen (2008) can be used. The resulting model can then serve as an excellent starting point, from where many existing variable selection methods (e.g., SCAD and Adaptive LASSO) can be applied directly. FR’s outstanding finite sample performances are confirmed by extensive numerical studies.

Testing Dependence Among Serially Correlated Multicategory Variables

Abstract: The contingency table literature on tests for dependence among discrete multicategory variables is extensive. Standard tests assume, however, that draws are independent and only limited results exist on the effect of serial dependency—a problem that is important in areas such as economics, finance, medical trials, and meteorology. This article proposes new tests of independence based on canonical correlations from dynamically augmented reduced rank regressions. The tests allow for an arbitrary number of categories as well as multiway tables of arbitrary dimension and are robust in the presence of serial dependencies that take the form of finite-order Markov processes. For three-way or higher order tables we propose new tests of joint and marginal independence. Monte Carlo experiments show that the proposed tests have good finite sample properties. An empirical application to microeconomic survey data on firms' forecasts of changes to their production and prices demonstrates the importance of correcting fo...

Nonparametric Bayes Modeling of Multivariate Categorical Data

Abstract: Modeling of multivariate unordered categorical (nominal) data is a challenging problem, particularly in high dimensions and cases in which one wishes to avoid strong assumptions about the dependence structure. Commonly used approaches rely on the incorporation of latent Gaussian random variables or parametric latent class models. The goal of this article is to develop a nonparametric Bayes approach, which defines a prior with full support on the space of distributions for multiple unordered categorical variables. This support condition ensures that we are not restricting the dependence structure a priori. We show this can be accomplished through a Dirichlet process mixture of product multinomial distributions, which is also a convenient form for posterior computation. Methods for nonparametric testing of violations of independence are proposed, and the methods are applied to model positional dependence within transcription factor binding motifs.

Bayesian Modeling of Uncertainty in Ensembles of Climate Models

Abstract: Projections of future climate change caused by increasing greenhouse gases depend critically on numerical climate models coupling the ocean and atmosphere (global climate models [GCMs]). However, different models differ substantially in their projections, which raises the question of how the different models can best be combined into a probability distribution of future climate change. For this analysis, we have collected both current and future projected mean temperatures produced by nine climate models for 22 regions of the earth. We also have estimates of current mean temperatures from actual observations, together with standard errors, that can be used to calibrate the climate models. We propose a Bayesian analysis that allows us to combine the different climate models into a posterior distribution of future temperature increase, for each of the 22 regions, while allowing for the different climate models to have different variances. Two versions of the analysis are proposed: a univariate analysis in w...

Local Multidimensional Scaling for Nonlinear Dimension Reduction, Graph Drawing, and Proximity Analysis

Abstract: In the past decade there has been a resurgence of interest in nonlinear dimension reduction. Among new proposals are “Local Linear Embedding,” “Isomap,” and Kernel Principal Components Analysis which all construct global low-dimensional embeddings from local affine or metric information. We introduce a competing method called “Local Multidimensional Scaling” (LMDS). Like LLE, Isomap, and KPCA, LMDS constructs its global embedding from local information, but it uses instead a combination of MDS and “force-directed” graph drawing. We apply the force paradigm to create localized versions of MDS stress functions with a tuning parameter to adjust the strength of nonlocal repulsive forces. We solve the problem of tuning parameter selection with a meta-criterion that measures how well the sets of K-nearest neighbors agree between the data and the embedding. Tuned LMDS seems to be able to outperform MDS, PCA, LLE, Isomap, and KPCA, as illustrated with two well-known image datasets. The meta-criterion can also be ...

Shrinkage Estimation of the Varying Coefficient Model

Abstract: The varying coefficient model is a useful extension of the linear regression model. Nevertheless, how to conduct variable selection for the varying coefficient model in a computationally efficient manner is poorly understood. To solve the problem, we propose here a novel method, which combines the ideas of the local polynomial smoothing and the Least Absolute Shrinkage and Selection Operator (LASSO). The new method can do nonparametric estimation and variable selection simultaneously. With a local constant estimator and the adaptive LASSO penalty, the new method can identify the true model consistently, and that the resulting estimator can be as efficient as the oracle estimator. Numerical studies clearly confirm our theories. Extension to other shrinkage methods (e.g., the SCAD, i.e., the Smoothly Clipped Absolute Deviation.) and other smoothing methods is straightforward.

Multivariate Statistical Process Control Using LASSO

Abstract: This article develops a new multivariate statistical process control (SPC) methodology based on adapting the LASSO variable selection method to the SPC problem. The LASSO method has the sparsity property of being able to select exactly the set of nonzero regression coefficients in multivariate regression modeling, which is especially useful in cases where the number of nonzero coefficients is small. In multivariate SPC applications, process mean vectors often shift in a small number of components. Our primary goals are to detect such a shift as soon as it occurs and to identify the shifted mean components. Using this connection between the two problems, we propose a LASSO-based multivariate test statistic, and then integrate this statistic into the multivariate EWMA charting scheme for Phase II multivariate process monitoring. We show that this approach balances protection against various shift levels and shift directions, and thus provides an effective tool for multivariate SPC applications. This article...

Jackknife Empirical Likelihood

Abstract: Empirical likelihood has been found very useful in many different occasions. However, when applied directly to some more complicated statistics such as U-statistics, it runs into serious computational difficulties. In this paper, we introduce a so-called jackknife empirical likelihood (JEL) method. The new method is extremely simple to use in practice. In particular, the JEL is shown to be very effective in handling one and two-sample U-statistics. The JEL can be potentially useful for other nonlinear statistics.

Locally Weighted Censored Quantile Regression

Abstract: Censored quantile regression offers a valuable supplement to Cox proportional hazards model for survival analysis. Existing work in the literature often requires stringent assumptions, such as unconditional independence of the survival time and the censoring variable or global linearity at all quantile levels. Moreover, some of the work uses recursive algorithms, making it challenging to derive asymptotic normality. To overcome these drawbacks, we propose a new locally weighted censored quantile regression approach that adopts the redistribution-of-mass idea and employs a local reweighting scheme. Its validity only requires conditional independence of the survival time and the censoring variable given the covariates, and linearity at the particular quantile level of interest. Our method leads to a simple algorithm that can be conveniently implemented with R software. Applying recent theory of M-estimation with infinite dimensional parameters, we establish the consistency and asymptotic normality of the pr...

Nonparametric Bayes Conditional Distribution Modeling With Variable Selection

Abstract: This article considers a methodology for flexibly characterizing the relationship between a response and multiple predictors. Goals are (1) to estimate the conditional response distribution addressing the distributional changes across the predictor space, and (2) to identify important predictors for the response distribution change both within local regions and globally. We first introduce the probit stick-breaking process (PSBP) as a prior for an uncountable collection of predictor-dependent random distributions and propose a PSBP mixture (PSBPM) of normal regressions for modeling the conditional distributions. A global variable selection structure is incorporated to discard unimportant predictors, while allowing estimation of posterior inclusion probabilities. Local variable selection is conducted relying on the conditional distribution estimates at different predictor points. An efficient stochastic search sampling algorithm is proposed for posterior computation. The methods are illustrated through simulation and applied to an epidemiologic study.

A Factor Model Approach to Multiple Testing Under Dependence

Abstract: The impact of dependence between individual test statistics is currently among the most discussed topics in the multiple testing of high-dimensional data literature, especially since Benjamini and Hochberg (1995) introduced the false discovery rate (FDR). Many papers have first focused on the impact of dependence on the control of the FDR. Some more recent works have investigated approaches that account for common information shared by all the variables to stabilize the distribution of the error rates. Similarly, we propose to model this sharing of information by a factor analysis structure for the conditional variance of the test statistics. It is shown that the variance of the number of false discoveries increases along with the fraction of common variance. Test statistics for general linear contrasts are deduced, taking advantage of the common factor structure to reduce the variance of the error rates. A conditional FDR estimate is proposed and the overall performance of multiple testing procedure is s...

Bayesian Model Averaging Continual Reassessment Method in Phase I Clinical Trials

Abstract: The continual reassessment method (CRM) is a popular dose-finding design for phase I clinical trials. This method requires that practitioners prespecify the toxicity probability at each dose. Such prespecification can be arbitrary, and different specifications of toxicity probabilities may lead to very different design properties. To overcome the arbitrariness and further enhance the robustness of the design, we propose using multiple parallel CRM models, each with a different set of prespecified toxicity probabilities. In the Bayesian paradigm, we assign a discrete probability mass to each CRM model as the prior model probability. The posterior probabilities of toxicity can be estimated by the Bayesian model averaging (BMA) approach. Dose escalation or deescalation is determined by comparing the target toxicity rate and the BMA estimates of the dose toxicity probabilities. We examine the properties of the BMA-CRM approach through extensive simulation studies, and also compare this new method and its vari...

Variable Selection for Partially Linear Models with Measurement Errors.

Abstract: This article focuses on variable selection for partially linear models when the covariates are measured with additive errors. We propose two classes of variable selection procedures, penalized least squares and penalized quantile regression, using the nonconvex penalized principle. The first procedure corrects the bias in the loss function caused by the measurement error by applying the so-called correction-for-attenuation approach, whereas the second procedure corrects the bias by using orthogonal regression. The sampling properties for the two procedures are investigated. The rate of convergence and the asymptotic normality of the resulting estimates are established. We further demonstrate that, with proper choices of the penalty functions and the regularization parameter, the resulting estimates perform asymptotically as well as an oracle procedure as proposed by Fan and Li. Choice of smoothing parameters is also discussed. Finite sample performance of the proposed variable selection procedures is asse...

Generalized Multilevel Functional Regression

Abstract: We introduce Generalized Multilevel Functional Linear Models (GMFLMs), a novel statistical framework for regression models where exposure has a multilevel functional structure We show that GMFLMs are, in fact, generalized multilevel mixed models Thus, GMFLMs can be analyzed using the mixed effects inferential machinery and can be generalized within a well-researched statistical framework We propose and compare two methods for inference: (1) a two-stage frequentist approach; and (2) a joint Bayesian analysis Our methods are motivated by and applied to the Sleep Heart Health Study, the largest community cohort study of sleep However, our methods are general and easy to apply to a wide spectrum of emerging biological and medical datasets Supplemental materials for this article are available online

Likelihood-Based Sufficient Dimension Reduction

Abstract: We obtain the maximum likelihood estimator of the central subspace under conditional normality of the predictors given the response. Analytically and in simulations we found that our new estimator can preform much better than sliced inverse regression, sliced average variance estimation and directional regression, and that it seems quite robust to deviations from normality.

Learn From Thy Neighbor: Parallel-Chain and Regional Adaptive MCMC

Abstract: Starting with the seminal paper of Haario, Saksman and Tamminen (Haario et al. (2001)), a substantial amount of work has been done to validate adaptive Markov chain Monte Carlo algorithms. In this paper we focus on two practical aspects of adaptive Metropolis samplers. First, we draw attention to the deficient performance of standard adaptation when the target distribution is multi-modal. We propose a parallel chain adaptation strategy that incorporates multiple Markov chains which are run in parallel. Second, we note that

Conditional Quantile Estimation for Generalized Autoregressive Conditional Heteroscedasticity Models

Abstract: Conditional quantile estimation is an essential ingredient in modern risk management. Although generalized autoregressive conditional heteroscedasticity (GARCH) processes have proven highly successful in modeling financial data, it is generally recognized that it would be useful to consider a broader class of processes capable of representing more flexibly both asymmetry and tail behavior of conditional returns distributions. In this article we study estimation of conditional quantiles for GARCH models using quantile regression. Quantile regression estimation of GARCH models is highly nonlinear; we propose a simple and effective two-step approach of quantile regression estimation for linear GARCH time series. In the first step, we use a quantile autoregression sieve approximation for the GARCH model by combining information over different quantiles. Then second-stage estimation for the GARCH model is carried out based on the first-stage minimum distance estimation of the scale process of the time series. ...

Empirical Bayes Estimates for Large-Scale Prediction Problems.

Abstract: Classical prediction methods, such as Fisher’s linear discriminant function, were designed for small-scale problems in which the number of predictors N is much smaller than the number of observations n. Modern scientific devices often reverse this situation. A microarray analysis, for example, might include n=100 subjects measured on N=10,000 genes, each of which is a potential predictor. This article proposes an empirical Bayes approach to large-scale prediction, where the optimum Bayes prediction rule is estimated employing the data from all of the predictors. Microarray examples are used to illustrate the method. The results demonstrate a close connection with the shrunken centroids algorithm of Tibshirani et al. (2002), a frequentist regularization approach to large-scale prediction, and also with false discovery rate theory.

Amplification of Sensitivity Analysis in Matched Observational Studies

Abstract: A sensitivity analysis displays the increase in uncertainty that attends an inference when a key assumption is relaxed In matched observational studies of treatment effects, a key assumption in some analyses is that subjects matched for observed covariates are comparable, and this assumption is relaxed by positing a relevant covariate that was not observed and not controlled by matching What properties would such an unobserved covariate need to have to materially alter the inference about treatment effects? For ease of calculation and reporting, it is convenient that the sensitivity analysis be of low dimension, perhaps indexed by a scalar sensitivity parameter, but for interpretation in specific contexts, a higher dimensional analysis may be of greater relevance An amplification of a sensitivity analysis is defined as a map from each point in a low-dimensional sensitivity analysis to a set of points, perhaps a “curve,” in a higher dimensional sensitivity analysis such that the possible inferences are

Likelihood-Based Analysis of Causal Effects of Job-Training Programs Using Principal Stratification

Abstract: Government-sponsored job-training programs must be subject to evaluation to assess whether their effectiveness justifies their cost to the public. The evaluation usually focuses on employment and total earnings, although the effect on wages is also of interest, because this effect reflects the increase in human capital due to the training program, whereas the effect on total earnings may be simply reflecting the increased likelihood of employment without any effect on wage rates. Estimating the effects of training programs on wages is complicated by the fact that, even in a randomized experiment, wages are “truncated” (or less accurately “censored”) by nonemployment, that is, they are only observed and well-defined for individuals who are employed. In this article, we develop a likelihood-based approach to estimate the wage effect of the US federally-funded Job Corps training program using “Principal Stratification”. Our estimands are formulated in terms of: (1) the effect of the training program on wages...

A Case Study in Exploratory Functional Data Analysis: Geometrical Features of the Internal Carotid Artery

Abstract: This pilot study is a product of the AneuRisk Project, a scientific program that aims at evaluating the role of vascular geometry and hemodynamics in the pathogenesis of cerebral aneurysms. By means of functional data analyses, we explore the AneuRisk dataset to highlight the relations between the geometric features of the internal carotid artery, expressed by its radius profile and centerline curvature, and the aneurysm location. After introducing a new similarity index for functional data, we eliminate ancillary variability of vessel radius and curvature profiles through an iterative registration procedure. We then reduce data dimension by means of functional principal components analysis. Last, a quadratic discriminant analysis of functional principal components scores allows us to discriminate patients with aneurysms in different districts.

Simultaneous Testing of Grouped Hypotheses: Finding Needles in Multiple Haystacks

Abstract: In large-scale multiple testing problems, data are often collected from heterogeneous sources and hypotheses form into groups that exhibit different characteristics. Conventional approaches, including the pooled and separate analyses, fail to efficiently utilize the external grouping information. We develop a compound decision theoretic framework for testing grouped hypotheses and introduce an oracle procedure that minimizes the false nondiscovery rate subject to a constraint on the false discovery rate. It is shown that both the pooled and separate analyses can be uniformly improved by the oracle procedure. We then propose a data-driven procedure that is shown to be asymptotically optimal. Simulation studies show that our procedures enjoy superior performance and yield the most accurate results in comparison with both the pooled and separate procedures. A real-data example with grouped hypotheses is studied in detail using different methods. Both theoretical and numerical results demonstrate that exploit...

Quantile Regression With Measurement Error

Abstract: Regression quantiles can be substantially biased when the covariates are measured with error. In this paper we propose a new method that produces consistent linear quantile estimation in the presence of covariate measurement error. The method corrects the measurement error induced bias by constructing joint estimating equations that simultaneously hold for all the quantile levels. An iterative EM-type estimation algorithm to obtain the solutions to such joint estimation equations is provided. The finite sample performance of the proposed method is investigated in a simulation study, and compared to the standard regression calibration approach. Finally, we apply our methodology to part of the National Collaborative Perinatal Project growth data, a longitudinal study with an unusual measurement error structure.

A Multivariate Extension of the Dynamic Logit Model for Longitudinal Data Based on a Latent Markov Heterogeneity Structure

Abstract: For the analysis of multivariate categorical longitudinal data, we propose an extension of the dynamic logit model. The resulting model is based on a marginal parameterization of the conditional distribution of each vector of response variables given the covariates, the lagged response variables, and a set of subject-specific parameters for the unobserved heterogeneity. The latter ones are assumed to follow a first-order Markov chain. For the maximum likelihood estimation of the model parameters, we outline an EM algorithm. The data analysis approach based on the proposed model is illustrated by a simulation study and an application to a dataset, which derives from the Panel Study on Income Dynamics and concerns fertility and female participation to the labor market.

Time Series Modelling with Semiparametric Factor Dynamics

Abstract: High-dimensional regression problems, which reveal dynamic behavior, are typically analyzed by time propagation of a few number of factors. The inference on the whole system is then based on the low-dimensional time series analysis. Such high-dimensional problems occur frequently in many different fields of science. In this article we address the problem of inference when the factors and factor loadings are estimated by semiparametric methods. This more flexible modeling approach poses an important question: Is it justified, from an inferential point of view, to base statistical inference on the estimated times series factors? We show that the difference of the inference based on the estimated time series and “true” unobserved time series is asymptotically negligible. Our results justify fitting vector autoregressive processes to the estimated factors, which allows one to study the dynamics of the whole high-dimensional system with a low-dimensional representation. We illustrate the theory with a simulati...