Summary. We propose the elastic net, a new regularization and variable selection method. Real world data and a simulation study show that the elastic net often outperforms the lasso, while enjoying a similar sparsity of representation. In addition, the elastic net encourages a grouping effect, where strongly correlated predictors tend to be in or out of the model together.The elastic net is particularly useful when the number of predictors (p) is much bigger than the number of observations (n). By contrast, the lasso is not a very satisfactory variable selection method in the

/pdf/regularization-and-variable-selection-via-the-elastic-net-4z8q6j5o0i.pdf

Regularization and variable selection via the elastic net

We develop fast algorithms for estimation of generalized linear models with convex penalties. The models include linear regression, two-class logistic regression, and multinomial regression problems while the penalties include l(1) (the lasso), l(2) (ridge regression) and mixtures of the two (the elastic net). The algorithms use cyclical coordinate descent, computed along a regularization path. The methods can handle large problems and can also deal efficiently with sparse features. In comparative timings we find that the new algorithms are considerably faster than competing methods.

/pdf/regularization-paths-for-generalized-linear-models-via-39r1t606ac.pdf

Regularization Paths for Generalized Linear Models via Coordinate Descent

Today's Web-enabled deluge of electronic data calls for automated methods of data analysis. Machine learning provides these, developing methods that can automatically detect patterns in data and then use the uncovered patterns to predict future data. This textbook offers a comprehensive and self-contained introduction to the field of machine learning, based on a unified, probabilistic approach. The coverage combines breadth and depth, offering necessary background material on such topics as probability, optimization, and linear algebra as well as discussion of recent developments in the field, including conditional random fields, L1 regularization, and deep learning. The book is written in an informal, accessible style, complete with pseudo-code for the most important algorithms. All topics are copiously illustrated with color images and worked examples drawn from such application domains as biology, text processing, computer vision, and robotics. Rather than providing a cookbook of different heuristic methods, the book stresses a principled model-based approach, often using the language of graphical models to specify models in a concise and intuitive way. Almost all the models described have been implemented in a MATLAB software package--PMTK (probabilistic modeling toolkit)--that is freely available online. The book is suitable for upper-level undergraduates with an introductory-level college math background and beginning graduate students.

Machine Learning : A Probabilistic Perspective

Summary. We consider the problem of selecting grouped variables (factors) for accurate prediction in regression. Such a problem arises naturally in many practical situations with the multifactor analysis-of-variance problem as the most important and well-known example. Instead of selecting factors by stepwise backward elimination, we focus on the accuracy of estimation and consider extensions of the lasso, the LARS algorithm and the non-negative garrotte for factor selection. The lasso, the LARS algorithm and the non-negative garrotte are recently proposed regression methods that can be used to select individual variables. We study and propose efficient algorithms for the extensions of these methods for factor selection and show that these extensions give superior performance to the traditional stepwise backward elimination method in factor selection problems. We study the similarities and the differences between these methods. Simulations and real examples are used to illustrate the methods.

/pdf/model-selection-and-estimation-in-regression-with-grouped-45w0oq79ry.pdf

Model selection and estimation in regression with grouped variables

The lasso is a popular technique for simultaneous estimation and variable selection. Lasso variable selection has been shown to be consistent under certain conditions. In this work we derive a necessary condition for the lasso variable selection to be consistent. Consequently, there exist certain scenarios where the lasso is inconsistent for variable selection. We then propose a new version of the lasso, called the adaptive lasso, where adaptive weights are used for penalizing different coefficients in the l1 penalty. We show that the adaptive lasso enjoys the oracle properties; namely, it performs as well as if the true underlying model were given in advance. Similar to the lasso, the adaptive lasso is shown to be near-minimax optimal. Furthermore, the adaptive lasso can be solved by the same efficient algorithm for solving the lasso. We also discuss the extension of the adaptive lasso in generalized linear models and show that the oracle properties still hold under mild regularity conditions. As a bypro...

/pdf/the-adaptive-lasso-and-its-oracle-properties-p1nmst08nh.pdf

The adaptive lasso and its oracle properties

Zero-inflated count data are very common in health surveys. This study develops new variable selection methods for the zero-inflated Poisson regression model. Our simulations demonstrate the negative consequences which arise from the ignorance of zero-inflation. Among the competing methods, the one-step SCAD method is recommended because it has the highest specificity, sensitivity, exact fit, and lowest estimation error. The design of the simulations is based on the special features of two large national databases commonly used in the alcoholism and substance abuse field so that our findings can be easily generalized to the real settings. Applications of the methodology are demonstrated by empirical analyses on the data from a well-known alcohol study.

New variable selection methods for zero-inflated count data with applications to the substance abuse field

There has been considerable attention on estimation of conditional variance function in the literature. We propose here a nonparametric model for conditional covariance matrix. A kernel estimator is developed accordingly, its asymptotic bias and variance are derived, and its asymptotic normality is established. A real data example is used to illustrate the proposed estimation procedure.

Nonparametric Covariance Model

Although genome-wide association studies (GWAS) have proven powerful for comprehending the genetic architecture of complex traits, they are challenged by a high dimension of single-nucleotide polymorphisms (SNPs) as predictors, the presence of complex environmental factors, and longitudinal or functional natures of many complex traits or diseases. To address these challenges, we propose a high-dimensional varying-coefficient model for incorporating functional aspects of phenotypic traits into GWAS to formulate a so-called functional GWAS or fGWAS. The Bayesian group lasso and the associated MCMC algorithms are developed to identify significant SNPs and estimate how they affect longitudinal traits through time-varying genetic actions. The model is generalized to analyze the genetic control of complex traits using subject-specific sparse longitudinal data. The statistical properties of the new model are investigated through simulation studies. We use the new model to analyze a real GWAS data set from the Framingham Heart Study, leading to the identification of several significant SNPs associated with age-specific changes of body mass index. The fGWAS model, equipped with the Bayesian group lasso, will provide a useful tool for genetic and developmental analysis of complex traits or diseases.

Bayesian group Lasso for nonparametric varying-coefficient models with application to functional genome-wide association studies

This article proposes some probability plots are proposed to test spherical and elliptical symmetry in terms of some invariant statistics under orthogonal transformations. Some correlation coefficients as numerical measures of detecting deviation from spherical or elliptical symmetry are recommended, and the empirical percentiles of these correlation coefficients are calculated by simulation. The simulation results for data sets from 12 different populations show that the new plots are useful for testing spherical and elliptical symmetry. Some discussion is given also.

Some Q-Q Probability Plots to Test Spherical and Elliptical Symmetry

Some new statistics are proposed to test the uniformity of random samples in the multidimensional unit cube [0, 1] d (d ≥ 2). These statistics are derived from number-theoretic or quasi-Monte Carlo methods for measuring the discrepancy of points in [0,1] d Under the null hypothesis that the samples are independent and identically distributed with a uniform distribution in [0,1] d , we captain some asymptotic properties of the new statistics. By Monte Carlo simulation, it is found that the finite-sample distributions of the new statistics are well approximated by the standard normal distribution, N(0,1), or the chi-squared distribution, X 2 (2). A power study is performed, and possible applications of the new statistics to testing general multivariate goodness-of-fit problems are discussed.

https://www.ams.org/mcom/2001-70-233/S0025-5718-00-01203-5/S0025-5718-00-01203-5.pdf

Runze Li

Papers

New variable selection methods for zero-inflated count data with applications to the substance abuse field

Nonparametric Covariance Model

Bayesian group Lasso for nonparametric varying-coefficient models with application to functional genome-wide association studies

Some Q-Q Probability Plots to Test Spherical and Elliptical Symmetry

Testing multivariate uniformity and its applications