Summary. We propose the elastic net, a new regularization and variable selection method. Real world data and a simulation study show that the elastic net often outperforms the lasso, while enjoying a similar sparsity of representation. In addition, the elastic net encourages a grouping effect, where strongly correlated predictors tend to be in or out of the model together.The elastic net is particularly useful when the number of predictors (p) is much bigger than the number of observations (n). By contrast, the lasso is not a very satisfactory variable selection method in the

/pdf/regularization-and-variable-selection-via-the-elastic-net-4z8q6j5o0i.pdf

Regularization and variable selection via the elastic net

We develop fast algorithms for estimation of generalized linear models with convex penalties. The models include linear regression, two-class logistic regression, and multinomial regression problems while the penalties include l(1) (the lasso), l(2) (ridge regression) and mixtures of the two (the elastic net). The algorithms use cyclical coordinate descent, computed along a regularization path. The methods can handle large problems and can also deal efficiently with sparse features. In comparative timings we find that the new algorithms are considerably faster than competing methods.

/pdf/regularization-paths-for-generalized-linear-models-via-39r1t606ac.pdf

Regularization Paths for Generalized Linear Models via Coordinate Descent

Today's Web-enabled deluge of electronic data calls for automated methods of data analysis. Machine learning provides these, developing methods that can automatically detect patterns in data and then use the uncovered patterns to predict future data. This textbook offers a comprehensive and self-contained introduction to the field of machine learning, based on a unified, probabilistic approach. The coverage combines breadth and depth, offering necessary background material on such topics as probability, optimization, and linear algebra as well as discussion of recent developments in the field, including conditional random fields, L1 regularization, and deep learning. The book is written in an informal, accessible style, complete with pseudo-code for the most important algorithms. All topics are copiously illustrated with color images and worked examples drawn from such application domains as biology, text processing, computer vision, and robotics. Rather than providing a cookbook of different heuristic methods, the book stresses a principled model-based approach, often using the language of graphical models to specify models in a concise and intuitive way. Almost all the models described have been implemented in a MATLAB software package--PMTK (probabilistic modeling toolkit)--that is freely available online. The book is suitable for upper-level undergraduates with an introductory-level college math background and beginning graduate students.

Machine Learning : A Probabilistic Perspective

Summary. We consider the problem of selecting grouped variables (factors) for accurate prediction in regression. Such a problem arises naturally in many practical situations with the multifactor analysis-of-variance problem as the most important and well-known example. Instead of selecting factors by stepwise backward elimination, we focus on the accuracy of estimation and consider extensions of the lasso, the LARS algorithm and the non-negative garrotte for factor selection. The lasso, the LARS algorithm and the non-negative garrotte are recently proposed regression methods that can be used to select individual variables. We study and propose efficient algorithms for the extensions of these methods for factor selection and show that these extensions give superior performance to the traditional stepwise backward elimination method in factor selection problems. We study the similarities and the differences between these methods. Simulations and real examples are used to illustrate the methods.

/pdf/model-selection-and-estimation-in-regression-with-grouped-45w0oq79ry.pdf

Model selection and estimation in regression with grouped variables

The lasso is a popular technique for simultaneous estimation and variable selection. Lasso variable selection has been shown to be consistent under certain conditions. In this work we derive a necessary condition for the lasso variable selection to be consistent. Consequently, there exist certain scenarios where the lasso is inconsistent for variable selection. We then propose a new version of the lasso, called the adaptive lasso, where adaptive weights are used for penalizing different coefficients in the l1 penalty. We show that the adaptive lasso enjoys the oracle properties; namely, it performs as well as if the true underlying model were given in advance. Similar to the lasso, the adaptive lasso is shown to be near-minimax optimal. Furthermore, the adaptive lasso can be solved by the same efficient algorithm for solving the lasso. We also discuss the extension of the adaptive lasso in generalized linear models and show that the oracle properties still hold under mild regularity conditions. As a bypro...

/pdf/the-adaptive-lasso-and-its-oracle-properties-p1nmst08nh.pdf

The adaptive lasso and its oracle properties

For theoretical properties of variable selection procedures for Cox's model, we study the asymptotic behavior of partial likelihood for the Cox model. We find that the partial likelihood does not behave like an ordinary likelihood, whose sample average typically tends to its expected value, a finite number, in probability. Under some mild conditions, we prove that the sample average of partial likelihood tends to infinity at the rate of the logarithm of the sample size, in probability. We apply the asymptotic results on the partial likelihood to study tuning parameter selection for penalized partial likelihood. We find that the penalized partial likelihood with the generalized cross-validation (GCV) tuning parameter proposed in Fan and Li (2002) enjoys the model selection consistency property, despite the fact that GCV, AIC and Cp , equivalent in the context of linear regression models, are not model selection consistent. Our empirical studies via Monte Carlo simulation and a data example confirm our theoretical findings.

/pdf/asymptotic-behavior-of-cox-s-partial-likelihood-and-its-50lucoo6g5.pdf

Asymptotic Behavior of Cox's Partial Likelihood and its Application to Variable Selection.

In partially linear single-index models, we obtain the semiparametrically efficient profile least-squares estimators of regression coefficients. We also employ the smoothly clipped absolute deviation penalty (SCAD) approach to simultaneously select variables and estimate regression coefficients. We show that the resulting SCAD estimators are consistent and possess the oracle property. Subsequently, we demonstrate that a proposed tuning parameter selector, BIC, identifies the true model consistently. Finally, we develop a linear hypothesis test for the parametric coefficients and a goodness-of-fit test for the nonparametric component, respectively. Monte Carlo studies are also presented.

Estimation and testing for partially linear single-index models

Motivated by an empirical analysis of stock reaction to COVID-19 pandemic, we propose a generalized mediation model with high-dimensional potential mediators to study the mediation effects of financial metrics that bridge company’s sector and stock value. We propose an estimation procedure for the direct effect via a partial penalized maximum likelihood method and establish its theoretical properties. We develop a Wald test for the indirect effect and show that the proposed test has a χ2 limiting null distribution. We also develop a partial penalized likelihood ratio test for the direct effect and show that the proposed test asymptotically follows a χ2-distribution under null hypothesis. A more efficient estimator of indirect effect under complete mediation model is also developed. Simulation studies are conducted to examine the finite sample performance of the proposed procedures and compare with some existing methods. We further illustrate the proposed methodology with an empirical analysis of stock reaction to COVID-19 pandemic via exploring the underlying mechanism of the relationship between companies’ sectors and their stock values. 

Estimations and Tests for Generalized Mediation Models with High-Dimensional Potential Mediators

The Heat Distribution Mechanism and Improved Temperature Model Based on Discrete Heat Source

Bacterial cytochrome P450s represent an emerging enzyme family that can modify ribosomally synthesized peptides to generate structurally complex macrocyclic skeletons. However, the functional sequence space of this type of enzyme is largely unexplored. In this study, we conduct a systematic genome mining of small ribosomal peptide-tailoring P450s from genomes of actinobacteria via a precursor-centric, primary sequence-, and structure-guided strategy. We uncovered 1,957 putative P450s, prioritized two representative families for functional study, and characterized two P450 enzymes that can respectively catalyze Tyrosine-to-Tryptophan and Tryptophan-to-Tryptophan crosslinks to form 3-mer or 4-mer macrocycle. These two P450 enzymes exhibit broad substrate selectivity, suggesting a promising starting template for engineering unnatural cyclic peptide construction. Our work expanded the enzymatic catalysis of P450s and could inspire the community to discover hidden peptide-modifying enzymes.

/pdf/bacterial-cytochrome-p450-catalyzed-post-translational-c753rqio.pdf

Runze Li

Papers

Asymptotic Behavior of Cox's Partial Likelihood and its Application to Variable Selection.

Estimation and testing for partially linear single-index models

Estimations and Tests for Generalized Mediation Models with High-Dimensional Potential Mediators

The Heat Distribution Mechanism and Improved Temperature Model Based on Discrete Heat Source

Bacterial Cytochrome P450-catalyzed Post-translational Macrocyclization