Scikit-learn is a Python module integrating a wide range of state-of-the-art machine learning algorithms for medium-scale supervised and unsupervised problems. This package focuses on bringing machine learning to non-specialists using a general-purpose high-level language. Emphasis is put on ease of use, performance, documentation, and API consistency. It has minimal dependencies and is distributed under the simplified BSD license, encouraging its use in both academic and commercial settings. Source code, binaries, and documentation can be downloaded from http://scikit-learn.sourceforge.net.

/pdf/scikit-learn-machine-learning-in-python-23clfh9q83.pdf

Scikit-learn: Machine Learning in Python

In comparative high-throughput sequencing assays, a fundamental task is the analysis of count data, such as read counts per gene in RNA-seq, for evidence of systematic changes across experimental conditions. Small replicate numbers, discreteness, large dynamic range and the presence of outliers require a suitable statistical approach. We present DESeq2, a method for differential analysis of count data, using shrinkage estimation for dispersions and fold changes to improve stability and interpretability of estimates. This enables a more quantitative analysis focused on the strength rather than the mere presence of differential expression. The DESeq2 package is available at http://www.bioconductor.org/packages/release/bioc/html/DESeq2.html
 .

/pdf/moderated-estimation-of-fold-change-and-dispersion-for-rna-3ywnbkcan0.pdf

Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2

SUMMARY We propose a new method for estimation in linear models. The 'lasso' minimizes the residual sum of squares subject to the sum of the absolute value of the coefficients being less than a constant. Because of the nature of this constraint it tends to produce some coefficients that are exactly 0 and hence gives interpretable models. Our simulation studies suggest that the lasso enjoys some of the favourable properties of both subset selection and ridge regression. It produces interpretable models like subset selection and exhibits the stability of ridge regression. There is also an interesting relationship with recent work in adaptive function estimation by Donoho and Johnstone. The lasso idea is quite general and can be applied in a variety of statistical models: extensions to generalized regression models and tree-based models are briefly described.

/pdf/regression-shrinkage-and-selection-via-the-lasso-6mx35txfr5.pdf

Regression Shrinkage and Selection via the Lasso

We propose a deep convolutional neural network architecture codenamed Inception that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14). The main hallmark of this architecture is the improved utilization of the computing resources inside the network. By a carefully crafted design, we increased the depth and width of the network while keeping the computational budget constant. To optimize quality, the architectural decisions were based on the Hebbian principle and the intuition of multi-scale processing. One particular incarnation used in our submission for ILSVRC14 is called GoogLeNet, a 22 layers deep network, the quality of which is assessed in the context of classification and detection.

/pdf/going-deeper-with-convolutions-1yobw2o2ds.pdf

Going deeper with convolutions

Scikit-learn is a Python module integrating a wide range of state-of-the-art machine learning algorithms for medium-scale supervised and unsupervised problems. This package focuses on bringing machine learning to non-specialists using a general-purpose high-level language. Emphasis is put on ease of use, performance, documentation, and API consistency. It has minimal dependencies and is distributed under the simplified BSD license, encouraging its use in both academic and commercial settings. Source code, binaries, and documentation can be downloaded from this http URL.

So far in this book, we have mostly focused on linear models. Linear models are relatively simple to describe and implement, and have advantages over other approaches in terms of interpretation and inference. However, standard linear regression can have significant limitations in terms of predictive power. This is because the linearity assumption is almost always an approximation, and sometimes a poor one. In Chapter 6 we see that we can improve upon least squares using ridge regression, the lasso, principal components regression, and other techniques. In that setting, the improvement is obtained by reducing the complexity of the linear model, and hence the variance of the estimates. But we are still using a linear model, which can only be improved so far! In this chapter we relax the linearity assumption while still attempting to maintain as much interpretability as possible. We do this by examining very simple extensions of linear models like polynomial regression and step functions, as well as more sophisticated approaches such as splines, local regression, and generalized additive models.

Moving Beyond Linearity

In this paper we purpose a blockwise descent algorithm for grouppenalized multiresponse regression. Using a quasi-newton framework we extend this to group-penalized multinomial regression. We give a publicly available implementation for these in R, and compare the speed of this algorithm to a competing algorithm | we show that our implementation is an order of magnitude faster than its competitor, and can solve gene-expression-sized problems in real time.

A Blockwise Descent Algorithm for Group-penalized Multiresponse and

Regression models with crossed random effect errors can be very expensive to compute. The cost of both generalized least squares and Gibbs sampling can easily grow as $N^{3/2}$ (or worse) for $N$ observations. Papaspiliopoulos et al. (2020) present a collapsed Gibbs sampler that costs $O(N)$, but under an extremely stringent sampling model. We propose a backfitting algorithm to compute a generalized least squares estimate and prove that it costs $O(N)$. A critical part of the proof is in ensuring that the number of iterations required is $O(1)$ which follows from keeping a certain matrix norm below $1-\delta$ for some $\delta>0$. Our conditions are greatly relaxed compared to those for the collapsed Gibbs sampler, though still strict. Empirically, the backfitting algorithm has a norm below $1-\delta$ under conditions that are less strict than those in our assumptions. We illustrate the new algorithm on a ratings data set from Stitch Fix.

Backfitting for large scale crossed random effects regressions

Undirected Graphical Models

The ubiquitous phenomenon of modern massive datasets, such as Web data and data streams, has brought in both challenges and opportunities to scientists and engineers. For example, simple operations like computing AA T become difficult when the data matrix A grows to the Web scale. Various sampling techniques have been developed to store massive data using small space and to recover the original summary statistics (e.g., pair-wise distances) from compact data representations. This thesis will elaborate two sampling techniques, stable random projections and Conditional Random Sampling (CRS). 
The method of stable random projections multiplies A ∈ Rn ×  D by a random matrix R ∈ RD ×  k, resulting in B=AR ∈ Rn ×  k. The entries of R are typically sampled i.i.d. from a symmetric α-stable distribution (0 < α ≤ 2); and we can estimate the original Lα properties of A from B. In the L2 case, the advantage of stable random projections is highlighted by the celebrated Johnson-Lindenstrauss (JL) Lemma, which says it suffices to let k = O(log n).
We prove an analog of the JL Lemma for 0 < α ≤ 2. The method of stable random projections boils down to a statistical estimation problem, for estimating the scale parameter of a symmetric α-stable distribution. We seek estimators that are both statistically accurate and computationally efficient. We compare various estimators including the arithmetic mean, the geometric mean, the harmonic mean, the fractional power, and the maximum likelihood estimators. 
The method of stable random projections does not take advantage of data sparsity while large-scale datasets are often sparse. Conditional Random Sampling (CRS) utilizes data sparsity by sampling only from non-zero data elements. In the estimation stage, CRS reconstructs equivalent (conditional) random samples on a pair-wise or group-wise basis, with the (equivalent) sample size retrospectively determined. Theoretical analysis will be conditional on the sample size. 
In practical scenarios, CRS often outperforms stable random projections . Also, CRS is a general-purpose technique while the method of stable random projections is associated with a particular Lα norm. These two methods can be complementary to each other, for solving large-scale science and engineering problems involving massive computations of distances.

Trevor Hastie

Papers

Moving Beyond Linearity

A Blockwise Descent Algorithm for Group-penalized Multiresponse and

Backfitting for large scale crossed random effects regressions

Undirected Graphical Models

Stable random projections and conditional random sampling, two sampling techniques for modern massive datasets