Maximum likelihood or restricted maximum likelihood (REML) estimates of the parameters in linear mixed-effects models can be determined using the lmer function in the lme4 package for R. As for most model-fitting functions in R, the model is described in an lmer call by a formula, in this case including both fixed- and random-effects terms. The formula and data together determine a numerical representation of the model from which the profiled deviance or the profiled REML criterion can be evaluated as a function of some of the model parameters. The appropriate criterion is optimized, using one of the constrained optimization functions in R, to provide the parameter estimates. We describe the structure of the model, the steps in evaluating the profiled deviance or REML criterion, and the structure of classes or types that represents such a model. Sufficient detail is included to allow specialization of these structures by users who wish to write functions to fit specialized linear mixed models, such as models incorporating pedigrees or smoothing splines, that are not easily expressible in the formula language used by lmer.

/pdf/fitting-linear-mixed-effects-models-using-lme4-4zzrpoy9lh.pdf

Fitting Linear Mixed-Effects Models Using lme4

Many problems of recent interest in statistics and machine learning can be posed in the framework of convex optimization. Due to the explosion in size and complexity of modern datasets, it is increasingly important to be able to solve problems with a very large number of features or training examples. As a result, both the decentralized collection or storage of these datasets as well as accompanying distributed solution methods are either necessary or at least highly desirable. In this review, we argue that the alternating direction method of multipliers is well suited to distributed convex optimization, and in particular to large-scale problems arising in statistics, machine learning, and related areas. The method was developed in the 1970s, with roots in the 1950s, and is equivalent or closely related to many other algorithms, such as dual decomposition, the method of multipliers, Douglas–Rachford splitting, Spingarn's method of partial inverses, Dykstra's alternating projections, Bregman iterative algorithms for l1 problems, proximal methods, and others. After briefly surveying the theory and history of the algorithm, we discuss applications to a wide variety of statistical and machine learning problems of recent interest, including the lasso, sparse logistic regression, basis pursuit, covariance selection, support vector machines, and many others. We also discuss general distributed optimization, extensions to the nonconvex setting, and efficient implementation, including some details on distributed MPI and Hadoop MapReduce implementations.

/pdf/distributed-optimization-and-statistical-learning-via-the-8s9gtddw0d.pdf

Distributed Optimization and Statistical Learning Via the Alternating Direction Method of Multipliers

Numerical Optimization presents a comprehensive and up-to-date description of the most effective methods in continuous optimization. It responds to the growing interest in optimization in engineering, science, and business by focusing on the methods that are best suited to practical problems. For this new edition the book has been thoroughly updated throughout. There are new chapters on nonlinear interior methods and derivative-free methods for optimization, both of which are used widely in practice and the focus of much current research. Because of the emphasis on practical methods, as well as the extensive illustrations and exercises, the book is accessible to a wide audience. It can be used as a graduate text in engineering, operations research, mathematics, computer science, and business. It also serves as a handbook for researchers and practitioners in the field. The authors have strived to produce a text that is pleasant to read, informative, and rigorous - one that reveals both the beautiful nature of the discipline and its practical side.

Numerical Optimization

Fitting Linear Mixed-Effects Models using lme4

/pdf/convex-analysisnoer-san-nojin-zhan-nituite-5dl2rbmoyr.pdf

Convex Analysisの二,三の進展について

The Sherman–Morrison–Woodbury formulas relate the inverse of a matrix after a small-rank perturbation to the inverse of the original matrix. The history of these fomulas is presented and various applications to statistics, networks, structural analysis, asymptotic analysis, optimization, and partial differential equations are discussed. The Sherman-Morrison-Woodbury formulas express the inverse of a matrix after a small rank perturbation in terms of the inverse of the original matrix. This paper surveys the history of these formulas and we examine some applications where these formulas are helpful

Updating the inverse of a matrix

A new nonlinear conjugate gradient method and an associated implementation, based on an inexact line search, are proposed and analyzed With exact line search, our method reduces to a nonlinear version of the Hestenes--Stiefel conjugate gradient scheme For any (inexact) line search, our scheme satisfies the descent condition ${\bf g}_k^{\sf T} {\bf d}_k \le -\frac{7}{8} \|{\bf g}_k\|^2$ Moreover, a global convergence result is established when the line search fulfills the Wolfe conditions A new line search scheme is developed that is efficient and highly accurate Efficiency is achieved by exploiting properties of linear interpolants in a neighborhood of a local minimizer High accuracy is achieved by using a convergence criterion, which we call the ``approximate Wolfe'' conditions, obtained by replacing the sufficient decrease criterion in the Wolfe conditions with an approximation that can be evaluated with greater precision in a neighborhood of a local minimum than the usual sufficient decrease criterion Numerical comparisons are given with both L-BFGS and conjugate gradient methods using the unconstrained optimization problems in the CUTE library

A New Conjugate Gradient Method with Guaranteed Descent and an Efficient Line Search

This paper reviews the development of dierent versions of nonlinear conjugate gradient methods, with special attention given to global convergence properties.

/pdf/a-survey-of-nonlinear-conjugate-gradient-methods-115o6h95fm.pdf

A survey of nonlinear conjugate gradient methods

CHOLMOD is a set of routines for factorizing sparse symmetric positive definite matrices of the form A or AAT, updating/downdating a sparse Cholesky factorization, solving linear systems, updating/downdating the solution to the triangular system Lx = b, and many other sparse matrix functions for both symmetric and unsymmetric matrices. Its supernodal Cholesky factorization relies on LAPACK and the Level-3 BLAS, and obtains a substantial fraction of the peak performance of the BLAS. Both real and complex matrices are supported. CHOLMOD is written in ANSI/ISO C, with both C and MATLABTM interfaces. It appears in MATLAB 7.2 as x = A\b when A is sparse symmetric positive definite, as well as in several other sparse matrix functions.

/pdf/algorithm-887-cholmod-supernodal-sparse-cholesky-15p7kbqjy7.pdf

Algorithm 887: CHOLMOD, Supernodal Sparse Cholesky Factorization and Update/Downdate

http://vdol.mae.ufl.edu/JournalPublications/unifiedFramework.pdf

William W. Hager

Papers

Updating the inverse of a matrix

A New Conjugate Gradient Method with Guaranteed Descent and an Efficient Line Search

A survey of nonlinear conjugate gradient methods

Algorithm 887: CHOLMOD, Supernodal Sparse Cholesky Factorization and Update/Downdate

Brief paper: A unified framework for the numerical solution of optimal control problems using pseudospectral methods