Many problems of recent interest in statistics and machine learning can be posed in the framework of convex optimization. Due to the explosion in size and complexity of modern datasets, it is increasingly important to be able to solve problems with a very large number of features or training examples. As a result, both the decentralized collection or storage of these datasets as well as accompanying distributed solution methods are either necessary or at least highly desirable. In this review, we argue that the alternating direction method of multipliers is well suited to distributed convex optimization, and in particular to large-scale problems arising in statistics, machine learning, and related areas. The method was developed in the 1970s, with roots in the 1950s, and is equivalent or closely related to many other algorithms, such as dual decomposition, the method of multipliers, Douglas–Rachford splitting, Spingarn's method of partial inverses, Dykstra's alternating projections, Bregman iterative algorithms for l1 problems, proximal methods, and others. After briefly surveying the theory and history of the algorithm, we discuss applications to a wide variety of statistical and machine learning problems of recent interest, including the lasso, sparse logistic regression, basis pursuit, covariance selection, support vector machines, and many others. We also discuss general distributed optimization, extensions to the nonconvex setting, and efficient implementation, including some details on distributed MPI and Hadoop MapReduce implementations.

/pdf/distributed-optimization-and-statistical-learning-via-the-8s9gtddw0d.pdf

Distributed Optimization and Statistical Learning Via the Alternating Direction Method of Multipliers

THE DESIGN AND ANALYSIS OF EXPERIMENTS. By Oscar Kempthorne. New York, John Wiley and Sons, Inc., 1952. 631 pp. $8.50. This book by a teacher of statistics (as well as a consultant for \"experimenters\") is a comprehensive study of the philosophical background for the statistical design of experiment. It is necessary to have some facility with algebraic notation and manipulation to be able to use the volume intelligently. The problems are presented from the theoretical point of view, without such practical examples as would be helpful for those not acquainted with mathematics. The mathematical justification for the techniques is given. As a somewhat advanced treatment of the design and analysis of experiments, this volume will be interesting and helpful for many who approach statistics theoretically as well as practically. With emphasis on the \"why,\" and with description given broadly, the author relates the subject matter to the general theory of statistics and to the general problem of experimental inference. MARGARET J. ROBERTSON

The Design and Analysis of Experiments

计量经济分析 = Econometric analysis

From the Publisher:
This book explores the meta-heuristics approach called tabu search, which is dramatically changing our ability to solve a hostof problems that stretch over the realms of resource planning,telecommunications, VLSI design, financial analysis, scheduling, spaceplanning, energy distribution, molecular engineering, logistics,pattern classification, flexible manufacturing, waste management,mineral exploration, biomedical analysis, environmental conservationand scores of other problems. The major ideas of tabu search arepresented with examples that show their relevance to multipleapplications. Numerous illustrations and diagrams are used to clarifyprinciples that deserve emphasis, and that have not always been wellunderstood or applied. The book's goal is to provide ''hands-on' knowledge and insight alike, rather than to focus exclusively eitheron computational recipes or on abstract themes. This book is designedto be useful and accessible to researchers and practitioners inmanagement science, industrial engineering, economics, and computerscience. It can appropriately be used as a textbook in a masterscourse or in a doctoral seminar. Because of its emphasis on presentingideas through illustrations and diagrams, and on identifyingassociated practical applications, it can also be used as asupplementary text in upper division undergraduate courses. 

Finally, there are many more applications of tabu search than canpossibly be covered in a single book, and new ones are emerging everyday. The book's goal is to provide a grounding in the essential ideasof tabu search that will allow readers to create successfulapplications of their own. Along with the essentialideas,understanding of advanced issues is provided, enabling researchers togo beyond today's developments and create the methods of tomorrow.

Tabu Search

https://leeds-faculty.colorado.edu/glover/TS%20-%20Future%20Paths%20for%20Integer%20Programming.pdf

Future paths for integer programming and links to artificial intelligence

A scenario-based stochastic programming model for water supplies from the highland lakes

Improvements in Capacitance-Resistive Modeling and Optimization of Large Scale Reservoirs

A bilevel program is a mathematical program involving functions defined implicitly as solutions to another mathematical program. We discuss a method for extracting derivative information on the implicit function, which is especially efficient when the lower-level problem has simple bounds on the variables and/or many inactive constraints. Computational experience on problems with up to 230 variables and 30 constraints is presented.

/pdf/derivative-evaluation-and-computational-experience-with-29bo8cqf0z.pdf

Derivative evaluation and computational experience with large bilevel mathematical programs

The problem considered is that of obtaining solutions to large nonlinear mathematical programs by coordinated solution of smaller subproblems. If all functions in the original problem are additively separable, this can be done by finding a saddle point for the associated Lagrangian function. Coordination is then accomplished by shadow prices, with these prices chosen to solve a dual program. Characteristics of the dual program are investigated, and an algorithm is proposed in which subproblems are solved for given shadow prices. These solutions provide the value and gradient of the dual function, and this information is used to update the shadow prices so that the dual problem is brought closer to solution. Application to two classes of problems is given. The first class is one whose constraints describe a system of coupled subsystems; the second is a class of multi-item inventory problems whose decision variables may be discrete.

Duality and Decomposition in Mathematical Programming

Gasoline blending is a critical refinery operation. In 1980, Texaco began developing an improved, optimization based, decision support system for planning and scheduling its blending operations. The system, OMEGA, is implemented on personal computers and on larger computer systems. It relies on refinery data bases and on-line data acquisition and exploits detailed nonlinear models of gasoline attributes. Texaco uses OMEGA in all seven US refineries and its Canadian and Welsh refineries. Its benefits include an estimated $30 million annually, better quality control, improved planning and marketing information, and the ability to conduct a variety of what-if studies.

Leon S. Lasdon

Papers

A scenario-based stochastic programming model for water supplies from the highland lakes

Improvements in Capacitance-Resistive Modeling and Optimization of Large Scale Reservoirs

Derivative evaluation and computational experience with large bilevel mathematical programs

Duality and Decomposition in Mathematical Programming

OMEGA: An Improved Gasoline Blending System for Texaco