Many problems of recent interest in statistics and machine learning can be posed in the framework of convex optimization. Due to the explosion in size and complexity of modern datasets, it is increasingly important to be able to solve problems with a very large number of features or training examples. As a result, both the decentralized collection or storage of these datasets as well as accompanying distributed solution methods are either necessary or at least highly desirable. In this review, we argue that the alternating direction method of multipliers is well suited to distributed convex optimization, and in particular to large-scale problems arising in statistics, machine learning, and related areas. The method was developed in the 1970s, with roots in the 1950s, and is equivalent or closely related to many other algorithms, such as dual decomposition, the method of multipliers, Douglas–Rachford splitting, Spingarn's method of partial inverses, Dykstra's alternating projections, Bregman iterative algorithms for l1 problems, proximal methods, and others. After briefly surveying the theory and history of the algorithm, we discuss applications to a wide variety of statistical and machine learning problems of recent interest, including the lasso, sparse logistic regression, basis pursuit, covariance selection, support vector machines, and many others. We also discuss general distributed optimization, extensions to the nonconvex setting, and efficient implementation, including some details on distributed MPI and Hadoop MapReduce implementations.

/pdf/distributed-optimization-and-statistical-learning-via-the-8s9gtddw0d.pdf

Distributed Optimization and Statistical Learning Via the Alternating Direction Method of Multipliers

THE DESIGN AND ANALYSIS OF EXPERIMENTS. By Oscar Kempthorne. New York, John Wiley and Sons, Inc., 1952. 631 pp. $8.50. This book by a teacher of statistics (as well as a consultant for \"experimenters\") is a comprehensive study of the philosophical background for the statistical design of experiment. It is necessary to have some facility with algebraic notation and manipulation to be able to use the volume intelligently. The problems are presented from the theoretical point of view, without such practical examples as would be helpful for those not acquainted with mathematics. The mathematical justification for the techniques is given. As a somewhat advanced treatment of the design and analysis of experiments, this volume will be interesting and helpful for many who approach statistics theoretically as well as practically. With emphasis on the \"why,\" and with description given broadly, the author relates the subject matter to the general theory of statistics and to the general problem of experimental inference. MARGARET J. ROBERTSON

The Design and Analysis of Experiments

计量经济分析 = Econometric analysis

From the Publisher:
This book explores the meta-heuristics approach called tabu search, which is dramatically changing our ability to solve a hostof problems that stretch over the realms of resource planning,telecommunications, VLSI design, financial analysis, scheduling, spaceplanning, energy distribution, molecular engineering, logistics,pattern classification, flexible manufacturing, waste management,mineral exploration, biomedical analysis, environmental conservationand scores of other problems. The major ideas of tabu search arepresented with examples that show their relevance to multipleapplications. Numerous illustrations and diagrams are used to clarifyprinciples that deserve emphasis, and that have not always been wellunderstood or applied. The book's goal is to provide ''hands-on' knowledge and insight alike, rather than to focus exclusively eitheron computational recipes or on abstract themes. This book is designedto be useful and accessible to researchers and practitioners inmanagement science, industrial engineering, economics, and computerscience. It can appropriately be used as a textbook in a masterscourse or in a doctoral seminar. Because of its emphasis on presentingideas through illustrations and diagrams, and on identifyingassociated practical applications, it can also be used as asupplementary text in upper division undergraduate courses. 

Finally, there are many more applications of tabu search than canpossibly be covered in a single book, and new ones are emerging everyday. The book's goal is to provide a grounding in the essential ideasof tabu search that will allow readers to create successfulapplications of their own. Along with the essentialideas,understanding of advanced issues is provided, enabling researchers togo beyond today's developments and create the methods of tomorrow.

Tabu Search

https://leeds-faculty.colorado.edu/glover/TS%20-%20Future%20Paths%20for%20Integer%20Programming.pdf

Future paths for integer programming and links to artificial intelligence

http://www.gbv.de/dms/ilmenau/toc/016265343.PDF

Optimization Theory for Large Systems

: The purpose of this paper is to describe a Generalized Reduced Gradient (GRG) algorithm for nonlinear programming, its implementation as a FORTRAN program for solving small to medium size problems, and some computational results. Our focus is more on the software implementation of the algorithm than on its mathematical properties. This is in line with the premise that robust, efficient, easy to use NLP software must be written and made accessible if nonlinear programming is to progress, both in theory and in practice.

Design and Testing of a Generalized Reduced Gradient Code for Nonlinear Programming

I Problem Formulation 1 The Nature and Organization of Optimization Problems 2 Developing Models for Optimization 3 Formulation of the Objective Function II Optimization Theory and Methods 4 Basic Concepts of Optimization 5 Optimization for Unconstrained Functions: One- Dimensional Search 6 Unconstrained Multivariable Optimization 7 Linear Programming and Applications 8 Nonlinear Programming with Constraints 9 Mixed-Integer Programming 10 Global Optimization for Problems Containing Continuous and Discrete Variables IIIApplications of Optimization 11 Heat Transfer and Energy Conservation 12 Separation Processes 13 Fluid Flow Systems 14 Chemical Reactor Design and Operation 15 Optimization in Large-Scale Plant Design and Operations 16 Integrated Planning, Scheduling, and Control in the Process Industries Appendixes

Optimization of Chemical Processes

The algorithm described here, called OptQuest/NLP or OQNLP, is a heuristic designed to find global optima for pure and mixed integer nonlinear problems with many constraints and variables, where all problem functions are differentiable with respect to the continuous variables. It uses OptQuest, a commercial implementation of scatter search developed by OptTek Systems, Inc., to provide starting points for any gradient-based local solver for nonlinear programming (NLP) problems. This solver seeks a local solution from a subset of these points, holding discrete variables fixed. The procedure is motivated by our desire to combine the superior accuracy and feasibility-seeking behavior of gradient-based local NLP solvers with the global optimization abilities of OptQuest. Computational results include 155 smooth NLP and mixed integer nonlinear program (MINLP) problems due to Floudas et al. (1999), most with both linear and nonlinear constraints, coded in the GAMS modeling language. Some are quite large for global optimization, with over 100 variables and 100 constraints. Global solutions to almost all problems are found in a small number of local solver calls, often one or two.

/pdf/scatter-search-and-local-nlp-solvers-a-multistart-framework-4witwchb5x.pdf

Scatter Search and Local NLP Solvers: A Multistart Framework for Global Optimization

In designing the spreadsheet optimizer that is bundled with Microsoft Excel, we and Microsoft made certain choices in designing its user interface, model processing, and solution algorithms for linear, nonlinear, and integer programs. We describe some of the common pitfalls users encounter and remedies available in the latest version of Microsoft Excel. The Solver has many applications and great impact in industry and education.

Leon S. Lasdon

Papers

Optimization Theory for Large Systems

Design and Testing of a Generalized Reduced Gradient Code for Nonlinear Programming

Optimization of Chemical Processes

Scatter Search and Local NLP Solvers: A Multistart Framework for Global Optimization

Design and Use of the Microsoft Excel Solver