Many problems of recent interest in statistics and machine learning can be posed in the framework of convex optimization. Due to the explosion in size and complexity of modern datasets, it is increasingly important to be able to solve problems with a very large number of features or training examples. As a result, both the decentralized collection or storage of these datasets as well as accompanying distributed solution methods are either necessary or at least highly desirable. In this review, we argue that the alternating direction method of multipliers is well suited to distributed convex optimization, and in particular to large-scale problems arising in statistics, machine learning, and related areas. The method was developed in the 1970s, with roots in the 1950s, and is equivalent or closely related to many other algorithms, such as dual decomposition, the method of multipliers, Douglas–Rachford splitting, Spingarn's method of partial inverses, Dykstra's alternating projections, Bregman iterative algorithms for l1 problems, proximal methods, and others. After briefly surveying the theory and history of the algorithm, we discuss applications to a wide variety of statistical and machine learning problems of recent interest, including the lasso, sparse logistic regression, basis pursuit, covariance selection, support vector machines, and many others. We also discuss general distributed optimization, extensions to the nonconvex setting, and efficient implementation, including some details on distributed MPI and Hadoop MapReduce implementations.

/pdf/distributed-optimization-and-statistical-learning-via-the-8s9gtddw0d.pdf

Distributed Optimization and Statistical Learning Via the Alternating Direction Method of Multipliers

THE DESIGN AND ANALYSIS OF EXPERIMENTS. By Oscar Kempthorne. New York, John Wiley and Sons, Inc., 1952. 631 pp. $8.50. This book by a teacher of statistics (as well as a consultant for \"experimenters\") is a comprehensive study of the philosophical background for the statistical design of experiment. It is necessary to have some facility with algebraic notation and manipulation to be able to use the volume intelligently. The problems are presented from the theoretical point of view, without such practical examples as would be helpful for those not acquainted with mathematics. The mathematical justification for the techniques is given. As a somewhat advanced treatment of the design and analysis of experiments, this volume will be interesting and helpful for many who approach statistics theoretically as well as practically. With emphasis on the \"why,\" and with description given broadly, the author relates the subject matter to the general theory of statistics and to the general problem of experimental inference. MARGARET J. ROBERTSON

The Design and Analysis of Experiments

计量经济分析 = Econometric analysis

From the Publisher:
This book explores the meta-heuristics approach called tabu search, which is dramatically changing our ability to solve a hostof problems that stretch over the realms of resource planning,telecommunications, VLSI design, financial analysis, scheduling, spaceplanning, energy distribution, molecular engineering, logistics,pattern classification, flexible manufacturing, waste management,mineral exploration, biomedical analysis, environmental conservationand scores of other problems. The major ideas of tabu search arepresented with examples that show their relevance to multipleapplications. Numerous illustrations and diagrams are used to clarifyprinciples that deserve emphasis, and that have not always been wellunderstood or applied. The book's goal is to provide ''hands-on' knowledge and insight alike, rather than to focus exclusively eitheron computational recipes or on abstract themes. This book is designedto be useful and accessible to researchers and practitioners inmanagement science, industrial engineering, economics, and computerscience. It can appropriately be used as a textbook in a masterscourse or in a doctoral seminar. Because of its emphasis on presentingideas through illustrations and diagrams, and on identifyingassociated practical applications, it can also be used as asupplementary text in upper division undergraduate courses. 

Finally, there are many more applications of tabu search than canpossibly be covered in a single book, and new ones are emerging everyday. The book's goal is to provide a grounding in the essential ideasof tabu search that will allow readers to create successfulapplications of their own. Along with the essentialideas,understanding of advanced issues is provided, enabling researchers togo beyond today's developments and create the methods of tomorrow.

Tabu Search

https://leeds-faculty.colorado.edu/glover/TS%20-%20Future%20Paths%20for%20Integer%20Programming.pdf

Future paths for integer programming and links to artificial intelligence

On detection of linear and nonlinear optimization problems

The algorithm described here, called OptQuest/NLP or OQNLP, is a heuristic designed to find global optima for pure and mixed integer nonlinear problems with many constraints and variables, where all problem functions are differentiable with respect to the continuous variables. It uses OptQuest, a commercial implementation of Scatter Search developed by OptTek Systems, Inc., to provide starting points for any gradient-based local NLP solver. This solver seeks a local solution from a subset of these points, holding discrete variables fixed. The procedure is motivated by our desire to combine the superior accuracy and feasibility-seeking behavior of gradient-based local NLP solvers with the global optimization abilities of OptQuest. Computational results include 155 smooth NLP and MINLP problems due to Floudas et al., most with both linear and nonlinear constraints, coded in the GAMS modeling language. Some are quite large for global optimization, with over 100 variables and 100 constraints. Global solutions to almost all problems are found in a small number of local solver calls, often one or two.

Scatter Search and Local NLP Solvers: A Multistart Framework for Global Optimization

Development and testing of a model for determining optimal pumping and recharge of large-scale aquifers.

This research project was established by the Texas Department of Transportation's (TxDOT’s) Research and Technology Implementation Office to address special studies required by the department’s Administration during FY 2009 Five short-term, quick turnaround tasks were completed and are documented These tasks included the following: Task 1: Relationships between Vehicle Operating Costs and Ride Quality; Task 2: Nationwide DOT Per Unit Production Cost Analysis and Comparison; Task 3: Optimization of Emergency Response Among TxDOT Maintenance Sections; Task 4: The Needs and Funding Options for Texas Mega-Bridge Replacement Projects; and Task 5: Tracking the US Fiscal Stimulus Investments in Texas Transportation Projects Supervised by TxDOT and Developing New Economic Impact Models for Project Selection

https://rosap.ntl.bts.gov/view/dot/17943/dot_17943_DS1.pdf

Special Studies for TxDOT Administration in FY 2009

In an earlier paper, the authors formulated an acoustic antenna array design problem as a nonlinear program. Computation times for large problems (about 400 sensors) were in the range of 8 to 10 hours on a Vax 11/780. Most of this time was spent in computing the objective function and its gradient. This paper describes how these computations (and these only) were recoded to exploit the vector processing capabilities of a Cray 1-M computer. Run times are reduced to less than one minute. The results have implications for many nonlinear programs whose function evaluations are very time consuming.

Leon S. Lasdon

Papers

On detection of linear and nonlinear optimization problems

Scatter Search and Local NLP Solvers: A Multistart Framework for Global Optimization

Development and testing of a model for determining optimal pumping and recharge of large-scale aquifers.

Special Studies for TxDOT Administration in FY 2009

Solving a large nonlinear programming problem on a vector processing computer