Many problems of recent interest in statistics and machine learning can be posed in the framework of convex optimization. Due to the explosion in size and complexity of modern datasets, it is increasingly important to be able to solve problems with a very large number of features or training examples. As a result, both the decentralized collection or storage of these datasets as well as accompanying distributed solution methods are either necessary or at least highly desirable. In this review, we argue that the alternating direction method of multipliers is well suited to distributed convex optimization, and in particular to large-scale problems arising in statistics, machine learning, and related areas. The method was developed in the 1970s, with roots in the 1950s, and is equivalent or closely related to many other algorithms, such as dual decomposition, the method of multipliers, Douglas–Rachford splitting, Spingarn's method of partial inverses, Dykstra's alternating projections, Bregman iterative algorithms for l1 problems, proximal methods, and others. After briefly surveying the theory and history of the algorithm, we discuss applications to a wide variety of statistical and machine learning problems of recent interest, including the lasso, sparse logistic regression, basis pursuit, covariance selection, support vector machines, and many others. We also discuss general distributed optimization, extensions to the nonconvex setting, and efficient implementation, including some details on distributed MPI and Hadoop MapReduce implementations.

/pdf/distributed-optimization-and-statistical-learning-via-the-8s9gtddw0d.pdf

Distributed Optimization and Statistical Learning Via the Alternating Direction Method of Multipliers

THE DESIGN AND ANALYSIS OF EXPERIMENTS. By Oscar Kempthorne. New York, John Wiley and Sons, Inc., 1952. 631 pp. $8.50. This book by a teacher of statistics (as well as a consultant for \"experimenters\") is a comprehensive study of the philosophical background for the statistical design of experiment. It is necessary to have some facility with algebraic notation and manipulation to be able to use the volume intelligently. The problems are presented from the theoretical point of view, without such practical examples as would be helpful for those not acquainted with mathematics. The mathematical justification for the techniques is given. As a somewhat advanced treatment of the design and analysis of experiments, this volume will be interesting and helpful for many who approach statistics theoretically as well as practically. With emphasis on the \"why,\" and with description given broadly, the author relates the subject matter to the general theory of statistics and to the general problem of experimental inference. MARGARET J. ROBERTSON

The Design and Analysis of Experiments

计量经济分析 = Econometric analysis

From the Publisher:
This book explores the meta-heuristics approach called tabu search, which is dramatically changing our ability to solve a hostof problems that stretch over the realms of resource planning,telecommunications, VLSI design, financial analysis, scheduling, spaceplanning, energy distribution, molecular engineering, logistics,pattern classification, flexible manufacturing, waste management,mineral exploration, biomedical analysis, environmental conservationand scores of other problems. The major ideas of tabu search arepresented with examples that show their relevance to multipleapplications. Numerous illustrations and diagrams are used to clarifyprinciples that deserve emphasis, and that have not always been wellunderstood or applied. The book's goal is to provide ''hands-on' knowledge and insight alike, rather than to focus exclusively eitheron computational recipes or on abstract themes. This book is designedto be useful and accessible to researchers and practitioners inmanagement science, industrial engineering, economics, and computerscience. It can appropriately be used as a textbook in a masterscourse or in a doctoral seminar. Because of its emphasis on presentingideas through illustrations and diagrams, and on identifyingassociated practical applications, it can also be used as asupplementary text in upper division undergraduate courses. 

Finally, there are many more applications of tabu search than canpossibly be covered in a single book, and new ones are emerging everyday. The book's goal is to provide a grounding in the essential ideasof tabu search that will allow readers to create successfulapplications of their own. Along with the essentialideas,understanding of advanced issues is provided, enabling researchers togo beyond today's developments and create the methods of tomorrow.

Tabu Search

https://leeds-faculty.colorado.edu/glover/TS%20-%20Future%20Paths%20for%20Integer%20Programming.pdf

Future paths for integer programming and links to artificial intelligence

UV−vis spectroscopy was utilized to measure the decomposition of aqueous HNO3 solutions above 300 °C, in some cases with added NaOH, H2O2, and/or NaNO2, to form NO2, HNO2, NO, N2O, and oxygen. Individual bands corresponding to HNO2 and NO2 were deconvoluted from the spectra up to 400 °C. With constrained nonlinear optimization, areas of these bands were used to determine extinction coefficients of NO2 and HNO2 and equilibrium constants for the reactions:  HNO3 ⇌ H+ + NO3-; 2HNO3 ⇌ H2O + 2NO2 + (1/2)O2; 2NO2 + H2O ⇌ HNO3 + HNO2; 2HNO2 ⇌ H2O + 2NO + (1/2)O2; 2NO ⇌ N2O + (1/2)O2; (Na+)(NO3-) ⇌ Na+ + NO3-.

Nitric/Nitrous Acid Equilibria in Supercritical Water

The algorithm described here, called OptQuest/NLP or OQNLP, is a heuristic designed to find global optima for pure and mixed integer nonlinear problems with many constraints and variables, where all problem functions are differentiable with respect to the continuous variables. It uses OptQuest, a commercial implementation of scatter search developed by OptTek Systems, Inc., to provide starting points for a gradient-based local NLP solver. This solver seeks a local solution from a subset of these points, holding discrete variables fixed. The procedure is motivated by our desire to combine the superior accuracy and feasibility-seeking behavior of gradient-based local NLP solvers with the global optimization abilities of OptQuest. Computational results include 144 smooth NLP and MINLP problems due to Floudas et al, most with both linear and nonlinear constraints, coded in the GAMS modeling language. Some are quite large for global optimization, with over 100 variables and many constraints. Global solutions to almost all problems are found in a small number of NLP solver calls, often one or two.

A Multistart Scatter Search Heuristic for Smooth NLP and MINLP Problems

Most large nonlinear optimization models are composed of “pieces”—subsets of decision variables—and constraints whose union is the entire model. Each piece represents an additional aspect of the situation being modeled. This opens the possibility of solving the simplest piece first, adding the constraints and variables of another piece, and solving this submodel from a starting point provided by the first solution. This process is repeated until the original model is solved. This “piece-by-piece” approach provides each submodel with a good starting point, which greatly increases the probability that a good nonlinear solver will find an optimal solution. We apply it to a large multiperiod nonlinear programming (NLP) model with 13,700 variables, 10,000 equations, and a high degree of nonlinearity (54.3% of the nonzero Jacobian elements are nonconstant), arising from water resources planning and operation in a river basin. Using the GAMS modeling language and the CONOPT2 NLP solver, the piece-by-piece method...

Piece-by-piece approach to solving large nonlinear water resources management models

: An algorithm for solving min cost or max flow multicommodity flow problems is described. It is a specialization of the simplex method, which takes advantage of the special structure of the multicommodity problem. The only non- graph or non-additive operations in a cycle involve the inverse of a working basis, whose dimension is the number of currently saturated arcs. Efficient relations for updating this inverse are derived.

A generalized upper bounding algorithm for multicommodity network flow problems

This paper is concerned with the modeling and resolution of the energy generation planning problem of a large hydroelectric system for a horizon up to two years, with emphasis on hydro plant generation, hydro system aggregation, and objective function modeling. The problem has been formulated as a deterministic discrete-time optimal control problem, with linear transition equations, a highly nonlinear objective, and bounds on both the state and the control variables. The solution algorithm is of the reduced gradient type, with the control variables nonbasic to provide good conditioning. Because of the linear relationship between the state and control variables, the set of feasible directions at a point is a polyhedral convex cone and the bounds both on the state and control variables can be accounted for by means of an algorithm for the orthogonal projection of the gradient on the cone. Some computational experience with this large-scale non-linear program is given, followed by a discussion of implementation.

https://www.jstor.org/stable/info/2630700

Leon S. Lasdon

Papers

Nitric/Nitrous Acid Equilibria in Supercritical Water

A Multistart Scatter Search Heuristic for Smooth NLP and MINLP Problems

Piece-by-piece approach to solving large nonlinear water resources management models

A generalized upper bounding algorithm for multicommodity network flow problems

Modeling and Resolution of the Medium Term Energy Generation Planning Problem for a Large Hydro-Electric System