Many problems of recent interest in statistics and machine learning can be posed in the framework of convex optimization. Due to the explosion in size and complexity of modern datasets, it is increasingly important to be able to solve problems with a very large number of features or training examples. As a result, both the decentralized collection or storage of these datasets as well as accompanying distributed solution methods are either necessary or at least highly desirable. In this review, we argue that the alternating direction method of multipliers is well suited to distributed convex optimization, and in particular to large-scale problems arising in statistics, machine learning, and related areas. The method was developed in the 1970s, with roots in the 1950s, and is equivalent or closely related to many other algorithms, such as dual decomposition, the method of multipliers, Douglas–Rachford splitting, Spingarn's method of partial inverses, Dykstra's alternating projections, Bregman iterative algorithms for l1 problems, proximal methods, and others. After briefly surveying the theory and history of the algorithm, we discuss applications to a wide variety of statistical and machine learning problems of recent interest, including the lasso, sparse logistic regression, basis pursuit, covariance selection, support vector machines, and many others. We also discuss general distributed optimization, extensions to the nonconvex setting, and efficient implementation, including some details on distributed MPI and Hadoop MapReduce implementations.

/pdf/distributed-optimization-and-statistical-learning-via-the-8s9gtddw0d.pdf

Distributed Optimization and Statistical Learning Via the Alternating Direction Method of Multipliers

THE DESIGN AND ANALYSIS OF EXPERIMENTS. By Oscar Kempthorne. New York, John Wiley and Sons, Inc., 1952. 631 pp. $8.50. This book by a teacher of statistics (as well as a consultant for \"experimenters\") is a comprehensive study of the philosophical background for the statistical design of experiment. It is necessary to have some facility with algebraic notation and manipulation to be able to use the volume intelligently. The problems are presented from the theoretical point of view, without such practical examples as would be helpful for those not acquainted with mathematics. The mathematical justification for the techniques is given. As a somewhat advanced treatment of the design and analysis of experiments, this volume will be interesting and helpful for many who approach statistics theoretically as well as practically. With emphasis on the \"why,\" and with description given broadly, the author relates the subject matter to the general theory of statistics and to the general problem of experimental inference. MARGARET J. ROBERTSON

The Design and Analysis of Experiments

计量经济分析 = Econometric analysis

From the Publisher:
This book explores the meta-heuristics approach called tabu search, which is dramatically changing our ability to solve a hostof problems that stretch over the realms of resource planning,telecommunications, VLSI design, financial analysis, scheduling, spaceplanning, energy distribution, molecular engineering, logistics,pattern classification, flexible manufacturing, waste management,mineral exploration, biomedical analysis, environmental conservationand scores of other problems. The major ideas of tabu search arepresented with examples that show their relevance to multipleapplications. Numerous illustrations and diagrams are used to clarifyprinciples that deserve emphasis, and that have not always been wellunderstood or applied. The book's goal is to provide ''hands-on' knowledge and insight alike, rather than to focus exclusively eitheron computational recipes or on abstract themes. This book is designedto be useful and accessible to researchers and practitioners inmanagement science, industrial engineering, economics, and computerscience. It can appropriately be used as a textbook in a masterscourse or in a doctoral seminar. Because of its emphasis on presentingideas through illustrations and diagrams, and on identifyingassociated practical applications, it can also be used as asupplementary text in upper division undergraduate courses. 

Finally, there are many more applications of tabu search than canpossibly be covered in a single book, and new ones are emerging everyday. The book's goal is to provide a grounding in the essential ideasof tabu search that will allow readers to create successfulapplications of their own. Along with the essentialideas,understanding of advanced issues is provided, enabling researchers togo beyond today's developments and create the methods of tomorrow.

Tabu Search

https://leeds-faculty.colorado.edu/glover/TS%20-%20Future%20Paths%20for%20Integer%20Programming.pdf

Future paths for integer programming and links to artificial intelligence

Beyond portfolio optimization

A generalized approach to the design of practical linear arrays, using recent mathematical optimization techniques, is outlined. The interelement spacing, element amplitudes, and phases are all permitted to vary within specified bounds. The technique can accommodate realistic radiators, since measured directivity patterns can be used in the design. An example is presented that shows that Dolph‐Chebychev shading does not provide optimal results or point‐source arrays.

Nonlinear Programming Applied to Linear‐Array Design

Motivated by the successful use of a pseudo-cut strategy within the setting of constrained nonlinear and nonconvex optimization in Lasdon et al. (2010), we propose a framework for general pseudo-cut strategies in global optimization that provides a broader and more comprehensive range of methods. The fundamental idea is to introduce linear cutting planes that provide temporary, possibly invalid, restrictions on the space of feasible solutions, as proposed in the setting of the tabu search metaheuristic in Glover (1989), in order to guide a solution process toward a global optimum, where the cutting planes can be discarded and replaced by others as the process continues. These strategies can be used separately or in combination, and can also be used to supplement other approaches to nonlinear global optimization. Our strategies also provide mechanisms for generating trial solutions that can be used with or without the temporary enforcement of the pseudo-cuts.

/pdf/pseudo-cut-strategies-for-global-optimization-269j6nj7jk.pdf

Pseudo-Cut Strategies for Global Optimization

Integration of planning, scheduling, and process control.

Constrained minimization is often done via interior penalty functions. Such functions can be very difficult to minimize using existing algorithms. In this paper, a new algorithm is described which is specially constructed to deal with such functions. It generates search directions by linearizing the objective and constraints about the current (interior) point, substituting these linearizations into the penalty function, and minimizing the result. Properties of the algorithm are derived, an efficient method for solving the direction finding problem is suggested, and computational results are presented. Preliminary results are also given on an extension to quasibarrier and exterior penalty functions.

Leon S. Lasdon

Papers

Beyond portfolio optimization

Nonlinear Programming Applied to Linear‐Array Design

Pseudo-Cut Strategies for Global Optimization

Integration of planning, scheduling, and process control.

An efficient algorithm for minimizing barrier and penalty functions