Can machine learning deliver AI? Theoretical results, inspiration from the brain and cognition, as well as machine learning experiments suggest that in order to learn the kind of complicated functions that can represent high-level abstractions (e.g. in vision, language, and other AI-level tasks), one would need deep architectures. Deep architectures are composed of multiple levels of non-linear operations, such as in neural nets with many hidden layers, graphical models with many levels of latent variables, or in complicated propositional formulae re-using many sub-formulae. Each level of the architecture represents features at a different level of abstraction, defined as a composition of lower-level features. Searching the parameter space of deep architectures is a difficult task, but new algorithms have been discovered and a new sub-area has emerged in the machine learning community since 2006, following these discoveries. Learning algorithms such as those for Deep Belief Networks and other related unsupervised learning algorithms have recently been proposed to train deep architectures, yielding exciting results and beating the state-of-the-art in certain areas. Learning Deep Architectures for AI discusses the motivations for and principles of learning algorithms for deep architectures. By analyzing and comparing recent results with different learning algorithms for deep architectures, explanations for their success are proposed and discussed, highlighting challenges and suggesting avenues for future explorations in this area.

/pdf/learning-deep-architectures-for-ai-2358znl654.pdf

Learning Deep Architectures for AI

Simulation and the monte carlo method

This paper addresses the solution of bound-constrained optimization problems using algorithms that require only the availability of objective function values but no derivative information. We refer to these algorithms as derivative-free algorithms. Fueled by a growing number of applications in science and engineering, the development of derivative-free optimization algorithms has long been studied, and it has found renewed interest in recent time. Along with many derivative-free algorithms, many software implementations have also appeared. The paper presents a review of derivative-free algorithms, followed by a systematic comparison of 22 related implementations using a test set of 502 problems. The test bed includes convex and nonconvex problems, smooth as well as nonsmooth problems. The algorithms were tested under the same conditions and ranked under several criteria, including their ability to find near-global solutions for nonconvex problems, improve a given starting point, and refine a near-optimal solution. A total of 112,448 problem instances were solved. We find that the ability of all these solvers to obtain good solutions diminishes with increasing problem size. For the problems used in this study, TOMLAB/MULTIMIN, TOMLAB/GLCCLUSTER, MCS and TOMLAB/LGO are better, on average, than other derivative-free solvers in terms of solution quality within 2,500 function evaluations. These global solvers outperform local solvers even for convex problems. Finally, TOMLAB/OQNLP, NEWUOA, and TOMLAB/MULTIMIN show superior performance in terms of refining a near-optimal solution.

/pdf/derivative-free-optimization-a-review-of-algorithms-and-2qmveq3j89.pdf

Derivative-free optimization: a review of algorithms and comparison of software implementations

This book is about the theoretical foundations of optimization algorithms, and also provides practical insights on how such methods should be implemented and applied. The goal of this text is not focused on in-depth details of state-of-the-art methods, but rather to provide a good foundation of the tried and trusted methods and how they can be used to solve real-world problems efficiently. The target audience for the text seems to be graduate students and engineers in industry as a good reference or self-study. However, it does not lend itself to easy adaption in a classroom setting due to the fact that only some chapters have problem sets and most chapters do not have a summary section. The book has four main parts each written by the one of the different authors. Two of the parts deal with standard unconstrained and constrained optimization techniques. The other two section deal with more complicated issues including solving problems that exhibit nonsmooth behavior and interior point methods. Solving optimization problems that have nonsmooth objectives or constraints is a formidable challenge because sensitivity information cannot be calculated using traditional approaches; the book provides a variety of ways to approach such problems. Interior point methods have been gaining popularity in the engineering optimization community for their efficiency and ability to maintain feasibility throughout the optimization process. There have been many textbooks written on the subject of numerical optimization. This book has the same flavor as many of these past books. A book with a very similar title by Nocedal and Wright [1] covered much of the same material, but it did not address nonsmooth problems. A classical book in the field by Flecther [2] considers nonsmooth problems but not interior point methods. Also, if one is more interested in comparing the performance of the various methods applied to engineering design problems, the book by Vanderplaats [3] may be more suitable. The appeal of this book is in the authors’ attention to detail and their advice on practical implementation issues. For many of the different optimization algorithms detailed flow charts are given. They also provide adequate examples to help the reader understand the methods better and explore possible pitfalls. There are, however, a few downsides of this book. The main one is that through translation of the book to English, it is evident that some of the finer points were lost and in some cases the text is hard to understand. The book could also have benefited from more figures; there are only 26. Finally, each of the dif

Numerical Optimization: Theoretical and Practical Aspects

In this paper, we first prove that the expansion and contraction steps of the Nelder-Mead simplex algorithm possess a descent property when the objective function is uniformly convex. This property provides some new insights on why the standard Nelder-Mead algorithm becomes inefficient in high dimensions. We then propose an implementation of the Nelder-Mead method in which the expansion, contraction, and shrink parameters depend on the dimension of the optimization problem. Our numerical experiments show that the new implementation outperforms the standard Nelder-Mead method for high dimensional problems.

/pdf/implementing-the-nelder-mead-simplex-algorithm-with-adaptive-47wblnu9vb.pdf

Implementing the Nelder-Mead simplex algorithm with adaptive parameters

In this paper formal definitions of exactness for penalty functions are introduced and sufficient conditions for a penalty function to be exact according to these definitions are stated, thus providing a unified framework for the study of both nondifferentiable and continuously differentiable penalty functions In this framework the best-known classes of exact penalty functions are analyzed, and new results are established concerning the correspondence between the solutions of the constrained problem and the unconstrained minimizers of the penalty functions

Exact penalty functions in constrained optimization

Fast Linear Algebra for Multiarc Trajectory Optimization (Nicolas Berend, J. Frederic Bonnans, Julien Laurent-Varin, MounirHaddou, Christophe Talbot).- Lagrange Multipliers with Optimal Sensitivity Properties in Constrained Optimization (Dimitri P. Bertsekas).- n O(n^2) Algorithm for Isotonic Regression (Oleg Burdakov, Oleg Sysoev, Anders Grimvall, Mohamed Hussian).- Knitro: An Integrated Package for Nonlinear Optimization (Richard H. Byrd, Jorge Nocedal, Richard A. Waltz).- On implicit-factorization constraint preconditioners (H. Sue Dollar, Nicholas I. M. Gould, Andrew J. Wathen).- Optimal algorithms for large sparse quadratic programming problems with uniformly bounded spectrum (Zdenek Dostal).- Numerical methods for separating two polyhedra (Yury G. Evtushenko, Alexander I. Golikov, Saed Ketabchi).- Exact penalty functions for generalized Nash problems (Francisco Facchinei, Jong-Shi Pang).- Parametric Sensitivity Analysis for Optimal Boundary Control of a 3D Reaction-Diffusion System (Roland Griesse, Stefan Volkwein).- Projected Hessians for Preconditioning in One-Step One-Shot Design Optimization (Andreas Griewank).- Conditions and parametric representations of approximate minimal elements of a set through scalarization (Cesar Gutierrez, Bienvenido Jimenez, Vicente Novo).- Efficient methods for large-scale unconstrained optimization (Ladislav Luksan, Jan Vlcek).- A variational approach for minimum cost flow problems (Giandomenico Mastroeni).- Multi-Objective Optimisation of Expensive Objective Functions with Variable Fidelity Models (Daniele Peri, Antonio Pinto, Emilio F. Campana).-Towards the Numerical Solution of a Large Scale PDAE Constrained Optimization Problem Arising in Molten Carbonate Fuel Cell Modeling (Hans Josef Pesch, Kati Sternberg, Kurt Chudej).- The NEWUOA software for unconstrained optimization without derivatives (M.J.D. Powell).

Large-Scale Nonlinear Optimization

Towards a Discrete Newton Method with Memory for Largescale Optimization R.H. Byrd, et al. On Regularity for Generalized Systems and Applications M. Castellani, et al. An Algorithm Using Quadratic Interpolation for Unconstrained Derivative Free Optimization A.R. Conn, P.L. Toint. Massively Parallel Solution of Large Scale Network Flow Problems R. De Leone, et al. Correlation Theorems for Nonsmooth Systems V.F. Dem'yanov. Successive Projection Methods for the Solution of Overdetermined Nonlinear Systems M.A. Diniz-Erhardt, J.M. Martinez. On Exact Augmented Lagrangian Functions in Nonlinear Programming G. Di Pillo, S. Lucidi. Spacetransformation Technique: The State of the Art Y.G. Evtushenko, G.Z. Vitali. Semismoothness and Superlinear Convergence in Nonsmooth Optimization and Nonsmooth Equations H. Jiang, et al. On the Solution of the Monotone and Nonmonotone Linear Complementarity Problem by an Infeasible Interior Point Algorithm J. Judice, et al. Ergodic Results in Subgradient Optimization T. Larsson, et al. Protoderivatives and the Geometry of Solution Mappings in Nonlinear Programming A.B. Levy, R.T. Rockafellar. Index.

Nonlinear optimization and applications

In this paper a new class of augmented Lagrangians is introduced, for solving equality constrained problems via unconstrained minimization techniques. It is proved that a solution of the constrained problem and the corresponding values of the Lagrange multipliers can be found by performing a single unconstrained minimization of the augmented Lagrangian. In particular, in the linear quadratic case, the solution is obtained by minimizing a quadratic function. Numerical examples are reported.

A New Class of Augmented Lagrangians in Nonlinear Programming

Exact penalty methods for the solution of constrained optimization problems are based on the construction of a function whose unconstrained minilnizing points are also solution of the constrained problem. In the first part of this paper we recall some definitions concerning exactness properties of penalty functions, of barrier functions, of augmented Lagrangian functions, and discuss under which assumptions on the constrained problem these properties can be ensured. In the second part of the paper we consider algorithmic aspects of exact penalty methods; in particular we show that, by making use of continuously differentiable functions that possess exactness properties, it is possible to define implementable algorithms that are globally convergent with superlinear convergence rate towards KKT points of the constrained problem.

G. Di Pillo

Papers

Exact penalty functions in constrained optimization

Large-Scale Nonlinear Optimization

Nonlinear optimization and applications

A New Class of Augmented Lagrangians in Nonlinear Programming

Exact Penalty Methods