Showing papers in "Optimization Methods & Software in 2017"
TL;DR: In this paper, the authors present a framework for distributed optimization that both allows the flexibility of arbitrary solvers to be used on each single machine locally and yet maintains competitive performance against other state-of-the-art distributed methods.
Abstract: With the growth of data and necessity for distributed optimization methods, solvers that work well on a single machine must be re-designed to leverage distributed computation. Recent work in this area has been limited by focusing heavily on developing highly specific methods for the distributed environment. These special-purpose methods are often unable to fully leverage the competitive performance of their well-tuned and customized single machine counterparts. Further, they are unable to easily integrate improvements that continue to be made to single machine methods. To this end, we present a framework for distributed optimization that both allows the flexibility of arbitrary solvers to be used on each single machine locally and yet maintains competitive performance against other state-of-the-art special-purpose distributed methods. We give strong primal–dual convergence rate guarantees for our framework that hold for arbitrary local solvers. We demonstrate the impact of local solver selection both theoretically and in an extensive experimental comparison. Finally, we provide thorough implementation details for our framework, highlighting areas for practical performance gains.
164 citations
TL;DR: The proposed algorithm is a sequential quadratic optimization method that employs Broyden-Fletcher-Goldfarb-Shanno quasi-Newton Hessian approximations and an exact penalty function whose parameter is controlled using a steering strategy.
Abstract: We propose an algorithm for solving nonsmooth, nonconvex, constrained optimization problems as well as a new set of visualization tools for comparing the performance of optimization algorithms. Our algorithm is a sequential quadratic optimization method that employs Broyden-Fletcher-Goldfarb-Shanno BFGS quasi-Newton Hessian approximations and an exact penalty function whose parameter is controlled using a steering strategy. While our method has no convergence guarantees, we have found it to perform very well in practice on challenging test problems in controller design involving both locally Lipschitz and non-locally-Lipschitz objective and constraint functions with constraints that are typically active at local minimizers. In order to empirically validate and compare our method with available alternatives—on a new test set of 200 problems of varying sizes—we employ new visualization tools which we call relative minimization profiles. Such profiles are designed to simultaneously assess the relative performance of several algorithms with respect to objective quality, feasibility, and speed of progress, highlighting the trade-offs between these measures when comparing algorithm performance.
106 citations
TL;DR: In this article, a stochastic gradient method called semi-stochastic coordinate descent was proposed for minimizing a strongly convex function represented as the average of a large number of smooth convolutions.
Abstract: We propose a novel stochastic gradient method—semi-stochastic coordinate descent—for the problem of minimizing a strongly convex function represented as the average of a large number of smooth conv...
57 citations
TL;DR: An accelerated HPE-type method based on general Bregman distances for solving convex–concave saddle-point (SP) problems that is superior to Nesterov's smoothing scheme and works for any constant choice of proximal stepsize.
Abstract: This paper describes an accelerated HPE-type method based on general Bregman distances for solving convex–concave saddle-point (SP) problems. The algorithm is a special instance of a non-Euclidean ...
50 citations
TL;DR: A new modified proximal point algorithm combined with Halpern's iteration process for nonexpansive mappings in the framework of CAT(0) spaces is proposed and a strong convergence theorem under some mild conditions is established.
Abstract: We propose a new modified proximal point algorithm combined with Halpern's iteration process for nonexpansive mappings in the framework of CAT0 spaces. We establish a strong convergence theorem under some mild conditions. Our results extend some known results which appeared in the literature.
46 citations
TL;DR: In this paper, the authors proposed a method for the algorithmic solution of structured smooth convex optimization problems, which is a natural choice for many machine learning and data fitting applications.
Abstract: Many recent applications in machine learning and data fitting call for the algorithmic solution of structured smooth convex optimization problems. Although the gradient descent method is a natural ...
42 citations
TL;DR: A Newton-like method applied to a perturbation of the optimality system that follows from a reformulation of the initial problem by introducing an augmented Lagrangian function is presented.
Abstract: We present a primal–dual augmented Lagrangian method to solve an equality constrained minimization problem. This is a Newton-like method applied to a perturbation of the optimality system that follows from a reformulation of the initial problem by introducing an augmented Lagrangian function. An important aspect of this approach is that, by a choice of suitable updating rules of parameters, the algorithm reduces to a regularized Newton method applied to a sequence of optimality systems. The global convergence is proved under mild assumptions. An asymptotic analysis is also presented and quadratic convergence is proved under standard regularity assumptions. Some numerical results show that the method is very efficient and robust.
42 citations
TL;DR: This paper proposes a distributed algorithm for solving coupled problems with chordal sparsity or an inherent tree structure which relies on primal–dual interior-point methods by distributing the computations at each iteration, using message-passing.
Abstract: In this paper, we propose a distributed algorithm for solving coupled problems with chordal sparsity or an inherent tree structure which relies on primal–dual interior-point methods. We achieve this by distributing the computations at each iteration, using message-passing. In comparison to existing distributed algorithms for solving such problems, this algorithm requires far fewer iterations to converge to a solution with high accuracy. Furthermore, it is possible to compute an upper-bound for the number of required iterations which, unlike existing methods, only depends on the coupling structure in the problem. We illustrate the performance of our proposed method using a set of numerical examples.
38 citations
TL;DR: In this article, a general approach to building non-asymptotic confidence bounds for stochastic optimization problems is presented, which is based on the observation that a sample average approximation of a problem supplies upper and lower bounds for the optimal value of the problem which are essentially better than the quality of the corresponding optimal solutions.
Abstract: We discuss a general approach to building non-asymptotic confidence bounds for Stochastic Optimization problems. Our principal contribution is the observation that a Sample Average Approximation of a problem supplies upper and lower bounds for the optimal value of the problem which are essentially better than the quality of the corresponding optimal solutions. At the same time, such bounds are more reliable than ‘standard’ confidence bounds obtained through the asymptotic approach. We also discuss bounding the optimal value of MinMax Stochastic Optimization and stochastically constrained problems. We conclude with a simulation study illustrating the numerical behaviour of the proposed bounds.
37 citations
TL;DR: Numerical results show that CONORBIT performs better than COBYLA (Powell 1994), a sequential penalty derivative-free method, an augmented Lagrangian method, a direct search method, and another RBF-based algorithm on the test problems and on the automotive application.
Abstract: This paper presents CONORBIT CONstrained Optimization by Radial Basis function Interpolation in Trust regions, a derivative-free algorithm for constrained black-box optimization where the objective and constraint functions are computationally expensive. CONORBIT employs a trust-region framework that uses interpolating radial basis function RBF models for the objective and constraint functions, and is an extension of the ORBIT algorithm [S.M. Wild, R.G. Regis and C.A. Shoemaker, ORBIT: optimization by radial basis function interpolation in trust-regions, SIAM J. Sci. Comput. 30 2008, pp. 3197–3219]. It uses a small margin for the RBF constraint models to facilitate the generation of feasible iterates, and extensive numerical tests confirm that such a margin is helpful in improving performance. CONORBIT is compared with other algorithms on 27 test problems, a chemical process optimization problem, and an automotive application. Numerical results show that CONORBIT performs better than COBYLA Powell 1994, a sequential penalty derivative-free method, an augmented Lagrangian method, a direct search method, and another RBF-based algorithm on the test problems and on the automotive application.
37 citations
TL;DR: Experimental results indicate that the proposed approach can significantly improve the production efficiency (i.e. increased production load balance, minimized reconstruction cost, and minimized delayed workload).
Abstract: We propose a multi-objective optimization scheduling model to improve the production efficiency of a reconfigurable assembly line. We aim to minimize the costs of assembly line reconstruction, achieve the production load equalization, and minimize the delayed workload using this model. However, the proposed multi-objective optimization model is significantly complex for conventional mathematical optimization methods. Thus, we present an efficient solution approach based on a distance sorting particle swarm optimization. Finally, a case study is conducted to illustrate the feasibility and efficiency of the proposed method. Experimental results indicate that our proposed approach can significantly improve the production efficiency i.e. increased production load balance, minimized reconstruction cost, and minimized delayed workload.
TL;DR: A new approach for generating spectral parameters is presented, where a new double-truncating technique, which can ensure both the sufficient descentproperty of the search directions and the bounded property of the sequence of spectral parameters, is introduced.
Abstract: The spectral conjugate gradient methods, with simple construction and nice numerical performance, are a kind of effective methods for solving large-scale unconstrained optimization problems. In this paper, based on quasi-Newton direction and quasi-Newton condition, and motivated by the idea of spectral conjugate gradient method as well as Dai-Kou's selecting technique for conjugate parameter [SIAM J. Optim. 23 2013, pp. 296–320], a new approach for generating spectral parameters is presented, where a new double-truncating technique, which can ensure both the sufficient descent property of the search directions and the bounded property of the sequence of spectral parameters, is introduced. Then a new associated spectral conjugate gradient method for large-scale unconstrained optimization is proposed. Under either the strong Wolfe line search or the generalized Wolfe line search, the proposed method is always globally convergent. Finally, a large number of comparison numerical experiments on large-scale instances from one thousand to two million variables are reported. The numerical results show that the proposed method is more promising.
TL;DR: This paper considers the problem of identifying the most influential (or central) group of nodes (of some predefined size) in a network that has the largest value of betweenness centrality or one of its variants, for example, the length-scaled or the bounded-distance betweennessCentrality concepts.
Abstract: In this paper we consider the problem of identifying the most influential or centralgroup of nodes of some predefined size in a network. Such a group has the largest value of betweenness centrality or one of its variants, for example, the length-scaled or the bounded-distance betweenness centralities. We demonstrate that this problem can be modelled as a mixed integer program MIP that can be solved for reasonably sized network instances using off-the-shelf MIP solvers. We also discuss interesting relations between the group betweenness and the bounded-distance betweenness centrality concepts. In particular, we exploit these relations in an algorithmic scheme to identify approximate solutions for the original problem of identifying the most central group of nodes. Furthermore, we generalize our approach for identification of not only the most central groups of nodes, but also central groups of graph elements that consists of either nodes or edges exclusively, or their combination according to some pre-specified criteria. If necessary, additional cohesiveness properties can also be enforced, for example, the targeted group should form a clique or a κ-club. Finally, we conduct extensive computational experiments with different types of real-life and synthetic network instances to show the effectiveness and flexibility of the proposed framework. Even more importantly, our experiments reveal some interesting insights into the properties of influential groups of graph elements modelled using the maximum betweenness centrality concept or one of its variations.
TL;DR: A new modified version of the CG formula that was introduced by Polak, Ribière, and Polyak is proposed for problems that are bounded below and have a Lipschitz-continuous gradient, and numerical results demonstrate the efficiency of the proposed CG parameter compared with the other CG parameters.
Abstract: The conjugate gradient (CG) method is one of the most popular methods for solving large-scale unconstrained optimization problems. In this paper, a new modified version of the CG formula that was introduced by Polak, Ribiere, and Polyak is proposed for problems that are bounded below and have a Lipschitz-continuous gradient. The new parameter provides global convergence properties when the strong Wolfe-Powell (SWP) line search or the weak Wolfe-Powell (WWP) line search is employed. A proof of a sufficient descent condition is provided for the SWP line search. Numerical comparisons between the proposed parameter and other recent CG modifications are made on a set of standard unconstrained optimization problems. The numerical results demonstrate the efficiency of the proposed CG parameter compared with the other CG parameters.
TL;DR: The p-regularized subproblem (p-RS) is the key content of a regularization technique in computing a Newton-like step for unconstrained optimization as mentioned in this paper.
Abstract: The p-regularized subproblem (p-RS) is the key content of a regularization technique in computing a Newton-like step for unconstrained optimization. The idea is to incorporate a local quadratic approximation of the objective function with a weighted regularization term and then globally minimize it at each iteration. In this paper, we establish a complete theory of the p-RSs for general p>2 that covers previous known results on p=3 or p=4. The theory features necessary and sufficient optimality conditions for the global and also for the local non-global minimizers of (p-RS). It gives a closed-form expression for the global minimum set of (p-RS) and shows that (p-RS), p>2 can have at most one local non-global minimizer. Our theory indicates that (p-RS) have all properties that the trust region subproblems do. In application, (p-RS) can appear in natural formulation for optimization problems. We found two examples. One is to utilize the Tikhonov regularization to stabilize the least square solution for an o...
TL;DR: The computational results with practical problems demonstrate that the proposed penalty proximal alternating linearized minimization method can find the suboptimal solutions of the problems efficiently and is competitive with some other local solution methods.
Abstract: In this paper, we propose a penalty proximal alternating linearized minimization method for the large-scale sparse portfolio problems in which a sequence of penalty subproblems are solved by utilizing the proximal alternating linearized minimization framework and sparse projection techniques. For exploiting the structure of the problems and reducing the computation complexity, each penalty subproblem is solved by alternately solving two projection problems. The global convergence of the method to a Karush-Kuhn-Tucker point or a local minimizer of the problem can be proved under the characteristic of the problem. The computational results with practical problems demonstrate that our method can find the suboptimal solutions of the problems efficiently and is competitive with some other local solution methods.
TL;DR: It is demonstrated on numerical examples that the regularized Laplacian method is robust with respect to the choice of the regularization parameter and outperforms the LaplACian-based heat kernel methods.
Abstract: We study a semi-supervised learning method based on the similarity graph and regularized Laplacian. We give convenient optimization formulation of the regularized Laplacian method and establish its various properties. In particular, we show that the kernel of the method can be interpreted in terms of discrete and continuous-time random walks and possesses several important properties of proximity measures. Both optimization and linear algebra methods can be used for efficient computation of the classification functions. We demonstrate on numerical examples that the regularized Laplacian method is robust with respect to the choice of the regularization parameter and outperforms the Laplacian-based heat kernel methods.
TL;DR: The worst-case behaviour of a general class of regularization algorithms is considered in the case where only objective function values and associated gradient vectors are evaluated, and complexity bounds vary according to the regularization power and the assumed Hölder exponent.
Abstract: The worst-case behaviour of a general class of regularization algorithms is considered in the case where only objective function values and associated gradient vectors are evaluated. Upper bounds are derived on the number of such evaluations that are needed for the algorithm to produce an approximate first-order critical point whose accuracy is within a user-defined threshold. The analysis covers the entire range of meaningful powers in the regularization term as well as in the Holder exponent for the gradient. The resulting complexity bounds vary according to the regularization power and the assumed Holder exponent, recovering known results when available.
TL;DR: New weak conditions that ensure the validity of necessary second-order optimality conditions (SOC) for nonlinear optimization are presented and it is proved that weak and strong SOCs hold for all Lagrange multipliers using Abadie-type assumptions.
Abstract: In this work we present new weak conditions that ensure the validity of necessary second-order optimality conditions SOC for nonlinear optimization. We are able to prove that weak and strong SOCs hold for all Lagrange multipliers using Abadie-type assumptions. We also prove weak and strong SOCs for at least one Lagrange multiplier imposing the Mangasarian–Fromovitz constraint qualification and a weak constant rank assumption.
TL;DR: Two different approaches based on eigenvalues and singular values of the matrix representing the search direction in conjugate gradient algorithms are considered and it is proved that both algorithms are significantly more efficient and more robust than CG-DESCENT algorithm.
Abstract: Two different approaches based on eigenvalues and singular values of the matrix representing the search direction in conjugate gradient algorithms are considered. Using a special approximation of the inverse Hessian of the objective function, which depends by a positive parameter, we get the search direction which satisfies both the sufficient descent condition and the Dai–Liao’s conjugacy condition. In the first approach the parameter in the search direction is determined by clustering the eigenvalues of the matrix defining it. The second approach uses the minimizing the condition number of the matrix representing the search direction. In this case the obtained conjugate gradient algorithm is exactly the three-term conjugate gradient algorithm proposed by Zhang, Zhou and Li. The global convergence of the algorithms is proved for uniformly convex functions. Intensive numerical experiments, using 800 unconstrained optimization test problems, prove that both these approaches have similar numerical performan...
TL;DR: Multi-objective optimization is a very active research area as mentioned in this paper, and the actuality of this subject stems from real-life applications as well as from its high theoretical importance.
Abstract: During several decades, multi-objective optimization is a very active research area. The actuality of this subject stems from real-life applications as well as from its high theoretical importance....
TL;DR: In this article, the alternating-current optimal power flow (ACOPF) problem is formulated as a non-convex nonlinear optimization problem, and a first-order method based on the parallel coordinate descent with a novel closed-form step based on roots of cubic polynomials is proposed.
Abstract: The alternating-current optimal power flow (ACOPF) is one of the best known non-convex nonlinear optimization problems. We present a novel re-formulation of ACOPF, which is based on lifting the rectangular power-voltage rank-constrained formulation, and makes it possible to derive alternative semidefinite programming relaxations. For those, we develop a first-order method based on the parallel coordinate descent with a novel closed-form step based on roots of cubic polynomials.
TL;DR: A hybrid genetic algorithm (HGA) is created and shown to be efficient in solving the three-dimensional container packing problem (3DCPP) and a tabu search algorithm is developed based on the existing UTP literature.
Abstract: The university timetabling problem UTP has been studied by numerous research groups for decades. In addition to addressing hard and soft constraints, we extend the UTP by considering consecutiveness and periodicity constraints of multi-session lectures, which are common in many eastern Asian universities. Because schedulers can decide the consecutiveness and periodicity constraints for the multi-session lectures within a limited ratio, we consider these novel decision variables in our model. We develop a mixed integer linear program for the UTP. For the analysis, we convert the UTP into the three-dimensional container packing problem 3DCPP and create a hybrid genetic algorithm HGA, which has been shown to be efficient in solving the 3DCPP. We also develop a tabu search algorithm based on the existing UTP literature and compare the findings with that of our HGA. The results show that our HGA obtains a better solution than the tabu search algorithm in a reasonable amount of time.
TL;DR: It is shown that under certain assumptions the resulting MPCC fulfills CQs for MPCCs being the basis for theory on MPCC optimality conditions and consequently for numerical solution techniques.
Abstract: In this paper an inverse optimal control problem in the form of a mathematical program with complementarity constraints MPCC is considered and numerical experiences are discussed. The inverse optimal control problem arises in the context of human navigation where the body is modelled as a dynamical system and it is assumed that the motions are optimally controlled with respect to an unknown cost function. The goal of the inversion is now to find a cost function within a given parametrized family of candidate cost functions such that the corresponding optimal motion minimizes the deviation from given data. MPCCs are known to be a challenging class of optimization problems typically violating all standard constraint qualifications CQs. We show that under certain assumptions the resulting MPCC fulfills CQs for MPCCs being the basis for theory on MPCC optimality conditions and consequently for numerical solution techniques. Finally, numerical results are presented for the discretized inverse optimal control problem of locomotion using different solution techniques based on relaxation and lifting.
TL;DR: A composite step method, designed for equality constrained optimization with partial differential equations, based on cubic regularization of the objective and an affine covariant damped Newton method for feasibility is proposed.
Abstract: We propose a composite step method, designed for equality constrained optimization with partial differential equations. Focus is laid on the construction of a globalization scheme, which is based on cubic regularization of the objective and an affine covariant damped Newton method for feasibility. We show finite termination of the inner loop and fast local convergence of the algorithm. Numerical results are shown for optimal control problems subject to a nonlinear heat equation.
TL;DR: This paper proposes a deterministic discrete approximation scheme due to Pflug and Pichler and demonstrates that the discrete probability measure and the ambiguity set of discrete probability measures approximate their continuous counterparts under the Kantorovich metric.
Abstract: Since the pioneering work by Dentcheva and Ruszczynski [Optimization with stochastic dominance constraints, SIAM J. Optim. 14 2003, pp. 548–566], stochastic programs with second-order dominance constraints SPSODC have received extensive discussions over the past decade from theory of optimality to numerical schemes and practical applications. In this paper, we investigate discrete approximation of SPSODC when a the true probability is known but continuously distributed and b the true probability distribution is unknown but it lies within an ambiguity set of distributions. Differing from the well-known Monte Carlo discretization method, we propose a deterministic discrete approximation scheme due to Pflug and Pichler [Approximations for Probability Distributions and Stochastic Optimization Problems, International Series in Operations Research & Management Science, Vol. 163, Springer, New York, 2011, pp. 343–387] and demonstrate that the discrete probability measure and the ambiguity set of discrete probability measures approximate their continuous counterparts under the Kantorovich metric. Stability analysis of the optimal value and optimal solutions of the resulting discrete optimization problems is presented and some comparative numerical test results are reported.
TL;DR: This paper proposes a proximal partially parallel splitting method for solving convex minimization problems, where the objective function is separable into m individual operators without any coupled variables, and the structural constraint set comprises only linear functions.
Abstract: In this paper, we propose a proximal partially parallel splitting method for solving convex minimization problems, where the objective function is separable into m individual operators without any coupled variables, and the structural constraint set comprises only linear functions. At each iteration of this algorithm, one selected subproblem is solved, and subsequently the remaining subproblems are solved in parallel, utilizing the new iterate information. Hence, the proposed method is a hybrid mechanism that combines the nice features of parallel decomposition methods and alternating direction methods, while simultaneously adopting the predictor–corrector strategy to ensure convergence of the algorithm. Our algorithmic framework is also amenable to admitting linearized versions of the subproblems, which frequently have closed-form solutions, thereby making the proposed method more implementable in practice. Furthermore, the worst-case convergence rate of the proposed method is obtained under both ergodic...
TL;DR: This work reexamines the work of Aupy et al. on optimal algorithms for hierarchical adjoint computations, and provides an optimal algorithm in constant time and space, with appropriate pre-processing, for the online problem, when the adjoint chain size is not known before-hand.
Abstract: We reexamine the work of Aupy et al. on optimal algorithms for hierarchical adjoint computations, where two levels of memories are available. The previous optimal algorithm had a quadratic execution time. Here, with structural arguments, namely periodicity, on the optimal solution, we provide an optimal algorithm in constant time and space, with appropriate pre-processing. We also provide an asymptotically optimal algorithm for the online problem, when the adjoint chain size is not known before-hand. Again, these algorithms rely on the proof that the optimal solution for hierarchical adjoint computations is weakly periodic. We conjecture a closed-form formula for the period. Finally, we assess the convergence speed of the approximation ratio for the online problem through simulations.
TL;DR: Through the solution of a set of benchmark problems from the literature, it is shown that the new global optimization algorithm can potentially lead to orders of magnitude reduction in optimality gap when compared to commercial solvers BARON and GloMIQO.
Abstract: Spatial branch-and-bound (B&B) is widely used for the global optimization of non-convex problems. It basically works by iteratively reducing the domain of the variables so that tighter relaxations can be achieved that ultimately converge to the global optimal solution. Recent developments for bilinear problems have brought us piecewise relaxation techniques that can prove optimality for a sufficiently large number of partitions and hence avoid spatial B&B altogether. Of these, normalized multiparametric disaggregation (NMDT) exhibits a good performance due to the logarithmic increase in the number of binary variables with the number of partitions. We now propose to integrate NMDT with spatial B&B for solving mixed-integer quadratically constrained minimization problems. Optimality-based bound tightening is also part of the algorithm so as to compute tight lower bounds in every step of the search and reduce the number of nodes to explore. Through the solution of a set of benchmark problems from the literat...
TL;DR: A numerical procedure based on the appropriately defined adjoint equations formulated for the discretized equations being the result of the numerical integration of systems equations by an implicit Runge–Kutta method to solve hybrid optimal control problems described by higher index differential–algebraic equations (DAEs).
Abstract: The paper deals with hybrid optimal control problems described by higher index differential–algebraic equations DAEs. We introduce a numerical procedure for solving these problems. The procedure has the following features: it is based on the appropriately defined adjoint equations formulated for the discretized equations being the result of the numerical integration of systems equations by an implicit Runge–Kutta method; the consistent initialization procedure is applied whenever control functions jumps, or state variables transition occurs. The procedure can cope with hybrid optimal control problems which are defined by DAEs with the index not exceeding three. Our approach does not require differentiation of some system equations in order to transform higher index DAEs to the underlying ordinary differential equations ODEs. The presented numerical examples show that the proposed approach can be used to solve efficiently hybrid optimal control problems with higher index DAEs.