This paper is devoted to bilevel optimization, a branch of mathematical programming of both practical and theoretical interest. Starting with a simple example, we proceed towards a general formulation. We then present fields of application, focus on solution approaches, and make the connection with MPECs (Mathematical Programs with Equilibrium Constraints).

/pdf/an-overview-of-bilevel-optimization-4x7xo28qzy.pdf

An overview of bilevel optimization

This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient descent. Extensive experiments on CIFAR-10, ImageNet, Penn Treebank and WikiText-2 show that our algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques. Our implementation has been made publicly available to facilitate further research on efficient architecture search algorithms.

DARTS: Differentiable Architecture Search

In this paper, we propose a novel coupled dictionary training method for single-image super-resolution (SR) based on patchwise sparse recovery, where the learned couple dictionaries relate the low- and high-resolution (HR) image patch spaces via sparse representation. The learning process enforces that the sparse representation of a low-resolution (LR) image patch in terms of the LR dictionary can well reconstruct its underlying HR image patch with the dictionary in the high-resolution image patch space. We model the learning problem as a bilevel optimization problem, where the optimization includes an l1-norm minimization problem in its constraints. Implicit differentiation is employed to calculate the desired gradient for stochastic gradient descent. We demonstrate that our coupled dictionary learning method can outperform the existing joint dictionary training method both quantitatively and qualitatively. Furthermore, for real applications, we speed up the algorithm approximately 10 times by learning a neural network model for fast sparse inference and selectively processing only those visually salient regions. Extensive experimental comparisons with state-of-the-art SR algorithms validate the effectiveness of our proposed approach.

Coupled Dictionary Training for Image Super-Resolution

This report proposes 15 large-scale benchmark problems as an extension to the existing CEC’2010 large-scale global optimization benchmark suite. The aim is to better represent a wider range of realworld large-scale optimization problems and provide convenience and flexibility for comparing various evolutionary algorithms specifically designed for large-s cale global optimization. Introducing imbalance between the contribution of various subcomponents, subcomponents with nonuniform sizes, and conforming and conflicting overlapping functions are among the major new features proposed in this report.

https://web.xidian.edu.cn/xlwang/files/20150312_175341.pdf

Benchmark Functions for the CEC'2013 Special Session and Competition on Large-Scale Global Optimization

Bilevel optimization is defined as a mathematical program, where an optimization problem contains another optimization problem as a constraint. These problems have received significant attention from the mathematical programming community. Only limited work exists on bilevel problems using evolutionary computation techniques; however, recently there has been an increasing interest due to the proliferation of practical applications and the potential of evolutionary algorithms in tackling these problems. This paper provides a comprehensive review on bilevel optimization from the basic principles to solution strategies; both classical and evolutionary. A number of potential application problems are also discussed. To offer the readers insights on the prominent developments in the field of bilevel optimization, we have performed an automated text-analysis of an extended list of papers published on bilevel optimization to date. This paper should motivate evolutionary computation researchers to pay more attention to this practical yet challenging area.

A Review on Bilevel Optimization: From Classical to Evolutionary Approaches and Applications

This paper provides an introductory survey of a class of optimization problems known as bilevel programming. We motivate this class through a simple application, and then proceed with the general formulation of bilevel programs. We consider various cases (linear, linear-quadratic, nonlinear), describe their main properties and give an overview of solution approaches.

Bilevel programming: A survey

We consider the approximation of nonlinear bilevel mathematical programs by solvable programs of the same type, i.e., bilevel programs involving linear approximations of the upper-level objective and all constraint-defining functions, as well as a quadratic approximation of the lower-level objective. We describe the main features of the algorithm and the resulting software. Numerical experiments tend to confirm the promising behavior of the method.

/pdf/a-trust-region-method-for-nonlinear-bilevel-programming-hx7pxy1gvh.pdf

A Trust-Region Method for Nonlinear Bilevel Programming: Algorithm and Computational Experience

We present an algorithm for solving nonlinear programming problems involving a partially separable objective function whose derivatives are assumed to be unavailable. At each iteration, we construct a quadratic interpolation model of the objective function around the current iterate and minimize this model to obtain a trial step. The whole process is embedded within a trust-region framework. We further propose to use ideas of Curtis, Powell and Reid to minimize the number of calls to the objective function in the part of the derivative-free algorithm that improves the geometry of the interpolation set. Numerical experiments tend to confirm the promising behaviour of the algorithm.

/pdf/optimizing-partially-separable-functions-without-derivatives-c7k4bxug3e.pdf

Optimizing partially separable functions without derivatives

Advanced nonlinear analyses developed for estimating structural responses for recent applications for the aerospace industry lead to expensive computational times. However optimization procedures are necessary to quickly provide optimal designs. Several possible optimization methods are available in the literature, based on either local or global approximations, which may or may not include sensitivities (gradient computations), and which may or may not be able to resort to parallelism facilities. In this paper Sequential Convex Programming (SCP), Derivative Free Optimization techniques (DFO), Surrogate Based Optimization (SBO) and Genetic Algorithm (GA) approaches are compared in the design of stiffened aircraft panels with respect to local and global instabilities (buckling and collapse). The computations are carried out with software developed for the European aeronautical industry. The specificities of each optimization method, the results obtained, computational time considerations and their adequacy to the studied problems are discussed.

/pdf/optimization-methods-for-advanced-design-of-aircraft-panels-3ds8p2nb48.pdf

Benoît Colson

Papers

An overview of bilevel optimization

Bilevel programming: A survey

A Trust-Region Method for Nonlinear Bilevel Programming: Algorithm and Computational Experience

Optimizing partially separable functions without derivatives

Optimization methods for advanced design of aircraft panels: a comparison