Big Data applications are typically associated with systems involving large numbers of users, massive complex software systems, and large-scale heterogeneous computing and storage architectures. The construction of such systems involves many distributed design choices. The end products (e.g., recommendation systems, medical analysis tools, real-time game engines, speech recognizers) thus involve many tunable configuration parameters. These parameters are often specified and hard-coded into the software by various developers or teams. If optimized jointly, these parameters can result in significant improvements. Bayesian optimization is a powerful tool for the joint optimization of design choices that is gaining great popularity in recent years. It promises greater automation so as to increase both product quality and human productivity. This review paper introduces Bayesian optimization, highlights some of its methodological aspects, and showcases a wide range of applications.

/pdf/taking-the-human-out-of-the-loop-a-review-of-bayesian-4olcznoudw.pdf

Taking the Human Out of the Loop: A Review of Bayesian Optimization

The use of metamodeling techniques in the design and analysis of computer experiments has progressed remarkably in the past 25 years, but how far has the field really come? This is the question addressed in this paper, namely, the extent to which the use of metamodeling techniques in multidisciplinary design optimization have evolved in the 25 years since the seminal paper on design and analysis of computer experiments by Sacks et al. (“Design and Analysis of Computer Experiments,” Statistical Science, Vol. 4, No. 4, 1989, pp. 409–435). Rather than a technical review of the entire body of metamodeling literature, the focus is on the evolution and motivation for advancements in metamodeling with some discussion on the research itself; not surprisingly, much of the current research motivation is the same as it was in the past. Based on current research thrusts in the field, multifidelity approximations and ensembles (i.e., sets) of metamodels, as well as the availability of metamodels within commercial soft...

Metamodeling in Multidisciplinary Design Optimization: How Far Have We Really Come?

Stepwise uncertainty reduction (SUR) strategies aim at constructing a sequence of points for evaluating a function  f in such a way that the residual uncertainty about a quantity of interest progressively decreases to zero. Using such strategies in the framework of Gaussian process modeling has been shown to be efficient for estimating the volume of excursion of f above a fixed threshold. However, SUR strategies remain cumbersome to use in practice because of their high computational complexity, and the fact that they deliver a single point at each iteration. In this article we introduce several multipoint sampling criteria, allowing the selection of batches of points at which f can be evaluated in parallel. Such criteria are of particular interest when f is costly to evaluate and several CPUs are simultaneously available. We also manage to drastically reduce the computational cost of these strategies through the use of closed form formulas. We illustrate their performances in various numerical experiment...

/pdf/fast-parallel-kriging-based-stepwise-uncertainty-reduction-21zx1dua6x.pdf

Fast Parallel Kriging-Based Stepwise Uncertainty Reduction With Application to the Identification of an Excursion Set

This article addresses the problem of derivative-free (single- or multi-objective) optimization subject to multiple inequality constraints. Both the objective and constraint functions are assumed to be smooth, non-linear and expensive to evaluate. As a consequence, the number of evaluations that can be used to carry out the optimization is very limited, as in complex industrial design optimization problems. The method we propose to overcome this difficulty has its roots in both the Bayesian and the multi-objective optimization literatures. More specifically, an extended domination rule is used to handle objectives and constraints in a unified way, and a corresponding expected hyper-volume improvement sampling criterion is proposed. This new criterion is naturally adapted to the search of a feasible point when none is available, and reduces to existing Bayesian sampling criteria--the classical Expected Improvement (EI) criterion and some of its constrained/multi-objective extensions--as soon as at least one feasible point is available. The calculation and optimization of the criterion are performed using Sequential Monte Carlo techniques. In particular, an algorithm similar to the subset simulation method, which is well known in the field of structural reliability, is used to estimate the criterion. The method, which we call BMOO (for Bayesian Multi-Objective Optimization), is compared to state-of-the-art algorithms for single- and multi-objective constrained optimization.

https://hal.archives-ouvertes.fr/hal-01207679/document

A Bayesian approach to constrained single- and multi-objective optimization

The expected improvement (EI) algorithm is a very popular method for expensive optimization problems. In the past twenty years, the EI criterion has been extended to deal with a wide range of expensive optimization problems. This paper gives a comprehensive review of the EI extensions designed for parallel optimization, multiobjective optimization, constrained optimization, noisy optimization, multi-fidelity optimization and high-dimensional optimization. The main challenges of extending the EI approach to solve these complex optimization problems are pointed out, and the ideas proposed in literature to tackle these challenges are highlighted. For each reviewed algorithm, the surrogate modeling method, the computation of the infill criterion and the internal optimization of the infill criterion are carefully studied and compared. In addition, the monotonicity properties of the multiobjective EI criteria and constrained EI criteria are analyzed in detail. Through this review, we give an organized summary about the EI developments in the past twenty years and show a clear picture about how the EI approach has advanced. In the end of this paper, several interesting problems and future research topics about the EI developments are given.

Expected improvement for expensive optimization: a review

We consider the problem of optimizing a real-valued continuous function f, which is supposed to be expensive to evaluate and, consequently, can only be evaluated a limited number of times. This article focuses on the Bayesian approach to this problem, which consists in combining evaluation results and prior information about f in order to efficiently select new evaluation points, as long as the budget for evaluations is not exhausted.

The algorithm called efficient global optimization (EGO), proposed by Jones, Schonlau and Welch (J. Global Optim., 13(4):455–492, 1998), is one of the most popular Bayesian optimization algorithms. It is based on a sampling criterion called the expected improvement (EI), which assumes a Gaussian process prior about f. In the EGO algorithm, the parameters of the covariance of the Gaussian process are estimated from the evaluation results by maximum likelihood, and these parameters are then plugged in the EI sampling criterion. However, it is well-known that this plug-in strategy can lead to very disappointing results when the evaluation results do not carry enough information about f to estimate the parameters in a satisfactory manner.

We advocate a fully Bayesian approach to this problem, and derive an analytical expression for the EI criterion in the case of Student predictive distributions. Numerical experiments show that the fully Bayesian approach makes EI-based optimization more robust while maintaining an average loss similar to that of the EGO algorithm.

/pdf/robust-gaussian-process-based-global-optimization-using-a-3lct7n9889.pdf

Robust gaussian process-based global optimization using a fully bayesian expected improvement criterion

We consider the problem of optimizing a real-valued continuous function f using a Bayesian approach, where the evaluations of f are chosen sequentially by combining prior information about f, which is described by a random process model, and past evaluation results. The main difficulty with this approach is to be able to compute the posterior distributions of quantities of interest which are used to choose evaluation points. In this article, we decide to use a Sequential Monte Carlo (SMC) approach.

https://hal-supelec.archives-ouvertes.fr/hal-00717195/document

Bayesian optimization using sequential monte carlo

Ce travail de these s'interesse au probleme de l'optimisation globale d'une fonction couteuse dans un cadre bayesien. Nous disons qu'une fonction est couteuse lorsque son evaluation necessite l’utilisation de ressources importantes (simulations numeriques tres longues, notamment). Dans ce contexte, il est important d'utiliser des algorithmes d'optimisation utilisant un faible nombre d'evaluations de cette derniere. Nous considerons ici une approche bayesienne consistant a affecter a la fonction a optimiser un a priori sous la forme d'un processus aleatoire gaussien, ce qui permet ensuite de choisir les points d'evaluation de la fonction en maximisant un critere probabiliste indiquant, conditionnellement aux evaluations precedentes, les zones les plus interessantes du domaine de recherche de l'optimum. Deux difficultes dans le cadre de cette approche peuvent etre identifiees : le choix de la valeur des parametres du processus gaussien et la maximisation efficace du critere. La premiere difficulte est generalement resolue en substituant aux parametres l'estimateur du maximum de vraisemblance, ce qui est une methode peu robuste a laquelle nous preferons une approche dite completement bayesienne. La contribution de cette these est de presenter un nouvel algorithme d'optimisation bayesien, maximisant a chaque etape le critere dit de l'esperance de l'amelioration, et apportant une reponse conjointe aux deux difficultes enoncees a l'aide d'une approche Sequential Monte Carlo. Des resultats numeriques, obtenus a partir de cas tests et d'applications industrielles, montrent que les performances de notre algorithme sont bonnes par rapport a celles d’algorithmes concurrents.

Nouvel algorithme d'optimisation bayésien utilisant une approche Monte-Carlo séquentielle.

We consider the problem of optimizing a real-valued continuous function $f$ using a Bayesian approach, where the evaluations of $f$ are chosen sequentially by combining prior information about $f$, which is described by a random process model, and past evaluation results. The main difficulty with this approach is to be able to compute the posterior distributions of quantities of interest which are used to choose evaluation points. In this article, we decide to use a Sequential Monte Carlo (SMC) approach.

Bayesian optimization using sequential Monte Carlo

The conception of analog and mixed-signal functions requires great effort because the complex analog parts should be recursively optimized based not only on system-level requirements but also on technological limitations and imperfections High-level behavioral models used for chip-level simulations can be employed using multi-domain hardware description languages (HDL), but they are usually manually written and lack technological characteristics Moreover, automatic resizing and optimization at the transistor level are very limited, and the behavioral models cannot be re-adjusted to changes at the transistor level In this paper, we present an efficient design methodology implying the automatic optimization of cells at the transistor level using a modified Bayesian Kriging approach and the extraction of robust analog macro-models, which can be directly regenerated during the optimization process Coherent results were obtained when using the proposed methodology for the conception of a sixth-order continuous-time (CT) Sigma-Delta (ΣΔ) modulator

/pdf/efficient-optimization-methodology-for-ct-functions-based-on-4rl6yvu01x.pdf

Romain Benassi

Papers

Robust gaussian process-based global optimization using a fully bayesian expected improvement criterion

Bayesian optimization using sequential monte carlo

Nouvel algorithme d'optimisation bayésien utilisant une approche Monte-Carlo séquentielle.

Bayesian optimization using sequential Monte Carlo

Efficient optimization methodology for CT functions based on a modified bayesian kriging approach