Derivative-Free and Blackbox Optimization

TLDR: DFO algorithms have principally fallen into one of two categories: direct search methods and modelbased methods, and researchers began mixing direct search and model-based methods to create hybrid methods with improved performance.

Abstract: Derivative-free optimization (DFO) is the mathematical study of the optimization algorithms that do not use derivatives. While a DFO algorithm was used to test one of the worlds first computers (the MANIAC in 1952), it was not until the 1990s that DFO algorithms were studied mathematically. Blackbox optimization (BBO) is the study of optimization problems where the objective function is a blackbox. That is, no analytic description of the function is available, but given an arbitrary input the blackbox returns a function value. As BBO naturally arises whenever a computer simulation is involved in an optimization problem, BBO is one of the most rapidly expanding areas of applied optimization. BBO can naturally be approached by DFO. DFO algorithms have principally fallen into one of two categories: direct search methods and modelbased methods. Direct search methods work from an incumbent solution and examine a collection of trial points; if improvement is found, then the incumbent solution is updated, otherwise a search radius parameter is decreased and a new collection of trial points is examined. Model-based methods approximate the objective function with a model function, and use the gradients or even second derivatives of the model function to help guide optimization. (Note that while DFO studies algorithms that do not use derivatives, this does not mean that the objective function is nondifferentiable – for example the objective could be a computer simulation using numerical integration.) It was not until very recently that researchers began mixing direct search and model-based methods to create hybrid methods with improved performance.