Best practices for comparing optimization algorithms
Reads0
Chats0
TLDR
This paper systematically review the benchmarking process of optimization algorithms, and provides suggestions for each step of the comparison process and highlights the pitfalls to avoid when evaluating the performance of optimized algorithms.Abstract:
Comparing, or benchmarking, of optimization algorithms is a complicated task that involves many subtle considerations to yield a fair and unbiased evaluation. In this paper, we systematically review the benchmarking process of optimization algorithms, and discuss the challenges of fair comparison. We provide suggestions for each step of the comparison process and highlight the pitfalls to avoid when evaluating the performance of optimization algorithms. We also discuss various methods of reporting the benchmarking results. Finally, some suggestions for future research are presented to improve the current benchmarking process.read more
Citations
More filters
Journal ArticleDOI
COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting
TL;DR: COCO as discussed by the authors is an open source platform for comparing continuous optimizers in a black-box setting, which aims at automatizing the tedious and repetitive task of benchmarking numerical optimization algorithms to the greatest possible extent.
Journal ArticleDOI
Performance assessment of the metaheuristic optimization algorithms: an exhaustive review
TL;DR: It is of utmost importance to use a correct tool for measuring the performance of the diverse set of metaheuristic algorithms to derive an appropriate judgment on the superiority of the algorithms and also to validate the claims raised by researchers for their specific objectives.
Journal ArticleDOI
Improving the Flexibility and Robustness of Model-based Derivative-free Optimization Solvers
TL;DR: Numerical experiments show that Py-BOBYQA is comparable to or better than existing general DFO solvers for noisy problems, and introduces an adaptive accuracy measure for data profiles of noisy functions, striking a balance between measuring the true and the noisy objective improvement.
Journal ArticleDOI
Building energy optimization: An extensive benchmark of global search algorithms
Christoph Waibel,Christoph Waibel,Thomas Wortmann,Ralph Evins,Ralph Evins,Jan Carmeliet,Jan Carmeliet +6 more
TL;DR: This study investigates the performance of a wide selection of single objective black-box optimization algorithms (optimizers) when applied to a large set of building energy simulation problems from the literature and develops a new metric based on ranks, which helps users to select an appropriate optimizer for BEO problems and the available evaluation budget.
Posted Content
Benchmarking in Optimization: Best Practice and Open Issues
Thomas Bartz-Beielstein,Carola Doerr,Jakob Bossek,Sowmya Chandrasekaran,Tome Eftimov,Andreas Fischbach,Pascal Kerschke,Manuel López-Ibáñez,Katherine M. Malan,Jason H. Moore,Boris Naujoks,Patryk Orzechowski,Vanessa Volz,Markus Wagner,Thomas Weise +14 more
TL;DR: The article discusses eight essential topics in benchmarking: clearly stated goals, well-specified problems, suitable algorithms, adequate performance measures, thoughtful analysis, effective and efficient designs, comprehensible presentations, and guaranteed reproducibility.
References
More filters
Book
Numerical Optimization
Jorge Nocedal,Stephen J. Wright +1 more
TL;DR: Numerical Optimization presents a comprehensive and up-to-date description of the most effective methods in continuous optimization, responding to the growing interest in optimization in engineering, science, and business by focusing on the methods that are best suited to practical problems.
Book
The Visual Display of Quantitative Information
TL;DR: The visual display of quantitative information is shown in the form of icons and symbols in order to facilitate the interpretation of data.
Journal ArticleDOI
Benchmarking optimization software with performance profiles
Elizabeth D. Dolan,Jorge J. Moré +1 more
TL;DR: It is shown that performance profiles combine the best features of other tools for performance evaluation to create a single tool for benchmarking and comparing optimization software.