EN FR
EN FR


Section: New Results

Benchmarking Methodology

Single-Objective Benchmarking

Benchmarking optimization algorithms seems trivial at first sight but is quite involved in practice and little decisions on the experimental setup can have a large effect on the displayed algorithm performance.

We have investigated some of these effects in the context of the Black-Box Optimization Benchmarking (bbob ) test suite of the COCO platform and the well-known quasi-Newton BFGS algorithm, default in MATLAB's fminunc and in Python's scipy.optimize module [5]. We realized in particular that the instance instantiation in the COCO platform has little impact while the initial search point has a larger one. The largest performance differences, however, stem from implementation details that are typically not documented and not exposed to the user via internal algorithm parameters. For example is the MATLAB implementation of the BFGS algorithm significantly worse than the Python implementation and the MATLAB 2017 version is worse than the MATLAB 2009 implementation.

Additionally, Nikolaus Hansen gave a hands-on tutorial on good benchmarking practice at the GECCO-2018 conference in Kyoto [11].

Multi-Objective Benchmarking

In terms of multiobjective benchmarking, our contributions are two-fold. Firstly, we wrote a scientific article on the scientific methodology for defining our new multiobjective benchmark suite. In [10] we introduce two new bi-objective test suites on the basis of the above mentioned, well-known 24 bbob test functions and propose a generic test suite generator for an arbitrary number of objectives. The former are implemented in our COCO platform and extensively documented in terms of search and objective space plots for each function.

Secondly, we realized with the proposal of the biobjective bbob test suites that there is a need for more theoretical analyses of simple test functions that still test for practical challenges such as ill-conditioning or search space rotations. In our upcoming EMO conference paper [3] we therefore characterize theoretically Pareto sets and Pareto fronts of combinations of two convex quadratic functions with arbitrary search space dimension. Based on this theoretical analysis, we suggest a wide set of new biobjective test functions.