Can I read On the Assessment of Benchmark Suites for Algorithm Comparison on EtoBox?

On the Assessment of Benchmark Suites for Algorithm Comparison by Mattos, David Issa; Ruud, Lucas; Bosch, Jan; Olsson, Helena Holmström is a scholarly article available to read on EtoBox.

What is On the Assessment of Benchmark Suites for Algorithm Comparison about?

Benchmark suites, i.e. a collection of benchmark functions, are widely used in the comparison of black-box optimization algorithms. Over the years, research has identified many desired qualities for benchmark suites, such as diverse topology, different difficulties, scalability, representativeness of real-world problems among others. However, while the topology characteristics have been subjected to previous studies, there is no study that has statistically evaluated the difficulty level of benchmark functions, how well they discriminate optimization algorithms and how suitable is a benchmark suite for algorithm comparison. In this paper, we propose the use of an item response theory (IRT) model, the Bayesian two-parameter logistic model for multiple attempts, to statistically evaluate these aspects with respect to the empirical success rate of algorithms. With this model, we can assess the difficulty level of each benchmark, how well they discriminate different algorithms, the ability score of an algorithm, and how much information the benchmark suite adds in the estimation of the ability scores. We demonstrate the use of this model in two well-known benchmark suites, the Black-Bo

Author

Mattos, David Issa; Ruud, Lucas; Bosch, Jan; Olsson, Helena Holmström

Published

2021

Language

EN