Evolution of Benchmark: Black-Box Optimization Benchmark Design through Large Language Model

Benchmark Design in Black-Box Optimization (BBO) is a fundamental yet open-ended topic. Early BBO benchmarks are predominantly human-crafted, introducing expert bias and constraining diversity. Automating this design process can relieve the human-in-the-loop burden while enhancing diversity and objectivity. We propose Evolution of Benchmark (EoB), an automated BBO benchmark designer empowered by the large language model (LLM) and its program evolution capability. Specifically, we formulate benchmark design as a bi-objective optimization problem towards maximizing (i) landscape similarity to target tasks and (ii) algorithm-differentiation ability across a portfolio of BBO solvers. Under this paradigm, EoB iteratively prompts LLM to evolve a population of benchmark programs and employs a reflection-based scheme to co-evolve the landscape and its corresponding program. Comprehensive experiments validate our EoB is a competitive candidate in multi-dimensional usages: 1) Benchmarking BBO algorithms; 2) Training and testing learning-assisted BBO algorithms; 3) Extending proxy for expensive real-world problems.

Optimization methods are widely used in science and engineering, and researchers need good test problems to compare them fairly. However, many of these benchmarks are still designed by hand, which is slow, labor-intensive, and can limit diversity. In this paper, we study whether large language models can help automate this process. We introduce Evolution of Benchmark (EoB), a system that uses a language model to repeatedly generate and improve small programs that define optimization tasks. The goal is to create benchmark problems that both resemble real-world tasks and highlight the differences between optimization methods. Our experiments show that this approach can produce useful benchmarks for several purposes: evaluating optimization algorithms, supporting the training of learning-based optimizers, and providing lower-cost stand-ins for expensive real-world problems.