Pith. sign in

REVIEW 3 major objections 5 minor 35 references

The Pitfalls of Benchmarking in Algorithm Selection: What We Are Getting Wrong

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Two common benchmarking habits can make algorithm-selection results look far better than they really are.

desk verdict A clean, well-scoped demonstration of two pitfalls in algorithm selection evaluation that deserves review but overclaims its generality. read the letter →

arxiv 2505.07750 v1 pith:3WQFDTWD submitted 2025-05-12 cs.LG

Gašper Petelin , Gjorgjina Cenikj This is my paper · ORCID
classification cs.LG
keywords algorithmselectionleave-instance-outevaluationleave-problem-outspuriouscorrelationsscale-sensitivemetricstargetprecisioncontinuousblack-boxoptimizationbenchmarkingpitfalls
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Algorithm selection in continuous black-box optimization is usually evaluated by training a meta-model on landscape features and checking how well it chooses an optimizer. The paper argues that two common evaluation choices can inflate those results: leave-instance-out testing on the COCO benchmark, where instances within the same problem class are similar enough that a feature encoding only the class identity predicts held-out performance; and scale-sensitive performance targets such as target precision, where a model can shrink its error by learning the problem's scale rather than which algorithm is better. Under leave-instance-out, deliberately non-informative features and a single class-identity feature both beat sensible baselines, while under leave-problem-out the advantage disappears. Under scale-sensitive targets, a meta-model using one scale-related feature beats the mean-precision baseline by an order of magnitude, yet its rank-based error is no better than the baseline. The authors conclude that such evaluation flaws can invalidate core claims in the field and should be replaced or supplemented by rank-based, scale-independent checks.

What carries the argument

The first load-bearing mechanism is the combination of a benchmark with 24 problem classes and several instances per class, a leave-instance-out (LIO) split in which instances of every class appear in both training and test sets, and Pairwise Ranking Error (PRE), the fraction of algorithm pairs whose predicted order disagrees with the true order. Because instances within a class resemble one another, class identity is a shortcut: a meta-model can lower PRE by memorizing which class an instance belongs to. The second mechanism is target precision, $f(x_{\mathrm{best}})-f(x_{\mathrm{opt}})$, which the paper defines as scale-sensitive because rescaling the objective function changes its value; the feature $f_{\mathrm{scale}}=\max(Y)-\min(Y)$ then lets a regressor reduce MSE simply by tracking scale. The paper constructs non-informative features by applying random aggregate, transformation, and scalar functions to sampled fitness values, and uses the class label as a worst-case feature.

What would settle it

Construct, or find, a benchmark whose instances within a class are deliberately made as dissimilar in algorithm ranking as instances across classes; if a class-identity meta-model still achieves near-0.05 pairwise ranking error under leave-instance-out evaluation, the paper's spurious-correlation explanation would be wrong, whereas if the advantage vanishes, the mechanism is confirmed.

Watch

Extended reading notes

Core claim

The authors claim that 'leave-instance-out' (LIO) evaluation is flawed in the COCO setting: because instances of the same problem class resemble each other in both landscape and algorithm performance, a meta-model can achieve low pairwise ranking error by learning which class an instance belongs to rather than by learning anything useful for selecting an algorithm. They support this with deliberately non-informative features built only from summary statistics of sampled function values, and with a worst-case feature that is simply the problem class; under LIO both reach PRE around 0.13 and 0.05 respectively, comparable to or better than exploratory landscape analysis features, while under leave-problem-out all meta-models perform similarly. They further claim that scale-sensitive performance metrics, specifically target precision $f(x_{\mathrm{best}})-f(x_{\mathrm{opt}})$, conflate problem scale with algorithm quality: a random-forest meta-model using only the feature $\max(Y)-\min(Y)$ beats a mean-precision baseline by an order of magnitude in MSE under leave-problem-out, but when predictions are converted to ranks, its pairwise ranking error is indistinguishable from the baseline. The conclusion is that these evaluation choices create spurious success, not genuine algorithm-selection ability.

Load-bearing premise

The demonstration that leave-instance-out evaluation rewards useless features rests on the assumption that COCO instances within the same problem class are similar enough, in landscape and in algorithm performance, that a feature encoding only the class identity can predict held-out performance; if within-class instances were as diverse as across-class instances, the class-only feature would not achieve its low error.

Editorial extensions

If this is right

  • A strong result obtained only under leave-instance-out evaluation on a benchmark whose instances cluster by problem class is not evidence of generalization; class memorization can reproduce it.
  • A meta-model that predicts target precision well, as measured by MSE or MAE, can fail entirely at the actual task of ranking algorithms, so rank-based metrics such as PRE should accompany any scale-sensitive regression target.
  • A baseline that predicts mean target precision is too weak when problems differ in scale; accounting for scale is necessary before a precision-prediction improvement means anything.
  • Feature-importance and explainability analyses on such meta-models are unreliable, because they will inflate the importance of class-correlated and scale-sensitive features.
  • Reporting leave-problem-out evaluation, or at least a permutation or class-label check, would expose the spurious success that LIO hides.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same leave-instance-out critique should apply to any performance-prediction task on benchmarks with near-duplicate instance families, such as runtime or difficulty prediction for shifted, scaled, and rotated problem instances.
  • A natural extension is to normalize objective values before computing target precision, or to use a relative gap; the paper's logic predicts that the apparent advantage of scale-aware models would then shrink or disappear.
  • A testable consequence is that a new feature set whose leave-instance-out performance lies between the mean baseline and the class-identity baseline cannot be distinguished from spurious correlation without a stronger evaluation protocol.
  • The scale-sensitivity argument suggests that meta-model results across papers are not comparable unless the scale distribution of the benchmark problems is matched or controlled.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that two common evaluation practices in algorithm selection (AS) for continuous black-box optimization are flawed. First, it claims that the leave-instance-out (LIO) evaluation on the COCO benchmark can reward non-informative features and meta-models through spurious correlations with problem class, producing over-optimistic results. This is demonstrated with three feature sets (ELA, hand-crafted non-informative statistics, and a single problem-class indicator) and five pymoo solvers on 5D COCO instances, where the class-only meta-model reaches PRE ≈ 0.05 under LIO but not under leave-problem-out (LPO). Second, it claims that scale-sensitive performance metrics, such as target precision, can mislead meta-model evaluation: a model trained with a single scale-correlated feature appears to outperform a mean baseline on MSE, but this advantage vanishes when the predictions are converted to rankings and evaluated with pairwise ranking error. The paper concludes with recommendations to avoid LIO and scale-sensitive metrics in AS evaluation.

Significance. If the claims hold, the paper makes a useful contribution by explicitly demonstrating two evaluation pitfalls that are likely widespread in the continuous black-box optimization AS community. The experiments are carefully constructed and the code is available, which supports reproducibility. The section on scale-sensitive metrics is particularly effective: the counterexample where a single scale feature yields an order-of-magnitude MSE improvement with no ranking improvement cleanly illustrates why scale-sensitive metric evaluation can be misleading. The LIO counterexample is also thought-provoking, although its strength depends on a similarity premise that is not independently established. The paper is appropriately cautious in places (e.g., noting that it cannot generally test feature informativeness), but the scope of the headline claims is broader than the supporting experiments.

major comments (3)
  1. [Section 2, paragraph before cases (a)-(d)] The label 'non-informative features' is not justified. These features are aggregates of fitness values (e.g., mean, median, standard deviation, quantiles) under random scalar and transformation choices. Such statistics can carry information about the scale, spread, or difficulty of the objective function, and may therefore be genuinely informative for algorithm performance rather than only for class identity. The paper should either rename these features as 'not intentionally designed for AS' or provide a control showing that they do not correlate with algorithm performance in a setting where class identity is removed (e.g., within-class regression or LPO with multiple classes).
  2. [Section 2.2, Figure 1] The LIO demonstration is limited to 5D problems, 15 instances per class, five default-configured pymoo solvers, and a single benchmark. The conclusion that the LIO methodology is flawed and should be avoided is stated broadly, but the experiments do not show whether the spurious-success effect persists in higher dimensions, with more instances per class, or with portfolios that have more varied behavior within a class. Since the paper recommends broad community-practice changes, this scope limitation is load-bearing rather than cosmetic. The paper should either add experiments in other settings or explicitly restrict the claim to the demonstrated configuration.
  3. [Section 3.3, recommendation bullets] The scale-sensitive metric argument is sound as a counterexample, but the final recommendation that 'metrics like PRE or classification accuracy are more suitable' may be too strong. Ranking metrics discard the magnitude of performance differences, which can be important when some algorithm pairs are nearly tied or when the cost of a wrong selection depends on the gap. The paper should temper this recommendation or discuss situations where scale-aware metrics are still needed and how they should be normalized.
minor comments (5)
  1. [Section 2.1, performance measurement] The text says each algorithm is run 30 times per instance to obtain mean rank, but the definition of PRE in Eq. (2) refers to rankings 'established through 100 runs'. Please align these numbers.
  2. [Section 2.1, feature sets] The sentence 'the second uses the so-called non-informative features. These features are designed to identify the class of a problem' is confusing: the non-informative features are not designed to identify class; the class feature is. Rephrase to avoid implying that the random-transform features were built as class indicators.
  3. [Equation (1)] The notation in the summation is garbled: it should be a sum over ordered pairs (a_i, a_j) in A^2 with a_i ≠ a_j, rather than 'A 2, a_i ≠ a_j'. Please fix the typesetting.
  4. [References] Reference [34] appears to duplicate reference [28] (same paper). Please consolidate.
  5. [Section 3.2, Figure 3] The left panel uses a logarithmic y-axis for MSE; consider stating this explicitly in the caption, and note that the Wilcoxon test is applied to 24 paired points.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's demonstrations are explicitly constructed counterexamples; the only self-citation is a stated assumption with external support.

full rationale

The paper's two central claims are established by deliberate counterexamples, not by fitting parameters and renaming them as predictions. In Section 2, the class feature is defined as the problem class itself, and the low PRE of the class meta-model follows from the explicitly stated assumption that algorithm performance is similar within COCO problem classes; this is a conditional demonstration of why LIO rewards class-correlated features, not a derivation of the conclusion from the conclusion. The assumption is flagged as "a mild assumption" and is supported by external work [16,26,28] as well as the authors' repository [22]; the self-citation is not load-bearing because the argument does not depend on accepting [22] as an independent theorem, and the paper itself warns that the methodology cannot test informativeness. In Section 3, the scale-sensitive metric result is a constructed counterexample: f_scale = max(Y)-min(Y) is intentionally chosen to track scale, and the demonstration that it improves MSE but not PRE is logically valid and does not reduce to a fitted prediction. No equation is defined in terms of a claimed result, and no uniqueness or ansatz is smuggled in via self-citation. The scope limitations (5D, 15 instances, one benchmark, five default solvers) are correctness risks rather than circularity.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The analysis relies on two domain assumptions about benchmark structure and metric invariance. It introduces no free fitted parameters and no new theoretical entities. The demonstrations are deliberately constructed feature sets rather than fitted models.

assumptions (2)
  • domain assumption Problem instances within the same COCO problem class are more similar to each other than to instances from other classes, and optimization algorithm performance is somewhat similar within a class.
    Invoked in Section 2's 'mild assumption' paragraph and in the Discussion to explain why class-identity features succeed under LIO. Cited to the authors' own repository [22].
  • domain assumption Rescaling an objective function leaves optimization algorithm search trajectories, and hence rankings, unchanged.
    Assumed in Section 3 to define rank as a scale-independent metric. The paper acknowledges this is not always true (see citation [16]) but relies on it for the COCO problems studied.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Pitfalls of Benchmarking in Algorithm Selection: What We Are Getting Wrong." pith.science (2026). https://pith.science/paper/3WQFDTWD

@misc{pith2026250507750,
  author       = {Pith},
  title        = {Pith review of: The Pitfalls of Benchmarking in Algorithm Selection: What We Are Getting Wrong},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3WQFDTWD}},
  note         = {Machine review of arXiv:2505.07750}
}
read the original abstract

Algorithm selection, aiming to identify the best algorithm for a given problem, plays a pivotal role in continuous black-box optimization. A common approach involves representing optimization functions using a set of features, which are then used to train a machine learning meta-model for selecting suitable algorithms. Various approaches have demonstrated the effectiveness of these algorithm selection meta-models. However, not all evaluation approaches are equally valid for assessing the performance of meta-models. We highlight methodological issues that frequently occur in the community and should be addressed when evaluating algorithm selection approaches. First, we identify flaws with the "leave-instance-out" evaluation technique. We show that non-informative features and meta-models can achieve high accuracy, which should not be the case with a well-designed evaluation framework. Second, we demonstrate that measuring the performance of optimization algorithms with metrics sensitive to the scale of the objective function requires careful consideration of how this impacts the construction of the meta-model, its predictions, and the model's error. Such metrics can falsely present overly optimistic performance assessments of the meta-models. This paper emphasizes the importance of careful evaluation, as loosely defined methodologies can mislead researchers, divert efforts, and introduce noise into the field

Figures

Figures reproduced from arXiv: 2505.07750 by the authors.

Figure 1
Figure 1. PRE of different meta-models with the LIO eval [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Comparison of target precision and rank metrics (y [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Left: MSE calculated between true and predicted [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 32 canonical work pages

  1. [1]

    Andrejaana Andova, Jordan N Cork, Aljoša Vodopija, Tea Tušar, and Bogdan Filipič. 2024. Predicting Algorithm Performance in Constrained Multiobjec- tive Optimization: A Tough Nut to Crack. In International Conference on the Applications of Evolutionary Computation (Part of EvoStar). Springer, 310–325

    work page 2024

  2. [2]

    Thomas Bäck. 2005. Evolution strategies: An alternative evolutionary algorithm. In Artificial Evolution: European Conference, AE 95 Brest, France, September 4–6, 1995 Selected Papers. Springer, 1–20

    work page 2005

  3. [3]

    Vahid Beiranvand, Warren Hare, and Yves Lucet. 2017. Best practices for compar- ing optimization algorithms. Optimization and Engineering 18 (2017), 815–848

    work page 2017

  4. [4]

    Blank and K

    J. Blank and K. Deb. 2020. pymoo: Multi-Objective Optimization in Python. IEEE Access 8 (2020), 89497–89509

    work page 2020

  5. [5]

    Gjorgjina Cenikj, Gašper Petelin, and Tome Eftimov. 2024. A cross-benchmark ex- amination of feature-based algorithm selector generalization in single-objective numerical optimization. Swarm and Evolutionary Computation 87 (2024), 101534

    work page 2024

  6. [6]

    Gjorgjina Cenikj, Gašper Petelin, Moritz Seiler, Nikola Cenikj, and Tome Efti- mov. 2025. Landscape Features in Single-Objective Continuous Optimization: Have We Hit a Wall in Algorithm Selection Generalization? arXiv preprint arXiv:2501.17663 (2025)

    work page Pith review arXiv 2025

  7. [7]

    Nikolaus Hansen, Anne Auger, Raymond Ros, Olaf Mersmann, Tea Tušar, and Dimo Brockhoff. 2021. COCO: A platform for comparing continuous optimizers in a black-box setting.Optimization Methods and Software 36, 1 (2021), 114–144

    work page 2021

  8. [8]

    Nikolaus Hansen and Andreas Ostermeier. 2001. Completely derandomized self-adaptation in evolution strategies. Evolutionary computation 9, 2 (2001), 159–195. The Pitfalls of Benchmarking in Algorithm Selection: What We Are Getting Wrong

    work page 2001

Show all 35 references
  1. [9]

    Hansika Hewamalage, Klaus Ackermann, and Christoph Bergmeir. 2023. Forecast evaluation for data scientists: common pitfalls and best practices. Data Mining and Knowledge Discovery 37, 2 (2023), 788–832

    2023

  2. [10]

    John N Hooker. 1995. Testing heuristics: We have it all wrong. Journal of heuristics 1 (1995), 33–42

    1995

  3. [11]

    Anja Jankovic and Carola Doerr. 2020. Landscape-aware fixed-budget perfor- mance regression and algorithm selection for modular CMA-ES variants. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference. 841–849

    2020

  4. [12]

    Sourabh Katoch, Sumit Singh Chauhan, and Vijay Kumar. 2021. A review on genetic algorithm: past, present, and future. Multimedia Tools and Applications 80 (2021), 8091–8126

    2021

  5. [13]

    James Kennedy and Russell Eberhart. 1995. Particle swarm optimization. In Proceedings of ICNN’95-international conference on neural networks, Vol. 4. IEEE, 1942–1948

    1995

  6. [14]

    Pascal Kerschke and Heike Trautmann. 2019. Automated algorithm selection on continuous black-box problems by combining exploratory landscape analysis and machine learning. Evolutionary computation 27, 1 (2019), 99–127

    2019

  7. [15]

    Benjamin Lacroix and John McCall. 2019. Limitations of benchmark sets and landscape features for algorithm selection and performance prediction. In Proceedings of the Genetic and Evolutionary Computation Conference Companion. 261–262

    2019

  8. [16]

    Fu Xing Long, Diederick Vermetten, Bas van Stein, and Anna V Kononova

  9. [17]

    Olaf Mersmann, Bernd Bischl, Heike Trautmann, Mike Preuss, Claus Weihs, and Günter Rudolph. 2011. Exploratory landscape analysis. In Proceedings of the 13th annual conference on Genetic and evolutionary computation. 829–836

    2011

  10. [18]

    Mario Andrés Muñoz and Michael Kirley. 2021. Sampling effects on algorithm selection for continuous black-box optimization. Algorithms 14, 1 (2021), 19

    2021

  11. [19]

    Curtis G Northcutt, Anish Athalye, and Jonas Mueller. 2021. Pervasive label errors in test sets destabilize machine learning benchmarks. arXiv preprint arXiv:2103.14749 (2021)

    2021 arXiv

  12. [20]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cour- napeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python.Journal of Machine L...

    2011

  13. [21]

    Gašper Petelin and Gjorgjina Cenikj. 2024. On Generalization of ELA Fea- ture Groups. In Proceedings of the Genetic and Evolutionary Computation Conference Companion. 419–422

    2024

  14. [22]

    Gašper Petelin and Gjorgjina Cenikj. 2025. Benchmarking Pitfalls. https://github. com/gasperpetelin/benchmarking-as-problems. Accessed: 2025-01-20

    2025

  15. [23]

    Gašper Petelin, Gjorgjina Cenikj, and Tome Eftimov. 2023. Towards understand- ing the importance of time-series features in automated algorithm performance prediction. Expert Systems with Applications 213 (2023), 119023

    2023

  16. [24]

    Raphael Patrick Prager and Heike Trautmann. 2023. Pflacco: Feature-based landscape analysis of continuous and constrained optimization problems in Python. Evolutionary Computation (2023), 1–25

    2023

  17. [25]

    Kenneth Price, Rainer M Storn, and Jouni A Lampinen. 2006. Differential evolution: a practical approach to global optimization. Springer Science & Busi- ness Media

    2006

  18. [26]

    Quentin Renau, Johann Dréo, Carola Doerr, and Benjamin Doerr. 2021. To- wards explainable exploratory landscape analysis: extreme feature selection for classifying BBOB functions. In Applications of Evolutionary Computation: 24th International Conference, EvoApplications 2021,...

    2021

  19. [27]

    Why should i trust you?

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. " Why should i trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 1135–1144

    2016

  20. [28]

    Urban Škvorc, Tome Eftimov, and Peter Korošec. 2020. Understanding the problem space in single-objective numerical optimization using exploratory landscape analysis. Applied Soft Computing 90 (2020), 106138

    2020

  21. [29]

    Urban Škvorc, Tome Eftimov, and Peter Korošec. 2022. Transfer learning analysis of multi-class classification for landscape-aware algorithm selection. mathematics 10, 3 (2022), 432

    2022

  22. [30]

    Eivind Strøm and Odd Erik Gundersen. 2024. Performance metrics for multi- step forecasting measuring win-loss, seasonal variance and forecast stability: an empirical study. Applied Intelligence 54, 21 (2024), 10490–10515

    2024

  23. [31]

    Ryoji Tanabe. 2022. Benchmarking Feature-Based Algorithm Selection Systems for Black-Box Numerical Optimization. IEEE Transactions on Evolutionary Computation 26, 6 (2022), 1321–1335

    2022

  24. [32]

    Guohua Wu, Rammohan Mallipeddi, and Ponnuthurai Suganthan. 2016. Prob- lem Definitions and Evaluation Criteria for the CEC 2017 Competition and Special Session on Constrained Single Objective Real-Parameter Optimization. Computational Intelligence Laboratory, Zhengzhou Univers...

    2016

  25. [33]

    Renjie Wu and Eamonn J Keogh. 2021. Current time series anomaly detec- tion benchmarks are flawed and are creating the illusion of progress. IEEE transactions on knowledge and data engineering 35, 3 (2021), 2421–2429

    2021

  26. [34]

    Urban Škvorc, Tome Eftimov, and Peter Korošec. 2020. Understanding the problem space in single-objective numerical optimization using exploratory landscape analysis. Applied Soft Computing 90 (2020), 106138. https://doi.org/ 10.1016/j.asoc.2020.106138

    2020

  27. [2023]

    In International Conference on the Applications of Evolutionary Computation (Part of EvoStar)

    BBOB instance analysis: Landscape properties and algorithm performance across problem instances. In International Conference on the Applications of Evolutionary Computation (Part of EvoStar). Springer, 380–395

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.