Pith. sign in

REVIEW 1 major objections 2 minor 18 references

Classifiers trained on run features can predict whether a stochastic optimizer's estimated number of runs is reliable within the same configuration.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Supervised classifiers trained on 23 features from Nevergrad runs on COCO can detect unreliable run-number estimates with high minority-class recall in within-optimizer settings.

T0 review reviewed 2026-06-29 challenge →

load-bearing objection The paper wraps a run-number heuristic with classifiers trained on 132k COCO runs to flag unreliable estimates, but the abstract gives no metrics or label details so the claim stays unverified. the 1 major comments →

arxiv 2605.28309 v1 pith:FVTIDE4R submitted 2026-05-27 cs.LG cs.NE

Learning to Assess the Reliability of Number-of-Runs Estimation in Stochastic Optimization

Sara Gjorgjieva , Eva Tuba , Tome Eftimov This is my paper
classification cs.LG cs.NE
keywords stochastic optimizationnumber of runs estimationreliability predictionmachine learning classifiersCOCO benchmarkwithin-configuration learningminority class recall
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tests whether machine learning can determine if an online estimate of the number of runs needed for reliable results from a stochastic optimizer is itself trustworthy. It draws on 132000 annotated runs across 24 COCO problems in 20 dimensions using 11 optimizers inside the Nevergrad framework. Twenty-three statistical, shape, and stability features are extracted from each run and used to train classifiers that focus on catching unreliable estimates via high minority-class recall. Training and testing occur in a within-configuration regime where the optimizer stays fixed. The central finding is that this prediction task is learnable under those constraints, though data diversity within one optimizer limits how far the models generalize.

Core claim

Using annotated outcomes from 132000 Nevergrad runs on COCO, classifiers trained on 23 statistical, energy-free, and shape and stability features can predict whether a run-number estimate is reliable in a within-configuration scenario, enabling detection of unreliable estimates with high minority-class recall.

What carries the argument

Classifiers that take 23 run-derived features as input and output a binary reliability label for the current estimate of required runs, trained and evaluated separately per optimizer.

Load-bearing premise

The human-assigned labels that mark each run-number estimate as reliable or unreliable are correct, and the 23 selected features contain enough signal to separate the two classes.

What would settle it

A fresh collection of optimization runs whose reliability labels are obtained by an independent verification procedure (for example, exhaustive additional runs) on which the trained classifiers show low minority-class recall would show the claim does not hold.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Reliability of run-number estimates becomes a learnable property rather than a purely heuristic one when the optimizer is held fixed.
  • High minority-class recall allows early detection of cases where the current run count is insufficient without needing exhaustive extra computation.
  • The learned predictor can be inserted into existing online heuristics that already use outlier handling and skewness checks.
  • Performance gains are bounded by the narrow variety of run trajectories that appear inside any single optimizer.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Integrating the reliability classifier into the stopping rule itself could produce an adaptive loop that both estimates and validates the run count on the fly.
  • Cross-optimizer training might overcome the diversity limit observed in within-configuration experiments and yield models that transfer to unseen algorithms.
  • The same feature set and labeling approach could be applied to other adaptive heuristics that decide when to stop sampling in stochastic settings.
  • If the feature set misses certain failure modes, adding cheap-to-compute descriptors such as trajectory curvature or gradient-norm statistics could be tested directly on the same 132000-run corpus.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 2 minor

Summary. The paper claims that a learning-based extension of an empirical online heuristic for estimating the number of runs in stochastic optimization can be assessed for reliability using classifiers trained on 23 statistical, energy-free, shape, and stability features from 132,000 annotated Nevergrad runs on COCO (24 problems, 20D, 10 instances, 11 optimizers). In a within-configuration protocol (train/test on data from the same optimizer), the classifiers detect unreliable estimates with high minority-class recall, although performance is constrained by limited data diversity within fixed configurations.

Significance. If the binary labels constitute independent ground truth, the work could enable more efficient large-scale benchmarking by adaptively flagging when run-number estimates are unreliable, thereby avoiding unnecessary computation. The within-configuration results provide evidence that reliability signals are learnable under matched conditions, and the emphasis on minority-class recall directly targets the practical cost of missing unreliable cases.

major comments (1)
  1. [Data Preparation / Annotation (Methods)] The annotation process that produces the 132,000 binary labels (reliable vs. unreliable run-number estimates) is not described with sufficient detail on the statistical checks, stability thresholds, skewness cutoffs, or reference budgets employed. Because these labels are generated from the same runs via checks that may correlate with the 23 features or the base heuristic, the high minority-class recall could arise by construction rather than from discovery of independent unreliability signals; this is load-bearing for the central claim that reliability 'can be learned'.
minor comments (2)
  1. [Abstract] The abstract states that results show 'high minority-class recall' but supplies no numerical values, confidence intervals, or baseline comparisons, which would strengthen the presentation of the within-configuration findings.
  2. [Feature Engineering] The 23 features are described only at a high level (statistical, energy-free, shape, stability); an explicit list or table with definitions would improve reproducibility.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the detailed review and constructive feedback. We address the single major comment below and agree that the manuscript requires expansion on the annotation process.

read point-by-point responses

  1. Referee: The annotation process that produces the 132,000 binary labels (reliable vs. unreliable run-number estimates) is not described with sufficient detail on the statistical checks, stability thresholds, skewness cutoffs, or reference budgets employed. Because these labels are generated from the same runs via checks that may correlate with the 23 features or the base heuristic, the high minority-class recall could arise by construction rather than from discovery of independent unreliability signals; this is load-bearing for the central claim that reliability 'can be learned'.

    Authors: We will revise the Methods section to provide a complete description of the annotation pipeline, including the exact statistical checks, stability thresholds, skewness cutoffs, and reference budgets used to assign the binary labels. Regarding potential circularity: the 23 features comprise trajectory-derived statistics, energy-free descriptors, shape metrics, and stability indicators computed directly from the observed runs, whereas the reliability labels are assigned via an independent post-hoc protocol that compares the heuristic's run-number estimate against convergence behavior on additional held-out runs and higher reference budgets. The base heuristic itself is not used in label generation; labels reflect whether the estimated run count would have produced a stable outcome under the verification criteria. We will add an explicit paragraph clarifying this separation to demonstrate that the prediction task is not tautological. These changes will be made in the next version. revision: yes

Circularity Check

0 steps flagged

No significant circularity; labels are externally annotated

full rationale

The paper trains supervised classifiers on 23 features to predict binary reliability labels for run-number estimates. These labels originate from an annotation process applied to 132000 runs, which is presented as independent ground truth rather than derived from the same equations or features used in the model. No self-definitional step, fitted-input-as-prediction, or self-citation chain appears in the abstract or described method; the within-configuration protocol further decouples training and test data. The derivation therefore remains self-contained against the external annotations.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claim rests on the assumption that the 132,000 annotated runs constitute a representative training distribution for reliability prediction and that the 23 features capture the necessary statistical signals; no free parameters or invented entities are introduced in the abstract.

axioms (2)
  • domain assumption The labels indicating whether each run-number estimate was reliable are accurate and unbiased.
    The entire supervised-learning pipeline depends on these labels; the abstract does not describe how they were generated.
  • domain assumption The 23 statistical, energy-free, shape and stability features are sufficient to discriminate reliable from unreliable estimates.
    Feature selection is presented as given; no ablation or justification appears in the abstract.

reviewed 2026-06-29 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning to Assess the Reliability of Number-of-Runs Estimation in Stochastic Optimization." pith.science (2026). https://pith.science/paper/FVTIDE4R

@misc{pith2026260528309,
  author       = {Pith},
  title        = {Pith review of: Learning to Assess the Reliability of Number-of-Runs Estimation in Stochastic Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FVTIDE4R}},
  note         = {Machine review of arXiv:2605.28309}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

In large-scale benchmarking of stochastic optimization algorithms, the key challenge is no longer whether repeated runs are needed for reliability, but how to determine when sufficient evidence has been collected without incurring unnecessary computational cost. We study a learning-based extension of a recent empirical online heuristic that adaptively estimates the required number of runs using outlier handling and skewness-based symmetry checks. Using annotated outcomes from 132{,}000 Nevergrad runs on COCO (24 problems in 20 dimensions, 10 instances each, 11 optimizers), we train classifiers on 23 statistical, energy-free, and shape and stability features to predict whether a run-number estimate is reliable, prioritizing detection of incorrect estimates via minority-class recall. We evaluate reliability prediction using a within-configuration learning setup, where models are trained and tested on data sharing the same optimizer. The results show that run-number reliability can be learned in a within-configuration scenario, enabling detection of unreliable estimates with high minority-class recall, although performance remains limited by the restricted data diversity within fixed configurations.

Figures

Figures reproduced from arXiv: 2605.28309 by Eva Tuba, Sara Gjorgjieva, Tome Eftimov.

Figure 1
Figure 1. Figure 1: Within-configuration learning experiment results. Three heatmaps correspond to outlier methods 1, 2, and 3. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 1 canonical work pages

  1. [1]

    Malan, Jason H

    Thomas Bartz-Beielstein, Carola Doerr, Daan van den Berg, Jakob Bossek, Sowmya Chandrasekaran, Tome Eftimov, Andreas Fischbach, Pascal Kerschke, William La Cava, Manuel Lopez-Ibanez, et al. Benchmarking in optimization: Best practice and open issues.arXiv preprint arXiv:2007.03488, 2020

    work page arXiv 2007

  2. [2]

    PhD thesis, INRIA, 2009

    Nikolaus Hansen, Steffen Finck, Raymond Ros, and Anne Auger.Real-parameter black-box optimization benchmarking 2009: Noiseless functions definitions. PhD thesis, INRIA, 2009

    2009

  3. [3]

    Jing J Liang, Bo Y Qu, and Ponnuthurai N Suganthan. Problem definitions and evaluation criteria for the cec 2014 special session and competition on single objective real-parameter numerical optimization.Technical report Zhengzhou, China, 635:490, 2013

    2014

  4. [4]

    A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms

    Joaquín Derrac, Salvador García, Daniel Molina, and Francisco Herrera. A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms. Swarm and Evolutionary Computation, 1(1):3–18, 2011

    2011

  5. [5]

    On the influence of the number of algorithms, problems, and independent runs in the comparison of evolutionary algorithms.Applied Soft Computing, 54:23–45, 2017

    Niki Veˇcek, Matej ˇCrepinšek, and Marjan Mernik. On the influence of the number of algorithms, problems, and independent runs in the comparison of evolutionary algorithms.Applied Soft Computing, 54:23–45, 2017

    2017

  6. [6]

    Adaptive estimation of the number of algorithm runs in stochastic optimization

    Tome Eftimov and Peter Korošec. Adaptive estimation of the number of algorithm runs in stochastic optimization. InProceedings of the Genetic and Evolutionary Computation Conference, pages 719–727, 2025

    2025

  7. [7]

    Learning to assess the reliability of number-of-runs estimation in stochastic optimization, jan 2026

    Sara Gjorgjieva, Eva Tuba, and Tome Eftimov. Learning to assess the reliability of number-of-runs estimation in stochastic optimization, jan 2026

    2026

  8. [8]

    Experimental research in evolutionary computation

    Thomas Bartz-Beielstein and Mike Preuss. Experimental research in evolutionary computation. InProceedings of the 9th annual conference companion on genetic and evolutionary computation, pages 3001–3020, 2007

    2007

  9. [9]

    A chess rating system for evolutionary algorithms: a new method for the comparison and ranking of evolutionary algorithms.Information Sciences, 277:656–679, 2014

    Niki Veˇcek, Marjan Mernik, and Matej ˇCrepinšek. A chess rating system for evolutionary algorithms: a new method for the comparison and ranking of evolutionary algorithms.Information Sciences, 277:656–679, 2014

    2014

  10. [10]

    Identifying practical significance through statistical comparison of meta- heuristic stochastic optimization algorithms.Applied Soft Computing, 85:105862, 2019

    Tome Eftimov and Peter Korošec. Identifying practical significance through statistical comparison of meta- heuristic stochastic optimization algorithms.Applied Soft Computing, 85:105862, 2019

    2019

  11. [11]

    Analyzing the impact of undersampling on the benchmarking and configuration of evolutionary algorithms

    Diederick Vermetten, Hao Wang, Manuel López-Ibañez, Carola Doerr, and Thomas Bäck. Analyzing the impact of undersampling on the benchmarking and configuration of evolutionary algorithms. InProceedings of the Genetic and Evolutionary Computation Conference, pages 867–875, 2022

    2022

  12. [12]

    Sample size estimation for power and accuracy in the experimental comparison of algorithms.Journal of Heuristics, 25:305–338, 2019

    Felipe Campelo and Fernanda Takahashi. Sample size estimation for power and accuracy in the experimental comparison of algorithms.Journal of Heuristics, 25:305–338, 2019. 5

    2019

  13. [13]

    Hansen, A

    N. Hansen, A. Auger, R. Ros, O. Mersmann, T. Tušar, and D. Brockhoff. COCO: A platform for comparing continuous optimizers in a black-box setting.Optimization Methods and Software, 36:114–144, 2021

    2021

  14. [14]

    Rapin and O

    J. Rapin and O. Teytaud. Nevergrad - A gradient-free optimization platform. https://GitHub.com/ FacebookResearch/Nevergrad, 2018

    2018

  15. [15]

    PhD thesis, East Tennessee State University, 2005

    Dewey Lonzo Whaley III.The interquartile range: Theory and estimation. PhD thesis, East Tennessee State University, 2005

    2005

  16. [16]

    A review of bootstrap confidence intervals.Journal of the Royal Statistical Society: Series B (Methodological), 50(3):338–354, 1988

    Thomas J Diciccio and Joseph P Romano. A review of bootstrap confidence intervals.Journal of the Royal Statistical Society: Series B (Methodological), 50(3):338–354, 1988

    1988

  17. [17]

    Labeling methods for identifying outliers.International Journal of Statistics and Systems, 10(2):231–238, 2015

    K Senthamarai Kannan, K Manoj, and S Arumugam. Labeling methods for identifying outliers.International Journal of Statistics and Systems, 10(2):231–238, 2015

    2015

  18. [18]

    Optuna: A next-generation hyperparameter optimization framework

    Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. InProceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623–2631, 2019. 6

    2019

This paper was first reviewed by grok-4.3 on June 29, 2026.