REVIEW 9 cited by
Benchmarking in Optimization: Best Practice and Open Issues
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Benchmarking in Optimization: Best Practice and Open Issues
read the original abstract
This survey compiles ideas and recommendations from more than a dozen researchers with different backgrounds and from different institutes around the world. Promoting best practice in benchmarking is its main goal. The article discusses eight essential topics in benchmarking: clearly stated goals, well-specified problems, suitable algorithms, adequate performance measures, thoughtful analysis, effective and efficient designs, comprehensible presentations, and guaranteed reproducibility. The final goal is to provide well-accepted guidelines (rules) that might be useful for authors and reviewers. As benchmarking in optimization is an active and evolving field of research this manuscript is meant to co-evolve over time by means of periodic updates.
Forward citations
Cited by 9 Pith papers
-
SPEC CPU: The Next Generation
SPEC CPU 2026 presents a new benchmark suite using open-source apps, expanded multithreading, and Rolling-Round-Robin Rate to address gaps in evaluating heterogeneous multiprogrammed CPU performance.
-
Block-Bench: A Framework for Controllable and Transparent Discrete Optimization Benchmarking
Block-Bench constructs controllable discrete optimization benchmarks from block functions and dependency graphs to enable transparent analysis of algorithm behavior beyond the objective value.
-
Learning to Assess the Reliability of Number-of-Runs Estimation in Stochastic Optimization
Supervised classifiers trained on 23 features from Nevergrad runs on COCO can detect unreliable run-number estimates with high minority-class recall in within-optimizer settings.
-
Large-scale benchmarking of multi-objective soft-computing metaheuristics for redundancy allocation in repairable k-out-of-n systems
A 65-algorithm benchmark on a bi-objective repairable redundancy-allocation problem shows that algorithm rankings are budget-dependent and that Scaled Binomial Initialization changes relative performance.
-
Standardizing case study descriptions for multi-energy systems and networks modeling
The authors adapt an existing standard into a unified description framework for multi-energy systems case studies, apply it to diverse cases, and develop a review checklist through cross-author evaluation.
-
Improving Evaluation of Recombination-based Cartesian Genetic Programming
Hyperparameter optimization yields performance improvements for recombination-based Cartesian Genetic Programming on SRBench.
-
From Heuristic Selection to Automated Algorithm Design: LLMs Benefit from Strong Priors
Prompting LLMs with strong benchmark algorithm code, rather than relying on linguistic instructions, improves LLM-driven black-box optimization; the proposed BAG method outperforms five baselines on pbo and bbob.
-
Time-Fair Benchmarking for Metaheuristics: A Restart-Fair Protocol for Fixed-Time Comparisons
A fixed-time, restart-fair benchmarking protocol for metaheuristics, using ERT and time-based performance profiles, is proposed without empirical validation.
-
Asymmetry PRISM: A CPU/GPU Portfolio Optimization Engine for Deadline-Bounded Institutional Rebalancing
Asymmetry PRISM-CPU achieves 4.5x-24.1x speedups over reference solvers on N=100-2000 problems and GPU completes all 500 accounts in 109.5s where OSQP completes 4.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.