Pith. sign in

REVIEW 3 major objections 4 minor 34 references

Bencher: Simple and Reproducible Benchmarking for Black-Box Optimization

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Bencher separates benchmark execution from optimizer code by isolating each benchmark in its own Python environment and exposing all 80 problems through one RPC client.

desk verdict The benchmark harness is real and the isolation idea is genuinely useful, but the paper asserts reproducibility without demonstrating it; it needs a round of validation before acceptance. read the letter →

arxiv 2505.21321 v1 pith:U5IXPJPF submitted 2025-05-27 cs.LG

Leonard Papenmeier , Luigi Nardi This is my paper
classification cs.LG
keywords black-boxoptimizationbenchmarkingframeworkbenchmarkisolationreproducibleevaluationremoteprocedurecallcontainerizationvirtualenvironmentsbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Bencher claims that the main barrier to reproducible black-box optimization is not missing benchmark problems but missing separation between benchmark code and optimizer code. Its fix is to run each benchmark inside its own virtual Python environment and expose it through a unified remote procedure call (RPC) interface, with the server packaged in a Docker or Singularity container and a minimal pip-installable client on the user side. The framework ships with 80 benchmarks across continuous, categorical, and binary domains, including 18 real-world problems such as the 124-dimensional Mopta08 vehicle-mass task. If this design holds, then one published container image can make every optimizer evaluation bit-comparable across machines, removing the setup variability that currently leads the same baseline to appear on different scales in different papers.

What carries the argument

The load-bearing mechanism is a client-server RPC boundary: each benchmark (or compatible group) runs as a service in its own virtual environment, and a coordinator service inside the container forwards every client call to the right environment. A benchmark registry maps names to internal ports, dimensions, and input type (continuous, binary, or categorical), and an optional Python-version file lets legacy benchmarks use older interpreters without affecting others. Containerization (Docker locally, Singularity when it inherits the Docker image on HPC) pins the full runtime, while the client remains dependency-light so users never install benchmark-specific packages.

What would settle it

Evaluate a fixed set of points on a benchmark such as mopta08 through Bencher on two machines or two container builds and check that the returned values are identical; any difference on the same benchmark version would falsify the reproducibility claim. Separately, compare Bencher's Mopta08 values against an independently archived copy of the original executable to test wrapper fidelity.

Watch

Extended reading notes

Core claim

The central claim is that a clean abstraction boundary between benchmark execution and optimization logic is enough to make diverse benchmarks coexist reproducibly. Bencher implements this boundary as a coordinator inside a container that receives RPCs from a thin client and delegates each call to the benchmark environment owning that function. Each environment has its own dependency set managed independently and may pin a different Python version, so mutually incompatible benchmarks run side by side. All internal traffic stays inside the container, so the client only sees one port and only needs gRPC and protobuf installed. The authors support the claim with an 80-benchmark suite whose container is rebuilt and exercised by an integration-test pipeline, and they present this suite as evidence that the boundary works at the scale of real-world benchmarks with complex, outdated, or dead-download software.

Load-bearing premise

The load-bearing premise is that sealing each benchmark in its own containerized virtual environment guarantees reproducible evaluations, and that the recreated wrappers for benchmarks whose original executables are gone—most notably Mopta08—faithfully reproduce the original problems.

Editorial extensions

If this is right

  • A user can evaluate all 80 included benchmarks through one pip-installed client without manually resolving any benchmark-specific dependency.
  • New benchmarks can be added as self-contained subprojects with their own Python versions and packages, so adding one cannot break the others.
  • The same evaluation can be run locally, in Docker, or on an HPC cluster via Singularity, giving a single recipe for reproducible comparisons.
  • The integration-test pipeline gives an up-to-date signal of which benchmarks still run, which matters for real-world benchmarks whose original executables have disappeared.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same per-benchmark isolation pattern should extend to constrained, multi-objective, and multi-fidelity problems once the RPC interface grows, which the paper lists as future work.
  • The decisive test of reproducibility is not containerization itself but wrapper fidelity: comparing Bencher's Mopta08 values against an independently archived original executable would settle whether the recreated benchmark matches the original.
  • A natural extension would have the framework record and expose a canonical set of function values for each benchmark version, allowing any optimizer paper to verify bit-level agreement with the published image.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents Bencher, a client-server benchmarking framework for black-box optimization in which each benchmark runs in its own virtual Python environment inside a Docker or Singularity container, accessed by clients through a gRPC-based RPC interface. The stated goals are to eliminate dependency conflicts, simplify benchmark integration, and provide a reproducible runtime for a claimed set of 80 benchmarks across continuous, categorical, and binary domains. The paper describes the architecture, client usage, container deployment, project structure, and integration testing, and it gives an overview of the included benchmarks with references to their sources.

Significance. If the reproducibility claims are substantiated, Bencher would be a practically useful contribution to the black-box optimization community: it offers a clean abstraction boundary between optimizers and benchmarks, modular per-benchmark environments, and concrete deployment artifacts (public repository, PyPI package, Docker image, Singularity recipe). The design is sensible and the integration-testing infrastructure is a positive sign. However, the central claim of reproducibility is asserted rather than demonstrated, and the benchmark scope claim is internally inconsistent. These issues currently prevent the paper from being accepted as a trustworthy systems contribution.

major comments (3)
  1. [Abstract and Section 4 vs. Appendix A] The abstract and Section 4 state that Bencher supports 80 benchmarks, but Appendix A lists exactly 72 named benchmarks. This discrepancy makes the scope claim unverifiable and must be fixed, either by correcting the text to 72 or by adding the missing benchmark names to the appendix.
  2. [Section 3.5 (Testing)] The integration tests described in Section 3.5 verify only that benchmarks start successfully inside the Docker container; there is no comparison of returned function values against reference implementations or known optima, and no experiment demonstrating that evaluations are stable across repeated runs, across different hosts, or between the Docker and Singularity runtimes. Since the paper's central claim is reproducibility, the testing section needs to validate value fidelity and cross-environment stability, not merely startup success.
  3. [Section 4 (Mopta08 provenance)] The discussion of Mopta08 concedes that the original executables are no longer available and that the version used in Bencher derives from files uploaded by other researchers. Without a provenance check or a comparison of the wrapper's outputs against reference values from the literature, the fidelity of this benchmark cannot be established, and the same concern may extend to other wrapped benchmarks. The paper should either provide such validation or clearly document the provenance and its limitations for each affected benchmark.
minor comments (4)
  1. [Section 3 (Design and Implementation)] The sentence 'The server is responsible for running the benchmarks and listening to RPCs from clients.' is duplicated verbatim in the opening paragraph of Section 3.
  2. [Section 3.1 (Client), Listing 1] In Listing 1, the variable 'values' is defined but the call uses 'client.evaluate_point(benchmark_name, point)' where 'point' is undefined; this should be 'values'.
  3. [Section 1 (Introduction)] The phrase 'have received considerably interest' should read 'have received considerable interest'.
  4. [Section 3.3 (HPC Setup)] The Singularity example uses curly quotes around INST_NAME and an unclear placeholder format; using a concrete shell example with SLURM_ARRAY_TASK_ID would improve clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular reasoning found: Bencher is a self-contained software/architecture description with no fitted parameters or self-referential predictions.

full rationale

The paper is a systems/software contribution describing an RPC-based benchmarking framework. Its central claims are architectural: that each benchmark is isolated in a virtual Python environment, that the server communicates via gRPC, and that Docker and Singularity containers provide reproducible deployment. There is no derived quantity fitted to benchmark data, no empirical prediction that depends on the framework's own output, and no uniqueness theorem imported from prior work. Self-citations (e.g., Papenmeier et al. 2022, 2023, 2025) are used only for context, benchmark provenance, or as examples of the benchmarks' adoption; they do not carry the load of the framework's correctness claims. The testing described in Section 3.5 checks that benchmarks run, but even if this under-validates reproducibility, that is a verification-strength concern, not a circularity concern. The Mopta08 provenance caveat in Section 4 is an honesty about external availability, not a circular derivation. No step in the paper reduces to its own inputs by construction.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The central claims that Bencher makes benchmarks reproducible and that 80 benchmarks are supported rest on the fidelity of the wrapped benchmarks and on container isolation being sufficient. No fitted parameters or new entities are introduced; the framework adds value as infrastructure, not as a new scientific quantity.

assumptions (2)
  • domain assumption Container isolation is sufficient to guarantee reproducible benchmark evaluations across machines and HPC environments.
    Sections 3.2 and 3.3 rely on Docker and Singularity to provide identical software environments, but they do not address hardware differences or nondeterminism in simulators such as MuJoCo.
  • domain assumption The benchmark implementations included in Bencher are correct and faithful to the original source benchmarks.
    Section 4 says most benchmarks are well-known and includes wrappers of Mopta08 whose original executables are dead; correctness is asserted by integration tests, not verified against reference values.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bencher: Simple and Reproducible Benchmarking for Black-Box Optimization." pith.science (2026). https://pith.science/paper/U5IXPJPF

@misc{pith2026250521321,
  author       = {Pith},
  title        = {Pith review of: Bencher: Simple and Reproducible Benchmarking for Black-Box Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U5IXPJPF}},
  note         = {Machine review of arXiv:2505.21321}
}
read the original abstract

We present Bencher, a modular benchmarking framework for black-box optimization that fundamentally decouples benchmark execution from optimization logic. Unlike prior suites that focus on combining many benchmarks in a single project, Bencher introduces a clean abstraction boundary: each benchmark is isolated in its own virtual Python environment and accessed via a unified, version-agnostic remote procedure call (RPC) interface. This design eliminates dependency conflicts and simplifies the integration of diverse, real-world benchmarks, which often have complex and conflicting software requirements. Bencher can be deployed locally or remotely via Docker or on high-performance computing (HPC) clusters via Singularity, providing a containerized, reproducible runtime for any benchmark. Its lightweight client requires minimal setup and supports drop-in evaluation of 80 benchmarks across continuous, categorical, and binary domains.

Figures

Figures reproduced from arXiv: 2505.21321 by the authors.

Figure 1
Figure 1. The Bencher architecture. The server runs in a Docker container and listens to RPCs from clients, which can be on the same or on a different machine. The server is composed of multiple Poetry environments, one for each benchmark. Bencher implements most of the benchmarks from IOHex￾perimenter, including the BBOB, pseudo-boolean, W-model, and submodular benchmarks. While all of the aforementioned benchmark suites adv… view at source ↗
Figure 2
Figure 2. The Bencher directory structure. 3.3. HPC Setup Bencher can be run on an HPC cluster using Singular￾ity with minimal manual setup by inheriting the Docker container. One constraint is that the Singularity container needs to be started as an instance in the background. To isolate instances running on the same cluster node, we rec￾ommend using a unique instance name for each HPC job. With slurm, this can be done by us… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 28 canonical work pages

  1. [1]

    Nevergrad: black-box optimization platform

    Bennet, P., Doerr, C., Moreau, A., Rapin, J., Teytaud, F., and Teytaud, O. Nevergrad: black-box optimization platform. ACM SIGEVOlution, 14 0 (1): 0 8--15, 2021

    work page 2021

  2. [2]

    Algorithms for Hyper-Parameter Optimization

    Bergstra, J., Bardenet, R., Bengio, Y., and K\' e gl, B. Algorithms for Hyper-Parameter Optimization . In Advances in Neural Information Processing Systems (NeurIPS), volume 24. Curran Associates, Inc., 2011

    work page 2011

  3. [3]

    M., Gusev, V

    Burger, B., Maffettone, P. M., Gusev, V. V., Aitchison, C. M., Bai, Y., Wang, X., Li, X., Alston, B. M., Li, B., Clowes, R., et al. A mobile robotic chemist. Nature, 583 0 (7815): 0 237--241, 2020

    work page 2020

  4. [4]

    Cosenza, Z., Astudillo, R., Frazier, P., Baar, K., and Block, D. E. Multi-Information Source Bayesian Optimization of Culture Media for Cellular Agriculture . Biotechnology and Bioengineering, 2022

    work page 2022

  5. [5]

    Iohexperimenter: Benchmarking platform for iterative optimization heuristics

    de Nobel, J., Ye, F., Vermetten, D., Wang, H., Doerr, C., and B \"a ck, T. Iohexperimenter: Benchmarking platform for iterative optimization heuristics. Evolutionary Computation, 32 0 (3): 0 205--210, 2024

    work page 2024

  6. [6]

    R., and Eriksson, D

    Deshwal, A., Ament, S., Balandat, M., Bakshy, E., Doppa, J. R., and Eriksson, D. Bayesian Optimization over High-Dimensional Combinatorial Spaces via Dictionary-based Embeddings . In International Conference on Artificial Intelligence and Statistics, pp.\ 7021--7039. PMLR, 2023

    work page 2023

  7. [7]

    Iohprofiler: A benchmarking and profiling tool for iterative optimization heuristics

    Doerr, C., Wang, H., Ye, F., Van Rijn, S., and B \"a ck, T. Iohprofiler: A benchmarking and profiling tool for iterative optimization heuristics. arXiv preprint arXiv:1810.05281, 2018

    arXiv 2018

  8. [8]

    Hpobench: A collection of reproducible multi-fidelity benchmark problems for hpo

    Eggensperger, K., M \"u ller, P., Mallik, N., Feurer, M., Sass, R., Klein, A., Awad, N., Lindauer, M., and Hutter, F. Hpobench: A collection of reproducible multi-fidelity benchmark problems for hpo. arXiv preprint arXiv:2109.06716, 2021

    arXiv 2021

Show all 34 references
  1. [9]

    and Jankowiak, M

    Eriksson, D. and Jankowiak, M. High-dimensional bayesian optimization with sparse axis-aligned subspaces. In Uncertainty in Artificial Intelligence, pp.\ 493--503. PMLR, 2021

    2021

  2. [10]

    and Poloczek, M

    Eriksson, D. and Poloczek, M. Scalable C onstrained B ayesian O ptimization. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp.\ 730--738. PMLR, 13--15 Apr 2021

    2021

  3. [11]

    D., and Poloczek, M

    Eriksson, D., Pearce, M., Gardner, J., Turner, R. D., and Poloczek, M. Scalable G lobal O ptimization via L ocal Bayesian O ptimization. In Advances in Neural Information Processing Systems (NeurIPS), pp.\ 5496--5507, 2019

    2019

  4. [12]

    H., and Hu, Q

    Fan, Z., Wang, W., Ng, S. H., and Hu, Q. Minimizing ucb: a better local search strategy in local bayesian optimization. Advances in Neural Information Processing Systems, 37: 0 130602--130634, 2024

    2024

  5. [13]

    A., Sampaio, P

    Hansen, N., Brockhoff, D., Mersmann, O., Tusar, T., Tusar, D., ElHara, O. A., Sampaio, P. R., Atamna, A., Varelas, K., Batu, U., Nguyen, D. M., Matzner, F., and Auger, A. Comparing continuous optimizers: numbbo/coco on github, March 2019. URL https://doi.org/10.5281/zenodo.2594848

    2019 doi

  6. [14]

    M., Requeima, J., Pyzer-Knapp, E

    Hern \'a ndez-Lobato, J. M., Requeima, J., Pyzer-Knapp, E. O., and Aspuru-Guzik, A. Parallel and Distributed Thompson Sampling for Large-scale Accelerated Exploration of Chemical Space . In Proceedings of the 34th International Conference on Machine Learning, volume 70, pp.\ 1...

    2017

  7. [15]

    O., and Nardi, L

    Hvarfner, C., Hellsten, E. O., and Nardi, L. Vanilla Bayesian Optimization Performs Great in High Dimensions . In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Mac...

    2024

  8. [16]

    Jones, D. R. Large-scale multi-disciplinary mass optimization in the auto industry . In MOPTA 2008 Conference (20 August 2008), 2008

    2008

  9. [17]

    Lam, R., Poloczek, M., Frazier, P., and Willcox, K. E. Advances in Bayesian optimization with applications in aerospace engineering . In 2018 AIAA Non-Deterministic Approaches Conference, pp.\ 1656, 2018

    2018

  10. [18]

    F., De Breuker, R., and Castro, S

    Maathuis, H. F., De Breuker, R., and Castro, S. G. High-Dimensional Bayesian Optimisation with Large-Scale Constraints-An Application to Aeroelastic Tailoring . In AIAA SCITECH 2024 Forum, pp.\ 2012, 2024

    2024

  11. [19]

    Local bayesian optimization via maximizing probability of descent

    Nguyen, Q., Wu, K., Gardner, J., and Garnett, R. Local bayesian optimization via maximizing probability of descent. Advances in neural information processing systems, 35: 0 13190--13202, 2022

    2022

  12. [20]

    Combinatorial Bayesian Optimization using the Graph Cartesian Product

    Oh, C., Tomczak, J., Gavves, E., and Welling, M. Combinatorial Bayesian Optimization using the Graph Cartesian Product . Advances in Neural Information Processing Systems (NeurIPS), 32, 2019

    2019

  13. [21]

    Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspaces

    Papenmeier, L., Nardi, L., and Poloczek, M. Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspaces . In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022

    2022

  14. [22]

    Bounce: Reliable high-dimensional bayesian optimization for combinatorial and mixed spaces

    Papenmeier, L., Nardi, L., and Poloczek, M. Bounce: Reliable high-dimensional bayesian optimization for combinatorial and mixed spaces. Advances in Neural Information Processing Systems, 36: 0 1764--1793, 2023

    2023

  15. [23]

    Understanding high-dimensional bayesian optimization

    Papenmeier, L., Poloczek, M., and Nardi, L. Understanding high-dimensional bayesian optimization. arXiv preprint arXiv:2502.09198, 2025

    2025 arXiv

  16. [24]

    and M \"u ller, R

    Sala, R. and M \"u ller, R. Benchmarking for metaheuristic black-box optimization: perspectives and open challenges. In 2020 IEEE Congress on Evolutionary Computation (CEC), pp.\ 1--8. IEEE, 2020

    2020

  17. [25]

    LassoBench: A High-Dimensional Hyperparameter Optimization Benchmark Suite for Lasso

    S ehi \'c , K., Gramfort, A., Salmon, J., and Nardi, L. LassoBench: A High-Dimensional Hyperparameter Optimization Benchmark Suite for Lasso . In First Conference on Automated Machine Learning (Main Track), 2022 a

    2022

  18. [26]

    LassoBench: A High-Dimensional Hyperparameter Optimization Benchmark Suite for Lasso

    S ehi \'c , K., Gramfort, A., Salmon, J., and Nardi, L. LassoBench: A High-Dimensional Hyperparameter Optimization Benchmark Suite for Lasso . In International Conference on Automated Machine Learning, pp.\ 2--1. PMLR, 2022 b

    2022

  19. [27]

    and Kingsford, C

    Shen, Y. and Kingsford, C. Computationally efficient high-dimensional bayesian optimization via variable selection. arXiv preprint arXiv:2109.09264, 2021

    2021 arXiv

  20. [28]

    Snoek, J., Larochelle, H., and Adams, R. P. Practical Bayesian Optimization of Machine Learning Algorithms . In Advances in Neural Information Processing Systems (NeurIPS), volume 25, 2012

    2012

  21. [29]

    E., Sahu, S., Kosa, N

    Tallorin, L., Wang, J., Kim, W. E., Sahu, S., Kosa, N. M., Yang, P., Thompson, M., Gilson, M. K., Frazier, P. I., Burkart, M. D., et al. Discovering de novo peptide substrates for enzymes using machine learning . Nature communications, 9 0 (1): 0 1--10, 2018

    2018

  22. [30]

    O., Hvarfner, C., Nardi, L., and Sj \"a lander, M

    T rring, J. O., Hvarfner, C., Nardi, L., and Sj \"a lander, M. CATBench: A Compiler Autotuning Benchmarking Suite for Black-box Optimization . arXiv preprint arXiv:2406.17811, 2024

    2024 arXiv

  23. [31]

    Bayesian optimization is superior to random search for machine learning hyperparameter tuning: Analysis of the black-box optimization challenge 2020

    Turner, R., Eriksson, D., McCourt, M., Kiili, J., Laaksonen, E., Xu, Z., and Guyon, I. Bayesian optimization is superior to random search for machine learning hyperparameter tuning: Analysis of the black-box optimization challenge 2020. In NeurIPS 2020 Competition and Demonstr...

    2020

  24. [32]

    Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Search

    Wang, L., Fonseca, R., and Tian, Y. Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Search . Advances in Neural Information Processing Systems (NeurIPS), 33: 0 19511--19522, 2020

    2020

  25. [33]

    Batched large-scale B ayesian optimization in high-dimensional spaces

    Wang, Z., Gehring, C., Kohli, P., and Jegelka, S. Batched large-scale B ayesian optimization in high-dimensional spaces. In International Conference on Artificial Intelligence and Statistics, pp.\ 745--754, 2018

    2018

  26. [34]

    M., and Zhe, S

    Xu, Z., Wang, H., Phillips, J. M., and Zhe, S. Standard gaussian process is all you need for high-dimensional bayesian optimization. In The Thirteenth International Conference on Learning Representations, 2024

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.