Pith Number

pith:QEP33ICZ

pith:2026:QEP33ICZFE3RFOJFEIZWY4ZEHN

not attested not anchored not stored refs resolved

BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Tasks

Apivich Hemachandra, Bryan Kian Hsiang Low, Ruth Wan Theng Chew, Zhiliang Chen

BoLT supplies lightweight surrogate models from thousands of real LLM runs so black-box optimization researchers can test methods on realistic expensive tasks without prohibitive costs.

arxiv:2605.17000 v1 · 2026-05-16 · cs.LG · cs.AI

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{QEP33ICZFE3RFOJFEIZWY4ZEHN}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

BoLT is the first LLM-centric benchmark that democratizes LLM research for the BBO community by releasing lightweight surrogate models fitted to the results of thousands of real LLM experiments, covering multi-fidelity, multi-objective, heteroscedastic noise, and high-dimensional search spaces; selected BO methods consistently outperform others across tasks.

C2weakest assumption

The surrogate models fitted to the real LLM experiment data accurately reproduce the optimization landscapes, noise characteristics, and relative performance ordering of methods that would be observed on the actual expensive LLM tasks.

C3one line summary

BoLT is a benchmark of surrogate models fitted to real LLM experiment data that enables evaluation of Bayesian and black-box optimization methods on multi-fidelity, multi-objective, high-dimensional LLM tasks.

References

83 extracted · 83 resolved · 7 Pith anchors

[1] T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama. Optuna: A next-generation hyperparameter optimization framework. InProceedings of the 25th ACM SIGKDD international conference on knowledge discov 2019

[2] A survey on data selection for language models 2024

[3] S. Ament, S. Daulton, D. Eriksson, M. Balandat, and E. Bakshy. Unexpected improvements to expected improvement for Bayesian optimization.Advances in Neural Information Processing Systems, 36:20577– 20 2023

[4] S. P. Arango, H. S. Jomaa, M. Wistuba, and J. Grabocka. Hpo-b: A large-scale reproducible benchmark for black-box hpo based on openml. InThirty-fifth Conference on Neural Information Processing System 2021

[5] Program Synthesis with Large Language Models 2021 · arXiv:2108.07732

Formal links

2 machine-checked theorem links

Receipt and verification

First computed	2026-05-20T00:03:35.329787Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

811fbda059293712b92522336c73243b4134627b6d7f43eb1144ba1b6a8cd345

Aliases

arxiv: 2605.17000 · arxiv_version: 2605.17000v1 · doi: 10.48550/arxiv.2605.17000 · pith_short_12: QEP33ICZFE3R · pith_short_16: QEP33ICZFE3RFOJF · pith_short_8: QEP33ICZ

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/QEP33ICZFE3RFOJFEIZWY4ZEHN \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 811fbda059293712b92522336c73243b4134627b6d7f43eb1144ba1b6a8cd345

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "a964e91e3a00451ef5cc7b438a185025a4363acea902531dfb9abfe24cf9a797",
    "cross_cats_sorted": [
      "cs.AI"
    ],
    "license": "http://creativecommons.org/licenses/by/4.0/",
    "primary_cat": "cs.LG",
    "submitted_at": "2026-05-16T13:53:44Z",
    "title_canon_sha256": "f3a2c3e37183d47e3fe991d4b3d6ba918a7df437d722b49400b7be09599d07a8"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2605.17000",
    "kind": "arxiv",
    "version": 1
  }
}