Pith. sign in

REVIEW 2 major objections 6 minor 12 references

Facility location on inexpensive semantic prompt embeddings selects small LLM benchmark coresets that preserve model scores better than score-based methods, without any prior model evaluations.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 16:37 UTC pith:5V2BQF7F

load-bearing objection Solid empirical result: facility location on cheap prompt embeddings beats a wide baseline suite for score-free multi-benchmark coreset selection, and the same objective stays competitive when scores are available. the 2 major comments →

arxiv 2607.09739 v1 pith:5V2BQF7F submitted 2026-07-02 cs.AI cs.CL

Coresets Before Score Sets: Evaluation-Unsupervised Prompt Subset Selection for LLM Benchmarks

Jihan Yao , Gantavya Bhatt , Arnav Das , Peter Jin , Ke Bao , Qiaolin Yu , Khushi Bhardwaj , Chang Su
show 11 more authors
Jialei Wang Yikai Zhu Sugam Devare Damon Mosk-Aoyama Zhen Dong Venkat Krishna Srinivasan Yineng Zhang Oleksii Kuchaiev Jiantao Jiao Banghua Zhu Jeff Bilmes
This is my paper
classification cs.AI cs.CL
keywords LLM evaluationbenchmark coresetsubmodular optimizationfacility locationsemantic embeddingsevaluation-unsupervised selectionDPPprompt subset selection
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Evaluating large language models on ever-growing suites of benchmarks is expensive, and many of those suites are redundant. This paper asks whether a small subset of prompts, chosen without looking at any model scores or responses, can still recover the scores and rankings that the full suite would have produced. Working at the fine grain of individual prompts across many benchmarks, the authors show that a classical submodular coverage objective—facility location—run on cheap text embeddings of prompts plus their ground-truth metadata does this better than a dozen score-based and diversity baselines. On a newly collected suite of 35 heterogeneous benchmarks, 18 frontier models, and more than 61,000 prompts, the same objective keeps both overall and per-capability relative error low across coreset budgets from 70 to 200. The same facility-location idea also works when the task is to pick whole benchmarks from public leaderboards that already have dense score matrices, matching or beating stronger score-driven baselines at lower compute cost. The practical upshot is that benchmark compression can start from the prompts themselves rather than from a large historical score matrix.

Core claim

On a large new suite of 35 heterogeneous benchmarks spanning five capability categories, 18 frontier LLMs, and over 61K prompts, the facility-location function that uses only inexpensive semantic prompt embeddings preserves LLM scores better than twelve score-based and diversity-based baselines across coreset budgets k in [70, 200], and the same objective matches or outperforms state-of-the-art baselines for whole-benchmark selection on MMLU and MTEB while being substantially cheaper to compute.

What carries the argument

Facility location: the submodular coverage function f(A) = sum over clients i of the maximum similarity of i to any selected facility in A, maximized greedily under a cardinality budget. Similarities are clipped cosine similarities of Qwen3-Embedding-4B vectors of (prompt + ground-truth/metadata) text, sparsified to top-t neighbors with a tuned density parameter alpha so that coverage does not saturate and memory stays linear.

Load-bearing premise

The claim rests on cosine similarity of the chosen text embeddings being a faithful enough proxy for the coverage that actually preserves per-model and per-capability scores; if that geometry misses the dimensions that drive real performance differences, the unsupervised selection guarantee collapses.

What would settle it

Re-run the same FL pipeline with a different embedding model or with deliberately corrupted prompt text (for example, stripping ground-truth fields or swapping answers across items) and check whether Cap-MRE and MRE on the held-out models rise above the random and DPP-Cov baselines across the same budget range; a large degradation would show that the present embeddings were carrying the result.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper introduces evaluation-unsupervised, prompt-level coreset selection for multi-benchmark LLM evaluation: a small subset of prompts is chosen without using model responses or scores, so that full-suite model scores (and, by claim, rankings) are approximately recovered. The authors cast selection as submodular maximization and compare facility location (FL), several DPP variants (including mutual-information forms), k-center, clustering, and score-based baselines. On a newly assembled suite of 35 heterogeneous benchmarks (five capability categories), 18 frontier models, and >61K prompts, FL on sparsified cosine similarities of Qwen3-Embedding-4B vectors of (prompt + ground-truth/metadata) attains the lowest MRE and Cap-MRE across budgets k∈[70,200], outperforming twelve baselines. The same FL objective, when applied to score-derived similarities, matches or beats DPP-style methods for whole-benchmark selection on MMLU and MTEB at lower cost. Supporting analyses cover text vs IRT embeddings, block-diagonal stratification, α sparsification, and an unsupervised benchmark–benchmark similarity via submodular mutual information.

Significance. Benchmark evaluation cost is a real bottleneck in LLM development; a method that compresses multi-benchmark suites without requiring historical model responses is practically valuable and less contaminated by leaderboard overfitting. The work contributes (i) a clear problem formulation distinguishing evaluation-unsupervised prompt-level coresets from supervised whole-benchmark selection, (ii) a large, recent multi-capability evaluation suite with frontier models, (iii) Cap-MRE as a metric that avoids cross-capability cancellation, and (iv) systematic evidence that classical facility location on inexpensive semantic embeddings is competitive with, and often superior to, score-based and DPP baselines in both regimes. The supervised MMLU/MTEB results and the SMI-based unsupervised benchmark ordering further show the objective is not confined to one setting. If the empirical findings hold under broader embedding and model regimes, the paper offers a simple, scalable default for benchmark compression.

major comments (2)
  1. Abstract and §1 claim that coresets approximate both model scores and rankings, but the primary metrics (Eqs. 4–5, Table 1, Figs. 1, 8–9) are only relative score errors (MRE, Cap-MRE). No rank correlation, top-k agreement, or pairwise ranking fidelity is reported on the 35-benchmark suite. Because cancellation of signed errors can preserve averages while distorting orderings, the ranking claim is currently unsupported by the main evaluation. Adding at least one ranking metric (e.g., Spearman/Kendall of model orderings overall and per capability) on the same folds would make the central claim match the reported evidence.
  2. §3.1–3.2 and §4.1: selection itself uses only embeddings, but the sparsification factor α is tuned by minimizing Cap-MRE on a calibration split of model scores. The pipeline is therefore not fully evaluation-unsupervised end-to-end; a default α (or a score-free heuristic for α) is not validated. Given that Fig. 7 shows α strongly affects marginal gains, the paper should either (a) report sensitivity of held-out MRE/Cap-MRE to α without re-tuning per fold, or (b) clearly bound the claim to “selection unsupervised, hyperparameter calibration lightly supervised,” and show that a fixed α chosen once still beats the baselines.
minor comments (6)
  1. Only Qwen3-Embedding-4B is used for text embeddings; a short ablation with one alternative embedding model (or frozen bag-of-words / TF-IDF) would strengthen the claim that the result is not embedding-specific (§3.1).
  2. Figure 1 normalizes MRE/Cap-MRE to [0,1] within each budget then averages; the raw per-budget curves (Figs. 8–9) are clearer for absolute error. Consider putting absolute mean±SE in the main figure or stating the normalization more prominently in the caption.
  3. Table 1 and the text sometimes report normalized aggregate MRE as 0.008 (abstract/Fig. 1) and elsewhere raw MRE ~0.05 at k=80; keep units consistent when citing “best overall” numbers.
  4. NeuMF imputation (§4.1, Eq. 6) fills 4.29% missing entries; a brief note on whether coreset fidelity changes if missing entries are left out or mean-imputed would help readers assess sensitivity.
  5. Appendix B embedding field list is useful; a one-sentence statement in the main text that ground-truth/metadata is concatenated (and that this is still model-response-free) would prevent misreading as leakage of model scores.
  6. Minor typos/notation: “force majeure factors” is informal; “evalutaion” in C.4; B(A) vs script-B consistency; and “SE-aware mean average rank” could be defined formally once.

Circularity Check

0 steps flagged

No significant circularity: unsupervised FL selection uses only prompt embeddings; scores appear solely for α tuning on calibration folds and held-out fidelity evaluation.

full rationale

The paper's central claim is empirical, not a first-principles derivation that could collapse into its inputs. Facility location (Eq. 1–2) is maximized over a cosine similarity matrix built exclusively from Qwen3-Embedding-4B vectors of (prompt + ground-truth/metadata) text; model scores never enter the selection objective. Scores are used only (i) to tune the sparsification hyper-parameter α on a calibration split of models and (ii) to compute MRE/Cap-MRE on held-out models under 3-fold CV—standard supervised evaluation of an unsupervised selector, not a circular fit. The secondary MMLU/MTEB experiments explicitly switch to a score-derived similarity matrix and compare against DPP baselines on the same matrix; this is an additional supervised application, not a re-derivation of the unsupervised result. Classic submodular guarantees (Nemhauser et al.) and Bilmes software are cited for the greedy algorithm, but no uniqueness theorem or self-citation is load-bearing for the fidelity claims. No equation reduces the reported error numbers to the selection inputs by construction. Minor self-citation of the authors' own submodularity survey is normal background and does not force the empirical outcome.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

The central empirical claim rests on classical submodular maximization guarantees, the modeling choice that embedding cosine similarity (after sparsification) proxies evaluation coverage, a small number of tunable hyper-parameters (chiefly α), and standard experimental assumptions about missing-score imputation and cross-validation. No new physical entities or free constants are invented; the free parameters are ordinary algorithmic knobs.

free parameters (3)
  • sparsification factor α
    Controls top-t retention t=⌈α n/k⌉ of the similarity matrix; tuned on calibration folds to avoid both under-coverage and early saturation (Section 3.1, Figure 7).
  • embedding model (Qwen3-Embedding-4B) and max length 8192
    Choice of which embedding model and truncation length determines the geometry used by FL and all embedding-based baselines; not ablated against alternatives.
  • NeuMF imputation hyper-parameters
    Missing 4.29 % of the model–prompt matrix are filled by a trained NeuMF model whose architecture and training details affect both the evaluation matrix and the IRT-embedding baselines.
axioms (4)
  • standard math Monotone submodular maximization under cardinality constraint admits a (1−1/e)-approximation via greedy (Nemhauser et al. 1978)
    Invoked to justify the greedy algorithm used for FL and related objectives (Section 3.1 and Appendix A).
  • domain assumption Clipped cosine similarity of prompt embeddings is a non-negative similarity that makes facility location submodular and a useful coverage proxy for score preservation
    Core modeling choice of Section 3.1; if the embedding geometry is misaligned with performance-relevant dimensions the unsupervised claim fails.
  • ad hoc to paper Block-diagonal (benchmark-stratified) similarity is a reasonable default that prevents concentration on a single large benchmark
    Design choice in Section 3.1; ablated in Figure 3 but still the primary reported setting.
  • domain assumption Cap-MRE and MRE on held-out models are faithful measures of coreset quality
    Defined in Section 3.2 and used as the sole ranking criteria for all methods.
invented entities (2)
  • evaluation-unsupervised benchmark coreset selection (prompt-level, multi-benchmark) no independent evidence
    purpose: Names the problem setting in which selection uses no model scores or responses
    Framing device that organizes the paper; not a physical entity, but a newly emphasized regime relative to prior score-dependent coresets.
  • Cap-MRE (mean capability relative error) no independent evidence
    purpose: Metric that averages absolute relative error inside each capability before averaging across capabilities, avoiding cancellation
    Defined in Eq. (4); useful but paper-specific aggregation choice.

pith-pipeline@v1.1.0-grok45 · 30322 in / 3131 out tokens · 31079 ms · 2026-07-14T16:37:19.596278+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Coresets Before Score Sets: Evaluation-Unsupervised Prompt Subset Selection for LLM Benchmarks." pith.science (2026). https://pith.science/paper/5V2BQF7F

@misc{pith2026260709739,
  author       = {Pith},
  title        = {Pith review of: Coresets Before Score Sets: Evaluation-Unsupervised Prompt Subset Selection for LLM Benchmarks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5V2BQF7F}},
  note         = {Machine review of arXiv:2607.09739}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We study LLM benchmark coreset selection: selecting a small subset of prompts over multiple benchmarks whose induced model scores and rankings approximate those obtained from the full benchmark suite. In evaluation-unsupervised benchmark coreset selection (our approach), the selection algorithm uses no model evaluation outcomes, and operates on a fine granularity by producing subsets of prompts over multiple benchmarks rather than producing a sub-collection of entire benchmarks. We use submodular subset selection, and we develop and evaluate many different submodular functions for this purpose, including determinantal point process (DPP) based approaches, submodular mutual information functions, and facility location-based functions. On a new large-scale suite of 35 heterogeneous benchmarks spanning five different capability categories, 18 frontier LLMs, and over 61K prompts, we find that the facility location (FL) function operating exclusively on inexpensive semantic prompt embeddings preserves LLM scores better than twelve separate score-based and diversity-based baselines, across a range of coreset budgets. Moreover, we show our proposed objective is not limited to the evaluation-unsupervised regime: in the setting where only a handful of whole benchmarks must be selected and a large amount of model scores are available, the same objective matches or outperforms state-of-the-art baselines on the MMLU and MTEB leaderboards, while being substantially cheaper to compute. Together, our results suggest that submodularity, in general, is a strong and reliable tool for benchmark compression.

Figures

Figures reproduced from arXiv: 2607.09739 by Arnav Das, Banghua Zhu, Chang Su, Damon Mosk-Aoyama, Gantavya Bhatt, Jeff Bilmes, Jialei Wang, Jiantao Jiao, Jihan Yao, Ke Bao, Khushi Bhardwaj, Oleksii Kuchaiev, Peter Jin, Qiaolin Yu, Sugam Devare, Venkat Krishna Srinivasan, Yikai Zhu, Yineng Zhang, Zhen Dong.

Figure 1
Figure 1. Figure 1: The lower x-axis reports mean normalized MRE and mean normalized Cap-MRE, where scores are linearly normalized to [0, 1] within each budget and then averaged across all budgets k ∈ [70, 200]. The upper x-axis reports the SE-aware mean average rank jointly over MRE and Cap-MRE, where a method is ranked above another only when their error intervals do not overlap. Lower values indicate better performance acr… view at source ↗
Figure 2
Figure 2. Figure 2: FL with text embeddings vs. FL with IRT em [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: FL with block-diagonal similarity (ours) vs. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Benchmark-level selection on MMLU and MTEB (same protocol as (Smola, 2026)), with FL run on the identical score-derived similarity matrix Σ used by greedy entropy and greedy MI. FL matches or beats both DPP-style objectives almost everywhere, and dom￾inates throughout on MTEB according to mean test R2 . 3 6 9 12 15 18 k (benchmarks selected) 10 1 Residual variance fraction MMLU 3 6 9 12 15 18 k (benchmarks… view at source ↗
Figure 5
Figure 5. Figure 5: Benchmark-level selection on MMLU and MTEB (same protocol as (Smola, 2026)), with FL run on the identical score-derived similarity matrix Σ used by greedy entropy and greedy MI. FL achieves the low￾est residual variance fraction across the entire range on both leaderboards according to mean residual variance fraction. Σˆ is estimated on the training split and used for both selection and imputation across a… view at source ↗
Figure 6
Figure 6. Figure 6: Normalized benchmark–benchmark similar￾ity matrix M (Eq. 12), computed purely from prompt embeddings. A bright entry Mbb′ indicates that bench￾marks b and b ′ redundantly cover the same region of embedding space; this matrix is the similarity used by the benchmark-level facility-location greedy that pro￾duces the ordering in [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Marginal gains of the greedy selection across [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Per-budget overall relative error (MRE) for all methods across budgets [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Per-budget capability-wise relative error (Cap-MRE) for all methods across budgets [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Benchmark-level selection on MMLU and MTEB (same protocol as ( [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Benchmark-level selection on MMLU and MTEB (same protocol as ( [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 8 linked inside Pith

  1. [1]

    Arnav Mohanty Das, Gantavya Bhatt, Sahil Verma, Yip- ing Wang, Viswa Virinchi Muppirala, and Jeff Bilmes

    Combinatorial optimization.Oberwolfach Rep., 5(4):2875–2942. Arnav Mohanty Das, Gantavya Bhatt, Sahil Verma, Yip- ing Wang, Viswa Virinchi Muppirala, and Jeff Bilmes

  2. [2]

    InThe Four- teenth International Conference on Learning Repre- sentations

    Matched data, better models: Target aligned data filtering with sparse autoencoders. InThe Four- teenth International Conference on Learning Repre- sentations. DatologyAI Team. 2026. Discriminative, faith- ful, and efficient VLM evaluations.Preprint, arXiv:2601.02316. Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Ger- stein, and Arman Cohan. 2024. Investi...

    arXiv 2026

  3. [3]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Ja- cob Steinhardt

    Measuring massive multitask language under- standing.arXiv preprint arXiv:2009.03300. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Ja- cob Steinhardt. 2021. Measuring mathematical prob- lem solving with the math dataset.arXiv preprint arXiv:2103.03874. Zhongzhan Huang, Guoming Ling, Shanshan Zhong, Hef...

    Pith/arXiv arXiv 2009

  4. [4]

    Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Au- mayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, and 1 others

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437. Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Au- mayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, and 1 others. 2024. Toolsand- box: A stateful, conversational, interactive evalua- tion benchmark for llm tool use capabilities.arXiv preprint arXiv:2408.04682. Adyasha Mah...

    Pith/arXiv arXiv 2024

  5. [5]

    Mike A Merrill, Alexander G Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E Kelly Buchanan, and 1 others

    D2 pruning: Message passing for balancing diversity and difficulty in data pruning.Preprint, arXiv:2310.07931. Mike A Merrill, Alexander G Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E Kelly Buchanan, and 1 others. 2026. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line ...

    Pith/arXiv arXiv 2026

  6. [6]

    InProceedings of the AAAI Conference on Artificial Intelligence, 1

    Lazier than lazy greedy. InProceedings of the AAAI Conference on Artificial Intelligence, 1. Baharan Mirzasoleiman, Jeff A. Bilmes, and Jure Leskovec. 2019. Data sketching for faster training of machine learning models.CoRR, abs/1906.01827. Aidar Myrzakhan, Sondos Mahmoud Bsharat, and Zhiqiang Shen. 2024. Open-llm-leaderboard: From multi-choice to open-st...

    Pith/arXiv arXiv 2019

  7. [7]

    Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hanna Hajishirzi

    tinybenchmarks: evaluating llms with fewer examples.arXiv preprint arXiv:2402.14992. Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hanna Hajishirzi. 2026. Gener- alizing verifiable instruction following.Advances in Neural Information Processing Systems, 38. Valentina Pyatkin, Saumya Malik...

    Pith/arXiv arXiv 2026

  8. [8]

    Maxim K Surkov and Ivan P Yamshchikov

    Simba: Simplifying benchmark analy- sis using performance matrices alone.Preprint, arXiv:2510.17998. Maxim K Surkov and Ivan P Yamshchikov. 2024. Vy- gotsky distance: measure for benchmark task similar- ity. InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 17...

    arXiv 2024

  9. [9]

    Nexusflow.ai team

    Kimi k2: Open agentic intelligence.arXiv preprint arXiv:2507.20534. Nexusflow.ai team. 2024. Nexusbench: Fc and agent benchmarking suite. The Terminal-Bench Team. 2025. Terminal-bench: A benchmark for ai agents in terminal environments. Rajan Vivek, Kawin Ethayarajh, Diyi Yang, and Douwe Kiela. 2024. Anchor points: Benchmarking models with much fewer exam...

    Pith/arXiv arXiv 2024

  10. [10]

    Zhexu Wang, Yiping Liu, Yejie Wang, Wenyang He, Bofei Gao, Muxi Diao, Yanxu Chen, Kelin Fu, Flood Sung, Zhilin Yang, and 1 others

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.Ad- vances in Neural Information Processing Systems, 37:95266–95290. Zhexu Wang, Yiping Liu, Yejie Wang, Wenyang He, Bofei Gao, Muxi Diao, Yanxu Chen, Kelin Fu, Flood Sung, Zhilin Yang, and 1 others. 2025. Ojbench: A competition level code benchmark for large language mode...

    arXiv 2025

  11. [11]

    In2014 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pages 4107–4111

    Unsupervised submodular subset selection for speech data. In2014 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pages 4107–4111. IEEE. Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxing Xu, and 1 others. 2024. Bench- marking complex instruction-following with...

    Pith/arXiv arXiv 2024

  12. [12]

    Prompt” refers to the original prompt field name used to construct the prompt shown to the model. “Target

    Qwen3 technical report.arXiv preprint arXiv:2505.09388. Jihan Yao, Peter Jin, Ke Bao, Qiaolin Yu, Khushi Bhard- waj, Chang Su, Jialei Wang, Yikai Zhu, Sugam De- vare, Damon Mosk-Aoyama, and 1 others. 2025. The measure of all measures: Quantifying llm benchmark quality. InNeurIPS 2025 Workshop on Evaluating the Evolving LLM Lifecycle: Benchmarks, Emergent ...

    Pith/arXiv arXiv 2025