Pith. sign in

REVIEW 2 major objections 6 minor 44 references

Sequential testing stops model evaluation once statistical needs are met, cutting cost without losing reliability.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 06:01 UTC pith:3O3XWA7W

load-bearing objection Solid packaging of classical sequential testing for expensive model eval, with real savings numbers on a large VLM leaderboard; immediately usable and worth engaging. the 2 major comments →

arxiv 2607.08522 v1 pith:3O3XWA7W submitted 2026-07-09 cs.LG

Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data

Ofir Arviv , Kristjan Greenewald , Yotam Perlitz , Hadar Mulian , Michal Shmueli-Scheuer , Leshem Choshen This is my paper
classification cs.LG
keywords sequential testingmodel evaluationgroup sequential designadaptive stoppingbenchmark efficiencyconfidence intervalsVLM leaderboard
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Fixed-size benchmarks force the same sample budget on every evaluation goal, so teams either waste compute after the answer is already clear or cut samples and lose statistical guarantees. This paper argues that group sequential testing, long used in clinical trials, should replace that rigidity. It supplies an adaptive framework that peeks at partial results under a Pocock spending rule and stops when a user-chosen criterion is met: confidence-interval width, efficacy, equivalence margin, diminishing returns, futility, or a performance threshold. On the Open VLM Leaderboard the method recovers an 80 percent sample reduction for a ±2.5-point CI allowance while still controlling Type-I error, and similar savings appear for pairwise ranking and early filtering of weak checkpoints. The result is that efficiency becomes an explicit, auditable choice rather than a hidden heuristic.

Core claim

An adaptive evaluation framework built on group sequential testing (Pocock spending function) plus practical stopping rules can cut evaluation cost by roughly 80 percent relative to a fixed-size benchmark while still delivering the statistical significance or confidence-interval precision the user requested, as shown on the Open VLM Leaderboard.

What carries the argument

Group sequential testing with the Pocock spending function: the evaluation is partitioned into pre-specified interim looks; at each look a constant critical value is applied so that the overall Type-I error stays at the target α, and any of several user-defined stopping rules may then halt sampling.

Load-bearing premise

The test statistic at every interim look is treated as approximately normal and the observations are treated as independent, so that the pre-chosen Pocock critical values still control the overall false-positive rate.

What would settle it

Run the same Open VLM Leaderboard comparisons under both the sequential procedure and a fixed-size test of equal total sample size; if the sequential procedure exceeds the nominal Type-I error or fails to produce the claimed sample savings while matching the fixed-size decisions, the central claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper argues that fixed-size benchmarks are inefficient for modern LLM/VLM evaluation because different objectives (ranking, selection, development debugging) require different statistical power. It proposes an adaptive evaluation framework based on group sequential testing (Pocock spending function) that stops when user-specified practical criteria are met while preserving Type-I error control. Stopping rules include efficacy, equivalence-margin, precision/MDES, threshold-crossing, futility, and diminishing-returns. Experiments on the Open VLM Leaderboard (206 models, 31 datasets) quantify large savings: ~80% fewer samples for a ±2.5-point CI half-width, 44% under diminishing returns, and 30–63% in three realistic case studies (compute-constrained ranking, meaningful-change deployment, candidate filtering), all while maintaining statistical significance under the stated assumptions.

Significance. If the reported savings hold under the usual sequential-testing assumptions, the work supplies a practical, statistically grounded alternative to both full fixed-size evaluation and ad-hoc sub-sampling. The combination of classical GST with evaluation-specific stopping rules (especially diminishing returns and MDES) is immediately usable by practitioners and is accompanied by concrete numbers on a public leaderboard. Code release is promised; the statistical machinery is standard and correctly cited. The contribution is therefore of clear practical value to the large-model evaluation community and strengthens the case for sequential methods in the field.

major comments (2)
  1. §5.2 and §6.2: when ranking or selecting among many models the framework reduces to pairwise tests and explicitly defers multiple-testing correction. For the top-50 and top-15 experiments this leaves family-wise error uncontrolled; a brief quantification (e.g., Bonferroni or sequential FDR) of how the reported sample savings change under a joint error guarantee would make the multi-model claims fully rigorous.
  2. §5.2: the Pocock critical values and Type-I control rest on approximate normality of Z_k (via CLT) and a pre-specified maximum number of looks t. With the chosen batch size of 100 the CLT is plausible, yet no diagnostic (QQ plots, empirical Type-I rates under permutation of labels, or sensitivity to smaller batches) is supplied. A short verification would strengthen confidence that the 80%/44% savings figures remain valid under the paper’s own assumptions.
minor comments (6)
  1. Abstract and §1 claim an 80% reduction for a 2.5-point CI width; Figure 1 caption and §7.1 use both ±2.5 and ±3. Align the exact threshold throughout.
  2. Contributions list in §1 jumps from 2) to 4); renumber.
  3. Eq. (6) writes the denominator as sqrt(σ̂^{2}_k · (1/N_k)); for a two-sample difference the usual form is sqrt(σ̂^{2}_A/n_A + σ̂^{2}_B/n_B). Clarify whether equal sample sizes and a pooled variance are assumed.
  4. §6.1: the LLM-as-Judge (Llama 3.1 8B) used to obtain scores from predictions is itself stochastic; a short note on seed control or variance of the judge would be helpful.
  5. Figure 2 x-axis is performance difference; a secondary axis or color indicating absolute scores would aid interpretation of the equivalence-margin region.
  6. Limitations paragraph correctly notes i.i.d. and reproducibility issues; a one-sentence pointer to non-parametric sequential methods (e.g., Bibaut et al. already cited) that relax normality would be a useful addition.

Circularity Check

0 steps flagged

No significant circularity; standard GST applied with empirical savings measured on external leaderboard data.

full rationale

The paper applies classical group sequential testing (Pocock spending function from the literature) plus user-specified stopping rules (precision/MDES, efficacy, equivalence margin, diminishing returns, futility) to model evaluation. Statistical guarantees rest on stated standard assumptions (approx. normality of Z_k via CLT for the chosen batch sizes, independent observations, pre-specified maximum looks t) that are not derived from the paper's own data or definitions. Reported efficiency numbers (e.g., 80% sample reduction for a ±2.5-point CI half-width, 44% under diminishing-returns stopping, pairwise savings under equivalence margins) are obtained by running the procedure on the external Open VLM Leaderboard records and comparing against fixed-size baselines; they are not forced by construction from any fitted parameter. Self-citations (Perlitz et al., Choshen et al., etc.) appear only in the related-work discussion of efficient evaluation heuristics and do not supply load-bearing uniqueness theorems or ansätze for the sequential-testing core. No equation reduces to its own input, no fitted quantity is relabeled a prediction, and no uniqueness claim is imported from the authors' prior work. The framework is therefore self-contained.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claim rests on classical sequential-testing theory plus a handful of implementation defaults and the usual i.i.d./normality assumptions of group sequential designs. No new physical entities are invented; free parameters are the usual design knobs of sequential testing and the user-chosen practical thresholds.

free parameters (4)
  • initial sample size b_init = 600
    Set to 600 (100 per model) as a reasonable default; not tuned.
  • batch size b = 100
    Set to 100 per iteration; controls frequency of interim looks and power loss.
  • beta (Type-II error) = 0.9
    Set to 0.9; affects power calculations in the design.
  • equivalence margin / CI half-width targets = ±2 or ±2.5 points (examples)
    User-chosen practical thresholds (e.g., ±2 or ±2.5 points) that directly determine when stopping occurs; different values change reported savings.
axioms (4)
  • domain assumption Observations are independent (or can be treated as such for the purpose of the variance estimator).
    Required for the sequential test statistic and Type-I error control; stated in §5.2.
  • standard math The test statistic Z_k is approximately normal under the null for sufficiently large cumulative sample size (CLT).
    Justifies the critical values of the Pocock design; §5.2.
  • domain assumption Maximum number of interim analyses t is pre-specified.
    Needed for the spending function to control overall α; §5.2.
  • domain assumption Mean-of-means aggregation across heterogeneous datasets is an appropriate overall score.
    Used throughout the Open VLM Leaderboard experiments; Eq. (1).

pith-pipeline@v1.1.0-grok45 · 17209 in / 2537 out tokens · 48747 ms · 2026-07-10T06:01:11.500471+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data." pith.science (2026). https://pith.science/paper/3O3XWA7W

@misc{pith2026260708522,
  author       = {Pith},
  title        = {Pith review of: Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3O3XWA7W}},
  note         = {Machine review of arXiv:2607.08522}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The inherent rigidity of fixed-size benchmarks makes them an inefficient tool for model evaluation. Diverse evaluation objectives, including model ranking, model selection and testing throughout development, demand varying levels of statistical power. The mismatch between fixed sample sizes and these diverse needs results in either excessive computational cost or compromised reliability - a critical concern for model evaluation. To overcome these limitations, we call for adoption of sequential testing in our field. We provide an adaptive evaluation framework, that provides a principled way to navigate the trade-off between efficiency and reliability in model evaluation. Our framework combines the established statistical paradigm of sequential testing with stopping criteria tailored to common evaluation needs such as diminishing returns detection, and minimum detectable effect size. We demonstrate its ability to adaptively manage the efficiency-reliability trade-off on the Open VLM Leaderboard, including, for example, a 80% reduction in computational cost compared to fixed-size evaluation (with a 2.5-point CI width allowance) while maintaining statistical significance.

Figures

Figures reproduced from arXiv: 2607.08522 by Hadar Mulian, Kristjan Greenewald, Leshem Choshen, Michal Shmueli-Scheuer, Ofir Arviv, Yotam Perlitz.

Figure 1
Figure 1. Figure 1: Half-width of the confidence interval as a [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Required sample size (%) for model comparison with an equivalence margin of 2 points, as a function of [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 44 canonical work pages · 2 internal anchors

  1. [1]

    2024 , eprint=

    Efficient Benchmarking of Language Models , author=. 2024 , eprint=

    work page 2024

  2. [2]

    2024 , eprint=

    From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline , author=. 2024 , eprint=

    work page 2024

  3. [3]

    2024 , eprint=

    Efficient Prompt Optimization Through the Lens of Best Arm Identification , author=. 2024 , eprint=

    work page 2024

  4. [4]

    2023 , eprint=

    A Survey on Evaluation of Large Language Models , author=. 2023 , eprint=

    work page 2023

  5. [5]

    The Hitchhiker`s Guide to Testing Statistical Significance in Natural Language Processing

    Dror, Rotem and Baumer, Gili and Shlomov, Segev and Reichart, Roi. The Hitchhiker`s Guide to Testing Statistical Significance in Natural Language Processing. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018. doi:10.18653/v1/P18-1128

    work page doi:10.18653/v1/p18-1128 2018

  6. [6]

    2021 , eprint=

    Active Testing: Sample-Efficient Model Evaluation , author=. 2021 , eprint=

    work page 2021

  7. [7]

    Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024): Tutorial Summaries , pages=

    Navigating the modern evaluation landscape: Considerations in benchmarks and frameworks for large language models (LLMs) , author=. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024): Tutorial Summaries , pages=

    work page 2024

  8. [8]

    2025 , eprint=

    How to Select Datapoints for Efficient Human Evaluation of NLG Models? , author=. 2025 , eprint=

    work page 2025

  9. [9]

    2024 , eprint=

    Anchor Points: Benchmarking Models with Much Fewer Examples , author=. 2024 , eprint=

    work page 2024

  10. [10]

    Annual Meeting of the Association for Computational Linguistics , year=

    Benchmarking Large Language Model Capabilities for Conditional Generation , author=. Annual Meeting of the Association for Computational Linguistics , year=

    work page

  11. [11]

    Annals of Mathematical Statistics , year=

    Sequential Tests of Statistical Hypotheses , author=. Annals of Mathematical Statistics , year=

    work page

  12. [12]

    Biometrika , year=

    Group sequential methods in the design and analysis of clinical trials , author=. Biometrika , year=

    work page

  13. [13]

    Biometrika , year=

    Discrete sequential boundaries for clinical trials , author=. Biometrika , year=

    work page

  14. [14]

    1999 , url=

    Group Sequential Methods with Applications to Clinical Trials , author=. 1999 , url=

    work page 1999

  15. [15]

    2024 , eprint=

    Near-Optimal Non-Parametric Sequential Tests and Confidence Sequences with Possibly Dependent Observations , author=. 2024 , eprint=

    work page 2024

  16. [16]

    ArXiv , year=

    LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models , author=. ArXiv , year=

    work page

  17. [17]

    ArXiv , year=

    A Survey on Benchmarks of Multimodal Large Language Models , author=. ArXiv , year=

    work page

  18. [18]

    Springer Series in Pharmaceutical Statistics , year=

    Group Sequential and Adaptive Designs , author=. Springer Series in Pharmaceutical Statistics , year=

    work page

  19. [19]

    MMBench: Is Your Multi-modal Model an All-around Player? , year =

    Yuan Liu and Haodong Duan and Yuanhan Zhang and Bo Li and Songyang Zhang and Wangbo Zhao and Yike Yuan and Jiaqi Wang and Conghui He and Ziwei Liu and Kai Chen and Dahua Lin , journal =. MMBench: Is Your Multi-modal Model an All-around Player? , year =

    work page

  20. [20]

    An introduction to group sequential methods: planning and multi-aspect optimization

    An introduction to group sequential methods: planning and multi-aspect optimization , author=. arXiv preprint arXiv:2303.01040 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  21. [21]

    ArXiv , year=

    Can We Predict Performance of Large Models across Vision-Language Tasks? , author=. ArXiv , year=

    work page

  22. [22]

    A ctive L earning for BERT : A n E mpirical S tudy

    Ein-Dor, Liat and Halfon, Alon and Gera, Ariel and Shnarch, Eyal and Dankin, Lena and Choshen, Leshem and Danilevsky, Marina and Aharonov, Ranit and Katz, Yoav and Slonim, Noam. A ctive L earning for BERT : A n E mpirical S tudy. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020. doi:10.18653/v1/2020.emnl...

    work page doi:10.18653/v1/2020.emnlp-main.638 2020

  23. [23]

    AdaStop: adaptive statistical testing for sound comparisons of Deep RL agents

    AdaStop: adaptive statistical testing for sound comparisons of Deep RL agents , author=. arXiv preprint arXiv:2306.10882 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  24. [24]

    2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=

    MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI , author=. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=

    work page 2024

  25. [25]

    Science China Information Sciences , year=

    OCRBench: on the hidden mystery of OCR in large multimodal models , author=. Science China Information Sciences , year=

    work page

  26. [26]

    ArXiv , year=

    Label-Efficient Model Selection for Text Generation , author=. ArXiv , year=

    work page

  27. [27]

    ArXiv , year=

    A Framework for Efficient Model Evaluation through Stratification, Sampling, and Estimation , author=. ArXiv , year=

    work page

  28. [28]

    ArXiv , year=

    Efficient multi-prompt evaluation of LLMs , author=. ArXiv , year=

    work page

  29. [29]

    ArXiv , year=

    AutoRAG-HP: Automatic Online Hyper-Parameter Tuning for Retrieval-Augmented Generation , author=. ArXiv , year=

    work page

  30. [30]

    ArXiv , year=

    tinyBenchmarks: evaluating LLMs with fewer examples , author=. ArXiv , year=

    work page

  31. [31]

    2016 , eprint=

    A Diagram Is Worth A Dozen Images , author=. 2016 , eprint=

    work page 2016

  32. [32]

    2024 , eprint=

    Are We on the Right Way for Evaluating Large Vision-Language Models? , author=. 2024 , eprint=

    work page 2024

  33. [33]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =

    Guan, Tianrui and Liu, Fuxiao and Wu, Xiyang and Xian, Ruiqi and Li, Zongxia and Liu, Xiaoyu and Wang, Xijun and Chen, Lichang and Huang, Furong and Yacoob, Yaser and Manocha, Dinesh and Zhou, Tianyi , title =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =. 2024 , pages =

    work page 2024

  34. [34]

    Efficient Benchmarking (of Language Models)

    Perlitz, Yotam and Bandel, Elron and Gera, Ariel and Arviv, Ofir and Ein-Dor, Liat and Shnarch, Eyal and Slonim, Noam and Shmueli-Scheuer, Michal and Choshen, Leshem. Efficient Benchmarking (of Language Models). Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (V...

    work page doi:10.18653/v1/2024.naacl-long.139 2024

  35. [35]

    Proceedings of the 32nd ACM International Conference on Multimedia , pages=

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models , author=. Proceedings of the 32nd ACM International Conference on Multimedia , pages=

    work page

  36. [36]

    FirstName LastName , title =

    work page

  37. [37]

    FirstName Alpher , title =

    work page

  38. [38]

    Journal of Foo , volume = 13, number = 1, pages =

    FirstName Alpher and FirstName Fotheringham-Smythe , title =. Journal of Foo , volume = 13, number = 1, pages =

    work page

  39. [39]

    Journal of Foo , volume = 14, number = 1, pages =

    FirstName Alpher and FirstName Fotheringham-Smythe and FirstName Gamow , title =. Journal of Foo , volume = 14, number = 1, pages =

    work page

  40. [40]

    FirstName Alpher and FirstName Gamow , title =

    work page

  41. [41]

    Journal of the American Statistical Association , volume=

    Sequential method of sampling for deciding between two courses of action , author=. Journal of the American Statistical Association , volume=. 1945 , publisher=

    work page 1945

  42. [42]

    1999 , publisher=

    Group sequential methods with applications to clinical trials , author=. 1999 , publisher=

    work page 1999

  43. [43]

    Wald , title =

    A. Wald , title =. The Annals of Mathematical Statistics , number =. 1945 , doi =

    work page 1945

  44. [44]

    , author=

    A multiple testing procedure for clinical trials. , author=. Biometrics , year=

    work page