Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Confidence in Large Language Model Evaluation: A Bayesian Approach to Limited-Sample Challenges

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a Bayesian interval posterior over anchor models yields statistically robust probabilistic LLM rankings with as few as 20 curated questions, resolving ambiguities that scalar metrics leave tied.

desk verdict A useful Bayesian interval-ranking idea, undermined by an internally inconsistent multi-trial likelihood (Eq. 6) that all the headline Bayes@10 results depend on. read the letter →

arxiv 2504.21303 v1 pith:E2W7POCQ submitted 2025-04-30 cs.CL

Xiao Xiao , Yu Su , Sijing Zhang , Zhang Chen , Yadong Chen , Tian Liu This is my paper · ORCID
classification cs.CL
keywords BayesianevaluationLLMrankinglimitedsamplecapabilityintervalsanchormodelsmaximumentropyprobabilistichypothesistesting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that LLM evaluation should be framed as Bayesian inference over capability intervals rather than as a scalar accuracy score. Treating each model's latent capability as an unknown parameter, the method places a test model among a set of anchor models whose per-question success rates are measured in advance. The paper claims that with about 20 curated questions the resulting posterior yields stable, interpretable probability statements—for instance, a model's chance of falling between GPT-4 and o1—and that these rankings separate models that traditional metrics tie, such as QwQ-32B and Llama-4-Maverick. A reader should care because the approach offers calibrated confidence in small-sample, real-world evaluation settings.

What carries the argument

The load-bearing object is the interval posterior $\Pr(\theta_i < \theta_x \le \theta_{i+1} \mid \mathcal{Q}=\mathbf{q})$, which assigns the test model a probability of falling into each gap between consecutive anchor capabilities. The key identity is Eq. 4: the likelihood of a single answer given that the test model lies in interval $(\theta_i,\theta_{i+1}]$ is taken to be the arithmetic mean of the boundary anchor probabilities, justified by maximum entropy as the most neutral choice when $\theta$ is assumed uniform inside the interval. The prior on the interval is then proportional to its width $\theta_{i+1}-\theta_i$ (Eq. 2), and the multi-trial likelihood (Eqs. 6 and 7) incorporates repeated Bernoulli draws. This machinery converts measured anchor success rates into a calibrated posterior over rankings.

What would settle it

For each GPT-series anchor, hold that anchor out as a test model, rebuild the posterior from the remaining five anchors, and check whether the true interval is the modal one; a systematic failure identifies the midpoint assumption as the culprit.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that the Bayesian interval posterior of Eq. 1, coupled with the maximum-entropy likelihood of Eq. 4 and its multi-trial generalization in Eqs. 6 and 7, provides statistically robust probabilistic rankings down to about 20 evaluation questions. The method treats the answer pattern on a curated query set as a binary readout of latent capability, and produces a probability distribution over seven intervals defined by six GPT-series anchors. Empirically, the paper reports that this procedure resolves performance ambiguities that accuracy, Pass@N, and mean-plus-standard-error reporting cannot, including separating QwQ-32B from Llama-4-Maverick despite overlapping mean estimates. It further claims that the posterior remains stable at 20 questions while degrading below 10, and that adding a test model to the anchor set refines the ranking of its neighbors.

Load-bearing premise

Eq. 4 assumes that inside each capability interval the probability that a test model answers correctly is the simple average of the two neighboring anchor probabilities, which requires that correctness varies linearly with the latent capability within each interval.

Editorial extensions

If this is right

  • Probabilistic ranking statements of the form 'the test model has a P% chance of lying between anchor i and i+1' remain decisive at M=20, with at least 65% posterior mass on the most likely interval.
  • The method separates models that scalar metrics tie, including QwQ-32B from Llama-4-Maverick, which accuracy and mean±std treat as comparable.
  • Multi-trial Bayesian reports are sharper than single-trial ones and keep the same category boundaries, while Pass@10 inflates accuracy and hides differences like Llama-4's inferiority to QwQ-32B.
  • Adding a test model to the anchor set can refine the ranking of nearby models, e.g., incorporating QwQ-32B would disambiguate it from DeepSeek-V3-0324.
  • The approach remains stable at M=20 questions and degrades only below M=10, where the peak interval probability falls below 50%.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test the linearity of Eq. 4 directly; holding out anchor models and rebuilding the posterior would reveal whether the midpoint likelihood is the cause of misranking arms.
  • The prior is proportional to interval width, so anchor models clustered in capability space will produce weak priors; anchor selection is therefore part of the statistical model, not just experimental design.
  • The same binary-readout machinery could apply to other evaluations where anchor performance profiles are measurable, such as code correctness or preference judgments, with the curated query set replaced by the relevant task distribution.
  • The conditional-independence caveat raised in the limitations section could be checked empirically by grouping the 50 questions by domain and seeing whether posterior intervals shift when a domain is dropped.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a Bayesian method for ranking LLMs into capability intervals defined by a set of anchor models. The method treats latent capability as a continuous variable with a uniform prior across anchor-defined intervals, models per-question responses as conditionally independent Bernoulli trials, and claims that the resulting posterior probabilities provide statistically robust rankings even with as few as 20 questions. The authors evaluate the method on five open-weight test models using GPT-series anchors and a curated 50-question set, comparing it with accuracy, Pass@N, and mean-plus-standard-deviation reporting. The central claim is that the Bayesian approach yields enhanced discriminative capability and stable analytical outcomes under limited sample sizes.

Significance. The paper addresses a real and practically important problem: how to make probabilistic, uncertainty-aware claims about LLM rankings when only a small number of evaluation questions can be afforded. The proposed framework is conceptually attractive, and the detailed query set and anchor-model protocol are useful contributions. However, the core multi-trial likelihood (Eq. 6) is not the marginal likelihood of the described model, and because all reported experiments use O=10 trials, the primary empirical claims rest on an internally inconsistent derivation. The paper also does not validate the key linearity assumption in Eq. 4 or the calibration of the reported posterior probabilities. If the likelihood is corrected and the assumptions are validated empirically, the method could be a useful addition to the LLM-evaluation toolkit; as it stands, the central claim is not supported.

major comments (4)
  1. [Section 3.4, Eq. (6)] Equation (6) is not the marginal likelihood of the model stated in Section 2.3. Under the uniform-θ prior in Eq. (2) and the linear-response justification of Eq. (4), the probability of observing K successes in O trials given that θ lies in an interval should be the integral of the binomial likelihood over θ, not the average of the two endpoint binomial likelihoods. Eq. (6) instead corresponds to a different generative model in which the test model behaves exactly like L_i for all O trials with probability 1/2 and like L_{i+1} otherwise. The discrepancy is not negligible: for p_i=0.2, p_{i+1}=0.8, O=2, K=1, Eq. (6) gives 0.32 while the exact uniform-θ integral gives 0.44. Since Figures 2 and 3 and all Bayes@10 comparisons use O=10, every reported multi-trial posterior is computed from a likelihood that is inconsistent with the paper's own model description. This directly undermines the central claim of statistically robust probabilistic rankings.
  2. [Section 2.3, Eq. (4)] The assertion that the probability of a correct response is linear in θ within each interval is presented as a consequence of maximum entropy, but maximum entropy over θ only justifies the uniform prior in Eq. (2); it does not imply linearity of the item-response function. If the true p_j(θ) is nonlinear, Eq. (4) is biased and the resulting posterior intervals will be miscalibrated. The paper does not provide a derivation of this linearity, nor does it check it against the anchor-model data (e.g., by testing whether success probabilities vary roughly linearly across the anchor points). The limitations section acknowledges only conditional query independence, not this load-bearing assumption, so the assumption's status is unclear.
  3. [Sections 2.3 and 3.3] The anchor capability values θ_i and the per-question anchor probabilities Pr(Q_j|L_i) are estimated from O=10 trials but are then treated as known constants in the prior (Eq. 2) and in the likelihood (Eqs. 4, 6, 7). This ignores the sampling uncertainty in the anchor responses and overstates the precision of the posterior probabilities reported in Figures 2 and 3. For example, the text states that M=20 retains 'at least 65% confidence' in the most likely interval, but no uncertainty is attached to that figure. A bootstrap or sensitivity analysis around the anchor estimates would be needed to support such quantitative claims.
  4. [Sections 3.2 and 4] The claim of 'statistical robustness' at M=20 is supported only by qualitative inspection of Figures 2 and 3. No calibration checks, coverage probabilities, or repeated-sampling simulations are reported, and the query set is selected post hoc based on the authors' knowledge of other model families. The paper would need to demonstrate, at least on synthetic data or via a bootstrap over queries, that the posterior intervals are actually calibrated and that the method's rankings track ground truth. Without this, the conclusion that the method 'maintains statistical robustness' is not established.
minor comments (5)
  1. [Section 5.1] The sentence 'the factorization in Eqs. 3 assumes assumes conditional query independence' contains a duplicated word ('assumes assumes').
  2. [Section 4, Figure 3 bullet 3] The text says 'It does not provide any statistica interpretation' — 'statistica' should be 'statistical'.
  3. [Figures 2 and 3] The exact numerical posterior probabilities are not reported in a table; the text mentions thresholds such as 'at least 65% confidence' but the reader cannot verify these values from the figures. A supplementary table with the full probability mass functions for each test model and each M would improve the paper.
  4. [Notation] The notation 'Prest' is nonstandard and appears without definition. Consider using a hat or a subscript, e.g., \hat{P} or P_{\mathrm{est}}, to clarify that these are estimated quantities.
  5. [Appendix A] Appendix A is referenced in the text as containing the complete success-rate distribution, but the figure or table is not present in the manuscript body. Please include the appendix material or remove the reference.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the claimed probabilistic rankings are computed from externally measured anchor probabilities and independent test-model observations, with no fitted parameter renamed as a prediction.

full rationale

The paper's derivation chain does not reduce any claimed prediction to its inputs. The anchor-model probabilities P(Q_j|L_i) are measured by repeated trials (Section 3.3, Eq. 7) and held fixed; the test model's posterior (Eq. 1) is then a function of the test model's own observed outcomes q_j. The prior in Eq. 2 and likelihood in Eqs. 3-4 both use anchor data, which is an empirical-Bayes-style reuse, but the claimed rankings are not forced by that reuse: a test model's interval assignment still depends on its own response pattern, and the paper's discriminations (e.g., QwQ-32B vs Llama-4-Maverick) follow from differing per-question response patterns, not from a parameter fitted to the test models. No load-bearing self-citation or imported uniqueness theorem is present; the limitations section flags conditional query independence as an acknowledged assumption rather than a circular step. The reviewer's noted inconsistency between Eq. 6 and the uniform-theta marginalization is a correctness/validity concern, not a constructional circularity, so it does not raise the circularity score.

Assumptions & free parameters 4 free parameters · 5 assumptions · 2 invented entities

The method introduces no new physical entities, but it depends on a latent capability construct and two artificial boundary anchors. The free parameters are mostly experimental design choices (O, epsilon, query selection rule) plus the empirically estimated anchor capability values. The most fragile addition is the linearity assumption in Eq. 4, which is an ad hoc modeling choice.

free parameters (4)
  • Anchor capability values theta_1..theta_6 = Cumulative success rates from 14% to 82%, quasi-uniform (Figure 1)
    Estimated from O=10 trials per question on the curated 50-question set; they define the ranking intervals and the prior weights in Eq. 2.
  • epsilon boundary adjustment = 0.01
    Hand-chosen to replace extreme anchor probabilities {0,1} with {0.01,0.99} and prevent division by zero; affects likelihood values at the boundary.
  • Number of trials per question O = 10
    Experimental choice used in the multi-trial likelihood (Eq. 7); larger O would reduce anchor probability variance but cost more API calls.
  • Discriminative query selection threshold = Between one and half of early-2025 models answer correctly
    Hand-chosen criterion in Section 3.2 to make questions discriminative; it is a design choice, not estimated from the reported test models.
assumptions (5)
  • domain assumption A scalar capability theta_i exists and is monotonically ordered across anchors.
    Section 2.1 defines theta as an IQ-like latent variable and assumes theta_1 < ... < theta_N without loss of generality.
  • domain assumption Conditional query independence.
    Eq. 3 factorizes the joint likelihood as a product over questions; Section 5.1 acknowledges this may fail for semantically correlated queries.
  • standard math Uniform theta prior via maximum entropy.
    Eq. 2 sets the prior proportional to interval width, citing Jaynes (2003); this is a standard uninformative-prior choice.
  • ad hoc to paper Per-question correctness probability is linear in theta within each interval.
    Eq. 4 averages the boundary anchor probabilities; no derivation or empirical check is given for this linearity, and it is load-bearing for the posterior.
  • domain assumption Anchor probabilities Pr(Q_j|L_i) are stable estimates from O=10 trials.
    Section 3.3 measures each anchor 10 times per question; sampling error in these probabilities is not propagated into the posterior.
invented entities (2)
  • Boundary anchor models L_0 and L_{N+1}
    purpose: Define the extreme ranking intervals with Pr=0 and Pr=1 for every question.
    Artificial constructs; no real model corresponds to always-wrong or always-correct responses on the query set.
  • Latent capability parameter theta
    purpose: Underlying scalar that the ranking intervals partition.
    Standard latent-variable construct, not directly observed; no independent probe is offered beyond the question set.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Confidence in Large Language Model Evaluation: A Bayesian Approach to Limited-Sample Challenges." pith.science (2026). https://pith.science/paper/E2W7POCQ

@misc{pith2026250421303,
  author       = {Pith},
  title        = {Pith review of: Confidence in Large Language Model Evaluation: A Bayesian Approach to Limited-Sample Challenges},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E2W7POCQ}},
  note         = {Machine review of arXiv:2504.21303}
}
read the original abstract

Large language models (LLMs) exhibit probabilistic output characteristics, yet conventional evaluation frameworks rely on deterministic scalar metrics. This study introduces a Bayesian approach for LLM capability assessment that integrates prior knowledge through probabilistic inference, addressing limitations under limited-sample regimes. By treating model capabilities as latent variables and leveraging a curated query set to induce discriminative responses, we formalize model ranking as a Bayesian hypothesis testing problem over mutually exclusive capability intervals. Experimental evaluations with GPT-series models demonstrate that the proposed method achieves superior discrimination compared to conventional evaluation methods. Results indicate that even with reduced sample sizes, the approach maintains statistical robustness while providing actionable insights, such as probabilistic statements about a model's likelihood of surpassing specific baselines. This work advances LLM evaluation methodologies by bridging Bayesian inference with practical constraints in real-world deployment scenarios.

Figures

Figures reproduced from arXiv: 2504.21303 by the authors.

Figure 1
Figure 1. Anchor Model Performance (a) Success rates of the six anchor models (measured over 𝑂 = 10 trials per question) for 𝑀 = 50 evaluation questions. (b) The success rate of the first 20 questions are shown here, where the complete success rate distribution is provided in Appendix. Extreme probability values {0%, 100%} were modulated to {1%, 99%} to ensure numerical stability during subsequent computations. The capability… view at source ↗
Figure 2
Figure 2. Bayesian Probability Ranking Analysis Probability distributions of the test model’s ranking relative to six anchor models across varying question counts (𝑀).The anchor models partitioned the ranking space into seven mutually exclusive intervals, with probabilities quantifying the likelihood of the test model falling into each interval. 6 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Method comparison Performance comparison between the proposed Bayesian approach and conventional evaluation metrics. Bayesian@1 and accuracy reporting are Single-trial evaluations, while Bayes@10, Pass@10 and Mean±std are aggregated from O=10 independent trials. All results are obtained from M=20 questions. tion, while 𝑀 = 20 retained at least 65% confidence in most likely interval resolution. Sample size reduction … view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility

    cs.LG 2026-08 conditional novelty 5.0 of 10

    The paper formalizes test-time scaling into three regimes, introduces a discovery-stability profile for repeated-sampling evaluation, and releases nearly two million reasoning traces.

  2. Correcting Prompt Dependence in LLM Benchmarks: A Bayesian Hierarchical Model with Embedding-Space Clustering

    cs.CR 2025-10 conditional novelty 5.0 of 10

    A Bayesian model that groups similar LLM test prompts into clusters gives better predictive scores than a no-clustering baseline but does not prove that it truly corrects prompt dependence.

Reference graph

Works this paper leans on

76 extracted references · 46 canonical work pages · cited by 2 Pith papers

  1. [1]

    Language models are few-shot learners.arXiv preprint arXiv:2005.14165, 2020

    Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, et al. Language models are few-shot learners.arXiv preprint arXiv:2005.14165, 2020. URLhttps: //arxiv.org/abs/2005.14165

    arXiv 2005

  2. [2]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeff Wu, Xu Jiang, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022. URLhttps://arxiv.org/abs/2203.02155

    arXiv 2022

  3. [3]

    Improving language understand- ing by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understand- ing by generative pre-training. Technical report, OpenAI, 2018. URLhttps://cdn.openai.com/ research-covers/language-unsupervised/language_understanding_paper.pdf

    work page 2018

  4. [4]

    Languagemodelsareunsupervisedmultitasklearners

    AlecRadford,JeffWu,RewonChild,etal. Languagemodelsareunsupervisedmultitasklearners. Techni- calreport,OpenAI,2019. URL https://cdn.openai.com/better-language-models/language_ models_are_unsupervised_multitask_learners.pdf

    work page 2019

  5. [5]

    A survey of large language models.arXiv preprint arXiv:2303.18223, 2023

    Wayne Xin Zhao, Kun Zhou, Junyi Li, et al. A survey of large language models.arXiv preprint arXiv:2303.18223, 2023. URLhttps://arxiv.org/abs/2303.18223

    arXiv 2023

  6. [6]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021. URLhttps://arxiv.org/abs/2107.03374

    arXiv 2021

  7. [7]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. URLhttps://arxiv.org/abs/2204.02311

    arXiv 2022

  8. [8]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. URLhttps://arxiv.org/abs/2302.13971

    arXiv 2023

Show all 76 references
  1. [9]

    Towards reproducible llm evaluation: Quantifying uncertainty in llm benchmark scores.arXiv preprint arXiv:2410.03492, 2024

    Robert E Blackwell, James Barry, and Anthony G Cohn. Towards reproducible llm evaluation: Quantifying uncertainty in llm benchmark scores.arXiv preprint arXiv:2410.03492, 2024. URL https://arxiv.org/abs/2410.03492

    2024 arXiv

  2. [10]

    Adding error bars to evals: A statistical approach to language model evaluations.arXiv preprint arXiv:2411.00640, 2024

    Ethan Miller. Adding error bars to evals: A statistical approach to language model evaluations.arXiv preprint arXiv:2411.00640, 2024. URLhttps://arxiv.org/abs/2411.00640

    2024 arXiv

  3. [11]

    MESA Press, Chicago, 50th anniversary edition, 1993

    Georg Rasch.Probabilistic models for some intelligence and attainment tests. MESA Press, Chicago, 50th anniversary edition, 1993. URLhttps://eric.ed.gov/?id=ED419814

    1993

  4. [12]

    tinybenchmarks: Evaluatingllmswithfewer examples

    FedericoMariaPolo,LucasWeber,LeshemChoshen,etal. tinybenchmarks: Evaluatingllmswithfewer examples. arXiv preprint arXiv:2402.14992, 2024. URLhttps://arxiv.org/abs/2402.14992

    2024 arXiv

  5. [13]

    Reliable and efficient amortized model-based evaluation

    Sang Truong, Yifan Tu, Percy Liang, et al. Reliable and efficient amortized model-based evaluation. arXiv preprint arXiv:2503.13335, 2025. URLhttps://arxiv.org/abs/2503.13335

    2025 arXiv

  6. [14]

    Chatbot arena: An open platform for evaluating llms by human preference.arXiv preprint arXiv:2403.04132, 2024

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, et al. Chatbot arena: An open platform for evaluating llms by human preference.arXiv preprint arXiv:2403.04132, 2024. URLhttps://arxiv.org/abs/ 2403.04132

    2024 arXiv

  7. [15]

    Probability theory: The logic of science

    Edwin T Jaynes. Probability theory: The logic of science . Cambridge University Press, 2003. URL https://www.cambridge.org/core/books/probability-theory/ 9CA08E224FF30123304E6D8935CF1A99. 9 Confidence in Large Language Model Evaluation: A Bayesian Approach to Limited-Sample Challenges

    2003

  8. [16]

    Introducing chatgpt and whisper apis, March 2023

    OpenAI. Introducing chatgpt and whisper apis, March 2023. URLhttps://openai.com/blog/ introducing-chatgpt-and-whisper-apis/. Accessed: 2023-03-01

    2023

  9. [17]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    OpenAI. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023. URLhttps://arxiv.org/ abs/2303.08774

    2023 arXiv

  10. [18]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    OpenAI. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024. URLhttps://arxiv.org/ abs/2410.21276

    2024 arXiv

  11. [19]

    Introducing gpt-4.5, February 2025

    OpenAI. Introducing gpt-4.5, February 2025. URL https://openai.com/blog/ introducing-gpt-4-5/. Accessed: 2025-02-27

    2025

  12. [20]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    OpenAI. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024. URLhttps://arxiv.org/ abs/2412.16720

    2024 arXiv

  13. [21]

    Openai o3-mini, January 2025

    OpenAI. Openai o3-mini, January 2025. URL https://openai.com/blog/openai-o3-mini/. Accessed: 2025-01-31

    2025

  14. [22]

    Superclue: A comprehensive chinese large language model benchmark

    Liang Xu, Aiping Li, Lei Zhu, et al. Superclue: A comprehensive chinese large language model benchmark. arXiv preprint arXiv:2307.15020, 2023. URLhttps://arxiv.org/abs/2307.15020

    2023 arXiv

  15. [23]

    Minillm: Knowledge distillation of large language models

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. arXiv preprint arXiv:2306.08543, 2023. URLhttps://arxiv.org/abs/2306.08543

    2023 arXiv

  16. [24]

    Lamini-lm: A diverse herd of distilled models from large-scale instructions

    Minghao Wu, Abdul Waheed, Chiyu Zhang, et al. Lamini-lm: A diverse herd of distilled models from large-scale instructions. InProceedings of the 18th Conference of the European Chapter of the AssociationforComputationalLinguistics(EACL2024) ,pages944–964.AssociationforComputati...

    2024

  17. [25]

    Supergpqa: Scaling llm evaluation across 285 graduate disciplines

    M-A-P Team, Xiaocong Du, Yuan Yao, et al. Supergpqa: Scaling llm evaluation across 285 graduate disciplines. arXiv preprint arXiv:2502.14739, 2025. URLhttps://arxiv.org/abs/2502.14739

    2025 arXiv

  18. [26]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.arXiv preprint arXiv:2406.01574, 2024

    Yizhong Wang, Xiang Ma, Guiming Zhang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.arXiv preprint arXiv:2406.01574, 2024. URLhttps://arxiv. org/abs/2406.01574

    2024 arXiv

  19. [27]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Benjamin L Hou, Asa Cooper Stickland, et al. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023. URLhttps://arxiv.org/abs/2311.12022

    2023 arXiv

  20. [28]

    Measuringmathematicalproblemsolvingwiththe math dataset.arXiv preprint arXiv:2103.03874, 2021

    DanHendrycks,CollinBurns,SauravKadavath,etal. Measuringmathematicalproblemsolvingwiththe math dataset.arXiv preprint arXiv:2103.03874, 2021. URLhttps://arxiv.org/abs/2103.03874

    2021 arXiv

  21. [29]

    Zebralogic: On the scaling limits of llms for logical reasoning.arXiv preprint arXiv:2502.01100, 2025

    Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, et al. Zebralogic: On the scaling limits of llms for logical reasoning.arXiv preprint arXiv:2502.01100, 2025. URLhttps://arxiv.org/abs/2502. 01100

    2025 arXiv

  22. [30]

    Kor-bench: Benchmarking language models on knowledge-orthogonal reasoning tasks.arXiv preprint arXiv:2410.06526, 2024

    Kaijun Ma, Xiaocong Du, Yizhong Wang, et al. Kor-bench: Benchmarking language models on knowledge-orthogonal reasoning tasks.arXiv preprint arXiv:2410.06526, 2024. URLhttps://arxiv. org/abs/2410.06526

    2024 arXiv

  23. [31]

    Procbench: Benchmark for multi-step reasoning and following procedure.arXiv preprint arXiv:2410.03117, 2024

    Itsuki Fujisawa, Shun Nobe, Hiroki Seto, et al. Procbench: Benchmark for multi-step reasoning and following procedure.arXiv preprint arXiv:2410.03117, 2024. URLhttps://arxiv.org/abs/2410. 03117

    2024 arXiv

  24. [32]

    Claude 3.7 sonnet and claude code, February 2025

    Anthropic. Claude 3.7 sonnet and claude code, February 2025. URLhttps://www.anthropic.com/ news/claude-3-7-sonnet. Accessed: 2025-02-24

    2025

  25. [33]

    Gemini 2.5: Our most intelligent ai model, March 2025

    Google DeepMind. Gemini 2.5: Our most intelligent ai model, March 2025. URLhttps://blog. google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/ .Ac- cessed: 2025-03-25

    2025

  26. [34]

    Transmamba: Flexibly switching between transformer and mamba

    Yuhang Li, Rui Xie, Zhen Yang, et al. Transmamba: Flexibly switching between transformer and mamba. arXiv preprint arXiv:2503.24067, 2025. URLhttps://arxiv.org/abs/2503.24067

    2025

  27. [35]

    Doubao-1.5-pro: Model release, January 2025

    Doubao Team. Doubao-1.5-pro: Model release, January 2025. URLhttps://team.doubao.com/ en/special/doubao_1_5_pro. Accessed: 2025-01-22. 10 Confidence in Large Language Model Evaluation: A Bayesian Approach to Limited-Sample Challenges

    2025

  28. [36]

    Qwen2.5technicalreport

    QwenTeam. Qwen2.5technicalreport. arXivpreprintarXiv:2412.15115 ,2025. URL https://arxiv. org/abs/2412.15115

    2025 arXiv

  29. [37]

    Statistical knowledge assessment for large language models.arXiv preprint arXiv:2305.10519, 2023

    Qingxiu Dong, Jing Xu, Lingpeng Kong, Zhifang Sui, and Lei Li. Statistical knowledge assessment for large language models.arXiv preprint arXiv:2305.10519, 2023. URLhttps://arxiv.org/abs/ 2305.10519

    2023 arXiv

  30. [38]

    Benchmarking knowledge boundary for large language models: A different perspective on model evaluation

    Xunjian Yin, Xinyu Zhang, Jian Ruan, and Xiaojun Wan. Benchmarking knowledge boundary for large language models: A different perspective on model evaluation. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page...

    2024 doi

  31. [39]

    A survey on data contamination for large language models

    Yu Cheng, Yupeng Chang, and Yuxin Wu. A survey on data contamination for large language models. arXiv preprint arXiv:2502.14425, 2025. URLhttps://arxiv.org/abs/2502.14425

    2025 arXiv

  32. [41]

    Data contamination quiz: A tool to detect and estimate contamination in large language models

    Shahriar Golchin and Mihai Surdeanu. Data contamination quiz: A tool to detect and estimate contamination in large language models. arXiv preprint arXiv:2311.06233, 2023. URL https: //arxiv.org/abs/2311.06233

    2023 arXiv

  33. [42]

    The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, April 2025

    Meta. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, April 2025. URL https://ai.meta.com/blog/llama-4-multimodal-intelligence/. Accessed: 2025-04- 05

    2025

  34. [43]

    Deepseek-v3-0324 release, March 2025

    DeepSeek-AI. Deepseek-v3-0324 release, March 2025. URLhttps://api-docs.deepseek.com/ news/news250325. Accessed: 2025-03-24

    2025

  35. [44]

    Deepseek-r1: Incentivizingreasoningcapabilityinllmsviareinforcementlearning

    DeepSeek-AI. Deepseek-r1: Incentivizingreasoningcapabilityinllmsviareinforcementlearning. arXiv preprint arXiv:2501.12948, 2025. URLhttps://arxiv.org/abs/2501.12948

    2025 arXiv

  36. [45]

    Dirty Laundry

    QwenTeam. Qwq-32b: Embracingthepowerofreinforcementlearning, 2025. URL https://qwenlm. github.io/blog/qwq-32b/. Accessed: 2025. 11 Confidence in Large Language Model Evaluation: A Bayesian Approach to Limited-Sample Challenges Appendix A Complete success rate distribution Comp...

    2025

  37. [47]

    Move one position to the right in the string

  38. [48]

    Decrement the step count by one

  39. [49]

    final":

    Continue until the step count reaches zero. You should report: - The **final** character found when the step count reaches zero. - The **initial** state as a pair [starting character, initial step count]. - The **intermediate** states as a list of [character, remaining steps] ...

    1948

  40. [50]

    Input: A conjunctive normal form S

  41. [51]

    Plausible

    Output: Output "Plausible" if S has a satisfying assignment; otherwise, output "Implausible."

  42. [52]

    Implausible

    Procedure: - Initialization: - Let S0 and S2 be empty sets. - Let S1 be the set of all simple disjunctive clauses in S. - Resolution with S0 and S1: - For each clause C1 in S0 and each clause C2 in S1, if they can be resolved, compute C = dispel(C1, C2). - If C = 0, output "Im...

    1931

  43. [53]

    The game begins with an initial word and identifies a target word at the end

  44. [54]

    You may change only one letter at each step, with every resulting intermediate word being valid

  45. [55]

    Transform the starting word into the target word using the fewest number of steps possible

  46. [56]

    HEAD" into

    The puzzle provides the starting and target words. Your task is to determine the minimum number of transformations required. Task: Convert the word "HEAD" into "TALE". Express the minimum number of steps needed in double square brackets. For instance, if it requires 3 steps, w...

  47. [57]

    Peter occupies the second house

  48. [58]

    Arnold lives immediately to the left of the person who drinks only water

  49. [59]

    reasoning

    The person who drinks only water is immediately to the left of the person whose favorite drink is milk. Please explain your reasoning and provide your final answer using the JSON format below: { "reasoning": "", "solution": { "House 1": { "Name": "", "Drink": ""}, "House 2": {...

  50. [60]

    The person who drives a Honda Civic is the one whose favorite color is yellow

  51. [61]

    The individual whose favorite color is red drives a Tesla Model 3

  52. [62]

    The owner of the BMW 3 Series does not reside in the fourth house

  53. [63]

    The person whose mother is named Aniya is the one who prefers blue

  54. [64]

    Eric’s favorite color is green

  55. [65]

    The resident with red as their favorite color lives somewhere to the left of the owner of a Ford F-150

  56. [66]

    Alice is immediately to the left of Eric

  57. [67]

    The resident whose mother is Holly lives in the first house

  58. [68]

    Arnold has white as his favorite color

  59. [69]

    The person whose mother is Janelle loves white

  60. [70]

    The resident whose mother is Kailyn is Alice

  61. [71]

    Arnold resides somewhere to the left of Peter

  62. [72]

    Eric is the owner of the BMW 3 Series

  63. [73]

    reasoning

    Bob lives adjacent to the house where the owner of the Ford F-150 resides. Please detail your reasoning process and provide your final solution using the JSON structure below: { "reasoning": "", "solution": { "House 1": { "Name": "", "CarModel": "", "Color": "", "Mother": "" }...

  64. [74]

    Nuclear power, also termed atomic power, is the energy generated from processes that alter the composition of an atom’s nucleus

    ⋆Definition: Specifies the origin or source of the entity described by the concept as the differentia. 3.†Definition: Uses the entity’s functional role or purpose as the differentia. 4.◦Definition: Defines a concept through its relational attributes to other entities. 5.∙Defin...

  65. [75]

    If a is a multiple of b, then a※b = a/b + 2

  66. [76]

    If b is a multiple of a, then a※b = b/a + 2

  67. [77]

    Problem: Using the rules above, evaluate the expression 25※5※14 sequentially

    If neither number is a multiple of the other, then a※b = 24. Problem: Using the rules above, evaluate the expression 25※5※14 sequentially. Your answer should be a single number, formatted within double square brackets (for example: [[your answer]]). [[4]] Reasoning KOR-Bench 26

  68. [2023]

    URL https://arxiv.org/abs/2311.09783

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.