Pith. sign in

REVIEW 4 major objections 5 minor 49 references

How Much Do Large Language Model Cheat on Evaluation? Benchmarking Overestimation under the One-Time-Pad-Based Framework

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ArxivRoll claims that the gap between a model's public and private benchmark scores directly measures the share of its performance that comes from contamination or training bias.

desk verdict Worth reading for the fresh-benchmark machinery, but the headline metric is not a proportion and the paper never validates it as one. read the letter →

arxiv 2507.19219 v2 pith:SX7PTBO5 submitted 2025-07-25 cs.CL cs.CR

Zi Liang , Liantong Yu , Shiyu Zhang , Qingqing Ye , Haibo Hu This is my paper · ORCID
classification cs.CLcs.CR
keywords LLMevaluationbenchmarkcontaminationoverestimationquantificationone-timepadprivatebenchmarksclozetaskssequencingRuggedScores
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ArxivRoll tries to make overestimation in LLM evaluation measurable rather than just suspected. The framework builds private test sets from very recent preprint articles, keeps them secret until evaluation time, and treats them as one-time-use secrets in the style of a one-time pad. The Rugged Scores then take the normalised difference between a model's public and private scores in the same domain (RSI) and the spread of its private scores across domains (RSII), reading both as estimates of how much of the public score is inflated. If the paper is right, a leaderboard can report not only which model is strongest but how much of each score is real. The paper reports that several open-source model families show substantial RSI and that private benchmarks correlate with human-preference rankings.

What carries the argument

The carrying objects are SCP and the Rugged Scores. SCP turns a short excerpt of a recent preprint into three objective multiple-choice tasks: Sequencing asks for the order of shuffled sentences, Cloze asks for the correct masked sentence in a paragraph, and Prediction asks which of four continuations follows the excerpt. This generator provides the confidentiality and freshness of the private set without human annotation. The Rugged Score RSI is the ratio-symmetric public-minus-private gap defined in Eq. (1), and RSII is the variance-based spread across private domains defined in Eq. (2). Together with the release-then-expire lifecycle of each private set, these form the mechanism by which the paper converts a suspected bias into a number.

What would settle it

Fine-tune a small open model on a held-out sample of ArxivRollBench while a matched control model sees none of it; if the fine-tuned model's RSI does not clearly exceed the control's, the gap is not mostly contamination.

Watch

Extended reading notes

Core claim

The paper's central claim is that overestimation is not a vague risk but a quantity, and that the quantity is the normalised gap between performance on public and private benchmarks. Concretely, RSI is defined by Eq. (1) as twice the average of $(M_p - M_c)/(M_p + M_c)$ over matched public–private pairs plus an analogous unmatched term, so RSI $=0$ means a model performs equally on both and higher RSI means its public score is increasingly inflated. RSII, defined in Eq. (2), is the standard deviation of the model's accuracy across private domains, with a normalised variant dividing by the mean, and is claimed to measure biased overtraining. The private benchmarks are generated automatically from papers too recent for training data, which is what makes the gap interpretable as contamination rather than as ordinary difficulty.

Load-bearing premise

The whole contamination reading rests on the assumption that the public-minus-private performance gap is caused by contamination or training bias, not by the private tasks being harder, differently formatted, or drawn from a different distribution.

Editorial extensions

If this is right

  • A leaderboard that lists RSI next to accuracy lets a reader see at a glance whether a model's ranking comes from public-benchmark inflation or genuine ability.
  • Because every evaluation period uses a fresh private set, earlier ArxivRollBench releases can be published freely without weakening later assessments.
  • Domain-level RSI identifies which fields a model has overtrained on, so a developer can target training away from benchmark-specific strengths.
  • RSII gives a second, orthogonal signal: a model with balanced capabilities across domains scores low, while one trained hard on only the evaluated domains scores high.
  • Existing public benchmarks can be retrofitted as the 'public' side of the pair, so the framework does not require abandoning established benchmark sets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural test of the contamination reading is to take a small open model, fine-tune it on a held-out slice of ArxivRollBench, and show that its RSI rises while a matched control model's RSI stays flat; the paper's Eq. (1) predicts that directly.
  • The same ratio-symmetric gap could be computed between two versions of one benchmark, an old public release and a refreshed one, giving a contamination estimate that needs no new task generator.
  • For closed API models, the 'private' side is fresh but the public side is uncertain, so RSI should be interpreted as an upper bound on contamination rather than an exact proportion until the benchmark-pair difficulty is matched.
  • If SCP tasks measure mainly text coherence rather than domain knowledge, RSI would partly reflect task-format sensitivity; comparing RSI across SCP variants would reveal how much of the gap is format-specific.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ArxivRoll, a dynamic LLM evaluation framework with two components: SCP, an automated generator of private test cases from recent arXiv articles, and Rugged Scores (RS), which are claimed to quantify the proportion of public benchmark contamination and training bias. The authors construct private benchmarks, meta-evaluate their quality, and use the public-private performance gap to compute RSI values for a range of open and closed LLMs. The central claim is that these rugged scores measure the proportion of cheating behavior or contamination in a model's performance on public benchmarks.

Significance. If the interpretation of RS were valid, the paper would offer a valuable, transparent, reproducible, and dynamically refreshed evaluation pipeline, and its quantification of overestimation would be a useful complement to static benchmarks. The SCP construction is genuinely interesting, the authors release their code, and the correlation analysis with ChatbotArena provides some evidence that the private benchmark ranks models sensibly. However, the central measurement claim is not supported: RSI as defined in Eq. (1) is not a proportion, the paper's own numbers exceed 100%, and no validation is provided that the public-private gap is attributable to contamination rather than to task-format differences.

major comments (4)
  1. [§1, §2.1, §2.3, Eq. (1), Table 3] The central claim that RSI quantifies the proportion of contamination is contradicted by the metric's own definition. For a single matched pair, the matched-pair term of Eq. (1) is 2(p−c)/(p+c), which is a symmetrized relative gap, not a proportion: it is unbounded above and approaches 2 as c → 0. Table 3 reports Absolute RSI values of 1.21 and 1.41, and Section 4.3 explicitly says these are 'even larger than 100%'. Section 2.3 itself concedes that 'RSI is not a normalized metric'. Even under the authors' assumption that the entire public-private gap is due to contamination, no standard contamination proportion equals 2(p−c)/(p+c); natural candidates such as (p−c)/p or (p−c)/(1−c) give different values (e.g., for p=0.5, c=0.25, the matched-pair term is 0.67 while these candidates are 0.5 and 0.33). The paper provides no derivation linking Eq. (1) to a proportion, so the headline claim that RS 'quantify the proportion of cheating behavior' is unsupported by the paper's own definition.
  2. [§2.1, §3.2, §4.3] The interpretation of the public-private performance gap as a contamination proportion requires that the gap be attributable to contamination rather than to differences in task format, content, difficulty, or domain shift. The public benchmarks (MMLU, MMLU Pro, and the vaguely described 'domain-specific datasets' in Appendix A) differ substantially from the private SCP tasks, which are sequencing, cloze, and prediction on arXiv text fragments. The paper provides no synthetic contamination experiment, no controlled benchmarks matched in format and difficulty, and no calibration against known contamination. The correlation analysis in Section 3.2 validates only that private-benchmark rankings resemble ChatbotArena rankings; it does not validate the gap interpretation. Consequently, Eq. (1) is at best a task-specific accuracy gap, and the contamination reading is an unvalidated assumption.
  3. [Appendix A, §2.3] The construction of the public-private pairs used in Eq. (1) is underspecified. Appendix A states that MMLU, MMLU Pro, and 'domain-specific datasets, such as those commonly used in Math' are used, but it does not report which public benchmarks are paired with which of the eight private domains, the number of pairs Np, the sizes of the unmatched sets, or how the aggregate terms in Eq. (1) are weighted. Since RSI depends on all of these choices, the RSI values in Table 3 are not reproducible from the manuscript as written. The authors should provide the exact benchmark inventory, the pairing rule, and the uncertainty propagated through Eq. (1).
  4. [Table 3, Tables 2, 6, 7] RSI values in Table 3 are reported without confidence intervals, despite the large standard errors in the underlying private-benchmark accuracies (e.g., several models in Table 2 have error bars of ±7–8 percentage points). For reported differences such as Llama3.1-70B (RSI 0.48) versus Llama3-8B (RSI 0.74), the gap is likely within the measurement noise. Without uncertainty propagation, the qualitative claims in Section 4.3 about which model families exhibit the highest overestimation are not supported.
minor comments (5)
  1. [§2.2, Appendix A] The free parameters N and Nf are only defined in Appendix A (N=1, Nf=80); they should be defined in the main text where the SCP algorithm is introduced, and a sensitivity analysis for these choices would strengthen the benchmark construction.
  2. [Figure 5] The axis labels in Figure 5 are garbled by font-encoding artifacts and are not readable, so the reader cannot verify the 'growing overestimation' claim from the figure.
  3. [Table 3] The column headed 'Relative Rank Changes' mixes arrow glyphs with numeric values and is never clearly defined; it should be renamed and explained, for example by stating whether it is a relative RSI score or a rank shift.
  4. [Appendix C, Table 13] In the ArxivRollBench2026a results, 'moonshotai/kimi-k2.6' is ranked first with a valid accuracy of 70.8% despite a raw accuracy of 5.6% and a coverage of 7.9%; the pooled valid-response protocol can materially reorder the leaderboard when coverage is low, and this should be discussed in the main text.
  5. [§1] The title's phrase 'How Much Do Large Language Model Cheat on Evaluation?' suggests intentionality, while the paper also addresses unintentional contamination and biased overtraining; a more neutral phrasing such as 'overestimation in evaluation' would better match the content.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: Rugged Scores are defined formulas evaluated on external benchmarks; the 'proportion' wording is an over-interpretation, not a circular derivation.

full rationale

The central derivation chain is self-contained in the sense required by the circularity pass. RSI (Eq. 1) and RSII (Eq. 2) are explicitly defined formulas computed from measured accuracies on separately constructed public and private benchmarks; no parameter is fitted to a target conclusion, and no 'prediction' is statistically forced by a fitted input. The private benchmarks are generated by SCP from fresh arXiv articles and validated against the external ChatbotArena in Section 3.2, providing independent support. The paper's self-citations (Liang et al. 2025a,b; Bai et al. 2025; Shang et al. 2025; Xiao et al. 2025) appear only for experimental conventions such as greedy search and for related work, not as load-bearing justification of the metric or its interpretation. The one substantive concern flagged by the paper itself is in Section 2.3: 'RSI is not a normalized metric,' and Table 3 reports values above 100%, which contradicts the wording in Sections 1 and 2.1 that RS quantifies 'the proportion' of contamination or cheating. That is an interpretive over-claim and a metric-validity problem, not a circular reduction: the paper never defines an independent contamination proportion and then derives RSI from it, nor does it define RSI in terms of that proportion. The public-private gap is a defined quantity, and calling it a proportion is an unsupported label rather than a self-referential derivation. Under the hard rules, this does not constitute circularity, so the score is low despite the wording concern.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on two load-bearing domain assumptions: the confidentiality of recent arXiv papers and the attribution of the public-private gap to contamination. The paper does not provide independent evidence for either. The hand-chosen generation parameters N and Nf are additional degrees of freedom.

free parameters (2)
  • N (number of consensus phrases per fragment) = 1
    Chosen by hand in Appendix A to control fragment length and test difficulty.
  • Nf (minimum word count for fragment filtering) = 80
    Chosen by hand in Appendix A to remove too-short fragments.
assumptions (3)
  • domain assumption Recent arXiv papers are unseen by the evaluated LLMs.
    Section 2.2 claims confidentiality, but no membership test is performed; models with later training cutoffs could have seen the papers.
  • ad hoc to paper The public-private performance gap reflects contamination and training bias.
    Eq. (1) and the interpretation in Section 4.3 assume the gap is caused by cheating rather than by task format or difficulty differences.
  • domain assumption ChatbotArena rankings are a reliable proxy for LLM capability.
    Section 3.2 uses ChatbotArena as the reference benchmark despite its crowdsourced and potentially gameable nature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How Much Do Large Language Model Cheat on Evaluation? Benchmarking Overestimation under the One-Time-Pad-Based Framework." pith.science (2026). https://pith.science/paper/SX7PTBO5

@misc{pith2026250719219,
  author       = {Pith},
  title        = {Pith review of: How Much Do Large Language Model Cheat on Evaluation? Benchmarking Overestimation under the One-Time-Pad-Based Framework},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SX7PTBO5}},
  note         = {Machine review of arXiv:2507.19219}
}
read the original abstract

Overestimation in evaluating large language models (LLMs) has become an increasing concern. Due to the contamination of public benchmarks or imbalanced model training, LLMs may achieve unreal evaluation results on public benchmarks, either intentionally or unintentionally, which leads to unfair comparisons among LLMs and undermines their realistic capability assessments. Existing benchmarks attempt to address these issues by keeping test cases permanently secret, mitigating contamination through human evaluation, or repeatedly collecting and constructing new samples. However, these approaches fail to ensure reproducibility, transparency, and high efficiency simultaneously. Moreover, the extent of overestimation in current LLMs remains unquantified. To address these issues, we propose ArxivRoll, a dynamic evaluation framework inspired by one-time pad encryption in cryptography. ArxivRoll comprises two key components: \emph{i) SCP (Sequencing, Cloze, and Prediction)}, an automated generator for private test cases, and \emph{ii) Rugged Scores (RS)}, metrics that measure the proportion of public benchmark contamination and training bias. Leveraging SCP, ArxivRoll constructs a new benchmark every six months using recent articles from ArXiv and employs them for one-time evaluations of LLM performance. Extensive experiments demonstrate the high quality of our benchmark, and we provide a systematic evaluation of current LLMs. The source code is available at https://github.com/liangzid/ArxivRoll/.

Figures

Figures reproduced from arXiv: 2507.19219 by the authors.

Figure 1
Figure 1. Framework of ArxivRoll, which categorizes benchmarks into two distinct groups: public benchmarks and private benchmarks (i.e., ArxivRollBench). These benchmarks are utilized to estimate both the overestimation proportion and per￾formance of large language models (LLMs). Notably, ArxivRoll represents a dynamic benchmarking system, where private benchmarks are utilized exclusively once and then expire for subsequent e… view at source ↗
Figure 2
Figure 2. An illustrative example of symbolic formatting for test samples, encompassing three formats: sequencing, cloze, and prediction (SCP). {T i p}Np ∪ {T j p }N′ p may be less reliable, and model m may be overfitted to the specific characteristics of them. Unfortunately, RSI is not a normalized metric and is un￾avoidably coupled with models and benchmarks used for evaluation. This means that RSI obtained for different se… view at source ↗
Figure 3
Figure 3. Performance of Llama3 (8B) across 32-time [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Performance of models across different domain benchmarks and the corresponding Absolute [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Evolution of series models. fairness of ArxivRollBench across these domains. Besides, it is observed that Absolute RSI are also significantly higher in the Econ, Q-Fin, Bio, and Phy domains, suggesting that advantages of these models on public benchmarks in these areas…
Figure 6
Figure 6. Figure 6: Sample numbers distribution of ArxivRollBench2024b among eight categories across sequencing (a), cloze (b), and [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: The details of private datasets generation, encompassing three formats: sequencing, cloze, and prediction (SCP). [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 23 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  4. [4]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  5. [5]

    Qwen2 Technical Report

    2024. Qwen2 Technical Report

    work page 2024

  6. [6]

    G.; and Chapelle, C

    Abraham, R. G.; and Chapelle, C. A. 1992. The Meaning of Cloze Test Scores: An Item Difficulty Perspective. The Modern Language Journal, 76(4): 468--479

    work page 1992

  7. [7]

    AI@Meta. 2024. Llama 3 Model Card

    2024

  8. [8]

    Alderson, J. C. 1979. The Cloze Procedure and Proficiency in English as a Foreign Language. TESOL Quarterly, 13(2): 219--227

    work page 1979

Show all 49 references
  1. [9]

    Bormuth, J. R. 1968. The Cloze Readability Procedure. Elementary English, 45(4): 429--436

    1968

  2. [10]

    Britannica. 2024. Gestalt psychology. Encyclopedia Britannica

    2024

  3. [11]

    N.; Li, T.; Li, D.; Zhu, B.; Zhang, H.; Jordan, M

    Chiang, W.; Zheng, L.; Sheng, Y.; Angelopoulos, A. N.; Li, T.; Li, D.; Zhu, B.; Zhang, H.; Jordan, M. I.; Gonzalez, J. E.; and Stoica, I. 2024. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. In Forty-first International Conference on Machine Learning,...

    2024

  4. [12]

    Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; Hesse, C.; and Schulman, J. 2021. Training Verifiers to Solve Math Word Problems. CoRR, abs/2110.14168

    2021 arXiv

  5. [13]

    Dong, Y.; Jiang, X.; Liu, H.; Jin, Z.; Gu, B.; Yang, M.; and Li, G. 2024. Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models. In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., Findings of the Association for Computational L...

    2024

  6. [14]

    Elo, A. 1967. The proposed USCF rating system, its development, theory, and applications. Chess Life XXII (8): 242--247

    1967

  7. [15]

    Gao, L.; Tow, J.; Abbasi, B.; Biderman, S.; Black, S.; DiPofi, A.; Foster, C.; Golding, L.; Hsu, J.; Le Noac'h, A.; Li, H.; McDonell, K.; Muennighoff, N.; Ociepa, C.; Phang, J.; Reynolds, L.; Schoelkopf, H.; Skowron, A.; Sutawika, L.; Tang, E.; Thite, A.; Wang, B.; Wang, K.; a...

    2024

  8. [16]

    Gunasekar, S.; Zhang, Y.; Aneja, J.; Mendes, C. C. T.; Giorno, A. D.; Gopi, S.; Javaheripi, M.; Kauffmann, P.; de Rosa, G.; Saarikivi, O.; Salim, A.; Shah, S.; Behl, H. S.; Wang, X.; Bubeck, S.; Eldan, R.; Kalai, A. T.; Lee, Y. T.; and Li, Y. 2023. Textbooks Are All You Need. ...

    2023 arXiv

  9. [17]

    Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2021. Measuring Massive Multitask Language Understanding. Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  10. [18]

    Huang, Z.; Wang, Z.; Xia, S.; Li, X.; Zou, H.; Xu, R.; Fan, R.; Ye, L.; Chern, E.; Ye, Y.; Zhang, Y.; Yang, Y.; Wu, T.; Wang, B.; Sun, S.; Xiao, Y.; Li, Y.; Zhou, F.; Chern, S.; Qin, Y.; Ma, Y.; Su, J.; Liu, Y.; Zheng, Y.; Zhang, S.; Lin, D.; Qiao, Y.; and Liu, P. 2024. Olympi...

    2024 arXiv

  11. [19]

    Javaheripi, M.; Bubeck, S.; Abdin, M.; Aneja, J.; Bubeck, S.; Mendes, C. C. T.; Chen, W.; Del Giorno, A.; Eldan, R.; Gopi, S.; et al. 2023. Phi-2: The surprising power of small language models. Microsoft Research Blog, 1(3): 3

    2023

  12. [20]

    Z.; Zhong, M.; Schaeffer, R.; Ouyang, S.; Han, J.; and Koyejo, S

    Jiang, M.; Liu, K. Z.; Zhong, M.; Schaeffer, R.; Ouyang, S.; Han, J.; and Koyejo, S. 2024. Investigating Data Contamination for Pre-training Language Models. CoRR, abs/2401.06059

    2024 arXiv

  13. [21]

    E.; Yang, J.; Wettig, A.; Yao, S.; Pei, K.; Press, O.; and Narasimhan, K

    Jimenez, C. E.; Yang, J.; Wettig, A.; Yao, S.; Pei, K.; Press, O.; and Narasimhan, K. R. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . Ope...

    2024

  14. [22]

    E.; and Stoica, I

    Li, T.; Chiang, W.-L.; Frick, E.; Dunlap, L.; Wu, T.; Zhu, B.; Gonzalez, J. E.; and Stoica, I. 2024 a . From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline. arXiv preprint arXiv:2406.11939

    2024 arXiv

  15. [23]

    E.; and Stoica, I

    Li, T.; Chiang, W.-L.; Frick, E.; Dunlap, L.; Zhu, B.; Gonzalez, J. E.; and Stoica, I. 2024 b . From Live Data to High-Quality Benchmarks: The Arena-Hard Pipeline

    2024

  16. [24]

    D.; Gunasekar, S.; and Lee, Y

    Li, Y.; Bubeck, S.; Eldan, R.; Giorno, A. D.; Gunasekar, S.; and Lee, Y. T. 2023. Textbooks Are All You Need II: phi-1.5 technical report. arXiv:2309.05463

    2023 arXiv

  17. [25]

    Li, Y.; Guo, Y.; Guerin, F.; and Lin, C. 2024 c . An Open-Source Data Contamination Report for Large Language Models. In Al-Onaizan, Y.; Bansal, M.; and Chen, Y.-N., eds., Findings of the Association for Computational Linguistics: EMNLP 2024, 528--541. Miami, Florida, USA: Ass...

    2024

  18. [26]

    Mather, G. 2006. Foundations of perception. Psychology Press

    2006

  19. [27]

    Microsoft. 2024. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv:2404.14219

    2024 arXiv

  20. [28]

    Miller, F. 1882. Telegraphic Code to Insure Privacy and Secrecy in the Transmission of Telegrams. C.M. Cornwell

  21. [29]

    Mirzadeh, S.; Alizadeh, K.; Shahrokhi, H.; Tuzel, O.; Bengio, S.; and Farajtabar, M. 2024. GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models. CoRR, abs/2410.05229

    2024 arXiv

  22. [30]

    OpenAI. 2024 a . GPT-4 Technical Report. arXiv:2303.08774

    2024 arXiv

  23. [31]

    OpenAI. 2024 b . GPT-4o System Card. arXiv:2410.21276

    2024 arXiv

  24. [32]

    Palavalli, M.; Bertsch, A.; and Gormley, M. 2024. A Taxonomy for Data Contamination in Large Language Models. In Sainz, O.; Garc \' a Ferrero, I.; Agirre, E.; Ander Campos, J.; Jacovi, A.; Elazar, Y.; and Goldberg, Y., eds., Proceedings of the 1st Workshop on Data Contaminatio...

    2024

  25. [33]

    Shannon, C. E. 1949. Communication theory of secrecy systems. The Bell System Technical Journal, 28(4): 656--715

    1949

  26. [34]

    Team, T. T.-B. 2025. Terminal-Bench: A Benchmark for AI Agents in Terminal Environments

    2025

  27. [35]

    Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; Bikel, D.; Blecher, L.; Ferrer, C. C.; Chen, M.; Cucurull, G.; Esiobu, D.; Fernandes, J.; Fu, J.; Fu, W.; Fuller, B.; Gao, C.; Goswami, V.; Goyal, N....

    2023 arXiv

  28. [36]

    Wang, B.; and Komatsuzaki, A. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model . https://github.com/kingoflolz/mesh-transformer-jax

    2021

  29. [37]

    R.; Zhang, S.; Sun, Y.; and Wang, W

    Wang, X.; Hu, Z.; Lu, P.; Zhu, Y.; Zhang, J.; Subramaniam, S.; Loomba, A. R.; Zhang, S.; Sun, Y.; and Wang, W. 2024 a . SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models. In Forty-first International Conference on Machine Learning...

    2024

  30. [38]

    Wang, Y.; Ma, X.; Zhang, G.; Ni, Y.; Chandra, A.; Guo, S.; Ren, W.; Arulraj, A.; He, X.; Jiang, Z.; Li, T.; Ku, M.; Wang, K.; Zhuang, A.; Fan, R.; Yue, X.; and Chen, W. 2024 b . MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. arXiv:2406.01574

    2024 arXiv

  31. [39]

    Wang, Z.; Bukharin, A.; Delalleau, O.; Egert, D.; Shen, G.; Zeng, J.; Kuchaiev, O.; and Dong, Y. 2024 c . HelpSteer2-Preference: Complementing Ratings with Preferences. arXiv:2410.01257

    2024 arXiv

  32. [40]

    White, C.; Dooley, S.; Roberts, M.; Pal, A.; Feuer, B.; Jain, S.; Shwartz - Ziv, R.; Jain, N.; Saifullah, K.; Naidu, S.; Hegde, C.; LeCun, Y.; Goldstein, T.; Neiswanger, W.; and Goldblum, M. 2024. LiveBench: A Challenging, Contamination-Free LLM Benchmark. CoRR, abs/2406.19314

    2024 arXiv

  33. [41]

    Wu, M.; Zhang, Z.; Dong, Q.; Xi, Z.; Zhao, J.; Jin, S.; Fan, X.; Zhou, Y.; Fu, Y.; Liu, Q.; Zhang, S.; and Zhang, Q. 2025. Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination. arXiv:2507.10532

    2025

  34. [42]

    Wu, Z.; Qiu, L.; Ross, A.; Aky \"u rek, E.; Chen, B.; Wang, B.; Kim, N.; Andreas, J.; and Kim, Y. 2024. Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks. In Duh, K.; Gomez, H.; and Bethard, S., eds., Proceedings ...

    2024

  35. [43]

    Xu, C.; Guan, S.; Greene, D.; and Kechadi, M. T. 2024. Benchmark Data Contamination of Large Language Models: A Survey. CoRR, abs/2406.04244

    2024 arXiv

  36. [44]

    E.; and Stoica, I

    Yang, S.; Chiang, W.; Zheng, L.; Gonzalez, J. E.; and Stoica, I. 2023. Rethinking Benchmark and Contamination for Language Models with Rephrased Samples. CoRR, abs/2311.04850

    2023 arXiv

  37. [45]

    Ye, T.; Xu, Z.; Li, Y.; and Allen-Zhu, Z. 2024. Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process. arXiv:2407.20311

    2024 arXiv

  38. [46]

    Yue, X.; Ni, Y.; Zheng, T.; Zhang, K.; Liu, R.; Zhang, G.; Stevens, S.; Jiang, D.; Ren, W.; Sun, Y.; Wei, C.; Yu, B.; Yuan, R.; Sun, R.; Yin, M.; Zheng, B.; Yang, Z.; Liu, Y.; Huang, W.; Sun, H.; Su, Y.; and Chen, W. 2024. MMMU: A Massive Multi-Discipline Multimodal Understand...

    2024

  39. [47]

    Zhang, Z.; Chen, J.; and Yang, D. 2024. DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph. In Globersons, A.; Mackey, L.; Belgrave, D.; Fan, A.; Paquet, U.; Tomczak, J. M.; and Zhang, C., eds., Advances in Neural Information Processing Systems 38: ...

    2024

  40. [48]

    Z.; Yang, D.; and Xie, X

    Zhu, K.; Chen, J.; Wang, J.; Gong, N. Z.; Yang, D.; and Xie, X. 2024 a . DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net

    2024

  41. [49]

    Zhu, K.; Wang, J.; Zhao, Q.; Xu, R.; and Xie, X. 2024 b . Dynamic Evaluation of Large Language Models by Meta Probing Agents. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.