Pith. sign in

REVIEW 3 major objections 7 minor 56 references

Towards Contamination Resistant Benchmarks

T0 review · 3 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that a benchmark built from Caesar ciphers is contamination resistant, because every test instance can be freshly generated, and that once contamination is controlled, current LLMs fail at this simple linear-mapping task.

desk verdict A cheap, honest benchmark paper whose shift-3 failure pattern is real, but whose 'contamination resistant' label is an inference, not a measurement—worth refereeing, not accepting as-is. read the letter →

arxiv 2505.08389 v1 pith:UAROUVVS submitted 2025-05-13 cs.CL

Rahmatullah Musawi , Sheng Lu This is my paper · ORCID
classification cs.CL
keywords contaminationresistanceCaesarcipherbenchmarklargelanguagemodelsdataleakagememorizationinconsistentcompetencegeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces the concept of contamination resistance: a benchmark that keeps assessing the same capability while generating an infinite supply of fresh test instances, so memorization from pre-training gives no advantage. It proposes Caesar ciphers as such a benchmark, since any text can be shifted by any number of alphabet positions at essentially zero curation cost. When the authors test widely used LLMs under this controlled setting, most score near zero on exact-match accuracy, and GPT-4o succeeds mainly on shift=3 with natural English words. They interpret this as evidence that benchmark scores are inflated by contamination and that current models lack genuine generalization on a trivial linear mapping.

What carries the argument

The load-bearing object is the Caesar-cipher instance generator: a plaintext (natural English words or random nonsense words) combined with one of four shift values (3, 6, 9, 12), posed as an encode/decode prompt. The generator's infinitude is the actual mechanism of contamination resistance—each text-shift pair is unique, so a model cannot have memorized the instance, and any correct answer must come from computing the shift. The paper also uses exact-match accuracy and character error rate as evaluation metrics, and a set of prompt variants (open, base, dict, code, plus zero- and few-shot settings) to distinguish genuine computation from memorized routines. The verbalization, lookup-table, and code behaviors serve as probes that separate a correct reasoning chain from a correct final answer.

What would settle it

Take one of the paper's shift=6, shift=9, or shift=12 cipher-text/plaintext pairs with a random-word plaintext and search the released training corpora of LLaMA and Qwen for that exact string; if it is found, the contamination-resistance claim for those instances fails. Alternatively, prompt GPT-4o with a fresh random-word plaintext at a shift the paper did not use, such as 5 or 7, and count exact-match successes: high accuracy on unseen shifts would show the failure at 6, 9, and 12 is not simply contamination.

Watch

Extended reading notes

Core claim

The central discovery is that the Caesar cipher satisfies a definition of contamination resistance: it probes genuine capabilities (logical deduction, arithmetic reasoning, generalization), admits infinitely many fresh instances so memorization gives no advantage, keeps task complexity constant across instances, and costs almost nothing to update. Empirically, when the authors control for contamination, widely used LLMs perform at or near zero exact-match accuracy on simple encoding/decoding; GPT-4o succeeds mainly at shift=3 with natural-language words, and its performance collapses for shifts 6, 9, and 12 and for random nonsense words. The authors attribute the shift=3 advantage to contamination, noting that shift=3 examples are prevalent in web corpora and that models volunteering definitions of the cipher tend to use shift=3 examples. They call the overall pattern "inconsistent competence": models can verbalize correct letter-by-letter reasoning and still output the wrong answer, can produce correct lookup tables and code while failing the final answer, and do not benefit from 50-shot demonstrations that already contain all letter mappings. The conclusion is that apparent LLM competence on benchmarks can be inflated by memorization, and that contamination-resistant benchmarks reveal a lack of true generalization.

Load-bearing premise

The benchmark's contamination resistance rests on the assumption that the freshly generated cipher texts with shifts 6, 9, and 12 and random nonsense-word plaintexts were never present in the tested models' training data; the paper does not verify this and relies instead on the infinitude of the cipher space.

Editorial extensions

If this is right

  • A continuously regenerated Caesar-cipher test set can serve as a low-cost contamination control: any model that scores high on it must be computing the shift, not retrieving memorized text.
  • Performance gaps between shift=3 and other shifts, and between natural and nonsense words, should be read as evidence of memorization rather than task difficulty.
  • The observation that correct reasoning chains coexist with wrong final answers means model explanations cannot be taken at face value as evidence of competence.
  • Because 50-shot demonstrations covering all letter mappings did not help GPT-4o, in-context learning of a deterministic linear function appears unreliable in these models.
  • Larger models' lower character error rates (LLaMA3.1-70B vs 8B, Qwen2.5-32B vs 7B) suggest scale helps partially, but not enough to reach reliable zero-shot performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The infinite-instance construction extends beyond Caesar ciphers: any parameterized family of deterministic text transformations (other affine ciphers, Vigenère with random keys, simple formatting rules) could generate contamination-resistant probes for symbol manipulation at near-zero curation cost.
  • A stronger test than the paper's would draw a fresh random shift at query time and compare models on shifts never seen in pre-training; the paper's use of only four shifts leaves the space of possible tasks sparsely sampled.
  • The lookup-table/code discrepancy suggests these scaffolding prompts may themselves be memorized routines; this can be probed by asking models to build a lookup table for a nonstandard alphabet, which training data would not contain.
  • If contamination is indeed what makes shift=3 easier, evaluation protocols should report scores on never-seen instances separately from static benchmarks, and models with a large gap should be flagged as relying on memorization.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces a definition of "contamination resistance" for evaluation benchmarks, proposes a Caesar-cipher-based benchmark as an instance, and tests six LLMs across four shift values, three prompting strategies, and two plain-text types (natural English and random words). The main empirical finding is that most models achieve near-zero exact-match accuracy on most cells, while GPT-4o shows partial success when the shift is 3 and the plain text is natural English; the authors attribute this shift-dependent success to contamination and summarize the overall pattern as "inconsistent competence." The paper also includes pilot analyses of prompt effects, output-position accuracy, few-shot prompting, and two reasoning models (QwQ-32B and DeepSeek-R1), plus an integrity check for shift-direction ambiguity.

Significance. If the central claim were established, the benchmark would be a valuable, nearly free dynamic evaluation that controls for memorization, and the observed failures would provide evidence about LLM generalization on a simple linear mapping. The paper has real strengths: the experimental protocol is clearly specified, code and data are made available, the broad failure pattern (all non-GPT-4o models at near-zero accuracy) is robust across models and prompts, and the integrity check for shift direction is a thoughtful addition. However, the central claim as stated—that the benchmark is contamination resistant and that the results demonstrate behavior "when contamination is controlled"—is not supported by the evidence. The only condition with substantial success (shift=3, natural English) is, by the paper's own analysis, a condition in which task-level contamination is likely. This gap reduces the force of the empirical contribution unless the claim is reframed or directly tested.

major comments (3)
  1. [Section 3; Section 5.1; Table 12] The contamination-resistance argument is instance-level, not task-level. Section 3 says the benchmark is contamination resistant because "users can generate an infinite number of unique texts," which prevents memorization of particular strings. It does not prevent a model from having memorized the general Caesar-cipher rule or a canonical shift value. The paper's own Section 5.1 attributes GPT-4o's shift-3 advantage to the prevalence of shift-3 ciphers in web corpora (citing McCoy et al., 2024a), and Table 12 shows that four of six models spontaneously give shift-3 examples when asked to define the Caesar cipher. Thus the one condition in which the benchmark "works" (GPT-4o, shift=3, natural text) is, according to the authors' own reasoning, a contaminated condition. The abstract's claim that models "struggle with this benchmark when contamination is controlled" is therefore an inference from the absence of success in the other cells, not a demonstrated measurement of contamination. I recommend either limiting the claim to instance-level contamination resistance and revising the abstract and conclusion accordingly, or directly testing task-level contamination, e.g., with a controlled pretraining experiment on small models or with per-instance membership checks.
  2. [Section 3; Tables 2 and 13] The sample size of 25 plain texts per cell is too small to support the precision of the quantitative claims. For example, GPT-4o's exact-match accuracy of 0.82 on the natural, shift=3, open-prompt cell (Table 2) has an approximate standard error of sqrt(0.82*0.18/25)=0.077, so a reported difference like 0.36 vs. 0.34 between shift=12 and other conditions (Table 13) is well within noise. No confidence intervals or significance tests are reported anywhere, yet the paper makes fine-grained comparative statements about prompt effects, output-position accuracies (Section 5.4), and few-shot improvements (Section 5.5). The broad qualitative pattern—most models at 0.00 in most cells—is robust, but the quantitative details about GPT-4o should be either supported by larger sample sizes or presented with appropriate uncertainty estimates and cautious language.
  3. [Section 5.1; Section 5.2] The attribution of the shift-3 advantage to contamination is circumstantial rather than demonstrated. The paper relies on McCoy et al. (2024a) for the claim that shift-3 ciphers are more prevalent in pretraining corpora, but that citation does not establish that the specific test instances, or the general shift-3 rule, are in the training data of the tested models. The alternative explanation—that the models have a canonical shift bias or have learned the rule as a known task—is equally consistent with Table 12, which shows that all tested models can define a Caesar cipher and most give shift-3 examples. The paper should distinguish "contamination" from "task familiarity," and if the contamination claim is to be maintained, it needs a more direct form of evidence, such as a controlled experiment in which pretraining data are known or a membership-style test on the specific cipher texts.
minor comments (7)
  1. [Table 8 caption] The caption says "first (id=1) and second (id=2)", but Table 5 and the surrounding text use id=0,1,2 for the first, second, and third output characters; this caption should say "second (id=1) and third (id=2)".
  2. [Multiple sections] There are several typos: "straight forward" should be "straightforward" (Section 2.2 and Section 4); Table 2 has "a verage" instead of "average"; Section 5.7 has "an addition set" instead of "an additional set"; Appendix B has "4bit" instead of "4-bit".
  3. [Section 5.3] The sentence "We only experimented with the code prompt using GPT-4o" is followed immediately by a figure reference; consider moving Figure 2 into the sentence that discusses the code-prompt results for readability.
  4. [Section 5.6] The transition between the decoding query shown in Table 15 and the encoding query "rakibo zlmqwe" is confusing. State explicitly that the decoding task used the query from Table 15 and that the encoding task used a different (reverse) query, so readers do not think the same query was used for both tasks.
  5. [Section 5.4] Tables 5 and 6 use different column labels ("id" vs. "char_id") for the same character-position variable; unify the notation.
  6. [Throughout] Use "contamination-resistant" consistently when the phrase is used as an adjective; the text currently switches between "contamination resistant" and "contamination-resistant."
  7. [Section 5.5] The conclusion that "fine-tuning models on Caesar ciphers may not be a viable solution" (Section 5.5) goes beyond the data, since the experiment only tests zero-shot and 50-shot prompting with the base prompt on GPT-4o, not fine-tuning. Either remove the sentence or label it as speculation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's claims are a definitional proposal plus an empirical evaluation, with no prediction derived from fitted parameters or load-bearing self-citations.

full rationale

The paper's central claims are (i) a definitional proposal for the term 'contamination resistance', (ii) a constructed Caesar-cipher benchmark, and (iii) empirical measurements of LLM performance on that benchmark. None of these is derived by fitting a parameter to a subset of data and then presenting a closely related quantity as a prediction. The shift-3 versus shift-6/9/12 performance gap is interpreted as evidence of contamination through an external citation (McCoy et al., 2024a), which is independent support rather than a self-citation, and the paper's own Table 12 provides additional observational evidence. The authors' self-citations (Lu et al., 2024a,b) appear only in the related-work discussion and are not load-bearing for the main argument. The closest issue is that the claim that the cipher space is contamination resistant is justified by the dynamism of the generator rather than by directly measuring contamination, and task-level contamination of the canonical shift-3 mapping is not explicitly controlled; however, this is an evidential or correctness limitation, not circularity, because the reported performance figures are not equivalent by construction to the paper's definition or to any fitted input. The benchmark results are self-contained empirical observations, so no circular step can be exhibited.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no fitted parameters or invented entities. Its central claim rests on domain assumptions about contamination control, capability assessment, task difficulty consistency, and external evidence for shift-3 prevalence.

assumptions (4)
  • domain assumption The generated cipher texts, especially those with shifts 6, 9, and 12 and random-word plain texts, are absent from the pre-training data of the tested models.
    The contamination-resistance claim in Section 3 asserts that the infinite instance space makes memorization impossible, but the paper does not measure actual overlap with training corpora. This premise is load-bearing for interpreting the shift-3 advantage as contamination.
  • domain assumption Solving Caesar ciphers requires logical deduction, arithmetic reasoning, and generalization.
    Section 3 and Figure 1 define the benchmark as assessing these capabilities. This is asserted rather than validated; a model could solve ciphers via memorized letter mappings without engaging in reasoning.
  • domain assumption Task complexity is the same across shifts and across natural versus random plain texts.
    Section 3 claims that regardless of shift and plain-text type, the required capabilities are the same, so performance differences indicate competence inconsistency rather than task difficulty. This is a stated premise, not a measured control.
  • domain assumption Shift-3 Caesar ciphers are more prevalent in web corpora than other shifts.
    Section 5.1 cites McCoy et al. 2024a for this external empirical claim and uses it to attribute the shift-3 performance advantage to contamination. The paper does not independently verify prevalence for the tested models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Contamination Resistant Benchmarks." pith.science (2026). https://pith.science/paper/UAROUVVS

@misc{pith2026250508389,
  author       = {Pith},
  title        = {Pith review of: Towards Contamination Resistant Benchmarks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UAROUVVS}},
  note         = {Machine review of arXiv:2505.08389}
}
read the original abstract

The rapid development of large language models (LLMs) has transformed the landscape of natural language processing. Evaluating LLMs properly is crucial for understanding their potential and addressing concerns such as safety. However, LLM evaluation is confronted by various factors, among which contamination stands out as a key issue that undermines the reliability of evaluations. In this work, we introduce the concept of contamination resistance to address this challenge. We propose a benchmark based on Caesar ciphers (e.g., "ab" to "bc" when the shift is 1), which, despite its simplicity, is an excellent example of a contamination resistant benchmark. We test this benchmark on widely used LLMs under various settings, and we find that these models struggle with this benchmark when contamination is controlled. Our findings reveal issues in current LLMs and raise important questions regarding their true capabilities. Our work contributes to the development of contamination resistant benchmarks, enabling more rigorous LLM evaluation and offering insights into the true capabilities and limitations of LLMs.

Figures

Figures reproduced from arXiv: 2505.08389 by the authors.

Figure 1
Figure 1. We propose the concept of contamination resistance and establish a contamination resistant benchmark based on Caesar ciphers, which meets the following criteria: (a) this benchmark tests several capabilities, including logical deduction, arithmetic reasoning, and generalization; (b) this benchmark is dynamic, evolving over time so that an LLM can not memorize all possible queries, and the level of task complexity is… view at source ↗
Figure 2
Figure 2. Exact match accuracy (EMA); GPT-4o; the zero-shot setting. 5.4 The effect of output position [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 21 canonical work pages

  1. [1]

    What learning algorithm is in-context learning? investigations with linear models

    Ekin Aky \" u rek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/forum?id=0g0X4H8yN4I

    2023

  2. [2]

    The surprising effectiveness of test-time training for abstract reasoning

    Ekin Aky \" u rek, Mehul Damani, Linlu Qiu, Han Guo, Yoon Kim, and Jacob Andreas. The surprising effectiveness of test-time training for abstract reasoning. CoRR, abs/2411.07279, 2024. doi:10.48550/ARXIV.2411.07279. URL https://doi.org/10.48550/arXiv.2411.07279

    arXiv doi:10.48550/arxiv.2411.07279 2024

  3. [3]

    Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source llms

    Simone Balloccu, Patr \' cia Schmidtov \' a , Mateusz Lango, and Ondrej Dusek. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source llms. In Yvette Graham and Matthew Purver, editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2024 - Volume 1: Long Paper...

    work page 2024

  4. [4]

    Do, Yan Xu, and Pascale Fung

    Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. In Jong C. Park, Yuki Arase, Baotian Hu, Wei Lu, Derry Wijaya, Ayu Purwarianti, a...

    2023

  5. [5]

    Managing extreme ai risks amid rapid progress

    Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Trevor Darrell, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, et al. Managing extreme ai risks amid rapid progress. Science, 384 0 (6698): 0 842--845, 2024

    2024

  6. [6]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, et al. Language models are few-shot learners. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria - Florina Balcan, and Hsuan - Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on ...

    work page 2020

  7. [7]

    Lundberg, Harsha Nori, Hamid Palangi, Marco T \' u lio Ribeiro, and Yi Zhang

    S \' e bastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco T \' u lio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4 . CoRR, abs/2303.12712, 2023. doi:10.48550/ARXIV.2303.12712. U...

    arXiv doi:10.48550/arxiv.2303.12712 2023

  8. [8]

    Private benchmarking to prevent contamination and improve comparative evaluation of llms

    Nishanth Chandran, Sunayana Sitaram, Divya Gupta, Rahul Sharma, Kashish Mittal, and Manohar Swaminathan. Private benchmarking to prevent contamination and improve comparative evaluation of llms. CoRR, abs/2403.00393, 2024. doi:10.48550/ARXIV.2403.00393. URL https://doi.org/10.48550/arXiv.2403.00393

    arXiv doi:10.48550/arxiv.2403.00393 2024

Show all 56 references
  1. [9]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24: 0 240:1--240:113, 2023. URL http://jmlr.org/papers/v24/22...

    2023

  2. [10]

    Scaling instruction-finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, et al. Scaling instruction-finetuned language models. CoRR, abs/2210.11416, 2022. doi:10.48550/ARXIV.2210.11416. URL https://doi.org/10.48550/arXiv.2210.11416

    2022 doi arXiv

  3. [11]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.14168, 2021. URL https://...

    arXiv

  4. [12]

    Why can GPT learn in-context? language models secretly perform gradient descent as meta-optimizers

    Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can GPT learn in-context? language models secretly perform gradient descent as meta-optimizers. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for ...

    2023

  5. [13]

    Generalization or memorization: Data contamination and trustworthy evaluation for large language models

    Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. Generalization or memorization: Data contamination and trustworthy evaluation for large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Com...

    2024

  6. [14]

    The llama 3 herd of models, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  7. [15]

    Cole, Fangyu Liu, and William W

    Julian Martin Eisenschlos, Jeremy R. Cole, Fangyu Liu, and William W. Cohen. Winodict: Probing language models for in-context word acquisition. In Andreas Vlachos and Isabelle Augenstein, editors, Proceedings of the 17th Conference of the European Chapter of the Association fo...

    2023 doi

  8. [16]

    What can transformers learn in-context? A case study of simple function classes

    Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? A case study of simple function classes. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing S...

    2022

  9. [17]

    Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in chatgpt

    Thilo Hagendorff, Sarah Fabi, and Michal Kosinski. Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in chatgpt. Nature Computational Science, 3 0 (10): 0 833--838, 2023

    2023

  10. [18]

    Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Bias

    Itay Itzhak, Gabriel Stanovsky, Nir Rosenfeld, and Yonatan Belinkov. Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Bias . Transactions of the Association for Computational Linguistics, 12: 0 771--785, 06 2024. ISSN 2307-387X. doi:10.1162/tacl...

    2024 doi

  11. [19]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen - Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar - Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. CoRR, abs/2403.07974, 2024. doi:10.48550/ARXIV....

    2024 doi arXiv

  12. [20]

    Does data contamination make a difference? insights from intentionally contaminating pre-training data for language models

    Minhao Jiang, Ken Liu, Ming Zhong, Rylan Schaeffer, Siru Ouyang, Jiawei Han, and Sanmi Koyejo. Does data contamination make a difference? insights from intentionally contaminating pre-training data for language models. In ICLR 2024 Workshop on Navigating and Addressing Data Pr...

    2024

  13. [21]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annua...

    2022

  14. [22]

    Task contamination: Language models may not be few-shot anymore

    Changmao Li and Jeffrey Flanigan. Task contamination: Language models may not be few-shot anymore. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovati...

    2024

  15. [23]

    Quantifying multilingual performance of large language models across languages

    Zihao Li, Yucheng Shi, Zirui Liu, Fan Yang, Ninghao Liu, and Mengnan Du. Quantifying multilingual performance of large language models across languages. CoRR, abs/2404.11553, 2024. doi:10.48550/ARXIV.2404.11553. URL https://doi.org/10.48550/arXiv.2404.11553

    2024 doi arXiv

  16. [24]

    Sheng Lu, Irina Bigoulaeva, Rachneet Sachdeva, Harish Tayyar Madabushi, and Iryna Gurevych. Are emergent abilities in large language models just in-context learning? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Associ...

    2024

  17. [25]

    Sheng Lu, Hendrik Schuff, and Iryna Gurevych. How are prompts different in terms of sensitivity? In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human La...

    2024

  18. [26]

    Leveraging online olympiad-level math problems for llms training and contamination-resistant evaluation

    Sadegh Mahdavi, Muchen Li, Kaiwen Liu, Christos Thrampoulidis, Leonid Sigal, and Renjie Liao. Leveraging online olympiad-level math problems for llms training and contamination-resistant evaluation. arXiv preprint arXiv:2501.14275, 2025

    2025 arXiv

  19. [27]

    Thomas McCoy, Shunyu Yao, Dan Friedman, Mathew D

    R. Thomas McCoy, Shunyu Yao, Dan Friedman, Mathew D. Hardy, and Thomas L. Griffiths. Embers of autoregression show how large language models are shaped by the problem they are trained to solve. Proceedings of the National Academy of Sciences, 121 0 (41): 0 e2322420121, 2024 a ...

    2024 doi

  20. [28]

    Thomas McCoy, Shunyu Yao, Dan Friedman, Mathew D

    R. Thomas McCoy, Shunyu Yao, Dan Friedman, Mathew D. Hardy, and Thomas L. Griffiths. When a language model is optimized for reasoning, does it still show embers of autoregression? an analysis of openai o1. CoRR, abs/2410.01792, 2024 b . doi:10.48550/ARXIV.2410.01792. URL https...

    doi arXiv

  21. [29]

    Sources of hallucination by large language models on inference tasks

    Nick McKenna, Tianyi Li, Liang Cheng, Mohammad Hosseini, Mark Johnson, and Mark Steedman. Sources of hallucination by large language models on inference tasks. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMN...

    2023 doi

  22. [30]

    Language models implement simple word2vec-style vector arithmetic

    Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Language models implement simple word2vec-style vector arithmetic. CoRR, abs/2305.16130, 2023. doi:10.48550/ARXIV.2305.16130. URL https://doi.org/10.48550/arXiv.2305.16130

    2023 doi arXiv

  23. [31]

    In-context learning generalizes, but not always robustly: The case of syntax

    Aaron Mueller, Albert Webson, Jackson Petty, and Tal Linzen. In-context learning generalizes, but not always robustly: The case of syntax. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Associati...

    2024 doi

  24. [32]

    Niels M \" u ndler, Jingxuan He, Slobodan Jenko, and Martin T. Vechev. Self-contradictory hallucinations of large language models: Evaluation, detection and mitigation. CoRR, abs/2305.15852, 2023. doi:10.48550/ARXIV.2305.15852. URL https://doi.org/10.48550/arXiv.2305.15852

    2023 doi arXiv

  25. [33]

    Know what you don't know: Unanswerable questions for squad

    Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July ...

    2018 doi

  26. [34]

    How much are llms contaminated? A comprehensive survey and the llmsanitize library

    Mathieu Ravaut, Bosheng Ding, Fangkai Jiao, Hailin Chen, Xingxuan Li, Ruochen Zhao, Chengwei Qin, Caiming Xiong, and Shafiq Joty. How much are llms contaminated? A comprehensive survey and the llmsanitize library. CoRR, abs/2404.00699, 2024. doi:10.48550/ARXIV.2404.00699. URL ...

    2024 doi arXiv

  27. [35]

    Prompt programming for large language models: Beyond the few-shot paradigm

    Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Yoshifumi Kitamura, Aaron Quigley, Katherine Isbister, and Takeo Igarashi, editors, CHI '21: CHI Conference on Human Factors in Computing Systems, Virtual Event / Y...

    2021

  28. [36]

    A natural experiment on LLM data contamination in code generation

    Manley Roberts, Himanshu Thakur, Christine Herlihy, Colin White, and Samuel Dooley. A natural experiment on LLM data contamination in code generation. In I Can't Believe It's Not Better Workshop: Failure Modes in the Age of Foundation Models, 2024. URL https://openreview.net/f...

    2024

  29. [37]

    NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark

    Oscar Sainz, Jon Ander Campos, Iker Garc \' a - Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the ...

    2023 doi

  30. [38]

    Language models are greedy reasoners: A systematic formal analysis of chain-of-thought

    Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net...

    2023

  31. [39]

    Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Confer...

    2023

  32. [40]

    Livexiv - A multi-modal live benchmark based on arxiv papers content

    Nimrod Shabtay, Felipe Maia Polo, Sivan Doveh, Wei Lin, Muhammad Jehanzeb Mirza, Leshem Choshen, Mikhail Yurochkin, Yuekai Sun, Assaf Arbelle, Leonid Karlinsky, and Raja Giryes. Livexiv - A multi-modal live benchmark based on arxiv papers content. CoRR, abs/2410.10783, 2024. d...

    doi arXiv

  33. [41]

    Language models are multilingual chain-of-thought reasoners

    Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on Le...

    2023

  34. [42]

    Scaling LLM test-time compute optimally can be more effective than scaling model parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. CoRR, abs/2408.03314, 2024. doi:10.48550/ARXIV.2408.03314. URL https://doi.org/10.48550/arXiv.2408.03314

    2024 doi arXiv

  35. [43]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions ...

    2023

  36. [44]

    Michal S tef \'a nik and Marek Kadl c \' k. Can in-context learners learn a reasoning concept from demonstrations? In Bhavana Dalvi Mishra, Greg Durrett, Peter Jansen, Danilo Neves Ribeiro, and Jason Wei, editors, Proceedings of the 1st Workshop on Natural Language Reasoning a...

    2023 doi

  37. [45]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'A...

    2019

  38. [46]

    Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning

    Xinyi Wang, Wanrong Zhu, and William Yang Wang. Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning. CoRR, abs/2301.11916, 2023. doi:10.48550/ARXIV.2301.11916. URL https://doi.org/10.48550/arXiv.2301.11916

    2023 doi arXiv

  39. [47]

    Emergent analogical reasoning in large language models

    Taylor Webb, Keith J Holyoak, and Hongjing Lu. Emergent analogical reasoning in large language models. Nature Human Behaviour, 7 0 (9): 0 1526--1541, 2023

    2023

  40. [48]

    Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. T...

    2022

  41. [49]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, ...

    2022

  42. [50]

    Livebench: A challenging, contamination-free LLM benchmark

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz - Ziv, Neel Jain, et al. Livebench: A challenging, contamination-free LLM benchmark. CoRR, abs/2406.19314, 2024. doi:10.48550/ARXIV.2406.19314. URL https://doi.org/10.48550/arX...

    2024 doi arXiv

  43. [51]

    Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts

    Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 20...

    2024

  44. [52]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  45. [53]

    A careful examination of large language model performance on grade school arithmetic

    Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati, and Summer Yue. A careful examination of large language model performance on grade school arithmetic. CoRR, a...

    2024 doi arXiv

  46. [54]

    Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A. Smith. How language model hallucinations can snowball. CoRR, abs/2305.13534, 2023 a . doi:10.48550/ARXIV.2305.13534. URL https://doi.org/10.48550/arXiv.2305.13534

    2023 doi arXiv

  47. [55]

    Bartlett

    Ruiqi Zhang, Spencer Frei, and Peter L. Bartlett. Trained transformers learn linear models in-context. CoRR, abs/2306.09927, 2023 b . doi:10.48550/ARXIV.2306.09927. URL https://doi.org/10.48550/arXiv.2306.09927

    2023 doi arXiv

  48. [56]

    Why does chatgpt fall short in answering questions faithfully? CoRR, abs/2304.10513, 2023

    Shen Zheng, Jie Huang, and Kevin Chen - Chuan Chang. Why does chatgpt fall short in answering questions faithfully? CoRR, abs/2304.10513, 2023. doi:10.48550/ARXIV.2304.10513. URL https://doi.org/10.48550/arXiv.2304.10513

    2023 doi arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.