Pith. sign in

REVIEW 4 major objections 4 minor 37 references

Simulating Training Data Leakage in Multiple-Choice Benchmarks for LLM Evaluation

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Under controlled leakage, n-gram detection beats other methods on MCQ benchmarks.

desk verdict The MMLU leakage rates are a false-positive artifact; the controlled simulation and permutation-Q are the real contributions, and the paper needs major revision. read the letter →

arxiv 2505.24263 v1 pith:V6QNSXTN submitted 2025-05-30 cs.CL

Naila Shafirni Hidayat , Muhammad Dehan Al Kautsar , Alfan Farizki Wicaksono , Fajri Koto This is my paper
classification cs.CL
keywords datacontaminationbenchmarkleakagemultiple-choiceQAn-gramdetectionpermutation-QcontinualpretrainingMMLUHellaSwag
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that contamination in multiple-choice benchmarks can be detected reliably before results are released, and that the cheapest accurate way to do it is an n-gram regeneration check. It creates a controlled leakage scenario by continually pretraining four LLMs on 300 randomly chosen MMLU and HellaSwag questions that the models initially answered incorrectly, then compares three detectors. Under that setup, the n-gram method—generate each answer option with earlier options as prompt and compare via ROUGE-L—consistently reaches the highest F1-score, above 81% across settings and up to 100% on HellaSwag. Applying it to the full benchmarks flags up to 39% of MMLU instances for Qwen-7B, and removing flagged instances shifts some subject-level scores and, under the weak leakage definition, the top ranking. The payoff is a practical, instance-level contamination check that works even on closed-weight models.

What carries the argument

The carrying mechanism is the n-gram regeneration test adapted to instance level: for each option $o_i$, prompt the model with the question and earlier options, generate the option, and score the generation against the reference with ROUGE-L; if the ratio of replicated options is at least $T=0.25$, the instance is labeled 'Leakage'. The competing machinery is the permutation family: the original method checks whether the original option order gives the highest log-probability over all $n!$ orders, while the new permutation-Q reduces this to $O(n^2)$ by testing whether the original two-option pair (A-B) maximizes log-probability among all pairs. A third method, semi-half, truncates the question to its final seven words and asks whether the model still selects the correct answer. These detectors are compared on 600-instance simulated leakage sets produced by LoRA continual pretraining.

What would settle it

Take a set of MMLU-style questions that are guaranteed absent from training data but share generic option templates like 'Not Wrong; Wrong; Wrong; Wrong', run the n-gram detector with $T=0.25$ on them, and compare the flagged rate to the reported MMLU leakage rates of 26–39%. If the control set is flagged at a similar rate instead of near zero, the detector is capturing template regularity rather than memorized content, and the cleaned-benchmark numbers overstate contamination.

Watch

Extended reading notes

Core claim

The central claim is that under controlled simulated leakage via continual pretraining, the n-gram method consistently achieves the highest F1-score and is therefore the recommended primary detector for MCQ benchmarks, with permutation-Q as a competitive, cheaper alternative. The n-gram refinement generates each option sentence in one inference, compares it to the reference with ROUGE-L at threshold 0.75, and labels an instance leaked when at least one option replicates, a ratio threshold of $T=0.25$. This instance-level adaptation lets the authors build clean subsets of MMLU and HellaSwag: they remove 6,547 of 14,042 MMLU instances (46.6%) and 38 of 10,042 HellaSwag instances under their strong-leakage definition, then re-evaluate models on the cleaned sets. The accuracy drops are uneven across subjects, with Anatomy falling up to 35.4% for Qwen-7B and Formal Logic declining noticeably, which supports the claim that part of reported performance reflects memorization rather than reasoning.

Load-bearing premise

The full-benchmark leakage numbers rest on the assumption that an instance is contaminated whenever a model can regenerate at least one answer option with ROUGE-L similarity above 0.75 using the question and earlier options as prompt; on repetitive option templates such as 'True/False' or the MMLU moral scenarios, that signal can reflect surface regularity rather than memorization.

Editorial extensions

If this is right

  • Using n-gram as a standard pre-release contamination check would remove roughly half of MMLU under the strong definition, and subject-level scores, especially in Anatomy and Formal Logic, would drop noticeably.
  • Because n-gram needs only generated text, it can audit closed-weight APIs such as GPT-4o and Gemini-2.0-Flash, not just open models.
  • Permutation-Q gives near-n-gram detection at $O(n^2)$ cost, making contamination screening feasible for benchmarks with many options like MMLU-Pro.
  • Leakage does not equal understanding: models answer leaked instances correctly only 39–96% of the time, so removal raises some accuracies under the weak definition, lowers others under the strong definition, and can change the top model.
  • HellaSwag shows very low leakage in these audits (0.12–0.25%), so its scores are more likely to reflect generalization than MMLU's.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the paper's own limitation remarks imply that on repetitive-option subjects the n-gram flag may measure template regularity, so the 46.6% MMLU removal figure is likely an upper bound on true contamination there.
  • Editorial extension: a direct test of the same protocol on free-form reasoning benchmarks, such as math word problems, would show whether n-gram's advantage is specific to the multiple-choice format.
  • Editorial extension: the ranking shift under the weak-leakage definition suggests contamination checks could eventually become a standard leaderboard adjustment rather than a one-off audit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes and evaluates methods for detecting training-data leakage in multiple-choice benchmarks. The authors simulate leakage by continually pre-training four LLMs on 300 MMLU/HellaSwag instances with ground-truth leaked/non-leaked labels, and compare three detection families: semi-half truncation, permutation-based methods (including new variants permutation-R and permutation-Q), and an instance-level n-gram method. They then apply the n-gram method to the full MMLU and HellaSwag benchmarks, report per-model leakage rates, remove flagged instances to create 'cleaned' benchmarks, and re-evaluate models to measure accuracy and ranking shifts.

Significance. The controlled-simulation comparison is a genuine contribution: it provides the first systematic evaluation of leakage detectors under known ground-truth leakage for MCQ benchmarks, and the permutation-Q variant reduces computational complexity from O(n!) to O(n^2) while remaining competitive. The HellaSwag results, where n-gram achieves 100% F1 for Gemma-7B and LLaMA-8B, are striking and indicate the method can be specific when options are natural continuations. The authors also release code and data. However, the significance of the full-benchmark contamination rates and cleaned benchmarks is not supported: on MMLU the n-gram detector flags known-clean instances at a rate comparable to its reported full-benchmark leakage rate, and the threshold/proportion hyperparameters are tuned on the same data used to report final F1 scores.

major comments (4)
  1. [Abstract; Table 1] The abstract's claim that 'the n-gram method consistently achieves the highest F1-score' is contradicted by Table 1. In MMLU, permutation-Q has higher F1 than n-gram for Qwen-7B (82.68 vs. 81.89) and LLaMA-8B (84.27 vs. 84.11), and the original permutation is highest for Gemma-7B (85.10 vs. 82.87). The n-gram method is highest in only two of the eight settings (Qwen-0.5B MMLU and all four HellaSwag settings). Please revise the claim to 'among the best' or report the actual pattern.
  2. [Section 4.2 (Figures 3 and 4)] The n-gram threshold T=0.25 and the permutation-R proportion p=50% are selected by measuring F1 on the same simulated data that later appears in Table 1. Because the hyperparameters are tuned on the test set, the reported F1 scores are in-sample fitted values, and the 'consistently highest' comparison is not a predictive evaluation. The paper should use a held-out validation split (or cross-validation) to select T and p, and report test-set performance separately.
  3. [Section 3.4; Table 6; Figure 6; Limitations] On MMLU, the n-gram detector's false-positive rate on known non-leaked instances is comparable to its reported full-benchmark leakage rate. From Table 6, the FPR on the 300 non-leaked MMLU instances is 25.0% for Qwen-0.5B, 41.3% for Qwen-7B, 41.3% for Gemma-7B, and 37.3% for LLaMA-8B. Figure 6 reports full-benchmark MMLU leakage rates of 26.15%, 39.10%, and 37.94% for Qwen-0.5B, Qwen-7B, and LLaMA-8B, respectively. A detector that flags known-clean instances at the same rate as the full benchmark provides no evidence of contamination. The Limitations section itself concedes the mechanism: repetitive option templates such as 'True/False' and moral_scenarios are regenerable without memorization. The full-benchmark leakage rates and the cleaned MMLU benchmark are therefore not substantiated.
  4. [Section 4.3 (Benchmark Reduction Under Strong Leakage Definition); Section 5.1] The paper introduces a 'strong leakage' definition (flagged and answered correctly) but then removes all instances flagged by n-gram in at least one LLM, regardless of correctness (6,547 of 14,042, 46.6%). There is no comparison against removing a random 46.6% of instances. Given the high false-positive rate documented in the previous comment, the accuracy drops in Figures 7-9 cannot be attributed to memorization; removing a large random subset of a benchmark would be expected to change accuracy, especially for subjects with few items. The assertion that the cleaned results 'provide a more realistic reflection of the models' logical reasoning skills' (Section 5.1) is unsupported without such a baseline.
minor comments (4)
  1. [Section 3.2] The method name 'semi-half' is misleading because the truncation keeps a fixed seven-word suffix rather than half of the question; please clarify or rename the method.
  2. [Section 3.3] In the MAD formula, the variables p_j^i and p_k^i are used but the summation index and the definition of z are only loosely described; please state that i ranges over questions and define p_j^i as the log-probability score for permutation j on question i, and clarify whether scores are length-normalized.
  3. [Figure 6] The MMLU and HellaSwag leakage percentages are shown on a shared axis, but all HellaSwag values are below 0.3% while MMLU values reach 39%; separate panels or a log scale would make the HellaSwag values readable.
  4. [Appendix F] In Table 6, 'Boldscores' should read 'Bold scores'.

Circularity Check

1 steps flagged · score 4.0 of 10

In-sample threshold tuning makes the reported n-gram F1 and full-benchmark leakage rates fitted outputs, not independent predictions.

  1. fitted input called prediction [Section 4.1–4.3, Figure 3, Figure 4, Table 1, Figure 6]
    "After completing the continual pre-training for all eight settings, we tune the threshold of n-gram and optimize the permutation method first. We then use this configuration to evaluate all methods and compare their performance. ... Across all experiments, a threshold of T=0.25 consistently yields the best or comparable F1-score."

    The n-gram threshold T and the permutation-R proportion p are selected by maximizing F1 on the same 600-instance simulated set whose ground-truth labels define the F1 scores reported in Table 1. The headline result that 'the n-gram method consistently achieves the highest F1-score' is therefore an in-sample fitted value, obtained after optimizing the detector's sensitivity on the exact evaluation set. The full-benchmark leakage rates in Figure 6 and the cleaned MMLU accuracy drops inherit this same fitted T, so they are outputs of the tuned rule applied to new inputs rather than predictions validated against independent contamination labels.

full rationale

The controlled leakage simulation has genuine ground-truth labels (300 leaked vs 300 non-leaked instances), so the method comparison is not circular in its design. The only circularity-like step is the in-sample tuning of T=0.25 and p=50% on the same simulated data used to compute Table 1's F1 scores and then reused to produce the full-benchmark 'leakage' rates; this is a fitted-input problem rather than a definitional equivalence. The paper's Limitations section candidly concedes that the n-gram signal misfires on repetitive option templates such as 'True/False' and moral_scenarios, and Table 6 implies false-positive rates of 25–41% on known non-leaked MMLU instances, close to the 26–39% leakage rates in Figure 6; that is a serious external-validity concern, but it is not circular because the full-benchmark instances are not the same as the simulation instances. No load-bearing self-citation chain or imported uniqueness theorem appears. Score 4 reflects the partial circularity from fitting thresholds on the evaluation set while acknowledging the independent content of the simulation methodology.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claims rest on several modeling assumptions about how leakage manifests and on thresholds fitted to the simulation data; no invented entities are introduced.

free parameters (4)
  • T (n-gram leakage threshold) = 0.25
    Selected by maximizing F1 in Figure 3 on the same simulated data later used to report detection performance.
  • p (permutation-R subset proportion) = 50%
    Chosen from Figure 4 because F1 is stable at 50-100% on the same simulated data.
  • t (per-option ROUGE-L similarity threshold) = 0.75
    Adopted from Xu et al. (2024) without re-derivation.
  • semi-half truncation length = 7 words
    Set to the average half-length of MMLU questions; an ad hoc heuristic for the new semi-half method.
assumptions (5)
  • domain assumption LoRA continual pretraining on a benchmark subset simulates real-world training data leakage.
    Section 3.1; the limitation section acknowledges a single benchmark corpus is not fully realistic, though mixing with random data is reported to preserve results.
  • domain assumption Memorized MCQ instances can be detected by generating option text and comparing with ROUGE-L.
    Section 3.4 and Algorithm 2; the n-gram signal assumes leakage manifests as reproducible option text.
  • domain assumption The original option order having maximum log-probability indicates memorization.
    Section 3.3 and Appendix A, following Ni et al. (2024).
  • domain assumption High-perplexity, initially wrong questions are unlikely to have been seen during the base model's pretraining.
    Section 3.1, used to ensure the simulated leakage is the main cause of any post-training memorization.
  • ad hoc to paper Removing instances flagged by n-gram yields a cleaner benchmark that better reflects generalization.
    Section 4.3 strong leakage definition; this is the paper's own construct and is weakened because removing correct answers lowers accuracy by construction and because no random-removal baseline is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Simulating Training Data Leakage in Multiple-Choice Benchmarks for LLM Evaluation." pith.science (2026). https://pith.science/paper/V6QNSXTN

@misc{pith2026250524263,
  author       = {Pith},
  title        = {Pith review of: Simulating Training Data Leakage in Multiple-Choice Benchmarks for LLM Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V6QNSXTN}},
  note         = {Machine review of arXiv:2505.24263}
}
read the original abstract

The performance of large language models (LLMs) continues to improve, as reflected in rising scores on standard benchmarks. However, the lack of transparency around training data raises concerns about potential overlap with evaluation sets and the fairness of reported results. Although prior work has proposed methods for detecting data leakage, these approaches primarily focus on identifying outliers and have not been evaluated under controlled simulated leakage conditions. In this work, we compare existing leakage detection techniques, namely permutation and n-gram-based methods, under a continual pretraining setup that simulates real-world leakage scenarios, and additionally explore a lightweight method we call semi-half question. Although semi-half offers a low-cost alternative, our analysis shows that the n-gram method consistently achieves the highest F1-score. We also refine these techniques to support instance-level detection and reduce computational overhead. Leveraging the best-performing method, we create cleaned versions of MMLU and HellaSwag, and re-evaluate several LLMs. Our findings present a practical path toward more reliable and transparent evaluations, and we recommend contamination checks as a standard step before releasing benchmark results.

Figures

Figures reproduced from arXiv: 2505.24263 by the authors.

Figure 1
Figure 1. Workflow for simulating data leakage and evaluating detection methods. Boxes represent different [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Semi-half truncation example A-B-C-D), it may have memorized that specific multiple-choice instance during training and indi￾cated potential contamination. The detailed method is explained in Appendix A. The algorithm complexity for this method is domi￾nated by computing log-probability scores for each option order variation in a question. In big-Oh no￾tation, the complexity is stated as O(n!), where n denotes the n… view at source ↗
Figure 4
Figure 4. Performance of permutation at different per￾centages p, used to reduce computational complexity. these rankings to identify the top 24 most similar pairs (see [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Performance of permutation-based meth￾ods: the original (P), reduced variant permutation-R (PR), and quick variant permutation-Q (PQ), evalu￾ated on MMLU (M) and HellaSwag (HS). Model ab￾breviations: Q = Qwen, L = Llama, G = Gemma. 4.3 Main Results Detection Performanc…
Figure 6
Figure 6. Figure 6: Data leakage rates for each model on MMLU [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Comparison of model performance on orig [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 9
Figure 9. Figure 9: Performance drops by subject group for each [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 8
Figure 8. Figure 8: Performance drops in selected MMLU sub￾jects for each model. Qwen-7B shows the largest accu￾racy drop in these subjects. Meanwhile, Moral Scenarios contains the high￾est number of detected leaked instances. How￾ever, as observed in previous experiments, this subject yi…
Figure 10
Figure 10. Figure 10: Comparison of model performance on orig [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

37 extracted references · 3 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Simone Balloccu, Patr \'i cia Schmidtov \'a , Mateusz Lango, and Ondrej Dusek. 2024. https://aclanthology.org/2024.eacl-long.5/ Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLM s . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), ...

    2024

  4. [4]

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. https://ojs.aaai.org/index.php/AAAI/article/view/6239 PIQA: reasoning about physical commonsense in natural language . In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence , AAAI, pages 7432--7439, New York, NY, USA

    work page 2020

  5. [5]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    arXiv 2020

  6. [6]

    Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. http://arxiv.org/abs/2202.07646 Quantifying memorization across neural language models

    arXiv 2023

  7. [7]

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1300 B ool Q : Exploring the surprising difficulty of natural yes/no questions . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language...

    doi:10.18653/v1/n19-1300 2019

  8. [8]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

Show all 37 references
  1. [9]

    Yujuan Fu, Ozlem Uzuner, Meliha Yetisgen, and Fei Xia. 2025. http://arxiv.org/abs/2410.18966 Does data contamination detection work (well) for llms? a survey and evaluation on detection assumptions

    2025 arXiv

  2. [10]

    Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile van Krieken, and Pasquale Minervini...

    2025 arXiv

  3. [11]

    Shahriar Golchin and Mihai Surdeanu. 2024. http://arxiv.org/abs/2308.08493 Time travel in llms: Tracing data contamination in large language models

    2024 arXiv

  4. [12]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  5. [13]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021 a . Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  6. [14]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021 b . http://arxiv.org/abs/2009.03300 Measuring massive multitask language understanding

    2021 arXiv

  7. [15]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021 c . Measuring mathematical problem solving with the math dataset. NeurIPS

    2021

  8. [16]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. http://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models

    2021 arXiv

  9. [17]

    Yu, and Xuyun Zhang

    Hongsheng Hu, Zoran Salcic, Lichao Sun, Gillian Dobbie, Philip S. Yu, and Xuyun Zhang. 2022. http://arxiv.org/abs/2103.07853 Membership inference attacks on machine learning: A survey

    2022 arXiv

  10. [18]

    Yucheng Li. 2023. http://arxiv.org/abs/2309.10677 Estimating contamination via perplexity: Quantifying memorisation in language model evaluation

    2023 arXiv

  11. [19]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  12. [20]

    Lin, Jacob Hilton, and Owain Evans

    Stephanie C. Lin, Jacob Hilton, and Owain Evans. 2021. https://api.semanticscholar.org/CorpusID:237532606 Truthfulqa: Measuring how models mimic human falsehoods . In Annual Meeting of the Association for Computational Linguistics

    2021

  13. [21]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  14. [22]

    Inbal Magar and Roy Schwartz. 2022. https://doi.org/10.18653/v1/2022.acl-short.18 Data contamination: From memorization to exploitation . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 157--165, Dublin...

    2022 doi

  15. [23]

    Shiwen Ni, Xiangtao Kong, Chengming Li, Xiping Hu, Ruifeng Xu, Jia Zhu, and Min Yang. 2024. Training on the benchmark is not all you need. arXiv preprint arXiv:2409.01790

    2024 arXiv

  16. [24]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  17. [25]

    Aleksandra Piktus, Christopher Akiki, Paulo Villegas, Hugo Lauren c on, G \'e rard Dupont, Sasha Luccioni, Yacine Jernite, and Anna Rogers. 2023. https://doi.org/10.18653/v1/2023.acl-demo.29 The ROOTS search tool: Data transparency for LLM s . In Proceedings of the 61st Annual...

    2023 doi

  18. [26]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025 arXiv

  19. [27]

    Joshua Robinson, Christopher Michael Rytting, and David Wingate. 2023. http://arxiv.org/abs/2210.12353 Leveraging large language models for multiple choice question answering

    2023 arXiv

  20. [28]

    Vinay Samuel, Yue Zhou, and Henry Peng Zou. 2024. http://arxiv.org/abs/2409.09927 Towards data contamination detection for modern large language models: Limitations, inconsistencies, and oracle challenges

    2024 arXiv

  21. [29]

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. http://arxiv.org/abs/1904.09728 Socialiqa: Commonsense reasoning about social interactions

    2019 arXiv

  22. [30]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  23. [31]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex...

    2024 arXiv

  24. [32]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan...

    2024 arXiv

  25. [33]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. http://arxiv.org/abs/2302.13971 Llama: ...

    2023 arXiv

  26. [34]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. http://arxiv.org/abs/2406.01574 Mmlu-pro: A more robust and ...

    2024 arXiv

  27. [35]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 arXiv

  28. [36]

    Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. 2024. https://arxiv.org/abs/2404.18824 Benchmarking benchmark leakage in large language models . arXiv preprint arXiv:2404.18824

    2024 arXiv

  29. [37]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. http://arxiv.org/abs/1905.07830 Hellaswag: Can a machine really finish your sentence?

    2019 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.