Pith. sign in

REVIEW 4 major objections 5 minor 34 references

Data Laundering: Artificially Boosting Benchmark Results through Knowledge Distillation

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Distillation can launder benchmark answers into a 74% GPQA score

desk verdict Worth engaging, but the headline GPQA number is partly a positional-prior artifact the paper never controls for; Appendix C's own randomized-input results show it. read the letter →

arxiv 2412.15255 v2 pith:FXEMNKQF submitted 2024-12-15 cs.CL cs.AI

Jonibek Mansurov , Akhmed Sakip , Alham Fikri Aji This is my paper · ORCID
classification cs.CLcs.AI
keywords datacontaminationknowledgedistillationbenchmarkmanipulationGPQAMMLU-Reduxevaluationintegrityleakagemodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper demonstrates that knowledge distillation, a standard technique for compressing models, can be subverted to transfer answers from a benchmark's test set into a student model without the student ever seeing the test data. The authors train a teacher on the GPQA test set, then distill it into a 2-layer BERT using the unrelated MedMCQA dataset as an intermediate step. The student reaches 73.94% accuracy on GPQA, close to the 77.30% of a top general-purpose reasoning model, despite having no genuine reasoning capability. The finding implies that public benchmark scores can be inflated through legitimate-looking training pipelines, intentionally or accidentally.

What carries the argument

The central object is the three-phase Data Laundering pipeline built on logit-based knowledge distillation. The loss mixes hard labels from an intermediate dataset with the teacher's soft logits, $L_{\text{student}} = (1-\alpha)L_{\text{hard}} + \alpha L_{\text{soft}}$, and the paper shows that MSE on the logits transfers test-set knowledge more effectively than KL divergence. The distillation step is what converts the teacher's memorized test answers into a student that performs well on the benchmark while having no direct contact with the test set.

What would settle it

Train a teacher on the GPQA training split rather than the test split, distill it through MedMCQA into a 2-layer BERT, and evaluate on the GPQA test. If the student still scores near 74%, the boost is not caused by test-set contamination; if it drops to random level, the laundering explanation is supported.

Watch

Extended reading notes

Core claim

The central claim is that benchmark-specific knowledge can be covertly transferred through knowledge distillation in a three-phase process the authors call Data Laundering: placement (train a teacher on the target benchmark's test set), layering (distill the teacher into a student through an innocuous intermediate dataset, mixing hard labels and the teacher's soft logits), and integration (evaluate on the benchmark). The evidence is that a 2-layer BERT distilled this way through MedMCQA reaches 73.94% on GPQA, compared to 25.76% when trained normally on the same intermediate data without a contaminated teacher. The transfer is robust: it persists across loss functions, alpha values, dataset sizes down to 500 samples, and even when the intermediate data's questions and answers are replaced with random characters, indicating that the format alone carries the leaked signal.

Load-bearing premise

The demonstration assumes the attacker (or unwitting researcher) has access to the full test set of a public benchmark such as GPQA in order to train the teacher; the attack does not apply to private benchmarks whose test answers are withheld.

Editorial extensions

If this is right

  • A small model can be made to look like a much larger one on a leaderboard without gaining any real capability, so reported benchmark scores become untrustworthy.
  • Standard contamination checks that look for n-gram overlap or memorized text will not catch this leak, since the student never sees the test questions.
  • Researchers who distill from a teacher with opaque training data can inflate their scores without intending to, making provenance checks on teachers a necessary safeguard.
  • Private benchmarks with hidden answers become a practical defense, since the teacher in this attack needs the actual test labels.
  • Even meaningless intermediate training data transfers the benchmark signal, so sanitizing the intermediate corpus does not stop the leak.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same logit-based laundering likely transfers to generation tasks or open-ended benchmarks, because the signature lives in the teacher's output distribution rather than in memorized text.
  • The iterative-distillation result implies that one contaminated model can poison a lineage of descendants, so a single leaked benchmark can quietly skew model comparisons for several generations.
  • A concrete detection test would compare a student's agreement with its teacher on benchmark questions against agreement on held-out control questions; anomalous agreement would flag laundering.
  • If this technique spreads, benchmark averages across many models could become systematically inflated over time, eroding the meaning of 'state of the art' even for honestly trained models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces "Data Laundering," a three-stage process (placement, layering, integration) in which a teacher model is first trained on benchmark test data, then distilled into a student model via an intermediate dataset such as MedMCQA or RACE, and finally evaluated on the original benchmark. The central empirical claim is that a 2-layer BERT student can reach 73.94% on GPQA Diamond and 62.31% on MMLU-Redux, approaching OpenAI o1's 77.30% and far exceeding the student's random baseline, with the implication that benchmark-specific knowledge can be covertly transferred through a seemingly legitimate distillation pipeline. The paper includes experiments on loss functions, alpha mixing, iterative distillation, training data size, and artificially degraded intermediate datasets. The authors frame the work as a cautionary tale about evaluation vulnerability and suggest private benchmarks and transparent teacher training as mitigations.

Significance. If the central claim is supported, the paper demonstrates a practically important vulnerability: knowledge distillation can propagate test-set contamination through an intermediate training step, producing models that achieve high benchmark scores without genuine reasoning, and this can occur unintentionally when the teacher's training data is opaque. The authors provide code, specify student-side hyperparameters in Appendix B, and include a useful control experiment in Appendix C (randomized intermediate datasets). However, as detailed in the major comments, the interpretation of the headline result is not yet established because the paper does not control for answer-order position priors, and the comparison to untouched state-of-the-art models is rhetorically misleading. With additional controls and a more careful framing, the paper could still serve as a valuable warning about benchmark integrity.

major comments (4)
  1. [Section 5.1, Table 1] The headline result of 73.94% on GPQA is not currently evidence of covert transfer of benchmark-specific knowledge, because the paper does not report an answer-order permutation control. Appendix C shows that the same 2-layer BERT distillation setup, with both questions and answers replaced by random characters, still achieves 48.99% accuracy on GPQA, far above the 25% random baseline. Since the student never saw real GPQA text in this condition, a large fraction of the observed accuracy can be explained by a learned option-position prior derived from the teacher's logits, not by content knowledge. The authors should report an always-pick-A baseline, analyze the label/answer-position distribution of GPQA, and run an evaluation with randomly permuted answer orders to separate position prior from content transfer. Without these controls, the claim that the method 'launders' test-set knowledge is not supported; at most, the paper demonstrates a benchmark-gaming technique based on fixed option ordering.
  2. [Section 4.1, Table 1] The teacher training setup is ambiguous. Contaminated models are labeled as trained on 'GPQA/MMLU-Redux,' but the paper does not specify whether the teacher is trained jointly on both benchmarks, independently per benchmark, or with a mixed objective, nor does it state the teacher training hyperparameters. Since the same teacher model (e.g., model (1)) is used to produce the GPQA and MMLU-Redux columns, the reader cannot determine whether the MMLU-Redux result reflects a teacher trained on MMLU-Redux test data in the same configuration. Please clarify the teacher training data composition and provide teacher-side hyperparameters (learning rate, epochs, batch size) in Appendix B for reproducibility.
  3. [Section 5.1, Section 5.5] The comparison to OpenAI o1, Claude 3.5 Sonnet, GPT-4o, and LLaMA3-70B is unfair and should be reframed or removed. Those models were not trained on GPQA test data, while the laundered student was distilled from a teacher that was. The claim of 'near state-of-the-art' performance is therefore an artifact of the attack setup, not a meaningful competitive result. The paper should either drop this comparison or explicitly label it as a vulnerability demonstration against an unprivileged baseline, avoiding the implication that the method achieves legitimate SOTA-level capability.
  4. [Section 5, Figures 2-4] All secondary results (loss-function comparison, alpha sweeps, iterative distillation, and data-size curves) are based on single runs, as stated in Section 5. Claims such as 'MSE loss consistently outperforms KLD' and 'the most favorable trade-off ... occurs in the range α=0.5–0.7' are therefore not statistically supported. The authors should either run multiple seeds for these experiments or soften the claims to qualitative observations. This does not affect the paper's main vulnerability message if the headline result survives the permutation control, but it limits the reliability of the design recommendations.
minor comments (5)
  1. [Section 1] In the second paragraph, 'reliance' is misspelled as 'reliance' in the phrase 'as the relience on automated evaluation metrics increases'.
  2. [Section 5.2] The discussion contains a typo: 'constatnt knowledge leakage' should be 'constant knowledge leakage'.
  3. [Section 3.1] The phrase 'trained prohibitively on test data' is unclear; it likely means 'trained on data that is prohibited' or 'trained in a prohibited manner on test data.' Please rephrase for precision.
  4. [Appendix C, Figure 5] The caption for Figure 5 lists four modifications, but the text describes the fourth condition as 'identical questions with identical answers' with 28.65% accuracy; please make the mapping between the caption bullet points and the reported numbers explicit and consistent.
  5. [Section 5.5] The sentence 'One potential way to prevent the unintended use ... is to ensure the teacher model is trained on known dataset like LLM360' contains a grammatical issue ('a known dataset'); please revise.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical attack demonstration; reported benchmark scores are measured outcomes, not quantities derived by construction from fitted inputs.

full rationale

The paper's central claim is that knowledge distillation from a teacher trained on a target benchmark's test set, through an unrelated intermediate dataset, inflates student scores on that benchmark. This is an experimental demonstration, not a derivational chain. The teacher's contamination (training on GPQA test data) is the stated attack premise, and the student's GPQA accuracy (73.94%) is a measured result after distillation on MedMCQA. No equation in Section 3 defines the student's benchmark accuracy as a function of the setup; Equation 1 only defines the distillation loss. Alpha and loss-function choices are swept across a range and reported exhaustively (Tables 2-4, Figure 2) rather than fitted to hit a target score, so there is no fitted-input-called-prediction pattern. The method is defined externally to the benchmarks and evaluated post hoc. The paper contains no load-bearing self-citation: all cited baselines and methods are external prior work. Appendix C's random-input experiments (e.g., 48.99% with random questions and answers) are a correctness/interpretation concern about positional priors, not a circularity concern; they complicate what knowledge is transferred but do not make the measured outcome equivalent to an input. Because the central result is a cautionary empirical demonstration rather than a first-principles derivation, no step reduces to its own inputs by construction.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

No free parameters fitted to data; alpha is swept, not optimized. The axioms are standard domain assumptions about distillation and benchmark accessibility.

assumptions (2)
  • domain assumption Knowledge distillation can transfer specific decision boundaries from teacher to student.
    The entire method relies on the standard property of KD that soft labels encode teacher behavior; prior distillation literature supports this.
  • domain assumption Public benchmark test sets are accessible for training a teacher.
    The attack scenario presumes the adversary can download test answers (e.g., GPQA Diamond is public); true for open benchmarks but not private ones.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Data Laundering: Artificially Boosting Benchmark Results through Knowledge Distillation." pith.science (2026). https://pith.science/paper/FXEMNKQF

@misc{pith2026241215255,
  author       = {Pith},
  title        = {Pith review of: Data Laundering: Artificially Boosting Benchmark Results through Knowledge Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FXEMNKQF}},
  note         = {Machine review of arXiv:2412.15255}
}
read the original abstract

In this paper, we show that knowledge distillation can be subverted to manipulate language model benchmark scores, revealing a critical vulnerability in current evaluation practices. We introduce "Data Laundering," a process that enables the covert transfer of benchmark-specific knowledge through seemingly legitimate intermediate training steps. Through extensive experiments with a 2-layer BERT student model, we show how this approach can achieve substantial improvements in benchmark accuracy (up to 75\% on GPQA) without developing genuine reasoning capabilities. Notably, this method can be exploited intentionally or even unintentionally, as researchers may inadvertently adopt this method and inflate scores without realising the implications. While our findings demonstrate the effectiveness of this technique, we present them as a cautionary tale highlighting the urgent need for more robust evaluation methods in AI. This work aims to contribute to the ongoing discussion about evaluation integrity in AI development and the need for benchmarks that more accurately reflect true model capabilities. The code is available at https://github.com/mbzuai-nlp/data_laundering.

Figures

Figures reproduced from arXiv: 2412.15255 by the authors.

Figure 1
Figure 1. The Data Laundering framework parallels traditional money laundering phases: Placement (knowledge [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Impact of Loss Function Type and Alpha Parameter on Training and Benchmark Accuracy. This plot shows the accuracy trends of a 2-layer BERT and GPT-2 student model across varying values of the balancing parameter α (0 to 1.0), comparing the effects of MSE and KLD loss functions on GPQA. Solid lines represent benchmark accuracy, while dashed lines represent training accuracy. The choice of training data matters (MedMC… view at source ↗
Figure 3
Figure 3. Impact of Iterative Knowledge Distillation on Training and Benchmark Accuracy. This plot shows the accuracy trends of a 2-layer BERT (circle) and GPT-2 (cross) student model in iterative knowledge distillation (5 iterations) with α 0.6 (blue line) and 1.0 (yellow line), MSE loss function. 5.3 Iterative Data Laundering [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Impact of Dataset Size on Training and Benchmark Accuracy. This plot shows the accuracy trends of a 2-layer BERT (circle) and GPT-2 (cross) student model across varying values of the training size (500 to 25000) with α 0.6 (blue line) and 1.0 (yellow line), MSE loss fu…
Figure 5
Figure 5. Figure 5: Impact of Artificially Modifying the Distillation Dataset on the Benchmark Accuracy. This bar plot shows the evaluation accuracy on GPQA using a 2-layer BERT teacher-student pair with α = 1.0 when 1) replacing each answer choice in MedMCQA with 10 random characters, 2)…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 6 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  5. [5]

    Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, and Zhifeng Chen et al

    Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, and Zhifeng Chen et al. 2023. http://arxiv.org/abs/2305.10403 Palm 2 technical report

    arXiv 2023

  6. [6]

    Simone Balloccu, Patr \' cia Schmidtov \'a , Mateusz Lango, and Ondrej Dusek. 2024. https://aclanthology.org/2024.eacl-long.5 Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLM s . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pa...

    2024

  7. [7]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  8. [8]

    Xu Cheng, Zhefan Rao, Yilan Chen, and Quanshi Zhang. 2020. Explaining knowledge distillation by quantifying the knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    work page 2020

Show all 34 references
  1. [9]

    Gonzalez, and Ion Stoica

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. http://arxiv.org/abs/2403.04132 Chatbot arena: An open platform for evaluating llms by human preference

    2024 arXiv

  2. [10]

    Xiao Cui, Yulei Qin, Yuting Gao, Enwei Zhang, Zihan Xu, Tong Wu, Ke Li, Xing Sun, Wengang Zhou, and Houqiang Li. 2024. https://aclanthology.org/2024.lrec-main.1293/ S inkhorn distance minimization for knowledge distillation . In Proceedings of the 2024 Joint International Conf...

    2024

  3. [11]

    Hashimoto

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. 2024. http://arxiv.org/abs/2404.04475 Length-controlled alpacaeval: A simple way to debias automatic evaluators

    2024 arXiv

  4. [12]

    Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile Van Krieken, and Pasquale Minervini...

    2025

  5. [13]

    Shahriar Golchin and Mihai Surdeanu. 2024. http://arxiv.org/abs/2308.08493 Time travel in llms: Tracing data contamination in large language models

    2024 arXiv

  6. [14]

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2024. Minillm: Knowledge distillation of large language models. In Proceedings of ICLR

    2024

  7. [15]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  8. [16]

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. http://arxiv.org/abs/1503.02531 Distilling the knowledge in a neural network

    2015 arXiv

  9. [17]

    Daphne Ippolito, Florian Tramer, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher Choquette Choo, and Nicholas Carlini. 2023. https://doi.org/10.18653/v1/2023.inlg-main.3 Preventing generation of verbatim memorization in language models gives a false se...

    2023 doi

  10. [18]

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, page 2. Minneapolis, Minnesota

    2019

  11. [19]

    Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. 2024. http://arxiv.org/abs/2402.03898 Distillm: Towards streamlined distillation for large language models

    2024 arXiv

  12. [20]

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. https://doi.org/10.18653/v1/D17-1082 RACE : Large-scale R e A ding comprehension dataset from examinations . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages...

    2017 doi

  13. [21]

    Gonzalez, and Ion Stoica

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. 2024. http://arxiv.org/abs/2406.11939 From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline

    2024 arXiv

  14. [22]

    Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Yonghao Zhuang, Guowei He, Haonan Li, Fajri Koto, Liping Tang, Nikhil Ranjan, Zhiqiang Shen, Xuguang Ren, Rob...

    2023 arXiv

  15. [23]

    Inbal Magar and Roy Schwartz. 2022. https://doi.org/10.18653/v1/2022.acl-short.18 Data contamination: From memorization to exploitation . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 157--165, Dublin...

    2022 doi

  16. [24]

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. https://proceedings.mlr.press/v174/pal22a.html Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering . In Proceedings of the Conference on Health, Inference, and Lea...

    2022

  17. [25]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf Language models are unsupervised multitask learners . OpenAI blog, 1(8):9

    2019

  18. [26]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. http://jmlr.org/papers/v21/20-074.html Exploring the limits of transfer learning with a unified text-to-text transformer . Journal of Machine Lea...

    2020

  19. [27]

    Tanmay Rajore, Nishanth Chandran, Sunayana Sitaram, Divya Gupta, Rahul Sharma, Kashish Mittal, and Manohar Swaminathan. 2024. http://arxiv.org/abs/2403.00393 Truce: Private benchmarking to prevent contamination and improve comparative evaluation of llms

    2024 arXiv

  20. [28]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. https://openreview.net/forum?id=Ti67584b98 GPQA : A graduate-level google-proof q&a benchmark . In First Conference on Language Modeling

    2024

  21. [29]

    Oscar Sainz, Jon Ander Campos, Iker Garc \' a-Ferrero, Julen Etxaniz, and Eneko Agirre. 2023. Did chatgpt cheat on your test. Last accessed: 18th July

    2023

  22. [30]

    Brown, Adam Santoro, Aditya Gupta, and Adrià Garriga-Alonso et al

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, and Adrià Garriga-Alonso et al. 2023. http://arxiv.org/abs/2206.04615 Beyond the imitation game: Quantifying and extrapolating the capabil...

    2023 arXiv

  23. [31]

    Geras, Samira Ebrahimi Kahou, Ozlem Aslan, Shengjie Wang, Abdelrahman Mohamed, Matthai Philipose, Matt Richardson, and Rich Caruana

    Gregor Urban, Krzysztof J. Geras, Samira Ebrahimi Kahou, Ozlem Aslan, Shengjie Wang, Abdelrahman Mohamed, Matthai Philipose, Matt Richardson, and Rich Caruana. 2017. https://openreview.net/forum?id=r10FA8Kxg Do deep convolutional nets really need to be deep and convolutional? ...

    2017

  24. [32]

    Gonzalez, and Ion Stoica

    Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and Ion Stoica. 2023. http://arxiv.org/abs/2311.04850 Rethinking benchmark and contamination for language models with rephrased samples

    2023 arXiv

  25. [33]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/91f18a1287b398d378ef22505bf41832-Pap...

    2023

  26. [34]

    Xiaosen Zheng, Tianyu Pang, Chao Du, Qian Liu, Jing Jiang, and Min Lin. 2024. http://arxiv.org/abs/2410.07137 Cheating automatic llm benchmarks: Null models achieve high win rates

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.