Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Position: AI Competitions Provide the Gold Standard for Empirical Rigor in GenAI Evaluation

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AI Competitions, not static benchmarks, should set the standard for GenAI evaluation.

desk verdict A clearly argued, well-disclosed position paper that makes a real case for competitions, but overreaches when it claims static benchmarks can never be trusted. read the letter →

arxiv 2505.00612 v2 pith:EX274QEH submitted 2025-05-01 cs.AI

D. Sculley , Will Cukierski , Phil Culliton , Sohier Dane , Maggie Demkin , Ryan Holbrook , Addison Howard , Paul Mooney
show 3 more authors
Walter Reade Megan Risdal Nate Keating
This is my paper · ORCID
classification cs.AI
keywords generativeAIevaluationdataleakagebenchmarkcontaminationLLMcompetitionsnovelty-centricgeneralizationstaticbenchmarksrobustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Machine learning's traditional paradigm of a fixed training set and a hidden test set, reused as a public benchmark, assumes test examples are drawn from the same distribution as training data and stay unseen. The paper argues that this paradigm breaks for generative models, whose input and output spaces are effectively unbounded and whose outputs feed back into context. It therefore treats leakage and contamination as the central dangers, adopting the rule of thumb that an evaluation is leaked the moment it is shared online or sent to a model over the wire. From that premise it concludes that public static benchmarks cannot be trusted for GenAI and that time-bound AI competitions, which test many models in parallel on data that does not exist at training time, provide the strongest available empirical rigor.

What carries the argument

The central object is the AI Competition, defined by the paper as a task with an objective evaluation function in which independent teams make parallel, time-bound attempts. Its load-bearing feature is that test data is kept secret and, in the strongest cases, does not exist during the training phase—via prospective ground truth, novel task generation, or post-deadline data collection. The paper also places the leakage rule of thumb—'leaked the moment it has been shared online or sent over the wire'—as the mechanism that invalidates static benchmarks.

What would settle it

Construct a matched pair of evaluations from the same source—one a long-public static benchmark, the other a fresh set written after model training cutoffs—and run the same models on both; the claim is falsified if rank ordering and score gaps on the public benchmark closely reproduce on the fresh set across many tasks and models.

Watch

Extended reading notes

Core claim

Generative AI cannot be reliably evaluated with the traditional train/test-split benchmark, because GenAI models have nearly unbounded input and output spaces, no well-defined ground truth, and feedback loops that break the IID assumption; and because any evaluation data that is shared online or sent to a model is, by the paper's rule of thumb, already leaked. The paper's central claim is that the field should therefore regard static public benchmarks as invalidated once published and treat AI Competitions—time-bound, parallel, independently attempted tasks with hidden test data—as the gold standard for empirical rigor. It argues that competition structures such as prospective ground truth, novel task generation, and post-deadline data collection can create leak-proof evaluations, and that reproducibility should be sacrificed for robustness by replacing immutable benchmarks with repeatable processes.

Load-bearing premise

The argument stands or falls on the empirical premise that any evaluation data shared online or sent over the wire to a model is effectively leaked and cannot be kept out of training data; if some shared test sets remain effectively unknown, or contamination can be detected and filtered, static benchmarks could stay valid and competitions lose their uniquely protected status.

Editorial extensions

If this is right

  • Once an evaluation is published online or sent to a model, treat its results as potentially contaminated; static benchmark scores should carry much less weight in GenAI comparisons.
  • Evaluation should be organized as simultaneous, time-bound attempts on novel tasks, so that every model sees the test material for the first time at the same moment.
  • In cases where reproducibility and robustness conflict, robustness wins: better a one-time trustworthy result than a repeatable result that may be contaminated.
  • The field should invest in meta-analyses of competition results, synthesizing methods and outcomes across many tasks rather than trusting any single static benchmark.
  • Competition-style anti-cheating structures—hidden test data, trusted offline execution, and post-deadline data collection—should become the model for general GenAI evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that high-stakes model choices, such as deploying a model in medicine or law, should not be made from public leaderboard scores once the leakage rule of thumb is accepted.
  • A testable extension of the paper's position: competition platforms could publish periodic 'contamination audits' by measuring model probabilities on held-out competition data before and after public release, giving an empirical handle on the leakage rule.
  • The same time-bound, secret-test structure could be ported to agentic and tool-use evaluations, where leakage through environment data is even harder to detect than in text benchmarks.
  • If the rule of thumb holds, benchmark designers should shift effort from bigger static question sets to renewable pipelines and agreements with API providers not to train on evaluation traffic.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This position paper, authored by Kaggle employees, argues that traditional IID-based static benchmarking is fundamentally inadequate for generative AI, and that the dominant problem is leakage and contamination. The authors assert that an evaluation is effectively leaked the moment its test data has been shared online or sent over the wire, and that consequently no published static benchmark can be robust to leakage. They argue that AI competitions, which combine time-bound secret tests, parallel independent attempts, and anti-cheating mechanisms, provide the gold standard for empirical rigor in GenAI evaluation, and they recommend that the field shift from reproducible static benchmarks to repeatable processes and value meta-analyses of competition results. The paper supports these claims with Kaggle leakage case studies, a review of existing leak-avoidance benchmark designs, and descriptions of competition structures that use prospective ground truth, novel task generation, and post-deadline data collection.

Significance. If the paper's central impossibility claim were established, it would be a significant and timely intervention in how the field evaluates generative models. The paper's real strengths are its concrete catalog of leakage mechanisms from a decade of Kaggle competitions (SETI, TalkingData, LANL, Predict AI Model Runtime), and its explicit, actionable descriptions of leak-resilient competition designs such as CAFA 5, AIMO, the WSDM Cup, and the Konwinski Prize. These examples provide a useful practical blueprint, and the paper is honest enough to discuss alternative viewpoints. The fundamental weakness is that the load-bearing empirical premise—that any shared evaluation is irrevocably leaked and that a static published benchmark cannot be trustworthy—is asserted rather than demonstrated, and it sits in tension with the paper's own citation of Recht et al. (2019) and Roelofs et al. (2019a,b), which show rank-order stability under heavy public reuse. As a position statement it is valuable, but the central generalization currently outruns the evidence.

major comments (4)
  1. [Section 4.1] The rule of thumb that an evaluation is 'leaked the moment it has been shared online or sent over the wire' and the ensuing claim that 'we simply cannot have a published static benchmark that is robust to leakage' are load-bearing for the paper's conclusion, but they are supported only by a small set of Kaggle leakage anecdotes. The case studies in Section 4 (SETI, TalkingData, LANL, and the AI Model Runtime competition) involve file metadata, row ordering, randomization seeds, and synthetic-signal artifacts; they do not establish that LLM results on static benchmarks are generally invalidated by contamination, nor do they rule out detection and filtering of contaminated examples. This is an overgeneralization from a selected sample, and since the 'gold standard' recommendation depends on the impossibility claim, it needs substantially stronger support or a clearly qualified formulation.
  2. [Section 2.2 vs 4.1] The paper's own Section 2.2 cites Recht et al. (2019), which showed that ImageNet rank ordering was preserved on brand new data despite massive reuse, and Roelofs et al. (2019a,b), which showed that public leaderboard performance on Kaggle competitions was a strong indicator of private holdout rank ordering. That evidence directly cuts against the universal 'leaked the moment shared' rule, which treats any public exposure as catastrophically invalidating. The authors should either explain why GenAI evaluation is qualitatively different in ways that make contamination destroy the validity of comparisons (e.g., unbounded output spaces, memorization), or soften the impossibility claim to a claim about individual model scores rather than rank ordering. Without such engagement, the central thesis is internally inconsistent with evidence the paper itself presents.
  3. [Section 6.2 and 8] The paper claims that AI competitions provide 'the gold standard for empirical rigor' but never defines the criteria for that designation, nor does it systematically compare competitions against the alternatives reviewed in Section 5 (unreleased holdout sets, dynamic benchmarks, and community benchmarks). The possible weaknesses of competitions—such as task-selection bias, incentive effects of prizes, deadline-driven submission strategies, and the fact that leakage also occurs in competitions, as Section 4 recounts—are not weighed transparently against the claimed benefits. For a prescriptive position paper, the comparison should be made explicit and the term 'gold standard' operationally defined (e.g., contention-resistance, statistical power, ecological validity, and reproducibility).
  4. [Section 4.1] The asserted 'fundamental tension' between reproducibility and robustness, likened to the Heisenberg uncertainty principle, is presented rhetorically rather than argued. The text offers no proof or evidence that no reproducible static benchmark can also be robust to leakage. The hybrid FACTS Grounding Leaderboard discussed in Section 5.1, which publishes half the test set while keeping the other half private precisely to allow reproducibility checks, suggests a tradeoff with design options rather than an impossibility. The claim should be demoted from a universal impossibility to a practical difficulty or supported with an explicit argument.
minor comments (5)
  1. [Author affiliations] The paper should include an explicit conflict-of-interest statement: the authors are Kaggle employees and the paper argues for increased use of AI competition results, so readers would benefit from a transparent disclosure beyond the affiliation line.
  2. [Section 8] The claim that ensembling submissions 'obtains little-to-no improvement to top ranked solutions' is presented without data or citation; either provide the underlying analysis or qualify it as an informal observation.
  3. [Abstract and Section 1] The phrase 'gold standard' is used as a conclusion rather than a defined criterion; a brief operational definition would make the position more testable and reduce the risk of circular reasoning.
  4. [Section 3.2] The quoted Nectar dataset description contains 'Antropic/hh-rlhf', which appears to be a typo; please mark it as '[sic]' or correct the quotation.
  5. [Section 5.1] The FACTS Grounding Leaderboard is mentioned as a hybrid approach but only briefly; one or two sentences on how the public/private split is used would help the reader understand the reproducibility-robustness tradeoff the paper discusses.

Circularity Check

1 steps flagged · score 4.0 of 10

The 'published static benchmarks cannot be robust' conclusion is stipulated by the Section 4.1 leakage rule of thumb, but the broader competitions-as-gold-standard claim retains independent content.

  1. self definitional [Section 4.1, applied in Section 8]
    "we argue that it is simplest and safest to adopt a leakage rule of thumb that an evaluation should be considered leaked the moment it has been shared online or sent over the wire."

    This rule makes 'shared online' the definition of 'leaked.' The paper then presents the impossibility of robust published static benchmarks as a discovered consequence, concluding in Section 8 that 'static benchmarks should be considered to have been effectively invalidated once they have been published.' But once a benchmark is published it has, by definition, been shared online, so the invalidation conclusion is just the Section 4.1 rule restated rather than a result established by the leakage case studies. The SETI, TalkingData, and LANL anecdotes show that leakage happened in specific competitions; they do not establish the universal rule, which is doing the load-bearing work in the conclusion.

full rationale

This is a position paper, not an empirical derivation. There are no fitted parameters, no equations, and no quantitative prediction that is obtained from a fitted input. The leakage case studies come from Kaggle competitions, and several are cited with Kaggle staff as authors, but the events themselves are external and partly documented outside the paper (for example, OpenVaccine appeared in Nature Machine Intelligence, and CAFA is a community challenge), so this self-citation is not the kind of load-bearing self-reference that forces a result. The genuine circularity-like step is Section 4.1's stipulation that an evaluation is 'leaked the moment it has been shared online or sent over the wire.' Section 8's conclusion that published static benchmarks are 'effectively invalidated' follows directly from that definition, not from the case studies. However, the central recommendation—to value time-bound, parallel, secret-test competitions—has independent structural support (prospective ground truth, post-deadline data collection, isolated code execution) and is not derived from a fitted quantity or a self-citation chain. The score is therefore 4 rather than 6 or higher: there is a definitional shortcut at the core of one sub-claim, but the overall thesis does not reduce to a fit or to the authors' prior work.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper introduces no free parameters and no invented entities. It relies on several domain assumptions about the nature of GenAI generalization and about the trustworthiness of competition platforms, plus two ad hoc assumptions that are effectively the paper's empirical premises: the leakage rule of thumb and the impossibility of leak-proof static benchmarks. These premises carry most of the argument's weight and are not proven.

assumptions (5)
  • domain assumption GenAI generalization should be evaluated as novelty-centric generalization: performance on tasks never seen in training or development.
    Section 3 introduces this as 'the form of generalization the field should most care about'; it is a value judgment, not a derived result.
  • ad hoc to paper Any evaluation data shared online or sent over the wire to a model is effectively leaked and cannot be trusted.
    Section 4.1 states this as a 'leakage rule of thumb'; it is the central empirical premise of the paper and is stated without systematic evidence.
  • ad hoc to paper No published static benchmark can be robust to leakage; 'we simply cannot have a published static benchmark that is robust to leakage'.
    Section 4.1 asserts an impossibility claim, supported only by examples and anecdote, not by a proof or a systematic study.
  • domain assumption Competition platforms can serve as trusted keepers of hidden test data, including running isolated code competitions without network access.
    Section 6.1 assumes platform security and integrity; no independent audit of a platform's handling of hidden test data is cited.
  • domain assumption A steady stream of time-bound competitions provides a renewable pipeline of novel tasks whose results are directly comparable across models.
    Section 7 recommends the field depend on this pipeline; the claim that competition results remain comparable over time is assumed rather than demonstrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Position: AI Competitions Provide the Gold Standard for Empirical Rigor in GenAI Evaluation." pith.science (2026). https://pith.science/paper/EX274QEH

@misc{pith2026250500612,
  author       = {Pith},
  title        = {Pith review of: Position: AI Competitions Provide the Gold Standard for Empirical Rigor in GenAI Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EX274QEH}},
  note         = {Machine review of arXiv:2505.00612}
}
read the original abstract

In this position paper, we observe that empirical evaluation in Generative AI is at a crisis point since traditional ML evaluation and benchmarking strategies are insufficient to meet the needs of evaluating modern GenAI models and systems. There are many reasons for this, including the fact that these models typically have nearly unbounded input and output spaces, typically do not have a well defined ground truth target, and typically exhibit strong feedback loops and prediction dependence based on context of previous model outputs. On top of these critical issues, we argue that the problems of leakage and contamination are in fact the most important and difficult issues to address for GenAI evaluations. Interestingly, the field of AI Competitions has developed effective measures and practices to combat leakage for the purpose of counteracting cheating by bad actors within a competition setting. This makes AI Competitions an especially valuable (but underutilized) resource. Now is time for the field to view AI Competitions as the gold standard for empirical rigor in GenAI evaluation, and to harness and harvest their results with according value.

Figures

Figures reproduced from arXiv: 2505.00612 by the authors.

Figure 1
Figure 1. IID Evaluations vs. Novelty-Centric Evaluations. In the IID evaluation, left, both training and test data are drawn from the same distribution, resulting in significant overlap in examples in each set. In the novelty-centric version, right, no test example is allowed to be too similar to any given training example. We argue that the latter conceptualization more closely mirrors desired behavior for GenAI evaluations… view at source ↗
Figure 2
Figure 2. Comparing sequential and parallelized evaluation structures. In the traditional research structure, top, each new idea is evaluated in a linear sequence that typically requires several months for a single pass. The parallelized structure, bottom, allows hundreds or thousands of approaches to be simultaneously. a level of trust. In particular, providers must be trusted not to change their policies, but even more impo… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security

    cs.CR 2026-07 conditional novelty 7.0 of 10

    Adaptive multi-turn LLM attacks raise attack success from near 0% to 5.4–14.0% on frontier defenders, with scenario-specific defender weaknesses that aggregate scores hide.

  2. CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfitting

    cs.LG 2025-05 accept novelty 7.0 of 10

    Capping achievable accuracy with randomized correct answers turns any model that exceeds the cap into a detectable contamination alarm.

Reference graph

Works this paper leans on

51 extracted references · 27 canonical work pages · cited by 2 Pith papers

  1. [1]

    Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs , 2024

    Balloccu, S., Schmidtová, P., Lango, M., and Dušek, O. Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs , 2024. URL https://arxiv.org/abs/2402.03927

    arXiv 2024

  2. [2]

    State of M achine L earning C ompetitions in 2024

    Carlens, H. State of M achine L earning C ompetitions in 2024 . ML Contests Research, 2025. https://mlcontests.com/state-of-machine-learning-competitions-2024

    work page 2024

  3. [3]

    O'Reilly Media, Inc

    Chen, C., Murphy, N. R., Parisa, K., Sculley, D., and Underwood, T. Reliable Machine Learning . " O'Reilly Media, Inc.", 2021 a

    work page 2021

  4. [4]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Ch...

    arXiv 2021

  5. [5]

    E., Stoica, I., Mooney, P., Dane, S., Howard, A., and Keating, N

    Chiang, W.-L., Zheng, L., Dunlap, L., Gonzalez, J. E., Stoica, I., Mooney, P., Dane, S., Howard, A., and Keating, N. LMSYS - Chatbot Arena Human Preference Predictions . https://kaggle.com/competitions/lmsys-chatbot-arena, 2024 a . Kaggle

    work page 2024

  6. [6]

    N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J

    Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J. E., and Stoica, I. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference , 2024 b . URL https://arxiv.org/abs/2403.04132

    arXiv 2024

  7. [7]

    On the Measure of Intelligence , 2019

    Chollet, F. On the Measure of Intelligence , 2019. URL https://arxiv.org/abs/1911.01547

    arXiv 2019

  8. [8]

    Training Verifiers to Solve Math Word Problems , 2021

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training Verifiers to Solve Math Word Problems , 2021. URL https://arxiv.org/abs/2110.14168

    arXiv 2021

Show all 51 references
  1. [9]

    and Ghemawat, S

    Dean, J. and Ghemawat, S. MapReduce: simplified data processing on large clusters . Commun. ACM, 51 0 (1): 0 107–113, January 2008. ISSN 0001-0782. doi:10.1145/1327452.1327492. URL https://doi.org/10.1145/1327452.1327492

    2008

  2. [10]

    ImageNet: A large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database . In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 248--255, 2009. doi:10.1109/CVPR.2009.5206848

    2009

  3. [11]

    Dennett, D. C. Consciousness Explained . Penguin Books, 1991

    1991

  4. [12]

    A Large Scale Benchmark for Uplift Modeling

    Diemert Eustache, Betlei Artem , Renaudin, C., and Massih-Reza, A. A Large Scale Benchmark for Uplift Modeling . In Proceedings of the AdKDD and TargetAd Workshop, KDD, London,United Kingdom, August, 20, 2018. ACM, 2018

    2018

  5. [13]

    Open LLM Leaderboard v2

    Fourrier, C., Habib, N., Lozovskaya, A., Szafer, K., and Wolf, T. Open LLM Leaderboard v2 . https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard, 2024

    2024

  6. [14]

    D., Piovesan, D., Joshi, P., Reade, W., and Howard, A

    Friedberg, I., Radivojac, P., Paolis, C. D., Piovesan, D., Joshi, P., Reade, W., and Howard, A. CAFA 5 Protein Function Prediction . https://kaggle.com/competitions/cafa-5-protein-function-prediction, 2023. Kaggle

    2023

  7. [15]

    C., Buzzard, K., Gowers, T., Liu, P

    Frieder, S., Bealing, S., Nikolaiev, A., Smith, G. C., Buzzard, K., Gowers, T., Liu, P. J., Loh, P.-S., Mackey, L., de Moura, L., Roberts, D., Sculley, D., Tao, T., Balduzzi, D., Coyle, S., Gerko, A., Holbrook, R., Howard, A., and Markets, X. AI Mathematical Olympiad - Progres...

    2024

  8. [16]

    Glazer, E., Erdil, E., Besiroglu, T., Chicharro, D., Chen, E., Gunning, A., Olsson, C. F., Denain, J.-S., Ho, A., de Oliveira Santos, E., Järviniemi, O., Barnett, M., Sandler, R., Vrzala, M., Sevilla, J., Ren, Q., Pratt, E., Levine, L., Barkley, G., Stewart, N., Grechuk, B., G...

    2024 arXiv

  9. [17]

    Measuring Massive Multitask Language Understanding , 2021

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring Massive Multitask Language Understanding , 2021. URL https://arxiv.org/abs/2009.03300

    2021 arXiv

  10. [18]

    S., Swirhun, J., Xing, J., Wang, L., Gurumurthy, M., Aaron, M., Ambar, M., Fellinger, R., Wang, R., Zhang, Z., Goldshtein, S., and Das, D

    Jacovi, A., Wang, A., Alberti, C., Tao, C., Lipovetz, J., Olszewska, K., Haas, L., Liu, M., Keating, N., Bloniarz, A., Saroufim, C., Fry, C., Marcus, D., Kukliansky, D., Tomar, G. S., Swirhun, J., Xing, J., Wang, L., Gurumurthy, M., Aaron, M., Ambar, M., Fellinger, R., Wang, R...

    2025 arXiv

  11. [19]

    LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

    Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code . arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  12. [20]

    E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K

    Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? , 2024. URL https://arxiv.org/abs/2310.06770

    2024 arXiv

  13. [21]

    Leakage in data mining: Formulation, detection, and avoidance

    Kaufman, S., Rosset, S., Perlich, C., and Stitelman, O. Leakage in data mining: Formulation, detection, and avoidance . ACM Transactions on Knowledge Discovery from Data (TKDD), 6 0 (4), December 2012. ISSN 1556-4681. doi:10.1145/2382577.2382579. URL https://doi.org/10.1145/23...

    2012

  14. [22]

    UCI Machine Learning Repository , 2025

    Kelly, M., Longjohn, R., and Nottingham, K. UCI Machine Learning Repository , 2025. URL https://archive.ics.uci.edu

    2025

  15. [23]

    Konwinski, A., Rytting, C., Shaw, J. F. A., Dane, S., Reade, W., and Demkin, M. Konwinski Prize . https://kaggle.com/competitions/konwinski-prize, 2024. Kaggle

    2024

  16. [24]

    and Cortes, C

    LeCun, Y. and Cortes, C. MNIST handwritten digit database, 2010. URL http://yann.lecun.com/exdb/mnist/

    2010

  17. [25]

    and Schwartz, R

    Magar, I. and Schwartz, R. Data Contamination: From Memorization to Exploitation . In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp.\ 157--165, Dublin, ...

    2022 doi

  18. [26]

    P., Santorini, B., and Marcinkiewicz, M

    Marcus, M. P., Santorini, B., and Marcinkiewicz, M. A. Building a large annotated corpus of E nglish: The P enn T reebank. Computational Linguistics, 19 0 (2): 0 313--330, 1993. URL https://aclanthology.org/J93-2004/

    1993

  19. [27]

    Mitchell, T. M. Machine Learning , volume 1. McGraw-hill New York, 1997

    1997

  20. [28]

    MTEB : M assive T ext E mbedding B enchmark, 2023

    Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. MTEB : M assive T ext E mbedding B enchmark, 2023. URL https://arxiv.org/abs/2210.07316

    2023 arXiv

  21. [29]

    Handbook of Statistical Analysis & Data Mining Applications

    Nisbet, R., Elder, J., and Miner, G. Handbook of Statistical Analysis & Data Mining Applications . Elsevier, Inc, 2009

    2009

  22. [30]

    Oren, Y., Meister, N., Chatterji, N., Ladhak, F., and Hashimoto, T. B. Proving Test Set Contamination in Black Box Language Models , 2023. URL https://arxiv.org/abs/2310.17623

    2023 arXiv

  23. [31]

    Y., Zhao, H., Yu, M., Gangal, V., Zou, C., Wang, Z., Wang, J

    Phan, L., Gatti, A., Han, Z., Li, N., Hu, J., Zhang, H., Shi, S., Choi, M., Agrawal, A., Chopra, A., Khoja, A., Kim, R., Hausenloy, J., Zhang, O., Mazeika, M., Anderson, D., Nguyen, T., Mahmood, M., Feng, F., Feng, S. Y., Zhao, H., Yu, M., Gangal, V., Zou, C., Wang, Z., Wang, ...

    2025 arXiv

  24. [32]

    Google - Fast or Slow? Predict AI Model Runtime

    Phothilimthana, M., Abu-El-Haija, S., Perozzi, B., Reade, W., and Chow, A. Google - Fast or Slow? Predict AI Model Runtime . https://kaggle.com/competitions/predict-ai-model-runtime, 2023. Kaggle

    2023

  25. [33]

    SQuAD : 100,000+ questions for machine comprehension of text, 2016

    Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. SQuAD : 100,000+ questions for machine comprehension of text, 2016. URL https://arxiv.org/abs/1606.05250

    2016 arXiv

  26. [34]

    Do I mage N et classifiers generalize to I mage N et? In Chaudhuri, K

    Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do I mage N et classifiers generalize to I mage N et? In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research...

    2019

  27. [35]

    and Bozsolik, T

    Risdal, M. and Bozsolik, T. M eta K aggle, 2022. URL https://www.kaggle.com/datasets/kaggle/meta-kaggle

    2022

  28. [36]

    LANL Earthquake Prediction

    RL, B., Pyrak-Nolte, L., Reade, W., and Howard, A. LANL Earthquake Prediction . https://kaggle.com/competitions/LANL-Earthquake-Prediction, 2019. Kaggle

    2019

  29. [37]

    A meta-analysis of overfitting in machine learning

    Roelofs, R., Fridovich-Keil, S., Miller, J., Shankar, V., Hardt, M., Recht, B., and Schmidt, L. A meta-analysis of overfitting in machine learning . In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2019 a . Curran...

    2019

  30. [38]

    A meta-analysis of overfitting in machine learning

    Roelofs, R., Shankar, V., Recht, B., Fridovich-Keil, S., Hardt, M., Miller, J., and Schmidt, L. A meta-analysis of overfitting in machine learning. In Wallach, H., Larochelle, H., Beygelzimer, A., d Alch\' e -Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Informa...

    2019

  31. [39]

    L., and Agirre, E

    Sainz, O., Campos, J., Garc \'i a-Ferrero, I., Etxaniz, J., de Lacalle, O. L., and Agirre, E. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computation...

    2023 doi

  32. [40]

    SEAL leaderboards

    Scale AI . SEAL leaderboards. https://https://scale.com/leaderboard, 2024

    2024

  33. [41]

    D., Reade, W., Wang, S., Croft, S., and Chen, Y

    Siemion, A., Alonso, D. D., Reade, W., Wang, S., Croft, S., and Chen, Y. SETI Breakthrough Listen - E.T. Signal Search . https://kaggle.com/competitions/seti-breakthrough-listen, 2021. Kaggle

    2021

  34. [42]

    Turing, A. M. Computing Machinery and Intelligence . Mind, 59 0 (236): 0 433--460, 1950. ISSN 00264423. URL http://www.jstor.org/stable/2251299

    1950

  35. [43]

    N., Bischl, B., and Torgo, L

    Vanschoren, J., van Rijn, J. N., Bischl, B., and Torgo, L. OpenML: networked science in machine learning . CoRR, abs/1407.7722, 2014. URL http://arxiv.org/abs/1407.7722

    2014 arXiv

  36. [44]

    The Nature of Statistical Learning Theory

    Vapnik, V. The Nature of Statistical Learning Theory . Springer: New York, 1999

    1999

  37. [45]

    Wayment-Steele, H., Kladwang, W., Watkins, A., Kim, D. S., Tunguz, B., Reade, W., Demkin, M., Romano, J., Wellington-Oguri, R., Nicol, J., Gao, J., Onodera, K., Fujikawa, K., Mao, H., Vandewiele, G., Tinti, M., Steenwinckel, B., Ito, T., Noumi, T., and Das, R. Deep learning mo...

    2022 doi

  38. [46]

    S., Naidu, S

    White, C., Dooley, S., Roberts, M., Pal, A., Feuer, B., Jain, S., Shwartz-Ziv, R., Jain, N., Saifullah, K., Dey, S., Shubh-Agrawal, Sandha, S. S., Naidu, S. V., Hegde, C., LeCun, Y., Goldstein, T., Neiswanger, W., and Goldblum, M. LiveBench: A Challenging, Contamination-Free L...

    2025

  39. [47]

    AI mathematical olympiad - progress prize 1

    XTX Investments . AI mathematical olympiad - progress prize 1. https://kaggle.com/competitions/ai-mathematical-olympiad-prize, 2024. Kaggle

    2024

  40. [48]

    TalkingData AdTracking Fraud Detection Challenge

    Yin, A., Kleinman, J., Yana, T., Reade, W., and Elliott, J. TalkingData AdTracking Fraud Detection Challenge . https://kaggle.com/competitions/talkingdata-adtracking-fraud-detection, 2018. Kaggle

    2018

  41. [49]

    HellaSwag: Can a Machine Really Finish Your Sentence? , 2019

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can a Machine Really Finish Your Sentence? , 2019. URL https://arxiv.org/abs/1905.07830

    2019 arXiv

  42. [50]

    P., Zhang, H., Gonzalez, J

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. J udging LLM -as-a- J udge with MT -bench and C hatbot A rena, 2023. URL https://arxiv.org/abs/2306.05685

    2023 arXiv

  43. [51]

    Zhu, B., Frick, E., Wu, T., Zhu, H., and Jiao, J. Nectar . https://huggingface.co/datasets/berkeley-nest/Nectar, 2024

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.