Pith. sign in

REVIEW 4 major objections 5 minor 7 cited by

AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AntiLeakBench prevents data contamination by building test questions from real-world knowledge dated after each LLM's cutoff, and the paper reports that model scores drop after that cutoff.

desk verdict The automated pipeline is a real step forward for dynamic benchmarks, but the 'strictly contamination-free' guarantee doesn't follow from the construction and should be either verified or dropped. read the letter →

arxiv 2412.13670 v2 pith:K23ULI54 submitted 2024-12-18 cs.CL cs.LG

Xiaobao Wu , Liangming Pan , Yuxi Xie , Ruiwen Zhou , Shuai Zhao , Yubo Ma , Mingzhe Du , Rui Mao
show 2 more authors
Anh Tuan Luu William Yang Wang
This is my paper · ORCID
classification cs.CLcs.LG
keywords datacontaminationLLMevaluationbenchmarkleakageknowledgecutoffWikidataWikipediaquestionansweringautomatedconstruction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes to end data contamination in LLM benchmarks by building test questions from facts that could not have been in a model's training data. Instead of collecting newly released text and hoping it is unseen, it tracks Wikidata fact statements whose object changes after a model's knowledge cutoff—the date its training data ends—and uses the corresponding Wikipedia revision as supporting context. The result, AntiLeakBench, is constructed and updated automatically without human annotation, and the paper reports that model performance drops on post-cutoff samples, which it reads as evidence that pre-cutoff benchmark scores are inflated by contamination. If the guarantee holds, contamination-free evaluation can be regenerated cheaply for every newly released model.

What carries the argument

The load-bearing object is the 'updated knowledge' triple: a Wikidata fact whose object changes after the cutoff time, extracted by grouping claims by subject and relation and sorting them by start time. Each triple is paired with a supporting document: the Wikipedia page revision made after the new object's start time, checked to contain the subject and object or their aliases. The workflow turns these triples into single-hop questions and multi-hop chains (questions that require linking one fact to a related fact), optionally adding irrelevant distractor documents, and emits them in generation or multiple-choice format; the multiple-choice options deliberately include the outdated pre-cutoff answer, giving a direct diagnostic for whether a model relies on stale memorized knowledge.

What would settle it

Take a post-cutoff AntiLeakBench sample and look up whether its supporting Wikipedia revision appears in a public pretraining corpus snapshot dated after the model's stated cutoff; finding it there for any evaluated model would falsify the strict guarantee. A sharper test is to expose an open model to that revision during fine-tuning and then see whether it answers the question without the provided context; a large gain over an unexposed model would show the benchmark's contamination-freedom depends on an unverified training-data boundary.

Watch

Extended reading notes

Core claim

The paper's central claim is that data contamination is avoidable by anchoring evaluation to dated real-world knowledge. For each evaluated LLM, the method finds a Wikidata claim (subject, relation, object) whose object changed after the model's cutoff time, retrieves the Wikipedia revision that records the new object, and asks a question whose answer is that object; because both the new fact and the supporting article postdate the cutoff, the sample cannot be in the model's training set. The paper further claims that pre-cutoff samples are contaminated: most tested models decline after their cutoff, and in multiple-choice questions they increasingly select the outdated pre-cutoff answer instead of the correct new one. AntiLeakBench is presented as a strictly contamination-free, fully automated, multilingual testbed that exposes contamination existing before cutoff times.

Load-bearing premise

The 'strictly contamination-free' guarantee rests on believing that no model was trained on any web text written after its advertised cutoff date, and those cutoff dates are partly estimated and never checked against actual training corpora.

Editorial extensions

If this is right

  • Scores obtained on pre-cutoff samples should no longer be treated as clean estimates of ability; leaderboard comparisons should restrict themselves to post-cutoff samples.
  • For each new LLM, the benchmark can be regenerated from the latest Wikidata dump without human labor, so contamination-free evaluation can keep pace with model releases.
  • The outdated-option design gives a measurable contamination signature: a model that favors the pre-cutoff answer over the correct post-cutoff answer shows the expected effect of training-set memorization.
  • Post-cutoff evaluation is substantially harder for open models, while the two proprietary models tested remain high-scoring even as distractor documents are added.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same revision-history machinery could be pointed at any temporal knowledge source (sports rosters, corporate leadership, political appointments) to produce a family of contamination-free benchmarks rather than a single static one.
  • The paper's observation that some models drop before their stated cutoff suggests contamination risk is graded by how close a fact sits to the cutoff; a natural extension is to sample knowledge by time distance and plot a per-model contamination curve.
  • Because construction is automated and per-model, the method could generate a personalized benchmark for each new model at its own cutoff, something human-built static benchmarks cannot do at the same cadence.
  • The benchmark covers only factual relational knowledge tracked by Wikidata, so contamination-free evaluation of procedural skills, code, and open-ended reasoning remains an open problem rather than something this method settles.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. AntiLeakBench proposes an automated pipeline that builds QA benchmarks from Wikidata claim changes that occur after a model's knowledge cutoff, using matching Wikipedia revisions as supporting documents. The paper evaluates 12 LLMs on single-hop and multi-hop questions in generation and multi-choice formats, reports a performance drop on post-cutoff samples, and concludes that pre-cutoff evaluation is contaminated while AntiLeakBench provides strictly contamination-free evaluation. The framework is fully automated, multilingual, and human-verified on 200 samples.

Significance. If the 'strictly contamination-free' guarantee were established, this would be a valuable contribution: the pipeline removes the human labor bottleneck of dynamic benchmarks, supports multilingual construction, and the experimental design is internally consistent. The human verification of answer and context accuracy (Tables 3 and 9) and the release of code and data are concrete strengths. However, the central guarantee is not established by the construction, and the contamination finding is weaker than the abstract claims. The resource is still useful as a dynamically updated real-world QA benchmark, but the paper must be substantially revised before the headline claims can be accepted.

major comments (4)
  1. [§3.2–3.3, Fig. 2] The inference from a Wikidata start time after the cutoff to absence from pretraining is invalid. A Wikidata qualifier records when a relation became effective, not when the fact first appeared in public text. Transfers, appointments, and election results are routinely announced or discussed before their official start date, and such pre-cutoff text can be in pretraining data. The paper therefore does not establish the central claim in the abstract that the constructed samples contain 'explicitly new knowledge absent from LLMs' training sets'. A concrete test would be to check pre-cutoff snapshots (e.g., Common Crawl or earlier Wikipedia dumps) for the object-relation phrasing in samples where the cutoff is known rather than estimated.
  2. [§3.3] The sentence 'the supporting document is revised after LLMs' cutoff time, so it is also nonexistent in their training sets' is a non sequitur. A revision after the cutoff may add unrelated content, copy text that appeared in earlier revisions of the same page, or repeat text that appeared on other crawled pages before the cutoff. The paper needs to verify that the specific sentence or fact expressing the updated knowledge is absent from earlier revisions and from other pre-cutoff web sources, or it must withdraw the 'strictly contamination-free' wording.
  3. [Table 8, §4.1] The hard-boundary assumption for knowledge cutoff times is load-bearing for both the guarantee and the contamination analysis. Several cutoffs are marked as estimated (*) — Mistral-7B, Qwen-2-7B, and Gemma-2-9B — and proprietary cutoffs are not independently verified. No check rules out post-cutoff web data in any model's training set. A sensitivity analysis using conservative earliest plausible cutoffs would show how much of the pre/post performance gap survives; without this, the strict guarantee is not supported.
  4. [§4.2, Fig. 4] The paper interprets the post-cutoff performance drop as evidence of data contamination, but the drop is also consistent with recency or difficulty effects: pre-cutoff facts are more familiar and more widely documented, while post-cutoff facts may be less represented even in uncontaminated training. The authors already acknowledge that some models drop before the cutoff, which weakens the contamination interpretation. To support the contamination claim, the paper should compare against a control set of post-cutoff facts that are verified absent from pretraining data, or explicitly state that the result measures perceived recency rather than contamination.
minor comments (5)
  1. [§4.2] The text refers to 'Mistral-Nemo-7B', but Table 8 and the model list name the model Mistral-Nemo-12B; please correct this.
  2. [Table 5] For Qwen-2-7B in the Single-Hop Gold column, the Acc is 89.1 while the F1 is 39.7; this is surprising for a multi-choice format and should be clarified or corrected, including how F1 is computed for multi-choice answers.
  3. [Appendix D] The example question 'What sports team is Duncan Cowan Ferguson a coach of?' is fine, but the template description in Appendix A, 'What sports team is a member of?', appears to be missing the subject slot and should be corrected.
  4. [Table 2] The context field has an apparent typo: 'Lionel Andrés Messi (; born 24 June 1987)' should not have the leading semicolon.
  5. [Limitations] The Limitations section discusses task diversity and data-source accuracy, but it does not acknowledge the key limitation that post-cutoff start times and revision dates do not by themselves guarantee absence from training sets; this should be added or replaced.

Circularity Check

1 steps flagged · score 1.0 of 10

No circular derivation; the 'strictly contamination-free' label rests on an unverified timestamp-to-pretraining equivalence, a validity threat rather than a circular reduction.

  1. other [Section 3.2, Identifying Updated Knowledge]
    "We emphasize that the LLM is unaware of this knowledge because it occurs after its cutoff time."

    The sample-selection step defines 'updated knowledge' as any Wikidata claim whose start-time qualifier follows the model's cutoff, then converts that temporal ordering into the stronger claim 'absent from LLMs' training sets' (and in Sec. 3.3, 'so it is also nonexistent in their training sets'). Absence from pretraining is not established by the Wikidata start time: a fact can be discussed before its official start time, and the same sentence can appear in earlier revisions or other crawled pages. Table 8 also marks several cutoffs as estimated. The 'strictly contamination-free' property is thus assumed by construction, so the pre-vs-post performance comparison is interpreted as contamination without an independent check.

full rationale

The paper does not fit any parameter and does not lean on a self-citation chain to derive its headline result. The benchmark construction, the QA samples, and the LLM evaluation are all implemented in the paper, and the empirical performance drops after the cutoff are genuine observations against external models. The only load-bearing reduction is definitional: post-cutoff (by Wikidata start time) is equated with 'not in the training set,' which makes the 'strictly contamination-free' guarantee an assumption rather than a demonstrated property. That is a threat to the interpretation of the experiment, but it is not a case where a prediction equals its input by construction, so the circularity score is low.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the model cutoff boundary and on Wikidata/Wikipedia fidelity. No free parameters are fitted; the main manually chosen inputs are cutoff dates and relation subsets, which act as assumptions rather than fitted values.

free parameters (1)
  • Per-model knowledge cutoff time = Table 8; e.g., Llama-2: 2022-09, Llama-3.1-8B: 2023-12, several marked as estimates
    Samples are labeled as pre-cutoff or post-cutoff using these dates. They are not measured, and some are estimated from release notes, so the central contamination-free claim depends on their accuracy.
assumptions (4)
  • domain assumption Wikidata and Wikipedia provide timely and accurate records of real-world changes.
    Sections 3.1-3.3 treat these sources as ground truth for identifying updates and building contexts; errors or delays would make samples wrong or miss updates.
  • domain assumption A model's stated knowledge cutoff time is a hard boundary for its training data.
    Section 4.1 and Table 8; several cutoff times are estimated, and no check confirms that post-cutoff web data is absent from pretraining.
  • domain assumption A change in Wikidata object value after cutoff implies the model is unaware of the new object.
    Section 3.2; this is the basis for the 'strictly contamination-free' claim, and it inherits the cutoff-boundary assumption.
  • domain assumption A Wikipedia revision whose lead contains subject and object is a sufficient supporting document.
    Section 3.3; the pipeline does not verify semantic content beyond string presence, so some documents may be weakly relevant.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge." pith.science (2026). https://pith.science/paper/K23ULI54

@misc{pith2026241213670,
  author       = {Pith},
  title        = {Pith review of: AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K23ULI54}},
  note         = {Machine review of arXiv:2412.13670}
}
read the original abstract

Data contamination hinders fair LLM evaluation by introducing test data into newer models' training sets. Existing studies solve this challenge by updating benchmarks with newly collected data. However, they fail to guarantee contamination-free evaluation as the newly collected data may contain pre-existing knowledge, and their benchmark updates rely on intensive human labor. To address these issues, we in this paper propose AntiLeak-Bench, an automated anti-leakage benchmarking framework. Instead of simply using newly collected data, we construct samples with explicitly new knowledge absent from LLMs' training sets, which thus ensures strictly contamination-free evaluation. We further design a fully automated workflow to build and update our benchmark without human labor. This significantly reduces the cost of benchmark maintenance to accommodate emerging LLMs. Through extensive experiments, we highlight that data contamination likely exists before LLMs' cutoff time and demonstrate AntiLeak-Bench effectively overcomes this challenge.

Figures

Figures reproduced from arXiv: 2412.13670 by the authors.

Figure 1
Figure 1. Illustration of AntiLeakBench. It constructs [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the automated benchmark building workflow without human labor. After data preparation, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Illustration of constructing multi-hop samples [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: EM and F1 performance at each time interval. Marker [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Correct and outdated option proportions at each time interval. Marker [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SCOPE: Compress Mathematical Reasoning Steps for Efficient Automated Process Annotation

    cs.AI 2025-05 conditional novelty 7.0 of 10

    SCOPE compresses sampled math reasoning traces into a prefix tree of AST-normalized code steps, yielding PRM training labels at O(N) cost and improving Best-of-N and ProcessBench scores over automated baselines.

  2. Unsupervised Hallucination Detection by Inspecting Reasoning Processes

    cs.CL 2025-09 conditional novelty 6.0 of 10

    IRIS detects LLM hallucinations by training a lightweight probe on hidden states elicited during the model's own step-by-step verification, using the model's verbalized confidence as soft pseudolabels.

  3. MRAG: A Modular Retrieval Framework for Time-Sensitive Question Answering

    cs.CL 2024-12 conditional novelty 6.0 of 10

    A modular, training-free retrieval framework that decomposes time-sensitive questions into semantic content and temporal constraints, then ranks evidence by combined semantic and symbolic temporal scores, outperforms ...

  4. RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios

    cs.CL 2024-12 conditional novelty 6.0 of 10

    RuleArena evaluates LLMs on realistic rule-guided reasoning and finds that even o1-preview solves only about half of the easiest problems and near zero of the hardest.

  5. Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddings

    cs.IR 2025-06 conditional novelty 5.0 of 10

    Semantic clustering of patch embeddings at the final model layer, plus fine-tuning, preserves 94.6 to 98.2% of retrieval performance while reducing stored embedding memory to 1.8 to 11.8% of original.

  6. Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks

    cs.CL 2025-04 conditional novelty 4.0 of 10

    A survey that frames the core problem of LLM evaluation as 'evaluation generalization': finite test sets cannot scale with unbounded model capabilities, and proposes two transitions in evaluation design.

  7. Aspect-Based Summarization with Self-Aspect Retrieval Enhanced Generation

    cs.CL 2025-04 conditional novelty 4.0 of 10

    SARESG prunes documents to aspect-relevant sentences via embedding similarity before LLM summarization, reporting gains over selective-context baselines on three datasets.

Reference graph

Works this paper leans on

68 extracted references · 14 canonical work pages · cited by 7 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024. https://arxiv.org/abs/2404.14219 Phi-3 technical report: A highly capable language model locally on your phone . arXiv preprint arXiv:2404.14219

    arXiv 2024

  4. [4]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . arXiv preprint arXiv:2303.08774

    arXiv 2023

  5. [5]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. https://doi.org/10.18653/v1/2024.acl-long.172 L ong B ench: A bilingual, multitask benchmark for long context understanding . In Proceedings of the 62nd Annual Meeting of the Association for ...

    doi:10.18653/v1/2024.acl-long.172 2024

  6. [6]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. https://lmsys.org/blog/2023-03-30-vicuna/ Vicuna: An open-source chatbot impressing gpt-4 with 90\

    2023

  7. [7]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. https://arxiv.org/pdf/1803.05457 Think you have solved question answering? try arc, the ai2 reasoning challenge . arXiv preprint arXiv:1803.05457

    arXiv 2018

  8. [8]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . arXiv preprint arXiv:2110.14168

    arXiv 2021

Show all 68 references
  1. [9]

    Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. 2024. https://arxiv.org/pdf/2311.09783 Investigating data contamination in modern benchmarks for large language models . In Proceedings of the 2024 Conference of the North American Chapter of the Associat...

    2024 arXiv

  2. [10]

    Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. 2024. https://doi.org/10.18653/v1/2024.findings-acl.716 Generalization or memorization: Data contamination and trustworthy evaluation for large language models . In Findings of the Association for Co...

    2024 doi

  3. [11]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . arXiv preprint arXiv:2407.21783

    2024 arXiv

  4. [12]

    Shahriar Golchin and Mihai Surdeanu. 2023 a . https://arxiv.org/abs/2311.06233 Data contamination quiz: A tool to detect and estimate contamination in large language models . arXiv preprint arXiv:2311.06233

    2023 arXiv

  5. [13]

    Shahriar Golchin and Mihai Surdeanu. 2023 b . https://arxiv.org/pdf/2308.08493 Time travel in llms: Tracing data contamination in large language models . arXiv preprint arXiv:2308.08493

    2023 arXiv

  6. [14]

    Zhouhong Gu, Xiaoxuan Zhu, Haoning Ye, Lin Zhang, Jianchen Wang, Yixin Zhu, Sihang Jiang, Zhuozhi Xiong, Zihan Li, Weijie Wu, et al. 2024. https://arxiv.org/abs/2306.05783 Xiezhi: An ever-updating benchmark for holistic domain knowledge evaluation . In Proceedings of the AAAI ...

    2024 arXiv

  7. [15]

    Kilem Li Gwet. 2008. https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=087be7b3d737ee817b64246e7c9a4fb6cd57dc24 Computing inter-rater reliability and its variance in the presence of high agreement . British Journal of Mathematical and Statistical Psychology, 61(1):29--48

    2008

  8. [16]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://arxiv.org/pdf/2009.03300 Measuring massive multitask language understanding . In International Conference on Learning Representations

    2021 arXiv

  9. [17]

    Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. 2023. https://arxiv.org/abs/2305.10160 Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks . In Proceedings of the 2023 Conference on Empirical Meth...

    2023 arXiv

  10. [18]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. https://arxiv.org/pdf/2403.07974 Livecodebench: Holistic and contamination free evaluation of large language models for code . arXiv prepr...

    2024 arXiv

  11. [19]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. https://arxiv.org/abs/2310.06825 Mistral 7b . arXiv preprint arXiv:2310.06825

    2023 arXiv

  12. [20]

    Minhao Jiang, Ken Ziyu Liu, Ming Zhong, Rylan Schaeffer, Siru Ouyang, Jiawei Han, and Sanmi Koyejo. 2024. https://arxiv.org/abs/2401.06059 Investigating data contamination for pre-training language models . arXiv preprint arXiv:2401.06059

    2024 arXiv

  13. [21]

    Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Velocity Yu, Dragomir Radev, Noah A Smith, Yejin Choi, and Kentaro Inui. 2023. https://arxiv.org/pdf/2207.13332 Realtime qa: what's the answer right now? In Proceedings of the 37th Internationa...

    2023 arXiv

  14. [22]

    Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. 2021. https://arxiv.org/pdf/2104.14337 Dynabench: Rethinking benchmarking in nlp . In Proceedings of the 2021 Conference ...

    2021 arXiv

  15. [23]

    Tom \'a s Ko c isk \`y , Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, G \'a bor Melis, and Edward Grefenstette. 2018. https://direct.mit.edu/tacl/article-pdf/doi/10.1162/tacl_a_00023/1567652/tacl_a_00023.pdf The narrativeqa reading comprehension challenge ....

    2018 doi

  16. [24]

    Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang

    Dacheng Li*, Rulin Shao*, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. 2023. https://lmsys.org/blog/2023-06-29-longchat How long can open-source llms truly promise on context length?

    2023

  17. [25]

    Yucheng Li. 2023. https://arxiv.org/abs/2310.17589 An open source data contamination report for llama series models . arXiv preprint arXiv:2310.17589

    2023 arXiv

  18. [26]

    Adam Liska, Tomas Kocisky, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, D’Autume Cyprien De Masson, Tim Scholtes, Manzil Zaheer, Susannah Young, et al. 2022. https://arxiv.org/pdf/2205.11388 Streamingqa: A benchmark for adaptation to new knowledge over time i...

    2022 arXiv

  19. [27]

    Inbal Magar and Roy Schwartz. 2022. https://arxiv.org/abs/2203.08242 Data contamination: From memorization to exploitation . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 157--165

    2022 arXiv

  20. [28]

    Seyed Mahed Mousavi, Simone Alghisi, and Giuseppe Riccardi. 2024. https://arxiv.org/pdf/2404.08700 Dyknow:dynamically verifying time-sensitive factual knowledge in llms . arXiv preprint arXiv:2404.08700

    2024 arXiv

  21. [29]

    Shiwen Ni, Xiangtao Kong, Chengming Li, Xiping Hu, Ruifeng Xu, Jia Zhu, and Min Yang. 2024. https://arxiv.org/abs/2409.01790 Training on the benchmark is not all you need . arXiv preprint arXiv:2409.01790

    2024 arXiv

  22. [30]

    Yonatan Oren, Nicole Meister, Niladri S Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. 2023. https://arxiv.org/abs/2310.17623 Proving test set contamination in black-box language models . In The Twelfth International Conference on Learning Representations

    2023 arXiv

  23. [31]

    Fengjun Pan, Xiaobao Wu, Zongrui Li, and Anh Tuan Luu. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.794 Are LLM s good zero-shot fallacy classifiers? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14338--14364, Miami, Flor...

    2024 doi

  24. [32]

    Liangming Pan, Xiaobao Wu, Xinyuan Lu, Anh Tuan Luu, William Yang Wang, Min-Yen Kan, and Preslav Nakov. 2023. https://aclanthology.org/2023.acl-long.386/ Fact-checking complex claims with program-guided reasoning . In Proceedings of the 61st Annual Meeting of the Association f...

    2023

  25. [33]

    Christopher Potts, Zhengxuan Wu, Atticus Geiger, and Douwe Kiela. 2021. https://aclanthology.org/2021.acl-long.186.pdf Dynasent: A dynamic benchmark for sentiment analysis . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th...

    2021

  26. [34]

    Kun Qian, Shunji Wan, Claudia Tang, Youzhi Wang, Xuanming Zhang, Maximillian Chen, and Zhou Yu. 2024. https://aclanthology.org/2024.findings-emnlp.946/ Varbench: Robust language model benchmarking through dynamic variable perturbation . In Findings of the Association for Compu...

    2024

  27. [35]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. https://insightcivic.s3.us-east-1.amazonaws.com/language-models.pdf Language models are unsupervised multitask learners . OpenAI blog, 1(8):9

    2019

  28. [36]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. https://doi.org/10.18653/v1/D16-1264 SQ u AD : 100,000+ questions for machine comprehension of text . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-...

    2016 doi

  29. [37]

    Manley Roberts, Himanshu Thakur, Christine Herlihy, Colin White, and Samuel Dooley. 2023. https://openreview.net/pdf?id=m2NVG4Htxs To the cutoff... and beyond? a longitudinal perspective on llm data contamination . In The Twelfth International Conference on Learning Representations

    2023

  30. [38]

    Oscar Sainz, Jon Ander Campos, Iker Garc \' a-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. https://aclanthology.org/2023.findings-emnlp.722 Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark . In The 2023 Confe...

    2023

  31. [39]

    Rylan Schaeffer. 2023. https://arxiv.org/abs/2309.08632 Pretraining on the test set is all you need . arXiv preprint arXiv:2309.08632

    2023 arXiv

  32. [40]

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, et al. 2022. https://arxiv.org/abs/2206.04615 Beyond the imitation game: Quantifying and extrapolating the capabil...

    2022 arXiv

  33. [41]

    Saurabh Srivastava, Anto PV, Shashank Menon, Ajay Sukumar, Alan Philipose, Stevin Prince, Sooraj Thomas, et al. 2024. https://arxiv.org/abs/2402.19450 Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap . arXiv preprint arXiv:2402.19450

    2024 arXiv

  34. [42]

    Gemma Team. 2024. https://doi.org/10.34740/KAGGLE/M/3301 Gemma . blog

    2024 doi

  35. [43]

    Tristan Thrush, Kushal Tirumala, Anmol Gupta, Max Bartolo, Pedro Rodriguez, Tariq Kane, William Gaviria Rojas, Peter Mattson, Adina Williams, and Douwe Kiela. 2022. https://aclanthology.org/2022.acl-demo.17.pdf Dynatask: A framework for creating dynamic ai benchmark tasks . In...

    2022

  36. [44]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. https://arxiv.org/abs/2307.09288 Llama 2: Open foundation and fine-tuned chat models . arXiv preprint arXiv:...

    2023 arXiv

  37. [45]

    Denny Vrande c i\' c and Markus Kr\" o tzsch. 2014. https://doi.org/10.1145/2629489 Wikidata: A free collaborative knowledgebase . Commun. ACM, 57(10):78–85

    2014 doi

  38. [46]

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. 2024. https://arxiv.org/pdf/2406.19314 Livebench: A challenging, contamination-free llm benchmark . arXiv preprint arXiv:2406.19314

    2024 arXiv

  39. [47]

    Xiaobao Wu. 2025. https://arxiv.org/abs/2505.02686 Sailing ai by the stars: A survey of learning from rewards in post-training and test-time scaling of large language models . arXiv preprint arXiv:2505.02686

    2025 arXiv

  40. [48]

    Xiaobao Wu, Xinshuai Dong, Thong Nguyen, Chaoqun Liu, Liang-Ming Pan, and Anh Tuan Luu. 2023. https://arxiv.org/pdf/2304.03544 InfoCTM : A mutual information maximization perspective of cross-lingual topic modeling . In AAAI Conference on Artificial Intelligence (AAAI), volume...

    2023 arXiv

  41. [49]

    Xiaobao Wu, Xinshuai Dong, Liangming Pan, Thong Nguyen, and Anh Tuan Luu. 2024 a . https://aclanthology.org/2024.findings-acl.183 Modeling dynamic topics in chain-free fashion by evolution-tracking contrastive learning and unassociated word exclusion . In Findings of the Assoc...

    2024

  42. [50]

    Xiaobao Wu, Chunping Li, Yan Zhu, and Yishu Miao. 2020. https://aclanthology.org/2020.emnlp-main.138.pdf Short text topic modeling with topic distribution quantization and negative sampling decoder . In Proceedings of the 2020 Conference on Empirical Methods in Natural Languag...

    2020

  43. [51]

    Xiaobao Wu, Anh Tuan Luu, and Xinshuai Dong. 2022. https://aclanthology.org/2022.emnlp-main.176.pdf Mitigating data sparsity for short text topic modeling by topic-semantic contrastive learning . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Pr...

    2022

  44. [52]

    Xiaobao Wu, Thong Nguyen, and Anh Tuan Luu. 2024 b . https://doi.org/10.1007/s10462-023-10661-7 A survey on neural topic models: Methods, applications, and challenges . Artificial Intelligence Review

    2024 doi

  45. [53]

    Xiaobao Wu, Thong Thanh Nguyen, Delvin Ce Zhang, William Yang Wang, and Anh Tuan Luu. 2024 c . https://arxiv.org/pdf/2405.17978 FASTopic : Pretrained transformer is a fast, adaptive, stable, and transferable topic model . In The Thirty-eighth Annual Conference on Neural Inform...

    2024 arXiv

  46. [54]

    Xiaobao Wu, Fengjun Pan, and Anh Tuan Luu. 2024 d . https://aclanthology.org/2024.acl-demos.4 Towards the T op M ost: A topic modeling system toolkit . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations),...

    2024

  47. [55]

    Xiaobao Wu, Fengjun Pan, Thong Nguyen, Yichao Feng, Chaoqun Liu, Cong-Duy Nguyen, and Anh Tuan Luu. 2024 e . https://arxiv.org/pdf/2401.14113.pdf On the affinity, rationality, and diversity of hierarchical topic modeling . In Proceedings of the AAAI Conference on Artificial In...

    2024 arXiv

  48. [56]

    Xiaobao Wu, Liangming Pan, William Yang Wang, and Anh Tuan Luu. 2024 f . https://aclanthology.org/2024.emnlp-main.843 AKEW : Assessing knowledge editing in the wild . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15118--15133,...

    2024

  49. [57]

    Zhikun Xu, Yinghui Li, Ruixue Ding, Xinyu Wang, Boli Chen, Yong Jiang, Xiaodong Deng, Jianxin Ma, Hai-Tao Zheng, Wenlian Lu, et al. 2024. https://arxiv.org/abs/2402.19248 Let llms take on the latest challenges! a chinese dynamic question answering benchmark . arXiv preprint ar...

    2024 arXiv

  50. [58]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. https://arxiv.org/abs/2407.10671 Qwen2 technical report . arXiv preprint arXiv:2407.10671

    2024 arXiv

  51. [59]

    Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E Gonzalez, and Ion Stoica. 2023. https://arxiv.org/abs/2311.04850 Rethinking benchmark and contamination for language models with rephrased samples . arXiv preprint arXiv:2311.04850

    2023 arXiv

  52. [60]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. https://arxiv.org/abs/1809.09600 Hotpotqa: A dataset for diverse, explainable multi-hop question answering . In Proceedings of the 2018 Conference on Empi...

    2018 arXiv

  53. [61]

    Jiahao Ying, Yixin Cao, Yushi Bai, Qianru Sun, Bo Wang, Wei Tang, Zhaojun Ding, Yizhe Yang, Xuanjing Huang, and Shuicheng Yan. 2024. https://arxiv.org/pdf/2402.11894 Automating dataset updates towards reliable and timely evaluation of large language models . In The Thirty-eigh...

    2024 arXiv

  54. [62]

    Jifan Yu, Xiaozhi Wang, Shangqing Tu, Shulin Cao, Daniel Zhang-Li, Xin Lv, Hao Peng, Zijun Yao, Xiaohan Zhang, Hanming Li, et al. 2023. https://arxiv.org/abs/2306.09296 Kola: Carefully benchmarking world knowledge of large language models . In The Twelfth International Confere...

    2023 arXiv

  55. [63]

    Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, et al. 2024. https://arxiv.org/pdf/2405.00332 A careful examination of large language model performance on grade school arithmetic . arXiv preprint arXiv:2...

    2024 arXiv

  56. [64]

    Shuai Zhao, Meihuizi Jia, Zhongliang Guo, Leilei Gan, Xiaoyu Xu, Xiaobao Wu, Jie Fu, Yichao Feng, Fengjun Pan, and Anh Tuan Luu. 2025. https://openreview.net/forum?id=wZLWuFHxt5 A survey of recent backdoor attacks and defenses in large language models . Transactions on Machine...

    2025

  57. [65]

    Zexuan Zhong, Zhengxuan Wu, Christopher Manning, Christopher Potts, and Danqi Chen. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.971 MQ u AKE : Assessing knowledge editing in language models via multi-hop questions . In Proceedings of the 2023 Conference on Empirical Meth...

    2023 doi

  58. [66]

    Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. 2023. https://arxiv.org/abs/2311.01964 Don't make your llm an evaluation benchmark cheater . arXiv preprint arXiv:2311.01964

    2023 arXiv

  59. [67]

    Ruiwen Zhou, Wenyue Hua, Liangming Pan, Sitao Cheng, Xiaobao Wu, En Yu, and William Yang Wang. 2024. https://arxiv.org/abs/2412.08972 Rulearena: A benchmark for rule-guided reasoning with llms in real-world scenarios . arXiv preprint arXiv:2412.08972

    2024 arXiv

  60. [68]

    Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. 2023. https://arxiv.org/abs/2309.17167 Dyval: Graph-informed dynamic evaluation of large language models . arXiv e-prints, pages arXiv--2309

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.