{"id":"ffaeda28-9542-496a-860f-b715449ea385","arxiv_id":"2412.13670","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"AntiLeakBench automatically constructs QA benchmarks from knowledge updated after each model's cutoff, and its experiments suggest that pre-cutoff evaluation overstates LLM ability.","lead":"This paper presents AntiLeakBench, a benchmark built automatically from Wikidata and Wikipedia updates that occur after a model's knowledge cutoff, so the test questions ask about facts the model could not have memorized. It also reports that models score lower on post-cutoff questions than on pre-cutoff questions, which the authors read as evidence of data contamination in standard benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'strictly contamination-free' guarantee conflates Wikidata start times with absence from pretraining; post-cutoff labels do not establish that the fact or the supporting revision's content was unseen before cutoff.","rationale":"The reader's weakest assumption centers on unverified and estimated model cutoff dates, allowing post-cutoff training data to leak into pretraining. My concern is broader but closely related: even if every cutoff date were exactly correct and honored, the construction still would not establish that the knowledge is absent, because the paper equates 'Wikidata claim start time after cutoff' with 'fact not in pretraining.' The fact may have been publicly announced, speculated about, or written into earlier Wikipedia revisions before the official start time; the selected post-start-time revision does not guarantee that its content is new. This is a logical gap in the central argument, not merely an external uncertainty about model training. It is load-bearing because the paper's headline contribution is the guarantee of strictly contamination-free evaluation; if that guarantee fails, the benchmark's advantage over LiveBench-style recency-based benchmarks is weakened, although the automated pipeline and real-world QA value remain. I therefore agree with the reader's conditional verdict: the method is promising and publishable, but the strong guarantee should be softened or verified. The proposed audit of earliest Wikipedia mentions is a direct, automatable check that would settle whether the Section 3.3 reasoning holds. Because the reader already recommended conditional acceptance, my read does not change the verdict.","tokens_in":18318,"tokens_out":6175,"duration_ms":57395,"concrete_test":"Use the MediaWiki API to audit the revision history of each post-cutoff sample's supporting document: identify the timestamp of the earliest revision whose rendered text contains the target object (or one of its aliases), and compare that timestamp to the model's stated cutoff from Table 8. If any appreciable fraction (e.g., more than 5%) of post-cutoff samples have an earliest-mention timestamp at or before the cutoff, then the Section 3.3 inference that the supporting document and updated knowledge are absent from pretraining is falsified, and the 'strictly contamination-free' claim should be withdrawn or explicitly qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that AntiLeakBench constructs samples whose knowledge is absent from LLMs' training sets, hence 'strictly contamination-free' evaluation. The load-bearing assumption is the inference in Sections 3.2-3.3 that a Wikidata claim with start time after a model's cutoff, supported by a Wikipedia revision made after that start time, was unseen by the LLM. This inference is not valid. A Wikidata start-time qualifier records when a relation became effective, not when the fact first appeared in public text; transfers, appointments, and election results are routinely announced, predicted, or discussed before their official start date, and such pre-cutoff text can be in pretraining data. Likewise, the selected Wikipedia revision being dated after the cutoff does not imply its content is new: the same sentence about the object may have appeared in earlier revisions of that page, or in other crawled pages, before the cutoff. The word 'so' in Section 3.3 ('the supporting document is revised after LLMs' cutoff time, so it is also nonexistent in their training sets') is a non sequitur. Table 8 compounds this by marking several cutoff times as estimated (*), and proprietary cutoffs are not independently verified. Therefore the condition 'explicitly new knowledge absent from LLMs' training sets' is not guaranteed by the construction; it is an empirical claim that the paper does not test. The framework remains useful as a dynamic real-world QA benchmark, but the central headline guarantee is unproven.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"AntiLeakBench proposes an automated pipeline that builds QA benchmarks from Wikidata claim changes that occur after a model's knowledge cutoff, using matching Wikipedia revisions as supporting documents. The paper evaluates 12 LLMs on single-hop and multi-hop questions in generation and multi-choice formats, reports a performance drop on post-cutoff samples, and concludes that pre-cutoff evaluation is contaminated while AntiLeakBench provides strictly contamination-free evaluation. The framework is fully automated, multilingual, and human-verified on 200 samples.","tokens_in":18583,"tokens_out":2922,"duration_ms":29094,"significance":"If the 'strictly contamination-free' guarantee were established, this would be a valuable contribution: the pipeline removes the human labor bottleneck of dynamic benchmarks, supports multilingual construction, and the experimental design is internally consistent. The human verification of answer and context accuracy (Tables 3 and 9) and the release of code and data are concrete strengths. However, the central guarantee is not established by the construction, and the contamination finding is weaker than the abstract claims. The resource is still useful as a dynamically updated real-world QA benchmark, but the paper must be substantially revised before the headline claims can be accepted.","major_comments":[{"comment":"The inference from a Wikidata start time after the cutoff to absence from pretraining is invalid. A Wikidata qualifier records when a relation became effective, not when the fact first appeared in public text. Transfers, appointments, and election results are routinely announced or discussed before their official start date, and such pre-cutoff text can be in pretraining data. The paper therefore does not establish the central claim in the abstract that the constructed samples contain 'explicitly new knowledge absent from LLMs' training sets'. A concrete test would be to check pre-cutoff snapshots (e.g., Common Crawl or earlier Wikipedia dumps) for the object-relation phrasing in samples where the cutoff is known rather than estimated.","section":"§3.2–3.3, Fig. 2"},{"comment":"The sentence 'the supporting document is revised after LLMs' cutoff time, so it is also nonexistent in their training sets' is a non sequitur. A revision after the cutoff may add unrelated content, copy text that appeared in earlier revisions of the same page, or repeat text that appeared on other crawled pages before the cutoff. The paper needs to verify that the specific sentence or fact expressing the updated knowledge is absent from earlier revisions and from other pre-cutoff web sources, or it must withdraw the 'strictly contamination-free' wording.","section":"§3.3"},{"comment":"The hard-boundary assumption for knowledge cutoff times is load-bearing for both the guarantee and the contamination analysis. Several cutoffs are marked as estimated (*) — Mistral-7B, Qwen-2-7B, and Gemma-2-9B — and proprietary cutoffs are not independently verified. No check rules out post-cutoff web data in any model's training set. A sensitivity analysis using conservative earliest plausible cutoffs would show how much of the pre/post performance gap survives; without this, the strict guarantee is not supported.","section":"Table 8, §4.1"},{"comment":"The paper interprets the post-cutoff performance drop as evidence of data contamination, but the drop is also consistent with recency or difficulty effects: pre-cutoff facts are more familiar and more widely documented, while post-cutoff facts may be less represented even in uncontaminated training. The authors already acknowledge that some models drop before the cutoff, which weakens the contamination interpretation. To support the contamination claim, the paper should compare against a control set of post-cutoff facts that are verified absent from pretraining data, or explicitly state that the result measures perceived recency rather than contamination.","section":"§4.2, Fig. 4"}],"minor_comments":[{"comment":"The text refers to 'Mistral-Nemo-7B', but Table 8 and the model list name the model Mistral-Nemo-12B; please correct this.","section":"§4.2"},{"comment":"For Qwen-2-7B in the Single-Hop Gold column, the Acc is 89.1 while the F1 is 39.7; this is surprising for a multi-choice format and should be clarified or corrected, including how F1 is computed for multi-choice answers.","section":"Table 5"},{"comment":"The example question 'What sports team is Duncan Cowan Ferguson a coach of?' is fine, but the template description in Appendix A, 'What sports team is a member of?', appears to be missing the subject slot and should be corrected.","section":"Appendix D"},{"comment":"The context field has an apparent typo: 'Lionel Andrés Messi (; born 24 June 1987)' should not have the leading semicolon.","section":"Table 2"},{"comment":"The Limitations section discusses task diversity and data-source accuracy, but it does not acknowledge the key limitation that post-cutoff start times and revision dates do not by themselves guarantee absence from training sets; this should be added or replaced.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The headline 'strictly contamination-free' guarantee is the paper's central selling point, but it rests on an invalid inference from effective dates and revision timestamps to absence from pretraining. I would require either direct verification against pre-cutoff snapshots or a substantial weakening of the claim, with the benchmark reframed as a dynamic real-world QA benchmark. The automated pipeline and the human-verified data are solid enough to be worth developing further, so this is a major-revision rather than a reject."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the core idea here is worth building on. AntiLeakBench uses Wikidata claim histories to spot when a fact changes after a model's cutoff, then grabs a Wikipedia revision from after that change as supporting context. That gives you a question-answering benchmark where the expected answer is keyed to a fact that was not in the knowledge base before cutoff. The workflow is fully automated, multilingual, and the authors human-verified 200 samples with high agreement. Compared with LiveBench or RealTimeQA, which just collect new data and hope it's unseen, this is a genuine improvement in targeting genuinely new facts.\n\nThe soft spot is the headline claim. Section 3.2 says the LLM is 'unaware' of the knowledge because it occurs after the cutoff, and Section 3.3 says the supporting document was revised after cutoff, 'so it is also nonexistent in their training sets.' That 'so' does not follow. A Wikidata start time records when a relation became effective, not when the fact first appeared in public text. Transfers, appointments, and elections are routinely discussed before the official start date, and that pre-cutoff text can easily end up in pretraining data. Same for the Wikipedia revision: the revision is after cutoff, but the sentence about the object may have appeared in earlier revisions or other crawled pages. So the guarantee 'strictly contamination-free' is unproven. Table 8 also marks several cutoff times as estimated, so the partition itself is approximate.\n\nThat said, the experimental finding is still interesting: performance drops after cutoff for most models, and models increasingly pick outdated options. That is consistent with contamination, but it is also consistent with models just knowing less about very recent events. The paper does not include controls or error bars, and it does not directly probe whether post-cutoff samples leak into training.\n\nWho is this for: anyone working on evaluation methodology or temporal generalization. The pipeline is reproducible and the code/data are available. I would cite it for the construction mechanism, not for the guarantee. For peer review, I would send it out with a major-revision request: either add direct contamination checks (e.g., membership-style probes, canary strings, or comparing pre- and post-cutoff variants of the same fact) or reframe the claim as 'designed to minimize contamination' rather than 'strictly free.' The paper deserves a serious referee; the overclaim is fixable.","headline":"The automated pipeline is a real step forward for dynamic benchmarks, but the 'strictly contamination-free' guarantee doesn't follow from the construction and should be either verified or dropped.","tokens_in":19115,"tokens_out":2352,"would_cite":true,"duration_ms":19756,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AntiLeakBench prevents data contamination by building test questions from real-world knowledge dated after each LLM's cutoff, and the paper reports that model scores drop after that cutoff.","keywords":["data contamination","LLM evaluation","benchmark leakage","knowledge cutoff","Wikidata","Wikipedia","question answering","automated benchmark construction"],"falsifier":"Take a post-cutoff AntiLeakBench sample and look up whether its supporting Wikipedia revision appears in a public pretraining corpus snapshot dated after the model's stated cutoff; finding it there for any evaluated model would falsify the strict guarantee. A sharper test is to expose an open model to that revision during fine-tuning and then see whether it answers the question without the provided context; a large gain over an unexposed model would show the benchmark's contamination-freedom depends on an unverified training-data boundary.","tokens_in":18140,"feed_emoji":"🛡️","tokens_out":9388,"duration_ms":77908,"temperature":0.7,"pith_summary":"The paper proposes to end data contamination in LLM benchmarks by building test questions from facts that could not have been in a model's training data. Instead of collecting newly released text and hoping it is unseen, it tracks Wikidata fact statements whose object changes after a model's knowledge cutoff—the date its training data ends—and uses the corresponding Wikipedia revision as supporting context. The result, AntiLeakBench, is constructed and updated automatically without human annotation, and the paper reports that model performance drops on post-cutoff samples, which it reads as evidence that pre-cutoff benchmark scores are inflated by contamination. If the guarantee holds, contamination-free evaluation can be regenerated cheaply for every newly released model.","feed_headline":"New benchmark writes test questions from facts too new for LLMs","feed_subtitle":"It uses Wikidata changes and Wikipedia history to block leaked test data; model scores drop after cutoff.","key_machinery":"The load-bearing object is the 'updated knowledge' triple: a Wikidata fact whose object changes after the cutoff time, extracted by grouping claims by subject and relation and sorting them by start time. Each triple is paired with a supporting document: the Wikipedia page revision made after the new object's start time, checked to contain the subject and object or their aliases. The workflow turns these triples into single-hop questions and multi-hop chains (questions that require linking one fact to a related fact), optionally adding irrelevant distractor documents, and emits them in generation or multiple-choice format; the multiple-choice options deliberately include the outdated pre-cutoff answer, giving a direct diagnostic for whether a model relies on stale memorized knowledge.","core_discovery":"The paper's central claim is that data contamination is avoidable by anchoring evaluation to dated real-world knowledge. For each evaluated LLM, the method finds a Wikidata claim (subject, relation, object) whose object changed after the model's cutoff time, retrieves the Wikipedia revision that records the new object, and asks a question whose answer is that object; because both the new fact and the supporting article postdate the cutoff, the sample cannot be in the model's training set. The paper further claims that pre-cutoff samples are contaminated: most tested models decline after their cutoff, and in multiple-choice questions they increasingly select the outdated pre-cutoff answer instead of the correct new one. AntiLeakBench is presented as a strictly contamination-free, fully automated, multilingual testbed that exposes contamination existing before cutoff times.","pith_inferences":["The same revision-history machinery could be pointed at any temporal knowledge source (sports rosters, corporate leadership, political appointments) to produce a family of contamination-free benchmarks rather than a single static one.","The paper's observation that some models drop before their stated cutoff suggests contamination risk is graded by how close a fact sits to the cutoff; a natural extension is to sample knowledge by time distance and plot a per-model contamination curve.","Because construction is automated and per-model, the method could generate a personalized benchmark for each new model at its own cutoff, something human-built static benchmarks cannot do at the same cadence.","The benchmark covers only factual relational knowledge tracked by Wikidata, so contamination-free evaluation of procedural skills, code, and open-ended reasoning remains an open problem rather than something this method settles."],"forward_implications":["Scores obtained on pre-cutoff samples should no longer be treated as clean estimates of ability; leaderboard comparisons should restrict themselves to post-cutoff samples.","For each new LLM, the benchmark can be regenerated from the latest Wikidata dump without human labor, so contamination-free evaluation can keep pace with model releases.","The outdated-option design gives a measurable contamination signature: a model that favors the pre-cutoff answer over the correct post-cutoff answer shows the expected effect of training-set memorization.","Post-cutoff evaluation is substantially harder for open models, while the two proprietary models tested remain high-scoring even as distractor documents are added."],"supporting_citations":[{"why":"Supplies Wikidata as the knowledge base whose claims, qualifiers, and update history are the raw material for identifying post-cutoff knowledge.","marker":"Vrandečić and Krötzsch, 2014"},{"why":"RealTimeQA is the dynamic benchmark that collects newspaper quizzes by recency; it is the main example of the weak contamination guarantee and human labor the paper argues against.","marker":"Kasai et al., 2023"},{"why":"LiveBench is the comparison benchmark collecting newly released exam and coding questions, used to show that newly collected data alone does not ensure contamination-free evaluation.","marker":"White et al., 2024"},{"why":"LiveCodeBench is the continuously crawled code benchmark whose reliance on platform questions illustrates the same limitation and motivates the updated-knowledge approach.","marker":"Jain et al., 2024"},{"why":"ADU is the automated benchmark-updating baseline using LLM paraphrasing, contrasted with AntiLeakBench's real-world grounded construction.","marker":"Ying et al., 2024"},{"why":"LongBench supplies the prompt format and reading-comprehension evaluation setting used in the experiments.","marker":"Bai et al., 2024"},{"why":"SQuAD is the source of the exact-match and token-level F1 metrics used to score generation-format answers.","marker":"Rajpurkar et al., 2016"}],"fun_headline_variants":["AntiLeakBench: test LLMs on post-cutoff facts only","Fresh-knowledge benchmark blocks data leakage in LLM eval","Automated benchmark builds questions from new real-world events","LLM test questions written from facts after training cutoff","AntiLeakBench: automatic, contamination-free LLM benchmarking"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 'strictly contamination-free' guarantee rests on believing that no model was trained on any web text written after its advertised cutoff date, and those cutoff dates are partly estimated and never checked against actual training corpora.","fun_headline_variants_meta":{"raw":{"variants":["AntiLeakBench: test LLMs on post-cutoff facts only","Fresh-knowledge benchmark blocks data leakage in LLM eval","Automated benchmark builds questions from new real-world events","LLM test questions written from facts after training cutoff","AntiLeakBench: automatic, contamination-free LLM benchmarking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000725,"raw_usage":{"total_tokens":3206,"prompt_tokens":858,"completion_tokens":2348,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":474,"completion_tokens_details":{"reasoning_tokens":2265}},"tokens_in":474,"tokens_out":2348,"duration_ms":17573,"temperature":1.0,"reasoning_tokens":2265,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:54:02.936329+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a post-cutoff AntiLeakBench sample and look up whether its supporting Wikipedia revision appears in a public pretraining corpus snapshot dated after the model's stated cutoff; finding it there for any evaluated model would falsify the strict guarantee. A sharper test is to expose an open model to that revision during fine-tuning and then see whether it answers the question without the provided context; a large gain over an unexposed model would show the benchmark's contamination-freedom depends on an unverified training-data boundary.","supporting_citations":[],"review_version":1}