{"id":"838451b0-f376-42f2-95cb-cc3f08d71b00","arxiv_id":"2608.07341","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new stratified per-question probability-gap metric shows that existing contamination mitigation strategies overstate restoration, and a step-wise decoding cap (RailCap) achieves the lowest measured gap.","lead":"Contaminated AI models can fake high scores by reproducing memorized test answers; this paper argues the standard gap metric hides this and proposes a per-question probability-gap score (SA-PPG) to expose it. It also introduces a decoding-time mitigation, RailCap, and shows in simulated contamination that prior methods overstate restoration while RailCap scores best.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Synthetic contamination strength is the load-bearing premise: 5-epoch LoRA fine-tuning on the test split makes the greedy-fallback signal RailCap exploits artificially strong, so the quantitative overestimation and ranking claims may not transfer to natural contamination.","rationale":"The reader's weakest assumption—that all conclusions rest on simulated contamination—is exactly the load-bearing premise I identify, and I agree with the CONDITIONAL verdict. My concern sharpens the reader's point by linking contamination strength to RailCap's mechanism: the 5-epoch LoRA fine-tuning of §5.1 is precisely what produces the strong greedy-trajectory collapse in Figure 1, which RailCap then exploits. Without a sweep over contamination strength or a naturally contaminated model, the quantitative claim that prior restoration is 'substantially overestimated' and the ranking claim that RailCap is best cannot be separated from the simulation's intensity. The paper deserves credit for a clean mathematical critique of G-AP: the cancellation argument in §3.2.2 and the stratification in §3.3 are sound, and the internal consistency of Tables 1-4 is good. The unresolvable issue is external validity, not internal correctness. A controlled contamination-strength experiment would settle whether the empirical claims generalize; until then, the conditional verdict stands. I do not recommend rejecting the paper because the conceptual contribution is valuable, but the quantitative claims should be read as established only for strong, LoRA-style contamination.","tokens_in":13638,"tokens_out":10334,"duration_ms":100964,"concrete_test":"Run a controlled contamination-strength sweep on Pythia under the §5.1 protocol, varying the injected test data: 1, 2, and 5 LoRA epochs on the 660 test questions, plus a pretraining-style injection at 1x, 5x, and 20x duplication in a small corpus. For each strength, recompute Figure 1's fall-back statistics and Table 3's SA-PPG for all strategies. If RailCap's lead shrinks or vanishes at low contamination strength, the headline claims should be restricted to strong-contamination regimes; if it persists at all strengths, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claims—that G-AP substantially overestimates prior restoration and that RailCap attains the lowest SA-PPG—depend entirely on the contamination simulation of §5.1: Mco is obtained by 5-epoch LoRA fine-tuning on the 660 test-split questions, in the 8-shot CoT format, with 1,840 OpenOrca examples. This creates an unusually strong contamination regime. Figure 1 then shows the contaminated model's samples collapse onto its greedy trajectory on leaked questions, and that the clean model's next token is often the contaminated model's runner-up; those are exactly the signals RailCap's step-wise cap exploits in §4. In natural contamination, test items typically appear only a few times in pretraining, with heterogeneous formatting and interleaving, so the greedy-fallback signal will be weaker and noisier. The measured magnitude of G-AP's overestimation (Table 1: LNE-blocking 0.0235 vs SA-PPG 0.2932) and RailCap's six-way win (Table 3) could therefore be artifacts of contamination strength rather than general properties. No validation on a naturally contaminated model, nor a controlled sweep of contamination strength or duplication frequency, is reported. The metric mathematics is internally sound and the experiments are self-consistent; the concern is about external validity of the empirical generalization.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses the evaluation and mitigation of benchmark contamination in language models. It argues that the prevailing metric, G-AP (Gap of Aggregate Performance), is flawed because it uses discrete per-question readouts and averages before differencing, so over- and under-suppression can cancel out. The authors propose A-PPG (Aggregate of Per-question Probability Gaps) and its stratified variant SA-PPG (Stratified Aggregate of Per-question Probability Gaps), which estimate each question's solve probability by sampling, difference it against a clean model per question, and aggregate within groups defined by the clean model's solve probability. They also propose RailCap, a decoding-time mitigation strategy that caps the probability of the next greedy-trajectory token to the runner-up whenever a sample falls back onto the greedy trajectory. The experimental section simulates contamination by further LoRA fine-tuning on test-split questions, and claims that SA-PPG reveals prior strategies' restoration to be substantially overestimated and that RailCap attains the lowest SA-PPG in all six tested settings. The mathematical definitions and the zero-set argument for A-PPG are correct, and the decomposition in Table 2 is instructive, but the empirical generalization rests on a single synthetic contamination regime and on point estimates without uncertainty quantification.","tokens_in":13905,"tokens_out":6240,"duration_ms":58602,"significance":"The paper's theoretical contribution is solid and clearly presented: the definitions of A-PPG and SA-PPG are self-consistent, the inequality G-APP ≤ A-PPG is proven, and the zero-set argument in Eq. (6) correctly identifies why aggregate gaps can vanish without per-question restoration. The decomposition into under- and over-suppression in Table 2 is a useful diagnostic, and the All-Zero example convincingly illustrates the failure of equal weighting. RailCap is a novel decoding-time strategy that is well motivated by the two observations in Figure 1. If the empirical claims held beyond the controlled simulation, the paper would materially improve how contamination mitigation is evaluated. At present, however, the quantitative conclusions are established only for a synthetic contamination regime, and the ranking claims lack uncertainty quantification; therefore the significance is conditional on additional evidence.","major_comments":[{"comment":"The empirical generalization that G-AP substantially overestimates prior restoration and that RailCap attains the lowest SA-PPG rests entirely on a synthetic contamination simulation: Mco is obtained by five-epoch LoRA fine-tuning of Mcl on 660 test-split questions in the 8-shot CoT format mixed with 1,840 OpenOrca examples. This is a strong, homogeneous contamination regime, and Figure 1 demonstrates the greedy-fallback signal for Llama-2 on GSM8K only. No naturally contaminated model, no duplication-frequency sweep, and no variation of contamination strength is reported. Since the magnitudes of the overestimation (Table 1: LNE-blocking 0.0235 vs. SA-PPG 0.2932) and the six-way ranking (Table 3) may depend on this regime, the paper should either restrict its claims to the controlled setting or add experiments varying contamination strength and repetition frequency, preferably including a naturally contaminated model. Without such evidence, the central empirical conclusions are not established beyond the synthetic setup.","section":"§5.1, Tables 1 and 3"},{"comment":"The sentence 'changing only the metric overturns the verdict entirely' overstates what Table 1 shows: the G-AP column uses a single 0/1 observation per question, while the SA-PPG column uses m=50 samples, so the readout estimator and the aggregation change simultaneously. Moreover, the caption's phrase 'Readings of the same responses' is inconsistent with the protocol in §5.1, where the G-AP readout uses one sample and SA-PPG uses fifty. The aggregation-only comparison in Table 2 (G-APP 0.0207 vs. SA-PPG 0.2932 for LNE-blocking) is the correct evidence for the rank-reversal claim, and the text should be revised to present Table 2 as the primary support for that claim.","section":"§5.2.1, Table 1"},{"comment":"The strategy-ranking claims are reported as point estimates with no uncertainty quantification. All SA-PPG values are derived from m=50 samples per question in a single run, and no bootstrap confidence intervals, multiple seeds, or significance tests are provided. Since some reported differences are small (e.g., Pythia × GSM8K: RailCap 0.1648 vs. Shortcut 0.1708), the reader cannot determine whether the six-way win is robust. The authors should report error bars or repeated-seed intervals for the metric values in Tables 1–3 before drawing strong comparative conclusions.","section":"§5.3.1, Tables 1–3"}],"minor_comments":[{"comment":"The zero-set property of SA-PPG is stated for the true solve probability rprob, but in practice this quantity is estimated from m=50 samples; the paper should note that finite-sample SA-PPG is a noisy estimator and that the exact per-question restoration guarantee holds only in the population limit.","section":"§3.3"},{"comment":"The n-gram index H is built as a dictionary from a trailing n-gram to its successor; if the same n-gram appears more than once on a greedy trajectory with different successors, the later entry overwrites the earlier one. The paper should state how such repeated windows are handled, since this affects the cap applied at those steps.","section":"§4, Eq. (14)"},{"comment":"The choices B=50 for SA-PPG and T=0.7 for sampling are not justified or varied; a sensitivity analysis for B and m, or at least a brief discussion, would strengthen the practical recommendations of the metric.","section":"§5.1"},{"comment":"In panels (b) and (d), the y-axis label 'share' is not defined; the caption should state whether the shares are computed over forking steps or over questions, and clarify the denominator for each rank category.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a CL venue, and the metric contribution is well-motivated. The main risk is that the authors state universal empirical conclusions ('G-AP substantially overestimates', 'RailCap attains the lowest SA-PPG') from a single synthetic contamination setup; the revision should either add the missing external-validity experiments or substantially qualify the claims. The lack of uncertainty quantification is a second point that should be addressed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is worth a serious read. It identifies a real flaw in the standard contamination mitigation metric, G-AP, and proposes a better one, SA-PPG, with a straightforward but effective fix. The math is solid: the cancellation argument (Eqs. 8-11) is correct, and the stratification to block trivial All-Zero strategies is a genuinely good idea. The RailCap strategy is also interesting: using online fall-back onto the greedy trajectory as a contamination signal is a clever move that avoids the fragile two-step estimate-then-suppress paradigm. The empirical demonstration is internally consistent: under SA-PPG, LNE-blocking's apparent near-perfect restoration collapses, and RailCap wins across all six settings. Credit where due: the paper ships no code or data, but the metric definition and the decomposition experiments are clear enough to be checked.\n\nThe biggest soft spot is the synthetic contamination setup. All conclusions rest on 5-epoch LoRA fine-tuning on the test split, which produces a much stronger contamination signal than natural pretraining leakage. Figure 1, which motivates RailCap, is only shown for Llama-2 on GSM8K. So the quantitative claims—'substantially overestimated' and 'lowest SA-PPG'—may not transfer to naturally contaminated models. This is a real limitation, but it doesn't undermine the metric contribution: SA-PPG's argument holds regardless of contamination strength. The RailCap effectiveness, however, is more conditional. Also, there are no error bars, and the Table 1 comparison confounds readout choice with sample size (m=1 vs m=50), though the Table 2 decomposition with G-APP largely addresses that.\n\nWho is it for: anyone working on contamination mitigation evaluation or benchmark leakage. The metric is a candidate standard for that subfield. The RailCap strategy needs external validation on natural contamination before I'd trust its claimed state-of-the-art status.\n\nRecommendation: send it to peer review. The metric alone justifies referee time, and the RailCap claims deserve scrutiny. Suggest asking for code release, natural-contamination validation or at least a contamination-strength sweep, and uncertainty quantification.","headline":"A clean metric contribution that deserves review, with empirical claims that rest on a strong synthetic contamination setup.","tokens_in":14465,"tokens_out":2432,"would_cite":true,"duration_ms":21369,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Zero aggregate performance gap is not proof of restoration; stratified per-question probabilities overturn prior mitigation verdicts, and a step-wise decoding cap wins.","keywords":["data contamination","benchmark leakage","evaluation metrics","solve probability","per-question probability gap","decoding intervention","memorization suppression","RailCap"],"falsifier":"Take a model whose pretraining corpus is public and verify that a held-out benchmark like GSM8K or a paraphrased variant actually appears in its training data, then apply Identity, TED, LNE-blocking, shortcut-neuron patching, and RailCap and compute SA-PPG with the same $m=50$, $T=0.7$, $B=50$ protocol. If any estimate-based strategy beats RailCap on that naturally contaminated model, the paper's claim that restoration is 'substantially overestimated' and that RailCap is best would be falsified. A cheaper check is to test whether the greedy fall-back signal of Figure 1 appears on naturally contaminated questions at all.","tokens_in":13410,"feed_emoji":"📊","tokens_out":11870,"duration_ms":91082,"temperature":0.7,"pith_summary":"Benchmark leakage inflates scores through memorization, and mitigation strategies that suppress memorization are judged by how close their aggregate accuracy comes to the clean model's. The paper argues that this G-AP metric is structurally wrong: a 0/1 readout cannot represent per-question performance, and averaging before differencing lets over- and under-suppression cancel, so a zero gap can coexist with zero restoration. It replaces the readout with each question's solve probability, differences per question, and stratifies by the clean model's solve probability to block trivial all-fail strategies, giving SA-PPG. On the mitigation side it proposes RailCap, which caps the greedy-trajectory token's logit to the runner-up whenever sampling falls back onto the greedy path, so suppression is decided online step by step. The payoff claimed is concrete: prior strategies' restoration is substantially overestimated, and RailCap achieves the lowest SA-PPG across six model-by-domain settings.","feed_headline":"Per-question scores overturn benchmark-mitigation verdicts","feed_subtitle":"New metric shows prior restoration claims were overstated; a step-wise cap wins across models.","key_machinery":"The load-bearing object is the per-question probability gap, $\\Delta_s(q) = r^{\\mathrm{prob}}_{M_{\\mathrm{co}}^s}(q) - r^{\\mathrm{prob}}_{M_{\\mathrm{cl}}}(q)$, where $r^{\\mathrm{prob}}$ is the solve probability estimated from repeated sampling. A-PPG averages $|\\Delta_s(q)|$ over questions, and SA-PPG first averages inside bins defined by the clean model's solve probability, then across bins, so a strategy cannot score well by pushing every question to the clean model's most frequent probability value. The other load-bearing mechanism is RailCap's cap: once the last $n$ generated tokens match a window of the contaminated model's greedy trajectory, the trajectory successor's logit is replaced by $\\min(\\ell_t[x], v_t^{(2)})$, the runner-up's logit, so the clean model's next likely choice becomes samplable and suppression accumulates across steps. This turns contamination handling from a pre-hoc estimate into online, per-step supervision of the greedy fall-back signal.","core_discovery":"The paper's central claim is that 'zero gap is not restoration.' Under the prevailing G-AP metric, LNE-blocking looks near-perfect (0.0235 on Llama-2/GSM8K) while RailCap looks worse (0.0728), yet under SA-PPG the order reverses: RailCap is best (0.1914) and LNE-blocking (0.2932) is barely better than no intervention (0.3261). The same responses, the same models, and the same strategies change verdict when discrete averaged readouts are replaced by per-question solve probabilities and stratified aggregation. The paper also claims that existing strategies—TED, LNE-blocking, shortcut-neuron patching—are estimate-then-operate methods whose correctness is limited by a one-shot contamination estimate, whereas RailCap's step-wise greedy-trajectory cap needs no estimate and attains the lowest SA-PPG in all six settings tested. The takeaway, stated in the paper's own terms, is that restoration should be judged question by question and intervention should be supervised during generation.","pith_inferences":["I would test whether the greedy fall-back cap transfers to natural contamination by taking a model whose public pretraining corpus provably contains a benchmark's test split and running the same six-way comparison; if a one-shot estimator wins there, the synthetic fine-tuning setup is the reason RailCap leads.","SA-PPG's stratification depends on choosing a number of bins and on having a trustworthy clean reference; a natural next step is a sensitivity analysis over bin counts and a reference-free approximation for models with no clean counterpart.","The same per-question probability-gap logic could evaluate decoding interventions beyond contamination, such as detoxification or privacy filters, where the goal is also to match a reference behavior question by question rather than on average."],"forward_implications":["Every prior mitigation result reported as a G-AP gap needs to be re-checked with per-question probability gaps; a small G-AP is no longer evidence that a strategy restored the model.","Mitigation strategies should be designed around online signals such as fall-back onto the greedy trajectory instead of one-shot estimates of where contamination lies.","Under SA-PPG, an all-fail strategy is exposed as worse than no intervention, so the trivial-strategy loophole in aggregate metrics is closed.","In the six settings tested across three models and both verbatim and paraphrased contamination, RailCap attains the lowest SA-PPG, with LNE-blocking performing worse than identity on the paraphrased domain.","Paraphrased contamination is harder for estimate-based strategies, and RailCap's step-wise supervision is the only method that does not lose ground there."],"supporting_citations":[{"why":"Supplies the TED mitigation strategy and the G-AP-style evaluation protocol that the paper argues is flawed.","marker":"Dong et al., 2024"},{"why":"Supplies LNE-blocking, the baseline whose near-zero G-AP reading is overturned by SA-PPG, and the G-AP readout practice.","marker":"Hou et al., 2025"},{"why":"Supplies shortcut-neuron patching, the runner-up baseline under SA-PPG and a representative estimate-then-operate strategy.","marker":"Zhu et al., 2025"},{"why":"Provides GSM8K, the benchmark whose test split is used to simulate contamination and to construct the paraphrased PQ set.","marker":"Cobbe et al., 2021"},{"why":"Supplies the chain-of-thought prompting format used for both fine-tuning data and evaluation.","marker":"Wei et al., 2022"},{"why":"Supplies Llama-2-7B, one of the three model families used in the experiments.","marker":"Touvron et al., 2023"},{"why":"Supplies Pythia-12B, the fully open model whose public corpus lets the authors verify that the base model is uncontaminated.","marker":"Biderman et al., 2023"},{"why":"Supplies Gemma-4-E2B, the third model family used in the cross-model comparison.","marker":"Gemma Team, 2026"},{"why":"Motivates constructing the clean model by fine-tuning on the training split, the basis for the clean-versus-contaminated comparison.","marker":"Dominguez-Olmedo et al., 2025"},{"why":"Supplies the LoRA fine-tuning framework used to create the clean and contaminated models.","marker":"Zheng et al., 2024"}],"fun_headline_variants":["Zero gap is not restoration: per-question scores say so","SA-PPG flips mitigation verdicts: blocking overrated, RailCap wins","Per-question probability gaps expose false restoration claims","RailCap: step-wise decoder cap beats all prior mitigation","Benchmark contamination: average scores mislead, per-question matters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that contaminating a model by additional fine-tuning on the benchmark's own test questions faithfully represents the contamination a model acquires during large-scale pretraining; if real memorization differs in scale or form, the measured overestimation and RailCap's advantage may not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Zero gap is not restoration: per-question scores say so","SA-PPG flips mitigation verdicts: blocking overrated, RailCap wins","Per-question probability gaps expose false restoration claims","RailCap: step-wise decoder cap beats all prior mitigation","Benchmark contamination: average scores mislead, per-question matters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000338,"raw_usage":{"total_tokens":1920,"prompt_tokens":1046,"completion_tokens":874,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":662,"completion_tokens_details":{"reasoning_tokens":787}},"tokens_in":662,"tokens_out":874,"duration_ms":8836,"temperature":1.0,"reasoning_tokens":787,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T05:57:04.872945+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a model whose pretraining corpus is public and verify that a held-out benchmark like GSM8K or a paraphrased variant actually appears in its training data, then apply Identity, TED, LNE-blocking, shortcut-neuron patching, and RailCap and compute SA-PPG with the same $m=50$, $T=0.7$, $B=50$ protocol. If any estimate-based strategy beats RailCap on that naturally contaminated model, the paper's claim that restoration is 'substantially overestimated' and that RailCap is best would be falsified. A cheaper check is to test whether the greedy fall-back signal of Figure 1 appears on naturally contaminated questions at all.","supporting_citations":[{"cited_title":"Establishing Trustworthy LLM Evaluation via Shortcut Neuron Analysis","cited_arxiv_id":null,"evidence_quote":"Supplies shortcut-neuron patching, the runner-up baseline under SA-PPG and a representative estimate-then-operate strategy."},{"cited_title":"and Zhou, Denny","cited_arxiv_id":null,"evidence_quote":"Supplies the chain-of-thought prompting format used for both fine-tuning data and evaluation."},{"cited_title":"Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling","cited_arxiv_id":null,"evidence_quote":"Supplies Pythia-12B, the fully open model whose public corpus lets the authors verify that the base model is uncontaminated."}],"review_version":1}