{"id":"82665808-dd02-4e54-ad7e-cdc91934fac3","arxiv_id":"2608.12652","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A zero-sum contrast on probe accuracy across layers, recentred on a placebo baseline from the reference set, holds nominal false positive rates in contamination detection where level-based and flat-profile statistics fail.","lead":"Probing a language model's internal activations can reveal whether it memorized benchmark questions, but the obvious way to do it produces false alarms. This paper specifies a statistically controlled probing protocol and shows it stays reliable where simpler alternatives fail.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The depth-contrast weight vector w in Eq. (14) is never defined, so the 'RSCP statistic' whose nominal rate and power are claimed is not fully specified; all of the validation results are conditional on an unreported choice.","rationale":"The reader's weakest_assumption concerns the analyst-chosen surface key used to build the placebo baseline. That is a real limitation on applicability, but not on the formal validity of the test: §4.6 measures that a degraded key costs power without inflating false positives, so the test remains valid while sometimes being blind. The more load-bearing gap is the undefined contrast weight in the very statistic being validated. A statistical protocol is not reproducible or auditable if its test statistic is only described as 'a zero-sum contrast.' The paper's release of code and generated-macro artifacts is genuine independent support, and if w is defined in the archived code the gap is readily fixable; that is exactly why the verdict should remain conditional rather than move to reject. The simulations themselves are carefully executed and honestly reported, including the 0.083 calibration offset, and I do not see an internally inconsistent statistical argument once the missing definition is supplied. The concern therefore reinforces the reader's CONDITIONAL verdict rather than overturning it.","tokens_in":26107,"tokens_out":9718,"duration_ms":109833,"concrete_test":"Extract w_l from the archived release (doi:10.5281/zenodo.21855510 or the GitHub snapshot) and re-run the Phase 0 validation (phase0c_validation.py) under two alternative zero-sum contrasts: a linear depth slope w_l = l - mean(l), and an early-vs-late two-block contrast w_l = +1 for the first half of layers and -1 for the second half. Compare the false-positive rates in Table 4 and the power curve in Figure 5 against the reported values using the same seeds and 1000 replicates per cell. If any cell moves outside Monte Carlo error from the reported 0.083 false-positive rate or the 0.8-power crossing near epsilon = 1.5, the central claim is not robust to the unspecified weight choice and the paper must specify and justify w. If the results are unchanged, the text still needs to state w explicitly for reproducibility.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim is that the RSCP statistic has controlled false positives and useful power. That statistic is T_adj = sum_l w_l (BA_l - BA_plc_l) (Eq. 14), but the zero-sum weight vector w is never defined anywhere in the text: no formula, no default, no pre-registration, and no sensitivity analysis. The paper only states that the weights sum to zero and calls the result a 'single pre-registered scalar.' Every number in §4, Table 7, and Figures 5-7 is therefore a property of one unreported implementation of w. If w was chosen after inspecting the validation curves, the calibration is circular; if it was chosen arbitrarily, the claim that the protocol 'holds the nominal rate' is contingent on that arbitrary choice. The archived code may pin w down, but the scientific text does not, and the paper deliberately presents the statistic as a protocol rather than as a code artifact. This is the load-bearing gap because the entire validity argument is about this specific contrast, not about the family of all zero-sum contrasts.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RSCP, a protocol for detecting benchmark contamination in transformer language models by probing residual-stream activations. The protocol extracts activations at the item prefix, trains cross-fitted ridge probes at every layer, and computes a zero-sum depth contrast of balanced accuracy recentred on a placebo baseline built from a split of the reference set, tested against a label-permutation null. The authors validate the procedure on two deliberately dissimilar synthetic simulators, compare it against four simpler alternatives (level-based separability, flat-profile contrast, item bootstrap, half-size baseline), and report a small real-model audit on Pythia/Pile, WikiMIA, and Oren et al.'s contaminated PIQA checkpoint. The central claims are that the corrected protocol controls false positives at near-nominal rates and reaches 0.8 power at separations of one to two accuracy points, while the simpler alternatives fail, and that real baseline depth profiles are non-flat and track surface separability.","tokens_in":26302,"tokens_out":4928,"duration_ms":50742,"significance":"If the load-bearing definitional gap noted below is repaired, this is a valuable contribution. The paper identifies a real confound in activation-based contamination detection—probe accuracy reflects any surface difference between suspect and reference sets—and proposes a placebo-baseline correction that is well motivated and clearly explained. The validation is unusually careful: two structurally different simulators, explicit measurement of the failure modes of alternative statistics, a released implementation with property tests, and honest limitation statements including the acknowledged dependence on the analyst-chosen surface key (L3b) and the calibration-transfer assumption (L5). The real-model audits, though small, are appropriately conservative: the protocol refuses a verdict on the temporal WikiMIA split and returns nulls on well-matched Pile arms. The paper also provides falsifiable predictions and a concrete validation plan, which strengthens its scientific standing.","major_comments":[{"comment":"The weight vector w in Eq. (14) is never defined. The text states only that the weights sum to zero (§4.1) and calls the result a 'single pre-registered scalar' (§3.6), but no formula, default, pre-registration, or sensitivity analysis for w is given anywhere in the manuscript. Every reported false-positive rate, power curve, and real-model outcome in §4, Table 7, and Figures 5–7 is therefore a property of an unreported implementation of w. This is load-bearing because the paper's central claim is about this specific contrast, not about the family of all zero-sum contrasts. The authors should either specify w explicitly in the text (e.g., a closed-form contrast with defined coefficients) and report how results vary under alternative zero-sum weight vectors, or re-frame the contribution as tied to a specific archived code artifact rather than a fully specified protocol.","section":"§4.1, Eq. (14); §3.6"},{"comment":"The inference section is internally inconsistent about which statistic is tested. Eq. (9) defines the permutation statistic as T = max_l [BA_l − BA_nuis], but the following paragraph says the statistic actually tested is the recentred contrast of Eq. (14). Since Eq. (14) is itself unspecified due to the missing w, the reader cannot determine the actual permutation procedure used to produce the reported p-values. This needs to be resolved: either Eq. (9) is a vestigial description of a rejected alternative and should be removed or clearly labeled as such, or the recentred contrast must be fully defined and the permutation algorithm restated in terms of it.","section":"§3.6, Eq. (9)"},{"comment":"The protocol selects the placebo split by searching over split coarseness and taking the split whose embedding-layer separability is closest to the observed value. This is a data-dependent model-selection step, and the paper does not state explicitly whether the false-positive-rate simulations in §4 include this selection or whether the selection is made blind to the test contrast. If the search uses the same data that are later tested, the selection could in principle distort the null distribution beyond what the simulations capture, particularly in the real audits where only one realization exists. The authors should clarify the exact selection rule and confirm that it was part of the simulated protocol; if it was not, the nominal-rate claims in Tables 3 and 4 are not directly applicable to the real-audit procedure.","section":"§4.4, placebo split selection"}],"minor_comments":[{"comment":"The sentence 'Calibration is close to nominal but not exact, sitting between 0.083 and 0.083 across the null cells' appears to contain a typo, since a range with identical endpoints is not a range; Table 4 also includes a value of 0.033 in the Sim-B 50% row.","section":"§4.8"},{"comment":"The phrase 'single pre-registered scalar' is asserted without any pre-registration being shown. If no pre-registration exists for the weight vector w or the analysis plan, the claim should be removed or replaced with a description of how the statistic was fixed before seeing the validation data.","section":"§3.6"},{"comment":"The description of the placebo-split search would benefit from exact details: the number of coarseness levels considered, the metric used to measure 'closest' embedding-layer separability, and whether this metric is computed on the suspect-vs-reference contrast or on a held-out portion of the reference set.","section":"§4.4"},{"comment":"The table in Figure 1 uses 'yes/no/partial' symbols that are not defined in the caption; adding a legend or footnote would improve readability.","section":"§2.5"}],"recommendation":"major_revision","confidential_remarks":"The undefined weight vector w is the critical issue; it is fixable but must be addressed head-on. The paper's honesty about limitations and its extensive validation are genuine strengths, and the authors seem aware of the main threats to their method. If the authors can specify w, resolve the Eq. (9)/Eq. (14) inconsistency, and clarify the placebo-split selection, the paper would be a solid candidate for acceptance. I would also encourage the editor to verify that the released code indeed contains a definition of w and that the reported numbers are reproducible from that definition."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The stress-test note is right, and it lands on the load-bearing part of the paper. The statistic T_adj in Eq. (14) is defined as a weighted sum over layers with zero-sum weights w_l, and the paper never says what those weights are. No formula, no default, no sensitivity analysis. Every false-positive and power number in §4 is therefore a property of one unreported implementation of the contrast. That is a serious reproducibility gap, not a cosmetic one: the whole validity claim is about this specific statistic, not about the whole family of zero-sum contrasts. If the weights were chosen after looking at validation curves, the calibration is circular; if chosen arbitrarily, the claimed nominal rate is contingent on that choice. The archived code may pin it down, but the scientific text doesn't.\n\nCredit where it's earned. The paper is unusually honest and methodologically careful. Stating exchangeability as a precondition, rather than a caveat, is a real contribution. The prefix extraction rule is a genuine fix for the competence confound, and the argument for it is clear. The placebo baseline is a sensible idea, and the demonstration that level-based statistics track the control set's dimension is useful. The permutation null that refits the probe through a label-free smoother is clever, and the paper is right that an item bootstrap misses fit variance. The validation under two deliberately dissimilar simulators is real work, and the Phase 1 audits are transparent: null on well-matched Pile arms, refusal to interpret the WikiMIA temporal split, no manufactured positives.\n\nThe soft spots are proportionate. The missing weights are the main one. The calibration null rejection rate of 0.083 instead of 0.05 is acknowledged, so near-threshold p-values are approximate. The placebo split is chosen by searching over coarseness on the same data, and the paper doesn't fully account for the selection effect. And the only positive result sits on a split where exchangeability fails; the paper says this itself, so it's a boundary, not a hidden flaw.\n\nThis paper deserves a serious referee. It's for people working on contamination detection, membership inference, and probing methodology, and it may shift how they think about baselines. But before anyone can use the protocol, the weights need to be defined, with a sensitivity analysis over reasonable choices. I would not currently cite the headline numbers, but I would bring this to a reading group to discuss what a reproducible protocol should require.","headline":"A disciplined contamination-detection protocol with real validation, but its headline statistic is undefined in the text — the weights of the depth contrast appear only as 'w' and are never specified.","tokens_in":26838,"tokens_out":2045,"would_cite":false,"duration_ms":23371,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that residual-stream probing can detect benchmark contamination only when the probe's depth profile is contrasted against a per-model placebo baseline, not when raw probe accuracy is reported.","keywords":["benchmark contamination","residual-stream probing","excess separability","membership inference","nuisance control","placebo baseline","permutation test","exchangeability"],"falsifier":"Run the protocol's internal-null control on a real transformer many times—split a reference set at random, compute the recentred contrast, repeat—and compare the empirical false positive rate against the nominal 0.05; the synthetic validation reports 0.083 across null cells, so a real-model rate far above 0.05 would falsify the validity claim. A second check: on the deliberately contaminated checkpoints used in the paper's Phase 3 audit, the recentred contrast should move toward rejection as the duplication count rises from 2 to 50; if it stays negative and non-monotone, the exposure-calibration premise fails.","tokens_in":25850,"feed_emoji":"🧪","tokens_out":9063,"duration_ms":76130,"temperature":0.7,"pith_summary":"This paper claims that the natural activation-probe approach to detecting benchmark contamination—train a linear probe on internal activations to separate suspect from reference items—measures the wrong quantity. Raw probe accuracy, even after subtracting a nuisance baseline, mixes memorisation with any surface difference between the item sets, and the paper measures how badly that fails: the false positive rate tracks the dimension of the analyst's control set, from 0.03 to 0.99 under a true null. The fix is to test a zero-sum contrast on the depth profile of probe accuracy, recentred on a placebo baseline built from the analyst's own reference set, against a permutation null that refits the probe. Under two dissimilar simulators this corrected protocol holds the nominal 0.05 rate and reaches 0.8 power at a separation of one to two accuracy points. On real transformers the placebo depth profiles are measurably non-flat—spanning up to 29.1 accuracy points on a temporal split—and their non-flatness tracks how different the item sets are as text, so the correction is largest exactly where a naive test would be most wrong.","feed_headline":"Probe-based contamination tests fail without a placebo baseline","feed_subtitle":"Raw probe accuracy mixes memorisation with topic drift; the placebo-recentred contrast restores a controlled error rate.","key_machinery":"The central object is the residual-stream contamination probing (RSCP) protocol. It trains cross-fitted ridge probes at every layer on prefix activations, never on full items; builds a nuisance baseline from bag-of-embeddings, character n-gram, length, and reference-language-model features; and forms the level-matched placebo by splitting the reference set on an analyst-chosen surface key at matched embedding-layer separability. The tested statistic is the recentred zero-sum depth contrast $T_{\\mathrm{adj}}=\\sum_\\ell w_\\ell(\\mathrm{BA}_\\ell-\\mathrm{BA}^{\\mathrm{plc}}_\\ell)$, and the null is a label permutation whose key property is that the cross-fitted smoother $\\Pi_\\ell$ is label-free, so each permuted draw costs one matrix-vector product and refits the probe. That label-free smoother is what makes the permutation test feasible and what captures fit variance an item bootstrap misses.","core_discovery":"The paper's central claim is that excess separability—the probe's balanced accuracy minus what a declared nuisance family (token embeddings, n-gram TF-IDF, length statistics, reference-model likelihood summaries) can already separate—is a meaningful contamination signal only when measured as a depth contrast, not as a level. It reports that reporting the level makes false positives track the control set's size, contrasting against a flat depth profile fails in both directions (rejecting a true null 0.72 of the time when surface decodability rises with depth, losing power when it falls), an item bootstrap understates variance relative to a permutation null that refits the probe, and a half-size baseline triples the error rate. The corrected statistic, $T_{\\mathrm{adj}}=\\sum_\\ell w_\\ell(\\mathrm{BA}_\\ell-\\mathrm{BA}^{\\mathrm{plc}}_\\ell)$, recentred on a level-matched placebo baseline split from the reference set and tested against label permutations, holds the nominal rate under both simulators and reaches 0.8 power near $\\varepsilon=1.5$. The empirical Phase 1 result is that real baseline depth profiles are not flat and their non-flatness scales with surface difference (correlation 0.87 over six audits), all well-matched held-out arms come back null, and the only positive sits on the temporal split where exchangeability fails, so the paper does not claim to have shown that transformers carry a familiarity direction.","pith_inferences":["Beyond the paper: if baseline non-flatness really tracks the surface difference between item sets, then any probe-based contamination detector that omits a placebo baseline is not merely noisy but systematically biased in the direction of the analyst's control set.","Beyond the paper: the paper's null on deliberately contaminated checkpoints at high duplication count suggests, if replicated, that linear residual-stream familiarity may be a weak or absent signal at practical duplication counts, which would redirect effort toward black-box order tests and corpus search for low-duplication leakage.","Beyond the paper: a testable extension is to apply the protocol to a benchmark and its commissioned twin to separate item memorisation from format overfitting; a significant contrast with a small detection floor would support the memorisation reading, while a null would point to overfitting.","Beyond the paper: the dependence of power on the analyst's surface key implies that publishing the chosen key and a sensitivity analysis over alternative keys should become part of any audit report using this protocol."],"forward_implications":["Existing probe-based contamination results reported on temporal splits should be read as upper bounds on the memorisation signal, not estimates, because a blind classifier already separates those item sets.","Audits that return null should report a detection floor $m^*$; otherwise the null cannot be distinguished from an underpowered instrument.","Benchmark designers should publish matched twin sets or canary strings at release, because most current benchmarks lack the exchangeable reference set the protocol requires.","The protocol's negative controls on well-matched held-out corpus arms suggest that on corpora seen roughly once, no depth-dependent familiarity signal is linearly decodable at current sensitivity.","A positive contamination verdict requires both a significant recentred contrast and a reference set satisfying exchangeability; on the temporal split the protocol refuses a verdict rather than reporting one."],"supporting_citations":[{"why":"Introduces the activation-probe contamination detector whose reported statistic RSCP corrects, including its AUC measurements on temporal splits and contaminated checkpoints.","marker":"[42]"},{"why":"Shows that temporal membership-inference splits are separable blind, establishing the nuisance confound RSCP is designed to control.","marker":"[20]"},{"why":"Provides the near-chance membership-inference baseline on Pythia-style corpora seen approximately once, which motivates the well-matched null arms.","marker":"[24]"},{"why":"Supplies the deliberately contaminated checkpoints with exact exchangeability used for the paper's Phase 3 null result.","marker":"[49]"},{"why":"Provides the control-task logic that RSCP's required controls extend to set-level nuisance features.","marker":"[34]"},{"why":"Supplies a commissioned twin benchmark, the clearest existing instance of the exchangeability-by-construction requirement.","marker":"[67]"},{"why":"Provides the model family with known data ordering and checkpoints used in the real-model Phase 1 audits.","marker":"[6]"},{"why":"Contributes the likelihood-based per-token statistics that are included in the declared nuisance family.","marker":"[55]"}],"fun_headline_variants":["Naive probe tests drown in false positives; placebo fixes it","Probe contamination scores need a placebo baseline to hold","Baseline drift breaks probe tests; depth contrast restores power","Residual-stream probes fail without a level-matched placebo","Probe depth contrast beats raw accuracy for contamination"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The placebo baseline is built by splitting the reference set on an analyst-chosen surface key, and the entire instrument assumes that key tracks the nuisance dimensions along which the suspect and reference sets actually differ; the paper reports that a degraded key cuts power from 0.825 to 0.450 while leaving false positives controlled.","fun_headline_variants_meta":{"raw":{"variants":["Naive probe tests drown in false positives; placebo fixes it","Probe contamination scores need a placebo baseline to hold","Baseline drift breaks probe tests; depth contrast restores power","Residual-stream probes fail without a level-matched placebo","Probe depth contrast beats raw accuracy for contamination"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000883,"raw_usage":{"total_tokens":3946,"prompt_tokens":1210,"completion_tokens":2736,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":826,"completion_tokens_details":{"reasoning_tokens":2657}},"tokens_in":826,"tokens_out":2736,"duration_ms":19396,"temperature":1.0,"reasoning_tokens":2657,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:02:39.847260+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the protocol's internal-null control on a real transformer many times—split a reference set at random, compute the recentred contrast, repeat—and compare the empirical false positive rate against the nominal 0.05; the synthetic validation reports 0.083 across null cells, so a real-model rate far above 0.05 would falsify the validity claim. A second check: on the deliberately contaminated checkpoints used in the paper's Phase 3 audit, the recentred contrast should move toward rejection as the duplication count rises from 2 to 50; if it stays negative and non-monotone, the exposure-calibration premise fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the activation-probe contamination detector whose reported statistic RSCP corrects, including its AUC measurements on temporal splits and contaminated checkpoints."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the near-chance membership-inference baseline on Pythia-style corpora seen approximately once, which motivates the well-matched null arms."},{"cited_title":"Hashimoto","cited_arxiv_id":null,"evidence_quote":"Supplies the deliberately contaminated checkpoints with exact exchangeability used for the paper's Phase 3 null result."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the control-task logic that RSCP's required controls extend to set-level nuisance features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies a commissioned twin benchmark, the clearest existing instance of the exchangeability-by-construction requirement."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the model family with known data ordering and checkpoints used in the real-model Phase 1 audits."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the likelihood-based per-token statistics that are included in the declared nuisance family."}],"review_version":1}