Pith. sign in

REVIEW 3 major objections 5 minor 38 references

Temporal Leakage in LLM Backtesting: Measurement, Validation, and Adjusted Scores

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The standard pre/post contamination check cannot separate recency from leakage; measuring contamination requires one external reference, supplied as a known cutoff or a matched clean control.

desk verdict A genuinely useful non-identification theorem plus a ground-truth validation, wrapped in measurement claims that outrun the assumptions; a referee can fix it. read the letter →

arxiv 2608.02985 v1 pith:QKMP4LJQ submitted 2026-08-04 cs.LG cs.CLstat.ML

Zeyu Zhang , Bradly C. Stadie This is my paper · ORCID
classification cs.LGcs.CLstat.ML
keywords temporalleakageLLMbacktestingtrainingcutoffrecencyconfoundpartialidentificationregressiondiscontinuitydifference-in-differencesleakage-adjustedscore
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper targets a practical question in LLM evaluation: when a model is backtested on questions whose outcomes resolved before the model's training cutoff, how much of its apparent skill is memorized leakage rather than genuine forecasting? The paper argues that the standard defense, comparing scores on questions before and after the cutoff, cannot answer this question. It proves that no passive backtest can separate leakage from legitimate recency, which also mimics the leakage pattern, and that a flat pre/post profile is therefore not evidence of cleanliness. The paper's constructive claim is that one defensible external reference—a known cutoff with continuity of honest skill, or a matched clean control model—restores measurement and yields a leakage-adjusted score. If this is right, backtests do not need to be discarded; they need one reference, and audits should report power floors rather than cleanliness verdicts.

What carries the argument

The load-bearing object is the operational risk decomposition $m(x,g)=m_0(x,g)-L(x,g)$: the observable loss surface splits into an honest surface $m_0$, which may vary freely with distance from the cutoff (recency), and a leakage term $L\ge 0$ that is zero for $g\ge 0$. Theorem 2 shows this decomposition is unidentifiable from passive scores, so the paper's identification routes work by importing a reference that pins down $m_0$ on the leaked side. The second central object is the per-question leakage law $L=b_0 w(2-w)$ derived under the convex-pull assumption, where $b_0=(P_{\mathrm{hon}}-Y)^2$ is the honest Brier stakes and $w\in[0,1]$ is the extraction weight; the quadratic factor makes partial memorization disproportionately rewarded. The named estimators are regression discontinuity at a known cutoff (which identifies the boundary jump $J$), a standardized difference-in-differences against a matched clean control (which identifies the global $B$), and a calibration-gated paraphrase residual covariance (which detects evidence leakage without measuring it).

What would settle it

One observation would settle the central measurement claim: take a model whose training corpus is fully known, plant outcome-stating documents at a known dose as in the paper's twin experiment, but use as the difference-in-differences control a twin trained on a deliberately different recency schedule; if the estimator returns a nonzero adjusted inflation on questions clean for both twins, the transportable-recency assumption, and with it the global adjusted score, fails.

Watch

Extended reading notes

Core claim

The paper's central result is a sharp non-identifiability theorem. Writing the observable conditional mean forecast loss as $m(x,g)=\mathbb{E}[(P-Y)^2\,|\,X=x,G=g]$, with $g<0$ for questions that resolve before the training cutoff, the paper defines the leakage inflation $B=\mathbb{E}[m_0(X,G)-m(X,G)\,|\,G<0]$, where $m_0$ is the loss the model would incur using only legitimate information. Under the minimal assumption that leakage is nonnegative before the cutoff and zero after it, Theorem 2 shows $B$ is not a functional of the passive law of $(P,Y,X,G)$: the sharp identified set is $[0,\mathbb{E}[1-m(X,G)\,|\,G<0]]$, and more scoring data does not shrink it. The practical corollary is that a flat pre/post profile is not evidence of a clean backtest, and the paper demonstrates the converse failure with four flagship models that fail the standard check on questions that resolved after their cutoffs. Identification is restored by one external reference: a known cutoff identifies the boundary leakage $J(x)=L(x,0^-)$ under honest-risk continuity, and a matched clean control identifies the global inflation $B=\Delta_M-\Delta_{M_0}$ under transportable recency. Under a convex-pull model of partial recall, per-question leakage equals $b_0 w(2-w)$, where $b_0$ is honest uncertainty and $w$ is the extraction weight, so leakage concentrates on surprising, well-covered questions. The paper validates the estimators by planting leakage in twin models, recovering the injected dose and returning null on clean questions, and on frontier models it detects one cutoff-localized signature and clears five models after adjustment.

Load-bearing premise

The load-bearing premise is that the auditor can trust one external reference: either a documented training cutoff at which honest risk is continuous, or a clean control model whose standardized pre-to-post change matches the target's, a condition the paper states is not fully testable.

Editorial extensions

If this is right

  • A flat pre/post performance profile cannot certify a clean backtest; under Theorem 2, any observed profile is compatible with zero leakage and with leakage up to $\mathbb{E}[1-m\,|\,G<0]$.
  • One external reference is sufficient: a documented cutoff yields a boundary leakage estimate, and a matched clean control yields a global leakage-adjusted score, with $\mathrm{Risk}_{\mathrm{adj}} = \mathrm{Risk}_{\mathrm{meas}} + B$ for risk metrics and $\mathrm{Score}_{\mathrm{adj}} = \mathrm{Score}_{\mathrm{meas}} - B$ for higher-is-better scores.
  • Leakage is not a uniform rate: it concentrates where the crowd was surprised and training coverage was dense, so audits should target high-stakes questions rather than an average contamination rate.
  • Partial memorization is enough to distort results: an extraction weight of $w=0.5$ already yields 75 percent of the full-memorization inflation.
  • Twin-model validation with planted leakage recovers the injected dose and returns null on clean questions, and the positive control on dated code problems fires only at documented cutoffs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: benchmark audits should routinely report the identified set or a power floor alongside any cleanliness claim, because a null pre/post gap alone proves nothing.
  • Editorial extension: if this recipe is adopted widely, model vendors have an incentive to publish dated clean vintages or archived real-time forecasts, since those are the one defensible reference that makes an adjusted score possible.
  • Editorial extension: the concentration law suggests a testable design principle—benchmarks that deliberately oversample surprising, well-covered questions should make leakage statistically visible sooner, improving audit power at fixed sample size.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies temporal leakage in LLM backtesting, where a model's training data may contain information about outcomes that resolve after the as-of date. It defines the leakage inflation B as the pre-cutoff shortfall of the observable loss relative to an honest surface, and proves (Theorem 2) that B is not identifiable from the passive joint law of forecasts, outcomes, and timing: the sharp identified set is [0, E[1-m | G<0]]. The paper then proposes three routes to restore measurement: regression discontinuity at a known cutoff, difference-in-differences with a clean control, and a paraphrase-based diagnostic. It derives a concentration law B = E[b0 w(2-w) | G<0], validates the estimators by re-analyzing the Hubble controlled-pretraining suite and by planting leakage in twin models, and deploys the audit on frontier models, reporting one cutoff-localized signature and several null results. The central negative claim is that flat pre/post profiles are uninformative, and the central positive claim is that one external reference yields a leakage-adjusted score.

Significance. The paper's negative result is a genuine and important contribution: Theorem 2 sharply characterizes why pre/post contamination checks cannot measure leakage, and the empirical demonstration in Figure 1 — four models flagged despite all scored questions resolving after their cutoffs — is a valuable caution for the community. The twin experiment M3 is a strong methodological innovation: it provides per-question ground truth for leakage, and the paper's pre-registration, claim-to-evidence contract, power floors, and reproducibility map set a high standard. If the measurement claims were confined to the footprint interpretation, this would be a substantial paper for both the forecasting and LLM-evaluation literatures. However, the current abstract and Corollary 6 present the DiD route as yielding the leakage-adjusted score that removes B exactly, whereas the paper's own validation shows the route returns a conservative footprint when Assumption 1 is violated. The practical promise of exact measurement is therefore not yet supported, and the headline claims need to be reconciled with the reported evidence.

major comments (3)
  1. [Section 6.2 / D.4 and Proposition 5] The clean-control DiD route point-identifies the inflation B only under Assumption 1's support restriction L=0 for g>=0, but the paper's own M3 twin experiment measurably violates that restriction: the treatment twin acquires an 83% NO base-rate lean that depresses post-cutoff loss by about 0.028 Brier, so the perfect clean-control DiD contrast is about +0.052 while the per-question ground-truth inflation is +0.024. Thus the clean-control route does not identify B when leakage shifts priors; it returns the total contamination footprint, overstating B by roughly a factor of two. Corollary 6 and the abstract present this as a 'leakage-adjusted score,' but Section 6.2 itself acknowledges the footprint reading. This is a load-bearing discrepancy: the paper should either revise the headline measurement claim to a conservative upper-bound adjusted score, or add a spillover correction that restores point identification under a weakened version of Assumption 1.
  2. [Section 6.1 / Figure 3 (right) and D.4] The 'zero-free-parameter law check' is not zero-free-parameter: the extraction weight w is estimated from the treatment twin's probability movement, and that fitted w is then plugged into b0 w(2-w) before comparing with the observed contrasts. The agreement is therefore a within-data consistency check, not an independent, parameter-free prediction of the law. Please either obtain w from an external source, report the prediction as using one fitted parameter with appropriate uncertainty propagation, or drop 'zero-free-parameter' from the claim. This matters because the law is presented as a central theoretical contribution, and its empirical support should be stated accurately.
  3. [Section 7.2 / Table 7 and Figure 4] The M5 nulls do not support the strong deployment claim that five models were 'cleared' of leakage. For the primary target, the adjusted estimate is +0.020 with 95% CI [-0.063, +0.074], which includes the 0.05 Brier-reduction effect that the design is said to detect with 97% power. The conclusion should be worded as 'no boundary-differential inflation detected above the stated power floor,' and the abstract's phrase 'clear five models whose apparent advantages were recency alone' should be tempered accordingly. The power-floor language in Section 6.2 is a step in this direction, but the abstract and conclusion still overstate what the non-significant estimates establish.
minor comments (5)
  1. [Section 1 / Figure 1] The caption 'models that cannot have leaked' is shorthand: the models cannot have memorized the outcomes of the specific post-cutoff questions, not that they cannot have leaked in general. Please rephrase to avoid ambiguity.
  2. [Section 6.1] The phrase 'zero-free-parameter prediction' in the main text conflicts with Appendix D.4, where w is solved per tercile from L/b0 = w(2-w). Replace the phrase with 'one-parameter prediction' and note the fitting procedure in the main text.
  3. [Table 5, M5 row] The row says 'paired questions' but the 265 pre- and 1,207 post-boundary questions are not paired across time; they are the same questions scored by multiple models. Clarify that 'paired' refers to a per-question target-minus-control difference.
  4. [Section 6.2 / Figure 9] The claim that PRC 'fires only where leakage was injected' should specify that this holds for the calibrated, twin-differenced PRC; the raw PRC is negative in every cell, as Remark 2 warns.
  5. [Appendix B.1.3] The statement that 'the counterfactual model cannot be instantiated for a deployed system' is correct, but the discussion could explicitly cite retraining-based audits from Section 8 to help readers see why those do not solve the problem.

Circularity Check

1 steps flagged · score 4.0 of 10

Central non-identifiability and identification theorems are self-contained; the concentration-law 'zero-free-parameter prediction' is a within-data consistency check because w is fitted from the same twin forecasts whose Brier contrasts are then compared with the law.

  1. fitted input called prediction [Section 6.1, Figure 3 (right); see also Section 9 and Appendix E.3]
    "Red squares: the zero-free-parameter prediction ¯b0 ˆw(2−ˆw), with ˆw fitted from the treatment twin’s probability movement alone, never from the plotted contrasts."

    Under Assumption 2 the Brier saving is an algebraic identity: if P=(1−w)Phon+wY then E[(Phon−Y)^2−(P−Y)^2]=E[b0 w(2−w)]. Therefore, once w is estimated from the treatment twin's probability movement, the 'predicted' contrast is a deterministic transformation of those same forecasts and outcomes, not an independent out-of-sample test. The paper's own Section 9 says 'the twins impose rather than test this restriction', and Appendix E.3 calls such checks 'consistency checks of the machinery, not tests of the convex-pull assumption itself.' The match across difficulty bins is thus a within-data consistency check; the 'zero-free-parameter' framing overstates its independence.

full rationale

Theorem 2 is a genuine non-identifiability result with an explicit construction (m̃0,t = m + t(1−m)) and does not assume its conclusion. The identification routes (RD, DiD, PRC) are algebraic consequences of stated, and largely testable, assumptions; they do not rely on a self-citation chain. The only self-citation (Zhang et al. 2026) appears in a related-work list and is not load-bearing. The one reduction-to-input is the M3 validation of the concentration law: w is fitted from the same twin model whose Brier contrasts are compared with b0 w(2−w), and the paper concedes the twins impose, not test, the convex-pull restriction. This weakens the evidential weight of the law's empirical confirmation but does not affect the central identification theorems. The clean-control DiD overstatement in M3 (footprint +0.052 vs per-question truth +0.024) is a validity/scope issue, explicitly labeled by the paper as a footprint, not a circularity. Overall score 4: central claims are independently argued; one secondary 'prediction' reduces to a within-data consistency check.

Assumptions & free parameters 4 free parameters · 7 assumptions · 0 invented entities

No new physical or ontological entities are introduced. The causal graph's bridge data and leakage signal are modeling constructs with no independent falsifiable handle outside the analysis. The free parameters and axioms above are the main commitments the central claims depend on.

free parameters (4)
  • Extraction weight w(q) = globally 0.24/0.23/0.33/0.42 at doses 1/4/16/64; per-difficulty 0.5/0.30/0.16
    Central latent parameter in Assumption 2. It is fitted from the treatment twin's probability movement in M3, then reused in the zero-free-parameter law check.
  • LLM overconfidence temperature T = T approximately 8, reported as a lower bound
    Fitted to Qwen3.5 leakage-free post-boundary forecasts for PRC recalibration; used to justify calibration and set the T4 sweep range.
  • RD/DiD bandwidths and windows = 90, 120, 160, and 240 days depending on experiment
    Hand-chosen analysis windows. Boundary estimates and power floors change with bandwidth; robustness checks are reported, but headline local estimates depend on these choices.
  • M3 uncertainty screen thresholds = exclude base P(YES)>0.85 when outcome YES and P(YES)<0.15 when outcome NO
    Pre-registered screen removing questions the base model already knows; it shapes the injected pool and therefore the measured dose response and law fit.
assumptions (7)
  • domain assumption Assumption 1: nonnegative pre-cutoff leakage and support restriction L=0 for g>=0
    Defines the operational target and underpins Theorem 2. The paper's own twin experiment measures a post-side base-rate lean that violates L=0 on g>=0, so the assumption is measurably false under concentrated injection.
  • ad hoc to paper Assumption 2: convex-pull leakage, P=(1-w)Phon+wY with w in [0,1]
    Defines the law B=E[b0 w(2-w)]. It excludes misleading leakage (w<0) and is not tested against non-convex extraction mechanisms in real data.
  • domain assumption Assumption 3: honest-risk continuity at the cutoff
    Route R1 identifies the boundary jump only if honest competence has no discontinuity at T; corpus density or prompt-date effects could violate it.
  • domain assumption Assumption 4: transportable recency, parallel honest pre/post change for target and control
    Route R2 and all M5 nulls rest on this. It is not fully testable because the target's pre-cutoff cell is contaminated by construction; the authors demonstrate that a mismatched weak control re-inflates the estimate.
  • standard math Assumption 5: RD support regularity
    Technical density and common-support conditions needed for local limits at the cutoff.
  • ad hoc to paper Assumption 6: uniquely extrapolable honest surface
    Only needed to convert the boundary jump into global B by clean-side extrapolation; the paper does not claim this holds for LiveCodeBench.
  • domain assumption Assumption 7: PRC population conditions including crowd-anchored calibration and homogeneous w
    Needed for the detection statistic PRC; the paper shows real LLMs violate calibration and requires fitting a temperature before use.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Temporal Leakage in LLM Backtesting: Measurement, Validation, and Adjusted Scores." pith.science (2026). https://pith.science/paper/QKMP4LJQ

@misc{pith2026260802985,
  author       = {Pith},
  title        = {Pith review of: Temporal Leakage in LLM Backtesting: Measurement, Validation, and Adjusted Scores},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QKMP4LJQ}},
  note         = {Machine review of arXiv:2608.02985}
}
read the original abstract

The standard check for contamination in LLM backtests is simple: compare scores before and after the training cutoff. We show this check is uninformative. Four flagship models fail it on questions they cannot have memorized: every scored question resolved after their cutoffs. The reason is structural. Models legitimately know more about times near their cutoff, so recency mimics leakage, and we prove no passive backtest can separate the two from genuine skill. Measurement, not just detection, requires information from outside the backtest. We supply it in two forms. A known cutoff identifies leakage at the boundary; a matched clean control identifies it globally and yields a leakage-adjusted score. We also derive where leakage hides: it concentrates on outcomes that surprised the crowd and were well covered in training, and partial memorization is disproportionately rewarded. We validate the estimators against ground truth by planting leakage in twin models, where they recover the injected dose and return null on clean questions. Deployed on frontier models, they detect one cutoff-localized signature and, at the audit's power floor, clear five models whose apparent advantages were recency alone. Backtests need not be discarded; they need one defensible reference.

Figures

Figures reproduced from arXiv: 2608.02985 by the authors.

Figure 1
Figure 1. M1: models that cannot have leaked still fail the naive contamination check. Each row is one model (cutoff month in parentheses), scored only on questions resolving at least 30 days after its cutoff, so no outcome can be in its training data. The x-axis is its naive gap ∆ = ¯searly − s¯late, the window split at its median resolution date, where s = (c0 − Y ) 2 − (P − Y ) 2 is the Brier improvement of the model’s for… view at source ↗
Figure 2
Figure 2. Temporal leakage is an indirect information flow through a common cause of the training text and the outcome, not verbatim overlap. The model’s training cutoff T postdates the as-of date t0; a question is leakage-eligible when it resolves inside (t0, T]. The later world state W+ resolves the outcome Y (black, top) and also generates the bridge data Dbridge, text written inside (t0, T]. The deployed model trains on D… view at source ↗
Figure 3
Figure 3. M3: injected leakage produces a strictly monotone dose-response (left) and concentrates on hard questions as the law b0 w(2 − w) predicts (right). Treatment and control twins of one base model train on the same documents; only the treatment copy sees realized outcomes. Both panels plot the twin contrast Bbtwin = ¯ℓctrl − ¯ℓtreat (ℓ = (P − Y ) 2 ; positive = leakage inflation); whiskers: bootstrap 95% CIs; ∗ : CI exc… view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: M5: naive leakage flags dissolve under the matched adjustment. Red: naive gap ∆ = ¯spre −s¯post. Blue: adjusted ∆b DiD = ∆target − ∆control against pre-registered matched control GPT-5. Grey diamond: primary target vs. mismatched GPT-3.5-Turbo. Whiskers: 95% CIs; ∗ : C…
Figure 5
Figure 5. Figure 5: Why leakage inflates a backtest: the decomposition (a) and the double benefit it implies (b). (a) What the clean model does not know about the outcome—the clean gap γ (green)—splits into the leakage signal S (blue), everything the leaked data reveal about Y , and the r…
Figure 6
Figure 6. Figure 6: Choosing a route: report a leakage-adjusted score only when global B is identified; otherwise report the boundary or detection estimand. Forward evaluation (B = 0) is preferred when post-cutoff out￾comes can be awaited. Otherwise, RD identifies the boundary jump when t…
Figure 7
Figure 7. Figure 7: M2: at pretraining scale, measured accuracy inflates strictly monotonically with the known contamination dose. Hubble pretrains a perturbed model with benchmark documents inserted r times and a standard twin trained identically without them. The x-axis is the dose r (l…
Figure 8
Figure 8. Figure 8: M2: the Hubble dose-response is robust across estimators, scales, and tasks. (a) Raw minimal￾pair, placebo-centered, and self-baseline estimates for the 8B/500B pair (whiskers: 95% intervals propagated from published SEMs). (b) The placebo-centered estimate across mode…
Figure 9
Figure 9. Figure 9: M3: the calibrated, twin-differenced PRC fires only where leakage was injected. Bars: the calibrated twin-differenced PRC per injection cell (dose zero and each positive dose), computed from three validated paraphrases per question; positive values signal evidence leak…
Figure 10
Figure 10. Figure 10: E1: the estimators recover known ground truth in simulation. (a) DiD estimates lie on the truth line (bootstrap coverage 94–100% across leakage levels). (b) RD jumps track injected boundary leakage; no-leak placebos stay at zero under smooth recency. (c) PRC is hump-s…
Figure 11
Figure 11. Figure 11: E1 supplementary arms: noise cannot fake inflation, and leakage concentrates where the law predicts. (a) T5: under outcome-orthogonal logit noise with zero true leakage, the DiD estimate stays at or below zero at every noise scale, as Corollary 10 requires (whiskers: …
Figure 12
Figure 12. Figure 12: M4-C: the LiveCodeBench boundary signal survives the robustness battery. (a) Composition controls show no local discontinuity at target cutoffs. (b) Own-cutoff estimates are stable across individual controls. (c) Monthly scans concentrate positive effects at documente…
Figure 13
Figure 13. Figure 13: M4-C identifies the local boundary jump, not a global correction. (a) The strictly clean GPT-4- 0613 control yields a large contrast but fails temporal-profile matching (Assumption 4). (b) Clean-side extrapolated corrections change sign across horizons and polynomial …
Figure 14
Figure 14. Figure 14: Matched real-data PRC: the calibration-stability gate fails, so PRC remains secondary. Treatment–control PRC excess by domain and pooled, under Platt and isotonic calibration. Result: Platt yields positive excesses, but isotonic does not replicate the pooled significa…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 19 canonical work pages

  1. [1]

    Look-ahead-bench: A standardized benchmark of look-ahead bias in point-in-time LLMs for finance.arXiv preprint arXiv:2601.13770,

    Mostapha Benhenda. Look-ahead-bench: A standardized benchmark of look-ahead bias in point-in-time LLMs for finance.arXiv preprint arXiv:2601.13770,

    arXiv

  2. [2]

    Figure 9 shows the calibrated twin-differenced statistic by cell. D.5 M4-F (forecasting-panel matrix and clean-anchor arm) Specificity matrix.Targets are DeepSeek-V3.1 (documented cutoff March 2025), Kimi-K2.6 (April 2025), GPT-5.4 (August 2025), and GPT-5.5 (December 2025); Gemini-3.1-Pro (January

    work page 2025

  3. [4]

    Provenance of the fitted overconfidence temperature.The valueT≈8cited in Remark 2 and used to set the T4 sweep range (Tover∈{2,4,8}) was fit on this forecasting pipeline’s data: temperature scaling pcal =σ(logit(p)/T)withTchosen to minimize log-loss on Qwen3.5’sleakage-freepost-boundary control forecasts (n= 168ex-ante-uncertain factual questions, crowd a...

    work page 2025

  4. [5]

    requires paraphrases that vary framing without revealing the outcome; the set docu- mented here illustrates the design principles and the generation protocol. Designprinciples.Effectiveparaphrasesmust(i)preservetheeventandresolutioncriteria(sameanswer), (ii) maximize variation in the reasoning channel, and (iii) reveal nothing about the outcome. We vary s...

    work page 1969

  5. [6]

    Benchmark inflation: Revealing LLM performance gaps using retro-holdouts.arXiv preprint arXiv:2410.09247,

    Jacob Haimes, Cenny Wenner, Kunvar Thaman, Vassil Tashev, Clement Neo, Esben Kran, and Jason Schreiber. Benchmark inflation: Revealing LLM performance gaps using retro-holdouts.arXiv preprint arXiv:2410.09247,

    arXiv

  6. [7]

    Table 4 is the map: each theoretical claim of Sections 3 to 5 and the experiment whose headline result carries it

    The synthetic validation study E1, the per-experiment robustness analyses, and all supplementary arms are collected in Section E. Table 4 is the map: each theoretical claim of Sections 3 to 5 and the experiment whose headline result carries it. D.1 Scope and coverage Table 5 summarizes, for each experiment, the data source, sample size, models, metric, an...

    work page 2025

  7. [9]

    Summoning the Oracle to Slay It: Mitigating Look-Ahead Bias in Financial Backtesting with Large Language Models

    Weixian Waylon Li, Mengyu Wang, and Tiejun Ma. Summoning the oracle to slay it: Mitigating look-ahead bias in financial backtesting with large language models.arXiv preprint arXiv:2605.24564,

    work page Pith review arXiv

  8. [10]

    ExAnte: A Benchmark for Ex-Ante Inference in Large Language Models

    arXiv:2505.19533. Alejandro Lopez-Lira, Yuehua Tang, and Mingyin Zhu. The memorization problem: Can we trust LLMs’ economic forecasts?arXiv preprint arXiv:2504.14765,

    work page Pith review arXiv

Show all 38 references
  1. [11]

    Daniel Paleka, Shashwat Goel, Jonas Geiping, and Florian Tramèr

    arXiv:2310.17623. Daniel Paleka, Shashwat Goel, Jonas Geiping, and Florian Tramèr. Pitfalls in evaluating language model forecasters. InInternational Conference on Learning Representations (ICLR),

    arXiv

  2. [12]

    14 Andrew J Patton and Allan Timmermann

    arXiv:2506.00723. 14 Andrew J Patton and Allan Timmermann. Forecast rationality tests based on multi-horizon bounds.Journal of Business & Economic Statistics, 30(1):1–17,

    arXiv

  3. [13]

    Martin Riddell, Ansong Ni, and Arman Cohan

    arXiv:2404.00699. Martin Riddell, Ansong Ni, and Arman Cohan. Quantifying contamination in evaluating code generation capabilities of language models. InProceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 14116–14137,

    arXiv

  4. [14]

    Manley Roberts, Himanshu Thakur, Christine Herlihy, Colin White, and Samuel Dooley

    arXiv:2403.04811. Manley Roberts, Himanshu Thakur, Christine Herlihy, Colin White, and Samuel Dooley. To the cutoff... and beyond? a longitudinal perspective on LLM data contamination. InInternational Conference on Learning Representations (ICLR),

    arXiv

  5. [15]

    NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark

    Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. InFindings of the Association for Computational Linguistics: EMNLP 2023, pp....

    2023

  6. [16]

    Quantifying the effect of test set contamination on generative evaluations.arXiv preprint arXiv:2601.04301,

    Rylan Schaeffer, Joshua Kazdan, Baber Abbasi, Ken Ziyu Liu, Brando Miranda, Ahmed Ahmed, Fazl Barez, Abhay Puri, Stella Biderman, Niloofar Mireshghallah, and Sanmi Koyejo. Quantifying the effect of test set contamination on generative evaluations.arXiv preprint arXiv:2601.04301,

  7. [18]

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, et al

    arXiv:2510.19811. Colin White, Samuel Dooley, Manley Roberts, Arka Pal, et al. LiveBench: A challenging, contamination- limited LLM benchmark. InInternational Conference on Learning Representations (ICLR),

  8. [19]

    Jeffrey M Wooldridge

    arXiv:2406.19314. Jeffrey M Wooldridge. Control function methods in applied econometrics.Journal of Human Resources, 50 (2):420–445,

    arXiv

  9. [20]

    Benchmark data contamination of large language models: A survey.arXiv preprint arXiv:2406.04244,

    Cheng Xu, Shuhao Guan, Derek Greene, and M-Tahar Kechadi. Benchmark data contamination of large language models: A survey.arXiv preprint arXiv:2406.04244,

    arXiv

  10. [21]

    DatedGPT: Preventing lookahead bias in large language models with time-aware pretraining.arXiv preprint arXiv:2603.11838,

    Yutong Yan, Raphael Tang, Zhenyu Gao, Wenxi Jiang, and Yao Lu. DatedGPT: Preventing lookahead bias in large language models with time-aware pretraining.arXiv preprint arXiv:2603.11838,

    arXiv

  11. [22]

    Gonzalez, and Ion Stoica

    Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and Ion Stoica. Rethinking benchmark and contamination for language models with rephrased samples.arXiv preprint arXiv:2311.04850,

    arXiv

  12. [23]

    arXiv:2510.27055. Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati, and Sum- mer Yue. A careful examination of large language model performan...

    arXiv

  13. [24]

    Zeyu Zhang, Ryan Chen, and Bradly C

    arXiv:2405.00332. Zeyu Zhang, Ryan Chen, and Bradly C. Stadie. All leaks count, some count more: Interpretable temporal contamination detection and mitigation in LLM backtesting.arXiv preprint arXiv:2602.17234,

    arXiv

  14. [28]

    Experiment Data sourcenModels Metric Inference M1 ForecastBench market panel 332–767post- cutoff ques- tions/model GPT-5, Gemini- 3.1-Pro, Kimi-K2.6, GPT-5.5, MiniMax-M3 anchored Brier re- duction source×month cluster bootstrap M2 Hubble published accuracies 4tasks×6 doses Hub...

    2025

  15. [29]

    collapses every gap to insignificance—GPT- 5.5 falls from+0.061∗ to+0.008, and MiniMax’s gap becomes the largest of the five. The naive statistic therefore measures how much recency contrast an evaluation window spans, not whether the audited model 30 Figure 7:M2: at pretraini...

    2026

  16. [30]

    is a controlled pretraining experiment: models are trained from scratch on a web corpus into which benchmark evaluation documents (MMLU, PIQA, HellaSwag, WinoGrande questions with answers) are deliberately inserted atknownduplication countsr∈ {0,1,4,16,64,256}. For each config...

  17. [32]

    Controls are MiniMax-M3 and Claude-Opus-4.7, whose January 2026 cutoffs leave no leakage discontinuity inside the tested window

    is exploratory because the panel has thin mass before its boundary. Controls are MiniMax-M3 and Claude-Opus-4.7, whose January 2026 cutoffs leave no leakage discontinuity inside the tested window. For each target and eachassumedboundary, thestatisticisthepre/postjump(90-daywin...

    2026

  18. [34]

    Eachproblemcarriesitscontest release date, so a model can only have trained on a problem’s solution if the contest occurred before the model’s training cutoff

    is a coding benchmark that collects problems from competitiveprogrammingcontests(LeetCode, Codeforces, AtCoder). Eachproblemcarriesitscontest release date, so a model can only have trained on a problem’s solution if the contest occurred before the model’s training cutoff. The ...

    2023

  19. [37]

    paraphrases

    Bootstrap CIs: 200 problem-level resamples, 95% percentile interval. We run 300 Monte Carlo repetitions perwand report the mean bias ˆB−B true and the fraction of repetitions in which the CI coversBtrue. T2 (RD recovery and placebo).Each questionihas a resolution-time gapδ i∼U...

    2000

  20. [38]

    Treatment–control PRC excess by domain and pooled, under Platt and isotonic calibration

    43 Figure 14:Matched real-data PRC: the calibration-stability gate fails, so PRC remains secondary. Treatment–control PRC excess by domain and pooled, under Platt and isotonic calibration. Result: Platt yields positive excesses, but isotonic does not replicate the pooled signi...

    2025

  21. [910]

    Because the leakage estimand is ajumprather than a level, protocol level effects cancel unless they vary sharply in time

    that we flag as protocol sensitivity; both are an order of magnitude smaller than the GPT-5.5 leakage signature. Because the leakage estimand is ajumprather than a level, protocol level effects cancel unless they vary sharply in time. The placebo-jump check tests this directly...

    2025

  22. [1978]

    Chronologically consistent large language models

    Songrun He, Linying Lv, Asaf Manela, and Jimmy Wu. Chronologically consistent large language models. arXiv preprint arXiv:2502.21206,

    arXiv

  23. [1987]

    Detecting lookahead bias in LLM forecasts.arXiv preprint arXiv:2512.23847,

    Zhenyu Gao, Wenxi Jiang, and Yutong Yan. Detecting lookahead bias in LLM forecasts.arXiv preprint arXiv:2512.23847,

  24. [2010]

    The sharp-bounds framing of Theorem 2 follows partial identification (Manski, 2003; Imbens & Manski, 2004)

    and difference-in-differences, with a fixed-question intervention in the spirit of control functions and specification tests (Wooldridge, 2015; Wu, 1973; Hausman, 1978). The sharp-bounds framing of Theorem 2 follows partial identification (Manski, 2003; Imbens & Manski, 2004)....

    2015

  25. [2017]

    Singh, Muhammed Yusuf Kocyigit, Andrew Poulton, David Esiobu, Maria Lomeli, Gergely Szilvasy, and Dieuwke Hupkes

    Aaditya K. Singh, Muhammed Yusuf Kocyigit, Andrew Poulton, David Esiobu, Maria Lomeli, Gergely Szilvasy, and Dieuwke Hupkes. Evaluation data contamination in LLMs: how do we measure it and (when) does it matter?arXiv preprint arXiv:2411.03923,

    arXiv

  26. [2022]

    Time machine GPT

    Felix Drinkall, Eghbal Rahimikia, Janet Pierrehumbert, and Stefan Zohren. Time machine GPT. InFindings of the Association for Computational Linguistics: NAACL 2024, pp. 3281–3292,

    2024

  27. [2023]

    Composition controls have cutoffsafterthe latest problem (Gemini-2.5-Pro, DeepSeek-R1, and the published 2025-cutoff pool)

    and Claude-3.5-Sonnet-20240620 (cutoff≈Apr 2024). Composition controls have cutoffsafterthe latest problem (Gemini-2.5-Pro, DeepSeek-R1, and the published 2025-cutoff pool). Because control cutoffs lie outside the evaluation window, they have no leakage discontinuity at any ta...

    2026

  28. [2024]

    Graham Elliott and Allan Timmermann.Economic Forecasting

    arXiv:2402.07841. Graham Elliott and Allan Timmermann.Economic Forecasting. Princeton University Press,

    arXiv

  29. [2025]

    Minhao Jiang, Ken Ziyu Liu, Ming Zhong, Rylan Schaeffer, Siru Ouyang, Jiawei Han, and Sanmi Koyejo

    arXiv:2403.07974. Minhao Jiang, Ken Ziyu Liu, Ming Zhong, Rylan Schaeffer, Siru Ouyang, Jiawei Han, and Sanmi Koyejo. Investigating data contamination for pre-training language models.arXiv preprint arXiv:2401.06059,

    arXiv

  30. [2026]

    Language models are few-shot learners

    Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901,

    1901

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.