DatedGPT: Preventing Lookahead Bias in Large Language Models with Time-Aware Pretraining

Raphael Tang; Wenxi Jiang; Yao Lu; Yutong Yan; Zhenyu Gao

REVIEW 4 major objections 5 minor 2 cited by

DatedGPT: Preventing Lookahead Bias in Large Language Models with Time-Aware Pretraining

T0 review · 4 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Twelve 1.3B-parameter language models, each pretrained from scratch on data with a strict annual cutoff between 2013 and 2024, keep their knowledge bounded to that cutoff year — a claim the paper supports with perplexity-based probing and o

desk verdict Genuinely useful model series, but the abstract advertises a finance result that is nowhere in the manuscript; the instruct models' cutoff is also unverified. read the letter →

arxiv 2603.11838 v2 pith:6L4BRBYY submitted 2026-03-12 cs.CL q-fin.GN

Yutong Yan , Raphael Tang , Zhenyu Gao , Wenxi Jiang , Yao Lu This is my paper · ORCID

classification cs.CLq-fin.GN

keywords lookaheadbiastemporaldatacutofftime-awarepretrainingperplexityprobingknowledgememorisationinstructiontuningfinancialforecastingevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper attempts to show that lookahead bias — a model having already seen the true future outcome during training — can be engineered away by training a series of language models on web data partitioned by year. The authors build twelve 1.3B-parameter models, each trained from scratch on roughly 100 billion tokens with a strict cutoff between 2013 and 2024, and add instruction-tuned variants whose training data is filtered to the same cutoff. Perplexity-based probing on news headlines shows a clear reversal: after a model's cutoff year, its perplexity on later news rises steadily, suggesting it has not absorbed those outcomes. If this holds, the series provides a rare testbed where predictions about future events cannot be attributed to memorisation, which matters for financial forecasting and for evaluating how LLMs actually reason over time. The paper also advertises, in its abstract, a stock-return prediction result with an annualised Sharpe ratio of 3.20, a detail not present in the submitted body.

What carries the argument

The load-bearing mechanism is the strict temporal cutoff enforced twice: at pretraining, by filtering FineWeb-Edu (a quality-filtered web corpus derived from Common Crawl, where each document carries a crawl timestamp) so each model sees only documents crawled on or before its cutoff year; and at instruction-tuning, where a teacher LLM classifies general instruction examples as time-sensitive or not and generates finance-specific instruction pairs anchored to publication dates. The verification mechanism is perplexity-based probing: each model's perplexity on quarterly news headlines rises after its cutoff, which the authors interpret as evidence that the model has not memorised post-cutoff

What would settle it

Take a DatedGPT-Instruct model (say, 2020) and ask it for a fact that became widely known after 2020 (e.g., that OpenAI released ChatGPT in late 2022) using a paraphrase unlikely to appear in the pretraining corpus; if it reliably produces the post-cutoff fact or shows low perplexity on post-cutoff news, the cutoff guarantee fails at the instruction-tuning stage. Second, replicate the stock-return forecast experiment described in the abstract: if the advertised Sharpe ratio of 3.20 is not reproduced with the released checkpoints, the paper's headline financial claim is unsupported.

Watch

Extended reading notes

Core claim

The central claim is that a language model can be trained from scratch on web data partitioned by crawl year so that its knowledge, measured by perplexity on quarterly news headlines, is effectively bounded by its cutoff date: the model shows low perplexity on news from before its cutoff and a clear rise on news from after, indicating it was not exposed to those post-cutoff outcomes. This temporal partitioning is applied at both pretraining and instruction-tuning stages, producing a family of base and instruction-tuned models whose knowledge aligns with their designated cutoff years while remaining competitive with similarly sized open models on standard benchmarks. The authors present this

Load-bearing premise

The entire cutoff guarantee rests on the assumption that the instruction-tuning data, generated and filtered by a teacher model (Llama-3.3-70B-Instruct) that was itself trained on data after every cutoff year, does not secretly encode post-cutoff facts; if it does, the strict-cutoff guarantee for the instruction-tuned models — the models used in the advertised financial claim — is broken.

Editorial extensions

If this is right

Researchers can compare predictions from models with different cutoff years against actual subsequent outcomes, so any predictive edge cannot be attributed to leakage.
The series enables econometric tests of lookahead bias with a long out-of-sample period, avoiding the short out-of-sample windows that limit tests on existing LLMs.
The perplexity-reversal method can be reused as a cheap audit for temporal contamination in other models.
The models stay competitive on standard benchmarks (average 40.1–42.7 on the reported suite), so temporal control does not come at the cost of general ability.
The web interface lets users directly observe cutoff-dependent behaviour, such as the DatedGPT-2020 model not knowing about ChatGPT.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the cutoff guarantee holds for the base models, the series turns the memorisation-versus-reasoning question into a controlled experiment: any correct post-cutoff answer by a model must be derived from pre-cutoff context, not memorised outcome.
The abstract advertises a stock-return experiment with an annualised Sharpe ratio of 3.20 and a lookahead premium of 26.4 basis points, but the submitted body does not report this experiment; until the released checkpoints are evaluated, that figure should be treated as unverified.
The guarantee for the instruction-tuned models is only as strong as the teacher model's own cutoff discipline — a teacher trained on post-cutoff data could leak future facts into the instruction data without leaving a trace in the perplexity probing, which the paper reports only for the base models.
Extending the series backwards (e.g., models cut off before 2008) would let researchers test well-documented events like the 2008 crash under the same controlled conditions.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my take. The paper has a real artifact: twelve 1.3B-parameter models trained from scratch on about 100B tokens each, with annual cutoffs from 2013 to 2024, plus instruction-tuned variants. That's a useful resource for anyone working on temporal evaluation, and it goes beyond StoriesLM and ChronoGPT in scale and coverage. The benchmark table shows the models are competitive at their size, and the perplexity probe on the BASE series shows a clear trend reversal around the cutoff year, with confidence intervals. The authors also deserve credit for flagging the crawl-timestamp limitation in Section 2.1.

But the abstract makes claims the submission doesn't contain. The stock-return experiment—61,000 headlines, Sharpe 3.20, lookahead premium of 26.4 b.p.—does not exist in the text. No methodology, no table, no result. That is the main advertised contribution. Without it, the title's promise of preventing lookahead bias in financial forecasting rests on a perplexity probe of base models only.

The second issue is the teacher model. Section 2.2 uses Llama-3.3-70B-Instruct to filter general data and generate finance-specific instruction pairs. That teacher has post-cutoff knowledge. The paper doesn't audit the teacher's outputs, and the perplexity probe in Section 4.3 only evaluates DATEDGPT-BASE, not the INSTRUCT models used in the claimed finance application. So the instruction-tuned models could have leakage re-introduced, and the paper doesn't rule it out.

The conclusion overclaims: it says the models 'eliminate' lookahead bias in time-sensitive domains, but the evidence only supports that for base models, and only at the level of perplexity.

The core artifact is genuinely useful, and the writing is clear. But the missing experiment is load-bearing. This needs a major revision: either add the finance experiment or cut the abstract claims. Given the resource itself, I'd send it to peer review rather than desk-reject, but the referee should demand the missing result and an audit or disclaimers on the instruct models. If the authors can't supply the experiment, the paper should be scaled back to the model series and perplexity analysis.

Referee Report

4 major / 5 minor

Summary. The paper introduces DatedGPT, a family of twelve 1.3B-parameter language models pretrained from scratch on approximately 100B tokens per year with strict annual cutoffs from 2013 to 2024, together with DatedInstruct, an instruction-following dataset filtered and generated under the same temporal constraints. The authors report competitive zero-shot performance on standard benchmarks, claim that a perplexity-based probe confirms each model's knowledge is bounded by its cutoff year, and describe an interactive web demo. The abstract additionally claims a stock-return prediction result with an annualised Sharpe ratio of 3.20 and a lookahead premium of 26.4 basis points, but no such experiment appears in the body of the paper.

Significance. If the temporal guarantees are verified, the DatedGPT series would be a valuable community resource for studying lookahead bias, temporal reasoning, and contamination in LLM evaluation, particularly for finance. The scale of the resource—twelve 1.3B models, each trained on about 100B tokens, plus instruction-tuned variants—is a meaningful step beyond prior time-indexed model releases such as StoriesLM and ChronoGPT. The data curation pipeline, including LLM-based filtering of general instruction data and teacher-generated finance instructions, is a useful recipe. However, the evidence presented for the central claim is incomplete: the perplexity probe is run only on base models, no leakage-prone control is included, and the headline financial result is absent from the manuscript.

major comments (4)

[Abstract vs. Sections 2–4] The abstract states that 'On stock return prediction over 61,000 firm-day news headlines, DatedGPT-instruct achieves an annualised Sharpe ratio of 3.20' and that 'lookahead-biased models ... add a lookahead premium of 26.4 b.p. per standard deviation, significant at the 1% level.' These numbers appear nowhere in Sections 2–7 or the appendices. Section 2.2 describes the construction of stock-return and capex instruction datasets, but no downstream financial evaluation is reported. Since these numbers are the only direct quantitative support for the claim that the series 'enables direct analysis of lookahead bias in financial forecasting', they must either be added with full experimental details (dataset, evaluation protocol, statistical tests, leakage controls) or the abstract and conclusions must be revised to remove them.
[§4.3] The perplexity-based probe is run only on the DATEDGPT-BASE series: 'we evaluate the DATEDGPT-BASE model series using news headlines.' Yet the abstract and conclusion state that 'each model's knowledge is effectively bounded by its cutoff year,' and the released/demo models are the DATEDGPT-INSTRUCT variants. Instruction tuning can introduce distributional changes and can re-inject post-cutoff information through the teacher model, so the bounded-knowledge claim cannot be transferred from base to instruct models without additional evidence. Please probe the INSTRUCT models or explicitly restrict the claim to the base series.
[§2.2] Llama-3.3-70B-Instruct is used both as a classifier to filter time-sensitive general instruction data and as a teacher to generate the finance-specific instruction pairs (news-to-return and earnings-call-to-capex). This teacher has a training cutoff well after every DatedGPT cutoff, so its outputs and classifications can encode facts that were not observable at the model's cutoff year. The paper does not audit the teacher's outputs nor test the resulting INSTRUCT models for post-cutoff leakage. Without such an audit—for example, human or model-based review of sampled generated instructions, or a time-sensitive QA probe on the INSTRUCT models—the 'strict cutoff' guarantee for the instruction-tuned models is unsupported.
[§4.3, Figure 2] The reversal in scaled perplexity at the cutoff is consistent with the model having been trained only on pre-cutoff text, but it is also consistent with a model whose training distribution is simply older and does not by itself rule out memorisation of specific post-cutoff outcomes. A leakage-positive control is needed to make the bounding claim informative. For example, evaluate DATEDGPT-BASE-2024 (or an undated model such as GPT2-XL) on the same 2013–2024 headline set and show the absence of a reversal for a model with post-cutoff exposure. In addition, 'Scaled PPLQ' and the construction of the 95% confidence interval are not defined; please clarify.

minor comments (5)

[Table 2] The header 'A vg.' should read 'Avg.'.
[Figure 2] The x-axis begins at 2010Q1, but the evaluation data are described as 2013–2024 news headlines. Please clarify the time range and the reason for the earlier axis start.
[Appendix C] The text says 'we provide more visualisations in the Appendix,' but only one additional figure (Figure 4, for 2017) is shown. Either include all visualisations or revise the statement.
[§5] The claim that DATEDGPT-2020 'responds with outdated information' would be more convincing with the actual model response reported or shown in the demo screenshot.
[Section 2.2 / References] The sentence 'These datasets include (Wei et al.; Wang et al., 2023; Taori et al., 2023; Zheng et al., 2023; Xu et al.)' contains incomplete citations; Wei et al. and Xu et al. appear in the reference list, but the in-text citations should include years and be formatted consistently.

Circularity Check

0 steps flagged · score 2.0 of 10

No construction-level circularity; knowledge-cutoff claim is checked against external headlines. Minor self-citation and teacher-leakage risk are correctness concerns, not circularity.

full rationale

The central derivation is not circular by construction. DatedGPT's 'strict cutoff' is implemented by filtering FineWeb-Edu on crawl year (Sec 2.1), which is a data-construction decision rather than a parameter fitted to make the conclusion true. The memorization probe (Sec 4.3) applies a pre-existing perplexity method (Cheng et al. 2024) to an external set of Bloomberg news headlines with publication timestamps; DatedGPT-BASE models were not trained on those headlines, and the probe could in principle have failed to show a post-cutoff perplexity rise. The probe is therefore genuine external evidence, not a renaming of the training filter. The main weaknesses are non-circular validity risks: (i) Sec 2.2 uses Llama-3.3-70B-Instruct as both time-sensitivity filter and finance teacher without auditing for post-cutoff knowledge, so the instruction-tuned models' temporal purity is unverified; (ii) the headline Sharpe-ratio and 26.4 b.p. lookahead-premium results appear only in the abstract and cannot be checked; (iii) crawl timestamps are acknowledged not to equal creation dates. There is one minor self-citation (Gao et al. 2025, by three of this paper's authors) as the source of the news-headline evaluation set and of an econometric lookahead-bias test; because that dataset is external, timestamped news and the probe is not fitted to the target claim, the citation is not load-bearing. These factors lower confidence but do not make the derivation equivalent to its inputs, so the circularity score is low.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claims rest on two curation mechanisms—crawl-date filtering and teacher-model filtering—and on the interpretability of the perplexity probe. No numeric free parameters are disclosed as fitted to data; the financial numbers in the abstract are reported without derivation.

assumptions (3)

domain assumption Crawl timestamp of a document bounds the time at which its content was available (a page crawled in year Y contains no events after Y).
Section 2.1 uses FineWeb-Edu crawl timestamps as strict annual cutoffs; the authors note that crawl time does not equal authorship time, but they assert this still prevents future exposure.
ad hoc to paper Llama-3.3-70B-Instruct correctly filters time-sensitive general instruction data and does not introduce post-cutoff knowledge into the financial instruction pairs it generates.
Section 2.2 relies on this model both as a classifier and as a teacher for domain data; no audit or false-negative analysis is provided.
domain assumption Lower perplexity on pre-cutoff news and higher perplexity on post-cutoff news is a valid indicator of temporal memorization and therefore of absence of lookahead bias.
Section 4.3 adopts the Cheng et al. (2024) probing method; the paper treats the observed perplexity reversal as confirmation that knowledge is bounded.

how reviews work

0 comments

Cite this review

Pith. "Pith review of DatedGPT: Preventing Lookahead Bias in Large Language Models with Time-Aware Pretraining." pith.science (2026). https://pith.science/paper/6L4BRBYY

@misc{pith2026260311838,
  author       = {Pith},
  title        = {Pith review of: DatedGPT: Preventing Lookahead Bias in Large Language Models with Time-Aware Pretraining},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6L4BRBYY}},
  note         = {Machine review of arXiv:2603.11838}
}

abstract

Large language models pretrained on internet-scale data risk lookahead bias in forecasting tasks, as they may have already seen the true outcome during training. To address this, we present DatedGPT, a family of twelve 1.3B-parameter language models trained from scratch on approximately 100 billion tokens each with strict annual data cutoffs spanning 2013 to 2024, together with DatedInstruct, an instruction dataset grounded in each year's documents to prevent leakage during post-training. The models are competitive with open models of similar scale, and perplexity-based probing confirms that each model's knowledge is bounded by its cutoff year. On stock return prediction over 61,000 firm-day news headlines, DatedGPT-instruct achieves an annualised Sharpe ratio of $3.20$ under the lookahead-bias-free setup. Lookahead-biased models, whose training data covers the outcome period, add a lookahead premium of $26.4$ b.p. per standard deviation, significant at the 1% level. The series thus enables direct analysis of lookahead bias in financial forecasting. We provide an interactive web demo that allows users to query and compare responses from models across different cutoff years, available at www.datedgpt.com.

Figures

Figures reproduced from arXiv: 2603.11838 by the authors.

**Figure 1.** Web interface for DATEDGPT. When queried about OpenAI’s chatbot, the DATEDGPT-2020 model, trained exclusively on data available before 2020, is unaware of ChatGPT. referred to as lookahead bias, poses a fundamental challenge for the reliable evaluation of LLMs in temporal prediction settings (Sarkar and Vafa, 2024; Gao et al., 2025). In economic and financial forecasts, ensuring uncontaminated predictions is a cor… view at source ↗

**Figure 2.** Average relative perplexity of DATEDGPTBASE-2020 evaluated on quarterly public company news headlines from 2013 to 2024. 5 Web Interface design We have developed a web interface for the DATEDGPT model series that enables users to interact with and explore model outputs grounded in different time periods. The interface is built as a Reactbased single-page application, designed to make temporal language model expl… view at source ↗

**Figure 3.** Training loss curves for 2013 and 2024 DATEDGPT-BASE models. All models exhibit smooth convergence without loss spikes or instability. 10 Q1 11 Q1 12 Q1 13 Q1 14 Q1 15 Q1 16 Q1 17 Q1 18 Q1 19 Q1 20 Q1 21 Q1 22 Q1 23 Q1 24 Q1 25 Q1 0.0 0.2 0.4 0.6 0.8 1.0 S c ale d P P L Q Mean Scaled PPLQ 95% Confidence Interval Model Cutoff (2017-12-31) [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Relative perplexity of DATEDGPT-BASE2017 evaluated on quarterly public company news headlines from 2013 to 2024. 9 [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

HindsightBench: A Black-Box Behavioral Audit Protocol for Parametric Hindsight in Time-Indexed LLM Decision Tasks
cs.LG 2026-07 conditional novelty 6.0 of 10

An integrated black-box audit protocol profiles parametric hindsight in LLMs and shows the date-trigger reflex tracks training generation, not scale, while effective knowledge cutoffs span 22 months.
Scaling Point-in-Time Language Models
cs.CL 2026-04 conditional novelty 5.5 of 10

Scaling point-in-time LLMs to 4B parameters and 1T temporally filtered tokens narrows the gap to unrestricted models to about 8–11 average points and yields positive out-of-sample Sharpe ratios from news embeddings.

Reference graph

Works this paper leans on

5 extracted references · 5 linked inside Pith · cited by 2 Pith papers

[5]

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou

Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information pro- cessing systems, 36:46595–46623. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models.Preprint, arXiv:2311.07911. 8 A Dataset Details Dataset Descr...

arXiv 2023
[2020]

Nathan Hu, Eric Mitchell, Christopher D Manning, and Chelsea Finn

Measuring massive multitask language under- standing.arXiv preprint arXiv:2009.03300. Nathan Hu, Eric Mitchell, Christopher D Manning, and Chelsea Finn. 2023. Meta-learning online adaptation of language models. InProceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 4418–4432. Manish Jha, Jialin Qian, Michael Webe...

arXiv 2009
[2023]

InInternational Conference on Machine Learning, pages 2397–2430

Pythia: A suite for analyzing large language models across training and scaling. InInternational Conference on Machine Learning, pages 2397–2430. PMLR. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, and 1 others. 2020. Piqa: Reasoning about physical commonsense in natural language. InProceedings of the AAAI conference on artificial intelligence, v...

arXiv 2020
[2024]

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, and 1 others

Dated data: Tracing knowledge cutoffs in large language models.First Conference on Lan- guage Modeling. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, and 1 others. 2023. Palm: Scaling language modeling with pathways.Journal of ma- chine lear...

arXiv 2023
[2025]

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt

Chronologically consistent large language models.arXiv preprint arXiv:2502.21206. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt

arXiv

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.

[1] [5]

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou

Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information pro- cessing systems, 36:46595–46623. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models.Preprint, arXiv:2311.07911. 8 A Dataset Details Dataset Descr...

arXiv 2023

[2] [2020]

Nathan Hu, Eric Mitchell, Christopher D Manning, and Chelsea Finn

Measuring massive multitask language under- standing.arXiv preprint arXiv:2009.03300. Nathan Hu, Eric Mitchell, Christopher D Manning, and Chelsea Finn. 2023. Meta-learning online adaptation of language models. InProceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 4418–4432. Manish Jha, Jialin Qian, Michael Webe...

arXiv 2009

[3] [2023]

InInternational Conference on Machine Learning, pages 2397–2430

Pythia: A suite for analyzing large language models across training and scaling. InInternational Conference on Machine Learning, pages 2397–2430. PMLR. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, and 1 others. 2020. Piqa: Reasoning about physical commonsense in natural language. InProceedings of the AAAI conference on artificial intelligence, v...

arXiv 2020

[4] [2024]

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, and 1 others

Dated data: Tracing knowledge cutoffs in large language models.First Conference on Lan- guage Modeling. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, and 1 others. 2023. Palm: Scaling language modeling with pathways.Journal of ma- chine lear...

arXiv 2023

[5] [2025]

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt

Chronologically consistent large language models.arXiv preprint arXiv:2502.21206. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt

arXiv