Pith. sign in

REVIEW 4 major objections 5 minor 43 references

A 4B-parameter language model trained only on text published before each date scores 58.1% on common-sense benchmarks and produces tradable news signals, showing that eliminating lookahead bias is largely a matter of scale.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 15:32 UTC pith:EMWIP24X

load-bearing objection A serious point-in-time LLM scaling study that deserves review, but the 'gap is mostly scale' claim needs a decontamination check and the financial numbers need error bars before the strong conclusion is accepted. the 4 major comments →

arxiv 2607.11889 v2 pith:EMWIP24X submitted 2026-04-24 cs.CL cs.AI

Scaling Point-in-Time Language Models

Bryan Kelly , Semyon Malamud , Johannes Schwab , Teng Andrea Xu This is my paper
classification cs.CL cs.AI
keywords point-in-time language modelslookahead biaschronologically filtered pretrainingtemporal validitycommon-sense reasoningnews embeddingsSharpe ratioout-of-sample backtest
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that the performance gap between point-in-time language models (trained only on text available as of each date) and unrestricted models is mostly a scale deficit, not an inherent limitation of the temporal constraint. It trains decoder-only transformers up to 4B parameters on 1 trillion timestamp-ordered tokens, producing monthly checkpoints from 2013 to 2024. The largest model, PIT-4B, scores 58.1% average zero-shot accuracy across seven common-sense benchmarks, approaching Gemma-3-4B (69.2%) and LLaMA-7B (66.1%). In a strict out-of-sample economic test, its news embeddings generate Sharpe ratios of 1.02 (base) and 1.53 (instruction-tuned), comparable to or better than full-sample models that violate temporal validity. The paper's conclusion matters because backtests and causal inference in finance and social science are only valid if the model has not seen the future.

Core claim

On its own terms, the paper claims that point-in-time language models—trained exclusively on text available up to each calendar date—do not have an inherent quality ceiling. Scaling to 4 billion parameters and 1 trillion timestamp-ordered tokens produces PIT-4B, which attains 58.1% average zero-shot accuracy across seven common-sense reasoning benchmarks, within a few points of Gemma-3-4B (69.2%) and LLaMA-7B (66.1%). In the economic evaluation, embeddings from the December checkpoints, applied to the following year's news, deliver out-of-sample Sharpe ratios of 1.02 (base) and 1.53 (instruction-tuned) over 2014–2020, close to or above the full-sample models that violate temporal validity. T

What carries the argument

The load-bearing mechanism is chronological filtering of the pretraining corpus: web documents are ordered by their publication timestamps, and training runs as one temporally ordered stream of tokens, so that the monthly checkpoint at date t has by construction seen only text dated on or before t. This construction converts the promise of 'no lookahead' from an assumption into a property of the training pipeline. The economic evaluation is carried by a second mechanism: news article embeddings (last-token hidden states) are averaged to stock-month level, residualized against firm characteristics, mapped into base portfolios, and combined via penalized Maximum Sharpe Ratio Regression to prod

Load-bearing premise

The entire no-lookahead guarantee rests on the web corpus timestamps accurately reflecting when each document first became publicly available, and on the pretraining corpus containing none of the benchmark questions used for evaluation; if either assumption fails, the reported benchmark scores and financial backtests cannot be interpreted as temporally valid.

What would settle it

Ask the December 2014 checkpoint to identify the winner of the 2016 U.S. presidential election; if it reliably produces the correct name rather than guesses, the no-lookahead guarantee has failed. A complementary calculation: measure token overlap between the 1T-token pretraining corpus and BoolQ, HellaSwag, ARC, and OBQA; any substantial overlap would invalidate the zero-shot benchmark comparisons.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Researchers can build backtests and causal studies on LLM-generated text signals without a lookahead caveat: the point-in-time portfolio is tradable by construction.
  • Scaling point-in-time training is a viable route to close the residual gap: the paper's own comparison of PIT-1.5B (53.8) and PIT-4B (58.1) attributes most of the improvement to scale.
  • Instruction tuning on temporally filtered data preserves chronological consistency while improving instruction following, making the models usable for historical simulations and policy counterfactuals.
  • The release of the full pipeline turns point-in-time pretraining into a reproducible recipe, so future work can extend the same methodology to other domains and languages.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the scaling story is causal, an 8B-parameter point-in-time model trained on 2T+ tokens should push the common-sense average above 60%; a plateau would reveal a temporal-information ceiling, not just a scale effect.
  • The point-in-time model beating its full-sample counterpart on Sharpe ratio (1.53 vs 1.18) suggests chronological restriction may act as an implicit regularizer; comparing the geometry of PIT and full-sample embeddings would show whether the PIT signal is cleaner or merely different.
  • Because decontamination is reported only for the instruction-tuning data, a token-level contamination audit of the pretraining corpus against the evaluation benchmarks is the nearest testable check, and the released pipeline makes that audit runnable.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper trains decoder-only point-in-time language models (1.5B and 4B parameters) on chronologically ordered FineWeb data with monthly checkpoints from 2013 to 2024. It reports zero-shot common-sense reasoning scores (Table 2), IFEval results after LoRA instruction tuning, and an asset-pricing application in which news embeddings from the PIT models are aggregated into portfolios via penalized maximum-Sharpe-ratio regression. The central claim is that chronological consistency costs little: PIT-4B approaches Gemma-3-4B and LLaMA-7B on average accuracy, and its embeddings achieve out-of-sample Sharpe ratios comparable to full-sample checkpoints, implying that the gap is mostly scale.

Significance. If valid, the paper would be an important scaling demonstration for temporally valid LLMs and a useful public resource. Strengths include the release of the full pipeline and model checkpoints, the use of programmatic IFEval rather than LLM-as-a-judge, the adoption of a standardized evaluation harness, and an explicitly out-of-sample design in the economic application. However, the central comparisons rest on unverified corpus decontamination and timestamp provenance, and the economic evidence lacks uncertainty quantification. The scale hypothesis is plausible but not yet established with the current evidence.

major comments (4)
  1. [§3 Pre-Training; §4.1 Table 2] The pretraining corpus is not decontaminated against the evaluation benchmarks. Table 1 flags 'heval decontaminated' only for SFT rows, and Appendix B describes temporal filtering for fine-tuning data, not benchmark decontamination of the 1T-token FineWeb pretraining set. All seven common-sense benchmarks predate the December 2024 cutoff, so their test items could legitimately appear in FineWeb under the chronological criterion. If they do, PIT-4B's 58.1% average and the conclusion that the gap to unrestricted models is mostly scale are inflated relative to Gemma-3-4B and LLaMA-7B, which are typically trained with decontamination. Please report n-gram overlap statistics between pretraining slices and benchmark test sets, and either retrain after decontamination or re-evaluate on contamination-free subsets.
  2. [§3 Pre-Training; §5] The 'eliminate leakage by construction' claim and the entire economic backtest assume FineWeb timestamps equal publication or availability dates. The paper gives no provenance or validation for these timestamps. If a timestamp is assigned at crawl ingestion or first-seen date rather than publication date, the monthly checkpoint may contain post-cutoff text, invalidating the point-in-time guarantee. Please describe how the timestamps are generated and provide a validation, such as timestamp distributions for documents with known dates or a manual audit of dated events.
  3. [§5, Figure 3] The Sharpe ratios are point estimates with no standard errors, confidence intervals, or t-statistics, over a short sample (2014–2020) from a single news dataset, and there is no transaction-cost analysis. With roughly six years of monthly data, a Sharpe of 1.02 or 1.53 is not interpretable as 'robust' without a null distribution. Please report bootstrapped or asymptotic uncertainties, turnover- and cost-adjusted returns, and sensitivity to the shrinkage grid and equal-weighted aggregation; ideally include a second dataset or state the single-dataset limitation more prominently.
  4. [§5 Full-sample benchmark models] The 'full-sample counterparts' are not fully unrestricted foundation models but the final checkpoints of the same PIT-4B models on the same FineWeb stream. This is a useful ablation for lookahead cost, but it does not address the more relevant comparison to a large model trained on the full internet, such as the embedding models used in Chen et al. (2022) or Didisheim et al. (2026). Please clarify this wording and, if possible, include such a comparison; otherwise the economic claim that PIT models 'perform close to full-sample counterparts' is narrower than the abstract and conclusion suggest.
minor comments (5)
  1. [§3] Typo: 'differnt' should be 'different'. Also, Figure 1 uses both 'DateGPT' and 'DatedGPT'; please standardize.
  2. [§4.1 Table 2, footnote 2] DatedGPT results are copied from the original paper because the weights could not be located. Please state explicitly whether evaluation settings match the original paper; otherwise the comparison in Table 2 is not strictly apples-to-apples.
  3. [§4.2 Figure 2] The baseline 'Qwen1.5-1.8B' appears to be a base model rather than an instruction-tuned chat variant. If so, the IFEval comparison is not a fair instruction-following baseline; consider Qwen1.5-1.8B-Chat or an equivalent instruction-tuned model.
  4. [§5] Please clarify whether the Sharpe ratio uses returns in excess of the risk-free rate, and cite Sharpe (1966 or 1994) rather than 'Sharpe et al. (1998)'.
  5. [Table 2] The DatedGPT row has only three entries; please mark missing values explicitly rather than leaving blank cells.

Circularity Check

1 steps flagged

Secondary IFEval result is trained on IFEval-like data, making that 'prediction' circular; central zero-shot and economic claims retain independent content.

specific steps
  1. fitted input called prediction [Section 3 (Data, Table 1) and Section 4.2 (Instruction Following)]
    "We cap the Argilla/IFEval-like (Xu et al., 2024) data at 270,000 examples to maintain an approximate balance whereby half of the dataset comprises coding and mathematical problems, while the remaining half focuses on rigorous adherence to user instructions. ... Figure 2 reports IFEval instruction-following accuracy for PIT-4B-FT, ChronoGPT-Instruct, and Qwen1.5-1.8B on four metrics: prompt-strict, prompt-loose, instruction-strict, and instruction-loose."

    The SFT stage includes the dataset 'argilla/ifeval-like-data' (Table 1; 456,304 examples, 386,584 retained), which by construction is designed to mimic IFEval. The paper then evaluates instruction-following with IFEval and reports PIT-4B-FT's accuracy as an independent result. Because the model was fine-tuned on examples drawn from the same distribution as the evaluation benchmark, the measured IFEval scores are a function of the training input, not an out-of-distribution prediction. The table does not mark this SFT set as decontaminated, so the reduction is visible in the paper's own data. This inflates the reported 31.3% vs 25.1% / 23.5% comparison.

full rationale

The central common-sense benchmarks (BoolQ, PIQA, HellaSwag, WinoGrande, ARC, OBQA) are external, fixed datasets, so the headline zero-shot comparison is not a circular fit; the economic Sharpe ratios are computed on real out-of-sample returns using standard portfolio methods, and the self-citations (Kelly-Xiu MSRR, JKP characteristics, Didisheim et al. pre-processing) are methodological tools rather than definitions of the result. The main circularity is confined to the secondary IFEval evaluation: the instruction-tuned model was trained on 'argilla/ifeval-like-data' and then evaluated on IFEval, making that particular 'prediction' reduce to its training distribution by construction. The potential absence of pretraining decontamination for the seven common-sense benchmarks is a correctness/contamination risk, not a logical circularity, since the paper does not fit those benchmark labels into the model. Overall, the core point-in-time scaling and economic claims retain independent content, so the score is moderate rather than high.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The central empirical claims rest on the fidelity of temporal filtering and on benchmark cleanliness. The ledger lists the assumptions that, if false, would invalidate the point-in-time interpretation. No new theoretical entities are introduced.

free parameters (5)
  • PIT-4B pretraining scale (4B params, 1T tokens)
    Chosen by the authors; no compute-matched or token-matched ablations are run, so the "scale" explanation is not separated from changes in data quantity and architecture.
  • PIT-1.5B pretraining budget (170B tokens)
    Chosen as the smaller comparison point; PIT-4B differs in both parameters and tokens, confounding the scaling attribution.
  • MSRR shrinkage grid Z and equal-weight aggregation = Z = {1e-6, ..., 1e-1, 1, 2, 5, 10, 1e2}
    Hand-chosen grid; the final portfolio equally weights all z values, so the reported Sharpe ratios depend on this selection and are not accompanied by uncertainty.
  • LoRA rank r=16 = 16
    Fixed across all fine-tuning experiments; no rank sensitivity analysis is reported.
  • gpt-5-nano temporal filter prompt and decision rule
    Appendix B uses an LLM classifier to label SFT examples as timeless; the prompt and threshold are hand-built and unvalidated against a gold set.
axioms (4)
  • domain assumption FineWeb publication timestamps reflect when text became publicly available
    Used throughout §3 and §5; if timestamps are crawl dates or noisy, monthly checkpoints may be misaligned with calendar time, weakening "point-in-time by construction".
  • domain assumption The 1T-token pretraining corpus contains no evaluation benchmark contamination
    No decontamination of FineWeb is reported; benchmark items are known to circulate on the web, so zero-shot accuracy in Table 2 could be inflated.
  • domain assumption gpt-5-nano temporal filtering is sufficient to remove time-sensitive SFT examples
    Appendix B; any filter errors leak future references into instruction-tuned models used in the economic evaluation.
  • domain assumption Scaling laws estimated on unrestricted corpora transfer to chronologically ordered training
    Cites Kaplan/Hoffmann/Henighan to conclude the remaining gap is a scale deficit, but no point-in-time scaling-curve fit is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 16650 in / 19408 out tokens · 197181 ms · 2026-08-02T15:32:48.564192+00:00 · methodology

0 comments
read the original abstract

Large language models trained on unrestricted internet corpora inevitably embed information from the future, introducing lookahead bias that compromises the validity of backtests and causal inference in finance and the social sciences. Point-in-time language models--trained exclusively on text available up to each calendar date--eliminate this leakage by construction, but existing efforts typically produce models that lag substantially behind their unconstrained counterparts. We show that this performance gap can be substantially narrowed through scale. Training decoder-only transformers with up to 4 billion parameters on 1 trillion chronologically filtered tokens from FineWeb, we construct a sequence of monthly model checkpoints spanning 2013-2024. Across a range of common-sense reasoning and language understanding benchmarks, our models approach the performance of leading open-weight models of comparable size (e.g., Gemma-3-4B and LLaMA-7B) trained on temporally unrestricted data, although a performance gap remains on several tasks. Instruction fine-tuning via LoRA further improves downstream usability. We release the complete pipeline--including dataset construction, training infrastructure, and evaluation code--to enable reproducible point-in-time language modeling and to support research applications that require strict temporal validity.

Figures

Figures reproduced from arXiv: 2607.11889 by Bryan Kelly, Johannes Schwab, Semyon Malamud, Teng Andrea Xu.

Figure 1
Figure 1. Figure 1: HellaSwag accuracy (%) over time for our PIT-1.5B (170B tokens) and PIT-4B (1T [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: IFEval instruction-following accuracy (%) for our fine-tuned PIT-4B-FT [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Out-of-sample annualized Sharpe ratio for each model. [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Prompt used for temporal leakage classification. [PITH_FULL_IMAGE:figures/full_fig_p023_4.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 39 linked inside Pith

  1. [1]

    Scalable second order optimization for deep learning.arXiv:2002.09018,

    Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable second order optimization for deep learning.arXiv:2002.09018,

    Pith/arXiv arXiv 2002

  2. [3]

    Deepseek llm: Scaling open-source language models with longtermism.arXiv:2401.02954,

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open-source language models with longtermism.arXiv:2401.02954,

    Pith/arXiv arXiv

  3. [4]

    Lora learns less and forgets less.arXiv:2405.09673,

    Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, et al. Lora learns less and forgets less.arXiv:2405.09673,

    Pith/arXiv arXiv

  4. [10]

    Gptscore: Evaluate as you desire

    Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. arXiv:2302.04166,

    Pith/arXiv arXiv

  5. [11]

    Representation degeneration problem in training natural language generation models.arXiv:1907.12009,

    Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. Representation degeneration problem in training natural language generation models.arXiv:1907.12009,

    Pith/arXiv arXiv 1907

  6. [12]

    The pile: An 800gb dataset of diverse text for language modeling.arXiv:2101.00027,

    15 Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling.arXiv:2101.00027,

    Pith/arXiv arXiv

  7. [13]

    Assessing look-ahead bias in stock return predictions generated by GPT sentiment analysis.arXiv:2309.17322,

    Paul Glasserman and Caden Lin. Assessing look-ahead bias in stock return predictions generated by GPT sentiment analysis.arXiv:2309.17322,

    Pith/arXiv arXiv

  8. [14]

    Continual pre-training of large language models: How to (re) warm your model?arXiv:2308.04014,

    Kshitij Gupta, Benjamin Th´ erien, Adam Ibrahim, Mats L Richter, Quentin Anthony, Eugene Belilovsky, Irina Rish, and Timoth´ ee Lesort. Continual pre-training of large language models: How to (re) warm your model?arXiv:2308.04014,

    Pith/arXiv arXiv

  9. [16]

    Chronologically consistent large language models.arXiv:2502.21206, 2025a

    Songrun He, Linying Lv, Asaf Manela, and Jimmy Wu. Chronologically consistent large language models.arXiv:2502.21206, 2025a. Songrun He, Linying Lv, Asaf Manela, and Jimmy Wu. Instruction tuning chronologically consistent language models.arXiv:2510.11677, 2025b. Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Hee- woo Ju...

    Pith/arXiv arXiv 2010

  10. [18]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    URLhttps://github.com/KellerJordan/modded-nanogpt. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv:2001.08361,

    Pith/arXiv arXiv 2001

  11. [19]

    Continual pre-training of language models.arXiv:2302.03241,

    Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pre-training of language models.arXiv:2302.03241,

    Pith/arXiv arXiv

  12. [20]

    Understanding catastrophic for- getting in language models via implicit inference.arXiv:2309.10105,

    Suhas Kotha, Jacob Mitchell Springer, and Aditi Raghunathan. Understanding catastrophic for- getting in language models via implicit inference.arXiv:2309.10105,

    Pith/arXiv arXiv

  13. [21]

    Tulu 3: Pushing frontiers in open language model post-training.arXiv:2411.15124,

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training.arXiv:2411.15124,

    Pith/arXiv arXiv

  14. [22]

    Learning rate matters: Vanilla lora may suffice for llm fine-tuning.arXiv:2602.04998,

    Yu-Ang Lee, Ching-Yun Ko, Pin-Yu Chen, and Mi-Yen Yeh. Learning rate matters: Vanilla lora may suffice for llm fine-tuning.arXiv:2602.04998,

    Pith/arXiv arXiv

  15. [23]

    17 Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gon- zalez, and Ion Stoica

    SSRN 5082861. 17 Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gon- zalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline.arXiv:2406.11939,

    Pith/arXiv arXiv

  16. [24]

    G-eval: Nlg evaluation using gpt-4 with better human alignment.arXiv:2303.16634, 2023a

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment.arXiv:2303.16634, 2023a. Yiqi Liu, Nafise Sadat Moosavi, and Chenghua Lin. Llms as narcissistic evaluators: When ego inflates evaluation scores.arXiv:2311.09766, 2023b. Alejandro Lopez-Lira and Yuehua Tang. Can chatg...

    Pith/arXiv arXiv

  17. [25]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback....

    Pith/arXiv arXiv

  18. [28]

    Lookahead bias in pretrained language models

    Suproteem K Sarkar and Keyon Vafa. Lookahead bias in pretrained language models. InICML 2025 Workshop on Reliable and Responsible Foundation Models,

    2025

  19. [29]

    Accessed: 2026-02-24. 19 Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhil...

    2026

  20. [30]

    Gemma 3 technical report.arXiv:2503.19786,

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram´ e, Morgane Rivi` ere, et al. Gemma 3 technical report.arXiv:2503.19786,

    Pith/arXiv arXiv

  21. [31]

    Gemma 2: Improving open language models at a practical size

    Gemma Team et al. Gemma 2: Improving open language models at a practical size. arXiv:2408.00118,

    Pith/arXiv arXiv

  22. [32]

    Llama: Open and efficient foundation language models.arXiv:2302.13971,

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ ee Lacroix, Baptiste Rozi` ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv:2302.13971,

    Pith/arXiv arXiv

  23. [33]

    Large language models are not fair evaluators.arXiv:2305.17926,

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators.arXiv:2305.17926,

    Pith/arXiv arXiv

  24. [35]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. arXiv:2109.01652,

    Pith/arXiv arXiv

  25. [36]

    Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing.arXiv:2406.08464,

    20 Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing.arXiv:2406.08464,

    Pith/arXiv arXiv

  26. [37]

    Datedgpt: Preventing looka- head bias in large language models with time-aware pretraining.arXiv:2603.11838,

    Yutong Yan, Raphael Tang, Zhenyu Gao, Wenxi Jiang, and Yao Lu. Datedgpt: Preventing looka- head bias in large language models with time-aware pretraining.arXiv:2603.11838,

    Pith/arXiv arXiv

  27. [38]

    Justice or prejudice? quantifying biases in llm-as-a- judge.arXiv:2410.02736,

    Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, et al. Justice or prejudice? quantifying biases in llm-as-a- judge.arXiv:2410.02736,

    Pith/arXiv arXiv

  28. [39]

    Hellaswag: Can a machine really finish your sentence?arXiv:1905.07830,

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?arXiv:1905.07830,

    Pith/arXiv arXiv 1905

  29. [40]

    Lora land: 310 fine-tuned llms that rival gpt-4, a technical report.arXiv:2405.00732,

    Justin Zhao, Timothy Wang, Wael Abid, Geoffrey Angus, Arnav Garg, Jeffery Kinnison, Alex Sherstinsky, Piero Molino, Travis Addair, and Devvret Rishi. Lora land: 310 fine-tuned llms that rival gpt-4, a technical report.arXiv:2405.00732,

    Pith/arXiv arXiv

  30. [41]

    Cheating automatic llm benchmarks: Null models achieve high win rates.arXiv:2410.07137,

    Xiaosen Zheng, Tianyu Pang, Chao Du, Qian Liu, Jing Jiang, and Min Lin. Cheating automatic llm benchmarks: Null models achieve high win rates.arXiv:2410.07137,

    Pith/arXiv arXiv

  31. [42]

    Instruction-following evaluation for large language models.arXiv:2311.07911,

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models.arXiv:2311.07911,

    Pith/arXiv arXiv

  32. [43]

    Value residual learning for alleviating attention concentration in transformers

    Zhanchao Zhou et al. Value residual learning for alleviating attention concentration in transformers. arXiv:2410.17897,

    Pith/arXiv arXiv

  33. [44]

    timeless

    21 A Training Configuration We train two GPT-2-style decoder-only transformer models on chronologically ordered FineWeb data. PIT-1.5B has 52 layers, 12 attention heads, and an embedding dimension of 1,536 (head dim = 128), totaling approximately 1.5B parameters trained on 170B tokens. PIT-4B has 20 layers, 32 attention heads, and an embedding dimension o...

    2023

  34. [1999]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    1901

  35. [2008]

    Training compute-optimal large language models.arXiv:2203.15556,

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models.arXiv:2203.15556,

    Pith/arXiv arXiv

  36. [2018]

    Scaling laws and compute-optimal training beyond fixed training durations

    Alexander H¨ agele et al. Scaling laws and compute-optimal training beyond fixed training durations. arXiv:2405.18392,

    Pith/arXiv arXiv

  37. [2019]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Turney, and Daniel Khashabi. Think you have solved question answering? try arc, the ai2 reasoning challenge. InarXiv:1803.05457,

    Pith/arXiv arXiv

  38. [2021]

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Can- wen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Ma...

    Pith/arXiv arXiv

  39. [2022]

    Reuse, don’t retrain: A recipe for continued pretraining of language models.arXiv:2407.07263,

    Jupinder Parmar, Sanjev Satheesh, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Reuse, don’t retrain: A recipe for continued pretraining of language models.arXiv:2407.07263,

    Pith/arXiv arXiv

  40. [2023]

    Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pel- lat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping...

    Pith/arXiv arXiv

  41. [2024]

    Qwen technical report, 2023a

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    Pith/arXiv arXiv

  42. [2025]

    Can large language models be an alternative to human evaluations?arXiv:2305.01937,

    Cheng-Han Chiang and Hung-yi Lee. Can large language models be an alternative to human evaluations?arXiv:2305.01937,

    Pith/arXiv arXiv

  43. [2026]

    Yann Dubois, Bal´ azs Galambosi, Percy Liang, and Tatsunori B Hashimoto

    SSRN:6540399. Yann Dubois, Bal´ azs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators.arXiv:2404.04475,

    Pith/arXiv arXiv