Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Warming up an LLM with as little as 0.1–0.3% procedural data—sequences generated by simple algorithms—before standard pretraining improves language, code, and math performance, and reaches the same loss with 55–86% of the original data.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 06:51 UTC pith:XAW366NP

load-bearing objection The diagnostic half is solid; the semantic-transfer half lacks a token-matched control, so the flagship data-efficiency claim is not yet nailed down. the 4 major comments →

arxiv 2601.21725 v2 pith:XAW366NP submitted 2026-01-29 cs.CL cs.LG

Procedural Pretraining: Warming Up Language Models with Abstract Data

Liangze Jiang , Zachary Shinnick , Anton van den Hengel , Hemanth Saratchandran , Damien Teney This is my paper
classification cs.CL cs.LG
keywords procedural pretraininglanguage model pretrainingsynthetic dataformal languagestransfer learningdata efficiencyalgorithmic reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that exposing a language model to a small amount of abstract, algorithm-generated data before standard pretraining acts as a beneficial warm-up. It claims this procedural pretraining improves performance on natural language, code, and informal mathematics, and that it dramatically reduces the data and compute needed to reach a given loss. The authors diagnose which algorithmic skills each type of procedural data imparts, localize the transferred knowledge in separate model components, and show that procedural data can complement or substitute for semantic data. A sympathetic reader would care because it suggests a cheap, simple way to accelerate LLM pretraining and to disentangle reasoning from knowledge acquisition.

Core claim

The central claim is that pretraining on a small amount of procedural data—outputs of formal languages, sequence transformations like sorting or set union, stack operations, or cellular automata—before standard semantic pretraining is both a complementary and an efficient substitute for standard data. Across C4, CodeParrot, and DeepMind-Math, front-loading only 0.1–0.3% procedural tokens outperforms standard pretraining, and the same loss is reached with only 55%, 67%, and 86% of the original semantic data respectively. The improvements are structure-dependent: shuffling the procedural sequences collapses gains back to baseline in the diagnostic tasks. The transferred knowledge is localized—

What carries the argument

The central object is "procedural data"—token sequences generated by explicit algorithms such as k-Dyck balanced brackets, STACK simulations, sequence transformations (IDENTITY, SET, SORT, UNION, REVERSE, DELETE), and ECA Rule 110 cellular automata. These sequences are used as a warm-up pretraining stage before standard semantic pretraining. The argumentative load is carried by measuring what transfers: the authors use diagnostic algorithmic tasks (needle-in-a-haystack recall, arithmetic, sorting) and selective layer transfer (attention-only vs. MLP-only) to attribute gains to structural regularities and to specific architectural components.

Load-bearing premise

The central premise is that the observed gains come from the structure of the procedural data itself, but the experiments that would prove this—replacing the procedural warm-up with equal amounts of random or shuffled tokens in the large-scale C4, CodeParrot, and DeepMind-Math runs—are not reported.

What would settle it

Train identical models on identical budgets where the warm-up phase uses the same number of tokens that are either randomly sampled or are the procedural sequences with their internal order shuffled, then compare downstream loss on C4, CodeParrot, and DeepMind-Math; if these non-structural warm-ups match or exceed the procedural pretraining gains, the claim that procedural structure is the cause would be falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • LLM pretraining can be made more data-efficient: reaching the same loss with 55–86% of the original data implies corresponding FLOPs savings.
  • Procedural pretraining may offer a practical way to improve code and mathematical reasoning in models, since attention-transferred procedural structure benefits structured domains.
  • The paper implies a distinction between knowledge acquisition and reasoning: an 'algorithmic reasoning prior' can be instilled before world knowledge is learned.
  • Benefits persist after downstream fine-tuning, suggesting the warm-up has lasting utility for practical model deployment.
  • Multiple procedural data types combine additively, either by mixing data or by stitching together attention and MLP weights, opening the door to optimized procedural curricula.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The absence of an equal-token control on the large-scale datasets leaves open whether any warm-up data—not necessarily structured—would yield similar gains; a direct test with random or shuffled tokens would settle whether "procedural" structure is the operative ingredient.
  • The localization results hint that procedural pretraining might be used deliberately as a modular initialization: one could, for instance, warm up MLPs for language and attention heads for code and then transfer only the relevant component.
  • If the effect scales to frontier-scale models, procedural warm-ups could be combined with data-mixture optimization to reduce web-scale pretraining budgets; this is a testable extension beyond the reported 1.3B scale.
  • The paper's mechanism evidence suggests that the benefits come from precise weight structure rather than distributional statistics, which implies that distillation or zero-shot transfer of these structures into other architectures may not be straightforward.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes 'procedural pretraining': an initial training stage on algorithmically generated data (sequence transformations such as SET/SORT/UNION, stack operations, Dyck languages, cellular automata) before standard pretraining on natural language, code, or mathematics. On small diagnostic tasks, different procedural data types are shown to improve specific algorithmic skills, and a shuffled-sequence control indicates that structure matters. On larger GPT-2 models, the authors report that adding tiny amounts (0.1–0.3%) of procedural tokens improves validation perplexity/accuracy on C4, CodeParrot, and DeepMind-Math, and that the same loss can be reached with 55/67/86% of the original semantic data. The paper also localizes the transferable information to attention or MLP layers and explores combining procedural data types at the data and weight level.

Significance. If the transfer results hold, procedural pretraining would provide a simple, compute-efficient complement to standard pretraining, with potential implications for data-centric LLM training. The paper has genuine strengths: a broad diagnostic sweep across many procedural data types, a shuffled-sequence control in the diagnostic setting, multi-seed reporting for the algorithmic tasks, scaling experiments up to 1.3B parameters, and mechanistic analyses using weight shuffling/noise. The central semantic-transfer claim, however, currently lacks the controls needed to separate the effect of procedural structure from the effect of simply training on more tokens and optimization steps. The missing control is directly load-bearing for the headline 'complement/substitute' and '55/67/86% data savings' claims.

major comments (4)
  1. [Sections 5.1–5.2, Figure 5] The additive setting compares T1 procedural + T2 semantic tokens against T2 semantic tokens only, and the substitutive setting compares T1 + (T2−ΔT2) against T2. In both settings, the procedural model is trained on strictly more tokens and updates, so the observed gains—including the 55/67/86% data-savings numbers—could be caused by extra training alone rather than by the algorithmic structure of the data. The shuffled-sequence control that would isolate structure is run only for the small algorithmic diagnostic tasks (Section 4.1, Figure 2), not for C4/CodeParrot/DeepMind-Math. Please add an equal-token control in both settings, e.g., pretraining on shuffled procedural sequences, random tokens, or additional semantic tokens. Without this control, the paper's central claim that procedural structure—not token count—drives semantic gains is not supported.
  2. [Section 5.2, Tables 3–4, Appendix M] The 'significantly outperforms' claims for C4, CodeParrot, DeepMind-Math, BLiMP, GLUE, PY150, and WikiText-103 rest on single runs with no variance or significance test. The small diagnostic experiments report means over 10 seeds, but the semantic-transfer results do not. Please provide at least 3 seeds (or error bars in Figures 4–6) for the headline comparisons, including the scaling results in Table 3.
  3. [Section 5.2, Appendices G–I] The procedural data types were selected as 'best-performing' using the same domain-specific validation sets (Figure 4), and the T1 budgets (sequence length, steps) are tuned on downstream validation. This selection on the evaluation criterion can overstate the magnitude of the gains. Please either report results for all procedural types and a range of T1 values without pre-selection, or describe the selection procedure and quantify its effect (e.g., by reporting the worst procedural type as a lower bound).
  4. [Section 5.2, Figure 5 bottom row] The data-savings percentages (55/67/86%) are derived from loss-equivalence curves, but the fitting/interpolation method is not described. It is unclear whether the reported numbers come from a single run or an average, which procedural type and T1 value they correspond to, and how 'matching baseline loss' is defined. Please specify the procedure explicitly and report confidence intervals or at least the range across the three procedural types.
minor comments (6)
  1. [Abstract, Section 5.2] The abstract states '0.1% procedural data', while Section 5.2 says 0.1–0.3%. Please align the numbers.
  2. [Section 1] 'C4 (Raffel et al., 2020))' contains a double closing parenthesis.
  3. [Figure 5] The bottom-row panels are under-explained; the caption should define what x and y are, which procedural type is used, and how the equivalence curve is computed.
  4. [Appendix N.1, Table 7] The row label 'BEST MODEL SHUFFLED' is not defined in the caption. Please clarify which model was shuffled and how.
  5. [Section 5.3] 'JAVACORPUSand CODEPARROTas different domains' is missing spaces and is hard to read; also Figure 6 labels are small and partially illegible.
  6. [Reproducibility Statement] The GitHub link is provided, but the code is described as 'in preparation'. Please make the repository accessible or state clearly that it will be populated for the final version.

Circularity Check

0 steps flagged

No circularity: headline claims are evaluated against external benchmarks; the missing equal-token control is a confound, not a circular reduction.

full rationale

The paper's central derivation is empirical: procedural pretraining (T1 procedural tokens followed by T2 semantic tokens) is compared with standard pretraining (T2 only) on external datasets (C4, CodeParrot, DeepMind-Math, WikiText, JavaCorpus, etc.). The 'savings' figures (55/67/86%) are defined through an equivalence curve — 'we plot curves whose points (x,y) achieve equivalent performance with x procedural tokens and y standard tokens' — and are read off measured training curves rather than derived from a model that already contains the answer. The choice of procedural types (UNION, SORT, SET) and T1 budgets is a grid-search/model-selection step, not a fitted parameter later relabeled as a prediction: the selection is made on Section 5.1 validation and then evaluated on different target corpora. The self-citation to Shinnick et al. (2025) appears only as a related-work aside ('A concurrent work ... shows that procedural data benefits visual learning') and is not load-bearing for the claims. The missing equal-token/shuffled control in Sections 5.1-5.2 is a genuine experimental confound (extra tokens/steps could explain part of the gain), but that is a correctness/identifiability limitation, not circularity: the reported improvements are not forced by the experimental design in a definitional way. No equations define the target quantity in terms of the input, and no result is imported solely from the authors' prior work. Therefore no significant circularity.

Axiom & Free-Parameter Ledger

8 free parameters · 5 axioms · 0 invented entities

The paper is empirical; it introduces no new theoretical entities or fitted equations. The listed free parameters are experimental/tuning choices that shape the headline numbers. The load-bearing domain assumptions are about transfer validity and the adequacy of controls.

free parameters (8)
  • Procedural data type selection (UNION/SORT/SET) = UNION, SORT, SET (others in Fig 2)
    Types reported in main transfer results were selected after observing downstream perplexity (Fig 4), so headline gains are conditional on this selection.
  • Procedural token budget T1 = 0–20M; key results use 2.1M / 4.1M / 82M
    T1 is swept (100–2500 steps x seq len) and best values used; no principled rule is given.
  • Procedural sequence length = 8–128 (swept)
    Appendices G/H show optimal length is task- and target-domain-dependent; main results use grid-selected lengths.
  • Procedural vocabulary size = 100 symbols (default)
    Appendix J shows small vocabularies hurt and ~100–200 works; chosen by hand/grid.
  • Curriculum thresholds for sequence transformations = length +2 when 99% accuracy reached
    Hand-chosen curriculum rule for small models in Section 4.
  • Weight decay during procedural pretraining = 0.1 (NL), 0.01 (code/math)
    Follows Hu et al. (2025); Appendix K shows the conclusion is unchanged, but the exact choice is a tuning parameter.
  • Entropy-regularizer target tau = 0.8
    Set to match observed head entropy for STACK; used only in the control experiment in Appendix F.
  • Procedural generation probabilities = STACK push/pop 0.75; DYCK p_open 0.49
    Generation hyperparameters chosen from prior work; affect data difficulty and the measured transfer.
axioms (5)
  • domain assumption Any extra pretraining tokens, even unstructured ones, would not reproduce the semantic gains; the shuffled-control is sufficient evidence of structure's importance.
    Weakened by the fact that shuffled control appears only for algorithmic tasks (Fig 2), not for C4/CodeParrot/DeepMind-Math.
  • domain assumption Selective transfer (attention-only / MLP-only) cleanly attributes learned skills to those layers.
    Reinitializing other layers changes the optimization trajectory; Sections 4.2 and 5.3 draw localization conclusions from this proxy.
  • domain assumption Reinitializing token embeddings breaks vocabulary correspondence while preserving transferable structure.
    Section 3.1; used in all semantic transfers, with no control varying the embedding initialization method.
  • domain assumption Loss-equivalence curves constructed by interpolation from single training runs reliably measure data savings.
    Section 5.2 bottom panel and the 55/67/86% numbers; no confidence intervals or repeated runs are provided.
  • domain assumption GPT-2/CodeParrot-small models up to 1.3B are adequate proxies for LLM pretraining.
    The Limitations section admits models are smaller than state-of-the-art; scaling claims rely on extrapolation.

pith-pipeline@v1.3.0-alltime-deepseek · 26869 in / 14132 out tokens · 131923 ms · 2026-08-03T06:51:42.154755+00:00 · methodology

0 comments
read the original abstract

Pretraining language models directly on web-scale corpora is the de facto paradigm. We study an alternative where the model is initially exposed to abstract structured data to ease the subsequent acquisition of rich semantic knowledge, much like humans learning simple logic and mathematics before higher reasoning. We focus on procedural data, generated by formal languages and other simple algorithms, as such abstract data. We first diagnose the algorithmic skills that different forms of procedural data can improve, often significantly. For example, the accuracy of context recall (Needle-in-a-haystack) jumps from 10 to 98% when a model is pretrained on Dyck sequences (balanced brackets). Second, we study how these gains are reflected in pretraining larger models (up to 1.3B). We find that front-loading as little as 0.1 to 0.3% procedural data significantly outperforms standard pretraining on natural language, code, and informal mathematics (C4, CodeParrot, and DeepMind-Math datasets). Notably, this also enables the models to reach the same loss value with only 55/67/86% of the original data and thus a comparable reduction in FLOPs. Third, we explore the mechanisms behind the benefits and find that procedural pretraining instills non-trivial structure in both attention and MLP layers. The former is particularly important for structured domains (e.g. code), and the latter for language. Finally, we lay a path for combining multiple forms of procedural data. Our results show that procedural pretraining is a simple, lightweight means of improving performance and accelerating language model pretraining, ultimately suggesting the promise of disentangling knowledge acquisition from reasoning in LLMs.

Figures

Figures reproduced from arXiv: 2601.21725 by Anton van den Hengel, Damien Teney, Hemanth Saratchandran, Liangze Jiang, Zachary Shinnick.

Figure 1
Figure 1. Figure 1 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Different types of procedural pretraining can significantly improve over standard training (dashed line) across various algorithmic tasks. If we remove the structure within the pro￾cedural data by shuffling the sequences (Best model shuffled), the performance falls to the baseline. Reported values are the means over 10 seeds (full results with variance in Appendix N.1). Take-away. Among different types of … view at source ↗
Figure 3
Figure 3. Figure 3: Selective transfer of MLP or attention layers can improve over full-model transfer, showing that procedural pretraining creates ‘modular’ structure localised in the selected model com￾ponents. Reported values are means across 10 seeds (full results with variance in Appendix N.1). Results [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The benefits of procedural pretraining transfer to semantic domains. Perplexity (lower is better) on natural language (left) and pure code (right). A little of procedural data is very effective: compare the number of procedural tokens (T1) in these plots with the amount of tokens from the target datasets (T2) being 15M for WIKITEXT and 105M for JAVACORPUS. Take-away. The benefits of procedural pretraining … view at source ↗
Figure 5
Figure 5. Figure 5: Procedural pretraining is complementary to standard data & highly data-efficient. Each column corresponds to a different semantic dataset. (Top) Training curves with different types of procedural data (UNION, SORT, SET). (Middle) Additive setting: a small amount of procedural data is sufficient to outperform standard pretraining. (Bottom) Substitutive setting: we plot curves whose points (x, y) achieve equ… view at source ↗
Figure 6
Figure 6. Figure 6: Localisation of transferable pretrained information for different semantic domains. (Top) Using selective weight transfer (see Section 3.1), we find that MLPs and attention layers are important respectively for natural language and pure code, across different types of procedural data. (Bottom) On larger datasets, MLP-only transfer works best for language. As expected, full transfer is optimal for domains i… view at source ↗
Figure 7
Figure 7. Figure 7: Mixtures of two types of procedural data. We vary the pro￾portion of SET and UNION (indicated by the small pie charts) while keep￾ing the total number of procedural to￾ken T1 fixed. Some choices achieve a clearly better perplexity (lower is bet￾ter) than either of the two types alone. Results [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: We pretrain transformers on various forms of procedural data generated from simple [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Head-wise attention entropy after fine-tuning. Procedural pretraining yields a subset of [PITH_FULL_IMAGE:figures/full_fig_p021_9.png] view at source ↗
Figure 11
Figure 11. Figure 11: Validation accuracy on HAYSTACK with en￾tropy regularisation. Models trained from scratch with ex￾plicitly enforced low-entropy heads (orange) underperform those with procedurally pretrained heads (blue), indicating that sharper attention alone is insufficient. Results are aver￾aged over 10 random seeds. 0 25 50 75 100 Accuracy (%) STACK → HAYSTACK SET → HAYSTACK SET → ADDITION SET → SORTING Attention-onl… view at source ↗
Figure 10
Figure 10. Figure 10: Validation accuracy after downstream fine-tuning when transferring subsets of proce [PITH_FULL_IMAGE:figures/full_fig_p022_10.png] view at source ↗
Figure 12
Figure 12. Figure 12: Layer-wise weight shuffling largely eliminates the benefits of procedural pretrain￾ing, despite preserving the overall distribution of weight values. This indicates that the advan￾tages arise from precise structural organisation of the weights, rather than from their distribu￾tion alone. 0 0.01 0.05 0.1 Noise level (σ) −1.0 −0.5 0.0 0.5 1.0 Relative improv. HAYSTACK ADDITION SORTING No procedural pretrain… view at source ↗
Figure 14
Figure 14. Figure 14: Validation perplexity for different configurations of procedural pretraining when fine [PITH_FULL_IMAGE:figures/full_fig_p023_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Validation perplexity for DYCK and DYCK SHUFFLE procedural pretraining when fine￾tuned on WIKITEXT (left) and JAVACORPUS (right), sweeping over sequence length and number of pretraining steps. Setup matches [PITH_FULL_IMAGE:figures/full_fig_p024_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Effect of extending sequence length during procedural pretraining on W [PITH_FULL_IMAGE:figures/full_fig_p024_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Transferability of procedural pretraining. Relationship between procedural validation [PITH_FULL_IMAGE:figures/full_fig_p024_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Effect of vocabulary size during procedural pretraining on W [PITH_FULL_IMAGE:figures/full_fig_p025_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Effect of weight decay during procedural pretraining on C4. Changing weight decay [PITH_FULL_IMAGE:figures/full_fig_p025_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Token level code completion accuracy on J [PITH_FULL_IMAGE:figures/full_fig_p030_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: BLiMP accuracy (Warstadt et al., 2020) after training on C4. We compare partial transfer [PITH_FULL_IMAGE:figures/full_fig_p030_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Comparison of MLP-only transfer and full-model transfer on C4 for U [PITH_FULL_IMAGE:figures/full_fig_p031_22.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Brain Score Tracks Shared Properties of Languages: Evidence from Many Natural Languages and Structured Sequences

    cs.CL 2026-04 unverdicted novelty 7.0

    Brain Score remains similar when language models are trained on diverse natural languages or on structured non-language data like DNA and code, indicating the metric tracks shared structural extraction but is not diag...

Reference graph

Works this paper leans on

46 extracted references · 30 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Transferring inductive biases through knowledge distillation.arXiv preprint arXiv:2006.00555,

    Samira Abnar, Mostafa Dehghani, and Willem Zuidema. Transferring inductive biases through knowledge distillation.arXiv preprint arXiv:2006.00555,

    Pith/arXiv arXiv 2006

  2. [2]

    We report results for full model transfer, MLP- transfer, and attention-transfer

    Pretraining task FULL TRANSFERMLPONLYATTENTION ONLY 4-DYCK98.3±1.1 8.7±0.5 11.6±0.5 16-DYCK SHUFFLE18.6±26.3 8.9±0.9 16.5±10.6 STACK55.2±39.3 7.1±0.6 98.9±0.8 IDENTITY18.8±14.3 7.0±0.9 99.0±1.7 SET18.9±26.6 8.3±0.7 88.9±27.1 UNION9.8±1.1 8.2±0.7 11.7±0.4 REVERSE33.3±22.4 7.3±1.2 98.6±0.8 DELETE52.6±22.4 8.4±0.8 91.8±3.5 ECA10.5±0.5 8.7±1.0 11.6±1.0 Table ...

    2021

  3. [4]

    The hyperparameters used for procedural pretraining are summarised in Table 2, with the exception of ECA RULE110, whose configuration is reported separately below

    18 Preprint. The hyperparameters used for procedural pretraining are summarised in Table 2, with the exception of ECA RULE110, whose configuration is reported separately below. Task SEQ.LENGTHLEARNING RATEVOCAB.SIZE IDENTITY4–205×10 −4 102 SET2–205×10 −4 102 STACK4–205×10 −4 103 k-DYCK1285×10 −5 2×k k-DYCKSHUFFLE1285×10 −5 2×k Table 2: Pretraining hyperpa...

    2024

  4. [5]

    Following Hu et al

    For all algorithmic procedural tasks used in this section (IDENTITY, SET, UNION, DELETE, SORT, REVERSE, and STACK), we train using AdamW with a batch size of 64 and no warmup steps. Following Hu et al. (2025), we pretrain models on procedural data with a weight decay of 0.1 for WIKITEXTand C4, and use 0.01 for JAVACORPUS, CODEPARROT, and DEEPMIND-MATH. Th...

    2025

  5. [8]

    The Pile: An 800GB dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027,

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027,

    Pith/arXiv arXiv

  6. [10]

    Meta-learning neural mechanisms rather than bayesian priors.arXiv preprint arXiv:2503.16048,

    Michael Goodale, Salvador Mascarenhas, and Yair Lakretz. Meta-learning neural mechanisms rather than bayesian priors.arXiv preprint arXiv:2503.16048,

    Pith/arXiv arXiv

  7. [11]

    Learning uni- versal predictors.arXiv preprint arXiv:2401.14953,

    Jordi Grau-Moya, Tim Genewein, Marcus Hutter, Laurent Orseau, Gr ´egoire Del´etang, Elliot Catt, Anian Ruoss, Li Kevin Wenliang, Christopher Mattern, Matthew Aitchison, et al. Learning uni- versal predictors.arXiv preprint arXiv:2401.14953,

    Pith/arXiv arXiv

  8. [12]

    The platonic representation hypothesis.arXiv preprint arXiv:2405.07987,

    Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis.arXiv preprint arXiv:2405.07987,

    Pith/arXiv arXiv

  9. [13]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    Pith/arXiv arXiv 2001

  10. [14]

    Questioning representational optimism in deep learning: The fractured entangled representation hypothesis.arXiv preprint arXiv:2505.11581,

    Akarsh Kumar, Jeff Clune, Joel Lehman, and Kenneth O Stanley. Questioning representational optimism in deep learning: The fractured entangled representation hypothesis.arXiv preprint arXiv:2505.11581,

    Pith/arXiv arXiv

  11. [15]

    Codexglue: A machine learning benchmark dataset for code understanding and generation.arXiv preprint arXiv:2102.04664,

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. Codexglue: A machine learning benchmark dataset for code understanding a...

    Pith/arXiv arXiv

  12. [16]

    Modeling rapid language learning by distilling bayesian priors into artificial neural networks.arXiv preprint arXiv:2305.14701,

    R Thomas McCoy and Thomas L Griffiths. Modeling rapid language learning by distilling bayesian priors into artificial neural networks.arXiv preprint arXiv:2305.14701,

    Pith/arXiv arXiv

  13. [17]

    Pointer sentinel mixture models.arXiv preprint arXiv:1609.07843,

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models.arXiv preprint arXiv:1609.07843,

    Pith/arXiv arXiv

  14. [19]

    Asano, Iro Laina, Christian Rupprecht, Nakamasa Inoue, Rio Yokota, and Hirokatsu Kataoka

    Ryo Nakamura, Ryu Tadokoro, Ryosuke Yamada, Yuki M. Asano, Iro Laina, Christian Rupprecht, Nakamasa Inoue, Rio Yokota, and Hirokatsu Kataoka. Scaling backwards: Minimal synthetic pre-training?arXiv preprint arXiv:2408.00677,

    Pith/arXiv arXiv

  15. [20]

    Injecting structural hints: Using language models to study inductive biases in language learning.arXiv preprint arXiv:2304.13060,

    Isabel Papadimitriou and Dan Jurafsky. Injecting structural hints: Using language models to study inductive biases in language learning.arXiv preprint arXiv:2304.13060,

    Pith/arXiv arXiv

  16. [21]

    How does code pretraining affect language model task performance?arXiv preprint arXiv:2409.04556,

    Jackson Petty, Sjoerd van Steenkiste, and Tal Linzen. How does code pretraining affect language model task performance?arXiv preprint arXiv:2409.04556,

    Pith/arXiv arXiv

  17. [22]

    Hadi Pouransari, David Grangier, C Thomas, Michael Kirchhof, and Oncel Tuzel

    12 Preprint. Hadi Pouransari, David Grangier, C Thomas, Michael Kirchhof, and Oncel Tuzel. Pretrain- ing with hierarchical memories: separating long-tail and common knowledge.arXiv preprint arXiv:2510.02375,

    arXiv

  18. [23]

    Pretraining with artificial language: Studying transferable knowledge in language models.arXiv preprint arXiv:2203.10326,

    Ryokan Ri and Yoshimasa Tsuruoka. Pretraining with artificial language: Studying transferable knowledge in language models.arXiv preprint arXiv:2203.10326,

    Pith/arXiv arXiv

  19. [24]

    Proce- dural knowledge in pretraining drives reasoning in large language models.arXiv preprint arXiv:2411.12580,

    Laura Ruis, Maximilian Mozes, Juhan Bae, Siddhartha Rao Kamalakara, Dwarak Talupuru, Acyr Locatelli, Robert Kirk, Tim Rockt ¨aschel, Edward Grefenstette, and Max Bartolo. Proce- dural knowledge in pretraining drives reasoning in large language models.arXiv preprint arXiv:2411.12580,

    Pith/arXiv arXiv

  20. [26]

    Zico Kolter

    Asher Trockman and J. Zico Kolter. Mimetic initialization of self-attention layers.arXiv preprint arXiv:2305.09828,

    Pith/arXiv arXiv

  21. [27]

    net/forum?id=rJ4km2R5t7

    URLhttps://openreview. net/forum?id=rJ4km2R5t7. Yanwei Wang, Ching-Yun Ko, and Pulkit Agrawal. Visual pre-training for navigation: What can we learn from noise?arXiv preprint arXiv:2207.00052,

    Pith/arXiv arXiv

  22. [28]

    Pre-training with synthetic data helps offline reinforcement learning.arXiv preprint arXiv:2310.00771,

    Zecheng Wang, Che Wang, Zixuan Dong, and Keith Ross. Pre-training with synthetic data helps offline reinforcement learning.arXiv preprint arXiv:2310.00771,

    Pith/arXiv arXiv

  23. [29]

    Filtering with self-attention and storing with mlp: One-layer trans- formers can provably acquire and extract knowledge.arXiv preprint arXiv:2508.00901,

    Ruichen Xu and Kexin Chen. Filtering with self-attention and storing with mlp: One-layer trans- formers can provably acquire and extract knowledge.arXiv preprint arXiv:2508.00901,

    Pith/arXiv arXiv

  24. [30]

    Initializing models with larger ones.arXiv preprint arXiv:2311.18823,

    Zhiqiu Xu, Yanjie Chen, Kirill Vishniakov, Yida Yin, Zhiqiang Shen, Trevor Darrell, Lingjie Liu, and Zhuang Liu. Initializing models with larger ones.arXiv preprint arXiv:2311.18823,

    Pith/arXiv arXiv

  25. [31]

    Instilling inductive biases with subnetworks

    Enyan Zhang, Michael A Lepori, and Ellie Pavlick. Instilling inductive biases with subnetworks. arXiv preprint arXiv:2310.10899,

    Pith/arXiv arXiv

  26. [32]

    Intelligence at the edge of chaos.arXiv preprint arXiv:2410.02536,

    Shiyang Zhang, Aakash Patel, Syed A Rizvi, Nianchen Liu, Sizhuang He, Amin Karbasi, Emanuele Zappala, and David van Dijk. Intelligence at the edge of chaos.arXiv preprint arXiv:2410.02536,

    Pith/arXiv arXiv

  27. [33]

    But recent results also question the value of the data, showing that some benefits of pretraining are attributable to the optimisation objective more than the actual data

    of pretraining data are empirically critical for the performance of large language models. But recent results also question the value of the data, showing that some benefits of pretraining are attributable to the optimisation objective more than the actual data. Balestriero & Huang (2024) compared models trained for text classification from random initial...

    2024

  28. [34]

    and burstiness to enable in-context learning (Chan et al., 2022). Computer code is empirically very effective as pretraining data for LLMs, as it improves their abilities for com- positional generalisation and math-related tasks (Aryabumi et al., 2024; Petty et al., 2024). This presumably results from the abundant compositional and recursive patterns in c...

    2022

  29. [35]

    Partial transfer from pretrained transformers.Zhang et al

    and noise-based images (Baradad et al., 2022). Partial transfer from pretrained transformers.Zhang et al. (2023) and (Xu et al.,

    2022

  30. [36]

    mimetic initialisation

    showed that copying subsets of pretrained weights could transfer specific capabilities. Abnar et al. (2020) used knowledge distillation to transfer the inductive biases of one architecture into another. The “mimetic initialisation” of self-attention (Trockman & Kolter,

    2020

  31. [37]

    Goodale et al

    and Solomonoff Induction (Grau-Moya et al., 2024). Goodale et al. (2025) showed that well-chosen formal languages can teach complex mechanisms (e.g. counters) to a sequence model. Pretraining can generally be seen as asoftinductive bias for subsequent fine-tuning. But there is a large gap in our understanding of its effects compared to those ofhardinducti...

    2024

  32. [41]

    The hyperparameters follow those in (Lu et al., 2021)

    We use a learning rate of8×10 −5 and no warmup steps. The hyperparameters follow those in (Lu et al., 2021). Sequences are tokenized with the CodeGPT (Lu et al.,

    2021

  33. [42]

    Sequences are tokenized with the GPT-2 tokenizer and truncated to 2,048 tokens

    We use a learning rate of5×10 −4 with cosine decay and no warmup steps. Sequences are tokenized with the GPT-2 tokenizer and truncated to 2,048 tokens. We evaluate models on the C4 validation split using 1,024 samples, reporting validation perplexity. To assess linguistic generalization, we also report accuracy on the BLiMP grammaticality judgment benchma...

    2020

  34. [43]

    We use a constant learning rate of8×10 −5 (as is done in the original paper), no warmup steps, and weight decay of0.1. Sequences are tokenized at the character-level (including digits, alphabet in upper and lower case, punctuation and whitespace, a total of 95 different tokens) and have a length 512 tokens. We evaluate models on the in-distribution valida...

    2016

  35. [44]

    Sequences are tokenized with the GPT-2 tokenizer, truncated to 2,048 tokens

    We use a learning rate of1×10 −4 with cosine decay and no warmup steps. Sequences are tokenized with the GPT-2 tokenizer, truncated to 2,048 tokens. GLUE: We finetune our language models on the GLUE benchmark (Wang et al., 2019). For all evalautions, fine-tuning is run for one epoch with a batch size of 16 and learning rate of5×10 −5 with a linear decay. ...

    2019

  36. [64]

    Appendix G analyses the effect of sequence length, while Appendix H examines the impact of extending lengths further. For consistency in token counts, we assume the output sequence is at most twice the length of the input, and thus estimate and report the total number of procedural tokens as2×the input length. Sequence Transformation Descriptions. IDENTIT...

    2023

  37. [500]

    K WEIGHTDECAYABLATION In the main paper, natural language experiments use a weight decay of 0.1 during procedural pre- training, following Hu et al

    Overall, procedural pretraining is most effective within a moderate vocabulary range, too small harms transfer, while too large brings no improvement or negative return. K WEIGHTDECAYABLATION In the main paper, natural language experiments use a weight decay of 0.1 during procedural pre- training, following Hu et al. (2025). To test this choice, we reduce...

    2025

  38. [2013]

    Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzieh Fadaee, Ahmet ¨Ust¨un, and Sara Hooker

    10 Preprint. Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzieh Fadaee, Ahmet ¨Ust¨un, and Sara Hooker. To code, or not to code? exploring impact of code in pre-training.arXiv preprint arXiv:2408.10914,

    Pith/arXiv arXiv

  39. [2016]

    Trans- formers can do bayesian inference.arXiv preprint arXiv:2112.10510,

    Samuel M¨uller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Trans- formers can do bayesian inference.arXiv preprint arXiv:2112.10510,

    Pith/arXiv arXiv

  40. [2019]

    Can you learn to see without images? procedural warm-up for vision transformers.arXiv preprint arXiv:2511.13945,

    Zachary Shinnick, Liangze Jiang, Hemanth Saratchandran, Damien Teney, and Anton van den Hen- gel. Can you learn to see without images? procedural warm-up for vision transformers.arXiv preprint arXiv:2511.13945,

    arXiv

  41. [2020]

    Transformer feed-forward layers are key-value memories.arXiv preprint arXiv:2012.14913,

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories.arXiv preprint arXiv:2012.14913,

    Pith/arXiv arXiv 2012

  42. [2021]

    Procedural image programs for representation learning.arXiv preprint arXiv:2211.16412,

    Manel Baradad, Chun-Fu Chen, Jonas Wulff, Tongzhou Wang, Rogerio Feris, Antonio Torralba, and Phillip Isola. Procedural image programs for representation learning.arXiv preprint arXiv:2211.16412,

    Pith/arXiv arXiv

  43. [2022]

    Emergent properties with repeated examples.arXiv preprint arXiv:2410.07041,

    Franc ¸ois Charton and Julia Kempe. Emergent properties with repeated examples.arXiv preprint arXiv:2410.07041,

    Pith/arXiv arXiv

  44. [2023]

    Attention retrieves, mlp memorizes: Disentangling trainable components in the transformer.arXiv preprint arXiv:2506.01115,

    Yihe Dong, Lorenzo Noci, Mikhail Khodak, and Mufan Li. Attention retrieves, mlp memorizes: Disentangling trainable components in the transformer.arXiv preprint arXiv:2506.01115,

    Pith/arXiv arXiv

  45. [2024]

    Learning to see by looking at noise.arXiv preprint arXiv:2106.05963,

    Manel Baradad, Jonas Wulff, Tongzhou Wang, Phillip Isola, and Antonio Torralba. Learning to see by looking at noise.arXiv preprint arXiv:2106.05963,

    Pith/arXiv arXiv

  46. [2025]

    Doge: Domain reweighting with generalization estimation.arXiv preprint arXiv:2310.15393,

    Simin Fan, Matteo Pagliardini, and Martin Jaggi. Doge: Domain reweighting with generalization estimation.arXiv preprint arXiv:2310.15393,

    Pith/arXiv arXiv