Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pretraining on procedural data instils distinct, complementary reasoning skills in small transformers, located in specific weight components and composable across models.

desk verdict Plausible, well-controlled study showing procedural pretraining creates transferable and partly composable inductive biases; the modular-localization claim is more tentative than the paper suggests. read the letter →

arxiv 2505.22308 v1 pith:ZNEB7KKZ submitted 2025-05-28 cs.LG

Zachary Shinnick , Liangze Jiang , Hemanth Saratchandran , Anton van den Hengel , Damien Teney This is my paper
classification cs.LG
keywords proceduralpretraininginductivebiasselectiveweighttransfermodularityalgorithmicreasoningtransformersattentionlayersMLPblocks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that pretraining small transformers on simple procedurally generated sequences—nested brackets, stack operations, identity copying, set deduplication, and cellular automata—does not just warm up the network in a generic way. Different procedural rules instil distinct, complementary inductive structures, and those structures settle in different architectural components: attention layers usually carry the most transferable information, while some rules put useful structure into MLP blocks instead. The paper supports this with selective-transfer experiments that initialize only attention or only MLP weights from a pretrained model, and with perturbation experiments showing the benefits depend on precise weight structure rather than mere weight magnitudes. The central constructive result is that modules from separately pretrained models can be recombined, so a model initialized with SET-pretrained attention and ECA-pretrained MLPs fine-tunes well on all four tested reasoning tasks. If right, this points toward training reasoning skills independently of semantic knowledge and assembling them into a generalist model.

What carries the argument

The machinery is the decomposition of a transformer's weights into three blocks, $T = (E, A, F)$, with $E$ the token and position embeddings, $A$ the attention layers, and $F$ the MLP layers. The paper's protocol is selective transfer: initialize a fine-tuning run with only one block from a pretrained model and random weights elsewhere, then fine-tune everything. A block is credited with carrying the pretraining structure when this beats both random initialization and full transfer. The perturbation controls, per-layer weight shuffling (which preserves magnitudes but destroys structure) and additive Gaussian noise of increasing variance, separate genuine learned mechanisms from trivial initialization effects. The composition step builds $T_{\mathrm{comb}} = (E_{\mathrm{rand}}, A_{\mathrm{pre}}^1, F_{\mathrm{pre}}^2)$ from two separately pretrained models, showing that the localized structures are mutually compatible.

What would settle it

Run the composition experiment of Section 5 with the ECA MLP block replaced by randomly initialized MLP weights of the same scale, keeping the SET attention block; if this control matches the composed model's accuracy on all four diagnostic tasks, the claimed complementarity of the two pretraining rules collapses.

Watch

Extended reading notes

Core claim

On the paper's terms: procedurally pretrained transformers contain modular inductive structures that are specific to the pretraining rule, localized in particular weight subsets, and transferable across models. Full-model transfer of weights from k-DYCK pretraining to a needle-in-a-haystack task reaches 98.3 percent accuracy versus 11.3 percent from random initialization, while attention-only transfer from STACK or IDENTITY pretraining reaches 98–99 percent, far above full transfer. Reversed addition, in contrast, benefits more from MLP-only or full transfer, showing that the residence of the structure is task-dependent. The perturbation analysis shows that adding Gaussian noise or shuffling weights steadily destroys the downstream benefit, with haystack and addition collapsing under shuffling, which rules out an explanation purely in terms of beneficial weight magnitudes. The composition experiment then shows that taking attention weights from a SET-pretrained model and MLP weights from an ECA-pretrained model gives a single initialization that performs well on haystack, addition, reversed addition, and sorting, where either source alone fails on at least one task.

Load-bearing premise

The load-bearing premise is that a component's benefit after fine-tuning tells you where the pretrained structure actually lives, since the whole model is fine-tuned after partial transfer and a component could help merely by making optimization easier rather than by storing the skill.

Editorial extensions

If this is right

  • Pretraining data can be chosen by the skill it installs, such as k-DYCK for memory recall, ECA for carry-based arithmetic, and SET for sorting, rather than by general language-modeling perplexity.
  • Partial weight transfer is a practical initialization strategy: copying only attention or only MLP weights can beat transferring the whole pretrained model.
  • Composed initializations from several small procedural pretrainings can support several downstream capabilities at once, suggesting a path to multi-skill models without a single massive pretraining corpus.
  • Weight-magnitude effects are not the explanation: the downstream gains require the precise internal arrangement of pretrained weights, so procedural pretraining learns mechanisms, not just calibration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same selective-transfer logic could serve as a diagnostic on much larger language models, locating where code pretraining or instruction tuning places its skills by fine-tuning only attention or only MLP blocks from a checkpoint.
  • A direct test of modular composition would replace one donated block with random weights of matched scale; if that control matches the composed model, the complementarity claim would need revision, and the paper does not report this control.
  • If procedural pretraining installs reasoning mechanisms independently of semantics, pre-pretraining curricula on such data could precede natural-language pretraining, potentially improving the data efficiency of general-purpose models.
  • The observed task-dependence of where structure resides suggests that capacity could be routed to attention or MLP blocks depending on which downstream skills are prioritized during initialization.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper pretrains small GPT-2-style transformers on procedurally generated data (k-DYCK, k-DYCK SHUFFLE, STACK, IDENTITY, SET, and ECA Rule 110) and then fine-tunes them on algorithmic diagnostic tasks (needle-in-a-haystack, addition, reversed addition, multiplication, sorting, and language modeling). It reports that different procedural pretraining tasks improve different downstream skills, that selective transfer of only attention or only MLP weights can outperform full-model transfer, that perturbing pretrained weights degrades performance in task-dependent ways, and that combining the attention weights of a SET-pretrained model with the MLP weights of an ECA-pretrained model can yield a strong multi-task initialization. The central claim is that procedural pretraining creates modular, composable inductive structures that reside in specific architectural components.

Significance. If the localization and composition claims hold, this would be a useful step toward understanding when and why synthetic procedural data helps downstream reasoning, and toward composing skills from separately pretrained models. The paper has real strengths: it includes random-initialization baselines, noise and shuffling perturbations, full-versus-selective transfer comparisons, and a clearly specified experimental pipeline. However, the main evidence for 'where the structure resides' and for 'composable modules' is inferential and, in its current form, does not uniquely exclude simpler explanations based on optimization-landscape effects of the initialization. The result is therefore promising but not yet established.

major comments (4)
  1. [§4.1, Tables 4–7] The claim that useful structures 'reside in specific architectural components' is not uniquely supported, because in the selective-transfer setup described in §4.1 'the entire model is fine-tuned' after initialization. A comparison such as STACK attention-only 98.9 ± 0.8 versus STACK full-transfer 55.2 ± 39.3 on HAYSTACK (Table 4) is equally consistent with the hypothesis that the pretrained MLP weights are a worse optimization starting point than random MLP weights, and that the attention weights themselves carry little task-specific structure. The same confound affects Table 7, where IDENTITY attention-only reaches 91.3 ± 10.1 on SORTING while IDENTITY full transfer reaches only 19.9 ± 0.5. To localize the effect, the authors should add controls where the non-transferred components are frozen rather than fine-tuned, or where each component is replaced individually by random weights while the others are held fixed, and they should report performance at initialization and during early fine-tuning.
  2. [§5, Table 1] The composition experiment lacks random-component controls. The table compares SET (attention only), ECA (MLPs only), and SET-attention + ECA-MLPs, but there is no row such as SET-attention + random-MLPs or random-attention + ECA-MLPs. Without these controls, the strong performance of the combined model on REVERSED ADDITION (82.9 ± 16.9) relative to SET attention alone (54.4 ± 10.4) does not uniquely establish that the ECA MLP weights contribute transferable structure; it could result from a more favorable random-like MLP initialization. Conversely, on ADDITION the combined model (80.3 ± 13.9) is not better than SET attention alone (81.1 ± 12.2). The authors should add random-component controls and also test at least one additional pair of pretraining tasks before claiming that structures 'can be composed' generally.
  3. [§3, Figure 2 and Table 3] The text states that 'procedurally pretrained models can largely outperform randomly initialized ones for every downstream task,' but Table 3 contains many configurations where procedural pretraining hurts relative to random initialization, e.g., 4-DYCK on ADDITION (52.7 ± 0.3 vs. 59.1 ± 7.0), STACK on SORTING (21.3 ± 0.6 vs. 82.7 ± 11.6), and 4-DYCK on REVERSED ADDITION (35.7 ± 2.5 vs. 76.4 ± 23.2). The paper should explicitly acknowledge that the beneficial effects are selective and task-dependent, and should avoid presenting 'best per task' cells as if they were representative of all procedural pretraining. Because the diagnostic tasks were chosen from the same benchmark family on which the positive results are demonstrated, the selection of favorable pretraining tasks per diagnostic should be justified as hypothesis-driven rather than post hoc.
  4. [§4.2, Table 8] The perturbation analysis applies noise or shuffling to the 'best transfer configurations,' which differ across tasks: attention-only for HAYSTACK, full transfer for ADDITION and SORTING, and so on. Differences in sensitivity across tasks may therefore reflect which components were transferred rather than the nature of the learned skill. For example, SORTING's robustness to shuffling (98.7 → 94.2) is measured on a full-transfer model, whereas HAYSTACK's sensitivity (98.9 → 17.2) is measured on an attention-only transfer. The authors should either apply the perturbations to the same transfer configuration across all tasks or explicitly discuss this confound and report both full- and selective-transfer perturbation results.
minor comments (5)
  1. [Appendix B.3] There are several typographical errors: 'Switzerlad' in the author affiliation, 'autoregressivley' in the SORTING description, 'occurance' in the STACK description, 'two-thrids' in the STACK data generation, and 'afforementioned' in the downstream training section.
  2. [Figure 2] The main-text figures omit error bars, and the caption says full statistics are in Appendix C; for a paper whose claims often rest on differences that are small relative to the reported standard deviations (e.g., REVERSED ADDITION in Table 6), the main figures should show at least some measure of variability.
  3. [§4.1, Tables 4–7] The claim that 'attention layers often carry the most transferable information' is nuanced by Table 6: on REVERSED ADDITION, MLP-only transfer frequently outperforms attention-only transfer (e.g., STACK MLP-only 74.4 ± 24.7 vs. attention-only 42.1 ± 8.1). The text should acknowledge this more explicitly to avoid overgeneralizing the attention-centric conclusion.
  4. [Appendix C, Table 8] The caption of Table 8 states that pretrained models were 'selected based on best individual performance per task,' but the choice for REVERSED ADDITION is 8-DYCK SHUFFLE (90.1 ± 5.9) even though ECA achieves 91.1 ± 16.1 in Table 3; the selection rule should be stated precisely, including how ties or overlapping confidence intervals were resolved.
  5. [§2, weight transfer paragraph] The description of embedding transfer is incomplete for ECA: Appendix B.1 explains that ECA replaces token embeddings with a linear projection, and that 'averaged embeddings' are used for transfer, but the main text's description of resetting to 'average pretrained vector' should reference this special case for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: modular-structure claims rest on empirical transfer and perturbation ablations, not on fitted predictions or self-citation chains.

full rationale

The derivation chain is empirical rather than definitional: pretraining on procedural data, then fine-tuning on fixed diagnostic tasks, with comparisons against random initialization, full-weight transfer, selective component transfer, and weight perturbations. The central claims (distinct inductive structures, attention/MLP residence, composability) are supported by measured accuracies in Figures 2-4 and Table 1; none of these measurements is mathematically forced by how a variable is defined. The 'residence' of structure is operationalized through selective transfer, and although full fine-tuning of transferred components means the attribution to attention vs MLP is an experimental inference rather than a controlled localization, that is a validity limitation, not self-definitionality: the conclusion is not identical to the operational definition. The perturbation experiments (noise and shuffling) provide an independent check that precise weights matter, so the claims are not merely restatements of the pretraining loss. The paper's self-citations (Teney et al. 2024, 2025) appear only in related-work context about hard inductive biases and are not load-bearing for the new results. The paper explicitly acknowledges it lacks a first-principle explanation of why specific data help specific tasks, which is consistent with an honest, non-circular empirical study. The mild concern that the best pretraining task for each diagnostic is selected from the same benchmark is a task-selection issue, not a reduction of the conclusion to the input.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a set of manually chosen procedural tasks, diagnostic tasks, and tiny architectures. No mathematical derivation is provided and no invented entities are postulated. The evidence is empirical and comparative, with several hand-set hyperparameters.

free parameters (4)
  • Model scale and hidden size = 16 (2 layers, 4 heads) for most tasks; 64 for language modeling; 512 for multiplication
    All modular-structure findings are established only at this tiny scale; larger models may behave differently.
  • k-DYCK bracket count and opening probability = k in {4, 8, 16}; popen = 0.49 or 0.50
    The specific formal languages and generation probabilities determine which skills are instilled.
  • Curriculum maximum sequence length = 20 for STACK, IDENTITY, SET
    Sets the difficulty and token budget of the pretraining tasks.
  • Diagnostic task sizes = k=30 haystack, n=5 addition and multiplication, n=10 reversed addition and sorting
    All capability conclusions are conditional on these small benchmark scales.
assumptions (4)
  • domain assumption Random initialization with the same architecture and fine-tuning is an adequate baseline
    All improvements are measured relative to this baseline in Section 3 Setup.
  • domain assumption Per-layer weight shuffling preserves weight-magnitude distributions while destroying structure, isolating magnitude-only benefits
    Used in Section 4.2 to argue pretraining creates precise structure; the shuffling operation is not fully specified.
  • domain assumption Small two-layer transformers are representative vehicles for studying algorithmic reasoning mechanisms
    All modular-structure findings come from hidden-size-16 models; scaling behavior is untested.
  • domain assumption Diagnostic tasks such as needle-in-a-haystack, addition, and sorting measure the intended algorithmic capabilities and transfer cleanly from pretraining
    The attribution of skills to specific pretraining rules depends on task validity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning." pith.science (2026). https://pith.science/paper/ZNEB7KKZ

@misc{pith2026250522308,
  author       = {Pith},
  title        = {Pith review of: Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZNEB7KKZ}},
  note         = {Machine review of arXiv:2505.22308}
}
read the original abstract

Pretraining on large, semantically rich datasets is key for developing language models. Surprisingly, recent studies have shown that even synthetic data, generated procedurally through simple semantic-free algorithms, can yield some of the same benefits as natural language pretraining. It is unclear what specific capabilities such simple synthetic data instils in a model, where these capabilities reside in the architecture, and how they manifest within its weights. In this short paper, we identify several beneficial forms of procedural data, together with specific algorithmic reasoning skills that improve in small transformers. Our core finding is that different procedural rules instil distinct but complementary inductive structures in the model. With extensive ablations and partial-transfer experiments, we discover that these structures reside in different parts of the model. Attention layers often carry the most transferable information, but some pretraining rules impart useful structure to MLP blocks instead. Most interestingly, the structures induced by multiple rules can be composed to jointly reinforce multiple capabilities. These results suggest an exciting possibility of disentangling the acquisition of knowledge from reasoning in language models, with the goal of improving their robustness and data efficiency.

Figures

Figures reproduced from arXiv: 2505.22308 by the authors.

Figure 1
Figure 1. (a) We pretrain small transformers on various forms of procedural data, then fine-tune them on a series of diagnostic [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Models pretrained on carefully-chosen procedural data (blue) significantly improve specific downstream diagnostic [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Results of the selective transfer of procedurally-pretrained weights. Each color is a different type of transfer: [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Relative downstream improvement using pretrained weights with pertur [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Learned replacement non-linearities show transformers are rarely optimal for algorithmic tasks, with benefits that are task-specific, while language/code gains are smaller and more transferable.

Reference graph

Works this paper leans on

36 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [1]

    Transferring inductive biases through knowledge distillation

    Abnar, S., Dehghani, M., and Zuidema, W. Transferring inductive biases through knowledge distillation. arXiv preprint arXiv:2006.00555,

    arXiv 2006

  2. [5]

    REVERSED ADDITION

    Accuracy is computed at the token level on the result digits. REVERSED ADDITION . We again use the implementation by Zhong & Andreas (2024). This task evaluates a model’s ability to perform multi-step arithmetic by adding two length n integers represented as sequences of digits. To simplify the positional dependencies, both the inputs and the output are r...

    work page 2024

  3. [7]

    Meta-Learning Neural Mechanisms rather than Bayesian Priors

    Goodale, M., Mascarenhas, S., and Lakretz, Y . Meta-learning neural mechanisms rather than bayesian priors.arXiv preprint arXiv:2503.16048,

    work page Pith review arXiv

  4. [9]

    General Intelligence Requires Reward-based Pretraining

    Han, S., Pari, J., Gershman, S. J., and Agrawal, P. General reasoning requires learning to reason from the get-go. arXiv preprint arXiv:2502.19402,

    work page Pith review arXiv

  5. [10]

    edu/˜johnhew/vocab-expansion.html

    URLhttps://www.cs.columbia. edu/˜johnhew/vocab-expansion.html. Hu, M. Y ., Petty, J., Shi, C., Merrill, W., and Linzen, T. Between circuits and chomsky: Pre-pretraining on formal languages imparts linguistic biases. arXiv preprint arXiv:2502.19249,

    arXiv

  6. [11]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

    arXiv 2001

  7. [12]

    McCoy, R. T. and Griffiths, T. L. Modeling rapid language learning by distilling bayesian priors into artificial neural networks. arXiv preprint arXiv:2305.14701,

    arXiv

  8. [13]

    P., Grabocka, J., and Hutter, F

    M¨uller, S., Hollmann, N., Arango, S. P., Grabocka, J., and Hutter, F. Transformers can do bayesian inference. arXiv preprint arXiv:2112.10510,

    arXiv

Show all 36 references
  1. [14]

    M., Laina, I., Rupprecht, C., Inoue, N., Yokota, R., and Kataoka, H

    Nakamura, R., Tadokoro, R., Yamada, R., Asano, Y . M., Laina, I., Rupprecht, C., Inoue, N., Yokota, R., and Kataoka, H. Scaling backwards: Minimal synthetic pre-training? arXiv preprint arXiv:2408.00677,

    arXiv

  2. [15]

    and Jurafsky, D

    Papadimitriou, I. and Jurafsky, D. Injecting structural hints: Using language models to study inductive biases in language learning. arXiv preprint arXiv:2304.13060,

    arXiv

  3. [16]

    How does code pretraining affect language model task performance? arXiv preprint arXiv:2409.04556,

    Petty, J., van Steenkiste, S., and Linzen, T. How does code pretraining affect language model task performance? arXiv preprint arXiv:2409.04556,

    arXiv

  4. [18]

    R., Talupuru, D., Locatelli, A., Kirk, R., Rockt ¨aschel, T., Grefenstette, E., and Bartolo, M

    Ruis, L., Mozes, M., Bae, J., Kamalakara, S. R., Talupuru, D., Locatelli, A., Kirk, R., Rockt ¨aschel, T., Grefenstette, E., and Bartolo, M. Procedural knowledge in pretraining drives reasoning in large language models. arXiv preprint arXiv:2411.12580,

    arXiv

  5. [19]

    and Kolter, J

    Trockman, A. and Kolter, J. Z. Mimetic initialization of self-attention layers. arXiv preprint arXiv:2305.09828,

    arXiv

  6. [20]

    Visual pre-training for navigation: What can we learn from noise?arXiv preprint arXiv:2207.00052,

    Wang, Y ., Ko, C.-Y ., and Agrawal, P. Visual pre-training for navigation: What can we learn from noise?arXiv preprint arXiv:2207.00052,

    arXiv

  7. [21]

    Pre-training with synthetic data helps offline reinforcement learning

    Wang, Z., Wang, C., Dong, Z., and Ross, K. Pre-training with synthetic data helps offline reinforcement learning. arXiv preprint arXiv:2310.00771,

    arXiv

  8. [22]

    Initializing models with larger ones

    Xu, Z., Chen, Y ., Vishniakov, K., Yin, Y ., Shen, Z., Darrell, T., Liu, L., and Liu, Z. Initializing models with larger ones. arXiv preprint arXiv:2311.18823,

    arXiv

  9. [23]

    Data mixing laws: Optimizing data mixtures by predicting language modeling performance

    Ye, J., Liu, P., Sun, T., Zhan, J., Zhou, Y ., and Qiu, X. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint arXiv:2403.16952,

    arXiv

  10. [24]

    A., and Pavlick, E

    6 Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning Zhang, E., Lepori, M. A., and Pavlick, E. Instilling inductive biases with subnetworks. arXiv preprint arXiv:2310.10899,

    arXiv

  11. [25]

    A., Liu, N., He, S., Karbasi, A., Zappala, E., and van Dijk, D

    Zhang, S., Patel, A., Rizvi, S. A., Liu, N., He, S., Karbasi, A., Zappala, E., and van Dijk, D. Intelligence at the edge of chaos. arXiv preprint arXiv:2410.02536,

    arXiv

  12. [26]

    But recent results also question the value of the data, showing that some benefits of pretraining are attributable to the optimization objective more than the actual data

    of pretraining data are empirically critical for the performance of large language models. But recent results also question the value of the data, showing that some benefits of pretraining are attributable to the optimization objective more than the actual data. Balestriero & ...

    2024

  13. [27]

    and burstiness to enable in-context learning (Chan et al., 2022). Computer code is empirically very effective as pretraining data for LLMs, as it improves their abilities for compositional generalization and math-related tasks (Aryabumi et al., 2024; Petty et al., 2024). This ...

    2025

  14. [28]

    Partial transfer from pretrained transformers

    and noise-based images (Baradad et al., 2022). Partial transfer from pretrained transformers. Zhang et al. (2023) and (Xu et al.,

    2023

  15. [29]

    mimetic initialization

    showed that copying subsets of pretrained weights could transfer specific capabilities. Abnar et al. (2020) used knowledge distillation to transfer the inductive biases of one architecture into another. The “mimetic initialization” of self-attention (Trockman & Kolter,

    2020

  16. [30]

    Goodale et al

    and Solomonoff Induction (Grau-Moya et al., 2024). Goodale et al. (2025) showed that well-chosen formal languages can teach complex mechanisms (e.g. counters) to a sequence model. Pretraining can generally be seen as a soft inductive bias for subsequent fine-tuning. But there ...

    2025

  17. [31]

    As stated by Hu et al

    Our opening token probability is (popen = 0.50). As stated by Hu et al. (2025), the final sequence may be invalid due to truncation, but we also did not see any negative consequences of this. STACK. Consists of sequences that simulate stack-based operations, where the first pa...

    2025

  18. [33]

    We adopt the training configuration from Zhang et al. (2024). Models are pretrained for next-token prediction using data generated from ECA rule

    2024

  19. [34]

    This setup effectively simulates infinite data

    In each epoch, a fresh dataset is generated from a new random initial state. This setup effectively simulates infinite data. Training proceeds for up to 10,000 epochs with early stopping based on validation loss. We use a batch size of 64 (60 time steps, 100 spatial dimensions...

    2024

  20. [100]

    For example, given an input sequence 6 3 5 and separator |, the expected output is 3 5 6

    The model receives the input sequence followed by a separator token and is trained to output the sorted version of the input immediately after the separator. For example, given an input sequence 6 3 5 and separator |, the expected output is 3 5 6 . The model is trained autoreg...

    2023

  21. [110]

    (2024), where data is procedurally generated from Elementary Cellular Automata (ECA) using Rule 110, a Class IV rule known for its complex, Turing- complete behavior

    We adopt the training setup and codebase released by Zhang et al. (2024), where data is procedurally generated from Elementary Cellular Automata (ECA) using Rule 110, a Class IV rule known for its complex, Turing- complete behavior. To enable next-token prediction over binary ...

    2024

  22. [2019]

    and Tsuruoka, Y

    Ri, R. and Tsuruoka, Y . Pretraining with artificial language: Studying transferable knowledge in language models.arXiv preprint arXiv:2203.10326,

    arXiv

  23. [2020]

    To code, or not to code? exploring impact of code in pre-training

    Aryabumi, V ., Su, Y ., Ma, R., Morisot, A., Zhang, I., Locatelli, A., Fadaee, M.,¨Ust¨un, A., and Hooker, S. To code, or not to code? exploring impact of code in pre-training. arXiv preprint arXiv:2408.10914,

    arXiv

  24. [2021]

    Procedural image programs for representation learning

    Baradad, M., Chen, C.-F., Wulff, J., Wang, T., Feris, R., Torralba, A., and Isola, P. Procedural image programs for representation learning. arXiv preprint arXiv:2211.16412,

    arXiv

  25. [2022]

    and Kempe, J

    Charton, F. and Kempe, J. Emergent properties with repeated examples. arXiv preprint arXiv:2410.07041,

    arXiv

  26. [2023]

    Doge: Domain reweighting with generalization estimation

    Fan, S., Pagliardini, M., and Jaggi, M. Doge: Domain reweighting with generalization estimation. arXiv preprint arXiv:2310.15393,

    arXiv

  27. [2024]

    Learning to see by looking at noise

    Baradad, M., Wulff, J., Wang, T., Isola, P., and Torralba, A. Learning to see by looking at noise. arXiv preprint arXiv:2106.05963,

    arXiv

  28. [2025]

    K., Mattern, C., Aitchison, M., et al

    Grau-Moya, J., Genewein, T., Hutter, M., Orseau, L., Del ´etang, G., Catt, E., Ruoss, A., Wenliang, L. K., Mattern, C., Aitchison, M., et al. Learning universal predictors. arXiv preprint arXiv:2401.14953,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.