Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

Extrapolation by Association: Length Generalization Transfer in Transformers

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Co-training with a longer related task transfers length generalization to a short main task.

desk verdict Genuine new phenomenon, but the best-of-five-seed reporting makes the headline effect look stronger than the data support. read the letter →

arxiv 2506.09251 v2 pith:4C7XPL7E submitted 2025-06-10 cs.CL cs.AI

Ziyang Cai , Nayoung Lee , Avi Schwarzschild , Samet Oymak , Dimitris Papailiopoulos This is my paper
classification cs.CLcs.AI
keywords lengthgeneralizationtasktransfermultitasktrainingtransformerinterpretabilityattentionheadsharingout-of-distributionpretrainedlanguagemodelsalgorithmictasks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that length generalization—a transformer's ability to handle inputs beyond its training lengths—can be passed from one task to another. The authors train a main task on short examples together with an auxiliary task on longer examples and report that the main task then succeeds at lengths it never saw, inheriting the auxiliary task's generalization range. This transfer appears in arithmetic, string manipulation, and maze navigation. The paper also finds that natural-language pretraining acts like an implicit auxiliary task: later pretraining checkpoints finetune into better length generalization on synthetic tasks. A control result anchors the claim: unrelated task pairs such as reverse add with copy-first-op show no transfer, suggesting relatedness is what carries the effect.

What carries the argument

The carrying object is task association realized through joint multitask training: at each step a task is sampled uniformly from a group containing a short main task and longer auxiliary tasks, so the model is forced to learn the auxiliary task at lengths the main task has never seen. The quantitative instrument is the generalization gap, the average difference between the main and auxiliary accuracy-versus-length curves normalized to [0,1], with smaller values meaning stronger transfer; the paper shows transfer is strongest when the auxiliary-to-main training-length ratio lies between 0.5 and 2. The mechanistic instrument is the attention-head mean-ablation map, a matrix recording the accuracy drop when each head's output is replaced by its mean activation; similarity of these maps between tasks tracks the generalization gap over training, identifying shared head usage as the correlate of transfer. RoPE, the rotary position encoding, is the positional scheme whose presence the paper finds supports transfer, with NoPE giving weaker transfer under the same task settings.

What would settle it

Pair reverse add (short, length 16) with a longer auxiliary task that is algorithmically related but uses a different surface format—for example, addition in a different numeral base—and check whether the main task still extrapolates; if it does not, the phenomenon depends on format overlap rather than structural relatedness, and if it does, the relatedness claim generalizes beyond the paper's exact formats.

Watch

Extended reading notes

Core claim

The paper's discovery is that extrapolation by association is real in transformers: co-training a task at a short length with a related task at a longer length makes the short task extrapolate to the longer task's range, even though the main task never saw examples at those lengths. The authors demonstrate this in three domains—reverse addition inheriting range from no-carry, carry-only, reverse-subtract, or chain-of-thought multiplication; string copy inheriting from multi-query associative recall; capitalize-reverse inheriting from capitalize and reverse; and maze DFS-trace and shortest-path, where transfer works in both directions. They further find that pretraining on natural language provides the same kind of transfer, with extrapolation improving steadily as pretraining progresses, and that rotary position encodings produce the effect more strongly than no positional encoding. Mechanistically, the paper reports that successful transfer coincides with the two tasks sharing attention heads: the per-head mean-ablation importance maps of the tasks converge as the generalization gap closes, while unrelated control pairs do not show this convergence. This is offered as evidence that length generalization transfer corresponds to reuse of the same internal computation across tasks.

Load-bearing premise

The results assume the observed improvement comes from transferring algorithmic structure between related tasks, not from the auxiliary task sharing superficial input-format features, and this assumption rests on only two unrelated control pairs.

Editorial extensions

If this is right

  • A practical recipe for improving length generalization is to co-train a short target task with a related task at the desired longer length, rather than changing the model architecture.
  • Task relatedness is a necessary condition for the effect: unrelated auxiliary tasks should not be expected to extend a main task's generalization range.
  • Pretrained language models carry reusable computational scaffolding from natural language that transfers to novel synthetic tasks, so finetuning a later pretraining checkpoint should yield better extrapolation.
  • Attention-head mean-ablation similarity can serve as an observable signature that transfer is occurring, allowing early detection or monitoring of successful multitask length generalization.
  • The auxiliary task's length should be within roughly a factor of two of the main task's training length; beyond that ratio, the model tends to overfit the length difference and transfer weakens.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If shared attention circuits are the mechanism, transfer should occur between tasks that share a subprocedure but differ in surface format; this could be tested by pairing reverse addition with digit-wise addition in another base or with a different delimiter format.
  • The length-ratio band of 0.5–2 suggests a curriculum strategy the authors do not explicitly propose: to reach very long lengths, one could stage auxiliary tasks, each new task within a factor of two of the previous, rather than jumping directly from short to very long.
  • The seed-to-seed instability the paper reports implies that in practice transfer is not guaranteed by the training setup alone; deployment would likely require multiple seeds or a checkpoint-selection rule, which the paper does not provide.
  • The pretraining result suggests a testable scaling prediction: models pretrained on corpora with more long-range structure should show progressively better downstream length generalization on synthetic tasks, even when the tasks share no surface format with the pretraining data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies whether length generalization can transfer across related tasks in transformers. In their setup, a 'main' task is trained only on short inputs while a related 'auxiliary' task is trained on longer inputs; the authors report that joint training lets the main task extrapolate to lengths it never saw, often matching the auxiliary task's generalization range. The phenomenon is demonstrated on three task families (reverse addition with related arithmetic auxiliaries, string transformations, and maze navigation), with two control pairs showing no transfer when tasks are unrelated. The authors further report that finetuning SmolLM-360M checkpoints at increasing pretraining steps improves OOD accuracy on reverse add and shortest path, and they provide attention-based analyses suggesting that transfer correlates with shared attention-head usage across tasks.

Significance. If the central claim holds, the paper identifies a new route to length generalization: rather than architectural changes or data augmentation, a model can inherit extrapolation behavior from a longer, related training task. The study is well positioned in the length-generalization literature and has practical implications for multitask training and for understanding why pretrained models generalize. Strengths include on-the-fly data generation with exact-match evaluation, three diverse task families, control pairs, public pretrained checkpoints, and an explicit acknowledgment of training instability. The main weaknesses are the best-seed presentation of the headline curves, the absence of an equivalent-compute baseline for the pretraining trend, and the informal nature of the mechanistic correlation analysis. The phenomenon is plausible and interesting, but the current evidence does not yet establish how typical the transfer effect is.

major comments (3)
  1. [Section 6.2 and Figures 3–7] The main quantitative evidence is presented through best-of-five-seed curves, and this selection rule is load-bearing for the central claim. The captions of Figures 3–7 state that 'best-performing runs are shown in bold,' while Section 6.2 states that 'not all random seeds exhibit successful length generalization transfer' and Figure 11 documents substantial run-to-run variability. Because both the co-trained conditions and the single-task baselines are selected by the same rule, the headline comparison can reflect the top order statistic of a noisy process rather than a typical training outcome. Please report the full seed distribution (individual runs, or median with spread) for every condition, quantify the fraction of seeds that achieve transfer in each task group, and state explicitly whether the conclusion holds for median runs as well as best runs.
  2. [Section 5 and Figure 9] The pretraining evidence is a checkpoint trend without a matched equivalent-compute baseline. Figure 9 shows that later SmolLM checkpoints finetune to higher OOD accuracy on reverse add and shortest path, but there is no from-scratch, randomly initialized, or early-step model finetuned for the same number of steps and with the same data. Without such a control, the monotonic improvement could reflect differences in finetuning dynamics or optimization landscapes rather than reusable scaffolding acquired during pretraining. Please add an equivalent-compute from-scratch baseline and report seed variation for the pretrained-model experiments, or substantially soften the causal wording in Section 5.
  3. [Section 6.4 and Figures 18–19] The mechanistic claim is stated more strongly than the evidence supports. The text says the attention similarity metrics correlate with the generalization gap 'in most cases,' but Figure 18 shows that the raw attention-matrix difference does not track the gap for the string task pairs, while the head-ablation-map difference does. Figure 19 also shows mixed behavior for the control pair. The analysis is an informal visual comparison across checkpoints without correlation coefficients or statistical summaries. Please either report quantitative correlations for the measures that actually track the gap, or explicitly present the circuit-sharing results as a qualitative, initial observation rather than a confirmed correlation.
minor comments (6)
  1. [Figure captions (Figures 3–7)] Please define the criterion for 'best-performing runs' and clarify that the bold curves correspond to a single selected seed; this information is essential for interpreting the plots.
  2. [Section 6.1 and Figure 10] The generalization gap is described only in the caption; please give an explicit formula in the main text and label the axes of the heatmap clearly so the reader knows whether rows or columns correspond to main versus auxiliary length.
  3. [Section 2, Related Work] Please clarify the precise difference from task hinting [Awasthi and Gupta, 2023], which also reports improved length generalization from auxiliary tasks; the current text does not explain why the longer-auxiliary setup here is a distinct phenomenon.
  4. [Section 4.4] There is a typo: 'or arithmetic' should be 'For arithmetic,' and 'with unrelated task' should agree in number with the plural examples ('unrelated tasks').
  5. [Section 6.3] The phrase 'significantly weaker transfer' is not supported by a statistical test; please use 'substantially weaker' or provide a significance test for the RoPE versus NoPE comparison.
  6. [Appendix B.2, Table 2] The notation 'n×3CoT multiply' is confusing because the example uses a three-digit multiplier (502); please define the notation or rename the task to clarify the length parameter.

Circularity Check

1 steps flagged · score 4.0 of 10

Central transfer claim is empirically independent, but the Section 6.1 ablation concluding that transfer is strongest at length ratios 0.5-2 uses a generalization-gap metric whose definition already forces curve alignment when training ranges are close.

  1. self definitional [Section 6.1 and Figure 10 caption]
    "The generalization gap is defined as the average difference in accuracy between the main and auxiliary tasks across evaluation lengths, normalized to the range [0, 1]. A lower value indicates better transfer. ... we find that the transfer effect is most effective when the ratio between the auxiliary and main lengths is between 0.5 and 2."

    Transfer is operationalized through the gap between the main and auxiliary accuracy curves. When the main and auxiliary training lengths are close, the two tasks are in-distribution and out-of-distribution at the same evaluation points, so the average curve difference is small by construction, including the equal-length diagonal where no longer auxiliary range exists to transfer. The dark diagonal band in Figure 10 therefore largely restates the definition of the metric rather than independently measuring transfer. This is a localized self-definitional step in the length-ratio ablation; the central transfer claim in Figures 3-7 is measured directly from the main task's held-out accuracy and is not reduced to this gap metric.

full rationale

The central claim — that co-training on a longer related auxiliary task lets a shorter main task extrapolate — is tested by direct exact-match accuracy on held-out lengths for the main task with and without the auxiliary, so it is not defined in terms of a fitted parameter, a self-citation, or a prior uniqueness theorem. The reverse-add format is adopted from Lee et al. 2023, a same-group citation, but that citation supplies only the task format and not the transfer conclusion. The circuit-sharing analysis in Section 6.4 is correlational rather than a derivation, and the paper's own Section 6.2 documents seed instability; the best-seed plotting rule is a reproducibility/statistical concern, not circularity. The one definitional issue is the Section 6.1 generalization gap: because the gap is literally the average main-minus-aux accuracy difference, the finding that transfer is 'most effective' at length ratios 0.5-2 is substantially built into the metric when the two tasks share similar training ranges. The main transfer result remains independently supported, so the paper is only partially circular.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No new physical or architectural entities are introduced. The phrase 'reusable computational scaffolding' is a descriptive label for a hypothesized internal reuse, not an entity with independent falsifiable predictions. The central claim rests on empirical observations and on the domain assumptions listed above.

assumptions (4)
  • domain assumption Tasks grouped as 'related' in Section 3 share algorithmic procedures sufficient for computational reuse.
    The transfer phenomenon is only observed for these task groups; the paper relies on this relatedness to distinguish transfer from mere length exposure.
  • domain assumption The test set of 1024 examples per length and the on-the-fly data sampler provide a stable estimate of model behavior.
    All accuracy curves are based on this fixed test set and sampled training data, with the dataset held fixed across seeds.
  • domain assumption The attention-head mean-ablation map difference reflects functional overlap in the computation used by the two tasks.
    Section 6.4 interprets ablation-map similarity as evidence of shared circuits; no causal intervention establishes that the shared heads are the ones responsible for transfer.
  • domain assumption Successive SmolLM-360M checkpoints differ only in the amount of pretraining, so the improvement in transfer can be attributed to pretraining progress.
    Section 5 attributes the trend to natural language pretraining, but no control for other differences between checkpoints is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Extrapolation by Association: Length Generalization Transfer in Transformers." pith.science (2026). https://pith.science/paper/4C7XPL7E

@misc{pith2026250609251,
  author       = {Pith},
  title        = {Pith review of: Extrapolation by Association: Length Generalization Transfer in Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4C7XPL7E}},
  note         = {Machine review of arXiv:2506.09251}
}
read the original abstract

Transformer language models have demonstrated impressive generalization capabilities in natural language domains, yet we lack a fine-grained understanding of how such generalization arises. In this paper, we investigate length generalization--the ability to extrapolate from shorter to longer inputs--through the lens of \textit{task association}. We find that length generalization can be \textit{transferred} across related tasks. That is, training a model with a longer and related auxiliary task can lead it to generalize to unseen and longer inputs from some other target task. We demonstrate this length generalization transfer across diverse algorithmic tasks, including arithmetic operations, string transformations, and maze navigation. Our results show that transformer models can inherit generalization capabilities from similar tasks when trained jointly. Moreover, we observe similar transfer effects in pretrained language models, suggesting that pretraining equips models with reusable computational scaffolding that facilitates extrapolation in downstream settings. Finally, we provide initial mechanistic evidence that length generalization transfer correlates with the re-use of the same attention heads between the tasks. Together, our findings deepen our understanding of how transformers generalize to out-of-distribution inputs and highlight the compositional reuse of inductive structure across tasks.

Figures

Figures reproduced from arXiv: 2506.09251 by the authors.

Figure 1
Figure 1. Trained separately, each task fails to generalize to [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the tasks used in our length generalization transfer experiments, spanning [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Length generalization results for addition-related task groups. The main task is [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (17 more)
Figure 4
Figure 4. Figure 4: Performance plots for string tasks. When trained alone (b, d), models fail to generalize [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: 8 × 8 mazes with number of nodes equal to 16, 32, and 64. We define length generalization as the ability to generalize to mazes with a higher number of nodes. We consider two maze tasks: (1) shortest path , where the model outputs the shortest path from start to end no…
Figure 6
Figure 6. Figure 6: Performance plots for maze tasks. Co-training [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Length generalization results for maze task group with reversed task roles. Co [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Control tasks for (a) addition and (b) string operations. These unrelated task pairs fail to [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Finetuning at different SmolLM-360M checkpoints reveals that length generalization [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: (a) The generalization gap is defined as the average difference in accuracy between the main and auxiliary tasks across evaluation lengths, normalized to the range [0, 1]. A lower value indi￾cates better transfer. (b) Generalization gap across different combinations o…
Figure 11
Figure 11. Figure 11: Training curves for the reverse add when co-trained with no carry and carry only . Accuracy in the transfer region (length 17–32) fluctuates significantly, illustrat￾ing unstable training dynamics in length generalization transfer. 9 [PITH_FULL_IMAGE:figures/full_fig…
Figure 12
Figure 12. Figure 12: Length generalization transfer results using NoPE model, under the same task settings. [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]
Figure 13
Figure 13. Figure 13: Comparison of generalization gap across several task groups shows that NoPE leads to [PITH_FULL_IMAGE:figures/full_fig_p011_13.png]
Figure 14
Figure 14. Figure 14: We use the arithmetic task group reverse add & reverse subtraction for the analysis. Evolution of generalization gap, attention matrix difference, and attention head mean￾ablation map difference across checkpoints. All three metrics closely align, suggesting successfu…
Figure 15
Figure 15. Figure 15: Additional results for arithmetic and string (copy) task groups. Each row shows perfor [PITH_FULL_IMAGE:figures/full_fig_p017_15.png]
Figure 16
Figure 16. Figure 16: Length generalization transfer with the pretrained model SmolLM-360M. (a–c): Arith [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]
Figure 17
Figure 17. Figure 17: Mean-ablation maps for reverse add and reverse subtract across training checkpoints. Each (i, j) entry indicates the accuracy drop after mean-ablating head i in layer j. Similar ablation maps suggest that both tasks rely on overlapping computational circuits. A.3.2 Fu…
Figure 18
Figure 18. Figure 18: Circuit sharing results for string task pairs. The attention matrix difference does not [PITH_FULL_IMAGE:figures/full_fig_p020_18.png]
Figure 19
Figure 19. Figure 19: Circuit sharing results for arithmetic tasks. Both attention matrix and head ablation map [PITH_FULL_IMAGE:figures/full_fig_p021_19.png]
Figure 20
Figure 20. Figure 20: Detailed example of maze data format. Each node is a random number selected from [PITH_FULL_IMAGE:figures/full_fig_p022_20.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Learned replacement non-linearities show transformers are rarely optimal for algorithmic tasks, with benefits that are task-specific, while language/code gains are smaller and more transferable.

  2. Resizing the giants: How modelling adiabatic interiors impacts predicted planetary radii

    astro-ph.EP 2026-03 unverdicted novelty 7.0 of 10

    Numerical treatment of the adiabatic gradient changes predicted giant-planet radii by up to 3.4 percent depending on the temperature-equation form and derivative method used.

Reference graph

Works this paper leans on

50 extracted references · 9 canonical work pages · cited by 2 Pith papers

  1. [1]

    Task generalization with autoregressive compositional structure: Can learning from d tasks generalize to d^t tasks? arXiv preprint arXiv:2502.08991, 2025

    Amirhesam Abedsoltan, Huaqing Zhang, Kaiyue Wen, Hongzhou Lin, Jingzhao Zhang, and Mikhail Belkin. Task generalization with autoregressive compositional structure: Can learning from d tasks generalize to d^t tasks? arXiv preprint arXiv:2502.08991, 2025

    arXiv 2025

  2. [2]

    On provable length and compositional generalization, 2024

    Kartik Ahuja and Amin Mansouri. On provable length and compositional generalization, 2024. URL https://arxiv.org/abs/2402.04875

    arXiv 2024

  3. [3]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024. URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md

    2024

  4. [4]

    Smollm - blazingly fast and remarkably powerful, 2024

    Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Leandro von Werra, and Thomas Wolf. Smollm - blazingly fast and remarkably powerful, 2024

    2024

  5. [5]

    Exploring length generalization in large language models

    Cem Anil, Yuhuai Wu, Anders Johan Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Venkatesh Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length generalization in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022....

    work page 2022

  6. [6]

    Zoology: Measuring and improving recall in efficient language models, 2023

    Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language models, 2023. URL https://arxiv.org/abs/2312.04927

    arXiv 2023

  7. [7]

    Improving length-generalization in transformers via task hinting, 2023

    Pranjal Awasthi and Anupam Gupta. Improving length-generalization in transformers via task hinting, 2023. URL https://arxiv.org/abs/2310.00726

    arXiv 2023

  8. [8]

    Round and round we go! what makes rotary positional encodings useful? ArXiv, abs/2410.06205, 2024

    Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Velivckovi'c. Round and round we go! what makes rotary positional encodings useful? ArXiv, abs/2410.06205, 2024

    arXiv 2024

Show all 50 references
  1. [9]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffr...

    2020 arXiv

  2. [10]

    Curve circuits

    Nick Cammarata, Gabriel Goh, Shan Carter, Chelsea Voss, Ludwig Schubert, and Chris Olah. Curve circuits. Distill, 2021. doi:10.23915/distill.00024.006. https://distill.pub/2020/circuits/curve-circuits

    2021 doi

  3. [11]

    Position coupling: Improving length generalization of arithmetic transformers using task structure

    Hanseul Cho, Jaeyoung Cha, Pranjal Awasthi, Srinadh Bhojanapalli, Anupam Gupta, and Chulhee Yun. Position coupling: Improving length generalization of arithmetic transformers using task structure. 2024. URL https://api.semanticscholar.org/CorpusID:273695226

    2024

  4. [12]

    Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang

    Yiran Ding, L. Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending llm context window beyond 2 million tokens. ArXiv, abs/2402.13753, 2024

    2024 arXiv

  5. [13]

    From interpolation to extrapolation: Complete length generalization for arithmetic transformers

    Shaoxiong Duan, Yining Shi, and Wei Xu. From interpolation to extrapolation: Complete length generalization for arithmetic transformers. arXiv preprint arXiv:2310.11984, 2023

    2023 arXiv

  6. [14]

    Location attention for extrapolation to longer sequences

    Yann Dubois, Gautier Dagan, Dieuwke Hupkes, and Elia Bruni. Location attention for extrapolation to longer sequences. arXiv preprint arXiv:1911.03872, 2019

    1911 arXiv

  7. [15]

    Looped transformers for length generalization

    Ying Fan, Yilun Du, Kannan Ramchandran, and Kangwook Lee. Looped transformers for length generalization. arXiv preprint arXiv:2409.15647, 2024

    2024 arXiv

  8. [16]

    Not all llm reasoners are created equal

    Arian Hosseini, Alessandro Sordoni, Daniel Toyama, Aaron Courville, and Rishabh Agarwal. Not all llm reasoners are created equal. arXiv preprint arXiv:2410.01748, 2024

    2024 arXiv

  9. [17]

    Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67: 0 757--795, 2020

    Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67: 0 757--795, 2020

    2020

  10. [18]

    The impact of positional encoding on length generalization in transformers

    Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36, 2024

    2024

  11. [19]

    Teaching arithmetic to small transformers

    Nayoung Lee, Kartik Sreenivasan, Jason D Lee, Kangwook Lee, and Dimitris Papailiopoulos. Teaching arithmetic to small transformers. arXiv preprint arXiv:2307.03381, 2023

    2023 arXiv

  12. [20]

    Self-improving transformers overcome easy-to-hard and length generalization challenges, 2025

    Nayoung Lee, Ziyang Cai, Avi Schwarzschild, Kangwook Lee, and Dimitris Papailiopoulos. Self-improving transformers overcome easy-to-hard and length generalization challenges, 2025. URL https://arxiv.org/abs/2502.01612

    2025 arXiv

  13. [21]

    On the power of convolution-augmented transformer

    Mingchen Li, Xuechen Zhang, Yixiao Huang, and Samet Oymak. On the power of convolution-augmented transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 18393--18402, 2025

    2025

  14. [22]

    Functional interpolation for relative positions improves long context transformers

    Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontanon, Manzil Zaheer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, and Srinadh Bhojanapalli. Functional interpolation for relative positions improves long context transformers. arXiv preprint arXiv:2310.04418, 2023

    arXiv

  15. [23]

    Transformers can do arithmetic with the right embeddings

    Sean McLeish, Arpit Bansal, Alex Stein, Neel Jain, John Kirchenbauer, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, Jonas Geiping, Avi Schwarzschild, et al. Transformers can do arithmetic with the right embeddings. arXiv preprint arXiv:2405.17399, 2024

    2024 arXiv

  16. [24]

    The eos decision and length extrapolation

    Benjamin Newman, John Hewitt, Percy Liang, and Christopher D Manning. The eos decision and length extrapolation. arXiv preprint arXiv:2010.07174, 2020

    2010 arXiv

  17. [25]

    In-context learning and induction heads, 2022

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kam...

    2022 arXiv

  18. [26]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774

    2024 arXiv

  19. [27]

    Yarn: Efficient context window extension of large language models

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. ArXiv, abs/2309.00071, 2023

    2023 arXiv

  20. [28]

    Train short, test long: Attention with linear biases enables input length extrapolation

    Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409, 2021

    2021 arXiv

  21. [29]

    Understanding addition in transformers

    Philip Quirke and Fazl Barez. Understanding addition in transformers. arXiv preprint arXiv:2310.13121, 2023

    2023 arXiv

  22. [30]

    Arithmetic in transformers explained, 2025

    Philip Quirke, Clement Neo, and Fazl Barez. Arithmetic in transformers explained, 2025. URL https://arxiv.org/abs/2402.02619

    2025 arXiv

  23. [31]

    Compositional capabilities of autoregressive transformers: A study on synthetic, interpretable tasks

    Rahul Ramesh, Ekdeep Singh Lubana, Mikail Khona, Robert P Dick, and Hidenori Tanaka. Compositional capabilities of autoregressive transformers: A study on synthetic, interpretable tasks. In Forty-first International Conference on Machine Learning

  24. [32]

    Randomized positional encodings boost length generalization of transformers

    Anian Ruoss, Gr \'e goire Del \'e tang, Tim Genewein, Jordi Grau-Moya, R \'o bert Csord \'a s, Mehdi Bennani, Shane Legg, and Joel Veness. Randomized positional encodings boost length generalization of transformers. arXiv preprint arXiv:2305.16843, 2023

    2023 arXiv

  25. [33]

    Explicitly encoding structural symmetry is key to length generalization in arithmetic tasks

    Mahdi Sabbaghi, George Pappas, Hamed Hassani, and Surbhi Goel. Explicitly encoding structural symmetry is key to length generalization in arithmetic tasks. arXiv preprint arXiv:2406.01895, 2024

    2024 arXiv

  26. [34]

    Multitask prompted training enables zero-shot task generalization

    Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyo...

    2022

  27. [35]

    Positional description matters for transformers arithmetic

    Ruoqi Shen, S \'e bastien Bubeck, Ronen Eldan, Yin Tat Lee, Yuanzhi Li, and Yi Zhang. Positional description matters for transformers arithmetic. arXiv preprint arXiv:2311.14737, 2023

    2023 arXiv

  28. [36]

    Roformer: Enhanced transformer with rotary position embedding, 2023

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. URL https://arxiv.org/abs/2104.09864

    2023 arXiv

  29. [37]

    Interpretability in the wild: a circuit for indirect object identification in gpt-2 small, 2022

    Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small, 2022. URL https://arxiv.org/abs/2211.00593

    2022 arXiv

  30. [38]

    Chi, Tatsunori Hashimoto, O

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus. Emergent abilities of large language models. ArXiv, abs/220...

    2022 arXiv

  31. [39]

    Generating random spanning trees more quickly than the cover time

    David Bruce Wilson. Generating random spanning trees more quickly than the cover time. In Proceedings of the Twenty-Eighth Annual ACM Symposium on Theory of Computing, STOC '96, page 296–303, New York, NY, USA, 1996. Association for Computing Machinery. ISBN 0897917855. doi:10...

    1996

  32. [40]

    Do large language models have compositional ability? an investigation into limitations and scalability, 2024

    Zhuoyan Xu, Zhenmei Shi, and Yingyu Liang. Do large language models have compositional ability? an investigation into limitations and scalability, 2024. URL https://arxiv.org/abs/2407.15720

    2024 arXiv

  33. [41]

    Exploring compositional generalization of large language models

    Haoran Yang, Hongyuan Lu, Wai Lam, and Deng Cai. Exploring compositional generalization of large language models. In Yang (Trista) Cao, Isabel Papadimitriou, Anaelia Ovalle, Marcos Zampieri, Francis Ferraro, and Swabha Swayamdipta, editors, Proceedings of the 2024 Conference o...

    2024

  34. [42]

    From local structures to size generalization in graph neural networks

    Gilad Yehudai, Ethan Fetaya, Eli Meirom, Gal Chechik, and Haggai Maron. From local structures to size generalization in graph neural networks. In International Conference on Machine Learning, pages 11975--11986. PMLR, 2021

    2021

  35. [43]

    Skill-mix: a flexible and expandable family of evaluations for ai models, 2023

    Dingli Yu, Simran Kaur, Arushi Gupta, Jonah Brown-Cohen, Anirudh Goyal, and Sanjeev Arora. Skill-mix: a flexible and expandable family of evaluations for ai models, 2023. URL https://arxiv.org/abs/2310.17567

    2023 arXiv

  36. [44]

    On the out-of-distribution generalization of multimodal large language models, 2024

    Xingxuan Zhang, Jiansheng Li, Wenjing Chu, Junjia Hai, Renzhe Xu, Yuqing Yang, Shikai Guan, Jiazheng Xu, and Peng Cui. On the out-of-distribution generalization of multimodal large language models, 2024. URL https://arxiv.org/abs/2402.06599

    2024 arXiv

  37. [45]

    Can models learn skill composition from examples?, 2025

    Haoyu Zhao, Simran Kaur, Dingli Yu, Anirudh Goyal, and Sanjeev Arora. Can models learn skill composition from examples?, 2025. URL https://arxiv.org/abs/2409.19808

    2025 arXiv

  38. [46]

    What algorithms can transformers learn? a study in length generalization

    Hattie Zhou, Arwen Bradley, Etai Littwin, Noam Razin, Omid Saremi, Josh Susskind, Samy Bengio, and Preetum Nakkiran. What algorithms can transformers learn? a study in length generalization. arXiv preprint arXiv:2310.16028, 2023

    arXiv

  39. [47]

    Transformers can achieve length generalization but not robustly

    Yongchao Zhou, Uri Alon, Xinyun Chen, Xuezhi Wang, Rishabh Agarwal, and Denny Zhou. Transformers can achieve length generalization but not robustly. arXiv preprint arXiv:2402.09371, 2024

    2024 arXiv

  40. [48]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  41. [49]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  42. [50]

    Gw4YHj hk 9(kIwxS4g6ŘT kI,pk QXf4!׵G 1F <Fi<5 Si,pk -)#cA M;5zt`vj

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.