Pith. sign in

REVIEW 4 major objections 6 minor 6 cited by

Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By repeatedly training on its own filtered solutions to slightly harder problems, a standard transformer can learn to add 100-digit numbers and solve mazes far beyond its training distribution.

desk verdict Real results with a missing measurement: the 100-digit addition headline works, but the paper never shows the label-quality trajectory that the self-improvement loop depends on. read the letter →

arxiv 2502.01612 v2 pith:57WYALBR submitted 2025-02-03 cs.LG cs.AI

Nayoung Lee , Ziyang Cai , Avi Schwarzschild , Kangwook Lee , Dimitris Papailiopoulos This is my paper · ORCID
classification cs.LGcs.AI
keywords self-improvementlengthgeneralizationeasy-to-hardtransformerarithmeticreasoningchain-of-thoughtmajorityvotingerroravalanche
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that transformers' failure to generalize beyond their training length or difficulty does not require architectural fixes: an iterative loop in which the model labels slightly harder problems, filters those labels, and fine-tunes on them can push performance far outside the original training distribution. Starting with supervised labels for roughly 1-to-16-digit reverse addition, the authors report near-perfect accuracy on additions beyond 100 digits, and similar one-step-at-a-time gains for string copying and reversing, forward addition, chain-of-thought multiplication, and maze shortest-path finding. The method relies on the observed "transcendence" property: a model trained on easy instances can produce correct answers for marginally harder instances. Simple unsupervised filters, such as dropping outputs that are too short relative to the batch and keeping only majority-voted answers across several seeds, keep the self-generated data clean enough to avoid an error avalanche. The paper also reports that pretrained models improve faster and that, under filtered self-training, out-of-distribution performance can grow exponentially across rounds.

What carries the argument

The load-bearing mechanism is the iterative self-improvement loop: train on initial supervised data $D_0$ of difficulty at most $d_0$; at round $r$, sample inputs of difficulty $d_r=d_0+r$ (or a finer schedule), let the current model generate labels, filter them, and fine-tune on $D_0\cup D_1\cup\cdots\cup D_r$. The two unsupervised filters are relative length filtering, which removes outputs shorter than the batch maximum minus a threshold, and majority voting, which keeps outputs agreed upon by at least a threshold fraction of several seed-trained models. This loop leverages the transcendence property: a model trained on easy problems can answer problems one difficulty step harder, giving the next round a stream of mostly correct self-generated labels.

What would settle it

Measure the base model's exact-match accuracy on the first out-of-distribution difficulty before any self-improvement round, for example 17-digit reverse addition after training on 1-to-16-digit examples. If this accuracy is near zero, the first self-improvement round injects mostly wrong labels and the loop cannot bootstrap, whereas the paper's account predicts it is well above zero.

Watch

Extended reading notes

Core claim

The central discovery is that the "transcendence" phenomenon, where a model trained on easy instances can solve slightly harder ones, can be chained into extreme length and difficulty generalization. By repeatedly collecting the model's predictions on the next difficulty level, filtering them, and continuing training on the combined old and new data, a standard transformer progresses one step at a time from its initial distribution to lengths and difficulties far beyond it. The authors demonstrate this on reverse addition, string copy and reverse, forward addition, chain-of-thought multiplication, and maze solving, with no changes to positional embeddings or the transformer architecture. They further show that unsupervised filtering based on relative output length and majority voting is sufficient to prevent label-noise avalanches, and that the safe range of difficulty the model can be asked to self-label widens with each round.

Load-bearing premise

The method depends on the empirical regularity that a model trained on easy problems can solve problems one step harder; if that one-step "transcendence" fails at any round, the self-generated labels become garbage and the loop breaks.

Editorial extensions

If this is right

  • On reverse addition, copying, and reversing, the loop reaches near-perfect accuracy far outside the training range, such as over 100-digit addition and over 120-character strings, even without explicit filtering.
  • For forward addition, multiplication, and maze solving, unsupervised length filtering and majority voting are what sustain the loop; without them, accuracy collapses after a few rounds.
  • The safe difficulty range widens with each round, so an accelerated schedule that samples several difficulties per round reaches the same generalization in fewer rounds, e.g., 10-by-10 multiplication in 19 rounds instead of 41.
  • Pretrained 1B and 3B models bootstrap the loop faster than a from-scratch 14M model, so the recipe composes with model scale.
  • Filtered self-improvement can yield exponential growth in out-of-distribution accuracy across rounds, while the error-avalanche analysis identifies structured short-answer noise, rather than pure random label noise, as the main failure mode.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the recipe only needs a difficulty ladder whose rungs are one step apart; any problem family with a monotone, quantifiable notion of hardness, such as proof length, code complexity, or planning horizon, is a candidate for this kind of bootstrapping.
  • The error-avalanche simulations suggest that filters targeting structurally short or incomplete outputs may be more valuable than generic noise filters; outside arithmetic, truncation and dropped reasoning steps are common failure patterns, so a domain-agnostic "completeness" filter may substitute for exact verification.
  • Because the accelerated schedule in Section 7.2 used test-set accuracy to choose next-round difficulties, a practical deployment would need an unsupervised confidence proxy, such as the agreement rate across majority-voted models or the length-filter survival rate, to decide how far to push each round.
  • The majority-voting ablation shows that independent training trajectories matter more than simply training more data, which hints that diversity in the models being voted over is the active ingredient rather than ensemble size alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes an iterative self-improvement procedure for transformer models: train on an initial labeled dataset of low difficulty, then at each round generate predictions on slightly harder instances, optionally filter those predictions with unsupervised criteria (relative length filtering and/or majority voting), and fine-tune on the combined data. The authors evaluate this recipe on reverse addition, copy/reverse, forward addition, chain-of-thought multiplication, and maze solving, reporting strong out-of-distribution extrapolation (e.g., from 10-digit to 100-digit reverse addition, from length-10 to length-120 strings, and from 5-by-5 to 10-by-10 multiplication with filtering). They also analyze an "error avalanche" failure mode, show that structured label noise is more harmful than uniform noise, and demonstrate that pretrained Llama models accelerate self-improvement. The framing positions the results as evidence that standard transformers, without new positional encodings or architectural changes, can achieve extreme easy-to-hard and length generalization through self-training with simple filters.

Significance. If the reported results hold, the paper would make a strong empirical contribution: it suggests that iterative self-training with weak, task-agnostic filtering can bootstrap a vanilla decoder-only transformer far beyond its training difficulty, complementing prior work that relies on architectural or positional-encoding modifications. Strengths include the breadth of tasks, the controlled difficulty schedules, the explicit analysis of self-generated label quality in Figure 5, the synthetic noise experiments in Section 8, and the ablation of majority-voting variants in Figure 31. The paper is generally well written and the experimental setup is described in enough detail to be reproducible. However, the significance of the headline claims is currently limited by (i) the test-set-leaked accelerated schedule used to claim exponential improvement, (ii) the absence of variance estimates for the flagship length-generalization curves, and (iii) the lack of a per-round measurement of self-generated label accuracy for the 100-digit reverse-addition trajectory that supports the central transcendence mechanism.

major comments (4)
  1. [§7.2, Fig. 18] The claim of "exponential improvement" is not supported as stated because the accelerated schedule directly uses test-set accuracy to choose which difficulty levels to sample. The paper acknowledges this: "the sampling schedule in this proof-of-concept experiment leverages information about test set accuracy to determine the extra OOD lengths to sample." Since the training data at each round is selected from lengths that achieve above 99% test accuracy, the reported extrapolation is partly a product of oracle knowledge of the test distribution, not of the self-improvement procedure alone. Moreover, "exponential" is not established: Figure 18 shows a steeply rising but concave curve with no exponential fit or functional form. Please remove or substantially qualify the exponential claim, and provide a leak-free version that selects difficulties using only a held-out validation set or a model-based proxy.
  2. [Figs. 3, 4, 10, 18] The headline results—reverse addition to 100 digits, copy/reverse to length 120, forward addition to length 75, and the accelerated schedules—are reported without error bars or multiple seeds. The paper's own Figure 26 shows that 11-digit OOD accuracy varies dramatically across seeds (e.g., 0.617 for seed 44 vs. 0.998 for seed 41), so the claim of "perfect generalization" in Figure 3 is not yet convincing. Please report mean and standard deviation across at least 5 independent seeds for the central length-generalization curves, and state explicitly how many runs underlie each plotted line.
  3. [§4.1, §5.1, Fig. 5] The success of the reverse-addition self-improvement loop depends on the transcendence property: at each of roughly 84 rounds, the model must generate enough correct labels at difficulty 16+r to avoid an error avalanche. Figure 5 provides a scatter of data-accuracy versus next-round accuracy for some rounds, but the paper never reports the per-round label accuracy trajectory for the specific experiment behind Figure 3. Without this trajectory, one cannot verify that transcendence held throughout the loop, assess the margin of safety near each step, or rule out that the result hinges on a few early high-accuracy rounds. Please include a plot of self-generated label accuracy (and, where filtering is used, filtered-data accuracy) versus round for the reverse-addition and copy/reverse experiments.
  4. [§5.2, §6.1, §6.2] The filtering thresholds are chosen ad hoc and no sensitivity analysis is provided. Forward addition uses a length threshold of 2, multiplication uses 10 tokens, majority voting uses 4-out-of-5 consensus for multiplication and 3-out-of-3 for mazes. The paper presents these filters as "general-purpose" and central to success, but it does not show how performance varies with the threshold τ or with the consensus level. Without such an analysis, the positive results for forward addition, multiplication, and mazes could be knife-edge phenomena tied to particular hand-selected constants. Please add ablations over τ and over the consensus threshold for at least one task in each category.
minor comments (6)
  1. [Abstract, §3] The abstract states the method works "without any changes to the positional embeddings," but Section 3 explicitly removes positional embeddings and uses NoPE. This is itself a deliberate design choice (indeed, NoPE is a known technique for length generalization). Please rephrase the abstract to acknowledge that NoPE is used, or clarify that "no additional" positional encoding beyond the absence of one is meant.
  2. [§4.2] The fine-tuning set is written as "D1 ∪ ⋯ ∪ Dr" without D0, unlike the reverse-addition description. If the original labeled data is intentionally omitted, say so; if this is a typo, correct it.
  3. [§5.2, Fig. 8] The symbol τ is used both for the length-filter threshold and for the majority-vote consensus threshold. Please use distinct symbols (e.g., τ_len and τ_maj) to avoid ambiguity.
  4. [§8.2] The label "drop-preturb" in Figure 24 should be "drop-perturb."
  5. [§7.2] The statement that "the amount of extra OOD generalization increases roughly linearly" with round is not quantified; Figure 17 shows different slopes for copy and reverse addition. A linear fit or a statement of the observed range would be more precise.
  6. [§7.3, Appendix C.3] For the pretrained Llama experiments, the paper says "keep the embedding component unchanged" after replacing the tokenizer with a character tokenizer, which is internally inconsistent because a different vocabulary size requires a different embedding. Please clarify how the embedding layer is handled.

Circularity Check

1 steps flagged · score 6.0 of 10

Accelerated self-improvement claim is test-informed: the sampling schedule uses the same >99% test-accuracy metric that is then reported as the result.

  1. fitted input called prediction [Section 7.2 (Self-Improvement can be Accelerated), Figure 18 and surrounding text]
    "It is important to note that the sampling schedule in this proof-of-concept experiment leverages information about test set accuracy to determine the extra OOD lengths to sample, which is not typically available in practical scenarios. ... At each round, the self-improvement dataset is uniformly sampled from all difficulty levels achieving over 99% evaluation accuracy."

    The accelerated-schedule experiment defines its training distribution by the very evaluation metric it then reports: only difficulties with over 99% test accuracy are sampled into the next round, and the headline result is the maximum input length achieving over 99% test accuracy. The training data is therefore selected using the test set, and the reported exponential improvement is not an independent prediction of a fixed self-improvement loop; it is a feedback loop in which test accuracy is an input to the schedule. The paper explicitly acknowledges this test-set leakage, which confirms that the exponential extrapolation claim reduces in part to its own evaluation signal rather than to a purely self-generated curriculum.

full rationale

The core self-improvement results are not circular: in the reverse-addition, copy/reverse, forward-addition, multiplication, and maze experiments (Sections 4 and 6), the model generates pseudo-labels at difficulty d_r without using test-set answers, and final exact-match accuracy is an independent measurement. The transcendence assumption is attributed to an external prior work (Zhang et al., 2024) rather than to a self-citation chain, so it is an empirical premise, not a circular step. The one genuine circular element is the accelerated-schedule experiment in Section 7.2: the schedule is chosen using test-set accuracy ('all difficulty levels achieving over 99% evaluation accuracy'), and the reported curve is exactly the maximum length achieving over 99% test accuracy. This makes the exponential-improvement claim partly constructed from the evaluation set, as the paper itself acknowledges. Because the main 10-to-100-digit reverse-addition result remains independent of this test-informed schedule, the circularity is partial rather than total; the score reflects that specific predicted outcome rather than the whole paper.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method introduces no new entities or forces. The central burden is the transcendence assumption and the hand-picked filtering thresholds. The accelerated schedule additionally uses test-set information, which is a free parameter that weakens the exponential-improvement claim.

free parameters (4)
  • Length filtering threshold tau = 2 tokens (forward addition), 10 tokens (multiplication)
    Chosen by hand with no ablation. Directly controls how much self-generated data is discarded, and results depend on it.
  • Majority voting consensus threshold = 4 out of 5 (multiplication), 3 out of 3 (maze)
    Chosen by hand; different tasks use different thresholds, and no sensitivity analysis is given.
  • Number of majority-vote models k = 5 for multiplication, 3 for maze
    Selected without systematic study; affects data quality and compute cost.
  • Accelerated sampling schedule = all difficulty levels achieving over 99% test accuracy
    The schedule is set using evaluation accuracy on the test set, which is a form of test-set leakage and a key free parameter for the exponential-improvement claim.
assumptions (3)
  • domain assumption Transcendence: models can generalize slightly beyond the difficulty of their training data, generating correct outputs for n+1 difficulty with nontrivial accuracy.
    Invoked throughout Section 1 and the self-improvement loop; if this property fails, the generated labels at each round are too noisy to bootstrap.
  • domain assumption Length and majority-vote filtering preserve enough correct labels to keep the self-improvement process stable.
    Section 5 and 6 assume these unsupervised filters yield high-quality training data without ground-truth labels; the paper provides empirical evidence but no guarantee.
  • standard math Standard arithmetic and graph-search properties, e.g., in a tree a path with no repeated nodes is the shortest path, and addition follows standard carry rules.
    Used for data generation and for the oracle verifier in Section 6.3.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges." pith.science (2026). https://pith.science/paper/57WYALBR

@misc{pith2026250201612,
  author       = {Pith},
  title        = {Pith review of: Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/57WYALBR}},
  note         = {Machine review of arXiv:2502.01612}
}
read the original abstract

Large language models often struggle with length generalization and solving complex problem instances beyond their training distribution. We present a self-improvement approach where models iteratively generate and learn from their own solutions, progressively tackling harder problems while maintaining a standard transformer architecture. Across diverse tasks including arithmetic, string manipulation, and maze solving, self-improving enables models to solve problems far beyond their initial training distribution-for instance, generalizing from 10-digit to 100-digit addition without apparent saturation. We observe that in some cases filtering for correct self-generated examples leads to exponential improvements in out-of-distribution performance across training rounds. Additionally, starting from pretrained models significantly accelerates this self-improvement process for several tasks. Our results demonstrate how controlled weak-to-strong curricula can systematically teach a model logical extrapolation without any changes to the positional embeddings, or the model architecture.

Figures

Figures reproduced from arXiv: 2502.01612 by the authors.

Figure 1
Figure 1. Overview of self-improvement results. Models trained with self-improvement can tackle [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of our self-improvement procedure. At each round, the training data is updated [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Results on the reverse addition task, where both operands and the output are represented in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (39 more)
Figure 4
Figure 4. Figure 4: Results on string manipulation tasks. (Top) Copy: the model replicates the input string [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Effect of self-generated data accu￾racy on length generalization performance in the reverse addition task. Each data point rep￾resents the accuracy of the self-improve data Dr (on n digit addition) generated by model Mr−1, and the resulting n + 1-digit perfor￾mance of …
Figure 6
Figure 6. Figure 6: OOD results are often short. (Left) Reverse addition task: the proportion of shorter answers [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Majority voting leverages label diversity. (Left & Mid): Mean and standard deviation [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Overview of the two data-filtering methods employed. Length filtering removes data points [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Models trained on forward addition over 10 self-improvement rounds. (Left) Without [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: Results on the forward addition task with length filtering. The model is initially trained on labeled forward addition data of lengths 1 to 10. Using the self-improvement framework over 60 rounds, with incremental increases in digit length by 1 per round, the model ac…
Figure 11
Figure 11. Figure 11: Results on multiplication without filtering (round [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]
Figure 12
Figure 12. Figure 12: Comparison of data filtering methods at round 7. (Top-Left) no filtering (Top-Right) length filtering (Bottom-Left) majority voting (Bottom￾Right) and a combination of majority voting and length filtering. Data filtering significantly improves self-improvement perform…
Figure 13
Figure 13. Figure 13: Results on combining majority voting with length filtering (at round [PITH_FULL_IMAGE:figures/full_fig_p013_13.png]
Figure 14
Figure 14. Figure 14: Maze-solving task with increasing hops ( [PITH_FULL_IMAGE:figures/full_fig_p014_14.png]
Figure 15
Figure 15. Figure 15: Maze-solving task with increasing nodes (fixed at 9 hops). Models are trained on graphs [PITH_FULL_IMAGE:figures/full_fig_p015_15.png]
Figure 16
Figure 16. Figure 16: Results on using verifier for data filtering. (Top) Increasing hops. (Bottom) Increasing [PITH_FULL_IMAGE:figures/full_fig_p016_16.png]
Figure 17
Figure 17. Figure 17: Number of extra OOD digit lengths achieving over 99% accuracy when self-improving with one additional digit per round, on (Left) copy and (Right) reverse addition. The growing OOD capability suggests the potential to sample more digits per round as self-improvement pr…
Figure 18
Figure 18. Figure 18: Maximum input length achieving over 99% accuracy at different self-improvement rounds [PITH_FULL_IMAGE:figures/full_fig_p017_18.png]
Figure 19
Figure 19. Figure 19: Accelerated self-improvement in multiplication. [PITH_FULL_IMAGE:figures/full_fig_p017_19.png]
Figure 20
Figure 20. Figure 20: Reverse addition results for pretrained models. (Left) Llama-1B model. (Right) Llama-3B [PITH_FULL_IMAGE:figures/full_fig_p018_20.png]
Figure 21
Figure 21. Figure 21: Accelerated reverse addition with pretrained models. Comparison of self-improvement acceleration for Llama￾3B, Llama-1B, and a smaller 14M param￾eter model trained from scratch. Larger pre￾trained models demonstrate faster and more robust self-improvement. Setting. To…
Figure 22
Figure 22. Figure 22: Error avalanche is a common failure case for self-improvement. As inaccuracies in self-generated data accumulate, they degrade future rounds of training, leading to eventual failure. (Left) The impact of inaccuracies in n-digit data on n + 1-digit generalization. (Rig…
Figure 23
Figure 23. Figure 23: Patterns in model errors. (Left) Most incorrect digits are off by 1. (Middle) Errors cluster [PITH_FULL_IMAGE:figures/full_fig_p019_23.png]
Figure 24
Figure 24. Figure 24: Simulating error avalanche. Synthetic mistakes of varying noise levels are injected at the [PITH_FULL_IMAGE:figures/full_fig_p020_24.png]
Figure 25
Figure 25. Figure 25: Models memorize their mistakes. Accuracy on incorrect training examples decreases with additional self-improvement rounds, indicating that repeated exposure reinforces errors instead of correcting them. Models can Generalize Despite Memorizing Past Mistakes. Since sel…
Figure 26
Figure 26. Figure 26: Effect of self-generated training data quantity and quality on model performance. Each [PITH_FULL_IMAGE:figures/full_fig_p021_26.png]
Figure 27
Figure 27. Figure 27: Distribution of carry occurrences in the standard 20-digit self-improve dataset. Models [PITH_FULL_IMAGE:figures/full_fig_p030_27.png]
Figure 28
Figure 28. Figure 28: Performance of the model at round 10 (trained with self-generated data up to 19 digits). [PITH_FULL_IMAGE:figures/full_fig_p030_28.png]
Figure 29
Figure 29. Figure 29: Test accuracy on 5 different seeds during the initial training phase. Models exhibit high [PITH_FULL_IMAGE:figures/full_fig_p031_29.png]
Figure 30
Figure 30. Figure 30: Test accuracy on models trained with the same seed data but different training seeds. [PITH_FULL_IMAGE:figures/full_fig_p031_30.png]
Figure 31
Figure 31. Figure 31: Ablations on majority voting. (Left) No majority voting, but larger self-improve data. [PITH_FULL_IMAGE:figures/full_fig_p032_31.png]
Figure 32
Figure 32. Figure 32: he first incorrect digit in model outputs tends to occur near the most significant digit in [PITH_FULL_IMAGE:figures/full_fig_p033_32.png]
Figure 33
Figure 33. Figure 33: Effect of training on randomized labels. The model is trained on 1-10 digit data, further [PITH_FULL_IMAGE:figures/full_fig_p033_33.png]
Figure 34
Figure 34. Figure 34: Maze-solving task with N = 30 nodes. (Left & Middle) Visualization of the maze task with 4 hops (ID) and 13 hops (OOD). (Right) Example of the data format: the input specifies the start and end nodes along with the graph structure, and the output lists the shortest pa…
Figure 35
Figure 35. Figure 35: Results for multiplication without filtering. Each cell represents the accuracy on [PITH_FULL_IMAGE:figures/full_fig_p038_35.png]
Figure 36
Figure 36. Figure 36: Results for multiplication with length filtering with length threshold of 10. [PITH_FULL_IMAGE:figures/full_fig_p038_36.png]
Figure 37
Figure 37. Figure 37: Multiplication with majority voting where filtering is based on agreement of at least 4 out [PITH_FULL_IMAGE:figures/full_fig_p039_37.png]
Figure 38
Figure 38. Figure 38: Multiplication task with majority voting with shared self-improve data (See Section [PITH_FULL_IMAGE:figures/full_fig_p039_38.png]
Figure 39
Figure 39. Figure 39: Combining majority voting with length filtering. This approach achieves near-perfect [PITH_FULL_IMAGE:figures/full_fig_p040_39.png]
Figure 40
Figure 40. Figure 40: Accelerated multiplication. We can significantly reduce the self-improvement rounds [PITH_FULL_IMAGE:figures/full_fig_p041_40.png]
Figure 41
Figure 41. Figure 41: Maze solving task with increasing hops. (Top to bottom) Exact match accuracy, move [PITH_FULL_IMAGE:figures/full_fig_p042_41.png]
Figure 42
Figure 42. Figure 42: Maze solving task with increasing nodes. (Top to bottom) Exact match accuracy, move [PITH_FULL_IMAGE:figures/full_fig_p042_42.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Symmetry-Aware Transformer Training for Automated Planning

    cs.AI 2025-08 conditional novelty 7.0 of 10

    A contrastive loss that aligns attention and hidden states between renamed copies helps transformers solve larger planning problems in some domains, but not all.

  2. A Task-Centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula

    cs.LG 2026-02 conditional novelty 6.0 of 10

    Iterative self-improvement provably keeps improving only when initial performance lies in a moderate difficulty interval, and easy-to-hard curricula beat fixed mixtures under moderate difficulty separation and suffici...

  3. Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers

    cs.CL 2025-12 conditional novelty 6.0 of 10

    Canon layers—residual 1-d causal convolutions over adjacent tokens—boost synthetic reasoning depth 2-4x, lift NoPE to RoPE level, and bring GLA up to Mamba2/GDN, with qualitative real-world confirmation.

  4. Maximizing Prefix-Confidence at Test-Time Efficiently Improves Mathematical Reasoning

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Selecting the most confident 32-token prefix and completing only it gives better accuracy per compute than majority voting on five math reasoning datasets, using only the model's own confidence as a selector.

  5. Uncovering a Universal Abstract Algorithm for Modular Addition in Neural Networks

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Trained MLPs and transformers solving modular addition can be unified under an approximate Chinese Remainder Theorem, and deep or embedding-based networks learn only O(log n) frequency features.

  6. Extrapolation by Association: Length Generalization Transfer in Transformers

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Length generalization on a short-trained main task can be inherited from a longer-trained related auxiliary task trained jointly with it.

Reference graph

Works this paper leans on

71 extracted references · 12 canonical work pages · cited by 6 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024. URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md

    2024

  3. [3]

    I., Babaei, H

    Alemohammad, S., Casco-Rodriguez, J., Luzi, L., Humayun, A. I., Babaei, H. R., LeJeune, D., Siahkoohi, A., and Baraniuk, R. Self-consuming generative models go mad. ArXiv, abs/2307.01850, 2023. URL https://api.semanticscholar.org/CorpusID:259341801

    arXiv 2023

  4. [4]

    Global lyapunov functions: a long-standing open problem in mathematics, with symbolic transformers

    Alfarano, A., Charton, F., and Hayat, A. Global lyapunov functions: a long-standing open problem in mathematics, with symbolic transformers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    work page 2024

  5. [5]

    Exploring length generalization in large language models

    Anil, C., Wu, Y., Andreassen, A., Lewkowycz, A., Misra, V., Ramasesh, V., Slone, A., Gur-Ari, G., Dyer, E., and Neyshabur, B. Exploring length generalization in large language models. Advances in Neural Information Processing Systems, 35: 0 38546--38556, 2022

    work page 2022

  6. [6]

    and Nagarajan, V

    Bachmann, G. and Nagarajan, V. The pitfalls of next-token prediction. arXiv preprint arXiv:2403.06963, 2024

    arXiv 2024

  7. [7]

    End-to-end algorithm synthesis with recurrent networks: Extrapolation without overthinking

    Bansal, A., Schwarzschild, A., Borgnia, E., Emam, Z., Huang, F., Goldblum, M., and Goldstein, T. End-to-end algorithm synthesis with recurrent networks: Extrapolation without overthinking. Advances in Neural Information Processing Systems, 35: 0 20232--20242, 2022

    work page 2022

  8. [8]

    Q., and Kazemi, M

    Bansal, H., Hosseini, A., Agarwal, R., Tran, V. Q., and Kazemi, M. Smaller, weaker, yet better: Training llm reasoners via compute-optimal sampling. arXiv preprint arXiv:2408.16737, 2024

    arXiv 2024

Show all 71 references
  1. [9]

    The pitfalls of memorization: When memorization hurts generalization

    Bayat, R., Pezeshki, M., Dohmatob, E., Lopez-Paz, D., and Vincent, P. The pitfalls of memorization: When memorization hurts generalization. 2024. URL https://api.semanticscholar.org/CorpusID:274610625

    2024

  2. [10]

    J., Duplessis, A., Jiralerspong, M., and Gidel, G

    Bertrand, Q., Bose, A. J., Duplessis, A., Jiralerspong, M., and Gidel, G. On the stability of iterative retraining of generative models on their own data. ArXiv, abs/2310.00429, 2023. URL https://api.semanticscholar.org/CorpusID:263334017

    2023 arXiv

  3. [11]

    Bagging predictors

    Breiman, L. Bagging predictors. Machine learning, 24: 0 123--140, 1996

    1996

  4. [12]

    Large language models suffer from their own output: An analysis of the self-consuming training loop

    Briesch, M., Sobania, D., and Rothlauf, F. Large language models suffer from their own output: An analysis of the self-consuming training loop. ArXiv, abs/2311.16822, 2023. URL https://api.semanticscholar.org/CorpusID:265466007

    2023 arXiv

  5. [13]

    H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., et al

    Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390, 2023

    2023 arXiv

  6. [14]

    S., Wagner, A

    Charton, F., Ellenberg, J. S., Wagner, A. Z., and Williamson, G. Patternboost: Constructions in mathematics with a little help from ai. arXiv preprint arXiv:2411.00566, 2024

    2024 arXiv

  7. [15]

    Teaching large language models to self-debug

    Chen, X., Lin, M., Sch \"a rli, N., and Zhou, D. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023

    2023 arXiv

  8. [16]

    Position coupling: Improving length generalization of arithmetic transformers using task structure

    Cho, H., Cha, J., Awasthi, P., Bhojanapalli, S., Gupta, A., and Yun, C. Position coupling: Improving length generalization of arithmetic transformers using task structure. 2024. URL https://api.semanticscholar.org/CorpusID:273695226

    2024

  9. [17]

    de Arcaute, G. M. R., Watson, L., Reviriego, P., Hern \'a ndez, J. A., Ju \'a rez, M., and Sarkar, R. Combining generative artificial intelligence (ai) and the internet: Heading towards evolution or degradation? ArXiv, abs/2303.01255, 2023. URL https://api.semanticscholar.org/...

    2023 arXiv

  10. [18]

    From explicit cot to implicit cot: Learning to internalize cot step by step

    Deng, Y., Choi, Y., and Shieber, S. From explicit cot to implicit cot: Learning to internalize cot step by step. arXiv preprint arXiv:2405.14838, 2024

    2024 arXiv

  11. [19]

    A tale of tails: Model collapse as a change of scaling laws

    Dohmatob, E., Feng, Y., Yang, P., Charton, F., and Kempe, J. A tale of tails: Model collapse as a change of scaling laws. ArXiv, abs/2402.07043, 2024. URL https://api.semanticscholar.org/CorpusID:267628004

    2024 arXiv

  12. [20]

    From interpolation to extrapolation: Complete length generalization for arithmetic transformers

    Duan, S., Shi, Y., and Xu, W. From interpolation to extrapolation: Complete length generalization for arithmetic transformers. arXiv preprint arXiv:2310.11984, 2023

    2023 arXiv

  13. [21]

    Location attention for extrapolation to longer sequences

    Dubois, Y., Dagan, G., Hupkes, D., and Bruni, E. Location attention for extrapolation to longer sequences. arXiv preprint arXiv:1911.03872, 2019

    1911 arXiv

  14. [22]

    L., Jiang, L., Lin, B

    Dziri, N., Lu, X., Sclar, M., Li, X. L., Jiang, L., Lin, B. Y., Welleck, S., West, P., Bhagavatula, C., Le Bras, R., et al. Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems, 36, 2024

    2024

  15. [23]

    Looped transformers for length generalization

    Fan, Y., Du, Y., Ramchandran, K., and Lee, K. Looped transformers for length generalization. arXiv preprint arXiv:2409.15647, 2024

    2024 arXiv

  16. [24]

    Beyond model collapse: Scaling up with synthesized data requires reinforcement

    Feng, Y., Dohmatob, E., Yang, P., Charton, F., and Kempe, J. Beyond model collapse: Scaling up with synthesized data requires reinforcement. arXiv preprint arXiv:2406.07515, 2024

    2024 arXiv

  17. [25]

    Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data

    Gerstgrasser, M., Schaeffer, R., Dey, A., Rafailov, R., Sleight, H., Hughes, J., Korbak, T., Agrawal, R., Pai, D., Gromov, A., et al. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data. arXiv preprint arXiv:2404.01413, 2024

    2024 arXiv

  18. [26]

    Self-correcting self-consuming loops for generative model training

    Gillman, N., Freeman, M., Aggarwal, D., Hsu, C.-H., Luo, C., Tian, Y., and Sun, C. Self-correcting self-consuming loops for generative model training. arXiv preprint arXiv:2402.07087, 2024

    2024 arXiv

  19. [27]

    L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., et al

    Gulcehre, C., Paine, T. L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998, 2023

    2023 arXiv

  20. [28]

    The unreasonable effectiveness of easy training data for hard tasks

    Hase, P., Bansal, M., Clark, P., and Wiegreffe, S. The unreasonable effectiveness of easy training data for hard tasks. arXiv preprint arXiv:2401.06751, 2024

    2024 arXiv

  21. [29]

    Will large-scale generative models corrupt future datasets? In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp.\ 20555--20565, October 2023

    Hataya, R., Bao, H., and Arai, H. Will large-scale generative models corrupt future datasets? In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp.\ 20555--20565, October 2023

    2023

  22. [30]

    V-star: Training verifiers for self-taught reasoners

    Hosseini, A., Yuan, X., Malkin, N., Courville, A., Sordoni, A., and Agarwal, R. V-star: Training verifiers for self-taught reasoners. arXiv preprint arXiv:2402.06457, 2024

    2024 arXiv

  23. [31]

    E., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., and Chen, W

    Hu, J. E., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., and Chen, W. Lora: Low-rank adaptation of large language models. ArXiv, abs/2106.09685, 2021. URL https://api.semanticscholar.org/CorpusID:235458009

    2021 arXiv

  24. [32]

    J., Rohatgi, D., Zhang, C., Simchowitz, M., Ash, J

    Huang, A., Block, A., Foster, D. J., Rohatgi, D., Zhang, C., Simchowitz, M., Ash, J. T., and Krishnamurthy, A. Self-improvement in language models: The sharpening mechanism, 2024. URL https://arxiv.org/abs/2412.01951

    2024 arXiv

  25. [33]

    S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J

    Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J. Large language models can self-improve. arXiv preprint arXiv:2210.11610, 2022

    2022 arXiv

  26. [34]

    Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67: 0 757--795, 2020

    Hupkes, D., Dankers, V., Mul, M., and Bruni, E. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67: 0 757--795, 2020

    2020

  27. [35]

    Length generalization in arithmetic transformers

    Jelassi, S., d'Ascoli, S., Domingo-Enrich, C., Wu, Y., Li, Y., and Charton, F. Length generalization in arithmetic transformers. arXiv preprint arXiv:2306.15400, 2023

    2023 arXiv

  28. [36]

    The impact of positional encoding on length generalization in transformers

    Kazemnejad, A., Padhi, I., Natesan Ramamurthy, K., Das, P., and Reddy, S. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36, 2024

    2024

  29. [37]

    D., Lee, K., and Papailiopoulos, D

    Lee, N., Sreenivasan, K., Lee, J. D., Lee, K., and Papailiopoulos, D. Teaching arithmetic to small transformers. arXiv preprint arXiv:2307.03381, 2023

    2023 arXiv

  30. [38]

    Functional interpolation for relative positions improves long context transformers

    Li, S., You, C., Guruganesh, G., Ainslie, J., Ontanon, S., Zaheer, M., Sanghai, S., Yang, Y., Kumar, S., and Bhojanapalli, S. Functional interpolation for relative positions improves long context transformers. arXiv preprint arXiv:2310.04418, 2023

    arXiv

  31. [39]

    I-sheep: Self-alignment of llm from scratch through an iterative self-enhancement paradigm

    Liang, Y., Zhang, G., Qu, X., Zheng, T., Guo, J., Du, X., Yang, Z., Liu, J., Lin, C., Ma, L., et al. I-sheep: Self-alignment of llm from scratch through an iterative self-enhancement paradigm. arXiv preprint arXiv:2408.08072, 2024

    2024 arXiv

  32. [40]

    Let's verify step by step

    Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023

    2023 arXiv

  33. [41]

    Self-refine: Iterative refinement with self-feedback

    Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024

    2024

  34. [42]

    R., Kailkhura, B., Bhatele, A., Geiping, J., Schwarzschild, A., et al

    McLeish, S., Bansal, A., Stein, A., Jain, N., Kirchenbauer, J., Bartoldson, B. R., Kailkhura, B., Bhatele, A., Geiping, J., Schwarzschild, A., et al. Transformers can do arithmetic with the right embeddings. arXiv preprint arXiv:2405.17399, 2024

    2024 arXiv

  35. [43]

    Newman, B., Hewitt, J., Liang, P., and Manning, C. D. The eos decision and length extrapolation. arXiv preprint arXiv:2010.07174, 2020

    2010 arXiv

  36. [44]

    Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., and Weston, J

    Pang, R. Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., and Weston, J. Iterative reasoning preference optimization, 2024. URL https://arxiv.org/abs/2404.19733

    2024 arXiv

  37. [45]

    Regenesis: Llms can grow into reasoning generalists via self-improvement

    Peng, X., Xia, C., Yang, X., Xiong, C., Wu, C.-S., and Xing, C. Regenesis: Llms can grow into reasoning generalists via self-improvement. arXiv preprint arXiv:2410.02108, 2024

    2024 arXiv

  38. [46]

    A., and Lewis, M

    Press, O., Smith, N. A., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409, 2021

    2021 arXiv

  39. [47]

    Recursive introspection: Teaching language model agents how to self-improve

    Qu, Y., Zhang, T., Garg, N., and Kumar, A. Recursive introspection: Teaching language model agents how to self-improve. arXiv preprint arXiv:2407.18219, 2024

    2024 arXiv

  40. [48]

    and Barez, F

    Quirke, P. and Barez, F. Understanding addition in transformers. arXiv preprint arXiv:2310.13121, 2023

    2023 arXiv

  41. [49]

    Deep learning is robust to massive label noise

    Rolnick, D. Deep learning is robust to massive label noise. arXiv preprint arXiv:1705.10694, 2017

    2017 arXiv

  42. [50]

    Randomized positional encodings boost length generalization of transformers

    Ruoss, A., Del \'e tang, G., Genewein, T., Grau-Moya, J., Csord \'a s, R., Bennani, M., Legg, S., and Veness, J. Randomized positional encodings boost length generalization of transformers. arXiv preprint arXiv:2305.16843, 2023

    2023 arXiv

  43. [51]

    Explicitly encoding structural symmetry is key to length generalization in arithmetic tasks

    Sabbaghi, M., Pappas, G., Hassani, H., and Goel, S. Explicitly encoding structural symmetry is key to length generalization in arithmetic tasks. arXiv preprint arXiv:2406.01895, 2024

    2024 arXiv

  44. [52]

    Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks

    Schwarzschild, A., Borgnia, E., Gupta, A., Huang, F., Vishkin, U., Goldblum, M., and Goldstein, T. Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks. Advances in Neural Information Processing Systems, 34: 0 6695--6706, 2021

    2021

  45. [53]

    T., Li, Y., and Zhang, Y

    Shen, R., Bubeck, S., Eldan, R., Lee, Y. T., Li, Y., and Zhang, Y. Positional description matters for transformers arithmetic. arXiv preprint arXiv:2311.14737, 2023

    2023 arXiv

  46. [54]

    Weak-to-strong generalization through the data-centric lens

    Shin, C., Cooper, J., and Sala, F. Weak-to-strong generalization through the data-centric lens. arXiv preprint arXiv:2412.03881, 2024

    2024 arXiv

  47. [55]

    The curse of recursion: Training on generated data makes models forget

    Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., and Anderson, R. The curse of recursion: Training on generated data makes models forget. arXiv preprint arXiv:2305.17493, 2023

    2023 arXiv

  48. [56]

    Ai models collapse when trained on recursively generated data

    Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., and Gal, Y. Ai models collapse when trained on recursively generated data. Nature, 631 0 (8022): 0 755--759, 2024

    2024

  49. [57]

    D., Agarwal, R., Anand, A., Patil, P., Garcia, X., Liu, P

    Singh, A., Co-Reyes, J. D., Agarwal, R., Anand, A., Patil, P., Garcia, X., Liu, P. J., Harrison, J., Lee, J., Xu, K., et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.06585, 2023

    2023 arXiv

  50. [58]

    Mind the gap: Examining the self-improvement capabilities of large language models

    Song, Y., Zhang, H., Eisenach, C., Kakade, S., Foster, D., and Ghai, U. Mind the gap: Examining the self-improvement capabilities of large language models. arXiv preprint arXiv:2412.02674, 2024

    2024 arXiv

  51. [59]

    Easy-to-hard generalization: Scalable alignment beyond human supervision

    Sun, Z., Yu, L., Shen, Y., Liu, W., Yang, Y., Welleck, S., and Gan, C. Easy-to-hard generalization: Scalable alignment beyond human supervision. arXiv preprint arXiv:2403.09472, 2024

    2024 arXiv

  52. [60]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  53. [61]

    Self-consistency improves chain of thought reasoning in language models

    Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022 a

    2022 arXiv

  54. [62]

    A., Khashabi, D., and Hajishirzi, H

    Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560, 2022 b

    2022 arXiv

  55. [63]

    Understanding warmup-stable-decay learning rates: A river valley loss landscape perspective

    Wen, K., Li, Z., Wang, J., Hall, D., Liang, P., and Ma, T. Understanding warmup-stable-decay learning rates: A river valley loss landscape perspective. arXiv preprint arXiv:2410.05192, 2024

    2024 arXiv

  56. [64]

    From local structures to size generalization in graph neural networks

    Yehudai, G., Fetaya, E., Meirom, E., Chechik, G., and Maron, H. From local structures to size generalization in graph neural networks. In International Conference on Machine Learning, pp.\ 11975--11986. PMLR, 2021

    2021

  57. [65]

    Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, J

    Yuan, W., Pang, R. Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, J. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024

    2024 arXiv

  58. [66]

    Zelikman, E., Wu, Y., and Goodman, N. D. Star: Bootstrapping reasoning with reasoning. 2022. URL https://api.semanticscholar.org/CorpusID:247762790

    2022

  59. [67]

    L., Tambe, M., Kakade, S

    Zhang, E., Zhu, V., Saphra, N., Kleiman, A., Edelman, B. L., Tambe, M., Kakade, S. M., and Malach, E. Transcendence: Generative models can outperform the experts that train them. arXiv preprint arXiv:2406.11741, 2024

    2024 arXiv

  60. [68]

    and Parkes, D

    Zhang, H. and Parkes, D. C. Chain-of-thought reasoning is a policy improvement operator. arXiv preprint arXiv:2309.08589, 2023

    2023 arXiv

  61. [69]

    Be your own teacher: Improve the performance of convolutional neural networks via self distillation

    Zhang, L., Song, J., Gao, A., Chen, J., Bao, C., and Ma, K. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 3713--3722, 2019

    2019

  62. [70]

    What algorithms can transformers learn? a study in length generalization

    Zhou, H., Bradley, A., Littwin, E., Razin, N., Saremi, O., Susskind, J., Bengio, S., and Nakkiran, P. What algorithms can transformers learn? a study in length generalization. arXiv preprint arXiv:2310.16028, 2023

    2023 arXiv

  63. [71]

    Transformers can achieve length generalization but not robustly

    Zhou, Y., Alon, U., Chen, X., Wang, X., Agarwal, R., and Zhou, D. Transformers can achieve length generalization but not robustly. arXiv preprint arXiv:2402.09371, 2024

    2024 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.