Pith. sign in

REVIEW 2 major objections 3 minor 94 references

Frontier LLMs fail to exactly copy even within context; the paper attributes this to positional encodings and shows a 2D row-column position scheme turns copying into fixed-offset retrieval.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 21:25 UTC pith:ZRM56CCE

load-bearing objection The copy-failure finding and 2D-RoPE idea are real contributions, but the headline pretraining table doesn't match the appendix LR sweep, so the controlled comparison as reported doesn't stand until reconciled. the 2 major comments →

arxiv 2607.16072 v1 pith:ZRM56CCE submitted 2026-07-17 cs.CL

Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D

Haodong Wen , Yiran Zhang , Yingfa Chen , Kaifeng Lyu This is my paper
classification cs.CL MSC 68T0768T50
keywords exact copyingpositional encodingRoPElength generalization2D text layoutinduction headscopy taskLLM capabilities
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Current LLMs, including frontier systems, often cannot reproduce an input string exactly even when it is short enough to fit in their context; accuracy drops as strings get longer or contain repeated substrings. The paper argues this is not a reasoning failure but an inductive-bias failure of standard positional encodings, which favor copying by matching local context rather than by retrieving a token at a known position. It introduces 2D-RoPE, which lays text out as a grid—line breaks define rows, token offsets define columns—so that copying a string means looking up the same column in the previous row, a fixed-offset operation a single attention layer can express. The authors show that shallow 2D-RoPE transformers copy strings hundreds to thousands of times longer than their training data, and that 2D-RoPE language models pretrained on web text keep common-sense accuracy while copying much longer strings than RoPE models. If correct, the layout of positional information, not model scale, is what gates a basic capability that agents rely on.

Core claim

The paper's central discovery is that exact copying in a transformer is hard not because it requires complex computation but because standard relative positional encodings make the required retrieval offset depend on the unknown input length. 2D-RoPE removes that length dependence by giving every token a row ID and a column ID, with the newline token as the row separator. Once the source string and the output string occupy separate rows, the token to be copied and the token being produced share the same column, so copying reduces to retrieving the previous row at this column. The paper proves a one-layer 2D-RoPE transformer can represent this rule (Theorem 4.1) and that constrained global mi

What carries the argument

2D-RoPE. The central object is a modified position encoding: each token receives a (row, column) position pair computed from line breaks, and the head dimension of query/key rotations is split so one half encodes row differences and the other half column differences. Copying from an input row to an output row becomes a fixed column-offset lookup, expressible by one attention layer without length-dependent arithmetic. Auto-2D-RoPE replaces the hand-set line-break rule with a learned affine update of coordinates, so the model can discover row boundaries even when the separator token is not a newline.

Load-bearing premise

The headline results rest on two premises the paper itself flags: Theorem 4.2's length generalization is proven only under Assumption J.5 (zero value vectors for non-binary tokens), and synthetic 2D-RoPE depends on newline tokens as row separators—without them it degenerates to standard RoPE (Appendix F).

What would settle it

Take a one-layer 2D-RoPE model trained on a copy template whose separator is a token other than a newline, with Auto-2D-RoPE disabled: if it still copies at 100× training length, the claim that row separation drives the result is wrong; if it does not, the newline dependence is confirmed. Equivalently, measure a frontier model's copying accuracy on long strings built to contain no repeated substrings: high accuracy there would show repetition, not length, is the true driver, contradicting the paper's local-matching explanation.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If 2D-RoPE's fixed-offset view is right, standard RoPE's failure to copy is a positional-encoding artifact, not a scale issue, so architecture changes can restore exact copying without larger models.
  • Shallow transformers can copy lengths hundreds to thousands of times their training length, so length generalization on structured formatting tasks can be obtained from short training data.
  • Pretrained 2D-RoPE models generalize copying to out-of-distribution strings far beyond RoPE, suggesting the benefit transfers across copy-like tasks.
  • Hybrid-RoPE, alternating standard and 2D layers, gives a practical migration path for existing architectures with modest changes.
  • Auto-2D-RoPE shows the reliance on explicit newlines can be relaxed by learning coordinates from hidden states, which matters for natural text without line breaks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: 2D-RoPE should transfer to any task with aligned rows—tables, parallel text, key-value blocks—so structured-data formatting and function calling are the most direct near-term applications.
  • Beyond the paper: exact-copy accuracy on repetitive strings could serve as a cheap probe for whether a model's positional representations are genuinely length-invariant; large jumps on this probe would signal that a positional-encoding change is working.
  • Beyond the paper: a testable extension is to apply learned-coordinate variants to long code documents, where indentation and newlines carry structure; if copying of code blocks improves, that would confirm the 2D layout hypothesis outside synthetic strings.
  • Beyond the paper: because the theory's length-generalization guarantee is proven for binary strings and one layer, the multi-layer, natural-language regime is open; a staged curriculum that introduces 2D coordinates only for aligned spans could be tried.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. This paper studies exact copying in LLMs. It reports that frontier models degrade sharply on binary strings with repeated structure, attributes the failure to the inductive bias of 1D positional encodings, and proposes 2D-RoPE, which assigns each token a row ID and a column ID and rotates query/key vectors along both axes. The paper reports synthetic copy experiments with 1- and 12-layer Transformers, a two-layer RoPE expressivity theorem, a one-layer 2D-RoPE representation theorem, and a global-minimum length-generalization theorem. It also reports DCLM pretraining experiments from 350M to 1.4B parameters followed by copy finetuning, claiming that 2D-RoPE preserves common-sense reasoning while substantially improving copy length generalization.

Significance. The central idea—that the coordinate system of a positional encoding changes which algorithms are easy to learn—is well motivated and potentially impactful. The synthetic length-generalization results in Figure 4 are striking, and the paper includes a detailed proof appendix, an LR sweep, and a code link, which are commendable. The DCLM pretraining comparison is a valuable scaling check. However, the large-scale evidence in Table 1 is not internally consistent with the stated LR protocol in Appendix E.4, and the main theoretical claim in Theorem 4.2 carries a restrictive assumption that should be stated more prominently. If these issues are corrected, the paper would be a solid contribution; in its current form, major revision is required.

major comments (2)
  1. [Table 1 / §5.2 / §E.4] The Recursive-Flip columns of Table 1 do not correspond to the stated finetuning LR of 5e-5. For example, Table 1 reports 2D-RoPE 1.4B Recursive-Flip 1K/2K/4K/8K as 100/98.1/92.6/87.3, while Table 5 at LR=5e-5 gives 66.0/54.4/26.6/1.8; the Table 1 values coincide with the 2D-RoPE 350M LR=3e-5 row. Similarly, H-RoPE 1.4B Table 1 values equal H-RoPE 350M at LR=3e-5. Since Table 1 is the sole evidence for the claim that the advantage 'consistently holds in large-scale pretraining,' this discrepancy is load-bearing. The authors must either replace Table 1 with the exact 5e-5 rows from Table 5, or state precisely which configuration was used for each column and why it differs from the protocol described in §5.
  2. [§4.5.2 / Assumption J.5] Theorem 4.2 is proved only for the parameter class Θ_M defined in Appendix J.3, which fixes the value vectors of all non-binary tokens to zero (Assumption J.5). The main-text statement 'Under Assumptions J.3–J.5' does not convey the substance of this restriction; the surrounding prose in §4.5.2 claims more generally that 'global minima ... learn binary copy with length generalization.' Please state Assumption J.5 explicitly in the main text, and discuss whether the length-generalization result depends on this restriction or whether it can be relaxed.
minor comments (3)
  1. [Appendix E.4] The text says the chosen LR is 'highlighted in orange,' but the table in the submitted version has no highlighting. Please either add the highlight or delete the reference.
  2. [Table 5] The header of Table 5 repeats '1K 2K 4K 8K' for both the Imbalanced and Recursive-Flip blocks without clear subheaders, making the table hard to read. Adding 'Copy (Imbalanced)' and 'Copy (Recursive-Flip)' as separate column groups would improve clarity.
  3. [Figure 6] The caption says 'The gray region indicates the training context length range during finetuning.' Please specify the exact length range and whether it applies to both subfigures.

Circularity Check

0 steps flagged

No load-bearing circularity: 2D-RoPE is an intentional construction with self-contained proofs, and the only self-citation is contextual.

full rationale

The paper's derivation chain is not circular in the sense prohibited here. The central move — assigning row/column IDs so that the source token and its copy share a column — is explicitly a design choice, not a fitted parameter or a result that is presumed in its own proof. Section 4.1 states this as the motivation ('The idea behind 2D-RoPE is to remove this length dependence by assigning each token a two-dimensional position ID'), and the theoretical claims are constructive: Theorem 4.1 exhibits explicit Q/K/V parameters and proves a norm-bound length guarantee; Theorem 4.2 analyzes global minima under disclosed Assumptions J.3–J.5 and proves a margin-to-length-generalization implication. The length-generalization experiments genuinely test lengths far beyond the 1–100 training range, so the reported accuracies are extrapolations, not re-statements of training fits. The reliance on line breaks is disclosed as a limitation, and the Assumption J.5 restriction (zero value vectors for non-binary tokens) is a stated parameter-class restriction rather than a hidden equivalence. The only self-citation (Wen et al. 2025, co-authored by K. Lyu) appears in related work and is not load-bearing. Separately, the apparent inconsistency between Table 1 and the LR-sweep rows of Table 5 is a reproducibility/correctness concern, not a circularity reduction, so it does not change this verdict.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The central claim depends on three classes of inputs: (i) the hypothesis that LLM copy failures stem from 1D positional encodings' context-matching bias; (ii) the line-break-based row/column coordinate assignment, which defines the inductive bias being tested; (iii) the theoretical analysis, which requires simplifications (one-hot embeddings, no MLP, and Assumption J.5 zeroing non-binary value vectors).

free parameters (4)
  • RoPE base frequency θx=θy = 1000 for LLM experiments, 100 for synthetic
    Hand-set rotational frequency for 2D-RoPE rotations; preliminary experiments show insensitivity, but the value is chosen by the authors.
  • Head-dimension split ratio between column and row rotations = 1:1 (d/4 each)
    Architectural choice; no ablation provided to justify the split.
  • Auto-2D-RoPE rescaling factor α = 1.0
    Hand-set scale for learned coordinate updates in Auto-2D-RoPE.
  • Finetuning max LR = 5e-5
    Swept over {3e-5, 5e-5, 1e-4}; chosen by best overall performance. Main Table 1 numbers do not match the reported 5e-5 rows in Table 5 for Recursive-Flip.
axioms (5)
  • domain assumption Copy failures of LLMs are caused by positional encodings' inductive bias toward context-matching (Sections 1, 3.3)
    Central hypothesis motivating the method; supported by correlation and finetuning evidence but not causally identified.
  • domain assumption Line break token reliably separates rows so matching columns aligns source and target tokens (Section 4.1)
    2D-RoPE's advantage depends on prompt templates using a newline separator; without line breaks it degenerates to 1D RoPE.
  • ad hoc to paper Assumption J.5: value vectors of non-binary tokens are fixed to zero in the global-minimum analysis (Appendix J.3)
    Restricts Theorem 4.2 to a parameter subclass Θ_M rather than the full model; not stated in Section 4.5.2.
  • domain assumption Assumptions J.3 and J.4: all strings up to length L have probability ≥ L·p; L > d^4, d > log³ L (Appendix J.3)
    Technical conditions guaranteeing the global-minimum analysis; plausible but limiting.
  • domain assumption Theoretical model uses one-hot embeddings and no MLP (Appendix G, J.1)
    Theory analyzes a simplified Transformer; application to real language models is by analogy.
invented entities (2)
  • 2D-RoPE position IDs (row, column) for text tokens no independent evidence
    purpose: Make copying a fixed-column-offset retrieval instead of a length-dependent 1D offset
    New positional encoding applied to text; validated within the paper's copy tasks and DCLM pretraining, but not yet externally benchmarked.
  • Auto-2D-RoPE learned affine coordinate updates no independent evidence
    purpose: Infer 2D structure without explicit line-break tokens
    Cumulative-product coordinate predictor; results shown only for the star-separator copy task.

pith-pipeline@v1.3.0-alltime-deepseek · 54544 in / 16533 out tokens · 142731 ms · 2026-08-01T21:25:49.407703+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D." pith.science (2026). https://pith.science/paper/ZRM56CCE

@misc{pith2026260716072,
  author       = {Pith},
  title        = {Pith review of: Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZRM56CCE}},
  note         = {Machine review of arXiv:2607.16072}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

While large language models (LLMs) can solve advanced reasoning problems in seconds, we show that even frontier models fail to perform a much simpler operation: exactly copying an input string that lies well within their context windows. We attribute this failure to positional encodings in Transformer architectures, whose inductive bias favors copying through a shortcut based on matching local contexts rather than carefully locating the corresponding input positions. To address this issue, we introduce 2D-RoPE, which organizes text into a 2D grid rather than a 1D sequence and assigns each token a row ID and a column ID. Under this view, copying becomes simply retrieving input tokens at a fixed column offset, which makes the task easy to learn. In synthetic copy experiments, shallow Transformers with 2D-RoPE achieve perfect copying at input lengths hundreds of times longer than those seen during training, whereas standard positional encodings fall far behind. We further show that the advantage of 2D-RoPE language models on copy tasks consistently holds in large-scale pretraining on DCLM with model sizes up to 1.4B parameters. Overall, our results suggest that viewing text in 2D can benefit language modeling, and we hope this encourages future work to further explore the potential of 2D positional encodings.

Figures

Figures reproduced from arXiv: 2607.16072 by Haodong Wen, Kaifeng Lyu, Yingfa Chen, Yiran Zhang.

Figure 1
Figure 1. Figure 1: Frontier language models struggle to perform two representative copy tasks, the binary copy [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Repeat-structure copy test for frontier LLMs. Each input is built by repeating a short base block (period 4–10) until length 1000, and then independently flipping each token with some probability. The x-axis, (1 − p) m, is the fraction of base blocks that survive uncorrupted, so a larger value means the input contains more exact repetitions; the y-axis is the exact copying accuracy. Across all three models… view at source ↗
Figure 3
Figure 3. Figure 3: Attention map of a RoPE model on a failed sample with sequence length within training [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: 2D-RoPE models exhibit stronger length generalization for synthetic experiments on binary copy. Models are trained only on lengths 1–100 and tested on much longer inputs. The x-axis is the sequence length (log scale) and the y-axis is the copying accuracy; the three columns correspond to the Uniform, Imbalanced, and Recursive-Flip distributions, and each curve is a different positional encoding. (a) 12 lay… view at source ↗
Figure 5
Figure 5. Figure 5: The training loss and evaluation loss during pretraining of the 2D-RoPE, H-RoPE, and [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Fine-tuning results on copy-related tasks using [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Length generalization for training-from-scratch experiments in 2 distributions. [PITH_FULL_IMAGE:figures/full_fig_p026_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Attention map of a RoPE-based model on a failed sample with sequence length beyond the [PITH_FULL_IMAGE:figures/full_fig_p026_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

94 extracted references · 17 linked inside Pith

  1. [1]

    High-dimensional Learning Dynamics 2025 , year =

    Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data , author=. High-dimensional Learning Dynamics 2025 , year =

    2025

  2. [2]

    Proceedings of the 41st International Conference on Machine Learning , pages=

    Why do you grok? a theoretical analysis on grokking modular addition , author=. Proceedings of the 41st International Conference on Machine Learning , pages=

  3. [3]

    2024 , eprint=

    The Llama 3 Herd of Models , author=. 2024 , eprint=

    2024

  4. [4]

    2019 , eprint=

    BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions , author=. 2019 , eprint=

    2019

  5. [5]

    2019 , eprint=

    PIQA: Reasoning about Physical Commonsense in Natural Language , author=. 2019 , eprint=

    2019

  6. [6]

    2016 , eprint=

    The LAMBADA dataset: Word prediction requiring a broad discourse context , author=. 2016 , eprint=

    2016

  7. [7]

    2019 , eprint=

    HellaSwag: Can a Machine Really Finish Your Sentence? , author=. 2019 , eprint=

    2019

  8. [8]

    2021 , eprint=

    Measuring Massive Multitask Language Understanding , author=. 2021 , eprint=

    2021

  9. [9]

    2019 , eprint=

    WinoGrande: An Adversarial Winograd Schema Challenge at Scale , author=. 2019 , eprint=

    2019

  10. [10]

    Advances in Neural Information Processing Systems , volume=

    Datacomp-lm: In search of the next generation of training sets for language models , author=. Advances in Neural Information Processing Systems , volume=

  11. [11]

    arXiv:1803.05457v1 , year =

    Peter Clark and Isaac Cowhey and Oren Etzioni and Tushar Khot and Ashish Sabharwal and Carissa Schoenick and Oyvind Tafjord , title =. arXiv:1803.05457v1 , year =

    Pith/arXiv arXiv

  12. [12]

    International Conference on Learning Representations , year=

    Decoupled Weight Decay Regularization , author=. International Conference on Learning Representations , year=

  13. [13]

    doi:10.5281/zenodo.12608602 , url =

    Gao, Leo and Tow, Jonathan and Abbasi, Baber and Biderman, Stella and Black, Sid and DiPofi, Anthony and Foster, Charles and Golding, Laurence and Hsu, Jeffrey and Le Noac'h, Alain and Li, Haonan and McDonell, Kyle and Muennighoff, Niklas and Ociepa, Chris and Phang, Jason and Reynolds, Laria and Schoelkopf, Hailey and Skowron, Aviya and Sutawika, Lintang...

    doi:10.5281/zenodo.12608602

  14. [14]

    First Conference on Language Modeling , year =

    Counting Like Transformers: Compiling Temporal Counting Logic Into Softmax Transformers , author=. First Conference on Language Modeling , year =

  15. [15]

    Songlin Yang and Yikang Shen and Kaiyue Wen and Shawn Tan and Mayank Mishra and Liliang Ren and Rameswar Panda and Yoon Kim , booktitle=. Pa. 2026 , url=

    2026

  16. [16]

    The Twelfth International Conference on Learning Representations , year =

    Feature emergence via margin maximization: case studies in algebraic tasks , author=. The Twelfth International Conference on Learning Representations , year =

  17. [17]

    International Conference on Machine Learning , pages=

    Implicit Bias of AdamW: _ -Norm Constrained Optimization , author=. International Conference on Machine Learning , pages=. 2024 , organization=

    2024

  18. [18]

    arXiv preprint arXiv:2308.16898 , year=

    Transformers as support vector machines , author=. arXiv preprint arXiv:2308.16898 , year=

    Pith/arXiv arXiv

  19. [19]

    Advances in neural information processing systems , volume=

    Max-margin token selection in attention mechanism , author=. Advances in neural information processing systems , volume=

  20. [20]

    Advances in Neural Information Processing Systems , volume=

    Regularization matters: Generalization and optimization of neural nets vs their induced kernel , author=. Advances in Neural Information Processing Systems , volume=

  21. [21]

    Advances in Neural Information Processing Systems , volume=

    The implicit bias of adam on separable data , author=. Advances in Neural Information Processing Systems , volume=

  22. [22]

    International Conference on Machine Learning , pages=

    Repeat After Me: Transformers are Better than State Space Models at Copying , author=. International Conference on Machine Learning , pages=. 2024 , organization=

    2024

  23. [23]

    The Thirteenth International Conference on Learning Representations , year =

    A Formal Framework for Understanding Length Generalization in Transformers , author=. The Thirteenth International Conference on Learning Representations , year =

  24. [24]

    arXiv preprint arXiv:2505.21785 , year=

    Born a Transformer--Always a Transformer? , author=. arXiv preprint arXiv:2505.21785 , year=

    arXiv

  25. [25]

    Advances in Neural Information Processing Systems , volume=

    Training Compute-Optimal Large Language Models , author=. Advances in Neural Information Processing Systems , volume=

  26. [26]

    Proceedings of the National Academy of Sciences , volume=

    Out-of-distribution generalization via composition: a lens through induction heads in transformers , author=. Proceedings of the National Academy of Sciences , volume=. 2025 , publisher=

    2025

  27. [27]

    Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers , volume =

    Chen, Siyu and Sheen, Heejune and Wang, Tianhao and Yang, Zhuoran , booktitle =. Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers , volume =. doi:10.52202/079017-2127 , editor =

    doi:10.52202/079017-2127

  28. [28]

    arXiv preprint arXiv:2410.11474 , year=

    How Transformers Get Rich: Approximation and Dynamics Analysis , author=. arXiv preprint arXiv:2410.11474 , year=

    Pith/arXiv arXiv

  29. [29]

    International Conference on Machine Learning , pages=

    How Transformers Learn Causal Structure with Gradient Descent , author=. International Conference on Machine Learning , pages=. 2024 , organization=

    2024

  30. [30]

    The Twelfth International Conference on Learning Representations , year =

    What Algorithms can Transformers Learn? A Study in Length Generalization , author=. The Twelfth International Conference on Learning Representations , year =

  31. [31]

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    On the Ability and Limitations of Transformers to Recognize Formal Languages , author=. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    2020

  32. [32]

    Advances in Neural Information Processing Systems , volume=

    Exploring length generalization in large language models , author=. Advances in Neural Information Processing Systems , volume=

  33. [33]

    Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    Length generalization of causal transformers without position encoding , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    2024

  34. [34]

    Advances in Neural Information Processing Systems , volume=

    The impact of positional encoding on length generalization in transformers , author=. Advances in Neural Information Processing Systems , volume=

  35. [35]

    arXiv preprint arXiv:2402.09371 , year=

    Transformers can achieve length generalization but not robustly , author=. arXiv preprint arXiv:2402.09371 , year=

    Pith/arXiv arXiv

  36. [36]

    arXiv preprint arXiv:2310.00726 , year=

    Improving length-generalization in transformers via task hinting , author=. arXiv preprint arXiv:2310.00726 , year=

    Pith/arXiv arXiv

  37. [37]

    The Thirteenth International Conference on Learning Representations , year =

    Language Models Need Inductive Biases to Count Inductively , author=. The Thirteenth International Conference on Learning Representations , year =

  38. [38]

    arXiv preprint arXiv:2306.15400 , year=

    Length generalization in arithmetic transformers , author=. arXiv preprint arXiv:2306.15400 , year=

    Pith/arXiv arXiv

  39. [39]

    International Conference on Machine Learning , pages=

    Thinking like transformers , author=. International Conference on Machine Learning , pages=. 2021 , organization=

    2021

  40. [40]

    Transactions of the Association for Computational Linguistics , volume=

    MasakhaNER: Named entity recognition for African languages , author=. Transactions of the Association for Computational Linguistics , volume=. 2021 , publisher=

    2021

  41. [41]

    Advances in Neural Information Processing Systems , volume=

    Representational strengths and limitations of transformers , author=. Advances in Neural Information Processing Systems , volume=

  42. [42]

    International Conference on Machine Learning , pages=

    Inductive biases and variable creation in self-attention mechanisms , author=. International Conference on Machine Learning , pages=. 2022 , organization=

    2022

  43. [43]

    Journal of Machine Learning Research , volume=

    Attention is turing-complete , author=. Journal of Machine Learning Research , volume=

  44. [44]

    Self-Attention Networks Can Process Bounded Hierarchical Languages , author=. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages=

  45. [45]

    Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Overcoming a Theoretical Limitation of Self-Attention , author=. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  46. [46]

    The Twelfth International Conference on Learning Representations , year =

    The Expressive Power of Transformers with Chain of Thought , author=. The Twelfth International Conference on Learning Representations , year =

  47. [47]

    Advances in Neural Information Processing Systems , volume=

    Towards revealing the mystery behind chain of thought: a theoretical perspective , author=. Advances in Neural Information Processing Systems , volume=

  48. [48]

    The Thirteenth International Conference on Learning Representations , year =

    RNNs are not Transformers (Yet): The Key Bottleneck on In-Context Retrieval , author=. The Thirteenth International Conference on Learning Representations , year =

  49. [49]

    arXiv preprint arXiv:2412.02975 , year=

    Theoretical limitations of multi-layer transformer , author=. arXiv preprint arXiv:2412.02975 , year=

    Pith/arXiv arXiv

  50. [50]

    arXiv preprint arXiv:2511.07378 , year=

    Transformers provably learn chain-of-thought reasoning with length generalization , author=. arXiv preprint arXiv:2511.07378 , year=

    arXiv

  51. [51]

    Proceedings of Thirty Seventh Conference on Learning Theory , pages =

    Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality , author =. Proceedings of Thirty Seventh Conference on Learning Theory , pages =. 2024 , editor =

    2024

  52. [52]

    Forty-first International Conference on Machine Learning , year =

    In-context Convergence of Transformers , author=. Forty-first International Conference on Machine Learning , year =

  53. [53]

    The Thirteenth International Conference on Learning Representations , year =

    From Sparse Dependence to Sparse Attention: Unveiling How Chain-of-Thought Enhances Transformer Sample Efficiency , author=. The Thirteenth International Conference on Learning Representations , year =

  54. [54]

    The Thirteenth International Conference on Learning Representations , year =

    Transformers Provably Solve Parity Efficiently with Chain of Thought , author=. The Thirteenth International Conference on Learning Representations , year =

  55. [55]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

    Multi-head Transformers Provably Learn Symbolic Multi-step Reasoning via Gradient Descent , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

  56. [56]

    Advances in Neural Information Processing Systems , volume=

    A phase transition between positional and semantic learning in a solvable model of dot-product attention , author=. Advances in Neural Information Processing Systems , volume=

  57. [57]

    arXiv preprint arXiv:2402.05738 , year=

    Implicit bias and fast convergence rates for self-attention , author=. arXiv preprint arXiv:2402.05738 , year=

    Pith/arXiv arXiv

  58. [58]

    arXiv preprint arXiv:2403.08699 , year=

    Implicit regularization of gradient flow on one-layer softmax attention , author=. arXiv preprint arXiv:2403.08699 , year=

    Pith/arXiv arXiv

  59. [59]

    International Conference on Learning Representations , volume=

    Forgetting transformer: Softmax attention with a forget gate , author=. International Conference on Learning Representations , volume=

  60. [60]

    arXiv preprint arXiv:2506.16640 , year=

    Long-context generalization with sparse attention , author=. arXiv preprint arXiv:2506.16640 , year=

    arXiv

  61. [61]

    arXiv preprint arXiv:2504.08719 , year=

    Swan-gpt: An efficient and scalable approach for long-context language modeling , author=. arXiv preprint arXiv:2504.08719 , year=

    Pith/arXiv arXiv

  62. [62]

    High-dimensional Learning Dynamics 2025 , year =

    Attention with Trained Embeddings Provably Selects Important Tokens , author=. High-dimensional Learning Dynamics 2025 , year =

    2025

  63. [63]

    Advances in neural information processing systems , volume=

    Attention is all you need , author=. Advances in neural information processing systems , volume=

  64. [64]

    International Conference on Learning Representations , year=

    Teaching Arithmetic to Small Transformers , author=. International Conference on Learning Representations , year=

  65. [65]

    European Conference on Computer Vision , pages=

    Rotary position embedding for vision transformer , author=. European Conference on Computer Vision , pages=. 2024 , organization=

    2024

  66. [66]

    European Conference on Computer Vision , pages=

    Visionllama: A unified llama backbone for vision tasks , author=. European Conference on Computer Vision , pages=. 2024 , organization=

    2024

  67. [67]

    Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages=

    Resource-efficient hybrid x-formers for vision , author=. Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages=

  68. [68]

    Neurocomputing , volume=

    Roformer: Enhanced transformer with rotary position embedding , author=. Neurocomputing , volume=. 2024 , publisher=

    2024

  69. [69]

    arXiv preprint arXiv:2209.11895 , year=

    In-context learning and induction heads , author=. arXiv preprint arXiv:2209.11895 , year=

    Pith/arXiv arXiv

  70. [70]

    International Conference on Learning Representations , year =

    Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation , author=. International Conference on Learning Representations , year =

  71. [71]

    Proceedings of the 31st International Conference on Computational Linguistics , pages=

    Icleval: evaluating in-context learning ability of large language models , author=. Proceedings of the 31st International Conference on Computational Linguistics , pages=

  72. [72]

    Advances in Neural Information Processing Systems , volume=

    Needle in a multimodal haystack , author=. Advances in Neural Information Processing Systems , volume=

  73. [73]

    arXiv preprint arXiv:2404.06654 , year=

    RULER: What's the real context size of your long-context language models? , author=. arXiv preprint arXiv:2404.06654 , year=

    Pith/arXiv arXiv

  74. [74]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    Pith/arXiv arXiv

  75. [75]

    Shengding Hu and Yuge Tu and Xu Han and Ganqu Cui and Chaoqun He and Weilin Zhao and Xiang Long and Zhi Zheng and Yewei Fang and Yuxiang Huang and Xinrong Zhang and Zhen Leng Thai and Chongyi Wang and Yuan Yao and Chenyang Zhao and Jie Zhou and Jie Cai and Zhongwu Zhai and Ning Ding and Chao Jia and Guoyang Zeng and dahai li and Zhiyuan Liu and Maosong Su...

    2024

  76. [76]

    Forty-first International Conference on Machine Learning , year =

    What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation , author=. Forty-first International Conference on Machine Learning , year =

  77. [77]

    International Conference on Learning Representations , year=

    Scaling Stick-Breaking Attention: An Efficient Implementation and In-depth Study , author=. International Conference on Learning Representations , year=

  78. [78]

    arXiv preprint arXiv:2512.14391 , year=

    RePo: Language Models with Context Re-Positioning , author=. arXiv preprint arXiv:2512.14391 , year=

    Pith/arXiv arXiv

  79. [79]

    The Fourteenth International Conference on Learning Representations , year =

    Mamba-3: Improved Sequence Modeling using State Space Principles , author=. The Fourteenth International Conference on Learning Representations , year =

  80. [80]

    arXiv preprint arXiv:2502.06807 , year=

    Competitive programming with large reasoning models , author=. arXiv preprint arXiv:2502.06807 , year=

    Pith/arXiv arXiv

Showing first 80 references.