Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Diagonal Batching Unlocks Parallelism in Recurrent Memory Transformers for Long Contexts

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Diagonal Batching reorders layer-level recurrent memory transformer computation into diagonal waves, running up to $N_{\text{layers}}$ operations in one GPU kernel launch and cutting 131,072-token inference latency by 3.3x.

desk verdict Solid engineering contribution with a correct scheduling lemma, honestly scoped in the limitations but over-sold in the title and abstract; worth a serious referee after some fixes. read the letter →

arxiv 2506.05229 v1 pith:RRAHKHQC submitted 2025-06-05 cs.LG cs.CL

Danil Sivtsov , Ivan Rodkin , Gleb Kuzmin , Yuri Kuratov , Ivan Oseledets This is my paper · ORCID
classification cs.LGcs.CL
keywords DiagonalBatchingRecurrentMemoryTransformerParallelAssociativelong-contextinferenceGPUschedulingsegment-levelrecurrencelinear-time
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's claim is that the long-context inference bottleneck in Recurrent Memory Transformer–style models is scheduling, not algorithmic complexity: recurrent dependencies force layers and segments to wait for one another and leave GPUs under-used. Diagonal Batching is a run-time reordering of the two-dimensional grid of layers and segments into diagonal waves, so all independent computations with the same segment-plus-layer index run together; up to $N_{\text{layers}}$ operations execute per GPU kernel launch while the exact recurrence is preserved. Because it is only a reordering, existing Parallel Recurrent Memory Transformers (PRMTs), such as the Associative Recurrent Memory Transformer (ARMT), adopt it with no retraining. On a one-billion-parameter ARMT, the paper reports a 3.3x speedup over full-attention inference and a 1.8x speedup over the sequential ARMT implementation at 131,072 tokens, with numerical drift under 2% and unchanged scores on a long-context reasoning benchmark.

What carries the argument

The central object is Diagonal Batching, a schedule for the layer-by-segment dependency graph. In a PRMT, node $(s, l)$ (segment $s$, layer $l$) depends only on $(s, l-1)$ and $(s-1, l)$, so all nodes with $s+l$ equal to the same value are mutually independent; Diagonal Batching groups them into one wave and launches the whole wave through a single grouped layer (stacked weights, grouped matrix multiplication, and batched attention). This makes the GPU treat the diagonal as a batch, recovering the utilization that normally requires many independent requests, without changing the recurrence the model computes.

What would settle it

Run one 131,072-token prompt through the same trained ARMT checkpoint twice, once with the sequential reference implementation and once with Diagonal Batching, on the same GPU, and compare final logits and wall-clock latency. If the relative logit error grows well beyond the reported roughly 2% as the number of segments increases, or if the grouped schedule is not faster than the sequential one for the claimed configurations, the exact-recurrence and speedup claims would be refuted. A direct boundary check is to apply the schedule to an original RMT with cross-layer memory flow and observe the output change, since the method is only valid for layer-level recurrent models.

Watch

Extended reading notes

Core claim

The central claim is that for models with layer-level recurrence—where each layer keeps its own memory and updates it once per segment—the whole forward pass over $N_{\text{segments}}$ segments and $N_{\text{layers}}$ layers can be executed as $N_{\text{segments}}+N_{\text{layers}}-1$ groups instead of $N_{\text{segments}}\times N_{\text{layers}}$ sequential steps. Group $i$ contains every node whose segment index plus layer index equals $i$, and Lemma 3.1 states that this is the minimum possible number of groups because the longest dependency path has exactly that many nodes. The implementation fuses the distinct transformer layers into one grouped layer with stacked weights and batched attention, so the GPU sees a large effective batch without batching multiple requests. The paper argues that the resulting computation is numerically close to the original—relative logit error below 2% for sequences up to 32,768 tokens—and that on a 1B ARMT it delivers up to 3.3x lower latency than full-attention inference and up to 1.8x lower latency than the sequential ARMT at 131,072 tokens.

Load-bearing premise

The load-bearing premise is the layer-level recurrent dependency graph: each (segment, layer) step may read only the same segment's previous layer and the previous segment's same-layer memory. Original Recurrent Memory Transformers violate this because the previous segment's final-layer memory enters every layer of the next segment, in which case the diagonal groups are no longer independent and the schedule is invalid.

Editorial extensions

If this is right

  • Any already-trained PRMT-style model can be switched to Diagonal Batching as a drop-in inference change, with no retraining or architecture modification, because the reordering computes the same recurrence.
  • The number of sequential synchronization barriers drops from $N_{\text{segments}} \times N_{\text{layers}}$ to $N_{\text{segments}} + N_{\text{layers}} - 1$, so models with more layers gain disproportionately more.
  • A single long-context request can saturate the GPU without waiting for other requests, which simplifies serving and load balancing compared with large-batch inference.
  • Developers can choose smaller segment sizes for quality without automatically paying the usual inference-speed penalty, because diagonal grouping decouples performance from segment size.
  • The observed numerical drift stays below 2% and generation scores on a long-context reasoning benchmark are unchanged up to 64k tokens, so the speedup is not bought with task quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the diagonal schedule is a generic property of any two-dimensional layer-by-segment dependency grid, so it should transfer to chunked inference in other per-layer recurrent architectures whenever their layer states update independently per segment.
  • Editorial extension: since the reported speedup grows with layer count, stacking more layers or designing grouped modular layers could compound the gain, while very shallow models will see little benefit.
  • Editorial extension: the backward pass described in the appendix suggests the same grouped schedule could be used during training, making long-context recurrent-memory training numerically consistent with inference and possibly faster.
  • Editorial extension: the observed 1–2% error budget is similar to replacing one attention kernel with another, so combining diagonal batching with quantization or speculative decoding is a plausible next test, though the paper does not demonstrate that combination.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Diagonal Batching, a scheduling scheme for layer-level recurrent models (which the authors call PRMTs, including ARMT, RWKV, and Mamba) that reorganizes the layer-segment computation grid into diagonal groups. The authors prove in Lemma 3.1 that this schedule completes the dependency DAG in the minimum number of groups, N_segments + N_layers - 1, and implement it for ARMT by replacing all layers with a single grouped layer. They report speedups over sequential ARMT and over standard full-attention Llama models at sequence lengths up to 131,072 tokens on A100/H100 GPUs, together with an error-accumulation study and BABILong evaluations.

Significance. If the claims hold, this is a useful systems contribution: it shows that a simple, purely runtime reordering can unlock inter-segment parallelism for PRMT-style architectures without retraining, and it ships publicly available code. The scheduling lemma is correct for the stated layer-level dependency graph, and the experimental sweep across model sizes (160M to 8B), sequence lengths, and GPUs is broad. The measurements are accompanied by an error analysis and a downstream benchmark, which strengthens the empirical picture. The main caveat is that the method does not apply to the original RMT recurrence, so the paper's broad 'RMT' framing overstates the contribution, and the 'exact recurrence' claim needs qualification given the reported numerical drift.

major comments (3)
  1. [Title, Abstract, and Section 2.2] The title and abstract claim applicability to 'Recurrent Memory Transformers' generally, but Section 2.2 and the Limitations explicitly state that standard RMT, defined by Eq. (1), does not satisfy the layer-level dependency assumption on which Diagonal Batching relies. In particular, the abstract's statement that 'existing RMT models adopt it with no retraining' is unsupported for the original RMT, because node (s, l) in standard RMT transitively depends on the final layer of the previous segment, not only on (s-1, l) and (s, l-1). This is a load-bearing overstatement of the method's scope; the paper should consistently frame the contribution as applying to layer-level recurrent models (PRMTs/ARMT) in the title, abstract, and conclusion, or should justify why the original RMT can be included despite the dependency structure shown in Figure 2 (left).
  2. [Abstract, Section 3.2, and Table 2] The paper repeatedly claims that Diagonal Batching 'preserves exact recurrence' and enables 'exact, linear-time inference,' but Table 2 reports up to 1.87% relative logit drift between the base ARMT implementation and the Diagonal Batching implementation, and Appendix A attributes this drift to implementation details such as optimized kernels. The schedule does preserve the dependency graph exactly, but the executed computation is not bitwise identical. The authors should explicitly distinguish between exact preservation of the recurrence structure and numerical equality of the logits, and adjust the wording in the abstract and introduction accordingly; otherwise the 'exact' claim is misleading.
  3. [Table 1] The Llama-3.2-1B full-attention baseline timing at 8192 tokens (0.026s) is internally inconsistent: it is barely larger than the 4096-token time (0.024s) and far too small relative to the 16384-token time (0.376s), which is close to the ~4x value expected under quadratic scaling from 8192 at ~0.096s. This inconsistency affects the credibility of the baseline and of the speedup figures derived from it, such as those in Table 8. The authors should correct the entry or explain the measurement conditions (e.g., a typo or a caching effect), and ideally report the number of trials and variance for the timing measurements.
minor comments (5)
  1. [Section 4.5] The sentence 'However, we the effect of error accumulation on downstream tasks is negligible' appears to be missing a verb; it should read something like 'However, we find that the effect of error accumulation on downstream tasks is negligible.'
  2. [Section 2.1, Eq. (1)] The notation in Eq. (1) is confusing: the input list [M_{s-1}, H_{s-1}, M_{s-1}] repeats the memory state, and the output is written as [_, _, M_s]. This looks like a typesetting artifact and should be corrected to match the original RMT formulation or clarified in the text.
  3. [Figure 6] The 'Ideal Even Load' curve is not precisely defined in the main text; the caption says it assumes all segment computations run with maximum achievable FLOPS, but the authors should specify how this ideal time was computed (e.g., measured peak FLOPS vs. Achieved FLOPS) so that the comparison is reproducible.
  4. [Tables 8 and 9] The table captions say 'speedup' but the columns contain values that appear to be execution times (e.g., 0.085s at 4096 tokens in Table 8); please clarify whether the tables report absolute times or speedup ratios, and label the columns consistently.
  5. [Global] There are several minor typos and inconsistent capitalizations across the paper, including 'LLama' versus 'Llama' and 'inplementation' in the Table 9 caption; a careful proofreading pass is recommended.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the DAG-scheduling lemma and the empirical speedups are self-contained, and the relevant self-citations are used only as architectural context, not as fitted inputs or predictions.

full rationale

The paper's central contribution is a scheduling algorithm plus an empirical evaluation. Lemma 3.1 proves a purely graph-theoretic lower bound and a matching schedule for the layer-segment DAG whose edges are defined explicitly in Section 3.1: each (segment, layer) node depends only on (segment-1, layer) and (segment, layer-1). The proof is self-contained and does not rely on any fitted constant, benchmark number, or prior result. The reported speedups are measured wall-clock times against ARMT and Llama baselines on A100/H100 GPUs; they are not derived from the method's definition, nor are any parameters fitted to produce them. The paper does cite the ARMT paper [28] by overlapping authors as the source of the architecture and of the PRMT family, but this is load-bearing only in the sense that the method targets that architecture; the correctness of the diagonal schedule does not depend on accepting [28]'s empirical claims, and the benchmark comparison independently exercises the implementation. The one genuine concern is a scope mismatch, not circularity: the abstract and title refer broadly to RMTs, while Section 2.2 and the Limitations section explicitly state that standard RMT, whose final-layer memory flows into all layers of the next segment, is not directly compatible. That limitation is honestly disclosed and affects the breadth of the claim, not the derivation chain. The technical result is therefore self-contained against external benchmarks and exhibits no reduction of a prediction to its own inputs.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The method has no fitted free parameters. The ledger instead shows scope conditions: only layer-recurrent models with uniform layer shapes qualify, and the speedup premise is a hardware scaling behavior measured in Figures 4-5. No new physical or conceptual entities are introduced; diagonal groups are a scheduling construct.

assumptions (4)
  • domain assumption Layer-level dependency graph: each (segment, layer) node depends only on (segment-1, layer) and (segment, layer-1).
    Invoked in Section 3.1 and Figure 2; excludes original RMT, which the title nevertheless references.
  • domain assumption Uniform layer geometry: all layers share identical shapes (hidden size, segment length, number of memory tokens) so weights can be stacked into a single grouped GEMM.
    Stated in Section 3.3 and Limitations; heterogeneous layers would require manual grouping.
  • domain assumption GPU FLOPS scale with group size similarly to batch size, as measured for grouped GEMM and attention.
    This is the premise for the claimed speedup; Figures 4 and 5 show it holds from group size 4 on A100 and H100.
  • standard math Topological sorting of the layer-segment DAG by key i+j gives each node its earliest feasible group.
    Used in Lemma 3.1; the longest-path argument is elementary and correct.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Diagonal Batching Unlocks Parallelism in Recurrent Memory Transformers for Long Contexts." pith.science (2026). https://pith.science/paper/RRAHKHQC

@misc{pith2026250605229,
  author       = {Pith},
  title        = {Pith review of: Diagonal Batching Unlocks Parallelism in Recurrent Memory Transformers for Long Contexts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RRAHKHQC}},
  note         = {Machine review of arXiv:2506.05229}
}
read the original abstract

Transformer models struggle with long-context inference due to their quadratic time and linear memory complexity. Recurrent Memory Transformers (RMTs) offer a solution by reducing the asymptotic cost to linear time and constant memory usage. However, their memory update mechanism leads to sequential execution, causing a performance bottleneck. We introduce Diagonal Batching, a scheduling scheme that unlocks parallelism across segments in RMTs while preserving exact recurrence. This approach eliminates the sequential constraint, enabling efficient GPU inference even for single long-context inputs without complex batching and pipelining techniques. Because the technique is purely a run-time computation reordering, existing RMT models adopt it with no retraining. Applied to a LLaMA-1B ARMT model, Diagonal Batching yields a 3.3x speedup over standard full-attention LLaMA-1B and a 1.8x speedup over the sequential RMT implementation on 131,072-token sequences. By removing sequential bottleneck, Diagonal Batching reduces inference cost and latency, thereby strengthening RMTs as a practical solution for real-world, long-context applications.

Figures

Figures reproduced from arXiv: 2506.05229 by the authors.

Figure 1
Figure 1. Diagonal Batching enables the Recurrent Memory Transformers (ARMT) to process 128k tokens sequences 3.3x faster than the LLama-3.2-1B model, with 167.1x memory savings. These results were obtained using an A100 GPU, and the segment size for the ARMT was set to 1,024 tokens. arXiv:2506.05229v1 [cs.LG] 5 Jun 2025 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Unlocking Parallelism in Recurrent Memory Transformers (RMT) with Diagonal Batching. Left: Standard RMT splits long sequences and processes segments sequentially. Each layer updates a memory state (mem0, mem1, . . . ) and the final-layer memory state is fed as input to the next segment; red arrows highlight the recurrent dependencies that force strictly sequential execution. Center: Parallel RMT generalizes a family… view at source ↗
Figure 3
Figure 3. Baseline compute schedule in PRMTs leads to n_layers x n_segments sequential operations. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Cutlass Group GEMM scales similarly to batch size 1 Linear layer’s matrix multiplication, [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Diagonal batching increase attention performance by treating groups as batches—similar to [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Ideal batch-size scaling vs grouped batching on Nvidia A100 for Llama models, time per [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Extending LLM Context via Associative Recurrent Memory

    cs.CL 2026-07 conditional novelty 5.0 of 10

    ARMT-augmented 1B-class LLMs, trained with continued pretraining, synthetic long data, curriculum, and selective memory layers, keep in-window quality while generalizing past 32k–65k tokens at constant memory and ~30%...

Reference graph

Works this paper leans on

37 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895–4901, 2023

    2023

  2. [2]

    Beyond attention: Breaking the limits of transformer context length with recurrent memory.Proceedings of the AAAI Conference on Artificial Intelligence, 38(16):17700–17708, Mar

    Aydar Bulatov, Yuri Kuratov, Yermek Kapushev, and Mikhail Burtsev. Beyond attention: Breaking the limits of transformer context length with recurrent memory.Proceedings of the AAAI Conference on Artificial Intelligence, 38(16):17700–17708, Mar. 2024

    work page 2024

  3. [3]

    Recurrent memory transformer.Advances in Neural Information Processing Systems, 35:11079–11091, 2022

    Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. Recurrent memory transformer.Advances in Neural Information Processing Systems, 35:11079–11091, 2022

    2022

  4. [4]

    Transformer-XL: Attentive language models beyond a fixed-length context

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988, Florence, Italy, July 2019. Association for Computational Linguistics

    work page 2019

  5. [5]

    FlashAttention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024

    2024

  6. [6]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  7. [7]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. InInternational Conference on Machine Learning, pages 10041–10071. PMLR, 2024

    2024

  8. [8]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, 2019

    2019

Show all 37 references
  1. [10]

    GPTQ: Accurate post-training compression for generative pretrained transformers.arXiv preprint arXiv:2210.17323, 2022

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training compression for generative pretrained transformers.arXiv preprint arXiv:2210.17323, 2022

    2022 arXiv

  2. [11]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  3. [12]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  4. [13]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. InInternational Conference on Learning Representations, 2021

    2021

  5. [14]

    Block- recurrent transformers

    DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. Block- recurrent transformers. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors,Advances in Neural Information Processing Systems, 2022

    2022

  6. [15]

    Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models.arXiv preprint arXiv:2309.14509, 2023

    Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models.arXiv preprint arXiv:2309.14509, 2023

    2023 arXiv

  7. [16]

    Repeat after me: Transformers are better than state space models at copying

    Samy Jelassi, David Brandfonbrener, Sham M Kakade, and Eran Malach. Repeat after me: Transformers are better than state space models at copying. InInternational Conference on Machine Learning, pages 21502–21521. PMLR, 2024. 10

    2024

  8. [17]

    Babilong: Testing the limits of llms with long context reasoning-in-a-haystack

    Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of llms with long context reasoning-in-a-haystack. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, ed...

    2024

  9. [18]

    xformers: A modular and hack- able transformer modelling library

    Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xformers: A modular and hack- able transforme...

    2022

  10. [19]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration.Proceedings of Machine Learning and Systems, 6:87–100, 2024

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration.Proceedings of Machine Learning and Systems, 6:87–100, 2024

    2024

  11. [20]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024

    2024 arXiv

  12. [21]

    Ringattention with blockwise transformers for near-infinite context

    Hao Liu, Matei Zaharia, and Pieter Abbeel. Ringattention with blockwise transformers for near-infinite context. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  13. [22]

    The illusion of state in state-space models

    William Merrill, Jackson Petty, and Ashish Sabharwal. The illusion of state in state-space models. InInternational Conference on Machine Learning, pages 35492–35506. PMLR, 2024

    2024

  14. [23]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023

    2023

  15. [24]

    RWKV: Reinventing RNNs for the transformer era

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartłomiej Koptyra, Hay...

    2023

  16. [25]

    Language models are unsupervised multitask learners

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019

    2019

  17. [26]

    Rae, Anna Potapenko, Siddhant M

    Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling. InInternational Conference on Learning Representations, 2020

    2020

  18. [27]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv pre...

    2024 arXiv

  19. [28]

    Associative recurrent memory transformer.CoRR, 2024

    Ivan Rodkin, Yuri Kuratov, Aydar Bulatov, and Mikhail Burtsev. Associative recurrent memory transformer.CoRR, 2024

    2024

  20. [29]

    Linear transformers are secretly fast weight programmers, 2021

    Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers, 2021

    2021

  21. [30]

    Fast transformer decoding: One write-head is all you need.arXiv preprint arXiv:1911.02150, 2019

    Noam Shazeer. Fast transformer decoding: One write-head is all you need.arXiv preprint arXiv:1911.02150, 2019

    1911 arXiv

  22. [31]

    What formal lan- guages can transformers express? a survey.Transactions of the Association for Computational Linguistics, 12, 2024

    Lena Strobl, William Merrill, Gail Weiss, David Chiang, and Dana Angluin. What formal lan- guages can transformers express? a survey.Transactions of the Association for Computational Linguistics, 12, 2024. 11

    2024

  23. [32]

    End-to-end memory networks, 2015

    Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-to-end memory networks, 2015

    2015

  24. [33]

    Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621, 2023

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621, 2023

    2023 arXiv

  25. [34]

    Attention is All you Need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is All you Need. InAdvances in neural information processing systems, pages 5998–6008, 2017

    2017

  26. [35]

    Memory networks

    Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. In Yoshua Bengio and Yann LeCun, editors,3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015

    2015

  27. [36]

    Roofline: an insightful visual performance model for multicore architectures.Communications of the ACM, 52(4):65–76, 2009

    Samuel Williams, Andrew Waterman, and David Patterson. Roofline: an insightful visual performance model for multicore architectures.Communications of the ACM, 52(4):65–76, 2009

    2009

  28. [37]

    Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation

    Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Findings of the Association for Computational Linguistics:...

    2023

  29. [38]

    Parallelizing linear transformers with the delta rule over sequence length

    Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems. 12 Task Length, tokens LLama-3.2-1B ARMT LLama-3.2-1B ARM...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.