Pith. sign in

REVIEW 4 major objections 4 minor 49 references

A linear-time sequence model preserves token memories over long contexts by routing writes to a sparse, input-dependent subset of memory slots, and shows near-perfect recall at 16x its training length.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 02:38 UTC pith:5CFSNSLR

load-bearing objection Raven is a genuinely interesting design point for recall in linear-time models, but the paper's headline configuration is internally inconsistent — the 400M results match a TopK=128 setup while the text recommends TopK=32. the 4 major comments →

arxiv 2607.25357 v1 pith:5CFSNSLR submitted 2026-07-28 cs.LG cs.AI

Raven: High-Recall Sequence Modeling with Sparse Memory Routing

Arshia Afzal , Aviv Bick , Eric P. Xing , Volkan Cevher , Albert Gu This is my paper
classification cs.LG cs.AI
keywords routing slot memoriessparse routinglinear-time sequence modelslong-context recallstate-space modelssliding-window attentionlength generalizationmemory slots
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Raven, a linear-time sequence model that keeps a fixed set of memory slots but updates only a sparse, input-dependent subset at each step, with decay applied only to those slots. The authors argue this combination fills a gap between dense state-space models, which overwrite all memory at every step and suffer interference, and sliding-window attention, which preserves tokens but evicts them by position. They report that Raven maintains near-perfect accuracy on needle-in-a-haystack retrieval at context lengths up to 16 times its training length, where both dense SSMs and windowed attention collapse. If correct, this would give fixed-size recurrent models a practical route to long-context recall without quadratic attention or short-range convolutions.

Core claim

The central claim is that sparse, input-dependent routing to memory slots, paired with explicit decay on only the selected slots, lets a linear-time model preserve individual token memories over long contexts. The authors formalize a class of 'routing slot memories' in which every slot is updated independently and a router vector decides write intensity per slot; sliding-window attention and state-space models appear as extremes on a sparsity axis. Raven uses a sigmoid-scored Top-K router and a per-head scalar decay, and the paper reports that this design achieves ≥99% retrieval accuracy up to 16K tokens at 400M scale, and >91% at 32K, 16x its training length, while remaining competitive on

What carries the argument

The central object is the Routing Slot Memory, a slot-separable linear recurrence St = (1 - rt) ⊙ S(t-1) + rt ⊙ (Dt S(t-1) At + Ut), where rt is a sparse input-dependent router and decay applies only to updated slots. In Raven, rt is a sigmoid-scored Top-K vector normalized to sum to one, and the write content is the stacked key/value vector; this lets the model protect unselected slots from overwrite while gradually forgetting selected ones, generalizing both sliding-window attention and dense state-space models.

Load-bearing premise

The learned router must distribute tokens across slots rather than collapsing onto a few; the paper's only safeguard is Gumbel noise during training, with no formal guarantee that specialization holds at larger scales or on very different data.

What would settle it

Train a Raven-scale model on a distribution where many retrieval-critical tokens compete for the same slots and measure whether recall degrades; or track the fraction of slots ever selected in a long run and check whether collapse drives most of them to zero usage.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, linear-time recurrent models can achieve long-context recall without short-range convolutions, simplifying the architecture.
  • Models trained at one context length can extrapolate to many multiples of it without a length curriculum, because each slot naturally sees a different effective sequence length.
  • In hybrid architectures, replacing dense recurrent components with Raven's sparse-routed memory preserves long-range retrieval while retaining linear-time inference.
  • The routing/forgetting decoupling gives future designs a separate axis to control: where to write and how long to keep, rather than conflating both in a decay matrix.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • One could apply the same sparse-slot principle to cache management in Transformers, routing important tokens to protected slots to improve long-context QA; the paper does not test this.
  • Monitoring slot-selection entropy during training could serve as a practical early warning for router collapse, and the Gumbel-noise safeguard may need re-tuning at larger scale.
  • The effective-sequence-length analysis suggests a testable prediction: training with shorter contexts may already expose the model to longer effective horizons, so extrapolation should scale smoothly with the number of slots.
  • Combining Raven's slot-level routing with head-level routing could produce even sparser writes, a direction the paper flags as future work.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Routing Slot Memories (RSMs), a framework that unifies linear sequence models by making the write location explicit through a routing vector, and introduces Raven, an instantiation with sparse input-dependent top-K routing and slot-wise decay. The authors argue that this combination sits between dense SSMs and sliding-window attention and preserves individual token memories far beyond the training context. They evaluate Raven at 400M and 800M scales on NIAH passkey retrieval, real-world extraction tasks (SWDE, FDA, SQuAD), zero-shot language modeling, and hybrid architectures. The paper also includes ablations of router design, memory shape, and top-K sparsity, plus analyses of slot specialization via effective sequence length and memory visualizations.

Significance. If the empirical claims hold, Raven is a valuable architecture: it is simple, linear-time, needs no convolutions, and reportedly retains high recall at 16x the training length, a regime where dense SSMs and SWA both fail. The RSM framework is a clean unifying perspective and the formal slot-separability proposition is a nice contribution. The paper is also candid about the router-collapse risk and provides a reasonable set of ablations. However, the strength of the paper rests on a small number of benchmark numbers, and the current manuscript has an internal inconsistency about the exact configuration used for the headline results, plus a lack of any variance information. These issues are fixable but need to be addressed before the empirical claims can be fully trusted.

major comments (4)
  1. [§6.4 vs. Table 2 and Table 7] The headline 400M Raven row in Table 2 (SWDE 34.1, FDA 22.7, SQuAD 35.4, NIAH-1 32K 91.4, NIAH-2 32K 23.0, NIAH-3 8K 13.4) is numerically identical to the M=256/K=128 row of Table 7, up to one NIAH-1 4K cell (99.8 vs. 99.4). Yet §6.4 concludes that K=32 "strikes the best balance," and Tables 5 and 6 use K=32. The paper never states which K is used in Table 2. This is load-bearing for reproducibility: K is not a nuisance parameter, since Table 7 shows NIAH-2 at 32K varies from 2.4 (K=32) to 23.0 (K=128). In addition, the sentence in §6.4 that "Smaller K ... improving recall on SWDE" is contradicted by Table 7 for M=256, where SWDE is 31.5 at K=32 and 34.1 at K=128. Please state the exact configuration of Table 2, reconcile it with §6.4, and if K=128 is the final choice, report the K=32 configuration on the headline benchmarks.
  2. [§5.1 and Tables 5–7] The central empirical claim is partly circular because the same benchmarks used as evidence in Table 2 appear in the model-selection ablations (Tables 5–7). The paper selects router type, Gumbel noise, M, K, and α using NIAH/SWDE/FDA/SQuAD performance, and then reports the best configuration on the same tasks as a headline result. The text in §6.2.3 says "we select the best-performing configuration per Table 6," but no held-out split or selection protocol is described. To support the claim that sparse routing itself (not the particular hyperparameter choice) provides the reported benefits, the authors should either pre-register a fixed configuration and report all configurations, or use a held-out validation split for model selection and then evaluate the final model only once.
  3. [Tables 2–4, 7, 9] All benchmark numbers are point estimates with no standard deviations, confidence intervals, or seed counts. NIAH is known to be noisy, and several of the claimed advantages are small (e.g., Raven 400M vs. GLA on SWDE, 34.1 vs. 29.0; Raven 400M vs. GDN on SWDE, 34.1 vs. 29.5). Without multiple seeds or at least a stability analysis, these differences may not be significant. Please report the number of runs, the aggregation method, and ideally error bars or at least the variance across seeds for the main tables.
  4. [§4.1, Eq. (20)] The router normalization α is set to 1 for the 400M model and 4 for the 800M model, but no ablation or sensitivity analysis is provided for this choice. Since α directly scales the routing weights and interacts with the top-K threshold and the decay rate, a scale-dependent heuristic is not sufficient justification. Please provide an ablation over α (at least for one scale) or a principled motivation for why α=1 and α=4 are correct.
minor comments (4)
  1. [Table 4 vs. Table 9] The 400M Raven hybrid row in Table 4 reports NIAH-3 at 1K as 90.0, while Table 9 reports 89.6 for the same configuration. Please verify and unify these numbers.
  2. [Table 2 caption] The caption mentions "Rec. mem. and Conv. mem." but the table body only shows a single memory column; clarify the column layout and what "12.5 / 0.0" means.
  3. [Paper front matter] The "Raven Code" and "Raven Blog" items are not hyperlinked; if code is meant to be released, provide a working URL or repository identifier.
  4. [§6.4] The phrase "K=32 ... strikes the best balance" is undefined. Is it based on a single task average, a qualitative judgment, or a computational constraint? Please state the criterion used.

Circularity Check

0 steps flagged

No definitional or derivation-to-data circularity; Raven's empirical claims are self-contained. Minor self-citations are motivational only, and the K=32-vs-K=128 configuration issue is a fitting/reporting concern, not circularity.

full rationale

The paper's derivation chain is architectural rather than predictive: RSM (Eq. 11) and Raven's recurrence (Eq. 18) are definitions, and the benchmark numbers in Tables 2-4 are measurements from trained models. No equation is shown to reduce to another by construction, and no fitted parameter is renamed as a prediction. The 'missing combination' claim is a design-space observation backed by Table 1, not by a uniqueness theorem imported from the authors' prior work. Self-citations (e.g., Bick et al. 2025c for retrieval-head specialization; Afzal 2026 for position handling) are used as motivation/analogy and do not carry the empirical conclusion. The one substantive concern is configuration selection: Section 6.4 states 'We find that K=32 (out of M=256 slots, i.e., 12.5% occupancy) strikes the best balance,' while the headline 400M numbers in Table 2 numerically match the M=256/TopK=128 row of Table 7 (SWDE 34.1, FDA 22.7, SQuAD 35.4, NIAH-1 32K 91.4, NIAH-2 32K 8.8, NIAH-3 8K 1.0). The paper never states that the final model uses K=128 and does not report the recommended K=32 config on the headline benchmarks. This is a reproducibility/fitting concern about how much of the reported advantage was selected in-sample, but it is not a circular derivation: the qualitative sparse-routing result holds across multiple rows of Table 7, and the architecture's behavior is externally benchmarked rather than inferred from the RSM equations.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

No new physical entities are postulated; 'Raven' and 'Routing Slot Memories' are architectural constructs. The free parameters listed above are the main hand-fitted degrees of freedom that the central empirical claims rest on.

free parameters (3)
  • Router normalization α = α=1 (400M), α=4 (800M)
    Set by hand per model scale (Eq. 20) and not ablated. It controls the magnitude of routing weights and directly affects how strongly selected slots are decayed and overwritten.
  • Top-K write budget K = K=32 of M=256
    Selected from Table 7 sweep. The central recall claims depend on this sparsity level; smaller K improves SWDE, larger K approaches dense SSM behavior.
  • Number of memory slots M = 256
    Chosen as default after budget-matched sweeps in Table 7. The paper shows slot granularity drives retrieval quality, so this choice is consequential.
axioms (3)
  • domain assumption The learned router will specialize rather than collapse, without a load-balancing loss.
    Section 6.2.2 and Section 7.3. The paper's recall benefit requires that retrieval-critical tokens are routed to dedicated slots; Gumbel noise is the only mechanism preventing collapse, verified only at two scales.
  • domain assumption Passkey-style NIAH-1 retrieval is representative of the long-context recall that matters for real tasks.
    The headline 16x extrapolation claim is based on NIAH-1 (Section 5.1). Real-world SWDE/FDA/SQuAD gains are real but smaller, so the strongest claim depends on this representativeness.
  • domain assumption Decay-based relative-position encoding is sufficient without RoPE or convolutions.
    Section 4.2 and A.6. Raven drops short-range convolutions and RoPE, relying on slot-wise decay to encode position. This is supported empirically but not theoretically.

pith-pipeline@v1.3.0-alltime-deepseek · 29164 in / 10844 out tokens · 107733 ms · 2026-08-01T02:38:58.671178+00:00 · methodology

0 comments
read the original abstract

Long-context recall in linear-time sequence models highlights a tradeoff in how they write to memory. State-based linear models, such as state-space models (SSMs) and linear Transformers, write densely, updating the entire state for each newly arrived token, which leads to interference and makes specific past tokens hard to recover. Sliding-window attention (SWA) exhibits the opposite behavior: it writes sparsely by storing explicit token representations, but only within a fixed window, so recall drops once the relevant token is evicted. Interpolating between these models, we introduce Raven, a linear-time sequence model that maintains a fixed set of memory slots and, at each step, decays and updates only a selected subset via learned, input-dependent routing. This lets Raven mitigate SWA's position-based overwriting and hard eviction while reducing interference from dense state updates in SSMs, thereby preserving long-range content much more effectively. Across recall-intensive benchmarks, Raven is competitive with or outperforms prior linear-time baselines, achieving strong long-context recall where both SWA and SSMs sharply degrade. It remains effective when extrapolating to context lengths as large as 16x its training length, with similar gains in hybrid architectures.

Figures

Figures reproduced from arXiv: 2607.25357 by Albert Gu, Arshia Afzal, Aviv Bick, Eric P. Xing, Volkan Cevher.

Figure 1
Figure 1. Figure 1: Raven Overview. Visualization of three different sequence mixers using Routing Slot Memories, with different router choices. (a) SWA memory allocation as first-in-first-out strategy using a one-hot vector et as router. (b) SSM memory allocation, which projects each token in all memory slots using a dense, all-ones router 1M (c) Raven memory allocation, which uses a selective router for writes. Visualizatio… view at source ↗
Figure 2
Figure 2. Figure 2: Effective Sequence Length Visualization. For a hidden state with M = 4 memory slots and a sequence of length T = 6, each memory slot processes a different effective sequence length (ESL), depending on the router. A q k v Linear Projection R RSM A q v SSM Conv N X N N Nonlinearity X N Raven SSMs N N N GDN, GLA, Mamba2, … Sequence Mixer k N N [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: NIAH Results. (Left) NIAH-1 for 400M and 800M SSMs and transformers. (Right) NIAH-2 and NIAH-3 for 800M hybrids. compares Raven against Mamba-2, GLA, GDN, and strong Transformer baselines including FoX (Lin et al., 2025) at both 400M and 800M parameter scales across standard zero-shot benchmarks. At 400M parameters, Raven matches or exceeds Mamba-2 and GLA on average accuracy while achieving the best Lamba… view at source ↗
Figure 5
Figure 5. Figure 5: Raven Effective Sequence Length. Normalized effective sequence length for a NIAH-1 sample at sequence length 16K. SWA : each token stored in exactly one slot (FIFO). Raven : hidden state St for layer 1, head 1 (256 slots, Top32). SSM : each token stored in all slots with decay. Slots are reordered by usage frequency; results are for 400M parameter models. 7.1 Effective Sequence Length in Routed Memory The … view at source ↗
Figure 6
Figure 6. Figure 6: Raven Memory Dynamics. Memory allocation for two different heads of Raven on a synthetic NIAH-style task. Red slots store tokens that are important for retrieval (e.g., passwords), Green slots store non-retrieval tokens, and Blue slots are shared memory slots between the two type of tokens (red and green). Different heads allocate different amounts of slots to retrieval-important tokens, showing non-unifor… view at source ↗
Figure 7
Figure 7. Figure 7: Memory Visualization. Memory of Raven while being asked the password within the prompt. The figures show the hidden state of Raven St at the end of processing the prompt and answering to the question (time point t = L) for (Top) Layer 24, (Middle) Layer 12, (Bottom) Layer 2 for the first head among 4 heads at 400M scale. Red slots are storing only retrieval (red) tokens, green slots are memory slots only s… view at source ↗
Figure 8
Figure 8. Figure 8: Effective Sequence Length of Raven. The normalized effective sequence length for a NIAH-1 sample with a sequence length of 16K. Results are from the first, 12th, and last (24th) layers, for all four heads of the 400M model. Slots are reordered from highest to lowest [PITH_FULL_IMAGE:figures/full_fig_p028_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

49 extracted references · 35 linked inside Pith

  1. [5]

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi

    URL https: //arxiv.org/abs/2602.11374. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language,

    arXiv

  2. [7]

    Yingfa Chen, Xinrong Zhang, Shengding Hu, Xu Han, Zhiyuan Liu, and Maosong Sun

    URLhttps://arxiv.org/abs/2306.15595. Yingfa Chen, Xinrong Zhang, Shengding Hu, Xu Han, Zhiyuan Liu, and Maosong Sun. Stuffed mamba: Oversized states lead to the inability to forget,

    Pith/arXiv arXiv

  3. [10]

    Mom: Linear sequence modeling with mixture-of-memories

    Jusen Du, Weigao Sun, Disen Lan, Jiaxi Hu, and Yu Cheng. Mom: Linear sequence modeling with mixture-of-memories. arXiv preprint arXiv:2502.13685,

    arXiv

  4. [11]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Pith/arXiv arXiv

  5. [15]

    URLhttps://arxiv.org/abs/2402.01032. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts.arXiv preprint arXiv:2401.04088,

    Pith/arXiv arXiv

  6. [16]

    Peng Jin, Bo Zhu, Li Yuan, and Shuicheng Yan

    URLhttps://kexue.fm/archives/11320. Peng Jin, Bo Zhu, Li Yuan, and Shuicheng Yan. Moh: Multi-head attention as mixture-of-head attention.arXiv preprint arXiv:2410.11842,

    arXiv

  7. [18]

    Forgetting transformer: Softmax attention with a forget gate.arXiv preprint arXiv:2503.02130,

    Zhixuan Lin, Evgenii Nikishin, Xu Owen He, and Aaron Courville. Forgetting transformer: Softmax attention with a forget gate.arXiv preprint arXiv:2503.02130,

    Pith/arXiv arXiv

  8. [19]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    Pith/arXiv arXiv

  9. [20]

    William Merrill, Jackson Petty, and Ashish Sabharwal

    URLhttps://arxiv.org/abs/2509.19633. William Merrill, Jackson Petty, and Ashish Sabharwal. The illusion of state in state-space models.arXiv preprint arXiv:2404.08819,

    arXiv

  10. [21]

    Landmark attention: Random-access infinite context length for transformers

    Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers. arXiv preprint arXiv:2305.16300,

    Pith/arXiv arXiv

  11. [22]

    Selective rotary position embedding.arXiv preprint arXiv:2511.17388,

    Sajad Movahedi, Timur Carstensen, Arshia Afzal, Frank Hutter, Antonio Orvieto, and Volkan Cevher. Selective rotary position embedding.arXiv preprint arXiv:2511.17388,

    Pith/arXiv arXiv

  12. [23]

    Yuqi Pan, Yongqi An, Zheng Li, Yuhong Chou, Ruijie Zhu, Xiaohui Wang, Mingxuan Wang, Jinqiao Wang, and Guoqi Li

    URL https: //arxiv.org/abs/2502.20339. Yuqi Pan, Yongqi An, Zheng Li, Yuhong Chou, Ruijie Zhu, Xiaohui Wang, Mingxuan Wang, Jinqiao Wang, and Guoqi Li. Scaling linear attention with sparse state expansion.arXiv preprint arXiv:2507.16577,

    Pith/arXiv arXiv

  13. [24]

    20 Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole

    URL https://arxiv.org/abs/2402.04248. 20 Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. InInternational Conference on Learning Representations (ICLR),

    Pith/arXiv arXiv

  14. [25]

    URL https://arxiv.org/abs/2309. 00071. Hao Peng, Jungo Kasai, Nikolaos Pappas, Dani Yogatama, Zhaofeng Wu, Lingpeng Kong, Roy Schwartz, and Noah A Smith. Abc: Attention with bounded-memory control.arXiv preprint arXiv:2110.02488,

    Pith/arXiv arXiv

  15. [26]

    URL https://arxiv.org/abs/2108.12409. Qwen. Latest advancements. https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd&from= research.latest-advancements-list,

    Pith/arXiv arXiv

  16. [27]

    Pranav Rajpurkar, Robin Jia, and Percy Liang

    Accessed: 2025-11-03. Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for squad.arXiv preprint arXiv:1806.03822,

    Pith/arXiv arXiv 2025

  17. [28]

    Samba: Simple hybrid state space models for efficient unlimited context language modeling, 2024.URL https://arxiv.org/abs/2406.07522, 2406:07522,

    Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling, 2024.URL https://arxiv.org/abs/2406.07522, 2406:07522,

    Pith/arXiv arXiv 2024

  18. [29]

    Understanding and improving length generalization in recurrent models.arXiv preprint arXiv:2507.02782,

    Ricardo Buitrago Ruiz and Albert Gu. Understanding and improving length generalization in recurrent models.arXiv preprint arXiv:2507.02782,

    arXiv

  19. [30]

    Imanol Schlag, Kazuki Irie, and J¨ urgen Schmidhuber

    URLhttps://arxiv.org/abs/1907.10641. Imanol Schlag, Kazuki Irie, and J¨ urgen Schmidhuber. Linear transformers are secretly fast weight programmers. In International Conference on Machine Learning, pages 9355–9366. PMLR,

    Pith/arXiv arXiv 1907

  20. [32]

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov

    URL https://huggingface.co/datasets/cerebras/ SlimPajama-627B. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting.The journal of machine learning research, 15(1):1929–1958,

    1929

  21. [34]

    Learning to (learn at test time): Rnns with expressive hidden states.arXiv preprint arXiv:2407.04620,

    Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, et al. Learning to (learn at test time): Rnns with expressive hidden states.arXiv preprint arXiv:2407.04620,

    Pith/arXiv arXiv

  22. [35]

    Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621,

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621,

    Pith/arXiv arXiv

  23. [36]

    Kimi linear: An expressive, efficient attention architecture.arXiv preprint arXiv:2510.26692,

    Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, Jiaxi Hu, Fanqing Meng, Chengyin Liu, Xin Men, Songlin Yang, Zhiyuan Li, et al. Kimi linear: An expressive, efficient attention architecture.arXiv preprint arXiv:2510.26692,

    Pith/arXiv arXiv

  24. [38]

    Junxiong Wang, Daniele Paliotta, Avner May, Alexander M

    URL https://arxiv.org/abs/ 2406.07887. Junxiong Wang, Daniele Paliotta, Avner May, Alexander M. Rush, and Tri Dao. The mamba in the llama: Distilling and accelerating hybrid models,

    Pith/arXiv arXiv

  25. [39]

    Kaiyue Wen, Xingyu Dang, and Kaifeng Lyu

    URLhttps://arxiv.org/abs/2408.15237. Kaiyue Wen, Xingyu Dang, and Kaifeng Lyu. Rnns are not transformers (yet): The key bottleneck on in-context retrieval,

    Pith/arXiv arXiv

  26. [40]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al

    URLhttps://arxiv.org/abs/2402.18510. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  27. [41]

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim

    URLhttps://github.com/fla-org/flash-linear-attention. Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training.arXiv preprint arXiv:2312.06635,

    Pith/arXiv arXiv

  28. [42]

    Gated delta networks: Improving mamba2 with delta rule.arXiv preprint arXiv:2412.06464, 2024a

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule.arXiv preprint arXiv:2412.06464, 2024a. Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length.arXiv preprint arXiv:2406.06484, 2024b. Rowan Zellers, Ari Holtzman, Yonatan ...

    Pith/arXiv arXiv

  29. [43]

    Tianyuan Zhang, Sai Bi, Yicong Hong, Kai Zhang, Fujun Luan, Songlin Yang, Kalyan Sunkavalli, William T Freeman, and Hao Tan

    URLhttps://arxiv.org/abs/1905.07830. Tianyuan Zhang, Sai Bi, Yicong Hong, Kai Zhang, Fujun Luan, Songlin Yang, Kalyan Sunkavalli, William T Freeman, and Hao Tan. Test-time training done right.arXiv preprint arXiv:2505.23884, 2025a. Yifan Zhang, Zhen Qin, and Quanquan Gu. Higher-order linear attention.arXiv preprint arXiv:2510.27258, 2025b. URL https://arx...

    Pith/arXiv arXiv 1905

  30. [44]

    Understanding transformer from the perspective of associative memory.arXiv preprint arXiv:2505.19488,

    Shu Zhong, Mingyu Xu, Tenglong Ao, and Guang Shi. Understanding transformer from the perspective of associative memory.arXiv preprint arXiv:2505.19488,

    Pith/arXiv arXiv

  31. [45]

    Then, S′ t[i]−S t[i] =D t[ik] ∆At ̸=0

    Now consider two states St−1 and S′ t−1 that differ only in rowk, with S′ t−1[ℓ] = ( St−1[k] + ∆, ℓ=k, St−1[ℓ], ℓ̸=k. Then, S′ t[i]−S t[i] =D t[ik] ∆At ̸=0. Thus, modifying only row k̸=i changes St[i], contradicting row-separability. Therefore all off-diagonal entries of Dt must be zero, and hence for slot-separable updateD t must be diagonal. A.3 Experim...

    2024

  32. [48]

    One can look at Raven update as applying sparse TTTand only updating selected slots

    the regularizer vanishes, so that slot incurs no weight decay and its content is preserved exactly. One can look at Raven update as applying sparse TTTand only updating selected slots. Models using the Delta update rule, such as DeltaNet and KDA, apply a different online learning objective, which takes the 26 Table 10: An overview of different attention m...

    2024

  33. [49]

    (2025) for more details on the TTT framework

    −⟨S⊤t−1kt,vt⟩+ 1 2 pDiag(1−at)St−1 2 F St = Diag(at)St−1+ktv⊤t Raven −(1−exp(atrt))⟨S⊤t−1kt,vt⟩+ 1 2 p1−exp(atrt)St−1 2 F St =S t−1Diag(exp(atrt)) + (1− exp(atrt))ktv⊤t form: L(S) = 1 2 ∥S⊤kt −v t∥2,(29) This online update rule has connections toAssociative Memoryintroduced in Hopfield Networks (Hopfield, 1982).We refer to Table 7 of Team et al. (2025) fo...

    1982

  34. [256]

    ot = (Sv t )⊤ softmax Sk t qt

    A.4 Raven Recurrence Raven Recurrence: mt =σ(W xt),g t = ( mt[i]i∈Top K(mt) 0.0o.w ,r t = gt α PM i=1 gt[i] , a t =−SoftPlus(w ⊤xt).exp(∆), Sk t = exp(atrt)⊙S k t−1 + (1−exp(a trt))k⊤ t ,S v t = exp(atrt)⊙S v t−1 + (1−exp(a trt))v⊤ t . ot = (Sv t )⊤ softmax Sk t qt . A.5 Efficient Training using Gated Linear Attention As Raven recurrence for keys and valu...

    2023

  35. [1991]

    Categorical reparameterization with gumbel-softmax.arXiv preprint arXiv:1611.01144,

    Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax.arXiv preprint arXiv:1611.01144,

    Pith/arXiv arXiv

  36. [1992]

    Jamba: A hybrid transformer-mamba language model.arXiv preprint arXiv:2403.19887,

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer-mamba language model.arXiv preprint arXiv:2403.19887,

    Pith/arXiv arXiv

  37. [2014]

    Roformer: Enhanced transformer with rotary position embedding.arXiv preprint arXiv:2104.09864,

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.arXiv preprint arXiv:2104.09864,

    Pith/arXiv arXiv

  38. [2015]

    The learning rate follows a cosine scheduler with 1B tokens of warmup

    with a learning rate of 4×10 −4 (Yang et al., 2024a). The learning rate follows a cosine scheduler with 1B tokens of warmup. We utilize the flash-linear-attention2 repository for baselines as well as our codebase. A.3.2 Models Configuration Our experiments encompass several state-of-the-art linear models, including GLA, GSA, Mamba-2, GDN, and SWA. We foll...

    2024

  39. [2016]

    Query-key normalization for transformers.arXiv preprint arXiv:2010.04245,

    Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen. Query-key normalization for transformers.arXiv preprint arXiv:2010.04245,

    Pith/arXiv arXiv 2010

  40. [2017]

    Mesanet: Sequence modeling by locally optimal test-time training.arXiv preprint arXiv:2506.05233,

    Johannes von Oswald, Nino Scherrer, Seijin Kobayashi, Luca Versari, Songlin Yang, Maximilian Schlegel, Kaitlin Maile, Yanick Schimpf, Oliver Sieberling, Alexander Meulemans, et al. Mesanet: Sequence modeling by locally optimal test-time training.arXiv preprint arXiv:2506.05233,

    Pith/arXiv arXiv

  41. [2018]

    Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al

    URLhttps://arxiv.org/abs/1803.05457. Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models.arXiv preprint arXiv:2401.06066,

    Pith/arXiv arXiv

  42. [2019]

    18 Aaron Blakeman, Aaron Grattafiori, Aarti Basant, Abhibha Gupta, Abhinav Khattar, Adi Renduchintala, Aditya Vavre, Akanksha Shukla, Akhiad Bercovich, Aleksander Ficek, et al

    URLhttps://arxiv.org/abs/1911.11641. 18 Aaron Blakeman, Aaron Grattafiori, Aarti Basant, Abhibha Gupta, Abhinav Khattar, Adi Renduchintala, Aditya Vavre, Akanksha Shukla, Akhiad Bercovich, Aleksander Ficek, et al. Nemotron 3 nano: Open, efficient mixture-of-experts hybrid mamba-transformer model for agentic reasoning.arXiv preprint arXiv:2512.20848, 2025a...

    Pith/arXiv arXiv 1911

  43. [2020]

    Llamba: Scaling distilled recurrent models for efficient language processing.arXiv preprint arXiv:2502.14458, 2025a

    Aviv Bick, Tobias Katsch, Nimit Sohoni, Arjun Desai, and Albert Gu. Llamba: Scaling distilled recurrent models for efficient language processing.arXiv preprint arXiv:2502.14458, 2025a. Aviv Bick, Kevin Y. Li, Eric P. Xing, J. Zico Kolter, and Albert Gu. Transformers to ssms: Distilling quadratic knowledge to subquadratic models, 2025b. URLhttps://arxiv.or...

    Pith/arXiv arXiv

  44. [2021]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538,

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538,

    Pith/arXiv arXiv

  45. [2022]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. InConference on Learning and Modeling (COLM 2024),

    2024

  46. [2023]

    URL https://arxiv.org/abs/2312. 04927. Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher R ´e. Simple linear attention language models balance the recall-throughput tradeoff.arXiv preprint arXiv:2402.18668,

    Pith/arXiv arXiv

  47. [2024]

    Longformer: The long-document transformer.arXiv preprint arXiv:2004.05150,

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer.arXiv preprint arXiv:2004.05150,

    Pith/arXiv arXiv 2004

  48. [2025]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord

    URLhttps://arxiv.org/abs/2410.07145. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge,

    arXiv

  49. [2026]

    github.io/blog/2026/pe/

    URL https://arshiaafzal. github.io/blog/2026/pe/. Quentin Anthony, Yury Tokpanov, Skyler Szot, Srivatsan Rajagopal, Praneeth Medepalli, Rishi Iyer, Vasu Shyam, Anna Golubeva, Ansh Chaurasia, Xiao Yang, et al. Training foundation models on a full-stack amd platform: Compute, networking, and system design.arXiv preprint arXiv:2511.17127,

    arXiv 2026