Pith. sign in

REVIEW 4 major objections 4 minor 77 references

MARCH: Scaling Recurrent Memory with Content-Routed State Anchors

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MARCH claims that a recurrent sequence model can escape its fixed-size memory bottleneck by periodically saving cumulative state snapshots and routing each token to the snapshot whose content matches what it needs to recall.

desk verdict MARCH is a solid, well-ablated extension of Gated DeltaNet with checkpointed states and content routing; the retrieval gains look real, but the key-content assumption is not directly tested and all headline numbers are single runs. read the letter →

arxiv 2608.12435 v1 pith:7TQ2TYPV submitted 2026-08-12 cs.LG

Ming Zhang , Kaisen Yang , Shu Yu , Ermo Hua , Ning Ding , Xia Hu , Bowen Zhou , Chaochao Lu
show 1 more author
Youbang Sun
This is my paper · ORCID
classification cs.LG
keywords MARCHstateanchorscontent-basedroutingrecurrentmemorylong-contextretrievallinearattentiongateddeltarulein-context
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Recurrent language models compress the whole history into one fixed-size state, so old details get overwritten; Transformers avoid this by keeping every token, at quadratic cost. This paper claims that a recurrent model can get Transformer-like long-range recall without giving up its efficient recurrence: checkpoint the evolving state every so often, attach each checkpoint to a compact key that describes its content, and let each token route to the checkpoints whose content matches what it needs. The resulting architecture, MARCH, is trained from scratch on 50B tokens and consistently beats its Gated DeltaNet backbone and a log-linear variant on commonsense reasoning, LongBench, needle-in-a-haystack retrieval, and six in-context retrieval benchmarks, and it keeps working at 32K despite being trained at 16K. If the claim holds, recurrent models can scale memory with context length while keeping a linear computation path.

What carries the argument

The central object is the state anchor: a checkpoint $A_m = S_{b_m}$ of the cumulative recurrent state taken at periodic text boundaries, paired with an occurrence of a shared anchor token whose hidden representation becomes a compact routing key $\boldsymbol{\kappa}_m = W_k \mathbf{u}_m$. Because the anchor token is aligned with its checkpoint and reads it before the next layer, the key at deeper layers is conditioned on what the checkpoint actually contains. At every text token a routing query $\boldsymbol{\rho}_t$ scores all causally visible anchor keys plus a learned null candidate through a softmax, and the resulting probabilities weight the anchor states, which are read with the token's normal state-read query and added to the current-state readout. This turns the otherwise transient state trajectory into a persistent, content-addressable memory bank whose size grows with context length.

What would settle it

Retrain or fine-tune MARCH with the anchor-key gradient detached from the state-conditioned representation, for example by driving keys only from a learned per-position embedding, keeping the routing pathway otherwise intact, and compare 8K NIAH accuracy; if the gap over vanilla Gated DeltaNet survives, content-conditioned routing is not load-bearing, and if it collapses, the claim is confirmed.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the fixed-state bottleneck of linear recurrences is not the recurrence itself but the loss of earlier state versions. MARCH shows that periodically saving cumulative snapshots $A_m$ of the Gated DeltaNet state, giving each snapshot a learned content-conditioned key $\boldsymbol{\kappa}_m$, and aggregating the relevant snapshots with a softmax over visible anchor keys plus a null option restores direct access to earlier memory. The current-state readout $S_t q_t$ is preserved, and the historical readout $\sum_j \pi_{t,j} A_j q_t$ is added as a residual branch, so the native recurrent path is untouched. Across single-needle and multi-needle tasks this raises retrieval sharply, with single-needle NIAH at 8K climbing from 55.3 to 81.7, and the gains also appear on general-purpose benchmarks.

Load-bearing premise

The whole retrieval gain rides on the compact anchor keys truly encoding what their checkpoints contain, so that the routing query can separate useful snapshots from irrelevant ones; if the keys barely vary with state content, the softmax averages all checkpoints and the benefit disappears.

Editorial extensions

If this is right

  • Needle-in-a-haystack recall jumps from 55.3 to 81.7 on single-needle NIAH at 8K, and across all 24 NIAH task-length settings MARCH wins 19 and ties the remaining five against the stronger recurrent baseline; at 32K it keeps nonzero accuracy on all six tasks where the Transformers and the log-linear baseline score zero.
  • On LongBench, MARCH lifts the average from 11.9 for vanilla Gated DeltaNet and 12.5 for the log-linear variant to 14.9, a 25% relative gain over the backbone, concentrated in multi-document QA and summarization.
  • On six in-context retrieval benchmarks, MARCH raises the average from 20.5 to 23.3, with relative gains between 8% and 23% over the stronger Gated DeltaNet baseline on every task.
  • The checkpoint interval sets a controllable accuracy-memory trade-off: chunk size 512 is the best default, and Top-4 sparse routing keeps most of the benefit while more than doubling training throughput at 128K and exceeding FlashAttention-2 at that length.
  • Because routing is content-based and the bank grows with context, MARCH extrapolates to 32K from a 16K training length without introducing anchor-specific parameters for the new anchors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the anchor keys are truly content-addressable, the same routing layer should transfer to other recurrent backbones, such as Mamba-style selective states, without retraining from scratch; the paper's Fenwick-tree ablation already hints that the router is not tied to positional indexing.
  • The routing probabilities and null-route mass are a ready-made interpretability signal: they could identify which tokens depend on old context and which checkpoint supplies a recalled fact, something this paper does not analyze.
  • The growing anchor bank is effectively an external, content-addressed memory written by the recurrence, which suggests uses the paper does not explore, such as editing or consolidating memory at test time for continual learning.
  • Fixed-interval checkpointing is a convenience, not a requirement; replacing the constant chunk size with novelty-triggered checkpointing, where rapid state change generates more anchors, is a direct testable extension that could help fast-changing contexts.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces MARCH (Memory-Anchor Routing across Context History), an architecture that augments Gated DeltaNet by periodically checkpointing the cumulative recurrent state into a bank of state anchors. Each anchor is paired with a compact learned routing key; at each text token, a routing query scores all causally visible anchors plus a learned null option, and the resulting weighted average of anchor states is read and added as a residual to the current-state readout. The paper claims that after 50B-token pretraining, MARCH consistently outperforms vanilla Gated DeltaNet and Log-Linear Gated DeltaNet on commonsense reasoning, LongBench, RULER/NIAH, and in-context retrieval, while extrapolating to 32K beyond the 16K training length. The authors also provide ablations on checkpoint interval, routing dimension, Top-K sparsity, null option, and an I/O-aware fused implementation with throughput measurements.

Significance. If the empirical claims hold, MARCH is a useful architectural contribution: it decouples the total available recurrent memory from the fixed state dimension without modifying the underlying recurrence, and it brings query-dependent content routing to historical state checkpoints. The design is clearly specified, end-to-end trainable, and the ablations in Tables 4 and 5 isolate the main design choices. The efficiency discussion in Figure 4 is a strength, showing that dense and Top-K variants have a practical implementation path. The paper does not provide formal guarantees, and its significance rests entirely on the empirical comparisons, so the robustness of those comparisons is what determines whether the central claims are supported.

major comments (4)
  1. [§3.1, Eqs. (10)-(11); Table 5] The central retrieval claim requires the compact anchor keys κ_m^{(ℓ)} to encode the content of their aligned state checkpoints A_m, but this property is not demonstrated. As the paper itself notes, at layer 1 all anchor positions receive the same shared embedding ξ, so κ_m^{(1)} is identical for all m; content dependence first enters only through the single projection o_m^{(1)} = A_m W_q^{(1)} ξ. This one shared query direction could encode coarse properties such as state norm or recency rather than the key–value associations a future query needs. Table 5 shows that routing precision matters (Top-4 drops NIAH average from 51.33 to 44.85), but it does not establish that the selected anchors are content-selective. I ask for a direct diagnostic: compare learned routing against recency-only or random-key routing, measure routing entropy as a function of anchor age, or inspect selected anchors on a controlled insertion/retrieval task.
  2. [§4.1, Tables 4-5] The two free hyperparameters C=512 and d_r=64 are selected after inspecting the same benchmark suites that are used for the main results. Table 4 chooses C by comparing NIAH and in-context retrieval scores, and Table 5 chooses d_r by comparing aggregated benchmark averages. This selection-on-the-evaluation-set procedure can inflate apparent gains and makes the reported improvements harder to interpret. The authors should either use a separate validation split for hyperparameter choice or explicitly report the selection procedure and show that the main conclusions are stable across a small grid around the chosen values.
  3. [Tables 1-3] Every reported number in the main comparison tables is a single point estimate with no error bars, no multiple seeds, and no significance testing. Some of the headline differences are small, for example the commonsense average of 41.5 for MARCH versus 41.3 for the 21-layer Transformer and 41.4 for the 24-layer Transformer in Table 1, and a 14.9 versus 15.4 difference in the LongBench average in Table 2. Without variance information, the claim of 'consistent' improvement is not fully supported. I request at least two or three seeds for the main configurations, or an explicit statement of which reported numbers are single runs and which are averaged.
  4. [§4.2, Figure 3] The 32K extrapolation result is presented as a strong piece of evidence, but the figure and text do not report confidence intervals or repeated evaluations for the 32K condition. Since the authors state that all baselines score zero on the 32K multi-needle and several single-needle tasks, it is important to exclude the possibility of evaluation artifacts at long context (e.g., truncation or metric decoding issues) and to report exact per-task scores rather than only the plotted curves. If the 32K numbers are confirmed, this is a valuable result; the manuscript should make the evaluation protocol at 32K fully explicit.
minor comments (4)
  1. [Table 4] Table 4 is difficult to read because several entries are run together, e.g., the chunk-256 row shows '36.76 49.2344.83 58.17 49.2544.83', with missing column separators. The table should be reformatted so each column is unambiguous.
  2. [§4.1] The text says 'six single-neddle and multi-needle tasks'; 'single-neddle' should be 'single-needle'.
  3. [Table 4, Fenwick row] The Fenwick tree row is described as reporting performance 'close to [Guo et al., 2026]', but no comparison numbers from that paper are given in the table or the text. Please add the reference numbers so the reader can verify the claim.
  4. [Figure 1] The figure caption contains spacing artifacts such as 'S t a t e S p a c e U p d a t e' and the right panel labels are hard to parse; please redraw the figure and caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MARCH's claims are empirical architectural results, self-contained against external baselines, with no fitted parameter renamed as a prediction and no load-bearing self-citation chain.

full rationale

The paper's central claim is an architectural one: augmenting Gated DeltaNet with periodically cached state anchors and content-routed retrieval improves long-context recall. The derivation chain is the model definition itself (Eqs. 8-14), and the evaluation is a standard pretrain-then-benchmark protocol. No result is derived from a fitted parameter that is then renamed as a prediction; the routing keys, anchor states, and readout are all jointly optimized under the language-modeling objective, and the benchmark scores are measured, not derived. Hyperparameter choices such as chunk size C=512 and routing dimension d_r=64 are selected from ablations (Tables 4 and 5), but this is ordinary model selection on held-out evaluation suites, not a case where a fitted quantity is later reported as an independent prediction. The paper's only notable self-referential element is its use of its own architecture's construction to justify the claim that anchor keys are content-conditioned (Eqs. 10-11); that is a definitional property of the model, not a circular derivation of an empirical result. The skeptic's concern that the compact anchor keys may not encode enough state content is a legitimate empirical risk, and the paper's own Top-4 ablation (Table 5) shows routing precision matters, but this is a correctness or robustness concern, not a circularity: the claim would fail only if the learned keys fail to behave as hoped, not because the conclusion is equivalent to its inputs. No load-bearing citation to the authors' own prior work appears; the primary baseline, Gated DeltaNet, and the Log-Linear Attention comparison are external prior work with no author overlap. Accordingly, the appropriate finding is no significant circularity, and the score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 1 invented entities

The central claim rests on standard machine-learning assumptions (differentiable training, consistent evaluation) and on two hyperparameters selected by ablating the same benchmarks used for the main results. The state anchor is an invented architectural component with no external evidence beyond the reported experiments.

free parameters (2)
  • Anchor chunk size C = 512
    Selected via ablation on the evaluation benchmarks (Section 5, Table 4) as the best balance between retrieval quality and anchor count; this is tuning on the test tasks.
  • Routing dimension d_r = 64
    Selected via ablation (Section 5, Table 5) trading retrieval ability against general performance; 192 yields higher retrieval but lower CS/LongBench.
assumptions (3)
  • domain assumption The learned routing keys are optimized end-to-end and retain sufficient state content information for correct retrieval.
    The method has no guarantee that keys are content-discriminative; the paper relies on the shared anchor token reading its aligned state checkpoint to condition keys (Section 3.1, Eq. 10-11).
  • domain assumption Training and evaluation configurations are identical across all compared models.
    Section 4.1 states all models use the same data, token budget, context length, and optimization, which is necessary for controlled comparison but not independently verified.
  • domain assumption The evaluation benchmarks (RULER, LongBench, etc.) are valid measures of long-context retrieval and reasoning.
    The paper's conclusions depend on the chosen benchmarks capturing the intended capabilities; there is no independent validation of this.
invented entities (1)
  • State anchor bank
    purpose: Stores periodic checkpoints of the recurrent state, each with a learned routing key, to enable content-based retrieval of historical memory.
    This is a new model component. The paper provides no external falsifiable prediction that would validate the anchor bank independently of its own training runs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MARCH: Scaling Recurrent Memory with Content-Routed State Anchors." pith.science (2026). https://pith.science/paper/7TQ2TYPV

@misc{pith2026260812435,
  author       = {Pith},
  title        = {Pith review of: MARCH: Scaling Recurrent Memory with Content-Routed State Anchors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7TQ2TYPV}},
  note         = {Machine review of arXiv:2608.12435}
}
read the original abstract

Transformers owe much of their strong long-context retrieval capability to a token-level memory that grows with context length. This flexibility, however, incurs a quadratic computation complexity during training and a key--value cache that grows linearly during autoregressive inference. Recurrent alternatives offer efficient decoding by compressing the entire history into a fixed-size state, but often underperform on recall-intensive tasks since earlier associations usually get overwritten by subsequent updates, and only the most recent contextual information is retained. In this paper, we introduce Memory-Anchor Routing across Context History (MARCH), a network architecture that effectively scales state-space models beyond a fixed-size dimension, while maintaining computational efficiency over long-sequences. MARCH periodically caches cumulative recurrent-state checkpoints as state anchors and associates each anchor with a compact, content-conditioned anchor key. This lets MARCH maintain a memory bank, which can grow as context length increases, providing a controllable trade-off between historical resolution and memory cost. At each token, MARCH produces an anchor query to attend all causally available state anchors, and the output is calculated as an attention-style aggregation over all historical anchors along the current state. We show that after standard pretraining, MARCH consistently outperforms multiple linear attention variants across commonsense reasoning, LongBench, and in-context retrieval. These results demonstrate that content-routed state caching substantially strengthens recurrent long-range memory while preserving its native computation path.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

77 extracted references · 51 canonical work pages

  1. [1]

    International Conference on Learning Representations , year =

    Log-Linear Attention , author =. International Conference on Learning Representations , year =

    work page

  2. [2]

    , title =

    Fenwick, Peter M. , title =. Software: Practice and Experience , volume =. 1994 , doi =

    work page 1994

  3. [3]

    Gated Delta Networks: Improving

    Yang, Songlin and Kautz, Jan and Hatamizadeh, Ali , booktitle =. Gated Delta Networks: Improving. 2025 , url =

    2025

  4. [4]

    Paperno, Denis and Kruszewski, Germán and Lazaridou, Angeliki and Pham, Ngoc Quan and Bernardi, Raffaella and Pezzelle, Sandro and Baroni, Marco and Boleda, Gemma and Fernández, Raquel , EDITOR =. The. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , YEAR =. doi:10.18653/v1/P16-1144 , PAGES =

    doi:10.18653/v1/p16-1144

  5. [5]

    The Thirty-Fourth

    Bisk, Yonatan and Zellers, Rowan and LeBras, Ronan and Gao, Jianfeng and Choi, Yejin , PUBLISHER =. The Thirty-Fourth. 2020 , DOI =

    work page 2020

  6. [6]

    Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , YEAR =

    Zellers, Rowan and Holtzman, Ari and Bisk, Yonatan and Farhadi, Ali and Choi, Yejin , EDITOR =. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , YEAR =. doi:10.18653/v1/P19-1472 , PAGES =

    doi:10.18653/v1/p19-1472

  7. [7]

    The Thirty-Fourth

    Sakaguchi, Keisuke and Bras, Ronan Le and Bhagavatula, Chandra and Choi, Yejin , PUBLISHER =. The Thirty-Fourth. 2020 , DOI =

    work page 2020

  8. [8]

    Think You Have Solved Question Answering? Try

    Clark, Peter and Cowhey, Isaac and Etzioni, Oren and Khot, Tushar and Sabharwal, Ashish and Schoenick, Carissa and Tafjord, Oyvind , URL =. Think You Have Solved Question Answering? Try. 2018 , JOURNAL =

    work page 2018

Show all 77 references
  1. [9]

    Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages=

    Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering , author=. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages=. 2018 , doi=

    2018

  2. [10]

    2019 , doi=

    Talmor, Alon and Herzig, Jonathan and Lourie, Nicholas and Berant, Jonathan , booktitle=. 2019 , doi=

    2019

  3. [11]

    Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

    Longbench: A bilingual, multitask benchmark for long context understanding , author=. Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=. 2024 , doi=

    2024

  4. [12]

    First Conference on Language Modeling , year=

    RULER: What’s the Real Context Size of Your Long-Context Language Models? , author=. First Conference on Language Modeling , year=

  5. [13]

    Know What You Don

    Rajpurkar, Pranav and Jia, Robin and Liang, Percy , EDITOR =. Know What You Don. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , YEAR =. doi:10.18653/v1/P18-2124 , PAGES =

    doi

  6. [14]

    Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , YEAR =

    Joshi, Mandar and Choi, Eunsol and Weld, Daniel and Zettlemoyer, Luke , EDITOR =. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , YEAR =. doi:10.18653/v1/P17-1147 , PAGES =

    doi

  7. [15]

    Proceedings of the 2019 Conference of the North

    Lockard, Colin and Shiralkar, Prashant and Dong, Xin Luna , EDITOR =. Proceedings of the 2019 Conference of the North. 2019 , DOI =

    2019

  8. [16]

    Language

    Arora, Simran and Yang, Brandon and Eyuboglu, Sabri and Narayan, Avanika and Hojel, Andrew and Trummer, Immanuel and Ré, Christopher , URL =. Language. 2023 , JOURNAL =

    2023

  9. [17]

    and Uszkoreit, Jakob and Le, Quoc and Petrov, Slav

    Kwiatkowski, Tom and Palomaki, Jennimaria and Redfield, Olivia and Collins, Michael and Parikh, Ankur and Alberti, Chris and Epstein, Danielle and Polosukhin, Illia and Devlin, Jacob and Lee, Kenton and Toutanova, Kristina and Jones, Llion and Kelcey, Matthew and Chang, Ming-W...

    2019 doi

  10. [18]

    Proceedings of the 2019 Conference of the North

    Dua, Dheeru and Wang, Yizhong and Dasigi, Pradeep and Stanovsky, Gabriel and Singh, Sameer and Gardner, Matt , EDITOR =. Proceedings of the 2019 Conference of the North. 2019 , DOI =

    2019

  11. [19]

    arXiv preprint arXiv:2606.10650 , year=

    Dynamic Linear Attention , author=. arXiv preprint arXiv:2606.10650 , year=

    arXiv

  12. [20]

    A framework for few-shot language model evaluation , VERSION =

    Gao, Leo and Tow, Jonathan and Biderman, Stella and Black, Sid and DiPofi, Anthony and Foster, Charles and Golding, Laurence and Hsu, Jeffrey and McDonell, Kyle and Muennighoff, Niklas and Phang, Jason and Reynolds, Laria and Tang, Eric and Thite, Anish and Wang, Ben and Wang,...

    2021

  13. [21]

    Simple Linear Attention Language Models Balance the Recall-Throughput Tradeoff , booktitle =

    Arora, Simran and Eyuboglu, Sabri and Zhang, Michael and Timalsina, Aman and Alberti, Silas and Zou, James and Rudra, Atri and R. Simple Linear Attention Language Models Balance the Recall-Throughput Tradeoff , booktitle =. 2024 , url =

    2024

  14. [22]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=. 2025 , doi=

    2025

  15. [23]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Mt-eval: A multi-turn capabilities evaluation benchmark for large language models , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=. 2024 , doi=

    2024

  16. [24]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    On many-shot in-context learning for long-context evaluation , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=. 2025 , doi=

    2025

  17. [25]

    Transactions of the Association for Computational Linguistics , volume=

    Lost in the Middle: How Language Models Use Long Contexts , author=. Transactions of the Association for Computational Linguistics , volume=. 2024 , url=

    2024

  18. [26]

    Advances in Neural Information Processing Systems , volume=

    Augmenting language models with long-term memory , author=. Advances in Neural Information Processing Systems , volume=. 2023 , url=

    2023

  19. [27]

    arXiv preprint arXiv:2501.00663 , year=

    Titans: Learning to memorize at test time , author=. arXiv preprint arXiv:2501.00663 , year=

    arXiv

  20. [28]

    and Kaiser, Lukasz and Polosukhin, Illia , EDITOR =

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N. and Kaiser, Lukasz and Polosukhin, Illia , EDITOR =. Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 20...

    2017

  21. [29]

    Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention , VOLUME =

    Katharopoulos, Angelos and Vyas, Apoorv and Pappas, Nikolaos and Fleuret, François , PUBLISHER =. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention , VOLUME =. Proceedings of the 37th International Conference on Machine Learning,. 2020 , PAGES =

    2020

  22. [30]

    Proceedings of the 41st International Conference on Machine Learning , series =

    Dao, Tri and Gu, Albert , title =. Proceedings of the 41st International Conference on Machine Learning , series =. 2024 , url =

    2024

  23. [31]

    arXiv preprint arXiv:2312.00752 , year =

    Mamba: Linear-Time Sequence Modeling with Selective State Spaces , author =. arXiv preprint arXiv:2312.00752 , year =

    arXiv

  24. [32]

    2021 , eprint=

    Linear Transformers Are Secretly Fast Weight Programmers , author=. 2021 , eprint=

    2021

  25. [33]

    Proceedings of 12th International Conference on Learning Representations (ICLR) , year=

    Zoology: Measuring and improving recall in efficient language models , author=. Proceedings of 12th International Conference on Learning Representations (ICLR) , year=

  26. [34]

    arXiv preprint arXiv:2507.16577 , year=

    Scaling Linear Attention with Sparse State Expansion , author=. arXiv preprint arXiv:2507.16577 , year=

  27. [35]

    arXiv preprint arXiv:2607.07386 , year=

    Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity , author=. arXiv preprint arXiv:2607.07386 , year=

    arXiv

  28. [36]

    2025 , url=

    Du, Jusen and Sun, Weigao and Lan, Disen and Hu, Jiaxi and Cheng, Yu , journal=. 2025 , url=

    2025

  29. [37]

    Forty-third International Conference on Machine Learning , year=

    Memory Caching: RNNs with Growing Memory , author=. Forty-third International Conference on Machine Learning , year=

  30. [38]

    2025 , url=

    Lu, Enzhe and Jiang, Zhejun and Liu, Jingyuan and Du, Yulun and Jiang, Tao and Hong, Chao and Liu, Shaowei and He, Weiran and Yuan, Enming and Wang, Yuzhi and others , journal=. 2025 , url=

    2025

  31. [39]

    arXiv preprint arXiv:2607.02980 , year=

    Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling , author=. arXiv preprint arXiv:2607.02980 , year=

    arXiv

  32. [40]

    and Ermon, Stefano and Rudra, Atri and R

    Dao, Tri and Fu, Daniel Y. and Ermon, Stefano and Rudra, Atri and R. FlashAttention: Fast and Memory-Efficient Exact Attention with. Advances in Neural Information Processing Systems , volume =. 2022 , url =

    2022

  33. [41]

    2025 , eprint=

    RATTENTION: Towards the Minimal Sliding Window Size in Local-Global Attention Models , author=. 2025 , eprint=

    2025

  34. [42]

    2025 , eprint=

    Short window attention enables long-term memorization , author=. 2025 , eprint=

    2025

  35. [43]

    and Weller, Adrian , PUBLISHER =

    Choromanski, Krzysztof Marcin and Likhosherstov, Valerii and Dohan, David and Song, Xingyou and Gane, Andreea and Sarlós, Tamás and Hawkins, Peter and Davis, Jared Quincy and Mohiuddin, Afroz and Kaiser, Lukasz and Belanger, David Benjamin and Colwell, Lucy J. and Weller, Adri...

    2021

  36. [44]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume =

    Xiong, Yunyang and Zeng, Zhanpeng and Chakraborty, Rudrasis and Tan, Mingxing and Fung, Glenn and Li, Yin and Singh, Vikas , title =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =. 2021 , doi =

    2021

  37. [45]

    2024 , BOOKTITLE =

    Dao, Tri , URL =. 2024 , BOOKTITLE =

    2024

  38. [46]

    Linear Attention Sequence Parallelism , VOLUME =

    Sun, Weigao and Qin, Zhen and Li, Dong and Shen, Xuyang and Qiao, Yu and Zhong, Yiran , URL =. Linear Attention Sequence Parallelism , VOLUME =. 2024 , JOURNAL =

    2024

  39. [47]

    arXiv preprint arXiv:1904.10509 , year=

    Generating Long Sequences with Sparse Transformers , author=. arXiv preprint arXiv:1904.10509 , year=

    1904 arXiv

  40. [48]

    arXiv preprint arXiv:2004.05150 , year=

    Longformer: The long-document transformer , author=. arXiv preprint arXiv:2004.05150 , year=

    2004 arXiv

  41. [49]

    Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , YEAR =

    Zaheer, Manzil and Guruganesh, Guru and Dubey, Kumar Avinava and Ainslie, Joshua and Alberti, Chris and Ontañón, Santiago and Pham, Philip and Ravula, Anirudh and Wang, Qifan and Yang, Li and Ahmed, Amr , EDITOR =. Advances in Neural Information Processing Systems 33: Annual C...

    2020

  42. [50]

    Efficient Content-Based Sparse Attention with Routing Transformers , VOLUME =

    Roy, Aurko and Saffar, Mohammad and Vaswani, Ashish and Grangier, David , EDITOR =. Efficient Content-Based Sparse Attention with Routing Transformers , VOLUME =. Transactions of the Association for Computational Linguistics , PAGES =. 2021 , DOI =

    2021

  43. [51]

    Advances in Neural Information Processing Systems , volume=

    Zhang, Zhenyu and Sheng, Ying and Zhou, Tianyi and Chen, Tianlong and Zheng, Lianmin and Cai, Ruisi and Song, Zhao and Tian, Yuandong and R. Advances in Neural Information Processing Systems , volume=. 2023 , url=

    2023

  44. [52]

    2024 , url=

    Tang, Jiaming and Zhao, Yilong and Zhu, Kan and Xiao, Guangxuan and Kasikci, Baris and Han, Song , booktitle=. 2024 , url=

    2024

  45. [53]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Native sparse attention: Hardware-aligned and natively trainable sparse attention , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=. 2025 , doi=

    2025

  46. [54]

    Chou, Yuhong and Yao, Man and Wang, Kexin and Pan, Yuqi and Zhu, Rui-Jie and Wu, Jibin and Zhong, Yiran and Qiao, Yu and XU, Bo and Li, Guoqi , URL =. Meta. The Thirty-eighth Annual Conference on Neural Information Processing Systems , YEAR =

  47. [55]

    The Tenth International Conference on Learning Representations , year=

    Efficiently Modeling Long Sequences with Structured State Spaces , author=. The Tenth International Conference on Learning Representations , year=

  48. [56]

    Retentive network: A successor to transformer for large language models , VOLUME =

    Sun, Yutao and Dong, Li and Huang, Shaohan and Ma, Shuming and Xia, Yuqing and Xue, Jilong and Wang, Jianyong and Wei, Furu , URL =. Retentive network: A successor to transformer for large language models , VOLUME =. 2023 , JOURNAL =

    2023

  49. [57]

    Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    RWKV: Reinventing RNNs for the Transformer Era , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=. 2023 , doi=

    2023

  50. [58]

    2024 , JOURNAL =

    Qin, Zhen and Yang, Songlin and Sun, Weixuan and Shen, Xuyang and Li, Dong and Sun, Weigao and Zhong, Yiran , URL =. 2024 , JOURNAL =

    2024

  51. [59]

    International Conference on Machine Learning , pages=

    Resurrecting recurrent neural networks for long sequences , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  52. [60]

    arXiv preprint arXiv:2407.14207 , year =

    Liu, Bo and Wang, Rui and Wu, Lemeng and Feng, Yihao and Stone, Peter and Liu, Qiang , title =. arXiv preprint arXiv:2407.14207 , year =

    arXiv

  53. [61]

    Advances in Neural Information Processing Systems , TITLE =

    Yang, Songlin and Wang, Bailin and Zhang, Yu and Shen, Yikang and Kim, Yoon , YEAR =. Advances in Neural Information Processing Systems , TITLE =

  54. [62]

    2026 , eprint=

    Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention , author=. 2026 , eprint=

    2026

  55. [63]

    2025 , note =

    Julien Siems and Timur Carstensen and Arber Zela and Frank Hutter and Massimiliano Pontil and Riccardo Grazzi , title =. 2025 , note =

    2025

  56. [64]

    The Thirteenth International Conference on Learning Representations , YEAR =

    Grazzi, Riccardo and Siems, Julien and Zela, Arber and Franke, J. The Thirteenth International Conference on Learning Representations , YEAR =

  57. [65]

    2024 , JOURNAL =

    Wen, Kaiyue and Dang, Xingyu and Lyu, Kaifeng , URL =. 2024 , JOURNAL =

    2024

  58. [66]

    Transformers are Multi-State RNNs , VOLUME =

    Oren, Matanel and Hassid, Michael and Adi, Yossi and Schwartz, Roy , URL =. Transformers are Multi-State RNNs , VOLUME =. 2024 , JOURNAL =

    2024

  59. [67]

    2019 , eprint=

    Large Memory Layers with Product Keys , author=. 2019 , eprint=

    2019

  60. [68]

    2024 , eprint=

    Memory Layers at Scale , author=. 2024 , eprint=

    2024

  61. [69]

    2026 , eprint=

    Fast-weight Product Key Memory , author=. 2026 , eprint=

    2026

  62. [70]

    arXiv preprint arXiv:2607.25357 , year=

    Raven: High-Recall Sequence Modeling with Sparse Memory Routing , author=. arXiv preprint arXiv:2607.25357 , year=

    arXiv

  63. [71]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    Adapting language models to compress contexts , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=. 2023 , doi=

    2023

  64. [72]

    Advances in Neural Information Processing Systems , volume=

    Learning to compress prompts with gist tokens , author=. Advances in Neural Information Processing Systems , volume=. 2023 , url=

    2023

  65. [73]

    arXiv preprint arXiv:2401.03462 , year=

    Long context compression with activation beacon , author=. arXiv preprint arXiv:2401.03462 , year=

    arXiv

  66. [74]

    arXiv preprint arXiv:2509.15763 , year=

    UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compression , author=. arXiv preprint arXiv:2509.15763 , year=

  67. [75]

    arXiv preprint arXiv:2504.08934 , year=

    Long context in-context compression by getting to the gist of gisting , author=. arXiv preprint arXiv:2504.08934 , year=

    arXiv

  68. [76]

    2026 , eprint =

    Simplified Sparse Attention via Gist Tokens , author =. 2026 , eprint =

    2026

  69. [77]

    Unlocking state-tracking in linear

    Riccardo Grazzi and Julien Siems and J. Unlocking state-tracking in linear. 2025 , note =

    2025

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.