Pith. sign in

REVIEW 3 major objections 7 minor 112 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

Sparse reads give RNNs a 1000x bigger memory at no extra compute cost

2026-07-09 12:22 UTC pith:OLDOQKND

load-bearing objection SDM is a genuinely new architecture with strong scaling experiments, but the headline 8B result is confounded by the learned M0 parameter count. the 3 major comments →

arxiv 2607.07386 v1 pith:OLDOQKND submitted 2026-07-08 cs.LG

Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

Lo\"ic Cabannes , Pierre-Emmanuel Mazar\'e , Gergely Szilvasy , Matthijs Douze , Maria Lomeli , Ilze Amanda Auzina , Justin Carpentier , Gabriel Synnaeve
show 1 more author
Herv\'e J\'egou
This is my paper
classification cs.LG
keywords statememorylinearsparseattentionhigherimprovessize
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Linear RNNs compress everything they've seen into a fixed-size hidden state, which keeps compute constant per token but caps how much they can remember. This paper introduces Sparse Delta Memory (SDM), which replaces the dense key-value update rule of Gated DeltaNet with sparse reads and writes to a large explicit memory table. The key insight is that the delta rule — which subtracts old associations before writing new ones — can be applied to just a handful of selected memory slots rather than the entire state. By using a product-key indexing scheme (which finds the top-k most relevant slots among millions without scanning all of them), SDM maintains the same per-token FLOP count as GDN while carrying a memory state roughly a thousand times larger. The paper shows that this enlarged state, matched for parameters and compute, substantially improves long-context recall (RULER scores rise from 20.0 to 31.2 at 1.4B and from 34.2 to 50.2 at 8B), yields lower training loss than GDN at every scale tested, and even edges past full softmax attention at 8B on both loss and short-context accuracy. Additionally, because the memory table is large enough to hold meaningful content, the paper shows that learning its initial values (as model parameters) injects pretraining knowledge that further improves reasoning and common-knowledge benchmarks — something the tiny GDN state cannot do.

Core claim

The central mechanism is the sparsification of the Gated DeltaNet update rule. In standard GDN, each token updates the entire dense matrix state, so doubling the state doubles the FLOPs. SDM instead maintains a large memory table of N slots but applies the gated delta update — decay, subtract old value, write new value — to only W=64 slots selected per token via product-key indexing. Reads similarly touch only R=64 slots. Because the product-key trick finds the top-k slots among N without materializing all N scores, the per-token cost stays at O((W+R) x d) regardless of N, decoupling memory capacity from compute. The paper demonstrates that this sparse update preserves enough information to:

What carries the argument

Sparse Delta Memory (SDM): a recurrent memory layer that replaces Gated DeltaNet's dense state update with sparse top-W writes and top-R reads to a large N-slot memory table, using product-key indexing for sublinear slot selection. The delta rule (subtract-before-write) is applied only to selected slots. The initial memory state M0 is a learnable parameter, turning the memory table into both a working buffer for in-context information and a parametric store for pretraining knowledge.

Load-bearing premise

The architecture rests on the premise that updating only 64 out of roughly 4 million memory slots per token preserves enough information for the recurrent state to function as a viable alternative to dense updates. The paper does not directly compare sparse access to dense access on the same total memory, so it cannot confirm whether the sparse selection pattern itself loses information that a dense update would capture.

What would settle it

If a dense-update model with the same memory size and the same FLOP budget (achieved by shrinking the state per FLOP) matched or beat SDM on long-context recall, the sparse-access mechanism would be shown to lose information that dense access retains.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If sparse delta updates suffice at 1000x state sizes, the primary bottleneck for linear RNN long-context performance shifts from compute to memory bandwidth — kernel engineering becomes the critical path for practical adoption.
  • The learned initial state M0 acting as parametric memory suggests that large recurrent states can partially absorb the role of feed-forward network parameters, potentially changing how parameter budgets are allocated in future architectures.
  • SDM's constant-FLOP, large-state design could enable agents that maintain memory over millions of tokens without the growing KV-cache cost of attention, making sustained reasoning over codebases or long videos tractable.
  • The three-way hybrid (sliding-window attention + full attention + SDM) showing the best scaling coefficient suggests these mechanisms capture complementary information types, pointing toward heterogeneous global-layer designs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 42% write-slot utilization (only 42% of slots ever written during training) raises the question of whether the memory table is over-provisioned; a smaller table with denser access might achieve similar quality with less memory footprint, though the paper notes read utilization reaches 100%.
  • If the sparse access pattern itself (not just total memory) determines information retention, then alternative slot-selection strategies — learned routing, content-based addressing, or multi-hop access — could improve coverage beyond what a single top-W write per token captures.
  • The finding that SDM matches or exceeds full attention on some RULER subtasks but lags substantially on multikey_2 suggests that tasks requiring simultaneous lookup by multiple independent keys may expose a fundamental limit of sparse single-pass access that denser attention mechanisms avoid.
  • The fp8/int4 quantization of memory snapshots during long-context fine-tuning showing no degradation hints that the memory table is robust to precision reduction, which could extend to inference-time quantization of the recurrent state itself.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper introduces Sparse Delta Memory (SDM), which extends Gated DeltaNet (GDN) by replacing the dense key-value outer product update with sparse reads and writes to a large explicit memory table indexed via product-key mechanisms. The core insight is that the GDN delta-rule update can be sparsified: only W out of N memory slots are updated per token, decoupling memory capacity from per-token FLOPs. Under an isoFLOP and iso-parameter constraint, SDM achieves memory states 1000x larger than GDN, improving long-context recall (RULER) and training loss across a scaling ladder from 257M to 8B parameters. The authors train 8B models on 1T+ tokens and report that SDM achieves lower training loss than both GDN and full attention. A learned initial memory state M0 is introduced as a parametric memory mechanism. The experimental design is thorough: scaling law fits (R²=0.999), isoFLOP comparisons, per-task RULER breakdowns, and ablations separating memory size from learned M0.

Significance. The paper makes a significant architectural contribution to linear RNNs by demonstrating that sparse addressing can decouple memory capacity from compute cost, achieving three orders of magnitude larger state sizes at matched FLOPs. The scaling law analysis with predictive validation at 8B is well-executed, and the ablation design (Table 3) separating memory size from learned M0 is methodologically sound. The provision of reproducible code and the training of models up to 8B on 1T+ tokens represents substantial experimental investment. The RULER results showing SDM at 50.2 vs GDN at 34.2 at 8B scale are a meaningful improvement. However, the significance of the headline claim that SDM outperforms full attention at 8B is tempered by the parameter accounting concern around the learned M0 (see major comments).

major comments (3)
  1. The headline claim that SDM achieves lower training loss than Full Attention at 8B scale (Section 5.1, Figure 3, Table 2) is confounded by the learned M0 parameter accounting. Table 1 reports the 8B SDM model has a state size of 7.963B with St:Param = 111%, and the footnote states 'Params include embedding weights but excludes SDM memory state.' Since M0 is a learnable parameter with the same shape as the state, the actual trainable parameter count of the 8B SDM model is approximately 8.14B + 7.96B ≈ 16B, roughly double the 8B FullAttn baseline it outperforms. The null-M0 ablation (Table 3, top) at 1.4B shows SDM-null still beats GDN (code NLL 0.845 vs 0.849), confirming the sparse architecture itself provides value. However, the gap between SDM-null and SDM-learned is substantial (0.845 vs 0.822, a 0.023 NLL difference), and the null-M0 ablation is only run at 1.4B — not at 8B. The Full
  2. The scaling law in Figure 3, which predicts and confirms SDM > FullAttn at 8B, is computed entirely with learned M0. Without an 8B null-M0 run or a scaling law fit to null-M0 results, we cannot determine whether SDM's advantage over FullAttn at scale comes from the sparse architecture or from having approximately 2x the trainable parameters. This directly undermines the paper's most striking claim. The authors should either (a) run an 8B null-M0 ablation, (b) fit a scaling law to null-M0 results and project to 8B, or (c) reframe the headline claim to clearly attribute the FullAttn comparison to the combined effect of sparse architecture plus learned M0, with transparent parameter accounting.
  3. Table 3 (bottom) ablates memory size by varying N, but does not test whether the sparse access pattern itself loses information compared to dense access to the same total memory. The write-key utilization analysis (Figure 7, Appendix B) shows only 42% of slots are ever written to during training at 1.4B, raising the question of whether untrained slots contribute meaningfully or are wasted memory. While the read utilization reaches 100% (Figure 8), the write utilization gap suggests the effective memory capacity may be lower than the nominal N. A discussion of this gap and its implications for the state-size claims would strengthen the paper.
minor comments (7)
  1. Section 5.4: The perplexity increase beyond 256k tokens is attributed to 'local, token-level perplexity being higher at those positions,' but this explanation is not fully convincing without additional analysis. Consider providing per-position loss statistics to support this claim.
  2. Table 2: The BoolQ result at 8B shows SDM at 67.13 vs GDN at 74.98, a -7.85 delta. This is the largest negative delta in the table and warrants discussion, as it may indicate a systematic weakness of the sparse memory approach on certain task types.
  3. Figure 3: The y-axis label 'Training Loss (avg last 10)' could be more precise — clarify whether this is the average of the last 10 log entries, last 10% of training, or last 10k steps.
  4. Section 3.2: The statement that PKM's top-k computation amounts to 'less than 1% of the layer's FLOPs' should specify at which model scale this was measured, as the ratio may change with N.
  5. Appendix A.3: 'this reduces memory consumption for the memory state checkpoints of an SDM layer from 226 GB to 8GB' — clarify whether the 226 GB figure is per-layer or total across all SDM layers.
  6. The paper would benefit from a clearer statement in the abstract and introduction that the learned M0 adds trainable parameters excluded from the parameter count, to set reader expectations early.
  7. Table 4 (Appendix D): The RULER column is averaged across only 6 sub-tasks 'for computational reasons,' which differs from the 13-task average in Table 2. This inconsistency should be noted more prominently to avoid confusion.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee raises three major comments, all centered on the learned initial memory state M0: (1) the parameter accounting concern that M0 roughly doubles the trainable parameter count of the 8B SDM model, confounding the headline comparison against Full Attention; (2) the absence of an 8B null-M0 ablation or null-M0 scaling law projection, leaving open whether SDM's advantage over FullAttn at scale comes from the sparse architecture or from extra parameters; and (3) the write utilization gap (42% of slots written during training) and its implications for effective memory capacity claims. We agree with the substance of all three comments. For (1) and (2), we will reframe the headline claim to clearly attribute the FullAttn comparison to the combined effect of sparse architecture plus learned M0, with transparent parameter accounting in Table 1 and throughout. We will also fit a scaling law to the null-M0 ablation results to project whether the sparse architecture alone would surpass FullAttn at 8B. For (3), we will add a discussion of the write utilization gap and its implications. We cannot commit to running a full 8B null-M0 ablation within the revision timeline due to compute costs (~1T tokens), but the scaling law projection and reframed claims address the core concern.

read point-by-point responses

  1. Referee: The headline claim that SDM achieves lower training loss than Full Attention at 8B scale is confounded by the learned M0 parameter accounting. Table 1 reports the 8B SDM model has a state size of 7.963B with St:Param = 111%, and the footnote states 'Params include embedding weights but excludes SDM memory state.' Since M0 is a learnable parameter with the same shape as the state, the actual trainable parameter count of the 8B SDM model is approximately 8.14B + 7.96B ≈ 16B, roughly double the 8B FullAttn baseline it outperforms.

    Authors: The referee is correct on the factual point. M0 is a learnable parameter with the same shape as the memory state, so the 8B SDM model with learned M0 has approximately 16B trainable parameters — roughly double the 8B FullAttn baseline. Our Table 1 footnote ('Params include embedding weights but excludes SDM memory state') was intended to signal this exclusion, but it does not make the parameter accounting transparent enough, and the headline claim in Section 5.1 ('SDM outperforms Full Attention at 8B') does not adequately acknowledge the parameter difference. We will revise the manuscript to: (a) add a column or explicit note in Table 1 reporting total trainable parameters including M0, (b) reframe the headline claim in Section 5.1 to clearly state that the FullAttn comparison involves SDM with learned M0, which approximately doubles the trainable parameter count, and (c) add a sentence in the abstract and conclusion clarifying that the comparison to FullAttn reflects the combined effect of sparse architecture plus learned M0. We agree this is a necessary correction. revision: yes

  2. Referee: The scaling law in Figure 3, which predicts and confirms SDM > FullAttn at 8B, is computed entirely with learned M0. Without an 8B null-M0 run or a scaling law fit to null-M0 results, we cannot determine whether SDM's advantage over FullAttn at scale comes from the sparse architecture or from having approximately 2x the trainable parameters. The authors should either (a) run an 8B null-M0 ablation, (b) fit a scaling law to null-M0 results and project to 8B, or (c) reframe the headline claim to clearly attribute the FullAttn comparison to the combined effect of sparse architecture plus learned M0, with transparent parameter accounting.

    Authors: We agree that the current evidence does not isolate whether the sparse architecture alone (without learned M0) would surpass FullAttn at 8B. The null-M0 ablation at 1.4B (Table 3, top) shows SDM-null beats GDN (code NLL 0.845 vs 0.849), confirming the sparse architecture provides value independently. However, the gap between SDM-null and SDM-learned (0.845 vs 0.822) is substantial, and we cannot extrapolate from a single 1.4B data point whether the null-M0 variant would surpass FullAttn at 8B. We will pursue option (b): we will train null-M0 SDM at multiple scales on the scaling ladder (at least 3-4 levels) and fit a scaling law to project to 8B. This is feasible within the revision timeline since the smaller ladder levels (1-6) are relatively inexpensive. We will also pursue option (c) regardless: the headline claim will be reframed to clearly attribute the FullAttn comparison to the combined effect of sparse architecture plus learned M0, with transparent parameter accounting. We cannot commit to option (a) — a full 8B null-M0 run on 1T+ tokens — within the revision timeline due to compute costs (~200k GPU-hours total for this project, with the 8B run being a substantial fraction). If the null-M0 scaling law projection is favorable (i.e., projects SDM-null > FullAttn at 8B), we will report this as evidence; if it is unfavorable or inconclusive, we will report that honestly and scope the claim accordingly. revision: partial

  3. Referee: Table 3 (bottom) ablates memory size by varying N, but does not test whether the sparse access pattern itself loses information compared to dense access to the same total memory. The write-key utilization analysis (Figure 7, Appendix B) shows only 42% of slots are ever written to during training at 1.4B, raising the question of whether untrained slots contribute meaningfully or are wasted memory. While the read utilization reaches 100% (Figure 8), the write utilization gap suggests the effective memory capacity may be lower than the nominal N. A discussion of this gap and its implications for the state-size claims would strengthen the paper.

    Authors: This is a fair observation. The 42% write utilization at 1.4B does suggest that the effective writeable memory is lower than the nominal N, and we should discuss this more transparently. However, we note several points that contextualize the gap: (1) Figure 8 shows write utilization growing steadily from 28% to 42% during training and not yet plateauing, suggesting it may increase further with longer training; (2) read utilization reaches 100%, meaning every slot is read from — slots that are never written to still serve as readable memory (initialized via M0 in the learned-M0 setting, or zero-initialized in the null-M0 setting); (3) the memory size ablation (Table 3, bottom) shows monotonic degradation as N shrinks, which would not occur if the unused slots were purely wasted — shrinking N removes both used and unused slots, and the degradation suggests the unused slots may still contribute indirectly (e.g., by providing readable structure or by being written to at lower frequency that our measurement window does not capture). That said, we agree the paper should discuss this gap explicitly. We will add a paragraph in Section 6 discussing the write utilization gap, its implications for effective capacity claims, and the observation that utilization is still growing at the end of training. We will also note that the state-size claims in the paper refer to nominal state size (N × d), and we will clarify that effective writeable capacity is lower. Regarding the broader question of sparse vs. dense access to the same total memory: a dense-access baseline with N=211MB slots would require O(N × d) FLOPs per token, which is exactly the cost SDM is designed to avoid — so this comparison is not feasible at the state sizes we use. We will note this as a conceptual rather than实验性 revision: partial

Circularity Check

0 steps flagged

No circularity found; derivation is empirically validated against external baselines

full rationale

The paper's central claims are empirical: SDM outperforms GDN and FullAttn on external benchmarks (RULER, DCLM NLL, reasoning tasks) under an isoFLOP/iso-parameter design (Section 3.2). The isoFLOP matching (W=R=d_qk^GDN) is a design constraint, not a definition that forces the outcome. The scaling law (Section 5.1) is fitted to levels 1-8 and then independently validated by training the 8B model — the prediction is an extrapolation, not a fitted quantity renamed as a result. The SDM update rule (Eqs. 3-5) is a genuine sparsification of GDN's delta rule using PKM indexing from Lample et al. (2019, external), and the paper shows it recovers GDN as a special case. The one self-citation (Cabannes et al., 2025) is for a minor SWA window-size heuristic and is not load-bearing for the central architectural claims. The skeptic's concern about learned M0 adding ~8B parameters excluded from the parameter count is a legitimate fairness/correctness concern, but it is not circularity: the paper does not define M0 in terms of the target result, nor does it rename a fit as a prediction. The null-M0 ablation (Table 3) shows SDM-null still beats GDN at 1.4B, providing independent evidence for the sparse architecture's value. Score 1 reflects the minor non-load-bearing self-citation.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 2 invented entities

SDM introduces one main architectural entity (the SDM layer) and one secondary technique (learned M0). Both are empirically validated with ablations and external benchmarks. The free parameters are mostly justified by the isoFLOP design constraint or standard GDN conventions, except H which uses a heuristic (state-to-param ratio ~1:1). The learned M0 adds substantial parameters not counted in the parameter-matched comparison — this is the most significant concern, partially mitigated by the null-M0 ablation showing improvement even without it.

free parameters (6)
  • W (write slots) = 64
    Set to match GDN's d_qk=64 for isoFLOP comparison (Section 3.2). Not fitted to data but chosen to match baseline FLOPs.
  • R (read slots) = 64
    Set equal to W by default (Section 4). Ablated in Appendix D.1 but default choice is not justified beyond symmetry with W.
  • H (SDM heads per layer) = 1 or 2 (scale-dependent)
    Chosen per scale to keep state-to-parameter ratio near 1:1 (Table 1). This is a design heuristic, not derived from theory.
  • A (forget gate decay parameter) = initialized U[0,16]
    Learnable per-head parameter, initialized to match GDN/FLA conventions (Section 4).
  • b_dt (time-step bias) = inv_softplus(U(0.001,0.1))
    Learnable bias, initialized from a uniform distribution matching GDN conventions (Section 4).
  • M0 (learned initial memory state) = learned during pretraining
    The initial state of the N×d memory table is a learnable parameter. At 8B scale this is ~8B additional parameters (Table 1: state 7.963B vs 8.14B params). This is a substantial number of additional learned parameters not counted in the 'parameter-matched' comparison.
axioms (4)
  • domain assumption The GDN delta-rule update can be meaningfully sparsified by applying it only to top-W selected slots without catastrophic information loss.
    This is the core architectural assumption (Section 3.1). It is empirically validated by the main results but not theoretically derived.
  • domain assumption Product-key memory indexing (PKM) provides sufficient slot selection quality for recurrent state updates, not just static lookups.
    PKM was designed for static FFN-like memory (Lample et al. 2019). Using it for dynamic recurrent updates is a new application assumption (Section 3.1, step 1).
  • ad hoc to paper The isoFLOP comparison is the right evaluation frame, even when wall-clock training time differs by 1.49x.
    Section 3.2 frames the comparison as isoFLOP, but the 10x kernel MFU gap means SDM is slower in practice. The paper acknowledges this (Section 6) but the headline results use isoFLOP framing.
  • domain assumption Long-context fine-tuning on 128k sequences (4B-16B tokens) is sufficient to evaluate long-context capabilities.
    Section 4 describes the fine-tuning stage. The RULER evaluation goes to 128k, and perplexity to 1M, but the fine-tuning data budget is small relative to pretraining.
invented entities (2)
  • Sparse Delta Memory (SDM) layer independent evidence
    purpose: A recurrent memory layer combining PKM sparse addressing with the GDN delta-rule update.
    The SDM layer is the paper's main contribution. It is evaluated against external baselines (GDN, Mamba2, FullAttn) on standard benchmarks (RULER, DCLM NLL, reasoning tasks) at multiple scales. Code is released. Falsifiable: if sparse access loses critical information, RULER multikey tasks would fail — and they partially do (multikey_2/3 near zero).
  • Learned initial state M0 as parametric memory independent evidence
    purpose: Using the large SDM memory table as a learnable knowledge store initialized at test time.
    Ablated in Table 3 (top) showing it helps SDM but not GDN. The improvement on reasoning tasks (Table 2) provides evidence it stores useful knowledge. However, the ~8B additional parameters at the largest scale are not counted in the 'parameter-matched' comparison, which inflates the apparent efficiency.

pith-pipeline@v1.1.0-glm · 23948 in / 3782 out tokens · 574462 ms · 2026-07-09T12:22:11.181335+00:00 · methodology

0 comments
read the original abstract

Linear attention models allow a fixed state size and a fixed amount of compute per token. However, due to their limited state size, linear attention models fall behind in long-context recall compared to softmax-attention-based transformer architectures. Increasing the state size of linear attention improves recall performance but at the cost of higher FLOPs. In this work, we introduce Sparse Delta Memory (SDM), an architecture that scales the hidden state of gated linear RNNs to orders of magnitude higher capacity using a sparse addressing scheme. SDM extends the Gated DeltaNet architecture by replacing the dense key-value outer product with sparse reads and writes to a large explicit memory. We show that, under an isoFLOP constraint and with an identical number of parameters, a higher state memory capacity significantly improves performance on in-context learning and long-context retrieval tasks. Moreover, by learning the initial state of the SDM memory and therefore using it as a parametric memory, we show that the model further improves on a wide range of common-knowledge and reasoning tasks.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

112 extracted references · 112 canonical work pages · 32 internal anchors

  1. [1]

    Scaling Learning Algorithms Towards

    Bengio, Yoshua and LeCun, Yann , booktitle =. Scaling Learning Algorithms Towards

    work page

  2. [2]

    and Osindero, Simon and Teh, Yee Whye , journal =

    Hinton, Geoffrey E. and Osindero, Simon and Teh, Yee Whye , journal =. A Fast Learning Algorithm for Deep Belief Nets , volume =

    work page

  3. [3]

    2016 , publisher=

    Deep learning , author=. 2016 , publisher=

    work page 2016

  4. [4]

    2019 , eprint=

    Large Memory Layers with Product Keys , author=. 2019 , eprint=

    work page 2019

  5. [5]

    2023 , eprint=

    Attention Is All You Need , author=. 2023 , eprint=

    work page 2023

  6. [6]

    2024 , eprint=

    Gated Linear Attention Transformers with Hardware-Efficient Training , author=. 2024 , eprint=

    work page 2024

  7. [8]

    2025 , eprint=

    Simple linear attention language models balance the recall-throughput tradeoff , author=. 2025 , eprint=

    work page 2025

  8. [9]

    2020 , eprint=

    Longformer: The Long-Document Transformer , author=. 2020 , eprint=

    work page 2020

  9. [10]

    2018 , month = nov, day =

    luckyq , title =. 2018 , month = nov, day =

    work page 2018

  10. [11]

    2017 , eprint=

    Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer , author=. 2017 , eprint=

    work page 2017

  11. [12]

    Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

    Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism , author=. arXiv preprint arXiv:1909.08053 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 1909

  12. [13]

    2024 , eprint=

    Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models , author=. 2024 , eprint=

    work page 2024

  13. [14]

    2025 , eprint=

    gpt-oss-120b & gpt-oss-20b Model Card , author=. 2025 , eprint=

    work page 2025

  14. [15]

    2024 , eprint=

    Hymba: A Hybrid-head Architecture for Small Language Models , author=. 2024 , eprint=

    work page 2024

  15. [16]

    2025 , eprint=

    RATTENTION: Towards the Minimal Sliding Window Size in Local-Global Attention Models , author=. 2025 , eprint=

    work page 2025

  16. [17]

    2025 , eprint=

    Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling , author=. 2025 , eprint=

    work page 2025

  17. [19]

    2017 , eprint=

    Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning , author=. 2017 , eprint=

    work page 2017

  18. [20]

    arXiv , primaryclass =

    Maximilian Beck and Korbinian Pöppel and Phillip Lippe and Sepp Hochreiter , year =. arXiv , primaryclass =

    work page

  19. [21]

    2014 , eprint=

    Neural Turing Machines , author=. 2014 , eprint=

    work page 2014

  20. [22]

    2024 , eprint=

    Mamba: Linear-Time Sequence Modeling with Selective State Spaces , author=. 2024 , eprint=

    work page 2024

  21. [23]

    2019 , eprint=

    Generating Long Sequences with Sparse Transformers , author=. 2019 , eprint=

    work page 2019

  22. [24]

    2024 , eprint=

    xLSTM: Extended Long Short-Term Memory , author=. 2024 , eprint=

    work page 2024

  23. [25]

    2024 , eprint=

    BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack , author=. 2024 , eprint=

    work page 2024

  24. [26]

    2023 , eprint=

    Hungry Hungry Hippos: Towards Language Modeling with State Space Models , author=. 2023 , eprint=

    work page 2023

  25. [27]

    2024 , journal =

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, Ion Stoica , title =. 2024 , journal =

    work page 2024

  26. [28]

    2024 , eprint=

    RULER: What's the Real Context Size of Your Long-Context Language Models? , author=. 2024 , eprint=

    work page 2024

  27. [29]

    2021 , eprint=

    Evaluating Large Language Models Trained on Code , author=. 2021 , eprint=

    work page 2021

  28. [30]

    2024 , eprint=

    SWE-bench: Can Language Models Resolve Real-World GitHub Issues? , author=. 2024 , eprint=

    work page 2024

  29. [31]

    2024 , eprint=

    Memory Layers at Scale , author=. 2024 , eprint=

    work page 2024

  30. [32]

    2016 , eprint=

    Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes , author=. 2016 , eprint=

    work page 2016

  31. [33]

    Nature , volume =

    Sophia Chen , title =. Nature , volume =. 2025 , month =. doi:10.1038/d41586-025-00616-z , url =

    work page doi:10.1038/d41586-025-00616-z 2025

  32. [34]

    2025 , month =

    Sam Altman , title =. 2025 , month =

    work page 2025

  33. [35]

    Psychological Review , volume =

    Rosenblatt, Frank , title =. Psychological Review , volume =. 1958 , doi =

    work page 1958

  34. [36]

    and Hinton, Geoffrey E

    Rumelhart, David E. and Hinton, Geoffrey E. and Williams, Ronald J. , title =. Nature , volume =. 1986 , doi =

    work page 1986

  35. [37]

    2024 , eprint=

    DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models , author=. 2024 , eprint=

    work page 2024

  36. [38]

    2025 , howpublished=

    Why Stacking Sliding Windows Can't See Very Far , author=. 2025 , howpublished=

    work page 2025

  37. [39]

    FLA: A Triton-Based Library for Hardware-Efficient Implementations of Linear Attention Mechanism , author =

    work page

  38. [40]

    2022 , eprint=

    FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness , author=. 2022 , eprint=

    work page 2022

  39. [41]

    2025 , eprint=

    A Systematic Analysis of Hybrid Linear Attention , author=. 2025 , eprint=

    work page 2025

  40. [42]

    2025 , eprint=

    Understanding Transformer from the Perspective of Associative Memory , author=. 2025 , eprint=

    work page 2025

  41. [43]

    2016 , eprint=

    Layer Normalization , author=. 2016 , eprint=

    work page 2016

  42. [44]

    2019 , eprint=

    Root Mean Square Layer Normalization , author=. 2019 , eprint=

    work page 2019

  43. [45]

    2025 , eprint=

    xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference , author=. 2025 , eprint=

    work page 2025

  44. [46]

    2021 , eprint=

    Program Synthesis with Large Language Models , author=. 2021 , eprint=

    work page 2021

  45. [47]

    2017 , eprint=

    RACE: Large-scale ReAding Comprehension Dataset From Examinations , author=. 2017 , eprint=

    work page 2017

  46. [50]

    2021 , eprint=

    Training Verifiers to Solve Math Word Problems , author=. 2021 , eprint=

    work page 2021

  47. [51]

    2019 , eprint=

    HellaSwag: Can a Machine Really Finish Your Sentence? , author=. 2019 , eprint=

    work page 2019

  48. [52]

    2019 , eprint=

    BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions , author=. 2019 , eprint=

    work page 2019

  49. [53]

    2019 , eprint=

    PIQA: Reasoning about Physical Commonsense in Natural Language , author=. 2019 , eprint=

    work page 2019

  50. [54]

    2019 , eprint=

    SocialIQA: Commonsense Reasoning about Social Interactions , author=. 2019 , eprint=

    work page 2019

  51. [55]

    2019 , eprint=

    WinoGrande: An Adversarial Winograd Schema Challenge at Scale , author=. 2019 , eprint=

    work page 2019

  52. [56]

    EMNLP , year=

    Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering , author=. EMNLP , year=

    work page

  53. [57]

    2018 , eprint=

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge , author=. 2018 , eprint=

    work page 2018

  54. [58]

    2025 , eprint=

    DataComp-LM: In search of the next generation of training sets for language models , author=. 2025 , eprint=

    work page 2025

  55. [59]

    2025 , eprint=

    Understanding and Improving Length Generalization in Recurrent Models , author=. 2025 , eprint=

    work page 2025

  56. [60]

    2023 , eprint=

    RoFormer: Enhanced Transformer with Rotary Position Embedding , author=. 2023 , eprint=

    work page 2023

  57. [62]

    Journal of Machine Learning Research , year =

    Nitish Srivastava and Geoffrey Hinton and Alex Krizhevsky and Ilya Sutskever and Ruslan Salakhutdinov , title =. Journal of Machine Learning Research , year =

    work page

  58. [63]

    2024 , eprint=

    Parallelizing Linear Transformers with the Delta Rule over Sequence Length , author=. 2024 , eprint=

    work page 2024

  59. [64]

    2021 , eprint=

    Linear Transformers Are Secretly Fast Weight Programmers , author=. 2021 , eprint=

    work page 2021

  60. [65]

    2025 , eprint=

    What is Wrong with Perplexity for Long-context Language Modeling? , author=. 2025 , eprint=

    work page 2025

  61. [66]

    2025 , eprint=

    Short window attention enables long-term memorization , author=. 2025 , eprint=

    work page 2025

  62. [67]

    2026 , eprint=

    MiMo-V2-Flash Technical Report , author=. 2026 , eprint=

    work page 2026

  63. [68]

    2025 , eprint=

    Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free , author=. 2025 , eprint=

    work page 2025

  64. [69]

    doi:10.5281/zenodo.5196577 , url =

    PENG Bo , title =. doi:10.5281/zenodo.5196577 , url =

    work page doi:10.5281/zenodo.5196577

  65. [70]

    2026 , eprint=

    Fast-weight Product Key Memory , author=. 2026 , eprint=

    work page 2026

  66. [71]

    2025 , eprint=

    Test-Time Training Done Right , author=. 2025 , eprint=

    work page 2025

  67. [72]

    2019 , eprint=

    Decoupled Weight Decay Regularization , author=. 2019 , eprint=

    work page 2019

  68. [73]

    2022 , eprint=

    Training Compute-Optimal Large Language Models , author=. 2022 , eprint=

    work page 2022

  69. [74]

    Transformers are

    Dao, Tri and Gu, Albert , booktitle=. Transformers are

    work page

  70. [75]

    2019 , eprint=

    CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge , author=. 2019 , eprint=

    work page 2019

  71. [76]

    Proceedings of the International Conference on Learning Representations (ICLR) , year=

    Measuring Massive Multitask Language Understanding , author=. Proceedings of the International Conference on Learning Representations (ICLR) , year=

    work page

  72. [78]

    Is Your Code Generated by Chat

    Liu, Jiawei and Xia, Chunqiu Steven and Wang, Yuyao and Zhang, Lingming , booktitle =. Is Your Code Generated by Chat. 2023 , url =

    work page 2023

  73. [79]

    2025 , eprint=

    Gated Delta Networks: Improving Mamba2 with Delta Rule , author=. 2025 , eprint=

    work page 2025

  74. [80]

    Simple linear attention language models balance the recall-throughput tradeoff

    Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the recall-throughput tradeoff, 2025. URL https://arxiv.org/abs/2402.18668

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  75. [81]

    xLSTM: Extended Long Short-Term Memory

    Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory, 2024. URL https://arxiv.org/abs/2405.04517

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  76. [82]

    Memory Layers at Scale

    Vincent-Pierre Berges, Barlas Oğuz, Daniel Haziza, Wen tau Yih, Luke Zettlemoyer, and Gargi Ghosh. Memory layers at scale, 2024. URL https://arxiv.org/abs/2412.09764

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  77. [83]

    PIQA: Reasoning about Physical Commonsense in Natural Language

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019. URL https://arxiv.org/abs/1911.11641

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  78. [84]

    Short window attention enables long-term memorization

    Loïc Cabannes, Maximilian Beck, Gergely Szilvasy, Matthijs Douze, Maria Lomeli, Jade Copet, Pierre-Emmanuel Mazaré, Gabriel Synnaeve, and Hervé Jégou. Short window attention enables long-term memorization, 2025. URL https://arxiv.org/abs/2509.24552

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  79. [85]

    BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019. URL https://arxiv.org/abs/1905.10044

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  80. [86]

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https://arxiv.org/abs/1803.05457

    work page internal anchor Pith review Pith/arXiv arXiv 2018

Showing first 80 references.