Pith. sign in

REVIEW 2 major objections 7 minor 27 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

Delta-rule attention matches softmax by key-dependent projection

2026-07-09 01:28 UTC pith:7XNBJK4P

load-bearing objection Clean experimental design and a useful theoretical framing of why delta rules approximate softmax; the Taylor error bound is loose but the paper's empirical work carries the argument anyway. the 2 major comments →

arxiv 2607.07706 v1 pith:7XNBJK4P submitted 2026-07-08 cs.LG

The Key to Going Linear: Analysis-Driven Transformer Linearization

Anna Kuzina , Paul N. Whatmough , Babak Ehteshami Bejnordi This is my paper
classification cs.LG
keywords transformer linearizationlinear attentiondelta rulerank-1 projectionsoftmax approximationgated delta networkspost-hoc linearizationattention replacement
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper explains why certain linear-time attention replacements preserve model quality better than others when swapped into a frozen pretrained transformer. The central claim is that softmax attention, when approximated to first order, naturally applies rank-1 orthogonal projections that depend on the key vectors at each position. Delta-rule updates (as in Gated Delta Networks) natively implement these key-dependent projections, while purely gated accumulation (as in Gated Linear Attention) uses key-independent diagonal decay and misses this geometric correction. The authors derive this connection through a chain of approximations: a first-order Taylor expansion of softmax, a projection that replaces mean-subtraction with orthogonal projection onto the complement of the mean key direction, and a concentration argument showing that a product of per-key rank-1 operators collapses to a single projection when keys are directionally aligned. They then identify where the approximation breaks down (early sequence positions, where keys are least concentrated and a uniform bias term is largest), and introduce structural compensations: sink tokens, disjoint sliding-window attention, and short convolutions. The final recipe, applied to LLaMA and Qwen models up to 32B parameters, matches or outperforms prior post-hoc linearization methods on MMLU and common reasoning benchmarks while using fewer training tokens.

Core claim

The paper's central object is the rank-1 orthogonal projection operator (I - beta_j k_j k_j^T), which appears both in the first-order expansion of softmax attention weights and in the delta-rule state update of Gated Delta Networks. The authors show that softmax attention, to first order, applies a sequence of these key-dependent projections that suppress components aligned with previous keys. Delta-rule mechanisms replicate this structure exactly; purely gated mechanisms do not. This structural alignment, rather than training tricks or added capacity, is what makes delta-based updates the most faithful linear replacement for softmax attention in a frozen-backbone setting. The authors empiri

What carries the argument

Rank-1 orthogonal projection operators of the form (I - beta_j k_j k_j^T), where k_j is the key vector at position j and beta_j is a learned scalar gate. These operators project onto the subspace orthogonal to each key, and their product over a sequence approximates a single projection onto the complement of the mean key direction when keys are concentrated. The delta-rule state update composes these operators natively; the softmax attention weight vector, expanded to first order, has the same structure.

Load-bearing premise

The derivation requires that normalized key vectors are concentrated around their mean direction, so that a product of per-key rank-1 projection operators collapses to a single projection. The paper's own diagnostics show this concentration is lowest at early sequence positions, which is exactly where approximation error is highest, suggesting the assumption fails in the regime that matters most for the sink-token motivation.

What would settle it

If a purely gated mechanism (GLA) with short convolutions but no delta-rule structure were shown to match GDN on all benchmarks including early-sequence-heavy tasks, the claim that key-dependent rank-1 projections are the load-bearing mechanism would be weakened. Alternatively, if key concentration were shown to be uniformly high across all positions in well-trained models, the predicted early-sequence approximation gap would not appear, undermining the diagnostic link between the theory and the practical interventions.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If delta-rule updates are structurally the closest linear approximation to softmax, then future linearization efforts should default to delta-based mechanisms rather than gated or kernelized alternatives, reducing the design space for post-hoc conversion.
  • The identification of early-sequence positions as the hardest to approximate suggests that any linear attention mechanism will systematically struggle with sink-token attention patterns, making sink-token retention a structural necessity rather than an engineering convenience.
  • The key-concentration diagnostic (cosine similarity between each key and the running mean direction) provides a layer-level hardness signal that can guide hybrid architectures, selectively retaining full attention only where the linear approximation is worst.
  • The finding that short convolutions erase the gap between GLA and GDN suggests that local feature mixing can substitute for the rank-1 geometric correction, implying the correction matters most when input projections are not already smoothed.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the concentration assumption fails precisely at early positions (as the paper's own Figure 3 shows), then the theoretical justification for delta-rule superiority is strongest in the regime where it is least needed (long sequences with aligned keys) and weakest where structural interventions (sink tokens) are doing the heavy lifting. The practical recipe may be succeeding for reasons partially
  • The uniform bias term 1/t from the Taylor expansion, which practical GDN implementations omit, predicts a systematic over-estimation of attention to early tokens by the linear mechanism. This is related to but distinct from the attention-sink phenomenon, and suggests that an explicit bias correction in the delta-rule update could reduce reliance on sink-token caching.
  • The layer-level key-concentration metric could serve as a pre-computable predictor of linearization difficulty for any pretrained model, before any training occurs, enabling model-specific hybridization strategies without trial-and-error.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. This paper studies post-hoc linearization of pretrained transformers, focusing on which linear state update best approximates softmax attention in a strict frozen-backbone regime. The authors derive a first-order approximation showing that softmax attention induces key-dependent rank-1 orthogonal projections, structurally aligning with delta-based updates (GDN) rather than purely gated accumulation (GLA). They introduce practical interventions (sink tokens, short convolutions, disjoint SWA) and a kGLA variant, then scale GDN-based linearization across LLaMA and Qwen models up to 32B, reporting competitive or superior MMLU and long-context retrieval versus prior baselines (LoLCATs, Liger-GLA, Lizard, LoLA). The three lemmas in Appendix A are self-contained, and the empirical ablations are systematic with 3-seed averaging and controlled component additions.

Significance. The paper's main contribution is a structural explanation for why delta-style updates outperform gated accumulation in post-hoc linearization, grounded in a first-order Taylor analysis of softmax. The controlled frozen-backbone experimental design is a genuine strength: it isolates the approximation capacity of each mechanism before reintroducing compensating components. The scaling experiments up to 32B parameters, the hybrid-layer analysis using theoretically motivated hardness signals, and the introduction of kGLA as a bridge between GLA and GDN all add practical value. The assumption checks (Fig. 3) for Lemmas 2 and 3 are a commendable practice. However, the central 'faithful approximation' claim rests on Lemma 1, whose error bound is never empirically validated, and the key concentration assumption (Lemma 3) appears to fail precisely in the regime the paper identifies as most important (early sequence positions). These issues affect the load-bearing connection between the theory and the empirical superiority of GDN.

major comments (2)
  1. §3.2.1, Lemma 1 (Appendix A.1): The first-order Taylor expansion of softmax around x=0 yields an error bound ||ε^(1)||_∞ ≤ (1/2)||x||²₂, where x_i = q_t^T k_i / √d. The paper empirically validates the assumptions of Lemmas 2 and 3 (Fig. 3) but never checks whether the Lemma 1 Taylor error is small. For realistic attention logits (routinely O(1)–O(10) for strongly attending tokens), ||x||₂ can be O(√t) or larger, making the error bound O(t) and potentially dwarfing the O(1/t) leading terms. Since the paper's central claim is that GDN 'faithfully captures the first-order dynamics of softmax attention' and this faithfulness derives from Lemma 1, the absence of any empirical check of the Taylor error is a significant gap. The authors should either (a) empirically measure ||x||₂ and the actual Taylor residual in the frozen backbones and show it is small in practice, or (b) reframe the claim:G
  2. §3.2.1, Lemma 3 and Fig. 3: The rank-1 product approximation requires keys to be concentrated around their mean direction so that the product of (I − β_j k_j k_j^T) approximates (I − c Π_kbar). The paper's own Figure 3(b) shows key concentration is lowest at early sequence positions, exactly where approximation error is highest (Fig. 2c). The paper acknowledges this qualitatively ('the first tokens exhibit the lowest concentration') but does not address the tension: the assumption underlying the derivation fails precisely in the regime that motivates the sink-token intervention. This weakens the claim that the theory 'derives' GDN from softmax and justifies its superiority. The authors should discuss whether the structural correspondence holds in the regime that matters most, or whether the sink-token and SWA interventions are compensating for a breakdown of the theoretical assumptions (
minor comments (7)
  1. §3.2.1, Eq. (11): The constant bias 1/t from the Taylor expansion is noted as absent in practical GDN implementations. The paper states it 'becomes negligible' for large t but does not quantify at what t this holds. A brief empirical check or a threshold value would be helpful.
  2. Table 3: The LoRA learning rate is reduced to 5e-4 'due to training instability,' and the authors hypothesize LoRA could benefit from longer training. Since LoRA underperforms short convolutions on most benchmarks, this caveat should be stated more prominently in the main text rather than only in §4, as it affects the comparison.
  3. §3.2.2, Eq. (12): The kGLA variant uses Diag(k_j ⊙ k_j) as a diagonal approximation to the rank-1 update k_j k_j^T. The relationship between this diagonal approximation and the off-diagonal structure of k_j k_j^T is not discussed. A brief note on what information is lost would clarify the motivation for kGLA as a bridge between GLA and GDN.
  4. Table 4: The comparison with prior works uses different training token budgets (0.01B for ours vs. 0.02–0.04B for baselines). While this favors the proposed method, the table caption should note that fewer training tokens is better, as this is not immediately obvious from the table itself.
  5. Appendix A.1, proof of Lemma 4: The bound ||R(x)||_∞ ≤ (1/2)||x||²₂ is stated without derivation. A one-line justification (e.g., citing the Lipschitz constant of the softmax Jacobian) would make the proof self-contained.
  6. Figure 2(b): The y-axis label 'Normalised MSE' uses a log scale but this is not indicated. Clarifying the scale would improve readability.
  7. §5, Table 9: The RULER results show GDN at 896 cache tokens is competitive on aggregation/QA tasks but weak on multi-query/multi-value retrieval. The abstract claims 'matching the long-context retrieval of complex adaptive-caching frameworks,' which is only partially supported. The abstract should be qualified (e.g., 'matching on aggregation tasks while remaining competitive overall').

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for a careful and constructive report. Both major comments identify genuine gaps between the theoretical claims and their empirical validation. We agree with both points and will revise the manuscript accordingly. Below we address each in detail.

read point-by-point responses

  1. Referee: §3.2.1, Lemma 1 (Appendix A.1): The first-order Taylor expansion of softmax around x=0 yields an error bound ||ε^(1)||_∞ ≤ (1/2)||x||²₂, where x_i = q_t^T k_i / √d. The paper empirically validates the assumptions of Lemmas 2 and 3 (Fig. 3) but never checks whether the Lemma 1 Taylor error is small. For realistic attention logits (routinely O(1)–O(10) for strongly attending tokens), ||x||₂ can be O(√t) or larger, making the error bound O(t) and potentially dwarfing the O(1/t) leading terms. Since the paper's central claim is that GDN 'faithfully captures the first-order dynamics of softmax attention' and this faithfulness derives from Lemma 1, the absence of any empirical check of the Taylor error is a significant gap. The authors should either (a) empirically measure ||x||₂ and the actual Taylor residual in the frozen backbones and show it is small in practice, or (b) reframe the claim.

    Authors: The referee is correct that we validated the assumptions of Lemmas 2 and 3 but not Lemma 1, and this is an oversight given that Lemma 1 is the foundation of the entire derivation. We will address this in two ways. First, we will add an empirical measurement of ||x||₂ and the actual Taylor residual ||ε^(1)||_∞ in the frozen LLaMA-3.1-8B and Qwen3-8B backbones, following the same methodology used for the assumption checks in Fig. 3 (i.e., computed on the same 8,000-sequence evaluation set, reported per token position and per layer). We expect this to reveal, consistent with the referee's concern, that the worst-case bound ||ε^(1)||_∞ ≤ (1/2)||x||²₂ is loose and that the actual residual is non-trivial at positions where attention logits are large (e.g., sink positions, strongly attending tokens). Second, and more importantly, we will reframe the central claim. The word 'faithfully' overstates what the first-order analysis establishes. The Taylor expansion shows that the *structural form* of softmax's first-order term—specifically, the emergence of a key-dependent projection (I − Π_k̄)—aligns with the delta rule's rank-1 operators and not with GLA's key-independent diagonal decay. This is a statement about structural correspondence of the leading-order term, not a guarantee that the Taylor residual is negligible in practice. We will revise the language throughout the paper (abstract, §3.2, §3.2.1, and the contributions list) to say that GDN 'captures the first-order structural dynamics' or 'aligns with the first-order geometry' of softmax, rather than 'faithfully captures.' We believe this reframing is both more accurate and still conveys the core insight: the delta rule's rank-1 key-dependent update is the natural linear-attention analogue of what softmax does at first ord revision: yes

  2. Referee: §3.2.1, Lemma 3 and Fig. 3: The rank-1 product approximation requires keys to be concentrated around their mean direction so that the product of (I − β_j k_j k_j^T) approximates (I − c Π_kbar). The paper's own Figure 3(b) shows key concentration is lowest at early sequence positions, exactly where approximation error is highest (Fig. 2c). The paper acknowledges this qualitatively ('the first tokens exhibit the lowest concentration') but does not address the tension: the assumption underlying the derivation fails precisely in the regime that motivates the sink-token intervention. This weakens the claim that the theory 'derives' GDN from softmax and justifies its superiority. The authors should discuss whether the structural correspondence holds in the regime that matters most, or whether the sink-token and SWA interventions are compensating for a breakdown of the theoretical assumptions.

    Authors: This is a fair and incisive observation. We agree that there is a genuine tension: the key-concentration assumption underlying Lemma 3 is most violated at early sequence positions, which are simultaneously the positions where approximation error is highest and where our sink-token intervention is most impactful. We do not think this invalidates the theory, but we agree it requires explicit discussion, which we will add. Our view is as follows. The first-order analysis explains why GDN's rank-1 key-dependent structure is the right inductive bias for approximating softmax in the regime where the assumptions hold—namely, the bulk of the sequence where keys are reasonably concentrated. The theory does not claim uniform validity across all positions, and the paper already notes in §6 (Limitations) that the assumptions 'may not hold uniformly across layers or very short prefixes.' What the referee correctly identifies is that the regime of assumption failure is not incidental: it is exactly the regime that matters most for downstream quality (early tokens carry disproportionate attention mass). The sink-token and SWA interventions are indeed compensating for the breakdown of the theoretical assumptions in this regime. This is not a contradiction of the theory but a consequence of it: the analysis identifies *where* the linear mechanism fails (early positions, low concentration) and *why* (the rank-1 product no longer approximates a structured projection), and the interventions target exactly this failure mode. We will make this argument explicit in a revised §3.3 or a new discussion paragraph, acknowledging that (i) the structural correspondence between softmax and GDN holds in the concentration regime, (ii) it breaks at early positions, and (iii) the practical interventions（ revision: yes

Circularity Check

0 steps flagged

No circularity: self-contained derivation with no self-citations or fitted-input-as-prediction

full rationale

The paper's derivation chain (Lemmas 1-3, Section 3.2, proofs in Appendix A) is entirely self-contained. Each lemma uses standard mathematical tools (Taylor expansion, orthogonal projections, matrix norm bounds) with full proofs provided. The authors do not appear in the reference list as authors of any cited work — there are zero self-citations. The central claim (GDN's rank-1 updates structurally correspond to softmax's first-order dynamics) is a qualitative structural correspondence derived from first principles, not a parameter fitted to data and then presented as a prediction. The kGLA variant (Section 3.2.2) is defined constructively from the analysis and then independently evaluated. Empirical comparisons use external baselines (LoLCATs, Liger-GLA, Lizard, LoLA, STILL) and external model families (LLaMA, Qwen). No step in the derivation chain reduces to its own inputs by construction. The skeptic's concern about Lemma 1's Taylor error bound being potentially large in practice is a correctness/approximation-quality issue, not a circularity issue.

Axiom & Free-Parameter Ledger

6 free parameters · 3 axioms · 1 invented entities

The paper introduces 5-6 families of trainable parameters (gate/beta projections, feature maps, short conv, LoRA) and 2 hand-set hyperparameters (window=56, sink=8). The key concentration and key normalization assumptions are domain-specific and empirically validated only partially. The kGLA variant is an invented architectural component without strong independent support.

free parameters (6)
  • Gate projection weights (W_r1_alpha, W_r2_alpha, b_alpha) = Trained on DCLM-Edu, 10M tokens
    Newly introduced parameters for GLA/kGLA/GDN gate computation, trained during linearization.
  • Beta projection weights (W_beta, b_beta) = Trained on DCLM-Edu, 10M tokens
    Per-head scalar beta for GDN delta updates, trained during linearization.
  • Hedgehog feature map weights (W) = Trained on DCLM-Edu, 10M tokens
    Feature map parameters for kernelized linear attention variant.
  • Short convolution kernel and bias = kernel_size=8, trained
    Parameters for the short convolution adaptation of Q/K/V projections.
  • LoRA parameters (rank 8, alpha 16) = Trained on DCLM-Edu, 10M tokens
    Low-rank adaptation weights for Q/K/V/O projections.
  • SWA window size and sink token count = w=56, s=8 (total budget 64)
    Fixed by hand following prior work convention; not fitted but chosen a priori.
axioms (3)
  • domain assumption Keys are approximately concentrated around their mean direction
    Invoked in Lemma 3 (Sec. 3.2.1) to replace the product of rank-1 operators with a single projection. Empirically checked in Fig. 3 but shown to fail at early sequence positions.
  • domain assumption Keys are normalized to unit norm
    Invoked in Lemmas 2 and 3 (Sec. 3.2.1). The paper applies L2 normalization to keys (Fig. 1) but this is an architectural choice, not a property of pretrained models.
  • standard math First-order Taylor expansion around x=0 is a valid approximation for softmax
    Invoked in Lemma 1 (Sec. 3.2.1). The error bound grows with ||x||^2, which may be large for typical attention pre-activations.
invented entities (1)
  • kGLA (Key-Gated Linear Attention) no independent evidence
    purpose: A GLA variant with key-dependent diagonal gating to better approximate softmax
    Introduced in Sec. 3.2.2 as a structural midpoint between GLA and GDN. Empirical results (Tables 1-3) show it rarely outperforms GLA after training, providing no independent evidence of utility.

pith-pipeline@v1.1.0-glm · 24150 in / 2193 out tokens · 331233 ms · 2026-07-09T01:28:18.717734+00:00 · methodology

0 comments
read the original abstract

The quadratic cost of causal self-attention severely bottlenecks long-context transformer inference. While numerous post hoc linearization pipelines exist, it is difficult to identify which components preserve model quality. This work isolates the effect of state update design in a strict frozen-backbone regime. We show that softmax relies on key-dependent, rank-1 orthogonal projections, elucidating why delta-style networks outperform purely gated accumulation. We identify a potential source of approximation errors and introduce structural interventions, specifically sink tokens, short convolutions, and fixed-budget cache routing, which reduces the remaining gap. We scale this linearization approach across LLaMA and Qwen models up to 32B parameters, outperforming prior post hoc baselines on MMLU and matching the long-context retrieval of complex adaptive-caching frameworks.

Figures

Figures reproduced from arXiv: 2607.07706 by Anna Kuzina, Babak Ehteshami Bejnordi, Paul N. Whatmough.

Figure 1
Figure 1. Figure 1: Attention replacement blocks evaluated in our strict frozen-backbone setting. Blue ` markers denote frozen pretrained parameters. The pretrained parameters including Wq,Wk,Wv, and RMSNorm parameters are kept frozen, while only the newly introduced mechanism-specific parameters are trained. (a) Standard Softmax. (b) Hedgehog [27] replaces the softmax kernel with learned feature maps φ. (c) GLA [23] introduc… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of linear attention variants for Llama3.1-8b linearization. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Key Concentration and Projection Residual for LLama3.1-8b [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Hybrid setting: keeping full attention for 10% to 50% layers . Finally, our approach is competitive with Llamba [3] on common reasoning despite Llamba relying on full-model finetuning over a much larger token budget (12B), highlighting the strength of delta-style updates plus minimal compensations in the post hoc setting. Several recent linearization works further improve performance by augmenting the cach… view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of linear attention variants for Qwen3-8b linearization. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Key Concentration and Projection Residual for Qwen3-8b [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 27 canonical work pages · 17 internal anchors

  1. [1]

    Smollm2: When smol goes big – data-centric training of a small language model, 2025

    Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíˇcek, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Wer...

    work page 2025

  2. [2]

    Physics of language models: Part 4.1, architecture design and the magic of canon layers.arXiv preprint arXiv:2512.17351, 2025

    Zeyuan Allen-Zhu. Physics of language models: Part 4.1, architecture design and the magic of canon layers.arXiv preprint arXiv:2512.17351, 2025

    work page arXiv 2025

  3. [3]

    Llamba: Scaling Distilled Recurrent Models for Efficient Language Processing

    Aviv Bick, Tobias Katsch, Nimit Sohoni, Arjun Desai, and Albert Gu. Llamba: Scaling distilled recurrent models for efficient language processing.arXiv preprint arXiv:2502.14458, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  4. [4]

    Rethinking Attention with Performers

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers.arXiv preprint arXiv:2009.14794, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2009

  5. [5]

    FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning.arXiv preprint arXiv:2307.08691, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  6. [6]

    Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality.arXiv preprint arXiv:2405.21060, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  7. [7]

    Myosotis: structured computation for attention like layer.arXiv preprint arXiv:2509.20503, 2025

    Evgenii Egorov, Hanno Ackermann, Markus Nagel, and Hong Cai. Myosotis: structured computation for attention like layer.arXiv preprint arXiv:2509.20503, 2025

    work page arXiv 2025

  8. [8]

    Liger: Linearizing Large Language Models to Gated Recurrent Structures

    Disen Lan, Weigao Sun, Jiaxi Hu, Jusen Du, and Yu Cheng. Liger: Linearizing large language models to gated recurrent structures.arXiv preprint arXiv:2503.01496, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  9. [9]

    Datacomp-lm: In search of the next generation of training sets for language models.Advances in Neural Information Processing Systems, 37:14200–14282, 2024

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. Datacomp-lm: In search of the next generation of training sets for language models.Advances in Neural Information Processing Systems, 37:14200–14282, 2024

    work page 2024

  10. [10]

    Longhorn: State Space Models are Amortized Online Learners

    Bo Liu, Rui Wang, Lemeng Wu, Yihao Feng, Peter Stone, and Qiang Liu. Longhorn: State space models are amortized online learners.arXiv preprint arXiv:2407.14207, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  11. [11]

    Lola: Low-rank linear attention with sparse caching.arXiv preprint arXiv:2505.23666, 2025

    Luke McDermott, Robert W Heath Jr, and Rahul Parhi. Lola: Low-rank linear attention with sparse caching.arXiv preprint arXiv:2505.23666, 2025

    work page arXiv 2025

  12. [12]

    Still: Selecting tokens for intra-layer hybrid attention to linearize llms.arXiv preprint arXiv:2602.02180, 2026

    Weikang Meng, Liangyu Huo, Yadan Luo, Jiawen Guan, Jingyi Zhang, Yingjian Li, and Zheng Zhang. Still: Selecting tokens for intra-layer hybrid attention to linearize llms.arXiv preprint arXiv:2602.02180, 2026

    work page arXiv 2026

  13. [13]

    Linearizing Large Language Models

    Jean Mercat, Igor Vasiljevic, Sedrick Keh, Kushal Arora, Achal Dave, Adrien Gaidon, and Thomas Kollar. Linearizing large language models.arXiv preprint arXiv:2405.06640, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  14. [14]

    cosFormer: Rethinking Softmax in Attention

    Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. cosformer: Rethinking softmax in attention.arXiv preprint arXiv:2202.08791, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  15. [15]

    On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention

    Yeonju Ro, Zhenyu Zhang, Souvik Kundu, Zhangyang Wang, and Aditya Akella. On- the-fly adaptive distillation of transformer to dual-state linear attention.arXiv preprint arXiv:2506.09316, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  16. [16]

    Retentive Network: A Successor to Transformer for Large Language Models

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  17. [17]

    Eleutherai/lm-evaluation- harness: v0

    Lintang Sutawika, Hailey Schoelkopf, Leo Gao, Baber Abbasi, Stella Biderman, Jonathan Tow, Charles Lovering, Jason Phang, Anish Thite, Thomas Wang, et al. Eleutherai/lm-evaluation- harness: v0. 4.9.Zenodo, 2025. 10

    work page 2025

  18. [18]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    work page 2023

  19. [19]

    Kimi Linear: An Expressive, Efficient Attention Architecture

    Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, Jiaxi Hu, Fanqing Meng, Chengyin Liu, Xin Men, Songlin Yang, Zhiyuan Li, et al. Kimi linear: An expressive, efficient attention architecture.arXiv preprint arXiv:2510.26692, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  20. [20]

    Lizard: An Efficient Linearization Framework for Large Language Models

    Chien Van Nguyen, Ruiyi Zhang, Hanieh Deilamsalehy, Puneet Mathur, Viet Dac Lai, Haoliang Wang, Jayakumar Subramanian, Ryan A Rossi, Trung Bui, Nikos Vlassis, et al. Lizard: An efficient linearization framework for large language models.arXiv preprint arXiv:2507.09025, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  21. [21]

    Efficient Streaming Language Models with Attention Sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks.arXiv preprint arXiv:2309.17453, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  22. [22]

    Gated Delta Networks: Improving Mamba2 with Delta Rule

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule.arXiv preprint arXiv:2412.06464, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  23. [23]

    Gated Linear Attention Transformers with Hardware-Efficient Training

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training.arXiv preprint arXiv:2312.06635, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  24. [24]

    Parallelizing linear transformers with the delta rule over sequence length.Advances in neural information processing systems, 37:115491–115522, 2024

    Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length.Advances in neural information processing systems, 37:115491–115522, 2024

    work page 2024

  25. [25]

    Fla: A triton-based library for hardware-efficient implementations of linear attention mechanism, January 2024

    Songlin Yang and Yu Zhang. Fla: A triton-based library for hardware-efficient implementations of linear attention mechanism, January 2024

    work page 2024

  26. [26]

    LoLCATs: On Low-Rank Linearizing of Large Language Models

    Michael Zhang, Simran Arora, Rahul Chalamala, Alan Wu, Benjamin Spector, Aaryan Singhal, Krithik Ramesh, and Christopher Ré. Lolcats: On low-rank linearizing of large language models. arXiv preprint arXiv:2410.10254, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  27. [27]

    The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry

    Michael Zhang, Kush Bhatia, Hermann Kumbong, and Christopher Ré. The hedgehog & the porcupine: Expressive linear attentions with softmax mimicry.arXiv preprint arXiv:2402.04347, 2024. 11 A Proofs A.1 Linear Approximation Lemma 4(First–order Taylor approximation of softmax).Let p(x) = softmax(x), xi =q ⊤ t ki, and ¯x= 1 t Pt j=1 xj.Then for each indexi, pi...

    work page internal anchor Pith review Pith/arXiv arXiv 2024