Pith. sign in

REVIEW 3 major objections 6 minor 12 references

Linear-attention architectures form a multi-objective frontier of loss, throughput, and memory control, and routing each layer’s write value into the shared residual stream modestly improves final validation loss for DeltaNet-style models.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-07-10 14:47 UTC pith:VHBJ5V5A

load-bearing objection Useful comparative map of DeltaNet-family mixers plus a small, honest CLVR finding; single-run deltas are the soft spot, not the algebra or the framing. the 3 major comments →

arxiv 2607.07953 v1 pith:VHBJ5V5A submitted 2026-07-08 cs.LG cs.AI

Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing

Tommaso Cerruti , Tim Rieder , George Rowlands , Lingfeng Jin , Imanol Schlag This is my paper
classification cs.LG cs.AI
keywords Linear attentionRecurrent associative memoryDeltaNetGated DeltaNetKimi Delta AttentionCross-layer routingCLVRTraining throughput
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Softmax attention is expressive but quadratic in sequence length, so this paper compares it with four recurrent linear-attention designs—DeltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2—under one shared recurrent-memory notation. In matched 350M-parameter runs to 15B tokens, no single design wins every objective: Kimi Delta Attention with Muon in a hybrid stack reaches the lowest final validation loss, pure Gated DeltaNet with AdamW is the fastest trainer, hybrid stacks generally improve loss at a throughput cost, and Muon consistently beats AdamW on final loss. The authors also test lightweight cross-layer routing for these memories: forwarding a lower layer’s delta-rule write error into the next layer’s value target does not help, but routing the write value into the aligned residual stream (CLVR) modestly lowers final validation loss for both DeltaNet and Gated DeltaNet while keeping linear-time structure. A sympathetic reader cares because the comparison makes the design space legible—selectivity, forgetting, erase/write control, and stack pattern—rather than declaring a single best mixer.

Core claim

Within the reported 350M-parameter, 15B-token sweep, Kimi Delta Attention with Muon in a hybrid stack reaches the lowest final validation loss, pure Gated DeltaNet with AdamW has the highest normalized training throughput, hybrid stacks generally improve loss at a throughput cost, and Muon consistently lowers final validation loss relative to AdamW. The natural DeltaNet-inspired cross-layer idea—forwarding write error into the next layer’s value target—does not improve over matched baselines; routing the write value into the shared residual stream (CLVR) does, modestly, for DeltaNet and Gated DeltaNet.

What carries the argument

A common recurrent-memory notation for the associative state W, the delta-rule residual r = v − Wκ, and scalar or channel-wise decay/erase/write gates, plus Cross-Layer Value Routing (CLVR): projecting each routing-capable layer’s write value into the shared residual stream through a zero-initialized projection so later layers read an aligned signal without a new depth-attention operator.

Load-bearing premise

That single-run final validation-loss gaps—especially CLVR gains on the order of a few thousandths to about one hundredth with no seed repeats—are stable enough to rank architectures and routing variants.

What would settle it

Repeated-seed matched training of the same 350M and larger DeltaNet/Gated DeltaNet hosts with and without CLVR showing that the final-loss advantage of value routing disappears or reverses once seed variance is measured.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper compares softmax attention with four recurrent linear-attention variants (DeltaNet, Gated DeltaNet, Kimi Delta Attention, Gated DeltaNet-2) under a shared recurrent-memory notation that isolates the memory matrix W, the delta residual r, and the decay/erase/write controls. On a controlled 350M-parameter, 15B-token FineWeb-Edu sweep it reports a multi-objective frontier: Kimi Delta Attention with Muon in a hybrid stack reaches the lowest final validation loss (2.273), pure Gated DeltaNet with AdamW has the highest normalized training throughput, hybrid stacks generally improve loss at a throughput cost, and Muon consistently improves matched final loss relative to AdamW. Sequence-length iteration-time measurements (4k–32k) confirm the expected scaling advantage of pure recurrent stacks. The authors further introduce lightweight cross-layer routing for DeltaNet-style memories: Cross-Layer Error Residuals (CLER), which injects a lower layer’s write error into the next layer’s value target, does not improve over matched baselines, while Cross-Layer Value Routing (CLVR), which injects the write value into the shared residual stream via a zero-initialized projection, yields small final-loss reductions on the reported DeltaNet and Gated DeltaNet runs. Code and launchers are released.

Significance. If the reported trade-offs hold, the work supplies a useful, reproducible map of the recent DeltaNet family that is currently missing from the literature: a common algebraic notation, matched training recipes, optimizer/LR sensitivity, hybrid-versus-pure stack effects, and long-context iteration-time scaling. The CLVR result, though modest, is a concrete and falsifiable design insight—that the write value, not the delta residual, is the useful cross-layer signal when injected into the residual stream—and is cleanly ablated against CLER and CLER-H with zero-init projections and matched hosts. The public Megatron implementations of KDA, Gated DeltaNet-2, and the routing variants lower the barrier for follow-up work. The contribution is primarily empirical systems/architecture rather than a new theoretical guarantee, but the careful scoping and open code make it a solid reference point for linear-attention design choices.

major comments (3)
  1. Section 6 and Appendix B state that every architecture and routing row is a single run with no seed averages or standard deviations; the authors themselves flag gaps below roughly 10^{-3}–10^{-2} as weak. Table 7’s CLVR gains (−0.0103/−0.0119 at 350M/1B, shrinking to −0.0059/−0.0016 at 350M/15B and −0.0019 at 1.3B/40B) sit inside or near that band. Without at least a small multi-seed check on the main CLVR vs. baseline pairs (or an explicit noise estimate from repeated short runs), the positive routing claim remains suggestive rather than established. The coarser frontier (Muon helps, pure is faster, hybrid recovers loss, pure scales better at 32k) is less sensitive to this issue and can stand; the fine ranking among close hybrid-Muon rows and the CLVR effect need either more seeds or a clearer “exploratory” framing in the abstract and conclusion.
  2. Table 7 and Section 5.3: the only larger-scale (1.3B/40B) hidden-stream routing comparison is for Gated DeltaNet; there is no matched DeltaNet row at that scale, and routing is never evaluated on Kimi Delta Attention or Gated DeltaNet-2. The abstract and conclusion present CLVR as lowering loss “for both DeltaNet and Gated DeltaNet,” which is true only at 350M. Either add the missing larger DeltaNet (and ideally one richer host) comparison, or restrict the claim language to the scales and hosts actually measured so that the generalization statement does not outrun the evidence.
  3. Section 5.2 / Figure 4 and Table 2: learning-rate preference is strongly architecture- and optimizer-dependent (AdamW linear variants prefer ~1e-3; Muon prefers ~3e-4). The main 15B sweep uses a single recorded peak LR per family rather than a full re-tune for every cell. While the paper correctly warns that a single default LR can distort comparisons, the reported “best final loss” ranking (KDA+Muon hybrid) could still shift under a more complete LR grid. A short note quantifying residual LR sensitivity for the top few rows, or an explicit statement that the ranking is conditional on the recorded LRs, would make the frontier claim more robust.
minor comments (6)
  1. Abstract and Section 1: the paper correctly notes that speed results are training throughput/iteration time only; a one-sentence reminder in the conclusion that inference latency and memory footprint remain unmeasured would help readers avoid over-generalizing the scaling claims.
  2. Figure 6 caption and surrounding text: the growth factors (+192% / +65% / +8%) are clear; adding the absolute iteration times already stated in the text into the figure legend would make the plot self-contained.
  3. Section 4.3: the distinction among CLER, CLER-H, and CLVR is carefully drawn, but the first use of “CLER-V” in Table 5 is not defined in the main text; either drop the alias or introduce it once.
  4. Table 2: “Relative speed” is normalized to pure Gated DeltaNet + AdamW = 100%; stating the absolute ktok/s/GPU (or TFLOP/s) for that reference row in the caption would aid external comparison.
  5. References: arXiv preprints for Kimi Linear, Gated DeltaNet-2, Attention Residuals, and Mixture-of-Depths Attention are recent; ensure final versions or DOIs are updated if available at camera-ready.
  6. Appendix B inclusion criteria are exemplary; a one-line pointer from Section 6 to Appendix B would help readers who skip the appendix understand why some exploratory controls are omitted from the main tables.

Circularity Check

0 steps flagged

No significant circularity: empirical architecture and routing comparisons measured against held-out data and matched baselines, not forced by definition or self-citation.

full rationale

This is a comparative systems/ML paper. Softmax attention, DeltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2 are rewritten in a shared recurrent-memory notation (Eqs. 1–24) for clarity; that is algebraic re-expression of prior mechanisms, not a derivation that claims to predict new quantities from fitted constants. The main claims—KDA+Muon hybrid lowest final validation loss (2.273), pure GDN+AdamW highest normalized throughput, hybrid vs pure trade-offs, Muon vs AdamW, and CLVR modestly lowering final val loss while CLER does not—are single-run empirical measurements on FineWeb-Edu held-out loss, training throughput, and matched non-routing baselines (Tables 2, 6, 7; Sections 5–6). Routing variants (CLER, CLER-H, CLVR) are ablations that inject internal signals and report measured deltas against parameter-matched hosts; they are not predictions forced by construction. Citations to Schlag et al., Yang et al., Kimi Team, Hatamizadeh et al., etc. identify host architectures and related work; they do not supply a uniqueness theorem or ansatz that forces the reported ranking or the CLVR result. Self-citation load is not load-bearing for the central claims. Weaknesses (single-run noise, small CLVR deltas) are statistical/evidence-strength issues, not circularity. Score 0 is appropriate.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 3 invented entities

The work is empirical. Load-bearing premises are standard LM-training assumptions plus design choices that define the comparison (token budget, hybrid pattern, optimizers, zero-init routing projections). No new physical entities. Free parameters are training hyperparameters and learned routing scalars/projections, not constants fitted to force a theoretical prediction.

free parameters (4)
  • Peak learning rates (AdamW ~3e-4 / 1e-3; Muon ~3e-4 / 1.5e-3)
    Chosen via ablation and defaults; preferred LR depends on optimizer and mixer and materially affects final loss rankings.
  • Hybrid interleave ratio (default 2:1 linear:softmax; some 3:1 GDN-2 variants)
    Stack composition is a free design choice that trades loss against throughput and is not derived.
  • CLVR/CLER projection Pl (zero-initialized, optionally low-rank) and scalar Γl
    Learned routing strength parameters; zero-init is a training choice that defines the baseline-matched start.
  • Token budgets and model depths (350M/15B main; 1B routing ablations; 1.3B/40B and 3B/60B DeltaNet)
    Scale choices that bound all quantitative claims; different FineWeb-Edu slices across budgets limit absolute-loss comparison.
axioms (4)
  • domain assumption Feature-map linear attention can be written as a recurrent memory update W(i) with constant-size state and linear training cost.
    Standard linear-attention premise from §2; underpins all DeltaNet-family comparisons.
  • domain assumption Held-out FineWeb-Edu cross-entropy under fixed tokenizer/sequence/batch is a valid primary quality metric for architecture ranking at this scale.
    §5 metric definitions; downstream tasks are only secondary checks.
  • ad hoc to paper Single-run final-checkpoint losses are comparable within a matched sweep when scale, optimizer, LR, and stack are recorded.
    Explicit inclusion rule in §5 and Appendix B; authors note no seed averages.
  • ad hoc to paper Routing only on DeltaNet/Gated DeltaNet hosts is sufficient to evaluate the proposed cross-layer idea.
    §5.3 and §6.6; no routing results on KDA or Gated DeltaNet-2.
invented entities (3)
  • Cross-Layer Error Residuals (CLER) no independent evidence
    purpose: Forward a lower layer’s delta-rule write residual into the next routing-capable layer’s value target.
    New routing pathway defined in §4.2; empirically inactive/neutral in matched runs.
  • Cross-Layer Value Routing (CLVR) no independent evidence
    purpose: Project each routing-capable layer’s write value into the shared residual stream via a zero-initialized projection.
    New pathway in §4.3; modest loss reductions reported in Table 7 only under the paper’s matched single runs.
  • CLER-H no independent evidence
    purpose: Ablation that routes the write error into the shared hidden stream instead of the value target.
    Control pathway isolating injection space vs signal content (§4.3).

reviewed 2026-07-10 · how reviews work

0 comments
read the original abstract

Self-attention lets each token retrieve information from the full context, but its quadratic cost in sequence length limits training and inference at long context. This paper presents a comparative study of softmax attention and four recent recurrent linear-attention architectures: DeltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2. We express these mechanisms in a common recurrent-memory notation, making explicit how they differ in expressivity, memory decay, erase and write control, training throughput, and implementation complexity. Our experiments center on 350M-parameter models trained for 15B tokens, and include optimizer and learning-rate comparisons, hybrid-versus-pure stack comparisons, sequence-length runtime measurements, larger DeltaNet runs at 1.3B and 3B parameters, and a small set of downstream evaluations. The reported speed results measure training throughput and iteration time; we do not provide an empirical inference-speed benchmark. Within the reported 350M-parameter, 15B-token sweep, Kimi Delta Attention with Muon reaches the lowest final validation loss, a pure Gated DeltaNet stack trained with AdamW has the highest normalized training throughput, hybrid stacks generally improve loss at a throughput cost, and Muon consistently lowers final validation loss relative to AdamW in the matched architecture settings we evaluate. We introduce and evaluate lightweight cross-layer routing mechanisms for DeltaNet-style memories. The most natural DeltaNet-inspired formulation, forwarding a lower layer's delta-rule write error into the next layer's value target, does not improve over matched baselines. Routing into the aligned hidden stream and forwarding the write value instead yields a modest improvement in the matched runs we report: Cross-Layer Value Routing (CLVR) lowers final validation loss for both DeltaNet and Gated DeltaNet.

Figures

Figures reproduced from arXiv: 2607.07953 by George Rowlands, Imanol Schlag, Lingfeng Jin, Tim Rieder, Tommaso Cerruti.

Figure 1
Figure 1. Figure 1: Cross-Layer Error Residuals (CLER). A lower routing-capable recurrent-memory layer computes a DeltaNet [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Cross-Layer Value Routing (CLVR). Each routing-capable layer projects its internal write value into the [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Validation-loss scaling for the 350M-parameter runs trained to 15B tokens. Lower is better. In this sweep, [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Learning-rate ablation for 350M-parameter hybrid models trained for 2000 steps, approximately 1.05B [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Learning-rate ablation iso plots for 350M-parameter hybrid models trained for 2000 steps. The iso plots [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Iteration-time scaling from 4k to 32k tokens. Softmax attention time grows much faster with sequence length [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 12 canonical work pages · 5 internal anchors

  1. [1]

    N., Kaiser, L., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. InAdvances in Neural Information Processing Systems, volume 30, pages 5998–6008, 2017

    work page 2017

  2. [2]

    Linear transformers are secretly fast weight programmers

    Schlag, I., Irie, K., and Schmidhuber, J. Linear transformers are secretly fast weight programmers. InProceedings of the 38th International Conference on Machine Learning, pages 9355–9366. PMLR, 2021

    work page 2021

  3. [3]

    Gated Delta Networks: Improving Mamba2 with Delta Rule

    Yang, S., Kautz, J., and Hatamizadeh, A. Gated Delta Networks: Improving Mamba2 with Delta Rule. InThe Thirteenth International Conference on Learning Representations, 2025

    work page 2025

  4. [4]

    Kimi Linear: An Expressive, Efficient Attention Architecture

    Kimi Team. Kimi Linear: An Expressive, Efficient Attention Architecture.arXiv preprint arXiv:2510.26692, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  5. [5]

    Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention

    Hatamizadeh, A., Choi, Y ., and Kautz, J. Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention. arXiv preprint arXiv:2605.22791, 2026. 18 Linear Attention Architectures

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  6. [6]

    Attention Residuals

    Kimi Team. Attention Residuals.arXiv preprint arXiv:2603.15031, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  7. [7]

    Mixture-of-Depths Attention.arXiv preprint arXiv:2603.15619, 2026

    Zhu, L., Fang, Y ., Liao, B., Wang, S., Cheng, T., Huang, Z., Chen, C., Wei, L., Zeng, Y ., Wang, Y ., Lin, Y ., Li, Y ., and Wang, X. Mixture-of-Depths Attention.arXiv preprint arXiv:2603.15619, 2026

    work page arXiv 2026

  8. [8]

    Value Residual Learning

    Zhou, Z., Wu, T., Jiang, Z., Obeid, F., and Lan, Z. Value Residual Learning. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 28341–28356. Association for Computational Linguistics, 2025

    work page 2025

  9. [9]

    The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

    Penedo, G., Kydlíˇcek, H., Ben Allal, L., Lozhkov, A., Mitchell, M., Raffel, C., von Werra, L., and Wolf, T. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale.arXiv preprint arXiv:2406.17557, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  10. [10]

    Llama 2: Open Foundation and Fine-Tuned Chat Models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., and others. Llama 2: Open Foundation and Fine-Tuned Chat Models.arXiv preprint arXiv:2307.09288, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  11. [11]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled Weight Decay Regularization. InInternational Conference on Learning Representations, 2019

    work page 2019

  12. [12]

    Muon: An optimizer for hidden layers in neural networks

    Jordan, K., Jin, Y ., Boza, V ., You, J., Cesista, F., Newhouse, L., and Bernstein, J. Muon: An optimizer for hidden layers in neural networks. 2024.https://kellerjordan.github.io/posts/muon/. 19 Linear Attention Architectures Table 8: Supplementary 350M baseline runs trained for approximately 1B tokens. These shorter-budget runs provide context before th...

    work page arXiv 2024

This paper was first reviewed by grok-4.5 on July 10, 2026.