REVIEW 21 cited by
Parallelizing Linear Transformers with the Delta Rule over Sequence Length
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Transformers with linear attention (i.e., linear transformers) and state-space models have recently been suggested as a viable linear-time alternative to transformers with softmax attention. However, these models still underperform transformers especially on tasks that require in-context retrieval. While more expressive variants of linear transformers which replace the additive update in linear transformers with the delta rule (DeltaNet) have been found to be more effective at associative recall, existing algorithms for training such models do not parallelize over sequence length and are thus inefficient to train on modern hardware. This work describes a hardware-efficient algorithm for training linear transformers with the delta rule, which exploits a memory-efficient representation for computing products of Householder matrices. This algorithm allows us to scale up DeltaNet to standard language modeling settings. We train a 1.3B model for 100B tokens and find that it outperforms recent linear-time baselines such as Mamba and GLA in terms of perplexity and zero-shot performance on downstream tasks. We also experiment with two hybrid models which combine DeltaNet layers with (1) sliding-window attention layers every other layer or (2) two global attention layers, and find that these hybrids outperform strong transformer baselines.
Forward citations
Cited by 21 Pith papers
-
PhyCheck: Fine-Grained Evidence-Grounded Dataset for Physical Law Understanding in Video-LLMs
PhyCheck is a 69,825-pair video QA benchmark that tests and improves Video-LLMs' ability to judge whether events obey physical laws, with fine-grained evidence questions and a context-sensitivity pilot.
-
The Orthogonalized Read Is a Removable Training Scaffold for Recurrent Memory
The orthogonalized read on noisy recall is a removable training scaffold: it multiplies escape hazard ~6.4x during plateau training, but a stock mLSTM achieves full accuracy once the read is annealed away.
-
pLSTM: parallelizable Linear Source Transition Mark networks
pLSTM extends linear recurrent networks to general directed acyclic graphs with a parallelizable scheme and two stabilization modes for long-range propagation.
-
What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents
KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.
-
Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
SDM sparsifies the Gated DeltaNet update rule to enable 1000x larger recurrent memory states at iso-FLOP, improving long-context recall and short-context reasoning over GDN and matching full attention at 8B scale.
-
ELiTeFormer: An Efficient Transformer for FPGAs
Hybrid linear attention plus ternary projections, co-designed with a multiplier-free PE, deliver 10× weight and 12.8× KV-cache compression with competitive MMLU and FPGA speedups over LLaMA 3 on A100.
-
Pretraining Recurrent Networks without Recurrence
SMT trains nonlinear RNNs by imitating one-step memory-transition labels generated by a Transformer, replacing BPTT's unrolled credit assignment with time-parallel supervised learning.
-
Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers
Canon layers—residual 1-d causal convolutions over adjacent tokens—boost synthetic reasoning depth 2-4x, lift NoPE to RoPE level, and bring GLA up to Mamba2/GDN, with qualitative real-world confirmation.
-
Cartridges: Lightweight and general-purpose long context representations via self-study
A per-corpus trained KV cache, called a Cartridge, matches full-context in-context learning quality on long-document benchmarks while using up to 38.6x less serving memory.
-
HAD: Hybrid Architecture Distillation Outperforms Teacher in Genomic Sequence Modeling
A compact hybrid GDN+attention model distilled from Nucleotide Transformer v2 outperforms similarly sized models and, on several tasks, its 500x larger teacher.
-
Solving Empirical Bayes via Transformers
A transformer pre-trained on synthetic Poisson data can beat the classical NPMLE estimator on several empirical Bayes tasks and run about 100x faster.
-
An Uncertainty Principle for Linear Recurrent Neural Networks
For linear RNNs, recalling an input K steps back with S hidden units has best-case error about 1-S/K when K exceeds S, with the filter's width scaling as K/S.
-
LADY: Linear Attention for Autonomous Driving Efficiency without Transformers
LADY shows that an end-to-end driving model using only linear attention can match transformer-based planners on NAVSIM/Bench2Drive while fusing arbitrary-length historical sensor frames at constant per-frame cost.
-
Elucidating the Design Space of Decay in Linear Attention
A controlled study of decay in linear attention finds median decay near 0.8 works best, vector decay generally beats scalar decay, and RoPE/TPE give little benefit for models with sub-unity decay.
-
A Survey on Latent Reasoning
A survey that organizes latent reasoning methods into vertical recurrence, horizontal recurrence, and infinite-depth diffusion, arguing that silent reasoning can beat explicit chain-of-thought.
-
TPTT: Transforming Pretrained Transformers into Titans
TPTT claims to convert pretrained transformers into memory-augmented linear-attention models with LoRA fine-tuning, but the reported MMLU results are marred by implausibly low baseline scores.
-
SeerAttention-R: Sparse Attention Adaptation for Long Reasoning
A learned gate selects the important KV blocks during long decoding, preserving math reasoning accuracy while skipping up to 90% of attention work.
-
Scaling Reasoning without Attention
A purely attention-free Mamba-2 model, fine-tuned with PromptCoT-style synthetic data, reaches state-of-the-art scores on AIME and LiveCodeBench among open models of its size and surpasses Gemma3-27B on three benchmarks.
-
Understanding Transformer from the Perspective of Associative Memory
Frames the Transformer as associative memory, derives retrieval SNR for linear, softmax, ReLU, and SoLU kernels, and proposes DeltaFormer, a softmax-plus-delta-rule variant claimed to exceed TC0 expressivity.
-
ModRWKV: Transformer Multimodality in Linear Time
A linear RNN backbone (RWKV7) with lightweight adapters can handle vision, speech, and time-series inputs, with competitive vision and speech results but unreliable time-series evaluation.
-
A Vision Toward Energy-Efficient Domain-Specific Artificial Intelligence Models and Agents
A position paper proposing compact, domain-specific AI agents as the path to ≥1000× energy efficiency, without demonstrating the claim.
Discussion (0). Continue with ORCID to comment.