Pith. sign in

REVIEW 46 cited by

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2006.16236 v3 pith:XBCSNFJQ submitted 2020-06-29 cs.LG stat.ML

Angelos Katharopoulos , Apoorv Vyas , Nikolaos Pappas , François Fleuret This is my paper
classification cs.LGstat.ML
keywords transformersautoregressivelinearachievecomplexityleftlengthlong
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from $\mathcal{O}\left(N^2\right)$ to $\mathcal{O}\left(N\right)$, where $N$ is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our linear transformers achieve similar performance to vanilla transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.

Discussion (0). Sign in to comment.

Forward citations

Cited by 46 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. WriteSAE: Sparse Autoencoders for Recurrent State

    cs.LG 2026-05 unverdicted novelty 8.0 of 10

    WriteSAE is the first sparse autoencoder that factors decoder atoms into the native d_k x d_v cache write shape of recurrent models and supplies a closed-form per-token logit shift for atom substitution.

  2. WriteSAE: Sparse Autoencoders for Recurrent State

    cs.LG 2026-05 unverdicted novelty 8.0 of 10

    WriteSAE decomposes recurrent model cache writes into substitutable atoms with a closed-form logit shift, achieving high substitution success and targeted behavioral installs on models like Qwen3.5 and Mamba-2.

  3. WriteSAE: Sparse Autoencoders for Recurrent State

    cs.LG 2026-05 unverdicted novelty 8.0 of 10

    WriteSAE introduces sparse autoencoders with rank-1 matrix atoms for recurrent state updates, allowing replacement tests that outperform deletion on 92.4% of positions and a formula predicting logit changes with R²=0.98.

  4. Chem-GMNet: A Sphere-Native Geometric Transformer for Molecular Property Prediction

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    Chem-GMNet uses sphere-native embeddings, DualSKA attention, and SH-FFN layers to match or beat ChemBERTa-2 on MoleculeNet tasks with fewer parameters and sometimes no pretraining.

  5. WriteSAE: Sparse Autoencoders for Recurrent State

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    WriteSAE factors sparse autoencoder decoder atoms to the native d_k x d_v cache write shape in recurrent models, provides a closed-form logit shift, and demonstrates high success in atom substitution and behavioral ed...

  6. Retrieval from Within: An Intrinsic Capability of Attention-Based Models

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    Attention-based models can intrinsically retrieve and reuse pre-encoded evidence chunks via decoder attention queries, unifying retrieval with generation and outperforming external RAG pipelines on QA benchmarks.

  7. Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation

    cs.LG 2025-09 unverdicted novelty 7.0 of 10

    Robust Filter Attention models self-attention as consistency-based state estimation under a linear SDE for token trajectories, matching standard attention complexity while showing lower perplexity and better zero-shot...

  8. Rethinking Attention with Performers

    cs.LG 2020-09 unverdicted novelty 7.0 of 10

    Performers approximate full-rank softmax attention in Transformers via FAVOR+ random features for linear complexity, with theoretical guarantees of unbiased estimation and competitive results on pixel, text, and prote...

  9. Planning as Emergent Behavior in Reinforcement Learning with Relational Hidden States

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Relational hidden states anchored to environment states are what let a model-free RL agent plan, and a free-slot control without that anchoring shows no planning signatures.

  10. Infrared Organization and Critical Cognitive Field Formation in Transformer Dynamics

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Pythia layer Jacobians reorganize during training into an approximately flat infrared TDOS ρ(λ)∼λ^{-0.1} with K(t)∼1/t kernels and a transient memory-self-energy maximum interpreted as critical cognitive-field formation.

  11. Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

    cs.LG 2026-07 conditional novelty 6.0 of 10

    SDM sparsifies the Gated DeltaNet update rule to enable 1000x larger recurrent memory states at iso-FLOP, improving long-context recall and short-context reasoning over GDN and matching full attention at 8B scale.

  12. ELiTeFormer: An Efficient Transformer for FPGAs

    cs.AR 2026-07 conditional novelty 6.0 of 10

    Hybrid linear attention plus ternary projections, co-designed with a multiplier-free PE, deliver 10× weight and 12.8× KV-cache compression with competitive MMLU and FPGA speedups over LLaMA 3 on A100.

  13. A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets

    cs.AI 2026-07 unverdicted novelty 6.0 of 10

    HOLA pairs a compressive delta-rule recurrent state with a residual-selected exact KV cache and decoupled RMSNorm-gamma read, yielding lower perplexity than both standard linear attention and full-attention baselines ...

  14. Memory-Managed Long-Context Attention: Bounded Editable Memory with a Hard Lifecycle and Calibrated Sparse Fallback

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    A hybrid attention mechanism with editable request-local memory slots and sparse fallback achieves high accuracy on synthetic overwrite, version, and anti-pollution tasks where pure fixed-state or sparse methods fail,...

  15. Gated Bidirectional Linear Attention for Generative Retrieval

    cs.IR 2026-06 unverdicted novelty 6.0 of 10

    GBLA extends kernelized linear attention with local causal mixing, key gating, and gated RMSNorm; a 1:2 hybrid with self-attention matches full bidirectional self-attention quality on Yandex Music data while deliverin...

  16. The Transformer as a Polar State Estimator

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    The standard Transformer block arises as a first-order approximation to a polar state estimator on the hypersphere, with a Polar Transformer retaining higher-order terms.

  17. The Transformer as a Polar State Estimator

    cs.LG 2026-05 conditional novelty 6.0 of 10

    The paper casts the standard Transformer block with RoPE as a first-order approximation of a radial–tangential state estimator and introduces a Polar Transformer variant that retains the discarded geometric corrections.

  18. The Transformer as a Polar State Estimator

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Transformer components arise as the natural solution to precision-weighted directional state estimation on the hypersphere.

  19. Structured Recurrent Mixers for Massively Parallelized Sequence Generation

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    Structured Recurrent Mixers enable algebraic switching between parallel training and recurrent inference representations, delivering higher efficiency, information capacity, and throughput than other linear-complexity models.

  20. Structured Recurrent Mixers for Massively Parallelized Sequence Generation

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    Structured Recurrent Mixers provide a dual parallel-recurrent representation for sequence models, claiming superior training efficiency, information capacity, and inference throughput over linear complexity alternatives.

  21. Structured Recurrent Mixers for Massively Parallelized Sequence Generation

    cs.CL 2026-05 conditional novelty 6.0 of 10

    Structured Recurrent Mixers enable algebraic switching between parallel training and recurrent inference representations, yielding higher throughput, concurrency, and training efficiency than comparable linear-complex...

  22. Retrieval from Within: An Intrinsic Capability of Attention-Based Models

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Attention-based models can retrieve evidence intrinsically by using decoder attention to score and reuse their own pre-encoded chunks, outperforming separate retrieval pipelines on QA benchmarks.

  23. HubRouter: A Pluggable Sub-Quadratic Routing Primitive for Hybrid Sequence Models

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    HubRouter is a sub-quadratic routing primitive using learned hubs that replaces attention layers in hybrid models while delivering competitive perplexity and large throughput gains.

  24. In-Place Test-Time Training

    cs.LG 2026-04 conditional novelty 6.0 of 10

    In-Place TTT adapts LLM MLP projection matrices at test time with a next-token-aligned objective and chunk-wise updates, enabling better long-context performance as a drop-in enhancement.

  25. Incremental Transformer Neural Processes

    cs.LG 2026-02 conditional novelty 6.0 of 10

    A causally masked, KV-cached Transformer Neural Process (incTNP) updates its context in linear time and matches or exceeds standard TNP accuracy on tabular and weather-stream benchmarks.

  26. Short window attention enables long-term memorization

    cs.LG 2025-09 unverdicted novelty 6.0 of 10

    Short sliding windows in hybrid attention-xLSTM models boost long-context performance by encouraging long-term memory use, and stochastic window sizing improves both short and long tasks.

  27. ICR-RL: Deep Reinforcement Learning via In-Context Regression

    cs.LG 2025-09 conditional novelty 6.0 of 10

    A pre-trained tabular regression transformer (TabPFN) can act as a Q-function approximator in fitted Q iteration, matching DQN on CartPole, MountainCar, and Acrobot without gradient updates.

  28. Customizing the Inductive Biases of Softmax Attention using Structured Matrices

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Structured-matrix scoring functions, BTT and MLR, let attention escape the low-rank bottleneck and add a distance-dependent compute bias, improving accuracy for fixed compute on regression, language modeling, and forecasting.

  29. Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation

    cs.LG 2025-09 reject novelty 6.0 of 10

    AFA re-derives self-attention as a precision-weighted robust state estimator for a linear SDE, recovering rotary encodings in a limit, but the abstract's language-modeling results are absent from the text.

  30. Deformable DETR: Deformable Transformers for End-to-End Object Detection

    cs.CV 2020-10 accept novelty 6.0 of 10

    Deformable DETR achieves higher accuracy than DETR, especially on small objects, while converging in one-tenth the training epochs by using sparse deformable attention on image features.

  31. Memory-Managed Long-Context Attention: Bounded Editable Memory with a Hard Lifecycle and Calibrated Sparse Fallback

    cs.CL 2026-06 conditional novelty 5.0 of 10

    Editable bounded memory slots with lifecycle control plus sparse fallback cover overwrite and no-signal long-context cases that pure fixed-state or pure sparse methods fail under controlled conditions.

  32. Q-Delta: Beyond Key-Value Associative State Evolution

    cs.AI 2026-06 unverdicted novelty 5.0 of 10

    Q-Delta extends linear attention by introducing a query-conditioned delta rule that incorporates mixed key-query errors into recurrent state updates for improved stability and performance.

  33. Structured Recurrent Mixers for Massively Parallelized Sequence Generation

    cs.CL 2026-05 conditional novelty 5.0 of 10

    By restricting token mixing to rank-1 decayed matrices, the same model trains in parallel but generates recurrently with constant memory, yielding large throughput gains in compute-matched benchmarks.

  34. StreamIndex: Memory-Bounded Compressed Sparse Attention via Streaming Top-k

    cs.LG 2026-05 accept novelty 5.0 of 10

    Chunked streaming top-k enables CSA indexer execution at 1M sequence length with 6.21 GB peak memory and >=0.998 recall on synthetic V4-shaped inputs.

  35. Toeplitz MLP Mixers are Low Complexity, Information-Rich Sequence Models

    cs.LG 2026-04 unverdicted novelty 5.0 of 10

    Toeplitz MLP Mixers replace attention with masked Toeplitz multiplications for sub-quadratic complexity while retaining more sequence information and outperforming on copying and in-context tasks.

  36. NVIDIA Nemotron 3: Efficient and Open Intelligence

    cs.CL 2025-12 unverdicted novelty 5.0 of 10

    NVIDIA releases the Nemotron 3 model family with hybrid Mamba-Transformer architecture, LatentMoE, NVFP4 training, MTP layers, and multi-environment RL post-training for reasoning and agentic tasks.

  37. StateX: Enhancing RNN Recall via Post-training State Expansion

    cs.CL 2025-09 unverdicted novelty 5.0 of 10

    StateX post-trains RNNs to expand recurrent state size, improving recall and in-context learning with negligible parameter growth.

  38. Infrared Organization and Critical Cognitive Field Formation in Transformer Dynamics

    cs.LG 2026-07 reject novelty 4.0 of 10

    Slow relaxation modes in Pythia transformers accumulate toward zero rate during training, yielding a near-flat infrared spectrum and 1/t memory kernels—but the claimed 'critical cognitive field formation' is not direc...

  39. GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference

    cs.AI 2026-06 reject novelty 4.0 of 10

    GLIDE allocates softmax attention only to early/middle transformer layers and linearizes later layers, cutting KV-cache I/O by up to ~62× while retaining 92–96% of baseline accuracy.

  40. Cognitive Field Theory: Memory-Dressed Collective Dynamics of Intelligence

    q-bio.NC 2026-01 reject novelty 4.0 of 10

    The paper asserts that Hopfield networks, RNNs, transformers, and the author's FHRN model are all special cases of a single stochastic field equation whose collective time-scale spectrum governs cognition.

  41. Evaluation of Finetuned LLMs in AMR Parsing

    cs.CL 2025-08 conditional novelty 4.0 of 10

    Simple finetuning of LLaMA 3.2 reaches SMATCH F1 0.804 on the AMR 3.0 test set, matching the APT+Silver parser and coming within 0.05 of the Graphene state of the art.

  42. LLMOrbit: A Circular Taxonomy of Large Language Models -From Scaling Walls to Agentic AI Systems

    cs.LG 2026-01 unverdicted novelty 3.0 of 10

    A survey taxonomy of LLMs identifies three scaling crises and six efficiency paradigms while tracing the shift from generation to tool-using agents.

  43. Cognitive Field Theory: Memory-Dressed Collective Dynamics of Intelligence

    q-bio.NC 2026-01 reject novelty 3.0 of 10

    A field-theoretic reframing in which cognition is the spectrum of slow collective modes of generic Langevin dynamics, whose central one-loop result is assumed and whose headline memory-dressing claims do not appear in...

  44. WaveLLDM: Design and Development of a Lightweight Latent Diffusion Model for Speech Enhancement and Restoration

    cs.SD 2025-08 conditional novelty 3.0 of 10

    WaveLLDM, a lightweight latent diffusion model with a neural codec, achieves low spectral distortion (LSD 0.48-0.60) on speech restoration but scores far below SOTA on PESQ and STOI.

  45. Rethinking Transformer Connectivity: TLinFormer, A Path to Exact, Full Context-Aware Linear Attention

    cs.LG 2025-08 reject novelty 3.0 of 10

    TLinFormer compresses long history into a fixed-size context state to make each full forward pass linear in sequence length, but the attention is not exact and per-token generation still costs O(N).

  46. A Vision Toward Energy-Efficient Domain-Specific Artificial Intelligence Models and Agents

    cs.AI 2025-10 unverdicted novelty 2.0 of 10

    A position paper proposing compact, domain-specific AI agents as the path to ≥1000× energy efficiency, without demonstrating the claim.

Pith tools