Pith. sign in

REVIEW 31 cited by

Simple linear attention language models balance the recall-throughput tradeoff

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.18668 v2 pith:VUBIJUV7 submitted 2024-02-28 cs.CL cs.LG

Simran Arora , Sabri Eyuboglu , Michael Zhang , Aman Timalsina , Silas Alberti , Dylan Zinsley , James Zou , Atri Rudra
show 1 more author
Christopher Ré
This is my paper · ORCID
classification cs.CLcs.LG
keywords attentionmodelslanguagelinearrecallsizestatetradeoff
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Recent work has shown that attention-based language models excel at recall, the ability to ground generations in tokens previously seen in context. However, the efficiency of attention-based models is bottle-necked during inference by the KV-cache's aggressive memory consumption. In this work, we explore whether we can improve language model efficiency (e.g. by reducing memory consumption) without compromising on recall. By applying experiments and theory to a broad set of architectures, we identify a key tradeoff between a model's state size and recall ability. We show that efficient alternatives to attention (e.g. H3, Mamba, RWKV) maintain a fixed-size recurrent state, but struggle at recall. We propose BASED a simple architecture combining linear and sliding window attention. By varying BASED window size and linear attention feature dimension, we can dial the state size and traverse the pareto frontier of the recall-memory tradeoff curve, recovering the full quality of attention on one end and the small state size of attention-alternatives on the other. We train language models up to 1.3b parameters and show that BASED matches the strongest sub-quadratic models (e.g. Mamba) in perplexity and outperforms them on real-world recall-intensive tasks by 6.22 accuracy points. Implementations of linear attention are often less efficient than optimized standard attention implementations. To make BASED competitive, we develop IO-aware algorithms that enable 24x higher throughput on language generation than FlashAttention-2, when generating 1024 tokens using 1.3b parameter models. Code for this work is provided at: https://github.com/HazyResearch/based.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 31 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Orthogonalized Read Is a Removable Training Scaffold for Recurrent Memory

    cs.LG 2026-07 accept novelty 7.0 of 10

    The orthogonalized read on noisy recall is a removable training scaffold: it multiplies escape hazard ~6.4x during plateau training, but a stock mLSTM achieves full accuracy once the read is annealed away.

  2. MesaNet: Sequence Modeling by Locally Optimal Test-Time Training

    cs.LG 2025-06 conditional novelty 7.0 of 10

    MesaNet uses conjugate-gradient-optimal test-time regression in a chunkwise-parallelizable recurrent layer, achieving strong language modeling and benchmark performance at up to 1B scale.

  3. Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism

    cs.LG 2025-04 conditional novelty 7.0 of 10

    In both Transformers and SSMs, in-context retrieval is performed by a small set of Gather and Aggregate heads, and the Transformer-SSM gap can be reduced to how well these few heads implement this mechanism.

  4. DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling

    cs.LG 2026-08 conditional novelty 6.0 of 10

    DART augments Mamba-2 with attention over its own chunk state contributions, improving associative recall and retrieval with a smaller inference cache than token-level attention.

  5. Raven: High-Recall Sequence Modeling with Sparse Memory Routing

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Raven is a linear-time sequence model whose sparse, input-dependent routing writes tokens into dedicated memory slots, preserving long-context recall and extrapolating 16x beyond training length.

  6. The Capability Convergence Hypothesis: Capability from Access Structure, Not Scale

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Long-context capability converges to hybrid architectures pairing O(1) state with a scalable index; pure scaling of SSMs or Transformers alone hits information, horizon, or circuit walls.

  7. Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

    cs.LG 2026-07 conditional novelty 6.0 of 10

    SDM sparsifies the Gated DeltaNet update rule to enable 1000x larger recurrent memory states at iso-FLOP, improving long-context recall and short-context reasoning over GDN and matching full attention at 8B scale.

  8. ELiTeFormer: An Efficient Transformer for FPGAs

    cs.AR 2026-07 conditional novelty 6.0 of 10

    Hybrid linear attention plus ternary projections, co-designed with a multiplier-free PE, deliver 10× weight and 12.8× KV-cache compression with competitive MMLU and FPGA speedups over LLaMA 3 on A100.

  9. Neural Attention Search Linear: Towards Adaptive Token-Level Hybrid Attention Models

    cs.CL 2026-02 conditional novelty 6.0 of 10

    NAtS-L learns per-chunk routing between Gated DeltaNet and softmax attention, improving long-context retrieval and length extrapolation over fixed hybrids.

  10. Controllably Efficient Language Models

    cs.LG 2025-11 conditional novelty 6.0 of 10

    A single transformer variant can compress past context into chunk summaries and use chunk size as a test-time knob to trade quality against speed and memory, outperforming many efficient baselines on recall benchmarks.

  11. Customizing the Inductive Biases of Softmax Attention using Structured Matrices

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Structured-matrix scoring functions, BTT and MLR, let attention escape the low-rank bottleneck and add a distance-dependent compute bias, improving accuracy for fixed compute on regression, language modeling, and forecasting.

  12. DTRNet: Dynamic Token Routing Network to Reduce Quadratic Costs in Transformers

    cs.LG 2025-08 conditional novelty 6.0 of 10

    A learned router sends about 90% of tokens through a linear projection path instead of quadratic attention, keeps the MLP for every token, and matches dense Transformer accuracy at roughly 85% of the FLOPs.

  13. Cartridges: Lightweight and general-purpose long context representations via self-study

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A per-corpus trained KV cache, called a Cartridge, matches full-context in-context learning quality on long-document benchmarks while using up to 38.6x less serving memory.

  14. Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltas

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Chipmunk speeds up diffusion transformer inference by recomputing, at each step, only the top percent of attention and MLP activation columns that change most between steps, caching the rest in column-sparse GPU kernels.

  15. Overflow Prevention Enhances Long-Context Recurrent LLMs

    cs.LG 2025-05 conditional novelty 6.0 of 10

    OPRM, a training-free chunk-and-select inference method, improves recurrent LLMs by 14-51% on LongBench and sets a 7B-class LongBench v2 record, while raising doubts about whether recurrent models exploit long-range d...

  16. Quantifying Memory Utilization with Effective State-Size

    cs.LG 2025-04 conditional novelty 6.0 of 10

    Effective state-size, the rank of a causal operator submatrix, is proposed and empirically validated as a memory-utilization proxy that beats cache size at predicting recall-task accuracy and reveals how LLMs reset co...

  17. Task-Agnostic Language Model Watermarking via High Entropy Passthrough Layers

    cs.CL 2024-12 reject novelty 6.0 of 10

    A backdoor watermark for LLMs using passthrough layers trained to output high-entropy text on a private key, with near-perfect extraction in benign settings but with layer-removal robustness contradicted by the paper'...

  18. Hymba: A Hybrid-head Architecture for Small Language Models

    cs.CL 2024-11 conditional novelty 6.0 of 10

    Hymba-1.5B, a hybrid-head model with parallel attention and Mamba heads plus learnable meta tokens, achieves state-of-the-art accuracy among sub-2B models and outperforms Llama-3.2-3B with an 11.67x smaller cache and ...

  19. Selective Attention: Enhancing Transformer through Principled Context Control

    cs.LG 2024-11 conditional novelty 6.0 of 10

    Selective Self-Attention adds query- and value-dependent temperature scaling to transformer attention, improving language modeling accuracy and passkey retrieval with under 0.5% extra parameters.

  20. MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map

    cs.LG 2024-11 reject novelty 6.0 of 10

    MetaLA removes the Key matrix from linear attention, adds input-dependent decay plus self-augmentation, and reports gains over many linear baselines.

  21. Distill-then-Replace: Efficient Task-Specific Hybrid Attention Model Construction

    cs.LG 2026-01 conditional novelty 5.0 of 10

    Distill-then-Replace builds task-specific hybrid attention LLMs by distilling each full-attention block into a linear counterpart and greedily replacing layers under a validation-performance constraint.

  22. Elucidating the Design Space of Decay in Linear Attention

    cs.CL 2025-09 conditional novelty 5.0 of 10

    A controlled study of decay in linear attention finds median decay near 0.8 works best, vector decay generally beats scalar decay, and RoPE/TPE give little benefit for models with sub-unity decay.

  23. SCOUT: Toward Sub-Quadratic Attention via Segment Compression for Optimized Utility in Transformers

    cs.LG 2025-08 conditional novelty 5.0 of 10

    SCOUT matches full-attention transformer scores on long-context language modeling and reasoning benchmarks at 400M and 1.3B scales while attending only to strided checkpoint tokens.

  24. ENA: Efficient N-dimensional Attention

    cs.LG 2025-08 conditional novelty 5.0 of 10

    ENA combines linear recurrence with hardware-friendly sliding tile attention to model images and videos efficiently, claiming Transformer-level accuracy at roughly 70% attention sparsity.

  25. LASP-2: Rethinking Sequence Parallelism for Linear Attention and Its Hybrid

    cs.LG 2025-02 conditional novelty 5.0 of 10

    LASP-2 trains linear-attention transformers with long sequences by exchanging GPU memory states in one all-gather step, improving throughput over prior sequence-parallel methods.

  26. Naju: A Native Discrete State-Space Model with Independent Retention and Writing for Long-Sequence Memory

    cs.AI 2026-07 conditional novelty 4.0 of 10

    Decoupling the forget gate from the input gate in a discrete state-space recurrence lets one fixed-size state both preserve old bindings over long horizons and overwrite stale ones.

  27. Convolution for Large Language Models

    cs.CL 2026-07 conditional novelty 4.0 of 10

    Adding a residual depthwise convolution (kernel 3) to QKV projections before attention raises average downstream accuracy in Qwen3-1.7B/4B by 1.6-3.8 points with negligible parameter cost.

  28. GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference

    cs.AI 2026-06 reject novelty 4.0 of 10

    GLIDE allocates softmax attention only to early/middle transformer layers and linearizes later layers, cutting KV-cache I/O by up to ~62× while retaining 92–96% of baseline accuracy.

  29. Scaling Context Requires Rethinking Attention

    cs.LG 2025-07 conditional novelty 4.0 of 10

    On 64k-token natural language training, power attention with degree 2 achieves lower loss per FLOP than both softmax attention and existing linear attention, and its GPU kernels run faster than Flash Attention at long...

  30. Softplus Attention with Re-weighting Boosts Length Extrapolation in Large Language Models

    cs.CL 2025-01 conditional novelty 4.0 of 10

    A two-stage softplus-based attention mechanism with re-weighting (LSSAR) is reported to keep validation loss nearly flat when a 124M-parameter GPT-2 is tested at up to 16x its 1024-token training length.

  31. MSWA: Refining Local Attention with Multi-ScaleWindow Attention

    cs.CL 2025-01 conditional novelty 4.0 of 10

    MSWA assigns exponentially increasing attention window sizes across heads and layers, improving sliding-window attention's language modeling and reasoning performance at lower relative cost.

Pith tools