Pith. sign in

REVIEW 8 cited by

The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2504.17768 v3 pith:Q2KJM2DX submitted 2025-04-24 cs.CL cs.LG

Piotr Nawrot , Robert Li , Renjie Huang , Sebastian Ruder , Kelly Marchisio , Edoardo M. Ponti This is my paper
classification cs.CLcs.LG
keywords sparseattentionmethodssparsityanalysiscostduringestimation
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Sparse attention offers a promising strategy to extend long-context capabilities in Transformer LLMs, yet its efficiency-accuracy trade-offs remain unclear due to the lack of comprehensive evaluation. We address this gap with the largest-scale empirical analysis to date of training-free sparse attention, evaluating six methods across multiple model families and sizes, sequences up to 128K tokens, and sparsity levels up to 0.95 (i.e., $1/20$ attention budget) on nine diverse tasks. We first organise the rapidly evolving landscape of sparse attention methods into a taxonomy along four design axes. Our analysis then yields actionable insights: 1) sparse attention is effective: larger sparse models outperform smaller dense ones at equivalent cost, improving the Pareto frontier; 2) for the training-free methods we study, fine-grained per-query importance estimation during prefilling remains impractical-due to both the cost of estimation and the lack of sparse kernels that translate fine-grained sparsity into wall-clock gains-forcing a task-dependent choice between global-to-token and block-to-block selection. Instead, during decoding, token-to-page selection becomes feasible, enabling better generalisation and higher sparsity tolerance; 3) longer sequences tolerate higher sparsity, suggesting that fixed-budget methods in production are suboptimal. Together, these findings provide practical guidance for deploying sparse attention and methodological recommendations for future evaluations. Our code is available at https://github.com/PiotrNawrot/sparse-frontier.

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding

    cs.LG 2026-07 accept novelty 7.0 of 10

    Page-local rank-8 spectral key summaries let sparse decode selection track the exact mass oracle and match FullKV quality at ~2% attended tokens with 2× latency cut at 1M context.

  2. Learning What Matters: Supervising Global Context Pruning with Causal Evidence Sets

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Attention is an unreliable supervision target for context pruning; causal evidence sets recovered by masking produce routers that match or beat attention-distilled routers, especially on multi-hop and conflicting-fact tasks.

  3. KVpop -- Key-Value Cache Compression with Predictive Online Pruning

    cs.LG 2026-07 conditional novelty 6.0 of 10

    KVpop supervises fixed-budget KV eviction with an efficiently computed future-attention target and delayed mLSTM scoring, retaining ~97–100% of full-attention math performance at 75–88% compression.

  4. Controllably Efficient Language Models

    cs.LG 2025-11 conditional novelty 6.0 of 10

    A single transformer variant can compress past context into chunk summaries and use chunk size as a test-time knob to trade quality against speed and memory, outperforming many efficient baselines on recall benchmarks.

  5. Attention's forward pass and Frank-Wolfe

    math.OC 2025-08 conditional novelty 6.0 of 10

    Hardmax self-attention is shown to be a Frank-Wolfe iteration; with positive-definite key-query it converges to Voronoi-cell vertices, and a Markov-chain version of soft attention is metastable there for exponential-i...

  6. SCOUT: Toward Sub-Quadratic Attention via Segment Compression for Optimized Utility in Transformers

    cs.LG 2025-08 conditional novelty 5.0 of 10

    SCOUT matches full-attention transformer scores on long-context language modeling and reasoning benchmarks at 400M and 1.3B scales while attending only to strided checkpoint tokens.

  7. Physics- and geometry-aware spatio-spectral graph neural operator for time-independent and time-dependent PDEs

    cs.LG 2025-08 unverdicted novelty 4.0 of 10

    A submission whose abstract describes a new graph neural operator for PDEs but whose full text is a different paper, leaving the claimed method and results unverifiable.

  8. Lifelong In-Context Learning with Transformers Requires Parametric Forms of Attention

    cs.LG 2026-06 unverdicted novelty 3.0 of 10

    Argues that parametric attention forms are necessary for lifelong in-context learning in transformers to maintain constant memory footprint over arbitrary sequence lengths.

Pith tools