Pith. sign in

REVIEW 3 cited by

Star Attention: Efficient LLM Inference over Long Sequences

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2411.17116 v3 pith:2WVXFXZ2 submitted 2024-11-26 cs.CL cs.AIcs.LG

Shantanu Acharya , Fei Jia , Boris Ginsburg This is my paper
classification cs.CLcs.AIcs.LG
keywords attentioninferencestaracrosshostsllmslongphase
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Inference with Transformer-based Large Language Models (LLMs) on long sequences is both costly and slow due to the quadratic complexity of the self-attention mechanism. We introduce Star Attention, a two-phase block-sparse approximation that improves computational efficiency by sharding attention across multiple hosts while minimizing communication overhead. In the first phase, the context is processed using blockwise-local attention across hosts, in parallel. In the second phase, query and response tokens attend to all prior cached tokens through sequence-global attention. Star Attention integrates seamlessly with most Transformer-based LLMs trained with global attention, reducing memory requirements and inference time by up to 11x while preserving 97-100% of accuracy.

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Elastic Attention Cores for Scalable Vision Transformers

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    VECA learns effective visual representations using core-periphery attention where patches interact exclusively via a resolution-invariant set of learned core embeddings, achieving linear O(N) complexity while maintain...

  2. FlashSVD: Memory-Efficient Inference with Streaming for Low-Rank Models

    cs.LG 2025-08 conditional novelty 6.0 of 10

    FlashSVD fuses low-rank SVD projections into attention and feed-forward GPU kernels so SVD-compressed transformers avoid materializing dense activations, cutting activation memory at a latency cost.

  3. A Survey of Context Engineering for Large Language Models

    cs.CL 2025-07 accept novelty 4.0 of 10

    The survey organizes Context Engineering into retrieval, processing, management, and integrated systems like RAG and multi-agent setups while identifying an asymmetry where LLMs handle complex inputs well but struggle...

Pith tools