Pith. sign in

REVIEW 17 cited by

Striped Attention: Faster Ring Attention for Causal Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2311.09431 v1 pith:O2Q6VE6I submitted 2023-11-15 cs.LG cs.CL

William Brandon , Aniruddha Nrusimha , Kevin Qian , Zachary Ankner , Tian Jin , Zhiye Song , Jonathan Ragan-Kelley This is my paper · ORCID
classification cs.LGcs.CL
keywords attentionringcausalsequencestripedtransformerableachieve
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

To help address the growing demand for ever-longer sequence lengths in transformer models, Liu et al. recently proposed Ring Attention, an exact attention algorithm capable of overcoming per-device memory bottle- necks by distributing self-attention across multiple devices. In this paper, we study the performance characteristics of Ring Attention in the important special case of causal transformer models, and identify a key workload imbal- ance due to triangular structure of causal attention computations. We propose a simple extension to Ring Attention, which we call Striped Attention to fix this imbalance. Instead of devices having contiguous subsequences, each device has a subset of tokens distributed uniformly throughout the sequence, which we demonstrate leads to more even workloads. In experiments running Striped Attention on A100 GPUs and TPUv4s, we are able to achieve up to 1.45x end-to-end throughput improvements over the original Ring Attention algorithm on causal transformer training at a sequence length of 256k. Furthermore, on 16 TPUv4 chips, we were able to achieve 1.65x speedups at sequence lengths of 786k. We release the code for our experiments as open source

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SlimPipe: Memory-Thrifty and Efficient Pipeline Parallelism for Long-Context LLM Training

    cs.LG 2025-04 conditional novelty 7.0 of 10

    A slice-level pipeline-parallel schedule with attention-work redistribution that cuts activation memory roughly by the pipeline size and reduces pipeline bubbles for long-context LLM training.

  2. LV-XAttn: Distributed Cross-Attention for Long Visual Inputs in Multimodal Large Language Models

    cs.CV 2025-02 conditional novelty 7.0 of 10

    LV-XAttn speeds up distributed cross-attention in multimodal LLMs by keeping large visual KV blocks local and rotating small query blocks among GPUs, achieving up to 10.62x wall-clock speedups over Ring Attention on l...

  3. FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelism

    cs.DC 2024-12 conditional novelty 7.0 of 10

    FlexSP is a training system that adaptively forms heterogeneous sequence-parallelism groups and time-balances sequence assignment, improving long-context LLM training throughput by up to 1.98x.

  4. From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving

    cs.DC 2026-06 accept novelty 6.5 of 10

    KV-cache serving systems concentrate into five archetypes under a four-axis taxonomy, with ownership explaining residual distributed design variance and seven measurement gaps blocking next steps.

  5. Libra: Taming Attention Workload Skew in Long-Context LLM Training with Bounded Sequence Pool

    cs.DC 2026-07 conditional novelty 6.0 of 10

    Libra bounds attention load-balancing to fixed-size sequence pools and combines variance-reduced sequence placement with tiled attention pooling, achieving up to 2.54x end-to-end throughput over Ulysses on 256K- and 1...

  6. A Training-Memory Regression in MLA Sequence Parallelism: Why Megatron-Core Forbids Absorption, and LAGA -- a Communication-Efficient Fix

    cs.DC 2026-07 conditional novelty 6.0 of 10

    LAGA replaces MLA training's per-head K/V all-to-all with a latent all-gather and local up-projection, matching explicit-form memory while cutting collective communication ~1.98x.

  7. DASH: Deterministic Attention Scheduling for High-throughput Reproducible LLM Training

    cs.LG 2026-01 conditional novelty 6.0 of 10

    DASH speeds up deterministic attention backprop by reversed and shift-based tile schedules, giving up to 1.28× throughput and ~5% end-to-end gains while keeping bitwise reproducibility.

  8. Arctic Long Sequence Training: Scalable And Efficient Training For Multi-Million Token Sequences

    cs.LG 2025-06 conditional novelty 6.0 of 10

    ALST combines tiled loss and MLP compute, Ulysses-style sequence parallelism for Hugging Face GQA and MQA models, and activation-checkpoint CPU offload to train Llama-8B at up to 500K tokens on one H100 and 15M tokens...

  9. DDiT: Dynamic Resource Allocation for Diffusion Transformer Model Serving

    cs.DC 2025-06 conditional novelty 6.0 of 10

    DDiT cuts text-to-video serving latency by up to 1.44x via DiT-VAE phase decoupling and step-level, starvation-aware GPU reassignment.

  10. Hetu v2: A General and Scalable Deep Learning System with Hierarchical and Heterogeneous Single Program Multiple Data Annotations

    cs.DC 2025-04 conditional novelty 6.0 of 10

    Hetu v2 introduces hierarchical and heterogeneous SPMD annotations plus graph specialization and dynamic graph switching, enabling efficient LLM training on heterogeneous, unstable, and mixed-length workloads.

  11. TurboAttention: Efficient Attention Approximation For High Throughputs LLMs

    cs.LG 2024-12 conditional novelty 6.0 of 10

    A hybrid INT8/INT4/INT2 quantization plus polynomial-LUT softmax lets LLM attention run without FP32 dequantization, giving up to 1.8x speedup and 2.37x throughput on an A100.

  12. Motif 3: Technical Report

    cs.AI 2026-08 conditional novelty 5.0 of 10

    A 314B-parameter mixture-of-experts language model with grouped differential latent attention and multi-teacher distillation reports competitive results on agentic, coding, and reasoning benchmarks.

  13. QLPO: Quadrant-weighted Sampling for Length-aware Policy Optimization

    cs.AI 2026-07 conditional novelty 5.0 of 10

    QLPO resamples GRPO training groups to favor short correct and long incorrect responses, cutting reasoning length substantially while keeping accuracy roughly unchanged.

  14. Efficient Scaling of LLM Training with Flexible Context Parallelism

    cs.DC 2026-02 reject novelty 5.0 of 10

    Regrouping context-parallel workers per micro-batch with non-power-of-two group sizes, chosen by a fitted cost model plus dynamic programming, is claimed to speed up multimodal LLM training by up to 1.36x (1.46x per t...

  15. Efficient Pretraining Length Scaling

    cs.CL 2025-04 conditional novelty 5.0 of 10

    PHD-Transformer repeats input tokens K times during pretraining with a custom attention mask, then drops the repeated tokens' KV cache, yielding average accuracy gains over a vanilla baseline at 1.2B scale.

  16. TokenRing: An Efficient Parallelism Framework for Infinite-Context LLMs via Bidirectional Communication

    cs.DC 2024-12 conditional novelty 4.0 of 10

    TokenRing claims faster long-sequence attention on multiple GPUs by overlapping forward query transfers with backward partial-output transfers, supported by one 4-GPU profiling comparison.

  17. Taming the Titans: A Survey of Efficient LLM Inference Serving

    cs.CL 2025-04 conditional novelty 3.0 of 10

    A survey that classifies LLM inference serving techniques into instance, cluster, emerging-scenario, and miscellaneous categories, with a companion GitHub index and four research directions.

Pith tools