Pith. sign in

REVIEW 4 cited by

DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Training

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.03294 v2 pith:3HBLRQPH submitted 2023-10-05 cs.LG cs.AIcs.DC

Dacheng Li , Rulin Shao , Anze Xie , Eric P. Xing , Xuezhe Ma , Ion Stoica , Joseph E. Gonzalez , Hao Zhang This is my paper
classification cs.LGcs.AIcs.DC
keywords distflashattnattentioncomparedllmsspeeduptrainingachievesdistributed
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

FlashAttention (Dao, 2023) effectively reduces the quadratic peak memory usage to linear in training transformer-based large language models (LLMs) on a single GPU. In this paper, we introduce DISTFLASHATTN, a distributed memory-efficient attention mechanism optimized for long-context LLMs training. We propose three key techniques: token-level workload balancing, overlapping key-value communication, and a rematerialization-aware gradient checkpointing algorithm. We evaluate DISTFLASHATTN on Llama-7B and variants with sequence lengths from 32K to 512K. DISTFLASHATTN achieves 8x longer sequences, 4.45 - 5.64x speedup compared to Ring Self-Attention, 2 - 8x longer sequences, 1.24 - 2.01x speedup compared to Megatron-LM with FlashAttention. It achieves 1.67x and 1.26 - 1.88x speedup compared to recent Ring Attention and DeepSpeed-Ulysses. Code is available at https://github.com/RulinShao/LightSeq.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling

    cs.AI 2026-08 conditional novelty 7.0 of 10

    Adding 12% cross-repository code dependency contexts to the long-context fine-tuning mix improves long-range retrieval, state tracking, repo code understanding, and agentic tool use, while largely preserving short-con...

  2. Scaling Generative Recommendations with Context Parallelism on Hierarchical Sequential Transducers

    cs.IR 2025-07 conditional novelty 6.0 of 10

    Context parallelism with AllToAll and load-balanced mini-chunks lets HSTU recommendation models support 5.3x longer interaction sequences at 1.55x throughput scaling.

  3. Arctic Long Sequence Training: Scalable And Efficient Training For Multi-Million Token Sequences

    cs.LG 2025-06 conditional novelty 6.0 of 10

    ALST combines tiled loss and MLP compute, Ulysses-style sequence parallelism for Hugging Face GQA and MQA models, and activation-checkpoint CPU offload to train Llama-8B at up to 500K tokens on one H100 and 15M tokens...

  4. QLPO: Quadrant-weighted Sampling for Length-aware Policy Optimization

    cs.AI 2026-07 conditional novelty 5.0 of 10

    QLPO resamples GRPO training groups to favor short correct and long incorrect responses, cutting reasoning length substantially while keeping accuracy roughly unchanged.

Pith tools