Pith. sign in

REVIEW 5 cited by

DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Training

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.03294 v2 pith:3HBLRQPH submitted 2023-10-05 cs.LG cs.AIcs.DC

Dacheng Li , Rulin Shao , Anze Xie , Eric P. Xing , Xuezhe Ma , Ion Stoica , Joseph E. Gonzalez , Hao Zhang This is my paper
classification cs.LGcs.AIcs.DC
keywords distflashattnattentioncomparedllmsspeeduptrainingachievesdistributed
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

FlashAttention (Dao, 2023) effectively reduces the quadratic peak memory usage to linear in training transformer-based large language models (LLMs) on a single GPU. In this paper, we introduce DISTFLASHATTN, a distributed memory-efficient attention mechanism optimized for long-context LLMs training. We propose three key techniques: token-level workload balancing, overlapping key-value communication, and a rematerialization-aware gradient checkpointing algorithm. We evaluate DISTFLASHATTN on Llama-7B and variants with sequence lengths from 32K to 512K. DISTFLASHATTN achieves 8x longer sequences, 4.45 - 5.64x speedup compared to Ring Self-Attention, 2 - 8x longer sequences, 1.24 - 2.01x speedup compared to Megatron-LM with FlashAttention. It achieves 1.67x and 1.26 - 1.88x speedup compared to recent Ring Attention and DeepSpeed-Ulysses. Code is available at https://github.com/RulinShao/LightSeq.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling

    cs.AI 2026-08 conditional novelty 7.0 of 10

    Adding 12% cross-repository code dependency contexts to the long-context fine-tuning mix improves long-range retrieval, state tracking, repo code understanding, and agentic tool use, while largely preserving short-con...

  2. Scaling Generative Recommendations with Context Parallelism on Hierarchical Sequential Transducers

    cs.IR 2025-07 conditional novelty 6.0 of 10

    Context parallelism with AllToAll and load-balanced mini-chunks lets HSTU recommendation models support 5.3x longer interaction sequences at 1.55x throughput scaling.

  3. Arctic Long Sequence Training: Scalable And Efficient Training For Multi-Million Token Sequences

    cs.LG 2025-06 conditional novelty 6.0 of 10

    ALST combines tiled loss and MLP compute, Ulysses-style sequence parallelism for Hugging Face GQA and MQA models, and activation-checkpoint CPU offload to train Llama-8B at up to 500K tokens on one H100 and 15M tokens...

  4. QLPO: Quadrant-weighted Sampling for Length-aware Policy Optimization

    cs.AI 2026-07 conditional novelty 5.0 of 10

    QLPO resamples GRPO training groups to favor short correct and long incorrect responses, cutting reasoning length substantially while keeping accuracy roughly unchanged.

  5. Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library

    cs.LG 2025-06 conditional novelty 5.0 of 10

    ROLL is a distributed RL training library for LLMs with sample-level rollout scheduling and flexible device mapping, demonstrated on RLVR and agentic tasks but without baseline or scaling measurements.

Pith tools