REVIEW 23 cited by
USP: A Unified Sequence Parallelism Approach for Long Context Generative AI
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Sequence parallelism (SP), which divides the sequence dimension of input tensors across multiple computational devices, is becoming key to unlocking the long-context capabilities of generative AI models. This paper investigates the state-of-the-art SP approaches, i.e. DeepSpeed-Ulysses and Ring-Attention, and proposes a unified SP approach, which is more robust to transformer model architectures and network hardware topology. This paper compares the communication and memory cost of SP and existing parallelism, including data/tensor/zero/pipeline parallelism, and discusses the best practices for designing hybrid 4D parallelism involving SP. We achieved 47% MFU on two 8xA800 nodes using SP for the LLAMA3-8B model training using sequence length 208K. Our code is publicly available at https://github.com/feifeibear/long-context-attention.
Forward citations
Cited by 23 Pith papers
-
Zellige: Moldable Sequence Placement for Mixed Image-Video DiT Training
Zellige assigns each training sequence its own parallelism configuration and GPU ranks, outperforming prior disjoint-group placement systems by 1.12 to 1.54x in end-to-end DiT training.
-
X-Stage: An Overlooked Pipeline Stage for Communication-Computation Overlap in DiT Inference
X-Stage is a measurable post-issue window for GPU remote stores; a Burst-Gap model predicts backpressure and guides scheduling, yielding up to 1.62x MoE and 1.43x attention speedups.
-
LiveAnimate: Stable Long-Form Streaming Human Animation in Real-Time
LiveAnimate is a 14-billion-parameter video diffusion system that generates stable, identity-consistent human animation at about 20 frames per second from a streaming pose input.
-
StateFlow: Sequence Pipeline Parallelism for Long-Context Modeling with Linear Recurrence
StateFlow is a sequence pipeline parallelism system that propagates boundary states across sequence chunks to reduce activation memory and pipeline bubbles when training linear-recurrence and hybrid models.
-
A Training-Memory Regression in MLA Sequence Parallelism: Why Megatron-Core Forbids Absorption, and LAGA -- a Communication-Efficient Fix
LAGA replaces MLA training's per-head K/V all-to-all with a latent all-gather and local up-projection, matching explicit-form memory while cutting collective communication ~1.98x.
-
Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videos
AV-Flamingo is an open audio-visual language model that outperforms similarly sized open models on many long-video understanding benchmarks, using a new 7M-question dataset and timestamped reasoning.
-
DiTango: Cost-Effective Parallel Diffusion Generation with Selective Attention State Reuse
Selective reuse of composed attention states across denoising steps lets DiTango skip both remote KV communication and attention compute for low-contribution sequence partitions, cutting multi-GPU diffusion latency by...
-
Wan-Dancer: A Hierarchical Framework for Minute-scale Coherent Music-to-Dance Generation
A hierarchical global-keyframe then local-refinement diffusion pipeline produces stable 720p/30fps music-to-dance videos longer than one minute across five genres.
-
LSRM: High-Fidelity Object-Centric Reconstruction via Scaled Context Windows
Scaling transformer context with sparse attention and 3D-aware block routing improves feed-forward 3D reconstruction and inverse rendering, closing much of the quality gap with dense-view optimization.
-
Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking
UPipe chunks attention by head so QKV and all-to-all buffers scale with a small tunable chunk size rather than head count, enabling 5M-token Llama3-8B training on one 8×H100 node with throughput close to Ulysses.
-
SwiftFusion: Scalable Sequence Parallelism for Distributed Inference of Diffusion Transformers on GPUs
StreamFusion places Ulysses-style all-to-all communication between GPU machines and ring-style communication inside machines, overlaps the inter-machine transfers with attention computation, and reports 1.35x average ...
-
HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelism
HelixPipe repartitions transformer layers so attention runs in parallel across pipeline stages, cutting idle time and memory imbalance, and reports a 26% speedup for a 7B model on 128k-length sequences.
-
Arctic Long Sequence Training: Scalable And Efficient Training For Multi-Million Token Sequences
ALST combines tiled loss and MLP compute, Ulysses-style sequence parallelism for Hugging Face GQA and MQA models, and activation-checkpoint CPU offload to train Llama-8B at up to 500K tokens on one H100 and 15M tokens...
-
SRDiffusion: Accelerate Video Diffusion Inference via Sketching-Rendering Cooperation
SRDiffusion accelerates video diffusion by switching from a large model to a smaller sibling model after early high-noise steps, using an adaptive threshold for the switch.
-
Domain Adaptation of Foundation LLMs for e-Commerce
Continued pretraining Llama 3.1 on 1 trillion e-commerce tokens produces e-Llama 8B/70B models that improve in-house e-commerce benchmarks by roughly 25-30% while retaining most general-domain accuracy.
-
Visko Orbis 1.0: A Live Model for Real-Time Interactive Long Video Generation
A chunk-wise streaming video model with bounded multi-scale memory and streaming 4K upscaling reports real-time interactive long-form generation and top Arena preference/stability scores.
-
MoWorld: A Flash World Model
MoWorld achieves up to 50 FPS real-time interactive world simulation on NPUs by combining a 3D-native data engine, curriculum cross-frame pretraining, autoregressive distillation, and mixed-precision parallel inference.
-
Seeing Voices: Generating A-Roll Video from Audio with Mirage
Mirage generates photorealistic A-roll videos of people speaking directly from audio, using only joint self-attention over audio, text, and video tokens.
-
SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformers
An audio-conditioned video diffusion transformer that animates portraits from image, video, text, and audio inputs with a sliding-window fusion for long videos.
-
Wan-S2V: Audio-Driven Cinematic Video Generation
Wan-S2V is an audio-driven video generator built on Wan, claiming better cinematic character animation than prior systems, though the evaluation is limited.
-
TokenRing: An Efficient Parallelism Framework for Infinite-Context LLMs via Bidirectional Communication
TokenRing claims faster long-sequence attention on multiple GPUs by overlapping forward query transfers with backward partial-output transfers, supported by one 4-GPU profiling comparison.
-
Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey
A comprehensive survey that frames multimodal understanding and generation as next token prediction and proposes a five-part taxonomy.
-
Efficient Diffusion Models: A Survey
The paper organizes research on efficient diffusion models into a taxonomy spanning algorithms, systems, and frameworks, and provides a curated reference list.
Discussion (0). Continue with ORCID to comment.