Expert-parallel scaling leaves per-expert routing imbalance flat; mock-token benchmarks overestimate real-text imbalance and fake a batch-size trend; architectures split into data-resilient (MHA, Mamba-2) and persistently concentrated (MLA, GDN) classes.
Demystifying the communication characteristics for distributed transformer models
2 Pith papers cite this work. Polarity classification is still indexing.
2
Pith papers citing it
citation-role summary
background 1
citation-polarity summary
fields
cs.DC 2years
2026 2roles
background 1polarities
background 1representative citing papers
TACO compresses tensor-parallel intermediate tensors with an adaptive FP8 scheme and fused kernels, yielding up to 1.87X throughput gains on GPT and Qwen models with near-lossless accuracy.
citing papers explorer
-
Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory
Expert-parallel scaling leaves per-expert routing imbalance flat; mock-token benchmarks overestimate real-text imbalance and fake a batch-size trend; architectures split into data-resilient (MHA, Mamba-2) and persistently concentrated (MLA, GDN) classes.
-
TACO: Efficient Communication Compression of Intermediate Tensors for Scalable Tensor-Parallel LLM Training
TACO compresses tensor-parallel intermediate tensors with an adaptive FP8 scheme and fused kernels, yielding up to 1.87X throughput gains on GPT and Qwen models with near-lossless accuracy.