A Training-Memory Regression in MLA Sequence Parallelism: Why Megatron-Core Forbids Absorption, and LAGA -- a Communication-Efficient Fix

Changzheng Ma

REVIEW 4 major objections 5 minor 15 references

Megatron-Core's ban on MLA's 'absorbed' training form is justified: its intermediates inflate activation memory 20–34% (up to 9.2 GB), and LAGA gets the same 1.98× communication cut with the explicit form's memory and numerics.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 17:24 UTC pith:YKUE67AD

load-bearing objection LAGA's communication optimization is real and clean; the memory-trap claim doesn't match its own data, so treat C1 as unproven. the 4 major comments →

arxiv 2607.17644 v1 pith:YKUE67AD submitted 2026-07-20 cs.DC

A Training-Memory Regression in MLA Sequence Parallelism: Why Megatron-Core Forbids Absorption, and LAGA -- a Communication-Efficient Fix

Changzheng Ma This is my paper

classification cs.DC

keywords multi-head latent attentionsequence parallelismactivation memorymemory traplatent all-gathercollective communicationMegatron-CoreDeepSeek-V3

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Multi-head Latent Attention (MLA), the low-rank attention used in the DeepSeek-V2/V3 family, ships in Megatron-Core in two forms: an explicit form used for training, and an 'absorbed' form that cuts collective communication by gathering only the compressed latent — yet the absorbed form is hard-asserted out of training with no documented reason. This paper establishes that the restriction is well-founded and quantifies it: ported to training, the absorbed form is a memory trap, because its intermediates (q_absorbed and the post-attention latent accumulator) live in n_h×d_kv = 128×512 dimensions per token — larger than the per-head K/V (128×256) they replace — inflating activation memory by 20–34%, up to 9.2 GB at seq=16384/SP=8 (19.2 GB under a fused kernel), enough to change device fit. The paper then supplies the missing training path: LAGA (Latent All-Gather Attention) keeps the absorbed form's latent-gather communication but reconstructs per-head K/V locally at head-shard granularity, cutting collective communication 1.98× while matching the explicit form's activation memory within 0.5% and its numerics to floating-point precision (bit-identical at SP=1). This matters because long-context MLA training currently has no low-communication sequence-parallelism path; LAGA is fastest precisely in the cross-node, long-sequence regime such training targets (1.07–1.24× under fused attention), and the paper's general lesson — inference KV-cache tricks can regress training memory — extends beyond MLA.

Core claim

On the paper's own terms: the absorb trick is an inference KV-cache optimization — attention runs against a shared d_kv-dimensional latent, and only that small latent crosses the collective (a ~32× cache saving at decode). In training there is no cache; every forward recomputes, and the same reformulation replaces per-head K/V (n_h×256 per token) with strictly larger latent-space intermediates (n_h×512), so the 'efficient' path is the memory-inefficient one: a measured 20–34% activation-memory inflation, up to 9.2 GB at DeepSeek-V3 scale (19.2 GB under fused attention), rising with sequence length and head count. The absorb reformulation then turns out to be unnecessary: LAGA all-gathers the

What carries the argument

The load-bearing comparison is between the two forms' per-token activation footprint in a training forward: the explicit form materializes per-head K/V in the head dimension (n_h × (d_nope+d_v) = 128×256 per token), while the absorbed form works in the shared latent dimension (n_h × d_kv = 128×512 per token). Because d_kv exceeds d_head, each absorbed intermediate — q_absorbed = W_kv_b⊤ q_nope and the post-attention latent accumulator — costs roughly double the memory of the per-head K/V it replaces, and because training recomputes rather than caches, that is pure overhead. LAGA's mechanism pairs a cheaper collective (all-gather the d_kv-dimensional latent plus the small RoPE band, instead o

Load-bearing premise

The central claim — that the absorbed form's n_h×d_kv intermediates cause the measured 20–34% memory inflation (9.2 GB eager, 19.2 GB fused) — rests on an attribution the paper cannot yet support from its stated numbers: no batch size or allocation model is given, the reported gaps grow faster with sequence length than the per-token footprint predicts, and the memory measurements are single-layer; if this attribution fails, the explanation of Megatron-Core's assert and the de

What would settle it

Measure the absorbed-minus-explicit peak-memory gap at S=4K/8K/16K with a fused attention kernel, a stated batch size, and a stated layer count: the paper's mechanism predicts a gap growing with sequence length on a slope matching n_h × (d_kv − (d_nope+d_v)) bytes per token, so a flat gap, or one fully explained by the score matrix, would falsify the trap attribution. The sharper test is parametric: shrink d_kv below d_head and the paper's inequality predicts the sign flips — absorbed becomes memory-cheaper than explicit; if it does not, the mechanism is wrong. The 1.98× communication ratio an

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Long-context MLA training can cut collective communication by a stable 1.98× at DeepSeek-V3 scale (n_h=128, across SP=4–16 and sequence lengths 4K–16K) with no activation-memory penalty; the ratio strengthens as head count grows.
Megatron-Core's hard assert against absorbed-MLA training is a guard, not a missing feature: the absorbed form's latent-dimension intermediates inflate peak activation memory 20–34%, enough to change device fit — but the guard leaves training with no low-communication MLA path, which is the gap LAGA fills.
LAGA and the explicit path compute the same function: bit-identical output and parameter gradients at SP=1, agreement to within ~1e-3 at SP=2–8 after gradient aggregation, and identical loss endpoints in a multi-step training run.
Under a fused attention kernel, LAGA beats the explicit path at every tested sequence length cross-node (1.07–1.24×, with the advantage growing with length) and at production head count single-node (1.04–1.06×) — the long-context regime MLA is deployed for.
The paper's generalizable lesson: an optimization designed for the inference KV cache can become a memory regression in training, so inference tricks ported to training should be checked for activation footprint, not just communication.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The latent-gather plus local-up-projection template should transfer to any attention variant with a shared low-rank KV latent, and the sign of the memory trade flips with the ratio: in a design with d_kv < d_head, the absorbed form would be memory-cheaper in training, so the justification for the assert is parameter-regime-dependent, not universal.
A fused up-projection-to-attention kernel — listed in the paper as future work — is the natural completion of LAGA and would likely close the residual compute-bound regime where the explicit path still wins short-sequence, single-node throughput.
The paper's incidental finding that top-k MoE routing turns sub-ULP floating-point noise into cliff-shaped, same-seed divergence on NPU hardware is a testable reproducibility claim of its own: if it reproduces on any backend with nondeterministic reduction order, MoE checkpoint-resume checks should rely on loss range rather than bit-equality.
For sequence lengths that push SP degree beyond head count, the latent-gather idea extends naturally to ring-style blockwise parallelism — each rank gathering only the latent chunks it needs — a direction the paper notes but does not build.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

LAGA's communication optimization is real and clean; the memory-trap claim doesn't match its own data, so treat C1 as unproven.

read the letter

Two things to know. LAGA — all-gather the latent, reconstruct per-head K/V locally — is a real, clean idea. The 1.98x communication reduction is a derived constant from the MLA dimensions, and the equivalence to explicit attention is by construction (same matmul, reordered views). If there is a production takeaway here, that's it.

The other headline — that the absorbed MLA form is a memory trap with 9–19 GB inflation — does not hold up against the paper's own tables. The mechanism they propose (n_h·d_kv per-token intermediates) predicts a gap that grows linearly in sequence length. Table 2 shows the gap growing about 4x per sequence doubling (0.6→2.4→9.2 GB at SP=8), which is quadratic, and the absolute magnitude at S=16384 is roughly 3x the back-of-envelope from their stated dims. No memory model, batch size, or allocation script is provided to bridge that. So the explanation for Megatron-Core's assert is a hypothesis, not a measurement. The authors' own quote from MCore — 'not doing true absorption' — suggests an incomplete implementation rather than a deliberate memory guard.

LAGA doesn't need the memory-trap claim; it stands alone as a communication optimization. The throughput numbers are modest (1.04–1.24x) and only on Ascend, with an A100 cross-check for the crossover shape. No code is shipped, which matters for a measurement-heavy argument. If a referee can pin down the memory model, this could become a solid systems paper. Without that, C1 is unsupported and the motivation weakens to 'give me a faster way to do what you already do' — still worth having.

I'd send this to peer review, with the memory analysis as the make-or-break issue. For a reading group, it's a maybe: the LAGA derivation is a good exercise, but the memory controversy needs external context to judge.

Referee Report

4 major / 5 minor

Summary. The paper addresses a puzzle in Megatron-Core's MLA implementation: the absorbed (latent-only) form of multi-head latent attention is fully implemented but hard-asserted out of training. The authors claim that porting the absorbed form to training creates a memory trap, because intermediates such as q_absorbed and the post-attention latent accumulator occupy n_h * d_kv dimensions per token (128 * 512 at DeepSeek-V3 scale), inflating activation memory by 20–34% and by up to 9.2 GB (19.2 GB under a fused kernel). They attribute Megatron-Core's training-time assert to this regression. They then propose LAGA (Latent All-Gather Attention), which keeps the absorbed form's low communication by all-gathering the latent but reconstructs per-head K/V locally, avoiding the latent-dimension intermediates. The paper reports communication reduction of 1.98x, memory matching the explicit form, bit-identical or floating-point-equivalent numerics, and throughput improvements on 8x Ascend 910B with cross-validation on A100.

Significance. If the central memory-trap claim is correct, the paper identifies a real and non-obvious pitfall in MLA sequence parallelism and offers a simple, sound fix: LAGA is mathematically equivalent to the explicit form by construction, and its communication reduction is structural. The equivalence proof in §5 is clean, and the cross-hardware checks are useful. However, the memory-trap measurement is not reproducible from the stated per-token costs, and the paper's own tables contradict the claimed linear-in-sequence-length scaling. Because the memory-trap claim is the load-bearing explanation for Megatron-Core's assert and the motivation for LAGA, these quantitative inconsistencies must be resolved before the paper's central empirical conclusion can be accepted. The LAGA communication/equivalence contributions are separately defensible and would survive a corrected memory analysis.

major comments (4)

[§3.2, §4.3, Table 2] The claimed linear-in-S validation is contradicted by the paper's own data. For SP=8, Table 2 gives B2−B1 = 0.576 GB at S=4096, 2.365 GB at S=8192, and 9.248 GB at S=16384, i.e., about a 4x increase per doubling of S. This is quadratic growth, not the linear growth predicted by the n_h*d_kv intermediate mechanism. The fused numbers in Table 4 behave similarly (1.591, 5.327, 19.237 GB). If the mechanism is q_absorbed and the latent accumulator, both scale linearly in S; the observed quadratic scaling requires an additional explanation. Please provide a memory model, batch size, allocation/checkpointing details, and a tensor-level breakdown so the measured peaks are derivable from the stated costs.
[§3.2, §4.6, Table 4] The +19.2 GB fused-kernel inflation at S=16384 is not explainable by the stated intermediates. At DeepSeek-V3 dimensions, q_absorbed and the post-attention latent accumulator are each S * n_h * d_kv * 2 bytes ≈ 2 GiB at S=16384, so even both together are about 4 GiB, roughly 5x smaller than the reported 19.2 GB gap. Either additional tensors (e.g., un-absorbed V, saved activations for backward, allocator copies, or a materialized score matrix in the absorbed fused path) dominate, or the mechanism description is incomplete. Without a tensor-level breakdown and the allocation script, the headline +9.2/+19.2 GB numbers are not reproducible.
[§4.2, Table 1] The communication table is not reproducible from the stated per-token costs. Using the paper's dimensions and a per-rank send-volume convention, at S=16384, SP=8, the explicit form sends approximately S*n_h*(d_q+d_k+d_v)/P + S*n_h*d_v/P ≈ 302 MB, while LAGA sends approximately S*n_h*d_q/P + (P−1)/P * S*(d_kv+d_rope) + S*n_h*d_v/P ≈ 184 MB, giving a ratio of about 1.64x, not 1.98x. The table reports 587.2 MB and 296.0 MB. If the table counts both send and receive directions, or total group traffic, or includes an additional collective, please state the convention explicitly and show the formula. This matters because the 1.98x reduction is a headline contribution.
[§1, §3.2] The causal claim that Megatron-Core's assert is 'an undocumented guard against this regression' is not established. The manuscript itself quotes Megatron-Core's comment 'not doing true absorption. We will add this support at a later time,' which suggests the absorbed training path may be incomplete rather than deliberately disabled for memory reasons. The memory measurement can show that the absorbed form would be harmful if used in training, but it cannot by itself establish the historical intent behind the assert. Please soften the causal wording or provide external evidence (e.g., issue tracker, commit history, developer statement).

minor comments (5)

[Abstract, §4.5, Prop. 1] The abstract says 'bit-identical to explicit at SP=1', but Prop. 1 says 'up to floating-point reordering' and §4.5 reports some gradients (h, Wkv_a, LN) at max |Δ| ≤ 2e-5, not exactly 0. Please state precisely which tensors are bit-identical and which are merely close.
[Table 3] The last column 'LAGA/B1' contains multiple numbers in one cell with no clear mapping to the three sequence lengths. For SP=2, the row has two ratios; for SP=8, three ratios. Please reformat so each ratio is tied to its sequence length.
[Table 2 vs Table 4] Table 2 reports a numeric B2 peak of 36,596 MB at S=16384/SP=8, while Table 4 lists B2 as OOM at the same configuration under eager attention. This is presumably because Table 2 uses a single layer and Table 4 uses a 4-layer stack, but the text should state that distinction explicitly at first use of 'OOM'.
[§4.8, Table 6] The A100 absorbed-inflation values (+8734 MB at S=16384) differ from Ascend (+9247 MB) by about 5.5%. The text calls this 'allocator noise (≤6%)', but allocator noise of this size deserves a sentence of explanation, since the proposed mechanism is deterministic and should not be allocator-dependent at the GB scale unless the allocator behavior is unusual.
[§8] The MoE routing nondeterminism observation is unrelated to LAGA and is reported without an experimental protocol (number of seeds, exact model, routing threshold, measurement method). If retained, it should be moved to an appendix with full details or clearly labeled as preliminary and out of scope.

Circularity Check

0 steps flagged

No significant circularity: the paper derives LAGA as a reordering of the same linear maps as the explicit baseline, makes no fitted-parameter prediction, and does not rely on self-citations.

full rationale

No step in the paper's chain of derivation reduces to its own inputs or to the author's prior work. The core correctness claim (Section 5) is an explicit proof of identity: LAGA's step 4 computes "the H_r slice of B1's W_kv_b(latent) — same parameters, same matmul, reordered views" (Section 3.3), and the bit-identical SP=1 result is a code-level equivalence, not a fitted prediction. The memory-trap claim (C1) is presented as a measurement plus a structural comparison (n_h*d_kv per-token intermediates vs per-head K/V), not as a parameter fit, and the paper validates it on two hardware stacks; the cited MCore assert is quoted verbatim from the library. There are no load-bearing self-citations: the reference list contains no prior works by the author, and the only internal claims are supported by the paper's own experiments. The LAGA communication reduction is a closed-form byte count (B1 K/V all-to-all n_h(d_nope+d_v) vs latent d_kv+d_rope), matching the measured 1.98x. One inconsistency should be noted as a correctness/reproducibility risk rather than circularity: the paper says the trap was "validated on two axes (linear in seq and n_h)" but Table 2's B2−B1 gaps at SP=8 (576→2365→9248 MB) grow about 4x when S doubles, i.e., ~quadratically, not linearly; this undermines the stated mechanism's quantitative support but is not a definitional or self-citational circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The central claims rely on standard MLA math and Ulysses SP semantics, plus the DeepSeek-V3 config (d_kv=512, d_head=128). No free parameters are fitted to data. The main unstated premise is that the Ascend fused attention kernel and 4-layer prototype represent production behavior.

axioms (4)

domain assumption Under Ulysses-style SP, each rank must own at least one per-head KV shard to scatter; MLA's shared latent (P1) violates this.
§2.2/3.1; standard Ulysses assumption, false for MLA, motivates LAGA's design.
domain assumption MLA's kv_b up-projection is linear and shared across heads; the latent LN(latent) is common to all heads.
§2.1 equations; used in Propositions 1 and 2 equivalence proof.
domain assumption For DeepSeek-V3, d_kv=512 > d_head=128, so n_h*d_kv > n_h*(d_nope+d_v).
§2.1 config; basis of the memory-trap mechanism.
domain assumption The fused npu attention kernel's measured behavior is representative of production fused attention.
§4.6; not independently verified, affects throughput and fused-memory claims.

pith-pipeline@v1.3.0-alltime-deepseek · 13534 in / 17819 out tokens · 160404 ms · 2026-08-01T17:24:12.665887+00:00 · methodology

0 comments

read the original abstract

Multi-head Latent Attention (MLA) ships two implementations in Megatron-Core: an explicit form used for training and an absorbed form -- which slashes collective communication by gathering only the compressed latent -- that is fully implemented but hard-asserted out of training (the forward opens with "assert not (self.training and self.cache_mla_latents)"), allowed only in inference decode. The library documents no reason. We show the restriction is well-founded and quantify why: ported to training, the absorbed form is a memory trap -- its intermediates live in n_h x d_kv dimensions per token, larger than the per-head K/V they replace -- inflating activation memory by 20-34%, up to 9.2 GB at DeepSeek-V3 scale (n_h=128, seq=16384, SP=8, eager kernel; the gap widens to 19.2 GB under a fused kernel), enough to change device-fit. This measurement, validated on two axes (linear in seq and n_h) and cross-verified on NVIDIA A100, explains the otherwise-undocumented restriction and leaves practitioners with no low-communication MLA training path. We then provide one. LAGA (Latent All-Gather Attention) keeps the absorbed form's latent-gather communication but rejects the absorb reformulation, instead reconstructing per-head K/V locally from the gathered latent. On 8x Ascend 910B at real DeepSeek-V3 dimensions, LAGA cuts collective communication 1.98x, matches explicit memory within 0.5%, is bit-identical to explicit at SP=1 and equivalent to within 1e-3 at SP=2-8, and under a fused attention kernel improves attention-block throughput 1.04-1.06x single-node and 1.07-1.24x cross-node -- leading at all sequence lengths in the cross-node regime MLA is deployed for.

Figures

Figures reproduced from arXiv: 2607.17644 by Changzheng Ma.

**Figure 1.** Figure 1: Teaser. Left: per-forward communication at DeepSeek-V3 scale (explicit → LAGA/absorbed ≈ 50%, 1.98×). Right: peak activation memory — absorbed +9.2 GB at seq=16384/SP=8 (the trap we measure and explain; +19.2 GB under fused attention without the eager score-matrix confound), LAGA = explicit (the consequent fix). The headline finding is the right panel’s memory trap and its cause; LAGA is the training path … view at source ↗

**Figure 2.** Figure 2: Dataflow. Explicit (B1): Wkv b → a2a(Q, K, V ) → attn → a2a. Absorbed (B2): a2a(Q) + ag(latent) → absorb → latent-attn → un-absorb → a2a. LAGA: a2a(Q) + ag(latent) → local up-proj(K, V ) → attn → a2a. LAGA shares B2’s left and right spine but replaces latent-space attention with head-dim attention on locally reconstructed K/V. 4096 8192 16384 sequence length 1.965 1.970 1.975 1.980 1.985 1.990 1.995 2.000 … view at source ↗

**Figure 3.** Figure 3: Communication reduction is structural and scaledependent. (a) Explicit/LAGA ratio holds ≈ 1.98× across SP and seq. (b) It climbs 1.88 → 1.98× as nh grows 16 → 128 — strengthens at production scale. (short S) LAGA is ≈21% slower. Note LAGA’s total FLOPs equal B1’s (Wkv b is merely redistributed) — the regression is kernel efficiency, not FLOPs: LAGA’s per-head-shard einsums are smaller GEMMs. B2 is consist… view at source ↗

**Figure 5.** Figure 5: Key result. LAGA/Explicit throughput ratio cross-node (SP=16, nh=128). Eager: LAGA loses short (0.73× at S=4096), wins long (+15%). Fused: the short-seq regression vanishes — LAGA wins at all seq lengths (1.07–1.24×), rising to +24%. dients ≤ 2×10−5 (backward accumulation noise, fp32). • SP=2/4/8: equivalent to floating-point precision. After SP-consistent gradient aggregation: output max |∆| ≤ 5.6×10−6 an… view at source ↗

**Figure 6.** Figure 6: Correctness. (a) Convergence to an identical endpoint. (b) |∆loss| vs Explicit: |LAGA − Absorbed| ≈ 0 (LAGA ≡ Absorbed, bit-identical) while |LAGA − Explicit| is the generic fpchaos. tical amounts, confirming LAGA introduces no pathology beyond the generic fp-equivalence class. 4.6 Fused-kernel validation The throughput results above use an eager softmax (scores=QK⊤ → softmax → @V ), which materializes t… view at source ↗

discussion (0)

Reference graph

Works this paper leans on

15 extracted references · 13 linked inside Pith

[1]

Bhatia, N., More, A., Borkar, R., Mitra, T., Matas, R., Zhao, R., Golub, M., Mudigere, D., Pharris, B., and Rouhani, B. D. Helix Parallelism: Rethinking sharding strategies for interactive multi-million-token LLM decoding.arXiv preprint arXiv:2507.07120,

Pith/arXiv arXiv
[5]

and Zhao, S

Fang, J. and Zhao, S. Usp: A unified sequence parallelism approach for long context generative ai.arXiv preprint arXiv:2405.07719,

Pith/arXiv arXiv
[6]

Gu, D., Sun, P., Hu, Q., Huang, T., Chen, X., Xiong, Y ., Wang, G., Chen, Q., Zhao, S., Fang, J., Wen, Y ., Zhang, T., Jin, X., and Liu, X

arXiv:2211.15841. Gu, D., Sun, P., Hu, Q., Huang, T., Chen, X., Xiong, Y ., Wang, G., Chen, Q., Zhao, S., Fang, J., Wen, Y ., Zhang, T., Jin, X., and Liu, X. LoongTrain: Efficient training of long-sequence LLMs with head-context parallelism. arXiv preprint arXiv:2406.18485,

Pith/arXiv arXiv
[7]

Jacobs, S

arXiv:2206.03382. Jacobs, S. A., Tanaka, M., Zhang, C., Zhang, M., Song, S. L., Rajbhandari, S., and He, Y . DeepSpeed-Ulysses: System optimizations for enabling training of extreme long sequence transformer models.arXiv preprint arXiv:2309.14509,

Pith/arXiv arXiv
[8]

Li, J., Jiang, Y ., Zhu, Y ., Wang, C., and Xu, H

arXiv:2205.05198. Li, J., Jiang, Y ., Zhu, Y ., Wang, C., and Xu, H. Acceler- ating distributed MoE training and inference with Lina. InUSENIX Annual Technical Conference (ATC),

Pith/arXiv arXiv
[9]

Liu, H., Zaharia, M., and Abbeel, P

arXiv:2210.17223. Liu, H., Zaharia, M., and Abbeel, P. Ring attention with blockwise transformers for near-infinite context. InInter- national Conference on Learning Representations (ICLR),

Pith/arXiv arXiv
[10]

Liu, S., Peng, H., Zhang, Z., Chen, Z., and Guo, Y

arXiv:2310.01889. Liu, S., Peng, H., Zhang, Z., Chen, Z., and Guo, Y . Multi-head low-rank attention. InInternational Confer- ence on Learning Representations (ICLR),

Pith/arXiv arXiv
[13]

Sun, W., Lan, D., Zhong, Y ., Qu, X., and Cheng, Y

Reference MLA im- plementation: multi_latent_attention.py un- dermegatron/core/transformer/. Sun, W., Lan, D., Zhong, Y ., Qu, X., and Cheng, Y . LASP- 2: Rethinking sequence parallelism for linear attention and its hybrid.arXiv preprint arXiv:2502.07563, 2025a. Linear attention, not softmax MLA – orthogonal to our work. Sun, W., Qin, Z., Li, D., Shen, X....

Pith/arXiv arXiv 2024
[14]

Inference/decode only

arXiv:2508.15881. Inference/decode only. Zhang, P. and Su, L. Memory analysis on the training course of DeepSeek models.arXiv preprint arXiv:2502.07846,

Pith/arXiv arXiv
[15]

DeepEP: an efficient expert- parallel communication library

Zhao, C., Zhou, S., Zhang, L., Deng, C., Xu, Z., Liu, Y ., Yu, K., Li, J., and Zhao, L. DeepEP: an efficient expert- parallel communication library. https://github. com/deepseek-ai/DeepEP, 2025

2025
[2021]

arXiv:2104.04473. NVIDIA. Megatron-Core. https://github.com/ NVIDIA/Megatron-LM,

Pith/arXiv arXiv
[2023]

DeepSeek-V2: A strong, economical, and ef- ficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434,

DeepSeek-AI. DeepSeek-V2: A strong, economical, and ef- ficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434,

Pith/arXiv arXiv
[2024]

DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437,

DeepSeek-AI. DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437,

Pith/arXiv arXiv
[2025]

Brandon, W., Nrusimha, A., Qian, K., Ankner, Z., Jin, T., Song, Z., and Ragan-Kelley, J

KV-parallelism for long-context decoding; inference. Brandon, W., Nrusimha, A., Qian, K., Ankner, Z., Jin, T., Song, Z., and Ragan-Kelley, J. Striped attention: Faster ring attention for causal transformers.arXiv preprint arXiv:2311.09431,

Pith/arXiv arXiv
[2026]

arXiv:2603.02188

Poster. arXiv:2603.02188. Dual-path attention enabling 4-way tensor-parallel decoding; inference. Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Pat- wary, M., Korthikanti, V . A., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., and Za- haria, M. Efficient large-scale language model training on GPU clusters using Mega...

arXiv

[1] [1]

Bhatia, N., More, A., Borkar, R., Mitra, T., Matas, R., Zhao, R., Golub, M., Mudigere, D., Pharris, B., and Rouhani, B. D. Helix Parallelism: Rethinking sharding strategies for interactive multi-million-token LLM decoding.arXiv preprint arXiv:2507.07120,

Pith/arXiv arXiv

[2] [5]

and Zhao, S

Fang, J. and Zhao, S. Usp: A unified sequence parallelism approach for long context generative ai.arXiv preprint arXiv:2405.07719,

Pith/arXiv arXiv

[3] [6]

Gu, D., Sun, P., Hu, Q., Huang, T., Chen, X., Xiong, Y ., Wang, G., Chen, Q., Zhao, S., Fang, J., Wen, Y ., Zhang, T., Jin, X., and Liu, X

arXiv:2211.15841. Gu, D., Sun, P., Hu, Q., Huang, T., Chen, X., Xiong, Y ., Wang, G., Chen, Q., Zhao, S., Fang, J., Wen, Y ., Zhang, T., Jin, X., and Liu, X. LoongTrain: Efficient training of long-sequence LLMs with head-context parallelism. arXiv preprint arXiv:2406.18485,

Pith/arXiv arXiv

[4] [7]

Jacobs, S

arXiv:2206.03382. Jacobs, S. A., Tanaka, M., Zhang, C., Zhang, M., Song, S. L., Rajbhandari, S., and He, Y . DeepSpeed-Ulysses: System optimizations for enabling training of extreme long sequence transformer models.arXiv preprint arXiv:2309.14509,

Pith/arXiv arXiv

[5] [8]

Li, J., Jiang, Y ., Zhu, Y ., Wang, C., and Xu, H

arXiv:2205.05198. Li, J., Jiang, Y ., Zhu, Y ., Wang, C., and Xu, H. Acceler- ating distributed MoE training and inference with Lina. InUSENIX Annual Technical Conference (ATC),

Pith/arXiv arXiv

[6] [9]

Liu, H., Zaharia, M., and Abbeel, P

arXiv:2210.17223. Liu, H., Zaharia, M., and Abbeel, P. Ring attention with blockwise transformers for near-infinite context. InInter- national Conference on Learning Representations (ICLR),

Pith/arXiv arXiv

[7] [10]

Liu, S., Peng, H., Zhang, Z., Chen, Z., and Guo, Y

arXiv:2310.01889. Liu, S., Peng, H., Zhang, Z., Chen, Z., and Guo, Y . Multi-head low-rank attention. InInternational Confer- ence on Learning Representations (ICLR),

Pith/arXiv arXiv

[8] [13]

Sun, W., Lan, D., Zhong, Y ., Qu, X., and Cheng, Y

Reference MLA im- plementation: multi_latent_attention.py un- dermegatron/core/transformer/. Sun, W., Lan, D., Zhong, Y ., Qu, X., and Cheng, Y . LASP- 2: Rethinking sequence parallelism for linear attention and its hybrid.arXiv preprint arXiv:2502.07563, 2025a. Linear attention, not softmax MLA – orthogonal to our work. Sun, W., Qin, Z., Li, D., Shen, X....

Pith/arXiv arXiv 2024

[9] [14]

Inference/decode only

arXiv:2508.15881. Inference/decode only. Zhang, P. and Su, L. Memory analysis on the training course of DeepSeek models.arXiv preprint arXiv:2502.07846,

Pith/arXiv arXiv

[10] [15]

DeepEP: an efficient expert- parallel communication library

Zhao, C., Zhou, S., Zhang, L., Deng, C., Xu, Z., Liu, Y ., Yu, K., Li, J., and Zhao, L. DeepEP: an efficient expert- parallel communication library. https://github. com/deepseek-ai/DeepEP, 2025

2025

[11] [2021]

arXiv:2104.04473. NVIDIA. Megatron-Core. https://github.com/ NVIDIA/Megatron-LM,

Pith/arXiv arXiv

[12] [2023]

DeepSeek-V2: A strong, economical, and ef- ficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434,

DeepSeek-AI. DeepSeek-V2: A strong, economical, and ef- ficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434,

Pith/arXiv arXiv

[13] [2024]

DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437,

DeepSeek-AI. DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437,

Pith/arXiv arXiv

[14] [2025]

Brandon, W., Nrusimha, A., Qian, K., Ankner, Z., Jin, T., Song, Z., and Ragan-Kelley, J

KV-parallelism for long-context decoding; inference. Brandon, W., Nrusimha, A., Qian, K., Ankner, Z., Jin, T., Song, Z., and Ragan-Kelley, J. Striped attention: Faster ring attention for causal transformers.arXiv preprint arXiv:2311.09431,

Pith/arXiv arXiv

[15] [2026]

arXiv:2603.02188

Poster. arXiv:2603.02188. Dual-path attention enabling 4-way tensor-parallel decoding; inference. Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Pat- wary, M., Korthikanti, V . A., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., and Za- haria, M. Efficient large-scale language model training on GPU clusters using Mega...

arXiv