REVIEW 3 major objections 3 minor 6 references
Memory Analysis on the Training Course of DeepSeek Models
T0 review · 3 major / 3 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read The paper derives closed-form per-GPU memory formulas for training DeepSeek-v3-scale MoE models, reducing the answer to layer counts and parallelism choices.
desk verdict Useful reference math for DeepSeek-v3 training memory, but the activation table misses the pipeline microbatch multiplier and is not a per-device number as claimed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is a per-stage arithmetic decomposition built from two counting rules. First, static parameters are counted per expert-parallel rank: with 256 routed experts and 1 shared expert per MoE layer, expert parallelism 8 gives 32 routed plus 1 shared expert per layer per rank, and a four-layer pipeline stage therefore holds 132 expert stacks of three matrices each ($h\times h_E$, $h\times h_E$, $h_E\times h$), yielding 5.813 billion expert parameters per device. Second, activation memory is expressed as a symbolic sum over the MLA and MoE dataflow, with the MoE term relying on the balanced-routing identity $E_{tok}=bsN_r/N$ for the average number of tokens per expert. These two rules, together with the ZeRO sharding denominators ($DP=32$ for dense parts, $EDP=8$ for experts), convert all memory categories into closed-form expressions.
What would settle it
Run one training step of the DeepSeek-v3 architecture under the paper's PP16, TP2, EP8, SP2, full-recomputation configuration with $b=1$ and $s=4096$, record peak per-GPU memory with a profiler, and compare the activation portion against $8bsh+8bsN_r$ after subtracting measured parameter, gradient, optimizer, fragmentation, and buffer usage; a systematic excess on hot experts would refute the balanced-routing formula.
Extended reading notes
Core claim
On its own terms, the paper's central claim is a decomposition identity. In the PP16@TP2@EP8 configuration, each pipeline stage holds four transformer layers, and each GPU on a stage holds 132 experts (128 routed experts and 4 shared experts across 4 layers), giving a static weight footprint of 6250364928 parameters, or 11.64 GB in BF16, with 10.84 GB of that coming from MoE experts. Applying ZeRO with data-parallel size 32 for dense parts and expert-data-parallel size 8 for experts yields 1.38 GB for parameters, 2.76 GB for gradients, and 5.52 GB for optimizer states. For activations, the paper derives per-device formulas for a four-layer stage: without recomputation the MLA term is $10bsh + 8bs(d_{cq}+d_c) + 16bsd_hn_h + 8bsd_{hr}n_h + 10bn_hs^2$ and the MoE term is $20bsh + 16bsN + 8bsN_r + 4bs(N_r/N)(96h+256h_E) + 32bsh_E$; with full recomputation the total collapses to $8bsh + 8bsN_r$. The claim is that these expressions predict the device memory peak once fragmentation and communication buffers are added.
Load-bearing premise
The activation estimate assumes routed tokens spread evenly across experts, so each expert's share is exactly the average $bsN_r/N$; if real routing is imbalanced, hot experts can need more activation memory than the formula credits them.
Editorial extensions
If this is right
- Under the paper's case configuration, static expert parameters dominate device memory (10.84 GB of 11.64 GB), so increasing expert parallelism is the most direct lever for shrinking per-GPU weight memory.
- Full activation recomputation lowers activation memory from a four-line formula involving several MLA constants to the simple expression $8bsh+8bsN_r$, meaning activation cost scales linearly with micro-batch size, sequence length, and the number of routed experts per token.
- The ZeRO os+g+params scenario puts parameters, gradients, and optimizer states at 9.66 GB per GPU for this configuration, implying that the remaining memory budget on a modern GPU is set by activations, fragmentation, and communication buffers.
- The same counting method can be applied to any MoE transformer with known layer counts and an MLA-style attention, giving a pre-training estimate without running a profiler.
Reading between the lines
- Extending the paper's arithmetic to other batch sizes is immediate: since the formulas are linear in $b$, doubling the micro-batch doubles the activation term, which lets a planner tune batch size against the measured per-GPU HBM before launch.
- The balanced-routing assumption is the natural stress point; real token-to-expert routing is often skewed, so a testable extension would be to walk the formulas with empirically observed per-expert token counts and compare the resulting activation estimate with the $E_{tok}$-based one.
- Because the paper leaves FP8 training and quantization scaling factors out of scope, applying the same decomposition to mixed-precision training would require adding per-tensor scaling memory and is a direct follow-up the authors do not perform.
- The fragmentation range (5-30%) and communication-buffer range (0.8-2 GB) quoted at the end suggest that the analytic formula is a lower-bound-style planning tool rather than a precise peak-memory guarantee for a specific runtime allocator.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript is a theoretical memory-usage analysis for training DeepSeek-v2/v3-style mixture-of-experts models. It first counts parameters per transformer layer and per pipeline stage, then derives per-device static parameter memory for a PP16@TP2@EP8 configuration, applies DeepSpeed ZeRO stages to optimizer states, gradients, and parameters, and finally gives closed-form activation-memory expressions for MLA and MoE blocks with and without full activation recomputation. The paper explicitly disclaims that the analyzed parallel configurations are not DeepSeek's official training setup. The headline numbers are about 6.25B parameters and 11.64 GiB static memory per device, the ZeRO totals in Table 8 (e.g., 9.66 GiB for os+g+params), and the activation totals in Table 10.
Significance. The parameter-level arithmetic and the mixed DP/EDP ZeRO formulas check out internally and are reproducible from public architectural constants, which is a genuine strength: a reader can recompute every static-memory number from Tables 1 and 2. The paper also usefully separates MoE-expert parameters from non-MoE parameters in the ZeRO calculation, which many simplified memory models do not. The utility of the activation half is currently limited by a first-order omission: the formulas in Section 5 are per-micro-batch per-stage quantities, while Table 10 presents them as per-device values under pipeline parallelism, and no multiplication by the number of in-flight micro-batches appears. A secondary gap is that the worst-case stage is selected by parameter count alone even though activation memory is not monotone in parameter count. If these issues are corrected, the paper would be a valuable reference for MoE training memory planning; as written, the activation-memory claims are not yet established.
major comments (3)
- [Section 5.3, Table 10] The activation formulas are written for one micro-batch (b is the micro-batch size, and no variable appears for the number of micro-batches in flight), but Table 10 labels the result as 'activation memory per device' under PP16. In a 1F1B pipeline schedule with PP16 and M micro-batches, a stage must retain activations for each micro-batch that has completed its forward pass but not its backward pass; for the first stage this can be all M micro-batches. The device-level total is therefore approximately M times the listed formulas, not the formulas themselves. For example, with b=1, s=4096 and no recomputation, the Table 10 total is about 24.7 GiB per micro-batch, so with M=16 the device would need on the order of 395 GiB, far beyond the paper's implied budget. The full-recomputation total 8bsh+8bsNr similarly carries an implicit M. The authors should either multiply by the number of in-flight micro-batches and state the pipeline schedule, or explicitly state that Table 10 is a per-micro-batch quantity and provide the missing M factor.
- [Section 2.2, Table 4] The paper selects Stages 1-14 as the worst case based solely on parameter count and then analyzes only MLA and MoE layers, but activation memory is not monotone in parameter count. Stage 0 contains layers 0-3, including three dense FFN layers with intermediate size hF=18432 and the embedding matrix; Section 5 provides no activation formulas for these dense layers. Under the same activation accounting as the shared-expert term in Section 5.2, one dense FFN layer contributes roughly 8bshF bytes for its intermediate activations, which at b=1 and s=4096 is about 0.60 GiB per micro-batch and is comparable to the per-layer MoE expert terms. The claim that Stages 1-14 determine the global per-device memory peak is therefore not justified unless the authors either extend the activation analysis to Stage 0 or prove that its activation memory is bounded by that of the MoE stages.
- [Section 5.2] The MoE activation formula uses Etoken = bs*Nr/N, the average number of tokens per expert under perfectly balanced routing. The paper does not state whether per-expert activation buffers are allocated dynamically according to the actual number of tokens routed to each expert or statically to a maximum capacity. With dynamic allocation, the total over all experts on a rank depends on the total number of expert-tokens routed to that rank, so per-expert hot spots are less of an issue but per-rank token variance still matters; with static allocation, a hot expert can cause per-expert memory to exceed the average-based estimate by an amount the paper does not bound. The authors should state the buffer-allocation model and, if static allocation is used, provide a worst-case bound or rely on a load-balancing guarantee from the training configuration.
minor comments (3)
- [Section 4] The prose sentence before Table 8 swaps the labels for optimizer states and gradients: it says 23.3 GB is for optimizer states and 46.6 GB is for gradients, while Table 8 and the ZeRO reduction formulas use gradients=23.3 GB and optimizer states=46.6 GB. Please correct the prose.
- [Throughout] The tables and text use 'GB' for what are actually binary units (GiB). For instance, 12,500,729,856 bytes is 11.64 GiB, not 12.5 GB. Please state the unit convention or use GiB consistently.
- [Section 5.2] The term 4bsN in the MoE activation formula is not explained; the authors should state that it represents the router-logit memory for N experts, or otherwise define the terms in the closed-form expression.
Circularity Check
No significant circularity: the report's memory counts are direct arithmetic from published architecture constants and external references, with no fitted parameter renamed as a prediction.
full rationale
The paper's derivation chain is self-contained arithmetic over externally given facts. Static parameter counts (Table 6: 11.64 GB per device) are obtained by multiplying published matrix shapes from Table 2 and DeepSeek-v3's layer count by 2 bytes per BF16 parameter; no quantity is fitted to a subset of data and then 'predicted' back. ZeRO memory numbers in Table 8 are the same parameter/gradient/optimizer bytes divided by the stated DP or EDP ranks, e.g. '(429,719,552/32 DP + 5,820,645,376/8 EDP) × 8', which is a direct application of the ZeRO sharding rule, not a renamed fit. Activation formulas in Section 5 are textbook per-microbatch activation accounting with the model's dimensions substituted, and the total in Table 10 sums the per-component terms. The paper cites external prior work (Megatron-LM, ZeRO, recomputation paper) for standard mechanisms; none of these citations is from the present authors, and none is invoked to forbid alternatives. The explicit disclaimer that the analyzed training policies are not DeepSeek's official configurations is a scope limitation, not a circular claim. Potential concerns such as unbalanced MoE routing or the absence of an explicit in-flight microbatch multiplier are soundness/accuracy criticisms of the model, not cases where the conclusion is equivalent to an input by construction. Accordingly, no circular step is present and the score is 0.
Assumptions & free parameters
free parameters (6)
- micro-batch size b =
1, 2, 4
- sequence length s =
4096
- routed experts per token Nr =
8
- parallel config (DP, TP, PP, EP, ETP, EDP) =
32, 2, 16, 8, 1, 8
- fragmentation overhead =
5% to 30% of allocated memory
- communication buffer size =
0.8 to 2 GB per device
assumptions (5)
- domain assumption Training runs in FP16/BF16; FP8 is excluded.
- domain assumption Expert load is balanced; each expert sees Etoken = bs*Nr/N tokens.
- domain assumption Megatron-LM's MLA implementation is an adequate proxy for DeepSeek-v3's MLA.
- domain assumption Sequence parallelism halves all activation tensors uniformly.
- domain assumption Full activation recomputation requires retaining only pre-RMSNorm inputs.
Cite this review
Pith. "Pith review of Memory Analysis on the Training Course of DeepSeek Models." pith.science (2026). https://pith.science/paper/AURWPLAY
@misc{pith2026250207846,
author = {Pith},
title = {Pith review of: Memory Analysis on the Training Course of DeepSeek Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/AURWPLAY}},
note = {Machine review of arXiv:2502.07846}
}
read the original abstract
We present a theoretical analysis of GPU memory consumption during the training of DeepSeek models such as DeepSeek-v2 and DeepSeek-v3. Our primary objective is to clarify the device-level memory requirements associated with various distributed training configurations. Specifically, we examine critical factors influencing memory usage, including micro-batch size, activation recomputation policies, 3D parallelism, and ZeRO optimizations. It is important to emphasize that the training policies discussed in this report are not representative of DeepSeek's official configurations. Instead, they are explored to provide a deeper understanding of memory dynamics in training of large-scale mixture-of-experts model.
Figures
Reference graph
Works this paper leans on
-
[1]
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024
arXiv 2024
-
[2]
Deepseek-v2: A strong, economical, and efficient mixture-of- experts language model
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of- experts language model. arXiv preprint arXiv:2405.04434, 2024
arXiv 2024
-
[3]
Efficient large-scale language model training on gpu clusters using megatron-lm
Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, S...
2021
-
[4]
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3506, 2020
work page 2020
-
[5]
Zero: Memory optimiza- tions toward training trillion parameter models
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimiza- tions toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020
2020
-
[6]
Reducing activation recomputation in large transformer models
Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mo- hammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5:341–353, 2023. 9
work page 2023
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.