{"id":"20ea77d7-5314-4c60-99a3-cf72d9fba285","arxiv_id":"2502.07846","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A parameter and activation memory model for DeepSeek-v3 training gives per-GPU footprints under PP/TP/EP and ZeRO, but with no empirical validation.","lead":"This paper calculates how much GPU memory it would take to train DeepSeek-style mixture-of-experts models under different parallel setups, activation recomputation, and ZeRO memory savings. It is a back-of-envelope engineering reference for planning large-scale model training rather than a new scientific result.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 5 activation formulas omit in-flight pipeline microbatches, so Table 10 understates per-device activation memory by roughly the microbatch count; the missing M factor is a first-order gap in the central claim.","rationale":"The paper's stated goal is device-level memory during training, and the reader's strongest claim treats Section 5.3's Total as activation memory per device. That identification is the least secure point of the argument. The formulas count tensors for one microbatch only, while pipeline execution introduces a multiplicative factor equal to the number of microbatches whose activations are stored simultaneously. Since no variable for this count is defined in Section 5, Table 10 cannot be a per-device total as written. The issue is independent of routing balance and applies to both recomputation modes, although full recomputation shrinks the per-microbatch footprint. The static-parameter and ZeRO analysis is largely self-consistent, so I would not reject the whole report; however the activation portion of the central claim needs a correction and an explicit microbatch schedule. The reader's balanced-routing concern is a genuine second-order effect on the MoE expert term, but it only matters after the missing microbatch factor is fixed. I therefore disagree with the reader's choice of weakest assumption, while agreeing that the verdict should remain conditional rather than accept.","tokens_in":6785,"tokens_out":23490,"duration_ms":213650,"concrete_test":"Re-derive Table 10 for PP16, b=1, s=4096, no recomputation, introducing M = number of in-flight microbatches. For a filled pipeline M=16, each entry in Table 10 must scale by M; compare M × 4(M_A1+M_E1) against the stated Table 10 total. If M is absent, stage 0 activation memory is understated by a factor of 16, producing a device-total far above an 80 GB GPU. Then check whether the authors can state M explicitly and apply the same multiplier to the 8bsh+8bsNr full-recomputation total.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that Table 10 gives per-device activation memory under PP16. But every Section 5 formula is written for a single microbatch: b is the micro-batch size, and no variable appears for the number of microbatches in flight in the pipeline schedule. With pipeline parallelism, each stage must retain activations for every microbatch that has completed its forward pass but not yet its backward pass. In a standard 1F1B schedule with PP16 and M microbatches, the first stage can store all M microbatches' activations, so the per-device total is approximately M times the formulas, not the formulas alone. For b=1 and s=4096 in a four-layer MoE stage with no recomputation, the per-microbatch total from Table 10 is tens of GB; multiplying by M yields hundreds of GB, far beyond a device budget. Full recomputation does not remove the issue because each pending microbatch still needs its retained input activations; the 8bsh+8bsNr total also carries an implicit M. The table therefore cannot be the claimed device-level activation memory unless the authors intend a per-microbatch quantity and state the microbatch count. The reader's balanced-routing concern is real but secondary; the missing M multiplier is first-order.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript is a theoretical memory-usage analysis for training DeepSeek-v2/v3-style mixture-of-experts models. It first counts parameters per transformer layer and per pipeline stage, then derives per-device static parameter memory for a PP16@TP2@EP8 configuration, applies DeepSpeed ZeRO stages to optimizer states, gradients, and parameters, and finally gives closed-form activation-memory expressions for MLA and MoE blocks with and without full activation recomputation. The paper explicitly disclaims that the analyzed parallel configurations are not DeepSeek's official training setup. The headline numbers are about 6.25B parameters and 11.64 GiB static memory per device, the ZeRO totals in Table 8 (e.g., 9.66 GiB for os+g+params), and the activation totals in Table 10.","tokens_in":7059,"tokens_out":14980,"duration_ms":136900,"significance":"The parameter-level arithmetic and the mixed DP/EDP ZeRO formulas check out internally and are reproducible from public architectural constants, which is a genuine strength: a reader can recompute every static-memory number from Tables 1 and 2. The paper also usefully separates MoE-expert parameters from non-MoE parameters in the ZeRO calculation, which many simplified memory models do not. The utility of the activation half is currently limited by a first-order omission: the formulas in Section 5 are per-micro-batch per-stage quantities, while Table 10 presents them as per-device values under pipeline parallelism, and no multiplication by the number of in-flight micro-batches appears. A secondary gap is that the worst-case stage is selected by parameter count alone even though activation memory is not monotone in parameter count. If these issues are corrected, the paper would be a valuable reference for MoE training memory planning; as written, the activation-memory claims are not yet established.","major_comments":[{"comment":"The activation formulas are written for one micro-batch (b is the micro-batch size, and no variable appears for the number of micro-batches in flight), but Table 10 labels the result as 'activation memory per device' under PP16. In a 1F1B pipeline schedule with PP16 and M micro-batches, a stage must retain activations for each micro-batch that has completed its forward pass but not its backward pass; for the first stage this can be all M micro-batches. The device-level total is therefore approximately M times the listed formulas, not the formulas themselves. For example, with b=1, s=4096 and no recomputation, the Table 10 total is about 24.7 GiB per micro-batch, so with M=16 the device would need on the order of 395 GiB, far beyond the paper's implied budget. The full-recomputation total 8bsh+8bsNr similarly carries an implicit M. The authors should either multiply by the number of in-flight micro-batches and state the pipeline schedule, or explicitly state that Table 10 is a per-micro-batch quantity and provide the missing M factor.","section":"Section 5.3, Table 10"},{"comment":"The paper selects Stages 1-14 as the worst case based solely on parameter count and then analyzes only MLA and MoE layers, but activation memory is not monotone in parameter count. Stage 0 contains layers 0-3, including three dense FFN layers with intermediate size hF=18432 and the embedding matrix; Section 5 provides no activation formulas for these dense layers. Under the same activation accounting as the shared-expert term in Section 5.2, one dense FFN layer contributes roughly 8bshF bytes for its intermediate activations, which at b=1 and s=4096 is about 0.60 GiB per micro-batch and is comparable to the per-layer MoE expert terms. The claim that Stages 1-14 determine the global per-device memory peak is therefore not justified unless the authors either extend the activation analysis to Stage 0 or prove that its activation memory is bounded by that of the MoE stages.","section":"Section 2.2, Table 4"},{"comment":"The MoE activation formula uses Etoken = bs*Nr/N, the average number of tokens per expert under perfectly balanced routing. The paper does not state whether per-expert activation buffers are allocated dynamically according to the actual number of tokens routed to each expert or statically to a maximum capacity. With dynamic allocation, the total over all experts on a rank depends on the total number of expert-tokens routed to that rank, so per-expert hot spots are less of an issue but per-rank token variance still matters; with static allocation, a hot expert can cause per-expert memory to exceed the average-based estimate by an amount the paper does not bound. The authors should state the buffer-allocation model and, if static allocation is used, provide a worst-case bound or rely on a load-balancing guarantee from the training configuration.","section":"Section 5.2"}],"minor_comments":[{"comment":"The prose sentence before Table 8 swaps the labels for optimizer states and gradients: it says 23.3 GB is for optimizer states and 46.6 GB is for gradients, while Table 8 and the ZeRO reduction formulas use gradients=23.3 GB and optimizer states=46.6 GB. Please correct the prose.","section":"Section 4"},{"comment":"The tables and text use 'GB' for what are actually binary units (GiB). For instance, 12,500,729,856 bytes is 11.64 GiB, not 12.5 GB. Please state the unit convention or use GiB consistently.","section":"Throughout"},{"comment":"The term 4bsN in the MoE activation formula is not explained; the authors should state that it represents the router-logit memory for N experts, or otherwise define the terms in the closed-form expression.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a useful reference calculation rather than a research breakthrough, and its static-parameter and ZeRO arithmetic are solid enough to be worth publishing after revision. The missing pipeline-micro-batch factor in the activation-memory claim is a first-order correctness issue that prevents the paper from fulfilling its stated purpose in its current form, so I cannot recommend acceptance until it is resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThis paper is a worked memory-accounting exercise for DeepSeek-v3-style MoE training: it plugs architecture constants from the DeepSeek-v3 report into the standard Megatron-LM and ZeRO formulas and produces per-device numbers for a PP16/TP2/EP8 configuration. That is the whole contribution, but it is a useful one. The static parameter math checks out (6.25B parameters per device, 11.64 GB in BF16, ZeRO os+g+params total 9.66 GB), and the authors correctly keep DP and EDP separate when sharding optimizer states and gradients. No new mechanisms or equations beyond the cited sources, but the adaptation to MLA and MoE shapes is careful.\n\nThe soft spots, in order of severity.\n\nFirst, Table 10 is not per-device activation memory as claimed. Every activation formula in Section 5 is written for a single microbatch (b is the micro-batch size), and no multiplier appears for the number of microbatches in flight. Under PP16 with a standard 1F1B schedule, a stage can hold activations for many microbatches; the first stage potentially all M of them. So the real per-device value is roughly M times the table's entries, not the table itself. For b=1, s=4096, no recomputation, that is hundreds of GB, far beyond a device. Full recomputation does not remove the issue because every pending microbatch still needs its retained inputs. The formulas are fine as per-microbatch quantities; the table's labeling is the problem. This is a first-order gap.\n\nSecond, the MoE activation estimate assumes perfectly balanced routing (Etoken = bs*Nr/N). Real routing is unbalanced, and hot experts can push activation memory higher. The paper gives no bound. Real concern, secondary to the missing M.\n\nThird, a small but real inconsistency: the ZeRO prose swaps the labels for gradients and optimizer states (23.3 vs 46.6 GB) relative to Table 8. The table itself is correct.\n\nFourth, there is no validation against measured memory traces. The whole thing is an unvalidated estimate. It would be far stronger with one real measurement or a small simulator.\n\nThe reader's conditional verdict is right, and the missing-M issue makes it more conditional than the reader originally thought. The central static-parameter and ZeRO analysis holds up; the activation analysis needs rework and re-labeling before its numbers are used for capacity planning.\n\nThis paper deserves a serious referee: it is useful, transparent, and mostly correct, but the activation table needs a revision. I'd send it back with a request to fix the microbatch accounting and either bound the routing imbalance or state the balanced-routing assumption clearly. Citation-wise, I'd wait for the revised version.","headline":"Useful reference math for DeepSeek-v3 training memory, but the activation table misses the pipeline microbatch multiplier and is not a per-device number as claimed.","tokens_in":7588,"tokens_out":3365,"would_cite":false,"duration_ms":29671,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper derives closed-form per-GPU memory formulas for training DeepSeek-v3-scale MoE models, reducing the answer to layer counts and parallelism choices.","keywords":["DeepSeek-v3","mixture of experts","GPU memory estimation","activation recomputation","ZeRO","pipeline parallelism","expert parallelism","tensor parallelism"],"falsifier":"Run one training step of the DeepSeek-v3 architecture under the paper's PP16, TP2, EP8, SP2, full-recomputation configuration with $b=1$ and $s=4096$, record peak per-GPU memory with a profiler, and compare the activation portion against $8bsh+8bsN_r$ after subtracting measured parameter, gradient, optimizer, fragmentation, and buffer usage; a systematic excess on hot experts would refute the balanced-routing formula.","tokens_in":6571,"feed_emoji":"🧮","tokens_out":8280,"duration_ms":69280,"temperature":0.7,"pith_summary":"This report tries to show that the per-GPU memory needed to train a DeepSeek-v3-scale mixture-of-experts (MoE) language model is an arithmetic consequence of layer counts and parallelism choices, not a quantity that must be measured empirically. It works through a concrete case (pipeline depth 16, tensor parallelism 2, expert parallelism 8) and derives exact device-level counts: about 6.25 billion parameters per GPU, dominated by the MoE experts, with the combined parameters, gradients, and optimizer states dropping to under 10 GB when ZeRO shards all three across data-parallel groups. It also gives closed-form activation formulas for the multi-head latent attention and MoE feed-forward blocks, with and without activation recomputation. If the formulas are right, a training team can predict memory before launch and choose micro-batch size and recomputation policy accordingly.","feed_headline":"Counting experts yields exact per-GPU memory for DeepSeek-v3 training","feed_subtitle":"Parameter, optimizer, activation, and fragmentation memory all fall out of simple arithmetic from layer counts and parallelism","key_machinery":"The carrying mechanism is a per-stage arithmetic decomposition built from two counting rules. First, static parameters are counted per expert-parallel rank: with 256 routed experts and 1 shared expert per MoE layer, expert parallelism 8 gives 32 routed plus 1 shared expert per layer per rank, and a four-layer pipeline stage therefore holds 132 expert stacks of three matrices each ($h\\times h_E$, $h\\times h_E$, $h_E\\times h$), yielding 5.813 billion expert parameters per device. Second, activation memory is expressed as a symbolic sum over the MLA and MoE dataflow, with the MoE term relying on the balanced-routing identity $E_{tok}=bsN_r/N$ for the average number of tokens per expert. These two rules, together with the ZeRO sharding denominators ($DP=32$ for dense parts, $EDP=8$ for experts), convert all memory categories into closed-form expressions.","core_discovery":"On its own terms, the paper's central claim is a decomposition identity. In the PP16@TP2@EP8 configuration, each pipeline stage holds four transformer layers, and each GPU on a stage holds 132 experts (128 routed experts and 4 shared experts across 4 layers), giving a static weight footprint of 6250364928 parameters, or 11.64 GB in BF16, with 10.84 GB of that coming from MoE experts. Applying ZeRO with data-parallel size 32 for dense parts and expert-data-parallel size 8 for experts yields 1.38 GB for parameters, 2.76 GB for gradients, and 5.52 GB for optimizer states. For activations, the paper derives per-device formulas for a four-layer stage: without recomputation the MLA term is $10bsh + 8bs(d_{cq}+d_c) + 16bsd_hn_h + 8bsd_{hr}n_h + 10bn_hs^2$ and the MoE term is $20bsh + 16bsN + 8bsN_r + 4bs(N_r/N)(96h+256h_E) + 32bsh_E$; with full recomputation the total collapses to $8bsh + 8bsN_r$. The claim is that these expressions predict the device memory peak once fragmentation and communication buffers are added.","pith_inferences":["Extending the paper's arithmetic to other batch sizes is immediate: since the formulas are linear in $b$, doubling the micro-batch doubles the activation term, which lets a planner tune batch size against the measured per-GPU HBM before launch.","The balanced-routing assumption is the natural stress point; real token-to-expert routing is often skewed, so a testable extension would be to walk the formulas with empirically observed per-expert token counts and compare the resulting activation estimate with the $E_{tok}$-based one.","Because the paper leaves FP8 training and quantization scaling factors out of scope, applying the same decomposition to mixed-precision training would require adding per-tensor scaling memory and is a direct follow-up the authors do not perform.","The fragmentation range (5-30%) and communication-buffer range (0.8-2 GB) quoted at the end suggest that the analytic formula is a lower-bound-style planning tool rather than a precise peak-memory guarantee for a specific runtime allocator."],"forward_implications":["Under the paper's case configuration, static expert parameters dominate device memory (10.84 GB of 11.64 GB), so increasing expert parallelism is the most direct lever for shrinking per-GPU weight memory.","Full activation recomputation lowers activation memory from a four-line formula involving several MLA constants to the simple expression $8bsh+8bsN_r$, meaning activation cost scales linearly with micro-batch size, sequence length, and the number of routed experts per token.","The ZeRO os+g+params scenario puts parameters, gradients, and optimizer states at 9.66 GB per GPU for this configuration, implying that the remaining memory budget on a modern GPU is set by activations, fragmentation, and communication buffers.","The same counting method can be applied to any MoE transformer with known layer counts and an MLA-style attention, giving a pre-training estimate without running a profiler."],"supporting_citations":[{"why":"Supplies the DeepSeek-v3 architecture, layer counts, and hyper-parameters (61 layers, hidden size 7168, 256 routed experts) that all counters use.","marker":"[1]"},{"why":"Establishes the multi-head latent attention and MoE design that the memory formulas are built on.","marker":"[2]"},{"why":"Defines the tensor-parallel layer implementation used as the basis for partitioning MLA and MoE parameter matrices.","marker":"[3]"},{"why":"Introduces the ZeRO optimizer-state, gradient, and parameter sharding techniques whose denominators are used in the memory tables.","marker":"[4]"},{"why":"Specifies the os, os+g, and os+g+params variants that determine how much of each memory category remains per device.","marker":"[5]"},{"why":"Gives the selective and full recomputation strategy whose memory-saving effect the activation formulas quantify.","marker":"[6]"}],"fun_headline_variants":["Exact per-GPU memory from layer counts and parallelism","DeepSeek-v3 memory: a simple arithmetic identity","Counting experts gives exact memory for DeepSeek training","Memory breakdown for DeepSeek falls out of arithmetic","Per-GPU memory peak predicted by layer and expert counts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The activation estimate assumes routed tokens spread evenly across experts, so each expert's share is exactly the average $bsN_r/N$; if real routing is imbalanced, hot experts can need more activation memory than the formula credits them.","fun_headline_variants_meta":{"raw":{"variants":["Exact per-GPU memory from layer counts and parallelism","DeepSeek-v3 memory: a simple arithmetic identity","Counting experts gives exact memory for DeepSeek training","Memory breakdown for DeepSeek falls out of arithmetic","Per-GPU memory peak predicted by layer and expert counts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000238,"raw_usage":{"total_tokens":1503,"prompt_tokens":933,"completion_tokens":570,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":491}},"tokens_in":549,"tokens_out":570,"duration_ms":5297,"temperature":1.0,"reasoning_tokens":491,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T12:51:46.943374+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run one training step of the DeepSeek-v3 architecture under the paper's PP16, TP2, EP8, SP2, full-recomputation configuration with $b=1$ and $s=4096$, record peak per-GPU memory with a profiler, and compare the activation portion against $8bsh+8bsN_r$ after subtracting measured parameter, gradient, optimizer, fragmentation, and buffer usage; a systematic excess on hot experts would refute the balanced-routing formula.","supporting_citations":[{"cited_title":"Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters","cited_arxiv_id":null,"evidence_quote":"Introduces the ZeRO optimizer-state, gradient, and parameter sharding techniques whose denominators are used in the memory tables."},{"cited_title":"Reducing activation recomputation in large transformer models","cited_arxiv_id":null,"evidence_quote":"Gives the selective and full recomputation strategy whose memory-saving effect the activation formulas quantify."}],"review_version":1}