{"id":"50236e85-ac9e-423f-a57e-fbaa0653e05c","arxiv_id":"2603.28768","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Per-layer, benefit-aware expert replication under a memory budget improves MoE serving goodput by 1.14x on average over uniform replication (EPLB) while using ~7x fewer replicas.","lead":"This paper describes CRAFT, a way to decide which layers of a Mixture-of-Experts language model get extra copies of \"hot\" experts, spending GPU memory only where it helps. In tests on 671B- and 1-trillion-parameter models, it raises serving throughput by about 14% compared to the standard EPLB replication approach, often while using several times less replica memory.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CRAFT's 1.14x gain lacks a same-memory uniform-replication control; the improvement may be due to preserving KV cache rather than to fine-grained per-layer allocation.","rationale":"The reader's weakest assumption concerns whether the offline-balancedness-gain matrix T and additive MCKP objective reliably predict end-to-end goodput. That is a real concern, but it is about the planner's optimality under drift and uneven replica splitting. My concern is a different, more basic experimental-control issue: the headline comparison conflates two variables—replica allocation granularity and total replica memory. The paper's own data in Section 5.3 show EPLB's KV cache is reduced by up to 75%, and that this alone can make EPLB slower than placement-only BASE; hence CRAFT's improvement over EPLB may be largely a memory-budget effect. The paper's novelty claim is specifically that fine-grained per-layer allocation beats uniform replication under a memory budget, but no uniform-replication-with-same-budget baseline is provided. This is an addressable experimental gap rather than a contradiction, so it does not overturn the reader's CONDITIONAL verdict, but it should be an explicit condition for acceptance: the authors should either add the same-memory uniform baseline or soften the attribution of the gain to fine-grained allocation. This is why I mark agreement as partial rather than full: the reader identified a different weakest point, but both point to the need for additional experimental controls before the central mechanism is established.","tokens_in":38428,"tokens_out":6937,"duration_ms":72921,"concrete_test":"In DE8, KE8, and KJ8, add a same-memory baseline: apply EPLB's uniform replication (one replica per GPU per selected layer) to the k layers with the highest estimated balancedness gain, with k chosen so total replica memory equals CRA8's budget (k = R, here 8). Measure goodput, TTFT curves, and achieved balancedness. If EPLB-subset matches CRA8 within noise, the fine-grained per-layer replica counts are not the driver of the 1.14x gain; if CRA8 still beats it, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim attributes CRAFT's end-to-end goodput improvement to benefit-driven, fine-grained per-layer replica allocation. But the main comparison in Section 5.2 pits CRA8 (R=8 replicas per GPU total) against EPLB with one replica per MoE layer per GPU (58 or 60 replicas per GPU), i.e., 7.25–7.5x more replica memory. Section 5.3 and Figure 9 show that EPLB's over-replication shrinks KV cache by 19–75%, and on K*6 EPLB goodput falls below the placement-only BASE. Thus a large component of CRAFT's measured advantage may simply be that it avoids EPLB's memory overcommitment and keeps a larger KV cache, not that its MCKP-based per-layer replica counts are load-bearing. The paper never compares CRAFT against a uniform-replication scheme with the same total replica budget, e.g., EPLB-style one-replica-per-GPU replication applied only to the 8 highest-benefit layers. Without that control, the specific claim that fine-grained cost-aware per-layer count allocation is what delivers the gain is underdetermined. Additionally, the artifact appendix ships a heuristic threshold-based allocator rather than the MCKP planner used in the evaluation, so the exact Algorithm 2 allocation is not independently reproducible.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CRAFT, a per-layer expert-replication framework for distributed MoE inference. CRAFT profiles per-layer expert load distributions, estimates the balancedness gain of different replica counts, solves a multiple-choice knapsack problem (MCKP) to allocate a fixed number of replicas across layers under a memory budget, and then uses capacity-aware greedy placement to map experts and replicas to GPUs. The work is integrated into SGLang and evaluated on DeepSeek-R1-671B and Kimi-K2-1000B on clusters of 6, 8, and 12 nodes. The headline result is that CRAFT with R=8 replicas per GPU achieves about 1.14x higher goodput on average (up to 1.2x) compared to EPLB, which uses one replica per MoE layer per GPU (58–60 replicas per GPU).","tokens_in":38725,"tokens_out":3502,"duration_ms":36748,"significance":"The problem is practically important: expert replication is a standard load-balancing technique in large MoE serving, and over-replication has a direct memory cost in the form of reduced KV cache. The characterization of diminishing returns and layer-dependent replication benefit is a useful contribution, and the evaluation is at realistic scale with two large models, multiple workloads, and profiling samples held out from evaluation. If the 1.14x gain is robust and attributable to fine-grained per-layer allocation, the paper would offer a simple, drop-in improvement over EPLB. However, the current experimental design does not cleanly separate the effect of the cost-aware allocation from the effect of simply using a much smaller replication memory budget. The paper is also weakened by the fact that the shipped artifact implements a heuristic allocator rather than the MCKP planner used in the evaluation.","major_comments":[{"comment":"The headline comparison is confounded by memory budget. CRA8 uses R=8 replicas per GPU, while EPLB uses L replicas per GPU (58 for DeepSeek-R1, 60 for Kimi-K2), i.e., 7.25–7.5x more replica memory. Section 5.3 reports that EPLB reduces KV cache by 19–75%, and Figures 9(b) and 9(c) show EPLB goodput falling below BASE. Thus the measured CRAFT advantage may be largely due to avoiding EPLB's memory overcommitment and preserving KV cache, not due to the MCKP-based per-layer replica counts. To support the central claim, add a same-memory uniform-replication control: allocate the same total replica budget as CRA8 uniformly across layers (or apply EPLB-style replication only to a fixed number of layers). Without such a control, the paper's attribution of the gain to fine-grained cost-aware allocation is underdetermined.","section":"Section 5.2, Figure 9"},{"comment":"The artifact does not reproduce the evaluated planner. The main text and Appendix A.2 describe an MCKP-based dynamic-programming allocation (Algorithm 2), but Appendix E.1 states that the shipped core implementation instead uses a threshold-based heuristic that automatically allocates replicas and selects a replication factor close to R=8. Since the evaluation uses CRA8 with the MCKP plan, the exact allocation evaluated in Sections 5.2–5.7 is not independently reproducible from the artifact. Please ship the evaluated MCKP solver, or clearly state that evaluation used the heuristic and evaluate that heuristic in the paper.","section":"Appendix E.1 vs Section 4.2.1/Appendix A.2"},{"comment":"The planning objective is a surrogate that is not validated against the end-to-end goal. The DP maximizes the sum of per-layer balancedness gains T[r][l], where Algorithm 1 computes each per-layer gain under the assumption that a replica splits load evenly as floor(W/rel). This ignores cross-layer interactions in the actual min-max GPU-load objective and assumes that the profiled distribution remains representative until the next rebalancing window. The paper does not include an ablation comparing CRAFT's MCKP allocation against a same-budget uniform allocation, a random allocation, or a greedy benefit allocation. Such an ablation is needed to show that the fine-grained allocation itself, rather than the memory saving, is load-bearing for the 1.14x claim.","section":"Section 4.2.1, Algorithms 1 and 2"}],"minor_comments":[{"comment":"R=8 is described as 'generally the best-performing setting selected by CRAFT across configurations.' Because R is selected on the same configurations used for the headline evaluation, the reported 1.14x average is a retrospectively selected value. Please report error bars or repeated runs, and state explicitly how R would be chosen in a deployment without access to the test workload.","section":"Section 5.1"},{"comment":"The goodput knee points are read from throughput-TTFT curves, but the paper does not define a quantitative procedure for determining the knee. Please provide the criterion used or report the raw data points so the knee determination is reproducible.","section":"Figure 8"},{"comment":"The abstract says '1.14x on average (up to 1.2x)' while Section 5.2 reports 1.15x average on DeepSeek-R1 and 1.12x on Kimi-K2. This is consistent if averaged, but the exact averaging method (across configurations? across model families?) should be stated.","section":"Section 5.2"},{"comment":"The artifact provides expert load distribution traces but not the full serving framework integration or the profiling scripts used to collect the traces. The paper would be stronger if the trace-collection pipeline were also released, since the load distributions are the primary input to CRAFT's planner.","section":"Appendix E.3.4"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this paper has a real contribution. It observes that expert replication benefits vary sharply by layer, formulates per-layer replica allocation as a multiple-choice knapsack under a memory budget, and shows on DeepSeek-R1 and Kimi-K2 that you can get most of EPLB's balancedness with roughly 7x less replica memory and better end-to-end goodput. The characterization of diminishing returns is genuinely useful, and the evaluation is large scale with held-out profiling traces.\n\nThe serious soft spot is attribution. The headline comparison is CRAFT (R=8 replicas/GPU) against EPLB (one replica per layer per GPU, i.e., 58-60 replicas/GPU). EPLB's over-replication shrinks the KV cache by 19-75% and in some setups makes it slower than placement-only BASE. So a large part of CRAFT's 1.14x gain may just be not over-replicating, not the per-layer fine-grained allocation. The paper never includes a same-memory control: e.g., uniform replication applied to the 8 highest-benefit layers, or uniformly choosing 8 replicas per GPU. Without that, the specific claim that the MCKP benefit allocation is what delivers the gain is underdetermined. This is fixable, but it's load-bearing for the stated contribution.\n\nSecond issue: the artifact ships a heuristic threshold-based allocator, not the MCKP solver used in the evaluation. The appendix says so bluntly. That means the exact Algorithm 2 allocation isn't independently reproducible. For a systems paper whose whole point is a planning algorithm, this should be addressed before publication.\n\nOther weaknesses are minor: R=8 is selected per configuration, there are no error bars on the throughput curves, and the rebalancing-window choice is a hyperparameter. None of these are fatal.\n\nOverall: the idea is sensible and the engineering is real. I'd send it to peer review, and ask for (1) a same-memory uniform baseline, (2) the exact MCKP planner in the artifact, and (3) error bars. A serious referee will get value from this paper even if it needs another round.","headline":"Useful, credible MoE serving paper with a clean idea and a real evaluation, but the key ablation is missing and the artifact under-delivers; deserves peer review.","tokens_in":39230,"tokens_out":2636,"would_cite":true,"duration_ms":25599,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that expert replication in Mixture-of-Experts serving should be allocated per layer under a memory budget, and shows this raises end-to-end throughput by 1.14x on average compared to uniform replication by freeing memory fo","keywords":["mixture of experts","expert parallelism","load balancing","expert replication","inference serving","KV cache","goodput","cost-aware allocation"],"falsifier":"Run the same serving workload with a dynamic workload that switches between two very different data distributions every few hundred iterations, with no rebalancing in between; if the proposed scheme's balancedness and sustained throughput fall to placement-only levels while the uniform baseline holds up, the representativeness assumption fails. More directly, instrument a hot expert's dispatch: if tokens are not split approximately evenly across its replicas in deployment, the benefit table overestimates the true gain.","tokens_in":38312,"feed_emoji":"⚡","tokens_out":3956,"duration_ms":39296,"temperature":0.7,"pith_summary":"This paper argues that the standard way of replicating hot experts in Mixture-of-Experts inference — giving every layer the same number of extra expert copies — is wasteful. It claims that most of the load-balancing benefit comes from a few high-skew layers, and that replicas beyond about 8 per GPU add almost nothing while consuming GPU memory that could otherwise hold a larger KV cache. It proposes a framework that profiles per-layer replication benefit, then allocates a fixed replica budget across layers with dynamic programming to maximize balancedness gain, and assigns replicas evenly across devices. In tests on large MoE models, this benefit-driven plan uses roughly 7x fewer replicas than the uniform baseline while achieving slightly better sustained throughput (1.14x on average, up to 1.2x) because the freed memory enlarges the KV cache. The paper's central claim is that replication should be treated as a memory budget allocation problem, not a one-size-fits-all heuristic.","feed_headline":"Replicating only the experts that matter boosts MoE serving by 1.14x","feed_subtitle":"A per-layer memory budget replaces uniform replication, cutting replica count ~7x while raising throughput.","key_machinery":"The load-bearing object is the per-layer balancedness gain matrix T, an L×K table where entry T[r][l] is the GPU balancedness gain for layer l when given r replicas, computed by replaying profiled tokens under a greedy placement. The framework turns replica allocation into a multiple-choice knapsack problem over this table — choose one replica count per layer so the sum of gains is maximized subject to the replica budget — solved by dynamic programming in O(L·C·|R|). A secondary mechanism is capacity-aware interleaved assignment: because layers have different replica counts, extra experts are assigned to the least-loaded GPUs with an interleaved tie-break across nodes, keeping per-device mem","core_discovery":"The central claim is that the balancedness gain from expert replication is layer-specific and sharply diminishing: once hot experts are replicated enough for placement to smooth the rest, additional copies add negligible load balance but cost real memory. The framework operationalizes this by measuring, for each MoE layer, the balancedness gain at a geometric sweep of replica counts (replayed from profiled token loads), solving a multiple-choice knapsack to pick per-layer replica counts under a total memory budget, and then using a capacity-aware greedy placement to map experts to GPUs. The result is near-uniform device load with far fewer replicas, and the memory saved is converted directly","pith_inferences":["A natural extension is to make the benefit table adaptive online: rather than replaying a static profile, the server could update the table from observed loads and re-solve the knapsack at each rebalancing window, which would strengthen robustness to workload drift.","The knapsack formulation assumes gains are additive across layers; in models where layers interact (e.g., shared or grouped experts), a joint allocation model might find additional gains, or the additivity assumption may break.","The principle — spend memory only where load skew is high — extends beyond replication to other memory/compute trade-offs, such as expert offloading to CPU or sharding a subset of experts, where the same benefit-per-memory curve could guide which experts to offload.","The reported 1.14x is on long-sequence workloads with heavy prefill; decoding-dominated workloads may show smaller gains since load imbalance is less severe at lower batch sizes, so the benefit should be tested there."],"forward_implications":["MoE serving systems can get most of the load-balancing benefit of expert replication with about 7x less replica memory, enabling larger KV caches and batch sizes.","Over-replication hurts sustained throughput: in small clusters, the standard uniform scheme can reduce throughput below placement-only because memory is taken from the KV cache.","Replication plans should be computed per layer, not per device: a fixed replica count under-allocates to skewed layers and over-allocates to balanced ones.","The benefit-estimation and planning steps run in about 10 seconds and add no runtime overhead, making the scheme viable as a drop-in replacement in existing serving stacks.","The framework adapts to workload shifts through periodic rebalancing, and stays within comparable balancedness of the uniform baseline while using less memory."],"fun_headline_variants":["Per-layer replication lifts MoE throughput 1.14x","Smarter expert replication: same balance, 7x fewer replicas","CRAFT: right experts on right layers, 1.2x faster MoE","Trim redundant replicas, boost MoE serving throughput","Budget-aware expert replication cuts memory, scales MoE"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The plan is only as good as the profiled load distribution: the paper assumes the token loads observed during offline replay stay representative until the next rebalancing window, and that a replica splits its expert's load evenly across copies.","fun_headline_variants_meta":{"raw":{"variants":["Per-layer replication lifts MoE throughput 1.14x","Smarter expert replication: same balance, 7x fewer replicas","CRAFT: right experts on right layers, 1.2x faster MoE","Trim redundant replicas, boost MoE serving throughput","Budget-aware expert replication cuts memory, scales MoE"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000613,"raw_usage":{"total_tokens":2676,"prompt_tokens":723,"completion_tokens":1953,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":1864}},"tokens_in":467,"tokens_out":1953,"duration_ms":13591,"temperature":1.0,"reasoning_tokens":1864,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T10:57:51.522022+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same serving workload with a dynamic workload that switches between two very different data distributions every few hundred iterations, with no rebalancing in between; if the proposed scheme's balancedness and sustained throughput fall to placement-only levels while the uniform baseline holds up, the representativeness assumption fails. More directly, instrument a hot expert's dispatch: if tokens are not split approximately evenly across its replicas in deployment, the benefit table overestimates the true gain.","supporting_citations":[],"review_version":1}