REVIEW 3 major objections 4 minor 1 cited by
CRAFT: Fine-Grained Cost-Aware Expert Replication For Efficient Mixture-of-Experts Serving
T0 review · 3 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read The paper argues that expert replication in Mixture-of-Experts serving should be allocated per layer under a memory budget, and shows this raises end-to-end throughput by 1.14x on average compared to uniform replication by freeing memory fo
desk verdict Useful, credible MoE serving paper with a clean idea and a real evaluation, but the key ablation is missing and the artifact under-delivers; deserves peer review. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the per-layer balancedness gain matrix T, an L×K table where entry T[r][l] is the GPU balancedness gain for layer l when given r replicas, computed by replaying profiled tokens under a greedy placement. The framework turns replica allocation into a multiple-choice knapsack problem over this table — choose one replica count per layer so the sum of gains is maximized subject to the replica budget — solved by dynamic programming in O(L·C·|R|). A secondary mechanism is capacity-aware interleaved assignment: because layers have different replica counts, extra experts are assigned to the least-loaded GPUs with an interleaved tie-break across nodes, keeping per-device mem
What would settle it
Run the same serving workload with a dynamic workload that switches between two very different data distributions every few hundred iterations, with no rebalancing in between; if the proposed scheme's balancedness and sustained throughput fall to placement-only levels while the uniform baseline holds up, the representativeness assumption fails. More directly, instrument a hot expert's dispatch: if tokens are not split approximately evenly across its replicas in deployment, the benefit table overestimates the true gain.
Extended reading notes
Core claim
The central claim is that the balancedness gain from expert replication is layer-specific and sharply diminishing: once hot experts are replicated enough for placement to smooth the rest, additional copies add negligible load balance but cost real memory. The framework operationalizes this by measuring, for each MoE layer, the balancedness gain at a geometric sweep of replica counts (replayed from profiled token loads), solving a multiple-choice knapsack to pick per-layer replica counts under a total memory budget, and then using a capacity-aware greedy placement to map experts to GPUs. The result is near-uniform device load with far fewer replicas, and the memory saved is converted directly
Load-bearing premise
The plan is only as good as the profiled load distribution: the paper assumes the token loads observed during offline replay stay representative until the next rebalancing window, and that a replica splits its expert's load evenly across copies.
Editorial extensions
If this is right
- MoE serving systems can get most of the load-balancing benefit of expert replication with about 7x less replica memory, enabling larger KV caches and batch sizes.
- Over-replication hurts sustained throughput: in small clusters, the standard uniform scheme can reduce throughput below placement-only because memory is taken from the KV cache.
- Replication plans should be computed per layer, not per device: a fixed replica count under-allocates to skewed layers and over-allocates to balanced ones.
- The benefit-estimation and planning steps run in about 10 seconds and add no runtime overhead, making the scheme viable as a drop-in replacement in existing serving stacks.
- The framework adapts to workload shifts through periodic rebalancing, and stays within comparable balancedness of the uniform baseline while using less memory.
Reading between the lines
- A natural extension is to make the benefit table adaptive online: rather than replaying a static profile, the server could update the table from observed loads and re-solve the knapsack at each rebalancing window, which would strengthen robustness to workload drift.
- The knapsack formulation assumes gains are additive across layers; in models where layers interact (e.g., shared or grouped experts), a joint allocation model might find additional gains, or the additivity assumption may break.
- The principle — spend memory only where load skew is high — extends beyond replication to other memory/compute trade-offs, such as expert offloading to CPU or sharding a subset of experts, where the same benefit-per-memory curve could guide which experts to offload.
- The reported 1.14x is on long-sequence workloads with heavy prefill; decoding-dominated workloads may show smaller gains since load imbalance is less severe at lower batch sizes, so the benefit should be tested there.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CRAFT, a per-layer expert-replication framework for distributed MoE inference. CRAFT profiles per-layer expert load distributions, estimates the balancedness gain of different replica counts, solves a multiple-choice knapsack problem (MCKP) to allocate a fixed number of replicas across layers under a memory budget, and then uses capacity-aware greedy placement to map experts and replicas to GPUs. The work is integrated into SGLang and evaluated on DeepSeek-R1-671B and Kimi-K2-1000B on clusters of 6, 8, and 12 nodes. The headline result is that CRAFT with R=8 replicas per GPU achieves about 1.14x higher goodput on average (up to 1.2x) compared to EPLB, which uses one replica per MoE layer per GPU (58–60 replicas per GPU).
Significance. The problem is practically important: expert replication is a standard load-balancing technique in large MoE serving, and over-replication has a direct memory cost in the form of reduced KV cache. The characterization of diminishing returns and layer-dependent replication benefit is a useful contribution, and the evaluation is at realistic scale with two large models, multiple workloads, and profiling samples held out from evaluation. If the 1.14x gain is robust and attributable to fine-grained per-layer allocation, the paper would offer a simple, drop-in improvement over EPLB. However, the current experimental design does not cleanly separate the effect of the cost-aware allocation from the effect of simply using a much smaller replication memory budget. The paper is also weakened by the fact that the shipped artifact implements a heuristic allocator rather than the MCKP planner used in the evaluation.
major comments (3)
- [Section 5.2, Figure 9] The headline comparison is confounded by memory budget. CRA8 uses R=8 replicas per GPU, while EPLB uses L replicas per GPU (58 for DeepSeek-R1, 60 for Kimi-K2), i.e., 7.25–7.5x more replica memory. Section 5.3 reports that EPLB reduces KV cache by 19–75%, and Figures 9(b) and 9(c) show EPLB goodput falling below BASE. Thus the measured CRAFT advantage may be largely due to avoiding EPLB's memory overcommitment and preserving KV cache, not due to the MCKP-based per-layer replica counts. To support the central claim, add a same-memory uniform-replication control: allocate the same total replica budget as CRA8 uniformly across layers (or apply EPLB-style replication only to a fixed number of layers). Without such a control, the paper's attribution of the gain to fine-grained cost-aware allocation is underdetermined.
- [Appendix E.1 vs Section 4.2.1/Appendix A.2] The artifact does not reproduce the evaluated planner. The main text and Appendix A.2 describe an MCKP-based dynamic-programming allocation (Algorithm 2), but Appendix E.1 states that the shipped core implementation instead uses a threshold-based heuristic that automatically allocates replicas and selects a replication factor close to R=8. Since the evaluation uses CRA8 with the MCKP plan, the exact allocation evaluated in Sections 5.2–5.7 is not independently reproducible from the artifact. Please ship the evaluated MCKP solver, or clearly state that evaluation used the heuristic and evaluate that heuristic in the paper.
- [Section 4.2.1, Algorithms 1 and 2] The planning objective is a surrogate that is not validated against the end-to-end goal. The DP maximizes the sum of per-layer balancedness gains T[r][l], where Algorithm 1 computes each per-layer gain under the assumption that a replica splits load evenly as floor(W/rel). This ignores cross-layer interactions in the actual min-max GPU-load objective and assumes that the profiled distribution remains representative until the next rebalancing window. The paper does not include an ablation comparing CRAFT's MCKP allocation against a same-budget uniform allocation, a random allocation, or a greedy benefit allocation. Such an ablation is needed to show that the fine-grained allocation itself, rather than the memory saving, is load-bearing for the 1.14x claim.
minor comments (4)
- [Section 5.1] R=8 is described as 'generally the best-performing setting selected by CRAFT across configurations.' Because R is selected on the same configurations used for the headline evaluation, the reported 1.14x average is a retrospectively selected value. Please report error bars or repeated runs, and state explicitly how R would be chosen in a deployment without access to the test workload.
- [Figure 8] The goodput knee points are read from throughput-TTFT curves, but the paper does not define a quantitative procedure for determining the knee. Please provide the criterion used or report the raw data points so the knee determination is reproducible.
- [Section 5.2] The abstract says '1.14x on average (up to 1.2x)' while Section 5.2 reports 1.15x average on DeepSeek-R1 and 1.12x on Kimi-K2. This is consistent if averaged, but the exact averaging method (across configurations? across model families?) should be stated.
- [Appendix E.3.4] The artifact provides expert load distribution traces but not the full serving framework integration or the profiling scripts used to collect the traces. The paper would be stronger if the trace-collection pipeline were also released, since the load distributions are the primary input to CRAFT's planner.
Circularity Check
No significant circularity: throughput claim is externally measured; minor methodology concerns around tuning R and reusing the profiled objective for balancedness reporting.
full rationale
The central 1.14x goodput claim is not circular: CRAFT's planner consumes a profiled load distribution W and an L×K balancedness-gain matrix T (Algorithm 1), and Algorithm 2 chooses replica counts to maximize the sum of T. This makes the balancedness curves in Figures 5 and 9 essentially the optimizer's own objective, so the statement that "CRA achieves most of the balancedness gains with far fewer replicas" is partly self-confirming. However, the paper never uses T as a surrogate for the reported throughput: end-to-end goodput is measured in a real SGLang deployment, the profiling sequences are excluded from evaluation inputs (Section 5.1), and R=8 is presented as a selected hyperparameter with full R sweeps in Appendix B. The main comparison is to EPLB, an external baseline, not to a self-citation. The absent same-memory uniform-replication control, and the artifact appendix's heuristic (rather than MCKP) allocator, are legitimate underdetermination/reproducibility concerns, but they are not cases where a prediction equals its input by construction. I therefore find no load-bearing circularity, only minor methodology burdens: tuning R on the same configurations and reusing the replayed balancedness objective when reporting balancedness.
Assumptions & free parameters
free parameters (3)
- Replication factor R =
8 (default in CRA8; range 1-32 swept in Appendix B)
- Artifact heuristic threshold =
not reported
- Rebalancing window =
1200 iterations (~20 min)
assumptions (5)
- domain assumption The expert load distribution collected offline (3000 inference batches per workload) is representative of the online serving load after deployment and during each rebalancing window.
- domain assumption Per-layer balancedness gains are additive: the MCKP objective maximizes the sum of per-layer gains, ignoring interactions between layers or between replicas across layers.
- domain assumption Balancedness (average GPU load / max GPU load) is a monotone proxy for end-to-end goodput after accounting for KV-cache memory.
- domain assumption Replica copies split an expert's token load evenly, so the replay in Algorithm 1 can assign floor(W[b][l][e]/rel) tokens per copy.
- standard math MCKP DP correctness and pseudo-polynomial bounds (Martello & Toth 1990) for small D, K, L.
Cite this review
Pith. "Pith review of CRAFT: Fine-Grained Cost-Aware Expert Replication For Efficient Mixture-of-Experts Serving." pith.science (2026). https://pith.science/paper/AG3PAJ22
@misc{pith2026260328768,
author = {Pith},
title = {Pith review of: CRAFT: Fine-Grained Cost-Aware Expert Replication For Efficient Mixture-of-Experts Serving},
year = {2026},
howpublished = {\url{https://pith.science/paper/AG3PAJ22}},
note = {Machine review of arXiv:2603.28768}
}
abstract
Mixture-of-Experts (MoE) has recently emerged as the mainstream architecture for efficiently scaling large language models while maintaining near-constant computational cost. Expert parallelism distributes parameters by partitioning experts across devices, but this introduces token-level load imbalance during inference. Expert replication is a widely adopted load-balancing technique in serving frameworks that alleviates load imbalance in large-scale deployments by replicating experts with high loads. In this work, we demonstrate that existing replication schemes often over-replicate, with many replicas providing marginal improvement. Replicas consume substantial GPU memory, which may lead to resource contention and throughput degradation. We present CRAFT, an efficient expert replication framework that maximizes load balance under a given memory budget by performing fine-grained, per-layer replication based on the estimated replication benefit. CRAFT can be seamlessly integrated into existing serving frameworks without any additional training or model changes. Our evaluation shows that CRAFT increases end-to-end serving throughput by $1.14\times$ on average (up to $1.2\times$) over existing replication techniques in large-scale deployments with models ranging from hundreds of billions to a trillion parameters.
Figures
Figures from the paper (12 more)
Forward citations
Cited by 1 Pith paper
-
HCRMap: Pressure-Aware Hot-Expert Residency Mapping for 3.5D MoE Chiplet Inference
Pressure-aware two-timescale residency mapping of hot MoE experts across 3.5D memory tiers cuts simulated end-to-end latency by about 33–47% versus Hydra, MoEntwine, and PIMoE.
Reference graph
Works this paper leans on
-
[2017]
Singh, S., Ruwase, O., Awan, A
URL https://openreview.net/forum ?id=B1ckMDqlg. Singh, S., Ruwase, O., Awan, A. A., Rajbhandari, S., He, Y ., and Bhatele, A. A hybrid tensor-expert-data paral- lelism approach to optimize mixture-of-experts training. InProceedings of the 37th International Conference on Supercomputing, pp. 203–214, 2023. Skiadopoulos, A., Zhao, M., Gandhi, S., Norrie, T....
arXiv 2023
-
[2025]
NVIDIA TensorRT LLM Team
Accessed: 2025-10-14. NVIDIA TensorRT LLM Team. Scaling expert parallelism in tensorrt llm (part 1: Design and implementation of large-scale ep). https://github.com/NVIDIA/ TensorRT-LLM/blob/main/docs/source/b logs/tech_blog/blog4_Scaling_Expert_ Parallelism_in_TensorRT-LLM.md#ep-loa d-balancer, Sep 2025a. Accessed: 2026-03-02. NVIDIA TensorRT LLM Team. S...
2025
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.