{"id":"fdeee36c-b36f-4ba3-8f9b-e904f772b29d","arxiv_id":"2602.19938","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Replicate-and-Quantize copies the busiest MoE expert as a quantized duplicate and compresses the least important expert, lowering a new Load-Imbalance Score by up to 1.4x while accuracy varies by roughly -1.2 to +3.0 points.","lead":"This paper proposes Replicate-and-Quantize, a training-free method that copies overloaded experts at lower precision and compresses underused experts to rebalance sparse mixture-of-experts LLM inference. It reports lower load-imbalance scores, but the mechanism for routing tokens to the replicas is not specified.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The method's central dispatch mechanism is undefined: Algorithm 3 adds a quantized replica while leaving the router unchanged, so the replica cannot receive tokens and the reported LIS reductions are not derivable from the described algorithm.","rationale":"The reader's weakest_assumption identifies the same load-bearing gap: with the router frozen, a replicated expert is not addressable, so token counts—and hence LIS—cannot change under the paper's own definitions. I reviewed the full text for any alternative mechanism. Section 2.5 only says replicas 'provide additional parallel capacity'; it never specifies how tokens are redirected. Figure 4's caption speaks of 'replicated experts mitigate routing bottlenecks' but again gives no dispatch rule. The empirical diagnostics (frequency vs. importance, batch-size effects) are plausible and could stand on their own, but the method's headline result depends entirely on this undefined step. The accuracy excursions in Table 4 further undercut the ±0.6% claim, but even a perfect accuracy table would not salvage the method without a defined replication-dispatch mechanism. A concrete implementation test can settle whether any token ever reaches the replica; if not, the LIS reductions in Table 4 cannot be explained by the described algorithm. I therefore agree with the reader's REJECT verdict.","tokens_in":23261,"tokens_out":3954,"duration_ms":35134,"concrete_test":"Implement Algorithm 3 exactly as specified on the Switch Transformer (8 experts) using the original pretrained router, and run the same evaluation harness used for Table 4 (e.g., MMLU, PIQA). Log, per layer, the number of tokens processed by the added quantized replica and compute LIS under Definition 1. If the replica receives zero tokens and LIS equals the raw model's LIS, the reported reductions are not reproducible by the described method, confirming the missing mechanism. If the replica does receive tokens, document the dispatch rule that made this possible and check whether it constitutes a router modification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.5, Algorithm 3, adds `quant(layer[re])` to each MoE layer and quantizes `layer[qe]`, explicitly stating 'this replication does not require any modification to the model’s routing mechanism.' Definition 1 defines LIS in terms of `n_{i,j}`, the number of tokens routed to expert j. With an unchanged router, the token-to-expert assignments are exactly those of the raw model; the replica is never selected, receives zero tokens, and LIS remains unchanged. Yet Table 4 reports large LIS reductions (e.g., 1.9709→1.3937 on GSM8K; 4.3504→3.2925 on PIQA). If the intended mechanism is that the serving system splits the heavy-hitter's token batch between the original and its replica, then the post-balancing token counts must be used in LIS, but the paper never defines this post-R&Q LIS or the dispatch rule. The only way the reported numbers could be produced is either a router change (contradicting the paper) or an unspecified system-level scheduler. Without this mechanism, the central claim of up to 1.4× imbalance reduction is unsupported, independent of the additional accuracy-report inconsistencies (e.g., Table 4 shows +3.0 on DeepSeek V2 Lite MMLU, outside the claimed ±0.6%).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript studies load imbalance in sparse Mixture-of-Experts (SMoE) LLMs at inference time. It defines a Load-Imbalance Score (LIS) as m·max_j n_{i,j}/(nk) and uses routing traces to argue that imbalance persists and grows with batch size, and that selection frequency is not a reliable proxy for expert importance. The proposed Replicate-and-Quantize (R&Q) framework identifies a heavy-hitter expert per layer (Algorithm 1), identifies a least-important expert via a Wanda-style score (Algorithm 2), appends a quantized replica of the heavy-hitter and quantizes the least-important expert (Algorithm 3), while leaving the router unchanged. Experiments on Switch Transformer, LLaMA-MoE, DeepSeek-MoE, and DeepSeek-V2 Lite report LIS reductions of up to about 1.4×, with accuracy claimed to be maintained within ±0.6%, plus streaming and ablation studies.","tokens_in":23587,"tokens_out":10114,"duration_ms":95826,"significance":"The diagnostic parts of the paper are potentially useful: LIS is a simple, interpretable statistic computed from routing traces, and the observation that heavily routed experts need not be the most important could inform post-hoc optimization. If R&Q worked as described, it would be a practical training-free intervention for deployment. However, the central result is not supported as written: the paper never specifies how an unchanged router can dispatch tokens to the added replica, so the reported LIS reductions are not derivable from Algorithm 3. Additionally, the accuracy claim in the abstract is contradicted by the paper's own Table 4. These are load-bearing problems, not presentation issues.","major_comments":[{"comment":"Algorithm 3 appends quant(layer[re]) to each MoE layer and quantizes layer[qe], while the text states that 'this replication does not require any modification to the model's routing mechanism.' Definition 1 defines LIS using n_{i,j}, the number of tokens routed to expert j. With an unchanged router, token-to-expert assignments are exactly the raw model's, the replica is never selected, and LIS cannot decrease. Yet Table 4 reports reductions such as 1.9709→1.3937 (Switch Transformer GSM8K) and 4.3504→3.2925 (DeepSeek V2 Lite PIQA). The paper never defines a post-R&Q LIS or a dispatch rule that splits the heavy-hitter's tokens between original and replica. Without such a mechanism, the reported reductions are not derivable from the described algorithm. If serving-level token splitting is intended, it must be specified and the LIS recomputation formalized.","section":"§2.5, Algorithm 3; Definition 1"},{"comment":"The abstract and Section 1 claim that accuracy is maintained within ±0.6% across all tasks, and Section 3.2 states that R&Q 'maintains or slightly improves' accuracy. Table 4 shows deviations far outside this bound: Switch Transformer (8 experts) MMLU +2.2, DeepSeek V2 Lite MMLU +3.0, and DeepSeek V2 Lite GSM8K -1.2. These entries are directly inconsistent with the headline claim. The bound must be corrected, or the table entries explained/recomputed; as written, the 'near-lossless' claim is contradicted by the paper's own data.","section":"Abstract; Section 1; Table 4"},{"comment":"The importance-score procedure is internally inconsistent. Section 2.4 says Wanda scores are sorted in ascending order and the bottom-s fraction is pruned, with 'lower means imply higher importance.' Algorithm 2 sorts scores in descending order, takes sorted_idx[1:int(Cin×s)], and sets IE_l = argmax_e IS[l][e]. These two descriptions select different experts, and the argmax conflicts with the stated interpretation of the score. Because Algorithm 2 determines which expert is quantized, the selective-quantization step and the validation in Table 2 are non-reproducible as stated.","section":"§2.4, Algorithm 2"}],"minor_comments":[{"comment":"The caption refers to a 'normalized gap value (Definition 1)', but Definition 1 defines LIS, not the expert activation gap described in the text. Define the gap metric explicitly.","section":"Figure 5"},{"comment":"The input 's' (sparsity factor) is declared but never used in the algorithm. Either remove it or explain its role.","section":"Algorithm 1"},{"comment":"The setup states that LIS is computed with batch size 1 and max_new_tokens=1, but Figure 3 reports LIS for batch size 32. Clarify how the batch-32 LIS is computed and whether it uses the same generation settings.","section":"§3.1 vs. Figure 3"},{"comment":"The row 'Replicate Heavy-Hitter Experts + Quantize' appears to describe the same combination as 'R&Q (Ours)'. Clarify the difference, e.g., whether the less-important expert is also quantized in the former.","section":"Table 6"},{"comment":"The citation for Dhasade et al. (2025) is titled 'Practical Federated Learning Without a Server', which does not match the 'MoEShard' work cited in Section 1. Verify and correct this reference.","section":"References"}],"recommendation":"reject","confidential_remarks":"The missing dispatch mechanism is not a minor gap: as written, Algorithm 3 cannot produce the reported LIS reductions, and the paper's own Table 4 contradicts the accuracy bound stated in the abstract. The LIS diagnostic and the frequency-vs-importance analysis are worth salvaging. If the authors can specify a concrete replica-aware dispatch rule, formally define post-R&Q LIS, and correct the accuracy inconsistencies, a substantially revised submission might be viable. In its current form, however, the central claim is not supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper has one genuinely useful diagnostic idea and one fatal gap. LIS is a reasonable, simple measure of routing skew, and the observation that imbalance grows with batch size and that frequency does not track importance is worth having. But the central claim—that you can replicate a heavy-hitter expert without touching the router and thereby reduce LIS—does not follow from the described algorithm. Algorithm 3 adds a quantized copy of the heavy expert and quantizes a low-importance expert, explicitly saying the routing mechanism is unchanged. Definition 1 defines LIS in terms of n_{i,j}, tokens routed to expert j. An unchanged router never sends tokens to the replica; the replica gets zero; the max token count is unchanged; LIS is unchanged. The paper never defines a post-R&Q LIS or a dispatch rule that would explain the Table 4 numbers. That is not a small omission; it is the load-bearing mechanism.\n\nWhat the paper does well: the empirical study of routing frequency versus importance is solid, and the Wanda-based importance identification seems reasonable (Table 2 gives some evidence). The observation about batch size is clear. If the authors had proposed an explicit scheduler that splits heavy-hitter tokens between original and replica, the approach would be a modest extension of existing replication and sharding work (MoEShard, GRACE-MoE, Lina).\n\nOther issues: the abstract's ±0.6% accuracy claim is contradicted by Table 4 (e.g., MMLU +2.2 for Switch 8, +3.0 for DeepSeek V2 Lite, GSM8K -1.2). No code or data are released. No comparison to the closest prior systems. The ablation table does not ground the method.\n\nRecommendation: I would desk reject this version. The missing dispatch mechanism means the headline result is unverifiable. The diagnostic section could be spun off as a short empirical note. If the authors respond with a precise description of how tokens are split, and with code, I would be happy to see it again.","headline":"The diagnostic observations are worth a look, but the claimed method is missing its central mechanism: with the router unchanged, the replicated expert receives no tokens and LIS cannot drop.","tokens_in":24139,"tokens_out":2385,"would_cite":false,"duration_ms":22466,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"R&Q claims that inference-time MoE load imbalance can be cut by up to 1.4x — without retraining or touching the router — by replicating the busiest expert as a quantized copy and quantizing a low-importance one.","keywords":["mixture-of-experts","load balancing","expert replication","quantization","inference efficiency","load imbalance score","large language models","training-free adaptation"],"falsifier":"On a fixed batch, trace per-expert token counts before and after applying R&Q with the router frozen; if the quantized replica receives zero tokens while the original heavy-hitter keeps all of its tokens, then the measured LIS cannot have moved, contradicting Table 4.","tokens_in":23105,"feed_emoji":"⚖️","tokens_out":6215,"duration_ms":54347,"temperature":0.7,"pith_summary":"Pre-trained sparse mixture-of-experts models route tokens unevenly: a small set of experts absorbs most of the load, and the skew grows with batch size. The paper proposes Replicate-and-Quantize (R&Q), a training-free, plug-and-play fix: in each MoE layer, add a quantized copy of the heaviest-loaded expert and aggressively quantize the least important expert, so memory stays roughly constant. To measure the effect, it defines the Load Imbalance Score (LIS), the ratio of the busiest expert's token count to the fair-share count; across Switch Transformer, LLaMA-MoE, DeepSeek-MoE, and DeepSeek-V2 Lite, R&Q lowers LIS by up to about 1.4x while keeping accuracy within ±0.6%. The core bet is that a busy expert need not be an important expert, which lets the method relieve the bottleneck without protecting the wrong parameters.","feed_headline":"One trick cuts MoE load skew by 1.4x, no retraining","feed_subtitle":"Quantized twin experts absorb the bottleneck while low-impact experts get compressed; accuracy holds within ±0.6%.","key_machinery":"The Load Imbalance Score (LIS), defined per layer as m·max_j n_{i,j}/(n k), converts token counts into a scalar skew measure: 1 is perfectly balanced, larger is more skewed. The operational pair is replication plus selective quantization — a quantized copy of the heavy-hitter is inserted to absorb extra tokens, while a pruning-scored low-importance expert is quantized to offset the memory cost. The router is not modified; the claim is that the replica itself changes the effective load distribution.","core_discovery":"R&Q identifies, per MoE layer, two experts: the heavy-hitter, found by counting tokens in a small calibration pass, and the least important expert, found by a pruning-based score that averages pruned weight-activation magnitudes. It appends a quantized replica of the heavy-hitter and quantizes the least important expert, leaving the router and all other weights untouched. The paper reports that, under this operation, LIS drops consistently — for example from 4.35 to 3.29 on PIQA for DeepSeek-V2 Lite, and from 1.97 to 1.39 on GSM8K for the 8-expert Switch Transformer — with per-task accuracy changes between -1.2 and +3.0 percentage points.","pith_inferences":["A real system must define how the unchanged router dispatches to the replica — the paper does not specify the split rule. A natural test is to route heavy-hitter tokens alternately (or by score threshold) between the original and the replica and verify that LIS drops as reported.","The calibration set can likely be replaced by a sliding window of recent tokens, enabling fully online hot-swapping of replicas as the input distribution drifts; the paper's streaming experiment is a step in that direction but still uses pre-identified quantized experts.","Because LIS only tracks the maximum load, R&Q could shift the hotspot rather than flatten the distribution. A stronger check, not in the paper, is to measure the full load profile (e.g., top-3 load or entropy) before and after replication.","The small accuracy gains on some tasks (e.g., +2–3% on MMLU) hint that quantizing a low-importance expert acts as structured noise that recalibrates routing; an ablation that quantizes a random expert instead would test whether the gain depends on the importance criterion."],"forward_implications":["If R&Q holds, serving engineers can apply it to existing MoE checkpoints without retraining or custom runtimes, and get more even expert utilization and lower latency at large batch sizes.","Because the paper shows imbalance worsens with batch size, the method's benefit is largest in exactly the high-throughput serving regime where tokens are batched heavily.","The decoupling of load from importance means load balancing and accuracy preservation are not in conflict at inference time: the busy expert is replicated (not sacrificed) and the low-importance expert absorbs the compression cost.","The method is architecture-agnostic, applying to top-1, top-2, and shared-expert routing schemes without changing the router, so it plugs into several deployed MoE families.","LIS gives a cheap per-layer diagnostic that operators could monitor online to detect routing skew as inputs drift."],"fun_headline_variants":["No retraining: replicate heavy experts, quantize the rest","MoE load skew drops 1.4x with zero training","Quantize a twin to balance MoE experts","Inference-only trick cuts expert load imbalance","Cut MoE imbalance 1.4x, accuracy within ±0.6%"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"R&Q assumes that adding a quantized replica of the heavy-hitter changes which tokens go where — but the paper leaves the router untouched and never says how tokens are split to the replica; if the router keeps sending every heavy-hitter token to the original expert, the reported LIS reduction cannot happen.","fun_headline_variants_meta":{"raw":{"variants":["No retraining: replicate heavy experts, quantize the rest","MoE load skew drops 1.4x with zero training","Quantize a twin to balance MoE experts","Inference-only trick cuts expert load imbalance","Cut MoE imbalance 1.4x, accuracy within ±0.6%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000276,"raw_usage":{"total_tokens":1524,"prompt_tokens":828,"completion_tokens":696,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":611}},"tokens_in":572,"tokens_out":696,"duration_ms":6882,"temperature":1.0,"reasoning_tokens":611,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T21:27:28.025660+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a fixed batch, trace per-expert token counts before and after applying R&Q with the router frozen; if the quantized replica receives zero tokens while the original heavy-hitter keeps all of its tokens, then the measured LIS cannot have moved, contradicting Table 4.","supporting_citations":[],"review_version":1}