{"id":"6e3c42b1-9536-4d3b-bf0f-4b2e7deb29a5","arxiv_id":"2411.16205","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A recipe for setting MH-MoE expert widths, counts, and top-k to match SMoE FLOPs is presented with small perplexity gains, but the recipe's core equation is wrong and the experimental parity is not exact.","lead":"This paper reconfigures Multi-Head Mixture-of-Experts so it is supposed to use the same compute and parameter count as standard sparse MoE, and reports small perplexity gains on a small language model. It is worth reading as a test of whether multi-head expert routing can improve quality for free, but the paper's cost-matching math has an arithmetic error.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The FLOPs-parity derivation is internally inconsistent: Eq.6 fails to divide activated-expert cost by h and conflicts with Eq.5, so Eq.8 and the claimed parity with SMoE are not established.","rationale":"I read the paper as making a constructive claim: a particular reparameterization of MH-MoE is FLOP- and parameter-matched to SMoE and improves perplexity. For that claim to hold, Eq.8 must be a correct consequence of a correct complexity model. The weakest point is the complexity model itself. Eq.5 explicitly defines the expert input/output dimension as d/h, and the natural count is 2B d dmoe k / h; Eq.6 reports (4B d dmoe - B d - B dmoe h)k. These cannot both describe the same architecture. The paper never resolves the discrepancy, and the experimental configurations do not have exact parity even under the paper's own Eq.6 and Eq.9. This is an internal inconsistency, not merely a difference from community consensus. The perplexity improvements are small and single-run, but the more fundamental issue is that the central advertised property, matching FLOPs and parameters, is unsupported. I agree with the reader's identification of this weakness, and the rejection verdict does not need to change. A single arithmetic recomputation of the published configurations under Eq.5/SwiGLU would settle the matter immediately.","tokens_in":7105,"tokens_out":15062,"duration_ms":132581,"concrete_test":"Run a one-line arithmetic check on the Table 1 configurations: recompute per-token scalar multiplications from Eq.5 with SwiGLU (three projections per expert) and also from Eq.6 for SMoE (d=768, dmoe=2048, top-1, E=8), MH-MoE h=2 (dmhmoe=768, top-2, E=40), and MH-MoE h=3 (dmhmoe=512, top-3, E=96), including head and merge layers. If the MH-MoE-to-SMoE FLOP ratios deviate from 1 by more than 1%, or if Eq.9 parameter parity fails by more than 1% for either h=2 or h=3, the central parity claim in the abstract and Eq.8 is false.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Eq.8 gives a recipe for FLOPs and parameter parity with SMoE, but Eq.8 rests on Eq.6, which is inconsistent with the paper's own expert definition in Eq.5. In Eq.5, each expert is a ReLU FFN with input/output dimension d/h and matrices W1 in R^{d/h x dmoe}, W2 in R^{dmoe x d/h}; one activated expert therefore costs 2B(d/h)dmoe scalar multiplications, i.e., 2B d dmoe k / h for k activated experts. Eq.6 instead writes the activated-expert cost as (4B d dmoe - B d - B dmoe h)k: the leading term has no factor 1/h and is a factor of 2 too large, and the subleading terms are not derived. Since Eq.8 is obtained by balancing this leading term against the SMoE leading term, the parity recipe is unsupported. The inconsistency is not merely formal: plugging the Table 1 configurations into Eq.6/8 gives unequal leading terms. For h=2, d=768, dmhmoe=768, k=2, the SMoE leading term 4B d dmoe is about 6.29M B, while the MH-MoE leading term 4Bd^2 + 4B d dmhmoe k is about 7.08M B, roughly 12.5% higher; the h=3 configuration has the same leading term and is also not equal. Eq.9 parameter counts are likewise off by about 1.6% (h=2) and 4.7% (h=3). Finally, all experiments replace the ReLU FFN of Eq.5 with SwiGLU, so even the two-matrix ReLU complexity count does not apply to the measured models. The advertised parity is therefore not established, and the perplexity improvements rest on an unsupported cost-matching claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a new implementation of Multi-Head Mixture-of-Experts (MH-MoE) that is claimed to keep both FLOPs and parameter parity with standard sparse Mixture-of-Experts (SMoE) models. The proposed method sets the MH-MoE expert intermediate dimension through Eq. (8) and the expert count through Eq. (9), and it reports lower validation perplexities than SMoE and fine-grained SMoE baselines on RedPajama, Wiki, and C4, including with a shared expert and 1-bit BitNet quantization. The paper also contains ablations on the head and merge layers.","tokens_in":93,"tokens_out":14233,"duration_ms":179021,"significance":"If the parity claim were correct, the paper would offer a simple recipe for converting a standard SMoE architecture into an MH-MoE architecture at equal compute and parameter cost with better perplexity, which would be of practical interest to the MoE community. The paper includes a useful breadth of experiments, including shared experts, 1-bit quantization, and ablations. However, the central FLOPs calculation is inconsistent with the paper's own expert definition, and the reported experimental configurations do not actually satisfy the proposed parity equations. The significance is therefore conditional on a correction that the current manuscript does not provide.","major_comments":[{"comment":"The complexity model is internally inconsistent. Equation (5) defines each expert with input/output dimension d/h and weight matrices W1 in R^{(d/h) x d_moe} and W2 in R^{d_moe x (d/h)}, so k activated experts cost 2B(d/h)d_moe k scalar multiplications. Equation (6) instead gives the activated-expert leading term as 4B d d_moe k, which is not divided by h and is a factor of two larger than a two-layer FFN count. Because Eq. (8) is derived by balancing this leading term against the SMoE leading term, the parity recipe is not established.","section":"Section 2.2, Eqs. (5) and (6)"},{"comment":"The experimental configurations do not satisfy the proposed Eq. (8). For the SMoE baseline with d=768, d_moe=2048, and k=1, Eq. (8) gives d_mhmoe = 2048 - 768/2 = 1664 for h=2, k=2, and d_mhmoe = 2048 - 768/3 = 1792 for h=3, k=3, while Section 3 reports d_mhmoe = 768 and 512. Using the paper's own Eq. (6) at the reported configurations gives an MH-MoE leading term of 7,077,888B versus 6,291,456B for the SMoE baseline, i.e., about 12.5% higher FLOPs. Parameter parity is also not exact: for h=3, d_mhmoe=512, E=96, Eq. (9) gives 26,345,472 parameters versus 25,165,824 for the SMoE baseline, about 4.7% higher.","section":"Section 2.3, Eq. (8), and Section 3"},{"comment":"All experiments use SwiGLU feedforward networks, as stated in Section 3, whereas Eq. (5) and Eq. (6) model a two-matrix ReLU FFN. A SwiGLU expert has three weight matrices, so the FLOPs count in Eq. (6) and the parameter count 2(d/h)d_mhmoe used in Eq. (9) do not apply to the evaluated models. The reported comparisons are therefore not controlled tests of the paper's FLOPs and parameter parity claim.","section":"Section 3, SwiGLU versus Eq. (5)"},{"comment":"Because the parity premise is not established, the lower perplexity values reported for MH-MoE in Tables 1 and 2 cannot be attributed to the multi-head mechanism; they may be due to unequal compute or parameter counts. The central conclusion in Section 4 that MH-MoE \"maintains both FLOPs and parameter parity\" while improving quality is not supported by the evidence presented.","section":"Section 3.1 and Section 4"}],"minor_comments":[{"comment":"The word \"ingratiating\" should be \"integrating\".","section":"Section 2.1"},{"comment":"The wording \"both outperform both vanilla SMoE models and fine-grained MoE models\" contains a doubled \"both\" and should be rephrased.","section":"Abstract and Section 4"},{"comment":"The ablation table values for MH-MoE (head=2) with head and merge layers are 11.46/9.98/14.89 in Table 4, while Table 1 reports 11.60/10.11/15.11 for what appears to be the same configuration; the discrepancy should be explained or corrected.","section":"Section 3.3 and Tables 4-5"},{"comment":"The sentence \"It is crucial to thoroughly investigate their contributions and understand their impact\" is missing a closing period.","section":"Section 3.3"}],"recommendation":"reject","confidential_remarks":"This is a short follow-up to the authors' own prior MH-MoE and BitNet work. The central derivational error in Eq. (6) is not a typo that can be corrected locally: the reported model sizes and top-k choices do not match the parity equation even when Eq. (6) is taken at face value, and all experiments replace the analyzed ReLU FFN with SwiGLU. Supporting the headline claim would require redoing the complexity analysis and rerunning the experiments under correctly matched configurations, which goes beyond a routine revision. I recommend rejection, while noting that a corrected and properly controlled study of MH-MoE could be a valid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one if you want a clean example of why FLOPs accounting matters. The new thing here is a recipe—Eq. 8 and 9—for setting MH-MoE's expert width, expert count, and top-k so that the model matches a standard SMoE in FLOPs and parameters, plus a BitNet compatibility check. If that recipe worked, it would be a modest, useful result for MoE practitioners. It probably doesn't, as written.\n\nThe paper does some things well. The ablations are clean: adding head and merge layers to a vanilla SMoE gives almost nothing, while the same layers in MH-MoE give a real drop in perplexity, and the head layer does most of the work. The empirical comparisons cover three datasets, two training horizons, and a shared-expert setting, and the MH-MoE numbers are consistently lower than both baselines. That consistency is worth something even at 768 hidden and 100k steps. The BitNet section is a bonus and the result there is also consistent.\n\nThe soft spot is the complexity analysis, and it's load-bearing. Eq. 5 defines each expert as a two-matrix FFN with input dimension d/h, so one expert costs 2B d dmoe scalar multiplications per token, and with k experts the leading term should carry a factor 1/h. Eq. 6 writes the activated-expert term as (4B d dmoe - B d - B dmoe h)k with no 1/h and a factor of 2 that is never justified. Eq. 8 is algebraically fine given Eq. 6's leading terms—d_mhmoe = d_moe - d/k does follow—but since Eq. 6 conflicts with the paper's own expert definition, the parity recipe is unsupported. Plugging the Table 1 configurations into the paper's own Eq. 6 gives MH-MoE about 7.08M B against SMoE's 6.29M B, a 12.5% gap, for both h=2 and h=3; parameter parity is off by 1.6% and 4.7%. And all experiments use SwiGLU FFNs, while the derivation assumes ReLU, so the count doesn't transfer directly either.\n\nA few smaller things: the gains are small and there are no error bars or seeds; the baselines' routing details are underspecified; and the paper leans heavily on the authors' own MH-MoE and BitNet papers. None of those is disqualifying by itself. The derivation is.\n\nMy take: the empirical pattern might survive a corrected accounting, but the advertised mechanism is not established. I'd send it to a serious referee because the problem is checkable and the experiments are cheap to reproduce, but my expectation would be rejection unless the authors rework the complexity model and re-check parity. I wouldn't cite it in its current form, though I might bring it to a reading group as a cautionary example of why cost models have to match the architecture.","headline":"The practical recipe is appealing, but the FLOPs-parity derivation has an internal factor-of-h error in the activated-expert cost, so the central claim is unsupported as written.","tokens_in":8095,"tokens_out":5170,"would_cite":false,"duration_ms":46454,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multi-head mixture-of-experts layer can be configured to match sparse MoE in FLOPs and parameters while improving language-model perplexity.","keywords":["mixture of experts","multi-head MoE","FLOPs parity","sparse MoE","language modeling","BitNet","Transformer","perplexity"],"falsifier":"Recompute the exact multiply counts for the reported configurations (e.g., head=3, $d=768$, $d_{\\mathrm{mhmoe}}=512$, top-3) including the head and merge projections; if the leading term is not the same as the baseline SMoE, the parity claim fails. A simpler empirical check is to time a training step of the MH-MoE model against the vanilla SMoE baseline at equal batch size; similar wall-clock time supports the claim, while a large gap would contradict it.","tokens_in":6878,"feed_emoji":"🧠","tokens_out":9199,"duration_ms":78558,"temperature":0.7,"pith_summary":"Multi-Head Mixture-of-Experts has previously shown quality gains, but its original parameter choices cost more FLOPs than a vanilla sparse MoE layer. This paper tries to establish that the gains can be kept without paying extra compute by shrinking each expert's intermediate dimension by $d/k$ and adding experts to restore the parameter count. In decoder-only Transformer pretraining, the resulting MH-MoE variants consistently obtain lower validation perplexity than both vanilla and fine-grained sparse MoE on the RedPajama, Wiki, and C4 validation sets at matched parameters and computation. The same ordering holds when all models are trained with 1-bit BitNet quantization. If the paper is right, multi-head routing is a drop-in, compute-neutral replacement for standard sparse MoE layers in language models.","feed_headline":"MH-MoE beats sparse MoE at the same FLOPs and parameter count","feed_subtitle":"New head-split expert routing matches vanilla and fine-grained sparse MoE on parameters and compute, yet lowers perplexity.","key_machinery":"The load-bearing object is the head/merge projection pair around a shrunken MoE layer. The input is projected by $W_{\\mathrm{head}}$, split into $h$ sub-token views of dimension $d/h$, passed through top-$k$ experts of intermediate dimension $d_{\\mathrm{mhmoe}}$, concatenated, and projected by $W_{\\mathrm{merge}}$. Equation 8, $d_{\\mathrm{mhmoe}} = d_{\\mathrm{moe}} - d/k$, sets the intermediate dimension so the first-order FLOP count of the whole layer equals the standard SMoE count $4B d d_{\\mathrm{moe}} k$, and Equation 9 derives the required expert count by matching $2 d d_{\\mathrm{moe}} E_{\\mathrm{moe}}$ to $2d^2 + 2 (d/h) d_{\\mathrm{mhmoe}} E_{\\mathrm{mhmoe}}$. The argument works because the two extra projections cost $4Bd^2$ in the leading term, exactly the amount the expert shrink removes.","core_discovery":"The paper claims that the correct sizing of a multi-head MoE layer makes its compute exactly equal to a standard sparse MoE layer. Given a baseline with intermediate dimension $d_{\\mathrm{moe}}$ and $k$ activated experts, setting $d_{\\mathrm{mhmoe}} = d_{\\mathrm{moe}} - d/k$ and increasing the expert count through Equation 9 keeps the leading FLOP term and the parameter count fixed. With this recipe, two configurations, head=2 with top-2 gating and 40 experts and head=3 with top-3 gating and 96 experts, beat the dense baseline, a vanilla 8-expert SMoE, and a fine-grained 16-expert SMoE at every reported checkpoint and validation dataset. The head-3 configuration is the strongest throughout, and the improvement persists in the 1-bit BitNet setting.","pith_inferences":["Beyond the paper's claims, the identity $d_{\\mathrm{mhmoe}} = d_{\\mathrm{moe}} - d/k$ suggests a family of FLOPs-neutral configurations beyond the two tested ones; any head count $h$ can be paired with $k$ activated experts as long as the formula is respected, which is a direct test of whether the quality gain scales with $h$.","Beyond the paper's claims, because the FLOPs argument counts multiply-accumulate operations, actual on-chip parity also depends on memory traffic and kernel efficiency of the two projections and the wider expert set; measuring wall-clock step time is a corollary the paper leaves implicit.","Beyond the paper's claims, the head layer's larger ablation gain suggests the benefit may come from creating multiple routed views before gating rather than from merging after experts; a dedicated study varying only the head projection would clarify the mechanism beyond the paper's reported ablations."],"forward_implications":["An existing sparse MoE model can be converted to MH-MoE without changing parameter count by applying Equation 8 and Equation 9, making the perplexity comparisons structurally fair.","With compute held fixed, more heads and more activated experts improve quality: the three-head, top-3 variant beats the two-head, top-2 variant on every reported dataset.","The recipe composes with shared-expert settings such as DeepSeek-style residual experts, where the MH-MoE advantage over fine-grained SMoE remains.","Under BitNet 1-bit training, MH-MoE still outperforms SMoE, indicating multi-head routing helps even when weights are aggressively quantized."],"supporting_citations":[{"why":"Introduces the original MH-MoE architecture and the head/merge projection design that this paper re-parameterizes.","marker":"[WHWW24]"},{"why":"Establishes the sparsely-gated MoE layer and top-k routing that serves as the baseline framework.","marker":"[SMM+17]"},{"why":"Supplies the position-wise FFN and multi-head design vocabulary used in the complexity analysis.","marker":"[VSP+17]"},{"why":"Defines the SwiGLU feedforward network used in every experimental model.","marker":"[Sha20]"},{"why":"Provides the shared-expert residual setting the paper augments onto all MoE baselines.","marker":"[DDZ+24]"},{"why":"Supplies BitNet, the 1-bit quantization method the paper tests for compatibility.","marker":"[MWM+24]"},{"why":"Provides the RedPajama training and validation data used in all experiments.","marker":"[Com23]"}],"fun_headline_variants":["MH-MoE matches sparse MoE cost, beats quality","Same FLOPs, same params, better perplexity with MH-MoE","Head-split experts: equal cost, lower perplexity","MH-MoE beats vanilla and fine-grained MoE at same compute","Multi-head MoE boosts quality without extra FLOPs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's equal-compute claim rests on its FLOPs count for the MH-MoE layer: the activated-expert term is written as $4Bd d_{\\mathrm{mhmoe}} k$ with no division by the head count $h$, even though each expert sees a $d/h$-dimensional input; if that count is wrong for the SwiGLU experts used in the experiments, the configurations are not actually FLOPs-matched.","fun_headline_variants_meta":{"raw":{"variants":["MH-MoE matches sparse MoE cost, beats quality","Same FLOPs, same params, better perplexity with MH-MoE","Head-split experts: equal cost, lower perplexity","MH-MoE beats vanilla and fine-grained MoE at same compute","Multi-head MoE boosts quality without extra FLOPs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000437,"raw_usage":{"total_tokens":2158,"prompt_tokens":818,"completion_tokens":1340,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":434,"completion_tokens_details":{"reasoning_tokens":1250}},"tokens_in":434,"tokens_out":1340,"duration_ms":10082,"temperature":1.0,"reasoning_tokens":1250,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:28:01.100277+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the exact multiply counts for the reported configurations (e.g., head=3, $d=768$, $d_{\\mathrm{mhmoe}}=512$, top-3) including the head and merge projections; if the leading term is not the same as the baseline SMoE, the parity claim fails. A simpler empirical check is to time a training step of the MH-MoE model against the vanilla SMoE baseline at equal batch size; similar wall-clock time supports the claim, while a large gap would contradict it.","supporting_citations":[],"review_version":1}