{"id":"13b662f6-57f3-4777-828e-4573cdb2f8cc","arxiv_id":"2509.10513","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MoCE groups experts by k-means clusters of sequence embeddings and routes tokens within the chosen expert group, improving instruction-tuned LLM benchmark scores over PESC and domain-specialized baselines.","lead":"A new 'mixture of experts' design for large language models first sorts each request into a cluster, then routes tokens only to the experts in that cluster. It shows small but consistent gains over strong baselines on math, code, and reasoning benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sequence-level cluster transfer is unvalidated; if cluster assignments are arbitrary, the reported gains may reflect increased expert count rather than routing specialization.","rationale":"The reader's weakest assumption—cluster transfer from training to evaluation—is the most load-bearing uncertainty in the paper. The method's novelty and claimed advantage over PESC rest on the sequence-level routing being meaningful. If that routing is arbitrary, the method reduces to a capacity increase with a fixed partition, and the central claim of 'expert specialization via clustering' collapses. The paper's Figure 5 is suggestive but insufficient: 100 samples per benchmark, no purity metrics, no statistical test. The reader's conditional verdict is appropriate: the mechanism is plausible and the results are consistent, but the absence of variance reporting and the unvalidated cluster transfer leave the magnitude and source of gains uncertain. I agree with the reader's assessment and recommend keeping the verdict conditional, with the concrete test above as a condition for acceptance.","tokens_in":90,"tokens_out":4370,"duration_ms":51412,"concrete_test":"Retrain MoCE with random cluster labels (preserving group sizes) and compare to reported MoCE and PESC on all eight benchmarks. If the random-label variant matches MoCE's performance (or stays close), the sequence-level routing is not the source of the gains; if it drops to PESC levels, the routing is real. Additionally, compute cluster purity for a larger sample of each evaluation benchmark (e.g., all GSM8K and BBH instances) and compare against a random-assignment baseline to quantify whether centroids transfer beyond the 100-sample visualization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that hierarchical dual-stage routing improves generalization—depends entirely on k-means centroids, fit on the training mixture (SlimOrca, Magicoder, MetaMathQA), transferring to evaluation sequences (§3.2–3.3). If an evaluation prompt is assigned to the wrong expert group, the sequence-level stage adds noise rather than specialization. The only evidence of transfer is Figure 5, a qualitative visualization of 100 samples per benchmark, with no quantitative measure of cluster purity, no comparison against random assignment, and no analysis of embedding distribution shift between training and evaluation. Without such evidence, the sequence-level stage could be a fixed random partition: experts still specialize on whatever training data they see, but inference-time routing becomes effectively arbitrary. In that case, the improvements in Table 1 might be due to MoCE's larger total expert bank (4 clusters × 4 experts = 16) rather than to meaningful routing, especially since Table 3 shows that simply increasing expert count improves performance. The paper does not report PESC's number of experts or any statistical significance/variance, so the margin cannot be cleanly attributed to the proposed mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Mixture-of-Clustered-Experts (MoCE), a sparse MoE architecture for instruction tuning in which experts are organized into groups. A dual-stage routing mechanism first assigns each input sequence to an expert group via k-means clustering over encoder-based sequence embeddings, then performs standard top-k token-level routing within the selected group. The method is implemented in an adapter-based, upcycled setting and evaluated on LLaMA2-7B, Mistral-7B, and Qwen-7B across mathematics, code, knowledge, and reasoning benchmarks. The authors report consistent improvements over adapter-based baselines, including PESC and domain-specialized PESC models, and provide ablations on routing strategies and expert-count scaling.","tokens_in":14223,"tokens_out":2306,"duration_ms":22359,"significance":"If the reported gains are robust, MoCE offers a simple, practical recipe for improving expert specialization and cross-domain generalization in instruction-tuned MoE models, with evidence across three model families and multiple public benchmarks. The paper's strengths include evaluation on standard benchmarks, ablations of both routing stages, and experiments with different backbone models and embedding models. The main weakness is that the central claim—that hierarchical cluster-based routing, rather than additional parameters or selection artifacts, drives the improvements—is not yet established with the necessary controls and statistical evidence.","major_comments":[{"comment":"The cluster count is partly selected after inspecting test performance, which blurs selection and validation. The text states that the elbow method identifies four and seven as the optimal counts for Instructor and E5 embeddings, respectively, and Figure 4 then shows that these same counts 'yield the best overall performance' on the evaluation benchmarks. Since the evaluation benchmarks are used to confirm the elbow-chosen count, the claim that the elbow method is validated as a reliable approach is circular. Please report the elbow-selected counts based only on training-set embeddings, and treat any performance-based adjustment as model selection on the test set, with the corresponding caveat about optimistic bias.","section":"§5.4, Figure 4"},{"comment":"The transfer of cluster centroids from the training mixture to evaluation inputs is not quantitatively validated. The centroids are fit on SlimOrca, Magicoder, and MetaMathQA and then used to assign evaluation sequences, but the only evidence is the qualitative 100-sample visualization in Figure 5. There is no measure of cluster purity, assignment stability, or comparison against random assignment of sequences to expert groups. Since the sequence-level stage is the defining contribution of MoCE, this unvalidated transfer is load-bearing: if assignment on evaluation inputs is effectively arbitrary, the gains in Table 1 could arise from the larger total expert bank rather than from meaningful routing. Please add a quantitative analysis of cluster-transfer quality, such as agreement between train and test embedding geometries, cluster-size balance on evaluation sets, or an ablation that randomizes the cluster assignment while holding the number of experts fixed.","section":"§3.2–3.3, Figure 5"},{"comment":"The comparison against PESC is confounded with total expert count. Table 3 shows that increasing the number of experts per cluster from 1 to 4 (i.e., from 4 to 16 total experts) improves the average score from 35.55 to 36.76, and the paper does not report how many experts PESC uses. If PESC uses fewer than 16 experts, the gains attributed to hierarchical routing in Table 1 may partly reflect parameter count. Please report the total expert count for PESC and, if feasible, include a PESC baseline with a matched total number of experts, or otherwise demonstrate that the improvement is not explained by the expert-count scaling shown in Table 3.","section":"Table 1 and Table 3"},{"comment":"All reported scores are single-run point estimates without error bars or significance tests. Many of the comparisons involve small margins (for example, MMLU-Pro differences of 0.3–0.9 points in Table 1), so it is unclear whether the improvements are statistically reliable. At minimum, please provide multiple seeds or bootstrap confidence intervals for the main comparisons against PESC and for the ablations in Table 4. This is important for the central claim of consistent superiority.","section":"Table 1, Figure 3, Table 4"}],"minor_comments":[{"comment":"The notation for the gating function is inconsistent: R(x)_i is used in Eq. (1) but TopK(R(x)_i, k) is written with the index as a subscript; please clarify the indexing convention.","section":"§3.1, Eq. (1)-(3)"},{"comment":"The caption says 'Comparative evaluation results with dense and MoE models' but the table only includes dense models of similar size and MoE-based models; it is not clear which rows are dense and which are MoE. Please label the model types explicitly.","section":"Table 2 caption"},{"comment":"The sentence 'MoCE demonstrates superior performance not only over general baselines but also over models individually optimized for specific domains' is stated before the domain-specialized results are shown; consider moving this claim to after the relevant comparison.","section":"§5.1, first paragraph"},{"comment":"The ablation 'w/o Clustering' is not precisely defined. If clustering is disabled, how is the expert group selected? Please specify the alternative assignment used in this ablation.","section":"Table 4"},{"comment":"The subfigure labels are numbered '(2)' twice and skip '(3)'; please correct the numbering.","section":"Figure 5"},{"comment":"The adapter dimension, number of experts, and top-k are stated, but the load-balancing loss is only mentioned in §4.1; please specify its coefficient or provide the exact loss formulation used.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The paper does not release code or trained models, which makes it harder to verify the exact cluster assignments and ablations. Given that the central claim depends on the transfer of cluster centroids, I would encourage the editors to request code release or, at minimum, detailed cluster-assignment statistics as part of the revision. The paper also does not seem to compare against recent sequence-level routing methods beyond BTX; a broader related-work comparison would strengthen positioning, though this is not a blocker."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competently executed empirical paper. The dual-stage routing idea — k-means clustering at the sequence level, then token-level top-k inside the chosen group, built on PESC adapters — is a modest but real extension of the MoE-for-instruction-tuning line. The ablations support the mechanism better than the stress-test note suggests.\n\nWhat is actually new: the specific combination of cluster-conditional expert groups with a one-to-one cluster-to-group map and token-level routing inside the group. Gou et al. have cluster-conditional LoRA experts for vision-language, but they don't do the hierarchical token-level stage. Task-level MoE exists, but it doesn't use latent embeddings to define groups. So the architecture is not a major departure, but it's a reasonable new data point.\n\nThe paper does several things well. The ablation in Table 4 is the most important: removing either stage degrades performance, and the 'w/o Clustering' row keeps the expert count fixed, which directly answers the 'more experts' confound that the stress-test worried about. They also scale experts within groups (Table 3), test on Mistral and Qwen, and compare against publicly released dense and MoE models. The writing is clear, and the limitations section is honest.\n\nThe soft spots are mostly about reporting discipline. No code release, no error bars or significance tests, and the elbow-chosen cluster count is partly validated against test performance (Figure 4) — the text says the elbow count 'yielding the best overall performance,' which is selection on the test set. The cluster-transfer evidence is thin: a 100-sample visualization per benchmark, no cluster purity numbers, no random-assignment baseline. The stress-test's concern that routing could be arbitrary is not fully closed, though the 'w/o Clustering' ablation does mitigate it. I would like to see a random cluster baseline and quantitative transfer metrics.\n\nAlso note the generalization claim is real but modest: math and code benchmarks are close to the training mixture (MetaMathQA, Magicoder); the general knowledge/reasoning gains are smaller. The paper does not oversell this much.\n\nBottom line: for anyone working on parameter-efficient MoE for instruction tuning, this is a useful, solid paper that deserves serious refereeing. The mechanism is plausible, the ablations are informative, and the gains are consistent. It needs code or at least variance reporting and a cleaner cluster-count selection story, but I would send it to reviewers.","headline":"Solid empirical MoE paper with a plausible dual-stage routing mechanism; the ablations hold up better than the stress-test suggests, but reporting gaps keep the effect size uncertain.","tokens_in":14720,"tokens_out":2710,"would_cite":true,"duration_ms":25555,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that adding a sequence-level clustering stage before token-level expert selection lets Mixture-of-Experts models specialize expert groups and generalize across math, code, knowledge, and reasoning tasks.","keywords":["mixture of experts","instruction tuning","expert specialization","dual-stage routing","k-means clustering","sequence-level routing","parameter-efficient fine-tuning","large language models"],"falsifier":"Permute the one-to-one mapping from cluster IDs to expert groups, retrain only the router on the same data, and evaluate on the same benchmarks; if average performance does not fall below the standard cluster mapping, the clustering content is not driving the gains.","tokens_in":13804,"feed_emoji":"🎯","tokens_out":9571,"duration_ms":84424,"temperature":0.7,"pith_summary":"The paper is trying to establish why a sparse Mixture-of-Experts model fails to specialize when instruction-tuning data is highly heterogeneous: token-level routing by itself cannot group inputs by task or knowledge requirement. MoCE's fix is a dual-stage router—first a k-means cluster of the whole input embedding picks one expert group, then a token-level top-k gate picks experts inside that group. The paper argues this forces expert groups to specialize on distinct input clusters while keeping the computational cost of ordinary MoE, and reports consistent gains over adapter-based MoE baselines and over models trained on a single domain. The significance, if the claim holds, is that expert specialization and cross-task generalization can be induced by a simple precomputed clustering without increasing active parameters.","feed_headline":"Cluster-first routing beats single-domain MoE baselines","feed_subtitle":"Grouping inputs by clusters before token routing lifts average benchmark scores at no extra compute.","key_machinery":"The load-bearing object is the grouped MoE layer with a two-stage router. In the first stage, a fixed encoder embedding model (E5 or Instructor) represents the full input sequence; k-means assigns it to one of $M$ clusters, and a one-to-one mapping activates the corresponding expert group $G_{\\alpha}$. In the second stage, a group-specific gating function $R_{\\alpha}$ applies the standard top-k sparse gating among the group's experts, producing $y = \\sum_i \\mathrm{TopK}(R_{\\alpha}(x)_i,k)\\,A_{G_{\\alpha},i}(x)$. In the adapter-based setting used here, each expert is a low-rank adaptation (an up/down projection pair with a residual connection) on a frozen feed-forward network. A variant sums in general experts and a general router, pooling knowledge from all clusters. The machinery forces different expert groups to be exposed only to inputs whose embeddings fall in their cluster, while top-k selection keeps token-level flexibility.","core_discovery":"MoCE's central claim is that a two-level routing hierarchy—sequence-level group allocation followed by token-level expert selection—makes experts specialize by knowledge cluster and generalize to out-of-domain inputs better than token-only MoE. On the paper's experiments, MoCE-E5 averages 37.99 and MoCE-Instructor 37.61 across math, code, knowledge, and reasoning benchmarks, against 35.62 for PESC, 32.44 for BTX (A), and 32.26 for LLaMA-Adapter; gains are largest on GSM8K (41.93 vs 33.21 for PESC) and HumanEval@1 (19.28 vs 16.00). Domain-tuned PESC models score higher than MoCE only within their narrow domain and fall behind elsewhere, which the paper reads as evidence that MoCE balances specialization with generalization. The method transfers to Mistral and Qwen backbones and, with four or seven clusters chosen by an elbow rule, beats publicly released dense and MoE chat models of comparable size.","pith_inferences":["Because the clustering encoder is frozen and external, a natural test is to replace it with the MoE model's own hidden states or with trainable centroids; that would isolate how much of the gain comes from the routing signal itself versus the fixed group structure.","The architecture could be deployed at lower memory cost by loading only the expert group assigned to the current cluster during inference; the paper does not test this, but the one-to-one cluster-to-group mapping makes it a direct extension.","The authors' own stated limits—adapters only, no multilingual or broader-domain evaluation—mean the specialization claim is currently demonstrated for English math, code, knowledge, and reasoning tasks; extending to full FFN MoE layers and other languages would test whether the grouping benefit is generic."],"forward_implications":["MoCE's dual-stage routing improves average performance over adapter-based MoE baselines on all four task categories without activating more experts per token.","Domain-specialized PESC models are matched or beaten by MoCE in their own domains on math and code while MoCE retains general-task performance, suggesting grouping avoids sacrificing cross-domain transfer.","The same cluster-group architecture transfers to Mistral and Qwen model families with consistent gains, so the mechanism is not tied to one backbone.","Scaling the number of experts per group increases accuracy on code, math, and general benchmarks while keeping the number of activated experts at top-2.","Adding a set of general experts alongside the clustered groups lifts average performance, so domain-grouped knowledge and general knowledge combine productively."],"supporting_citations":[{"why":"Supplies the adapter-based PESC expert structure, the main efficiency mechanism, and the central baseline that MoCE must outperform.","marker":"Wu et al. (2024a)"},{"why":"Provides E5 sequence embeddings, one of the two encoders used to build k-means clusters for sequence-level group routing.","marker":"Wang et al. (2022a)"},{"why":"Provides Instructor sequence embeddings, the other encoder used for cluster construction and the MoCE variant.","marker":"Su et al. (2023)"},{"why":"Gives the sparse-upcycling recipe for initializing MoE experts from a pretrained dense model.","marker":"Komatsuzaki et al. (2023)"},{"why":"BTX, the branch-train-mix baseline that routes among independently trained experts, is compared against MoCE.","marker":"Sukhbaatar et al. (2024)"},{"why":"Supplies soft-merging and top-k parameter-efficient MoE techniques that MoCE uses in token-level routing and comparisons.","marker":"Zadouri et al. (2024)"},{"why":"Defines the sparse top-k gating mechanism that MoCE's token-level stage reuses.","marker":"Shazeer et al. (2017)"},{"why":"Provides the evaluation harness used to measure the reported benchmark scores.","marker":"Gao et al. (2024)"}],"fun_headline_variants":["MoCE: cluster-first routing generalizes beyond single-domain MoE","Two-stage MoE routing boosts expert specialization and scope","Group-then-token MoE beats domain-tuned baselines","MoCE: sequence-level clustering then token selection lifts MoE","Cluster-based expert specialization gains without extra compute"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The first routing stage assumes the clusters inferred from training-sequence embeddings keep their meaning on test inputs; if a new benchmark's examples do not sit near the training cluster centers, the cluster-to-group mapping is effectively arbitrary.","fun_headline_variants_meta":{"raw":{"variants":["MoCE: cluster-first routing generalizes beyond single-domain MoE","Two-stage MoE routing boosts expert specialization and scope","Group-then-token MoE beats domain-tuned baselines","MoCE: sequence-level clustering then token selection lifts MoE","Cluster-based expert specialization gains without extra compute"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001037,"raw_usage":{"total_tokens":4353,"prompt_tokens":922,"completion_tokens":3431,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":3349}},"tokens_in":538,"tokens_out":3431,"duration_ms":21881,"temperature":1.0,"reasoning_tokens":3349,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:33:04.732698+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Permute the one-to-one mapping from cluster IDs to expert groups, retrain only the router on the same data, and evaluate on the same benchmarks; if average performance does not fall below the standard cluster mapping, the clustering content is not driving the gains.","supporting_citations":[],"review_version":1}