{"id":"87aeb9e9-b3ba-4e10-a6e0-04e52aeb63d5","arxiv_id":"2506.23266","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Sub-MoE compresses MoE LLMs by K-means clustering of experts plus frequency-weighted merging of right singular vectors after a shared SVD, and claims 96 and 86 percent retained accuracy at 25 and 50 percent expert reduction on Mixtral-8x7B.","lead":"This paper describes Sub-MoE, a training-free method for compressing Mixture-of-Experts language models by grouping similar experts and merging them in a shared subspace. It reports retaining 94 to 96 percent of Mixtral-8x7B performance while removing a quarter of its experts, which matters because MoE models are expensive to store and serve.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"At full rank the subspace merge is algebraically identical to frequency-weighted direct averaging; the paper never states a truncation rank for the base method, so the claimed conflict-reduction mechanism is unsupported.","rationale":"The reader's weakest_assumption is the same one I would flag. The mathematical identity is airtight: a full-rank exact SVD makes Eq. (8) reproduce Eq. (4) with α_i = f_i/Σf, so the shared U does no conflict-reduction work. The paper's own text never says the base method truncates; only the extended Sub-MoE† does. The ablations compare Union SVD against an undefined 'Vanilla SVD', not against direct averaging, so they cannot demonstrate the subspace mechanism. Calibration transfer and numerical inconsistencies are secondary. The proposed check would settle the question without new theory: if the identity holds and scores match direct averaging, the novelty reduces to clustering plus frequency weighting; if truncation is intended, the authors must state the rank and show that truncation, not averaging, is responsible. Thus the verdict remains CONDITIONAL rather than ACCEPT or UNCHANGED.","tokens_in":14776,"tokens_out":6685,"duration_ms":73876,"concrete_test":"Reimplement Eqs. (5)–(8) for the Mixtral-8x7B 8→6 configuration with no rank truncation, and compare the resulting W_merged elementwise to Σ_i f_i W_i / Σ_i f_i (Eq. 4 with α_i = f_i/Σf). If they agree to numerical precision, rerun Table 1's Sub-MoE row using the direct frequency-weighted average; identical scores would prove the subspace mechanism is inert and the reported advantage is due to clustering and frequency weighting only. If the authors instead intend truncated SVD, repeat with a stated rank or energy threshold and compare against direct averaging at the same per-expert size; this would establish whether subspace merging adds anything beyond the weighted average.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Equations (5)–(8) define the mechanism: joint SVD of the group's concatenated weights, frequency-weighted merging of the V blocks, and reconstruction as W_merged = UΣV_merged^T. If this SVD is exact (full rank), then each expert satisfies W_i = UΣV_i^T, and Eq. (7) gives W_merged = UΣ(Σ_i f_i V_i^T)/(Σ_i f_i) = (Σ_i f_i W_i)/(Σ_i f_i). That is precisely the vanilla frequency-weighted average of Eq. (4); the shared U is only a basis change and performs no conflict reduction. The paper never specifies the rank r for the base Sub-MoE method: Section 3.3 presents no truncation, and the implementation note that singular values are multiplied into the shared U is consistent with full rank. Truncation appears only for Sub-MoE† in Section 3.4 (Eqs. 10–12). Therefore the central novelty—that merging in the shared subspace reduces parameter conflicts—is either algebraically vacuous (full rank) or empirically unverified (unspecified truncated rank). The reported gains over HC-SMoE could then come entirely from the clustering and frequency-weighting recipe, not from subspace alignment. None of the ablations isolates this: the 'Vanilla SVD' baseline in Table 3(B) is not described as full-rank versus truncated and is not the direct-average baseline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Sub-MoE, a training-free MoE compression method with two stages: adaptive K-means clustering of experts based on cosine similarity of their outputs, and 'subspace expert merging' that applies a joint SVD to concatenated expert weights, merges the per-expert V components with frequency weights, and reconstructs a single merged expert from shared U, Sigma, and the merged V. The method is evaluated on Mixtral-8x7B, Qwen1.5-MoE-A2.7B, Qwen3-30B-A3B, and DeepSeek-MoE-16B, against expert pruning and merging baselines, and is extended to Sub-MoE-dagger with activation-aware truncated SVD for intra-expert compression. The paper reports substantial gains, such as maintaining 0.58 average accuracy with 4 experts on Mixtral versus 0.51 for HC-SMoE, and includes ablations over clustering settings, merging settings, and calibration size.","tokens_in":15044,"tokens_out":5689,"duration_ms":58492,"significance":"If the subspace-alignment mechanism were genuine and fully specified, Sub-MoE would be a practically valuable training-free MoE compression method, and the broad empirical coverage across four models and multiple baselines is a real strength. The paper also provides useful ablations and an analysis of calibration sample size. However, the central algebraic claim is currently under-specified: at full rank the proposed subspace merge reduces exactly to frequency-weighted direct averaging, so the claimed conflict-reduction role of the shared U is not established. With a clarified truncation rank and an ablation that isolates the subspace effect, the empirical results could support a useful contribution; as written, the mechanism is either vacuous or unverified.","major_comments":[{"comment":"The manuscript never states the rank r or a truncation step for the base Sub-MoE method. If the SVD in Eq. (5) is exact (full rank), then, under the intended horizontal concatenation, each expert satisfies W_i = U Sigma V_i^T, and substituting Eq. (7) into Eq. (8) gives W_merged = U Sigma (sum_i f_i V_i^T)/(sum_i f_i) = (sum_i f_i W_i)/(sum_i f_i), which is exactly the frequency-weighted direct averaging of Eq. (4). In that case the shared U performs no conflict reduction; it is merely a basis change. Section 3.3 describes no truncation, and the implementation note that singular values are multiplied into U is consistent with full rank. Thus the paper's central mechanism is either algebraically equivalent to vanilla weighted averaging or, if a truncated SVD is actually used, empirically unverified because the truncation rank is not reported. Please specify the rank and truncation used in every reported experiment, and add an ablation comparing full-rank subspace merging against direct frequency-weighted averaging.","section":"Section 3.3, Eqs. (5)-(8)"},{"comment":"The concatenation geometry is inconsistent and load-bearing. The text says 'concatenate their weight matrices vertically', but the notation U in R^{O x r}, V in R^{r x nI}, with V partitioned into n blocks, only works for horizontal concatenation [W^(1), ..., W^(n)]. Under vertical concatenation, the SVD of the nO x I matrix has a single common V^T, and one would need to partition U into row blocks U_i to reconstruct W_i = U_i Sigma V^T, so the per-expert V_i used in Eq. (7) is not well-defined. Please clarify the concatenation direction and the corresponding block structure; this is essential for reproducing the method and for the correctness of Eqs. (7)-(8).","section":"Eq. (5), Section 3.3"},{"comment":"The ablation does not isolate the proposed subspace-sharing mechanism. The 'Vanilla SVD' row is not defined in the experimental setup, and no baseline of direct frequency-weighted averaging is reported. Without a comparison between full-rank Sub-MoE and direct averaging, and without truncated-rank variants of the joint SVD, the reported gains over HC-SMoE cannot be attributed to subspace alignment; they could come entirely from the expert clustering and frequency-weighting recipe. Please define 'Vanilla SVD', add a direct frequency-weighted average baseline, and vary the SVD truncation rank to show that the shared-subspace operation itself contributes to the results.","section":"Table 3(B), Section 4.4"},{"comment":"The adaptive allocation of cluster counts is underspecified. The paper states that experts on multiple MoE layers are jointly clustered 'while maintaining a target overall compression ratio' and that the process 'automatically' determines the number of clusters per layer, but no algorithm, objective, or mapping from the target compression ratio to the layer-wise cluster count k_l is given. Table 3(A) reports 1-, 2-, and 3-layer variants with different outcomes, yet the mechanism for choosing layer-wise cluster counts is not described, which prevents reproduction of the method's core compression schedule.","section":"Section 3.2, Multi-layer Adaptive Allocation"}],"minor_comments":[{"comment":"The reported retained accuracy for Mixtral-8x7B is inconsistent: the abstract says 96% and 86% for 25% and 50% expert reduction, while the introduction says 94% and 87%, and Table 1 implies 0.64/0.67 = 95.5% and 0.58/0.67 = 86.6%. Please align these numbers.","section":"Abstract and Introduction"},{"comment":"The model list in Section 4.1 mentions Qwen3-235B-A22B, but Table 1 reports results for Qwen3-30B-A3B. Please clarify which Qwen3 model was actually evaluated and correct the setup description.","section":"Section 4.1 and Table 1"},{"comment":"The caption says 'Performance of Sub-MoE and MC-SMoE', but the proposed method rows are labeled Sub-MoE-dagger; please adjust the caption to match the table entries.","section":"Table 2 caption"},{"comment":"There is a typo in 'expert-specific V T -matrixs'; it should read 'V^T matrices'.","section":"Section 1, paragraph 2"},{"comment":"The sentence 'while still preserving capabilities from experts' is incomplete; it should say something like 'while still preserving the specialized capabilities of less frequent experts'.","section":"Section 3.3, paragraph after Eq. (7)"},{"comment":"Reference [45] is cited for Qwen3-MoE but points to the Qwen2.5 technical report; please verify and update the citation.","section":"Reference [45]"}],"recommendation":"major_revision","confidential_remarks":"The paper reports broad and internally consistent empirical gains, and the promise of code release is welcome. My main concern is the gap between the advertised subspace-alignment mechanism and the actual algebra: at full rank, the proposed merge is exactly frequency-weighted direct averaging. If the authors can supply the truncation rank used in the base method and an ablation that isolates the shared-subspace effect, the paper may be repairable as a major revision. If the method is full-rank in all reported experiments, the contribution should be reframed as clustering plus frequency-weighted merging, with the significance claims adjusted accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — read this one for the gap between its story and its equations. The paper ships a broad empirical recipe: cluster MoE experts by output similarity, joint-SVD each cluster's concatenated weights, frequency-average the V blocks, reconstruct. On Mixtral, Qwen1.5, Qwen3, and DeepSeek MoE models it reports consistent gains over MC-SMoE and HC-SMoE at 25% and 50% expert reduction. That empirical surface is wide and the gains are large enough to take seriously. But the central mechanism — merging in a shared subspace reduces parameter conflicts — is unsupported as written. If the SVD is full rank, Eqs. (5)-(8) collapse to W_merged = sum f_i W_i / sum f_i, exactly the direct frequency-weighted average of the original experts. The shared U is only a basis change. The paper never states a truncation rank for the base Sub-MoE method, and the implementation note that singular values are folded into U is consistent with full rank; truncation appears only for Sub-MoE†. So either the subspace story is algebraically vacuous or the rank is an unstated free parameter. That's load-bearing, not cosmetic.\n\nWhat is genuinely new is the combination — output-similarity K-means plus frequency-weighted V merging — and the ablations in Table 3 are reasonably informative, including the calibration-size curve. The citation pattern looks fine; prior merging work is cited and the positioning over HC-SMoE/MC-SMoE is clear. Minor soft spots need fixing: abstract says 96/86 while body says 94/87; intro names Qwen3-235B-A22B but Table 1 uses Qwen3-30B-A3B; D2-MoE is discussed as a competitor without numeric comparison; no code or error bars; and 128 WikiText-2 samples are assumed to transfer to all eval tasks. None is fatal by itself.\n\nWho gets value: anyone working on training-free MoE compression. The recipe may transfer and is worth testing. But before trusting the subspace-alignment framing, I'd want the authors to state the rank for the base method, add the direct weighted-average baseline to the ablation, and release code. As is, I'd send it to review rather than desk-reject — the problem matters and the empirical work is broad — but reviewers should hold the mechanism to the algebra.","headline":"Broad, plausible empirical recipe for MoE compression, but the central subspace-merge mechanism is either algebraically identical to weighted averaging or rests on an unstated truncation rank.","tokens_in":15629,"tokens_out":4121,"would_cite":false,"duration_ms":42104,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sub-MoE claims that merging experts in a shared SVD subspace preserves specialized knowledge, keeping 94–96% of Mixtral-8x7B accuracy after dropping 25% of experts.","keywords":["mixture of experts","expert merging","singular value decomposition","subspace alignment","LLM compression","frequency-aware merging","K-means clustering","parameter conflict"],"falsifier":"Run Sub-MoE on a cluster with a full-rank joint SVD, meaning no singular values are truncated, and compare the reconstructed merged weight matrix to the direct frequency-weighted average of the original expert matrices: if the two matrices are equal, the claimed subspace mechanism has no effect beyond clustering and frequency weighting. Separately, re-run clustering and merging with a held-out calibration set from a different domain and check whether the reported 94–96% and 86–87% retention figures on Mixtral persist.","tokens_in":14521,"feed_emoji":"🧠","tokens_out":6048,"duration_ms":59775,"temperature":0.7,"pith_summary":"Sub-MoE is a compression method for mixture-of-experts language models that replaces several expert networks with one merged expert, without fine-tuning. The paper's central claim is that expert weights should not be averaged directly; instead, experts in the same functional cluster are first projected into a common subspace via a joint singular value decomposition, and only the expert-specific right-singular components are frequency-weighted and merged. This is said to reduce parameter conflicts caused by expert specialization and to preserve more specialized knowledge than standard expert merging or pruning. On Mixtral-8x7B, the method reports keeping 94–96% of zero-shot accuracy with 25% of experts removed and 86–87% with 50% removed, with similar patterns on Qwen and DeepSeek MoE models. A calibration set of 128 WikiText-2 samples is needed for clustering and merging.","feed_headline":"Merging experts in a shared SVD subspace keeps 96% of accuracy","feed_subtitle":"Sub-MoE drops a quarter of Mixtral's experts with a 4–6% accuracy cost; half of experts still retains 86–87%.","key_machinery":"The central mechanism is Experts Union Decomposition: the joint SVD of the concatenated expert weight matrices in one cluster, yielding a shared left-singular matrix $U$, a singular-value matrix $\\Sigma$, and per-expert right-singular matrices $V^{(i)}$. The method proceeds in three steps: K-means clustering of experts by cosine similarity of their outputs, frequency-based merging of the $V^{(i)}$ matrices using router activation counts, and reconstruction of one weight matrix per cluster from $U$, $\\Sigma$, and the merged $V$. The shared $U$ is the component claimed to eliminate conflicting parameters, while the frequency weights are the component claimed to preserve the most-used experts.","core_discovery":"The paper's discovery is that the obstacle to expert merging—low inter-expert similarity, which makes direct weight averaging destructive—can be addressed by decomposing the concatenated weights of a cluster of experts as $U\\Sigma[V^{(1)};\\dots;V^{(n)}]^T$, sharing the left basis $U$ across all experts in the cluster, and merging only the $V$ blocks. The merged expert is reconstructed as $W_{\\text{merged}} = U\\Sigma[V_{\\text{merged}}]^T$. The paper argues that this forces the experts to be expressed in one common coordinate system before fusion, so the merged parameters inherit a shared functional subspace rather than averaging away divergent representations. It reports consistent gains over frequency pruning, output pruning, MC-SMoE, and HC-SMoE across Mixtral, Qwen1.5-MoE, Qwen3-MoE, and DeepSeek-MoE, and it presents an extended version, Sub-MoE$^\\dagger$, that adds activation-aware truncation for intra-expert compression.","pith_inferences":["Editorial inference: at full rank, the joint SVD reconstruction is algebraically the frequency-weighted average of the original experts, so the reported gains may be driven by rank truncation and clustering rather than by the shared $U$ per se; an ablation that fixes the same clusters and compares direct weighted averaging with the Sub-MoE reconstruction at the same truncation would isolate the me","Editorial inference: the method's dependence on 128 WikiText-2 samples for clustering and frequencies suggests a stress test the paper does not report—vary the calibration distribution and measure retention.","Editorial inference: the same merge-in-a-shared-subspace recipe might transfer to other parameter-sharing settings, such as merging layers or attention heads, where the objects being merged have low similarity; this is a natural extension the paper leaves implicit."],"forward_implications":["If the claim holds, MoE LLMs can be compressed by removing a quarter to half of their experts without fine-tuning, keeping roughly 94–96% and 86–87% of zero-shot accuracy on Mixtral-8x7B.","Sub-MoE$^\\dagger$ extends the same subspace merging to intra-expert compression, reporting 1.1–1.3$\\times$ throughput on 8x H800 GPUs and more stable performance than intra-expert truncation applied after standard merging.","Because the method needs only a small calibration set and no training, it can be applied to models whose training pipeline is unavailable.","Frequency-based $V$-merging gives more weight to experts activated often, so the compressed model should preserve behavior on common inputs more than on rare ones.","The method reports consistent gains across four MoE architectures, suggesting the subspace-alignment mechanism is not tied to one router design."],"supporting_citations":[{"why":"HC-SMoE is the main hierarchical-clustering merging baseline that Sub-MoE must beat across all four models.","marker":"[3]"},{"why":"D2-MoE documents the low inter-expert similarity, typically 0.1–0.3 for Mixtral, which motivates the parameter-conflict problem Sub-MoE targets.","marker":"[11]"},{"why":"MoE-Compression supplies the reproduced frequency-prune and output-prune baselines used in Table 1.","marker":"[12]"},{"why":"MC-SMoE is the routing-policy-based merging baseline compared in both expert-level and intra-expert compression experiments.","marker":"[23]"},{"why":"The Mixtral-8x7B model is the primary evaluation architecture for the central accuracy-retention claims.","marker":"[18]"},{"why":"SVD-LLM provides the activation-aware truncation technique that Sub-MoE$^\\dagger$ adapts for intra-expert compression.","marker":"[41]"}],"fun_headline_variants":["SVD subspace merging trims MoE experts, keeps 96% accuracy","Sub-MoE: merge weights in shared SVD subspace, cut 25% experts","Shared U-matrix merging trims MoE experts, keeps 96% accuracy","Sub-MoE: subspace expert merging hits 96% accuracy at 25% fewer experts","Joint SVD merges expert weights in one subspace, keeping 96%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that sharing $U$ and merging only $V$ is genuinely different from merging the original weights; if the joint SVD is run at full rank with no singular values dropped, the reconstruction is algebraically equivalent to a frequency-weighted average of the original experts, so the subspace would do no conflict-reduction work. The method also depends on 128 WikiText-2 samples being representative of the evaluation tasks.","fun_headline_variants_meta":{"raw":{"variants":["SVD subspace merging trims MoE experts, keeps 96% accuracy","Sub-MoE: merge weights in shared SVD subspace, cut 25% experts","Shared U-matrix merging trims MoE experts, keeps 96% accuracy","Sub-MoE: subspace expert merging hits 96% accuracy at 25% fewer experts","Joint SVD merges expert weights in one subspace, keeping 96%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000793,"raw_usage":{"total_tokens":3575,"prompt_tokens":1106,"completion_tokens":2469,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":722,"completion_tokens_details":{"reasoning_tokens":2360}},"tokens_in":722,"tokens_out":2469,"duration_ms":16424,"temperature":1.0,"reasoning_tokens":2360,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:47:05.212428+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Sub-MoE on a cluster with a full-rank joint SVD, meaning no singular values are truncated, and compare the reconstructed merged weight matrix to the direct frequency-weighted average of the original expert matrices: if the two matrices are equal, the claimed subspace mechanism has no effect beyond clustering and frequency weighting. Separately, re-run clustering and merging with a held-out calibration set from a different domain and check whether the reported 94–96% and 86–87% retention figures on Mixtral persist.","supporting_citations":[{"cited_title":"Delta decompression for moe-based LLMs compression","cited_arxiv_id":null,"evidence_quote":"D2-MoE documents the low inter-expert similarity, typically 0.1–0.3 for Mixtral, which motivates the parameter-conflict problem Sub-MoE targets."},{"cited_title":"Merge, then compress: Demystify efficient SMoe with hints from its routing policy","cited_arxiv_id":null,"evidence_quote":"MC-SMoE is the routing-policy-based merging baseline compared in both expert-level and intra-expert compression experiments."},{"cited_title":"Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, et al","cited_arxiv_id":null,"evidence_quote":"The Mixtral-8x7B model is the primary evaluation architecture for the central accuracy-retention claims."}],"review_version":1}