{"id":"13ebc64e-210e-4ae8-90fa-394d8fb53418","arxiv_id":"2607.28308","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Across six MoE models, expert subspaces overlap substantially, routes stay coherent, the actual prefix narrows the selected expert's geometric advantage in all 39 tested cells, yet later experts still improve next-token prediction in most frozen-route tests.","lead":"This paper measures how overlapping expert subspaces are inside mixture-of-experts language models, and shows that even when these subspaces overlap heavily, routers still pick the right experts and multiple experts still improve predictions. It introduces a way to separate route quality from true expert complementarity, and finds that the actual context narrows, rather than amplifies, the selected expert's geometric advantage.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim rests on a single fixed rank-128 input-subspace metric; no rank sweep or centered-basis check rules out a measurement artifact.","rationale":"I read the paper as a carefully scoped empirical study: it distinguishes route coherence, candidate quality, and interaction, reports complete cell-level tables, and includes sensitivity analyses. The reader's conditional verdict identifies the rank-128 input-subspace metric as the weakest assumption; my stress-test agrees and sharpens it. The manuscript itself flags the limitation in Section 6 and Appendix S7.1, so the authors are not overclaiming relative to their stated scope. However, the central claim 'without disjoint linear coverage' is still conditional on one fixed linear substrate, and no rank sweep or centered-basis variant is provided. This is a genuine unverified assumption rather than an internal inconsistency. The functional NLL and matched-training results provide independent evidence that later experts retain value, which keeps the central claim plausible, but they do not by themselves establish the 'shared neighborhood' assertion. Since the reader already conditioned on this point, my analysis does not change the verdict; it specifies the concrete check that would upgrade the paper to full acceptance or expose the metric dependence.","tokens_in":28147,"tokens_out":14016,"duration_ms":144827,"concrete_test":"On the same 39 factorial cells and the same train/evaluation splits, recompute ESSI and the interaction D for p = 64, 256, and 512 (capped at the hidden dimension), and recompute D using centered rank-128 expert bases with residual energy normalized by centered token energy. The concern is settled if all D 95% confidence intervals remain below zero and ESSI stays within roughly 0.7-1.2 across all conditions; any cell whose interval crosses zero, or an ESSI jump above about 1.2, would show that the fixed p=128 and the uncentered factorial construction are load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim depends on interpreting ESSI near one and the all-negative interaction D as 'coherent overlap'. The load-bearing condition is that the rank-128 linear subspace of router inputs is the right substrate for this conclusion. Section 2.1 fixes p=128 for both centered ESSI bases and uncentered factorial bases; the sensitivity analyses in Section 4.3 and Appendix S5 vary normalization and matching, but never the representation or its rank. Section 6 and Appendix S7.1 explicitly concede that overlap in this metric can coexist with different nonlinear features, output directions, or logit effects. Therefore 'without disjoint linear coverage' is established only for one linear measurement. If D ceases to be negative or ESSI rises well above one at a different rank, or if centering the factorial bases changes the sign of D, the observed narrowing would be an artifact of the chosen projection rather than evidence against geometric complementarity. The functional NLL results are separate and only 24 of 39 additions are positive, so they do not independently establish the 'shared token-relevant neighborhood' half of the claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a decomposition of MoE route quality into three separable quantities—route coherence, candidate quality, and candidate-by-context interaction—and introduces an Expert Subspace Separation Index (ESSI) together with a prefix-controlled 2x2 factorial design to measure them. Across six open MoE architectures the authors report that expert subspaces overlap substantially (ESSI near one), while actual routes still fit token representations better than matched alternatives. In 39 factorial cells spanning OLMoE, Mixtral, and DeepSeek, the selected candidate explains more residual representation than the strongest unselected rival, yet the actual prefix narrows this advantage: all interaction estimates D are negative and all primary 95% confidence intervals are below zero. Functional probes show that adding later experts improves next-token NLL in 24 of 39 frozen-route additions (15 inconclusive), and a controlled Top-1/Top-2 training comparison favors Top-2 in all three seeds. The paper concludes that multi-expert benefit can arise from distinct computations within a shared token-relevant neighborhood without disjoint linear coverage, a pattern it calls coherent overlap.","tokens_in":28337,"tokens_out":5223,"duration_ms":50210,"significance":"If the findings are taken as measured, the paper makes a useful contribution: it separates quantities that prior work often conflates, it supplies a reusable factorial protocol for testing geometric complementarity, and it demonstrates that overlap in router-input subspaces does not by itself imply redundancy or pruning value. The statistical design is careful—train/eval splits, source-context paired bootstrap, multiple sensitivity variants, an independent tall-SVD numerical cross-check, and honest labeling of inconclusive functional estimates. The paper also gives full cell-level tables in the appendices. The main risk is that the central negative-interaction result is established for one fixed linear metric (rank-128 router-input subspaces), and the paper's broad conclusion is only partially protected by the explicit scope limitations in Section 6 and Appendix S7.1.","major_comments":[{"comment":"The central claim that experts do not provide disjoint linear coverage and that the actual prefix narrows the selected candidate's advantage rests entirely on a single rank value p=128 for both the centered ESSI bases and the uncentered factorial bases. The sensitivity analyses in Section 4.3 vary normalization and matching but never vary the representation's rank or its centering. Because Section 6 and S7.1 explicitly concede that overlap in this metric can coexist with different nonlinear features, output directions, or logit effects, the broad conclusion “without disjoint linear coverage” is not fully established for geometric complementarity in general. I therefore ask the authors to either (a) run a rank sweep (e.g., p=32, 64, 256, 512) and a centered-basis version of the factorial D computation, reporting whether ESSI and the sign of D are stable, or (b) explicitly scope the abstract and conclusion to say “as measured by rank-128 linear router-input subspaces.” This is load-bearing because the paper's title and concluding sentence make a general statement, while the evidence is metric-specific.","section":"Section 2.1, Section 4.3, and Section 6 / S7.1"}],"minor_comments":[{"comment":"The abstract states that “every 95% confidence interval lies below zero” for the interaction; this is true for the primary factorial analysis but not for the strict-caliper sensitivity variant in Appendix S5, where some intervals overlap zero. Please add a qualifier such as “in the primary analysis” for precision.","section":"Abstract"},{"comment":"The choice p=128 is described as “throughout” without a rationale or a reference to prior conventions. A sentence justifying the rank choice, or a pointer to a rank-sensitivity result in the appendix, would help readers assess the metric's robustness.","section":"Section 2.1"},{"comment":"The percentile values (0.006–0.154 and 0.007–0.079) are reported in the text but the figure caption does not state how the percentile is defined across the alternative contexts. Please spell out the definition in the caption or in the associated text.","section":"Figure 4D"},{"comment":"When describing the alternative contexts W1,...,WM, the text says they “exclude the complete actual route and the rival” but does not state whether they are allowed to include the selected candidate s at other positions. Clarifying this would remove a possible ambiguity about the counterfactual being tested.","section":"Appendix S3.3"},{"comment":"The column header “Pctl.” is defined in the caption, but a reminder in the table itself or a more explicit header such as “Actual-prefix percentile” would improve readability.","section":"Table S5 caption"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical study with a clean decomposition and honest reporting. My main reservation is the fixed-rank, fixed-centering geometry: the paper's own limitations show the authors are aware of the issue, but the abstract and conclusion still make a general claim. The requested rank sweep and centered-basis check are feasible and would substantially strengthen the paper; alternatively, a careful scoping revision would also be acceptable. I would be comfortable with acceptance after that point is addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a careful, honest empirical paper that deserves a serious referee. It separates candidate quality from context interaction, introduces ESSI as a normalized overlap index, and reports a robust negative interaction across 39 factorial cells. If the main results hold up, it reframes the pruning argument: linear overlap of expert input subspaces is not evidence of redundancy.\n\nWhat is genuinely new: the 2x2 factorial design isolates the candidate-by-context interaction, and the changing-context versus same-context sign reversal in Figure 3 is a nice demonstration that naive route comparisons conflate quality with opportunity. The paper also does the reporting properly: full cell tables, paired source-context bootstrap, sensitivity variants that keep the negative sign, an independent tall-SVD check, and honest labels for the functional results. The matched-compute Top-2/Top-1 study is modest but controlled. Related-work coverage looks reasonable.\n\nThe soft spots are real but not fatal. Everything geometric rests on a single fixed rank-128 PCA basis on router inputs; there is no rank sweep and no centered-basis check in the factorial, so if D changes sign at another rank or after centering, the 'coherent overlap' story would look like an artifact of that projection. The authors state this limitation clearly in Section 6 and S7.1, but the abstract and title generalize beyond the metric. The functional evidence is also weaker than the geometric evidence: only 24 of 39 frozen-route additions are positive, with 15 inconclusive, so the 'useful multi-expert computation persists' half of the claim is suggestive rather than proven. And there is no released code, which makes independent replication harder than it should be.\n\nWho it is for: people working on MoE interpretability, routing analysis, and expert pruning. I would bring it to reading group. My recommendation: send it to peer review, and ask the authors for code plus a rank-sensitivity analysis at least at one layer before acceptance. With that, the paper would be solid.","headline":"A careful, honest MoE study that deserves a serious referee; its fixed rank-128 metric is the one real soft spot.","tokens_in":28887,"tokens_out":2728,"would_cite":true,"duration_ms":24400,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that multi-expert benefit in sparse mixture-of-experts language models does not require experts to occupy disjoint linear subspaces of the token representation; instead, value can come from distinct computations inside a…","keywords":["mixture-of-experts","sparse routing","expert subspaces","geometric complementarity","coherent overlap","expert subspace separation index","factorial interaction","expert pruning"],"falsifier":"Recompute the prefix-controlled $2\\times2$ factorial using expert output representations or final-logit effects instead of rank-128 router-input projections; if the interaction $D$ becomes positive in a majority of the 39 cells under such an output-level metric, the paper's coherent-overlap conclusion would be an artifact of measuring only linear router-input coverage.","tokens_in":27917,"feed_emoji":"🧩","tokens_out":10515,"duration_ms":82624,"temperature":0.7,"pith_summary":"Sparse mixture-of-experts language models route each token through several experts, and the usual explanation is geometric complementarity: co-selected experts should cover disjoint directions of the token's representation. This paper argues that this account conflates three separate quantities—whether a route fits the token, whether the selected expert is individually better than a rival, and whether the selected expert becomes especially useful because of its co-selected context. Using a new separation index and a prefix-controlled factorial design across six MoE architectures and 39 model-layer-position cells, it finds expert subspaces overlap substantially while actual routes still fit tokens better than alternatives, and the selected candidate beats the strongest unselected rival in every cell—yet the actual prefix narrows rather than amplifies that advantage, with all 39 interactions negative and every 95% confidence interval below zero. This narrowing does not make later experts redundant: frozen-route additions improve next-token prediction in 24 of 39 comparisons, and a matched-compute Top-2 model beats Top-1 in all three seeds. The joint pattern, called coherent overlap, implies multi-expert benefit can arise from distinct computations inside a shared token-relevant neighborhood without disjoint linear coverage.","feed_headline":"Overlapping experts still earn their place in MoE routes","feed_subtitle":"In 39 cells, routing narrows geometric advantage; later experts still improve prediction in 24 of 39 tests.","key_machinery":"The carrying machinery is a pair of diagnostics. The Expert Subspace Separation Index (ESSI) divides the average chordal distance between experts' rank-128 global subspaces by the load-weighted average distance to each expert's local tangent subspace, so values near one mean between-expert separation is comparable to within-expert variation rather than evidence of a hard partition. The prefix-controlled $2\\times2$ factorial fixes the actual preceding experts $S$ and matched alternative contexts $W$, then crosses the selected candidate $s$ with the strongest unselected rival $r$, measuring fractional novelty $q_x(c|P)$ of each candidate's uncentered rank-128 basis against residual router-input energy; the difference-in-differences $D = T_s - T_r$ tests whether the actual context amplifies the selected candidate's advantage. Functional probes—frozen-route adjacent-prefix NLL recovery and a matched-compute Top-1/Top-2 training comparison—do the work of separating geometric novelty from predictive value.","core_discovery":"The paper's central discovery is that positive geometric complementarity, defined as the selected expert's advantage growing in its actual co-selection context, is not supported: in all 39 factorial cells across OLMoE, Mixtral, and DeepSeek, the candidate-by-context interaction $D = T_s - T_r$ is negative and every paired 95% bootstrap interval lies below zero. Candidate quality and route coherence still hold—the selected candidate explains more residual representation than the strongest unselected rival in all 39 cells, and actual routes leave less unexplained energy than matched alternatives—so the negative interaction is read as geometric saturation: a prefix drawn from the same router-input distribution removes directions available to both candidates, and the better candidate loses more marginal novelty. Functional probes then show that this linear narrowing is not functional redundancy: later experts reduce next-token NLL in 24 of 39 frozen-route additions, replacement damage concentrates in the leader in seven of nine configurations but the later set can dominate at some layers, and a matched-compute Top-2 configuration beats Top-1 in all three seeds. The paper concludes that multi-expert benefit can arise from distinct computations within a shared token-relevant neighborhood without disjoint linear coverage.","pith_inferences":["A direct next test the paper leaves open is to run the same prefix-controlled factorial on expert output subspaces or output-logit effects; if the interaction $D$ turns positive there, coherent overlap is a property of the input metric rather than of the computation.","Because expert count and width change together in the matched-compute study, an equally natural extension is to sweep the number of active experts while holding active width exactly constant across several model scales, checking whether the Top-2 advantage persists when capacity is perfectly matched.","If later experts improve prediction despite high input-subspace overlap, adaptive-k routers could plausibly replace novelty-based routing signals with predicted output gain of the next expert under the retained prefix; the factorial estimator in this paper is a ready-made way to compute that gain."],"forward_implications":["Expert overlap in router-input subspaces cannot by itself justify pruning, merging, or skipping; removal should be evaluated under the retained route because it changes the context of the remaining experts.","Router score order is informative: replacing the leader does more NLL damage than replacing the entire later set in seven of nine configurations, while the later set can dominate at some layers.","Marginal value of later experts declines with router rank but does not vanish: 24 of 39 frozen-route additions improve next-token prediction, and none show significant harm.","Under near-matched capacity, parameters, and compute, a Top-2 MoE reaches lower validation loss than a Top-1 MoE in all three seeds, so multi-expert routing can help even without expanding active capacity.","Geometric similarity alone should not be read as redundancy; route coherence, candidate quality, candidate-context interaction, and functional value are separable and can disagree."],"supporting_citations":[{"why":"Defines the normalized chordal (projection) distance on the Grassmann manifold that ESSI uses for between- and within-expert separation.","marker":"Edelman et al., 1998"},{"why":"Provides the subspace-distance formalism for comparing equal-rank linear spans of expert bases.","marker":"Ye and Lim, 2016"},{"why":"Gives the paired bootstrap used to build the 95% source-context confidence intervals on every factorial cell.","marker":"Efron and Tibshirani, 1993"},{"why":"Supplies OLMoE, one of the three architectures carrying the factorial and functional analyses.","marker":"Muennighoff et al., 2025"},{"why":"Supplies Mixtral, the Top-2 architecture whose cells all show negative interaction.","marker":"Jiang et al., 2024"},{"why":"Supplies DeepSeek-MoE, the shared-expert architecture used for the post-hoc factorial cells, and states the specialization design motivating geometric complementarity.","marker":"Dai et al., 2024"},{"why":"Frames the capacity-matching conventions used to design the controlled Top-1/Top-2 training comparison.","marker":"Clark et al., 2022"},{"why":"Supplies the fine-grained MoE scaling analysis used to hold active intermediate capacity fixed in the Top-1/Top-2 study.","marker":"Ludziejewski et al., 2024"}],"fun_headline_variants":["Overlap doesn't mean redundancy in MoE","Experts can share space and still help","No geometric edge, but experts earn their keep","Coherent overlap: similar experts, distinct value","Why similar experts still improve MoE routes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the leading rank-128 linear subspace of router-input representations is the right substrate for judging whether experts complement each other; if true complementarity lives in output directions or nonlinear interactions, the measured negative interaction would be an artifact of that linear choice.","fun_headline_variants_meta":{"raw":{"variants":["Overlap doesn't mean redundancy in MoE","Experts can share space and still help","No geometric edge, but experts earn their keep","Coherent overlap: similar experts, distinct value","Why similar experts still improve MoE routes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00026,"raw_usage":{"total_tokens":1654,"prompt_tokens":1074,"completion_tokens":580,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":690,"completion_tokens_details":{"reasoning_tokens":512}},"tokens_in":690,"tokens_out":580,"duration_ms":5796,"temperature":1.0,"reasoning_tokens":512,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:22:48.243276+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the prefix-controlled $2\\times2$ factorial using expert output representations or final-logit effects instead of rank-128 router-input projections; if the interaction $D$ becomes positive in a majority of the 39 cells under such an output-level metric, the paper's coherent-overlap conclusion would be an artifact of measuring only linear router-input coverage.","supporting_citations":[],"review_version":2}