Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing

Bonan Xu; Huiyuan Tian; Shijian Li

REVIEW 1 major objections 5 minor 9 references

Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing

T0 review · 1 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that multi-expert benefit in sparse mixture-of-experts language models does not require experts to occupy disjoint linear subspaces of the token representation; instead, value can come from distinct computations inside a…

desk verdict A careful, honest MoE study that deserves a serious referee; its fixed rank-128 metric is the one real soft spot. read the letter →

arxiv 2607.28308 v1 pith:XJYMM5JA submitted 2026-07-30 cs.LG

Huiyuan Tian , Bonan Xu , Shijian Li This is my paper · ORCID

classification cs.LG

keywords mixture-of-expertssparseroutingexpertsubspacesgeometriccomplementaritycoherentoverlapsubspaceseparationindexfactorialinteractionpruning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Sparse mixture-of-experts language models route each token through several experts, and the usual explanation is geometric complementarity: co-selected experts should cover disjoint directions of the token's representation. This paper argues that this account conflates three separate quantities—whether a route fits the token, whether the selected expert is individually better than a rival, and whether the selected expert becomes especially useful because of its co-selected context. Using a new separation index and a prefix-controlled factorial design across six MoE architectures and 39 model-layer-position cells, it finds expert subspaces overlap substantially while actual routes still fit tokens better than alternatives, and the selected candidate beats the strongest unselected rival in every cell—yet the actual prefix narrows rather than amplifies that advantage, with all 39 interactions negative and every 95% confidence interval below zero. This narrowing does not make later experts redundant: frozen-route additions improve next-token prediction in 24 of 39 comparisons, and a matched-compute Top-2 model beats Top-1 in all three seeds. The joint pattern, called coherent overlap, implies multi-expert benefit can arise from distinct computations inside a shared token-relevant neighborhood without disjoint linear coverage.

What carries the argument

The carrying machinery is a pair of diagnostics. The Expert Subspace Separation Index (ESSI) divides the average chordal distance between experts' rank-128 global subspaces by the load-weighted average distance to each expert's local tangent subspace, so values near one mean between-expert separation is comparable to within-expert variation rather than evidence of a hard partition. The prefix-controlled $2\times2$ factorial fixes the actual preceding experts $S$ and matched alternative contexts $W$, then crosses the selected candidate $s$ with the strongest unselected rival $r$, measuring fractional novelty $q_x(c|P)$ of each candidate's uncentered rank-128 basis against residual router-input energy; the difference-in-differences $D = T_s - T_r$ tests whether the actual context amplifies the selected candidate's advantage. Functional probes—frozen-route adjacent-prefix NLL recovery and a matched-compute Top-1/Top-2 training comparison—do the work of separating geometric novelty from predictive value.

What would settle it

Recompute the prefix-controlled $2\times2$ factorial using expert output representations or final-logit effects instead of rank-128 router-input projections; if the interaction $D$ becomes positive in a majority of the 39 cells under such an output-level metric, the paper's coherent-overlap conclusion would be an artifact of measuring only linear router-input coverage.

Watch

Extended reading notes

Core claim

The paper's central discovery is that positive geometric complementarity, defined as the selected expert's advantage growing in its actual co-selection context, is not supported: in all 39 factorial cells across OLMoE, Mixtral, and DeepSeek, the candidate-by-context interaction $D = T_s - T_r$ is negative and every paired 95% bootstrap interval lies below zero. Candidate quality and route coherence still hold—the selected candidate explains more residual representation than the strongest unselected rival in all 39 cells, and actual routes leave less unexplained energy than matched alternatives—so the negative interaction is read as geometric saturation: a prefix drawn from the same router-input distribution removes directions available to both candidates, and the better candidate loses more marginal novelty. Functional probes then show that this linear narrowing is not functional redundancy: later experts reduce next-token NLL in 24 of 39 frozen-route additions, replacement damage concentrates in the leader in seven of nine configurations but the later set can dominate at some layers, and a matched-compute Top-2 configuration beats Top-1 in all three seeds. The paper concludes that multi-expert benefit can arise from distinct computations within a shared token-relevant neighborhood without disjoint linear coverage.

Load-bearing premise

The load-bearing premise is that the leading rank-128 linear subspace of router-input representations is the right substrate for judging whether experts complement each other; if true complementarity lives in output directions or nonlinear interactions, the measured negative interaction would be an artifact of that linear choice.

Editorial extensions

If this is right

Expert overlap in router-input subspaces cannot by itself justify pruning, merging, or skipping; removal should be evaluated under the retained route because it changes the context of the remaining experts.
Router score order is informative: replacing the leader does more NLL damage than replacing the entire later set in seven of nine configurations, while the later set can dominate at some layers.
Marginal value of later experts declines with router rank but does not vanish: 24 of 39 frozen-route additions improve next-token prediction, and none show significant harm.
Under near-matched capacity, parameters, and compute, a Top-2 MoE reaches lower validation loss than a Top-1 MoE in all three seeds, so multi-expert routing can help even without expanding active capacity.
Geometric similarity alone should not be read as redundancy; route coherence, candidate quality, candidate-context interaction, and functional value are separable and can disagree.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct next test the paper leaves open is to run the same prefix-controlled factorial on expert output subspaces or output-logit effects; if the interaction $D$ turns positive there, coherent overlap is a property of the input metric rather than of the computation.
Because expert count and width change together in the matched-compute study, an equally natural extension is to sweep the number of active experts while holding active width exactly constant across several model scales, checking whether the Top-2 advantage persists when capacity is perfectly matched.
If later experts improve prediction despite high input-subspace overlap, adaptive-k routers could plausibly replace novelty-based routing signals with predicted output gain of the next expert under the retained prefix; the factorial estimator in this paper is a ready-made way to compute that gain.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a careful, honest empirical paper that deserves a serious referee. It separates candidate quality from context interaction, introduces ESSI as a normalized overlap index, and reports a robust negative interaction across 39 factorial cells. If the main results hold up, it reframes the pruning argument: linear overlap of expert input subspaces is not evidence of redundancy.

What is genuinely new: the 2x2 factorial design isolates the candidate-by-context interaction, and the changing-context versus same-context sign reversal in Figure 3 is a nice demonstration that naive route comparisons conflate quality with opportunity. The paper also does the reporting properly: full cell tables, paired source-context bootstrap, sensitivity variants that keep the negative sign, an independent tall-SVD check, and honest labels for the functional results. The matched-compute Top-2/Top-1 study is modest but controlled. Related-work coverage looks reasonable.

The soft spots are real but not fatal. Everything geometric rests on a single fixed rank-128 PCA basis on router inputs; there is no rank sweep and no centered-basis check in the factorial, so if D changes sign at another rank or after centering, the 'coherent overlap' story would look like an artifact of that projection. The authors state this limitation clearly in Section 6 and S7.1, but the abstract and title generalize beyond the metric. The functional evidence is also weaker than the geometric evidence: only 24 of 39 frozen-route additions are positive, with 15 inconclusive, so the 'useful multi-expert computation persists' half of the claim is suggestive rather than proven. And there is no released code, which makes independent replication harder than it should be.

Who it is for: people working on MoE interpretability, routing analysis, and expert pruning. I would bring it to reading group. My recommendation: send it to peer review, and ask the authors for code plus a rank-sensitivity analysis at least at one layer before acceptance. With that, the paper would be solid.

Referee Report

1 major / 5 minor

Summary. The paper proposes a decomposition of MoE route quality into three separable quantities—route coherence, candidate quality, and candidate-by-context interaction—and introduces an Expert Subspace Separation Index (ESSI) together with a prefix-controlled 2x2 factorial design to measure them. Across six open MoE architectures the authors report that expert subspaces overlap substantially (ESSI near one), while actual routes still fit token representations better than matched alternatives. In 39 factorial cells spanning OLMoE, Mixtral, and DeepSeek, the selected candidate explains more residual representation than the strongest unselected rival, yet the actual prefix narrows this advantage: all interaction estimates D are negative and all primary 95% confidence intervals are below zero. Functional probes show that adding later experts improves next-token NLL in 24 of 39 frozen-route additions (15 inconclusive), and a controlled Top-1/Top-2 training comparison favors Top-2 in all three seeds. The paper concludes that multi-expert benefit can arise from distinct computations within a shared token-relevant neighborhood without disjoint linear coverage, a pattern it calls coherent overlap.

Significance. If the findings are taken as measured, the paper makes a useful contribution: it separates quantities that prior work often conflates, it supplies a reusable factorial protocol for testing geometric complementarity, and it demonstrates that overlap in router-input subspaces does not by itself imply redundancy or pruning value. The statistical design is careful—train/eval splits, source-context paired bootstrap, multiple sensitivity variants, an independent tall-SVD numerical cross-check, and honest labeling of inconclusive functional estimates. The paper also gives full cell-level tables in the appendices. The main risk is that the central negative-interaction result is established for one fixed linear metric (rank-128 router-input subspaces), and the paper's broad conclusion is only partially protected by the explicit scope limitations in Section 6 and Appendix S7.1.

major comments (1)

[Section 2.1, Section 4.3, and Section 6 / S7.1] The central claim that experts do not provide disjoint linear coverage and that the actual prefix narrows the selected candidate's advantage rests entirely on a single rank value p=128 for both the centered ESSI bases and the uncentered factorial bases. The sensitivity analyses in Section 4.3 vary normalization and matching but never vary the representation's rank or its centering. Because Section 6 and S7.1 explicitly concede that overlap in this metric can coexist with different nonlinear features, output directions, or logit effects, the broad conclusion “without disjoint linear coverage” is not fully established for geometric complementarity in general. I therefore ask the authors to either (a) run a rank sweep (e.g., p=32, 64, 256, 512) and a centered-basis version of the factorial D computation, reporting whether ESSI and the sign of D are stable, or (b) explicitly scope the abstract and conclusion to say “as measured by rank-128 linear router-input subspaces.” This is load-bearing because the paper's title and concluding sentence make a general statement, while the evidence is metric-specific.

minor comments (5)

[Abstract] The abstract states that “every 95% confidence interval lies below zero” for the interaction; this is true for the primary factorial analysis but not for the strict-caliper sensitivity variant in Appendix S5, where some intervals overlap zero. Please add a qualifier such as “in the primary analysis” for precision.
[Section 2.1] The choice p=128 is described as “throughout” without a rationale or a reference to prior conventions. A sentence justifying the rank choice, or a pointer to a rank-sensitivity result in the appendix, would help readers assess the metric's robustness.
[Figure 4D] The percentile values (0.006–0.154 and 0.007–0.079) are reported in the text but the figure caption does not state how the percentile is defined across the alternative contexts. Please spell out the definition in the caption or in the associated text.
[Appendix S3.3] When describing the alternative contexts W1,...,WM, the text says they “exclude the complete actual route and the rival” but does not state whether they are allowed to include the selected candidate s at other positions. Clarifying this would remove a possible ambiguity about the counterfactual being tested.
[Table S5 caption] The column header “Pctl.” is defined in the caption, but a reminder in the table itself or a more explicit header such as “Actual-prefix percentile” would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central quantities are held-out measurements under operationalized metrics, not fitted constants repackaged as predictions.

full rationale

The paper's derivation chain is observational rather than definitional. ESSI (Eq. 3), the route-coherence residual ratio, and the factorial interaction D (Eq. 9) are computed on held-out tokens using bases fit on the train split; none of these quantities is a fitted parameter later renamed as a prediction. The sign of D is an empirical difference-in-differences: the paper defines positive geometric complementarity as D>0 (Figure 1) and then reports D<0 in all 39 cells, so the conclusion is a falsification of that defining prediction, not a restatement of it. No load-bearing step appeals to a self-citation: the Grassmannian chordal distance is cited to Edelman et al. and Ye and Lim, and the reference list contains no first-party prior work. The matched alternative contexts are constructed by the authors' own load-matching rules, and the rank-128 linear router-input metric is a stated scope condition (Sections 6 and S7.1) rather than a hidden input; the paper explicitly separates geometric interaction from functional value and tests the latter with NLL interventions and a controlled Top-1/Top-2 training study. Consequently, there is no quoted reduction of any claimed result to its own definition.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The contributions rest on the router-input representation being well captured by rank-128 linear subspaces, on the validity of the load-matched alternative contexts as counterfactuals, and on the fairness of using the selected expert's weight for both candidates. These are domain assumptions, plus the standard linear algebra of Grassmann distances. The free parameters are all hand-chosen analysis settings (rank, neighborhood, anchors, thresholds, alternative-context count, caliper percentile) rather than fitted values.

free parameters (8)

subspace rank p = 128
All ESSI and factorial bases are rank-128; chosen by hand, no rank sweep reported.
local tangent neighborhood size = 256
Number of nearest representations used to fit each local tangent basis for the ESSI denominator; chosen by hand (Appendix S2.1).
anchor count = 2048 (OLMoE/Mixtral/DeepSeek), 512 (Qwen3/Gemma4/Qwen3.6)
Anchors per eligible expert for local tangent fitting; chosen by hand, varies by model family (Appendix S2.1).
expert support threshold = 2048 (ESSI eligibility), 1024 (factorial basis fit), 1024 (Qwen-series geometry)
Minimum number of selected tokens for an expert to enter the analysis; chosen by hand (Appendices S2, S3).
number of alternative contexts M = 5
Alternative contexts per factorial cell; fixed by the authors' construction (Appendix S3.3).
load-near neighbor window = 8
Nearest experts by routing-load difference used to build alternative routes (Appendix S3.2).
strict caliper threshold = 5th percentile of training nearest-match distances
Post hoc sensitivity variant that excludes 92% of token-cells; chosen as a robustness check (Appendix S5).
matched-compute model widths = Top-1: 4 experts x 1024; Top-2: 8 experts x 512
Joint choice of expert count and width in the controlled training study; authors note both change together (Section 5, Appendix S6.3).

assumptions (5)

standard math Principal-angle chordal distance on the Grassmann manifold is an appropriate distance between equal-rank expert subspaces.
Used to define ESSI in Eq. 2; standard linear-algebraic tool.
domain assumption The router-input representation z_x is well described by the leading rank-128 linear subspace; residual energy after projection captures what a candidate adds.
Load-bearing for both the factorial and the overlap conclusions; stated in Section 2.1 and Appendix S7.1.
domain assumption Load-matched alternative routes are valid counterfactual contexts for the same token.
The entire 2x2 factorial and the D interaction depend on this; construction in Appendix S3.3.
domain assumption Evaluating both the selected candidate and the rival with the selected expert's actual rank-j router weight is a fair comparison.
Adopted in Appendix S3.1 to change expert identity rather than mixture scale; if the rival's natural weight differs, the comparison is partly an artifact.
domain assumption Matched active intermediate capacity isolates the benefit of Top-2 over Top-1 topology in the training study.
Expert count and width change jointly; authors acknowledge this in Section 5, so the comparison is specific to this configuration.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing." pith.science (2026). https://pith.science/paper/XJYMM5JA

@misc{pith2026260728308,
  author       = {Pith},
  title        = {Pith review of: Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XJYMM5JA}},
  note         = {Machine review of arXiv:2607.28308}
}

abstract

Sparse mixture-of-experts (MoE) language models route each token to multiple experts, suggesting a geometric account of their benefit: co-selected experts should contribute distinct representation directions. Existing evidence often conflates route coherence, candidate quality, and candidate-by-context interaction. We distinguish these quantities using an Expert Subspace Separation Index (ESSI), matched-route residuals, and a prefix-controlled $2\times2$ factorial; frozen-route interventions and a controlled Top-$k$ study assess functional value. Three paired contrasts organize the findings. First, across six MoE architectures, expert subspaces overlap substantially, yet actual routes explain token representations better than matched alternatives. Second, across the 39 factorial cells in OLMoE, Mixtral, and DeepSeek, the selected candidate explains more of the residual representation than the strongest unselected rival in every cell, yet the actual prefix narrows this advantage throughout: all interactions are negative, and every 95% confidence interval lies below zero. Third, this geometric narrowing does not imply functional redundancy: adding later experts improves next-token prediction in 24 of 39 frozen-route comparisons, while the other 15 estimates are inconclusive; a controlled training study also favors Top-2 over Top-1 in all three seeds. We call this joint pattern coherent overlap: routing selects token-relevant experts from a shared geometric neighborhood, while useful multi-expert computation persists without disjoint linear coverage. Separating these quantities clarifies why geometric similarity alone cannot determine redundancy or pruning value.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

9 extracted references · 1 canonical work pages

[4]

13 Coherent Overlap in Sparse Mixture-of-Experts Routing Michael I

URL https://arxiv.org/abs/2401.04088. 13 Coherent Overlap in Sparse Mixture-of-Experts Routing Michael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computation, 6(2):181–214, 1994. doi: 10.1162/neco.1994.6.2.181. Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural netwo...

arXiv 1994
[8]

doi: 10.18653/v1/2024.findings-emnlp.361

Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.361. URL https://aclanthology.org/2024.findings-emnlp.361/. Zeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu, and Jianfeng Gao. Diversifying the expert knowledge for task-agnostic pruning in sparse mixture-of-experts. InFindings of the Association for Computational Linguis...

doi:10.18653/v1/2024.findings-emnlp.361 2024
[9]

URL https://proceedings.neurips.cc/paper_files/paper/2022/ file/2f00ecd787b432c1d36f3de9800728eb-Paper-Conference.pdf

doi: 10.52202/068431-0515. URL https://proceedings.neurips.cc/paper_files/paper/2022/ file/2f00ecd787b432c1d36f3de9800728eb-Paper-Conference.pdf. Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models, 2022. URL https://arxiv.org/abs/2202.089...

arXiv 2022
[2018]

Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian

URL https://arxiv.org/abs/1803.05457. Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. A discourse-aware attention model for abstractive summarization of long documents. InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Languag...

arXiv 2018
[2021]

Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer

URL https://openreview.net/forum?id=qrwe7XHTmYb. Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. InProceedings of the 38th International Conference on Machine Learning, volume 139 ofProceedings of Machine Learning Research, pages 6265–6274. PMLR, 18–24 Jul 2021. URL ht...

work page 2021
[2022]

Gemma Team

URL http://jmlr.org/papers/v23/21-0998.html. Gemma Team. Gemma 4 technical report, 2026. URL https://arxiv.org/abs/2607.02770. Wentao Hu, Mingkuan Zhao, Shuangyong Song, Xiaoyan Zhu, Xin Lai, and Jiayin Wang. Mosaic pruning: A hierarchical framework for generalizable pruning of mixture-of-experts models. In Proceedings of the AAAI Conference on Artificial...

arXiv 2026
[2024]

doi: 10.18653/v1/2024.acl-long.70

Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.70. URL https: //aclanthology.org/2024.acl-long.70/. 12 Coherent Overlap in Sparse Mixture-of-Experts Routing Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conv...

doi:10.18653/v1/2024.acl-long.70 2024
[2025]

Ke Ye and Lek-Heng Lim

URL https://arxiv.org/abs/2505.09388. Ke Ye and Lek-Heng Lim. Schubert varieties and distances between subspaces of different di- mensions.SIAM Journal on Matrix Analysis and Applications, 37(3):1176–1197, 2016. doi: 10.1137/15M1054201. URL https://doi.org/10.1137/15M1054201. Youngsik Yoon, Siwei Wang, Wei Chen, and Jungseul Ok. When are experts misrouted...

arXiv 2016

Show all 9 references

[2026]

standing committee

Association for Computational Linguistics. ISBN 979-8-89176-380-7. doi: 10.18653/v1/ 2026.eacl-long.104. URL https://aclanthology.org/2026.eacl-long.104/. Bo Wang, Junzhuo Li, Hong Chen, Yuanlin Chu, Yuxuan Fan, and Xuming Hu. Deconstructing pre-training: Knowledge attribution...

2026 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [4]

13 Coherent Overlap in Sparse Mixture-of-Experts Routing Michael I

URL https://arxiv.org/abs/2401.04088. 13 Coherent Overlap in Sparse Mixture-of-Experts Routing Michael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computation, 6(2):181–214, 1994. doi: 10.1162/neco.1994.6.2.181. Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural netwo...

arXiv 1994

[2] [8]

doi: 10.18653/v1/2024.findings-emnlp.361

Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.361. URL https://aclanthology.org/2024.findings-emnlp.361/. Zeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu, and Jianfeng Gao. Diversifying the expert knowledge for task-agnostic pruning in sparse mixture-of-experts. InFindings of the Association for Computational Linguis...

doi:10.18653/v1/2024.findings-emnlp.361 2024

[3] [9]

URL https://proceedings.neurips.cc/paper_files/paper/2022/ file/2f00ecd787b432c1d36f3de9800728eb-Paper-Conference.pdf

doi: 10.52202/068431-0515. URL https://proceedings.neurips.cc/paper_files/paper/2022/ file/2f00ecd787b432c1d36f3de9800728eb-Paper-Conference.pdf. Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models, 2022. URL https://arxiv.org/abs/2202.089...

arXiv 2022

[4] [2018]

Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian

URL https://arxiv.org/abs/1803.05457. Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. A discourse-aware attention model for abstractive summarization of long documents. InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Languag...

arXiv 2018

[5] [2021]

Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer

URL https://openreview.net/forum?id=qrwe7XHTmYb. Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. InProceedings of the 38th International Conference on Machine Learning, volume 139 ofProceedings of Machine Learning Research, pages 6265–6274. PMLR, 18–24 Jul 2021. URL ht...

work page 2021

[6] [2022]

Gemma Team

URL http://jmlr.org/papers/v23/21-0998.html. Gemma Team. Gemma 4 technical report, 2026. URL https://arxiv.org/abs/2607.02770. Wentao Hu, Mingkuan Zhao, Shuangyong Song, Xiaoyan Zhu, Xin Lai, and Jiayin Wang. Mosaic pruning: A hierarchical framework for generalizable pruning of mixture-of-experts models. In Proceedings of the AAAI Conference on Artificial...

arXiv 2026

[7] [2024]

doi: 10.18653/v1/2024.acl-long.70

Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.70. URL https: //aclanthology.org/2024.acl-long.70/. 12 Coherent Overlap in Sparse Mixture-of-Experts Routing Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conv...

doi:10.18653/v1/2024.acl-long.70 2024

[8] [2025]

Ke Ye and Lek-Heng Lim

URL https://arxiv.org/abs/2505.09388. Ke Ye and Lek-Heng Lim. Schubert varieties and distances between subspaces of different di- mensions.SIAM Journal on Matrix Analysis and Applications, 37(3):1176–1197, 2016. doi: 10.1137/15M1054201. URL https://doi.org/10.1137/15M1054201. Youngsik Yoon, Siwei Wang, Wei Chen, and Jungseul Ok. When are experts misrouted...

arXiv 2016