Pith. sign in

REVIEW 4 major objections 6 minor 88 references

UniMoMo: Expert Merging-Based MoE Acceleration for Large Recommendation Models

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read UniMoMo converts a trained eight-expert MoE recommender to a four- or two-expert checkpoint that retains 98–104% of source NDCG@10 while running 1.28–2.21x faster.

desk verdict Useful post-training MoE compressor for recommenders with a sound core idea, but an internal Table 1 vs Table 3 inconsistency in the source checkpoint makes the headline preservation ratios unverifiable until fixed. read the letter →

arxiv 2608.08627 v1 pith:2DXJJBOC submitted 2026-08-09 cs.AI

Lei Xin , Bin Gu , Peize Li , Zitong Wang , Jianbo Zhao , Changjiang Jiang , Yanyue Xie , Chao Huang
show 4 more authors
Xuyang Zhao Zunhai Su Fanhu Zeng Zhenglun Kong
This is my paper · ORCID
classification cs.AI
keywords mixtureofexpertsexpertmergingmodelcompressionrecommendersystemsgraphcoarseningpost-trainingroutingexposureadaptive
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

UniMoMo addresses the serving cost of trained mixture-of-experts (MoE) recommender systems by converting an existing eight-expert checkpoint into a smaller standard MoE with four or two experts per block, rather than retraining or adding a compression module. The conversion treats expert reduction as a constrained graph-coarsening problem: experts are grouped by how similarly they respond to shared recommendation states on a short unlabeled calibration stream, while experts that carry heavy routing traffic are protected from repeated merging. Reconstructed experts combine traffic-weighted parameter averaging with a least-squares correction of intermediate activations, and a brief supervised adaptation stage adjusts the resized router and experts. Across three datasets and MoE depths of 2, 4, and 6 blocks, the four-expert checkpoints keep 99.92% to 102.30% of source mean NDCG@10 with 1.28x to 1.63x measured A100 speedups, and the two-expert operating point keeps 98.36% to 104.24% with up to 2.21x speedup. The result matters because it turns a trained recommender checkpoint into a deployable smaller model after training, under an explicit expert budget.

What carries the argument

The load-bearing object is a calibration-conditioned constrained graph coarsening. For each MoE layer, every expert is evaluated on the same calibration states, and the mean/scalar-variance summaries yield affinity scores $s_{ij} = 1/(1+D_{ij})$ from a symmetrized KL divergence; simultaneously, routing mass defines a protected set via the normalized entropy deficit. The greedy average-link coarsening repeatedly merges the most similar admissible pair until the target expert count is reached, with protection acting as a deterministic merge priority rather than a hard constraint. Reconstruction then fuses each cluster into one SwiGLU expert using routing-mass-weighted projection averages plus a least-squares correction $T_m = P^+ Q$ folded into the down projection, so the compressed layer keeps standard top-$k$ routing and no extra online computation. This one machinery couples behavioral compatibility with routing exposure under the same expert-count budget.

What would settle it

Take a trained eight-expert MoE checkpoint and build two UniMoMo conversions from disjoint calibration streams—one drawn from a narrow time window or a single user cohort, one from the full traffic distribution—while keeping the supervised adaptation stage fixed. If the two four-expert checkpoints differ in five-run mean NDCG@10 by more than the matched-baseline gap reported in the paper (about 0.0004–0.0051), the calibration-representativeness premise is falsified; alternatively, replacing the isotropic Gaussian affinity with a full-covariance affinity and showing a material NDCG shift would falsify the summary-statistics premise.

Watch

Extended reading notes

Core claim

The paper's central claim is that a trained sparse MoE recommender can be resized to a smaller standard MoE at deployment time without a compression-specific online module, provided the merge plan is conditioned on recommendation traffic. Instead of comparing expert parameters, UniMoMo runs every expert on the same unlabeled calibration inputs and summarizes each expert's outputs as an isotropic Gaussian (mean vector and scalar variance); the symmetrized KL divergence between these summaries defines an expert-affinity graph. A normalized entropy deficit of routing mass marks high-exposure experts as protected, so the greedy average-link coarsening merges behaviorally similar clusters while limiting repeated modification of traffic-heavy experts. Each merged cluster is initialized by a routing-mass-weighted average of the up, gate, and down projections, corrected by a least-squares map on the intermediate SwiGLU activations, and the router rows are set to the arithmetic centroid before supervised adaptation. The paper reports that this complete conversion-and-adaptation workflow preserves source-relative five-run mean NDCG@10 at 99.92%–102.30% for four-expert checkpoints and 98.36%–104.24% for the aggressive two-expert, top-1 point, with measured A100 speedups of 1.28x–1.63x and 1.47x–2.21x respectively.

Load-bearing premise

The result depends on the 20 unlabeled training batches used for calibration being representative of the states and routing traffic the recommender will see at serving time, and on the mean-plus-scalar-variance summary capturing the expert differences that matter for ranking; behavior absent from that stream cannot influence the merge plan.

Editorial extensions

If this is right

  • A trained eight-expert MoE recommender can be exported to a four-expert, top-2 serving shape that runs 1.28x–1.63x faster while keeping 99.92%–102.30% of source mean NDCG@10.
  • An aggressive two-expert, top-1 deployment is also viable, retaining 98.36%–104.24% of ranking quality with 1.47x–2.21x speedup.
  • Because the compressed model retains standard top-$k$ routing and folds all corrections into stored parameters, it can drop into existing serving pipelines without a compression-specific module.
  • The conversion generalizes across depths of 2, 4, and 6 MoE blocks and across three datasets, and the portability results on four other backbones indicate the partition does not depend on the RankMixer architecture.
  • The ablation shows both calibrated behavior affinity and routing-exposure protection contribute to final ranking, meaning neither signal alone reproduces the full conversion quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same calibration-conditioned coarsening should transfer to MoE language models, since the procedure only needs shared hidden states, routing mass, and a target expert count; a testable extension is applying UniMoMo to a Transformer MoE checkpoint and checking whether the 8-to-4 quality retention carries over.
  • The isotropic Gaussian summary is a strong simplification; comparing merge plans built from full-covariance summaries on a smaller layer would show whether the scalar-variance assumption hides expert differences that matter for ranking.
  • The dependence on a 20-batch calibration stream suggests a stress test: draw the calibration stream from a narrow time window or a single user cohort, and measure how far the resulting NDCG ratios move from the reported 99.92%–102.30% range.
  • Because routing-exposure protection sets per-layer protection from entropy alone, a natural extension is making the target expert count per layer an output of the same entropy signal rather than a fixed global budget.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes UniMoMo, a post-training compression method that converts a trained MoE-based recommendation checkpoint (RankMixer-style, 8 experts per MoE block) into a smaller standard MoE with a user-specified expert budget (4 or 2 experts). The method builds an expert-affinity graph from functional output summaries computed on unlabeled calibration tokens, protects high-exposure experts using a routing-entropy-based schedule, greedily coarsens the graph, reconstructs merged experts via traffic-weighted averaging plus a least-squares activation correction, and then applies a short supervised adaptation. Experiments on Amazon Beauty, KuaiRec, and TenRec with 2, 4, and 6 MoE blocks report source-relative five-run mean NDCG@10 ratios of 99.92%--102.30% for the four-expert conversion and 98.36%--104.24% for the two-expert operating point, with measured A100 speedups of 1.28x--2.21x. An appendix reports results on four additional recommendation backbones, where UniMoMo is claimed to rank first in all 24 reported metrics.

Significance. If the central results hold, the paper makes a useful contribution: it defines a checkpoint-level expert-budget conversion problem for recommendation MoEs, solves it with a deterministic, unlabeled-calibration-driven coarsening procedure, and preserves standard top-k inference with no method-specific online module. Strengths include the explicit serving-budget framing, the joint use of functional affinity and routing exposure, the included complexity analysis, measured latency, and a matched comparison against MergeMoE. However, the manuscript currently contains an internal inconsistency in the source checkpoint that defines every source-relative ratio, and the five-run means are reported without any per-run or variance information. These issues must be resolved before the stated preservation ratios and speedups can be considered reproducible.

major comments (4)
  1. [§4.4, Tables 1 and 3] The 8-expert source is reported inconsistently. Table 3, Amazon Beauty, 8 experts, gives NDCG@10 = 0.2703, HR@10 = 0.4492, AUC = 0.6815, latency = 5.243 ms. Table 1, Amazon Beauty Origin MoE, which is the same six-block eight-expert source with the same 24.97M parameters, gives NDCG@10 = 0.2649, HR@10 = 0.4391, AUC = 0.6778, latency = 4.833 ms. Section 4.4 explicitly uses the Table 3 value to state that four experts retain 99.37% of NDCG@10, while the Abstract, Section 1, Section 4.2, and Section 6 report the lower bound as 99.92%. If Table 3 is the correct reference, the four-expert ratio is 0.2686/0.2703 = 99.37% and the latency speedup is 5.243/3.127 = 1.68x; if Table 1 is the correct reference, Table 3's 8-expert row has no explained provenance. This discrepancy is larger than the 0.08%--2.30% margins being claimed. The authors must identify the correct source reference, recompute all source-relative ratios and speedups, and report per-run values so that the corrected numbers can be checked.
  2. [§4.1--§4.2, Tables 1 and 2] All headline claims are based on five-run means without standard deviations, ranges, or per-run values. The matched gains over MergeMoE are 0.0004, 0.0028, and 0.0051 in NDCG@10, and the ablation differences are 0.0018 and 0.0034, so the central claim rests on differences at the third decimal place. Without any measure of run-to-run variability, it is not possible to determine whether these differences are stable or whether the source-relative ratios are within noise. Please report the per-run numbers or at least standard deviations for the main tables.
  3. [§3.3, Eqs. (4)--(6)] The functional affinity is computed from an isotropic Gaussian summary consisting of a mean vector and a single scalar variance per expert. The memory argument for using a scalar variance is reasonable, but the manuscript provides no evidence that this summary preserves the merge decisions that would be obtained from a full-covariance or otherwise richer affinity measure. Since the affinity graph is the primary input to the coarsening algorithm, this is a load-bearing modeling choice. Please add a validation on at least one dataset/layer comparing the isotropic summary against a full-covariance or alternative affinity measure, or demonstrate that the resulting partition is stable under perturbations of the summary statistics.
  4. [§3.4, Eq. (14)] The protection schedule uses hand-set hyperparameters gamma_min = 0.1, gamma_max = 0.4, and beta = 1, with no sensitivity analysis. Because the ablation study (Table 2) is used to claim that the exposure constraint contributes to the result, the reader needs to know how sensitive the final partition and NDCG@10 are to these choices. Please report a sensitivity sweep over gamma_min, gamma_max, and beta for at least one dataset, or state explicitly that the conclusions are robust across a tested range.
minor comments (6)
  1. [Table 4] The 8-block row in Table 4 repeats NDCG@10 = 0.2703 from Table 3's 8-expert row. Please clarify whether this is a copy error or whether the depth sweep and the expert-count sweep share the same observation.
  2. [Abstract and §1] The Abstract and Section 1 say the experiments cover 2, 4, and 6 MoE blocks, but Table 4 and its surrounding text include 8 blocks. Please make the scope statement consistent.
  3. [Table 5] In the SDIM row, the two numbers '69.6369.03' are concatenated without a separator. This should be formatted as two separate values.
  4. [Figure 3] Figure 3 has no colorbar or axis labels, and the text does not define the displayed similarity scale. Since the figure is presented as a routing-pattern diagnostic, please add a quantitative legend or explicitly state what the cells and shading represent.
  5. [§2.2] Several references in the recommendation-MoE acceleration discussion, such as [22, 23, 24, 52, 59], concern AIGC detection, image/video interpretability, or document intelligence and are not connected to recommendation MoE acceleration. Please remove them or explicitly connect them to the topic.
  6. [§4.1] The sentence 'an internal variable named BPR does not denote pairwise BPR loss' suggests a misleading variable name in the code or protocol description. Please rename the variable or clarify the actual objective function being used.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: UniMoMo's compression partition is derived from unlabeled calibration states and an input expert budget, not from the ranking labels used for evaluation.

full rationale

UniMoMo's derivation is self-contained. The expert-affinity graph is built from functional summaries computed by evaluating all experts on shared calibration tokens (Eqs. 4-6), and the protection mechanism comes from routing mass observed on the same stream (Eqs. 12-14). Neither quantity is defined in terms of the final NDCG@10, and the paper explicitly states that the target expert count is 'a serving budget, not a hyperparameter selected to maximize a ranking metric' (Section 3.1). The protection hyperparameters gamma_min=0.1, gamma_max=0.4, beta=1 are fixed and shared across datasets rather than fitted to the evaluation metric. Reconstruction is a traffic-weighted average plus a least-squares activation correction computed on calibration states (Eqs. 8-10), and the final rankings come from an independent supervised adaptation and held-out evaluation with test interactions excluded. Self-citations in the related-work sections (e.g., refs. 14, 27, 62) are catalog-style mentions of model-merging and pruning work and are not load-bearing for the central claim. The stated limitation that calibration-stream behavior bounds what the partition can see is a genuine robustness caveat, not a circularity. The Table 1 vs Table 3 inconsistency in the Amazon Beauty 8-expert source numbers (0.2649/4.833 ms vs 0.2703/5.243 ms) is an internal consistency and reproducibility concern that affects which baseline anchors the reported ratios, but it does not make the derivation circular: the ratios are still computed from an empirically measured source model and an empirically measured compressed model. No step in the paper reduces by construction to its own input.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities, forces, or conserved quantities. Its assumptions are modeling choices: function-space summaries, entropy-based protection, greedy coarsening, and least-squares reconstruction. The free parameters are shared hyperparameters of the protection schedule and the calibration/adaptation budgets, not per-dataset fits to the evaluation metric. The main unvalidated premise is that the isotropic Gaussian summary and the 20-batch calibration stream capture the expert behaviors that matter for ranking.

free parameters (5)
  • gamma_min = 0.1
    Minimum protected-expert fraction in Eq. 14; chosen by hand and shared across layers and datasets.
  • gamma_max = 0.4
    Maximum protected-expert fraction in Eq. 14; chosen by hand and shared across layers and datasets.
  • beta = 1
    Exponent in the protection schedule Eq. 14; set to 1, with the paper noting the linear map is used because experiments do not tune a transition point.
  • Calibration batch count = 20
    Number of training batches used to profile experts (Sections 3.2 and 3.6); chosen by the authors and it affects all affinity, exposure, and reconstruction estimates.
  • Adaptation epochs = 5
    Fine-tuning budget for all matched baselines and UniMoMo; chosen to give the compressed model a comparable supervised adaptation stage.
assumptions (4)
  • domain assumption Isotropic Gaussian summary (mean vector and scalar variance) is a sufficient statistic for expert functional behavior on calibration states.
    Eq. 4-5; the paper chooses mean plus scalar variance for memory reasons and uses KL divergence on these summaries as the affinity metric. This assumes the summaries capture the decision-relevant differences between experts, which is not validated against full-covariance affinities.
  • ad hoc to paper Routing entropy deficit S_l is an appropriate proxy for the risk of modifying experts, and the gamma_l function in Eq. 14 is a valid protection schedule.
    Eq. 13-14; the paper introduces gamma_min, gamma_max, and beta without deriving them from a principled risk model. The protection rule is a design choice, not a consequence of an optimization problem.
  • domain assumption Greedy average-link coarsening with the protected-vertex merge rule produces a partition that preserves ranking utility after fine-tuning.
    Section 3.3; no optimality or approximation guarantee is given for the greedy partition, and the fallback that disables protection when the target count conflicts with the constraint is a heuristic.
  • domain assumption The least-squares activation correction plus supervised fine-tuning is sufficient to recover the traffic-weighted cluster output.
    Section 3.5; Eq. 16 bounds the reconstruction error but does not bound the final ranking loss after fine-tuning, and the pseudoinverse solution assumes the calibration states span the relevant activation space.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UniMoMo: Expert Merging-Based MoE Acceleration for Large Recommendation Models." pith.science (2026). https://pith.science/paper/2DXJJBOC

@misc{pith2026260808627,
  author       = {Pith},
  title        = {Pith review of: UniMoMo: Expert Merging-Based MoE Acceleration for Large Recommendation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2DXJJBOC}},
  note         = {Machine review of arXiv:2608.08627}
}
abstract

Sparse mixture-of-experts (MoE) layers expand recommendation capacity through conditional computation, yet a trained checkpoint still stores and routes over its full expert bank. We study a deployment problem: convert that checkpoint to a smaller standard MoE under an explicit expert budget, without adding a compression-specific online module. To address this, we introduce UniMoMo, a post-training compression framework formulated as a constrained graph coarsening problem. Rather than relying on parameter distance, UniMoMo groups experts based on their functional similarity, using an unlabeled calibration set to measure how similarly experts respond to shared recommendation states. To prevent performance degradation, we introduce a layer-adaptive protection mechanism that restricts the merging of high-traffic experts based on their routing exposure. Across Amazon Beauty, KuaiRec, and TenRec with 2, 4, and 6 MoE blocks, the final four-expert checkpoints obtain source-relative five-run mean NDCG@10 ratios of 99.92%--102.30% and measured A100 speedups of 1.28$\times$--1.63$\times$. An aggressive two-expert, top-1 operating point obtains ratios of 98.36%--104.24% and speedups of 1.47$\times$--2.21$\times$. These endpoint results evaluate the complete conversion-and-adaptation workflow and show that a trained recommendation MoE can be exported at multiple serving budgets.

Figures

Figures reproduced from arXiv: 2608.08627 by the authors.

Figure 1
Figure 1. Input–output overview. a) Offline, UniMoMo com [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of UniMoMo. Given a trained MoE recommender and unlabeled calibration interactions, dual-view profiling [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Layer-0 routing-pattern diagnostic on the same cali [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Efficiency-normalized performance across MoE [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

88 extracted references · 24 canonical work pages

  1. [1]

    Szymon Antoniak, Michał Krutul, Maciej Pióro, Jakub Krajewski, Jan Ludziejew- ski, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóźdź, Marek Cygan, and Sebas- tian Jaszczur. 2024. Mixture of tokens: Continuous moe through cross-example aggregation.Advances in Neural Information Processing Systems37 (2024), 103873– 103896

    2024

  2. [2]

    Arindam Banerjee, Srujana Merugu, Inderjit S Dhillon, and Joydeep Ghosh. 2005. Clustering with Bregman divergences.Journal of machine learning research6, Oct (2005), 1705–1749

    2005

  3. [3]

    I Chen, Hsu-Shen Liu, Wei-Fang Sun, Chen-Hao Chao, Yen-Chang Hsu, Chun-Yi Lee, et al. 2024. Retraining-free merging of sparse moe via hierarchical clustering. arXiv preprint arXiv:2410.08589(2024)

    arXiv 2024

  4. [4]

    Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al

  5. [5]

    Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. 2024. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. InProceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1280–1297

    2024

  6. [6]

    Gabriel de Souza Pereira Moreira, Sara Rabhi, Jeong Min Lee, Ronay Ak, and Even Oldridge. 2021. Transformers4rec: Bridging the gap between nlp and sequential/session-based recommendation. InProceedings of the 15th ACM con- ference on recommender systems. 143–153

    2021

  7. [7]

    Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al

  8. [8]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research23, 120 (2022), 1–39

    2022

Show all 88 references
  1. [9]

    Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. InInternational conference on machine learning. PMLR, 10323–10337

    2023

  2. [10]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323(2022)

    2022 arXiv

  3. [11]

    Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. 2023. Megablocks: Efficient sparse training with mixture-of-experts.Proceedings of Machine Learning and Systems5 (2023), 288–304

    2023

  4. [12]

    Chongming Gao, Shijun Li, Wenqiang Lei, Jiawei Chen, Biao Li, Peng Jiang, Xiangnan He, Jiaxin Mao, and Tat-Seng Chua. 2022. KuaiRec: A fully-observed dataset and insights for evaluating recommender systems. InProceedings of the 31st ACM International Conference on Information ...

    2022

  5. [13]

    Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. DeepFM: a factorization-machine based neural network for CTR prediction.arXiv preprint arXiv:1703.04247(2017)

    2017 arXiv

  6. [14]

    Haiyang Guo, Fanhu Zeng, Ziwei Xiang, Fei Zhu, Da-Han Wang, Xu-Yao Zhang, and Cheng-Lin Liu. 2025. Hide-llava: Hierarchical decoupling for continual instruction tuning of multimodal large language model. InProceedings of the 63rd Annual Meeting of the Association for Computati...

    2025

  7. [15]

    Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdhery, Maheswaran Sathiamoor- thy, Yihua Chen, Rahul Mazumder, Lichan Hong, and Ed Chi. 2021. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning.Advances in Neural Information Processi...

    2021

  8. [16]

    Shwai He, Run-Ze Fan, Liang Ding, Li Shen, Tianyi Zhou, and Dacheng Tao

  9. [17]

    Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. InProceedings of the 26th international conference on world wide web. 173–182

    2017

  10. [18]

    Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk

  11. [19]

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2022. Editing models with task arithmetic.arXiv preprint arXiv:2212.04089(2022)

    2022 arXiv

  12. [20]

    Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. 1991. Adaptive mixtures of local experts.Neural computation3, 1 (1991), 79–87

    1991

  13. [21]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al . 2024. Mixtral of experts.arXiv preprint 9 Conference’17, July 2017, Washington, DC, USA Lei...

    2024 arXiv

  14. [22]

    Changjiang Jiang, Wenhui Dong, Zhonghao Zhang, Fengchang Yu, Wei Peng, Xinbin Yuan, Yifei Bi, Ming Zhao, Zian Zhou, Chenyang Si, and Caifeng Shan

  15. [23]

    Changjiang Jiang, Xinkuan Sha, Fengchang Yu, Jingjing Liu, Jian Liu, Mingqi Fang, Chenfeng Zhang, and Wei Lu. 2026. Fake-HR1: Rethinking Reasoning of Vision Language Model for Synthetic Image Detection. InICASSP 2026 - 2026 IEEE International Conference on Acoustics, Speech an...

    2026

  16. [24]

    Changjiang Jiang, Fengchang Yu, Haihua Chen, Wei Lu, and Jin Zeng. 2025. TABDSR: Decompose, Sanitize, and Reason for Complex Numerical Reasoning in Tabular Data. InFindings of the Association for Computational Linguistics: EMNLP

    2025

  17. [25]

    Yuchen Jiang, Jie Zhu, Xintian Han, Hui Lu, Kunmin Bai, Mingyu Yang, Shikang Wu, Ruihao Zhang, Wenlin Zhao, Shipeng Bai, et al. 2026. TokenMixer-Large: Scaling Up Large Ranking Models in Industrial Recommenders.arXiv preprint arXiv:2602.06563(2026)

    2026

  18. [26]

    Wang-Cheng Kang and Julian McAuley. 2018. Self-attentive sequential recom- mendation. In2018 IEEE international conference on data mining (ICDM). IEEE, 197–206

    2018

  19. [27]

    Zhenglun Kong, Yize Li, Fanhu Zeng, Lei Xin, Shvat Messica, Xue Lin, Pu Zhao, Manolis Kellis, Hao Tang, and Marinka Zitnik. 2025. Token Reduction Should Go Beyond Efficiency in Generative Models–From Vision, Language to Multimodality. arXiv preprint arXiv:2505.18227(2025)

    2025

  20. [28]

    Yehuda Koren, Robert Bell, and Chris Volinsky. 2009. Matrix factorization tech- niques for recommender systems.Computer42, 8 (2009), 30–37

    2009

  21. [29]

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional computation and automatic sharding.arXiv preprint arXiv:2006.16668(2020)

    2020 arXiv

  22. [30]

    Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer

  23. [31]

    Jiacheng Li, Yujie Wang, and Julian McAuley. 2020. Time interval aware self- attention for sequential recommendation. InProceedings of the 13th international conference on web search and data mining. 322–330

    2020

  24. [32]

    Lujun Li, Qiyuan Zhu, Jiacheng Wang, Xiaoyu Qin, Wei Li, Hao Gu, Sirui Han, and Yike Guo. 2026. Sub-moe: Efficient mixture-of-expert llms compression via subspace expert merging. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. 22994–23002

    2026

  25. [33]

    Xiao Lin, Zhicheng Tang, Weilin Cong, Mengyue Hang, Kai Wang, Yajuan Wang, Zhichen Zeng, Ting-Wei Li, Hyunsik Yoo, Zhining Liu, et al. 2026. Mixture of Se- quence: Theme-Aware Mixture-of-Experts for Long-Sequence Recommendation. InProceedings of the ACM Web Conference 2026. 6469–6480

    2026

  26. [34]

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al . 2024. Deepseek- v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434(2024)

    2024 arXiv

  27. [35]

    Andreas Loukas. 2019. Graph reduction with spectral and cut guarantees.Journal of Machine Learning Research20, 116 (2019), 1–42

    2019

  28. [36]

    Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. 2018. Modeling task relationships in multi-task learning with multi-gate mixture-of- experts. InProceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 1930–1939

    2018

  29. [37]

    Ruijie Miao, Yilun Yao, Zihan Wang, Zhiming Wang, Bairen Yi, LingJun Liu, Yikai Zhao, and Tong Yang. 2025. MergeMoE: Efficient Compression of MoE Models via Expert Output Merging.arXiv preprint arXiv:2510.14436(2025)

    2025

  30. [38]

    Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, and Neil Houlsby. 2024. From sparse to soft mixtures of experts. InInternational Conference on Learning Representations, Vol. 2024. 28435–28445

    2024

  31. [39]

    Chenfan Qu, Yiwu Zhong, Xuekang Zhu, Junchi Li, Changjiang Jiang, Lianwen Jin, et al. 2026. Detect Any AI-Counterfeited Text Image. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 35437–35450

    2026

  32. [40]

    Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Am- inabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. 2022. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale. InInternational conference on machine...

    2022

  33. [41]

    David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. 2024. Mixture-of-depths: Dynamically allocating compute in transformer-based language models.arXiv preprint arXiv:2404.02258 (2024)

    2024 arXiv

  34. [42]

    Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme

  35. [43]

    Stephen Roller, Sainbayar Sukhbaatar, Jason Weston, et al. 2021. Hash layers for large sparse models.advances in neural information processing systems34 (2021), 17555–17566

    2021

  36. [44]

    Noam Shazeer. 2020. Glu variants improve transformer.arXiv preprint arXiv:2002.05202(2020)

    2020 arXiv

  37. [45]

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538(2017)

    2017 arXiv

  38. [46]

    Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. 2019. Autoint: Automatic feature interaction learning via self- attentive neural networks. InProceedings of the 28th ACM international conference on information and knowledge management....

    2019

  39. [47]

    Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang

  40. [48]

    Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. 2024. A simple and effective pruning approach for large language models. InInternational Conference on Learning Representations, Vol. 2024. 4942–4964

    2024

  41. [49]

    Shawn Tan, Yikang Shen, Rameswar Panda, and Aaron Courville. 2024. Scattered mixture-of-experts implementation.arXiv preprint arXiv:2403.08245(2024)

    2024 arXiv

  42. [50]

    Hongyan Tang, Junning Liu, Ming Zhao, and Xudong Gong. 2020. Progressive layered extraction (ple): A novel multi-task learning (mtl) model for personalized recommendations. InProceedings of the 14th ACM conference on recommender systems. 269–278

    2020

  43. [51]

    Jiaxi Tang and Ke Wang. 2018. Personalized top-n sequential recommenda- tion via convolutional sequence embedding. InProceedings of the eleventh ACM international conference on web search and data mining. 565–573

    2018

  44. [52]

    GuangJian Team. 2026. Venus-DeFakerOne: Unified Fake Image Detection & Localization.arXiv preprint arXiv:2605.14091(2026)

    2026 arXiv

  45. [53]

    Ulrike Von Luxburg. 2007. A tutorial on spectral clustering.Statistics and computing17, 4 (2007), 395–416

    2007

  46. [54]

    Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. 2017. Deep & cross network for ad click predictions. InProceedings of the ADKDD’17. 1–7

    2017

  47. [55]

    Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. 2021. Dcn v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. InProceedings of the web conference 2021. 1785–1797

    2021

  48. [56]

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Si- mon Kornblith, et al. 2022. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increas...

    2022

  49. [57]

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal

  50. [58]

    Xican Yang, Yuanhe Tian, and Yan Song. 2025. MoE Pathfinder: Trajectory-driven Expert Pruning.arXiv preprint arXiv:2512.18425(2025)

    2025

  51. [59]

    Dehao Ying, Fengchang Yu, Haihua Chen, Changjiang Jiang, Yurong Li, and Wei Lu. 2026. Beyond Human Annotation: Recent Advances in Data Generation Methods for Document Intelligence.arXiv preprint arXiv:2601.12318(2026)

    2026

  52. [60]

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning

    2024

  53. [61]

    Guanghu Yuan, Fajie Yuan, Yudong Li, Beibei Kong, Shujie Li, Lei Chen, Min Yang, Chenyun Yu, Bo Hu, Zang Li, et al . 2022. Tenrec: A large-scale multipurpose benchmark dataset for recommender systems.Advances in Neural Information Processing Systems35 (2022), 11480–11493

    2022

  54. [62]

    Fanhu Zeng, Haiyang Guo, Fei Zhu, Li Shen, and Hao Tang. 2025. Robustmerge: Parameter-efficient model merging for mllms with direction robustness.Advances in Neural Information Processing Systems38 (2025), 71071–71095

    2025

  55. [63]

    Buyun Zhang, Liang Luo, Yuxin Chen, Jade Nie, Xi Liu, Daifeng Guo, Yanli Zhao, Shen Li, Yuchen Hao, Yantao Yao, et al. 2024. Wukong: Towards a scaling law for large-scale recommendation.arXiv preprint arXiv:2403.02545(2024)

    2024 arXiv

  56. [64]

    Buyun Zhang, Liang Luo, Xi Liu, Jay Li, Zeliang Chen, Weilin Zhang, Xiaohan Wei, Yuchen Hao, Michael Tsang, Wenjun Wang, et al. 2022. DHEN: A deep and hierarchical ensemble network for large-scale click-through rate prediction. arXiv preprint arXiv:2203.11014(2022)

    2022 arXiv

  57. [65]

    Geng Zhang, Yuxuan Han, Yuxuan Lou, Yiqi Zhang, Wangbo Zhao, and Yang You

  58. [66]

    Junjie Zhang, Ruobing Xie, Hongyu Lu, Wenqi Sun, Wayne Xin Zhao, Yu Chen, and Zhanhui Kang. 2025. Frequency-augmented mixture-of-heterogeneous- experts framework for sequential recommendation. InProceedings of the ACM on Web Conference 2025. 2596–2605

    2025

  59. [67]

    Ties-merging: Resolving interference when merging models.Advances in neural information processing systems36 (2023), 7093–7115

    2023

  60. [68]

    Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou

  61. [69]

    Yushu Zhao, Zheng Wang, and Minjia Zhang. 2025. PuzzleMoE: Efficient Com- pression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference.arXiv preprint arXiv:2511.04805(2025)

    2025 arXiv

  62. [70]

    Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. 2019. Deep interest evolution network for click-through rate prediction. InProceedings of the AAAI conference on artificial intelligence, Vol. 33. 5941–5948

    2019

  63. [71]

    Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep interest network for click-through rate prediction. InProceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. ...

    2018

  64. [72]

    Kun Zhou, Hui Yu, Wayne Xin Zhao, and Ji-Rong Wen. 2022. Filter-enhanced MLP is all you need for sequential recommendation. InProceedings of the ACM web conference 2022. 2388–2399

    2022

  65. [73]

    Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, An- drew M Dai, Quoc V Le, James Laudon, et al . 2022. Mixture-of-experts with expert choice routing.Advances in Neural Information Processing Systems35 (2022), 7103–7114

    2022

  66. [74]

    Jie Zhu, Zhifang Fan, Xiaoxie Zhu, Yuchen Jiang, Hangyu Wang, Xintian Han, Haoran Ding, Xinmin Wang, Wenlin Zhao, Zhen Gong, et al. 2025. Rankmixer: Scaling up ranking models in industrial recommenders. InProceedings of the 34th ACM International Conference on Information and ...

    2025

  67. [75]

    Barret Zoph. 2022. Designing effective sparse expert models. In2022 IEEE In- ternational Parallel and Distributed Processing Symposium Workshops (IPDPSW). IEEE, 1044–1044

    2022

  68. [76]

    Mone: Replacing redundant experts with lightweight novices for structured pruning of moe.arXiv preprint arXiv:2507.00390(2025)

    2025

  69. [78]

    Yukun Zhang, Si Dong, Xu Wang, Bo Chen, Qinglin Jia, Shengzhe Wang, Jinlong Jiao, Runhan Li, Jiaqing Liu, Chaoyi Ma, et al . 2026. SMES: Towards Scalable Multi-Task Recommendation via Expert Sparsity.arXiv preprint arXiv:2602.09386 10 UniMoMo: Expert Merging-Based MoE Accelera...

    2026

  70. [80]

    In Findings of the Association for Computational Linguistics: ACL 2022

    Moefication: Transformer feed-forward layers are mixtures of experts. In Findings of the Association for Computational Linguistics: ACL 2022. 877–890

    2022

  71. [88]

    Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. 2022. St-moe: Designing stable and transfer- able sparse expert models.arXiv preprint arXiv:2202.08906(2022). 11 Conference’17, July 2017, Washington, DC, USA Lei Xin, Bi...

    2022 arXiv

  72. [2012]

    BPR: Bayesian personalized ranking from implicit feedback.arXiv preprint arXiv:1205.2618(2012)

    2012 arXiv

  73. [2015]

    Session-based recommendations with recurrent neural networks.arXiv preprint arXiv:1511.06939(2015)

    2015 arXiv

  74. [2016]

    InProceedings of the 1st workshop on deep learning for recommender systems

    Wide & deep learning for recommender systems. InProceedings of the 1st workshop on deep learning for recommender systems. 7–10

  75. [2019]

    InProceedings of the 28th ACM international conference on information and knowledge management

    BERT4Rec: Sequential recommendation with bidirectional encoder rep- resentations from transformer. InProceedings of the 28th ACM international conference on information and knowledge management. 1441–1450

  76. [2021]

    InInternational Conference on Machine Learning

    Base layers: Simplifying training of large, sparse models. InInternational Conference on Machine Learning. PMLR, 6265–6274

  77. [2022]

    In International conference on machine learning

    Glam: Efficient scaling of language models with mixture-of-experts. In International conference on machine learning. PMLR, 5547–5569

  78. [2023]

    InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

    Merging experts into one: Improving computational efficiency of mixture of experts. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 14685–14691

    2023

  79. [2025]

    doi:10.18653/v1/2025.findings-emnlp.169

    3172–3196. doi:10.18653/v1/2025.findings-emnlp.169

    2025 doi

  80. [2026]

    InProceedings of the 2026 International Conference on Mul- timedia Retrieval (ICMR ’26)

    Ivy-Fake: A Unified Explainable Framework and Benchmark for Image and Video AIGC Detection. InProceedings of the 2026 International Conference on Mul- timedia Retrieval (ICMR ’26). Association for Computing Machinery, 2438–2447. doi:10.1145/3805622.3810615

    2026

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.