Pith. sign in

REVIEW 6 cited by

ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2412.14711 v2 pith:R3FCDIIU submitted 2024-12-19 cs.LG

Ziteng Wang , Jun Zhu , Jianfei Chen This is my paper
classification cs.LG
keywords remoeacrosscomputationdifferentiableexpertsfullymixture-of-expertsmodel
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Sparsely activated Mixture-of-Experts (MoE) models are widely adopted to scale up model capacity without increasing the computation budget. However, vanilla TopK routers are trained in a discontinuous, non-differentiable way, limiting their performance and scalability. To address this issue, we propose ReMoE, a fully differentiable MoE architecture that offers a simple yet effective drop-in replacement for the conventional TopK+Softmax routing, utilizing ReLU as the router instead. We further propose methods to regulate the router's sparsity while balancing the load among experts. ReMoE's continuous nature enables efficient dynamic allocation of computation across tokens and layers, while also exhibiting domain specialization. Our experiments demonstrate that ReMoE consistently outperforms vanilla TopK-routed MoE across various model sizes, expert counts, and levels of granularity. Furthermore, ReMoE exhibits superior scalability with respect to the number of experts, surpassing traditional MoE architectures. The implementation based on Megatron-LM is available at https://github.com/thu-ml/ReMoE.

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DTop-p MoE: Sparsity-Controlled Dynamic Top-p MoE for Foundation Model Pre-training

    cs.AI 2025-12 conditional novelty 6.0 of 10

    A PI-controller-tuned, layerwise-normalized Top-p router trains sparse MoE models that beat Top-k at matched average activated-expert count.

  2. Robust Experts: the Effect of Adversarial Training on CNNs with Sparse Mixture-of-Experts Layers

    cs.CV 2025-09 conditional novelty 6.0 of 10

    Single deep MoE layers improve PGD/AutoPGD robust accuracy of adversarially trained ResNets on CIFAR-100, and routing collapse under switch loss produces individual experts that are more robust than the full MoE.

  3. RankMixer: Scaling Up Ranking Models in Industrial Recommenders

    cs.IR 2025-07 conditional novelty 6.0 of 10

    RankMixer scales an industrial ranking model to 1B dense parameters with 10x MFU improvement and unchanged latency, gaining 1.08% in app duration in Douyin A/B tests.

  4. BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A ReLU-routed MoE with chunk-level sparsity training objectives and custom kernels combining activation sparsity with speculative decoding achieves over 70% 8-token chunk sparsity and up to 3.67x end-side speedup.

  5. Maximum Score Routing For Mixture-of-Experts

    cs.LG 2025-08 unverdicted novelty 5.0 of 10

    MaxScore casts MoE routing as min-cost max-flow with SoftTopk and claims better loss and eval scores at equal FLOPs; unverified because the full text is unreadable.

  6. Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    Grove MoE uses unequal-size adjugate experts with complexity-based activation to run 33B-parameter models at roughly 3.1 to 3.3B active parameters while matching larger open models in benchmarks.

Pith tools