Pith. sign in

REVIEW 19 cited by

From Sparse to Soft Mixtures of Experts

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2308.00951 v2 pith:RRYTNBHY submitted 2023-08-02 cs.LG cs.AIcs.CV

Joan Puigcerver , Carlos Riquelme , Basil Mustafa , Neil Houlsby This is my paper
classification cs.LGcs.AIcs.CV
keywords softexpertsmoesinferencesparsetokenscapacitychoice
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Sparse mixture of expert architectures (MoEs) scale model capacity without significant increases in training or inference costs. Despite their success, MoEs suffer from a number of issues: training instability, token dropping, inability to scale the number of experts, or ineffective finetuning. In this work, we propose Soft MoE, a fully-differentiable sparse Transformer that addresses these challenges, while maintaining the benefits of MoEs. Soft MoE performs an implicit soft assignment by passing different weighted combinations of all input tokens to each expert. As in other MoEs, experts in Soft MoE only process a subset of the (combined) tokens, enabling larger model capacity (and performance) at lower inference cost. In the context of visual recognition, Soft MoE greatly outperforms dense Transformers (ViTs) and popular MoEs (Tokens Choice and Experts Choice). Furthermore, Soft MoE scales well: Soft MoE Huge/14 with 128 experts in 16 MoE layers has over 40x more parameters than ViT Huge/14, with only 2% increased inference time, and substantially better quality.

Discussion (0). Sign in to comment.

Forward citations

Cited by 19 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. GeMoE: Gating Entropy is All You Need for Uncertainty-aware Adaptive Routing in MoE-based Large Vision-Language Models

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    GeMoE adaptively sets the number of experts per token via gating entropy, retaining 99.5% of static-routing performance while raising average sparsity by 36.5%.

  2. SplatWeaver: Learning to Allocate Gaussian Primitives for Generalizable Novel View Synthesis

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    SplatWeaver dynamically allocates Gaussian primitives via cardinality experts and pixel-level routing guided by high-frequency cues for improved generalizable novel view synthesis.

  3. SplatWeaver: Learning to Allocate Gaussian Primitives for Generalizable Novel View Synthesis

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    SplatWeaver uses cardinality Gaussian experts and pixel-level routing to dynamically allocate varying numbers of Gaussian primitives for generalizable novel view synthesis.

  4. When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    Standard top-k routers in MoE language models often select suboptimal routes for difficult tokens, and updating only the final router layer raises pass@K on AIME and HMMT benchmarks across multiple models.

  5. Agent-Centric Observation Adaptation for Robust Visual Control under Dynamic Perturbations

    cs.RO 2026-04 unverdicted novelty 7.0 of 10

    ACO-MoE recovers 95.3% of clean-input performance in visual control tasks under Markov-switching corruptions by routing restoration experts and anchoring representations to clean foreground masks.

  6. Agent-Centric Observation Adaptation for Robust Visual Control under Dynamic Perturbations

    cs.RO 2026-04 unverdicted novelty 7.0 of 10

    ACO-MoE employs agent-centric mixture-of-experts to decouple task-relevant features from dynamic visual perturbations in RL, recovering 95.3% of clean performance on the new VDCS benchmark.

  7. B-MoE: A Body-Part-Aware Mixture-of-Experts "All Parts Matter" Approach to Micro-Action Recognition

    cs.CV 2026-03 unverdicted novelty 7.0 of 10

    B-MoE framework achieves state-of-the-art performance on micro-action recognition by using region-specific experts and cross-attention routing.

  8. Path-Constrained Mixture-of-Experts

    cs.LG 2026-03 unverdicted novelty 7.0 of 10

    PathMoE constrains expert paths in MoE models by sharing router parameters across layer blocks, yielding more concentrated paths, better performance on perplexity and tasks, and no need for auxiliary losses.

  9. Mixture of Cognitive Experts in Large Vision-Language Models

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Routing CV experts into atomic evidence then Bloom-staged verbalization improves LVLM benchmarks and yields measurable query-conditioned reasoning traces.

  10. FaceMoE: Mixture of Experts for Low-Resolution Face Recognition

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    FaceMoE introduces a MoE transformer with top-k routed specialized FFN experts for resolution-aware feature extraction in low-resolution face recognition, outperforming prior methods on eleven datasets.

  11. SCOPE and SCION: A Benchmark and an Auditable Reference Pipeline for Schema Induction and Fusion from Text

    cs.AI 2026-05 conditional novelty 6.0 of 10

    A 24-dataset benchmark for inducing schema graphs from raw text, plus an auditable LLM-based pipeline that reports the highest scores on the benchmark's four schema-similarity metrics.

  12. AGoQ: Activation and Gradient Quantization for Memory-Efficient Distributed Training of LLMs

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    AGoQ delivers up to 52% lower memory use and 1.34x faster training for 8B-32B LLaMA models by using near-4-bit adaptive activations and 8-bit gradients while preserving pretraining convergence and downstream accuracy.

  13. Design and Behavior of Sparse Mixture-of-Experts Layers in CNN-based Semantic Segmentation

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    Patch-wise sparse MoE layers in CNNs for semantic segmentation yield architecture-dependent gains up to 3.9 mIoU on Cityscapes and BDD100K with low overhead, but show strong design sensitivity.

  14. MP-ISMoE: Mixed-Precision Interactive Side Mixture-of-Experts for Efficient Transfer Learning

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    MP-ISMoE uses Gaussian noise perturbed iterative quantization and interactive side mixture-of-experts to deliver higher accuracy than prior memory-efficient transfer learning methods while keeping similar parameter an...

  15. HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap

    cs.DC 2025-08 conditional novelty 6.0 of 10

    HierMoE reduces MoE training time by removing duplicate token copies at each GPU-hierarchy level and swapping experts for load balance, measured at 1.18-1.27x end-to-end speedup on 32 GPUs.

  16. Tight Clusters Make Specialized Experts

    cs.LG 2025-02 unverdicted novelty 6.0 of 10

    Introduces Adaptive Clustering router for MoE models that scales features to identify tight expert clusters, yielding faster convergence, robustness to corruption, and performance gains.

  17. Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Survey

    cs.LG 2026-05 accept novelty 5.0 of 10

    A literature survey that categorizes how Mixture-of-Experts architectures address multimodal learning challenges and identifies open research gaps.

  18. AGoQ: Activation and Gradient Quantization for Memory-Efficient Distributed Training of LLMs

    cs.CL 2026-05 unverdicted novelty 5.0 of 10

    AGoQ cuts LLM training memory by up to 52% and speeds it up by 1.34x using tailored 4-bit activations and 8-bit gradients with special communication, matching baseline accuracy on LLaMA models.

  19. YOLO-Master: MOE-Accelerated with Specialized Transformers for Enhanced Real-time Detection

    cs.CV 2025-12 conditional novelty 4.0 of 10

    YOLO-Master inserts a sparse Mixture-of-Experts block into a YOLO backbone, reporting 42.4% COCO AP at 1.62 ms, +0.8 AP and 18% faster than YOLOv13-N.

Pith tools