REVIEW 8 cited by
From Sparse to Soft Mixtures of Experts
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Sparse mixture of expert architectures (MoEs) scale model capacity without significant increases in training or inference costs. Despite their success, MoEs suffer from a number of issues: training instability, token dropping, inability to scale the number of experts, or ineffective finetuning. In this work, we propose Soft MoE, a fully-differentiable sparse Transformer that addresses these challenges, while maintaining the benefits of MoEs. Soft MoE performs an implicit soft assignment by passing different weighted combinations of all input tokens to each expert. As in other MoEs, experts in Soft MoE only process a subset of the (combined) tokens, enabling larger model capacity (and performance) at lower inference cost. In the context of visual recognition, Soft MoE greatly outperforms dense Transformers (ViTs) and popular MoEs (Tokens Choice and Experts Choice). Furthermore, Soft MoE scales well: Soft MoE Huge/14 with 128 experts in 16 MoE layers has over 40x more parameters than ViT Huge/14, with only 2% increased inference time, and substantially better quality.
Forward citations
Cited by 8 Pith papers
-
Mixture of Cognitive Experts in Large Vision-Language Models
Routing CV experts into atomic evidence then Bloom-staged verbalization improves LVLM benchmarks and yields measurable query-conditioned reasoning traces.
-
SCOPE and SCION: A Benchmark and an Auditable Reference Pipeline for Schema Induction and Fusion from Text
A 24-dataset benchmark for inducing schema graphs from raw text, plus an auditable LLM-based pipeline that reports the highest scores on the benchmark's four schema-similarity metrics.
-
HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap
HierMoE reduces MoE training time by removing duplicate token copies at each GPU-hierarchy level and swapping experts for load balance, measured at 1.18-1.27x end-to-end speedup on 32 GPUs.
-
Automatic Expert Discovery in LLM Upcycling via Sparse Interpolated Mixture-of-Experts
Sparse Interpolated Mixture-of-Experts learns neuron-level masks and a per-prompt router to upcycle a dense LLM into a sparse expert model, improving instruction-tuning benchmarks with lower inference cost.
-
FairHuman: Boosting Hand and Face Quality in Human Image Generation with Minimum Potential Delay Fairness in Diffusion Models
A multi-objective fine-tuning method with Minimum Potential Delay fairness improves hand and face quality in human image generation while maintaining global quality.
-
Dynamical Multimodal Fusion with Mixture-of-Experts for Localizations
A mixture-of-experts network that adaptively fuses wireless signal fingerprints across frequency bands and jointly localizes short trajectories achieves sub-meter errors on simulated urban 6G test cases.
-
NaSh: Guardrails for an LLM-Powered Natural Language Shell
NaSh is a proposed natural-language shell whose key mechanism, the inverse overlay, keeps old copies of changed files so LLM-generated commands can be inspected, committed, or undone.
-
YOLO-Master: MOE-Accelerated with Specialized Transformers for Enhanced Real-time Detection
YOLO-Master inserts a sparse Mixture-of-Experts block into a YOLO backbone, reporting 42.4% COCO AP at 1.62 ms, +0.8 AP and 18% faster than YOLOv13-N.
Discussion (0). Sign in to comment.