REVIEW 2 cited by
Scattered Mixture-of-Experts Implementation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We present ScatterMoE, an implementation of Sparse Mixture-of-Experts (SMoE) on GPUs. ScatterMoE builds upon existing implementations, and overcoming some of the limitations to improve inference and training speed, and memory footprint. This implementation achieves this by avoiding padding and making excessive copies of the input. We introduce ParallelLinear, the main component we use to build our implementation and the various kernels used to speed up the operation. We benchmark our implementation against Megablocks, and show that it enables a higher throughput and lower memory footprint. We also show how ParallelLinear enables extension of the Mixture-of-Experts concept by demonstrating with an implementation of Mixture of Attention.
Forward citations
Cited by 2 Pith papers
-
Cross-Platform Fused MoE Dispatch in Triton: Portable Expert Routing Without CUDA
A pure-Triton fused MoE dispatch kernel reaches 89–131% of CUDA Megablocks throughput at inference batch sizes and passes all tests on both NVIDIA A100 and AMD MI300X with zero code changes.
-
Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
At 56B total parameters, fine-grained MoE with smaller, more numerous experts beats standard Switch and Mixtral-style MoE on validation loss and average downstream accuracy at matched FLOPs.
Discussion (0). Continue with ORCID to comment.