REVIEW 3 cited by
Expert Race: A Flexible Routing Strategy for Scaling Diffusion Transformer with Mixture of Experts
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Expert Race: A Flexible Routing Strategy for Scaling Diffusion Transformer with Mixture of Experts
read the original abstract
Diffusion models have emerged as mainstream framework in visual generation. Building upon this success, the integration of Mixture of Experts (MoE) methods has shown promise in enhancing model scalability and performance. In this paper, we introduce Race-DiT, a novel MoE model for diffusion transformers with a flexible routing strategy, Expert Race. By allowing tokens and experts to compete together and select the top candidates, the model learns to dynamically assign experts to critical tokens. Additionally, we propose per-layer regularization to address challenges in shallow layer learning, and router similarity loss to prevent mode collapse, ensuring better expert utilization. Extensive experiments on ImageNet validate the effectiveness of our approach, showcasing significant performance gains while promising scaling properties.
Forward citations
Cited by 3 Pith papers
-
Focusing on What Matters: Saliency-Harnessing Accurate Routing for Diffusion MoE
SharpMoE is a plug-and-play post-training method that uses clean latent features and a trajectory routing loss to enable accurate saliency-based routing in diffusion MoE models for improved visual generation.
-
Expert-Data Alignment Governs Generation Quality in Decentralized Diffusion Models
Sparse Top-2 routing beats full ensemble in decentralized diffusion models, and the paper attributes this to expert-data alignment rather than numerical stability — though much of the supporting evidence is circular.
-
UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learning
A redesigned memory-layer architecture with five engineering improvements reaches performance parity with 8-expert MoE at similar compute, with lower memory access and stronger long-context memorization.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.