REVIEW 13 cited by
Scaling Diffusion Transformers to 16 Billion Parameters
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
In this paper, we present DiT-MoE, a sparse version of the diffusion Transformer, that is scalable and competitive with dense networks while exhibiting highly optimized inference. The DiT-MoE includes two simple designs: shared expert routing and expert-level balance loss, thereby capturing common knowledge and reducing redundancy among the different routed experts. When applied to conditional image generation, a deep analysis of experts specialization gains some interesting observations: (i) Expert selection shows preference with spatial position and denoising time step, while insensitive with different class-conditional information; (ii) As the MoE layers go deeper, the selection of experts gradually shifts from specific spacial position to dispersion and balance. (iii) Expert specialization tends to be more concentrated at the early time step and then gradually uniform after half. We attribute it to the diffusion process that first models the low-frequency spatial information and then high-frequency complex information. Based on the above guidance, a series of DiT-MoE experimentally achieves performance on par with dense networks yet requires much less computational load during inference. More encouragingly, we demonstrate the potential of DiT-MoE with synthesized image data, scaling diffusion model at a 16.5B parameter that attains a new SoTA FID-50K score of 1.80 in 512$\times$512 resolution settings. The project page: https://github.com/feizc/DiT-MoE.
Forward citations
Cited by 13 Pith papers
-
SonicWeave: Chunk-Routed Mixture-of-Experts for Unified Audio Scene Generation
SonicWeave routes chunks of audio through specialized experts, using a learned gate between text-derived prior and local evidence, improving compositional fidelity in unified text-to-audio scene generation.
-
X-Stage: An Overlooked Pipeline Stage for Communication-Computation Overlap in DiT Inference
X-Stage is a measurable post-issue window for GPU remote stores; a Burst-Gap model predicts backpressure and guides scheduling, yielding up to 1.62x MoE and 1.43x attention speedups.
-
What Makes for Text to 360-degree Panorama Generation with Stable Diffusion?
Freezing query and key LoRA weights and strengthening the output weight with a mixture of experts gives a memory-efficient, state-of-the-art text-to-360-degree-panorama generator.
-
TinyFusion: Diffusion Transformers Learned Shallow
A learnable depth-pruning method that optimizes post-fine-tuning recoverability produces a 14-layer DiT-XL with FID 2.86 and a 2x speedup at 7% of the original training cost.
-
Unveiling the Secret of AdaLN-Zero in Diffusion Transformer
Zero-initializing the modulation weights is the dominant reason adaLN-Zero outperforms adaLN, and replacing it with a Gaussian initialization of std 0.001 improves FID at the same training steps.
-
Exploring Diffusion Transformer Designs via Grafting
Grafting uses activation distillation and lightweight fine-tuning to edit pretrained diffusion transformers into hybrid architectures with near-baseline quality at under 2% pretraining compute.
-
InfLVG: Reinforce Inference-Time Consistent Long Video Generation with GRPO
InfLVG uses a GRPO-optimized context selection policy to choose top-K relevant video tokens for consistent, prompt-aligned long video generation.
-
Ingredients: Blending Custom Photos with Video Diffusion Transformers
Ingredients adds a mask-supervised identity router to a video diffusion transformer, enabling multi-person videos from a few reference photos without per-identity fine-tuning.
-
Mr. DETR++: Instructive Multi-Route Training for Detection Transformers with Mixture-of-Experts
Multi-route training with instructive self-attention tokens and a route-aware mixture-of-experts raises detection mAP by 2 to 4 points across several DETR baselines at no inference cost.
-
Scaling Properties of Diffusion Models for Perceptual Tasks
Diffusion models for depth, optical flow, and amodal segmentation improve along power laws as training and test-time compute scale, and the fitted recipes match prior specialist models with less data.
-
Expert-Data Alignment Governs Generation Quality in Decentralized Diffusion Models
Sparse Top-2 routing beats full ensemble in decentralized diffusion models, and the paper attributes this to expert-data alignment rather than numerical stability — though much of the supporting evidence is circular.
-
Causal Diffusion Transformers for Generative Modeling
A decoder-only transformer that factors generation over both token order and noise level, coupling autoregressive and diffusion training, achieves competitive ImageNet generation and in-context editing.
-
Video Diffusion Transformers are In-Context Learners
Concatenating multiple video clips into one input and fine-tuning a LoRA adapter lets a pretrained video diffusion transformer produce consistent multi-scene videos from a single prompt.
Discussion (0). Continue with ORCID to comment.