Pith. sign in

REVIEW 4 major objections 6 minor 52 references

Unveiling the Secret of AdaLN-Zero in Diffusion Transformer

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that zero-initialization in DiT's adaLN-Zero wins because it puts the modulation weights at a good starting location, and that a small Gaussian start ($\mathrm{std}=10^{-3}$) works even better.

desk verdict The adaLN-Gaussian initialization trick is a real, cheap win for DiT training, but the causal story about why zero-init matters rests on an ablation that is never specified. read the letter →

arxiv 2608.09438 v1 pith:CZKFNGA4 submitted 2026-08-10 cs.CV

Jie Zhu , Mingyu Ding , Boqiang Duan , Leye Wang , Jingdong Wang This is my paper · ORCID
classification cs.CV
keywords diffusiontransformeradaLN-ZerozeroinitializationGaussianconditioningmechanismimagegenerationSqueeze-and-Excitationtrainingefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to explain why adaLN-Zero outperforms plain adaLN in diffusion transformers: it examines three differences between the two mechanisms—an added scaling path, zero-initialization of the weights that produce the scale, and a stagger in which weights update first. Through an ablation that decouples these factors, it argues that the zero-initialized starting location itself is the dominant factor, with the extra scaling path helping second and the staggered update order contributing least. Building on that diagnosis, it shows that initializing those scale-producing weights from a small Gaussian ($\mathcal{N}(0, 10^{-3})$, called adaLN-Gaussian) reaches the same image quality faster: FID 17.86 versus 20.02 at 400K steps on ImageNet 256×256, and FID 2.27 at roughly 3,800K steps, matching what adaLN-Zero reaches at 7,000K. A sympathetic reader would care because this turns an unexplained architectural trick into a transferable initialization principle, and it also yields a parameter-reduced variant, SE-adaLN-Zero, that improves FID while using fewer parameters.

What carries the argument

The load-bearing object is the adaLN-Zero conditioning block: the adaptive layer-norm style modulation in which a small MLP reads the conditioning embedding and outputs scale ($\gamma$), shift ($\beta$), and an extra per-block scaling factor ($\alpha$), with the linear layer producing $\alpha$ (and also $\gamma,\beta$ in DiT) zero-initialized. The paper's analysis instrument is the gradient table for the first iterations of a simplified DiT block, which shows that zero initialization makes weights update gradually—first the final layer, then patch embedding and $\alpha$ producers, then everything—and an experiment dubbed adaLN-Mix that tries to isolate this update order from the initial value. The proposed method replaces the zero start with $\mathcal{N}(0,10^{-3})$ for the modulation weights, matching the Gaussian-like distribution those weights converge to, which is what carries the claimed training-efficiency gain.

What would settle it

Train DiT-XL/2 with adaLN-Step1 initialization but with the update order enforced exactly as specified by the paper's own gradient table—only the final layer updates on step 1, then patch embedding, final-layer modulations, and $\alpha$ producers on step 2, then everything—implemented by freezing the parameter groups that the table marks as having zero gradient, with no other changes. If this transparently enforced schedule closes most of the gap to adaLN-Zero's FID 20.02 at 400K, the paper's claim that the starting value dominates would be wrong; if the gap stays near the adaLN-Mix level, the attribution holds. A second check: initialize only $W_\alpha$ from $\mathcal{N}(0,10^{-3})$ and keep $W_\gamma, W_\beta$ at zero; if the result does not match adaLN-Zero's 400K FID in the paper's setting, the claim that $W_\alpha$'s initial value is the decisive factor would be falsified.

Watch

Extended reading notes

Core claim

The central discovery is that adaLN-Zero's advantage comes from where its modulation weights start, not from the residual shortcut or the delayed update pattern it introduces. By comparing adaLN, adaLN-Step1 (scaling added, default initialization), adaLN-Mix (default initialization but with adaLN-Zero's update order enforced), and adaLN-Zero, the paper separates three candidate causes and finds that the near-zero initial value of the scale weights ($W_\alpha$, and similarly $W_\gamma, W_\beta$) accounts for most of the performance gap. It further observes that as training proceeds these weights converge to Gaussian-like distributions, so starting them from $\mathcal{N}(0, 10^{-3})$ (adaLN-Gaussian) places the model closer to where it is heading, which speeds convergence and improves FID at matched step counts. The paper claims this is evidence that zero-initialization is "a well-optimized location," not a special inductive bias of exact zero, and that a suitably chosen Gaussian is a better starting point.

Load-bearing premise

The whole attribution to "initial location" rests on the adaLN-Mix experiment, which the paper describes as using adaLN-Step1's initialization while enforcing adaLN-Zero's update order, but gives no mechanism, pseudocode, or code for how that enforcement is done; if the enforcement changes gradients on other weights (for example by masking or rescaling), the separation between starting point and update dynamics is not clean.

Editorial extensions

If this is right

  • Initializing the modulation weights of DiT with $\mathcal{N}(0,10^{-3})$ consistently improves FID at matched training steps across DiT-B/2, DiT-L/2, DiT-L/4, and 512×512 ImageNet.
  • At roughly 3,800K steps, adaLN-Gaussian reaches FID 2.27 that adaLN-Zero reaches at 7,000K, a roughly 46% saving in training steps at matched quality.
  • The benefit transfers to training systems built on DiT (SiT, FasterDiT), to DiT-based models (LlamaVision, U-DiT), and to text-to-image generation, where adaLN-Gaussian lowers COCO FID-30K from 71.41 to 65.51 at 50K steps.
  • SE-adaLN-Zero, an SE-inspired compression of the conditioning MLP, reduces parameters from 676M to 582M and improves FID from 20.02 to 19.13 at 400K steps; combining it with Gaussian initialization gives FID 18.76.
  • The paper's diagnosis implies that exact zero is not special: any initialization that lands the modulation weights near their converged distribution should give similar or better early-training behavior.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension suggested by the appendix is applying the same distribution-matching idea to ControlNet's zero convolutions, which also converge to Gaussian-like distributions; initializing those convolutions with small Gaussian noise could be measured for faster conditioning control.
  • The paper's entropy reasoning suggests a general recipe: estimate the converged weight distribution of a modulation path early in training and initialize at that mean and standard deviation; the optimal std would then be architecture-specific rather than universally $10^{-3}$.
  • Because adaLN-Gaussian does not change model capacity, the paper's own long-run tables imply the payoff is compute savings (reaching the same FID in fewer steps) rather than a higher final quality ceiling; in very long training regimes, the gap narrows.
  • The block-wise initialization experiments in the appendix hint that the optimal Gaussian std varies across blocks and across $W_\alpha, W_\gamma, W_\beta$, so a per-block, per-modulation initialization schedule is a natural next test of the same principle.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper investigates why the adaLN-Zero conditioning mechanism outperforms plain adaLN in diffusion transformers. It decomposes the difference into three candidate causes: an SE-like scaling structure, the zero-initialized starting location of the modulation weights, and the staggered 'gradual' update order induced by zero initialization. A simplified gradient derivation is given to explain the update order, and an adaLN-Mix experiment is introduced to separate location from update order. The paper then observes that the modulation weights W_alpha, W_gamma, and W_beta converge to Gaussian-like distributions and proposes adaLN-Gaussian (Gaussian initialization with std 1e-3), adaLN-Gaussian-v2 with per-module stds, and an SE-adaLN-Zero architecture with a compression ratio. Extensive experiments on ImageNet, additional datasets, and text-to-image generation report FID improvements at moderate training budgets and claim parity with the converged adaLN-Zero baseline at substantially fewer steps.

Significance. If the central attribution is accepted, the paper offers a simple, practical, and analysis-guided initialization that improves training efficiency without changing the architecture or learning algorithm. Strengths include the one-line PyTorch change, the breadth of experiments across model sizes, DiT variants, datasets, and text-to-image tasks, and the explicit falsifiable claim about initialization location. However, the main causal claim depends on the underspecified adaLN-Mix experiment, several long-horizon comparisons mix implementations, and the Gaussian std is tuned on the same benchmark used for the headline improvement. The empirical method may be useful, but the paper's central mechanistic conclusion needs stronger support before publication.

major comments (4)
  1. [Section III-B2, Fig. 1, Table III] The central conclusion that 'it is the zero-initialized location that accounts for the remarkable performance difference' rests on the adaLN-Mix experiment. The text says adaLN-Mix adopts adaLN-Step1's initialization while enforcing adaLN-Zero's update order 'simultaneously,' but no mechanism, pseudocode, or code is provided. Since adaLN-Step1's W_alpha is nonzero, the vanishing-gradient pattern in Table I does not occur naturally; the order must be imposed via freezing, gradient masking, or per-parameter learning-rate schedules. Any such intervention changes the optimization trajectory and can interact with the nonzero W_alpha, so the FID gap between adaLN-Mix and adaLN-Zero does not cleanly separate initial location from update dynamics. The Table III result that adaLN-Step1-Gaussian matches adaLN-Zero is supportive but does not by itself separate the two factors. Please specify the exact implementation and provide at least one alternative enforcement mechanism to show the conclusion is mechanism-independent.
  2. [Section IV, Tables II and IV] The Gaussian std for adaLN-Gaussian (and the per-module stds for v2) is selected by grid search on ImageNet 256x256 at 50K/400K steps, and the same benchmark and training length is then used as the headline evidence of improvement. This makes the comparison at 400K partially self-referential: the hyperparameter has been tuned on the testbed used for the claim. The paper should report performance on a held-out dataset or training budget without re-tuning, or provide sensitivity analysis demonstrating that the conclusion is robust across a range of std values. It would also help to report multiple seeds or confidence intervals; margins such as DiT-B/2 (42.72 vs 42.55) and U-DiT-L (10.87 vs 10.47) are small enough that run-to-run noise may affect the conclusion.
  3. [Section IV, Table IV and footnote 9] The long-horizon comparison is not apples-to-apples. The paper states that all experiments in Table IV use the fast-DiT implementation with gradient checkpointing, mixed precision, and pre-extracted VAE features, and that results 'may be slightly different from that of the original paper.' Yet the 2,352K and 7,000K adaLN-Zero baselines are borrowed from previous work and were not trained with this implementation. The headline claim that adaLN-Gaussian reaches 2.27 FID at 3,800K, matching adaLN-Zero at 7,000K, therefore mixes implementations across the comparison. The authors should either run the adaLN-Zero baseline under the identical fast-DiT pipeline for the long horizons, or clearly restrict the claim to the matched portions of the table.
  4. [Section III-C, Fig. 5, footnote 6] The entropy/thermodynamics argument is not a valid theoretical justification. The statement that systems evolve toward higher entropy (the second law) is applied to neural-network weight distributions without a closed system, a defined ensemble, or a connection between entropy increase and optimization ease. Moreover, the 'Gaussian-like' evidence in footnote 6 computes KL divergence by fitting a Gaussian to the same adaLN-Zero weights used for the comparison, so it measures the quality of a Gaussian fit rather than an independent target distribution. Please reframe this as a heuristic empirical observation, or provide a rigorous statement of the assumed dynamics.
minor comments (6)
  1. [Section IV, text-to-image paragraph] There is a typo: 'adaLN-Gaussin' should be 'adaLN-Gaussian'.
  2. [Section III-B1, text after Table I] The sentence listing zero gradients repeats W_gamma2 and W_beta2: 'dL/dW_gamma2, dL/dW_beta2, dL/dW_gamma2, and dL/dW_beta2' should list each weight once.
  3. [Eq. (1)] The identity map I in 'Sigmoid(I*c)' is not defined; aligning the notation with the element-wise multiplication and the 1-vector in Eq. (2) would make the SE analogy clearer.
  4. [Discussion section] There are typos: 'ensensially' should be 'essentially' and 'provid' should be 'provide'.
  5. [Footnote 9] The footnote should state which rows of Table IV use the fast-DiT implementation; the current phrasing 'all of which are employed in experiments of Tab. IV if not specified' leaves ambiguity about the 400K and 800K baseline rows.
  6. [Section V, SE-like variants] The sentence 'Our SE-like v2 user=4' should read 'uses r=4'.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor self-referential Gaussian-likeness check; the main initial-location attribution is not circular.

  1. self definitional [Footnote 6 (Section III-C, page 6)]
    "To demonstrate that adaLN-Zero exhibits a Gaussian-like distribution, we employ KL-Divergence to measure the distance between its distribution and a true Gaussian. We use the weights of adaLN-Zero at 50K steps to compute its mean and standard deviation. These parameters are then used to initialize a Gaussian distribution, from which we sample the same number of weight points as adaLN-Zero."

    The 'true Gaussian' is not an independent reference distribution: its mean and standard deviation are computed from the very same adaLN-Zero weights whose Gaussian-likeness is being asserted. The reported KL divergence therefore measures deviation from the best-fitting Gaussian in that family, so the conclusion 'adaLN-Zero exhibits a Gaussian-like distribution' is partly manufactured by the fitting step rather than established against an external baseline. This self-referential check motivates adaLN-Gaussian, but it is not the load-bearing evidence for the paper's central attribution, which rests on the controlled adaLN, adaLN-Step1, adaLN-Mix, and adaLN-Step1-Gaussian comparisons.

full rationale

The paper's central claim, that the zero-initialized starting location of the modulation weights is the dominant factor, is supported by controlled variants whose definitions do not presuppose the conclusion. In particular, adaLN-Step1-Gaussian uses a small nonzero W_alpha initialization and matches adaLN-Zero, independently separating initial location from the staggered zero-init update order. The adaLN-Mix experiment is incompletely specified, but that is a reproducibility gap rather than a circular reduction. The only identifiable self-referential element is the KL-based 'Gaussian-like' verification, where the reference Gaussian is fitted to the same weights being tested; this is a methodological weakness in a supporting observation rather than a forced derivation of the main result. The proposed initialization is also evaluated across other datasets, model sizes, and DiT-based models with the same fixed std, providing out-of-sample evidence. Overall, the derivation chain is not equivalent to its inputs, and no load-bearing step reduces by construction.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical or architectural entities are introduced. The ledger shows the method depends on a standard deviation fitted to performance, a compression ratio chosen by ablation, and two informal assumptions (Gaussian-likeness and the thermodynamics analogy).

free parameters (4)
  • Gaussian std for adaLN-Gaussian (uniform) = 0.001
    Selected by grid search on FID at 50K steps on ImageNet 256x256 (Table II); it is the value that best matches the observed W_alpha distribution but is also tuned to the evaluation metric.
  • Per-module Gaussian stds for adaLN-Gaussian-v2 = (W_alpha 8e-4, W_gamma 1.2e-3, W_beta 8e-4)
    Grid search in App. J (Table XI); best FID at 50K steps.
  • SE-like compression ratio r = 2 (variant v2)
    Ablation over r in {2,4,8} in Table VI; r=2 chosen as SE-adaLN-Zero for best FID at 50K steps.
  • Block-wise std polynomial coefficients = 7th/5th degree polynomial coefficients in App. K
    Fitted to observed peak densities of W_alpha, W_gamma, W_beta per block; explored only as a preliminary variant.
assumptions (5)
  • domain assumption The simplified DiT with one block, linear attention and FFN, and no bias terms preserves the gradient update order of the full DiT.
    Invoked in Section III-B1; supported only by a qualitative remark that the order matches original DiT variants.
  • standard math LayerNorm in DiT is learning-free and can be dropped from the gradient analysis.
    DiT's LayerNorm is not learned, so omission is standard practice.
  • ad hoc to paper The trained conditioning weights are well approximated by a Gaussian, making Gaussian initialization suitable.
    Section IV and footnote 6; the Gaussian-likeness is measured by KL against a Gaussian fitted to the same weights, which is partly self-confirming.
  • ad hoc to paper Entropy changes of weight distributions follow the second law of thermodynamics, so a distribution that increases entropy is easier to optimize.
    Section III-C uses this analogy to explain faster convergence; no formal connection between thermodynamic entropy and optimization dynamics is given.
  • standard math MSE loss on predicted noise is the training objective for the gradient derivation.
    Standard for diffusion training; used in App. C equation (3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unveiling the Secret of AdaLN-Zero in Diffusion Transformer." pith.science (2026). https://pith.science/paper/CZKFNGA4

@misc{pith2026260809438,
  author       = {Pith},
  title        = {Pith review of: Unveiling the Secret of AdaLN-Zero in Diffusion Transformer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CZKFNGA4}},
  note         = {Machine review of arXiv:2608.09438}
}
read the original abstract

Diffusion transformer (DiT), a rapidly emerging architecture for image generation, has gained much attention. However, despite ongoing efforts to improve its performance, the understanding of DiT remains superficial. In this work, we delve into and investigate a critical conditioning mechanism within DiT, adaLN-Zero, which achieves superior performance compared to adaLN. Our work studies three potential elements driving this performance, including an SE-like structure, zero-initialization, and a "gradual" update order, among which zero-initialization is proved to be the most influential. Building on this understanding, we propose an analysis-guided initialization strategy, termed adaLN-Gaussian, which serves both as an empirical validation of our analysis and as a practical initialization method that consistently improves optimization efficiency. On the other hand, inspired by the SE-like structure, we introduce an improved conditioning mechanism called SE-adaLN-Zero. Extensive experiments following DiT on four datasets, especially on ImageNet1K demonstrate the effectiveness and generalization of adaLN-Gaussian and SE-adaLN-Zero. Beyond class-to-image generation, we also evaluate the generalization of the two improved methods on text-to-image generation.

Figures

Figures reproduced from arXiv: 2608.09438 by the authors.

Figure 1
Figure 1. Comparing adaLN-Zero with adaLN, adaLN-Step1, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Performance of different variants in the initial stage. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Value distributions of all Wα during the training process. AdaLN-Zero and adaLN-Mix are initialization strategies and 160, 10K, and 50K are timestamps. adaLN-Step1, it still lags significantly behind adaLN-Zero. This first indicates that the update order resulting from zero￾initialization does contribute independently to performance. However, this contribution is not the primary reason for the substantial performanc… view at source ↗
Figures from the paper (19 more)
Figure 5
Figure 5. Figure 5: An abstract illustration of the entropy analysis on [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Value distributions of WL α in different blocks. tropy (the second law of thermodynamics). Therefore, adaLN￾Zero is comparatively easier to optimize and obtains better performance. One might question, though we have globally analyzed all Wα in DiT, is it possible that …
Figure 7
Figure 7. Figure 7: Value distributions of the whole Wγ in DiT blocks during the training process [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Value distributions of Wγf in FinalLayer during the training process. TABLE III: Ablation study for Wα. (0, 0, 0): adaLN-Zero Std (Wα, Wγ, Wβ) FID IS 0, 0, 0 78.99 14.19 1e-3, 0, 0 78.62 14.42 1e-3, 1e-3, 1e-3 76.21 15.01 To leverage Gaussian distribution to initialize…
Figure 9
Figure 9. Figure 9: Detailed illustration of our SE-like structure variants. [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Comparing adaLN-Zero with adaLN as well as different initialization strategies on Inception Score (IS). We use the [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: The structure of SE module. In [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: Value distributions of the whole Wβ in DiT blocks during the training process. E. Value Distributions of WL γ and WL β in Different Blocks We also present the value distributions of WL γ and WL β in different blocks of DiT-XL/2 using adaLN-Zero trained at a very early…
Figure 13
Figure 13. Figure 13: Value distributions of WL γ in different blocks [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Value distributions of WL β in different blocks [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: Value distributions of Wβf in FinalLayer during the training process [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: Value distributions of Wf in FinalLayer during the training process. F. Value Distributions of Wβf and Wf TABLE IX: A grid search of std for Wβf . 0: AdaLN￾Gaussian-v1 Std FID IS 0 76.21 15.01 2e-4 78.22 14.53 5e-4 82.05 13.78 1e-3 80.43 14.03 2e-3 77.45 14.74 3e-3 77…
Figure 17
Figure 17. Figure 17: Value distributions of Attention module including qkv and proj during training process. [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]
Figure 18
Figure 18. Figure 18: Value distributions of Mlp module including fc1 and fc2 during training process. [PITH_FULL_IMAGE:figures/full_fig_p019_18.png]
Figure 19
Figure 19. Figure 19: Value distributions of PatchEmbed during training process. [PITH_FULL_IMAGE:figures/full_fig_p020_19.png]
Figure 20
Figure 20. Figure 20: Value distributions of LabelEmbedder and TimestepEmbedder during training process. [PITH_FULL_IMAGE:figures/full_fig_p020_20.png]
Figure 21
Figure 21. Figure 21: Weight distributions of zero convolution in four ControlNet variants. [PITH_FULL_IMAGE:figures/full_fig_p021_21.png]
Figure 22
Figure 22. Figure 22: Value distributions of Wα with different std in Gaussian initialization [PITH_FULL_IMAGE:figures/full_fig_p021_22.png]
Figure 23
Figure 23. Figure 23: Three polynomial functions to fit the peak [PITH_FULL_IMAGE:figures/full_fig_p022_23.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 19 canonical work pages

  1. [1]

    Scalable diffusion models with transformers,

    W. Peebles and S. Xie, “Scalable diffusion models with transformers,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 4195–4205

    2023

  2. [2]

    Diffusion models beat gans on image synthesis,

    P. Dhariwal and A. Nichol, “Diffusion models beat gans on image synthesis,”Advances in neural information processing systems, vol. 34, pp. 8780–8794, 2021

    2021

  3. [3]

    High- resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10 684–10 695

    2022

  4. [4]

    Visionllama: A unified llama backbone for vision tasks,

    X. Chu, J. Su, B. Zhang, and C. Shen, “Visionllama: A unified llama backbone for vision tasks,” inEuropean Conference on Computer Vision, vol. 3, 2024

    work page 2024

  5. [5]

    Sit: Exploring flow and diffusion-based gen- erative models with scalable interpolant transformers,

    N. Ma, M. Goldstein, M. S. Albergo, N. M. Boffi, E. Vanden- Eijnden, and S. Xie, “Sit: Exploring flow and diffusion-based gen- erative models with scalable interpolant transformers,”arXiv preprint arXiv:2401.08740, 2024

    arXiv 2024

  6. [6]

    Fit: Flexible vision transformer for diffusion model,

    Z. Lu, Z. Wang, D. Huang, C. Wu, X. Liu, W. Ouyang, and L. Bai, “Fit: Flexible vision transformer for diffusion model,”arXiv preprint arXiv:2402.12376, 2024

    arXiv 2024

  7. [7]

    U-dits: Downsample tokens in u-shaped diffusion transformers,

    Y . Tian, Z. Tu, H. Chen, J. Hu, C. Xu, and Y . Wang, “U-dits: Downsample tokens in u-shaped diffusion transformers,”arXiv preprint arXiv:2405.02730, 2024

    arXiv 2024

  8. [8]

    SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformer

    R. Zhu, Y . Pan, Y . Li, T. Yao, Z. Sun, T. Mei, and C. W. Chen, “Sd- dit: Unleashing the power of self-supervised discrimination in diffusion transformer,”arXiv preprint arXiv:2403.17004, 2024

    work page Pith review arXiv 2024

Show all 52 references
  1. [9]

    Fasterdit: Towards faster diffusion transformers training without architecture modification,

    J. Yao, C. Wang, W. Liu, and X. Wang, “Fasterdit: Towards faster diffusion transformers training without architecture modification,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems

  2. [10]

    Roformer: En- hanced transformer with rotary position embedding,

    J. Su, M. Ahmed, Y . Lu, S. Pan, W. Bo, and Y . Liu, “Roformer: En- hanced transformer with rotary position embedding,”Neurocomputing, vol. 568, p. 127063, 2024

    2024

  3. [11]

    Glu variants improve transformer,

    N. Shazeer, “Glu variants improve transformer,”arXiv preprint arXiv:2002.05202, 2020

    2002 arXiv

  4. [12]

    Scaling rectified flow transformers for high-resolution image synthesis,

    P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. M ¨uller, H. Saini, Y . Levi, D. Lorenz, A. Sauer, F. Boeselet al., “Scaling rectified flow transformers for high-resolution image synthesis,” inForty-first International Conference on Machine Learning, 2024

    2024

  5. [13]

    Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis,

    J. Chen, J. Yu, C. Ge, L. Yao, E. Xie, Y . Wu, Z. Wang, J. Kwok, P. Luo, H. Luet al., “Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis,”arXiv preprint arXiv:2310.00426, 2023

    2023 arXiv

  6. [14]

    Pixart-\sigma: Weak-to-strong training of dif- fusion transformer for 4k text-to-image generation,

    J. Chen, C. Ge, E. Xie, Y . Wu, L. Yao, X. Ren, Z. Wang, P. Luo, H. Lu, and Z. Li, “Pixart-\sigma: Weak-to-strong training of dif- fusion transformer for 4k text-to-image generation,”arXiv preprint arXiv:2403.04692, 2024

    2024 arXiv

  7. [15]

    Exploring the role of large language models in prompt encoding for diffusion models,

    B. Ma, Z. Zong, G. Song, H. Li, and Y . Liu, “Exploring the role of large language models in prompt encoding for diffusion models,”arXiv preprint arXiv:2406.11831, 2024

    2024 arXiv

  8. [16]

    Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding,

    Z. Li, J. Zhang, Q. Lin, J. Xiong, Y . Long, X. Deng, Y . Zhang, X. Liu, M. Huang, Z. Xiaoet al., “Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding,”arXiv preprint arXiv:2405.08748, 2024

    2024 arXiv

  9. [17]

    Elucidating the design space of diffusion-based generative models,

    T. Karras, M. Aittala, T. Aila, and S. Laine, “Elucidating the design space of diffusion-based generative models,”Advances in neural infor- mation processing systems, vol. 35, pp. 26 565–26 577, 2022

    2022

  10. [18]

    Analyzing and improving the training dynamics of diffusion models,

    T. Karras, M. Aittala, J. Lehtinen, J. Hellsten, T. Aila, and S. Laine, “Analyzing and improving the training dynamics of diffusion models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 24 174–24 184

    2024

  11. [19]

    Mole: En- hancing human-centric text-to-image diffusion via mixture of low-rank experts,

    J. Zhu, Y . Chen, M. Ding, P. Luo, L. Wang, and J. Wang, “Mole: En- hancing human-centric text-to-image diffusion via mixture of low-rank experts,” inThe Thirty-eighth Annual Conference on Neural Information Processing Systems

  12. [20]

    Squeeze-and-excitation networks,

    J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7132–7141

    2018

  13. [21]

    ImageNet Large Scale Visual Recognition Challenge,

    O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “ImageNet Large Scale Visual Recognition Challenge,” International Journal of Computer Vision (IJCV), pp. 211–252, 2015

    2015

  14. [22]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in neural information processing systems, vol. 30, 2017

    2017

  15. [23]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gellyet al., “An image is worth 16x16 words: Transformers for image recognition at scale,”arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  16. [24]

    Layoutdm: Transformer-based diffusion model for layout generation,

    S. Chai, L. Zhuang, and F. Yan, “Layoutdm: Transformer-based diffusion model for layout generation,” inProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, 2023, pp. 18 349– 18 358

    2023

  17. [25]

    Masked diffusion trans- former is a strong image synthesizer,

    S. Gao, P. Zhou, M.-M. Cheng, and S. Yan, “Masked diffusion trans- former is a strong image synthesizer,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 23 164–23 173

    2023

  18. [26]

    Dit- 3d: Exploring plain diffusion transformers for 3d shape generation,

    S. Mo, E. Xie, R. Chu, L. Hong, M. Niessner, and Z. Li, “Dit- 3d: Exploring plain diffusion transformers for 3d shape generation,” Advances in neural information processing systems, vol. 36, pp. 67 960– 67 971, 2023

    2023

  19. [27]

    Diffuser: efficient transformers with multi-hop attention diffusion for long sequences,

    A. Feng, I. Li, Y . Jiang, and R. Ying, “Diffuser: efficient transformers with multi-hop attention diffusion for long sequences,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 11, 2023, pp. 12 772–12 780

    2023

  20. [28]

    Latent diffusion transformer for probabilistic time series forecasting,

    S. Feng, C. Miao, Z. Zhang, and P. Zhao, “Latent diffusion transformer for probabilistic time series forecasting,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 11, 2024, pp. 11 979– 11 987

    2024

  21. [29]

    All are worth words: A vit backbone for diffusion models,

    F. Bao, S. Nie, K. Xue, Y . Cao, C. Li, H. Su, and J. Zhu, “All are worth words: A vit backbone for diffusion models,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 22 669–22 679

    2023

  22. [30]

    Scaling diffusion transformers to 16 billion parameters,

    Z. Fei, M. Fan, C. Yu, D. Li, and J. Huang, “Scaling diffusion transformers to 16 billion parameters,”arXiv preprint arXiv:2407.11633, 2024

    2024 arXiv

  23. [31]

    Gentron: Diffusion transformers for image and video generation,

    S. Chen, M. Xu, J. Ren, Y . Cong, S. He, Y . Xie, A. Sinha, P. Luo, T. Xi- ang, and J.-M. Perez-Rua, “Gentron: Diffusion transformers for image and video generation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 6441–6451

    2024

  24. [32]

    Dlt: Conditioned layout generation with joint discrete-continuous diffusion layout trans- former,

    E. Levi, E. Brosh, M. Mykhailych, and M. Perez, “Dlt: Conditioned layout generation with joint discrete-continuous diffusion layout trans- former,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 2106–2115

    2023

  25. [33]

    Scalable high-resolution pixel-space image synthesis JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 12 with hourglass diffusion transformers,

    K. Crowson, S. A. Baumann, A. Birch, T. M. Abraham, D. Z. Kaplan, and E. Shippole, “Scalable high-resolution pixel-space image synthesis JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 12 with hourglass diffusion transformers,” inForty-first International Con- ferenc...

    2021

  26. [34]

    Diffit: Diffusion vision transformers for image generation,

    A. Hatamizadeh, J. Song, G. Liu, J. Kautz, and A. Vahdat, “Diffit: Diffusion vision transformers for image generation,”arXiv preprint arXiv:2312.02139, 2023

    2023 arXiv

  27. [35]

    A review on weight initialization strategies for neural networks,

    M. V . Narkhede, P. P. Bartakke, and M. S. Sutaone, “A review on weight initialization strategies for neural networks,”Artificial intelligence re- view, pp. 291–322, 2022

    2022

  28. [36]

    Understanding the difficulty of training deep feedforward neural networks,

    X. Glorot and Y . Bengio, “Understanding the difficulty of training deep feedforward neural networks,” inProceedings of the thirteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2010, pp. 249–256

    2010

  29. [37]

    Caffe: Convolutional architecture for fast feature embedding,

    Y . Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell, “Caffe: Convolutional architecture for fast feature embedding,” inProceedings of the 22nd ACM international conference on Multimedia, 2014, pp. 675–678

    2014

  30. [38]

    Rectified linear units improve restricted boltz- mann machines,

    V . Nair and G. E. Hinton, “Rectified linear units improve restricted boltz- mann machines,” inProceedings of the 27th international conference on machine learning (ICML-10), 2010, pp. 807–814

    2010

  31. [39]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,

    K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in Proceedings of the IEEE international conference on computer vision, 2015, pp. 1026–1034

    2015

  32. [40]

    Accurate, large minibatch sgd: Training imagenet in 1 hour,

    P. Goyal, P. Doll ´ar, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y . Jia, and K. He, “Accurate, large minibatch sgd: Training imagenet in 1 hour,”arXiv preprint arXiv:1706.02677, 2017

    2017 arXiv

  33. [41]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in neural information processing systems, vol. 33, pp. 6840– 6851, 2020

    2020

  34. [42]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium,

    M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,”Advances in neural information processing systems, vol. 30, 2017

    2017

  35. [43]

    Improved techniques for training gans,

    T. Salimans, I. Goodfellow, W. Zaremba, V . Cheung, A. Radford, and X. Chen, “Improved techniques for training gans,”Advances in neural information processing systems, vol. 29, 2016

    2016

  36. [44]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778

    2016

  37. [45]

    Adding conditional control to text-to-image diffusion models,

    L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 3836–3847

    2023

  38. [46]

    Generating images with sparse representations,

    C. Nash, J. Menick, S. Dieleman, and P. Battaglia, “Generating images with sparse representations,” inInternational Conference on Machine Learning. PMLR, 2021, pp. 7958–7968

    2021

  39. [47]

    Im- proved precision and recall metric for assessing generative models,

    T. Kynk ¨a¨anniemi, T. Karras, S. Laine, J. Lehtinen, and T. Aila, “Im- proved precision and recall metric for assessing generative models,” Advances in neural information processing systems, 2019

    2019

  40. [48]

    Tiny imagenet visual recognition challenge,

    Y . Le and X. Yang, “Tiny imagenet visual recognition challenge,”CS 231N, vol. 7, no. 7, p. 3, 2015

    2015

  41. [49]

    Stargan v2: Diverse image synthesis for multiple domains,

    Y . Choi, Y . Uh, J. Yoo, and J.-W. Ha, “Stargan v2: Diverse image synthesis for multiple domains,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 8188– 8197

    2020

  42. [50]

    Progressive growing of GANs for improved quality, stability, and variation,

    T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of GANs for improved quality, stability, and variation,” inInternational Conference on Learning Representations, 2018

    2018

  43. [51]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning. PmLR, 2021, pp. 8748–8763

    2021

  44. [52]

    Computing higher order deriva- tives of matrix and tensor expressions,

    S. Laue, M. Mitterreiter, and J. Giesen, “Computing higher order deriva- tives of matrix and tensor expressions,”Advances in neural information processing systems, vol. 31, 2018. Jie Zhuis currently working toward the Ph.D. degree in the School of Computer Science, Peking Univ...

    2018

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.