Pith. sign in

REVIEW 3 major objections 5 minor 44 references

Chunking fused weights lets Muon train DiTs twice as fast as AdamW

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 05:56 UTC pith:PE3UR3JG

load-bearing objection Useful optimizer tweak with a plausible mechanism, but the headline 2× speedup rests on a 0.03 FID gap from single runs — needs multi-seed evidence before believing it. the 3 major comments →

arxiv 2608.02502 v1 pith:PE3UR3JG submitted 2026-08-03 cs.AI

CMuon: Accelerating and Stabilizing Diffusion Transformer Training via Chunked Momentum Orthogonalization

Chuyan Chen , Peng Sun , Kun Yuan This is my paper
classification cs.AI
keywords MuonChunked MuonDiffusion Transformeroptimizermomentum orthogonalizationsubspace couplingImageNet 256FID
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that Muon's disappointing late-stage convergence on diffusion transformers is not an optimizer flaw but a consequence of how standard DiT code fuses functionally distinct weight matrices. The paper proposes chunking each fused matrix (QKV, AdaLN modulation, FFN gate+up) into its semantic sub-blocks before orthogonalizing momentum, so each block gets its own preconditioner. This single change, called CMuon, removes the plateau and reaches FID 1.18 on ImageNet 256 in 200 epochs, about twice as fast as AdamW, with no auxiliary training tricks.

Core claim

Subspace interference: when Muon orthogonalizes a stacked gradient matrix G = [G1; ...; GN], the update to each block Gi uses the shared preconditioner (Σj Gj^T Gj)^{-1/2} rather than its own (Gi^T Gi)^{-1/2}. The paper shows this shared preconditioner mixes gradient covariances of functionally unrelated blocks, distorting each block's preferred descent direction. Chunking restores block-local preconditioners, giving Gi(Gi^T Gi)^{-1/2}. Empirically, CMuon maintains a fast FID reduction through late training and surpasses vanilla Muon and AdamW, reaching FID 1.18 at 200 epochs with a 675M DiT-XL.

What carries the argument

The load-bearing mechanism is per-chunk orthogonalization: split a fused weight matrix along its longer dimension into semantic sub-matrices, apply Newton-Schulz iterations to each sub-matrix independently, and concatenate the orthogonalized chunks. The step is paired with the paper's adopted scaling rule, which recomputes the 0.2/sqrt(max(dout,din)) factor from per-chunk dimensions so the global Frobenius norm of the update is preserved while being redistributed across chunks. An optional sqrt(Nchunk) learning-rate rescale accelerates early-stage convergence.

Load-bearing premise

The speedup claim rests on single-run FID-50K evaluations with no seeds or confidence intervals; if run-to-run FID variance for the 675M model is on the order of 0.1, the decisive gap between CMuon at 200 epochs (1.18) and AdamW at 400 epochs (1.21) would not be meaningful.

What would settle it

Run the same DiT-XL on ImageNet 256 with AdamW, Muon, and CMuon under the paper's exact hyperparameters for at least three seeds each, computing FID-50K at 200 and 400 epochs. The 2x claim holds only if every CMuon 200-epoch FID beats every AdamW 400-epoch FID by more than the seed variance. A second, mechanism-specific check: re-implement the architecture with separate (unfused) QKV and AdaLN matrices and run vanilla Muon; if the late-stage plateau persists, the root-cause story is wrong.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • CMuon reaches FID 1.18 on ImageNet 256 with a 675M DiT-XL at 200 epochs, surpassing AdamW's 400-epoch FID of 1.21 — a more than 2x training speedup.
  • The fix is near-zero-overhead: only the optimizer step changes, no architecture modification, and with system-level kernel optimization CMuon's throughput is close to Adam's on 8xH100 nodes.
  • The subspace-interference account is architectural: any fused 2D weight tensor holding heterogeneous gradient statistics becomes a candidate for chunking, not just the three layer types listed.
  • Chunking preserves late-stage convergence, so Muon-family optimizers can be used for complete training runs rather than only early-stage acceleration.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Extension: the same fused-matrix pathology should appear in other momentum-orthogonalization optimizers that share Muon's block-wise update; chunking by semantic sub-blocks could be imported there.
  • Testable extension: fused QKV/MLP matrices in large language models trained with Muon-style optimizers may exhibit the same late-stage slowdown, making CMuon's chunking rule directly transferable.
  • Ablation prediction: if the coupling story is right, running vanilla Muon on unfused (separate-tensor) versions of the same layers should reproduce CMuon's late-stage behaviour, isolating the coupling mechanism from any other optimizer difference.
  • Design hint: the sqrt(Nchunk) rescaling result suggests that per-chunk learning-rate control is an independent, composable knob; one could try per-chunk adaptive rescaling rather than a fixed sqrt factor.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper identifies a bottleneck in applying the Muon optimizer to Diffusion Transformers: standard DiT implementations fuse functionally distinct weight matrices (QKV, AdaLN, and FFN gate+up projections) into single tensors, and vanilla Muon orthogonalizes these fused tensors jointly, which the authors argue causes 'subspace interference' and hurts late-stage convergence. The proposed fix, CMuon, splits each fused matrix into its semantic chunks before Newton-Schulz orthogonalization, with a Moonlight-style scaling rule and an optional √N_chunk learning-rate rescale. Experiments on ImageNet 256×256 with DiT-B and DiT-XL report consistent FID improvements over both AdamW and vanilla Muon across VA-VAE, SD-VAE, and DC-AE setups; the headline result is FID 1.18 at 200 epochs for a 675M DiT-XL, claimed as a 'more than 2x training speedup' over AdamW's FID 1.21 at 400 epochs. The paper includes ablations on chunked blocks, scaling rules, learning-rate rescaling, and a system-level throughput comparison.

Significance. If the empirical claims are reproducible, CMuon is a simple, practical, and broadly applicable modification: it changes only the optimizer's orthogonalization granularity, adds no learnable parameters, and the paper shows consistent improvements over vanilla Muon across model scales, VAE choices, and resolution settings. The paper also provides a useful system-level analysis showing that, with custom kernels, CMuon can match AdamW's throughput. The attempt to derive a parameter-free scaling rule (Section 3.4, Appendix B.2) is a strength, though it is approximate. The main weakness is that the central quantitative claim rests on single training runs without seeds or uncertainty estimates; for a purely empirical paper, this is the load-bearing gap. The subspace-interference explanation is plausible but currently supported only by a static-gradient toy analysis, not by the promised theory.

major comments (3)
  1. [§4.2, Table 2] The headline 'more than 2x training speedup' rests on a single-run comparison: CMuon FID 1.18 at 200 epochs vs AdamW FID 1.21 at 400 epochs, a gap of 0.03. No seeds, confidence intervals, or repeated evaluations are reported in Tables 2, 3, 5, 6, 9, 10, or 11. For a 675M DiT on ImageNet-256 FID-50K, run-to-run variability from data order, EMA initialization, and sampling is commonly of order 0.05–0.1 FID; the decisive gap is within that range. Consequently the 2x speedup claim is not established, and ablation differences as small as 0.1–0.3 (e.g., Table 3) may be within noise. Please report multiple seeds or a defensible variance estimate for the central comparisons, or explicitly soften the speedup claim.
  2. [§3.3, Eqs. (9)–(12)] The subspace-interference argument compares one-step preconditioners for static gradient submatrices G_i. It does not include the momentum buffer of Eqs. (5)–(6), the Frobenius-normalized Newton-Schulz approximation of Algorithm 3, or any optimization dynamics; it shows only that the preconditioners differ, not that the shared preconditioner harms the flow-matching objective. The text promises 'further theoretical discussions and analyses... in Appendix B,' but Appendix B contains only the Newton-Schulz implementation and RMS-norm calculation; the promised analysis of subspace interference is absent. Please supply the analysis or label the root-cause explanation as a heuristic motivation rather than a proven cause.
  3. [§3.4, Eqs. (15)–(17)] The norm-preservation argument assumes Orth(G) is exactly the polar factor with orthonormal columns. Algorithm 3 normalizes by Frobenius norm and runs K=5–6 Newton-Schulz iterations, so the returned matrix is only approximately orthogonal; consequently the equality ||α Orth(G)||_F = 0.2√(N d_out d_in) and the RMS-preservation claim in §B.2 are approximations. Since the √N_chunk rescaling and the claim of direct hyperparameter reuse from AdamW rest on this calculation, please quantify the approximation error (e.g., measure ||O_t O_t^T − I||_F during training) or state the calculation as approximate.
minor comments (5)
  1. [§4.1 vs Appendix A.1] The main text says 'all experiments are evaluated with 30 NFEs,' and Table 2 lists NFE=30 for SD-VAE rows, but Appendix A.1 states that sampling uses 40 steps for SD-VAE and DC-AE. Table 10 does not specify NFE. Please reconcile so the FID numbers are reproducible.
  2. [§5.3] The phrase 'As mentioned at the end of Section 9' is an orphan reference; there is no Section 9. The intended pointer appears to be §3.4 or Algorithm 2.
  3. [Table 3 / Table 9] For AdaLN-only chunking, FID@80ep is 6.00, which is substantially worse than the 'None' baseline of 5.50. The text states that individual block chunking gives 'marginal improvements' without acknowledging this early-stage regression. Please discuss or explain.
  4. [Conclusion vs Table 12] The conclusion calls CMuon a 'zero-overhead modification,' but Table 12 shows that without system-level optimization CMuon is noticeably slower than Adam (6.24 vs 9.51 it/s for DiT-B). The overhead is negligible only after the custom Triton kernels are applied. Please qualify the claim.
  5. [Figure 2] The rendered axis labels in Figure 2 contain font artifacts (e.g., '/uni00000014/uni00000011...'), making the figures hard to read. Please replace with a clean rendering.

Circularity Check

1 steps flagged

No significant circularity: headline FID gains are externally measured on ImageNet, and the only self-citation (UCGM [30]) supplies the training recipe, not the proof. The missing-seeds/error-bar issue is a statistical concern, not circularity.

specific steps
  1. other [Appendix A.1 (Experimental Settings); also §3.4, §5.4]
    "we use two types of DiT backbones following the implementation of [30]. ... For image generation, we use the same training objective and sampling hyperparameters as UCGM [30]. ... Motivated by the commonly adopted setting in UCGM [30], we evaluate learning rates in the neighborhood of 2×10−4."

    UCGM [30] is co-authored by Peng Sun, an author of this paper, so the experimental substrate (backbone implementation, training objective, sampling hyperparameters, LR-search neighborhood) is inherited from the authors' own prior work. This is a minor self-citation, but it is not load-bearing for the central claim: the AdamW vs Muon vs CMuon comparison is an external ImageNet FID-50K measurement taken inside that fixed setup, and the CMuon advantage is not defined in terms of UCGM's outputs. No 'prediction' reduces to the cited work; Table 3 even shows chunking AdaLN alone hurts (FID 6.00 vs 5.50), confirming the benefits are empirical, not definitional.

full rationale

This paper's contribution is empirical: a 675M DiT-XL trained with CMuon attains FID 1.18 at 200 epochs versus AdamW's 1.21 at 400 epochs on the external ImageNet-256 FID-50K benchmark. The headline quantity is measured against an independent, publicly defined evaluation, not defined in terms of the authors' own outputs. The mechanism argument in §3.3 (Eqs. 9-12) is an algebraic identity showing that stacked Muon uses a shared preconditioner (Σ_j G_j^T G_j)^{-1/2} while chunked Muon uses per-block (G_i^T G_i)^{-1/2}; the claim that this coupling 'distorts' optimization is validated empirically (Table 3) rather than assumed, and the AdaLN-only row (FID 6.00 vs 5.50) shows chunking can hurt, so the benefit is not forced by the definition. The RMS-norm calculation in §B.2 is a consistency check (both AdamW and Moonlight-scaled Muon updates yield RMS ≈ 0.2) and involves no fitted parameter renamed as a prediction. Learning rates are ablated for both optimizers in Table 6, so the speedup claim is not presented as an artifact of an unfavorable AdamW setting. The only self-citation is UCGM [30] (co-authored by Peng Sun), which supplies the backbone implementation, training objective, and sampling recipe; this is a baseline choice that does not by itself determine the optimizer ranking. The genuine weakness — single training runs with no seeds, confidence intervals, or repeated FID evaluations, making the decisive 1.18 vs 1.21 gap potentially within run-to-run noise — is a statistical-validity concern and, per the rubric, correctness risk rather than circularity: no equation or fitted value reduces to its own input. Score 1 reflects one minor self-citation that is not load-bearing.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claim rests on a small set of tuned hyperparameters (LR rescale, NS iterations, chunk configuration) and on standard assumptions about Newton-Schulz, the DiT/UCGM setup, and single-run FID measurements. No new physical or model entities are introduced. The paper's own promise of a theoretical analysis in Appendix B is not fulfilled, so the subspace-interference mechanism remains an assumption rather than a derivation.

free parameters (3)
  • Learning-rate rescaling factor for chunked layers = sqrt(N_chunk), enabled
    Extra multiplier introduced in Algorithm 2 (r=True) to improve early-stage FID; ablation Table 5 shows gains, but it is a hand-selected hyperparameter, not derived from theory.
  • Newton-Schulz iterations K = 5 (DiT-B), 6 (DiT-XL)
    Chosen per model for numerical accuracy/speed (Table 8); affects update quality and is not fixed by theory.
  • Chunking configuration = QKV + FFN + AdaLN chunked
    The set of layers to chunk is selected after ablations (Tables 3 and 9); the subspace-interference analysis alone does not predict which blocks should be chunked.
axioms (4)
  • domain assumption Newton-Schulz with K iterations yields a sufficiently accurate approximation of the polar factor Orth(.)
    Used throughout Algorithm 1; accuracy depends on K and coefficients inherited from Muon; no convergence certificate is provided relative to the exact polar factor.
  • ad hoc to paper The static stacked-gradient Gram-matrix analysis (Eq. 9–12) captures the actual Muon update behavior in training
    The toy model ignores momentum, the Nesterov term, and Newton-Schulz approximation; it illustrates a possible mechanism but is not a proof that fused tensors cause the plateau.
  • domain assumption The fused tensor shapes in Table 1 correspond exactly to functionally independent blocks in the reference DiT implementation
    The paper inherits the UCGM DiT setup and does not list its architectural modifications; if the reference implementation differs, the chunk shapes and results may not transfer.
  • domain assumption Single-run FID-50K measurements represent optimizer performance
    No seeds or confidence intervals are reported; all comparisons assume the observed FID gaps are larger than run-to-run noise.

pith-pipeline@v1.3.0-daily-deepseek · 16045 in / 11196 out tokens · 111093 ms · 2026-08-04T05:56:54.583265+00:00 · methodology

0 comments
read the original abstract

Diffusion Transformers (DiTs) have achieved state-of-the-art (SOTA) performance in visual generative modeling, yet their training remains computationally prohibitive. While the recently proposed Momentum Orthogonalization (Muon) optimizer offers a promising alternative to AdamW, its direct application to DiTs yields suboptimal late-stage convergence. In this paper, we identify the root cause of this bottleneck: standard DiT architectures fuse functionally distinct weights (e.g., within AdaLN and QKV layers) into unified tensors for computational efficiency. Applying Muon to these fused tensors inadvertently induces implicit subspace coupling, which distorts update directions and degrades global optimization. To address this, we introduce Chunked Muon (CMuon), a simple yet highly effective strategy that partitions these matrices into independent sub-components prior to orthogonalization. Extensive experiments demonstrate that a 675M-parameter DiT trained with CMuon achieves a FID of 1.18 on ImageNet 256 in just 200 epochs. This represents more than a 2x training speedup over AdamW, while effectively overcoming the late-stage convergence plateaus of vanilla Muon.

Figures

Figures reproduced from arXiv: 2608.02502 by Chuyan Chen, Kun Yuan, Peng Sun.

Figure 1
Figure 1. Figure 1: Overview and empirical validation of CMuon. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: DiT training comparison on ImageNet-1K 256 × 256. FID@50k (lower is better) as a function of training progress for AdamW, Muon, and CMuon at two model scales. Muon and CMuon improve markedly faster in early training, while CMuon further preserves late-stage convergence and achieves better final quality. diffusion-modeling work [5]. All models adopt DiTs [20] as the backbone.1 We perform latent-space genera… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative comparison on ImageNet-1K 256 × 256. Random class￾conditional samples generated by DiT-XL trained with Muon (left) and CMuon (right). All images are sampled with the same sampling configuration and NFE = 30. in just 200 epochs, outperforming AdamW’s 400-epoch FID of 1.21 ( [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 26 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2505.16932 (2025)

    Amsel, N., Persson, D., Musco, C., Gower, R.M.: The polar express: Optimal ma- trix sign methods and their application to the muon algorithm. arXiv preprint arXiv:2505.16932 (2025)

    Pith/arXiv arXiv 2025

  2. [2]

    1 kontext: Flow matching for in-context image generation and editing in latent space

    Batifol, S., Blattmann, A., Boesel, F., Consul, S., Diagne, C., Dockhorn, T., En- glish, J., English, Z., Esser, P., Kulal, S., et al.: Flux. 1 kontext: Flow matching for in-context image generation and editing in latent space. arXiv e-prints pp. arXiv–2506 (2025)

    2025

  3. [3]

    arXiv preprint arXiv:2511.22699 (2025)

    Cai, H., Cao, S., Du, R., Gao, P., Hoi, S., Hou, Z., Huang, S., Jiang, D., Jin, X., Li, L., et al.: Z-image: An efficient image generation foundation model with single-stream diffusion transformer. arXiv preprint arXiv:2511.22699 (2025)

    Pith/arXiv arXiv 2025

  4. [4]

    In: 2009 IEEE conference on computer vision and pattern recognition

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large- scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)

    2009

  5. [5]

    Advances in neural information processing systems34, 8780–8794 (2021)

    Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. Advances in neural information processing systems34, 8780–8794 (2021)

    2021

  6. [6]

    Advances in neural information processing systems30(2017)

    Heusel,M.,Ramsauer,H.,Unterthiner,T.,Nessler,B.,Hochreiter,S.:Ganstrained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems30(2017)

    2017

  7. [7]

    Advances in neural information processing systems33, 6840–6851 (2020) 16 C

    Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems33, 6840–6851 (2020) 16 C. Chen et al

    2020

  8. [8]

    arXiv preprint arXiv:2207.12598 (2022)

    Ho, J., Salimans, T.: Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 (2022)

    Pith/arXiv arXiv 2022

  9. [9]

    URL https://kellerjordan

    Jordan, K., Jin, Y., Boza, V., Jiacheng, Y., Cesista, F., Newhouse, L., Bern- stein, J.: Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan. github. io/posts/muon6(3), 4 (2024)

    2024

  10. [10]

    arXiv preprint arXiv:1412.6980 (2014)

    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

    Pith/arXiv arXiv 2014

  11. [11]

    arXiv preprint arXiv:1312.6114 (2013)

    Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)

    Pith/arXiv arXiv 2013

  12. [12]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Leng, X., Singh, J., Hou, Y., Xing, Z., Xie, S., Zheng, L.: Repa-e: Unlocking vae for end-to-end tuning of latent diffusion transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 18262–18272 (2025)

    2025

  13. [13]

    arXiv preprint arXiv:2210.02747 (2022)

    Lipman, Y., Chen, R.T., Ben-Hamu, H., Nickel, M., Le, M.: Flow matching for generative modeling. arXiv preprint arXiv:2210.02747 (2022)

    Pith/arXiv arXiv 2022

  14. [14]

    arXiv preprint arXiv:2502.16982 (2025)

    Liu, J., Su, J., Yao, X., Jiang, Z., Lai, G., Du, Y., Qin, Y., Xu, W., Lu, E., Yan, J., et al.: Muon is scalable for llm training. arXiv preprint arXiv:2502.16982 (2025)

    Pith/arXiv arXiv 2025

  15. [15]

    arXiv preprint arXiv:2209.03003 (2022)

    Liu, X., Gong, C., Liu, Q.: Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003 (2022)

    Pith/arXiv arXiv 2022

  16. [16]

    arXiv preprint arXiv:1711.05101 (2017)

    Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)

    Pith/arXiv arXiv 2017

  17. [17]

    arXiv preprint arXiv:2601.22158 (2026)

    Lu, Y., Lu, S., Sun, Q., Zhao, H., Jiang, Z., Wang, X., Li, T., Geng, Z., He, K.: One-step latent-free image generation with pixel mean flows. arXiv preprint arXiv:2601.22158 (2026)

    Pith/arXiv arXiv 2026

  18. [18]

    In: European Conference on Computer Vision

    Ma, N., Goldstein, M., Albergo, M.S., Boffi, N.M., Vanden-Eijnden, E., Xie, S.: Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In: European Conference on Computer Vision. pp. 23–40. Springer (2024)

    2024

  19. [19]

    In: Dokl akad nauk Sssr

    Nesterov, Y.: A method for solving the convex programming problem with conver- gence rate o (1/k2). In: Dokl akad nauk Sssr. vol. 269, p. 543 (1983)

    1983

  20. [20]

    Peebles,W.,Xie,S.:Scalablediffusionmodelswithtransformers.In:Proceedingsof the IEEE/CVF international conference on computer vision. pp. 4195–4205 (2023)

    2023

  21. [21]

    arXiv preprint arXiv:2502.07529 (2025)

    Pethick, T., Xie, W., Antonakopoulos, K., Zhu, Z., Silveti-Falls, A., Cevher, V.: Training deep learning models with norm-constrained lmos. arXiv preprint arXiv:2502.07529 (2025)

    Pith/arXiv arXiv 2025

  22. [22]

    arXiv preprint arXiv:2505.13416 (2025)

    Riabinin, A., Shulgin, E., Gruntkowska, K., Richtárik, P.: Gluon: Making muon & scion great again!(bridging theory and practice of lmo-based optimizers for llms). arXiv preprint arXiv:2505.13416 (2025)

    Pith/arXiv arXiv 2025

  23. [23]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)

    2022

  24. [24]

    In: International Conference on Medical image computing and computer-assisted intervention

    Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedi- cal image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. pp. 234–241. Springer (2015)

    2015

  25. [25]

    arXiv preprint arXiv:2510.19376 (2025)

    Schaipp, F.: Optimization benchmark for diffusion models on dynamical systems. arXiv preprint arXiv:2510.19376 (2025)

    arXiv 2025

  26. [26]

    arXiv preprint arXiv:2509.01440 (2025)

    Semenov, A., Pagliardini, M., Jaggi, M.: Benchmarking optimizers for large lan- guage model pretraining. arXiv preprint arXiv:2509.01440 (2025)

    Pith/arXiv arXiv 2025

  27. [27]

    arXiv preprint arXiv:2010.02502 (2020) CMuon 17

    Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020) CMuon 17

    Pith/arXiv arXiv 2010

  28. [28]

    arXiv preprint arXiv:2011.13456 (2020)

    Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score- based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 (2020)

    Pith/arXiv arXiv 2011

  29. [29]

    cn/archives/11267

    Su, J.: Why is the adam update rms 0.2? (Sep 2025),https://www.spaces.ac. cn/archives/11267

    2025

  30. [30]

    arXiv preprint arXiv:2505.07447 (2025)

    Sun, P., Jiang, Y., Lin, T.: Unified continuous generative models. arXiv preprint arXiv:2505.07447 (2025)

    Pith/arXiv arXiv 2025

  31. [31]

    arXiv preprint arXiv:2502.12154 (2025)

    Tang, Z., Bao, J., Chen, D., Guo, B.: Diffusion models without classifier-free guid- ance. arXiv preprint arXiv:2502.12154 (2025)

    Pith/arXiv arXiv 2025

  32. [32]

    arXiv preprint arXiv:2507.20534 (2025)

    Team, K., Bai, Y., Bao, Y., Chen, G., Chen, J., Chen, N., Chen, R., Chen, Y., Chen, Y., Chen, Y., et al.: Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534 (2025)

    Pith/arXiv arXiv 2025

  33. [33]

    arXiv preprint arXiv:2512.07584 (2025)

    Team, M.L., Ma, H., Tan, H., Huang, J., Wu, J., He, J.Y., Gao, L., Xiao, S., Wei, X., Ma, X., et al.: Longcat-image technical report. arXiv preprint arXiv:2512.07584 (2025)

    Pith/arXiv arXiv 2025

  34. [34]

    Advances in neural information pro- cessing systems30(2017)

    Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information pro- cessing systems30(2017)

    2017

  35. [35]

    arXiv preprint arXiv:2504.05741 (2025)

    Wang, S., Tian, Z., Huang, W., Wang, L.: Ddt: Decoupled diffusion transformer. arXiv preprint arXiv:2504.05741 (2025)

    Pith/arXiv arXiv 2025

  36. [36]

    arXiv preprint arXiv:2509.02046 (2025)

    Wen, K., Hall, D., Ma, T., Liang, P.: Fantastic pretraining optimizers and where to find them. arXiv preprint arXiv:2509.02046 (2025)

    Pith/arXiv arXiv 2025

  37. [37]

    arXiv preprint arXiv:2511.18870 (2025)

    Wu, B., Zou, C., Li, C., Huang, D., Yang, F., Tan, H., Peng, J., Wu, J., Xiong, J., Jiang, J., et al.: Hunyuanvideo 1.5 technical report. arXiv preprint arXiv:2511.18870 (2025)

    Pith/arXiv arXiv 2025

  38. [38]

    arXiv preprint arXiv:2508.02324 (2025)

    Wu, C., Li, J., Zhou, J., Lin, J., Gao, K., Yan, K., Yin, S.m., Bai, S., Xu, X., Chen, Y., et al.: Qwen-image technical report. arXiv preprint arXiv:2508.02324 (2025)

    Pith/arXiv arXiv 2025

  39. [39]

    arXiv preprint arXiv:2410.10629 (2024)

    Xie, E., Chen, J., Chen, J., Cai, H., Tang, H., Lin, Y., Zhang, Z., Li, M., Zhu, L., Lu, Y., et al.: Sana: Efficient high-resolution image synthesis with linear diffusion transformers. arXiv preprint arXiv:2410.10629 (2024)

    Pith/arXiv arXiv 2024

  40. [40]

    generation: Taming optimization dilemma in latent diffusion models

    Yao, J., Yang, B., Wang, X.: Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 15703–15712 (2025)

    2025

  41. [41]

    arXiv preprint arXiv:2410.06940 (2024)

    Yu, S., Kwak, S., Jang, H., Jeong, J., Huang, J., Shin, J., Xie, S.: Representation alignment for generation: Training diffusion transformers is easier than you think. arXiv preprint arXiv:2410.06940 (2024)

    Pith/arXiv arXiv 2024

  42. [42]

    arXiv preprint arXiv:2511.20645 (2025)

    Yu, Y., Xiong, W., Nie, W., Sheng, Y., Liu, S., Luo, J.: Pixeldit: Pixel diffusion transformers for image generation. arXiv preprint arXiv:2511.20645 (2025)

    Pith/arXiv arXiv 2025

  43. [43]

    arXiv preprint arXiv:2512.24176 (2025) 18 C

    Zhou, X., Li, Q., Hu, X., Chen, H., Gu, S.: Guiding a diffusion transformer with the internal dynamics of itself. arXiv preprint arXiv:2512.24176 (2025) 18 C. Chen et al. A Experimental Settings A.1 Detailed Settings of Training and Sampling In al our experiments, we use two types of DiT backbones following the imple- mentation of [30], namely DiT-B and D...

    arXiv 2025

  44. [44]

    This stability holds across models of different sizes and configurations, indicating a fundamental property of the AdamW op- timizer withβ 1 = 0.9andβ 2 = 0.95

    observed that the RMS norm of updates in AdamW typically stabilizes around 0.2-0.3 during training. This stability holds across models of different sizes and configurations, indicating a fundamental property of the AdamW op- timizer withβ 1 = 0.9andβ 2 = 0.95. By approximating the RMS value of AdamW’s update, we obtain: RMS(Uadam)≈0.2. (20) For the Moonli...