Pith. sign in

REVIEW 2 major objections 5 minor 77 references

Sparse Guidance uses the capacity gap between two token-sparsity levels to guide diffusion models, achieving FID 1.58 on ImageNet-256 with 25% fewer FLOPs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 12:45 UTC pith:BRSBSTGY

load-bearing objection SG is a genuinely new guidance mechanism for sparse diffusion models, but the headline 1.58 FID belongs to a confounded configuration (early checkpoint + sparsity schedule), not the plain two-branch method. the 2 major comments →

arxiv 2601.01608 v2 pith:BRSBSTGY submitted 2026-01-04 cs.CV

Guiding Token-Sparse Diffusion Models

Felix Krause , Stefan Andreas Baumann , Johannes Schusterbauer , Olga Grebenkova , Ming Gui , Vincent Tao Hu , Bj\"orn Ommer This is my paper
classification cs.CV
keywords sparse guidancetoken sparsitydiffusion modelsclassifier-free guidancecapacity gaptoken routingtext-to-image generationflow matching
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper proposes Sparse Guidance (SG), a method that turns token-level sparsity—the practice of processing only a subset of image tokens—into a test-time guidance signal for diffusion models trained with sparsity. SG runs two conditional predictions with different sparsity rates and extrapolates between them, creating a 'capacity gap' that steers sampling toward higher fidelity. On ImageNet-256, SG reaches an FID of 1.58 while using 25% fewer FLOPs than a dense guided baseline, and up to 58% FLOP savings at matched quality. Because both branches are conditional, SG preserves the high variance of the conditional prediction that classifier-free guidance tends to collapse. The same recipe improves a 2.5B text-to-image model's human-preference and composition scores while increasing throughput.

Core claim

The central claim is that the capacity gap between two conditional predictions, obtained by applying different token-sparsity rates at inference, can serve as a guidance signal for sparsely trained diffusion models, replacing the unconditional branch used by classifier-free guidance. The paper demonstrates that SG, D_SG = ω D_strong(c;γ_strong) + (1−ω) D_weak(c;γ_weak), yields an FID of 1.58 on ImageNet-256 with 25% fewer FLOPs than a dense guided baseline, and up to 58% FLOP savings at matched quality. It also shows the method scales to a 2.5B text-to-image model, where SG improves both GenEval and HPSv3 scores over CFG while increasing throughput. The key claim is that SG embraces—rather t

What carries the argument

The central mechanism is the capacity gap: a controlled difference in effective model capacity produced by two token-sparsity rates, γ_strong and γ_weak, applied to the same checkpoint. Token routing (bypassing layers with identity and reinserting tokens) and token masking (replacing tokens with learnable embeddings) are the two sparsity implementations; SG inherits the training-time sparsity settings and only varies γ at inference. The guidance formula D_SG = ω D_strong(c) + (1−ω) D_weak(c) is the operational identity, and the paper shows that larger guidance scales ω tolerate higher total sparsity, giving a smooth quality–throughput trade-off.

Load-bearing premise

The method assumes that increasing token-sparsity γ monotonically 'softens' the conditional prediction in a direction that makes extrapolating from weak to strong point toward higher-fidelity samples; if the capacity gap does not align with the direction of improved quality, SG cannot steer the generation.

What would settle it

Measure unguided FID of a fixed checkpoint as a function of γ across layers. If increasing γ does not monotonically degrade FID (i.e., a more sparse branch occasionally produces better samples than the strong branch), the assumed capacity-gap direction is unreliable and SG's extrapolation would be expected to fail. A simpler test: for a given (γ_strong, γ_weak), if the optimal guidance scale ω is negative, the gap points the wrong way.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Sparse-trained diffusion models can now be guided at inference without any dense finetuning, removing a key barrier to their adoption.
  • On ImageNet-256, SG achieves FID 1.58 with 25% fewer FLOPs than the dense guided baseline, and 58% FLOP savings at matched quality.
  • SG outperforms CFG, AutoGuidance, and ICG on the same sparsely pretrained backbone, and improves recall (higher output variance) while doing so.
  • SG is composable: it combines with CFG and AutoGuidance, and its sparsity controls broaden the range of usable auxiliary checkpoints in AutoGuidance.
  • At 2.5B scale, SG improves GenEval composition scores and HPSv3 human-preference scores over CFG while increasing throughput from 0.32 to 0.49 images/s.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The capacity-gap idea likely transfers to other sparsity axes—temporal sparsity in video diffusion, layer-wise sparsity, or mixture-of-experts routing—where a 'weaker' version of the same model can be produced at test time without retraining.
  • Because SG keeps both branches conditional, it offers a principled alternative to CFG for domains where unconditional training is costly or where variance collapse is a known failure mode, such as rare compositional prompts.
  • The observed relationship between ω and tolerable sparsity suggests an adaptive sampling schedule that anneals sparsity during generation, potentially extracting further FLOP savings at fixed quality.
  • Routing-based sparsity appears more robust than masking for SG (broader (γ_strong, γ_weak) landscape), implying that preserving token identity during sparsification is important for the guidance signal; this could inform the design of future sparse-training methods.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The manuscript introduces Sparse Guidance (SG), a guidance scheme for diffusion models trained with token-level sparsity. SG computes two conditional predictions from the same network at two token-sparsity levels (γstrong < γweak) and combines them as ωDstrong + (1−ω)Dweak, replacing the unconditional branch of classifier-free guidance with a 'weaker' sparse prediction. The authors report an ImageNet-256 FID of 1.58 with 24.6% fewer FLOPs than dense CFG, up to 58% FLOP savings at matched quality, and demonstrate the approach on a 2.5B text-to-image model, where they report improved GenEval and HPSv3 scores.

Significance. The core idea is attractive: rather than finetuning sparse-trained diffusion models to restore CFG, the capacity gap created by inference-time sparsity is used as a guidance signal. The paper's experimental coverage is broad—masking and routing backbones, multiple guidance baselines, and a billion-parameter T2I model. The SGFLOPS configuration (same checkpoint, static γstrong=0.5, γweak=0.9) is a clean demonstration that token-sparsity guidance can improve FID while reducing FLOPs. However, the headline 1.58 FID result relies on an early-checkpoint auxiliary and a sparsity schedule that are not part of Eq. (7), and hyperparameters are selected on the same benchmarks used for evaluation. The contribution therefore needs to be reframed and the result disentangled before the paper's main claims are accepted.

major comments (2)
  1. [App. B.1; §3.2 Eqs. (6)-(7); §4.2 Tables 2-3; Abstract] The headline result 'SG achieves FID 1.58 with 25% fewer FLOPs' is not produced by the method defined in Eq. (7). Appendix B.1 states that SGFID uses a 50k-iteration early checkpoint as the auxiliary branch and applies a cosine sparsity schedule (0.6→0.0 on the auxiliary model, inverse on the main model). This introduces a second capacity gap that is not token sparsity of the same checkpoint: it is the AutoGuidance mechanism of [30] plus an annealing schedule absent from Eq. (7). The same-checkpoint SGFLOPS configuration (γstrong=0.5, γweak=0.9, FID 2.14, 97.67 GFLOPS) is consistent with Eq. (7) and supports a weaker claim, but the 1.58 number and the associated '25% fewer FLOPs' contribution cannot be attributed to the proposed method as defined. The paper should either recompute the headline with a same-checkpoint configuration or explicitly present SGFID as an SG+early-checkpoint vari
  2. [§4.3 and App. B.2] The configurations reported in Tables 2–5 are selected by grid search on the metrics used for evaluation: the ImageNet-256 results use FID@5k heatmaps (§4.3) to choose (γstrong, γweak, ω); the T2I GenEval/HPSv3 settings (App. B.2) are chosen using GenEval and a 250-prompt HPSv3 subset. This selection on the test benchmarks makes the reported gains hard to interpret as predictive performance, especially the 1.58 FID point in Table 3, which appears to be an extremum of the search. The authors should provide a validation split or a pre-specified selection rule and report the range of FID over reasonable neighboring hyperparameters.
minor comments (5)
  1. [§4.2] The terms SGFID and SGFLOPS are used without definitions in the main text; only App. B.1 explains them. Define them where first used.
  2. [Table 2] The '+SGFID' row lists '#Epoch 400', but App. B.1 states the auxiliary branch is a 50k-iteration checkpoint. Clarify whether 400 refers to the main training run or to the method.
  3. [Table 5] The SG-vs-CFG GenEval improvement is 0.01 overall (0.61→0.62); no variances or repeated evaluations are reported, so 'consistent improvement' should be softened.
  4. [Eq. (4)] The notation for the masked branch is difficult to parse; the set T_m and the case distinction should be formalized more clearly.
  5. [Contributions, §1] The paper calls SG a 'post-hoc scheduling mechanism' and later introduces a cosine schedule only for SGFID; the terminology should be aligned with the actual method.

Circularity Check

1 steps flagged

Headline 1.58 FID is produced by SGFID, whose weak branch is an early checkpoint with a cosine sparsity schedule (App. B.1), not the same-checkpoint SG defined in Eq. (6)-(7); the reported number therefore does not isolate the proposed mechanism.

specific steps
  1. fitted input called prediction [Abstract / Section 3.2 Eqs. (6)-(7) / Appendix B.1]
    "SGFID (see Section 4.2, Table 3) is obtained through the usage of an early checkpoint of the same model training run. More specifically, we utilize a checkpoint with 50k training iterations. Furthermore, we apply cosine decay from 0.6 to 0.0 on the auxiliary model and the inverse on the main model. This aligns with the findings from Figure 7 where γstrong, γweak can be used to make up for undertrained auxiliary models."

    Section 3.2 defines the weak branch as the same network evaluated at a higher sparsity rate: Dweakθ(c) := Dθ(xt,t,c;γweak) with 0≤γstrong<γweak<1, and Eq. (7) extrapolates between these two evaluations of one network. The headline 'SG achieves FID 1.58' is, however, reported for SGFID, whose weak branch is a different 50k-iteration checkpoint and whose gammas follow a cosine schedule. That configuration folds in AutoGuidance's undertrained-auxiliary mechanism plus a benchmark-fitted schedule, neither of which is present in Eq. (6)-(7). The reported number is therefore not a prediction from SG-as-defined but a fitted composite attributed to the method, so the abstract's central quantitative claim does not isolate the proposed token-sparsity capacity gap.

full rationale

The paper contains no definitional identity in which an output equation equals its input by construction, no imported uniqueness theorem, and no load-bearing self-citation chain: dependence on TREAD [32] is supplemented by external works [74,77] and by the paper's own reproduced baselines. The FID/HPSv3 numbers are empirical measurements after hyperparameter search, which is model selection rather than derivation and would not by itself constitute circularity. The one problematic attribution is the headline SGFID result: Appendix B.1 discloses that SGFID uses a 50k-iteration early checkpoint and a cosine sparsity schedule, whereas Section 3.2 defines SG as the same network evaluated at two static sparsity levels. The 1.58 FID therefore cannot be read as a validation of Eq. (6)-(7) alone; it is a composite that includes AutoGuidance's early-checkpoint mechanism. The same-checkpoint SGFLOPS result (2.14 FID) provides independent evidence for a weaker version of the claim, so the circularity is partial rather than total.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The paper adds three tuned hyperparameters (γstrong, γweak, ω) plus a per-benchmark schedule; the guidance mechanism rests on an unproven monotonicity assumption about sparsity's effect on the conditional distribution.

free parameters (4)
  • γstrong (strong branch sparsity) = 0.5 for SGFLOPS; 0.1 for T2I HPSv3; 0.2 for T2I GenEval; scheduled for SGFID
    Chosen by grid search on FID@5k / HPSv3; controls the high-capacity branch.
  • γweak (weak branch sparsity) = 0.9 for SGFLOPS; 0.8 for T2I HPSv3; 0.7 for T2I GenEval; scheduled for SGFID
    Chosen by grid search; controls the low-capacity guidance branch.
  • ω (guidance scale) = 2.5 for ImageNet (Figs. A5/A6), 1.8 for T2I HPSv3, 2.5 for T2I GenEval
    Chosen per benchmark; extrapolation strength.
  • Cosine sparsity schedule for SGFID = decay 0.6→0.0 on auxiliary, inverse on main (not fully specified)
    Adds a temporal schedule; described in B.1 without exact form or timestep mapping.
axioms (4)
  • domain assumption Increasing token sparsity monotonically softens the conditional distribution
    Section 3.2 claims this without proof; central to SG's guidance signal.
  • domain assumption Extrapolating from a weak (high-sparsity) conditional branch toward a strong (low-sparsity) branch improves fidelity
    Same section; empirically motivated by CFG/AutoGuidance, not derived.
  • domain assumption Random Bernoulli token masks at test time approximate training-time sparsity distribution
    Section 3.2: masks sampled from Bernoulli(1−γ); no importance-based selection.
  • domain assumption An early checkpoint of the same training run is a valid weak branch (for SGFID)
    Appendix B.1 uses a 50k-iteration checkpoint; relies on AutoGuidance [30] finding that an undertrained version guides well.

pith-pipeline@v1.3.0-alltime-deepseek · 18843 in / 15108 out tokens · 146071 ms · 2026-08-03T12:45:55.871052+00:00 · methodology

0 comments
read the original abstract

Diffusion models deliver high quality in image synthesis but remain expensive during training and inference. Recent works have leveraged the inherent redundancy in visual content to make training more affordable by training only on a subset of visual information. While these methods were successful in providing cheaper and more effective training, sparsely trained diffusion models struggle in inference. This is due to their lacking response to Classifier-free Guidance (CFG) leading to underwhelming performance during inference. To overcome this, we propose Sparse Guidance (SG). Instead of using conditional dropout as a signal to guide diffusion models, SG uses token-level sparsity. As a result, SG preserves the high-variance of the conditional prediction better, achieving good quality and high variance outputs. Leveraging token-level sparsity at inference, SG improves fidelity at lower compute, achieving 1.58 FID on the commonly used ImageNet-256 benchmark with 25% fewer FLOPs, and yields up to 58% FLOP savings at matched baseline quality. To demonstrate the effectiveness of Sparse Guidance, we train a 2.5B text-to-image diffusion model using training time sparsity and leverage SG during inference. SG achieves improvements in composition and human preference score while increasing throughput at the same time.

Figures

Figures reproduced from arXiv: 2601.01608 by Bj\"orn Ommer, Felix Krause, Johannes Schusterbauer, Ming Gui, Olga Grebenkova, Stefan Andreas Baumann, Vincent Tao Hu.

Figure 1
Figure 1. Figure 1: Sparse Guidance provides effective, efficient, structure-preserving guidance for sparsely trained diffusion models. (Left) Unlike Classifier-free Guidance, SG stays closer to the conditional prediction, yielding higher-variance, non-collapsed samples. (Right, top) On ImageNet-256, SG (Quality) attains an FID of 1.58 without any previously required dense finetuning while also increasing throughput, and SG (… view at source ↗
Figure 2
Figure 2. Figure 2: Classifier-free Guidance (CFG) provides limited benefits for token-sparse diffusion models. While token-sparse training produces stronger conditional diffusion models than standard dense training, their practical impact has been constrained by poor compatibility with CFG, which limits inference quality and slows adoption in practice. Sparse Guidance (SG) overcomes this limitation, restoring strong guidance… view at source ↗
Figure 3
Figure 3. Figure 3: Masking and Routing as two types of token-level sparsity. Masking replaces tokens with learnable mask token [74] while routing preserves information by reintroducing tokens [32]. more, feature-caching approaches such as DeepCache and Learning-to-Cache accelerate diffusion U-Nets and trans￾formers by reusing intermediate activations across timesteps or layers [45, 46]. Our method builds on train-time sparsi… view at source ↗
Figure 4
Figure 4. Figure 4: Without Sparse Guidance, image quality and compo￾sition worsens consistently with increased token-sparsity ratios. while decreasing γ yields a sharper, higher-capacity predic￾tor. We propose instantiating guidance by using a high-γ (weak) branch to steer a low-γ (strong) branch during sam￾pling. The resulting capacity gap provides the guidance signal. In this view, γ is a single, continuous hyperparameter … view at source ↗
Figure 5
Figure 5. Figure 5: Sparse Guidance improves both convergence and training-time sample quality for sparsely trained diffusion models. Left FID over training iterations comparing CFG, CFG with dense finetuning, and Sparse Guidance (SG). where SG achieves the lowest FID using the best CFG scale ω for each method. Right Training-time sample progress using SG, showing that sparsely trained models already produce high-fidelity sam… view at source ↗
Figure 6
Figure 6. Figure 6: Our method achieves lower FID robustly across different ω by adaptation of γstrong and γweak. We show the combination of AutoGuidance with Sparse Guidance and demonstrate how SG allows for fine grained control over the capacity gap between the D strong θ and D weak θ that drives guidance. Notably, the area of viable settings is broad and shifts under increasing ω towards higher γstrong and γweak. Guidance … view at source ↗
Figure 8
Figure 8. Figure 8: Sparse Guidance provides qualitative improvements on routing and masking models and demonstrates well behaved trade-off between (γstrong, γweak and ω) where larger ω allows for higher rates of sparsity and therefore also higher throughput. constraints. Instead of relying on a precise reference check￾point, (near-) optimal auxiliary models can be recovered from a broad range of training steps by tuning the … view at source ↗
Figure 7
Figure 7. Figure 7: (Left) SG demonstrates smaller LPIPS between the output with guidance and the conditional prediction. (Right) SG allows for better usage of other, less flexible guidance meth￾ods, like AutoGuidance by offering the capability to adjust net￾work capacities without training for fine-grained capacity gaps. 0.0 0.2 0.4 0.6 0.8 0.0 0.2 0.4 0.6 0.8 Routing ( =1.4) 0.0 0.2 0.4 0.6 0.8 0.0 0.2 0.4 0.6 0.8 Routing (… view at source ↗
Figure 9
Figure 9. Figure 9: Selected examples: Sparse Guidance keeps more of the structure of the conditional prediction leading to higher variance in sample distribution while staying truthful to the prompt. on otherwise common objects (see [PITH_FULL_IMAGE:figures/full_fig_p008_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

77 extracted references · 20 linked inside Pith

  1. [1]

    Self-rectifying diffu- sion sampling with perturbed-attention guidance

    Donghoon Ahn, Hyoungwon Cho, Jaewon Min, Wooseok Jang, Jungwoo Kim, SeonHwa Kim, Hyun Hee Park, Ky- ong Hwan Jin, and Seungryong Kim. Self-rectifying diffu- sion sampling with perturbed-attention guidance. InEuropean Conference on Computer Vision, pages 1–17. Springer, 2024. 3, 5

    2024

  2. [2]

    Stochastic interpolants: A unifying framework for flows and diffusions.arXiv preprint arXiv:2303.08797, 2023

    Michael S Albergo, Nicholas M Boffi, and Eric Vanden- Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions.arXiv preprint arXiv:2303.08797, 2023. 2, 3

    Pith/arXiv arXiv 2023

  3. [3]

    Lumiere: A space-time diffusion model for video generation

    Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Guanghui Liu, Amit Raj, et al. Lumiere: A space-time diffusion model for video generation. InSIGGRAPH Asia 2024 Conference Papers, pages 1–11, 2024. 2

    2024

  4. [4]

    Improving image generation with better captions.Computer Science

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions.Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023. 8

    2023

  5. [5]

    Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023. 2

    Pith/arXiv arXiv 2023

  6. [6]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. 2

    2024

  7. [7]

    Coyo-700m: Image-text pair dataset

    Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset. https://github.com/kakaobrain/ coyo-dataset, 2022. 5, 1

    2022

  8. [8]

    Pixart- σ: Weak-to-strong training of diffusion transformer for 4k text-to-image generation, 2024

    Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- σ: Weak-to-strong training of diffusion transformer for 4k text-to-image generation, 2024. 8

    2024

  9. [9]

    Pixart- α: Fast training of diffusion transformer for photorealistic text-to-image synthe- sis.ICLR, 2024

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- α: Fast training of diffusion transformer for photorealistic text-to-image synthe- sis.ICLR, 2024. 8

    2024

  10. [10]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling.arXiv preprint arXiv:2501.17811, 2025

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling.arXiv preprint arXiv:2501.17811, 2025. 8

    Pith/arXiv arXiv 2025

  11. [11]

    Scalable high-resolution pixel-space image synthesis with hourglass diffusion transformers

    Katherine Crowson, Stefan Andreas Baumann, Alex Birch, Tanishq Mathew Abraham, Daniel Z Kaplan, and Enrico Ship- pole. Scalable high-resolution pixel-space image synthesis with hourglass diffusion transformers. InForty-first Interna- tional Conference on Machine Learning, 2024. 1

    2024

  12. [12]

    Diffusion models beat gans on image synthesis.NeurIPS, 2021

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis.NeurIPS, 2021. 3, 5

    2021

  13. [13]

    Continuous conditional generative adversarial networks: Novel empirical losses and label input mechanisms

    Xin Ding, Yongwei Wang, Zuheng Xu, William J Welch, and Z Jane Wang. Continuous conditional generative adversarial networks: Novel empirical losses and label input mechanisms. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 45(7):8143–8158, 2022. 5

    2022

  14. [14]

    An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy. An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020. 1

    Pith/arXiv arXiv 2010

  15. [15]

    Scaling rectified flow transformers for high-resolution image synthesis.ICML, 2024

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim En- tezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis.ICML, 2024. 2, 8

    2024

  16. [16]

    Flux-reason-6m & prism-bench: A million- scale text-to-image reasoning dataset and comprehensive benchmark.arXiv preprint arXiv:2509.09680, 2025

    Rongyao Fang, Aldrich Yu, Chengqi Duan, Linjiang Huang, Shuai Bai, Yuxuan Cai, Kun Wang, Si Liu, Xihui Liu, and Hongsheng Li. Flux-reason-6m & prism-bench: A million- scale text-to-image reasoning dataset and comprehensive benchmark.arXiv preprint arXiv:2509.09680, 2025. 5, 1

    Pith/arXiv arXiv 2025

  17. [17]

    Masked diffusion transformer is a strong image synthe- sizer

    Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is a strong image synthe- sizer. InProceedings of the IEEE/CVF International Confer- ence on Computer Vision (ICCV), pages 23164–23173, 2023. 2, 3, 4, 5, 6

    2023

  18. [18]

    Geneval: An object-focused framework for evaluating text-to- image alignment.Advances in Neural Information Processing Systems, 36:52132–52152, 2023

    Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to- image alignment.Advances in Neural Information Processing Systems, 36:52132–52152, 2023. 7, 8, 2

    2023

  19. [19]

    Gemini 2.0 flash: Model card

    Google DeepMind. Gemini 2.0 flash: Model card. https : / / modelcards . withgoogle . com / assets / documents / gemini - 2 - flash . pdf ,

  20. [20]

    Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis, 2024

    Jian Han, Jinlai Liu, Yi Jiang, Bin Yan, Yuqi Zhang, Zehuan Yuan, Bingyue Peng, and Xiaobing Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis, 2024. 8

    2024

  21. [21]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bern- hard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. InNeurIPS, 2017. 5

    2017

  22. [22]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. InNeurIPS Workshop, 2021. 2, 3, 4, 6 9

    2021

  23. [23]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. InNeurIPS, 2020. 2

    2020

  24. [24]

    Video diffusion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. InarXiv, 2022. 2

    2022

  25. [25]

    Improving sample quality of diffusion models us- ing self-attention guidance

    Susung Hong, Gyuseong Lee, Wooseok Jang, and Seungry- ong Kim. Improving sample quality of diffusion models us- ing self-attention guidance. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 7462– 7471, 2023. 3

    2023

  26. [26]

    Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models

    Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Lup- ing Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. InInternational Con- ference on Machine Learning, pages 13916–13932. PMLR,

  27. [27]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024. 2

    Pith/arXiv arXiv 2024

  28. [28]

    Spatiotemporal skip guidance for enhanced video diffusion sampling

    Junha Hyung, Kinam Kim, Susung Hong, Min-Jung Kim, and Jaegul Choo. Spatiotemporal skip guidance for enhanced video diffusion sampling. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 11006– 11015, 2025. 5

    2025

  29. [29]

    The unreasonable effectiveness of guidance for diffusion models

    Tim Kaiser, Nikolas Adaloglou, and Markus Kollmann. The unreasonable effectiveness of guidance for diffusion models. arXiv preprint arXiv:2411.10257, 2024. 3

    Pith/arXiv arXiv 2024

  30. [30]

    Guiding a diffusion model with a bad version of itself.Advances in Neural Infor- mation Processing Systems, 37:52996–53021, 2024

    Tero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding a diffusion model with a bad version of itself.Advances in Neural Infor- mation Processing Systems, 37:52996–53021, 2024. 3, 4, 6, 7

    2024

  31. [31]

    Kolors: Effective training of diffusion model for photorealistic text-to-image synthesis

    Kolors Team. Kolors: Effective training of diffusion model for photorealistic text-to-image synthesis. https: //huggingface.co/Kwai-Kolors/Kolors , 2024. Technical report and model card. 8

    2024

  32. [32]

    Tread: Token routing for efficient architecture-agnostic diffusion training

    Felix Krause, Timy Phan, Ming Gui, Stefan Andreas Bau- mann, Vincent Tao Hu, and Björn Ommer. Tread: Token routing for efficient architecture-agnostic diffusion training. arXiv preprint arXiv:2501.04765, 2025. 2, 3, 4, 5, 6, 7, 1

    arXiv 2025

  33. [33]

    Improved precision and recall metric for assessing generative models.Advances in neural information processing systems, 32, 2019

    Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models.Advances in neural information processing systems, 32, 2019. 5, 6

    2019

  34. [34]

    Applying guidance in a limited interval improves sample and distribution quality in diffusion models.NeurIPS, 2024

    Tuomas Kynkäänniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in a limited interval improves sample and distribution quality in diffusion models.NeurIPS, 2024. 4

    2024

  35. [35]

    Black Forest Labs. Flux. https://github.com/ black-forest-labs/flux, 2024. 2, 8

    2024

  36. [36]

    Playground v2.5: Three insights to- wards enhancing aesthetic quality in text-to-image generation

    Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2.5: Three insights to- wards enhancing aesthetic quality in text-to-image generation. arXiv preprint arXiv:2402.17245, 2024. 8

    Pith/arXiv arXiv 2024

  37. [37]

    Hunyuan-dit: A powerful multi-resolution dif- fusion transformer with fine-grained chinese understanding

    Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Min- bin Huang, Zedong Xiao, Dayou Chen, Jiajun He, Jiahao Li, Wenyue Li, Chen Zhang, Rongwei Quan, Jianxiang Lu, Jiabin Huang, Xiaoyan Yuan, Xiaoxiao Zheng, Yixuan Li, Jihong Zhang, Chao Zhang, Meng Chen, Jie Liu, Zheng Fang, Weiyan Wang, Jinbao Xu...

    Pith/arXiv arXiv 2024

  38. [38]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. InICLR, 2023. 2, 3

    2023

  39. [39]

    Audi- oldm: Text-to-audio generation with latent diffusion models

    Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audi- oldm: Text-to-audio generation with latent diffusion models. InICML, 2023. 2

    2023

  40. [40]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. InICLR, 2023. 2, 3

    2023

  41. [41]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InICLR, 2019. 5, 1

    2019

  42. [42]

    Exploring the role of large language models in prompt encoding for diffusion models.arXiv preprint arXiv:2406.11831, 2024

    Bingqi Ma, Zhuofan Zong, Guanglu Song, Hongsheng Li, and Yu Liu. Exploring the role of large language models in prompt encoding for diffusion models.arXiv preprint arXiv:2406.11831, 2024. 5, 1

    Pith/arXiv arXiv 2024

  43. [43]

    Sit: Exploring flow and diffusion-based generative models with scalable in- terpolant transformers

    Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable in- terpolant transformers. InEuropean Conference on Computer Vision, pages 23–40. Springer, 2024. 2, 6

    2024

  44. [44]

    Albergo, Nicholas M

    Nanye Ma, Mark Goldstein, Michael S. Albergo, Nicholas M. Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers, 2024. 5

    2024

  45. [45]

    Deepcache: Accelerating diffusion models for free, 2023

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free, 2023. 3

    2023

  46. [46]

    Learning-to-cache: Accelerating diffusion transformer via layer caching, 2024

    Xinyin Ma, Gongfan Fang, Michael Bi Mi, and Xinchao Wang. Learning-to-cache: Accelerating diffusion transformer via layer caching, 2024. 3

    2024

  47. [47]

    Hpsv3: Towards wide-spectrum human preference score

    Yuhang Ma, Xiaoshi Wu, Keqiang Sun, and Hongsheng Li. Hpsv3: Towards wide-spectrum human preference score. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15086–15095, 2025. 1, 7, 8, 2

    2025

  48. [48]

    Improved denoising diffusion probabilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. InICML, 2021. 2

    2021

  49. [49]

    Diff-a-riff: Musical accompani- ment co-creation via latent diffusion models.arXiv preprint arXiv:2406.08384, 2024

    Javier Nistal, Marco Pasini, Cyran Aouameur, Maarten Grachten, and Stefan Lattner. Diff-a-riff: Musical accompani- ment co-creation via latent diffusion models.arXiv preprint arXiv:2406.08384, 2024. 2

    Pith/arXiv arXiv 2024

  50. [50]

    Journeydb: A benchmark for generative image understanding, 2023

    Junting Pan, Keqiang Sun, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Journeydb: A benchmark for generative image understanding, 2023. 5, 1

    2023

  51. [51]

    Scalable diffusion models with transformers.ICCV, 2023

    William Peebles and Saining Xie. Scalable diffusion models with transformers.ICCV, 2023. 3, 5, 6, 1 10

    2023

  52. [52]

    SDXL: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. InThe Twelfth International Conference on Learning Representations, 2024. 8

    2024

  53. [53]

    Hierarchical text-conditional image genera- tion with clip latents.arXiv preprint arXiv:2204.06125, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image genera- tion with clip latents.arXiv preprint arXiv:2204.06125, 2022. 2

    Pith/arXiv arXiv 2022

  54. [54]

    Mixture-of- depths: Dynamically allocating compute in transformer-based language models.arXiv preprint arXiv:2404.02258, 2024

    David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of- depths: Dynamically allocating compute in transformer-based language models.arXiv preprint arXiv:2404.02258, 2024. 3

    Pith/arXiv arXiv 2024

  55. [55]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InCVPR, 2022. 2, 5, 8, 1

    2022

  56. [56]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation

  57. [57]

    No training, no problem: Rethinking classifier-free guidance for diffusion models.arXiv preprint arXiv:2407.02687, 2024

    Seyedmorteza Sadat, Manuel Kansy, Otmar Hilliges, and Romann M Weber. No training, no problem: Rethinking classifier-free guidance for diffusion models.arXiv preprint arXiv:2407.02687, 2024. 3, 6

    Pith/arXiv arXiv 2024

  58. [58]

    Improved techniques for training gans

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. InNeurIPS, 2016. 5

    2016

  59. [59]

    Baumann, Vincent Tao Hu, and Björn Ommer

    Johannes Schusterbauer, Ming Gui, Pingchuan Ma, Nick Stracke, Stefan A. Baumann, Vincent Tao Hu, and Björn Ommer. Boosting latent diffusion with flow matching. In ECCV, 2024. 2

    2024

  60. [60]

    Stretching each dollar: Diffu- sion training from scratch on a micro-budget.arXiv preprint arXiv:2407.15811, 2024

    Vikash Sehwag, Xianghao Kong, Jingtao Li, Michael Spranger, and Lingjuan Lyu. Stretching each dollar: Diffu- sion training from scratch on a micro-budget.arXiv preprint arXiv:2407.15811, 2024. 2, 6

    Pith/arXiv arXiv 2024

  61. [61]

    Glu variants improve transformer.arXiv preprint arXiv:2002.05202, 2020

    Noam Shazeer. Glu variants improve transformer.arXiv preprint arXiv:2002.05202, 2020. 1

    Pith/arXiv arXiv 2002

  62. [62]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. InICML, 2015. 2

    2015

  63. [63]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. InICLR, 2021. 2

    2021

  64. [64]

    Improved techniques for training score-based generative models.Advances in neural information processing systems, 33:12438–12448, 2020

    Yang Song and Stefano Ermon. Improved techniques for training score-based generative models.Advances in neural information processing systems, 33:12438–12448, 2020

    2020

  65. [65]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020. 2

    Pith/arXiv arXiv 2011

  66. [66]

    Stable diffusion 2.0 release

    Stability AI. Stable diffusion 2.0 release. https : //stability.ai/news/stable-diffusion-v2- release, 2022. 8

    2022

  67. [67]

    Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063,

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063,

  68. [68]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Am- jad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 1

    Pith/arXiv arXiv 2023

  69. [69]

    Attention is all you need.NeurIPS, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.NeurIPS, 2017. 3, 1

    2017

  70. [70]

    DeepMind Veo2-Team. Veo 2. 2024. 2

    2024

  71. [71]

    Fasterdit: Towards faster diffusion transformers training with- out architecture modification.Advances in Neural Informa- tion Processing Systems, 37:56166–56189, 2024

    Jingfeng Yao, Cheng Wang, Wenyu Liu, and Xinggang Wang. Fasterdit: Towards faster diffusion transformers training with- out architecture modification.Advances in Neural Informa- tion Processing Systems, 37:56166–56189, 2024. 6

    2024

  72. [72]

    Representa- tion alignment for generation: Training diffusion transformers is easier than you think.arXiv preprint arXiv:2410.06940,

    Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representa- tion alignment for generation: Training diffusion transformers is easier than you think.arXiv preprint arXiv:2410.06940,

    Pith/arXiv arXiv

  73. [73]

    Root mean square layer normalization.Advances in neural information processing systems, 32, 2019

    Biao Zhang and Rico Sennrich. Root mean square layer normalization.Advances in neural information processing systems, 32, 2019. 1

    2019

  74. [74]

    Fast training of diffusion models with masked trans- formers.TMLR, 2024

    Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anand- kumar. Fast training of diffusion models with masked trans- formers.TMLR, 2024. 2, 3, 5, 6, 1

    2024

  75. [75]

    Cogview4-6b

    ZhipuAI. Cogview4-6b. https://huggingface.co/ zai-org/CogView4-6B, 2025. 8

    2025

  76. [76]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shen- glong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025. 5, 1

    Pith/arXiv arXiv 2025

  77. [77]

    Describe the image by detailing the color, shape, size, texture, quantity, text, and spatial relationships of the objects

    Rui Zhu, Yingwei Pan, Yehao Li, Ting Yao, Zhenglong Sun, Tao Mei, and Chang Wen Chen. Sd-dit: Unleashing the power of self-supervised discrimination in diffusion transformer. In CVPR, pages 8435–8445, 2024. 2, 3, 4, 6 11 A. Implementation Details A.1. Training Details for T2I ArchitectureWe implement our transformer models [ 14, 69] largely following the ...

    2024