Pith. sign in

REVIEW 3 major objections 4 minor 16 references

A $600 training run yields 1024px text-to-image

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 04:29 UTC pith:MVJWM4GW

load-bearing objection A credible and refreshingly honest low-budget training recipe for T2I diffusion; the abstract oversells 'competitive quality' but the body delivers a useful existence proof with released code and model. the 3 major comments →

arxiv 2509.06068 v1 pith:MVJWM4GW submitted 2025-09-07 cs.CV

Home-made Diffusion Model from Scratch to Hatch

Shih-Ying Yeh This is my paper
classification cs.CV
keywords text-to-image diffusionflow matchingU-shaped transformercross-attention skip connectionsconsumer hardware trainingshifted square cropposition map camera controlefficient scaling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that a small text-to-image diffusion model, with a 343M-parameter backbone, can be trained from scratch to 1024x1024 output on four consumer GPUs for roughly $535-620, producing quality it calls competitive with far larger systems. To do this, it proposes the Cross-U-Transformer (XUT), a U-shaped transformer whose skip connections use cross-attention rather than concatenation, alongside a training recipe with TREAD token routing, a shifted-square-crop scheme for arbitrary aspect ratios, and progressive resolution scaling. The training scheme also yields two emergent behaviors the paper highlights: composition consistency when individual prompt attributes are edited, and camera-like pan/zoom control by shifting or scaling a position map at inference. If the claims hold, the main consequence is that architectural design and training efficiency can substitute for raw scale, lowering the barrier to entry for individual researchers.

Core claim

The central claim is that XUT's cross-attention skip connections give the decoder context-dependent access to encoder features, improving compositional consistency, and that training on randomly shifted square crops with a normalized position map teaches the model to inherit spatial control, so that at inference the position map acts as a camera. The paper demonstrates this on Danbooru2023 (7.6M images) with a 343M XUT-base model using Qwen3-0.6B text embeddings, a fine-tuned SDXL VAE, a flow-matching objective, and TREAD at 50% token selection, reaching 1024x1024 after a 385-hour schedule on four RTX5090s. The contribution is framed as an existence proof: a competitive text-to-image model c

What carries the argument

Two load-bearing components carry the argument. The first is the Cross-U-Transformer (XUT), which replaces concatenation-based skip connections with cross-attention between symmetric encoder and decoder depths, letting the decoder selectively attend to encoder features based on the current decoding context. The second is the shifted square crop with a position map: each training image is resized so the short side equals the target size, a random square crop is taken, and a 2D axial RoPE position map whose per-axis ranges are sqrt(H/W) and sqrt(W/H) is attached to the latent. TREAD token routing (50% selection during most stages, 0% at the final 1024 stage) provides convergence acceleration,

Load-bearing premise

The central claims rest on the untested assumption that the architecture and training components—cross-attention skips, shifted square crop, and positional encoding—are the causes of the observed quality and control, since the report includes no ablations or baseline comparisons.

What would settle it

Run the identical 385-hour training schedule with XUT's cross-attention skips replaced by concatenation skips at the same compute, dataset, and seed budget; if composition consistency under prompt edits and camera-control scores are unchanged, the paper's central attribution is undercut. A weaker test is to measure pairwise structural similarity across seed-matched prompt edits for XUT versus a concatenation-skip baseline.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A 1024x1024 text-to-image model can be trained for $535-620 on four RTX5090s, lowering the financial barrier by roughly an order of magnitude compared with data-center runs.
  • XUT's cross-attention skips yield consistent global composition when prompt attributes are swapped under a fixed seed, enabling fine-grained attribute editing without retraining.
  • Training on shifted square crops with position maps produces arbitrary aspect-ratio generation at inference and lets users pan and zoom by editing the position map.
  • Using different TREAD selection rates for conditional and unconditional prompts provides auto-guidance at inference without an extra model.
  • The recipe supports larger future datasets (40M+ images) because it avoids latent/text caching and relies on on-the-fly preprocessing.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Without an ablation, the compositional consistency attributed to XUT could in principle be produced by the shifted-crop/position-map training or by the dataset; a controlled XUT-versus-concatenation-skip comparison at equal budget is the decisive missing test.
  • The camera-control mechanism follows from normalizing positions by aspect ratio, suggesting that any model trained with such position maps—not just anime-domain models—should exhibit similar pan/zoom control.
  • A frozen or lightly trained small causal language model with no position embedding may be sufficient as a text encoder for T2I, further cutting memory and cost if validated against CLIP/T5 at equal scale.
  • If reproducible, the recipe implies that the bottleneck for individual T2I research shifts from GPU cost to data quality and curation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces HDM, a text-to-image diffusion model designed to be trained on consumer hardware. The main technical proposals are: (i) XUT, a U-shaped transformer that replaces concatenation/additive skip connections with cross-attention; (ii) a shifted square crop training strategy with position maps, enabling arbitrary aspect ratios and emergent camera control; and (iii) a progressive-resolution recipe (256²→1024²) using TREAD acceleration, an SDXL/EQ-VAE latent space, and a Qwen3-0.6B text encoder. The paper reports a training cost of $535–620 on four RTX5090 GPUs over 385 GPU-hours and presents qualitative examples for prompt-consistency and position-map manipulation. Section C explicitly reframes the contribution as an existence proof and acknowledges that no extensive ablation or benchmarking was performed.

Significance. If substantiated, the existence proof is valuable: it shows that a 343M-parameter latent diffusion transformer can be trained for roughly $600 on consumer hardware, and the shifted-crop position-map camera control is a genuinely interesting emergent phenomenon. The paper also ships code and model weights, which strengthens reproducibility. However, the headline claims—'competitive 1024×1024 generation quality', 'superior feature integration', and 'remarkable compositional consistency'—are not supported by the evidence in the current manuscript. The qualitative examples are useful but curated, and the paper itself concedes in Section C that it cannot claim superiority over UViT/HDiT-style baselines without a comparative analysis. The paper is best received as a reproducible existence proof, not yet as a validated architecture contribution.

major comments (3)
  1. [Abstract; Section 5; Section C] The abstract claims 'competitive 1024×1024 generation quality', but Section 5 and Appendix B provide only a handful of selected images. There are no quantitative metrics (FID, CLIP score, human preference), no comparison baselines, and no negative examples. Section C explicitly states that the model was not benchmarked against established metrics. This is load-bearing because 'competitive' is the central quality claim. Either add a quantitative comparison (even on a small set with standard metrics) or rewrite the abstract/conclusion to claim only a feasible low-cost existence proof.
  2. [Section 3.1; Section 5.1; Section C] The paper attributes the observed 'compositional consistency' to XUT's cross-attention skip connections, and the abstract claims that cross-attention provides 'superior feature integration'. No controlled comparison is provided against UViT, HDiT, or a matched transformer with concatenation/additive skip connections. Section C states: 'we cannot definitively claim its superiority over alternative U-shaped transformer designs.' This admission is in tension with the core contribution as presented in Sections 1 and 3.1. A matched ablation, at least at 256² resolution with the same training recipe, is needed to isolate the effect of XUT; otherwise the observed behaviors could equally arise from the shifted-crop/position-map training or the overall recipe.
  3. [Section 3.4, Eq. (15); Appendix A.1] There is a concrete inconsistency in the definition of the position map. Eq. (15) defines P[i,j] = (i·rH/Hresized, j·rW/Wresized), which ranges from 0 to rH/rW and is not centered at zero. The reference implementation in Appendix A.1 uses linspace(-h_range, h_range, h) and linspace(-w_range, w_range, w), i.e., values centered at zero with negative and positive extents. These two definitions imply different semantics for 'camera' translation and zoom in Section 3.7 and are not interchangeable. Since the emergent camera-control claim rests on this mechanism, the paper must specify the exact normalization—including whether the position map is centered, scaled to [-rH, rH] etc.—and make Eq. (15) and the code consistent.
minor comments (4)
  1. [Section 4.3] The energy figure is implausible as stated. 385 training hours on four RTX5090 GPUs would consume several hundred kWh under realistic load, not ~125 kWh 'across four RTX5090 GPUs'. Please clarify whether this is per-GPU, excludes non-GPU power, or is a corrected measurement.
  2. [Section 3.6] Pixtral-11M is likely a typo for a model with a much larger scale (e.g., Pixtral-12B). Please verify the model name.
  3. [Abstract] The phrase 'Cross-U-Transformer (XUT), a novel U-shape transformer, Cross-U-Transformer (XUT)' is repeated; the abstract should be copy-edited.
  4. [Section 3.2] The text says HDM 'directly concatenates text and image features as input to the entire backbone' but then says it 'retains adaLN for conditional information'. Please clarify what conditioning information goes to adaLN and what is concatenated as tokens, because Eq. (8) refers to 'other conditioning information'.

Circularity Check

0 steps flagged

No circular derivation: the cost and training-recipe claims are direct measurements, and the unsupported XUT-quality attribution is an empirical gap, not a logical circle.

full rationale

The paper does not disguise a fitted quantity as a prediction or derive a conclusion from its own definition. The training cost ($535–620) is computed from wall-clock hours and rental prices; the position-map/camera-control behavior is shown through image examples and follows from explicit conditioning on normalized coordinate maps, not from an equation that assumes the result. The abstract's assertion that XUT cross-attention skip connections yield 'remarkable compositional consistency' is not backed by ablation, but Section C explicitly concedes: 'we have not conducted extensive ablation studies or benchmarked our model against established metrics' and 'we cannot definitively claim its superiority over alternative U-shaped transformer designs.' That is an unsupported empirical claim, not circularity: no parameter was fitted to the claimed outcome and then renamed as a prediction. The positional-encoding constraints (Eqs. 10–12) are solved self-consistently from stated definitions, and the shifted-square-crop procedure is specified independently of the emergent behaviors it is said to explain. No self-citation chain is load-bearing, no uniqueness theorem is imported from the authors, and no ansatz is smuggled in via citation. The main weakness is missing comparative evaluation, which the paper itself flags; that affects evidentiary strength, not logical circularity.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 0 invented entities

The central claim depends on standard diffusion theory, two modules (TREAD, EQ-VAE) taken from prior work, a specific dataset/caption setup, and the unsupported assumption that a few curated images can support the 'competitive' label.

free parameters (1)
  • Position map normalization = r_H * r_W = 1.0
    Chosen by hand in Section 3.3 (Eq 10-11) to fix the scale of the RoPE coordinate ranges; it is not derived from data or a uniqueness theorem.
axioms (5)
  • standard math Flow matching objective (Eq 4) provides a valid training signal for generative modeling
    Assumed from Lipman et al. 2023; not proven in this paper.
  • domain assumption The SDXL VAE fine-tuned with EQ-VAE yields a latent space more suitable for diffusion than standard VAEs
    Adopted from Kouzelis et al. 2025; no internal verification.
  • domain assumption TREAD token routing accelerates convergence without degrading final quality
    Taken from Krause et al. 2025b; the paper does not ablate TREAD.
  • domain assumption The Danbooru2023 dataset with Pixtral-generated captions is sufficient to train a competitive text-to-image model
    Used as the training data; no general-domain evaluation.
  • ad hoc to paper A handful of curated generated images is evidence of competitive quality
    Section 5 only shows favorable examples; no quantitative metric.

pith-pipeline@v1.3.0-alltime-deepseek · 16428 in / 10809 out tokens · 105969 ms · 2026-08-05T04:29:12.092069+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Home-made Diffusion Model from Scratch to Hatch." pith.science (2026). https://pith.science/paper/MVJWM4GW

@misc{pith2026250906068,
  author       = {Pith},
  title        = {Pith review of: Home-made Diffusion Model from Scratch to Hatch},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MVJWM4GW}},
  note         = {Machine review of arXiv:2509.06068}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We introduce Home-made Diffusion Model (HDM), an efficient yet powerful text-to-image diffusion model optimized for training (and inferring) on consumer-grade hardware. HDM achieves competitive 1024x1024 generation quality while maintaining a remarkably low training cost of $535-620 using four RTX5090 GPUs, representing a significant reduction in computational requirements compared to traditional approaches. Our key contributions include: (1) Cross-U-Transformer (XUT), a novel U-shape transformer, Cross-U-Transformer (XUT), that employs cross-attention for skip connections, providing superior feature integration that leads to remarkable compositional consistency; (2) a comprehensive training recipe that incorporates TREAD acceleration, a novel shifted square crop strategy for efficient arbitrary aspect-ratio training, and progressive resolution scaling; and (3) an empirical demonstration that smaller models (343M parameters) with carefully crafted architectures can achieve high-quality results and emergent capabilities, such as intuitive camera control. Our work provides an alternative paradigm of scaling, demonstrating a viable path toward democratizing high-quality text-to-image generation for individual researchers and smaller organizations with limited computational resources.

Figures

Figures reproduced from arXiv: 2509.06068 by Shih-Ying Yeh.

Figure 1
Figure 1. Figure 1: HDM system overview showing the complete pipeline from input processing through XUT backbone [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Detailed XUT backbone architecture showing the encoder-decoder structure with cross-attention [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: An visualized example of position map creation and shifted square crop workflow. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 4 canonical work pages

  1. [1]

    All are worth words: A vit backbone for diffusion models

    Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. InComputer Vision and Pattern Recognition 2023, pages 22669–22679, 06

    work page 2023

  2. [5]

    14 By KohakuBlueLeaf(Shih-Ying Yeh) from Kohaku-Lab Xingjian Leng, Jaskirat Singh, Yunzhong Hou, Zhenchang Xing, Saining Xie, and Liang Zheng

    URLhttps://arxiv.org/abs/2506.15742. 14 By KohakuBlueLeaf(Shih-Ying Yeh) from Kohaku-Lab Xingjian Leng, Jaskirat Singh, Yunzhong Hou, Zhenchang Xing, Saining Xie, and Liang Zheng. Repa-e: Unlocking vae for end-to-end tuning with latent diffusion transformers,

    Pith/arXiv arXiv

  3. [6]

    URLhttps://arxiv.org/ abs/2504.10483. Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, Dayou Chen, Jiajun He, Jiahao Li, Wenyue Li, Chen Zhang, Rongwei Quan, Jianxiang Lu, Jiabin Huang, Xiaoyan Yuan, Xiaoxiao Zheng, Yixuan Li, Jihong Zhang, Chao Zhang, Meng Chen, Jie Liu...

    arXiv

  4. [7]

    org/abs/2405.08748

    URLhttps://arxiv. org/abs/2405.08748. Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. InThe Eleventh International Conference on Learning Representations,

    Pith/arXiv arXiv

  5. [8]

    William Peebles and Saining Xie

    URL https://openreview.net/forum?id=PqvMRDCJT9t. William Peebles and Saining Xie. Scalable diffusion models with transformers.arXiv preprint arXiv:2212.09748,

    Pith/arXiv arXiv

  6. [9]

    AlecRadford, JongWookKim, ChrisHallacy, AdityaRamesh, GabrielGoh, SandhiniAgarwal, GirishSastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever

    URLhttps://arxiv.org/abs/2503.21758. AlecRadford, JongWookKim, ChrisHallacy, AdityaRamesh, GabrielGoh, SandhiniAgarwal, GirishSastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision,

    Pith/arXiv arXiv

  7. [11]

    Jascha Sohl-Dickstein, Eric A

    URLhttps://arxiv.org/ abs/2502.14831. Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. InProceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML’15, page 2256–2265. JMLR.org,

    Pith/arXiv arXiv

  8. [12]

    doi: 10.1016/j.neucom.2023.127063

    ISSN 0925-2312. doi: 10.1016/j.neucom.2023.127063. URLhttps://doi.org/10.1016/j.neucom.2023.127063. Jie Wang, Tao Ji, Yuanbin Wu, Hang Yan, Tao Gui, Qi Zhang, Xuanjing Huang, and Xiaoling Wang. Length generalization of causal transformers without position encoding,

    work page arXiv 2023

  9. [13]

    Zhendong Wang, Yifan Jiang, Huangjie Zheng, Peihao Wang, Pengcheng He, Zhangyang Wang, Weizhu Chen, and Mingyuan Zhou

    URLhttps://arxiv.org/abs/ 2404.12224. Zhendong Wang, Yifan Jiang, Huangjie Zheng, Peihao Wang, Pengcheng He, Zhangyang Wang, Weizhu Chen, and Mingyuan Zhou. Patch diffusion: Faster and more data-efficient training of diffusion models. In Thirty-seventh Conference on Neural Information Processing Systems,

    Pith/arXiv arXiv

  10. [14]

    Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, and Song Han

    URLhttps://arxiv.org/abs/2508.02324. Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, and Song Han. SANA: Efficient high-resolution text-to-image synthesis with linear diffusion transformers. InThe Thirteenth International Conference on Learning Representations,

    Pith/arXiv arXiv

  11. [15]

    Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie

    URL https://arxiv.org/abs/2505.09388. Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. In The Thirteenth International Conference on Learning Representations,

    Pith/arXiv arXiv

  12. [16]

    ij")) returnposition_map A.2 Shifted Square Crop Strategy Following pseudo code provide a reference implementation of the

    URLhttps://openreview. net/forum?id=DJSZGGZYVi. 16 By KohakuBlueLeaf(Shih-Ying Yeh) from Kohaku-Lab Appendix Table of Contents A Implementation Details 18 A.1 Position Map Generation . . . . . . . . . . . . . . . . . . . . . . . . . 18 A.2 Shifted Square Crop Strategy . . . . . . . . . . . . . . . . . . . . . . . 18 B Image Examples 19 C Limitations 29 D ...

    work page 2025

  13. [2021]

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer

    URLhttps://arxiv.org/abs/2103.00020. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models,

    Pith/arXiv arXiv

  14. [2023]

    Chen Chen, Rui Qian, Wenze Hu, Tsu-Jui Fu, Jialing Tong, Xinze Wang, Lezhi Li, Bowen Zhang, Alex Schwing, Wei Liu, and Yinfei Yang

    doi: 10.1109/CVPR52729.2023.02171. Chen Chen, Rui Qian, Wenze Hu, Tsu-Jui Fu, Jialing Tong, Xinze Wang, Lezhi Li, Bowen Zhang, Alex Schwing, Wei Liu, and Yinfei Yang. Dit-air: Revisiting the efficiency of diffusion model architecture design in text to image generation, 2025a. URLhttps://arxiv.org/abs/2503.10618. Chubin Chen, Jiashu Zhu, Xiaokun Feng, Nish...

    work page arXiv 2023

  15. [2024]

    Jonathan Ho, Ajay Jain, and Pieter Abbeel

    URLhttps://arxiv.org/abs/2403.03206. Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. InProceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA,

    Pith/arXiv arXiv

  16. [2025]

    Felix Krause, Timy Phan, Ming Gui, Stefan Andreas Baumann, Vincent Tao Hu, and Björn Ommer

    URLhttps://arxiv.org/abs/ 2502.09509. Felix Krause, Timy Phan, Ming Gui, Stefan Andreas Baumann, Vincent Tao Hu, and Björn Ommer. Tread: Token routing for efficient architecture-agnostic diffusion training - inference implementation, 2025a. URLhttps://github.com/CompVis/tread/blob/b1f5cc8707a83ffcd6a1dba2b0fd6985f325f4ff/ inference.py#L31. GitHub reposito...

    Pith/arXiv arXiv