Pith. sign in

REVIEW 3 major objections 6 minor 34 references

Decoder-Only LLMs are Better Controllers for Diffusion Models

T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that replacing encoder-based text encoders with block-wise hidden states of decoder-only LLMs, via a lightweight adapter, makes text-to-image diffusion models more controllable and higher quality.

desk verdict The adapter is a genuine, clean idea, but the title-level claim is confounded by model scale and missing uncertainty quant. read the letter →

arxiv 2502.04412 v1 pith:CZJZH3KI submitted 2025-02-06 cs.CV cs.AIcs.CL

Ziyi Dong , Yao Xiao , Pengxu Wei , Liang Lin This is my paper · ORCID
classification cs.CVcs.AIcs.CL
keywords Text-to-ImageGenerationDiffusionModelsDecoder-onlyLLMLLMDiff-AdapterTextEncodingCross-AttentionSemanticUnderstandingControllability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the text encoders used by current text-to-image diffusion models, typically encoder-only models like CLIP or T5, are a bottleneck: they miss entity relationships, counts, and details in complex prompts. The authors propose replacing them with a decoder-only large language model (LLM) whose block-wise hidden states are turned into text encodings through a small adapter module attached to the diffusion U-Net's cross-attention. They support this with a derivation that views a transformer LLM as a denoising diffusion process, so text encodings can be sampled by Langevin dynamics from the difference between sentence-level and single-word LLM predictions. Experiments on Stable Diffusion 1.5 show higher SigLIP alignment scores and better human-rated logic than SD1.5, SDXL, and T5-based variants, with the gains increasing with LLM size.

What carries the argument

The central object is the LLMDiff-Adapter, a plug-in module that combines two cross-attention branches: the original branch aligned with the existing text encoder and a new branch fed by block-wise hidden states from a decoder-only LLM. The theoretical engine is the 'LLM as diffusion model' view, in which each transformer block acts as a denoising step and the score of the text-encoding distribution is approximated as the difference between sentence-level and single-word block predictions (Eq. 5). Algorithm 1 turns this score into actual text encodings by Langevin dynamics sampling from the LLM's hidden states. The adapter's two branches are mixed with learnable weights, initialized so the original branch dominates at the start of training.

What would settle it

Run Algorithm 1 on a set of prompts and compare the resulting text encodings against a reference encoding obtained by a method that does not rely on the sentence-minus-word difference, such as directly optimizing the encoding to reproduce the LLM's next-token predictions; if the two disagree substantially on context-dependent prompts, the theoretical claim is contradicted. A cheaper check: use a single-word prompt, where sentence-level and single-word scores are equal by construction; the derivation predicts no useful conditioning signal, so if the generated images still respond meaningfully to the prompt, the approximation is not the source of control.

Watch

Extended reading notes

Core claim

The central claim is that decoder-only LLMs are better controllers for diffusion models than encoder-based text encoders. The paper shows that, although decoder-only LLMs do not expose text features directly, their per-block hidden states can be interpreted as the result of a diffusion process. Under this view, the text-encoding distribution can be estimated by comparing the LLM's prediction for a full sentence with its prediction for a single word at each block, and the resulting score is used in Langevin dynamics to sample text encodings. These encodings are fed through a lightweight adapter, formed by two cross-attention branches, one aligned to the original text encoder and one learned from the LLM encoding, into a pre-trained diffusion U-Net. With the LLM and U-Net frozen, only the adapter is trained, and the resulting model improves generation quality, detail, and logical consistency, including counting entities, respecting spatial relations, and inferring physical consequences.

Load-bearing premise

The derivation's load-bearing step is the approximation in Eq. (5), which asserts that the score of the text-encoding distribution is the difference between the LLM's sentence-level and single-word block predictions; if that equality does not hold, Algorithm 1 loses its theoretical justification.

Editorial extensions

If this is right

  • Stronger decoder-only LLMs yield stronger controllability: Vicuna-7B outperforms phi-1.5, which itself improves over the Stable Diffusion 1.5 baseline.
  • The adapter needs only about 45 million trainable parameters while both the LLM and the diffusion U-Net remain frozen.
  • The approach captures counts, entity relations, and physical reasoning that encoder-based baselines miss, as the qualitative examples show.
  • The method generalizes across LLM scales, since both a compact model (phi-1.5) and a 7B model (Vicuna-7B) improve generation quality and alignment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The theoretical backing rests on an unproven approximation; if Eq. (5) does not hold, the method still works as an empirically motivated heuristic, but its claimed principled grounding weakens.
  • A natural extension is to apply the same adapter to other conditional diffusion models, such as video or 3D generation, where prompt controllability is also a known limitation.
  • The block-wise hidden-state requirement rules out closed-source LLMs; the authors note this, which implies the approach currently depends on open-weight models with accessible internal states.
  • One could isolate the contribution of the theory by abating the sentence-minus-word score with a simpler pooling of hidden states; if that baseline matches the full method, the theoretical derivation is not the source of the gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes LLMDiff-Adapter, a plug-and-play module that lets pre-trained text-to-image diffusion models (Stable Diffusion 1.5) use block-wise hidden states from a decoder-only LLM (Phi-1.5 or Vicuna-7B) as text conditioning, in place of encoder-based text encoders such as CLIP or T5. The authors argue, from a probabilistic perspective, that a decoder-only LLM can be viewed as a reverse diffusion process, and that text encodings for image generation can be obtained by running Langevin dynamics on a score function approximated as the difference between sentence-level and single-word LLM prediction scores. They report quantitative results (SigLIP score, CLIP-IQA quality/complexity/beauty, and a described user study) and qualitative examples. The central claim is that decoder-only LLMs are better controllers for diffusion models than encoder-based text encoders, and that the proposed adapter yields superior controllability and image quality.

Significance. If established, the claim would be significant for text-to-image generation because it offers a simple way to exploit the semantic and reasoning capabilities of large decoder-only LLMs without retraining the diffusion model. The adapter design is straightforward, keeps the base diffusion model and LLM frozen, and is potentially applicable to other diffusion architectures. The paper makes a genuine attempt at theoretical grounding, which is rare in this area. However, the empirical evidence is currently not conclusive: the only large improvement over baselines comes from a 7B LLM, while the size-matched 1.3B variant is markedly worse, and the reported user-study metric is never actually presented. The theoretical derivation also contains an unproven core approximation. These issues need to be addressed before the central claim can be accepted.

major comments (3)
  1. [Section 5.2, Table 1] The empirical evidence does not isolate the effect of the decoder-only architecture from model scale. The strongest result (SigLIP 8.5) uses Vicuna-7B, which has roughly 7B parameters, whereas the text encoders of the baseline models are much smaller (CLIP is roughly 0.4B and T5-XL roughly 2.8B). More importantly, the size-matched attempt, Ours (phi-1.5) with 1.3B parameters, achieves SigLIP 5.8, which is below SD1.5+(T5-XL) at 7.4. This outcome directly undercuts the title-level claim that decoder-only LLMs are better controllers: if a similar-scale decoder-only model is worse than an encoder-decoder model, the Vicuna-7B gain is more plausibly attributed to scale and pretraining data than to the decoder-only structure. The statement in Section 5.1 that "Phi1.5 is close to that of the text encoders of CLIP and T5" is inaccurate for T5-XL, which has about twice the parameters. A proper comparison should include a decoder-only model of comparable size to T5-XL, or a T5 variant sized to match Phi-1.5.
  2. [Section 3.4, Eq. (5)] The approximation in Eq. (5) is the load-bearing step for Algorithm 1, but it is asserted rather than derived. Equation (4) gives a proportionality for the posterior p(x^t_<d | ...), not a statement about the score of p(c_<d | ...). The jump from these posterior ratios to the score function of the text encoding c_<d, with an unspecified time-dependent scalar g(t), involves an implicit and unjustified relationship between the token sequence x and the implicit continuous encoding c. Equation (6) then further replaces score functions with log-probability differences in a way that does not follow from Eq. (1) or Eq. (4). Consequently, the "supporting theoretical analysis" does not actually justify the sentence-minus-word score in Algorithm 1. The authors should either supply a rigorous derivation of this approximation, or explicitly reframe Section 3.4 as an intuition/heuristic and remove the claim that the method is theoretically grounded.
  3. [Section 5.1, Metrics] The user study, which is described as the metric for logicality, is never reported anywhere in the paper. This is a significant omission because the qualitative claims about reasoning and logical coherence (Section 5.4) rest on this evaluation. Additionally, no error bars, confidence intervals, or significance tests are provided for any of the SigLIP or CLIP-IQA scores in Table 1; several differences between methods are small (e.g., Complexity 24.9 vs. 23.9, Beauty 91.0 vs. 90.9), and it is unclear whether these are reproducible. The authors should report the user study results (number of participants, prompts, and outcome) and provide variance or significance information for the quantitative metrics.
minor comments (6)
  1. [Abstract] There is a typo: "stat-of-the-art" should be "state-of-the-art."
  2. [Section 3.2] "For similarity" should read "For simplicity."
  3. [Figure 2 caption] The caption contains the typo "Scentence"; it should be "Sentence."
  4. [Section 5.2] The claim that the Vicuna-7B SigLIP score of 8.5 is "31% higher" than SDXL's 6.2 is arithmetically incorrect; the increase is approximately 37%. The percentage for phi-1.5 compared to SD1.5 is correct.
  5. [Section 5.1 / Table 1] The paper does not state how many prompts or images were used for the SigLIP and CLIP-IQA evaluations, nor how images were sampled (e.g., seeds, number of samples per prompt). This information is needed to assess the reliability of the reported scores.
  6. [Section 3.4, Algorithm 1] The notation for the score functions S_theta is introduced only informally in the text preceding Algorithm 1; the arguments of S_theta in lines 4 and 5 of the algorithm (e.g., S_theta t(x^{t-1}_d, x^t_d, x^t_<d)) do not match the notation of Eq. (5), where the scores are taken with respect to different variables. Please define these score functions explicitly.

Circularity Check

1 steps flagged · score 6.0 of 10

The Sec. 3.4 'derivation' of decoder-only text encodings is self-definitional: Eq. (5) postulates the sentence-minus-word score update that Algorithm 1 then executes, so the theoretical analysis restates the algorithm by construction; the empirical evaluation remains independent.

  1. self definitional [Section 3.4, Eq. (5); Algorithm 1, line 6]
    "there exists a c<d for decoder-only LLMs, which is the unbiased estimator of x<d. ... In accordance with Eqn. (4), the score function of pθt(c<d|xdt,xd t−1) can be approximated as follows: ∇c log pθt(c<d|xdt,xd t−1)≈g(t)(∇x log pθt(xd t−1|xdt,x t<d)−∇x log pθt(xd t−1|xdt)), (5)"

    The derivation chain is: Eq. (4) yields a ratio p(x_<d | ...) ∝ p(sentence)/p(word); Eq. (5) then asserts without proof that the score of the text-encoding distribution p(c_<d | ...) equals g(t) times the difference of the sentence and single-word scores. This is exactly the update executed in Algorithm 1, line 6: '∇ log pθt(c<d|xdt,xd t−1)← s_sentence−s_word'. Because c_<d is never given an independent definition—it is only declared to be an unbiased estimator of x_<d—the 'derived' score is a definition of the text encoding in terms of the sentence-minus-word hidden-state difference. The theoretical analysis therefore reduces by construction to the algorithm it is supposed to justify, with no independent content beyond the unconstrained scalar g(t).

full rationale

The only strong circularity candidate is Section 3.4. The paper claims to derive text encodings from decoder-only LLMs, but Eq. (5) introduces an approximation that is literally the Algorithm 1 update; the target quantity c_<d is not defined independently, so the derivation is a restatement of the algorithm rather than a derivation. This makes the paper's 'supporting theoretical analysis' partially self-definitional. However, the main empirical claim—that a decoder-only LLM plus the LLMDiff-Adapter improves text-to-image quality and controllability—is evaluated with external metrics (SigLIP, CLIP-IQA) against CLIP-, T5-, and SDXL-based baselines, and the adapter is trained with a standard diffusion MSE loss. That part is not circular and provides independent evidence for the method. There are no load-bearing self-citations or imported uniqueness theorems. The stated limitation in Section 6 (incompatibility with closed-source LLMs because block outputs are required) is an applicability constraint, not a circularity. The size mismatch between Vicuna-7B and the smaller CLIP/T5 encoders is a confound for the title-level architectural claim, but confounds are correctness/fairness concerns, not circularity. Overall: partial circularity in the theoretical justification (score 6), while the experimental evaluation stands apart.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method rests on a loose analogy between LLM layers and diffusion steps, several unproven approximations, and learned scaling functions; the empirical results provide independent grounding but the theoretical contribution is largely post hoc.

free parameters (3)
  • g(t) = unspecified
    Scalar function of timestep t in Eq. (5); its form and values are never given, yet it scales the estimated score.
  • h(t) = learned
    Learnable function in the Langevin sampling of Eq. (7); the paper does not specify its parameterization.
  • adapter weights a1, a2, b1, b2 = initialized a1=1, a2=0.1, b1=0, b2=0; trained
    Learnable combination weights for the two cross-attention branches in Eq. (9); they are fit to data.
assumptions (3)
  • ad hoc to paper Transformer blocks of a decoder-only LLM can be viewed as reverse diffusion steps (Eq. 1).
    This analogy is asserted without a formal noise schedule or proof; it underpins the entire theoretical derivation.
  • ad hoc to paper The score of p(c_<d | ...) can be approximated by the difference of sentence-level and word-level score functions (Eq. 5).
    The key approximation is stated, not derived; it directly produces Algorithm 1.
  • domain assumption A text encoding c_<d for a decoder-only LLM exists and is an unbiased estimator of x_<d.
    Postulated in Section 3.4 without proof.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Decoder-Only LLMs are Better Controllers for Diffusion Models." pith.science (2026). https://pith.science/paper/CZJZH3KI

@misc{pith2026250204412,
  author       = {Pith},
  title        = {Pith review of: Decoder-Only LLMs are Better Controllers for Diffusion Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CZJZH3KI}},
  note         = {Machine review of arXiv:2502.04412}
}
read the original abstract

Groundbreaking advancements in text-to-image generation have recently been achieved with the emergence of diffusion models. These models exhibit a remarkable ability to generate highly artistic and intricately detailed images based on textual prompts. However, obtaining desired generation outcomes often necessitates repetitive trials of manipulating text prompts just like casting spells on a magic mirror, and the reason behind that is the limited capability of semantic understanding inherent in current image generation models. Specifically, existing diffusion models encode the text prompt input with a pre-trained encoder structure, which is usually trained on a limited number of image-caption pairs. The state-of-the-art large language models (LLMs) based on the decoder-only structure have shown a powerful semantic understanding capability as their architectures are more suitable for training on very large-scale unlabeled data. In this work, we propose to enhance text-to-image diffusion models by borrowing the strength of semantic understanding from large language models, and devise a simple yet effective adapter to allow the diffusion models to be compatible with the decoder-only structure. Meanwhile, we also provide a supporting theoretical analysis with various architectures (e.g., encoder-only, encoder-decoder, and decoder-only), and conduct extensive empirical evaluations to verify its effectiveness. The experimental results show that the enhanced models with our adapter module are superior to the stat-of-the-art models in terms of text-to-image generation quality and reliability.

Figures

Figures reproduced from arXiv: 2502.04412 by the authors.

Figure 1
Figure 1. Comparison of our LLMDiff with DALL-E 3 [ [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison with other neural network structures employed for computing text encoding in diffusion models. Our [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Our LLMDiff-Adapter framework, wherein the parameters of both the LLM and the diffusion U-Net (including the [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: In comparison with existing approaches, LLMDiff exhibits superior capabilities in both language comprehension and [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Model evaluation on the capability of causal and logical reasoning for text-to-image generation. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The scale factor of newly added attentions and the [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 19 canonical work pages

  1. [1]

    Seungho Baek, Hyerin Im, Jiseung Ryu, Juhyeong Park, and Tak Yeon Lee. 2023. PromptCrafter: Crafting Text-to-Image Prompt through Mixed-Initiative Dia- logue with LLM. CoRR abs/2307.08985 (2023). arXiv:2307.08985

    work page Pith review arXiv 2023

  2. [2]

    James Betker, Gabriel Goh, Li Jing, TimBrooks, Jianfeng Wang, Linjie Li, Lon- gOuyang, JuntangZhuang, JoyceLee, YufeiGuo, WesamManassra, PrafullaDhari- wal, CaseyChu, YunxinJiao, and Aditya Ramesh. [n. d.]. Improving Image Gener- ation with Better Captions. https://cdn.openai.com/papers/dall-e-3.pdf

    work page

  3. [3]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    work page 2020

  4. [4]

    Zhihong Chen, Guiming Chen, Shizhe Diao, Xiang Wan, and Benyou Wang. 2023. On the Difference of BERT-style and CLIP-style Text Encoders. In ACL

    work page 2023

  5. [5]

    Jun Cheng, Fuxiang Wu, Yanling Tian, Lei Wang, and Dapeng Tao. 2020. RiFeGAN: Rich Feature Generation for Text-to-Image Synthesis From Prior Knowledge. In IEEE/CVF Conference on Computer Vision and Pattern Recognition . 10908–10917

    work page 2020

  6. [6]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. https://lmsys.org/blog/2023-03-30-vicuna/

    2023

  7. [7]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bra...

    2023

  8. [8]

    Dan Hendrycks

    et.al. Dan Hendrycks. 2021. Measuring Massive Multitask Language Understand- ing. In ICLR

    work page 2021

Show all 34 references
  1. [9]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT 2019, Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). 4171–4186

    2019

  2. [10]

    Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. 2021. CogView: Mastering Text-to-Image Generation via Transformers. In Advances in Neural Information Processing Systems. 19822–19835

    2021

  3. [11]

    Khan, and Peter Wonka

    Hanan Gani, Shariq Farooq Bhat, Muzammal Naseer, Salman H. Khan, and Peter Wonka. 2023. LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts. CoRR abs/2310.10640 (2023). arXiv:2310.10640

    2023 arXiv

  4. [12]

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising Diffusion Probabilistic Models. In Advances in Neural Information Processing Systems

    2020

  5. [13]

    Jonathan Ho and Tim Salimans. 2021. Classifier-Free Diffusion Guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications

    2021

  6. [14]

    et. al. Hugo Touvron. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. CoRR abs/2307.09288 (2023). arXiv:2307.09288

    2023 arXiv

  7. [15]

    Barron, Pieter Abbeel, and Ben Poole

    Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel, and Ben Poole

  8. [16]

    Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip H. S. Torr. 2019. Con- trollable Text-to-Image Generation. In NeurIPS 2019, Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.)

    2019

  9. [17]

    Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023. Textbooks Are All You Need II: phi-1.5 technical report. CoRR abs/2309.05463 (2023). arXiv:2309.05463

    2023 arXiv

  10. [18]

    Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. 2023. LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models. CoRR abs/2305.13655 (2023). arXiv:2305.13655

    2023 arXiv

  11. [19]

    Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. 2021. More Control for Free! Image Synthesis with Semantic Diffusion Guidance. CoRR abs/2112.05744 (2021). arXiv:2112.05744

    2021 arXiv

  12. [20]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR abs/1907.11692 (2019). arXiv:1907.11692

    2019 arXiv

  13. [21]

    Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. 2022. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. In International Conference on Machine Learnin...

    2022

  14. [22]

    OpenAI. 2023. GPT-4 Technical Report. CoRR abs/2303.08774 (2023). arXiv:2303.08774

    2023 arXiv

  15. [23]

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. 2023. Kosmos-2: Grounding Multimodal Large Language Models to the World. ArXiv abs/2306.14824 (2023)

    2023 arXiv

  16. [24]

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. 2023. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. CoRR abs/2307.01952 (2023). arXiv:2307.01952

    2023 arXiv

  17. [25]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. InProceedings ...

    2021

  18. [26]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.J. Mach. Learn. Res. 21 (2020), 140:1–140:67

    2020

  19. [27]

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-Resolution Image Synthesis with Latent Diffusion Models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition . 10674–10685

    2022

  20. [28]

    Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Den- ton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Moham- mad Norouzi. 2022. Photorealistic Text-to-Image Di...

    2022 arXiv

  21. [29]

    Jianyi Wang, Kelvin C. K. Chan, and Chen Change Loy. 2023. Exploring CLIP for Assessing the Look and Feel of Images. In AAAI 2023,, Brian Williams, Yiling Chen, and Jennifer Neville (Eds.). AAAI Press, 2555–2563

    2023

  22. [30]

    wanng. 2023. midjourney-v5-202304-clean. https://huggingface.co/datasets/ wanng/midjourney-v5-202304-clean

    2023

  23. [31]

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vi- jay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu

  24. [32]

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid Loss for Language Image Pre-Training. CoRR abs/2303.15343 (2023). arXiv:2303.15343

    2023 arXiv

  25. [33]

    CoRR abs/2206.10789 (2022)

    Scaling Autoregressive Models for Content-Rich Text-to-Image Generation. CoRR abs/2206.10789 (2022). arXiv:2206.10789

    2022 arXiv

  26. [2022]

    In IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zero-Shot Text-Guided Object Generation with Dream Fields. In IEEE/CVF Conference on Computer Vision and Pattern Recognition . 857–866

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.