Pith. sign in

REVIEW 4 major objections 4 minor 67 references

A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Averaging all layers of an LLM text encoder, not just the last one, improves text-to-image alignment.

desk verdict Credible controlled study of LLM text encoders, but the headline gains rely on an unvalidated GPT-4o VQAScore; no human eval, so the effect size is unproven. read the letter →

arxiv 2506.08210 v1 pith:UJTTIYFB submitted 2025-06-09 cs.CV cs.AIcs.CLcs.LG

Andrew Z. Wang , Songwei Ge , Tero Karras , Ming-Yu Liu , Yogesh Balaji This is my paper
classification cs.CVcs.AIcs.CLcs.LG
keywords text-to-imagegenerationdecoder-onlylargelanguagemodelstextencoderlayer-normalizedaveragingVQAScorecompositionaldiffusionembedding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether modern decoder-only large language models can replace T5 and CLIP as the text encoders that guide text-to-image diffusion models. The authors train 27 diffusion models that differ only in the text encoder, holding the architecture, data, and training recipe fixed. They report that the standard practice of feeding the last layer's embeddings into the model is suboptimal for LLMs, and instead propose layer-normalized average embeddings across all layers. With this conditioning, most decoder-only LLMs outperform the T5-XXL baseline, with the best model reaching the highest VQAScore in the study. If true, the paper provides a simple recipe: use all layers of the LLM, not just the final one, to get better compositional text-to-image generation.

What carries the argument

The load-bearing mechanism is layer-normalized average embeddings: for each layer of the decoder-only LLM, the per-token embeddings are mean-normalized, then averaged across all layers to form the conditioning input to the diffusion U-Net's cross-attention. This aggregation is what lets the text encoder contribute a balanced mixture of low-level linguistic structure and high-level semantics, rather than the next-token-prediction-skewed representation of the final layer. The paper also uses a fixed Stable Diffusion v2 U-Net with a linear projection to a common 1024-dimensional embedding space, so the only varying component across the 27 trained models is the text encoder and its embedding extraction strategy.

What would settle it

A human evaluation in which users see paired images from T5-XXL last-layer and Mistral-7B norm-avg models on the GenAI-Bench prompts and prefer the T5-XXL outputs at or above chance, or a replication showing that the norm-avg advantage disappears when the original CLIP-FlanT5 VQAScore or a different metric like TIFA is used, would contradict the paper's central ranking.

Watch

Extended reading notes

Core claim

The central discovery is that the de facto last-layer embedding extraction for text encoders does not transfer well to decoder-only LLMs, and that layer-normalized averaging across all layers is a substantially better conditioning signal. In the authors' controlled setup, replacing the last-layer embeddings of Mistral-7B with layer-normalized average embeddings raises its average VQAScore from 0.675 to 0.769, surpassing the T5-XXL baseline of 0.741; the best model, bge-Gemma2 with layer-normalized averaging, reaches 0.789. The paper argues that later layers of decoder-only LLMs specialize in next-token prediction, while middle layers carry more balanced semantic abstractions, so aggregating all layers combines complementary strengths. It also finds that fine-tuned embedding models can help, but do not always, and that scaling LLM size improves performance without uniformly improving all compositional skills.

Load-bearing premise

The paper's rankings assume that GPT-4o-based VQAScore measures text-image alignment accurately enough that a score gap like 0.741 versus 0.789 reflects a real, perceptible quality difference, and no human evaluation is provided to validate this.

Editorial extensions

If this is right

  • Text-to-image systems that currently use last-layer embeddings from T5 or other encoders could improve compositional alignment by switching to layer-normalized average embeddings, without changing the diffusion backbone.
  • Decoder-only LLMs become a viable drop-in replacement for T5 as text encoders, which matters because encoder-decoder model development has slowed relative to LLM scaling.
  • Fine-tuned embedding models, such as bge-Gemma2, are a promising untapped source of text encoders for generation, not just retrieval.
  • Scaling LLM size helps image alignment, but unevenly across skills, suggesting that skill-specific fine-tuning or hybrid encoders may be more compute-efficient than simply scaling up.
  • Evaluation of text-to-image models should consider using a stronger VQA model than the original CLIP-FlanT5, since the original VQAScore fails to differentiate similar models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is whether layer-normalized averaging also helps with other conditioning architectures, such as diffusion transformers that inject text at multiple blocks, since the paper only tests a U-Net with cross-attention.
  • The paper's explanation for why averaging helps—later layers overfit to next-token prediction—could be tested directly by probing which layers carry image-relevant semantics and comparing those probes to the observed per-layer VQAScores.
  • The lack of human evaluation means the reported gaps could be partly an artifact of GPT-4o's scoring; a human preference study on the same 1,600 prompts would settle whether 0.741 versus 0.789 is perceptible.
  • The finding that gte-Qwen2 degrades sharply even with norm-avg suggests that fine-tuning for pooling destroys per-token representations; a hybrid that uses pooled embeddings as an extra conditioning stream may recover some of this loss, as the paper itself hints.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents a controlled empirical study of decoder-only LLMs as text encoders for text-to-image diffusion models. Using a Stable Diffusion v2 U-Net pipeline with fixed training data and recipe, the authors train 27 models with 12 text encoders (T5-XXL, CLIP, several decoder-only LLMs, and fine-tuned embedding models) and compare embedding extraction strategies: last-layer embeddings, individual middle layers, average embeddings, and layer-normalized average embeddings. The main finding is that last-layer LLM embeddings underperform T5, while layer-normalized averaging across all layers improves compositional alignment, with the best model (bge-Gemma2 norm avg) exceeding T5 on the GPT-4o-based VQAScore across all GenAI-Bench aspects. The paper also reports scaling trends and experiments with pooled embeddings. Supplementary material includes results with the original CLIP-FlanT5 VQAScore.

Significance. If the central finding is validated, the paper makes a practically useful contribution: it identifies a simple, inexpensive modification to how LLM embeddings are extracted for conditioning text-to-image models, and it demonstrates through a carefully controlled pipeline that layer-normalized averaging is substantially better than the common last-layer practice. The controlled training setup, the breadth of encoders tested, and the consistent experimental protocol are strengths. However, the main quantitative evidence depends on an unvalidated GPT-4o-based VQAScore, and the supplementary results with the original VQAScore show much smaller gaps that are frequently within the reported noise. The credibility of the central claim therefore hinges on additional validation with human judgments or a meta-evaluation of the modified metric.

major comments (4)
  1. [§3.3, Tables C–E] The central claim in Sec. 6 that layer-normalized average embeddings 'outperform the baseline T5 model across all aspects' rests entirely on the modified GPT-4o VQAScore. The paper states that the original CLIP-FlanT5 VQAScore 'is not sufficient to differentiate between similar models' (§3.3) and asserts that GPT-4o gives 'a closer match to human-perceived quality,' but no human evaluation or meta-evaluation is provided. Supplementary Tables C–E, using the original VQAScore, show much smaller gaps: Mistral-7B norm avg 0.799 vs T5-XXL last-layer 0.795 (Δ=0.004) and bge-Gemma2 norm avg 0.801 vs 0.795 (Δ=0.006), both within the reported ±0.003–0.004 variation (Supp. C.5). The large margins in Table 3 (0.769 vs 0.741 and 0.789 vs 0.741) appear only with GPT-4o. Without human preference judgments or a comparison of GPT-4o VQAScore against human ratings, the 'significant improvements across all aspects' claim is not established.
  2. [§4.3, Tables 3 and E] The claim that the best model outperforms T5 'across all aspects' is contradicted by the original VQAScore on at least one category. In Table E, T5-XXL last-layer scores 0.793 on Differentiation while bge-Gemma2 norm avg scores 0.789, so T5 is higher on that aspect under the original metric. Under the GPT-4o-based metric (Table 3), bge-Gemma2 is higher on Differentiation (0.744 vs 0.675). Since the paper explicitly discards the original metric as insufficient, the discrepancy should be resolved with human evaluation rather than simply relying on the modified metric.
  3. [§3.3, Supp. B] The evaluation prompts are upsampled with Gemma2-9B (Supp. B), which is also one of the tested text encoders. No results on the original GenAI-Bench prompts are reported, so the interaction between the upsampler and the tested encoders is uncontrolled. This could systematically favor decoder-only LLM encoders, since the upsampled text distribution is generated by a decoder-only LLM and may be out-of-distribution for T5 and CLIP. Reporting results on both original and upsampled prompts would isolate the effect of the upsampler on the ranking.
  4. [§3.1, §3.3] Each of the 27 models is trained once (Sec. 3.1), and the reported ±0.003–0.004 variation covers only evaluation randomness (Supp. C.5). The paper does not quantify training stochasticity. For the central comparison (e.g., Mistral-7B norm avg vs T5-XXL last layer, Table E gaps of 0.004), a single training run cannot support a claim of significant improvement beyond the evaluation noise. At minimum, the paper should state that training variance is unquantified; ideally it should report repeated runs for the key comparisons.
minor comments (4)
  1. [Table A] In the 'Text Encoders' column, 'gte-Gwen2' appears to be a typo for 'gte-Qwen2'.
  2. [References] References [51] and [52] are the same paper (Gemma 2 technical report) with different page numbers; they should be merged or renumbered.
  3. [Figure 1] The caption says 'VQA scores' but should specify that these are the GPT-4o-based VQAScore values from Table 3, to avoid ambiguity with the original VQAScore reported in the supplementary tables.
  4. [§3.3] The GPT-4o VQAScore implementation is described only briefly ('replicating the code and swapping in the GPT-4o API'). The prompt template, image input format, and answer-parsing logic should be detailed in the supplementary material to enable reproducibility.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; the central ranking is an empirical comparison on a shared benchmark, not an equation-level derivation.

full rationale

This is an empirical study, not a derivation. The central claim—that layer-normalized averaging of decoder-only LLM embeddings outperforms last-layer T5 conditioning—is not equivalent to any input by construction. All models are trained with the same U-Net, same LAION-Aesthetics subset, same VFC-upsampled captions, and are evaluated on the same 1,600 GenAI-Bench prompts, so the norm-avg versus last-layer comparison is isolated to the text-encoder representation. The GPT-4o VQAScore is an external oracle; whether it is a valid human proxy is a measurement-validity concern, not circularity. Importantly, the supplementary CLIP-FlanT5 VQAScore tables show the same qualitative direction (e.g., Mistral norm-avg 0.799 vs T5-XXL 0.795; bge-Gemma2 norm-avg 0.801 vs T5-XXL 0.795), so the result is not solely an artifact of the chosen metric. The paper's self-citations (VFC caption upsampling, eDiff-I) are shared across all conditions and are not load-bearing; using Gemma2-9B to upsample evaluation prompts is a possible distributional confound, but all models receive identical prompts, so it does not make the LLM result true by definition.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central method has no fitted parameters; the only hand-chosen evaluation setting is the CFG scale. The paper relies on several domain assumptions about the validity of its evaluation and the isolation of the text encoder effect. No invented entities are introduced.

free parameters (1)
  • Classifier-free guidance scale = 7.0
    Chosen by hand as a standardized evaluation setting across all models; Figure 3 shows score depends on this value, so the ranking could shift with a different choice.
assumptions (4)
  • domain assumption Freezing the text encoder and training only the U-Net, with a linear projection, isolates the effect of the text encoder.
    Invoked in Sec 3.1; if sequence length or dimension interactions affect training dynamics, differences may not be purely attributable to the encoder.
  • domain assumption VQAScore with GPT-4o is a valid and sufficiently discriminative measure of text-image alignment.
    Invoked in Sec 3.3; the original VQAScore CLIP-FlanT5 model was found insufficient, and no human evaluation is provided.
  • domain assumption GenAI-Bench and its skill annotations fairly represent compositional text-to-image challenges.
    Invoked in Sec 3.3; the conclusions about 'advanced visio-linguistic reasoning' are drawn from this single benchmark.
  • domain assumption Layer-normalized averaging of all layers is a sensible aggregation with no learned weights.
    The paper motivates this via LLM interpretability but does not provide a theoretical guarantee; the choice is validated only empirically on a subset of models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation." pith.science (2026). https://pith.science/paper/UJTTIYFB

@misc{pith2026250608210,
  author       = {Pith},
  title        = {Pith review of: A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UJTTIYFB}},
  note         = {Machine review of arXiv:2506.08210}
}
read the original abstract

Both text-to-image generation and large language models (LLMs) have made significant advancements. However, many text-to-image models still employ the somewhat outdated T5 and CLIP as their text encoders. In this work, we investigate the effectiveness of using modern decoder-only LLMs as text encoders for text-to-image diffusion models. We build a standardized training and evaluation pipeline that allows us to isolate and evaluate the effect of different text embeddings. We train a total of 27 text-to-image models with 12 different text encoders to analyze the critical aspects of LLMs that could impact text-to-image generation, including the approaches to extract embeddings, different LLMs variants, and model sizes. Our experiments reveal that the de facto way of using last-layer embeddings as conditioning leads to inferior performance. Instead, we explore embeddings from various layers and find that using layer-normalized averaging across all layers significantly improves alignment with complex prompts. Most LLMs with this conditioning outperform the baseline T5 model, showing enhanced performance in advanced visio-linguistic reasoning skills.

Figures

Figures reproduced from arXiv: 2506.08210 by the authors.

Figure 1
Figure 1. VQA scores for text-to-image models using T5-XXL [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Visual comparison of images generated with different text encoders. We use last-layer embeddings ( [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. VQAScore as a function of classifier-free guidance [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Heatmap visualization. Images generated with Mistral-7B using standard last-layer embeddings ( [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

67 extracted references · 30 canonical work pages

  1. [1]

    Edify image: High-quality im- age generation with pixel space laplacian diffusion models

    Yuval Atzmon, Maciej Bala, Yogesh Balaji, Tiffany Cai, Yin Cui, Jiaojiao Fan, Yunhao Ge, Siddharth Gururani, Jacob Huffman, Ronald Isaac, et al. Edify image: High-quality im- age generation with pixel space laplacian diffusion models. arXiv preprint arXiv:2411.07126, 2024. 1

    arXiv 2024

  2. [2]

    ediff-i: Text-to-image dif- fusion models with an ensemble of expert denoisers.arXiv preprint arXiv:2211.01324, 2022

    Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Ji- aming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, et al. ediff-i: Text-to-image dif- fusion models with an ensemble of expert denoisers.arXiv preprint arXiv:2211.01324, 2022. 1, 2, 8

    arXiv 2022

  3. [3]

    Imagen 3.arXiv preprint arXiv:2408.07009, 2024

    Jason Baldridge, Jakob Bauer, Mukul Bhutani, Nicole Brichtova, Andrew Bunner, Lluis Castrejon, Kelvin Chan, Yichang Chen, Sander Dieleman, Yuqing Du, et al. Imagen 3.arXiv preprint arXiv:2408.07009, 2024. 4, 5

    arXiv 2024

  4. [4]

    Llm2vec: Large language models are secretly pow- erful text encoders.arXiv preprint arXiv:2404.05961, 2024

    Parishad BehnamGhader, Vaibhav Adlakha, Marius Mos- bach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. Llm2vec: Large language models are secretly pow- erful text encoders.arXiv preprint arXiv:2404.05961, 2024. 2, 6

    arXiv 2024

  5. [5]

    Scalable Performance Analysis for Vision-Language Models

    Santiago Castro, Oana Ignat, and Rada Mihalcea. Scal- able performance analysis for vision-language models.arXiv preprint arXiv:2305.18786, 2023. 5

    work page Pith review arXiv 2023

  6. [6]

    PixArt-alpha: Fast Training of Diffu- sion Transformer for Photorealistic Text-to-Image Synthesis

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. PixArt-alpha: Fast Training of Diffu- sion Transformer for Photorealistic Text-to-Image Synthesis. arXiv preprint arXiv:2310.00426, 2023. 1

    arXiv

  7. [7]

    Textdiffuser-2: Unleashing the power of language models for text rendering

    Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei. Textdiffuser-2: Unleashing the power of language models for text rendering. InECCV, 2024. 2

    work page 2024

  8. [8]

    Bge m3-embedding: Multi- lingual, multi-functionality, multi-granularity text embed- dings through self-knowledge distillation.arXiv preprint arXiv:2402.03216, 2024

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi- lingual, multi-functionality, multi-granularity text embed- dings through self-knowledge distillation.arXiv preprint arXiv:2402.03216, 2024. 2, 4

    arXiv 2024

Show all 67 references
  1. [9]

    Visual pro- gramming for step-by-step text-to-image generation and evaluation

    Jaemin Cho, Abhay Zala, and Mohit Bansal. Visual pro- gramming for step-by-step text-to-image generation and evaluation. InNeurIPS, 2023. 4

    2023

  2. [10]

    Flashattention: Fast and memory-efficient exact attention with io-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christo- pher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. InNeurIPS, 2022. 1

    2022

  3. [11]

    An- alyzing transformers in embedding space.arXiv preprint arXiv:2209.02535, 2022

    Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. An- alyzing transformers in embedding space.arXiv preprint arXiv:2209.02535, 2022. 2, 6

    2022 arXiv

  4. [12]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. InICML, 2024. 1

    2024

  5. [13]

    Lumina-t2x: Transforming text into any modality, reso- lution, and duration via flow-based large diffusion transform- ers.arXiv preprint arXiv:2405.05945, 2024

    Peng Gao, Le Zhuo, Ziyi Lin, Chris Liu, Junsong Chen, Ruoyi Du, Enze Xie, Xu Luo, Longtian Qiu, Yuhang Zhang, et al. Lumina-t2x: Transforming text into any modality, reso- lution, and duration via flow-based large diffusion transform- ers.arXiv preprint arXiv:2405.05945, 2024. 2, 5

    2024 arXiv

  6. [14]

    Visual fact checker: en- abling high-fidelity detailed caption generation

    Yunhao Ge, Xiaohui Zeng, Jacob Samuel Huffman, Tsung- Yi Lin, Ming-Yu Liu, and Yin Cui. Visual fact checker: en- abling high-fidelity detailed caption generation. InCVPR,

  7. [15]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Ab- hinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024. 2, 4

    2024 arXiv

  8. [16]

    Mars: Mixture of auto-regressive mod- els for fine-grained text-to-image synthesis.arXiv preprint arXiv:2407.07614, 2024

    Wanggui He, Siming Fu, Mushui Liu, Xierui Wang, Wenyi Xiao, Fangxun Shu, Yi Wang, Lei Zhang, Zhelun Yu, Haoyuan Li, et al. Mars: Mixture of auto-regressive mod- els for fine-grained text-to-image synthesis.arXiv preprint arXiv:2407.07614, 2024. 2

    2024 arXiv

  9. [17]

    Clipscore: A reference-free evaluation met- ric for image captioning.arXiv preprint arXiv:2104.08718,

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation met- ric for image captioning.arXiv preprint arXiv:2104.08718,

    arXiv

  10. [18]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. InNeurIPS, 2017. 4

    2017

  11. [19]

    Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022. 5

    2022 arXiv

  12. [20]

    Ella: Equip diffusion models with llm for enhanced semantic alignment.arXiv preprint arXiv:2403.05135, 2024

    Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with llm for enhanced semantic alignment.arXiv preprint arXiv:2403.05135, 2024. 2

    2024 arXiv

  13. [21]

    Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering

    Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Os- tendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. InICCV, 2023. 4

    2023

  14. [22]

    T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation

    Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xi- hui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. In NeurIPS, 2023. 4

    2023

  15. [23]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 5, 2

    2024 arXiv

  16. [24]

    What does bert learn about the structure of language? InACL,

    Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does bert learn about the structure of language? InACL,

  17. [25]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lam- ple, Lucile Saulnier, et al. Mistral 7B.arXiv preprint arXiv:2310.06825, 2023. 4, 6

    arXiv

  18. [26]

    Analyzing the role of semantic representations in the era of large language models.arXiv preprint arXiv:2405.01502,

    Zhijing Jin, Yuen Chen, Fernando Gonzalez, Jiarui Liu, Jiayi Zhang, Julian Michael, Bernhard Schölkopf, and Mona Diab. Analyzing the role of semantic representations in the era of large language models.arXiv preprint arXiv:2405.01502,

    arXiv

  19. [27]

    Elucidating the design space of diffusion-based generative models

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. InNeurIPS, 2022. 1

    2022

  20. [28]

    Analyzing and improving the training dynamics of diffusion models

    Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. InCVPR, 2024. 1

    2024

  21. [29]

    Nv- embed: Improved techniques for training llms as generalist embedding models.arXiv preprint arXiv:2405.17428, 2024

    Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv- embed: Improved techniques for training llms as generalist embedding models.arXiv preprint arXiv:2405.17428, 2024. 7

    2024 arXiv

  22. [30]

    Genai-bench: Evaluating and improv- ing compositional text-to-visual generation.arXiv preprint arXiv:2406.13743, 2024

    Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Gra- ham Neubig, et al. Genai-bench: Evaluating and improv- ing compositional text-to-visual generation.arXiv preprint arXiv:2406.13743, 2024. 2, 4

    2024 arXiv

  23. [31]

    Towards general text embeddings with multi-stage contrastive learning.arXiv preprint arXiv:2308.03281, 2023

    Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning.arXiv preprint arXiv:2308.03281, 2023. 2, 4

    2023 arXiv

  24. [32]

    Llm- grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models

    Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. Llm- grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. InTLMR, 2024. 2

    2024

  25. [33]

    Common diffusion noise schedules and sample steps are flawed

    Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. InWACV, 2024. 5

    2024

  26. [34]

    Evaluating text-to-visual generation with image-to-text gen- eration

    Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text gen- eration. InECCV, 2024. 4, 5, 1, 2

    2024

  27. [35]

    Playground v3: Im- proving text-to-image alignment with deep-fusion large lan- guage models.arXiv preprint arXiv:2409.10695, 2024

    Bingchen Liu, Ehsan Akhgari, Alexander Visheratin, Aleks Kamko, Linmiao Xu, Shivam Shrirao, Chase Lambert, Joao Souza, Suhail Doshi, and Daiqing Li. Playground v3: Im- proving text-to-image alignment with deep-fusion large lan- guage models.arXiv preprint arXiv:2409.10695, 2024. 1, 2

    2024 arXiv

  28. [36]

    Character-aware models improve visual text rendering.arXiv preprint arXiv:2212.10562, 2022

    Rosanne Liu, Dan Garrette, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, RJ Mical, Mo- hammad Norouzi, and Noah Constant. Character-aware models improve visual text rendering.arXiv preprint arXiv:2212.10562, 2022. 2

    2022 arXiv

  29. [37]

    Fan- tastic semantics and where to find them: Investigating which layers of generative llms reflect lexical semantics.arXiv preprint arXiv:2403.01509, 2024

    Zhu Liu, Cunliang Kong, Ying Liu, and Maosong Sun. Fan- tastic semantics and where to find them: Investigating which layers of generative llms reflect lexical semantics.arXiv preprint arXiv:2403.01509, 2024. 2, 6

    2024 arXiv

  30. [38]

    Decoupled weight decay regularization, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2017. 1

    2017

  31. [39]

    Salesforce AI Research’s SFR-embedding, the top performing text-embedding model

    Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Salesforce AI Research’s SFR-embedding, the top performing text-embedding model. Salesforce AI Research Blog, 2024. 4

    2024

  32. [40]

    Mteb: Massive text embedding benchmark.arXiv preprint arXiv:2210.07316, 2022

    Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. Mteb: Massive text embedding benchmark.arXiv preprint arXiv:2210.07316, 2022. 1, 4

    2022 arXiv

  33. [41]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InICCV, 2023. 1

    2023

  34. [42]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023. 1, 6, 7

    2023 arXiv

  35. [43]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. InICML, 2021. 1, 2, 3, 5

    2021

  36. [44]

    Exploring the limits of transfer learning with a unified text-to-text transformer.JMLR, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.JMLR, 2020. 1, 2, 3

    2020

  37. [45]

    Hierarchical text-conditional image gen- eration with clip latents.arXiv preprint arXiv:2204.06125,

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gen- eration with clip latents.arXiv preprint arXiv:2204.06125,

    arXiv

  38. [46]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image syn- thesis with latent diffusion models. InCVPR, 2022. 2, 3, 5, 1

    2022

  39. [47]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. InMICCAI, 2015. 3, 1

    2015

  40. [48]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. InNeurIPS, 2022. 1, 2, 5, 7

    2022

  41. [49]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. InNeurIPS, 2022. 3, 1

    2022

  42. [50]

    Repetition improves lan- guage model embeddings.arXiv preprint arXiv:2402.15449,

    Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, and Aditi Raghunathan. Repetition improves lan- guage model embeddings.arXiv preprint arXiv:2402.15449,

    arXiv

  43. [52]

    Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118,

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupati- raju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118,

    arXiv

  44. [53]

    Stable diffusion training with mo- saicml.Mosaic Research Blog, 2023

    The Mosaic ML Team. Stable diffusion training with mo- saicml.Mosaic Research Blog, 2023. 1

    2023

  45. [54]

    What do you learn from context? probing for sentence struc- ture in contextualized word representations.arXiv preprint arXiv:1905.06316, 2019

    Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. What do you learn from context? probing for sentence struc- ture in contextualized word representations.arXiv preprint arXiv:190...

    1905 arXiv

  46. [55]

    Winoground: Probing vision and language models for visio- linguistic compositionality

    Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio- linguistic compositionality. InCVPR, 2022. 5

    2022

  47. [56]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 2

    2023 arXiv

  48. [57]

    Diffusers: State-of-the-art diffusion models.GitHub repository, 2022

    Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffusion models.GitHub repository, 2022. 1

    2022

  49. [58]

    Text embeddings by weakly-supervised contrastive pre- training.arXiv preprint arXiv:2212.03533, 2022

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre- training.arXiv preprint arXiv:2212.03533, 2022. 2

    2022 arXiv

  50. [59]

    Improving text embeddings with large language models

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Ran- gan Majumder, and Furu Wei. Improving text embeddings with large language models. InACL, 2024. 2, 6

    2024

  51. [60]

    Revisiting text-to-image evaluation with gecko: On met- rics, prompts, and human ratings.arXiv preprint arXiv:2404.16820, 2024

    Olivia Wiles, Chuhan Zhang, Isabela Albuquerque, Ivana Kaji´c, Su Wang, Emanuele Bugliarello, Yasumasa Onoe, Chris Knutsen, Cyrus Rashtchian, Jordi Pont-Tuset, et al. Revisiting text-to-image evaluation with gecko: On met- rics, prompts, and human ratings.arXiv preprint arXiv:...

    2024 arXiv

  52. [61]

    Sana: Ef- ficient high-resolution image synthesis with linear diffusion transformers.arXiv preprint arXiv:2410.10629, 2024

    Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Yujun Lin, Zhekai Zhang, Muyang Li, Yao Lu, and Song Han. Sana: Ef- ficient high-resolution image synthesis with linear diffusion transformers.arXiv preprint arXiv:2410.10629, 2024. 2, 5

    arXiv

  53. [62]

    ByT5: Towards a token-free future with pre-trained byte- to-byte models.TACL, 2022

    Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raf- fel. ByT5: Towards a token-free future with pre-trained byte- to-byte models.TACL, 2022. 2

    2022

  54. [63]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024. 4

    2024 arXiv

  55. [64]

    What you see is what you read? improving text- image alignment evaluation

    Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor. What you see is what you read? improving text- image alignment evaluation. InNeurIPS, 2023. 4

    2023

  56. [65]

    Inves- tigating layer importance in large language models.arXiv preprint arXiv:2409.14381, 2024

    Yang Zhang, Yanfei Dong, and Kenji Kawaguchi. Inves- tigating layer importance in large language models.arXiv preprint arXiv:2409.14381, 2024. 2, 6

    2024 arXiv

  57. [66]

    Pytorch fsdp: experi- ences on scaling fully sharded data parallel.arXiv preprint arXiv:2304.11277, 2023

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien- Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experi- ences on scaling fully sharded data parallel.arXiv preprint arXiv:2304.11277, 2023. 1

    2023 arXiv

  58. [67]

    Lumina-next: Making lumina-t2x stronger and faster with next-dit.arXiv preprint arXiv:2406.18583,

    Le Zhuo, Ruoyi Du, Han Xiao, Yangguang Li, Dongyang Liu, Rongjie Huang, Wenze Liu, Lirui Zhao, Fu-Yun Wang, Zhanyu Ma, et al. Lumina-next: Making lumina-t2x stronger and faster with next-dit.arXiv preprint arXiv:2406.18583,

    arXiv

  59. [2024]

    a beautiful morning in the woods with the sun peaking through the trees

    2, 5 A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation Supplementary Material A. Training details We follow the U-Net [47] based latent diffusion architecture from Stable Diffusion v2 [46] with a replication training framework by MosaicML [53]. We use Dif...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.