{"id":"1e2d9053-7e9f-4d44-b6f5-0fd6772b2ef2","arxiv_id":"2506.08210","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Layer-normalized averaging of all decoder-only LLM hidden states, rather than last-layer embeddings, improves text-to-image compositional alignment and beats T5 on GenAI-Bench.","lead":"This paper trains 27 text-to-image models with 12 different text encoders to isolate how text embeddings affect generation quality. It finds that averaging normalized embeddings from all LLM layers significantly improves compositional prompt alignment, outperforming the standard T5 baseline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim hinges on an unvalidated GPT-4o-based VQAScore; the supplementary original CLIP-FlanT5 VQAScore shows much smaller gaps (≈0.004–0.006), so 'significant improvement' needs a human preference check before it is accepted.","rationale":"Reader and I agree. The study is well-controlled: same U-Net, same data, same training recipe, 27 models; the within-model contrast (last-layer vs norm-avg for Mistral and bge-Gemma2) is a clean ablation and the strongest evidence that the aggregation method matters. The descriptive visual examples and cross-attention maps are supportive but anecdotal. However, the headline claim about 'outperforming T5 across all aspects' is a comparative claim whose quantitative foundation is the GPT-4o VQAScore. The supplementary inclusion of original CLIP-FlanT5 VQAScore is honest and shows correlated trends, but it also reveals how much of the reported effect depends on the chosen metric: under the original metric the differences are 0.004–0.006, essentially within noise. Therefore the decisive missing piece is human validation of the GPT-4o-based score on this exact model set. If human pairwise preference confirms the ordering and effect size, the paper's central claim is credible and practically useful; if not, the conclusion overreaches. This does not change the reader's CONDITIONAL verdict—the request for human evaluation is already there—but it sharpens why that request is load-bearing rather than a routine reproducibility ask.","tokens_in":19805,"tokens_out":5293,"duration_ms":63092,"concrete_test":"Run a human pairwise preference study on 200 prompts stratified across GenAI-Bench categories, comparing T5-XXL last-layer vs Mistral-7B norm-avg and bge-Gemma2 norm-avg, using the same upsampled prompts and seeds. If the human preference rate for the norm-avg models is not consistent with the GPT-4o VQAScore gaps (0.741→0.769/0.789)—for example, if humans do not significantly prefer them—the central claim is unsupported. This single check would settle whether the metric substitution, rather than a perceptible quality gain, drives the reported improvements.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that decoder-only LLM embeddings with layer-normalized averaging outperform T5-XXL across all GenAI-Bench aspects. All quantitative support for this claim comes from a modified VQAScore that replaces the original CLIP-FlanT5 VQA model with GPT-4o. The paper itself states that the original metric is 'not sufficient to differentiate between similar models' (§3.3), then asserts that GPT-4o gives 'a closer match to human-perceived quality' without providing human judgments or a meta-evaluation. This is not merely a cosmetic issue: the supplementary Tables C–E, using the original CLIP-FlanT5 VQAScore, show Mistral-7B norm-avg at 0.799 vs T5-XXL last-layer at 0.795 (Δ=0.004) and bge-Gemma2 norm-avg at 0.801 vs 0.795 (Δ=0.006), both within the reported ±0.003–0.004 noise. The large gaps in the main text (0.741→0.769 and 0.741→0.789) therefore arise from the GPT-4o variant alone. If GPT-4o's scores are biased in any way correlated with LLM conditioning—for example, preferring more visually ornate outputs that detailed LLM embeddings tend to produce—the ranking underlying the conclusion would not reflect human-perceived alignment. The evaluation is also run on prompts upsampled by Gemma2-9B, an LLM, which further couples the benchmark distribution to the tested encoder family; no comparison on the original GenAI-Bench prompts is reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a controlled empirical study of decoder-only LLMs as text encoders for text-to-image diffusion models. Using a Stable Diffusion v2 U-Net pipeline with fixed training data and recipe, the authors train 27 models with 12 text encoders (T5-XXL, CLIP, several decoder-only LLMs, and fine-tuned embedding models) and compare embedding extraction strategies: last-layer embeddings, individual middle layers, average embeddings, and layer-normalized average embeddings. The main finding is that last-layer LLM embeddings underperform T5, while layer-normalized averaging across all layers improves compositional alignment, with the best model (bge-Gemma2 norm avg) exceeding T5 on the GPT-4o-based VQAScore across all GenAI-Bench aspects. The paper also reports scaling trends and experiments with pooled embeddings. Supplementary material includes results with the original CLIP-FlanT5 VQAScore.","tokens_in":20172,"tokens_out":4310,"duration_ms":46510,"significance":"If the central finding is validated, the paper makes a practically useful contribution: it identifies a simple, inexpensive modification to how LLM embeddings are extracted for conditioning text-to-image models, and it demonstrates through a carefully controlled pipeline that layer-normalized averaging is substantially better than the common last-layer practice. The controlled training setup, the breadth of encoders tested, and the consistent experimental protocol are strengths. However, the main quantitative evidence depends on an unvalidated GPT-4o-based VQAScore, and the supplementary results with the original VQAScore show much smaller gaps that are frequently within the reported noise. The credibility of the central claim therefore hinges on additional validation with human judgments or a meta-evaluation of the modified metric.","major_comments":[{"comment":"The central claim in Sec. 6 that layer-normalized average embeddings 'outperform the baseline T5 model across all aspects' rests entirely on the modified GPT-4o VQAScore. The paper states that the original CLIP-FlanT5 VQAScore 'is not sufficient to differentiate between similar models' (§3.3) and asserts that GPT-4o gives 'a closer match to human-perceived quality,' but no human evaluation or meta-evaluation is provided. Supplementary Tables C–E, using the original VQAScore, show much smaller gaps: Mistral-7B norm avg 0.799 vs T5-XXL last-layer 0.795 (Δ=0.004) and bge-Gemma2 norm avg 0.801 vs 0.795 (Δ=0.006), both within the reported ±0.003–0.004 variation (Supp. C.5). The large margins in Table 3 (0.769 vs 0.741 and 0.789 vs 0.741) appear only with GPT-4o. Without human preference judgments or a comparison of GPT-4o VQAScore against human ratings, the 'significant improvements across all aspects' claim is not established.","section":"§3.3, Tables C–E"},{"comment":"The claim that the best model outperforms T5 'across all aspects' is contradicted by the original VQAScore on at least one category. In Table E, T5-XXL last-layer scores 0.793 on Differentiation while bge-Gemma2 norm avg scores 0.789, so T5 is higher on that aspect under the original metric. Under the GPT-4o-based metric (Table 3), bge-Gemma2 is higher on Differentiation (0.744 vs 0.675). Since the paper explicitly discards the original metric as insufficient, the discrepancy should be resolved with human evaluation rather than simply relying on the modified metric.","section":"§4.3, Tables 3 and E"},{"comment":"The evaluation prompts are upsampled with Gemma2-9B (Supp. B), which is also one of the tested text encoders. No results on the original GenAI-Bench prompts are reported, so the interaction between the upsampler and the tested encoders is uncontrolled. This could systematically favor decoder-only LLM encoders, since the upsampled text distribution is generated by a decoder-only LLM and may be out-of-distribution for T5 and CLIP. Reporting results on both original and upsampled prompts would isolate the effect of the upsampler on the ranking.","section":"§3.3, Supp. B"},{"comment":"Each of the 27 models is trained once (Sec. 3.1), and the reported ±0.003–0.004 variation covers only evaluation randomness (Supp. C.5). The paper does not quantify training stochasticity. For the central comparison (e.g., Mistral-7B norm avg vs T5-XXL last layer, Table E gaps of 0.004), a single training run cannot support a claim of significant improvement beyond the evaluation noise. At minimum, the paper should state that training variance is unquantified; ideally it should report repeated runs for the key comparisons.","section":"§3.1, §3.3"}],"minor_comments":[{"comment":"In the 'Text Encoders' column, 'gte-Gwen2' appears to be a typo for 'gte-Qwen2'.","section":"Table A"},{"comment":"References [51] and [52] are the same paper (Gemma 2 technical report) with different page numbers; they should be merged or renumbered.","section":"References"},{"comment":"The caption says 'VQA scores' but should specify that these are the GPT-4o-based VQAScore values from Table 3, to avoid ambiguity with the original VQAScore reported in the supplementary tables.","section":"Figure 1"},{"comment":"The GPT-4o VQAScore implementation is described only briefly ('replicating the code and swapping in the GPT-4o API'). The prompt template, image input format, and answer-parsing logic should be detailed in the supplementary material to enable reproducibility.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The controlled pipeline and breadth of experiments are genuinely valuable, and the proposed layer-normalized averaging is a plausible and simple improvement. However, the central quantitative claim depends on a modified metric that the paper itself identifies as a divergence from the original, and no human validation is supplied. The supplementary original-metric results substantially weaken the claim as written. The issues are fixable: add a human preference study (even on a subset), report results on original GenAI-Bench prompts without upsampling, and discuss training stochasticity. If the authors can provide such evidence, the paper could be suitable for publication; without it, the headline claim is not yet supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on Wang et al. The paper is a large, carefully controlled comparison of text encoders for text-to-image diffusion, and it makes a concrete finding: if you take a decoder-only LLM, normalize each layer's embeddings and average them, you get better compositional alignment than T5's last layer. The pipeline is a real strength—same U-Net, same training recipe, only the encoder changes. Training 27 models that way is a serious effort, and the layer-wise analysis (Table 2) is a nice addition.\n\nThe main caveat is the evaluation. The headline numbers come from a GPT-4o-based VQAScore that the authors modified because the original CLIP-FlanT5 metric 'is not sufficient to differentiate between similar models.' They claim GPT-4o gives a closer match to human-perceived quality, but no human judgments are reported. The supplementary tables C–E are worrying in that respect: on the original metric, Mistral norm-avg scores 0.799 vs T5 last-layer 0.795 (Δ=0.004), and bge-Gemma2 norm-avg 0.801 vs 0.795 (Δ=0.006). Those gaps are within the reported ±0.003–0.004 noise. The large gaps in the main text (0.741→0.769 and 0.741→0.789) appear only when you switch to GPT-4o. Without a human preference study, I can't tell whether the 'significant improvement' is real or an artifact of the GPT-4o scoring. That's not a minor point—it's the load-bearing evidence for the paper's central claim. No repeated training runs either, so stochastic differences are unquantified.\n\nThere's also a more subtle confound: the evaluation uses prompts upsampled by Gemma2-9B, which is itself one of the tested encoders. That couples the benchmark distribution to a member of the family under study. They don't report results on the original GenAI-Bench prompts, so the effect size on non-upsampled text is unknown. I'd like to see that.\n\nThese are fixable. The authors could run a human preference comparison on a subset of the 1,600 prompts, report original-prompt scores, and release code and model checkpoints. The lack of code/data is another limitation, especially for a study whose value is a 27-model comparison.\n\nFor whom is this paper? Practitioners building text-to-image models who are choosing an encoder, and researchers studying how to extract features from LLMs. The finding that last-layer conditioning is suboptimal for LLMs is plausible and likely to influence practice, but I'd be cautious about adopting it before the human eval exists. The paper deserves a serious referee. I'd accept it for peer review, with conditionality on the human evaluation and artifacts.\n\nRecommendation: send it to review, flag the metric validation and the Gemma2 upsampling confound as major concerns, and ask for code, data, and a human preference test.","headline":"Credible controlled study of LLM text encoders, but the headline gains rely on an unvalidated GPT-4o VQAScore; no human eval, so the effect size is unproven.","tokens_in":20680,"tokens_out":4054,"would_cite":true,"duration_ms":42065,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Averaging all layers of an LLM text encoder, not just the last one, improves text-to-image alignment.","keywords":["text-to-image generation","decoder-only large language models","text encoder","layer-normalized averaging","VQAScore","compositional generation","diffusion models","embedding models"],"falsifier":"A human evaluation in which users see paired images from T5-XXL last-layer and Mistral-7B norm-avg models on the GenAI-Bench prompts and prefer the T5-XXL outputs at or above chance, or a replication showing that the norm-avg advantage disappears when the original CLIP-FlanT5 VQAScore or a different metric like TIFA is used, would contradict the paper's central ranking.","tokens_in":19632,"feed_emoji":"🖼️","tokens_out":4382,"duration_ms":40372,"temperature":0.7,"pith_summary":"This paper asks whether modern decoder-only large language models can replace T5 and CLIP as the text encoders that guide text-to-image diffusion models. The authors train 27 diffusion models that differ only in the text encoder, holding the architecture, data, and training recipe fixed. They report that the standard practice of feeding the last layer's embeddings into the model is suboptimal for LLMs, and instead propose layer-normalized average embeddings across all layers. With this conditioning, most decoder-only LLMs outperform the T5-XXL baseline, with the best model reaching the highest VQAScore in the study. If true, the paper provides a simple recipe: use all layers of the LLM, not just the final one, to get better compositional text-to-image generation.","feed_headline":"Averaging all LLM layers beats the last-layer trick for text-to-image","feed_subtitle":"A controlled study of 27 models finds that layer-normalized averaging makes decoder-only LLMs outperform T5 on compositional prompts.","key_machinery":"The load-bearing mechanism is layer-normalized average embeddings: for each layer of the decoder-only LLM, the per-token embeddings are mean-normalized, then averaged across all layers to form the conditioning input to the diffusion U-Net's cross-attention. This aggregation is what lets the text encoder contribute a balanced mixture of low-level linguistic structure and high-level semantics, rather than the next-token-prediction-skewed representation of the final layer. The paper also uses a fixed Stable Diffusion v2 U-Net with a linear projection to a common 1024-dimensional embedding space, so the only varying component across the 27 trained models is the text encoder and its embedding extraction strategy.","core_discovery":"The central discovery is that the de facto last-layer embedding extraction for text encoders does not transfer well to decoder-only LLMs, and that layer-normalized averaging across all layers is a substantially better conditioning signal. In the authors' controlled setup, replacing the last-layer embeddings of Mistral-7B with layer-normalized average embeddings raises its average VQAScore from 0.675 to 0.769, surpassing the T5-XXL baseline of 0.741; the best model, bge-Gemma2 with layer-normalized averaging, reaches 0.789. The paper argues that later layers of decoder-only LLMs specialize in next-token prediction, while middle layers carry more balanced semantic abstractions, so aggregating all layers combines complementary strengths. It also finds that fine-tuned embedding models can help, but do not always, and that scaling LLM size improves performance without uniformly improving all compositional skills.","pith_inferences":["A natural next test is whether layer-normalized averaging also helps with other conditioning architectures, such as diffusion transformers that inject text at multiple blocks, since the paper only tests a U-Net with cross-attention.","The paper's explanation for why averaging helps—later layers overfit to next-token prediction—could be tested directly by probing which layers carry image-relevant semantics and comparing those probes to the observed per-layer VQAScores.","The lack of human evaluation means the reported gaps could be partly an artifact of GPT-4o's scoring; a human preference study on the same 1,600 prompts would settle whether 0.741 versus 0.789 is perceptible.","The finding that gte-Qwen2 degrades sharply even with norm-avg suggests that fine-tuning for pooling destroys per-token representations; a hybrid that uses pooled embeddings as an extra conditioning stream may recover some of this loss, as the paper itself hints."],"forward_implications":["Text-to-image systems that currently use last-layer embeddings from T5 or other encoders could improve compositional alignment by switching to layer-normalized average embeddings, without changing the diffusion backbone.","Decoder-only LLMs become a viable drop-in replacement for T5 as text encoders, which matters because encoder-decoder model development has slowed relative to LLM scaling.","Fine-tuned embedding models, such as bge-Gemma2, are a promising untapped source of text encoders for generation, not just retrieval.","Scaling LLM size helps image alignment, but unevenly across skills, suggesting that skill-specific fine-tuning or hybrid encoders may be more compute-efficient than simply scaling up.","Evaluation of text-to-image models should consider using a stronger VQA model than the original CLIP-FlanT5, since the original VQAScore fails to differentiate similar models."],"supporting_citations":[{"why":"T5-XXL is the baseline text encoder that the paper compares all LLMs against.","marker":"[44]"},{"why":"Imagen established the last-layer T5 embedding approach that the paper argues is suboptimal for LLMs.","marker":"[48]"},{"why":"Stable Diffusion v2's U-Net latent diffusion architecture provides the fixed training and conditioning pipeline.","marker":"[46]"},{"why":"GenAI-Bench and the VQAScore metric are the primary evaluation benchmark and scoring method for compositional alignment.","marker":"[34]"},{"why":"GPT-4o is used to implement VQAScore because the original CLIP-FlanT5 VQAScore could not differentiate similar models.","marker":"[23]"},{"why":"bge-Gemma2, the best-performing text encoder in the paper, is a fine-tuned embedding model; its result motivates the promise of embedding models.","marker":"[8]"}],"fun_headline_variants":["Layer-average LLM embeddings beat last-layer for text-to-image","Why last-layer LLM embeddings fail for text-to-image","Mistral-7B surpasses T5 with layer-averaged embeddings","LLM text encoders: average all layers, not just the last","Decoder-only LLMs excel with averaged embeddings in T2I"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's rankings assume that GPT-4o-based VQAScore measures text-image alignment accurately enough that a score gap like 0.741 versus 0.789 reflects a real, perceptible quality difference, and no human evaluation is provided to validate this.","fun_headline_variants_meta":{"raw":{"variants":["Layer-average LLM embeddings beat last-layer for text-to-image","Why last-layer LLM embeddings fail for text-to-image","Mistral-7B surpasses T5 with layer-averaged embeddings","LLM text encoders: average all layers, not just the last","Decoder-only LLMs excel with averaged embeddings in T2I"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1438,"prompt_tokens":921,"completion_tokens":517,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":426}},"tokens_in":537,"tokens_out":517,"duration_ms":5515,"temperature":1.0,"reasoning_tokens":426,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:16:47.168268+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A human evaluation in which users see paired images from T5-XXL last-layer and Mistral-7B norm-avg models on the GenAI-Bench prompts and prefer the T5-XXL outputs at or above chance, or a replication showing that the norm-avg advantage disappears when the original CLIP-FlanT5 VQAScore or a different metric like TIFA is used, would contradict the paper's central ranking.","supporting_citations":[{"cited_title":"Exploring the limits of transfer learning with a unified text-to-text transformer.JMLR, 2020","cited_arxiv_id":null,"evidence_quote":"T5-XXL is the baseline text encoder that the paper compares all LLMs against."},{"cited_title":"Photorealistic text-to-image diffusion models with deep language understanding","cited_arxiv_id":null,"evidence_quote":"Imagen established the last-layer T5 embedding approach that the paper argues is suboptimal for LLMs."},{"cited_title":"High-resolution image syn- thesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Stable Diffusion v2's U-Net latent diffusion architecture provides the fixed training and conditioning pipeline."},{"cited_title":"Evaluating text-to-visual generation with image-to-text gen- eration","cited_arxiv_id":null,"evidence_quote":"GenAI-Bench and the VQAScore metric are the primary evaluation benchmark and scoring method for compositional alignment."}],"review_version":1}