{"id":"b318b9ba-1c94-4295-b575-192e1cf33f2d","arxiv_id":"2502.04412","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Using hidden states from a decoder-only LLM as the text conditioning, via a learned adapter, improves text-to-image generation fidelity in the authors' experiments.","lead":"This paper proposes a plug-in adapter that lets text-to-image diffusion models use the internal representations of decoder-only large language models (LLMs) as their text controller. In the paper's tests, the adapted model followed complex prompts more faithfully than models using standard text encoders.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The title-level claim is confounded: the only strong win uses a 7B decoder-only model, while the size-matched phi-1.5 variant loses to T5-XL, and no error bars or user-study results are reported.","rationale":"The reader correctly identifies Section 3.4's Eq. (5) as a weak theoretical step: the derivation jumps from p(x_t^<d | x_{t-1}^d, x_t^d) to a score function for the text encoding c_<d without establishing the relationship between c and the token hidden states, and Algorithm 1 is not a faithful instantiation of the equations. That is a real concern. However, I find an even more load-bearing issue in the empirical evaluation, because the title claim is fundamentally comparative and architectural. Even if the theoretical derivation were repaired, the paper would still need to show that the observed gains come from the decoder-only structure rather than from simply using a much larger, more recently pretrained language model. Table 1's phi-1.5 result actually provides evidence against the architectural claim, since it underperforms a T5-XL baseline despite the paper's assertion of parameter proximity. The absence of error bars and the complete omission of the promised user-study results further weaken the quantitative basis for 'superior' and 'reliable.' The reader's verdict of CONDITIONAL remains appropriate, because the paper could be revised with matched-size baselines and proper uncertainty reporting; my concern does not change the verdict but sharpens the condition under which the central claim would be acceptable.","tokens_in":13382,"tokens_out":11530,"duration_ms":102695,"concrete_test":"Reproduce the adapter pipeline with architecture-matched language models of similar parameter count and pretraining scale, keeping training data, adapter capacity, and evaluation protocol identical. Concretely: (a) replace Vicuna-7B with a ~7B encoder-only or encoder-decoder text model (e.g., FLAN-T5-XXL or a 7B CLIP-style encoder) and evaluate on the same SigLIP/CLIP-IQA prompts; (b) replace Phi-1.5 with a ~1.3B T5 or encoder-only model and compare; (c) run each configuration with at least 3 random seeds and report mean ± std with a paired significance test, and include the missing user-study results. If the decoder-only model still significantly outperforms its size-matched encoder counterpart, the architectural claim is supported; if not, the reported gains are attributable to model scale rather than decoder-only architecture.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the decoder-only architecture, not model scale, makes LLMs better text controllers for diffusion models (Abstract, Section 1, Table 1). The empirical support does not isolate architecture from capacity. Table 1 compares SD1.5, SDXL, SD1.5+(T5-XL), Ours(phi1.5), and Ours(Vicuna-7B). The strongest result (SigLIP 8.5) uses Vicuna-7B, which has roughly 7B parameters, far larger than the CLIP text encoder (~0.4B) and T5-XL (~2.8B) used in baselines. The matched-size attempt, Ours(phi1.5) at 1.3B, scores 5.8 on SigLIP, below SD1.5+(T5-XL) at 7.4. This directly undercuts the architectural claim: if a similar-sized decoder-only model is worse than an encoder-decoder model, the reported advantage of Vicuna is more plausibly due to scale and pretraining data than to the decoder-only structure. The paper states in Section 5.1 that Phi1.5 is close in parameters to CLIP and T5, but T5-XL is not close in size; this makes the 'fair comparison' assertion inaccurate. Furthermore, no error bars, confidence intervals, or significance tests are provided for any SigLIP or CLIP-IQA score, and the user study described in Section 5.1 as a metric for logicality is never reported in Table 1 or anywhere else. Without a size-matched comparison and uncertainty quantification, the evidence does not establish that decoder-only LLMs are better controllers; it only suggests that a large frozen LLM can improve controllability when combined with an adapter.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LLMDiff-Adapter, a plug-and-play module that lets pre-trained text-to-image diffusion models (Stable Diffusion 1.5) use block-wise hidden states from a decoder-only LLM (Phi-1.5 or Vicuna-7B) as text conditioning, in place of encoder-based text encoders such as CLIP or T5. The authors argue, from a probabilistic perspective, that a decoder-only LLM can be viewed as a reverse diffusion process, and that text encodings for image generation can be obtained by running Langevin dynamics on a score function approximated as the difference between sentence-level and single-word LLM prediction scores. They report quantitative results (SigLIP score, CLIP-IQA quality/complexity/beauty, and a described user study) and qualitative examples. The central claim is that decoder-only LLMs are better controllers for diffusion models than encoder-based text encoders, and that the proposed adapter yields superior controllability and image quality.","tokens_in":13846,"tokens_out":4054,"duration_ms":39691,"significance":"If established, the claim would be significant for text-to-image generation because it offers a simple way to exploit the semantic and reasoning capabilities of large decoder-only LLMs without retraining the diffusion model. The adapter design is straightforward, keeps the base diffusion model and LLM frozen, and is potentially applicable to other diffusion architectures. The paper makes a genuine attempt at theoretical grounding, which is rare in this area. However, the empirical evidence is currently not conclusive: the only large improvement over baselines comes from a 7B LLM, while the size-matched 1.3B variant is markedly worse, and the reported user-study metric is never actually presented. The theoretical derivation also contains an unproven core approximation. These issues need to be addressed before the central claim can be accepted.","major_comments":[{"comment":"The empirical evidence does not isolate the effect of the decoder-only architecture from model scale. The strongest result (SigLIP 8.5) uses Vicuna-7B, which has roughly 7B parameters, whereas the text encoders of the baseline models are much smaller (CLIP is roughly 0.4B and T5-XL roughly 2.8B). More importantly, the size-matched attempt, Ours (phi-1.5) with 1.3B parameters, achieves SigLIP 5.8, which is below SD1.5+(T5-XL) at 7.4. This outcome directly undercuts the title-level claim that decoder-only LLMs are better controllers: if a similar-scale decoder-only model is worse than an encoder-decoder model, the Vicuna-7B gain is more plausibly attributed to scale and pretraining data than to the decoder-only structure. The statement in Section 5.1 that \"Phi1.5 is close to that of the text encoders of CLIP and T5\" is inaccurate for T5-XL, which has about twice the parameters. A proper comparison should include a decoder-only model of comparable size to T5-XL, or a T5 variant sized to match Phi-1.5.","section":"Section 5.2, Table 1"},{"comment":"The approximation in Eq. (5) is the load-bearing step for Algorithm 1, but it is asserted rather than derived. Equation (4) gives a proportionality for the posterior p(x^t_<d | ...), not a statement about the score of p(c_<d | ...). The jump from these posterior ratios to the score function of the text encoding c_<d, with an unspecified time-dependent scalar g(t), involves an implicit and unjustified relationship between the token sequence x and the implicit continuous encoding c. Equation (6) then further replaces score functions with log-probability differences in a way that does not follow from Eq. (1) or Eq. (4). Consequently, the \"supporting theoretical analysis\" does not actually justify the sentence-minus-word score in Algorithm 1. The authors should either supply a rigorous derivation of this approximation, or explicitly reframe Section 3.4 as an intuition/heuristic and remove the claim that the method is theoretically grounded.","section":"Section 3.4, Eq. (5)"},{"comment":"The user study, which is described as the metric for logicality, is never reported anywhere in the paper. This is a significant omission because the qualitative claims about reasoning and logical coherence (Section 5.4) rest on this evaluation. Additionally, no error bars, confidence intervals, or significance tests are provided for any of the SigLIP or CLIP-IQA scores in Table 1; several differences between methods are small (e.g., Complexity 24.9 vs. 23.9, Beauty 91.0 vs. 90.9), and it is unclear whether these are reproducible. The authors should report the user study results (number of participants, prompts, and outcome) and provide variance or significance information for the quantitative metrics.","section":"Section 5.1, Metrics"}],"minor_comments":[{"comment":"There is a typo: \"stat-of-the-art\" should be \"state-of-the-art.\"","section":"Abstract"},{"comment":"\"For similarity\" should read \"For simplicity.\"","section":"Section 3.2"},{"comment":"The caption contains the typo \"Scentence\"; it should be \"Sentence.\"","section":"Figure 2 caption"},{"comment":"The claim that the Vicuna-7B SigLIP score of 8.5 is \"31% higher\" than SDXL's 6.2 is arithmetically incorrect; the increase is approximately 37%. The percentage for phi-1.5 compared to SD1.5 is correct.","section":"Section 5.2"},{"comment":"The paper does not state how many prompts or images were used for the SigLIP and CLIP-IQA evaluations, nor how images were sampled (e.g., seeds, number of samples per prompt). This information is needed to assess the reliability of the reported scores.","section":"Section 5.1 / Table 1"},{"comment":"The notation for the score functions S_theta is introduced only informally in the text preceding Algorithm 1; the arguments of S_theta in lines 4 and 5 of the algorithm (e.g., S_theta t(x^{t-1}_d, x^t_d, x^t_<d)) do not match the notation of Eq. (5), where the scores are taken with respect to different variables. Please define these score functions explicitly.","section":"Section 3.4, Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is formatted as a camera-ready for ACM Multimedia 2024, and the experimental section is quite short for a journal version. The editor should consider whether the expected scope of this journal requires a more thorough evaluation than what is currently presented. The central empirical claim is not yet supported because of the model-size confound and the unreported user study; the theoretical derivation also needs substantial rework. These are fixable with additional experiments and a more cautious framing, so I do not recommend rejection at this point."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline is this: the paper has a clean, genuinely new adapter idea—block-wise decoder-only LLM hidden states as text conditioning—and the qualitative results with Vicuna-7B are impressive. But the title-level claim that decoder-only architecture is the reason falls apart under a size-matched comparison: the phi-1.5 variant scores 5.8 SigLIP, below the T5-XL baseline at 7.4. So what the data actually show is that a large frozen LLM helps, not that decoder-only beats encoder-decoder.\n\nCredit where due: nothing I know of feeds LLM hidden states directly into cross-attention this way; prior work rewrites prompts or predicts layouts. The adapter itself is simple and sensible—keep the original cross-attention, add a new one with learnable scale/offset, initialize to preserve original behavior. They also report the phi-1.5 result, which is an honest data point even though it undercuts their own claim. Training on ~1M images and showing gains on entity/relationship prompts is a useful incremental contribution.\n\nThe soft spots are real. Eq. (5) is asserted, not derived; the whole diffusion-LLM analogy is loose and doesn't do the work. There are no error bars, and the user study described in Sec. 5.1 never appears in the results despite being the stated metric for logicality. The 'fair comparison' statement about phi-1.5 being close in size to T5-XL is wrong—T5-XL is roughly twice the size. These don't sink the adapter as a heuristic, but they do sink the architectural inference. A properly matched study with, say, a 2B-3B decoder-only model, or a scaling curve, would be needed to justify the title.\n\nWho is this for? Practitioners who want a plug-and-play way to swap a text encoder for an LLM, and who can afford a 7B frozen backbone. The paper is a legitimate incremental contribution, not a field-changer. I'd send it to review—it deserves referee time to sort out the size confound and missing uncertainty numbers—but I'd tell the authors the central claim needs another experiment. I wouldn't cite it as evidence that decoder-only is better; I'd cite it as an adapter design.","headline":"The adapter is a genuine, clean idea, but the title-level claim is confounded by model scale and missing uncertainty quant.","tokens_in":14227,"tokens_out":3132,"would_cite":true,"duration_ms":30592,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that replacing encoder-based text encoders with block-wise hidden states of decoder-only LLMs, via a lightweight adapter, makes text-to-image diffusion models more controllable and higher quality.","keywords":["Text-to-Image Generation","Diffusion Models","Decoder-only LLM","LLMDiff-Adapter","Text Encoding","Cross-Attention","Semantic Understanding","Controllability"],"falsifier":"Run Algorithm 1 on a set of prompts and compare the resulting text encodings against a reference encoding obtained by a method that does not rely on the sentence-minus-word difference, such as directly optimizing the encoding to reproduce the LLM's next-token predictions; if the two disagree substantially on context-dependent prompts, the theoretical claim is contradicted. A cheaper check: use a single-word prompt, where sentence-level and single-word scores are equal by construction; the derivation predicts no useful conditioning signal, so if the generated images still respond meaningfully to the prompt, the approximation is not the source of control.","tokens_in":13226,"feed_emoji":"🖼️","tokens_out":4771,"duration_ms":41706,"temperature":0.7,"pith_summary":"This paper argues that the text encoders used by current text-to-image diffusion models, typically encoder-only models like CLIP or T5, are a bottleneck: they miss entity relationships, counts, and details in complex prompts. The authors propose replacing them with a decoder-only large language model (LLM) whose block-wise hidden states are turned into text encodings through a small adapter module attached to the diffusion U-Net's cross-attention. They support this with a derivation that views a transformer LLM as a denoising diffusion process, so text encodings can be sampled by Langevin dynamics from the difference between sentence-level and single-word LLM predictions. Experiments on Stable Diffusion 1.5 show higher SigLIP alignment scores and better human-rated logic than SD1.5, SDXL, and T5-based variants, with the gains increasing with LLM size.","feed_headline":"Decoder-only LLMs beat text encoders at guiding image generation","feed_subtitle":"A plug-in adapter using LLM block states lifts text-to-image alignment above CLIP/T5 baselines.","key_machinery":"The central object is the LLMDiff-Adapter, a plug-in module that combines two cross-attention branches: the original branch aligned with the existing text encoder and a new branch fed by block-wise hidden states from a decoder-only LLM. The theoretical engine is the 'LLM as diffusion model' view, in which each transformer block acts as a denoising step and the score of the text-encoding distribution is approximated as the difference between sentence-level and single-word block predictions (Eq. 5). Algorithm 1 turns this score into actual text encodings by Langevin dynamics sampling from the LLM's hidden states. The adapter's two branches are mixed with learnable weights, initialized so the original branch dominates at the start of training.","core_discovery":"The central claim is that decoder-only LLMs are better controllers for diffusion models than encoder-based text encoders. The paper shows that, although decoder-only LLMs do not expose text features directly, their per-block hidden states can be interpreted as the result of a diffusion process. Under this view, the text-encoding distribution can be estimated by comparing the LLM's prediction for a full sentence with its prediction for a single word at each block, and the resulting score is used in Langevin dynamics to sample text encodings. These encodings are fed through a lightweight adapter, formed by two cross-attention branches, one aligned to the original text encoder and one learned from the LLM encoding, into a pre-trained diffusion U-Net. With the LLM and U-Net frozen, only the adapter is trained, and the resulting model improves generation quality, detail, and logical consistency, including counting entities, respecting spatial relations, and inferring physical consequences.","pith_inferences":["The theoretical backing rests on an unproven approximation; if Eq. (5) does not hold, the method still works as an empirically motivated heuristic, but its claimed principled grounding weakens.","A natural extension is to apply the same adapter to other conditional diffusion models, such as video or 3D generation, where prompt controllability is also a known limitation.","The block-wise hidden-state requirement rules out closed-source LLMs; the authors note this, which implies the approach currently depends on open-weight models with accessible internal states.","One could isolate the contribution of the theory by abating the sentence-minus-word score with a simpler pooling of hidden states; if that baseline matches the full method, the theoretical derivation is not the source of the gains."],"forward_implications":["Stronger decoder-only LLMs yield stronger controllability: Vicuna-7B outperforms phi-1.5, which itself improves over the Stable Diffusion 1.5 baseline.","The adapter needs only about 45 million trainable parameters while both the LLM and the diffusion U-Net remain frozen.","The approach captures counts, entity relations, and physical reasoning that encoder-based baselines miss, as the qualitative examples show.","The method generalizes across LLM scales, since both a compact model (phi-1.5) and a 7B model (Vicuna-7B) improve generation quality and alignment."],"supporting_citations":[{"why":"Supplies the base Stable Diffusion model and the diffusion U-Net architecture that the LLMDiff-Adapter is plugged into.","marker":"[27]"},{"why":"Provides the CLIP text encoder used as a primary baseline and the standard for text-image alignment evaluation.","marker":"[25]"},{"why":"Provides the T5 encoder-decoder baseline used in comparisons.","marker":"[26]"},{"why":"DALL-E 3 is the state-of-the-art encoder-based model compared against in the qualitative examples.","marker":"[2]"},{"why":"Denoising Diffusion Probabilistic Models is the framework used to reinterpret transformer blocks as denoising steps.","marker":"[12]"},{"why":"phi-1.5 is one of the two decoder-only LLMs used to derive text encodings in the experiments.","marker":"[17]"},{"why":"Vicuna-7B is the stronger decoder-only LLM used to demonstrate that model capacity improves controllability.","marker":"[6]"},{"why":"SigLIP is the evaluation metric used to measure text-image alignment independently of CLIP.","marker":"[32]"}],"fun_headline_variants":["LLM hidden states as diffusion signals boost text-to-image","Adapter lets diffusion models tap decoder-only LLM semantics","Decoder-only LLMs better guide diffusion via hidden-state diffusion","Text-to-image improved by treating LLM blocks as diffusion steps","From LLM to diffusion: a simple adapter lifts image alignment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The derivation's load-bearing step is the approximation in Eq. (5), which asserts that the score of the text-encoding distribution is the difference between the LLM's sentence-level and single-word block predictions; if that equality does not hold, Algorithm 1 loses its theoretical justification.","fun_headline_variants_meta":{"raw":{"variants":["LLM hidden states as diffusion signals boost text-to-image","Adapter lets diffusion models tap decoder-only LLM semantics","Decoder-only LLMs better guide diffusion via hidden-state diffusion","Text-to-image improved by treating LLM blocks as diffusion steps","From LLM to diffusion: a simple adapter lifts image alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00026,"raw_usage":{"total_tokens":1601,"prompt_tokens":972,"completion_tokens":629,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":547}},"tokens_in":588,"tokens_out":629,"duration_ms":6629,"temperature":1.0,"reasoning_tokens":547,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T23:51:04.179561+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Algorithm 1 on a set of prompts and compare the resulting text encodings against a reference encoding obtained by a method that does not rely on the sentence-minus-word difference, such as directly optimizing the encoding to reproduce the LLM's next-token predictions; if the two disagree substantially on context-dependent prompts, the theoretical claim is contradicted. A cheaper check: use a single-word prompt, where sentence-level and single-word scores are equal by construction; the derivation predicts no useful conditioning signal, so if the generated images still respond meaningfully to the prompt, the approximation is not the source of control.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the base Stable Diffusion model and the diffusion U-Net architecture that the LLMDiff-Adapter is plugged into."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CLIP text encoder used as a primary baseline and the standard for text-image alignment evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DALL-E 3 is the state-of-the-art encoder-based model compared against in the qualitative examples."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Denoising Diffusion Probabilistic Models is the framework used to reinterpret transformer blocks as denoising steps."}],"review_version":1}