Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Text Embedding is Not All You Need: Attention Control for Text-to-Image Semantic Alignment with Text Self-Attention Maps

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that guiding cross-attention maps with the text encoder's own self-attention maps, through test-time latent optimization, reduces missing objects and attribute misbinding in text-to-image generation.

desk verdict Neat idea, conditional evidence: the syntax-transfer trick works, but the paper needs a scrambled-target control before we believe the mechanism. read the letter →

arxiv 2411.15236 v1 pith:6OIOEZYL submitted 2024-11-21 cs.CV cs.LG

Jeeyung Kim , Erfan Esmaeili , Qiang Qiu This is my paper · ORCID
classification cs.CVcs.LG
keywords text-to-imagediffusioncross-attentionmapstextself-attentionsemanticalignmentattributebindingmissingobjectstest-timeoptimizationattentionsink
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that text embeddings alone are an incomplete channel for telling a text-to-image model how words relate, and that the missing syntax already lives inside the model. It shows that tokens with similar embeddings end up with overlapping cross-attention maps, while syntactically bound pairs such as attribute-object and unbound pairs such as object-object are not separated in embedding similarity. At the same time, the text encoder's self-attention maps do separate bound from unbound pairs, but this information is largely lost before the embeddings leave the encoder because attention is concentrated on the token. The proposed method, T-SAM, transfers the syntax by optimizing the latent noise during inference so that the cross-attention similarity matrix matches the text self-attention matrix. If correct, this offers a self-contained, test-time way to reduce missing objects and attribute misbinding across arbitrary prompts, with TIFA score rising from 0.79 to 0.83 on the benchmark.

What carries the argument

The central object is the text self-attention matrix T, formed by averaging self-attention maps over text encoder layers and heads and renormalizing the rows after removing the <bos> and <eos> tokens. The complementary object is the cross-attention similarity matrix S, whose entries are cosine similarities between the cross-attention maps of pairs of tokens. The method minimizes a loss L(z_t) = sum_i sum_{j <= i} rho_i |T^gamma_ij - S_ij(z_t)|, with gamma controlling temperature and rho_i = i/s, by taking gradient steps on the latent z_t at selected denoising steps. Two propositions carry the analytic weight: one derives cos(A_i, A_j) approximately equal to exp(-1/2 (k_i - k_j)^T W^T Sigma W (k_i - k_j)), showing that text-embedding similarity controls cross-attention overlap; the other shows that attention sink on the <bos> token drives output token vectors to nearly identical cosine similarities, so the syntax visible in self-attention maps is not transferred into embeddings.

What would settle it

Take a large sample of TIFA prompts, compute T, and compare its bound/unbound separation with human dependency annotations or a parser's judgments; the central claim would be falsified if a substantial fraction of prompts show T rating an unbound pair as close to a bound pair. A more direct test: choose a prompt whose T visibly violates syntax after renormalization, run T-SAM, and measure whether TIFA or human judgment worsens relative to unguided Stable Diffusion.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the text encoder's self-attention maps contain syntactic relations that are not present in the text embeddings used to condition generation, and that transferring those relations to cross-attention maps at test time improves semantic alignment. The claim has two empirical parts: text-embedding similarity strongly drives cross-attention map similarity, and text-embedding similarity does not reflect whether tokens are syntactically bound. The authors attribute the loss of relational information to attention sink on the <bos> token, and they prove approximate statements showing why embedding similarity propagates to cross-attention maps and why attention sink makes self-attention-layer outputs nearly collinear. The resulting method optimizes the latent noise with a loss that pulls S toward T, and the paper reports consistent gains over Stable Diffusion, including a TIFA score of 0.83 versus 0.79, higher CLIP similarities, and qualitative recovery of missing objects and correctly bound attributes.

Load-bearing premise

The load-bearing premise is that the averaged and renormalized text self-attention matrix is a trustworthy map of which words should bind together in the image, for every prompt; if that matrix is contaminated by attention-sink leftovers or lexical co-occurrence rather than genuine syntax, the optimization pushes cross-attention toward the wrong binding pattern.

Editorial extensions

If this is right

  • T-SAM needs no external text parser and no manually selected token groups, so it applies to free-form prompts where parser- or selection-based methods cannot run.
  • On the TIFA benchmark, T-SAM raises the TIFA score from 0.79 to 0.83 and improves CLIP image-text similarity over unguided Stable Diffusion.
  • On structured object-attribute prompts, T-SAM matches the contrastive baseline CONFORM while avoiding the artificial separations that contrastive grouping can introduce.
  • The method improves most TIFA question categories, including color, shape, counting, and activity, indicating the transfer benefits a range of word relationships, not only adjective-noun binding.
  • All gains come from optimizing only a subset of denoising steps at inference time, so the method requires no training or additional model parameters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: if attention sink is the root cause, then suppressing <bos>/<eos> attention inside the text encoder during the forward pass, not just when building the target matrix, should partly restore syntax in the embeddings themselves and could make latent optimization lighter or unnecessary.
  • Inference: because T is treated as ground truth, T-SAM's failures should track T's failures; prompts where the encoder's self-attention is dominated by lexical co-occurrence or tokenization artifacts could be over-constrained, so testing on non-compositional prompts would map the method's boundary.
  • Inference: Proposition 1 suggests the same alignment could be achieved by reweighting key embeddings according to the W^T Sigma W metric before conditioning, which would be a cheaper, training-free alternative and a direct test of the proposed mechanism.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies text-to-image semantic misalignment in Stable Diffusion and argues that (1) the cosine similarity of text embeddings drives the similarity of cross-attention maps, (2) text embeddings do not encode syntactic relations well, and (3) the text encoder's self-attention maps do encode such relations, but this information is attenuated by an attention-sink effect on the <bos> token. Based on these observations, the authors propose T-SAM, a test-time optimization that updates the latent variable z_t to minimize the distance between a row-normalized cross-attention cosine-similarity matrix S and a powered, renormalized text self-attention matrix T^gamma (Eq. 14). The method requires no external parser or manual token grouping. Experiments on the TIFA benchmark report TIFA 0.83 for T-SAM versus 0.79 for Stable Diffusion, with smaller gains on structured attribution prompts, plus qualitative examples showing improved object presence and attribute binding.

Significance. The proposed mechanism is appealing: if syntactic structure already present in the text encoder's self-attention maps can be transferred to cross-attention at test time, it offers a self-contained and generalizable alternative to parser-based or manually annotated guidance. The paper also contributes a formal analysis (Propositions 1 and 2) connecting text-embedding similarity to cross-attention map similarity, and an attention-sink explanation for why syntactic information is diluted in embeddings. These are useful ideas for the community. However, the significance of the empirical claim is currently limited by the absence of a control that isolates the syntactic content of T as the active ingredient, by the small scale of the reported gains, and by hyperparameter selection performed on the same metrics used for evaluation. The central hypothesis is plausible but not yet convincingly demonstrated.

major comments (4)
  1. [Section 4.2, Eq. (14); Appendix B] The central claim that the syntactic relations encoded in the text self-attention matrix T cause the reported improvements is not yet supported, because the ablations vary only alpha and gamma, never the target matrix T itself. Appendix B states that alpha and gamma were selected by maximizing the CLIP-full and CLIP-min similarity scores, which are the same metrics on which gains are reported. A control experiment that replaces T with a shuffled version (e.g., row/column permutations), a constant matrix, or an identity matrix is necessary to determine whether the syntactic content of T is load-bearing or whether any attention-alignment regularization of this form produces similar gains. Without such a control, the improvement over SD could be attributed to a generic sharpening or smoothing of cross-attention similarity rather than to the specific linguistic structure in T.
  2. [Section 5.1, Table 1] The key quantitative result for the TIFA benchmark is based on a single generated image per prompt and reports no variance, confidence intervals, or significance tests. The reported TIFA improvement of 0.79 to 0.83 is modest, and since the hyperparameters were tuned on the same benchmark (Appendix B), the quoted number may be optimistically biased. Please report means and standard deviations across multiple seeds, and either use a held-out split or a nested selection procedure for alpha and gamma when reporting performance on the same benchmark.
  3. [Section 4.1, Figure 6a; Eq. (14)] The paper validates the syntactic informativeness of T only through averaged distributions of text self-attention values for bound versus unbound token pairs (Figure 6a). However, the loss function in Eq. (14) acts per-prompt on individual T_ij values, and the diagnostic in Figure 4 shows that correct images require low cross-attention similarity for unbound object-object pairs and high similarity for bound attribute-object pairs. The manuscript does not demonstrate that per-prompt T^gamma satisfies this requirement for the prompts used, and tokenization artifacts or residual <bos>/<eos> influence could make T unreliable on some prompts. I ask for a per-prompt analysis or a failure-mode study, and a discussion of how such cases are handled by the method.
  4. [Appendix A.4, Proposition 2, Eq. (23)] The condition in Eq. (23), |R_mn|/R_11 ~ O(1/epsilon) for 1 < m,n, with epsilon << 1, requires the Gram entries of non-<bos> value-projected embeddings to be much larger in magnitude than the <bos> diagonal entry. Since R is a positive-semidefinite Gram matrix, |R_mn| <= sqrt(R_mm R_nn), so the condition can hold only if R_mm and R_nn are themselves large relative to R_11, which is not established. The proof's cancellation of O(epsilon) terms depends on this condition; please provide empirical validation of the condition on the CLIP text encoder layers, or revise the proof to derive the result under weaker and better-justified assumptions.
minor comments (5)
  1. [Section 2, paragraph 1] The phrase 'sematic alignment' appears to be a typo for 'semantic alignment'.
  2. [Figure 8] The label 'T-SAML' in the figure header appears to be a typo for 'T-SAM'.
  3. [Eq. (7) and surrounding text] The definition of S_ij as a row-normalized version of C_ij is somewhat confusing because C_ij is already a cosine similarity; please clarify why the row normalization is needed and how it affects the comparison with the row-stochastic matrix T in Eq. (14).
  4. [Eq. (14), text below] The description of gamma as controlling 'temperature' is nonstandard; since gamma is applied as an exponent, consider renaming it 'sharpening exponent' or 'power-scaling exponent' to avoid confusion with softmax temperature.
  5. [Appendix B, first paragraph] The appendix notes that the first row of the attention matrix is omitted in the loss, but the text does not specify whether the first column entries in other rows are fully removed by the renormalization in Eq. (3); please state explicitly how the <bos> and <eos> tokens are treated in T^gamma.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor metric-selection overlap in the CLIP results; the central TIFA claim and the text-self-attention target are independent, so no significant circularity.

  1. fitted input called prediction [Appendix B (hyperparameter grid search) and Table 2 / Section 5.1]
    "We conducted a grid search for α within the set {5, 10, 15, 25, 40}, and for γ, we explored values in {2, 3, 4}. The parameters that maximize the CLIP-full and CLIP-min similarity scores are chosen."

    The same CLIP-full and CLIP-min scores that determined α and γ are then reported in Table 2 as evidence that T-SAM 'achieves higher CLIP similarity scores than the baselines, confirming its superior semantic alignment.' The reported CLIP gains are therefore values of the objective used for hyperparameter selection, not independent predictions; the grid search makes higher scores statistically likely by construction. The TIFA score and the text-self-attention target itself remain independent of this selection, so this is a partial rather than central circularity.

full rationale

The central derivation is not circular: the target matrix T is computed from the frozen CLIP text encoder and does not depend on the generated image, so minimizing L(z_t) = sum ρ_i |T^γ_ij − S_ij(z_t)| aligns cross-attention similarity to a fixed, image-independent target rather than fitting to the evaluation outcome. The paper contains no load-bearing self-citations and invokes no uniqueness theorems from its own authors. The TIFA benchmark improvement (0.79 to 0.83) is an external metric that was not used to select α or γ, providing independent support for the method. The one identified issue is that Appendix B states the hyperparameters were chosen to maximize CLIP-full and CLIP-min, and Table 2 later presents those same CLIP scores as confirmation of superior semantic alignment; this is a fitted-input-called-prediction pattern for the CLIP-based evidence. The skeptical concern about the absence of a scrambled-target control is a missing-control limitation, not a demonstration that the result reduces to its inputs by construction. Overall the paper is largely self-contained, with only a minor non-independent metric overlap.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method does not introduce new physical or conceptual entities. It relies on several empirical assumptions about attention distributions and the reliability of text self-attention maps as syntax priors. The free parameters are standard hyperparameters, but their tuning against the evaluation metrics is a concern. The attention sink and Gaussian query assumptions are plausible but not rigorously proven, and the R matrix condition is ad hoc.

free parameters (4)
  • alpha (gradient step size) = 40 for TIFA, 10 for Attend-n-Excite
    Grid-searched over {5, 10, 15, 25, 40} to maximize CLIP-full and CLIP-min similarity scores on the evaluation benchmarks.
  • gamma (temperature exponent on T) = 4
    Grid-searched over {2, 3, 4}; chosen to maximize CLIP similarity scores on the same benchmarks used for evaluation.
  • rho_i (token weighting in loss) = i/s
    Arbitrary linear weighting giving more weight to later tokens; not justified theoretically.
  • Update step schedule = Steps 1 to 25 for TIFA; steps {0, 10, 20} for Attend-n-Excite
    Chosen empirically; different schedules for different datasets without a principled justification.
assumptions (5)
  • domain assumption Attention sink assumption: In the text encoder and cross-attention, the sum of attention probabilities to non-<bos> tokens over the <bos> token is less than epsilon << 1 (Eq. 1 and 2 in Appendix A.2).
    This is an empirical claim about the structure of attention in CLIP and Stable Diffusion. It underpins the approximations in Propositions 1 and 2. In practice, attention sink does occur, but the magnitude of epsilon ~ 0.1 is not universally verified.
  • domain assumption Query vectors in cross-attention are jointly Gaussian with mean mu and covariance Sigma (Assumption ii in Appendix A.3).
    Needed for the moment-generating function calculation in the proof of Proposition 1. The authors note it is strictly true at the first denoising step for the initial Gaussian latent, but less so at later steps.
  • domain assumption Assumption iii: mu_1 >> mu_i, mu_1 >> sigma_1, and sigma_1 ~ sigma_i for i = 2, ..., s.
    This assumption ensures that the exponential of the attention score difference to the <bos> token is small, making the approximation (14) valid. It is stated as an empirical observation.
  • ad hoc to paper Matrix R condition in Proposition 2: |R_mn|/R_11 ~ O(1/epsilon) and |R_1m|/R_11 ~ O(1).
    This condition is stated without empirical verification and seems contrived to make the proposition true. It is not established that value-projection Gram matrices in CLIP satisfy this scaling.
  • domain assumption Text self-attention maps T are a reliable and complete encoding of syntactic relations for diverse prompts.
    This is the central empirical premise of the method. The paper shows a few examples (Figures 6a, A, B) but does not provide a quantitative validation on a broad set of prompts with syntactic annotations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Text Embedding is Not All You Need: Attention Control for Text-to-Image Semantic Alignment with Text Self-Attention Maps." pith.science (2026). https://pith.science/paper/6OIOEZYL

@misc{pith2026241115236,
  author       = {Pith},
  title        = {Pith review of: Text Embedding is Not All You Need: Attention Control for Text-to-Image Semantic Alignment with Text Self-Attention Maps},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6OIOEZYL}},
  note         = {Machine review of arXiv:2411.15236}
}
read the original abstract

In text-to-image diffusion models, the cross-attention map of each text token indicates the specific image regions attended. Comparing these maps of syntactically related tokens provides insights into how well the generated image reflects the text prompt. For example, in the prompt, "a black car and a white clock", the cross-attention maps for "black" and "car" should focus on overlapping regions to depict a black car, while "car" and "clock" should not. Incorrect overlapping in the maps generally produces generation flaws such as missing objects and incorrect attribute binding. Our study makes the key observations investigating this issue in the existing text-to-image models:(1) the similarity in text embeddings between different tokens -- used as conditioning inputs -- can cause their cross-attention maps to focus on the same image regions; and (2) text embeddings often fail to faithfully capture syntactic relations already within text attention maps. As a result, such syntactic relationships can be overlooked in cross-attention module, leading to inaccurate image generation. To address this, we propose a method that directly transfers syntactic relations from the text attention maps to the cross-attention module via a test-time optimization. Our approach leverages this inherent yet unexploited information within text attention maps to enhance image-text semantic alignment across diverse prompts, without relying on external guidance.

Figures

Figures reproduced from arXiv: 2411.15236 by the authors.

Figure 1
Figure 1. The overview of our method. We leverage text self [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. For the analysis, we use the prompt sets from [ [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The generated images and cross-attention maps ( [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Comparison for the distributions of cosine similarity [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Correlation between the cosine similarity of text em [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 7
Figure 7. Figure 7: Accuracy for each question type in the TIFA benchmark. Our method achieves the best performance in most categories. The [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Comparison of our method (T-SAM) with recent state-of-the-art methods on prompts from [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Qualitative comparison using prompts from MSCOCO [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Detail++: Training-Free Detail Enhancer for T2I Diffusion Models

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Detail++ uses progressive multi-branch prompt injection and test-time attention optimization to improve attribute binding in text-to-image generation.

Reference graph

Works this paper leans on

41 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    A-star: Test-time attention segregation and retention for text-to-image synthesis

    Aishwarya Agarwal, Srikrishna Karanam, KJ Joseph, Apoorv Saxena, Koustava Goswami, and Balaji Vasan Srini- vasan. A-star: Test-time attention segregation and retention for text-to-image synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2283– 2293, 2023. 1, 2, 3, 4

    work page 2023

  2. [2]

    AlignIT: Enhancing Prompt Alignment in Customization of Text-to-Image Models

    Aishwarya Agarwal, Srikrishna Karanam, and Balaji Vasan Srinivasan. Alignit: Enhancing prompt alignment in customization of text-to-image models. arXiv preprint arXiv:2406.18893, 2024. 3

    work page Pith review arXiv 2024

  3. [3]

    Attend-and-excite: Attention-based se- mantic guidance for text-to-image diffusion models

    Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based se- mantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 42(4):1–10, 2023. 2, 3, 4, 5, 6

    work page 2023

  4. [4]

    Pixart- α: Fast training of diffusion transformer for photorealistic text-to-image synthesis

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart- α: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023. 2

    arXiv

  5. [5]

    Pixart- \sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation

    Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- \sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint arXiv:2403.04692, 2024. 2

    arXiv 2024

  6. [6]

    Dall-eval: Probing the reasoning skills and social biases of text-to- image generation models

    Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to- image generation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3043– 3054, 2023. 6

    work page 2023

  7. [7]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learn- ing, 2024. 2

    2024

  8. [8]

    Training-free structured diffusion guidance for compositional text-to-image synthesis

    Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint arXiv:2212.05032, 2022. 1, 2, 3, 4

    arXiv 2022

Show all 41 references
  1. [9]

    When attention sink emerges in language models: An empirical view

    Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. arXiv preprint arXiv:2410.10781, 2024. 3

    2024 arXiv

  2. [10]

    Prompt-to-prompt im- age editing with cross attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt im- age editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022. 1, 3

    2022 arXiv

  3. [11]

    spacy 2: Natural lan- guage understanding with bloom embeddings, convolutional neural networks and incremental parsing

    Matthew Honnibal and Ines Montani. spacy 2: Natural lan- guage understanding with bloom embeddings, convolutional neural networks and incremental parsing. To appear, 7(1): 411–420, 2017. 6

    2017

  4. [12]

    Ella: Equip diffusion models with llm for enhanced semantic alignment

    Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with llm for enhanced semantic alignment. arXiv preprint arXiv:2403.05135, 2024. 3

    2024 arXiv

  5. [13]

    Tifa: Accu- rate and interpretable text-to-image faithfulness evaluation with question answering

    Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accu- rate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, page...

    2023

  6. [14]

    Mc2: Multi-concept guidance for customized multi-concept generation

    Jiaxiu Jiang, Yabo Zhang, Kailai Feng, Xiaohe Wu, and Wangmeng Zuo. Mc2: Multi-concept guidance for customized multi-concept generation. arXiv preprint arXiv:2404.05268, 2024. 3

    2024 arXiv

  7. [15]

    Dense text-to-image generation with attention modulation

    Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Woo Ha, and Jun-Yan Zhu. Dense text-to-image generation with attention modulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7701–7711, 2023. 3

    2023

  8. [16]

    mplug: Effective and efficient vision-language learning by cross-modal skip-connections

    Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, et al. mplug: Effective and efficient vision-language learning by cross-modal skip-connections. arXiv preprint arXiv:2205.12005, 2022. 6

    2022 arXiv

  9. [17]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Interna- tional conference on machine learning, pages 12888–12900. PMLR, 2022. 6

    2022

  10. [18]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  11. [19]

    Improving text- to-image consistency via automatic prompt optimization

    Oscar Ma ˜nas, Pietro Astolfi, Melissa Hall, Candace Ross, Jack Urbanek, Adina Williams, Aishwarya Agrawal, Adri- ana Romero-Soriano, and Michal Drozdzal. Improving text- to-image consistency via automatic prompt optimization. arXiv preprint arXiv:2403.17804, 2024. 3

    2024 arXiv

  12. [20]

    Attention overlap is responsible for the entity missing problem in text-to-image diffusion models! arXiv preprint arXiv:2410.20972, 2024

    Arash Marioriyad, Mohammadali Banayeeanzade, Reza Ab- basi, Mohammad Hossein Rohban, and Mahdieh Soleymani Baghshah. Attention overlap is responsible for the entity missing problem in text-to-image diffusion models! arXiv preprint arXiv:2410.20972, 2024. 3

    arXiv

  13. [21]

    Conform: Contrast is all you need for high- fidelity text-to-image diffusion models

    Tuna Han Salih Meral, Enis Simsar, Federico Tombari, and Pinar Yanardag. Conform: Contrast is all you need for high- fidelity text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9005–9014, 2024. 1, 2, 3, 4, 6

    2024

  14. [22]

    Openai gpt-3 api [gpt-3.5-turbo], 2024

    OpenAI. Openai gpt-3 api [gpt-3.5-turbo], 2024. Available at: https://platform.openai.com. 6

    2024

  15. [23]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 2

    2023 arXiv

  16. [24]

    Not all noises are created equally: Diffusion noise selection and optimization

    Zipeng Qi, Lichen Bai, Haoyi Xiong, et al. Not all noises are created equally: Diffusion noise selection and optimization. arXiv preprint arXiv:2407.14041, 2024. 3 9

    2024 arXiv

  17. [25]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  18. [26]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 2

    2020

  19. [27]

    Hierarchical text-conditional image gener- ation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gener- ation with clip latents. arXiv preprint arXiv:2204.06125, 1 (2):3, 2022. 2

    2022 arXiv

  20. [28]

    Linguistic bind- ing in diffusion models: Enhancing attribute correspondence through attention map alignment

    Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Rav- fogel, Yoav Goldberg, and Gal Chechik. Linguistic bind- ing in diffusion models: Enhancing attribute correspondence through attention map alignment. Advances in Neural Infor- mation Processing Systems, 36, 2024. 1, 2, 3, 4, 6

    2024

  21. [29]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 2, 3, 6

    2022

  22. [30]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information...

    2022

  23. [31]

    Rethinking the spatial inconsistency in classifier- free diffusion guidance

    Dazhong Shen, Guanglu Song, Zeyue Xue, Fu-Yun Wang, and Yu Liu. Rethinking the spatial inconsistency in classifier- free diffusion guidance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9370–9379, 2024. 3

    2024

  24. [32]

    Massive activations in large language models.arXiv preprint arXiv:2402.17762, 2024

    Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models.arXiv preprint arXiv:2402.17762, 2024. 2, 3, 5

    2024 arXiv

  25. [33]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 3

    2017

  26. [34]

    Tokencompose: Grounding diffusion with token-level supervision

    Zirui Wang, Zhizhou Sha, Zheng Ding, Yilin Wang, and Zhuowen Tu. Tokencompose: Grounding diffusion with token-level supervision. arXiv preprint arXiv:2312.03626 ,

    arXiv

  27. [35]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023. 2, 3, 4, 5

    2023 arXiv

  28. [36]

    Dynamic prompt learning: Addressing cross- attention leakage for text-based image editing

    Fei Yang, Shiqi Yang, Muhammad Atif Butt, Joost van de Weijer, et al. Dynamic prompt learning: Addressing cross- attention leakage for text-based image editing. Advances in Neural Information Processing Systems , 36:26291–26303,

  29. [37]

    Towards understanding the working mechanism of text-to-image dif- fusion model

    Mingyang Yi, Aoxue Li, Yi Xin, and Zhenguo Li. Towards understanding the working mechanism of text-to-image dif- fusion model. arXiv preprint arXiv:2405.15330 , 2024. 3, 5

    2024 arXiv

  30. [38]

    Uncovering the text embedding in text-to-image diffusion models

    Hu Yu, Hao Luo, Fan Wang, and Feng Zhao. Uncovering the text embedding in text-to-image diffusion models. arXiv preprint arXiv:2404.01154, 2024. 3

    2024 arXiv

  31. [39]

    Scaling autoregres- sive models for content-rich text-to-image generation

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gun- jan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yin- fei Yang, Burcu Karagol Ayan, et al. Scaling autoregres- sive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2(3):5, 2022. 6

    2022 arXiv

  32. [40]

    When and why vision- language models behave like bags-of-words, and what to do about it? arXiv preprint arXiv:2210.01936, 2022

    Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision- language models behave like bags-of-words, and what to do about it? arXiv preprint arXiv:2210.01936, 2022. 2, 5

    2022 arXiv

  33. [41]

    is there a green backpack?

    Yang Zhang, Teoh Tze Tzun, Lim Wei Hern, Tiviatis Sim, and Kenji Kawaguchi. Enhancing semantic fidelity in text- to-image synthesis: Attention regulation in diffusion models. arXiv preprint arXiv:2403.06381, 2024. 3 10 Text Embedding is Not All You Need: Attention Control for ...

    2024 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.