Pith. sign in

REVIEW 3 major objections 6 minor 3 cited by

Rethinking Cross-Modal Interaction in Multimodal Diffusion Transformers

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Temperature scaling of cross-modal attention rebalances text and image tokens to improve prompt following.

desk verdict TACA is a simple, plausibly useful temperature-scaling tweak for MM-DiT attention, but the paper's causal story about token imbalance is not backed by direct logit measurements; the empirical gains are real but moderate. read the letter →

arxiv 2506.07986 v3 pith:O34J6CFG submitted 2025-06-09 cs.CV

Zhengyao Lv , Tianlin Pan , Chenyang Si , Zhaoxi Chen , Wangmeng Zuo , Ziwei Liu , Kwan-Yee K. Wong This is my paper · ORCID
classification cs.CV
keywords multimodaldiffusiontransformertext-to-imagealignmentcross-modalattentiontemperaturescalingdenoisingtimestepLoRAfine-tuningcompositionalgenerationsuppression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal diffusion transformers power today's leading text-to-image models, yet these models often omit or misplace objects described in the prompt. The paper traces this failure to two properties of the unified attention mechanism: the large number of image tokens dilutes the attention image tokens pay to text tokens, and the attention weights do not adapt to the phase of denoising. It proposes Temperature-Adjusted Cross-modal Attention (TACA), which multiplies visual-text attention logits by a scalar $\gamma(t)$ that is active early in denoising and turns off later, so text guidance dominates during layout formation. Paired with a small LoRA fine-tune to remove artifacts, TACA improves compositional alignment on T2I-CompBench for both FLUX.1-Dev and SD3.5-Medium with almost no added computation.

What carries the argument

The load-bearing object is a single scalar temperature coefficient $\gamma(t)$ inserted into the unified softmax of MM-DiT. It multiplies only the visual-text logits, leaving visual-visual and text-text logits untouched, so it acts as a signal booster for text guidance. The threshold $t_{thresh}$ makes the boost conditional on denoising phase, matching the observation that global layout is fixed in the first steps. A lightweight LoRA adapter on the attention layers compensates for the distribution shift the boosted logits introduce; TACA itself adds no learnable parameters.

What would settle it

Measure the average visual-text attention probability $P^{(i,j)}_{vis-txt}$ in a frozen MM-DiT model over many prompts and denoising steps, and compare it with the probability a typical cross-attention layer would give for the same text tokens. If early-step cross-modal attention is not substantially lower than the typical cross-attention baseline once logit scales are taken into account, then token imbalance is not suppressing cross-modal attention and TACA's performance would be a heuristic artifact rather than a fix for the diagnosed mechanism.

Watch

Extended reading notes

Core claim

The paper's central claim is that the semantic misalignment of MM-DiT models is caused by suppression of visual-to-text attention inside a single softmax over concatenated image and text tokens, together with timestep-insensitive query-key weighting. Because $N_{vis}\gg N_{txt}$ (e.g., 4096 visual versus 512 text tokens for a 1024$\times$1024 FLUX image), the denominator of the softmax is dominated by visual-visual terms, so $P^{(i,j)}_{vis-txt}$ is much smaller than it would be in a typical cross-attention layer. TACA counteracts this by replacing $s^{vt}_{ij}$ with $\gamma(t)\,s^{vt}_{ij}$ (Eq. 10), where $\gamma(t)=\gamma_0>1$ for early denoising steps $t\ge t_{thresh}$ and $\gamma(t)=1$ afterwards (Eq. 11). With LoRA fine-tuning to restore the real-image distribution, the paper reports consistent gains on T2I-CompBench: for FLUX.1-Dev, relative gains of 16.4% in spatial relationships and 5.9% in shape, and for SD3.5-Medium, 28.3% in spatial relationships and 2.9% in shape.

Load-bearing premise

The argument assumes that the unnormalized visual-text and visual-visual logits have comparable magnitudes, so the larger number of visual tokens mechanically pushes down cross-modal attention; if the pretrained network has learned larger visual-text logits that compensate, the proposed suppression mechanism would not be the true cause of misalignment.

Editorial extensions

If this is right

  • If TACA's diagnosis is right, any future MM-DiT model can improve prompt adherence by rebalancing cross-modal logits rather than by adding cross-attention modules or changing the tokenizer.
  • The gains are largest on spatial relationships and attribute binding, the exact cases where missing or misbound objects appear in baseline samples.
  • Because $\gamma(t)$ is active only in the early denoising steps, the method does not slow down the full sampling loop much; the selective recomposition implementation runs a 30-step 1024$\times$1024 generation in about 16 seconds versus 14 seconds unmodified.
  • TACA is complementary to classifier-free guidance: the ablations show it improves scores at multiple CFG scales, including the default scales of both tested models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same token-imbalance argument should apply more strongly to video MM-DiTs, where the visual token count is much larger relative to text; the paper notes training-free text-to-video experiments showed gains but LoRA training diluted them, so a video-specific schedule for $\gamma(t)$ is a natural extension.
  • A per-layer or per-head temperature, instead of one global $\gamma_0$, could let the model boost exactly the heads whose cross-modal attention is most suppressed, at the cost of a few more hyperparameters.
  • One can test the mechanism directly: measure the mean of unnormalized logits $s^{vt}$ versus $s^{vv}$ in a frozen pretrained model; if $s^{vt}$ already compensates for the token count, the suppression premise would fail and the success of TACA would be heuristic.
  • A calibration rule could replace grid search: set $\gamma_0$ proportional to the observed ratio $N_{vis}/N_{txt}$ or to the measured attention suppression at early steps.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper identifies two alleged defects in the attention mechanism of Multimodal Diffusion Transformers (MM-DiTs): suppression of cross-modal attention caused by the larger number of visual tokens relative to text tokens, and lack of timestep-aware weighting of cross-modal interactions. To address these, it proposes Temperature-Adjusted Cross-modal Attention (TACA), which multiplies visual-text attention logits by a timestep-dependent temperature factor gamma(t) (with gamma(t)=gamma0 for early denoising steps and 1 otherwise), optionally combined with LoRA fine-tuning to suppress artifacts. The method is evaluated on T2I-CompBench for FLUX.1-Dev and SD3.5-Medium, with ablations on gamma0 and tthresh, image-quality metrics, and a user study. The central claim is that TACA rebalances cross-modal attention and significantly improves text-image alignment with minimal computational overhead.

Significance. If the mechanism claims were fully supported, TACA would be an attractive practical contribution: it adds no trainable parameters, requires only a few lines of code around the attention computation, is compatible with two current state-of-the-art MM-DiT families, and the authors provide public code and implementation details (including a speed comparison in the supplementary material). The main tables show consistent gains on several T2I-CompBench attribute categories, and the qualitative examples are often compelling. However, the paper's causal story, which is the basis for its title and framing, rests on an unverified assumption about the scale of unnormalized attention logits, and the benchmark evaluation suffers from selection-on-evaluation and a LoRA confound. The empirical effects are therefore not yet convincingly attributable to the proposed rebalancing mechanism, although the method may still work as a heuristic.

major comments (3)
  1. [Section 3.2, Eqs. 7-9] The suppression mechanism is not yet supported by direct evidence. The argument that N_vis >> N_txt suppresses P_vis-txt assumes that the unnormalized logits s_vt and s_vv are of comparable scale, but the paper never measures these logits; Fig. 4 shows only the resulting attention mass averaged over 50 samples, which conflates the token-count effect with any learned rescaling of s_vt. If the pretrained network has compensated by learning larger s_vt values, then the low text-attention mass in the baseline MM-DiT is an intended allocation, and TACA's gains would be a heuristic reweighting rather than a correction of a diagnosed defect. Please report direct statistics of s_vt and s_vv (e.g., mean and variance per layer and per timestep group) and, ideally, a controlled experiment that varies the token counts while holding the semantic content fixed. In addition, the comparison to 'typical cross-attention' in Fig. 4 is not a controlled baseline: in MM-DiT a visual token must attend to all visual keys, so a lower text-attention mass than a pure text-key softmax is expected and may be optimal rather than pathological.
  2. [Section 4.3, Table 4 and Section 4.2, Table 1] The hyperparameters gamma0=1.2 and tthresh=970 are selected by ablation on T2I-CompBench prompts (Table 4, 100 prompts per attribute) and then evaluated on the same benchmark in Table 1. This selection-on-evaluation setup can inflate the reported gains; a hold-out split or nested evaluation is needed. In addition, Table 1 reports single-seed scores without error bars or significance tests, and several differences are very small (e.g., FLUX.1-Dev Non-Spatial 0.3035 to 0.3041 for r=64; SD3.5-Medium Complex 0.4441 to 0.4470 for r=64). Please provide multiple seeds or bootstrapped confidence intervals for the main quantitative claims.
  3. [Section 4.2, Table 1 and Section 4.3, Table 4] The benchmark improvements are reported for TACA combined with LoRA, not for TACA alone. The abstract and introduction attribute the gains to TACA, but the only training-free evidence is qualitative (Figs. 5 and 8), and the only LoRA-only quantitative result is on the 100-prompt subset in Table 4. Consequently, the specific contribution of the temperature scaling over LoRA fine-tuning is not isolated on the full benchmark; note that the LoRA-only baseline in Table 4 already improves Color and Texture over the frozen model. Please report full-benchmark results for training-free TACA, LoRA-only, and TACA+LoRA under the same evaluation protocol.
minor comments (6)
  1. [Section 4.1] Please clarify whether the 10K LAION image-text pairs used for LoRA training overlap with the T2I-CompBench evaluation prompts; the fixed seed and default Diffusers settings are helpful but do not rule out distributional leakage.
  2. [Fig. 3 and Section 3.2] The timestep-insensitivity claim is motivated by a single qualitative example; a quantitative curve such as CLIP similarity or attention mass per denoising step, as in Fig. 8b, would make the claim more convincing.
  3. [Supplementary Section B.3] The reference to 'Fig. ??' for the long-prompt results is unresolved; please fix the cross-reference.
  4. [References, [24]] The LLaVA reference cited in Section 4.1 is missing the year and venue; please update the entry.
  5. [Table 3, User Study] The user study reports percentages but not raw counts or confidence intervals; with 25 prompts and 50 participants, a binomial confidence interval would help assess the strength of the preference.
  6. [Table 2, Image Quality] Some image-quality differences are at the fourth decimal (e.g., FLUX MUSIQ 0.7186 to 0.7212); please state whether these differences are within the metrics' run-to-run variability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TACA's derivation and evaluation are self-contained; self-citations are contextual, and the main weakness is evidentiary, not circular.

full rationale

The paper does not exhibit a circular derivation chain. Its central diagnostic argument in Eqs. 7-9 is an internal softmax analysis that compares unified MM-DiT attention to a hypothetical typical cross-attention denominator; the conclusion that token imbalance suppresses cross-modal attention is an empirical hypothesis, and the supporting evidence in Fig. 4 reports attention mass rather than raw logits. That is a limitation in support, not a reduction of the conclusion to its inputs. The proposed fix, temperature scaling in Eq. 10 with the timestep schedule in Eq. 11, is an explicit architectural modification, not a fitted parameter renamed as a prediction. The values gamma0=1.2 and tthresh=970 are selected by ablation in Section 4.3 and then used in the main comparison; evaluating on the same benchmark after tuning is a selection-on-evaluation concern, not a circularity under the stated definitions, because the benchmark result is not mathematically forced by the construction. The self-citations in the related-work section (e.g., refs. [26], [46], [47], [52]) are contextual and do not carry any load-bearing step of the argument. The LoRA training objective in Eq. 13 is a standard velocity-prediction loss. The paper's claim that TACA improves T2I-CompBench is an external empirical claim, and its mechanism story may be under-supported, but the derivation does not assume what it purports to show.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method adds no learned parameters, but the two scalars gamma0 and tthresh are hand-set based on the evaluation benchmark. The LoRA configuration is a standard hyperparameter. No new entities are introduced. The paper's non-circular core is the analytic observation that token imbalance scales softmax denominators; everything after that is an empirical device.

free parameters (4)
  • gamma0 (base temperature scaling factor) = 1.2
    Base temperature scale for cross-modal logits, selected by ablation on 100 prompts per attribute from T2I-CompBench (Table 4).
  • tthresh (timestep threshold) = 970
    Timestep above which gamma applies; chosen to cover the first 3 of 30 sampling steps; robustness tested in Table 5.
  • LoRA rank r = 64 (also 16 tested)
    LoRA rank in Eq. 12; both r=16 and r=64 evaluated, r=64 generally better for attribute and spatial metrics.
  • LoRA scaling alpha = 64 (matching r)
    LoRA scale hyperparameter, set equal to rank as is common practice; affects the strength of the low-rank update.
assumptions (3)
  • standard math Attention in MM-DiT follows the softmax over concatenated visual and textual tokens (Eq. 4).
    Standard transformer attention, not an assumption specific to the paper.
  • domain assumption The early denoising steps (large t) require stronger text guidance than later steps.
    Motivates the step function gamma(t) in Eq. 11; supported by Fig. 3 but only a few qualitative examples.
  • domain assumption Unnormalized attention logits for visual-visual and visual-text pairs have comparable scale, so token count alone suppresses cross-modal attention.
    Core premise behind Eq. 7-9; not proven for the trained models, only illustrated with averages over 50 samples.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Cross-Modal Interaction in Multimodal Diffusion Transformers." pith.science (2026). https://pith.science/paper/O34J6CFG

@misc{pith2026250607986,
  author       = {Pith},
  title        = {Pith review of: Rethinking Cross-Modal Interaction in Multimodal Diffusion Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O34J6CFG}},
  note         = {Machine review of arXiv:2506.07986}
}
read the original abstract

Multimodal Diffusion Transformers (MM-DiTs) have achieved remarkable progress in text-driven visual generation. However, even state-of-the-art MM-DiT models like FLUX struggle with achieving precise alignment between text prompts and generated content. We identify two key issues in the attention mechanism of MM-DiT, namely 1) the suppression of cross-modal attention due to token imbalance between visual and textual modalities and 2) the lack of timestep-aware attention weighting, which hinder the alignment. To address these issues, we propose \textbf{Temperature-Adjusted Cross-modal Attention (TACA)}, a parameter-efficient method that dynamically rebalances multimodal interactions through temperature scaling and timestep-dependent adjustment. When combined with LoRA fine-tuning, TACA significantly enhances text-image alignment on the T2I-CompBench benchmark with minimal computational overhead. We tested TACA on state-of-the-art models like FLUX and SD3.5, demonstrating its ability to improve image-text alignment in terms of object appearance, attribute binding, and spatial relationships. Our findings highlight the importance of balancing cross-modal attention in improving semantic fidelity in text-to-image diffusion models. Our codes are publicly available at \href{https://github.com/Vchitect/TACA}

Figures

Figures reproduced from arXiv: 2506.07986 by the authors.

Figure 1
Figure 1. We propose TACA, a parameter-efficient method that dynamically rebalances cross-modal attention in multimodal diffusion transformers to improve text-image alignment. Abstract Multimodal Diffusion Transformers (MM-DiTs) have achieved remarkable progress in text-driven visual gen￾eration. However, even state-of-the-art MM-DiT mod￾els like FLUX struggle with achieving precise alignment between text prompts and generate… view at source ↗
Figure 2
Figure 2. Object missing in text-to-image models. Even in [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The denoising process. This figure shows the predicted [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (13 more)
Figure 6
Figure 6. Figure 6: Attention map differences. We conducted a visualization [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 5
Figure 5. Figure 5: Temperature scaling helps text-image alignment. From [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 7
Figure 7. Figure 7: Comparison of samples generated by FLUX.1 Dev and Stable Diffusion 3.5 Medium with and without TACA. [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: The effect of LoRA training. (a) On the left, we show qualitative results comparing training-free image generation to generation [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Visual comparisons on text-image alignment (FLUX.1 Dev, short prompts) [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Visual comparisons on text-image alignment (FLUX.1 Dev, short prompts) [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Visual comparisons on text-image alignment (FLUX.1 Dev, short prompts) [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Visual comparisons on text-image alignment (FLUX.1 Dev, short prompts) [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Visual comparisons on text-image alignment (SD3.5 Medium, short prompts) [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: Visual comparisons on text-image alignment (SD3.5 Medium, short prompts) [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: Visual comparisons on text-image alignment (FLUX.1 Dev, long prompts) [PITH_FULL_IMAGE:figures/full_fig_p019_15.png]
Figure 16
Figure 16. Figure 16: Visual comparisons on text-image alignment (FLUX.1 Dev, long prompts) [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]
Figure 17
Figure 17. Figure 17: Visual comparisons on text-image alignment (FLUX.1 Dev, long prompts) [PITH_FULL_IMAGE:figures/full_fig_p021_17.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. General Incomplete Multimodal Learning via Dynamic Quality Perception

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A unified multimodal learning framework models modality degradation as a continuous variable and uses a noise-aware quality estimator to adaptively weight fused representations under both intra- and inter-modality mis...

  2. Prompt Reinjection: Alleviating Prompt Forgetting in Multimodal Diffusion Transformers for Text-to-Image Generation

    cs.CV 2026-02 unverdicted novelty 6.0 of 10

    Re-injecting shallow text features into deeper MMDiT blocks counteracts measured 'prompt forgetting' and improves instruction following in SD3, SD3.5, FLUX, and Qwen-Image without retraining.

  3. Detail++: Training-Free Detail Enhancer for T2I Diffusion Models

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Detail++ uses progressive multi-branch prompt injection and test-time attention optimization to improve attribute binding in text-to-image generation.

Reference graph

Works this paper leans on

52 extracted references · 22 canonical work pages · cited by 3 Pith papers

  1. [1]

    arXiv preprint arXiv:2010.11929, 2020

    An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 2

    arXiv 2010

  2. [2]

    Aishwarya Agarwal, Srikrishna Karanam, K. J. Joseph, Apoorv Saxena, Koustava Goswami, and Balaji Vasan Srini- vasan. A-star: Test-time attention segregation and retention for text-to-image synthesis. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 2283–2293,

    work page 2023

  3. [3]

    Training diffusion models with reinforce- ment learning

    Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforce- ment learning. arXiv preprint arXiv:2305.13301, 2023. 3

    arXiv 2023

  4. [4]

    Attend-and-excite: Attention-based se- mantic guidance for text-to-image diffusion models

    Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based se- mantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 42:1 – 10, 2023. 3

    work page 2023

  5. [5]

    Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James T. Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-α: Fast training of dif- fusion transformer for photorealistic text-to-image synthesis. The Twelfth International Conference on Learning Represen- tations, 2023. 1, 2, 4

    work page 2023

  6. [6]

    Training- free layout control with cross-attention guidance

    Minghao Chen, Iro Laina, and Andrea Vedaldi. Training- free layout control with cross-attention guidance. 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 5331–5341, 2023. 3

    work page 2024

  7. [7]

    Training-free layout control with cross- attention guidance

    Minghao Chen et al. Training-free layout control with cross- attention guidance. arXiv preprint arXiv:2304.03373, 2023. 3

    arXiv 2023

  8. [8]

    Be yourself: Bounded attention for multi-subject text-to-image generation

    Omer Dahary, Or Patashnik, Kfir Aberman, and Daniel Cohen-Or. Be yourself: Bounded attention for multi-subject text-to-image generation. arXiv preprint arXiv:2403.16990,

    arXiv

Show all 52 references
  1. [9]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. arXiv preprint arXiv:2105.05233,

    arXiv

  2. [10]

    Patrick Esser, Sumith Kulal, A. Blattmann, Rahim Entezari, Jonas Muller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dock- horn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow trans- fo...

    2024 arXiv

  3. [11]

    Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee

    Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, P. Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learn- ing for fine-tuning text-to-image diffusion models. arXiv preprint arXiv:2305.16381, 2023. 3

    2023 arXiv

  4. [12]

    Galatolo, Mario G.C.A

    Federico A. Galatolo, Mario G.C.A. Cimino, and Gigliola Vaglini. Generating images from caption and vice versa via clip-guided generative latent space search. arXiv preprint arXiv:2102.01645, 2021. 3

    2021 arXiv

  5. [13]

    Jonathan Ho, Ajay Jain, and P. Abbeel. Denoising diffu- sion probabilistic models. arXiv preprint arXiv:2006.11239,

    2006 arXiv

  6. [14]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868, 2022. 2

    2022 arXiv

  7. [15]

    Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen

    J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 2, 5, 8

    2021 arXiv

  8. [16]

    T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation

    Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xi- hui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. arXiv preprint arXiv:2307.06350, 2023. 2, 5, 6, 7

    2023 arXiv

  9. [17]

    Musiq: Multi-scale image quality transformer

    Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer. arXiv preprint arXiv:2108.05997, 2021. 7

    2021 arXiv

  10. [18]

    Dense text-to-image generation with attention modulation

    Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Woo Ha, and Jun-Yan Zhu. Dense text-to-image generation with attention modulation. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7667–7677, 2023. 3

    2023

  11. [19]

    Hunyuanvideo: A systematic framework for large video generative models

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024. 2

    2024 arXiv

  12. [20]

    Black Forest Labs. Flux. https://github.com/ black-forest-labs/flux, 2024. 2, 4, 6

    2024

  13. [21]

    Divide & bind your attention for improved generative seman- tic nursing

    Yumeng Li, Margret Keuper, Dan Zhang, and Anna Khoreva. Divide & bind your attention for improved generative seman- tic nursing. arXiv preprint arXiv:2307.10864, 2023. 3

    2023 arXiv

  14. [22]

    Gligen: Open-set grounded text-to-image generation

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. 2023 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 22511–22521, 2023. 3

    2023

  15. [23]

    Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maxi- milian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022. 7

    2022 arXiv

  16. [24]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485. 5

    arXiv

  17. [25]

    Fusedream: Training-free text-to-image generation with improved clip+gan space op- timization

    Xingchao Liu, Chengyue Gong, Lemeng Wu, Shujian Zhang, Hao Su, and Qiang Liu. Fusedream: Training-free text-to-image generation with improved clip+gan space op- timization. arXiv preprint arXiv:2112.01573, 2021. 3

    2021 arXiv

  18. [26]

    Smartcontrol: Enhancing controlnet for handling rough visual conditions

    Xiaoyu Liu, Yuxiang Wei, Ming Liu, Xianhui Lin, Peiran Ren, Xuansong Xie, and Wangmeng Zuo. Smartcontrol: Enhancing controlnet for handling rough visual conditions. In European Conference on Computer Vision , pages 1–17. Springer, 2024. 1

    2024

  19. [27]

    Conform: Contrast is all you need for high- fidelity text-to-image diffusion models

    Tuna Han Salih Meral, Enis Simsar, Federico Tombari, and Pinar Yanardag. Conform: Contrast is all you need for high- fidelity text-to-image diffusion models. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9005–9014, 2023. 3

    2024

  20. [28]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In Interna- tional Conference on Machine Learning, 2021. 1

    2021

  21. [29]

    Ai toolkit, 2025

    Ostris. Ai toolkit, 2025. 6

    2025

  22. [30]

    Peebles and Saining Xie

    William S. Peebles and Saining Xie. Scalable diffusion mod- els with transformers. 2023 IEEE/CVF International Confer- ence on Computer Vision (ICCV), 2022. 1, 2, 4

    2023

  23. [31]

    Courville

    Ethan Perez, Florian Strub, Harm de Vries, Vincent Du- moulin, and Aaron C. Courville. Film: Visual reasoning with a general conditioning layer. In AAAI Conference on Artificial Intelligence, 2017. 2

    2017

  24. [32]

    Grounded text-to-image synthesis with attention refocusing

    Quynh Phung, Songwei Ge, and Jia-Bin Huang. Grounded text-to-image synthesis with attention refocusing. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7932–7942, 2023. 3

    2024

  25. [33]

    Blattmann, Tim Dockhorn, Jonas Muller, Joe Penna, and Robin Rombach

    Dustin Podell, Zion English, Kyle Lacey, A. Blattmann, Tim Dockhorn, Jonas Muller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 1

    2023 arXiv

  26. [34]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Con...

    2021

  27. [35]

    Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J

    Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2019. 3

    2019

  28. [36]

    Linguistic bind- ing in diffusion models: Enhancing attribute correspon- dence through attention map alignment

    Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Rav- fogel, Yoav Goldberg, and Gal Chechik. Linguistic bind- ing in diffusion models: Enhancing attribute correspon- dence through attention map alignment. arXiv preprint arXiv:2306.08877, 2023. 3

    2023 arXiv

  29. [37]

    Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer

    Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10674–10685, 2021. 1

    2022

  30. [38]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. arXiv preprint arXiv:1505.04597, 2015. 1, 2, 3

    2015 arXiv

  31. [39]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. 5

    2021 arXiv

  32. [40]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 1, 2

    2010 arXiv

  33. [41]

    Stable diffusion 3.5, 2024

    Stability-AI. Stable diffusion 3.5, 2024. 2, 6

    2024

  34. [42]

    Dream- sync: Aligning text-to-image generation with image under- standing feedback

    Jiao Sun, Deqing Fu, Yushi Hu, Su Wang, Royi Rassin, Da-Cheng Juan, Dana Alon, Charles Herrmann, Sjoerd van Steenkiste, Ranjay Krishna, and Cyrus Rashtchian. Dream- sync: Aligning text-to-image generation with image under- standing feedback. arXiv preprint arXiv:2311.17946, 2023. 3

    2023 arXiv

  35. [43]

    Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N

    Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural Infor- mation Processing Systems, 2017. 2

    2017

  36. [44]

    Diffusers: State-of-the-art diffu- sion models

    Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffu- sion models. https://github.com/huggingface/ diffusers...

    2022

  37. [45]

    Tokencompose: Text-to-image diffusion with token-level supervision

    Zirui Wang, Zhizhou Sha, Zheng Ding, Yilin Wang, and Zhuowen Tu. Tokencompose: Text-to-image diffusion with token-level supervision. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8553–8564, 2023. 3

    2024

  38. [46]

    Elite: Encoding visual con- cepts into textual embeddings for customized text-to-image generation

    Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual con- cepts into textual embeddings for customized text-to-image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15943–15953, 2023. 1

    2023

  39. [47]

    Personalized image generation with deep generative models: A decade survey

    Yuxiang Wei, Yiheng Zheng, Yabo Zhang, Ming Liu, Zhi- long Ji, Lei Zhang, and Wangmeng Zuo. Personalized image generation with deep generative models: A decade survey. arXiv preprint arXiv:2502.13081, 2025. 1

    2025 arXiv

  40. [48]

    Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion

    Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wen- tian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7418–7427, 2023. 3

    2023

  41. [49]

    Mastering text-to-image diffu- sion: Recaptioning, planning, and generating with multi- modal llms

    Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Ste- fano Ermon, and Bin Cui. Mastering text-to-image diffu- sion: Recaptioning, planning, and generating with multi- modal llms. arXiv preprint arXiv:2401.11708, 2024. 3

    2024 arXiv

  42. [50]

    Maniqa: Multi-dimension attention network for no-reference image quality assessment

    Sidi Yang, Tianhe Wu, Shuwei Shi, Shanshan Lao, Yuan Gong, Mingdeng Cao, Jiahao Wang, and Yujiu Yang. Maniqa: Multi-dimension attention network for no-reference image quality assessment. arXiv preprint arXiv:2204.08958,

    arXiv

  43. [51]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 2

    2024 arXiv

  44. [52]

    sweet spot

    Yabo Zhang, Xinpeng Zhou, Yihan Zeng, Hang Xu, Hui Li, and Wangmeng Zuo. Framepainter: Endowing interactive image editing with video diffusion priors. arXiv preprint arXiv:2501.08225, 2025. 1 Rethinking Cross-Modal Interaction in Multimodal Diffusion Transformers Supplementary...

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.