Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

VMix: Improving Text-to-Image Diffusion Model with Cross-Attention Mixing Control

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VMix is a plug-and-play adapter that separates content from aesthetics in the prompt and injects fine-grained aesthetic conditions through value-mixed cross-attention, improving image beauty without sacrificing text alignment.

desk verdict A clean but small adapter for aesthetics in text-to-image models, with an evaluation that needs fixing before the headline numbers can be trusted. read the letter →

arxiv 2412.20800 v1 pith:GWZOLBXS submitted 2024-12-30 cs.CV

Shaojin Wu , Fei Ding , Mengqi Huang , Wei Liu , Qian He This is my paper · ORCID
classification cs.CV
keywords text-to-imagegenerationdiffusionmodelsaestheticqualitycross-attentionadapterplug-and-playStablepromptdisentanglement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VMix is a plug-and-play adapter for text-to-image diffusion models that aims to close the gap between generated images and real-world aesthetics. The paper argues that current text encoders capture high-level semantics but lack awareness of finer-grained visual qualities such as natural light, coherent color, and composition, so merely adding aesthetic words to prompts is insufficient. VMix splits the prompt into a content description and an aesthetic description, then feeds the aesthetic part through a value-mixed cross-attention branch that reuses the content attention map. Experiments report that VMix raises aesthetic scores on MJHQ-30K and a newly built LAION-HQ10K benchmark while keeping CLIP scores competitive, and that it can be attached to personalized community models without retraining.

What carries the argument

The load-bearing mechanism is value-mixed cross-attention, a dual-branch cross-attention in which both branches share the query and key computed from the content text, so both use the same attention map and only the value source differs. This lets the model apply aesthetic guidance without re-weighting which text tokens each image patch attends to, protecting image-text alignment. A second component is the aesthetic embedding initialization (AesEmb): paired positive and negative aesthetic labels are passed through a frozen CLIP text encoder, and their [CLS] tokens are concatenated to form a fixed lookup table; a projection layer with a zero-initialized final linear upscales the selected token subset to match the content embedding's token dimension.

What would settle it

Generate images for a set of concrete, non-anthropomorphic objects (cups, phones, furniture) with all aesthetic labels active and measure object-class fidelity and aesthetic score; the paper's own limitation note predicts entity bias toward humans, so a sharp drop in object fidelity or an unexpected entity switch would place a boundary on the method's generality.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that fine-grained aesthetic conditions can be injected into a frozen diffusion U-Net through a modification of cross-attention in which the aesthetic branch computes Attention(Q, K_c, V_a) while sharing the query and key of the content branch. Because the attention map QK_c^T is identical for both branches, the base model's layout and text-binding decisions are preserved, and the only new parameters per attention layer are the value projection W_Va and a projection layer that maps a set of aesthetic label tokens to the same feature space as the content embedding. The aesthetic embeddings are initialized once by encoding paired positive/negative labels with a frozen CLIP model, and a zero-initialized linear layer keeps the initial injection harmless. With LoRA used during training and discarded at inference, the trained module can be lifted out and attached to other base models and community modules.

Load-bearing premise

The load-bearing premise is that the aesthetic predictor used for the AES score and the LAION-HQ10K test images represent genuine human aesthetic preferences, and that the curated 200k training set does not overlap the test set in distribution; if the scorer is gameable or the test set resembles training images, the measured gains may not transfer to other images.

Editorial extensions

If this is right

  • The adapter adds only a small number of parameters (one value projection per cross-attention layer plus a projection layer), so it can be trained on a modest budget and used as a drop-in module.
  • Because the attention maps are untouched, VMix output preserves the base model's spatial layout and composition, which is why it combines with ControlNet, IP-Adapter, and LoRA without conflicts.
  • The inference-time hyperparameter lambda offers a continuous trade-off between aesthetic strength and text fidelity, letting users tune the effect per prompt.
  • The method transfers across SD1.5 and SDXL with the same training recipe, suggesting the cross-attention control is architecture-agnostic within the U-Net family.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own limitation section points out that the closed set of aesthetic labels and label-entity correlations (emotional labels binding to humans or animals) could bias generation; a natural extension is to learn open-set or disentangled label embeddings that suppress entity leakage.
  • The shared-attention value-mixing trick is a general condition-injection primitive; it could be applied to other guidance signals such as style, lighting, or camera parameters, or to video diffusion models with cross-frame attention.
  • The quantitative evaluation relies heavily on an aesthetic scorer and a test set built from the same LAION distribution as training; a stronger test would use out-of-distribution aesthetic images and human preference judgments across varied prompts.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces VMix, a plug-and-play aesthetic adapter for text-to-image diffusion models (SD1.5 and SDXL). The method disentangles the input prompt into a content description and a set of fine-grained aesthetic labels, maps these labels to CLIP [CLS] tokens (AesEmb), and injects them into the U-Net via a value-mixed cross-attention branch that shares the query and key (and thus the attention map) with the content branch, while using a newly learned value projection. Only a projection layer, LoRA, and the extra value matrix are trained, and zero-initialized linear layers are used for stable integration. The paper reports qualitative and quantitative evaluations against FreeU, DPO, textual inversion, and SFT on MJHQ-30K and a new LAION-HQ10K benchmark, plus compatibility demonstrations with ControlNet, IP-Adapter, and community models.

Significance. The central idea is timely and practically valuable: a lightweight, plug-and-play adapter that improves fine-grained aesthetics without retraining the base model is directly useful for the open-source ecosystem. The paper's design is clearly specified, and the ablations (Table 3, Figs. 7-8) plausibly isolate the contributions of the two core components. The compatibility experiments with community models are a strength. However, the quantitative case is not yet fully persuasive: the main tables omit the inference-time λ, the LAION-HQ10K benchmark shares its source corpus with the training set, and the user study is reported only qualitatively. These gaps are addressable, but they are load-bearing for the claims of state-of-the-art aesthetic performance and lossless alignment.

major comments (4)
  1. [§3.3, §4.4, Tables 1-2] The value of the inference-time hyperparameter λ is not reported for the headline results in Tables 1 and 2. Section 3.3 sets λ=1 during training, but Section 4.4 and Fig. 7 show that λ is adjustable at inference and that increasing λ from 1 to 2 raises AES while lowering CLIP. If the numbers in Tables 1 and 2 were obtained at a λ selected from this favorable end of the sweep, the comparison to baselines (which use their default settings) would be unfair, and the claim of retaining alignment would be weaker than presented. Please state the exact λ used for every quantitative comparison, and preferably fix λ=1 for the main tables, or show the results as a function of λ with the baseline ranges.
  2. [§4.1, Tables 1-2] The LAION-HQ10K benchmark is created from the LAION collection, the same source corpus from which the 200k training subset is curated. Without a deduplication or overlap analysis, the AES and FID gains on this benchmark may partly reflect the model's proximity to its training distribution rather than a general aesthetic improvement. Please report the overlap (e.g., exact or near-duplicate image count) between the training subset and LAION-HQ10K, or evaluate on a held-out source, and indicate how the reported numbers change after removing overlapping images.
  3. [§4.2, Fig. 6] The user study is described (20 evaluators, 300 prompts) but no numerical outcome is reported; the text says results are 'As shown in Fig. 5' (which is actually a qualitative comparison of personalized models) and Fig. 6 only shows a bar chart without percentages, error bars, or a significance test. Because the AES score is a learned proxy, the human study is the main direct evidence for the aesthetic claim. Please report the preference rates for each comparison pair, the inter-rater variability, and a statistical test (e.g., Wilcoxon signed-rank test) to support the claim that VMix is preferred.
  4. [Tables 1-3] All quantitative metrics are single-run point estimates without error bars or significance tests. The reported differences are sometimes small (e.g., Table 1: CLIP 30.50 vs. 30.24; AES 5.79 vs. 5.54), so it is important to show variance across multiple seeds or sampling runs, especially since sampling is stochastic. Please include means and standard deviations over at least three seeds, or otherwise justify that the differences are beyond sampling noise.
minor comments (5)
  1. [Fig. 2 caption] Typo: 'Illustration of of VMix' should be 'Illustration of VMix'.
  2. [Tables 1-2] The citation for DPO is given as [8] in the tables, but DPO refers to Wallace et al. [27]; [8] is the LoRA reference. Please correct the citation.
  3. [§4.1, Evaluation Metrics] Typo: 'We assess our the performance' should be 'We assess the performance'.
  4. [§4.2, User study] The sentence 'As shown in Fig. 5, the results indicate...' refers to the user study results, but Fig. 5 shows qualitative examples of personalized models; the correct reference appears to be Fig. 6.
  5. [§6.3, Limitations] The limitation about emotional labels becoming bound to specific entities (e.g., humans) is an important caveat and should be mentioned earlier, possibly in the main text, since it affects the 'all positive aesthetic labels' inference default.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the core adapter derivation is self-contained and the evaluation concerns are transparency issues rather than circular reductions.

full rationale

No load-bearing circular step could be exhibited. VMix's mechanism is defined by external inputs (aesthetic labels) and standard operations: the training loss in Eq. (1) is the ordinary diffusion noise-prediction objective, the projection layer is a zero-initialized linear map in Eq. (6), and the value-mixed cross-attention in Eqs. (7)-(9) is a new combination of a shared attention map and a separately learned value branch. None of these equations is defined in terms of the reported AES/CLIP/FID results, and no fitted parameter is renamed as a prediction. The AES predictor is external, the aesthetic labels are human-defined text categories rather than outputs of the model, and the paper does not optimize the AES objective during training. The training-data filtering and the LAION-HQ10K benchmark both originate from LAION and both use aesthetic scoring, which is a legitimate benchmark-distribution concern, but the same benchmark is applied to all baselines and the relative comparison with SFT on the same curated data is not forced by construction. The inference-time lambda is swept and disclosed in Figure 7, though the lambda used in Tables 1 and 2 is not reported; that is an evaluation-transparency weakness, not a circular derivation. There are also no self-citations by the authors carrying a load-bearing argument. Therefore the central technical claim is self-contained and the paper is not circular.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or conceptual entities. Its free parameters are the inference-time mixing weight lambda, the hand-defined aesthetic label set, and the learned weights of the projection and value heads. The axioms are the disentanglement assumption, the shared-attention-map preservation claim, the zero-init safety heuristic, and the validity of the AES evaluation proxy.

free parameters (2)
  • lambda (lambda) = 1 during training; adjustable 1-2 at inference (Fig. 7)
    Balances the aesthetic cross-attention branch against the content branch. The paper reports a sweep over lambda (1 to 2) and does not state which lambda produced the main table results, so it is a hand- or eval-chosen scale.
  • aesthetic label set and count N = unspecified; examples include color, lighting, composition, focus, emotion
    The closed set of aesthetic labels is hand-defined (Section 3.2 and limitations) and determines the AesEmb size. The limitations section acknowledges the set is closed and may not cover all aesthetic dimensions.
assumptions (4)
  • domain assumption Style and content can be disentangled in text-to-image diffusion models, so aesthetic labels can be encoded as independent [CLS] tokens (Section 3.2, citing [29]).
    The entire design separates the prompt into content and aesthetic streams. If CLIP [CLS] tokens of short aesthetic labels carry content semantics, the aesthetic branch could corrupt subject identity.
  • domain assumption Sharing the cross-attention map Q Kc^T between content and aesthetic branches preserves text-image alignment (Section 3.3, Eq. 9, and Supplementary Fig. 10).
    The method claims no harm to text fidelity because the attention map is unchanged. This is plausible but demonstrated only on a few qualitative examples.
  • domain assumption Zero-initialized projection layers prevent harmful perturbation of the pre-trained model at the start of training (Section 3.3, following ControlNet [36]).
    This is a known heuristic from ControlNet, and the paper relies on it to claim plug-and-play preservation of base model capabilities.
  • domain assumption The external aesthetic predictor (improved-aesthetic-predictor) is a valid and sufficient measure of human aesthetic quality (Section 4.1 evaluation metrics).
    The AES score is a learned proxy, not a direct human judgment, and the reported user study is small and lacks statistical detail.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VMix: Improving Text-to-Image Diffusion Model with Cross-Attention Mixing Control." pith.science (2026). https://pith.science/paper/GWZOLBXS

@misc{pith2026241220800,
  author       = {Pith},
  title        = {Pith review of: VMix: Improving Text-to-Image Diffusion Model with Cross-Attention Mixing Control},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GWZOLBXS}},
  note         = {Machine review of arXiv:2412.20800}
}
read the original abstract

While diffusion models show extraordinary talents in text-to-image generation, they may still fail to generate highly aesthetic images. More specifically, there is still a gap between the generated images and the real-world aesthetic images in finer-grained dimensions including color, lighting, composition, etc. In this paper, we propose Cross-Attention Value Mixing Control (VMix) Adapter, a plug-and-play aesthetics adapter, to upgrade the quality of generated images while maintaining generality across visual concepts by (1) disentangling the input text prompt into the content description and aesthetic description by the initialization of aesthetic embedding, and (2) integrating aesthetic conditions into the denoising process through value-mixed cross-attention, with the network connected by zero-initialized linear layers. Our key insight is to enhance the aesthetic presentation of existing diffusion models by designing a superior condition control method, all while preserving the image-text alignment. Through our meticulous design, VMix is flexible enough to be applied to community models for better visual performance without retraining. To validate the effectiveness of our method, we conducted extensive experiments, showing that VMix outperforms other state-of-the-art methods and is compatible with other community modules (e.g., LoRA, ControlNet, and IPAdapter) for image generation. The project page is https://vmix-diffusion.github.io/VMix/.

Figures

Figures reproduced from arXiv: 2412.20800 by the authors.

Figure 1
Figure 1. Comparison of text fidelity and visual aesthetics be [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of of VMix. (a)In the initialization stage, pre-defined aesthetic labels are transformed into [CLS] tokens through [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Qualitative comparison with various state-of-the-art methods. All results are based on Stable Diffusion [ [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison with various state-of-the-art methods. All the results of the methods are based on the SDXL [ [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 8
Figure 8. Figure 8: Ablation Study for AesEmb of VMix. Left: The effects of using all aesthetic labels versus not using them. Right: The effects of using single-dimensional aesthetic labels. thereby enhancing the overall aesthetic presentation of the images. 4.4. Ablation Study Effect of …
Figure 6
Figure 6. Figure 6: User study. We report the user preference between using [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Ablation Study for λ of VMix. (a)Visual performance changes of λ. (b)Performance metrics for VMix, evaluated across a range of λ values from 1 to 2 from right to left. the incorporation of aesthetic embeddings has not detracted from the model’s inherent capabilities. O…
Figure 9
Figure 9. Figure 9: Visualization results of different training steps. Prompts: [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: Visualization results of attention maps. VMix is ca [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 11
Figure 11. Figure 11: Qualitative comparison. Prompts: (1) Kitten in the for [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: Qualitative results about VMix with ControlNet[ [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]
Figure 13
Figure 13. Figure 13: Qualitative comparison between results with VMix(on the right) and without VMix(on the left), shows that VMix significantly [PITH_FULL_IMAGE:figures/full_fig_p013_13.png]
Figure 14
Figure 14. Figure 14: Qualitative comparison between results with VMix(on the right) and without VMix(on the left), shows that VMix significantly [PITH_FULL_IMAGE:figures/full_fig_p014_14.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Hierarchical Anti-Aesthetics: Protecting Facial Privacy against Customized Diffusion Models

    cs.CV 2026-07 unverdicted novelty 6.0 of 10

    Hierarchical anti-aesthetic adversarial noise, guided by global and face-local preference reward models, degrades customized diffusion outputs and reduces facial identity leakage more than prior cloaking methods.

  2. USO: Unified Style and Subject-Driven Generation via Disentangled and Reward Learning

    cs.CV 2025-08 conditional novelty 6.0 of 10

    USO trains one DiT model for subject-driven, style-driven, and joint generation by disentangling content and style from triplet data and adding a style-reward objective, claiming SOTA on USO-Bench.

  3. IQA-Adapter: Exploring Knowledge Transfer from Image Quality Assessment to Diffusion-based Generative Models

    cs.CV 2024-12 conditional novelty 6.0 of 10

    An adapter module trained on IQA/IAA scores gives SDXL controllable quality-aware generation, improving perceived quality and enabling reference-based distortion transfer.

  4. Hunyuan-Game: Industrial-grade Intelligent Game Creation Model

    cs.CV 2025-05 reject novelty 4.0 of 10

    Tencent's Hunyuan-Game applies diffusion transformers to game asset creation across nine image and video generation tasks, with self-reported gains that are partly contradicted by its own evaluation table.

Reference graph

Works this paper leans on

37 extracted references · 16 canonical work pages · cited by 4 Pith papers

  1. [1]

    Layer normalization

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- ton. Layer normalization. arXiv preprint arXiv:1607.06450,

    arXiv

  2. [2]

    Attend-and-excite: Attention-based se- mantic guidance for text-to-image diffusion models

    Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based se- mantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 42(4):1–10, 2023. 5

    work page 2023

  3. [3]

    Emu: Enhanc- ing image generation models using photogenic needles in a haystack

    Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xi- aofang Wang, Abhimanyu Dubey, et al. Emu: Enhanc- ing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.15807, 2023. 1, 2, 7

    arXiv 2023

  4. [4]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learn- ing, 2024. 2

    2024

  5. [5]

    Ernie-vilg 2.0: Improving text-to- image diffusion model with knowledge-enhanced mixture- of-denoising-experts

    Zhida Feng, Zhenyu Zhang, Xintong Yu, Yewei Fang, Lanxin Li, Xuyi Chen, Yuxiang Lu, Jiaxiang Liu, Weichong Yin, Shikun Feng, et al. Ernie-vilg 2.0: Improving text-to- image diffusion model with knowledge-enhanced mixture- of-denoising-experts. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 10135–10145, 2023. 1

    work page 2023

  6. [6]

    An image is worth one word: Personalizing text-to- image generation using textual inversion

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patash- nik, Amit H Bermano, Gal Chechik, and Daniel Cohen- Or. An image is worth one word: Personalizing text-to- image generation using textual inversion. arXiv preprint arXiv:2208.01618, 2022. 3, 7

    arXiv 2022

  7. [7]

    Freestyle: Free lunch for text- guided style transfer using diffusion models

    Feihong He, Gang Li, Mengyuan Zhang, Leilei Yan, Lingyu Si, and Fanzhang Li. Freestyle: Free lunch for text- guided style transfer using diffusion models. arXiv preprint arXiv:2401.15636, 2024. 1

    arXiv 2024

  8. [8]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 2, 6, 7

    arXiv 2021

Show all 37 references
  1. [9]

    Pick-a-pic: An open dataset of user preferences for text-to-image generation

    Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Ma- tiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. arXiv preprint arXiv:2305.01569, 2023. 1

    2023 arXiv

  2. [10]

    Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image genera- tion, 2024

    Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Lin- miao Xu, and Suhail Doshi. Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image genera- tion, 2024. 7

    2024

  3. [11]

    Rich human feedback for text-to-image generation

    Youwei Liang, Junfeng He, Gang Li, Peizhao Li, Arseniy Klimovskiy, Nicholas Carolan, Jiao Sun, Jordi Pont-Tuset, Sarah Young, Feng Yang, et al. Rich human feedback for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio...

    2024

  4. [12]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 6

    2017 arXiv

  5. [13]

    T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhon- gang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453, 2023. 3

    2023 arXiv

  6. [14]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models.arXiv preprint arXiv:2112.10741, 2021

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models.arXiv preprint arXiv:2112.10741, 2021. 1

    2021 arXiv

  7. [15]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 1, 2, 6

    2023 arXiv

  8. [16]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, pages 8748–8763. PMLR, 2021. 2, 3, 4

    2021

  9. [17]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 2

    2020

  10. [18]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Confer- ence on Machine Learning, pages 8821–8831. PMLR, 2021. 2

    2021

  11. [19]

    Hierarchical text-conditional image gener- ation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gener- ation with clip latents. arXiv preprint arXiv:2204.06125, 1 (2):3, 2022. 1

    2022 arXiv

  12. [20]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1, 2, 3, 4, 5, 7

    2022

  13. [21]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234–241. Springer, 2015. 2, 3

    2015

  14. [22]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500...

    2023

  15. [23]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. 7

    2021 arXiv

  16. [24]

    Freeu: Free lunch in diffusion u-net

    Chenyang Si, Ziqi Huang, Yuming Jiang, and Ziwei Liu. Freeu: Free lunch in diffusion u-net. arXiv preprint arXiv:2309.11497, 2023. 1, 2, 7

    2023 arXiv

  17. [25]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 6

    2010 arXiv

  18. [26]

    Df-gan: A simple and effec- tive baseline for text-to-image synthesis

    Ming Tao, Hao Tang, Fei Wu, Xiao-Yuan Jing, Bing-Kun Bao, and Changsheng Xu. Df-gan: A simple and effec- tive baseline for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16515–16525, 2022. 2

    2022

  19. [27]

    Diffusion model align- ment using direct preference optimization

    Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model align- ment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision a...

    2024

  20. [28]

    Elite: Encoding visual con- cepts into textual embeddings for customized text-to-image generation

    Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual con- cepts into textual embeddings for customized text-to-image generation. arXiv preprint arXiv:2302.13848, 2023. 3

    2023 arXiv

  21. [29]

    Uncovering the disentanglement capability in text- to-image diffusion models

    Qiucheng Wu, Yujian Liu, Handong Zhao, Ajinkya Kale, Trung Bui, Tong Yu, Zhe Lin, Yang Zhang, and Shiyu Chang. Uncovering the disentanglement capability in text- to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , p...

    1900

  22. [30]

    Imagere- ward: Learning and evaluating human preferences for text- to-image generation

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagere- ward: Learning and evaluating human preferences for text- to-image generation. Advances in Neural Information Pro- cessing Systems, 36, 2024. 1

    2024

  23. [31]

    Attngan: Fine- grained text to image generation with attentional generative adversarial networks

    Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine- grained text to image generation with attentional generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , page...

    2018

  24. [32]

    Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models. arXiv preprint arXiv:2308.06721,

    arXiv

  25. [33]

    Scaling autoregres- sive models for content-rich text-to-image generation

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gun- jan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yin- fei Yang, Burcu Karagol Ayan, et al. Scaling autoregres- sive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2(3):5, 2022. 2

    2022 arXiv

  26. [34]

    Scaling autoregressive multi- modal models: Pretraining and instruction tuning

    Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive multi- modal models: Pretraining and instruction tuning. arXiv preprint arXiv:2309.02591, 2023. 2

    2023 arXiv

  27. [35]

    Cross-modal contrastive learning for text-to- image generation

    Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modal contrastive learning for text-to- image generation. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 833–842, 2021. 2

    2021

  28. [36]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 2, 3, 4, 1 10

    2023

  29. [37]

    More Qualitative Comparison VMix decouples aesthetic knowledge from content knowl- edge and introduces a novel conditional control method

    Supplementary 6.1. More Qualitative Comparison VMix decouples aesthetic knowledge from content knowl- edge and introduces a novel conditional control method. To further verify its effectiveness, we provide additional exper- imental results here. Training Stability. In Sec. 3, ...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.