Pith. sign in

REVIEW 4 major objections 4 minor 49 references

DreamBlend: Advancing Personalized Fine-tuning of Text-to-Image Diffusion Models

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DreamBlend claims that cross-attention guidance at inference time lets an early underfit checkpoint and a later overfit checkpoint jointly produce images with higher subject fidelity, prompt fidelity, and diversity than either checkpoint…

desk verdict A neat inference-time trick with a real human-preference win, but the quantitative attribution to the attention guidance is under-supported because the α=0 control is missing. read the letter →

arxiv 2411.19390 v1 pith:RQG2FDW7 submitted 2024-11-28 cs.CV cs.GRcs.LG

Shwetha Ram , Tal Neiman , Qianli Feng , Andrew Stuart , Son Tran , Trishul Chilimbi This is my paper · ORCID
classification cs.CVcs.GRcs.LG
keywords text-to-imagediffusionpersonalizedimagegenerationcross-attentionguidancefine-tuningtrade-offcatastrophicattentioncollapsesubjectfidelitypromptDDIM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the familiar trade-off in personalized text-to-image generation—early fine-tuning checkpoints follow prompts but not the subject, while later checkpoints capture the subject but forget prompts—can be sidestepped rather than accepted. DreamBlend's claim is that by using an early checkpoint's cross-attention maps as a layout guide while denoising with a later checkpoint, one image can have high subject fidelity, high prompt fidelity, and high diversity simultaneously. If true, this is an inference-only fix: no change to how models are fine-tuned, only an extra guided generation step at test time. The paper demonstrates this on standard personalized-generation benchmarks and reports improved subject- and prompt-fidelity metrics as well as human preference over existing fine-tuning approaches.

What carries the argument

The central object is the text-image cross-attention map, $A = \mathrm{softmax}(QK^T/\sqrt{d})$, which records how much each text token attends to each image patch, and the observation that these maps collapse onto the subject in overfit checkpoints. The mechanism is Algorithm 1: generate a reference image with the guidance model, storing $A_{\mathrm{ref}}[t]$ at every DDIM step; then, with the edit model, at each step compute $R = |A_{\mathrm{ref}}[t] - A_{\mathrm{current}}|$ and update the latent as $l \leftarrow l - \alpha \nabla_l R$ before the DDIM update. The scalar $\alpha$ controls how strongly layout is transferred, and the same initial latent for both models makes the attention maps directly comparable.

What would settle it

Take a subject and prompt from the paper's benchmark, fix the guidance and edit checkpoints, and compare $\alpha = 0$ (edit model alone) against $\alpha > 0$ over the same initial latents; if no value of $\alpha$ improves prompt fidelity while keeping subject fidelity at least as high as the edit model alone, the central claim that cross-attention guidance is what resolves the trade-off is false. A second check is whether attention maps at later fine-tuning steps really collapse: if token attention entropy stays flat as training proceeds, the motivating phenomenon is absent.

Watch

Extended reading notes

Core claim

As a later checkpoint overfits, the attention of every text token gradually concentrates on the subject—a pattern the paper calls catastrophic attention collapse—and this is what erases prompt fidelity and diversity. DreamBlend records the cross-attention maps produced by an early checkpoint while generating a reference image, then generates the final image from a later checkpoint using the same initial latent, at each DDIM step updating the latent to reduce the L1 difference between the later checkpoint's attention maps and the recorded reference maps. The paper claims this produces images that keep the early image's layout and prompt adherence while gaining the later checkpoint's subject identity, advancing the Pareto front of image-text alignment.

Load-bearing premise

The claimed gain depends on attention maps recorded from an early checkpoint actually transferring to a later checkpoint when the later model's denoising is nudged to match them, without the nudge eroding the subject identity the later checkpoint has learned.

Editorial extensions

If this is right

  • Practitioners can improve personalized generation without retraining or changing the fine-tuning loss, by keeping an early checkpoint and running guided synthesis at inference.
  • The best operating point for generation is no longer a single checkpoint; pairs of early and late checkpoints form a new search space, and the paper reports that dense sampling of this space advances the Pareto frontier.
  • Because the guidance operates on cross-attention, it transfers to any diffusion model with cross-attention and to several fine-tuning methods, including full fine-tuning, low-rank adapters, and attention-only fine-tuning.
  • The paper's human preference studies find the method preferred over both fine-tuning baselines on overall quality and on diversity, with statistical significance reported.
  • The approach also extends to personalized editing of real images, since the same guidance procedure can be applied from a DDIM-inverted latent.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not develop is continuous interpolation between checkpoints: instead of choosing one guidance model and one edit model, the same attention-matching objective could let users dial subject fidelity versus prompt fidelity smoothly.
  • The method's success suggests that attention-map distance, not just pixel-space loss, is a useful measure of how much fine-tuning has drifted; that measure could also inform early stopping in other fine-tuning settings.
  • If the transfer works because attention maps are relatively stable across nearby checkpoints, guidance could likely be applied with only a sparse subset of timesteps or only coarse-scale attention layers, cutting the inference cost that the paper does not quantify.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper proposes DreamBlend, an inference-time procedure for personalized text-to-image diffusion fine-tuning. The idea is to generate a reference image with an early (underfit) checkpoint, record its cross-attention maps, and then generate the final image with a later (overfit) checkpoint while adding a gradient penalty that pulls the later model's cross-attention maps toward the reference at each DDIM step. The authors call the later-checkpoint phenomenon 'catastrophic attention collapse' and argue that the guidance transfers layout from the underfit model while subject identity comes from the overfit model. Experiments on the DreamBooth benchmark cover SD1.5 full fine-tuning and SDXL LoRA, report CLIP-I, CLIP-T, DINO, a Pareto-front analysis, and a human preference study (overall and diversity) against DreamBooth and Custom Diffusion.

Significance. If the main claim is accepted, DreamBlend would be a practically useful, low-cost contribution: it acts only at inference, avoids changing the fine-tuning objective, and the reported human preference results are statistically significant. The paper is also honest about its limitations, including the failure mode when the edit model is too overfit and the failure case when the subject in the guidance image is too different from the actual subject. The main uncertainty is whether the quantitative gains in Table 1 isolate the cross-attention guidance mechanism itself or merely the choice of an edit checkpoint, and whether the benchmark selection protocol leads to optimistic numbers. These issues are fixable with a focused ablation and clarified experimental protocol, so the work is worth a revision rather than rejection. No code or model checkpoints are released, which limits reproducibility of the checkpoint-selection procedure.

major comments (4)
  1. [§4.2, §4.4, Table 1, Fig. 9] The central quantitative claim is not supported by an α=0 control. Table 1 reports DreamBlend at the best of 28 operating points formed by edit checkpoints (steps 100 and 200) and guidance checkpoints, but it never reports the edit model alone under the same sampling, classifier-free guidance, and initial latent. Fig. 9 shows α=0 versus α=0.1/0.2 for one subject only, without CLIP-I, CLIP-T, or DINO numbers. Since the prompt-fidelity component is, by construction, inherited from an underfit guidance model selected for high prompt fidelity, the reported CLIP-T gain over DreamBooth and Custom Diffusion could in principle arise from checkpoint selection alone. Add a quantitative α=0 ablation over the same benchmark and selection rule, and report the resulting DINO, CLIP-I, and CLIP-T values.
  2. [§4.2, Table 1] The automatic metrics are best-case values: for each method an operating point is selected on the benchmark itself using an undefined F1 score between CLIP-T and DINO, and Table 1 reports this best value without variance over the 30 subjects and 25 prompts or over random seeds. The human preference study (Table 2) provides external support, but the abstract-level claim of 'best DINO, CLIP-I and CLIP-T' needs either error bars or confidence intervals derived from per-subject metrics, or a held-out checkpoint-selection split. The CLIP-I gain over DreamBooth is 0.808 versus 0.805; without variance this difference is not interpretable.
  3. [§4.5, Table 2] The human preference study does not state which checkpoints or operating points were used for the DreamBooth and Custom Diffusion baselines. If the baselines were not selected by the same F1-based protocol used for DreamBlend, or if they used different classifier-free guidance or sampling settings, then the large preference percentages could conflate the cross-attention guidance mechanism with operating-point selection. Specify the baseline operating points and settings, and ideally include an α=0 arm in the human study to directly test whether users prefer the CAG result over the edit model alone.
  4. [§5, Limitations; Fig. 3] The paper's own Limitations paragraph concedes that cross-attention guidance becomes ineffective when the edit model is too overfit and can fail when the subject shape in the guidance image is too different from the actual subject. This narrows the domain of the central claim: the reported success depends on selecting an edit checkpoint before catastrophic attention collapse. That selection is currently made with knowledge of the test benchmark, as shown in Fig. 10 for a single subject. I do not regard this as disqualifying, but the paper should acknowledge that the practical benefit is tied to a checkpoint-selection rule and provide evidence that the rule transfers across subjects without test-set peeking.
minor comments (4)
  1. [Algorithm 1, line 14] The left-hand side of the assignment is missing the noise variable; it should read `eps, crossAttn <- E(l, t, c)` for consistency with line 6, and the text should clarify that the gradient of R is back-propagated through the cross-attention computation.
  2. [§4.2] The phrase 'F1 score between CLIP-T and DINO scores' is not defined; specify the normalization and the exact combination rule, since this score determines the reported operating points.
  3. [Appendix D, Fig. 18] The abbreviation 'gs' is used without definition; the main text uses 'classifier-free guidance scale,' so keep the notation consistent across the paper.
  4. [§4.4, Table 1] Diversity is claimed as a quantitative benefit, but Table 1 contains no diversity metric; either add a pairwise diversity metric or state explicitly that diversity is established through the human study and qualitative comparisons.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DreamBlend's improvements are empirical claims validated against external benchmarks; the attention-guidance loss is a design mechanism, not a restatement of the evaluation metrics.

full rationale

I walked the claimed derivation chain. DreamBlend's contribution is an inference-time procedure (Algorithm 1) that stores cross-attention maps from an underfit guidance model G and then uses an L1 regularization loss R = |A_ref[t] - crossAttn| while denoising with an overfit edit model E from the same initial latent. The paper's headline results — best DINO, CLIP-I, CLIP-T in Table 1 and human-preference wins in Table 2 — are measured with external embedding similarity and user studies, not defined in terms of the attention-map distance that the algorithm optimizes. No fitted parameter is renamed as a prediction: the F1-based selection of operating points is a model-selection rule applied to both DreamBlend and baselines, and the chosen alpha/checkpoint values are hyperparameters, not quantities derived from the reported metrics. The reliance on the known result that cross-attention maps influence layout is supported by external citations ([3,8,13,29]) rather than by a self-citation chain, and there is no imported uniqueness theorem. The absence of a quantitative alpha=0 ablation is an experimental-comparison weakness (the same edit model without guidance is not numerically isolated), but that concerns control of confounds, not circular equivalence. No equation in the paper makes the claimed improvement true by construction, so the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method is empirical and hyperparameter-dependent. No new physical or model entities are introduced. The main free parameters are the guidance scales and the checkpoint choices, with the best operating point selected on the evaluation benchmark. Several claims rest on empirical observations about fine-tuning trajectories and attention maps rather than on derivations.

free parameters (3)
  • Cross-attention guidance scale alpha = 0.1 for step-100 edit, 0.07 for step-200 edit
    Controls the strength of attention-map alignment; chosen by hand, not derived. Paper studies values from 0.0 to 0.2 in Figs. 9, 17, 18.
  • Classifier-free guidance scale = 3.0 for step-100 edit, 2.0 for step-200 edit
    Set per edit model in Appendix E; affects subject fidelity and prompt fidelity.
  • Edit and guidance checkpoint selections = Edit steps 100 and 200; guidance steps below 100
    These operating points are designated in Appendix E and the best one per subject is chosen on the DreamBooth test benchmark via F1 of CLIP-T and DINO in Section 4.2, so the choice is tuned to the evaluation set.
assumptions (4)
  • domain assumption Text-image cross-attention maps control generated image layout.
    Taken from prior work [3,8,13,28,29] and used as the basis for guidance in Section 3.
  • domain assumption Early checkpoints have high prompt fidelity and diversity with low subject fidelity; later checkpoints have the reverse.
    Empirical observation from fine-tuning trajectories in Section 1 and Fig. 2; not proven and varies by subject.
  • domain assumption A single initial latent gives compatible denoising trajectories across checkpoints, so attention guidance can steer the edit model.
    Algorithm 1 reuses the same initial latent for guidance and edit models; no analysis of trajectory divergence is given.
  • domain assumption CLIP and DINO embeddings capture subject fidelity, and CLIP text similarity captures prompt fidelity.
    Evaluation relies on these proxy metrics as in prior personalization work, stated in Section 4.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DreamBlend: Advancing Personalized Fine-tuning of Text-to-Image Diffusion Models." pith.science (2026). https://pith.science/paper/RQG2FDW7

@misc{pith2026241119390,
  author       = {Pith},
  title        = {Pith review of: DreamBlend: Advancing Personalized Fine-tuning of Text-to-Image Diffusion Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RQG2FDW7}},
  note         = {Machine review of arXiv:2411.19390}
}
read the original abstract

Given a small number of images of a subject, personalized image generation techniques can fine-tune large pre-trained text-to-image diffusion models to generate images of the subject in novel contexts, conditioned on text prompts. In doing so, a trade-off is made between prompt fidelity, subject fidelity and diversity. As the pre-trained model is fine-tuned, earlier checkpoints synthesize images with low subject fidelity but high prompt fidelity and diversity. In contrast, later checkpoints generate images with low prompt fidelity and diversity but high subject fidelity. This inherent trade-off limits the prompt fidelity, subject fidelity and diversity of generated images. In this work, we propose DreamBlend to combine the prompt fidelity from earlier checkpoints and the subject fidelity from later checkpoints during inference. We perform a cross attention guided image synthesis from a later checkpoint, guided by an image generated by an earlier checkpoint, for the same prompt. This enables generation of images with better subject fidelity, prompt fidelity and diversity on challenging prompts, outperforming state-of-the-art fine-tuning methods.

Figures

Figures reproduced from arXiv: 2411.19390 by the authors.

Figure 1
Figure 1. DreamBlend merges prompt fidelity and diversity from underfit checkpoints with subject fidelity from overfit checkpoints during [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Images generated by different checkpoints for prompt: ‘a [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Attention Guidance and Attention Collapse: Images generated at 5, 250, and 1000 steps of DreamBooth fine-tuning, with text [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Cross Attention Guided Image Synthesis: Across vari [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Comparison with fine-tuning methods: Our approach successfully generates images with better subject fidelity, prompt fidelity [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: Image alignment (DINO) - text alignment (CLIP-T) [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 11
Figure 11. Figure 11: Failure Cases. Top: SD and underfit model fail to [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]
Figure 8
Figure 8. Figure 8: DreamBlend applied on different backbones, different [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Effect of Cross Attention Guidance Scale [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Effect of using different checkpoints as guidance and [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 12
Figure 12. Figure 12: Guided Image Synthesis: Across various subjects and prompts, our approach successfully preserves the layout of the reference [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]
Figure 13
Figure 13. Figure 13: Comparison with prior works: Our approach successfully generates images with better subject fidelity, prompt fidelity and [PITH_FULL_IMAGE:figures/full_fig_p013_13.png]
Figure 14
Figure 14. Figure 14: Qualitative results on SDXL backbone (a) cat (b) dog2 (c) colorful sneaker (d) dog8 (e) fancy boot (f) monster toy (g) pink sunglasses (h) wolf plushie [PITH_FULL_IMAGE:figures/full_fig_p014_14.png]
Figure 15
Figure 15. Figure 15: Image alignment (DINO) - text alignment (CLIP-T) space spanned by densely sampled operating points of DreamBooth (gray), [PITH_FULL_IMAGE:figures/full_fig_p014_15.png]
Figure 16
Figure 16. Figure 16: Human preference study interface [PITH_FULL_IMAGE:figures/full_fig_p015_16.png]
Figure 17
Figure 17. Figure 17: Effect of cross attention guidance scale [PITH_FULL_IMAGE:figures/full_fig_p016_17.png]
Figure 18
Figure 18. Figure 18: Effect of varying classifier-free guidance scale (gs) and cross attention guidance scale ( [PITH_FULL_IMAGE:figures/full_fig_p016_18.png]
Figure 19
Figure 19. Figure 19: Comparison with non-fine-tuning based methods Textual Inversion and BLIP-Diffusion [PITH_FULL_IMAGE:figures/full_fig_p017_19.png]
Figure 20
Figure 20. Figure 20: Comparison with non-fine-tuning based methods IP-Adapter and AnyDoor [PITH_FULL_IMAGE:figures/full_fig_p017_20.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 18 canonical work pages

  1. [1]

    A neural space-time representation for text- to-image personalization

    Yuval Alaluf, Elad Richardson, Gal Metzer, and Daniel Cohen-Or. A neural space-time representation for text- to-image personalization. ACM Transactions on Graphics (TOG), 42(6):1–10, 2023. 2

    work page 2023

  2. [2]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In Pro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 9650–9660, 2021. 5, 11

    work page 2021

  3. [3]

    Attend-and-excite: Attention-based se- mantic guidance for text-to-image diffusion models

    Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based se- mantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 42(4):1–10, 2023. 3

    2023

  4. [4]

    Subject-driven text-to-image generation via apprenticeship learning

    Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Wei Chang, and William W Cohen. Subject-driven text-to-image generation via apprenticeship learning. Advances in Neural Information Processing Sys- tems, 36, 2024. 3, 5

    work page 2024

  5. [5]

    Anydoor: Zero-shot object-level im- age customization

    Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level im- age customization. arXiv preprint arXiv:2307.09481, 2023. 5, 6

    arXiv 2023

  6. [6]

    Custom-edit: Text-guided image edit- ing with customized diffusion models

    Jooyoung Choi, Yunjey Choi, Yunji Kim, Junho Kim, and Sungroh Yoon. Custom-edit: Text-guided image edit- ing with customized diffusion models. arXiv preprint arXiv:2305.15779, 2023. 3

    arXiv 2023

  7. [7]

    Diffedit: Diffusion-based seman- tic image editing with mask guidance

    Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion-based seman- tic image editing with mask guidance. arXiv preprint arXiv:2210.11427, 2022. 3

    arXiv 2022

  8. [8]

    Diffusion self-guidance for control- lable image generation

    Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self-guidance for control- lable image generation. Advances in Neural Information Processing Systems, 36:16222–16239, 2023. 3

    2023

Show all 49 references
  1. [9]

    An image is worth one word: Personalizing text-to- image generation using textual inversion

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patash- nik, Amit H Bermano, Gal Chechik, and Daniel Cohen- Or. An image is worth one word: Personalizing text-to- image generation using textual inversion. arXiv preprint arXiv:2208.01618, 2022. 2, 5, 6

    2022 arXiv

  2. [10]

    Encoder-based domain tuning for fast personalization of text-to-image models.ACM Transactions on Graphics (TOG), 42(4):1–13, 2023

    Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. Encoder-based domain tuning for fast personalization of text-to-image models.ACM Transactions on Graphics (TOG), 42(4):1–13, 2023. 2, 3

    2023

  3. [11]

    Photoswap: Personalized subject swapping in images

    Jing Gu, Yilin Wang, Nanxuan Zhao, Tsu-Jui Fu, Wei Xiong, Qing Liu, Zhifei Zhang, He Zhang, Jianming Zhang, Hyun- Joon Jung, et al. Photoswap: Personalized subject swapping in images. Advances in Neural Information Processing Sys- tems, 36, 2024. 3

    2024

  4. [12]

    Vico: Plug-and-play visual condition for personalized text-to-image generation

    Shaozhe Hao, Kai Han, Shihao Zhao, and Kwan-Yee K Wong. Vico: Plug-and-play visual condition for personalized text-to-image generation. arXiv preprint arXiv:2306.00971,

    arXiv

  5. [13]

    Prompt-to-prompt im- age editing with cross attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt im- age editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022. 3

    2022 arXiv

  6. [14]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 3

    2020

  7. [15]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 5

    2021 arXiv

  8. [16]

    Taming encoder for zero fine-tuning image customization with text-to-image diffusion models

    Xuhui Jia, Yang Zhao, Kelvin CK Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, and Yu-Chuan Su. Taming encoder for zero fine-tuning image customization with text-to-image diffusion models. arXiv preprint arXiv:2304.02642, 2023. 2, 3

    2023 arXiv

  9. [17]

    Multi-concept customization of text-to-image diffusion

    Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1931–1941, 2023. 2, 5, 6

    1931

  10. [18]

    Counter- ing language drift via visual grounding

    Jason Lee, Kyunghyun Cho, and Douwe Kiela. Counter- ing language drift via visual grounding. arXiv preprint arXiv:1909.04499, 2019. 2

    1909 arXiv

  11. [19]

    Blip-diffusion: Pre- trained subject representation for controllable text-to-image generation and editing

    Dongxu Li, Junnan Li, and Steven Hoi. Blip-diffusion: Pre- trained subject representation for controllable text-to-image generation and editing. Advances in Neural Information Pro- cessing Systems, 36, 2024. 2, 3, 5, 6

    2024

  12. [20]

    Dreamedit: Subject-driven image editing

    Tianle Li, Max Ku, Cong Wei, and Wenhu Chen. Dreamedit: Subject-driven image editing. arXiv preprint arXiv:2306.12624, 2023. 3

    2023 arXiv

  13. [21]

    Gligen: Open-set grounded text-to-image generation

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jian- wei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22511–22521, 2023. 3

    2023

  14. [22]

    Countering language drift with seeded iterated learning

    Yuchen Lu, Soumye Singhal, Florian Strub, Aaron Courville, and Olivier Pietquin. Countering language drift with seeded iterated learning. In International Conference on Machine Learning, pages 6437–6447. PMLR, 2020. 2

    2020

  15. [23]

    Image segmenta- tion using text and image prompts

    Timo L ¨uddecke and Alexander Ecker. Image segmenta- tion using text and image prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7086–7096, 2022. 3, 5, 11

    2022

  16. [24]

    Unified multi-modal latent diffusion for joint subject and text conditional image generation.arXiv preprint arXiv:2303.09319, 2023

    Yiyang Ma, Huan Yang, Wenjing Wang, Jianlong Fu, and Jiaying Liu. Unified multi-modal latent diffusion for joint subject and text conditional image generation.arXiv preprint arXiv:2303.09319, 2023. 2, 3

    2023 arXiv

  17. [25]

    Sdedit: Guided image synthesis and editing with stochastic differential equa- tions

    Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jia- jun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equa- tions. arXiv preprint arXiv:2108.01073, 2021. 3

    2021 arXiv

  18. [26]

    Locating and editing factual associations in gpt

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Be- linkov. Locating and editing factual associations in gpt. In Neural Information Processing Systems, 2022. 2

    2022

  19. [27]

    Null-text inversion for editing real im- ages using guided diffusion models

    Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real im- ages using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6038–6047, 2023. 3

    2023

  20. [28]

    Dragondiffusion: Enabling drag-style manipula- tion on diffusion models

    Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. Dragondiffusion: Enabling drag-style manipula- tion on diffusion models. arXiv preprint arXiv:2307.02421,

    arXiv

  21. [29]

    Zero-shot image-to-image translation

    Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu. Zero-shot image-to-image translation. In ACM SIGGRAPH 2023 Conference Proceed- ings, pages 1–11, 2023. 3

    2023

  22. [30]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 11

    2023 arXiv

  23. [31]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  24. [32]

    Hierarchical text-conditional image gen- eration with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gen- eration with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022. 2

    2022 arXiv

  25. [33]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Confer- ence on Machine Learning, pages 8821–8831. PMLR, 2021. 2

    2021

  26. [34]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1, 2, 11

    2022

  27. [35]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500...

    2023

  28. [36]

    Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat...

    2024

  29. [37]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information...

    2022

  30. [38]

    Instant- booth: Personalized text-to-image generation without test- time finetuning

    Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instant- booth: Personalized text-to-image generation without test- time finetuning. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) , pages 8543–8552, June 2024. 2, 3

    2024

  31. [39]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 3

    2010 arXiv

  32. [40]

    Generative pretraining in multi- modality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multi- modality. arXiv preprint arXiv:2307.05222, 2023. 2

    2023 arXiv

  33. [41]

    Key-locked rank one editing for text-to-image personaliza- tion

    Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editing for text-to-image personaliza- tion. In ACM SIGGRAPH 2023 Conference Proceedings , pages 1–11, 2023. 2

    2023

  34. [42]

    Diffusers: State-of-the-art diffu- sion models

    Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffu- sion models. https://github.com/huggingface/ diffusers...

    2022

  35. [43]

    p+: Extended textual conditioning in text-to- image generation

    Andrey V oynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman. p+: Extended textual conditioning in text-to- image generation. arXiv preprint arXiv:2303.09522, 2023. 2

    2023 arXiv

  36. [44]

    Edict: Exact diffusion inversion via coupled transformations

    Bram Wallace, Akash Gokul, and Nikhil Naik. Edict: Exact diffusion inversion via coupled transformations. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22532–22541, 2023. 3

    2023

  37. [45]

    Instantid: Zero-shot identity-preserving gener- ation in seconds

    Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and An- thony Chen. Instantid: Zero-shot identity-preserving gener- ation in seconds. arXiv preprint arXiv:2401.07519, 2024. 2, 3

    2024 arXiv

  38. [46]

    Elite: Encoding visual con- cepts into textual embeddings for customized text-to-image generation

    Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual con- cepts into textual embeddings for customized text-to-image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15943–15953, 2023. 2

    2023

  39. [47]

    Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models. arXiv preprint arXiv:2308.06721,

    arXiv

  40. [48]

    Scaling autoregres- sive models for content-rich text-to-image generation

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gun- jan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yin- fei Yang, Burcu Karagol Ayan, et al. Scaling autoregres- sive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2(3):5, 2022. 2

    2022 arXiv

  41. [49]

    a stuffed animal∗ on the beach

    Yanbing Zhang, Mengping Yang, Qin Zhou, and Zhe Wang. Attention calibration for disentangled text-to-image person- alization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4764– 4774, 2024. 2, 3 Appendix In Appendix A, we present ...

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.