Pith. sign in

REVIEW 3 major objections 5 minor 56 references

Nested Attention: Semantic-aware Attention Values for Concept Personalization

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Replacing a subject token's fixed value with query-dependent values preserves identity without breaking prompt control.

desk verdict A genuinely different value-injection mechanism, backed by unusually clean controlled comparisons; the semantic-routing explanation is under-supported, but the empirical tradeoff claim is strong enough to warrant serious review. read the letter →

arxiv 2501.01407 v1 pith:FJVQKAOD submitted 2025-01-02 cs.CV cs.GRcs.LG

Or Patashnik , Rinon Gal , Daniil Ostashev , Sergey Tulyakov , Kfir Aberman , Daniel Cohen-Or This is my paper · ORCID
classification cs.CVcs.GRcs.LG
keywords text-to-imagepersonalizationnestedattentioncross-attentionvaluesidentitypreservationencoder-baseddiffusionmodelsconceptinjectionmulti-subjectgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Personalizing a text-to-image model usually forces a tradeoff: a single learned token is too poor to capture a person's identity, while injecting many image tokens overwhelms the model and degrades prompt alignment. This paper claims the tradeoff can be sidestepped by keeping the subject attached to one textual token but replacing that token's fixed cross-attention value with values computed per image region. A small nested attention layer lets each region of the generated image query a rich multi-token representation of the subject and pull out the relevant part, such as eyes or nose, rather than one averaged embedding. The authors show this yields stronger identity preservation than common injection mechanisms, including decoupled cross-attention, under matched training budgets, and that the same mechanism supports combining a person and a pet in one image. If correct, it gives a general encoder-based personalization method that does not need face-identity losses or specialized datasets.

What carries the argument

The load-bearing object is the nested attention layer, a second cross-attention operation inserted inside each existing cross-attention layer of the frozen diffusion model. For the subject token $s^*$, its equation is $v^*_{q_{ij}} = \operatorname{softmax}(q_{ij} \tilde{K}^T / \sqrt{d}) \tilde{V}$, where $q_{ij}$ is the external layer's query at spatial location $(i,j)$ and $\tilde{K}, \tilde{V}$ are keys and values projected from encoder tokens. The output $v^*_{q_{ij}}$ replaces the value of $s^*$ at that location, while all other textual token values and all keys stay identical to the base model. The mechanism does two jobs: it gives the single subject token a rich, multi-vector backing representation, and it makes the routing of that representation depend on the semantic content of each generated region. Its training is bounded by a norm regularizer, $\|v^*_{q_{ij}}\| = \alpha |V[s^*]|$ with $\alpha = 2$, which prevents the injected values from magnifying the subject's attention and breaking the prior.

What would settle it

Log the dominant encoder token for each query's nested attention map during generation and compare it against a human-labeled part segmentation of the input subject; if identity preservation stays high in cases where the dominant tokens consistently point to semantically wrong regions, then the query-as-semantic-selector premise is not the actual cause of the gains.

Watch

Extended reading notes

Core claim

The central claim is that the expressiveness bottleneck in single-token personalization is the attention value, not the token's key or the number of tokens. In standard cross-attention, the value attached to the word 'person' is the same vector for every spatial location, so all identity detail must be compressed into one embedding. Nested Attention replaces that value with $v^*_{q_{ij}} = \operatorname{softmax}(q_{ij} \tilde{K}^T / \sqrt{d}) \tilde{V}$, a query-dependent vector computed by a second attention layer whose keys and values come from an image encoder (a frozen CLIP plus a trained Q-Former). Because only the value of the subject token is modified and its norm is regularized to a fixed multiple of the original value, the model's prior is disturbed far less than by adding parallel cross-attention layers, while the per-query values can encode fine-grained semantic parts. The paper argues, and supports with comparisons and a user study, that this combination yields a better identity-versus-prompt tradeoff than decoupled cross-attention, global value averaging, or using multiple tokens, and that it generalizes to non-human domains such as pets.

Load-bearing premise

The method assumes that the query vector at each generated-image location reliably encodes which part of the subject is needed there, so that the nested attention routes the correct subject feature to the correct region.

Editorial extensions

If this is right

  • Provided the nested attention layers train as described, an encoder-based personalization system can keep a single textual token as the handle for a subject while carrying as many identity details as a Q-Former with 256 to 1024 learned queries.
  • The identity-versus-prompt tradeoff becomes a single scalar: raising $\lambda$ increases the subject token's attention logit, so users can dial between likeness and editability at inference without retraining.
  • Because each subject's representation is anchored to its own token, multiple subjects from different domains can be combined in one image by running one encoder and one nested layer set per subject, with no extra training.
  • Supplying several photos of the same subject at test time, by concatenating their encoded tokens, improves identity fidelity without architectural changes, which handles occluded or ambiguous single views.
  • The method extends beyond faces: the same training procedure on pet images works without face-recognition features or identity losses, so the mechanism is not tied to a human-face prior.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: If the query vector is a reliable semantic selector, the same nested value routing could replace decoupled cross-attention inside other encoder-based personalization pipelines, potentially improving their prompt adherence; the paper hints at this by noting PulID's ideas are largely orthogonal.
  • Inference: The mechanism implicitly learns a soft segmentation of the subject by assigning different encoder tokens to different generated regions, so a direct test would be whether routing maps align with annotated part segmentations of the input image; the paper only shows qualitative evidence for two face regions.
  • Inference: For same-domain multi-subject generation, the paper reports overlap and self-attention leakage as a remaining failure; a plausible next step is to combine nested attention with attention-bounding or mask control to separate two faces or two pets, but this is not demonstrated here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Nested Attention, a mechanism for injecting a personalized subject's image representation into a pretrained text-to-image diffusion model. The subject is tied to a single text token s* in existing cross-attention layers, but instead of using a fixed value for that token, a nested attention layer computes query-dependent values v*_q = softmax(q K~^T / sqrt(d)) V~ from tokens produced by a CLIP backbone and a Q-Former encoder. This design is intended to combine the expressiveness of multi-token encoders with the prior-preservation of single-token injection. The authors train the encoder and nested layers on FFHQ-Wild for faces and on synthetic/AFHQ images for pets, and compare against alternative injection mechanisms (decoupled cross-attention, simple adapter, global value, multiple tokens) under equal training budgets, as well as against recent face personalization methods (IP-Adapter, InstantID, PhotoMaker, LCM-Lookahead, PuLID). They report improved identity-versus-prompt tradeoffs, multi-subject generation, multiple-input-image support, and an analysis of the learned queries and routing behavior.

Significance. If the claims hold, the contribution is a simple, domain-general injection mechanism that improves on decoupled cross-attention without requiring identity-recognition backbones or specialized losses. The controlled comparison of injection mechanisms under identical encoder and training budgets (Figure 10) is a genuine strength, as is the user study showing preference over several baselines. The paper also clearly specifies the attention equations and includes ablations of the norm regularization and the number of Q-Former queries. However, the work is primarily empirical, and the main quantitative results currently lack error bars and significance tests, the user study relies on 22 participants, and the claimed semantic routing is supported only by a small qualitative analysis. These gaps do not invalidate the approach, but they limit the strength of the comparative claims and leave the mechanism's explanatory story under-verified.

major comments (3)
  1. [Section 4, Figure 5] The central explanatory claim—that the external query q_ij is a reliable semantic selector and that v*_q = softmax(q K~^T / sqrt(d)) V~ routes eyes to eyes and nose to nose—is supported only by three hand-picked queries from one layer at one denoising step. The text itself notes that the arm query fails to find a matching region and falls back to neck/shirt. The paper reports no quantitative routing accuracy, no variation over layers, denoising steps, prompts, or domains, and no comparison to a random-query baseline. Because this routing is the stated mechanism behind the identity-preservation benefit (Section 3.2), the authors should either provide a quantitative routing evaluation (for example, aligning the Q-Former attention regions of dominant encoder tokens with semantic parts across many generated images and time steps) or explicitly reframe the contribution as a generic query-dependent value injection with evidence that the benefit does not reduce to added representational capacity. Without one of these, the 'semantic-aware' explanation is under-supported.
  2. [Section 5.2, Figures 10-11, Table 1] The quantitative comparisons lack error bars, standard deviations, and significance tests. The curves in Figure 10 overlap, and it is not clear whether the reported ordering is stable across random seeds or prompt sets. The user study (Appendix A) has only 22 participants; Table 1 reports win rates without confidence intervals or inter-annotator agreement, yet the text uses 'significantly outperforms' (Section 5.2). Please add per-method means and standard deviations over multiple seeds, statistical tests on the paired comparisons, per-prompt breakdowns, and user-study confidence intervals or exact p-values. This is needed to substantiate claims such as 'nested attention provides the best trade-off' and 'significantly outperforms IPA-Face in user evaluations.'
  3. [Section 5, Appendix A] Reproducibility details are incomplete for an empirical paper whose strongest evidence is the controlled comparison. The text does not state the optimizer, learning rate, total number of training iterations or their equivalence across baselines, the initialization of nested attention projections, or the number of random seeds and prompt samples used for the quantitative metrics. No code or trained weights are released. These omissions are load-bearing for verification of the central claim: a reader cannot currently rerun the comparison or check whether the reported tradeoff curves are robust. Please include a complete training and evaluation protocol and release code, and where possible the trained checkpoints.
minor comments (5)
  1. [Section 3.2, Eq. (1)] The notation for the nested attention equation uses sqrt(d) and K~, V~ without defining the dimension d or the projection parameterizations; the surrounding text defines W_K~ and W_V~, but the reader has to infer that d is the query/key dimension.
  2. [Section 5.1, Eq. for attention adjustment] The expression max(QK^T[s*], λQK^T[s*]) is mathematically ambiguous: for negative logits and λ ≥ 1 it has no effect, which may be intended, but the sentence should explain the intended clamping behavior explicitly.
  3. [Figure 5] The graphs in Figure 5 lack axis labels, a legend for the encoder-token points, and an explicit statement of which cross-attention layer and denoising step is probed; this makes the qualitative routing analysis harder to interpret.
  4. [Section 5.2, Table 1] The method name is written inconsistently as 'PulID' in the text and Table 1, while the reference is 'PuLID' (Ref. [19]); please use consistent spelling.
  5. [Section 6] The paper does not include a limitations paragraph; given the acknowledged difficulty with multiple subjects from the same domain and the routing-evidence gap, a brief limitations discussion would help readers calibrate the method's scope.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the nested-attention mechanism is an architectural proposal trained and evaluated against external benchmarks, with no load-bearing prediction reducing to its own inputs.

full rationale

Walking the paper's derivation chain, the central mechanism in Section 3.2 is defined constructively by the equations v*_qij = softmax(q_ij K~^T / sqrt(d)) V~ and V_qij[s] = v*_qij when s = s*, else V[s]. This is a new architectural proposal, not a derivation that reduces to its inputs. The encoder in Section 3.3 is trained by ordinary diffusion noise prediction on (input image, prompt, target image) triplets; no evaluation metric is used as a training target, so no fitted parameter is renamed as a prediction. Identity preservation is measured with a face-recognition network and by a 22-participant user study, both external to the method's own CLIP-based encoder. Prompt alignment is measured with CLIP, which overlaps with the frozen CLIP backbone used in the encoder, but this overlap is a mild evaluation caveat rather than a circular derivation: the CLIP similarity score is not optimized during training and is a standard external benchmark. The semantic-routing claim is supported by qualitative attention visualizations in Figures 5 and 6 rather than by construction; the lack of a quantitative routing test is a correctness and robustness risk, not a circularity. Self-citations appear, for example references [2], [11], and [45], but they support background statements, design motivations, or limitation remarks, and no load-bearing assertion reduces to an unverified self-citation. No uniqueness theorem is imported from the authors' prior work, and no known result is merely renamed. The empirical comparisons are against external baselines and include a user study, so the central claims have independent content. The honest finding is therefore no significant circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The method is an empirical neural architecture. The load-bearing assumptions are the query-as-semantic-selector assumption, the value-only injection assumption, and the encoder capacity assumption. Free parameters are the norm scale alpha and the Q-Former query count, both tuned via ablations; the query-count ablation is evaluated on the test set.

free parameters (2)
  • Value norm regularization constant alpha = 2
    Controls the norm of the nested attention values relative to the original text value; chosen via ablation in Appendix C to balance identity and prior preservation.
  • Number of Q-Former learned queries = 1024 for faces, 256 for pets
    Determines the number of nested keys and values; selected via ablation on the test set ID score (Appendix C, Figure 18).
assumptions (3)
  • domain assumption The query vector q_{ij} in the external cross-attention layer carries enough semantic information to select the correct subject features for each spatial region.
    The nested attention output v*_q = softmax(q K~^T) V~ relies on this; only qualitative support is provided in Figure 5. Section 3.2.
  • domain assumption Replacing only the value of token s* while keeping its key unchanged does not unacceptably distort the pretrained model's prior.
    The prior-preservation claim depends on this; the norm regularization alpha=2 is an empirical patch. Section 3.2 and Appendix C.
  • domain assumption A Q-Former with 1024 or 256 learned queries can represent a subject's identity from a single image without identity-specific losses or repeated-identity data.
    The encoder is trained on (input, prompt, target) triplets only, with no face-recognition loss. Section 3.3 and Appendix A.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Nested Attention: Semantic-aware Attention Values for Concept Personalization." pith.science (2026). https://pith.science/paper/FJVQKAOD

@misc{pith2026250101407,
  author       = {Pith},
  title        = {Pith review of: Nested Attention: Semantic-aware Attention Values for Concept Personalization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FJVQKAOD}},
  note         = {Machine review of arXiv:2501.01407}
}
read the original abstract

Personalizing text-to-image models to generate images of specific subjects across diverse scenes and styles is a rapidly advancing field. Current approaches often face challenges in maintaining a balance between identity preservation and alignment with the input text prompt. Some methods rely on a single textual token to represent a subject, which limits expressiveness, while others employ richer representations but disrupt the model's prior, diminishing prompt alignment. In this work, we introduce Nested Attention, a novel mechanism that injects a rich and expressive image representation into the model's existing cross-attention layers. Our key idea is to generate query-dependent subject values, derived from nested attention layers that learn to select relevant subject features for each region in the generated image. We integrate these nested layers into an encoder-based personalization method, and show that they enable high identity preservation while adhering to input text prompts. Our approach is general and can be trained on various domains. Additionally, its prior preservation allows us to combine multiple personalized subjects from different domains in a single image.

Figures

Figures reproduced from arXiv: 2501.01407 by the authors.

Figure 1
Figure 1. Our nested attention mechanism attaches a localized, expressive representation of a subject to a single text token. This approach [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. The nested attention mechanism. We replace the value [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. We visualize the values Vq[s ∗ ] generated for a subject in two different layers, with a vanilla cross-attention, and with our nested approach. Vanilla layers use the same value to represent the subject throughout the entire image (column 3). Nested attention assigns a different subject-value per query (columns 4 and 5), en￾coding fine-grained semantic information. having to represent the individual’s entire appeara… view at source ↗
Figures from the paper (15 more)
Figure 6
Figure 6. Figure 6: Attention maps between Q-Former learned queries and [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Qualitative results of our method trained on human faces (left) and pets (right). The sampled noise is fixed across each column. [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: By manipulating the attention given to the personalized [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: Comparing nested attention with decoupled cross attention. [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: Quantitative comparison of various personalization in [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]
Figure 11
Figure 11. Figure 11: Left: qualitative comparison of human faces personalization methods. Our method successfully changes expressions and pose [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]
Figure 12
Figure 12. Figure 12: Multi-subject generation comparison. Input “in a living room” [PITH_FULL_IMAGE:figures/full_fig_p008_12.png]
Figure 13
Figure 13. Figure 13: Using multiple images of the same concept increases [PITH_FULL_IMAGE:figures/full_fig_p008_13.png]
Figure 14
Figure 14. Figure 14: Our method allows mixing two identities by encoding [PITH_FULL_IMAGE:figures/full_fig_p012_14.png]
Figure 15
Figure 15. Figure 15: By simply changing at inference time the token [PITH_FULL_IMAGE:figures/full_fig_p012_15.png]
Figure 16
Figure 16. Figure 16: Qualitative comparison of injection mechanism. [PITH_FULL_IMAGE:figures/full_fig_p013_16.png]
Figure 17
Figure 17. Figure 17: Ablating the regularization performed on [PITH_FULL_IMAGE:figures/full_fig_p014_17.png]
Figure 18
Figure 18. Figure 18: Results of models trained with varying number of [PITH_FULL_IMAGE:figures/full_fig_p014_18.png]
Figure 19
Figure 19. Figure 19: Additional results on human faces. The initial noise is fixed across each column. [PITH_FULL_IMAGE:figures/full_fig_p015_19.png]
Figure 20
Figure 20. Figure 20: Additional results on pets. The initial noise is fixed across each column. [PITH_FULL_IMAGE:figures/full_fig_p016_20.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 32 canonical work pages

  1. [1]

    Cross-image attention for zero- shot appearance transfer, 2023

    Yuval Alaluf, Daniel Garibi, Or Patashnik, Hadar Averbuch- Elor, and Daniel Cohen-Or. Cross-image attention for zero- shot appearance transfer, 2023. 3

    work page 2023

  2. [2]

    A neural space-time representation for text- to-image personalization

    Yuval Alaluf, Elad Richardson, Gal Metzer, and Daniel Cohen-Or. A neural space-time representation for text- to-image personalization. ACM Transactions on Graphics (TOG), 42(6):1–10, 2023. 2, 4

    work page 2023

  3. [3]

    Ghost- facenets: Lightweight face recognition model from cheap operations

    Mohamad Alansari, Oussama Abdul Hay, Sajid Javed, Ab- dulhadi Shoufan, Yahya Zweiri, and Naoufel Werghi. Ghost- facenets: Lightweight face recognition model from cheap operations. IEEE Access, 11:35429–35446, 2023. 6

    work page 2023

  4. [4]

    Moab Arar, Rinon Gal, Yuval Atzmon, Gal Chechik, Daniel Cohen-Or, Ariel Shamir, and Amit H. Bermano. Domain- agnostic tuning-encoder for fast personalization of text-to- image models. In SIGGRAPH Asia 2023 Conference Papers, pages 1–10, 2023. 1, 2

    work page 2023

  5. [5]

    Palp: Prompt aligned personalization of text-to- image models

    Moab Arar, Andrey V oynov, Amir Hertz, Omri Avrahami, Shlomi Fruchter, Yael Pritch, Daniel Cohen-Or, and Ariel Shamir. Palp: Prompt aligned personalization of text-to- image models. arXiv preprint arXiv:2401.06105, 2024. 1

    arXiv 2024

  6. [6]

    Break-a-scene: Extracting multi- ple concepts from a single image

    Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen- Or, and Dani Lischinski. Break-a-scene: Extracting multi- ple concepts from a single image. In SIGGRAPH Asia 2023 Conference Papers, New York, NY , USA, 2023. Association for Computing Machinery. 2

    work page 2023

  7. [7]

    The chosen one: Consistent characters in text-to- image diffusion models

    Omri Avrahami, Amir Hertz, Yael Vinker, Moab Arar, Shlomi Fruchter, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski. The chosen one: Consistent characters in text-to- image diffusion models. arXiv preprint arXiv:2311.10093,

    arXiv

  8. [8]

    Masactrl: Tuning-free mu- tual self-attention control for consistent image synthesis and editing

    Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xi- aohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mu- tual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV) , pages 22560–22570,

Show all 56 references
  1. [9]

    Wenhu Chen, Hexiang Hu, Y ANDONG LI, Nataniel Ruiz, Xuhui Jia, Ming-Wei Chang, and William W. Cohen. Subject-driven text-to-image generation via apprenticeship learning. In Thirty-seventh Conference on Neural Informa- tion Processing Systems, 2023. 2

    2023

  2. [10]

    Stargan v2: Diverse image synthesis for multiple domains

    Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020. 5, 12

    2020

  3. [11]

    Be yourself: Bounded attention for multi-subject text-to-image generation, 2024

    Omer Dahary, Or Patashnik, Kfir Aberman, and Daniel Cohen-Or. Be yourself: Bounded attention for multi-subject text-to-image generation, 2024. 7

    2024

  4. [12]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Informa- tion Processing Systems, 34:8780–8794, 2021. 1

    2021

  5. [13]

    Drea- martist: Towards controllable one-shot text-to-image gen- eration via contrastive prompt-tuning

    Ziyi Dong, Pengxu Wei, and Liang Lin. Drea- martist: Towards controllable one-shot text-to-image gen- eration via contrastive prompt-tuning. arXiv preprint arXiv:2211.11337, 2022. 2

    2022 arXiv

  6. [14]

    Implicit style-content separation using b-lora

    Yarden Frenkel, Yael Vinker, Ariel Shamir, and Daniel Cohen-Or. Implicit style-content separation using b-lora. In European Conference on Computer Vision, pages 181–198. Springer, 2025. 2

    2025

  7. [15]

    Bermano, Gal Chechik, and Daniel Cohen-Or

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image gen- eration using textual inversion, 2022. 1, 2

    2022

  8. [16]

    Encoder-based domain tuning for fast personalization of text-to-image models.ACM Transactions on Graphics (TOG), 42(4):1–13, 2023

    Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. Encoder-based domain tuning for fast personalization of text-to-image models.ACM Transactions on Graphics (TOG), 42(4):1–13, 2023. 1, 2

    2023

  9. [17]

    Bermano, Gal Chechik, and Daniel Cohen-Or

    Rinon Gal, Or Lichter, Elad Richardson, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. Lcm- lookahead for encoder-based text-to-image personalization,

  10. [18]

    Tokenflow: Consistent diffusion features for consistent video editing

    Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arxiv:2307.10373, 2023. 3

    2023 arXiv

  11. [19]

    Pulid: Pure and lightning id customization via con- trastive alignment

    Zinan Guo, Yanze Wu, Zhuowei Chen, Lang Chen, and Qian He. Pulid: Pure and lightning id customization via con- trastive alignment. arXiv preprint arXiv:2404.16022, 2024. 1, 2, 6

    2024 arXiv

  12. [20]

    Svdiff: Compact pa- rameter space for diffusion fine-tuning

    Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact pa- rameter space for diffusion fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7323–7334, 2023. 2

    2023

  13. [21]

    Prompt-to-prompt image editing with cross attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. 2022. 3

    2022

  14. [22]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020. 1

    2020

  15. [23]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. ArXiv, abs/2106.09685, 2021. 2

    2021 arXiv

  16. [24]

    Taming encoder for zero fine-tuning image customization with text-to-image diffusion models

    Xuhui Jia, Yang Zhao, Kelvin CK Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, and Yu-Chuan Su. Taming encoder for zero fine-tuning image customization with text-to-image diffusion models. arXiv preprint arXiv:2304.02642, 2023. 2 9

    2023 arXiv

  17. [25]

    Customizing text-to-image models with a single image pair

    Maxwell Jones, Sheng-Yu Wang, Nupur Kumari, David Bau, and Jun-Yan Zhu. Customizing text-to-image models with a single image pair. arXiv preprint arXiv:2405.01536, 2024. 2

    2024 arXiv

  18. [26]

    A style-based generator architecture for generative adversarial networks

    Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4401–4410, 2019. 2, 5, 12

    2019

  19. [27]

    Multi-concept customization of text-to-image diffusion

    Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. arXiv, 2022. 1, 2

    2022

  20. [28]

    Keras insightface

    Leondgarse. Keras insightface. https://github.com/ leondgarse/Keras_insightface, 2022. 6

    2022

  21. [29]

    Blip- diffusion: Pre-trained subject representation for control- lable text-to-image generation and editing

    Dongxu Li, Junnan Li, and Steven CH Hoi. Blip- diffusion: Pre-trained subject representation for control- lable text-to-image generation and editing. arXiv preprint arXiv:2305.14720, 2023. 2, 4

    2023 arXiv

  22. [30]

    Photomaker: Customizing realistic human photos via stacked id embedding

    Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming- Ming Cheng, and Ying Shan. Photomaker: Customizing realistic human photos via stacked id embedding. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 1, 2, 6, 12

    2024

  23. [31]

    Self-distilled stylegan: Towards generation from internet photos, 2022

    Ron Mokady, Michal Yarom, Omer Tov, Oran Lang, Michal Irani Daniel Cohen-Or, Tali Dekel, and Inbar Mosseri. Self-distilled stylegan: Towards generation from internet photos, 2022. 5, 12

    2022

  24. [32]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models.arXiv preprint arXiv:2112.10741, 2021

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models.arXiv preprint arXiv:2112.10741, 2021. 1

    2021 arXiv

  25. [33]

    Zero-shot image-to-image translation

    Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu. Zero-shot image-to-image translation. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Proceed- ings, page 1–11. ACM, 2023. 3

    2023

  26. [34]

    Portraitbooth: A versatile portrait model for fast identity-preserved personalization

    Xu Peng, Junwei Zhu, Boyuan Jiang, Ying Tai, Donghao Luo, Jiangning Zhang, Wei Lin, Taisong Jin, Chengjie Wang, and Rongrong Ji. Portraitbooth: A versatile portrait model for fast identity-preserved personalization. arXiv preprint arXiv:2312.06354, 2023. 2

    2023 arXiv

  27. [35]

    SDXL: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth Inter- national Conference on Learning Representations, 2024. 3, 5

    2024

  28. [36]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In International Conference on Machine Learning...

    2021

  29. [37]

    Hierarchical text-conditional image gen- eration with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gen- eration with clip latents. arXiv preprint arXiv:2204.06125,

    arXiv

  30. [38]

    High-resolution image syn- thesis with latent diffusion models, 2021

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models, 2021. 3

    2021

  31. [39]

    Rb-modulation: Training-free personal- ization of diffusion models using stochastic optimal control

    L Rout, Y Chen, N Ruiz, A Kumar, C Caramanis, S Shakkot- tai, and W Chu. Rb-modulation: Training-free personal- ization of diffusion models using stochastic optimal control

  32. [40]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. 2022. 1, 2

    2022

  33. [41]

    Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models, 2023

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models, 2023. 1, 2

    2023

  34. [42]

    Low-rank adaptation for fast text-to-image diffusion fine-tuning

    Simo Ryu. Low-rank adaptation for fast text-to-image diffusion fine-tuning. https : / / github . com / cloneofsimo/lora, 2023. 2

    2023

  35. [43]

    Ziplora: Any subject in any style by effectively merging loras

    Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svet- lana Lazebnik, Yuanzhen Li, and Varun Jampani. Ziplora: Any subject in any style by effectively merging loras. In European Conference on Computer Vision, pages 422–438. Springer, 2025. 2

    2025

  36. [44]

    Instant- booth: Personalized text-to-image generation without test- time finetuning

    Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instant- booth: Personalized text-to-image generation without test- time finetuning. arXiv preprint arXiv:2304.03411, 2023. 2

    2023 arXiv

  37. [45]

    Key-locked rank one editing for text-to-image personaliza- tion

    Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editing for text-to-image personaliza- tion. In ACM SIGGRAPH 2023 Conference Proceedings , pages 1–11, 2023. 2, 4, 6

    2023

  38. [46]

    Training-free con- sistent text-to-image generation, 2024

    Yoad Tewel, Omri Kaduri, Rinon Gal, Yoni Kasten, Lior Wolf, Gal Chechik, and Yuval Atzmon. Training-free con- sistent text-to-image generation, 2024. 2

    2024

  39. [47]

    Plug-and-play diffusion features for text-driven image-to-image translation

    Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1921–1930, 2023. 3

    1921

  40. [48]

    Unitune: Text-driven image editing by fine tuning an image generation model on a single image.arXiv preprint arXiv:2210.09477, 2022

    Dani Valevski, Matan Kalman, Yossi Matias, and Yaniv Leviathan. Unitune: Text-driven image editing by fine tuning an image generation model on a single image.arXiv preprint arXiv:2210.09477, 2022. 2

    2022 arXiv

  41. [49]

    Face0: Instantaneously conditioning a text-to- image model on a face

    Dani Valevski, Danny Lumen, Yossi Matias, and Yaniv Leviathan. Face0: Instantaneously conditioning a text-to- image model on a face. InSIGGRAPH Asia 2023 Conference Papers, New York, NY , USA, 2023. Association for Comput- ing Machinery. 1, 2

    2023

  42. [50]

    p+: Extended textual conditioning in text-to- image generation

    Andrey V oynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman. p+: Extended textual conditioning in text-to- image generation. arXiv preprint arXiv:2303.09522, 2023. 2

    2023 arXiv

  43. [51]

    Moa: Mixture-of-attention for subject-context disentanglement in personalized image gen- eration, 2024

    Kuan-Chieh Wang, Daniil Ostashev, Yuwei Fang, Sergey Tulyakov, and Kfir Aberman. Moa: Mixture-of-attention for subject-context disentanglement in personalized image gen- eration, 2024. 1, 2

    2024

  44. [52]

    Instantid: Zero-shot identity-preserving gener- 10 ation in seconds

    Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and An- thony Chen. Instantid: Zero-shot identity-preserving gener- 10 ation in seconds. arXiv preprint arXiv:2401.07519, 2024. 1, 2, 6

    2024 arXiv

  45. [53]

    Elite: Encoding visual concepts into textual embeddings for customized text-to-image gener- ation

    Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image gener- ation. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision (ICCV), pages 15943–15...

    2023

  46. [54]

    Freeman, Fr ´edo Durand, and Song Han

    Guangxuan Xiao, Tianwei Yin, William T. Freeman, Fr ´edo Durand, and Song Han. Fastcomposer: Tuning-free multi- subject image generation with localized attention. arXiv,

  47. [55]

    Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models. arXiv preprint arXiv:2308.06721,

    arXiv

  48. [56]

    person” “child

    Ge Yuan, Xiaodong Cun, Yong Zhang, Maomao Li, Chenyang Qi, Xintao Wang, Ying Shan, and Huicheng Zheng. Inserting anybody in diffusion models via celeb ba- sis. arXiv preprint arXiv:2306.00926, 2023. 2 11 Appendices A. Implementation Details Method We train the human face model...

    2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.