Pith. sign in

REVIEW 4 major objections 6 minor 7 cited by

Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot Subject-Driven Image Generator

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Diptych Prompting claims that a large text-to-image model with an inpainting module is a zero-shot subject-driven generator: a segmented reference in the left panel, masked inpainting on the right, with cross-panel attention carrying the…

desk verdict Clever zero-shot trick, but the headline comparison is confounded by base-model scale; revisit with matched baselines before believing the claimed margin. read the letter →

arxiv 2411.15466 v2 pith:HEY7POI2 submitted 2024-11-23 cs.CV

Chaehun Shin , Jooyoung Choi , Heeseung Kim , Sungroh Yoon This is my paper · ORCID
classification cs.CV
keywords subject-driventext-to-imagegenerationzero-shotpersonalizationdiptychpromptingimageinpaintingFLUXdiffusiontransformerattentionrescalingbackgroundremoval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a large text-to-image model with a separate inpainting module can do zero-shot subject-driven generation, with no per-subject fine-tuning and no specially trained image encoder. The move is to recast the task as completing a diptych: a background-stripped reference photo occupies the left panel, the right panel is generated by text-conditioned inpainting, and the model's existing cross-panel attention transfers the subject's identity. If this is right, subject-driven personalization becomes a side effect of scale plus generic inpainting ability rather than a bespoke trained capability, which matters because it removes the optimization cost of personalization and makes the method improve automatically as the base model improves. The authors support the claim with DreamBench evaluations and a human preference study in which their method is preferred over the tested zero-shot encoder-based baselines for both subject and text alignment.

What carries the argument

The load-bearing object is the incomplete diptych $I_{\mathrm{diptych}} = [G_{\mathrm{seg}}(I_{\mathrm{ref}}); \varnothing]$ with mask $M_{\mathrm{diptych}} = [0_{h\times w}; 1_{h\times w}]$, fed through the inpainting equation $\hat{I}_{\mathrm{diptych}} = F_\theta(I_{\mathrm{diptych}}, M_{\mathrm{diptych}}, T_{\mathrm{diptych}})$, where $G_{\mathrm{seg}}$ is a grounded-detection-plus-segmentation background remover and $F_\theta$ is FLUX with a ControlNet inpainting module. The identity-carrying mechanism is the reference attention block inside FLUX's transformer: the submatrix of attention weights from right-panel queries to left-panel keys is rescaled by $\lambda > 1$ (set to $1.3$ in experiments) to amplify fine subject details. Background removal exists to stop the right panel from mirroring pose, location, and background from the reference, so the two auxiliary mechanisms jointly enforce text alignment and subject alignment.

What would settle it

Run the full pipeline on a subject whose identity lives in fine texture, such as the DreamBench colorful sneaker or the patterned bowl. If the generated right panel repeatedly loses the identifying pattern at the paper's chosen $\lambda=1.3$, while a text-encoder baseline keeps it, then the claim that cross-panel attention rescaling preserves granular details fails. A second check: remove the background-segmentation step on a varied set of subjects; if CLIP-T does not drop relative to the segmented version, the paper's stated mechanism for preventing content leakage is not doing the claimed work.

Watch

Extended reading notes

Core claim

The paper's central claim is that the diptych-generation ability of the FLUX text-to-image model can be repurposed as a training-free subject-driven generator by turning the task into diptych inpainting. Given one reference image, Diptych Prompting isolates the subject with a detection-and-segmentation step, places it in the left half of a 1:2 canvas, masks the right half, and fills that right half from a prompt describing the desired context in diptych form. The identity transfer happens through attention: queries from the right panel attend to keys from the left panel, and the method rescales those attention weights by $\lambda=1.3$ to preserve fine-grained details. Reported results on DreamBench show the method matching or exceeding encoder-based zero-shot baselines on DINO and CLIP-T scores, and winning pairwise human preference comparisons against every tested baseline at $p<0.01$. The same recipe is extended to style-driven generation and to subject-driven editing with no architectural change.

Load-bearing premise

The method assumes that FLUX's skill at generating two related panels from one text prompt still works when one panel is a real, background-stripped photograph and the other is produced by masked inpainting, so that identity crosses panels without training.

Editorial extensions

If this is right

  • Zero-shot subject-driven generation becomes a single forward pass: no per-subject optimization, no image-encoder training, and no test-time fine-tuning.
  • The same diptych framing transfers to stylized image generation: a style reference in the left panel produces right-panel images in that style, quantitatively comparable to dedicated zero-shot style methods.
  • Subject-driven editing is a special case of the same mechanism: put the target image in the right panel, mask only the edit region, and the reference subject is inpainted into it.
  • Performance is tied to the base model, so stronger text-to-image models and better inpainting modules should directly improve Diptych Prompting without any rework of the method.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to decouple identity from pose: the paper removes background but does not explicitly prevent the right panel from copying the reference's viewpoint, so a depth- or pose-controlled variant is a testable next step.
  • The attention-rescaling factor $\lambda$ acts like an identity-strength knob; one could expose it per layer or schedule it adaptively, which the paper does not explore and which may reduce the blur and color shifts the authors note at high $\lambda$.
  • Because the method needs only a model that can generate coherent multi-panel canvases, any future large text-to-image model with that emergent ability and a generic inpainting module should inherit the same trick, making this a recipe rather than a one-model fix.
  • Multi-subject generation is the obvious open boundary: the current method is demonstrated with a single reference panel, and the paper itself lists multi-subject generation as future work.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Diptych Prompting, a training-free method for zero-shot subject-driven text-to-image generation. The key idea is to reinterpret the task as an inpainting problem on a diptych canvas: the reference subject is placed (after background removal) in the left panel and the right panel is generated by a FLUX-based text-conditioned inpainting model, using a diptych text prompt that instructs the model to replicate the subject. A reference attention enhancement step rescales attention from the right-panel query to the left-panel key by a factor λ. The method is evaluated on DreamBench with DINO/CLIP metrics, a human preference study, ablations, and extensions to stylized generation and subject-driven editing.

Significance. If the central claim is sustained, this is a useful and conceptually clean contribution: it shows that an emergent capability of a large text-to-image model (diptych generation) can be repurposed as a zero-shot personalization mechanism without any training or fine-tuning, with concrete and reproducible components (background removal and attention rescaling). The paper is clearly written, the component ablations support the design choices, and the authors release code and weights, which strengthens reproducibility. The main risk is not internal circularity but external validity: the headline comparison to zero-shot image-prompting baselines is confounded by base-model scale and by undertuning of the only FLUX-based baseline, so the claimed margin over prior methods is not yet established.

major comments (4)
  1. [§4.2, Table 2; Appendix A] The headline claim that Diptych Prompting "significantly outperforms zero-shot image prompting methods" rests on a comparison that does not control for base-model scale. Most baselines use SD-v1.4, SD-v1.5, SD-XL, or Kandinsky 2.2, while Diptych Prompting uses FLUX.1-dev with a separately trained ControlNet. The only FLUX-based baseline, IP-Adapter (FLUX), is evaluated at a single conditioning scale of 0.6 (Appendix A states this is fixed for all IP-Adapter variants) with no tuning, no per-subject breakdown, and no error bars. Since Section 3.1 and Appendix I establish that the diptych capability is currently unique to FLUX, the observed margins may reflect base-model capacity rather than the inpainting reinterpretation. A matched-base, properly tuned IP-Adapter (FLUX) baseline (with several conditioning scales, per-subject results, and confidence intervals) is needed before the central comparison is secure.
  2. [§4.1, §4.3, Tables 3–4] The two key hyperparameters, ControlNet conditioning scale (0.95) and reference attention rescaling λ (1.3), are selected by looking at DreamBench metrics in Tables 3 and 4, yet the same DreamBench subjects are then used for the headline comparison in Table 2. There is no held-out validation split and no measure of variance (standard errors or confidence intervals) for any of the reported DINO/CLIP-I/CLIP-T numbers. Because the reported margins over baselines are sometimes small (e.g., CLIP-T 0.344 vs. IP-Adapter (FLUX) 0.351 in Table 2), the absence of variance estimates makes it difficult to assess whether the differences are meaningful. The authors should report error bars over subjects and, ideally, select hyperparameters on a separate validation set.
  3. [§3.1, Table S1] The paper's premise that only FLUX can generate accurate diptychs is stated strongly in Section 3.1, but Table S1 appears to contradict it: SD-XL achieves higher DINO (0.941) and CLIP-I (0.954) than FLUX (0.720 and 0.828), while only CLIP-T favors FLUX (0.352 vs. 0.288). The text should explain why higher DINO/CLIP-I in SD-XL does not indicate better diptych generation (for example, the model may copy the same image to both panels, inflating pairwise similarity while failing text alignment). Without this clarification, the quantitative evidence for FLUX's unique diptych capability is ambiguous, and this capability is load-bearing for the method, as Appendix I concedes that FLUX is currently the only usable base model.
  4. [§4.2, Table 1; Appendix C] The human preference study is an important part of the central claim, but the reporting is incomplete. Table 1 gives win/tie/lose percentages without any measure of inter-participant or per-subject variability, and the text states only that p<0.01 in a Wilcoxon signed-rank test. Appendix C says 450 responses were collected from 150 participants for each baseline and perspective, but it is not stated how ties are handled in the test, whether multiple responses per participant are treated as independent, or whether the comparison is per-image or per-subject. Without this information and without error bars on the win percentages, the strength of the human-preference evidence is difficult to gauge, especially for IP-Adapter (FLUX), where the win margin is the smallest (69.8% subject alignment, 65.2% text alignment).
minor comments (6)
  1. [§3.3, Eq. (7)] The notation in Eq. (7) introduces Q = [Qt; Qli; Qri], but Eq. (1) only defined Q = [Qt; Qi]. Please clarify that in the diptych inpainting setting the image tokens Qi are partitioned into left and right panels.
  2. [Fig. 3(b)] The schematic of reference attention enhancement would be easier to read if the submatrix W(Qri, Kli) that is multiplied by λ were explicitly outlined or colored; currently the λ symbol is shown but the exact block being rescaled is not visually delimited.
  3. [Table 3] The row labeled "ControlNet 0.95" appears directly below the SD-3 row, making it ambiguous whether ControlNet inpainting is applied to SD-3 or only to FLUX. Please clarify what base model and inpainting weights were used for each row.
  4. [Appendix C] The sentence "we collected 450 responses from 150 participants for each baseline and each perspective" is ambiguous about the number of pairwise comparisons per participant and the total number of responses per baseline. Please specify the exact experimental design and whether the Wilcoxon test was computed per subject or per image.
  5. [§4.3, Table 4] The ablation without Gseg yields higher DINO and CLIP-I (0.759, 0.783) than the chosen configuration (0.689, 0.758), and the text explains this as copy-and-paste behavior. To make this interpretation more than a post-hoc explanation, consider reporting a diversity metric or per-prompt text-alignment results for the no-Gseg condition.
  6. [Abstract and Fig. S5 caption] Minor language issues: the abstract says "reinterprets as an inpainting task" (likely "reinterprets subject-driven generation as an inpainting task"), and Fig. S5's caption mixes capitalization and formatting in "we control the λ from 1.0". These should be polished.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Diptych Prompting is an empirical prompting method whose central claim is evaluated against external baselines, not derived from its own inputs.

full rationale

The paper contains no derivation-level circularity. Diptych Prompting is an empirical prompting method: Eq. (6) defines the inpainting operation I_diptych = [G_seg(I_ref); I_gen] = F_theta(I_diptych, M_diptych, T_diptych) rather than assuming the claimed outcome. The central premise that FLUX generates coherent diptychs is established independently in Sec. 3.1 and Sec. D (Tab. S1) by quantitative comparisons across four models, and the inpainting extension is then tested against external DreamBench baselines and human preference judgments. The two tuning choices mentioned in Sec. 4.1 (ControlNet conditioning scale 0.95 and lambda = 1.3) are selected from the same benchmark used for the reported tables, and the FLUX-based IP-Adapter baseline is run at a fixed conditioning scale of 0.6 (Appendix A); these are evaluation-fairness and generalization concerns, not circularity. The method's components, background removal and reference attention rescaling, are ablated in Tab. 4, with the non-monotonic behavior in lambda indicating that improvement is not trivially forced. Section I's limitation that only FLUX currently has sufficient diptych capability narrows the scope but does not make the comparison circular, since FLUX's capability is treated as an external empirical property. No load-bearing self-citation or imported uniqueness theorem appears in the derivation.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method adds no new physical or model entities; it composes existing models (FLUX, ControlNet, Grounding DINO, SAM). The central claim rests on empirical assumptions about FLUX's diptych behavior, the segmentation pipeline, and hand-tuned inference hyperparameters that are selected on the evaluation benchmark.

free parameters (4)
  • lambda (reference attention rescaling) = 1.3
    Multiplier on the right-query/left-key attention submatrix; chosen from the sweep {1.0, 1.3, 1.5} evaluated on DreamBench in Table 4.
  • ControlNet conditioning scale = 0.95
    Chosen from the sweep {0.5, 0.8, 0.95} on DreamBench in Table 3; directly controls how strongly the inpainting module shapes the output.
  • Classifier-free guidance scale = 3.5
    Fixed inference hyperparameter; no reported sweep, but it affects text alignment and is standard for FLUX.
  • Number of denoising steps = 30
    Fixed inference hyperparameter; no reported sweep, but it affects sample quality.
assumptions (5)
  • domain assumption FLUX exhibits reliable diptych generation such that left-panel subject identity transfers to an inpainted right panel under text control.
    Section 3.1 and Figure 2 demonstrate diptych generation for selected prompts; the entire method depends on this emergent property holding for arbitrary DreamBench subjects.
  • domain assumption Grounding DINO plus SAM segmentation of the reference image by subject name yields a clean subject cutout that preserves identity-defining details.
    Section 3.2 Equation 4 removes the background; if detection or segmentation fails, the left panel either leaks background or discards subject details.
  • domain assumption Rescaling the right-query/left-key attention submatrix by lambda > 1 is a safe intervention that strengthens reference detail transfer without destabilizing generation.
    Section 3.3 introduces this heuristic; the paper provides no derivation from model internals, only empirical ablations.
  • domain assumption DreamBench DINO and CLIP metrics are valid proxies for subject alignment and text alignment.
    The metrics are standard in the field, but the paper does not validate them against human judgments for this specific method.
  • domain assumption Baseline numbers taken from the original papers are directly comparable to the authors' own re-evaluated numbers.
    Table 2 mixes reported values from ELITE, BLIP-Diffusion, Kosmos-G, Subject-Diffusion, MS-Diffusion, and IP-Adapter SD-XL with the authors' re-evaluation of IP-Adapter FLUX; different evaluation pipelines may not be directly comparable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot Subject-Driven Image Generator." pith.science (2026). https://pith.science/paper/HEY7POI2

@misc{pith2026241115466,
  author       = {Pith},
  title        = {Pith review of: Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot Subject-Driven Image Generator},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HEY7POI2}},
  note         = {Machine review of arXiv:2411.15466}
}
read the original abstract

Subject-driven text-to-image generation aims to produce images of a new subject within a desired context by accurately capturing both the visual characteristics of the subject and the semantic content of a text prompt. Traditional methods rely on time- and resource-intensive fine-tuning for subject alignment, while recent zero-shot approaches leverage on-the-fly image prompting, often sacrificing subject alignment. In this paper, we introduce Diptych Prompting, a novel zero-shot approach that reinterprets as an inpainting task with precise subject alignment by leveraging the emergent property of diptych generation in large-scale text-to-image models. Diptych Prompting arranges an incomplete diptych with the reference image in the left panel, and performs text-conditioned inpainting on the right panel. We further prevent unwanted content leakage by removing the background in the reference image and improve fine-grained details in the generated subject by enhancing attention weights between the panels during inpainting. Experimental results confirm that our approach significantly outperforms zero-shot image prompting methods, resulting in images that are visually preferred by users. Additionally, our method supports not only subject-driven generation but also stylized image generation and subject-driven image editing, demonstrating versatility across diverse image generation applications. Project page: https://diptychprompting.github.io/

Figures

Figures reproduced from arXiv: 2411.15466 by the authors.

Figure 1
Figure 1. Given a single reference image, our Diptych Prompting performs zero-shot subject-driven text-to-image generation through [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Diptych Generation Comparisons. We generate the diptych images with various TTI models from the following dip￾tych text: “A diptych with two side-by-side images of same cat. On the left, a photo of a cat in front of Eiffel Tower. On the right, replicate this cat exactly but as a photo of a cat in the jungle”. The first category [11, 22, 41, 48] involves fine-tuning on a small set of subject images (e.g., 3-5 images)… view at source ↗
Figure 3
Figure 3. (a) Overall Diptych Prompting Framework. Given the incomplete diptych Idiptych, text prompt Tdiptych describing the diptych, and the binary mask Mdiptych specifying the right panel as the inpainting target, FLUX with ControlNet module performs text-conditioned inpainting on the right panel while referencing the subject in the left panel. (b) Reference Attention Enhancement. To capture the granular details of the sub… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Background Removal Effects. Simple diptych inpaint￾ing exhibits content leakage from the reference image, including background, pose, and location. We mitigate this unwanted leak￾age through background removal by Gseg. plate used in Sec. 3.1. From the target text promp…
Figure 5
Figure 5. Figure 5: Qualitative Comparisons. Please zoom in for a more detailed view and better comparison. 6 [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 7
Figure 7. Figure 7 [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FreeLoRA: Enabling Training-Free LoRA Fusion for Autoregressive Multi-Subject Personalization

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A method for multi-subject image personalization that fuses independently trained LoRA modules at inference time on visual autoregressive models.

  2. IP-Prompter: Training-Free Theme-Specific Image Generation via Dynamic Visual Prompting

    cs.CV 2025-01 conditional novelty 6.0 of 10

    A training-free pipeline that uses dynamic visual prompts in an inpainting model to generate theme-consistent images without any model fine-tuning.

  3. IMAGINE-E: Image Generation Intelligence Evaluation of State-of-the-art Text-to-Image Models

    cs.CV 2025-01 conditional novelty 6.0 of 10

    A new evaluation suite finds that CLIPScore, HPSv2, and Aesthetic Score misjudge challenging text-to-image outputs, while GPT-4o and human ratings favor FLUX.1 and Ideogram2.0.

  4. From Wardrobe to Canvas: Wardrobe Polyptych LoRA for Part-level Controllable Human Image Generation

    cs.CV 2025-07 conditional novelty 5.0 of 10

    Wardrobe Polyptych LoRA lets a single diffusion model compose a person's face and clothing from multiple reference photos into new full-body images, generalizing to unseen identities without inference-time fine-tuning.

  5. DreamPoster: A Unified Framework for Image-Conditioned Generative Poster Design

    cs.CV 2025-07 conditional novelty 5.0 of 10

    DreamPoster fine-tunes Seedream3.0 with a deconstruction-recaptioning dataset pipeline and a three-stage curriculum to turn image-plus-text inputs into finished posters, reporting substantially higher usability than G...

  6. Insert Anything: Image Insertion via In-Context Editing in DiT

    cs.CV 2025-04 conditional novelty 5.0 of 10

    Insert Anything is a single model fine-tuned on 159,908 prompt-image pairs that performs mask- or text-guided insertion of people, objects, and garments from reference images into target scenes.

  7. In-Context Brush: Zero-shot Customized Subject Insertion with Context-Aware Latent Space Manipulation

    cs.CV 2025-05 conditional novelty 4.0 of 10

    In-Context Brush performs zero-shot customized subject insertion by amplifying prompt and reference attention and reweighting attention heads in a pre-trained Flux-Fill diffusion transformer.

Reference graph

Works this paper leans on

57 extracted references · 31 canonical work pages · cited by 7 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    The chosen one: Consistent characters in text- to-image diffusion models

    Omri Avrahami, Amir Hertz, Yael Vinker, Moab Arar, Shlomi Fruchter, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski. The chosen one: Consistent characters in text- to-image diffusion models. In ACM SIGGRAPH 2024 con- ference papers, pages 1–12, 2024. 3

    2024

  3. [3]

    ediff-i: Text-to-image dif- fusion models with an ensemble of expert denoisers

    Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Ji- aming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, et al. ediff-i: Text-to-image dif- fusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324, 2022. 2

    arXiv 2022

  4. [4]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jian- feng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023. 2, 3, 4

    work page 2023

  5. [5]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, S...

    work page 2020

  6. [6]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In Pro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 9650–9660, 2021. 5

    2021

  7. [7]

    Muse: Text-to-image generation via masked generative transformers

    Huiwen Chang, Han Zhang, Jarred Barber, Aaron Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Patrick Murphy, William T Freeman, Michael Ru- binstein, et al. Muse: Text-to-image generation via masked generative transformers. In International Conference on Machine Learning, pages 4055–4075. PMLR, 2023. 2

    work page 2023

  8. [8]

    Attend-and-excite: Attention-based se- mantic guidance for text-to-image diffusion models

    Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based se- mantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 42(4):1–10, 2023. 5

    2023

Show all 57 references
  1. [9]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first International Conference on Mach...

    2024

  2. [10]

    Make-a-scene: Scene- based text-to-image generation with human priors

    Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene- based text-to-image generation with human priors. In European Conference on Computer Vision , pages 89–106. Springer, 2022. 2

    2022

  3. [11]

    An image is worth one word: Personalizing text-to-image gener- ation using textual inversion

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth one word: Personalizing text-to-image gener- ation using textual inversion. In The Eleventh International Conference on Learning Representations, 2023. 2, 3

    2023

  4. [12]

    Renoise: Real image inversion through iterative noising

    Daniel Garibi, Or Patashnik, Andrey V oynov, Hadar Averbuch-Elor, and Daniel Cohen-Or. Renoise: Real image inversion through iterative noising. In European Conference on Computer Vision, 2024. 2

    2024

  5. [13]

    Prompt-to-prompt image editing with cross attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. In International Confer- ence on Learning Representations, 2023. 2, 5

    2023

  6. [14]

    Style aligned image generation via shared atten- tion

    Amir Hertz, Andrey V oynov, Shlomi Fruchter, and Daniel Cohen-Or. Style aligned image generation via shared atten- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 4775–4785,

  7. [15]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. 5

    2022 arXiv

  8. [16]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 2

    2020

  9. [17]

    In-context lora for diffusion transformers, 2024

    Lianghua Huang, Wei Wang, Zhi-Fan Wu, Yupeng Shi, Huanzhang Dou, Chen Liang, Yutong Feng, Yu Liu, and Jin- gren Zhou. In-context lora for diffusion transformers, 2024. 3

    2024

  10. [18]

    An edit friendly ddpm noise space: Inversion and manipulations

    Inbar Huberman-Spiegelglas, Vladimir Kulikov, and Tomer Michaeli. An edit friendly ddpm noise space: Inversion and manipulations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12469– 12478, 2024. 2

    2024

  11. [19]

    Hq-edit: A high-quality dataset for instruction-based image editing

    Mude Hui, Siwei Yang, Bingchen Zhao, Yichun Shi, Heng Wang, Peng Wang, Yuyin Zhou, and Cihang Xie. Hq-edit: A high-quality dataset for instruction-based image editing. arXiv preprint arXiv:2404.09990, 2024. 3

    2024 arXiv

  12. [20]

    Elucidating the design space of diffusion-based generative models

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35:26565–26577, 2022. 2

    2022

  13. [21]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- 9 head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4015–4026, 2023. 4

    2023

  14. [22]

    Multi-concept customization of text- to-image diffusion

    Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shecht- man, and Jun-Yan Zhu. Multi-concept customization of text- to-image diffusion. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 1931–1941, 2023. 2, 3

    1931

  15. [23]

    Flux.1-dev

    Black Forest Labs. Flux.1-dev. https : / / huggingface . co / black - forest - labs / FLUX.1-dev, 2024. 1, 2, 3, 4, 5, 12

    2024

  16. [24]

    Blip-diffusion: Pre- trained subject representation for controllable text-to-image generation and editing

    Dongxu Li, Junnan Li, and Steven Hoi. Blip-diffusion: Pre- trained subject representation for controllable text-to-image generation and editing. 2024. 2, 3, 5, 7, 12

    2024

  17. [25]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 12

    2023

  18. [26]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  19. [27]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Con- ference on Computer Vision, 2024. 4

    2024

  20. [28]

    Subject- diffusion: Open domain personalized text-to-image genera- tion without test-time fine-tuning

    Jian Ma, Junhao Liang, Chen Chen, and Haonan Lu. Subject- diffusion: Open domain personalized text-to-image genera- tion without test-time fine-tuning. In ACM SIGGRAPH 2024 Conference Papers, pages 1–12, 2024. 2, 3, 5, 7, 12

    2024

  21. [29]

    On distillation of guided diffusion models

    Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 14297–14306, 2023. 5

    2023

  22. [30]

    Null-text inversion for editing real images using guided diffusion models

    Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. 2023 ieee. In CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) , pages 6038–6047, 2022. 2

    2023

  23. [31]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In Interna- tional Conference on Machine Learning, 2022. 2

    2022

  24. [32]

    Kosmos-g: Generating images in context with multimodal large language models

    Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei. Kosmos-g: Generating images in context with multimodal large language models. In The Twelfth International Conference on Learning Representa- tions, 2024. 2, 3, 5, 7, 12

    2024

  25. [33]

    λ-eclipse: Multi-concept personalized text-to-image diffusion models by leveraging CLIP latent space

    Maitreya Patel, Sangmin Jung, Chitta Baral, and Yezhou Yang. λ-eclipse: Multi-concept personalized text-to-image diffusion models by leveraging CLIP latent space. arXiv preprint arXiv:2402.05195, 2024. 2, 3, 5, 7, 12

    2024 arXiv

  26. [34]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  27. [35]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations ,

  28. [36]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  29. [37]

    Hierarchical text-conditional image gener- ation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gener- ation with clip latents. arXiv preprint arXiv:2204.06125, 1 (2):3, 2022. 2

    2022 arXiv

  30. [38]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 2, 4, 12

    2022

  31. [39]

    U- net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, pa...

    2015

  32. [40]

    Rb-modulation: Training-free personalization of diffu- sion models using stochastic optimal control

    Litu Rout, Yujia Chen, Nataniel Ruiz, Abhishek Kumar, Constantine Caramanis, Sanjay Shakkottai, and Wen-Sheng Chu. Rb-modulation: Training-free personalization of diffu- sion models using stochastic optimal control. arXiv preprint arXiv:2405.17401, 2024. 2, 14

    2024 arXiv

  33. [41]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2250...

    2023

  34. [42]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information...

    2022

  35. [43]

    Styledrop: Text-to-image synthesis of any style

    Kihyuk Sohn, Lu Jiang, Jarred Barber, Kimin Lee, Nataniel Ruiz, Dilip Krishnan, Huiwen Chang, Yuanzhen Li, Irfan Essa, Michael Rubinstein, et al. Styledrop: Text-to-image synthesis of any style. Advances in Neural Information Pro- cessing Systems, 36, 2024. 2, 8, 14

    2024

  36. [44]

    Denois- ing diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois- ing diffusion implicit models. In International Conference on Learning Representations, 2021. 2

    2021

  37. [45]

    Score-based 10 generative modeling through stochastic differential equa- tions

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based 10 generative modeling through stochastic differential equa- tions. In International Conference on Learning Represen- tations, 2021. 2, 3, 7

    2021

  38. [46]

    Training-free con- sistent text-to-image generation

    Yoad Tewel, Omri Kaduri, Rinon Gal, Yoni Kasten, Lior Wolf, Gal Chechik, and Yuval Atzmon. Training-free con- sistent text-to-image generation. In ACM Transactions on Graphics (TOG), 2024. 3

    2024

  39. [47]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017. 2

    2017

  40. [48]

    p+: Extended textual conditioning in text-to- image generation

    Andrey V oynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman. p+: Extended textual conditioning in text-to- image generation. arXiv preprint arXiv:2303.09522 , 2023. 2, 3

    2023 arXiv

  41. [49]

    Oneactor: Consistent subject generation via cluster- conditioned guidance

    Jiahao Wang, Caixia Yan, Haonan Lin, Weizhan Zhang, Mengmeng Wang, Tieliang Gong, Guang Dai, and Hao Sun. Oneactor: Consistent subject generation via cluster- conditioned guidance. In Advances in Neural Information Processing Systems, 2024. 3

    2024

  42. [50]

    Ms-diffusion: Multi-subject zero-shot im- age personalization with layout guidance

    X Wang, Siming Fu, Qihan Huang, Wanggui He, and Hao Jiang. Ms-diffusion: Multi-subject zero-shot im- age personalization with layout guidance. arXiv preprint arXiv:2406.07209, 2024. 2, 3, 5, 7, 12

    2024 arXiv

  43. [51]

    Elite: Encoding visual concepts into textual embeddings for customized text-to-image gener- ation

    Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image gener- ation. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 15943–15953, 20...

    2023

  44. [52]

    Flux context window editing v3.3f (fill model) fix anything in any context

    xiaozhijason. Flux context window editing v3.3f (fill model) fix anything in any context. https://civitai.com/ models / 933018 ? modelVersionId = 1044405 ,

  45. [53]

    Inversion-free image editing with natural language

    Sihan Xu, Yidong Huang, Jiayi Pan, Ziqiao Ma, and Joyce Chai. Inversion-free image editing with natural language. arXiv preprint arXiv:2312.04965, 2023. 2

    2023 arXiv

  46. [54]

    Paint by example: Exemplar-based image editing with diffusion mod- els

    Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing with diffusion mod- els. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 18381–18391,

  47. [55]

    Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models. arXiv preprint arXiv:2308.06721,

    arXiv

  48. [56]

    Jedi: Joint- image diffusion models for finetuning-free personalized text- to-image generation

    Yu Zeng, Vishal M Patel, Haochen Wang, Xun Huang, Ting- Chun Wang, Ming-Yu Liu, and Yogesh Balaji. Jedi: Joint- image diffusion models for finetuning-free personalized text- to-image generation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recogni...

    2024

  49. [57]

    A diptych with two side-by- side images of same style. On the left, {original image de- scription}. On the right, replicate this style exactly but as {target image description}

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 3, 15 11 Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot...

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.