Pith. sign in

REVIEW 4 major objections 4 minor 76 references

FreeCus: Free Lunch Subject-driven Customization in Diffusion Transformers

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FreeCus claims that a single reference image plus attention steering is enough to personalize a diffusion transformer with no training.

desk verdict FreeCus is a credible training-free customization recipe with real engineering value, but its evaluation is optimistic and its key assumption about random-noise reference trajectories is hand-wavy. read the letter →

arxiv 2507.15249 v1 pith:SCDBYG5E submitted 2025-07-21 cs.CV

Yanbing Zhang , Zhe Wang , Qin Zhou , Mengping Yang This is my paper
classification cs.CV
keywords training-freecustomizationsubject-drivengenerationdiffusiontransformersattentionsharingzero-shotpersonalizationrectifiedflowsemanticfeaturecompensationFlux.1
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that subject-driven personalization of a modern diffusion transformer needs no training at all: no per-subject token optimization and no trained image encoder. FreeCus instead manipulates the pretrained model's attention at inference time, sharing keys and values from one reference image through a small set of 'vital' layers, reversing the model's noise-shift schedule to keep those shared features detailed, and augmenting the prompt with a short MLLM-generated subject caption. The reported experiments put FreeCus's subject similarity above most trained baselines while retaining text control. If the claim holds, one-image customization becomes immediate, cheap, and usable with any compatible diffusion-transformer pipeline.

What carries the argument

Three objects carry the argument. Pivotal attention sharing (Eq. 3) restricts reference-feature injection to a set $\mathcal{V}$ of ten 'vital' layers, using a subject mask $m_r$ and scalar weights $\lambda_r, \lambda_p$ to balance identity and prompt adherence. Adjusted noise shifting replaces the standard schedule with $\sigma'_t = \frac{e^{-\mu}}{e^{-\mu}+1} t - 1$, so the reference trajectory $z_t=(1-\sigma'_t)z_{\mathrm{ref}}+\sigma'_t\epsilon$ emphasizes low noise and fine details. Semantic feature compensation appends a short LLM-filtered vision-language caption to the prompt. Together these are intended to transfer layout and identity while avoiding the copy-paste effect of full attention sharing.

What would settle it

Use one fixed subject and prompt, and replace the random-noise reference trajectory of Sec. 3.2 with a true inversion trajectory (for example, a rectified-flow or DDIM inversion) while keeping pivotal attention sharing, adjusted noise shifting, and semantic compensation fixed; if the reported CLIP-I and DINO subject-similarity scores drop by a large margin, the self-correction assumption about $\sigma_0=0$ is the load-bearing part and the method does not transfer to other noise schedules.

Watch

Extended reading notes

Core claim

The paper's central claim is that a diffusion transformer such as Flux.1 already contains enough knowledge to keep a subject consistent across new contexts, and that this ability can be unlocked without any training by steering attention during denoising. The steering has three parts: pivotal attention sharing, which concatenates the reference image's keys and values into the attention computation at ten vital layers while masking the background; adjusted noise shifting, which builds the reference trajectory with a reversed shift so that lower-noise, detail-rich samples are attended to; and semantic feature compensation, which adds a concise filtered caption of the subject's appearance to the text prompt. With these components, the paper reports subject similarity scores (CLIP-I 0.853, DINO 0.696) that exceed most trained competitors on the benchmark, and text controllability (CLIP-T 0.308) that remains competitive, leading to the claim that genuinely zero-shot subject-driven synthesis is achievable.

Load-bearing premise

The load-bearing premise is that random noise added to the reference image yields a diffusion trajectory whose attention maps reliably carry the subject's identity, with early noise-induced errors automatically corrected as the schedule reaches $z_0=z_{\mathrm{ref}}$.

Editorial extensions

If this is right

  • If FreeCus's claim holds, a single reference photo and a text prompt are enough to generate the subject in new scenes, styles, and poses without per-subject finetuning.
  • Because the method is training-free and layer-localized, it can be attached to any DiT-based pipeline; the paper demonstrates this with inpaint and depth-control variants and by boosting a vision-language-guided Flux model.
  • The reported tension between subject fidelity and text control is tunable: raising the reference/text attention weights or the negative shift magnitude raises similarity at the cost of text alignment, so users can pick an operating point.
  • If stronger vision-language models produce more accurate subject captions, the semantic compensation component should improve without retraining the diffusion model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A reader could push the paper's logic further: if identity lives in a few vital layers of Flux.1, the same layer selection may transfer to other diffusion transformers, making the method a general zero-shot personalization primitive.
  • A direct test not run in the paper would swap the random-noise trajectory for a true inversion trajectory; if fidelity collapses, the self-correction assumption about the trajectory is the crux.
  • The artifact-control dropout mentioned in the paper suggests an adaptive, per-subject dropout rate as a natural extension to trade artifacts against fidelity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper presents FreeCus, an inference-time, training-free framework for subject-driven image customization with the Flux.1 diffusion transformer. The method has three components: pivotal attention sharing (PAS), which injects keys/values from a noisy reference trajectory into ten selected DiT layers; adjusted noise shifting (ANS), which reverses the noise schedule for the reference to emphasize fine detail; and semantic feature compensation (SFC), which uses MLLMs to generate a filtered subject-specific caption that augments the prompt. Experiments on DreamBench++ compare FreeCus with optimization-based and encoder-based personalization methods in terms of CLIP-T, CLIP-I, and DINO; the authors also demonstrate style transfer, inpainting, and integration with other DiT-based pipelines. The central claim is that FreeCus achieves state-of-the-art or comparable subject fidelity without any training or per-subject optimization.

Significance. If the empirical results are taken at face value, FreeCus offers a practical and simple alternative to training-based personalization for DiT models, with the advantage of being immediately applicable to a new subject from a single reference image. The paper includes a thorough component-wise ablation and qualitative evidence that the method balances subject fidelity with text controllability. The strengths are the clear engineering recipe, the use of publicly available models for segmentation and captions, and the demonstration of compatibility with inpainting and control modules. However, the claims are currently weakened by the evaluation protocol, as detailed below, and the mechanism lacks a rigorous account of why random-noise reference trajectories provide faithful subject features.

major comments (4)
  1. [Sec. 4.3 and Table 1] The final hyperparameters (λr=λp=1.10, shift multiplier -1.0, and the filtered LVLM caption strategy) are selected by scanning values on DreamBench++ (Sec. 4.3, 'Hyperparameter analysis' and 'Shift type analysis'), and the same benchmark is then used to report the final results in Table 1. This constitutes test-set tuning; without a separate validation split or cross-validation, the reported scores are optimistic and the comparison with competitors is not on equal footing. The differences against some baselines in Table 1 are small (e.g., CLIP-T 0.308 vs 0.314 for IP-Adapter FLUX.1, DINO 0.696 vs 0.729 for MS-Diffusion), so an independent selection would be necessary to support the state-of-the-art/comparable claim. Please provide a validation protocol or a sensitivity analysis that does not touch the final test metrics.
  2. [Sec. 3.2] The reference trajectory is built via Eq. (4) with random noise ε, and the paper asserts that inaccuracies in higher-timestep attention are 'progressively corrected' because σ0=0. This justification is incomplete: the target denoising process establishes global layout at early steps, precisely where the reference is noise-dominated, so unreliable reference key/value features could mislead the target before any correction occurs. ANS (Sec. 3.3) reduces the reference noise level but introduces a mismatch between reference (σ'_t) and target (σ_t) schedules; no analysis is provided of attention fidelity versus timestep or of the effect of this distribution shift. Please provide evidence (e.g., attention map visualizations or a fidelity metric as a function of t) that the random-noise trajectory yields faithful subject features at the timesteps where PAS is applied.
  3. [Sec. 7 and Sec. 4.1] The evaluation in Table 1 mixes different resolutions across methods. The paper's method uses 512×512 (Sec. 4.1), while Qwen2VL-Flux is generated at 1024×1024 (supplementary Sec. 7). Resolution affects CLIP and DINO scores (typically higher resolution improves fine-grained similarity), so the comparison is confounded. Please evaluate all methods at a common resolution, or at least report results for the proposed method at both 512×512 and 1024×1024 to isolate the resolution factor.
  4. [Sec. 4.1 and Table 1] No error bars or statistical significance are reported. With only four generated images per subject-prompt (Sec. 4.1), the averaged CLIP-I, DINO, and CLIP-T values carry considerable variance; several differences in Table 1 are within a few hundredths. Reporting standard errors or a significance test is necessary to support the claim of state-of-the-art or comparable performance.
minor comments (4)
  1. [Sec. 3.3 and Sec. 4.3] The notation 'µ*' used in the 'Shift type analysis' of Sec. 4.3 is never defined in the main text; the relationship between this parameter and the shift µ in Eq. (5) should be clarified.
  2. [Sec. 4.2] The statement 'surpasses most competitors in subject similarity' is imprecise because IP-Adapter-Plus and MS-Diffusion in Table 1 have higher CLIP-I and DINO; please explicitly qualify which competitors are surpassed.
  3. [Sec. 3.2] The notation switches between z0 and z_ref in Eq. (4) and the surrounding text without stating they are identical; please align the notation throughout.
  4. [Sec. 4.1 and supplementary Sec. 6] The selection of the ten 'vital layers' is not described precisely in the main text; providing the layer indices or the criterion would make the method more reproducible and the supplementary ablation easier to interpret.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: FreeCus is an inference-time recipe whose components are not defined in terms of the reported metrics; the single self-citation is a related-work pointer and is not load-bearing.

full rationale

The paper makes no formal derivation claim: PAS (Eq. 3) directly concatenates reference keys and values into attention, which is the intended mechanism for raising CLIP-I/DINO rather than a fitted parameter renamed as a prediction. The Sec. 3.2 claim that the random-noise trajectory remains valid ('Inaccuracies in the attention computed at high timesteps are progressively corrected as the noise diminishes, since sigma_0 = 0 ensures z_0 = z_ref') is an asserted assumption, not a circular step; if it fails, the method is less robust, but the claim is not equivalent to its own input. ANS and SFC are heuristic ablations whose hyperparameters (lambda_r, lambda_p, shift magnitude, caption style) are selected on DreamBench++ and then re-reported on the same benchmark; this is a benchmark tuning or overfitting concern, not a derivation that reduces to the metric by construction. The only self-citation, [66], appears in the related-work taxonomy ('The first [17,26,33,66] fine-tunes base models...') and is not used to justify any unique choice or to forbid alternatives. No uniqueness theorem from the authors is invoked. Therefore no step in the paper is circular.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

No new entities are introduced; the method is built entirely from pretrained components (Flux.1, BirefNet, Qwen2-VL, Qwen2.5). The main free parameters are the attention scaling weights and the shift multiplier, both tuned on the evaluation benchmark. The key assumptions are that attention sharing, vital layers, segmentation, random-noise trajectories, and MLLM captions transfer reliably to the customization task.

free parameters (2)
  • lambda_r and lambda_p attention scaling = 1.10
    Selected in Sec. 4.3 by scanning values on DreamBench++ and choosing the configuration that balances CLIP-T and DINO. Affects the relative influence of reference image vs. target prompt.
  • shift direction multiplier k = -1.0
    Chosen from the ablation in Sec. 4.3 with values 0, -0.5, -1.0, -2.0. More negative shift improves subject similarity but lowers CLIP-T; -1.0 was selected as optimal.
assumptions (5)
  • domain assumption Flux.1's MM-DiT blocks support concatenated key/value injection from a reference image (Eq. 2-3).
    The method relies on the pretrained model's attention mechanism accepting extra reference keys and values without retraining or finetuning.
  • domain assumption The vital layers identified by Stable Flow [6] for editing are also the right layers for subject customization in Flux.1.
    The paper selects ten critical layers based on [6] and validates via ablations in the supplementary, but does not re-derive the layer selection for Flux.1.
  • domain assumption A subject mask from BirefNet [67] accurately isolates the reference subject.
    Masked attention sharing in Eq. 3 depends on segmentation quality; the paper does not analyze failure cases where the mask leaks background or cuts off part of the subject.
  • domain assumption Rectified flow forward process with random noise yields valid reference trajectories for attention extraction.
    Sec. 3.2 argues trajectories are valid because sigma'_0 = 0, but the behavior at high noise levels is not formally bounded.
  • domain assumption MLLM-generated and LLM-filtered captions improve semantic fidelity without reducing text controllability.
    Sec. 3.4 and caption ablations support this empirically; the filtering step assumes the LLM can reliably remove background or action information from the caption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FreeCus: Free Lunch Subject-driven Customization in Diffusion Transformers." pith.science (2026). https://pith.science/paper/SCDBYG5E

@misc{pith2026250715249,
  author       = {Pith},
  title        = {Pith review of: FreeCus: Free Lunch Subject-driven Customization in Diffusion Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SCDBYG5E}},
  note         = {Machine review of arXiv:2507.15249}
}
read the original abstract

In light of recent breakthroughs in text-to-image (T2I) generation, particularly with diffusion transformers (DiT), subject-driven technologies are increasingly being employed for high-fidelity customized production that preserves subject identity from reference inputs, enabling thrilling design workflows and engaging entertainment. Existing alternatives typically require either per-subject optimization via trainable text embeddings or training specialized encoders for subject feature extraction on large-scale datasets. Such dependencies on training procedures fundamentally constrain their practical applications. More importantly, current methodologies fail to fully leverage the inherent zero-shot potential of modern diffusion transformers (e.g., the Flux series) for authentic subject-driven synthesis. To bridge this gap, we propose FreeCus, a genuinely training-free framework that activates DiT's capabilities through three key innovations: 1) We introduce a pivotal attention sharing mechanism that captures the subject's layout integrity while preserving crucial editing flexibility. 2) Through a straightforward analysis of DiT's dynamic shifting, we propose an upgraded variant that significantly improves fine-grained feature extraction. 3) We further integrate advanced Multimodal Large Language Models (MLLMs) to enrich cross-modal semantic representations. Extensive experiments reflect that our method successfully unlocks DiT's zero-shot ability for consistent subject synthesis across diverse contexts, achieving state-of-the-art or comparable results compared to approaches that require additional training. Notably, our framework demonstrates seamless compatibility with existing inpainting pipelines and control modules, facilitating more compelling experiences. Our code is available at: https://github.com/Monalissaa/FreeCus.

Figures

Figures reproduced from arXiv: 2507.15249 by the authors.

Figure 1
Figure 1. Given the user-provided image as a reference, our proposed method synthesizes a consistent subject while adhering to flexible [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Issues with StyleAligned [22] in personalization. At￾tention sharing causes text misalignment, e.g., it neither renders the anime style nor synthesizes the intended hat. Note: a mask is applied with StyleAligned to avoid fully replicating the input. applications, subject-driven generation [17, 38, 50], also termed customization or personalization, has gained promi￾nence for enabling contextually diverse image genera… view at source ↗
Figure 3
Figure 3. Method overview. Our approach transfers characteristics from a reference image z0 to a target image z˜0 through three mecha￾nisms: (1) pivotal attention sharing, masking attention in critical layers to inject structural features while preserving editing flexibility; (2) adjusted dynamic shifting, deriving an improved diffusion trajectory (z1, ..., zT ) processed via rectified flow to enhance detail alignment between… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: The noise scaling σ under different shift directions across all timesteps at a target resolution of 512 × 512. these intermediate samples. However, such methods often fail [39] or produce erroneous trajectories [27]. Instead, we inject random noise ϵ into z ref via a r…
Figure 5
Figure 5. Figure 5: Illustration of the subject caption generation process with Multimodal LLMs. further evaluate different shift directions to identify the op￾timal configuration. 3.4. Semantic Features Compensation (SFC) In addition to fine details, semantic features, such as color, can…
Figure 6
Figure 6. Figure 6: Qualitative evaluation results. Comparison across various subjects and contexts reveals: OminiControl and DreamBooth-L lack subject fidelity; IP-Adapter-Plus and Qwen2VL-Flux fail at text alignment; MS-Diffusion generates background artifacts (rows 1 and 4). In contras…
Figure 7
Figure 7. Figure 7: Impact visualization of each proposed component. optimization-free methods) and carefully adjusts the output distribution of the base model via the proposed strategies. Qualitative comparisons. Considering page constraints, we present qualitative results from five cont…
Figure 8
Figure 8. Figure 8: (a). This adaptation requires only a modification to the prompt for semantic subject caption generation (details provided in supplementary materials). Compatibility with other methods. The zero-shot na￾ture of our approach enables seamless integration with other DiT-ba…
Figure 9
Figure 9. Figure 9: Ablations on vital layer selection. Target Prompt: “ An axolotl lying on the sandy bottom of a freshwater stream. ” Reference Ours (Qwen2.5 VL) Ours (Qwen2 VL) a pale body with translucent skin, blue eyes, and feathery extemal gills a white body, pink fringed limbs, bl…
Figure 14
Figure 14. Figure 14: Prompt for Detailed Subject Caption [PITH_FULL_IMAGE:figures/full_fig_p014_14.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

76 extracted references · 43 canonical work pages

  1. [1]

    Cross-image attention for zero- shot appearance transfer

    Yuval Alaluf, Daniel Garibi, Or Patashnik, Hadar Averbuch- Elor, and Daniel Cohen-Or. Cross-image attention for zero- shot appearance transfer. In ACM SIGGRAPH 2024 Confer- ence Papers, pages 1–12, 2024. 3

    work page 2024

  2. [2]

    Build- ing normalizing flows with stochastic interpolants

    Michael Samuel Albergo and Eric Vanden-Eijnden. Build- ing normalizing flows with stochastic interpolants. In The Eleventh International Conference on Learning Representa- tions, 2023. 4

    work page 2023

  3. [3]

    Blended diffusion for text-driven editing of natural images

    Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18208–18218, 2022. 2, 3

    work page 2022

  4. [4]

    Break-a-scene: Extracting multi- ple concepts from a single image

    Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen- Or, and Dani Lischinski. Break-a-scene: Extracting multi- ple concepts from a single image. In SIGGRAPH Asia 2023 Conference Papers, pages 1–12, 2023. 2

    work page 2023

  5. [5]

    Diffuhaul: A training-free method for object dragging in images

    Omri Avrahami, Rinon Gal, Gal Chechik, Ohad Fried, Dani Lischinski, Arash Vahdat, and Weili Nie. Diffuhaul: A training-free method for object dragging in images. In SIG- GRAPH Asia 2024 Conference Papers, pages 1–12, 2024. 3

    work page 2024

  6. [6]

    Stable flow: Vital layers for training-free image editing

    Omri Avrahami, Or Patashnik, Ohad Fried, Egor Nemchi- nov, Kfir Aberman, Dani Lischinski, and Daniel Cohen- Or. Stable flow: Vital layers for training-free image editing. arXiv preprint arXiv:2411.14430, 2024. 2, 3, 4

    arXiv 2024

  7. [7]

    ediff-i: Text-to-image dif- fusion models with an ensemble of expert denoisers

    Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Ji- aming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, et al. ediff-i: Text-to-image dif- fusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324, 2022. 2

    arXiv 2022

  8. [8]

    Announcing black forest labs

    Black Forest Labs. Announcing black forest labs. https: / / blackforestlabs . ai / announcing - black - forest-labs/, 2023. Accessed: 2024-4. 1, 2, 3, 5

    work page 2023

Show all 76 references
  1. [9]

    Masactrl: Tuning-free mu- tual self-attention control for consistent image synthesis and editing

    Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xi- aohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mu- tual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF international con- ference on computer vision, pages 22560–2257...

    2023

  2. [10]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In Pro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 9650–9660, 2021. 5

    2021

  3. [11]

    Pixart- α: Fast training of diffusion transformer for photorealistic text-to-image synthesis

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart- α: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023. 2

    arXiv

  4. [12]

    Pixart- σ: Weak-to-strong training of diffusion transformer for 4k text-to-image generation

    Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- σ: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. In European Conference on Computer Vision , pages 74–91. Spri...

    2024

  5. [13]

    Anydoor: Zero-shot object-level im- age customization

    Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level im- age customization. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 6593–6602, 2024. 2

    2024

  6. [14]

    Turboedit: Text-based image editing using few-step diffusion models

    Gilad Deutch, Rinon Gal, Daniel Garibi, Or Patashnik, and Daniel Cohen-Or. Turboedit: Text-based image editing using few-step diffusion models. In SIGGRAPH Asia 2024 Con- ference Papers, pages 1–12, 2024. 1

    2024

  7. [15]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural informa- tion processing systems, 34:8780–8794, 2021. 2

    2021

  8. [16]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first International Conference on Mach...

    2024

  9. [17]

    An image is worth one word: Personalizing text-to-image gener- ation using textual inversion

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth one word: Personalizing text-to-image gener- ation using textual inversion. In The Eleventh International Conference on Learning Representations. 2, 5

  10. [18]

    Encoder-based domain tuning for fast personalization of text-to-image models.ACM Transactions on Graphics (TOG), 42(4):1–13, 2023

    Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. Encoder-based domain tuning for fast personalization of text-to-image models.ACM Transactions on Graphics (TOG), 42(4):1–13, 2023. 3

    2023

  11. [19]

    Lumina-t2x: Transforming text into any modality, reso- lution, and duration via flow-based large diffusion transform- ers

    Peng Gao, Le Zhuo, Ziyi Lin, Chris Liu, Junsong Chen, Ruoyi Du, Enze Xie, Xu Luo, Longtian Qiu, Yuhang Zhang, et al. Lumina-t2x: Transforming text into any modality, reso- lution, and duration via flow-based large diffusion transform- ers. arXiv preprint arXiv:2405.05945, 2024. 2

    2024 arXiv

  12. [20]

    Tokenflow: Consistent diffusion features for consistent video editing

    Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arXiv:2307.10373, 2023. 4

    2023 arXiv

  13. [21]

    Prompt-to-prompt image editing with cross-attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. Prompt-to-prompt image editing with cross-attention control. In The Eleventh Inter- national Conference on Learning Representations, 2023. 3

    2023

  14. [22]

    Style aligned image generation via shared atten- tion

    Amir Hertz, Andrey V oynov, Shlomi Fruchter, and Daniel Cohen-Or. Style aligned image generation via shared atten- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 4775–4785,

  15. [23]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 2

    2020

  16. [24]

    Cascaded diffu- sion models for high fidelity image generation

    Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffu- sion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1–33, 2022. 2

    2022

  17. [25]

    Lora: Low- rank adaptation of large language models

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low- rank adaptation of large language models. In International Conference on Learning Representations, 2022. 5

    2022

  18. [26]

    In-context lora for diffusion transformers

    Lianghua Huang, Wei Wang, Zhi-Fan Wu, Yupeng Shi, Huanzhang Dou, Chen Liang, Yutong Feng, Yu Liu, and Jin- gren Zhou. In-context lora for diffusion transformers. arXiv preprint arXiv:2410.23775, 2024. 2

    2024 arXiv

  19. [27]

    An edit friendly ddpm noise space: Inversion and manipulations

    Inbar Huberman-Spiegelglas, Vladimir Kulikov, and Tomer Michaeli. An edit friendly ddpm noise space: Inversion and manipulations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12469– 12478, 2024. 4

    2024

  20. [28]

    Open- clip, 2021

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Han- naneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Open- clip, 2021. If you use this software, please cite it as below. 2

    2021

  21. [29]

    Auto-encoding variational bayes

    Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. 3

    2013 arXiv

  22. [30]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4015–4026, 2023. 5

    2023

  23. [31]

    Anymaker: Zero-shot gen- eral object customization via decoupled dual-level id injec- tion

    Lingjie Kong, Kai Wu, Xiaobin Hu, Wenhui Han, Jin- long Peng, Chengming Xu, Donghao Luo, Jiangning Zhang, Chengjie Wang, and Yanwei Fu. Anymaker: Zero-shot gen- eral object customization via decoupled dual-level id injec- tion. arXiv preprint arXiv:2406.11643, 2024. 3

    2024 arXiv

  24. [32]

    Flowedit: Inversion-free text-based editing using pre-trained flow models

    Vladimir Kulikov, Matan Kleiner, Inbar Huberman- Spiegelglas, and Tomer Michaeli. Flowedit: Inversion-free text-based editing using pre-trained flow models. arXiv preprint arXiv:2412.08629, 2024. 3

    2024 arXiv

  25. [33]

    Multi-concept customization of text-to-image diffusion

    Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1931–1941, 2023. 2, 3

    1931

  26. [34]

    Blip-diffusion: Pre- trained subject representation for controllable text-to-image generation and editing

    Dongxu Li, Junnan Li, and Steven Hoi. Blip-diffusion: Pre- trained subject representation for controllable text-to-image generation and editing. Advances in Neural Information Pro- cessing Systems, 36, 2024. 3, 5, 2

    2024

  27. [35]

    Flow matching for genera- tive modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maxim- ilian Nickel, and Matthew Le. Flow matching for genera- tive modeling. In The Eleventh International Conference on Learning Representations, 2023. 4

    2023

  28. [36]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations (ICLR), 2023. 2, 4

    2023

  29. [37]

    Qwen2vl-flux: Unifying image and text guidance for controllable image generation, 2024

    Pengqi Lu. Qwen2vl-flux: Unifying image and text guidance for controllable image generation, 2024. 5, 2

    2024

  30. [38]

    Realcus- tom++: Representing images as real-word for real-time cus- tomization

    Zhendong Mao, Mengqi Huang, Fei Ding, Mingcong Liu, Qian He, Xiaojun Chang, and Yongdong Zhang. Realcus- tom++: Representing images as real-word for real-time cus- tomization. arXiv preprint arXiv:2408.09744 , 2024. 2, 3, 5

    2024

  31. [39]

    Null-text inversion for editing real im- ages using guided diffusion models

    Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real im- ages using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6038–6047, 2023. 1, 3, 4

    2023

  32. [40]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models.arXiv preprint arXiv:2112.10741, 2021

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models.arXiv preprint arXiv:2112.10741, 2021. 2

    2021 arXiv

  33. [41]

    Kosmos-g: Generating images in context with multimodal large language models

    Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei. Kosmos-g: Generating images in context with multimodal large language models. In The Twelfth International Conference on Learning Representa- tions, 2024. 3

    2024

  34. [42]

    λ-eclipse: Multi-concept personalized text-to-image diffusion models by leveraging clip latent space

    Maitreya Patel, Sangmin Jung, Chitta Baral, and Yezhou Yang. λ-eclipse: Multi-concept personalized text-to-image diffusion models by leveraging clip latent space. arXiv preprint arXiv:2402.05195, 2024. 3

    2024 arXiv

  35. [43]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  36. [44]

    Dreambench++: A human-aligned bench- mark for personalized image generation

    Yuang Peng, Yuxin Cui, Haomiao Tang, Zekun Qi, Runpei Dong, Jing Bai, Chunrui Han, Zheng Ge, Xiangyu Zhang, and Shu-Tao Xia. Dreambench++: A human-aligned bench- mark for personalized image generation. arXiv preprint arXiv:2406.16855, 2024. 5, 2

    2024 arXiv

  37. [45]

    W ¨urstchen: An ef- ficient architecture for large-scale text-to-image diffusion models

    Pablo Pernias, Dominic Rampas, Mats Leon Richter, Christopher Pal, and Marc Aubreville. W ¨urstchen: An ef- ficient architecture for large-scale text-to-image diffusion models. In The Twelfth International Conference on Learn- ing Representations. 2

  38. [46]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth Interna- tional Conference on Learning Representations. 2, 5

  39. [47]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  40. [48]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 2, 3

    2022

  41. [49]

    Semantic im- age inversion and editing using rectified stochastic differen- tial equations

    Litu Rout, Yujia Chen, Nataniel Ruiz, Constantine Carama- nis, Sanjay Shakkottai, and Wen-Sheng Chu. Semantic im- age inversion and editing using rectified stochastic differen- tial equations. arXiv preprint arXiv:2410.10792, 2024. 4

    arXiv

  42. [50]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2250...

    2023

  43. [51]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information...

    2022

  44. [52]

    Denois- ing diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois- ing diffusion implicit models. In International Conference on Learning Representations, 2021. 3, 4

    2021

  45. [53]

    Moma: Multimodal llm adapter for fast personalized image generation

    Kunpeng Song, Yizhe Zhu, Bingchen Liu, Qing Yan, Ahmed Elgammal, and Xiao Yang. Moma: Multimodal llm adapter for fast personalized image generation. In European Confer- ence on Computer Vision, pages 117–132. Springer, 2024. 2, 3

    2024

  46. [54]

    Generative multimodal mod- els are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiy- ing Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal mod- els are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , ...

    2024

  47. [55]

    Ominicontrol: Minimal and uni- versal control for diffusion transformer

    Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang. Ominicontrol: Minimal and uni- versal control for diffusion transformer. arXiv preprint arXiv:2411.15098, 2024. 5, 8, 2

    2024 arXiv

  48. [56]

    Key-locked rank one editing for text-to-image personaliza- tion

    Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editing for text-to-image personaliza- tion. In ACM SIGGRAPH 2023 Conference Proceedings , pages 1–11, 2023. 3

    2023

  49. [57]

    Add-it: Training-free object in- sertion in images with pretrained diffusion models

    Yoad Tewel, Rinon Gal, Dvir Samuel, Yuval Atzmon, Lior Wolf, and Gal Chechik. Add-it: Training-free object in- sertion in images with pretrained diffusion models. arXiv preprint arXiv:2411.07232, 2024. 2, 3, 4

    2024 arXiv

  50. [58]

    Training-free consis- tent text-to-image generation

    Yoad Tewel, Omri Kaduri, Rinon Gal, Yoni Kasten, Lior Wolf, Gal Chechik, and Yuval Atzmon. Training-free consis- tent text-to-image generation. ACM Transactions on Graph- ics (TOG), 43(4):1–18, 2024. 3, 1

    2024

  51. [59]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 2

    2017

  52. [60]

    Instantstyle: Free lunch towards style-preserving in text-to-image generation

    Haofan Wang, Matteo Spinelli, Qixun Wang, Xu Bai, Zekui Qin, and Anthony Chen. Instantstyle: Free lunch towards style-preserving in text-to-image generation. arXiv preprint arXiv:2404.02733, 2024. 1

    2024 arXiv

  53. [61]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Jun- yang Lin. Qwen2-vl: Enhancing vision-language model’s ...

    2024 arXiv

  54. [62]

    Wang, Siming Fu, Qihan Huang, Wanggui He, and Hao Jiang

    X. Wang, Siming Fu, Qihan Huang, Wanggui He, and Hao Jiang. Ms-diffusion: Multi-subject zero-shot image per- sonalization with layout guidance. CoRR, abs/2406.07209,

    arXiv

  55. [63]

    Elite: Encoding visual con- cepts into textual embeddings for customized text-to-image generation

    Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual con- cepts into textual embeddings for customized text-to-image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15943–15953, 2023. 3

    2023

  56. [64]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  57. [65]

    Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models. arXiv preprint arXiv:2308.06721,

    arXiv

  58. [66]

    Attention calibration for disentangled text-to-image person- alization

    Yanbing Zhang, Mengping Yang, Qin Zhou, and Zhe Wang. Attention calibration for disentangled text-to-image person- alization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4764– 4774, 2024. 2

    2024

  59. [67]

    Bilateral refer- ence for high-resolution dichotomous image segmentation

    Peng Zheng, Dehong Gao, Deng-Ping Fan, Li Liu, Jorma Laaksonen, Wanli Ouyang, and Nicu Sebe. Bilateral refer- ence for high-resolution dichotomous image segmentation. CAAI Artificial Intelligence Research , 3:9150038, 2024. 2, 4, 5

    2024

  60. [68]

    Lumina-next: Making lumina-t2x stronger and faster with next-dit

    Le Zhuo, Ruoyi Du, Han Xiao, Yangguang Li, Dongyang Liu, Rongjie Huang, Wenze Liu, Xiangyang Zhu, Fu-Yun Wang, Zhanyu Ma, et al. Lumina-next: Making lumina-t2x stronger and faster with next-dit. In The Thirty-eighth An- nual Conference on Neural Information Processing Systems,

  61. [70]

    Experiments Ablations on vital layer selection. We investigate: Does the benefit arise from simply reducing layers or specifically using vital layers? Do non-vital layers impact generation? Does attention-dropout [58] suffice? Two ablations address this: 1) sharing attention i...

  62. [71]

    Designs for captions

    sharing with random dropout in all 57 layers, dropping 5/6 to approximate 1−Nv/57 (ours-D’). Other components remain unchanged. Results (Fig. 9) show key detail loss in both settings: ours-N alters hairstyle and removes leg fea- tures, while ours-D’ shifts clothing color (purp...

  63. [72]

    For IPA (Flux.1) implemen- tation, we use the third-party code from XLabs-AI

    Compared Methods and Implementation Details IP-Adapter (IPA) [65] IPA introduces a lightweight adapter that decouples image and text features, addressing limitations in fine-grained control when merging these fea- tures in cross-attention layers. For IPA (Flux.1) implemen- tat...

  64. [73]

    Shape - Main body outline - Overall structure - Proportions and composition - Spatial organization

  65. [74]

    Color - Color palette and schemes - Saturation levels - Brightness/contrast - Color distribution patterns

  66. [75]

    Texture - Surface qualities - Detail clarity - Visual patterns - Material appearance

  67. [76]

    Prompt for Detailed Subject Caption

    Subject-Specific Features - If human/animal: facial features, expressions, poses - If object: distinctive characteristics, condition - If landscape: environmental elements , atmosphere [Description Quality Levels] Your description should aim for the highest level of detail: Le...

  68. [2024]

    2 FreeCus: Free Lunch Subject-driven Customization in Diffusion Transformers Supplementary Material

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.