Pith. sign in

REVIEW 4 major objections 4 minor 66 references

TKG-DM: Training-free Chroma Key Content Generation Diffusion Model

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By shifting the per-channel mean of the initial latent noise and blending with a Gaussian mask, TKG-DM makes vanilla Stable Diffusion generate prompt-aligned foregrounds on a specifiable, uniform chroma key background without fine-tuning.

desk verdict Clever, training-free noise-shift trick for chroma key backgrounds, but the quantitative claims outrun the evidence. read the letter →

arxiv 2411.15580 v3 pith:YYAYW2PF submitted 2024-11-23 cs.CV

Ryugo Morita , Stanislav Frolov , Brian Bernhard Moser , Takahiro Shirakawa , Ko Watanabe , Andreas Dengel , Jinjia Zhou This is my paper · ORCID
classification cs.CV
keywords text-to-imagegenerationdiffusionmodelschromakeybackgroundinitialnoisemanipulationchannelmeanshiftforeground-backgroundseparationtraining-freelatentspacecolorcontrol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a vanilla Stable Diffusion model can generate a foreground object on a clean, specifiable chroma key background without any fine-tuning, dataset curation, or prompt engineering. It does this by changing the mean of each channel of the initial latent noise (a channel mean shift) so that the background the model would otherwise paint becomes a uniform chosen color, and by combining that color-shifted noise with the ordinary noise through a Gaussian mask so the foreground keeps following the text prompt. The authors report that this training-free trick beats prompt-based baselines and matches or exceeds a fine-tuned layer-generation model on FID, mask-FID, CLIP image/text alignment, and a user study, and that it carries over to ControlNet, consistency models, and text-to-video generation. If this holds, expensive chroma key asset pipelines can be replaced by a few lines of noise manipulation.

What carries the argument

The load-bearing object is channel mean shift, a deterministic transformation of the initial noise tensor $z_T$ that adds a per-channel offset until the channel's positive pixel ratio reaches a target value (Eqs. 1-2), producing init color noise $z_T^*$. A second component, the 2D Gaussian mask $A(i,j)$ of Eq. 3, blends $z_T$ in the foreground region with $z_T^*$ in the background region to form the keyed noise fed to the unmodified diffusion model. The first component is what fixes the background hue; the second is what preserves prompt-aligned foreground content and gives layout control.

What would settle it

Run TKG-DM with the paper's green recipe (positive shifts on channels 2 and 3 at +7%) on a diverse set of prompts, measure the background's pixel variance outside the foreground mask; if any prompt set yields a background that deviates substantially from a uniform target green, or produces green spill on the foreground, the assumed prompt-independent noise-to-color map fails.

Watch

Extended reading notes

Core claim

The central claim is that the color of an image generated by Stable Diffusion is largely fixed by the statistics of the initial latent noise, in particular the positive-to-negative pixel ratio per channel. TKG-DM asserts that shifting the channel means so that this ratio changes by a small target amount (e.g., +7% on channels 2 and 3 for green) makes the background of a prompt-conditional generation collapse to that color, while the foreground remains semantic because cross-attention links the prompt to the object region. A 2D Gaussian mask blends the shifted noise into the background region and keeps the original noise in the foreground, and varying the mask center and width controls object position, size, and number. The paper argues this decoupling works because self-attention and cross-attention biases in the base model assign the prompt to the foreground and let the modified noise dominate the background, so background color becomes independent of the text prompt and of classifier-free guidance scale.

Load-bearing premise

The method assumes that the same per-channel positive-ratio shift produces the same background color regardless of the text prompt, the object being drawn, and the specific diffusion model; that assumption rests on a blog post and a handful of qualitative examples.

Editorial extensions

If this is right

  • Background color becomes a specifiable parameter: shifting channels 2 and 3 positively gives green, and other shifts give blue, yellow, red, orange, and other hues while keeping the same foreground prompt.
  • Foreground layout becomes controllable without training: moving the Gaussian center places the object elsewhere, enlarging sigma enlarges it, and multiple masks create multiple foreground objects in one image.
  • Because the base model is untouched, the recipe transfers to other latent-space generators: ControlNet conditioning, consistency models for few-step generation, and frame-wise text-to-video via AnimateDiff are all reported to keep the same chroma key behavior.
  • The method outperforms prompt-engineering baselines such as green background prompts on FID and mask-FID, and rivals the fine-tuned LayerDiffuse, while requiring no dataset licenses or extra training compute.
  • Negative prompts can be used to remove unwanted colors from the foreground without disturbing the background, because the background color lives in the initial noise rather than in the text prompt.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A likely extension is a calibration step that maps an arbitrary target RGB color to per-channel shift values from a small set of prompts, replacing the manual channel search used in the paper.
  • The reported stability of background color under changes in guidance scale suggests that other low-frequency background properties, such as gradients or simple textures, might also be imposed through initial-noise editing, provided they can be encoded in per-channel statistics.
  • Because the paper's green recipe is demonstrated on SD1.5, SDXL, and FLUX, the positive-ratio-to-color map appears model-specific; a practical deployment would need to re-derive channel roles for each new backbone rather than assuming the same shift values transfer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes TKG-DM, a training-free method for generating a foreground object described by a text prompt over a uniform, user-specifiable chroma-key background with Stable Diffusion. The method works by shifting the per-channel mean of the initial latent noise (channel mean shift) to bias the background color, and by combining the original and shifted noise through a Gaussian mask so that the foreground region retains standard sampling behavior. The authors evaluate against prompt-based baselines, a LoRA fine-tune, and LayerDiffuse, and demonstrate extensions to ControlNet, consistency models, text-to-video, and FLUX. The central claim is that this noise manipulation makes vanilla Stable Diffusion match or surpass fine-tuned models for chroma-key content generation without any training.

Significance. The core idea is simple, practical, and potentially significant: if the noise-to-color mapping is robust, TKG-DM offers a training-free, dataset-free way to control background color and foreground placement in existing text-to-image models. The paper is also commendable for openly reporting hyperparameters such as TargetShift and Gaussian mask settings, for documenting failure cases, and for demonstrating applicability beyond image generation. However, the current quantitative evidence is not sufficient to establish the two main claims: that the background color is reliably specifiable and prompt-independent, and that the method matches or surpasses fine-tuned models. The evaluation relies on an unvalidated self-defined metric, lacks error bars and significance tests, and the SDXL comparison does not show superiority over LayerDiffuse on any objective metric. The contribution is promising, but the evaluation needs substantial strengthening before the claims can be accepted.

major comments (4)
  1. [Section 4.1, Eq. (2)] The load-bearing assumption that raising the per-channel positive ratio by a fixed TargetShift produces a prompt-independent, uniform, specifiable background color is never directly measured. Figs. 3, 7, and supplementary Figs. 12-15 provide only qualitative examples, and supplementary Section 9.1 explicitly documents a prompt-dependent trade-off: low ratios leave object features in the background, while high ratios degrade foreground text alignment. Since the chosen +7% setting is used for all 3,000 prompts in Table 1, the paper should report quantitative background-color statistics (e.g., mean and standard deviation of the difference between the generated background and the target color, or the fraction of background pixels within a tolerance) across prompts and seeds, together with foreground alignment as a function of TargetShift. Without such measurements, the central claim of a specifiable chroma-key background is not established.
  2. [Section 5.4, Table 1] In the SDXL setting, the proposed method is worse than LayerDiffuse on every reported objective metric: FID 41.81 vs 29.34, m-FID 31.43 vs 29.82, CLIP-I 0.763 vs 0.778, and CLIP-S 0.273 vs 0.276. The abstract claim of matching or surpassing fine-tuned models is therefore supported only by the SD1.5 LoRA comparison and the user study in Fig. 8. The user study reports a 53.9% vs 46.1% preference over LayerDiffuse, but no confidence interval, significance test, or inter-rater agreement is provided, and the number of stimuli is small. The comparative claim needs direct statistical support and a discussion of what 'matching or surpassing' means when the main objective table points the other way.
  3. [Section 5.2, Table 1] The m-FID metric is introduced as a novel evaluation measure but is never validated. It depends on a BiRefNet mask extraction that is itself imperfect, and no comparison with human judgments or with established matting/segmentation metrics is given. Moreover, none of the FID, m-FID, CLIP-I, or CLIP-S values in Tables 1 and 2 are accompanied by error bars, multiple seeds, or statistical tests. Without these, differences such as the SDXL m-FID gap (31.43 vs 29.82) cannot be interpreted, and the claim that TKG-DM outperforms existing methods is not quantitatively supported.
  4. [Section 5.2 and Table 1] The ground-truth construction biases the comparison. The ground-truth images are created by overlaying MAGICK alpha images onto a lime-green background, and the MAGICK dataset itself was generated using DeepFloyd with a green background prompt followed by SDEdit and SDXL img2img. The paper acknowledges DeepFloyd's inherent advantage, but the same construction likely favors any method that outputs a uniform lime-green background, including the proposed method and LayerDiffuse, depending on how LayerDiffuse outputs are composited. The evaluation protocol for each baseline should be specified precisely: for example, whether LayerDiffuse's transparent-layer outputs are composited onto the same lime-green background before computing FID, and whether metrics are computed on full images or foreground-only crops. Without this information, the comparative conclusions in Table 1 are fragile.
minor comments (4)
  1. [Section 3.1, Eq. (1)-(2)] The definition of TargetShift as a percentage-point change in the positive ratio is only implicit; please state explicitly that TargetShift is measured in percentage points and describe how the iterative mean-shift search is initialized and terminated.
  2. [Supplementary Section 10.3] The citation 'FLUX [25]' points to the Flow Matching paper rather than to the FLUX model; a proper citation for FLUX is needed.
  3. [Section 5.5, Fig. 8] The sentence '30 image pairs per method, totaling 60 images' is ambiguous; please clarify whether 30 pairs were used for each comparison (60 pairs total) or 30 pairs overall, and report how many participants evaluated each pair.
  4. [Section 7] There is an ungrammatical phrase in the limitations section: 'if the size parameter is too small, ignore generating the foreground object' should be reworded for clarity.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the noise-shift mechanism is an empirical, externally benchmarked technique; hand-set hyperparameters and peripheral self-citations do not make the central claim circular.

full rationale

The paper's derivation chain is empirical rather than self-referential. The core construction (Eqs. 1-3) defines how initial noise is shifted and Gaussian-masked; it does not define the generated background color or foreground quality as an output of those equations. The color-shift relationship is explicitly borrowed from an external source ("Inspired by previous research [50], we control the chroma key background color by applying channel mean shift," Section 4.1) and is demonstrated qualitatively (Figs. 3, 7, 12-15), not derived from the method's own definitions. The choice TargetShiftc = ±7% and sigma = 0.5 is an openly reported hyperparameter, not a parameter fitted to reproduce the evaluation numbers; the evaluation itself uses the external MAGICK dataset, FID/m-FID/CLIP metrics, and a user study against fine-tuned and prompt-based baselines. The paper's self-citations ([30], [31], [32], [46]) appear in related-work, failure-case, or methodology contexts and are not load-bearing for the central claim. There is no equation in which a predicted quantity reduces to an input by construction, no uniqueness theorem imported from the authors' prior work, and no ansatz smuggled in through a self-citation chain. Concerns that the background-color mapping is under-validated, or that the SDXL FID does not surpass LayerDiffuse, are correctness/evidence concerns rather than circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on an empirical relationship between latent noise statistics and output color, plus the persistence of spatial noise blending. Both are plausible but are not derived from first principles and are validated only through qualitative examples and a small number of hand-chosen hyperparameters.

free parameters (3)
  • TargetShift = 7% (channels 2 and 3 positive for green)
    Hand-chosen to balance background uniformity and foreground alignment; the supplementary shows a trade-off as the ratio varies.
  • Gaussian mask sigma and center = sigma=0.5, mu_i=w/2, mu_j=h/2
    Chosen to control foreground size and position; these are layout parameters, not derived from data.
  • Channel selection per color = Channels 2 and 3 for green; channel 2 for blue
    Identified empirically from latent space observations [50] and qualitative experiments.
assumptions (3)
  • ad hoc to paper Per-channel positive ratio in SD initial noise correlates monotonicly with output color
    Invoked in Section 4.1; supported only by a blog post [50] and the paper's own qualitative figures.
  • domain assumption Cross-attention and self-attention biases make foreground follow text and background follow initial noise
    Assumed in Section 4.2 to explain content separation; not directly measured or tested.
  • domain assumption Spatial blending of initial noise via the Gaussian mask persists through the full denoising process
    Required for foreground/background separation; empirically displayed in the supplementary denoising visualizations but not theoretically justified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TKG-DM: Training-free Chroma Key Content Generation Diffusion Model." pith.science (2026). https://pith.science/paper/YYAYW2PF

@misc{pith2026241115580,
  author       = {Pith},
  title        = {Pith review of: TKG-DM: Training-free Chroma Key Content Generation Diffusion Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YYAYW2PF}},
  note         = {Machine review of arXiv:2411.15580}
}
read the original abstract

Diffusion models have enabled the generation of high-quality images with a strong focus on realism and textual fidelity. Yet, large-scale text-to-image models, such as Stable Diffusion, struggle to generate images where foreground objects are placed over a chroma key background, limiting their ability to separate foreground and background elements without fine-tuning. To address this limitation, we present a novel Training-Free Chroma Key Content Generation Diffusion Model (TKG-DM), which optimizes the initial random noise to produce images with foreground objects on a specifiable color background. Our proposed method is the first to explore the manipulation of the color aspects in initial noise for controlled background generation, enabling precise separation of foreground and background without fine-tuning. Extensive experiments demonstrate that our training-free method outperforms existing methods in both qualitative and quantitative evaluations, matching or surpassing fine-tuned models. Finally, we successfully extend it to other tasks (e.g., consistency models and text-to-video), highlighting its transformative potential across various generative applications where independent control of foreground and background is crucial.

Figures

Figures reproduced from arXiv: 2411.15580 by the authors.

Figure 1
Figure 1. TKG-DM provides a training-free generation of fore [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Starting with random noise zT ∼ N (0, I), init color noise z ∗ T is generated by applying channel mean shift Fc. This produces a single-colored image x ∗ 0 without a prompt. By combining normal noise with init color noise via a Gaussian mask, TKG-DM generates a chroma key image x key 0 with the specified foreground (e.g., “the cat runs in the park”) over a uniform background, effectively separating the foreground fr… view at source ↗
Figure 3
Figure 3. Fig. 3a illustrates the relationship between positive/negative channel mean shift with initial noise and color variations in the generated images. Different channel shifts across noise channels result in changes in the hue of the generated image. Fig. 3b shows how simultaneous shifts across multiple channels facilitate additive and subtractive color mixing, providing intuitive and flexible color control. black, resp… view at source ↗
Figures from the paper (29 more)
Figure 4
Figure 4. Figure 4: Visualization of self- and cross-attention maps. The self-attention maps, segmented using a foreground segmentation model [62], show the regions attended to during generation. The cross-attention maps illustrate how the model attends to relevant regions based on the fo…
Figure 5
Figure 5. Figure 5: Qualitative comparison in SD1.5. Existing methods fail to produce an accurate chroma key background. In contrast, ours produces a highly accurate chroma key background while generating high-quality foregrounds without Green Background Prompt (GBP). 5. Experiments 5.1. …
Figure 6
Figure 6. Figure 6: Qualitative comparison in SDXL. Existing training-free methods fail to isolate foreground objects from the background. While LayerDiffuse, a fine-tuned model, successfully generates the transparent foregrounds on the lime green background, it may struggle with accurate…
Figure 7
Figure 7. Figure 7: Qualitative various color result in SDXL. By adjusting the channel mean shift, TKG-DM controls the background color while generating a high-quality foreground. cluded in the text prompt ( [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Results from our user study show the preference rates for [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: Qualitative comparison in ControlNET. Existing methods typically generate images with issues such as color erosion in the foreground, uneven backgrounds, or unintended elements from the conditioning input. In contrast, our model generates images with the foreground iso…
Figure 11
Figure 11. Figure 11: Failure Cases. TKG-DM fails to generate a foreground object when the size is too small or when the text prompt lacks ob￾ject information, resulting in only the background being generated. AnimateDiff [15] on each frame while maintaining a uni￾form background for chrom…
Figure 10
Figure 10. Figure 10: Application results of TKG-DM across various tasks. TKG-DM effectively supports layout-aware text-to-image generation, consistency models and text-to-video. Each row demonstrates its versatility in adapting to different domains, from realistic object and character pla…
Figure 12
Figure 12. Figure 12: Relationship between the positive-to-negative ratio and single-channel mean shift (channel = 2) without a text prompt. As the [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: Relationship between the positive-to-negative ratio and single-channel mean shift (channel = 2) with the text prompt “red apple [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: Relationship between the positive-to-negative ratio and multi-channel mean shift (channels = 2 and 3) without a text prompt. [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]
Figure 15
Figure 15. Figure 15: Relationship between the positive-to-negative ratio and multi-channel mean shift (channels = 2 and 3) with the text prompt “red [PITH_FULL_IMAGE:figures/full_fig_p015_15.png]
Figure 16
Figure 16. Figure 16: Generated images in SDXL at various denoising steps (1 to 50). The images demonstrate the progression of generated content. [PITH_FULL_IMAGE:figures/full_fig_p016_16.png]
Figure 17
Figure 17. Figure 17: Generated images in SDXL with Green Background Prompt at various denoising steps (1 to 50). The input prompt is “An [PITH_FULL_IMAGE:figures/full_fig_p016_17.png]
Figure 18
Figure 18. Figure 18: Generated images using TKG-DM at various denoising steps (1 to 50). The input prompt is “An avocado”. Our method [PITH_FULL_IMAGE:figures/full_fig_p016_18.png]
Figure 19
Figure 19. Figure 19: Denoising process progression in SDXL at various steps. The images demonstrate the evolution of generated content from [PITH_FULL_IMAGE:figures/full_fig_p017_19.png]
Figure 20
Figure 20. Figure 20: Denoising process progression in SDXL with Green Background Prompt at various steps. The images demonstrate the evolution [PITH_FULL_IMAGE:figures/full_fig_p017_20.png]
Figure 21
Figure 21. Figure 21: Denoising process progression in our TKG-DM at various steps. The images demonstrate the evolution of generated content [PITH_FULL_IMAGE:figures/full_fig_p017_21.png]
Figure 22
Figure 22. Figure 22: Effect of varying guidance scales in SDXL. The guidance scale ranges from 1.0 to 10.0. The input prompt is “The cat”. Higher [PITH_FULL_IMAGE:figures/full_fig_p018_22.png]
Figure 23
Figure 23. Figure 23: Effect of varying guidance scales in SDXL with Green Background Prompt. The input prompt is “The cat”. Increasing the [PITH_FULL_IMAGE:figures/full_fig_p018_23.png]
Figure 24
Figure 24. Figure 24: Effect of varying guidance scales in TKG-DM. The input prompt is “The cat”. Our method maintains consistent background [PITH_FULL_IMAGE:figures/full_fig_p018_24.png]
Figure 25
Figure 25. Figure 25: Comparison of results using negative prompts to modify colors and control size in generated images. For color adjustments, [PITH_FULL_IMAGE:figures/full_fig_p019_25.png]
Figure 26
Figure 26. Figure 26: Additional Result in Green Background with SD1.5 [PITH_FULL_IMAGE:figures/full_fig_p020_26.png]
Figure 27
Figure 27. Figure 27: Additional Result in Green Background with SDXL [PITH_FULL_IMAGE:figures/full_fig_p021_27.png]
Figure 28
Figure 28. Figure 28: Additional Result in various color Background with SDXL [PITH_FULL_IMAGE:figures/full_fig_p022_28.png]
Figure 29
Figure 29. Figure 29: TKG-DM applied to the FLUX flow-based model. By adjusting the mean shift in specific channels, TKG-DM generates green and blue backgrounds without fine-tuning the model itself. The differences in each channel’s color representation from SD1.5 and SDXL highlight TKG-DM…
Figure 30
Figure 30. Figure 30: Additional Result in ControlNet with SD1.5 [PITH_FULL_IMAGE:figures/full_fig_p023_30.png]
Figure 31
Figure 31. Figure 31: Additional Result in ControlNet with SDXL [PITH_FULL_IMAGE:figures/full_fig_p024_31.png]
Figure 32
Figure 32. Figure 32: Additional Result of Application track 14 [PITH_FULL_IMAGE:figures/full_fig_p025_32.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 35 canonical work pages

  1. [1]

    The crystal ball hypoth- esis in diffusion models: Anticipating object positions from initial noise.arXiv preprint arXiv:2406.01970, 2024

    Yuanhao Ban, Ruochen Wang, Tianyi Zhou, Boqing Gong, Cho-Jui Hsieh, and Minhao Cheng. The crystal ball hypoth- esis in diffusion models: Anticipating object positions from initial noise.arXiv preprint arXiv:2406.01970, 2024. 2

    arXiv 2024

  2. [2]

    Multidiffusion: Fusing diffusion paths for controlled image generation, 2023

    Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation, 2023. 2

    work page 2023

  3. [3]

    Onegan: Simultaneous unsuper- vised learning of conditional image generation, foreground segmentation, and fine-grained clustering

    Yaniv Benny and Lior Wolf. Onegan: Simultaneous unsuper- vised learning of conditional image generation, foreground segmentation, and fine-grained clustering. InECCV, pages 514–530. Springer, 2020. 2

    work page 2020

  4. [4]

    In- structpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. In- structpix2pix: Learning to follow image editing instructions. InCVPR, pages 18392–18402, 2023. 2

    work page 2023

  5. [5]

    Magick: A large-scale captioned dataset from matting generated images using chroma keying

    Ryan D Burgert, Brian L Price, Jason Kuen, Yijun Li, and Michael S Ryoo. Magick: A large-scale captioned dataset from matting generated images using chroma keying. In CVPR, pages 22595–22604, 2024. 1, 2, 5

    work page 2024

  6. [6]

    Masactrl: Tuning-free mu- tual self-attention control for consistent image synthesis and editing

    Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xi- aohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mu- tual self-attention control for consistent image synthesis and editing. InICCV, pages 22560–22570, 2023. 2

    2023

  7. [7]

    Lay- outdm: Transformer-based diffusion model for layout gener- ation

    Shang Chai, Liansheng Zhuang, and Fengying Yan. Lay- outdm: Transformer-based diffusion model for layout gener- ation. InCVPR, pages 18349–18358, 2023. 2

    work page 2023

  8. [8]

    Attend-and-excite: Attention-based se- mantic guidance for text-to-image diffusion models.ACM Transactions on Graphics (TOG), 42(4):1–10, 2023

    Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based se- mantic guidance for text-to-image diffusion models.ACM Transactions on Graphics (TOG), 42(4):1–10, 2023. 1, 2

    work page 2023

Show all 66 references
  1. [9]

    Anyscene: Customized image synthesis with composited foreground

    Ruidong Chen, Lanjun Wang, Weizhi Nie, Yongdong Zhang, and An-An Liu. Anyscene: Customized image synthesis with composited foreground. InCVPR, pages 8724–8733,

  2. [10]

    Improving diffusion models for vir- tual try-on.arXiv preprint arXiv:2403.05139, 2024

    Yisol Choi, Sangkyung Kwak, Kyungmin Lee, Hyungwon Choi, and Jinwoo Shin. Improving diffusion models for vir- tual try-on.arXiv preprint arXiv:2403.05139, 2024. 1

    2024 arXiv

  3. [11]

    Layerfusion: Harmo- nized multi-layer text-to-image generation with generative priors.arXiv preprint arXiv:2412.04460, 2024

    Yusuf Dalva, Yijun Li, Qing Liu, Nanxuan Zhao, Jianming Zhang, Zhe Lin, and Pinar Yanardag. Layerfusion: Harmo- nized multi-layer text-to-image generation with generative priors.arXiv preprint arXiv:2412.04460, 2024. 2

    2024 arXiv

  4. [12]

    Diffusion mod- els beat gans on image synthesis.NeurIPS, 34:8780–8794,

    Prafulla Dhariwal and Alexander Nichol. Diffusion mod- els beat gans on image synthesis.NeurIPS, 34:8780–8794,

  5. [13]

    Reno: Enhancing one-step text-to-image models through reward-based noise optimiza- tion.arXiv preprint arXiv:2406.04312, 2024

    Luca Eyring, Shyamgopal Karthik, Karsten Roth, Alexey Dosovitskiy, and Zeynep Akata. Reno: Enhancing one-step text-to-image models through reward-based noise optimiza- tion.arXiv preprint arXiv:2406.04312, 2024. 2

    2024 arXiv

  6. [14]

    Initno: Boosting text-to-image diffu- sion models via initial noise optimization

    Xiefan Guo, Jinlin Liu, Miaomiao Cui, Jiankai Li, Hongyu Yang, and Di Huang. Initno: Boosting text-to-image diffu- sion models via initial noise optimization. InCVPR, pages 9380–9389, 2024. 2

    2024

  7. [15]

    Animatediff: Animate your personalized text- to-image diffusion models without specific tuning.arXiv preprint arXiv:2307.04725, 2023

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text- to-image diffusion models without specific tuning.arXiv preprint arXiv:2307.04725, 2023. 2, 8

    2023 arXiv

  8. [16]

    Prompt-to-prompt im- age editing with cross attention control.arXiv preprint arXiv:2208.01626, 2022

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt im- age editing with cross attention control.arXiv preprint arXiv:2208.01626, 2022. 2

    2022 arXiv

  9. [17]

    Clipscore: A reference-free evaluation met- ric for image captioning.arXiv preprint arXiv:2104.08718,

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation met- ric for image captioning.arXiv preprint arXiv:2104.08718,

    arXiv

  10. [18]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium.NeurIPS, 30, 2017

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium.NeurIPS, 30, 2017. 5

    2017

  11. [19]

    Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022. 2

    2022 arXiv

  12. [20]

    Denoising dif- fusion probabilistic models.NeurIPS, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models.NeurIPS, 33:6840–6851, 2020. 2

    2020

  13. [21]

    Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models

    Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. InICML, pages 13916–13932. PMLR, 2023. 2

    2023

  14. [22]

    Layerdiff: Exploring text-guided multi-layered composable image synthesis via layer-collaborative diffu- sion model

    Runhui Huang, Kaixin Cai, Jianhua Han, Xiaodan Liang, Renjing Pei, Guansong Lu, Songcen Xu, Wei Zhang, and Hang Xu. Layerdiff: Exploring text-guided multi-layered composable image synthesis via layer-collaborative diffu- sion model. InEuropean Conference on Computer Vision, pa...

    2024

  15. [23]

    Layoutdm: Discrete diffusion model for controllable layout generation

    Naoto Inoue, Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi. Layoutdm: Discrete diffusion model for controllable layout generation. InCVPR, pages 10167–10176, 2023. 2

    2023

  16. [24]

    Diffwave: A versatile diffusion model for audio synthesis.arXiv preprint arXiv:2009.09761, 2020

    Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis.arXiv preprint arXiv:2009.09761, 2020. 2

    2009 arXiv

  17. [25]

    Flow matching for generative mod- eling.arXiv preprint arXiv:2210.02747, 2022

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximil- ian Nickel, and Matt Le. Flow matching for generative mod- eling.arXiv preprint arXiv:2210.02747, 2022. 2

    2022 arXiv

  18. [26]

    Repaint: Inpainting using denoising diffusion probabilistic models

    Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. InCVPR, pages 11461–11471, 2022. 2

    2022

  19. [27]

    Guided image synthesis via initial image editing in diffusion model

    Jiafeng Mao, Xueting Wang, and Kiyoharu Aizawa. Guided image synthesis via initial image editing in diffusion model. arXiv preprint arXiv:2305.03382, 2023. 2

    2023 arXiv

  20. [28]

    Semantic-driven initial image construction for guided image synthesis in diffusion model.arXiv preprint arXiv:2312.08872, 2023

    Jiafeng Mao, Xueting Wang, and Kiyoharu Aizawa. Semantic-driven initial image construction for guided image synthesis in diffusion model.arXiv preprint arXiv:2312.08872, 2023. 2, 8

    2023 arXiv

  21. [29]

    SDEdit: Guided image synthesis and editing with stochastic differential equa- tions

    Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jia- jun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equa- tions. InICLR, 2022. 5

    2022

  22. [30]

    Interactive image manipulation with complex text instruc- tions

    Ryugo Morita, Zhiqiang Zhang, Man M Ho, and Jinjia Zhou. Interactive image manipulation with complex text instruc- tions. InWACV, pages 1053–1062, 2023. 2 9

    2023

  23. [31]

    Batinet: Background-aware text to image synthesis and manipulation network

    Ryugo Morita, Zhiqiang Zhang, and Jinjia Zhou. Batinet: Background-aware text to image synthesis and manipulation network. InICIP, pages 765–769. IEEE, 2023. 2

    2023

  24. [32]

    Yoda: You only diffuse ar- eas

    Brian B Moser, Stanislav Frolov, Federico Raue, Sebastian Palacio, and Andreas Dengel. Yoda: You only diffuse ar- eas. an area-masked diffusion approach for image super- resolution.arXiv preprint arXiv:2308.07977, 2023. 7

    2023 arXiv

  25. [33]

    Diffusion models, image super-resolution, and everything: A survey.IEEE Transactions on Neural Networks and Learn- ing Systems, 2024

    Brian B Moser, Arundhati S Shanbhag, Federico Raue, Stanislav Frolov, Sebastian Palacio, and Andreas Dengel. Diffusion models, image super-resolution, and everything: A survey.IEEE Transactions on Neural Networks and Learn- ing Systems, 2024. 1

    2024

  26. [34]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models.arXiv preprint arXiv:2112.10741, 2021

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models.arXiv preprint arXiv:2112.10741, 2021. 2

    2021 arXiv

  27. [35]

    Improved denoising diffusion probabilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. InICML, pages 8162–8171. PMLR, 2021. 2

    2021

  28. [36]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023. 5, 6, 7

    2023 arXiv

  29. [37]

    Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors.arXiv preprint arXiv:2306.17843,

    Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin-Ying Lee, Ivan Sko- rokhodov, Peter Wonka, Sergey Tulyakov, et al. Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors.arXiv preprint arXiv:2306.17843,

    arXiv

  30. [38]

    Alfie: Democratising rgba image generation with no $$$

    Fabio Quattrini, Vittorio Pippi, Silvia Cascianelli, and Rita Cucchiara. Alfie: Democratising rgba image generation with no $$$. InEuropean Conference on Computer Vision, pages 38–55. Springer, 2025. 2

    2025

  31. [39]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. InICML, pages 8821–

  32. [40]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. InCVPR, pages 10684– 10695, 2022. 1, 2, 5, 6

    2022

  33. [41]

    Photorealistic text-to-image diffusion models with deep language understanding.NeurIPS, 35:36479–36494, 2022

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding.NeurIPS, 35:36479–36494, 2022. ...

    2022

  34. [42]

    Image super- resolution via iterative refinement.IEEE TPAMI, 45(4): 4713–4726, 2022

    Chitwan Saharia, Jonathan Ho, William Chan, Tim Sali- mans, David J Fleet, and Mohammad Norouzi. Image super- resolution via iterative refinement.IEEE TPAMI, 45(4): 4713–4726, 2022. 7

    2022

  35. [43]

    Norm-guided latent space exploration for text-to-image generation.NeurIPS, 36, 2024

    Dvir Samuel, Rami Ben-Ari, Nir Darshan, Haggai Maron, and Gal Chechik. Norm-guided latent space exploration for text-to-image generation.NeurIPS, 36, 2024. 2

    2024

  36. [44]

    Generating images of rare concepts using pre- trained diffusion models

    Dvir Samuel, Rami Ben-Ari, Simon Raviv, Nir Darshan, and Gal Chechik. Generating images of rare concepts using pre- trained diffusion models. InAAAI, pages 4695–4703, 2024. 2

    2024

  37. [45]

    Laion-5b: An open large-scale dataset for training next generation image-text models.Advances in Neural In- formation Processing Systems, 35:25278–25294, 2022

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models.Advances in Neural In- fo...

    2022

  38. [46]

    Noisecollage: A layout-aware text-to-image diffusion model based on noise cropping and merging

    Takahiro Shirakawa and Seiichi Uchida. Noisecollage: A layout-aware text-to-image diffusion model based on noise cropping and merging. InCVPR, pages 8921–8930, 2024. 2, 8

    2024

  39. [47]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. InICML, pages 2256–

  40. [48]

    Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020. 2

    2010 arXiv

  41. [49]

    Consistency models.arXiv preprint arXiv:2303.01469, 2023

    Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models.arXiv preprint arXiv:2303.01469, 2023. 8

    2023 arXiv

  42. [50]

    Explaining the sdxl latent space.https : / / huggingface

    Timothy Alexis Vass. Explaining the sdxl latent space.https : / / huggingface . co / blog / TimothyAlexisVass / explaining - the - sdxl - latent-space, 2024. 3

    2024

  43. [51]

    Sv3d: Novel multi-view syn- thesis and 3d generation from a single image using latent video diffusion.arXiv preprint arXiv:2403.12008, 2024

    Vikram V oleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view syn- thesis and 3d generation from a single image using latent video diffusion.arXiv preprint arXiv:2403.12008, 2024. 2

    2024 arXiv

  44. [52]

    Sketch-guided text-to-image diffusion models

    Andrey V oynov, Kfir Aberman, and Daniel Cohen-Or. Sketch-guided text-to-image diffusion models. InACM SIG- GRAPH 2023 Conference Proceedings, pages 1–11, 2023. 1

    2023

  45. [53]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In ICCV, pages 7623–7633, 2023. 2

    2023

  46. [54]

    Good seed makes a good crop: Discovering secret seeds in text-to- image diffusion models.arXiv preprint arXiv:2405.14828,

    Katherine Xu, Lingzhi Zhang, and Jianbo Shi. Good seed makes a good crop: Discovering secret seeds in text-to- image diffusion models.arXiv preprint arXiv:2405.14828,

    arXiv

  47. [55]

    Oot- diffusion: Outfitting fusion based latent diffusion for control- lable virtual try-on.arXiv preprint arXiv:2403.01779, 2024

    Yuhao Xu, Tao Gu, Weifeng Chen, and Chengcai Chen. Oot- diffusion: Outfitting fusion based latent diffusion for control- lable virtual try-on.arXiv preprint arXiv:2403.01779, 2024. 1

    2024 arXiv

  48. [56]

    Paint by example: Exemplar-based image editing with diffusion mod- els

    Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing with diffusion mod- els. InCVPR, pages 18381–18391, 2023. 2

    2023

  49. [57]

    Reco: Region-controlled text-to-image genera- tion

    Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, et al. Reco: Region-controlled text-to-image genera- tion. InCVPR, pages 14246–14255, 2023. 1 10

    2023

  50. [58]

    Transparent image layer diffusion using latent transparency.arXiv preprint arXiv:2402.17113, 2024

    Lvmin Zhang and Maneesh Agrawala. Transparent image layer diffusion using latent transparency.arXiv preprint arXiv:2402.17113, 2024. 1, 2, 5, 6

    2024 arXiv

  51. [59]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, pages 3836–3847, 2023. 1, 7

    2023

  52. [60]

    Text2layer: Layered image generation using latent diffusion model.arXiv preprint arXiv:2307.09781, 2023

    Xinyang Zhang, Wentian Zhao, Xin Lu, and Jeff Chien. Text2layer: Layered image generation using latent diffusion model.arXiv preprint arXiv:2307.09781, 2023. 2

    2023 arXiv

  53. [61]

    Layoutdiffusion: Controllable diffu- sion model for layout-to-image generation

    Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li. Layoutdiffusion: Controllable diffu- sion model for layout-to-image generation. InCVPR, pages 22490–22499, 2023. 1

    2023

  54. [62]

    Bilateral refer- ence for high-resolution dichotomous image segmentation

    Peng Zheng, Dehong Gao, Deng-Ping Fan, Li Liu, Jorma Laaksonen, Wanli Ouyang, and Nicu Sebe. Bilateral refer- ence for high-resolution dichotomous image segmentation. arXiv preprint arXiv:2401.03407, 2024. 4, 5, 7

    2024 arXiv

  55. [63]

    Zero- shot subject-centric generation for creative application using entropy fusion.arXiv preprint arXiv:2503.10697, 2025

    Kaifeng Zou, Xiaoyi Feng, Peng Wang, Tao Huang, Zizhou Huang, Zhang Haihang, Yuntao Zou, and Dagang Li. Zero- shot subject-centric generation for creative application using entropy fusion.arXiv preprint arXiv:2503.10697, 2025. 2 11 TKG-DM: Training-free Chroma Key Content Gene...

    2025 arXiv

  56. [64]

    Additional Analysis To further illustrate the capabilities of TKG-DM in gener- ating chroma key content, we present comprehensive visu- alizations and analyses. These include examinations of the positive-to-negative ratio, detailed observations of the de- noising process, an e...

  57. [65]

    Additional Results: TKG-DM with Green Chroma Key Backgrounds Fig

    Additional Results 10.1. Additional Results: TKG-DM with Green Chroma Key Backgrounds Fig. 26 and Fig. 27 provide additional examples based on SD1.5 and SDXL, demonstrating TKG-DM’s effectiveness in generating high-quality, chroma-keyed foreground im- ages against a green back...

  58. [66]

    red apple and glass of juice

    More Results of TKG-DM with application track Beyond text-to-image tasks, TKG-DM enables chroma key content generation across various applications. The main paper highlights applications involving ControlNet, layout- aware text-to-image, consistency models, and text-to-video m...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.