Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Consistent Human Image and Video Generation with Spatially Conditioned Diffusion

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that consistent human image and video synthesis can be reframed as spatially conditioned inpainting, where a single denoising network extracts reference appearance features and generates pose-compliant targets, and that…

desk verdict Useful architectural simplification in human animation, but the empirical case is too thin (10 videos, no error bars) to support the SOTA claim as written. read the letter →

arxiv 2412.14531 v1 pith:ZVA5OFX5 submitted 2024-12-19 cs.CV

Mingdeng Cao , Chong Mou , Ziyang Yuan , Xintao Wang , Zhaoyang Zhang , Ying Shan , Yinqiang Zheng This is my paper · ORCID
classification cs.CV
keywords spatially-conditioneddiffusionhumanimageanimationpose-guidedgenerationreference-basedappearanceconsistencymodelsvisualtry-onvideo
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the standard recipe for human pose transfer and animation, which uses a separate Reference-Net to copy appearance from a reference image, is unnecessary. Instead, the task can be framed as spatial inpainting: concatenate the reference image latents with the noisy target latents and let the same denoising U-Net both extract reference appearance and generate the target. The authors argue this removes the domain gap between reference features and target features because both live in the same feature manifold. If true, a single fine-tuned diffusion backbone can handle human animation, visual try-on, and face reenactment without per-instance fine-tuning, while matching or beating dedicated reference-branch methods on reconstruction and video fidelity metrics.

What carries the argument

The central mechanism is spatial conditioning with causal self-attention. Reference latents and noisy target latents are concatenated along the spatial axis and fed into the same denoising U-Net, so both feature sets occupy the same manifold. In each self-attention layer, the target's query attends to key/value pairs formed by concatenating reference and target features ($K = [K^r, K]$, $V = [V^r, V]$), while the reference tokens are causally masked to attend only to themselves. The practical implementation splits this into two passes that share one network: first the clean reference image is passed through the denoising network at $t=0$ to extract reference features, then the noisy target is denoised while conditioning on those features. A lightweight pose encoder injects both reference and target poses so that the correspondence between reference and target tokens is spatially aligned.

What would settle it

Run the reference feature extraction with $t$ set to a small positive value instead of 0 while keeping the target pass identical; if SSIM and LPIPS do not drop, the $t=0$ convention is not load-bearing. Also evaluate SCD-V on a benchmark of extreme zoom and complex-background videos, since the paper itself reports that these conditions break appearance and identity, so a large performance drop there would bound the generalization claim.

Watch

Extended reading notes

Core claim

The central discovery is that reference-conditioned human generation is better treated as a spatially conditioned inpainting problem than as a two-network feature-transfer problem. On the TikTok benchmark, the video model SCD-V reaches SSIM 0.731, PSNR 18.81, LPIPS 0.236, and FVD 136.60, surpassing the reported scores of AnimateAnyone, MagicAnimate, MagicPose, and DisCo on reconstruction and fidelity metrics. The paper attributes this to a causal feature interaction: reference features query only themselves while target features query both reference and target features, and the whole process is implemented as two stages sharing one denoising network, with a trainable pose encoder injecting pose information into the same feature space.

Load-bearing premise

The method relies on a denoising network trained on noisy inputs being able to extract useful appearance features from a clean reference image at $t=0$, a setting the training loss never directly optimizes.

Editorial extensions

If this is right

  • If the central claim holds, separate Reference-Net branches become unnecessary; one fine-tuned diffusion backbone can handle both appearance extraction and conditioned generation.
  • The spatial-conditioning recipe transfers to visual try-on on VITON-HD and to animating real and cartoon characters, so a single framework covers multiple appearance-preservation tasks.
  • Because the method requires no per-instance fine-tuning, animating a new person from a single reference image becomes a single forward pass through the shared network, lowering the cost of deployment.
  • Swapping in stronger base diffusion models, such as video or newer image backbones, is a direct path to improved fidelity since the approach is formulated at the level of the denoising network rather than as a task-specific head.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the paper does not analyze the $t=0$ reference pass, which feeds a clean image to a network trained on noisy inputs; if this distribution shift matters, one could train with reference noise and anneal the reference timestep during sampling.
  • Editorial inference: causal masking could be extended from spatial tokens to temporal tokens, so earlier video frames condition later ones, potentially strengthening temporal consistency beyond frame-wise reference conditioning.
  • Editorial inference: the reported advantage over Reference-Net methods is strongest on reconstruction metrics; testing on large pose gaps and complex backgrounds, which the paper itself lists as failure cases, would clarify whether the unified-manifold claim holds where it matters most.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes spatially-conditioned diffusion (SCD) for reference-based human image and video generation. Instead of using a separate Reference-Net, SCD passes the clean reference image through the same denoising U-Net as the noisy target, with spatial concatenation and a causal self-attention mask that lets target features attend to reference features while preventing reference features from attending to target features. The method is split into reference feature extraction and conditioned target generation stages, and is evaluated on the TikTok benchmark against GAN-based and diffusion-based baselines, with additional experiments on VITON-HD for virtual try-on. The headline claim is that the video model SCD-V surpasses AnimateAnyone, MagicAnimate, MagicPose, and DisCo on reconstruction and several fidelity metrics.

Significance. If the empirical claims hold, the paper makes a useful conceptual simplification: reference appearance features and target generation features live in the same feature space because they are produced by the same denoising network, avoiding the domain gap of Reference-Net approaches. The causal feature interaction is simple and the ablation comparing Reference-Net with and without Stable Diffusion initialization is informative. However, the paper does not provide formal proofs or parameter-free derivations, so the contribution rests entirely on the experimental evidence. That evidence is currently thin: the evaluation uses only 10 test videos, no uncertainty estimates are reported, and the paper's own FID numbers contradict part of the superiority claim. These issues are fixable and the central idea is plausible, but the paper is not yet ready in its present form.

major comments (4)
  1. [Section 3.1, Eq. (1)] The training objective in Eq. (1) writes the reference feature extraction as epsilon_theta(xr_t, t, ∅), but the immediately following note states that 'the t for the reference feature extraction is set to 0 by default.' This is internally inconsistent: the denoising network is trained to predict noise from noisy latents at t>0, and its behavior at exactly t=0 is neither trained nor evaluated. If the reference is always passed as a clean image, the paper must state this explicitly and verify the assumption, for example by comparing against a reference pass at the same noise level as the target or against a small frozen encoder. As written, the central mechanism of the method rests on an untested assumption about the network's behavior at t=0.
  2. [Section 4.2, Table 1] The quantitative support for the headline superiority claim is not established. All comparisons are reported on 10 TikTok-style videos, with no per-video standard deviations, confidence intervals, or significance tests. Several reported gaps are small (SSIM 0.731 vs 0.718 for AnimateAnyone; LPIPS 0.236 vs 0.239 for MagicAnimate), so the ranking could plausibly change with a different split or with repeated evaluation. In addition, Table 1 shows that SCD-V has FID 34.44, which is worse than DisCo (28.31), MagicPose (26.67), and MagicAnimate (32.09); the text's claim of superiority on 'fidelity metrics (LPIPS, FID-VID, FVD)' omits this contrary metric. The paper should report variance and make the superiority claim cover the full metric set.
  3. [Section 4.1, Datasets] The claim of 'strong generalization to unseen human identities and poses' is not supported by the experimental design. The training set includes the TikTok dataset and about 3,500 self-collected videos, while the evaluation uses 10 TikTok-style videos. The paper does not state that these 10 test videos are disjoint from the training videos, nor does it demonstrate that the test identities and poses are representative of unseen conditions. With n=10 and no leakage check, the generalization claim is too strong; the authors should clarify the train/test split and, if possible, evaluate on a larger held-out set.
  4. [Section 4.3, Table 2] The ablation for causal feature interaction is mixed, but the text describes it as an improvement. Comparing SCD-I† (without causal interaction) to SCD-I, PSNR improves from 18.59 to 18.64 and LPIPS from 0.242 to 0.240, but FID worsens from 33.13 to 33.63, SSIM worsens from 0.728 to 0.726, and FID-VID worsens from 32.43 to 33.15. The sentence 'we observe additional improvements in reconstruction quality metrics' is therefore only partially supported. The authors should report the full pattern of changes and avoid selective emphasis on the metrics that improve.
minor comments (5)
  1. [Section 4.3] The paragraph beginning 'While our practical implementation of causal feature interaction is similar to that of the Reference-Net' appears twice, nearly verbatim; one copy should be removed.
  2. [Table 1] There are typos in the table header ('Image Mtrics' should be 'Image Metrics') and in the caption ('following Disco' should be 'following DisCo').
  3. [Section 3.1, Eq. (1)] In Eq. (1), the symbol ∅ is used without explanation, and the notation xr_t is confusing when t is later said to be 0; if the reference is clean, the equation should use xr_0 consistently.
  4. [Section 3.1] The training loss for the spatially conditioned model is only referenced as 'Eq. 4 in the Appendix' rather than stated in the main text; since the appendix equation is the standard diffusion loss, the paper should define the actual inpainting loss in the main text to make the method self-contained.
  5. [Figure 5] The qualitative comparison would be more informative if it included zoomed crops of faces, hands, and garment details, since these are the regions where the paper claims the largest differences from baselines.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central method is tested against external benchmarks and ablations, and self-citations are background only.

full rationale

No load-bearing circular step is present. The core mechanism is a conditional diffusion model fine-tuned on human video data and evaluated on the TikTok benchmark using official baseline implementations and the DisCo evaluation script, so the headline comparisons are against independent external systems. The claimed benefits of spatial conditioning and causal feature interaction are explicitly supported by ablations against CLIP embedding, channel concatenation, and Reference-Net (Table 2), meaning the contribution does not reduce to its inputs by construction. The self-citations to Cao et al. (2023) and Mou et al. (2024) are background references for attention-based editing and pose encoders; neither is used as the sole justification of the paper's central claim. The note that the reference-feature extraction time step is set to 0 by default is a potential train/inference consistency risk, but it is not a circular reduction: the reference features are an input to the training loss, not a fitted parameter later renamed as a prediction. No uniqueness theorem is imported from prior work by the same authors, and no known empirical result is merely renamed. The fragility of the benchmark ranking due to the 10-video test set and missing error bars is a statistical correctness concern, not circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the behavior of pretrained diffusion backbones and an untested t=0 reference pass; no free scalars are fitted to external data beyond learned weights, and no new entities are introduced.

assumptions (4)
  • domain assumption Pretrained Stable Diffusion and AnimateDiff retain their ability to synthesize coherent humans and transfer appearance after fine-tuning with the spatial conditioning loss.
    The method is built on fine-tuning SD1.5 and AnimateDiff as stated in Section 4.1; if the base model geometry did not support spatial concatenation conditioning, the method would fail.
  • domain assumption Self-attention layers, not convolutions, are the primary channel for appearance transfer between reference and target, justifying restricting causal interaction to self-attention.
    Section 3.1, in the discussion of how convolutions and attention contribute, describes an ablation showing attention alone preserves similarity, which is an empirical premise for the design.
  • ad hoc to paper Reference features extracted at t=0 by the denoising network are informative despite the network being trained for denoising at t>0.
    Section 3.1, Practical Implementation and Eq. 1, set t=0 for the reference pass without a corresponding training loss at t=0, so this is an unverified assumption about the behavior of the network.
  • domain assumption Reference pose injection aligns reference and target in a common feature space, improving pose control and appearance consistency.
    Section 3.1, Reference Pose Information Injection, asserts this without an independent mechanism; the ablation shows performance drops without it, but the reason is inferred.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Consistent Human Image and Video Generation with Spatially Conditioned Diffusion." pith.science (2026). https://pith.science/paper/ZVA5OFX5

@misc{pith2026241214531,
  author       = {Pith},
  title        = {Pith review of: Consistent Human Image and Video Generation with Spatially Conditioned Diffusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZVA5OFX5}},
  note         = {Machine review of arXiv:2412.14531}
}
read the original abstract

Consistent human-centric image and video synthesis aims to generate images or videos with new poses while preserving appearance consistency with a given reference image, which is crucial for low-cost visual content creation. Recent advances based on diffusion models typically rely on separate networks for reference appearance feature extraction and target visual generation, leading to inconsistent domain gaps between references and targets. In this paper, we frame the task as a spatially-conditioned inpainting problem, where the target image is inpainted to maintain appearance consistency with the reference. This approach enables the reference features to guide the generation of pose-compliant targets within a unified denoising network, thereby mitigating domain gaps. Additionally, to better maintain the reference appearance information, we impose a causal feature interaction framework, in which reference features can only query from themselves, while target features can query appearance information from both the reference and the target. To further enhance computational efficiency and flexibility, in practical implementation, we decompose the spatially-conditioned generation process into two stages: reference appearance extraction and conditioned target generation. Both stages share a single denoising network, with interactions restricted to self-attention layers. This proposed method ensures flexible control over the appearance of generated human images and videos. By fine-tuning existing base diffusion models on human video data, our method demonstrates strong generalization to unseen human identities and poses without requiring additional per-instance fine-tuning. Experimental results validate the effectiveness of our approach, showing competitive performance compared to existing methods for consistent human image and video synthesis.

Figures

Figures reproduced from arXiv: 2412.14531 by the authors.

Figure 1
Figure 1. We propose spatially-conditioned diffusion (SCD) for consistent human image and video [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the spatially-conditioned diffusion model. Our framework achieves consistent [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Examples of content consistency through spatial conditioning. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Separating the causal spatial condition￾ing process into the reference feature extraction and target image generation. Practical Implementation. To implement a model with the causal spatial conditioning strategy, we ef￾fectively divide the spatially-conditioned generat…
Figure 5
Figure 5. Figure 5: Qualitative comparison results against state-of-the-art human animation methods on the [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Ablation of different reference image conditioning methods and the key components of [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Qualitative comparison on the VITON-HD dataset (paired setting). [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Qualitative comparison on the VITON-HD dataset (unpaired setting). [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Failure cases under complex background and large poses. [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Results of sequentially performing try-on and animation. [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Animation results. Our method demonstrates highly competitive performance in animat [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AniCrafter: Customizing Realistic Human-Centric Animation via Avatar-Background Conditioning in Video Diffusion Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A diffusion model animates a character into arbitrary dynamic backgrounds by conditioning on a rendered 3D-avatar video, reframing open-domain animation as a restoration problem.

Reference graph

Works this paper leans on

81 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Conditional gan with discriminative filter generation for text-to-video synthesis

    Yogesh Balaji, Martin Renqiang Min, Bing Bai, Rama Chellappa, and Hans Peter Graf. Conditional gan with discriminative filter generation for text-to-video synthesis. In IJCAI, volume 1, pp.\ 2, 2019

    work page 2019

  3. [3]

    Person image synthesis via denoising diffusion model

    Ankan Kumar Bhunia, Salman Khan, Hisham Cholakkal, Rao Muhammad Anwer, Jorma Laaksonen, Mubarak Shah, and Fahad Shahbaz Khan. Person image synthesis via denoising diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 5968--5976, 2023

    work page 2023

  4. [4]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023 a

    arXiv 2023

  5. [5]

    Align your latents: High-resolution video synthesis with latent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 22563--22575, 2023 b

    2023

  6. [6]

    Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing

    Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 22560--22570, 2023

    work page 2023

  7. [7]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv\'e J\'egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021

    2021

  8. [8]

    Pix2video: Video editing using image diffusion

    Duygu Ceylan, Chun-Hao P Huang, and Niloy J Mitra. Pix2video: Video editing using image diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 23206--23217, 2023

    work page 2023

Show all 81 references
  1. [9]

    Everybody dance now

    Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros. Everybody dance now. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 5933--5942, 2019

    2019

  2. [10]

    Magicdance: Realistic human dance video generation with motions & facial expressions transfer

    Di Chang, Yichun Shi, Quankai Gao, Jessica Fu, Hongyi Xu, Guoxian Song, Qing Yan, Xiao Yang, and Mohammad Soleymani. Magicdance: Realistic human dance video generation with motions & facial expressions transfer. arXiv preprint arXiv:2311.12052, 2023

    2023 arXiv

  3. [11]

    Anydoor: Zero-shot object-level image customization

    Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level image customization. arXiv preprint arXiv:2307.09481, 2023

    2023 arXiv

  4. [12]

    Viton-hd: High-resolution virtual try-on via misalignment-aware normalization

    Seunghwan Choi, Sunghyun Park, Minsoo Lee, and Jaegul Choo. Viton-hd: High-resolution virtual try-on via misalignment-aware normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 14131--14140, 2021

    2021

  5. [13]

    Improving diffusion models for virtual try-on

    Yisol Choi, Sangkyung Kwak, Kyungmin Lee, Hyungwon Choi, and Jinwoo Shin. Improving diffusion models for virtual try-on. arXiv preprint arXiv:2403.05139, 2024

    2024 arXiv

  6. [14]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34: 0 8780--8794, 2021

    2021

  7. [15]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M \"u ller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Mach...

    2024

  8. [16]

    An image is worth one word: Personalizing text-to-image generation using textual inversion

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618, 2022

    2022 arXiv

  9. [17]

    Parser-free virtual try-on via distilling appearance flows

    Yuying Ge, Yibing Song, Ruimao Zhang, Chongjian Ge, Wei Liu, and Ping Luo. Parser-free virtual try-on via distilling appearance flows. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 8485--8493, 2021

    2021

  10. [18]

    Tokenflow: Consistent diffusion features for consistent video editing

    Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arXiv:2307.10373, 2023

    2023 arXiv

  11. [19]

    Taming the power of diffusion models for high-quality virtual try-on with appearance flow

    Junhong Gou, Siyu Sun, Jianfu Zhang, Jianlou Si, Chen Qian, and Liqing Zhang. Taming the power of diffusion models for high-quality virtual try-on with appearance flow. In Proceedings of the 31st ACM International Conference on Multimedia, pp.\ 7599--7607, 2023

    2023

  12. [20]

    Animatediff: Animate your personalized text-to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023

    2023 arXiv

  13. [21]

    Viton: An image-based virtual try-on network

    Xintong Han, Zuxuan Wu, Zhe Wu, Ruichi Yu, and Larry S Davis. Viton: An image-based virtual try-on network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 7543--7552, 2018

    2018

  14. [22]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  15. [23]

    Prompt-to-prompt image editing with cross attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022

    2022 arXiv

  16. [24]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017

    2017

  17. [25]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33: 0 6840--6851, 2020

    2020

  18. [26]

    Imagen video: High definition video generation with diffusion models

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022 a

    arXiv

  19. [27]

    Video diffusion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35: 0 8633--8646, 2022 b

    2022

  20. [28]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868, 2022

    2022 arXiv

  21. [29]

    Image quality metrics: Psnr vs

    Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference on pattern recognition, pp.\ 2366--2369. IEEE, 2010

    2010

  22. [30]

    Animate anyone: Consistent and controllable image-to-video synthesis for character animation

    Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. arXiv preprint arXiv:2311.17117, 2023

    2023 arXiv

  23. [31]

    Composer: Creative and controllable image synthesis with composable conditions

    Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Composer: Creative and controllable image synthesis with composable conditions. arXiv preprint arXiv:2302.09778, 2023

    2023 arXiv

  24. [32]

    Learning high fidelity depths of dressed humans by watching social media dance videos

    Yasamin Jafarian and Hyun Soo Park. Learning high fidelity depths of dressed humans by watching social media dance videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 12753--12762, 2021

    2021

  25. [33]

    Dreampose: Fashion video synthesis with stable diffusion

    Johanna Karras, Aleksander Holynski, Ting-Chun Wang, and Ira Kemelmacher-Shlizerman. Dreampose: Fashion video synthesis with stable diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 22680--22690, 2023

    2023

  26. [34]

    Text2video-zero: Text-to-image diffusion models are zero-shot video generators

    Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. In Proceedings of the IEEE/CVF International Conference on Computer ...

    2023

  27. [35]

    Stableviton: Learning semantic correspondence with latent diffusion model for virtual try-on, 2023

    Jeongho Kim, Gyojung Gu, Minho Park, Sunghyun Park, and Jaegul Choo. Stableviton: Learning semantic correspondence with latent diffusion model for virtual try-on, 2023

    2023

  28. [36]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  29. [37]

    Video-p2p: Video editing with cross-attention control

    Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. Video-p2p: Video editing with cross-attention control. arXiv preprint arXiv:2303.04761, 2023

    2023 arXiv

  30. [38]

    Sdedit: Guided image synthesis and editing with stochastic differential equations

    Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073, 2021

    2021 arXiv

  31. [39]

    T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 429...

    2024

  32. [40]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021

    2021 arXiv

  33. [41]

    Improved denoising diffusion probabilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pp.\ 8162--8171. PMLR, 2021

    2021

  34. [42]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 4195--4205, 2023

    2023

  35. [43]

    Fatezero: Fusing attentions for zero-shot text-based video editing

    Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 15932--15942, 2023

    2023

  36. [44]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp...

    2021

  37. [45]

    Hierarchical text-conditional image generation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1 0 (2): 0 3, 2022

    2022 arXiv

  38. [46]

    Deep spatial transformation for pose-guided person image generation and animation

    Yurui Ren, Ge Li, Shan Liu, and Thomas H Li. Deep spatial transformation for pose-guided person image generation and animation. IEEE Transactions on Image Processing, 29: 0 8622--8635, 2020

    2020

  39. [47]

    Neural texture extraction and distribution for controllable person image synthesis

    Yurui Ren, Xiaoqing Fan, Ge Li, Shan Liu, and Thomas H Li. Neural texture extraction and distribution for controllable person image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 13535--13544, 2022

    2022

  40. [48]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj \"o rn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 10684--10695, 2022

    2022

  41. [49]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention--MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part...

    2015

  42. [50]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 22500-...

    2023

  43. [51]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information...

    2022

  44. [52]

    First order motion model for image animation

    Aliaksandr Siarohin, St \'e phane Lathuili \`e re, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Advances in neural information processing systems, 32, 2019

    2019

  45. [53]

    Motion representations for articulated animation

    Aliaksandr Siarohin, Oliver J Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion representations for articulated animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 13653--13662, 2021

    2021

  46. [54]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792, 2022

    2022 arXiv

  47. [55]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp.\ 2256--2265. PMLR, 2015

    2015

  48. [56]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020 a

    2010 arXiv

  49. [57]

    Generative modeling by estimating gradients of the data distribution

    Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019

    2019

  50. [58]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020 b

    2011 arXiv

  51. [59]

    Plug-and-play diffusion features for text-driven image-to-image translation

    Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 1921--1930, 2023

    1921

  52. [60]

    Towards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018

    2018 arXiv

  53. [61]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  54. [62]

    Disco: Disentangled control for referring human dance generation in real world

    Tan Wang, Linjie Li, Kevin Lin, Chung-Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Disco: Disentangled control for referring human dance generation in real world. arXiv e-prints, pp.\ arXiv--2307, 2023

    2023

  55. [63]

    Image quality assessment: from error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13 0 (4): 0 600--612, 2004

    2004

  56. [64]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference...

    2023

  57. [65]

    Gp-vton: Towards general purpose virtual try-on via collaborative local-flow global-parsing learning

    Zhenyu Xie, Zaiyu Huang, Xin Dong, Fuwei Zhao, Haoye Dong, Xijin Zhang, Feida Zhu, and Xiaodan Liang. Gp-vton: Towards general purpose virtual try-on via collaborative local-flow global-parsing learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern ...

    2023

  58. [66]

    Ootdiffusion: Outfitting fusion based latent diffusion for controllable virtual try-on

    Yuhao Xu, Tao Gu, Weifeng Chen, and Chengcai Chen. Ootdiffusion: Outfitting fusion based latent diffusion for controllable virtual try-on. arXiv preprint arXiv:2403.01779, 2024

    2024 arXiv

  59. [67]

    Magicanimate: Temporally consistent human image animation using diffusion model

    Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. arXiv preprint arXiv:2311.16498, 2023

    2023 arXiv

  60. [68]

    Paint by example: Exemplar-based image editing with diffusion models

    Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 18381--18391, 2023 a

    2023

  61. [69]

    Towards photo-realistic virtual try-on by adaptively generating-preserving image content

    Han Yang, Ruimao Zhang, Xiaobao Guo, Wei Liu, Wangmeng Zuo, and Ping Luo. Towards photo-realistic virtual try-on by adaptively generating-preserving image content. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 7850--7859, 2020

    2020

  62. [70]

    Rerender a video: Zero-shot text-guided video-to-video translation

    Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender a video: Zero-shot text-guided video-to-video translation. In SIGGRAPH Asia 2023 Conference Papers, pp.\ 1--11, 2023 b

    2023

  63. [71]

    Dwnet: Dense warp-based network for pose-guided human video generation

    Polina Zablotskaia, Aliaksandr Siarohin, Bo Zhao, and Leonid Sigal. Dwnet: Dense warp-based network for pose-guided human video generation. arXiv preprint arXiv:1910.09139, 2019

    1910 arXiv

  64. [72]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 3836--3847, 2023

    2023

  65. [73]

    Exploring dual-task correlation for pose guided person image generation

    Pengze Zhang, Lingxiao Yang, Jian-Huang Lai, and Xiaohua Xie. Exploring dual-task correlation for pose guided person image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 7713--7722, 2022

    2022

  66. [74]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 586--595, 2018

    2018

  67. [75]

    Thin-plate spline motion model for image animation

    Jian Zhao and Hui Zhang. Thin-plate spline motion model for image animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 3657--3666, 2022

    2022

  68. [76]

    Uni-controlnet: All-in-one control to text-to-image diffusion models

    Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan-Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  69. [77]

    Tryondiffusion: A tale of two unets

    Luyang Zhu, Dawei Yang, Tyler Zhu, Fitsum Reda, William Chan, Chitwan Saharia, Mohammad Norouzi, and Ira Kemelmacher-Shlizerman. Tryondiffusion: A tale of two unets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 4606--4615, 2023

    2023

  70. [78]

    Champ: Controllable and consistent human image animation with 3d parametric guidance

    Shenhao Zhu, Junming Leo Chen, Zuozhuo Dai, Yinghui Xu, Xun Cao, Yao Yao, Hao Zhu, and Siyu Zhu. Champ: Controllable and consistent human image animation with 3d parametric guidance. arXiv preprint arXiv:2403.14781, 2024

    2024 arXiv

  71. [79]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  72. [80]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  73. [81]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.