Pith. sign in

REVIEW 4 major objections 6 minor 53 references

Re-Attentional Controllable Video Diffusion Editing

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims spatial control in text-guided video editing can be achieved without retraining by steering cross-attention maps and re-injecting the original unchanged regions.

desk verdict Training-free attention-refocusing for video editing works better than its theory; the VISOR leap is real but partly by construction, and the IRJS distribution claim is wrong as written. read the letter →

arxiv 2412.11710 v1 pith:JPS75QUZ submitted 2024-12-16 cs.CV cs.AI

Yuanzhi Wang , Yong Li , Mengyi Liu , Xiaoya Zhang , Xin Liu , Zhen Cui , Antoni B. Chan This is my paper · ORCID
classification cs.CVcs.AI
keywords text-guidedvideoeditingcross-attentioncontroltraining-freecontrollablegenerationdiffusionmodelsattentionrefocusinginvariantregionpreservationspatialrelationship
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ReAtCo argues that the two persistent failures of prompt-based video editing—objects appearing in the wrong place or in the wrong number—can be fixed without any training, by manipulating the cross-attention maps that connect each word to pixels. During each denoising step it defines a loss that rewards high attention inside user-provided masks and low attention outside them, then takes a small gradient step on the noisy video latent. To keep untouched regions faithful, it stitches the edited latent inside the masks with the diffused source video outside them at every timestep. If correct, this gives a plug-and-play layer of spatial control that works on top of existing video editing models such as Tune-A-Video.

What carries the argument

The load-bearing mechanism is the cross-attention map between text and video frame in the latent diffusion backbone, $A_i^{(t)} = \mathrm{Softmax}(Q_i^{(t)}K^\top/\sqrt{d}) \in \mathbb{R}^{L\times H\times W}$, which records how strongly each word attends to each pixel region. RAD turns each word's attention map into a differentiable objective: maximize the top-k activations inside the user's mask and minimize the top-k activations outside it, then back-propagate through the denoiser to update the noisy latent. IRJS is the second mechanism: at each step it replaces the latent outside the object masks with the source video's own noised latent, so the invariant region is regenerated from the original content rather than sampled freely.

What would settle it

Compute the edited video for a prompt identical to the source prompt under IRJS; if the output deviates measurably from the source video in the invariant region or drifts frame-to-frame, the stitched latent is not behaving like a valid sample from the transition distribution. A second direct test is to compare IRJS against simply decoding the generated latent and compositing the clean source background, using PSNR and LPIPS on the invariant region; comparable scores would show that per-step injection, not final compositing, is responsible for the reported fidelity gains.

Watch

Extended reading notes

Core claim

The central claim is that controllability of text-guided video editing, specifically the spatial location and number of multiple foreground objects, can be achieved by refocusing cross-attention activation responses during denoising rather than by retraining or finetuning. The paper proposes Re-Attentional Diffusion (RAD), which defines inner-region and outer-region constraints on the cross-attention maps of the words of interest, uses top-k responses to stay robust to imprecise masks, and updates the noisy sample as $X^{(t)} \leftarrow X^{(t)} - \alpha_t \nabla(\mathcal{L}_{\mathrm{IR}}(t)+\mathcal{L}_{\mathrm{OR}}(t))$. It further claims that an Invariant Region-guided Joint Sampling (IRJS) strategy removes sampling drift in unchanged regions by forming $\tilde{X}^{(t-1)} = X^{(t-1)}\times M + V^{(t-1)}\times (1-M)$, arguing that this mixture still follows the denoiser's Gaussian transition distribution. On the evaluated benchmarks ReAtCo reports the best scores in frame consistency, textual alignment, PickScore, and especially VISOR, with the spatial-relationship metric rising from 25.31 for the strongest baseline to 70.62.

Load-bearing premise

IRJS assumes that a latent formed by stitching two independent noisy samples, the edited object region and the diffused source background, is still a valid sample from the denoiser's next-step Gaussian distribution, even though no proof is given that the mixture is in-distribution.

Editorial extensions

If this is right

  • Spatial control becomes a training-free add-on: any video diffusion editing model built on cross-attention can inherit location control by injecting masks and gradient updates at denoising time.
  • Multi-object edits with explicit left, middle, and right relations become feasible, as demonstrated with two-object and three-object scenes in the paper.
  • Untouched regions stay faithful: the appendix reports that IRJS raises invariant-region PSNR from 29.29 to 36.57 dB and lowers LPIPS from 0.2706 to 0.0206.
  • The method needs only bounding-box style masks from the user, not per-video training or finetuning.
  • The attention maps become inspectable: the paper shows that with RAD, the jellyfish word's attention converges into the user-specified region as denoising progresses, while without RAD it drifts away.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same top-k attention loss could be applied per object to enforce counts, for example by penalizing attention mass outside a single connected mask per word, a setting the paper does not test.
  • Because IRJS injects the source video's noised latent outside masks, the quality ceiling of the invariant region is set by DDIM inversion; a source video that inverts poorly should show background artifacts even with IRJS.
  • RAD's inner and outer loss could transfer directly to text-to-image generation, since they only assume a backbone with cross-attention maps; the paper demonstrates them only for video editing.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes ReAtCo, a training-free inference-time method for text-guided video editing built on Tune-A-Video. It introduces two components: Re-Attentional Diffusion (RAD), which uses gradient updates on the noisy latent to maximize cross-attention responses of selected words inside user-specified masks and suppress them outside, and Invariant Region-guided Joint Sampling (IRJS), which stitches the diffused source video's invariant region into the denoised latent at each step. Experiments on LOVEU-TGVE-2023 and additional samples compare ReAtCo against Tune-A-Video, FateZero, StableVideo, TCVE, and TokenFlow, reporting a large VISOR gain and modest improvements in Frame Consistency, Textual Alignment, and PickScore. The paper also provides ablations, cross-attention visualizations, and a sensitivity study of the top-k hyperparameter K.

Significance. If the results hold, RAD provides a practical, plug-and-play spatial control for video editing without retraining, and the release of code supports reproducibility. The ablation study and cross-attention visualizations give some evidence for RAD's effectiveness. However, the distributional claim underlying IRJS is unsupported, the headline VISOR metric is closely aligned with the RAD objective and its hyperparameter is selected on the evaluation set, and the quantitative comparisons lack variance estimates. These issues currently temper the significance of the paper's central claims, although the core idea is promising and the limitations appear addressable.

major comments (4)
  1. [Invariant Region-guided Joint Sampling (Eq. 8)] In the paragraph immediately after Eq. (8), the paper claims that because X(t-1) ~ N(µθ, Σθ) and V(t-1) ~ N(√ᾱ_t V(0), (1-ᾱ_t)I) are 'approximately the same distribution,' the masked mixture eX(t-1) = X(t-1)M + V(t-1)(1-M) is 'abided by the distribution of X(t-1).' This is not a derivation. For a fixed binary mask M, the covariance of eX(t-1) is MΣθM^T + (1-M)(1-ᾱ_t)I(1-M)^T, which equals Σθ only in the special case Σθ = (1-ᾱ_t)I. In DDPM the posterior variance is β̃_t I, which is smaller than the forward marginal variance, and in DDIM with η=0 the transition is deterministic, so the mixture is generally not a draw from N(µθ, Σθ) and the noise level changes discontinuously at mask boundaries. The Appendix's PSNR/LPIPS improvements show that the injection helps empirically, but they do not validate the distributional premise. Since IRJS is a core contribution, the authors should either provide a correct distributional analysis (e.g., showing the mismatch is negligible at the resolutions and timesteps used) or explicitly reframe IRJS as an empirical heuristic and quantify boundary artifacts.
  2. [Exploring the effective K in topk(·, K) (Fig. 7)] The value K=20% is selected by maximizing VISOR on the evaluation set. This is problematic because the RAD objective directly raises cross-attention responses inside the masks and suppresses them outside, while VISOR measures exactly whether objects appear in the expected spatial relation (left/right/above/below) derived from those masks. Part of the reported VISOR gain (70.62 vs. 25.31 in Table 1) is therefore by construction rather than an independent measure of editing quality. The authors should report how K was chosen (e.g., on a validation split), give VISOR for K values not selected, and ideally also report an object-detection-based metric that is not aligned with the loss. Without this, the headline quantitative claim is overstated.
  3. [Quantitative results (Tables 1 and 2)] Tables 1 and 2 report point estimates only, with no error bars, significance tests, or number of videos per condition. The text states that ReAtCo 'consistently improves' and 'achieves superior video editing performance,' but the margins on Frame Consistency (95.24 vs. 94.79 for TCVE) and PickScore (20.70 vs. 20.58) are small relative to typical run-to-run variation for diffusion-based editing. Please report standard deviations over multiple runs or a paired per-video test, and state how many videos and prompts contribute to each number. This is necessary to support the 'consistently' claim.
  4. [Appendix: Ablation Study of IRJS (Table 3)] Table 3 reports PSNR/LPIPS for 'Ours w/o IRJS' vs. 'Ours.' Because IRJS literally replaces the invariant region with the diffused source video sample V(t-1), which is an approximation of the original source content, it is expected by construction that PSNR rises and LPIPS falls. This does not by itself demonstrate that the generated object region is 'harmonized' with the invariant region or that border artifacts are reduced, which is the stated motivation of IRJS. Please add a quantitative measure of boundary artifacts or a user study, or at least show cropped boundary regions for Ours w/o IRJS and Ours. The single frame in Fig. 3(d) is not sufficient.
minor comments (6)
  1. [Notation (Eq. 8)] The symbol V(t-1) in Eq. (8) conflicts with V used for the source video in the problem statement; please use a distinct notation such as V_src(t-1) or Z(t-1).
  2. [Eqs. (3)-(5)] The action of topk(·, K) is used in the equations but not formally defined; as written, it is ambiguous whether it returns the set of K largest values or their sum. Please define it explicitly.
  3. [Terminology] The paper calls the method 'training-free,' but RAD requires backpropagation through the UNet at inference; please clarify that no parameter updates are performed and that the gradient is used only to update the latent.
  4. [Appendix: More Implementation Details] The statement that 'αt decays linearly from 1 to 0.5 during the denoising process' should specify the timestep range over which this decay occurs, given that the model uses 50 denoising steps.
  5. [Table 2 caption] The row with '× ×' corresponds to the Tune-A-Video baseline; please state this explicitly in the caption for clarity.
  6. [Introduction and Abstract] There are minor typographical errors: 'Co ntrollable' in the Abstract and 'editied' in the Introduction should be corrected.

Circularity Check

1 steps flagged · score 6.0 of 10

IRJS's invariant-region preservation is confirmed by construction: the final latent is a direct paste of the source background, so the PSNR/LPIPS ablation measures the injection operation itself.

  1. self definitional [Invariant Region-guided Joint Sampling, Eq. (8); Appendix 'Ablation Study of IRJS in the Invariant Region', Table 3]
    "eX (t−1) = X (t−1) × M + V(t−1) × (1−M) ... V(t − 1) ∼ N(√¯αtV(0), (1 − ¯αt)I) ... These results support the fact that our proposed IRJS effectively maintains the invariant region content during video editing."

    At the final denoising step, t−1 = 0, the term V(0) is the source video latent, so eX(0) contains the source latent in every pixel outside M. The invariant region of the output is thus the original source invariant region by definition. The PSNR/LPIPS gain in Table 3 therefore measures the copy/paste operation in Eq. (8) rather than a mitigation of sampling errors; the evaluated property coincides with the operation that defines IRJS.

full rationale

The central spatial-controllability claim rests on RAD, a training-free optimization of cross-attention maps. That mechanism is not circular: it is an external, falsifiable intervention on latents, and the VISOR improvements, although aligned with the objective, require the diffusion model to actually render the requested objects in the masked regions. The main circular element is IRJS's invariant-region fidelity: because V(0) is the source video, Eq. (8) literally pastes the source background into the final latent, so the PSNR/LPIPS ablation in Table 3 is a self-fulfilling measure. The distributional assertion that eX(t−1) ~ N(µθ,Σθ) is not circular but is an unsupported (and generally false) claim about masked Gaussian mixtures; this is a correctness risk, not a circularity. No load-bearing self-citation chain appears in the paper. Overall, the headline VISOR and editing-quality results retain independent content, so the circularity is partial and confined to the invariant-region evaluation.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on standard diffusion editing assumptions, two hand-set hyperparameters (K and α_t), and one unproved distributional assumption in IRJS. No new physical or conceptual entities are introduced.

free parameters (2)
  • K in topk for RAD loss = 20% of mask region count (adaptively set)
    Selected via ablation on VISOR metric (Fig. 7); controls how many top responses are considered in the inner/outer constraints.
  • α_t step size schedule = linearly decays from 1.0 to 0.5
    Hand-set per the appendix; controls the gradient update strength at each denoising step.
assumptions (4)
  • domain assumption Cross-attention response maps localize where each text token is generated in the image.
    Invoked in the Method section to justify using cross-attention maps for spatial control; the paper states 'the higher the response, the more the word concept is being attended to in that region'.
  • domain assumption The user-provided masks M accurately separate objects of interest from the invariant region.
    The entire RAD and IRJS rely on mask correctness; masks are specified manually or by an object detector, and no error analysis of mask quality is provided.
  • ad hoc to paper The stitched latent eX(t-1) from Eq. 8 follows the same Gaussian transition as X(t-1).
    Stated in the IRJS section: the paper approximates the model's transition distribution with the diffused source distribution and assumes the mixture remains in-distribution. This is not proven and is the fragile step.
  • domain assumption DDIM inversion of the source video reliably produces an editable noise trajectory.
    The method builds on Tune-A-Video's DDIM inversion; if inversion fails, editing quality degrades, but this is a standard assumption in the field.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Re-Attentional Controllable Video Diffusion Editing." pith.science (2026). https://pith.science/paper/JPS75QUZ

@misc{pith2026241211710,
  author       = {Pith},
  title        = {Pith review of: Re-Attentional Controllable Video Diffusion Editing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JPS75QUZ}},
  note         = {Machine review of arXiv:2412.11710}
}
read the original abstract

Editing videos with textual guidance has garnered popularity due to its streamlined process which mandates users to solely edit the text prompt corresponding to the source video. Recent studies have explored and exploited large-scale text-to-image diffusion models for text-guided video editing, resulting in remarkable video editing capabilities. However, they may still suffer from some limitations such as mislocated objects, incorrect number of objects. Therefore, the controllability of video editing remains a formidable challenge. In this paper, we aim to challenge the above limitations by proposing a Re-Attentional Controllable Video Diffusion Editing (ReAtCo) method. Specially, to align the spatial placement of the target objects with the edited text prompt in a training-free manner, we propose a Re-Attentional Diffusion (RAD) to refocus the cross-attention activation responses between the edited text prompt and the target video during the denoising stage, resulting in a spatially location-aligned and semantically high-fidelity manipulated video. In particular, to faithfully preserve the invariant region content with less border artifacts, we propose an Invariant Region-guided Joint Sampling (IRJS) strategy to mitigate the intrinsic sampling errors w.r.t the invariant regions at each denoising timestep and constrain the generated content to be harmonized with the invariant region content. Experimental results verify that ReAtCo consistently improves the controllability of video diffusion editing and achieves superior video editing performance.

Figures

Figures reproduced from arXiv: 2412.11710 by the authors.

Figure 1
Figure 1. Edited samples from the common video diffusion [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The framework of our proposed ReAtCo. Given a source video [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Edited video frames by different methods. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Visual comparisons of different methods in various scenes. Compared with these state-of-the-arts, ReAtCo can edit [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Visualization of cross-attention maps. Exploring the effective K in topk (·, K). We conduct the ablation studies to explore the effective K in topk (·, K) [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: illustrates the performance of our method with various K in topk (·, K) under VISOR metric, and we can observe that the best performance is reached when K is set to 20%. Subsequently, the performance is degraded as K increases. The above phenomenon demonstrates the fac…
Figure 8
Figure 8. Figure 8: An example of controlling the objects in the form [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 20 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Avrahami, O.; Hayes, T.; Gafni, O.; Gupta, S.; Taigman, Y.; Parikh, D.; Lischinski, D.; Fried, O.; and Yin, X. 2023. Spatext: Spatio-textual representation for controllable image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 18370--18380

    2023

  4. [4]

    Bar-Tal, O.; Ofri-Amar, D.; Fridman, R.; Kasten, Y.; and Dekel, T. 2022. Text2live: Text-driven layered image and video editing. In European Conference on Computer Vision, 707--723. Springer

    work page 2022

  5. [5]

    W.; Fidler, S.; and Kreis, K

    Blattmann, A.; Rombach, R.; Ling, H.; Dockhorn, T.; Kim, S. W.; Fidler, S.; and Kreis, K. 2023. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 22563--22575

    2023

  6. [6]

    Chai, W.; Guo, X.; Wang, G.; and Lu, Y. 2023. Stablevideo: Text-driven consistency-aware diffusion video editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 23040--23050

    work page 2023

  7. [7]

    Chen, W.; Wu, J.; Xie, P.; Wu, H.; Li, J.; Xia, X.; Xiao, X.; and Lin, L. 2023. Control-A-Video: Controllable Text-to-Video Generation with Diffusion Models. arXiv preprint arXiv:2305.13840

    arXiv 2023

  8. [8]

    Daras, G.; Dagan, Y.; Dimakis, A.; and Daskalakis, C. 2024. Consistent diffusion models: Mitigating sampling drift by learning to be consistent. Advances in Neural Information Processing Systems, 36

    work page 2024

Show all 53 references
  1. [9]

    Ge, S.; Nah, S.; Liu, G.; Poon, T.; Tao, A.; Catanzaro, B.; Jacobs, D.; Huang, J.-B.; Liu, M.-Y.; and Balaji, Y. 2023. Preserve your own correlation: A noise prior for video diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 22930--22941

    2023

  2. [10]

    Geyer, M.; Bar-Tal, O.; Bagon, S.; and Dekel, T. 2024. TokenFlow: Consistent Diffusion Features for Consistent Video Editing. In The Twelfth International Conference on Learning Representations

    2024

  3. [11]

    Gokhale, T.; Palangi, H.; Nushi, B.; Vineet, V.; Horvitz, E.; Kamar, E.; Baral, C.; and Yang, Y. 2022. Benchmarking spatial relationships in text-to-image generation. arXiv preprint arXiv:2212.10015

    2022 arXiv

  4. [12]

    Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014. Generative adversarial nets. Advances in neural information processing systems, 27

    2014

  5. [13]

    Hertz, A.; Mokady, R.; Tenenbaum, J.; Aberman, K.; Pritch, Y.; and Cohen-or, D. 2023. Prompt-to-Prompt Image Editing with Cross-Attention Control. In The Eleventh International Conference on Learning Representations

    2023

  6. [14]

    P.; Poole, B.; Norouzi, M.; Fleet, D

    Ho, J.; Chan, W.; Saharia, C.; Whang, J.; Gao, R.; Gritsenko, A.; Kingma, D. P.; Poole, B.; Norouzi, M.; Fleet, D. J.; et al. 2022 a . Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303

    2022 arXiv

  7. [15]

    Ho, J.; Jain, A.; and Abbeel, P. 2020. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33: 6840--6851

    2020

  8. [16]

    Ho, J.; Salimans, T.; Gritsenko, A.; Chan, W.; Norouzi, M.; and Fleet, D. J. 2022 b . Video diffusion models. Advances in Neural Information Processing Systems

    2022

  9. [17]

    Kasten, Y.; Ofri, D.; Wang, O.; and Dekel, T. 2021. Layered neural atlases for consistent video editing. ACM Transactions on Graphics (TOG), 40(6): 1--12

    2021

  10. [18]

    Kirstain, Y.; Polyak, A.; Singer, U.; Matiana, S.; Penna, J.; and Levy, O. 2023. Pick-a-pic: An open dataset of user preferences for text-to-image generation. arXiv preprint arXiv:2305.01569

    2023 arXiv

  11. [19]

    Li, Y.; and Shan, S. 2023. Contrastive learning of person-independent representations for facial action unit detection. IEEE Transactions on Image Processing, 32: 3212--3225

    2023

  12. [20]

    Li, Y.; Zeng, J.; and Shan, S. 2020. Learning representations for facial actions from unlabeled videos. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1): 302--317

    2020

  13. [21]

    Li, Y.; Zeng, J.; Shan, S.; and Chen, X. 2018. Occlusion aware facial expression recognition using CNN with attention mechanism. IEEE Transactions on Image Processing, 28(5): 2439--2450

    2018

  14. [22]

    Luo, J.; Wang, Y.; Gu, Z.; Qiu, Y.; Yao, S.; Wang, F.; Xu, C.; Zhang, W.; Wang, D.; and Cui, Z. 2024. MMM-RS: A Multi-modal, Multi-GSD, Multi-scene Remote Sensing Dataset and Benchmark for Text-to-Image Generation. In The Thirty-eight Conference on Neural Information Processin...

    2024

  15. [23]

    L.; and Salakhutdinov, R

    Mansimov, E.; Parisotto, E.; Ba, J. L.; and Salakhutdinov, R. 2016. Generating images from captions with attention. In International Conference on Learning Representations

    2016

  16. [24]

    Phung, Q.; Ge, S.; and Huang, J.-B. 2024. Grounded text-to-image synthesis with attention refocusing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7932--7942

    2024

  17. [25]

    Qi, C.; Cun, X.; Zhang, Y.; Lei, C.; Wang, X.; Shan, Y.; and Chen, Q. 2023. Fatezero: Fusing attentions for zero-shot text-based video editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision

    2023

  18. [26]

    Qing, Z.; Zhang, S.; Wang, J.; Wang, X.; Wei, Y.; Zhang, Y.; Gao, C.; and Sang, N. 2023. Hierarchical Spatio-temporal Decoupling for Text-to-Video Generation. arXiv preprint arXiv:2312.04483

    2023 arXiv

  19. [27]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748--8763. PMLR

    2021

  20. [28]

    Ramesh, A.; Dhariwal, P.; Nichol, A.; Chu, C.; and Chen, M. 2022. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125

    2022 arXiv

  21. [29]

    Ramesh, A.; Pavlov, M.; Goh, G.; Gray, S.; Voss, C.; Radford, A.; Chen, M.; and Sutskever, I. 2021. Zero-shot text-to-image generation. In International Conference on Machine Learning, 8821--8831. PMLR

    2021

  22. [30]

    Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 10684--10695

    2022

  23. [31]

    L.; Ghasemipour, K.; Gontijo Lopes, R.; Karagol Ayan, B.; Salimans, T.; et al

    Saharia, C.; Chan, W.; Saxena, S.; Li, L.; Whang, J.; Denton, E. L.; Ghasemipour, K.; Gontijo Lopes, R.; Karagol Ayan, B.; Salimans, T.; et al. 2022. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Syste...

    2022

  24. [32]

    Shen, F.; Jiang, X.; He, X.; Ye, H.; Wang, C.; Du, X.; Li, Z.; and Tang, J. 2024 a . Imagdressing-v1: Customizable virtual dressing. arXiv preprint arXiv:2407.12705

    2024 arXiv

  25. [33]

    Shen, F.; and Tang, J. 2024. IMAGPose: A Unified Conditional Framework for Pose-Guided Person Generation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  26. [34]

    Shen, F.; Ye, H.; Liu, S.; Zhang, J.; Wang, C.; Han, X.; and Yang, W. 2024 b . Boosting consistency in story visualization with rich-contextual conditional diffusion models. arXiv preprint arXiv:2407.02482

    2024 arXiv

  27. [35]

    Shen, F.; Ye, H.; Zhang, J.; Wang, C.; Han, X.; and Wei, Y. 2024 c . Advancing Pose-Guided Image Synthesis with Progressive Conditional Diffusion Models. In The Twelfth International Conference on Learning Representations

    2024

  28. [36]

    Song, J.; Meng, C.; and Ermon, S. 2021. Denoising Diffusion Implicit Models. In International Conference on Learning Representations

    2021

  29. [37]

    N.; Kaiser, L.; and Polosukhin, I

    Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  30. [38]

    Videvo. 2024. Free stock video footage. https://www.videvo.net/

    2024

  31. [39]

    Wang, F.-Y.; Chen, W.; Song, G.; Ye, H.-J.; Liu, Y.; and Li, H. 2023. Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising. arXiv preprint arXiv:2305.18264

    2023 arXiv

  32. [40]

    Wang, Y.; Cui, Z.; and Li, Y. 2023. Distribution-consistent modal recovering for incomplete multimodal learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 22025--22034

    2023

  33. [41]

    Wang, Y.; Li, Y.; and Cui, Z. 2024. Incomplete multimodality-diffused emotion recognition. Advances in Neural Information Processing Systems, 36

    2024

  34. [42]

    B.; and Cui, Z

    Wang, Y.; Li, Y.; Zhang, X.; Liu, X.; Dai, A.; Chan, A. B.; and Cui, Z. 2024. Edit Temporal-Consistent Videos with Image Diffusion Model. ACM Transactions on Multimedia Computing, Communications, and Applications, 20(12)

    2024

  35. [43]

    Wang, Y.; Lu, T.; Zhang, Y.; Wang, Z.; Jiang, J.; and Xiong, Z. 2022. FaceFormer: Aggregating global and local representation for face hallucination. IEEE Transactions on Circuits and Systems for Video Technology, 33(6): 2533--2545

    2022

  36. [44]

    Z.; Ge, Y.; Wang, X.; Lei, W.; Gu, Y.; Hsu, W.; Shan, Y.; Qie, X.; and Shou, M

    Wu, J. Z.; Ge, Y.; Wang, X.; Lei, W.; Gu, Y.; Hsu, W.; Shan, Y.; Qie, X.; and Shou, M. Z. 2023 a . Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision

    2023

  37. [45]

    Z.; Li, X.; Gao, D.; Dong, Z.; Bai, J.; Singh, A.; Xiang, X.; Li, Y.; Huang, Z.; Sun, Y.; He, R.; Hu, F.; Hu, J.; Huang, H.; Zhu, H.; Cheng, X.; Tang, J.; Shou, M

    Wu, J. Z.; Li, X.; Gao, D.; Dong, Z.; Bai, J.; Singh, A.; Xiang, X.; Li, Y.; Huang, Z.; Sun, Y.; He, R.; Hu, F.; Hu, J.; Huang, H.; Zhu, H.; Cheng, X.; Tang, J.; Shou, M. Z.; Keutzer, K.; and Iandola, F. 2023 b . CVPR 2023 Text Guided Video Editing Competition. arXiv:2310.16003

    2023 arXiv

  38. [46]

    Wu, Q.; Liu, Y.; Zhao, H.; Bui, T.; Lin, Z.; Zhang, Y.; and Chang, S. 2023 c . Harnessing the spatial-temporal attention of diffusion models for high-fidelity text-to-image synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 7766--7776

    2023

  39. [47]

    Z.; Shen, C.; Cheng, L.; Li, Y.; Gao, T.; Zhang, D.; and Wang, Z

    Wu, W.; Li, Z.; He, Y.; Shou, M. Z.; Shen, C.; Cheng, L.; Li, Y.; Gao, T.; Zhang, D.; and Wang, Z. 2023 d . Paragraph-to-Image Generation with Information-Enriched Diffusion Model. arXiv preprint arXiv:2311.14284

    2023 arXiv

  40. [48]

    Yang, Z.; Wang, J.; Gan, Z.; Li, L.; Lin, K.; Wu, C.; Duan, N.; Liu, Z.; Liu, C.; Zeng, M.; et al. 2023. Reco: Region-controlled text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14246--14255

    2023

  41. [49]

    Zhang, D.; Zhang, H.; Tang, J.; Wang, M.; Hua, X.; and Sun, Q. 2020. Feature pyramid transformer. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XXVIII 16, 323--339. Springer

    2020

  42. [50]

    Zhang, L.; Rao, A.; and Agrawala, M. 2023. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 3836--3847

    2023

  43. [51]

    A.; Shechtman, E.; and Wang, O

    Zhang, R.; Isola, P.; Efros, A. A.; Shechtman, E.; and Wang, O. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, 586--595

    2018

  44. [52]

    Zhang, Y.; Wei, Y.; Jiang, D.; Zhang, X.; Zuo, W.; and Tian, Q. 2023. ControlVideo: Training-free Controllable Text-to-Video Generation. arXiv preprint arXiv:2305.13077

    2023 arXiv

  45. [53]

    Zhao, M.; Wang, R.; Bao, F.; Li, C.; and Zhu, J. 2023. ControlVideo: Adding Conditional Control for One Shot Text-to-Video Editing. arXiv preprint arXiv:2305.17098

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.