Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

UniPaint: Unified Space-time Video Inpainting via Mixture-of-Experts

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read UniPaint claims that video inpainting, outpainting, and frame interpolation are all one mask-filling task, and that training them together improves every one of them.

desk verdict A sensible unified masking framework with MoE attention, but the mutual-enhancement claim is under-supported by small metric differences and some own-table contradictions. read the letter →

arxiv 2412.06340 v2 pith:AA6WOPRX submitted 2024-12-09 cs.CV

Zhen Wan , Chenyang Qi , Zhiheng Liu , Tao Gui , Yue Ma This is my paper · ORCID
classification cs.CV
keywords videoinpaintinginterpolationoutpaintingdiffusionmodelmixtureofexpertstext-guidededitingspace-timetemporalconsistency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

UniPaint claims that video inpainting, video outpainting, and video frame interpolation are all instances of one problem: filling masked space-time regions of a video. The paper proposes a single diffusion-based framework that adds a plug-and-play mask-conditioned adapter and a Mixture-of-Experts attention module to a pretrained text-to-video model, then trains it with a mixed masking strategy covering spatial and temporal cases. Its central assertion is that training on spatial and temporal inpainting together improves both tasks, rather than hurting either. If correct, this collapses separately engineered tasks into one mask-filling formulation, giving video-editing pipelines a single model and a shared training recipe.

What carries the argument

The load-bearing mechanism is a mask-conditioned Mixture-of-Experts (MoE) attention inserted into the motion module of a pretrained video diffusion model. A gating network takes the resized binary mask through 3D downsampling convolutions and adaptive pooling, then outputs a weight vector over several expert feedforward networks, and the layer's output is a weighted sum of the experts' outputs. This lets the same network route differently for spatially local masks, marginal outpainting masks, and full-frame temporal masks. A plug-and-play Space-time Inpainting Adapter, a separate control branch whose features are added to the pretrained backbone with a user-specified scale, supplies pixel-level alignment with unmasked content, while a mixed-mask training protocol combining segmentation-based, random, marginal, and interpolation masks trains the model on all scenarios at once.

What would settle it

Re-run the unified model and both task-specific ablations on the same evaluation set with multiple random seeds and compute confidence intervals on the background-preservation and PSNR differences; if the unified model does not consistently beat the single-task models beyond the noise range, the mutual-enhancement claim fails. A complementary check is to train the spatial-only and temporal-only ablations for exactly the same number of steps and with the same mixed-mask curriculum minus the other task's masks, ruling out that the gain comes from extra training signal or mask diversity rather than from cross-task enhancement.

Watch

Extended reading notes

Core claim

The paper's central claim is that spatial inpainting, outpainting, and temporal interpolation can be unified under a single mask-filling objective, and that joint training produces mutual enhancement. In the paper's own framing, different tasks correspond to different mask shapes—localized regions for inpainting, marginal bands for outpainting, entire frames for interpolation—so a model that learns to condition on the mask can handle all of them. The authors report that the unified model outperforms its own task-specific ablations on both spatial inpainting (background-preservation L1 41.8 versus 42.2) and temporal interpolation (PSNR 20.01 versus 19.82), and that a Mixture-of-Experts attention module is what lets the model adapt to mask shape. They also show that the adapter is plug-and-play, able to extend a personalized text-to-video backbone without retuning the backbone's parameters.

Load-bearing premise

The paper's mutual-enhancement conclusion assumes the small measured advantages of the unified model over task-specific training (background preservation 41.8 versus 42.2, PSNR 20.01 versus 19.82) are genuine improvements and not random variation, since the reported numbers come without error bars, repeated runs, or significance tests.

Editorial extensions

If this is right

  • A single checkpoint can replace separate models for video inpainting, outpainting, and interpolation, simplifying video-editing pipelines.
  • Users can tune the adapter's control scale to trade off how strongly the unmasked background is preserved against how freely the model generates inside the mask.
  • Because the adapter is plug-and-play, the same training recipe can transfer to new or personalized text-to-video backbones without retraining the backbone.
  • Mask-shape-conditioned routing suggests that other editing operations, such as object removal, environment swap, and re-texturing, can be expressed as mask-filling and handled by the same model.
  • The mixed-mask training protocol provides a concrete template for learning multiple video-editing capabilities jointly rather than as separate models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • As an extension the paper leaves implicit: if the mutual-enhancement result survives repeated runs, it suggests a general recipe for generative video models—tasks that share a mask-conditioning interface can be trained jointly to regularize one another, potentially extending to video super-resolution and video prediction as the authors list for future work.
  • A testable follow-up is to probe whether the MoE gating interpolates smoothly between mask-shape categories; if it does, the same gating could handle partially specified or noisy masks, such as user scribbles or imperfect automatic segmentation.
  • The paper does not quantify the compute saved by unification—it reports about three days of training on eight GPUs but not per-task baseline training costs—so a direct efficiency comparison between one unified model and the ensemble of specialists remains an open measurement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes UniPaint, a unified diffusion-based framework for spatial video inpainting, outpainting, and temporal interpolation. The method builds on a pretrained text-to-video model and adds a plug-and-play Space-time Inpainting Adapter, a Mixture-of-Experts (MoE) attention module that gates expert feedforward networks based on the mask shape, and a mixed spatial-temporal masking training strategy. The central claims are that spatial and temporal inpainting can be treated as a single mask-filling task, that the two tasks mutually enhance each other when trained jointly, and that UniPaint achieves state-of-the-art quantitative results across these tasks. The paper reports quantitative comparisons on DAVIS for inpainting, outpainting, and interpolation, plus ablations of the MoE module and the adapter control scale.

Significance. If the mutual-enhancement claim were firmly established, UniPaint would be a useful contribution: it provides a single adapter that handles several video editing tasks, uses a mask-gated MoE to adapt to different mask geometries, and releases code and checkpoints. The paper also honestly includes failure cases and limitations. However, the central quantitative evidence is currently too weak to support the headline claims. The differences between the unified model and task-specific ablations are small, no error bars or significance tests are given, and the paper's own tables contradict the 'best quantitative results' statement on two metrics. The idea is plausible and the architectural components are reasonable, but the evidence base needs substantial strengthening before the claims can be accepted.

major comments (4)
  1. [Section 4.1, Tables 2 and 3] The central mutual-enhancement claim rests on small metric gaps between Ours and the task-specific ablations, e.g., BP 41.8 vs 42.2 for spatial inpainting and PSNR 20.01 vs 19.82 for temporal inpainting, but no standard deviations, number of seeds, or significance tests are reported. Without repeated-run statistics, these differences could easily be within run-to-run or dataset-sample noise, so the claim that training on both tasks improves both is not established. Please report mean and variance over multiple seeds or otherwise justify the reliability of these differences.
  2. [Abstract, Section 1, Section 4.1] The statement that UniPaint 'achieves the best quantitative results across various tasks and scale setups' is contradicted by the paper's own tables: in Table 2, CoCoCo attains a higher TC (97.6 vs 97.5), and in Table 3, VIDIM attains a lower FVD (199.32 vs 201.35). The claims should be revised to specify exactly which metrics and configurations UniPaint improves upon, and the defeats or ties should be acknowledged and discussed.
  3. [Section 4.1] The quantitative evaluation protocol is under-specified. The paper does not state which subset of DAVIS is used, how the inpainting, outpainting, and interpolation masks are generated for evaluation, what text prompts are used, how many video clips are averaged, or whether the same masks and prompts are presented to all baseline methods. For A VID, the numbers appear to be taken from the publication rather than rerun. Without this protocol, the comparisons in Tables 2 and 3 cannot be reproduced or fully assessed.
  4. [Section 3.4 and Section 4.1] The comparison between Ours and Ours Spatial./Ours Temporal. may not isolate the effect of unified training. If the task-specific ablations are trained with the same total number of optimizer steps but only a subset of mask types, the models see different numbers of examples per mask type; if they are trained with fewer steps, the comparison conflates compute with task mixing. Please specify the exact training schedule, number of steps, and data exposure for each ablation so that the mutual-enhancement conclusion is not confounded by training budget.
minor comments (6)
  1. [Section 3.3] The module is called 'MoE attention,' but the gating in Eqs. (4)-(5) applies only to expert feedforward networks, not to the temporal or cross-attention layers; please either clarify the terminology or adjust the architecture description to match the name.
  2. [Section 4.1] The text says 'A VID [85]' in the comparison paragraph, but A VID is reference [82] in the bibliography; please correct the citation.
  3. [Section 4 and Figure 7 caption] There are typos in the manuscript: 'Experiemnts' in the Section 4 heading and 'previsou methods' in the Figure 7 caption; please proofread.
  4. [Table 1] The check marks in Table 1 indicate capabilities, but 'Shape-Aware' is not defined anywhere and it is not clear how this property is evaluated; please add a definition or a reference.
  5. [Section 4.2] The ablation 'Ours w/o MoE' replaces the MoE with a single FFN, but the paper does not specify whether that single FFN has the same parameter count or initialization as the full MoE; this confounds the interpretation of the ablation.
  6. [References] Several references are to 2025 preprints that may not yet be peer-reviewed; please cite published versions where available, and ensure all cited works are relevant to the claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: UniPaint's central claims are empirical and evaluated against external baselines; self-citations are present in related work but are not load-bearing.

full rationale

The paper contains no derivation chain in which a predicted quantity reduces by construction to a fitted input or to a self-citation. The central claim—that spatial and temporal inpainting can be unified and mutually improve each other—is an empirical architecture-and-training claim. It is supported by ablations comparing 'Ours Spatial' (trained only with spatial inpainting cases) and 'Ours Temporal' (trained only with temporal inpainting cases) against the jointly trained 'Ours' in Tabs. 2 and 3. These are ablations over the training mask mixture, not fitted parameters renamed as predictions; the unified model is then evaluated against external methods (VideoComposer, CoCoCo, A VID, LDMVFI, VIDIM) on the DAVIS benchmark. The MoE gating network is trained with the mask as input, and the claim that experts specialize is an interpretation of ablation outputs, not a result forced by definition. Self-citations (e.g., Refs. [40], [43], [51]) appear only in the related-work survey and do not carry the load of the method's validity or the experimental comparison. The main weaknesses—small metric gaps without error bars or significance tests, and the abstract's 'best quantitative results' wording conflicting with VIDIM's lower FVD and CoCoCo's higher TC in the paper's own tables—are correctness and evidence-quality concerns, not circularity under the specified criteria.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method relies on standard pretrained diffusion components and public video datasets. The hand-chosen hyperparameters (mask probabilities, number of experts, guidance scale, control scale, inference steps) are not fitted to the evaluation targets in a circular way, but they do affect the reported numbers. No new physical or conceptual entities are introduced beyond the architectural MoE module.

free parameters (5)
  • Mask mixing probabilities = 0.4, 0.1, 0.2, 0.3
    Hand-chosen to balance segmentation, random, marginal, and interpolation masks during training (Sec. 3.4); results may depend on this schedule.
  • Number of experts = 4
    Set for MoE attention; no ablation over the number of experts is provided (Sec. 3.3).
  • Classifier-free guidance scale = 12.5
    Used for all inference results (Sec. 4, Implementation details).
  • Adapter control scale = 0.0, 0.5, 1.0 (ablated)
    User-controllable; default value not specified; qualitative effect shown in Fig. 8.
  • Number of inference steps = 100
    DDIM steps in Sec. 4.
assumptions (4)
  • domain assumption Pretrained Stable Diffusion v1.5, AnimateDiff, and BrushNet provide a reliable generative prior for video inpainting.
    The whole method is built on these frozen backbones (Sec. 3 and Implementation details).
  • domain assumption WebVid-10M and YouTubeVOS are representative enough for learning both spatial and temporal inpainting jointly.
    Training data choice (Sec. 4).
  • domain assumption GroundingDINO and SAM2 produce accurate text-mask pairs for training.
    Segmentation-based mask generation (Sec. 3.4 and Fig. 6).
  • domain assumption The evaluation metrics (BP, TA, TC, PSNR, SSIM, LPIPS, FVD) reliably capture the intended video inpainting quality.
    Used for all quantitative claims (Sec. 4.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of UniPaint: Unified Space-time Video Inpainting via Mixture-of-Experts." pith.science (2026). https://pith.science/paper/AA6WOPRX

@misc{pith2026241206340,
  author       = {Pith},
  title        = {Pith review of: UniPaint: Unified Space-time Video Inpainting via Mixture-of-Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AA6WOPRX}},
  note         = {Machine review of arXiv:2412.06340}
}
abstract

In this paper, we present UniPaint, a unified generative space-time video inpainting framework that enables spatial-temporal inpainting and interpolation. Different from existing methods that treat video inpainting and video interpolation as two distinct tasks, we leverage a unified inpainting framework to tackle them and observe that these two tasks can mutually enhance synthesis performance. Specifically, we first introduce a plug-and-play space-time video inpainting adapter, which can be employed in various personalized models. The key insight is to propose a Mixture of Experts (MoE) attention to cover various tasks. Then, we design a spatial-temporal masking strategy during the training stage to mutually enhance each other and improve performance. UniPaint produces high-quality and aesthetically pleasing results, achieving the best quantitative results across various tasks and scale setups. The code and checkpoints are available at $\href{https://github.com/mmmmm-w/UniPaint}{this \ repository}$.

Figures

Figures reproduced from arXiv: 2412.06340 by the authors.

Figure 1
Figure 1. The visual results of unified space-time video inpainting. We introduce a space-time video inpainting method that is versatile across a spectrum of tasks. Displayed frames are uniformly selected from videos of different space-time inpainting scenarios. For inpainting and outpainting, the first row in the figure contains the source videos and the target regions, while the bottom row shows the results. For interpolati… view at source ↗
Figure 2
Figure 2. Previous attention v.s. MoE attention. We incorporate our Space-time Inpainting Adapter with MoE attention, providing better adaptability and textual alignment. adapter to facilitate powerful generative ability. Then, the Mixture of Experts (MoE) attention and spatial￾temporal masking strategy are designed to handle task diversity and enhance performance. • We conduct extensive quantitative and qualitative eval￾uati… view at source ↗
Figure 3
Figure 3. Overview of our method. As shown in the figure, UniPaint outputs an inpainted video given the mask and masked video input. The noise, masked frames, and masks are concatenated as the input to the Space-Time Inpainting Adapter. The feature extracted by the adapter is added to the pretrained video model with a custom scale. The mask is also input to the gating network of the MoE Motion Module. ensure frame consistency… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Inpainting on videos of different cases. We employ our method on various scenarios of inpainting. Our method can be applied to both spatial and temporal inpainting cases with arbitrary mask shapes. The caption in the middle represents the inpainting type and prompt gui…
Figure 5
Figure 5. Figure 5: Masks for different space-time inpainitng scenar￾ios. The mask region is shown as white while conserved region is black. In inpainting tasks, the masks are continuous regions cov￾ering a part of each frame. In outpainting tasks, the masks cover the desired expansion re…
Figure 6
Figure 6. Figure 6: Segmentation-based mask generation. We enhance the training with object-aware masks. We first detect the bound￾ing box and corresponding phrase for objects in the first frame with GroundingDINO [38], then input the box and source video to SAM2 [52]. SAM2 propagates thr…
Figure 7
Figure 7. Figure 7: Comparison with previsou methods. We compare our method against several approaches, including VideoComposer [66], AVID [82] and CoCoCo [85]. The results of AVID directly comes from its puclication, other methods are evaluated using their default hyper-parameters as spe…
Figure 9
Figure 9. Figure 9: Analysis of MoE attention. The source video is shown on the left. We show the results of each individual expert FFN in comparison with the result of MoE. Some experts are more spe￾cialized in outpainting tasks.The Mask-Gated MoE adpaptively synthesizes the outputs of e…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Follow-Your-Creation: Empowering 4D Creation through Video Inpainting

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Follow-Your-Creation fine-tunes the Wan2.1 video inpainting model on composite point-cloud and editing masks so a single monocular video can be converted into editable 4D video with new camera motion.

  2. SkipVAR: Accelerating Visual Autoregressive Modeling via Adaptive Frequency-Aware Skipping

    cs.CV 2025-06 conditional novelty 4.0 of 10

    SkipVAR selects, per sample, between step skipping and unconditional branch replacement using handcrafted frequency features and a trained logistic regression, to accelerate visual autoregressive generation.

Reference graph

Works this paper leans on

84 extracted references · 31 canonical work pages · cited by 2 Pith papers

  1. [1]

    Paint by word

    Alex Andonian, Sabrina Osmany, Audrey Cui, YeonHwan Park, Ali Jahanian, Antonio Torralba, and David Bau. Paint by word. arXiv preprint arXiv:2103.10951, 2021. 2, 3

    arXiv 2021

  2. [2]

    Blended latent diffusion

    Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. arXiv preprint arXiv:2206.02779, 2022. 3

    arXiv 2022

  3. [3]

    Blended diffusion for text-driven editing of natural images

    Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18208–18218, 2022. 3

    2022

  4. [5]

    Frozen in time: A joint video and image encoder for end-to-end retrieval

    Max Bain, Arsha Nagrani, G ¨ul Varol, and Andrew Zisser- man. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 1728–1738,

  5. [6]

    A database and evaluation methodology for optical flow

    Simon Baker, Daniel Scharstein, James P Lewis, Stefan Roth, Michael J Black, and Richard Szeliski. A database and evaluation methodology for optical flow. International journal of computer vision, 92:1–31, 2011. 3

    2011

  6. [7]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 2

    arXiv 2023

  7. [8]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators

  8. [9]

    A naturalistic open source movie for opti- cal flow evaluation

    Daniel J Butler, Jonas Wulff, Garrett B Stanley, and Michael J Black. A naturalistic open source movie for opti- cal flow evaluation. In Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part VI 12 , pages 611–

    2012

Show all 84 references
  1. [10]

    Pix2video: Video editing using image diffusion

    Duygu Ceylan, Chun-Hao P Huang, and Niloy J Mitra. Pix2video: Video editing using image diffusion. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 23206–23217, 2023. 3

    2023

  2. [11]

    Videocrafter2: Overcoming data limitations for high-quality video diffusion models

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. arXiv preprint arXiv:2401.09047, 2024. 2

    2024 arXiv

  3. [12]

    Follow-your-canvas: Higher-resolution video outpainting with extensive content generation.arXiv preprint arXiv:2409.01055, 2024

    Qihua Chen, Yue Ma, Hongfa Wang, Junkun Yuan, Wenzhe Zhao, Qi Tian, Hongmei Wang, Shaobo Min, Qifeng Chen, and Wei Liu. Follow-your-canvas: Higher-resolution video outpainting with extensive content generation.arXiv preprint arXiv:2409.01055, 2024. 2

    2024 arXiv

  4. [13]

    Control-a-video: Controllable text-to-video generation with diffusion models,

    Weifeng Chen, Yatai Ji, Jie Wu, Hefeng Wu, Pan Xie, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video generation with diffusion models,

  5. [14]

    Diffedit: Diffusion-based seman- tic image editing with mask guidance

    Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion-based seman- tic image editing with mask guidance. arXiv preprint arXiv:2210.11427, 2022. 3

    2022 arXiv

  6. [15]

    Ldmvfi: Video frame interpolation with latent diffusion models

    Duolikun Danier, Fan Zhang, and David Bull. Ldmvfi: Video frame interpolation with latent diffusion models. Pro- ceedings of the AAAI Conference on Artificial Intelligence , 38(2):1472–1480, Mar. 2024. 3, 4, 6, 7, 10

    2024

  7. [16]

    Cogview2: Faster and better text-to-image generation via hierarchical transformers

    Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. Advances in Neural Information Processing Systems, 35:16890–16902, 2022. 3

    2022

  8. [17]

    Video frame interpolation: A comprehensive survey

    Jiong Dong, Kaoru Ota, and Mianxiong Dong. Video frame interpolation: A comprehensive survey. ACM Trans. Multi- media Comput. Commun. Appl., 19(2s), May 2023. 3

    2023

  9. [18]

    Structure and content-guided video synthesis with diffusion models

    Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7346–7356, 2023. 2

    2023

  10. [19]

    Hierar- chical masked 3d diffusion model for video outpainting

    Fanda Fan, Chaoxu Guo, Litong Gong, Biao Wang, Tiezheng Ge, Yuning Jiang, Chunjie Luo, and Jianfeng Zhan. Hierar- chical masked 3d diffusion model for video outpainting. In Proceedings of the 31st ACM International Conference on Multimedia, pages 7890–7900, 2023. 2

    2023

  11. [20]

    Dit4edit: Dif- fusion transformer for image editing

    Kunyu Feng, Yue Ma, Bingyuan Wang, Chenyang Qi, Haozhe Chen, Qifeng Chen, and Zeyu Wang. Dit4edit: Dif- fusion transformer for image editing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 2969–2977, 2025. 2

    2025

  12. [21]

    Gen-2: The next step forward for generative ai

    Gen-2. Gen-2: The next step forward for generative ai. https://research.runwayml.com/gen2/ , 2023. 2

    2023

  13. [22]

    Tokenflow: Consistent diffusion features for consistent video editing

    Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arXiv:2307.10373, 2023. 3

    2023 arXiv

  14. [23]

    Sparsectrl: Adding sparse controls to text-to-video diffusion models, 2023

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models, 2023. 2

    2023

  15. [24]

    Animatediff: Animate your personalized text-to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023. 2, 4, 6

    2023 arXiv

  16. [25]

    Imagen video: High definition video generation with diffusion mod- els

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion mod- els. arXiv preprint arXiv:2210.02303, 2022. 2

    arXiv

  17. [26]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 2, 4

    2020

  18. [27]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. 4

    2022 arXiv

  19. [28]

    Real-time intermediate flow estimation for video frame interpolation

    Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. Real-time intermediate flow estimation for video frame interpolation. In European Conference on Com- puter Vision, pages 624–642. Springer, 2022. 10

    2022

  20. [29]

    Video inter- polation with diffusion models, 2024

    Siddhant Jain, Daniel Watson, Eric Tabellion, Aleksander Hoły´nski, Ben Poole, and Janne Kontkanen. Video inter- polation with diffusion models, 2024. 3, 4, 6, 7

    2024

  21. [30]

    Videobooth: Diffusion-based video generation with image prompts

    Yuming Jiang, Tianxing Wu, Shuai Yang, Chenyang Si, Dahua Lin, Yu Qiao, Chen Change Loy, and Ziwei Liu. Videobooth: Diffusion-based video generation with image prompts. arXiv preprint arXiv:2312.00777, 2023. 2

    2023 arXiv

  22. [31]

    Brushnet: A plug-and-play image inpainting model with decomposed dual-branch diffusion, 2024

    Xuan Ju, Xian Liu, Xintao Wang, Yuxuan Bian, Ying Shan, and Qiang Xu. Brushnet: A plug-and-play image inpainting model with decomposed dual-branch diffusion, 2024. 2, 3, 4, 6

    2024

  23. [32]

    Text2video-zero: Text-to- image diffusion models are zero-shot video generators.arXiv preprint arXiv:2303.13439, 2023

    Levon Khachatryan, Andranik Movsisyan, Vahram Tade- vosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to- image diffusion models are zero-shot video generators.arXiv preprint arXiv:2303.13439, 2023. 2

    2023 arXiv

  24. [33]

    Deep video inpainting

    Dahun Kim, Sanghyun Woo, Joon-Young Lee, and In So Kweon. Deep video inpainting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5792–5801, 2019. 10

    2019

  25. [34]

    Auto-encoding varia- tional bayes

    Diederik P Kingma and Max Welling. Auto-encoding varia- tional bayes. arXiv preprint arXiv:1312.6114, 2013. 4

    2013 arXiv

  26. [35]

    Videopoet: A large language model for zero-shot video generation

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, Jos ´e Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14125, 2023. 2

    2023 arXiv

  27. [36]

    Kuaishou. Kling. 2024. 2

    2024

  28. [37]

    Pseudo numerical methods for diffusion models on manifolds.arXiv preprint arXiv:2202.09778, 2022

    Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds.arXiv preprint arXiv:2202.09778, 2022. 4

    2022 arXiv

  29. [38]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 6

    2023 arXiv

  30. [39]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 10

    2017 arXiv

  31. [40]

    Magic- stick: Controllable video editing via control handle transfor- mations

    Yue Ma, Xiaodong Cun, Yingqing He, Chenyang Qi, Xin- tao Wang, Ying Shan, Xiu Li, and Qifeng Chen. Magic- stick: Controllable video editing via control handle transfor- mations. arXiv preprint arXiv:2312.03047, 2023. 2

    2023 arXiv

  32. [41]

    Follow-your-creation: Empowering 4d creation through video inpainting

    Yue Ma, Kunyu Feng, Xinhua Zhang, Hongyu Liu, David Junhao Zhang, Jinbo Xing, Yinhan Zhang, Ayden Yang, Zeyu Wang, and Qifeng Chen. Follow-your-creation: Empowering 4d creation through video inpainting. arXiv preprint arXiv:2506.04590, 2025. 2

    2025 arXiv

  33. [42]

    Follow your pose: Pose- guided text-to-video generation using pose-free videos

    Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Siran Chen, Xiu Li, and Qifeng Chen. Follow your pose: Pose- guided text-to-video generation using pose-free videos. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 38, pages 4117–4125, 2024. 2

    2024

  34. [43]

    Follow-your-click: Open-domain regional image animation via motion prompts

    Yue Ma, Yingqing He, Hongfa Wang, Andong Wang, Leqi Shen, Chenyang Qi, Jixuan Ying, Chengfei Cai, Zhifeng Li, Heung-Yeung Shum, et al. Follow-your-click: Open-domain regional image animation via motion prompts. In Proceed- ings of the AAAI Conference on Artificial Intelligence...

    2025

  35. [44]

    Follow-your-emoji: Fine-controllable and expressive freestyle portrait animation

    Yue Ma, Hongyu Liu, Hongfa Wang, Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung-Yeung Shum, Wei Liu, et al. Follow-your-emoji: Fine-controllable and expressive freestyle portrait animation. In SIGGRAPH Asia 2024 Conference Papers, pages 1–12, 2024. 2

    2024

  36. [45]

    Follow-your-motion: Video motion transfer via efficient spatial-temporal decoupled finetuning

    Yue Ma, Yulong Liu, Qiyuan Zhu, Ayden Yang, Kunyu Feng, Xinhua Zhang, Zhifeng Li, Sirui Han, Chenyang Qi, and Qifeng Chen. Follow-your-motion: Video motion transfer via efficient spatial-temporal decoupled finetuning. arXiv preprint arXiv:2506.05207, 2025. 2

    2025

  37. [46]

    Visual knowledge graph for human action rea- soning in videos

    Yue Ma, Yali Wang, Yue Wu, Ziyu Lyu, Siran Chen, Xiu Li, and Yu Qiao. Visual knowledge graph for human action rea- soning in videos. In Proceedings of the 30th ACM Interna- tional Conference on Multimedia , pages 4132–4141, 2022. 2

    2022

  38. [47]

    Simvtp: Sim- ple video text pre-training with masked autoencoders

    Yue Ma, Tianyu Yang, Yin Shan, and Xiu Li. Simvtp: Sim- ple video text pre-training with masked autoencoders. arXiv preprint arXiv:2212.03490, 2022. 2

    2022 arXiv

  39. [48]

    Magic-me: Identity-specific video customized diffu- sion, 2024

    Ze Ma, Daquan Zhou, Chun-Hsiao Yeh, Xue-She Wang, Xi- uyu Li, Huanrui Yang, Zhen Dong, Kurt Keutzer, and Jiashi Feng. Magic-me: Identity-specific video customized diffu- sion, 2024. 2

    2024

  40. [49]

    Pika labs

    Pika Labs. Pika labs. https://www.pika.art/, 2023. 2

    2023

  41. [50]

    The 2017 davis challenge on video object segmentation.arXiv preprint arXiv:1704.00675, 2017

    Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Ar- bel´aez, Alex Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation.arXiv preprint arXiv:1704.00675, 2017. 3, 6

    2017 arXiv

  42. [51]

    Fatezero: Fus- ing attentions for zero-shot text-based video editing

    Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fus- ing attentions for zero-shot text-based video editing. arXiv preprint arXiv:2303.09535, 2023. 3

    2023 arXiv

  43. [52]

    Sam 2: Segment anything in images and videos,

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junt- ing Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Doll´ar, and Christoph Feic...

  44. [53]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 3, 6

    2022

  45. [54]

    U- net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Pa...

    2015

  46. [55]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information...

    2022

  47. [56]

    Edit-a-video: Single video editing with object-aware consistency

    Chaehun Shin, Heeseung Kim, Che Hyun Lee, Sang-gil Lee, and Sungroh Yoon. Edit-a-video: Single video editing with object-aware consistency. arXiv preprint arXiv:2303.07945,

    arXiv

  48. [57]

    Denois- ing diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois- ing diffusion implicit models. In 9th International Con- ference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. 2, 3, 4, 6

    2021

  49. [58]

    Diffusion guided do- main adaptation of image generators

    Kunpeng Song, Ligong Han, Bingchen Liu, Dimitris Metaxas, and Ahmed Elgammal. Diffusion guided do- main adaptation of image generators. arXiv preprint arXiv:2212.04473, 2022. 2

    2022 arXiv

  50. [59]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    K Soomro. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 ,

    arXiv

  51. [60]

    To- wards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. 7, 8

    2018 arXiv

  52. [61]

    Mcvd-masked conditional video diffusion for prediction, generation, and interpolation

    Vikram V oleti, Alexia Jolicoeur-Martineau, and Chris Pal. Mcvd-masked conditional video diffusion for prediction, generation, and interpolation. Advances in neural informa- tion processing systems, 35:23371–23385, 2022. 3

    2022

  53. [62]

    Cove: Unleashing the diffusion feature correspondence for consistent video editing

    Jiangshan Wang, Yue Ma, Jiayi Guo, Yicheng Xiao, Gao Huang, and Xiu Li. Cove: Unleashing the diffusion feature correspondence for consistent video editing. arXiv preprint arXiv:2406.08850, 2024. 2

    2024 arXiv

  54. [63]

    Modelscope text-to-video technical report, 2023

    Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report, 2023. 2

    2023

  55. [64]

    Imagen editor and editbench: Advancing and evaluating text-guided im- age inpainting

    Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont- Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J Fleet, Radu Soricut, et al. Imagen editor and editbench: Advancing and evaluating text-guided im- age inpainting. In Proceedings of the IEEE/CVF C...

    2023

  56. [65]

    Zero-shot video editing using off-the-shelf image diffusion models

    Wen Wang, Kangyang Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang, and Chunhua Shen. Zero-shot video editing using off-the-shelf image diffusion models. arXiv preprint arXiv:2303.17599, 2023. 3

    2023 arXiv

  57. [66]

    Videocomposer: Compositional video synthesis with motion controllability

    Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. arXiv preprint arXiv:2306.02018, 2023. 2, 3, 4, 6, 7

    2023 arXiv

  58. [67]

    Image quality assessment: from error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Si- moncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004. 7, 8

    2004

  59. [68]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference...

    2023

  60. [69]

    Smartbrush: Text and shape guided object inpainting with diffusion model

    Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 22428–22437, 2023. 2

    2023

  61. [70]

    Shaoan Xie, Yang Zhao, Zhisheng Xiao, Kelvin C. K. Chan, Yandong Li, Yanwu Xu, Kun Zhang, and Tingbo Hou. Dreaminpainter: Text-guided subject-driven image inpaint- ing with diffusion models, 2023. 2

    2023

  62. [71]

    Dynamicrafter: Animating open-domain images with video diffusion priors

    Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Xin- tao Wang, Tien-Tsin Wong, and Ying Shan. Dynamicrafter: Animating open-domain images with video diffusion priors. arXiv preprint arXiv:2310.12190, 2023. 2

    2023 arXiv

  63. [72]

    Youtube-vos: A large-scale video object segmentation benchmark

    Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. Youtube-vos: A large-scale video object segmentation benchmark. arXiv preprint arXiv:1809.03327, 2018. 6, 10

    2018 arXiv

  64. [73]

    Deep flow-guided video inpainting

    Rui Xu, Xiaoxiao Li, Bolei Zhou, and Chen Change Loy. Deep flow-guided video inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3723–3732, 2019. 10

    2019

  65. [74]

    Video enhancement with task-oriented flow

    Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman. Video enhancement with task-oriented flow. International Journal of Computer Vision, 127:1106– 1125, 2019. 3

    2019

  66. [75]

    Eedit: Rethinking the spatial and temporal redundancy for efficient image editing

    Zexuan Yan, Yue Ma, Chang Zou, Wenteng Chen, Qifeng Chen, and Linfeng Zhang. Eedit: Rethinking the spatial and temporal redundancy for efficient image editing. arXiv preprint arXiv:2503.10270, 2025. 2

    2025

  67. [76]

    Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory, 2023

    Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory, 2023. 2

    2023

  68. [77]

    Learning joint spatial-temporal transformations for video inpainting

    Yanhong Zeng, Jianlong Fu, and Hongyang Chao. Learning joint spatial-temporal transformations for video inpainting. In Computer Vision–ECCV 2020: 16th European Confer- ence, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16, pages 528–543. Springer, 2020. 10

    2020

  69. [78]

    Show-1: Marrying pixel and latent diffusion models for text-to-video generation, 2023

    David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation, 2023. 2

    2023

  70. [79]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 2, 4

    2023

  71. [80]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 586–595, 2018. 7, 8

    2018

  72. [81]

    Magiccolor: Multi-instance sketch colorization

    Yinhan Zhang, Yue Ma, Bingyuan Wang, Qifeng Chen, and Zeyu Wang. Magiccolor: Multi-instance sketch colorization. arXiv preprint arXiv:2503.16948, 2025. 2

    2025 arXiv

  73. [82]

    Avid: Any-length video inpainting with diffusion model, 2024

    Zhixing Zhang, Bichen Wu, Xiaoyan Wang, Yaqiao Luo, Luxin Zhang, Yinan Zhao, Peter Vajda, Dimitris Metaxas, and Licheng Yu. Avid: Any-length video inpainting with diffusion model, 2024. 2, 3, 4, 7, 8, 9, 10

    2024

  74. [83]

    Instantswap: Fast customized concept swapping across sharp shape differ- ences

    Chenyang Zhu, Kai Li, Yue Ma, Longxiang Tang, Chengyu Fang, Chubin Chen, Qifeng Chen, and Xiu Li. Instantswap: Fast customized concept swapping across sharp shape differ- ences. arXiv preprint arXiv:2412.01197, 2024. 2

    2024 arXiv

  75. [84]

    A task is worth one word: Learning with task prompts for high-quality versatile image inpainting

    Junhao Zhuang, Yanhong Zeng, Wenran Liu, Chun Yuan, and Kai Chen. A task is worth one word: Learning with task prompts for high-quality versatile image inpainting. ECCV,

  76. [85]

    Cococo: Improving text-guided video inpainting for better consistency, controllability and compatibility, 2024

    Bojia Zi, Shihao Zhao, Xianbiao Qi, Jianan Wang, Yukai Shi, Qianyu Chen, Bin Liang, Kam-Fai Wong, and Lei Zhang. Cococo: Improving text-guided video inpainting for better consistency, controllability and compatibility, 2024. 2, 3, 4, 6, 7, 8, 9, 10

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.