Pith. sign in

REVIEW 4 major objections 6 minor 11 cited by

Follow-Your-Creation: Empowering 4D Creation through Video Inpainting

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that 4D video creation can be reduced to video inpainting, with a single tuned inpainting model producing editable, multi-view-consistent video from one monocular clip.

desk verdict A clever and well-abalated framing of 4D creation as video inpainting, but the evidence overreaches on 4D consistency and the current artifacts are missing. read the letter →

arxiv 2506.04590 v1 pith:YHRGUX6F submitted 2025-06-05 cs.CV

Yue Ma , Kunyu Feng , Xinhua Zhang , Hongyu Liu , David Junhao Zhang , Jinbo Xing , Yinhan Zhang , Ayden Yang
show 2 more authors
Zeyu Wang Qifeng Chen
This is my paper
classification cs.CV
keywords 4Dvideogenerationinpaintingcameratrajectorycontroldynamicpointcloudcompositemaskself-iterativetuningmulti-viewconsistencyprompt-basedediting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that 4D video creation—producing new views of a dynamic scene from a single monocular video under a user-specified camera trajectory—can be reformulated as a video inpainting problem. The idea is to let a powerful pretrained video inpainting model fill in the holes left when a depth-based dynamic point cloud is reprojected along the new trajectory, and to reuse the same machinery for user edits by adding editing masks. If this works, a single monocular clip plus a camera path yields multi-view-consistent 4D video, and the same model can propagate a first-frame edit or a text prompt through the scene. The paper reports that this beats existing camera-retargeting and 4D generation methods on frame quality, camera pose accuracy, and cross-view consistency metrics.

What carries the argument

The central mechanism is composite masked inpainting training, built on double reprojection. Double reprojection takes visibility masks from rendering a dynamic point cloud along a new trajectory and projects them back to the original camera pose, producing paired corrupted-video/mask/clean-video triples that keep the original frames as ground truth. Alongside point-cloud masks, editing masks and their union form a composite mask set that is randomly sampled during training. Two supporting mechanisms carry the argument: self-iterative tuning, which grows the camera angle stage by stage using the model's own previous outputs as new training data, and temporal-packing inference, which packs tokens of previously generated overlapping views into the current inpainting input so the base model's global self-attention enforces multi-view consistency.

What would settle it

Render a generated 4D video from a fresh camera viewpoint and compare its geometry—for example, its depth maps or reconstructed point cloud—against ground-truth range data from a synchronized multi-camera capture; if the gains are only in PSNR, LPIPS, or CLIP while the 3D structure of the inpainted regions deviates from the measured scene, the central claim of geometric completion is falsified.

Watch

Extended reading notes

Core claim

Reformulating 4D video creation as video inpainting, rather than as a separate generation task, lets a pretrained video inpainting foundation model (Wan2.1) be adapted with LoRA to both generate and edit 4D content. The authors construct training pairs by estimating per-frame depth with DepthCrafter, unprojecting frames into a dynamic point cloud, rendering the point cloud along a target trajectory, and double-reprojecting the result back to the original viewpoint so that the input video itself is ground truth. The composite masks combine point-cloud visibility masks with editing masks (first frame masked to '0' so it acts as guidance), and during training one mask type is sampled per instance. A self-iterative tuning loop starts with small camera angles, uses the model's own output as training data for larger angles, and thereby extends the range of camera motion to over 40 degrees. At inference, a temporal-packing module concatenates tokens from previously generated overlapping views with the current hole video so that global self-attention keeps multi-view content coherent.

Load-bearing premise

The pipeline assumes that the per-frame monocular depth maps produced by DepthCrafter are temporally consistent and geometrically accurate enough that the double-reprojected masks align with the true scene structure; if depth drifts or jumps between frames, the inpainter is trained to paint plausible pixels rather than the correct unseen geometry.

Editorial extensions

If this is right

  • A single monocular video plus a specified camera trajectory becomes a multi-view-consistent 4D video, with the same model supporting prompt-based content editing.
  • Composite mask training preserves the base inpainting model's original performance while adding 4D generation and editing capability, instead of requiring large-scale dataset retraining.
  • Self-iterative tuning extends large-angle camera motion beyond what a vanilla inpainting model handles, so cinematic shots with more than 40 degrees of rotation become feasible from one clip.
  • Temporal-packing inference improves consistency across overlapping generated views without adding new attention layers or manual selection of consistent regions.
  • On Kubric-4D and VBench-style evaluations, the method reports better frame similarity, camera rotation and translation error, and cross-view CLIP and FVD scores than the compared baselines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if per-frame monocular depth is the bottleneck, then feeding the same pipeline with a more temporally consistent depth estimator or multi-view stereo should push quality further, since the masks and training targets are entirely derived from that depth.
  • Beyond the paper: the temporal-packing idea could be applied during training as well as inference, teaching the model to maintain consistency across arbitrary pairs of trajectories rather than only sequentially generated overlapping views.
  • Beyond the paper: because the base model is text-conditioned, explicit text prompts during inpainting could allow semantic control over what fills the holes, a capability the paper demonstrates for editing but does not systematically explore for novel-view completion.
  • Beyond the paper: the reported metrics measure appearance and camera pose, not 3D geometry, so a geometric evaluation of the generated 4D structure would reveal whether the high scores correspond to correct scene structure or plausible hallucinated content.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. Follow-Your-Creation proposes to turn monocular video into 4D video by reformulating the task as video inpainting. It estimates per-frame depth with DepthCrafter, builds dynamic point clouds, renders target camera trajectories, and uses a double-reprojection strategy to obtain point-cloud masks. These masks are combined with editing masks into a composite-mask training set, and the Wan2.1 video inpainting model is fine-tuned with LoRA on this data. A self-iterative tuning scheme progressively increases viewing angles by reusing the model's own outputs as next-stage training data, and a temporal-packing inference module concatenates tokens from previously generated views to improve multi-view consistency. The method is evaluated on Kubric-4D, VBench, and custom metrics against GCD, Trajectory-Attention, ReCamMaster, and TrajectoryCrafter, with additional ablations in Table 4.

Significance. The paper addresses an important and timely problem, and the high-level idea of leveraging a strong video inpainting prior instead of training a new camera-conditioned generator is attractive. The composite-mask construction, self-iterative angle curriculum, and temporal-packing inference are concrete, and Table 4 shows that each proposed component improves the reported metrics. Training cost is modest (single A800 GPU, about two hours), and the Kubric-4D comparison provides an external test with ground truth. However, the central '4D consistency' claim is only as strong as the underlying monocular-depth geometry, and the manuscript reports no geometric validation of the depth or of the derived masks. In addition, the comparison against prior work is confounded by the choice of base model. If the missing geometric checks and controlled comparisons confirm the current numbers, this would be a valuable contribution; at present the central claim is not yet fully supported.

major comments (4)
  1. [Sec. 3.1-3.2, Eq. (1), Sec. 4.2, Tabs. 2-3] The geometric backbone is per-frame DepthCrafter depth combined with 'empirically configured camera intrinsics' (Sec. 4.1). The paper never validates the depth accuracy or temporal consistency of these estimates, nor the correctness of the double-reprojection masks M'' from Sec. 3.2. Because the training objective is appearance-based and the masks determine what the model is allowed to generate at inference, systematic depth error or temporal jitter can make the model fill arbitrary holes rather than true disoccluded content. The reported metrics (PSNR, SSIM, LPIPS, RotErr, TransErr, FVD-V, CLIP-V) do not measure geometric correctness. Please add depth error against ground truth (e.g., on Kubric-4D or a monocular-depth benchmark), mask IoU against true disocclusion masks, and a point-cloud or geometry consistency measure such as Chamfer distance on the synthesized novel views.
  2. [Sec. 4.1-4.2, Tabs. 2-3] The headline superiority claim is confounded by base-model choice. Ours fine-tunes Wan2.1, a large and strong video inpainting foundation model, while the main baselines (ReCamMaster, TrajectoryCrafter) are camera-conditioned generators built on different text-to-video or image-to-video models, and GCD and Trajectory-Attention use different conditioning mechanisms. A fair test requires either applying Wan2.1 inpainting to the same point-cloud/rendering pipeline without the proposed LoRA tuning, or tuning the proposed components on the same base model as a baseline. Without such a control, the gains in Tabs. 2-3 could be attributed to the base prior rather than to the proposed method's contributions.
  3. [Sec. 3.3, Eqs. (3)-(6), Tab. 4] The self-iterative tuning generates next-stage training data from the model's own outputs. This procedure can amplify systematic depth or inpainting biases, and the paper provides no per-iteration diagnostics or convergence/stability analysis. Additionally, L_cycle in Eq. (6) is only described as 'spatial-temporal consistency MSE losses [28]', where reference [28] is The Elements of Statistical Learning, which is not the source of this loss. Please define L_cycle precisely, cite the actual method, and report metrics after each tuning iteration, or otherwise justify why self-training collapse is not a concern for the reported improvements.
  4. [Sec. 4.2, Fig. 5] The prompt-based editing capability is a central claimed advantage, but it is only demonstrated qualitatively in Fig. 5. No quantitative editing metric or user study is reported in the main text; both the user study and the limitations are deferred to an appendix that is not part of this manuscript. Since the abstract explicitly claims support for prompt-based content editing and 'significantly outperforming' in versatility, the paper needs either a quantitative editing evaluation (e.g., CLIP similarity to the edited prompt, identity preservation, or a user study) or a clearly specified editing benchmark.
minor comments (6)
  1. [Sec. 3.4, Fig. 3] There are several typos in this section, including 'trajactory' and 'Recaputre'; please proofread the text and references.
  2. [Sec. 3.2-3.3, Eqs. (3) and (6)] The loss notation is inconsistent: L in Eq. (3) is used before it is defined, the flow-matching loss in Sec. 3.2 is described only verbally, and Eq. (6) introduces L_cycle. Please unify the loss definitions and notation.
  3. [References] References [61] and [94] appear to describe the same TrajectoryCrafter work with inconsistent author formatting; please merge them into a single citation.
  4. [Reference [28]] The citation for the spatial-temporal consistency MSE loss is incorrect; The Elements of Statistical Learning does not define this loss. Please replace it with the appropriate reference.
  5. [Tables 2-3] TrajectoryCrafter is cited as [94] in Table 2 and in the main text but as [61] in Table 3; please make the citation consistent.
  6. [Sec. 4.2, Fig. 5] For the editing comparison, the paper says an 'advanced video editing tool [42]' was used to edit the first frame for baselines, but no details are given about how the editing masks or edits were generated; please clarify the protocol so the comparison is reproducible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the pipeline's components are externally sourced or benchmarked, and no prediction reduces to a fitted input.

full rationale

Follow-Your-Creation is an empirical pipeline: per-frame depths from the external DepthCrafter (Eq. 1) are reprojected into double-reprojection masks (Sec. 3.2) following TrajectoryCrafter, and a Wan2.1 inpainting model is LoRA-fine-tuned with a flow-matching loss on the masked videos. The claimed 4D consistency is an empirical outcome measured against the external Kubric-4D benchmark (Tab. 3) and VBench (Tab. 1), and against external baselines (TrajectoryCrafter, ReCamMaster, GCD, Trajectory-Attention). No fitted parameter is renamed as a prediction: the only tuned quantities are LoRA weights and self-iterative schedule hyperparameters, and the reported numbers are frame, pose, and cross-view similarity scores, not derived from these fits by construction. The self-iterative tuning (Eqs. 4-6) does train on the model's own outputs, but that is a data-augmentation or self-training choice rather than a derivation that reduces to its inputs; the central performance claim is benchmarked externally. The only mild self-citation is adoption of the double-reprojection strategy from [94], co-authored by J. Xing, but this component is a published technique used as a tool, not an unverified uniqueness premise, and it does not force the reported results. The view-synchronization metrics FVD-V and CLIP-V are author-defined rather than standardized, but they are computed from generated outputs, not fitted, so this is a measurement-validity concern, not circularity. Absent any equation that reduces a prediction to its input by definition, the paper shows no significant circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on roughly four domain assumptions: monocular depth quality, transferability of Wan2.1's prior, cross-view identity via self-attention, and first-frame edit propagation. None is verified with a dedicated experiment beyond end-to-end ablations. The free parameters are mostly hand-set knobs; the empirically configured camera intrinsics are the most consequential because they silently control all geometry.

free parameters (5)
  • Camera intrinsics K = empirically configured (not measured)
    Sec 4.1 states 'empirically configured camera intrinsics'. The entire point cloud geometry, reprojection masks, and training targets depend on K; a wrong K shifts every rendered hole and can misalign the inpainted content.
  • Viewing angle thresholds for self-iterative tuning = initial angles below 30 degrees; target angles above 40 degrees
    Hand-chosen thresholds in Sec 3.3 that define the difficulty progression; the number of iterations and the exact angle schedule are not specified.
  • Inference LoRA weight and text-guidance scale = LoRA weight 0.7, guidance scale 6.5
    Sec 4.1. Manual inference knobs; no sensitivity analysis is reported.
  • Top-k frame selection for temporal packing = unspecified k
    Eq. 7 selects the K frames with the largest inpainting area from the previous view; the value of K is never given.
  • Training hyperparameters = LoRA rank 128, lr 1e-5, weight decay 0.1, 2000 steps, 81 frames at 512x512
    Sec 4.1. Standard but hand-set; no ablations over these values are reported.
assumptions (5)
  • standard math Pinhole camera model and perspective projection (Eq. 1, 2)
    Standard projective geometry, accepted without issue.
  • domain assumption DepthCrafter monocular depth estimates are temporally consistent and geometrically reliable across frames
    Sec 3.1. The entire dynamic point cloud, visibility masks, and training supervision rest on this; the paper provides no geometric accuracy check.
  • domain assumption Wan2.1's generative prior transfers to rendering-hole inpainting after LoRA fine-tuning
    Sec 1 and Sec 3. The paper states that Wan2.1's masks fall outside its training distribution but assumes LoRA adaptation suffices; support is empirical only.
  • domain assumption Self-attention over temporally packed tokens enforces cross-view identity without geometric correspondence
    Sec 3.4, Eq. 8. No new attention or alignment module is added; the claim that global self-attention is sufficient is untested in isolation.
  • domain assumption First-frame edit propagation (editing mask set to 0 on frame 0) yields temporally consistent edits
    Sec 3.2, following ReVideo [65]; adopted without validation against drift or identity preservation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Follow-Your-Creation: Empowering 4D Creation through Video Inpainting." pith.science (2026). https://pith.science/paper/YHRGUX6F

@misc{pith2026250604590,
  author       = {Pith},
  title        = {Pith review of: Follow-Your-Creation: Empowering 4D Creation through Video Inpainting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YHRGUX6F}},
  note         = {Machine review of arXiv:2506.04590}
}
read the original abstract

We introduce Follow-Your-Creation, a novel 4D video creation framework capable of both generating and editing 4D content from a single monocular video input. By leveraging a powerful video inpainting foundation model as a generative prior, we reformulate 4D video creation as a video inpainting task, enabling the model to fill in missing content caused by camera trajectory changes or user edits. To facilitate this, we generate composite masked inpainting video data to effectively fine-tune the model for 4D video generation. Given an input video and its associated camera trajectory, we first perform depth-based point cloud rendering to obtain invisibility masks that indicate the regions that should be completed. Simultaneously, editing masks are introduced to specify user-defined modifications, and these are combined with the invisibility masks to create a composite masks dataset. During training, we randomly sample different types of masks to construct diverse and challenging inpainting scenarios, enhancing the model's generalization and robustness in various 4D editing and generation tasks. To handle temporal consistency under large camera motion, we design a self-iterative tuning strategy that gradually increases the viewing angles during training, where the model is used to generate the next-stage training data after each fine-tuning iteration. Moreover, we introduce a temporal packaging module during inference to enhance generation quality. Our method effectively leverages the prior knowledge of the base model without degrading its original performance, enabling the generation of 4D videos with consistent multi-view coherence. In addition, our approach supports prompt-based content editing, demonstrating strong flexibility and significantly outperforming state-of-the-art methods in both quality and versatility.

Figures

Figures reproduced from arXiv: 2506.04590 by the authors.

Figure 1
Figure 1. Showcases of our Follow-Your-Creation. We reformulate 4D video creation as a video inpainting task. Given an input video, Follow-Your-Creation enables 4D video creation with various camera trajectories (bottom left on the input video’s first frame) and edited first frame (bottom right), while maintaining multi-view consistency. In addition, it supports flexible prompt-based content editing (e.g., adding a robot dog … view at source ↗
Figure 2
Figure 2. Overview of our method. We reformulate the 4D video creation as video inpainting task. Left: given a video, we first generate the composite masks from the dynamic point cloud and feed them into video inpainting model to unlock its 4D video creation capability. Right: To unlock the capability of generating 4D video with larger motion, we first generate videos with small motion, then feed them into the model to improv… view at source ↗
Figure 3
Figure 3. Motivation of temporal-packing inference [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Gallery of our proposed method. Our Follow-Your-Creation enables achieving flexible and high-quality 4D video creation using the given camera trajectory and the edited first frame (2nd row). Additionally, it also supports the 4D video creation using various prompts in …
Figure 5
Figure 5. Figure 5: Qualitative comparison results with the state-of-the-art methods. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Ablation study about composite mask (a) and self-iterative tuning (b). [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Ablation study of temporal-packing infer [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LiveEdit: Towards Real-Time Diffusion-Based Streaming Video Editing

    cs.CV 2026-06 unverdicted novelty 6.5 of 10

    LiveEdit distills a bidirectional video foundation model into a unidirectional streaming editor via three-stage training plus mask caching to reach 12.66 FPS with stable edits.

  2. EmoWorld: A Decoupled Affective Field for Controllable Emotional Video Generation

    cs.CV 2026-08 conditional novelty 6.0 of 10

    EmoWorld adds three training-free steering operators to a frozen video diffusion transformer that separately control atmosphere, affect-bearing cues, and temporal emotion transitions in generated videos.

  3. LiveLight: Real-time Streaming Video Relighting with Interactive Control

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A diffusion-based system performs real-time, interactive video relighting by injecting multi-plane light irradiance conditions and streaming latent chunks.

  4. FlexComposer: Unified Video Compositing from Images to Dynamic Footage with Flexible Trajectory Control

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A single video-diffusion framework composites both static images and dynamic footage along user-defined trajectories by transporting canonical foreground latents directly into the background latent sequence.

  5. GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation

    cs.RO 2026-07 conditional novelty 6.0 of 10

    Long-horizon action-faithful consistency, not short-term visual realism, dominates world-model reliability for robot policy evaluation; GigaWorld-1 implements that roadmap and gains 14.9% on evaluator-alignment metrics.

  6. UniGlyph: Unified Segmentation-Conditioned Diffusion for Precise Visual Text Synthesis

    cs.CV 2025-07 conditional novelty 6.0 of 10

    UniGlyph replaces pre-rendered glyph conditions with segmentation-derived masks in a ControlNet diffusion model, reporting gains on visual text rendering benchmarks.

  7. MagicPrompt: Ultra-Lightweight Prompt Tuning for Video Generation

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Soft prompts inserted into attention key/value streams plus a dual pixel/latent reward let video diffusion models be tuned with under 1% trainable parameters at competitive quality.

  8. Follow-Your-Instruction: A Comprehensive MLLM Agent for World Data Synthesis

    cs.CV 2025-08 conditional novelty 5.0 of 10

    An MLLM-driven pipeline that composes 3D scenes from assets, optimizes them with multi-view VLM feedback, and renders videos, yielding synthetic data that modestly improves several 2D, 3D, and 4D generative baselines.

  9. GR-Gaussian: Graph-Based Radiative Gaussian Splatting for Sparse-View CT Reconstruction

    eess.IV 2025-08 conditional novelty 5.0 of 10

    A graph-augmented 3D Gaussian Splatting method with denoised point cloud initialization improves sparse-view CT reconstruction by about 0.7 to 0.9 dB PSNR over R2-Gaussian.

  10. FADE: Adversarial Concept Erasure in Flow Models

    cs.CV 2025-07 reject novelty 4.0 of 10

    FADE combines adversarial training with trajectory preservation to erase concepts from diffusion models, reporting state-of-the-art erasure on Stable Diffusion benchmarks, but the evidence is incomplete and the theore...

  11. SkipVAR: Accelerating Visual Autoregressive Modeling via Adaptive Frequency-Aware Skipping

    cs.CV 2025-06 conditional novelty 4.0 of 10

    SkipVAR selects, per sample, between step skipping and unconditional branch replacement using handcrafted frequency features and a trained logistic regression, to accelerate visual autoregressive generation.

Reference graph

Works this paper leans on

105 extracted references · 23 canonical work pages · cited by 11 Pith papers

  1. [28]

    The elements of statistical learning, 2009

    Trevor Hastie, Robert Tibshirani, Jerome Friedman, et al. The elements of statistical learning, 2009

    2009

  2. [1]

    Gen-1.https://runwayml.com/research/gen-1, 2023

    2023

  3. [2]

    Klingai.https://klingai.com/cn/, 2024

    2024

  4. [3]

    Lindell, and Sergey Tulyakov

    Sherwin Bahmani, Ivan Skorokhodov, Guocheng Qian, Aliaksandr Siarohin, Willi Menapace, Andrea Tagliasacchi, David B. Lindell, and Sergey Tulyakov. Ac3d: Analyzing and improving 3d camera control in video diffusion transformers.arXiv preprint arXiv:2411.18673, 2024

    arXiv 2024

  5. [4]

    Vd3d: Taming large video diffusion transformers for 3d camera control

    Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siarohin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hsin-Ying Lee, Chaoyang Wang, Jiaxu Zou, Andrea Tagliasacchi, et al. Vd3d: Taming large video diffusion transformers for 3d camera control. InICLR, 2025

    2025

  6. [5]

    Recammaster: Camera-controlled generative rendering from a single video.ArXiv, abs/2503.11647, 2025

    Jianhong Bai, Menghan Xia, Xiao Fu, Xintao Wang, Lianrui Mu, Jinwen Cao, Zuozhu Liu, Haoji Hu, Xiang Bai, Pengfei Wan, and Di Zhang. Recammaster: Camera-controlled generative rendering from a single video.ArXiv, abs/2503.11647, 2025

    arXiv 2025

  7. [6]

    Syncammaster: Synchronizing multi-camera video generation from diverse viewpoints, 2024

    Jianhong Bai, Menghan Xia, Xintao Wang, Ziyang Yuan, Xiao Fu, Zuozhu Liu, Haoji Hu, Pengfei Wan, and Di Zhang. Syncammaster: Synchronizing multi-camera video generation from diverse viewpoints, 2024

    2024

  8. [7]

    Lumiere: A space-time diffusion model for video generation

    Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Guanghui Liu, Amit Raj, et al. Lumiere: A space-time diffusion model for video generation. InSIGGRAPH Asia 2024 Conference Papers, 2024

    2024

Show all 105 references
  1. [8]

    Videopainter: Any-length video inpainting and editing with plug-and-play context control

    Yuxuan Bian, Zhaoyang Zhang, Xuan Ju, Mingdeng Cao, Liangbin Xie, Ying Shan, and Qiang Xu. Videopainter: Any-length video inpainting and editing with plug-and-play context control. arXiv preprint arXiv:2503.05639, 2025

    2025 arXiv

  2. [9]

    Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    2023 arXiv

  3. [10]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024

    2024

  4. [11]

    Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing

    Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. InICCV, 2023

    2023

  5. [12]

    Pix2video: Video editing using image diffusion

    Duygu Ceylan, Chun-Hao P Huang, and Niloy J Mitra. Pix2video: Video editing using image diffusion. InICCV, 2023

    2023

  6. [13]

    Stablevideo: Text-driven consistency- aware diffusion video editing

    Wenhao Chai, Xun Guo, Gaoang Wang, and Yan Lu. Stablevideo: Text-driven consistency- aware diffusion video editing. InICCV, 2023

    2023

  7. [14]

    Magicpose: Realistic human poses and facial expressions retargeting with identity-aware diffusion

    Di Chang, Yichun Shi, Quankai Gao, Hongyi Xu, Jessica Fu, Guoxian Song, Qing Yan, Yizhe Zhu, Xiao Yang, and Mohammad Soleymani. Magicpose: Realistic human poses and facial expressions retargeting with identity-aware diffusion. InICML, 2024

    2024

  8. [15]

    VideoCrafter1: Open diffusion models for high-quality video generation.arXiv preprint arXiv:2310.19512, 2023

    Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. VideoCrafter1: Open diffusion models for high-quality video generation.arXiv preprint arXiv:2310.19512, 2023. 10

    arXiv

  9. [16]

    Follow-your-canvas: Higher-resolution video outpainting with extensive content generation.arXiv preprint arXiv:2409.01055, 2024

    Qihua Chen, Yue Ma, Hongfa Wang, Junkun Yuan, Wenzhe Zhao, Qi Tian, Hongmei Wang, Shaobo Min, Qifeng Chen, and Wei Liu. Follow-your-canvas: Higher-resolution video outpainting with extensive content generation.arXiv preprint arXiv:2409.01055, 2024

    2024 arXiv

  10. [17]

    M-bev: Masked bev perception for robust au- tonomous driving

    Siran Chen, Yue Ma, Yu Qiao, and Yali Wang. M-bev: Masked bev perception for robust au- tonomous driving. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1183–1191, 2024

    2024

  11. [18]

    Attentive snippet prompting for video retrieval.IEEE Transactions on Multimedia, 26:4348–4359, 2023

    Siran Chen, Qinglin Xu, Yue Ma, Yu Qiao, and Yali Wang. Attentive snippet prompting for video retrieval.IEEE Transactions on Multimedia, 26:4348–4359, 2023

    2023

  12. [19]

    Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions

    Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions. 2024

    2024

  13. [20]

    Lu- ciddreamer: Domain-free generation of 3d gaussian splatting scenes.arXiv preprint arXiv:2311.13384, 2023

    Jaeyoung Chung, Suyoung Lee, Hyeongjin Nam, Jaerin Lee, and Kyoung Mu Lee. Lu- ciddreamer: Domain-free generation of 3d gaussian splatting scenes.arXiv preprint arXiv:2311.13384, 2023

    2023 arXiv

  14. [21]

    Flatten: optical flow-guided attention for consistent text-to-video editing.arXiv preprint arXiv:2310.05922, 2023

    Yuren Cong, Mengmeng Xu, Christian Simon, Shoufa Chen, Jiawei Ren, Yanping Xie, Juan- Manuel Perez-Rua, Bodo Rosenhahn, Tao Xiang, and Sen He. Flatten: optical flow-guided attention for consistent text-to-video editing.arXiv preprint arXiv:2310.05922, 2023

    arXiv

  15. [22]

    Structure and content-guided video synthesis with diffusion models

    Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. InICCV, 2023

    2023

  16. [23]

    Dit4edit: Diffusion transformer for image editing

    Kunyu Feng, Yue Ma, Bingyuan Wang, Chenyang Qi, Haozhe Chen, Qifeng Chen, and Zeyu Wang. Dit4edit: Diffusion transformer for image editing. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 2969–2977, 2025

    2025

  17. [24]

    I2vcontrol-camera: Precise video camera control with adjustable motion strength

    Wanquan Feng, Jiawei Liu, Pengqi Tu, Tianhao Qi, Mingzhen Sun, Tianxiang Ma, Songtao Zhao, Siyu Zhou, and Qian He. I2vcontrol-camera: Precise video camera control with adjustable motion strength. InICLR, 2025

    2025

  18. [25]

    3dtrajmaster: Mastering 3d trajectory for multi-entity motion in video generation

    Xiao Fu, Xian Liu, Xintao Wang, Sida Peng, Menghan Xia, Xiaoyu Shi, Ziyang Yuan, Pengfei Wan, Di Zhang, and Dahua Lin. 3dtrajmaster: Mastering 3d trajectory for multi-entity motion in video generation. InICLR, 2025

    2025

  19. [26]

    Sparsectrl: Adding sparse controls to text-to-video diffusion models

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models. InEuropean Conference on Computer Vision, pages 330–348. Springer, 2024

    2024

  20. [27]

    Animatediff: Animate your personalized text-to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. InICLR, 2024

    2024

  21. [29]

    CameraCtrl: Enabling camera control for text-to-video generation

    Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. CameraCtrl: Enabling camera control for text-to-video generation. InICLR, 2025

    2025

  22. [30]

    Prompt-to-prompt image editing with cross attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. InICLR, 2023

    2023

  23. [31]

    Imagen video: High definition video generation with diffusion models.arXiv preprint arXiv:2210.02303, 2022

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models.arXiv preprint arXiv:2210.02303, 2022

    arXiv

  24. [32]

    Audio-visual controlled video diffusion with masked selective state spaces modeling for natural talking head generation, 2025

    Fa-Ting Hong, Zunnan Xu, Zixiang Zhou, Jun Zhou, Xiu Li, Qin Lin, Qinglin Lu, and Dan Xu. Audio-visual controlled video diffusion with masked selective state spaces modeling for natural talking head generation, 2025. 11

    2025

  25. [33]

    Generative camera dolly: Extreme monocular dynamic novel view synthesis.ArXiv, abs/2405.14868, 2024

    Basile Van Hoorick, Rundi Wu, Ege Ozguroglu, Kyle Sargent, Ruoshi Liu, Pavel Tokmakov, Achal Dave, Changxi Zheng, and Carl V ondrick. Generative camera dolly: Extreme monocular dynamic novel view synthesis.ArXiv, abs/2405.14868, 2024

    2024 arXiv

  26. [34]

    Training-free camera control for video generation.arXiv preprint arXiv:2406.10126, 2024

    Chen Hou, Guoqiang Wei, Yan Zeng, and Zhibo Chen. Training-free camera control for video generation.arXiv preprint arXiv:2406.10126, 2024

    2024 arXiv

  27. [35]

    Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen

    J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.ArXiv, abs/2106.09685, 2021

    2021 arXiv

  28. [36]

    Animate anyone: Consistent and controllable image-to-video synthesis for character animation.arXiv preprint arXiv:2311.17117, 2023

    Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and controllable image-to-video synthesis for character animation.arXiv preprint arXiv:2311.17117, 2023

    2023 arXiv

  29. [37]

    Depthcrafter: Generating consistent long depth sequences for open-world videos.ArXiv, abs/2409.02095, 2024

    Wenbo Hu, Xiangjun Gao, Xiaoyu Li, Sijie Zhao, Xiaodong Cun, Yong Zhang, Long Quan, and Ying Shan. Depthcrafter: Generating consistent long depth sequences for open-world videos.ArXiv, abs/2409.02095, 2024

    2024 arXiv

  30. [38]

    Depthcrafter: Generating consistent long depth sequences for open-world videos

    Wenbo Hu, Xiangjun Gao, Xiaoyu Li, Sijie Zhao, Xiaodong Cun, Yong Zhang, Long Quan, and Ying Shan. Depthcrafter: Generating consistent long depth sequences for open-world videos. InCVPR, 2025

    2025

  31. [39]

    Proposal-based video completion

    Yuan-Ting Hu, Heng Wang, Nicolas Ballas, Kristen Grauman, and Alexander G Schwing. Proposal-based video completion. InComputer Vision–ECCV 2020: 16th European Confer- ence, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVII 16, pages 38–54. Springer, 2020

    2020

  32. [40]

    Vbench: Comprehensive benchmark suite for video generative models.2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21807–21818, 2023

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Vbench: Comprehensive benchmark suite for video generative models.20...

    2024

  33. [41]

    Reangle-a-video: 4d video generation as video-to-video translation.ArXiv, abs/2503.09151, 2025

    Hyeonho Jeong, Suhyeon Lee, and Jong Chul Ye. Reangle-a-video: 4d video generation as video-to-video translation.ArXiv, abs/2503.09151, 2025

    2025 arXiv

  34. [42]

    Vace: All-in-one video creation and editing.arXiv preprint arXiv:2503.07598, 2025

    Zeyinzi Jiang, Zhen Han, Chaojie Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu. Vace: All-in-one video creation and editing.arXiv preprint arXiv:2503.07598, 2025

    2025 arXiv

  35. [43]

    Imagic: Text-based real image editing with diffusion models

    Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. InCVPR, 2023

    2023

  36. [44]

    Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    2024 arXiv

  37. [45]

    Collaborative video diffusion: Consistent multi-video generation with camera control

    Zhengfei Kuang, Shengqu Cai, Hao He, Yinghao Xu, Hongsheng Li, Leonidas J Guibas, and Gordon Wetzstein. Collaborative video diffusion: Consistent multi-video generation with camera control. InNeurIPS, 2024

    2024

  38. [46]

    Realcam- i2v: Real-world image-to-video generation with interactive complex camera control.arXiv preprint arXiv:2502.10059, 2025

    Teng Li, Guangcong Zheng, Rui Jiang, Tao Wu, Yehao Lu, Yining Lin, Xi Li, et al. Realcam- i2v: Real-world image-to-video generation with interactive complex camera control.arXiv preprint arXiv:2502.10059, 2025

    2025 arXiv

  39. [47]

    Motionclone: Training-free motion cloning for controllable video generation.arXiv preprint arXiv:2406.05338, 2024

    Pengyang Ling, Jiazi Bu, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Tong Wu, Huaian Chen, Jiaqi Wang, and Yi Jin. Motionclone: Training-free motion cloning for controllable video generation.arXiv preprint arXiv:2406.05338, 2024

    2024 arXiv

  40. [48]

    Human motionformer: Transferring human motions with vision transformers.arXiv preprint arXiv:2302.11306, 2023

    Hongyu Liu, Xintong Han, Chengbin Jin, Lihui Qian, Huawei Wei, Zhe Lin, Faqiang Wang, Haoye Dong, Yibing Song, Jia Xu, et al. Human motionformer: Transferring human motions with vision transformers.arXiv preprint arXiv:2302.11306, 2023. 12

    2023 arXiv

  41. [49]

    Rethinking image inpainting via a mutual encoder-decoder with feature equalizations

    Hongyu Liu, Bin Jiang, Yibing Song, Wei Huang, and Chao Yang. Rethinking image inpainting via a mutual encoder-decoder with feature equalizations. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 725–741. Sp...

    2020

  42. [50]

    Pd-gan: Probabilistic diverse gan for image inpainting

    Hongyu Liu, Ziyu Wan, Wei Huang, Yibing Song, Xintong Han, and Jing Liao. Pd-gan: Probabilistic diverse gan for image inpainting. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9371–9381, 2021

    2021

  43. [51]

    Deflocnet: Deep image editing via flexible low-level controls

    Hongyu Liu, Ziyu Wan, Wei Huang, Yibing Song, Xintong Han, Jing Liao, Bin Jiang, and Wei Liu. Deflocnet: Deep image editing via flexible low-level controls. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10765–10774, 2021

    2021

  44. [52]

    Avatarartist: Open-domain 4d avatarization

    Hongyu Liu, Xuan Wang, Ziyu Wan, Yue Ma, Jingye Chen, Yanbo Fan, Yujun Shen, Yibing Song, and Qifeng Chen. Avatarartist: Open-domain 4d avatarization. InCVPR, 2025

    2025

  45. [53]

    Decoupled spatial-temporal transformer for video inpainting

    Rui Liu, Hanming Deng, Yangyi Huang, Xiaoyu Shi, Lewei Lu, Wenxiu Sun, Xiaogang Wang, Jifeng Dai, and Hongsheng Li. Decoupled spatial-temporal transformer for video inpainting. arXiv preprint arXiv:2104.06637, 2021

    2021 arXiv

  46. [54]

    Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.ArXiv, abs/2206.00927, 2022

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.ArXiv, abs/2206.00927, 2022

    2022 arXiv

  47. [55]

    Magicstick: Controllable video editing via control handle transformations

    Yue Ma, Xiaodong Cun, Sen Liang, Jinbo Xing, Yingqing He, Chenyang Qi, Siran Chen, and Qifeng Chen. Magicstick: Controllable video editing via control handle transformations. In WACV, 2025

    2025

  48. [56]

    Follow your pose: Pose-guided text-to-video generation using pose-free videos

    Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Siran Chen, Xiu Li, and Qifeng Chen. Follow your pose: Pose-guided text-to-video generation using pose-free videos. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4117–4125, 2024

    2024

  49. [57]

    Follow your pose: Pose-guided text-to-video generation using pose-free videos, 2024

    Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Siran Chen, Ying Shan, Xiu Li, and Qifeng Chen. Follow your pose: Pose-guided text-to-video generation using pose-free videos, 2024

    2024

  50. [58]

    Follow-your-click: Open-domain regional image animation via short prompts.arXiv preprint arXiv:2403.08268, 2024

    Yue Ma, Yingqing He, Hongfa Wang, Andong Wang, Chenyang Qi, Chengfei Cai, Xiu Li, Zhifeng Li, Heung-Yeung Shum, Wei Liu, et al. Follow-your-click: Open-domain regional image animation via short prompts.arXiv preprint arXiv:2403.08268, 2024

    2024 arXiv

  51. [59]

    Follow-your-emoji: Fine-controllable and expressive freestyle portrait animation

    Yue Ma, Hongyu Liu, Hongfa Wang, Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung-Yeung Shum, Wei Liu, et al. Follow-your-emoji: Fine-controllable and expressive freestyle portrait animation. InSIGGRAPH Asia 2024 Conference Papers, pages 1–12, 2024

    2024

  52. [60]

    Visual knowledge graph for human action reasoning in videos

    Yue Ma, Yali Wang, Yue Wu, Ziyu Lyu, Siran Chen, Xiu Li, and Yu Qiao. Visual knowledge graph for human action reasoning in videos. InProceedings of the 30th ACM International Conference on Multimedia, pages 4132–4141, 2022

    2022

  53. [62]

    Sdedit: Guided image synthesis and editing with stochastic differential equations

    Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022

    2022

  54. [63]

    Anidoc: Animation creation made easier.arXiv preprint arXiv:2412.14173, 2024

    Yihao Meng, Hao Ouyang, Hanlin Wang, Qiuyu Wang, Wen Wang, Ka Leong Cheng, Zhiheng Liu, Yujun Shen, and Huamin Qu. Anidoc: Animation creation made easier.arXiv preprint arXiv:2412.14173, 2024

    2024 arXiv

  55. [64]

    Null-text inversion for editing real images using guided diffusion models

    Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. InCVPR, 2023. 13

    2023

  56. [65]

    Revideo: Remake a video with motion and content control

    Chong Mou, Mingdeng Cao, Xintao Wang, Zhaoyang Zhang, Ying Shan, and Jian Zhang. Revideo: Remake a video with motion and content control. InNeurIPS, 2024

    2024

  57. [66]

    Codef: Content deformation fields for temporally consistent video processing.arXiv preprint arXiv:2308.07926, 2023

    Hao Ouyang, Qiuyu Wang, Yuxi Xiao, Qingyan Bai, Juntao Zhang, Kecheng Zheng, Xiaowei Zhou, Qifeng Chen, and Yujun Shen. Codef: Content deformation fields for temporally consistent video processing.arXiv preprint arXiv:2308.07926, 2023

    2023 arXiv

  58. [67]

    Dreamdance: Animating human images by enriching 3d geometry cues from 2d poses.arXiv preprint arXiv:2412.00397, 2024

    Yatian Pang, Bin Zhu, Bin Lin, Mingzhe Zheng, Francis EH Tay, Ser-Nam Lim, Harry Yang, and Li Yuan. Dreamdance: Animating human images by enriching 3d geometry cues from 2d poses.arXiv preprint arXiv:2412.00397, 2024

    2024 arXiv

  59. [68]

    Con- trolnext: Powerful and efficient control for image and video generation, 2025

    Bohao Peng, Jian Wang, Yuechen Zhang, Wenbo Li, Ming-Chang Yang, and Jiaya Jia. Con- trolnext: Powerful and efficient control for image and video generation, 2025

    2025

  60. [69]

    Movie Gen: A cast of media foundation models.arXiv preprint arXiv:2410.13720, 2024

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, et al. Movie Gen: A cast of media foundation models.arXiv preprint arXiv:2410.13720, 2024

    2024 arXiv

  61. [70]

    D-nerf: Neural radiance fields for dynamic scenes

    Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318–10327, 2021

    2021

  62. [71]

    Fatezero: Fusing attentions for zero-shot text-based video editing

    Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. InICCV, 2023

    2023

  63. [72]

    Gen3c: 3d-informed world- consistent video generation with precise camera control.ArXiv, abs/2503.03751, 2025

    Xuanchi Ren, Tianchang Shen, Jiahui Huang, Huan Ling, Yifan Lu, Merlin Nimier-David, Thomas Muller, Alexander Keller, Sanja Fidler, and Jun Gao. Gen3c: 3d-informed world- consistent video generation with precise camera control.ArXiv, abs/2503.03751, 2025

    2025 arXiv

  64. [73]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. InICLR, 2023

    2023

  65. [74]

    Dimensionx: Create any 3d and 4d scenes from a single image with controllable video diffusion.arXiv preprint arXiv:2411.04928, 2024

    Wenqiang Sun, Shuo Chen, Fangfu Liu, Zilong Chen, Yueqi Duan, Jun Zhang, and Yikai Wang. Dimensionx: Create any 3d and 4d scenes from a single image with controllable video diffusion.arXiv preprint arXiv:2411.04928, 2024

    2024 arXiv

  66. [75]

    Edtalk: Efficient disentanglement for emotional talking head synthesis, 2024

    Shuai Tan, Bin Ji, Mengxiao Bi, and Ye Pan. Edtalk: Efficient disentanglement for emotional talking head synthesis, 2024

    2024

  67. [76]

    Unipaint: Unified space-time video inpainting via mixture-of-experts.arXiv preprint arXiv:2412.06340, 2024

    Zhen Wan, Yue Ma, Chenyang Qi, Zhiheng Liu, and Tao Gui. Unipaint: Unified space-time video inpainting via mixture-of-experts.arXiv preprint arXiv:2412.06340, 2024

    2024 arXiv

  68. [77]

    Wan: Open and advanced large-scale video generative models.ArXiv, abs/2503.20314, 2025

    Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Xiaofeng Meng, Ningying Zhang, Pandeng Li, P...

    2025 arXiv

  69. [78]

    Cove: Un- leashing the diffusion feature correspondence for consistent video editing.arXiv preprint arXiv:2406.08850, 2024

    Jiangshan Wang, Yue Ma, Jiayi Guo, Yicheng Xiao, Gao Huang, and Xiu Li. Cove: Un- leashing the diffusion feature correspondence for consistent video editing.arXiv preprint arXiv:2406.08850, 2024

    2024 arXiv

  70. [79]

    Taming rectified flow for inversion and editing.arXiv preprint arXiv:2411.04746, 2024

    Jiangshan Wang, Junfu Pu, Zhongang Qi, Jiayi Guo, Yue Ma, Nisha Huang, Yuxin Chen, Xiu Li, and Ying Shan. Taming rectified flow for inversion and editing.arXiv preprint arXiv:2411.04746, 2024. 14

    2024 arXiv

  71. [80]

    Motion inversion for video customization.arXiv preprint arXiv:2403.20193, 2024

    Luozhou Wang, Ziyang Mai, Guibao Shen, Yixuan Liang, Xin Tao, Pengfei Wan, Di Zhang, Yijun Li, and Yingcong Chen. Motion inversion for video customization.arXiv preprint arXiv:2403.20193, 2024

    2024 arXiv

  72. [81]

    Akira: Augmentation kit on rays for optical video generation.arXiv preprint arXiv:2412.14158, 2024

    Xi Wang, Robin Courant, Marc Christie, and Vicky Kalogeiton. Akira: Augmentation kit on rays for optical video generation.arXiv preprint arXiv:2412.14158, 2024

    2024 arXiv

  73. [82]

    Lavie: High-quality video generation with cascaded latent diffusion models.IJCV, pages 1–20, 2024

    Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models.IJCV, pages 1–20, 2024

    2024

  74. [83]

    Motionctrl: A unified and flexible motion controller for video generation

    Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. InACM SIGGRAPH 2024 Conference Proceedings, 2024

    2024

  75. [84]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. InICCV, 2023

    2023

  76. [85]

    Trajectory attention for fine-grained video motion control.ArXiv, abs/2411.19324, 2024

    Zeqi Xiao, Wenqi Ouyang, Yifan Zhou, Shuai Yang, Lei Yang, Jianlou Si, and Xingang Pan. Trajectory attention for fine-grained video motion control.ArXiv, abs/2411.19324, 2024

    2024 arXiv

  77. [86]

    ToonCrafter: Generative cartoon interpolation.ACM TOG, 43(6):1–11, 2024

    Jinbo Xing, Hanyuan Liu, Menghan Xia, Yong Zhang, Xintao Wang, Ying Shan, and Tien-Tsin Wong. ToonCrafter: Generative cartoon interpolation.ACM TOG, 43(6):1–11, 2024

    2024

  78. [87]

    Make-your-video: Customized video generation using textual and structural guidance.IEEE TVCG, 31(2):1526–1541, 2024

    Jinbo Xing, Menghan Xia, Yuxin Liu, Yuechen Zhang, Y He, H Liu, H Chen, X Cun, X Wang, Y Shan, et al. Make-your-video: Customized video generation using textual and structural guidance.IEEE TVCG, 31(2):1526–1541, 2024

    2024

  79. [88]

    Dynamicrafter: Animating open-domain images with video diffusion priors

    Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Dynamicrafter: Animating open-domain images with video diffusion priors. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Rus- sakovsky, Torsten S...

    2024

  80. [89]

    Enhancing image generation fidelity via progressive prompts.arXiv preprint arXiv:2501.07070, 2025

    Zhen Xiong, Yuqi Li, Chuanguang Yang, Tiao Tan, Zhihong Zhu, Siyuan Li, and Yue Ma. Enhancing image generation fidelity via progressive prompts.arXiv preprint arXiv:2501.07070, 2025

    2025 arXiv

  81. [90]

    CamCo: Camera-controllable 3D-consistent image-to-video generation.arXiv preprint arXiv:2406.02509, 2024

    Dejia Xu, Weili Nie, Chao Liu, Sifei Liu, Jan Kautz, Zhangyang Wang, and Arash Vah- dat. CamCo: Camera-controllable 3D-consistent image-to-video generation.arXiv preprint arXiv:2406.02509, 2024

    2024 arXiv

  82. [91]

    Hallo: Hierarchical audio-driven visual synthesis for portrait image animation, 2024

    Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang, Yao Yao, and Siyu Zhu. Hallo: Hierarchical audio-driven visual synthesis for portrait image animation, 2024

    2024

  83. [92]

    Follow-your-pose v2: Multiple-condition guided character image animation for stable pose control.arXiv preprint arXiv:2406.03035, 2024

    Jingyun Xue, Hongfa Wang, Qi Tian, Yue Ma, Andong Wang, Zhiyuan Zhao, Shaobo Min, Wenzhe Zhao, Kaihao Zhang, Heung-Yeung Shum, et al. Follow-your-pose v2: Multiple-condition guided character image animation for stable pose control.arXiv preprint arXiv:2406.03035, 2024

    2024 arXiv

  84. [93]

    Eedit: Rethinking the spatial and temporal redundancy for efficient image editing.arXiv preprint arXiv:2503.10270, 2025

    Zexuan Yan, Yue Ma, Chang Zou, Wenteng Chen, Qifeng Chen, and Linfeng Zhang. Eedit: Rethinking the spatial and temporal redundancy for efficient image editing.arXiv preprint arXiv:2503.10270, 2025

    2025

  85. [94]

    Trajectorycrafter: Redirecting camera trajectory for monocular videos via diffusion models.arXiv preprint arXiv:2503.05638, 2025

    Mark YU, Wenbo Hu, Jinbo Xing, and Ying Shan. Trajectorycrafter: Redirecting camera trajectory for monocular videos via diffusion models.arXiv preprint arXiv:2503.05638, 2025

    2025 arXiv

  86. [95]

    Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis.arXiv preprint arXiv:2409.02048, 2024

    Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang, Xiangjun Gao, Tien-Tsin Wong, Ying Shan, and Yonghong Tian. Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis.arXiv preprint arXiv:2409.02048, 2024. 15

    2024 arXiv

  87. [96]

    Identity-preserving text-to-video generation by frequency decomposition.arXiv preprint arXiv:2411.17440, 2024

    Shenghai Yuan, Jinfa Huang, Xianyi He, Yunyuan Ge, Yujun Shi, Liuhan Chen, Jiebo Luo, and Li Yuan. Identity-preserving text-to-video generation by frequency decomposition.arXiv preprint arXiv:2411.17440, 2024

    2024 arXiv

  88. [97]

    Follow- your-multipose: Tuning-free multi-character text-to-video generation via pose guidance.arXiv preprint arXiv:2412.16495, 2024

    Beiyuan Zhang, Yue Ma, Chunlei Fu, Xinyang Song, Zhenan Sun, and Ziqiang Li. Follow- your-multipose: Tuning-free multi-character text-to-video generation via pose guidance.arXiv preprint arXiv:2412.16495, 2024

    2024 arXiv

  89. [98]

    Recapture: Generative video camera controls for user-provided videos using masked video fine-tuning.arXiv preprint arXiv:2411.05003, 2024

    David Junhao Zhang, Roni Paiss, Shiran Zada, Nikhil Karnad, David E Jacobs, Yael Pritch, Inbar Mosseri, Mike Zheng Shou, Neal Wadhwa, and Nataniel Ruiz. Recapture: Generative video camera controls for user-provided videos using masked video fine-tuning.arXiv preprint arXiv:241...

    2024 arXiv

  90. [99]

    Magiccolor: Multi- instance sketch colorization.arXiv preprint arXiv:2503.16948, 2025

    Yinhan Zhang, Yue Ma, Bingyuan Wang, Qifeng Chen, and Zeyu Wang. Magiccolor: Multi- instance sketch colorization.arXiv preprint arXiv:2503.16948, 2025

    2025 arXiv

  91. [100]

    Avid: Any-length video inpainting with diffusion model.arXiv preprint arXiv:2312.03816, 2023

    Zhixing Zhang, Bichen Wu, Xiaoyan Wang, Yaqiao Luo, Luxin Zhang, Yinan Zhao, Peter Vajda, Dimitris Metaxas, and Licheng Yu. Avid: Any-length video inpainting with diffusion model.arXiv preprint arXiv:2312.03816, 2023

    2023 arXiv

  92. [101]

    Cami2v: Camera- controlled image-to-video diffusion model.arXiv preprint arXiv:2410.15957, 2024

    Guangcong Zheng, Teng Li, Rui Jiang, Yehao Lu, Tao Wu, and Xi Li. Cami2v: Camera- controlled image-to-video diffusion model.arXiv preprint arXiv:2410.15957, 2024

    2024 arXiv

  93. [102]

    Propainter: Im- proving propagation and transformer for video inpainting

    Shangchen Zhou, Chongyi Li, Kelvin CK Chan, and Chen Change Loy. Propainter: Im- proving propagation and transformer for video inpainting. InProceedings of the IEEE/CVF international conference on computer vision, pages 10477–10486, 2023

    2023

  94. [103]

    Multibooth: Towards generating all your concepts in an image from text

    Chenyang Zhu, Kai Li, Yue Ma, Chunming He, and Xiu Li. Multibooth: Towards generating all your concepts in an image from text. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 10923–10931, 2025

    2025

  95. [104]

    Instantswap: Fast customized concept swapping across sharp shape differences

    Chenyang Zhu, Kai Li, Yue Ma, Longxiang Tang, Chengyu Fang, Chubin Chen, Qifeng Chen, and Xiu Li. Instantswap: Fast customized concept swapping across sharp shape differences. arXiv preprint arXiv:2412.01197, 2024

    2024 arXiv

  96. [105]

    One model to edit them all: Free-form text-driven image manipulation with semantic modulations.Advances in Neural Information Processing Systems, 35:25146–25159, 2022

    Yiming Zhu, Hongyu Liu, Yibing Song, Ziyang Yuan, Xintong Han, Chun Yuan, Qifeng Chen, and Jue Wang. One model to edit them all: Free-form text-driven image manipulation with semantic modulations.Advances in Neural Information Processing Systems, 35:25146–25159, 2022

    2022

  97. [106]

    Cococo: Improving text-guided video inpainting for better consistency, controllability and compatibility.ArXiv, abs/2403.12035, 2024

    Bojia Zi, Shihao Zhao, Xianbiao Qi, Jianan Wang, Yukai Shi, Qianyu Chen, Bin Liang, Kam-Fai Wong, and Lei Zhang. Cococo: Improving text-guided video inpainting for better consistency, controllability and compatibility.ArXiv, abs/2403.12035, 2024. 16

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.