Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

I2VControl: Disentangled and Unified Video Motion Synthesis Control

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read I2VControl claims a single pipeline can orchestrate camera motion, object dragging, and motion brush without conflicts.

desk verdict A genuinely new three-in-one control adapter whose 'dense trajectory' framing overstates what brush units actually receive, and whose conflict-free claim needs a quantitative benchmark. read the letter →

arxiv 2411.17765 v3 pith:ZLWYPNKI submitted 2024-11-26 cs.CV

Wanquan Feng , Tianhao Qi , Jiawei Liu , Mingzhen Sun , Pengqi Tu , Tianxiang Ma , Fei Dai , Songtao Zhao
show 2 more authors
Siyu Zhou Qian He
This is my paper · ORCID
classification cs.CV
keywords image-to-videosynthesismotioncontrollabilitycameracontrolobjectdraggingbrushdensepointtrajectoriesspatialpartitioningdiffusionadapter
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the three standard families of video-motion control—camera extrinsics, object dragging, and region-based motion brush—are not separate problems but one problem: specifying dense point trajectories. I2VControl tries to show that by rewriting every control as a trajectory signal, partitioning the image into motion units (drag, brush, borderland), and injecting per-unit controls through a small adapter into a frozen image-to-video diffusion model, all three controls can be exercised in a single inference pass without logical conflicts. If this holds, a video editor would no longer choose between moving the camera and moving an object: both can be specified together, and the model resolves which pixels obey which command. The paper reports single-task results that match or beat dedicated baselines on camera, dragging, and brush benchmarks, and demonstrates combined controls that purely single-task methods cannot express.

What carries the argument

The load-bearing device is the unit-wise trajectory decomposition with the control tensor. The image domain is sliced into motion units $\Omega^{(p)}$ (brush-units, drag-units, and the borderland) with masks $\chi^{(p)}$, and on each unit the world-frame trajectory is decomposed as $D(\lambda,x)=R_\lambda^{(p)}\circ x+G^{(p)}(\lambda,x)$, where $R_\lambda^{(p)}\in SE(3)$ is the best-fit rigid transform recovered from tracked points (the identity for brush-units and borderland) and $G^{(p)}$ carries residual motion summarized by the scalar strength $m_\lambda^{(p)}$. A global camera transform $E_\lambda$ is applied afterward, giving the final camera-frame trajectory $T_\lambda=\sum_p \Pi(E_\lambda^{-1}\circ R_\lambda^{(p)}\circ(\chi^{(p)}\odot\Omega))$. The control signal handed to the network is the concatenation of trajectory, motion-strength, partition-index, and category maps $(T_\lambda,M_\lambda,P_\lambda,C_\lambda)$, encoded by convolutional layers and injected through a few trainable attention layers (the adapter) before the frozen diffusion model's self-attention, then removed after attention.

What would settle it

On held-out videos of a moving camera with a strongly non-rigid object (cloth, liquid, or hair), run the paper's pipeline to extract $E_\lambda$ and per-unit $R_\lambda^{(p)}$ and measure the residual $G^{(p)}(\lambda,x)$. If that residual grows with deformation speed and cannot be summarized by the scalar $m_\lambda^{(p)}$, the factorization fails; equivalently, generated brush-control videos on such scenes should visibly fail to reproduce the non-rigid motion, which a reader can check directly.

Watch

Extended reading notes

Core claim

The paper tries to establish that camera control, object dragging, and motion brush are not fundamentally separate mechanisms but three instances of one mechanism: prescribing dense point trajectories. Camera motion is a global 6-DOF transform $E_\lambda$ acting on all points; a dragged region is a per-unit rigid transform $R_\lambda^{(p)}$ fit to the tracked motion of that mask; a brushed region is governed only by a scalar motion strength $m_\lambda^{(p)}$, with an identity rigid part; the unselected borderland is left to natural motion. With the image partitioned into such units and each pixel carrying its trajectory, the generated video motion is specified in a single coordinate frame with explicit per-unit instructions, so the previous conflicts between camera motion and object or background motion are removed at the representation level. The paper substantiates this by reporting camera rotation and translation errors, drag accuracy (ObjMC), and brush motion-strength metrics on six datasets, along with qualitative combined-control sequences that single-task baselines cannot produce.

Load-bearing premise

The whole scheme assumes that real video motion can be decomposed into a global camera transform, per-region rigid motion, and a single scalar residual per region, and that the tracker and depth estimator used to build training labels recover that decomposition accurately.

Editorial extensions

If this is right

  • A user can set a camera path, drag masks with 6-DOF motion, and brush masks with motion-strength scalars in one inference call, and the generated frames follow all these signals simultaneously.
  • Because the adapter plugs into a frozen base model and the control representation is expressed as pixel-wise maps, the same adapter design transfers to other image-to-video diffusion backbones, as demonstrated in the paper on Seedance and Wan 2.1.
  • The automatic data pipeline turns ordinary RGB video clips into training supervision through tracking and motion-unit decomposition, so unified control training does not require per-task 3D or optical-flow datasets.
  • Single-task performance is not sacrificed for unification: the paper's reported camera errors, drag ObjMC, and brush motion-strength metrics improve on or match the dedicated methods it compares against.
  • Spatial controls can be composed dynamically per frame from masks, so switching from dragging one object to brushing a fluid region is only a change of category map and strength value, not a retrained model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the trajectory-factorization view generalizes, other motion controls—box trajectories, text-prompt motion, audio-driven motion—could be encoded as additional per-unit signal channels inside the same adapter rather than as separate frameworks; the paper only builds the three categories it evaluates.
  • The scalar motion-strength per unit is intentionally coarse, so for strongly non-rigid deformations such as cloth, liquid, or hair a single residual scalar may under-specify motion; a natural extension would carry per-unit residual flow fields alongside the rigid transform, and this is testable with the paper's own brush datasets.
  • The paper's data pipeline can be read as a video-motion-analysis tool: the same decomposition into global camera transform, per-unit rigid motion, and residual strength could be used to retarget or edit the motion of existing videos, an application the paper does not demonstrate.
  • The claimed conflict-free combination depends on the per-unit masks covering the relevant motion; a testable extension is whether partially overlapping or erroneously segmented masks degrade the camera-drag-brush combination more than single-control settings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes I2VControl, an image-to-video motion control framework that unifies camera extrinsics control, object dragging, and motion brush into a single dense point-trajectory representation. It introduces a motion-unit spatial partitioning scheme in which the image is divided into brush-units, drag-units, and a borderland, each governed by a dedicated formulation, and it presents a plug-in adapter for pre-trained video diffusion models along with a data pipeline that extracts control signals from RGB videos. The method is trained on 970K clips and evaluated on six datasets against DragAnything, MOFA-Video, MotionCtrl, CameraCtrl, and Motion-I2V, reporting strong quantitative results on single-control benchmarks. The central claim is that the three control types can be combined without logical conflicts in a single inference pipeline.

Significance. If the central claim holds, I2VControl would be a practically significant contribution: it is the first single pipeline supporting camera, drag, and brush control together, and it introduces a unified representation that makes combined control feasible. The paper contains substantial experimental work, including six datasets, three newly curated test sets, user studies, and adapter training on multiple base models (Seedance and Wanx 2.1), with a code release mentioned for Wan 2.1. The adapter design is architecture-agnostic and the data pipeline is clearly described. However, the flagship 'without conflicts' claim currently rests on qualitative examples, and the brush-control signal is information-limited by construction, so the contribution is not yet established at the level claimed in the abstract.

major comments (4)
  1. [Sec. 4.2 (Eqs. 5, 7, 10)] The 'dense trajectory' control signal is not dense for brush units. In Eq. (5), D decomposes into R^(p) and G^(p), but Eq. (10) feeds only R^(p) into T_λ. For brush units Eq. (7) sets R^(p) ≡ I, so T_λ is the camera-parallax projection of a static point set; the only brush information is the scalar m^(p)_λ computed by Eq. (7). Infinitely many ground-truth residual motions G^(p) are consistent with the same control signal. The model therefore cannot learn a deterministic mapping from the control signal to the requested residual motion; it must regress to a conditional mean or hallucinate. Please either include the residual in the trajectory signal (e.g., sampled trajectories from G^(p)) or provide an explicit analysis or bound showing that the scalar bottleneck preserves user intent for the brush task.
  2. [Sec. 5.3.2 and Fig. 7] The paper's first contribution is conflict-free combined control, but the only evidence for the combined setting is qualitative (Fig. 7, columns 7–8). There is no quantitative metric for multi-control consistency, such as camera RotErr/TransErr measured under simultaneous drag and brush, ObjMC under camera motion, brush mask IoU/MSC under drag or camera, or an ablation that removes one control and measures the effect on the others. Without such a benchmark, the central 'without conflicts' claim is not empirically supported.
  3. [Sec. 4.2 (Eq. 3) and Table 4] The brush metric is circular with respect to the training objective. The motion strength M in Eq. (3) is taken from the authors' own I2VControl-Camera [8], used as supervision for brush and drag units in Eqs. (7) and (9), and then reused as the MSC evaluation metric in Table 4. High MSC therefore partly measures adherence to the training definition rather than agreement with an external notion of requested motion. Please supplement MSC with an external metric, such as tracked optical-flow consistency with the user-supplied motion strength or a forced-choice user study comparing different residual motions at a fixed scalar strength.
  4. [Sec. 4.3] The data pipeline assumes that SpatialTracker plus the I2VControl-Camera decomposition recovers E_λ and D(λ, x) accurately, and that Unidepth provides a reliable initial point set Ω during inference. No error analysis or validation of these upstream estimates is reported. Since T_λ and M_λ are computed from these quantities, systematic tracking or depth errors would corrupt the training signal and the projected trajectories at inference. Please report tracking and depth accuracy on a validation subset, or provide an ablation that perturbs E_λ and Ω, to bound the sensitivity of the control signal to these assumptions.
minor comments (5)
  1. [Sec. 5.3.2, Tables 2–4] Please report the number of raters and videos in the user study, and provide confidence intervals or significance tests; the reported score differences are large, but the scale and variance are not stated.
  2. [Fig. 6] The notation (B, T, 5, H, W) for the control signal is inconsistent with the text description of a (T, 5, H, W)-shaped tensor and with the channel breakdown (T_λ, M_λ, P_λ, C_λ); please clarify how the five channels are arranged.
  3. [Eq. (4)] The masks χ^(p) are not explicitly required to be disjoint; please state the partition constraint (e.g., Σ_p χ^(p) = 1) to make the spatial partitioning well-defined.
  4. [Sec. 4.2 and Figs. 1, 7] The user-facing motion-strength values (e.g., 300, 1000, 5000) are not normalized or calibrated; please specify the scale, a default value, or the mapping from user input to m^(p)_λ.
  5. [References] References [2] and [3] are duplicated (both cite the same Stable Video Diffusion preprint); please merge or correct the citation list.

Circularity Check

2 steps flagged · score 4.0 of 10

Brush-control evaluation is self-referential: the MSC metric is the same motion-strength functional used as the training label (borrowed from the authors' I2VControl-Camera), and the 'dense trajectory' for brush units is definitional because Eq. 10 omits the non-rigid term.

  1. fitted input called prediction [Eq. (3), Eqs. (7)/(9), Sec. 5.1 Metrics]
    "We refer to I2VControl-Camera [8] and define a general motion strength function ... M(Γ, H, λ) = 1/|Γ| ∫_Γ ∥∂H(x, λ)/∂λ∥^2 dx ... To judge motion brush, we firstly refer to previous work [8] to compute a motion strength score MSC, and then refer to SAM [21] to employ a IoU metric."

    The brush training label is m_λ^(p) = M(Ω^(p), G^(p), λ) (Eqs. 7 and 9), i.e. the average speed under the same functional M borrowed from the authors' prior I2VControl-Camera [8]. The brush evaluation metric MSC is computed with that same M on the generated video. Thus the quantitative brush result is scored by the very functional that defines the training target: a model trained to match m will score high on MSC by construction, while the comparison method Motion-I2V, which is trained on optical flow, is evaluated on a metric tailored to the authors' formulation. The brush benchmark therefore measures adherence to the training definition rather than an externally grounded motion intent.

  2. renaming known result [Sec. 4.1-4.2, Eq. (10), Sec. 5.2]
    "For motion brush, we adopt the definition from motion-I2V [30], which enables the selected region to move (without specifying a trajectory or direction). ... T_λ = Σ_p Π(E_λ^{-1} ◦ R_λ^(p) ◦ (χ^(p) ⊙ Ω)) ... On the brush-units, we set R_λ^(p) ≡ I, m_λ^(p) = M(Ω^(p), G^(p), λ)."

    For brush-units R_λ^(p) ≡ I, so the 'dense point trajectory' T_λ in Eq. 10 contains no brush-specific motion; it is just the static point set with camera parallax. The only brush-dependent signal is the scalar m_λ^(p) carried in M_λ. The paper's own definition of motion brush is 'without specifying a trajectory or direction,' so presenting the unified representation as 'dense point trajectories' is a renaming: the brush control remains the scalar motion-strength control of Motion-I2V, merely concatenated into a tensor with static projected points. The claimed unification of brush into the trajectory representation is therefore definitional rather than derived.

full rationale

The camera-control and object-dragging components are evaluated against external datasets (RealEstate10K, MO, VIPSeg, Manual Dragging) with standard metrics (RotErr, TransErr, ObjMC, FID), and the spatial-partitioning adapter has independent engineering content, so the paper is not globally circular. The circularity is concentrated in the motion-brush modality: the motion-strength functional M is taken from the authors' own I2VControl-Camera [8], used to create the brush training labels, and then reused as the MSC evaluation metric, so the headline brush numbers partly reflect the training definition. In addition, Eq. 10 omits the non-rigid residual G for brush-units, so the supposed dense-trajectory unification is achieved by renaming the known scalar brush control rather than by actually representing brush motion as a trajectory. These issues affect the brush benchmark and the 'unified representation' claim, but they do not nullify the camera/drag results, hence a moderate score of 4.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The central claim rests on a factorization of video motion into camera extrinsics plus per-unit rigid and residual terms, on the accuracy of upstream tracking, depth, and segmentation tools, and on several hand-set experimental choices (unit-selection threshold, scalar motion strengths). No new physical entities are introduced; the 'motion unit' is an internal representational abstraction.

free parameters (2)
  • Unit selection overlap threshold = >50%
    Hand-chosen threshold in the data pipeline (Sec 4.3) determining which semantic segments become brush or drag units; not varied or justified, and it shapes all training control signals.
  • Motion-strength scalar per unit = User-set values such as 1000, 300, 5000
    Scalar strength governs brush and residual drag motion; values are chosen by users or examples without a canonical scale, so comparisons and creative results depend on these arbitrary magnitudes.
assumptions (5)
  • domain assumption Video motion factorizes into camera extrinsics E_lambda and per-unit deformations, where each motion unit's displacement is a rigid transform R_lambda^(p) plus a residual G^(p) (Eq. 5).
    Invoked in Sec 4.1; Eq. 10 feeds only the rigid part R into the trajectory signal while the residual appears only as a scalar strength, so non-rigid motion is represented lossily.
  • domain assumption SpatialTracker and the I2VControl-Camera decomposition recover accurate dense trajectories D(lambda,x) and camera extrinsics E_lambda from monocular RGB video.
    Sec 4.3 uses these off-the-shelf tools to build all training labels; the paper provides no error analysis, and errors would propagate into every control signal.
  • domain assumption UniDepth monocular metric depth estimation provides a sufficiently accurate initial 3D point set Omega for user controls at inference.
    Sec 4.3 lifts 2D user inputs to 3D with UniDepth; depth errors change the projection in Eq. 10 and can distort camera and drag control.
  • ad hoc to paper The borderland region, assigned R_lambda = I and m_lambda = 0, will still move naturally through the base model without explicit control.
    Stated in Sec 4.2 as a desired property rather than derived; it assumes the pretrained model hallucinates coherent background motion despite a zero control signal.
  • standard math SE(3) rigid transforms, projective camera models, and the inverse-extrinsic mapping in Eq. (1) are valid for video frames.
    Used without proof throughout Sec 3.2 and Sec 4; standard background for camera geometry.
invented entities (1)
  • Motion-unit abstraction (brush-unit, drag-unit, borderland)
    purpose: Spatial partitioning of the input image into control regions so that different control types can coexist without logical conflict.
    This is an internal representational construct for the framework; no external observable prediction is made beyond the generated videos themselves, and its validity rests on the paper's own experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of I2VControl: Disentangled and Unified Video Motion Synthesis Control." pith.science (2026). https://pith.science/paper/ZLWYPNKI

@misc{pith2026241117765,
  author       = {Pith},
  title        = {Pith review of: I2VControl: Disentangled and Unified Video Motion Synthesis Control},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZLWYPNKI}},
  note         = {Machine review of arXiv:2411.17765}
}
read the original abstract

Motion controllability is crucial in video synthesis. However, most previous methods are limited to single control types, and combining them often results in logical conflicts. In this paper, we propose a disentangled and unified framework, namely I2VControl, to overcome the logical conflicts. We rethink camera control, object dragging, and motion brush, reformulating all tasks into a consistent representation based on point trajectories, each managed by a dedicated formulation. Accordingly, we propose a spatial partitioning strategy, where each unit is assigned to a concomitant control category, enabling diverse control types to be dynamically orchestrated within a single synthesis pipeline without conflicts. Furthermore, we design an adapter structure that functions as a plug-in for pre-trained models and is agnostic to specific model architectures. We conduct extensive experiments, achieving excellent performance on various control tasks, and our method further facilitates user-driven creative combinations, enhancing innovation and creativity. Project page: https://wanquanf.github.io/I2VControl .

Figures

Figures reproduced from arXiv: 2411.17765 by the authors.

Figure 1
Figure 1. We propose an all-in-one disentangled and unified framework for image-to-video motion synthesis control, named I2VControl. In the illustration, we show several scenarios of controls, including camera movement (camera dollies in and gets closer to the sculpture), motion brush (smoke flows in the wind, with a given motion strength value), object movement (the astronaut walks for￾ward), and the combination of all above… view at source ↗
Figure 2
Figure 2. Examples of control conflicts in multiple control tasks. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Examples of dual tasks, where the dual perception task [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: The process of obtaining the motion units. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Details of adapter architecture and calculation. [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: We list some (not all) capabilities of I2VControl. The first two columns illustrate dragging operations within the xy-plane and [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Comparison on dragging control task. 5. Experiments In this section, we introduce our experiments. Sec. 5.1 presents our implementation details and settings. Sec. 5.2 presents the feature comprehensiveness comparison. In Sec. 5.3, we show our results and experimental c…
Figure 9
Figure 9. Figure 9: Comparison on camera control. on the same base model and data as ours. For the remaining methods, although the base models are different, they are at a similar level, all of which are U-Net methods (rather than recently developed high-quality DiT models). Further￾more,…
Figure 10
Figure 10. Figure 10: Comparison on motion brush for fluid effects. [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: Comparison on motion brush for object movements. [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Perception-as-Control: Fine-grained Controllable Image Animation with 3D-aware Motion Representation

    cs.CV 2025-01 conditional novelty 7.0 of 10

    A diffusion-based image animation method that uses rendered colored spheres and a checkerboard world envelope as 3D-aware motion control signals, enabling joint camera and object motion control with fine granularity.

  2. VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generation

    cs.CV 2025-02 conditional novelty 6.0 of 10

    VidCRAFT3 is a single image-to-video diffusion system that accepts camera, object, and lighting direction controls separately or jointly, trained in three stages with a new synthetic lighting dataset.

  3. Motion Prompting: Controlling Video Generation with Motion Trajectories

    cs.CV 2024-12 conditional novelty 5.0 of 10

    A single-stage ControlNet on the Lumiere video model, conditioned only on dense point tracks, generalizes to sparse and dense trajectory control for object, camera, and transferred motions.

Reference graph

Works this paper leans on

49 extracted references · 18 canonical work pages · cited by 3 Pith papers

  1. [8]

    I2vcontrol-camera: Precise video camera control with adjustable motion strength

    Wanquan Feng, Jiawei Liu, Pengqi Tu, Tianhao Qi, Mingzhen Sun, Tianxiang Ma, Songtao Zhao, Siyu Zhou, and Qian He. I2vcontrol-camera: Precise video camera control with adjustable motion strength. arXiv preprint arXiv:2411.06525, 2024. 3, 4, 5, 6

    arXiv 2024

  2. [1]

    Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models, 2024

    Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng, Min Zhao, Shilong Liu, Yaole Wang, and Jun Zhu. Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models, 2024. 1

    work page 2024

  3. [3]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 3

    arXiv 2023

  4. [4]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023. 3

    2023

  5. [5]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators

  6. [6]

    Videocrafter1: Open diffusion models for high-quality video generation

    Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512, 2023. 3

    arXiv

  7. [7]

    Seine: Short-to-long video diffu- sion model for generative transition and prediction

    Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffu- sion model for generative transition and prediction. In The Twelfth International Conference on Learning Representa- tions, 2023. 3

    2023

  8. [9]

    Seedance 1.0: Exploring the boundaries of video generation models

    Yu Gao, Haoyuan Guo, Tuyen Hoang, Weilin Huang, Lu Jiang, Fangyuan Kong, Huixia Li, Jiashi Li, Liang Li, Xi- aojie Li, et al. Seedance 1.0: Exploring the boundaries of video generation models. arXiv preprint arXiv:2506.09113,

    arXiv

Show all 49 references
  1. [10]

    Long video generation with time-agnostic vqgan and time- sensitive transformer

    Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vqgan and time- sensitive transformer. In European Conference on Computer Vision, pages 102–118. Springer, 2022. 2

    2022

  2. [11]

    Emu video: Factoriz- ing text-to-video generation by explicit image conditioning

    Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Du- val, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factoriz- ing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:2311.10709, 2023. 3

    2023 arXiv

  3. [12]

    I2v-adapter: A general image-to-video adapter for diffusion models

    Xun Guo, Mingwu Zheng, Liang Hou, Yuan Gao, Yufan Deng, Pengfei Wan, Di Zhang, Yufan Liu, Weiming Hu, Zhengjun Zha, et al. I2v-adapter: A general image-to-video adapter for diffusion models. In ACM SIGGRAPH 2024 Conference Papers, pages 1–12, 2024. 3

    2024

  4. [13]

    Photorealistic video generation with diffusion models

    Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and Jos ´e Lezama. Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.06662, 2023. 3

    2023 arXiv

  5. [14]

    Trackgo: A flexible and efficient method for controllable video generation

    Zhou Haitao, Wang Chuang, Nie Rui, Lin Jinxiao, Yu Dong- dong, Yu Qian, and Wang Changhu. Trackgo: A flexible and efficient method for controllable video generation. 2024. 6, 7

    2024

  6. [15]

    Cameractrl: Enabling camera control for text-to-video generation

    Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.02101, 2024. 2, 3, 6, 7

    2024 arXiv

  7. [16]

    Latent video diffusion models for high-fidelity long video generation

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211.13221 ,

    arXiv

  8. [17]

    Imagen video: High definition video generation with diffusion mod- els

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion mod- els. arXiv preprint arXiv:2210.02303, 2022. 3

    arXiv

  9. [18]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868, 2022. 2

    2022 arXiv

  10. [19]

    Animate anyone: Consistent and controllable image- to-video synthesis for character animation

    Li Hu. Animate anyone: Consistent and controllable image- to-video synthesis for character animation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8153–8163, 2024. 3

    2024

  11. [20]

    Peekaboo: Interactive video generation via masked- diffusion

    Yash Jain, Anshul Nasery, Vibhav Vineet, and Harkirat Behl. Peekaboo: Interactive video generation via masked- diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 2

    2024

  12. [21]

    Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C. Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick. Segment anything. arXiv:2304.02643, 2023. 2, 5, 6

    2023 arXiv

  13. [22]

    Kling, 2024

    KuaiShou. Kling, 2024. 1

    2024

  14. [23]

    Video generation from text

    Yitong Li, Martin Min, Dinghan Shen, David Carlson, and Lawrence Carin. Video generation from text. InProceedings of the AAAI conference on artificial intelligence, 2018. 2

    2018

  15. [24]

    Vidm: Video implicit diffusion models

    Kangfu Mei and Vishal Patel. Vidm: Video implicit diffusion models. In Proceedings of the AAAI conference on artificial intelligence, pages 9117–9125, 2023. 3

    2023

  16. [25]

    Large-scale video panoptic seg- mentation in the wild: A benchmark

    Jiaxu Miao, Xiaohan Wang, Yu Wu, Wei Li, Xu Zhang, Yun- chao Wei, and Yi Yang. Large-scale video panoptic seg- mentation in the wild: A benchmark. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, 2022. 6, 7 9

    2022

  17. [26]

    Mofa-video: Control- lable image animation via generative motion field adaptions in frozen image-to-video diffusion model

    Muyao Niu, Xiaodong Cun, Xintao Wang, Yong Zhang, Ying Shan, and Yinqiang Zheng. Mofa-video: Control- lable image animation via generative motion field adaptions in frozen image-to-video diffusion model. arXiv preprint arXiv:2405.20222, 2024. 3, 7

    2024 arXiv

  18. [27]

    UniDepth: Universal monocular metric depth estimation

    Luigi Piccinelli, Yung-Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. UniDepth: Universal monocular metric depth estimation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 5

    2024

  19. [28]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. 2021. 2

    2021

  20. [29]

    Gen2, 2023

    Runway. Gen2, 2023. 2, 3

    2023

  21. [30]

    Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling

    Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, et al. Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling. SIGGRAPH 2024, 2024. 2, 3, 7

    2024

  22. [31]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 ,

    arXiv

  23. [32]

    Wan: Open and advanced large-scale video gen- erative models

    Wan Team. Wan: Open and advanced large-scale video gen- erative models. 2025. 1, 8

    2025

  24. [33]

    Phenaki: Variable length video generation from open domain textual descriptions

    Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kin- dermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Le...

    2022

  25. [34]

    Pre- dicting video with vqvae

    Jacob Walker, Ali Razavi, and A ¨aron van den Oord. Pre- dicting video with vqvae. arXiv preprint arXiv:2103.01950,

    arXiv

  26. [35]

    Boximator: Generat- ing rich and controllable motions for video synthesis, 2024

    Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guo- qiang Wei, Liping Yuan, and Hang Li. Boximator: Generat- ing rich and controllable motions for video synthesis, 2024. 2, 3, 7

    2024

  27. [37]

    Magicvideo-v2: Multi- stage high-aesthetic video generation

    Weimin Wang, Jiawei Liu, Zhijie Lin, Jiangqiao Yan, Shuo Chen, Chetwin Low, Tuyen Hoang, Jie Wu, Jun Hao Liew, Hanshu Yan, et al. Magicvideo-v2: Multi- stage high-aesthetic video generation. arXiv preprint arXiv:2401.04468, 2024. 6

    2024 arXiv

  28. [38]

    Motionctrl: A unified and flexible motion controller for video generation

    Zhouxia Wang, Ziyang Yuan, Xintao Wang, Tianshui Chen, Menghan Xia, Ping Luo, and Yin Shan. Motionctrl: A unified and flexible motion controller for video generation

  29. [39]

    Dreamvideo: Composing your dream videos with customized subject and motion

    Yujie Wei, Shiwei Zhang, Zhiwu Qing, Hangjie Yuan, Zhi- heng Liu, Yu Liu, Yingya Zhang, Jingren Zhou, and Hong- ming Shan. Dreamvideo: Composing your dream videos with customized subject and motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog...

    2024

  30. [40]

    Draganything: Motion control for anything using entity representation, 2024

    Yuchao Gu Rui Zhao Yefei He David Junhao Zhang Mike Zheng Shou Yan Li Tingting Gao Di Zhang Weijia Wu, Zhuang Li. Draganything: Motion control for anything using entity representation, 2024. 2, 3, 6, 7

    2024

  31. [41]

    Godiva: Gen- erating open-domain videos from natural descriptions

    Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Gen- erating open-domain videos from natural descriptions. arXiv preprint arXiv:2104.14806, 2021. 2

    2021 arXiv

  32. [42]

    N ¨uwa: Visual synthesis pre- training for neural visual world creation

    Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. N ¨uwa: Visual synthesis pre- training for neural visual world creation. In European con- ference on computer vision, pages 720–736, 2022. 2

    2022

  33. [43]

    Spatialtracker: Tracking any 2d pixels in 3d space

    Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 5

    2024

  34. [44]

    Dynamicrafter: Animating open-domain images with video diffusion priors

    Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Xin- tao Wang, Tien-Tsin Wong, and Ying Shan. Dynamicrafter: Animating open-domain images with video diffusion priors. arXiv preprint arXiv:2310.12190, 2023. 3

    2023 arXiv

  35. [45]

    Direct-a-video: Customized video generation with user- directed camera movement and object motion.arXiv preprint arXiv:2402.03162, 2024

    Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user- directed camera movement and object motion.arXiv preprint arXiv:2402.03162, 2024. 3

    2024 arXiv

  36. [46]

    Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory, 2023

    Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory, 2023. 3, 7

    2023

  37. [47]

    Video probabilistic diffusion models in projected latent space

    Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 18456–18466,

  38. [48]

    Make pixels dance: High- dynamic video generation

    Yan Zeng, Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li. Make pixels dance: High- dynamic video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8850–8860, 2024. 3

    2024

  39. [49]

    I2vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models

    Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. I2vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint arXiv:2311.04145, 2023. 3

    2023 arXiv

  40. [50]

    Pia: Your personalized image animator via plug-and-play modules in text-to-image models

    Yiming Zhang, Zhening Xing, Yanhong Zeng, Youqing Fang, and Kai Chen. Pia: Your personalized image animator via plug-and-play modules in text-to-image models. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7747–7756, 2024. 3

    2024

  41. [51]

    Stereo magnification: Learning view syn- thesis using multiplane images

    Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view syn- thesis using multiplane images. In SIGGRAPH, 2018. 2, 6 10

    2018

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.