Pith. sign in

REVIEW 5 major objections 5 minor 57 references

AnyI2V: Animating Any Conditional Image with Motion Control

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AnyI2V claims a single frozen video diffusion model can animate any conditional image—depth, sketch, mesh, point cloud—along user-drawn trajectories, with no training.

desk verdict Useful training-free I2V method with an overblown headline and a fixable technical hole in the loss definition. read the letter →

arxiv 2507.02857 v1 pith:MKOJOG7P submitted 2025-07-03 cs.CV

Ziye Li , Hao Luo , Xincheng Shuai , Henghui Ding This is my paper
classification cs.CV
keywords training-freevideogenerationimage-to-videoanimationtrajectorycontrolfeatureinjectionPCAalignmentsemanticmaskconditionalimagediffusionmodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AnyI2V claims that a single frozen text-to-video diffusion model can turn any conditional image—a depth map, sketch, skeleton, segmentation map, mesh, or point cloud—into the first frame of a video, and then move the content along user-defined trajectories. The authors argue this is possible without any training: they extract features from the conditional image by DDIM inversion, inject a debiased subset of those features to control layout, and align PCA-reduced attention-query features across frames inside auto-generated semantic masks to enforce motion. If correct, this removes the need for paired training data that blocks trajectory control on unusual modalities, and it lets the same model switch between conditions and backbones at zero cost. The paper reports competitive FID, FVD, and object-motion error against trained methods, and notes that very large motion ranges and ambiguous occlusions remain difficult.

What carries the argument

The load-bearing mechanism is the cross-frame alignment of PCA-reduced attention-query features under a semantic mask. After DDIM inversion of the conditional image with the temporal module removed, features are extracted at timestep $t_\alpha = 201$; residual hidden states are debiased with patchified AdaIN to keep structure without leaking appearance, and the query is injected so the first frame defines the content. For motion, the query features from decoder up-blocks are reduced to their top $M = 64$ principal components, and the latents are optimized so that, within user-defined bounding boxes, each later frame's reduced query matches the stop-gradiented first-frame query (Eq. 8), masked by the overlap of binary K-Means clusters of cosine similarity to salient first-frame points. Optimization runs only for early sampling steps ($t' \ge 20$, every 5 steps), while later steps sample without gradient.

What would settle it

Take a point-cloud or mesh condition with a prescribed trajectory whose displacement exceeds half the frame width, run AnyI2V, and compare the tracked object trajectory against the target using the paper's ObjMC metric (lower is better; the reported full-method value is 16.39). If large translations or frame-to-frame occlusions cause the PCA-query alignment to diverge from the trajectory, the central premise of the method is not supported.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that the query features of spatial self-attention in a text-to-video diffusion model are temporally consistent and entity-aware across frames, which makes them a usable alignment target for zero-shot trajectory control. The authors show that attention maps drift over time and residual hidden features overrepresent fine appearance details, while the PCA-reduced query keeps the object coherent and traceable. They exploit this by optimizing the latent noise so that, inside user-specified bounding boxes, the high-ranked principal components of each later frame's query match those of the first frame, with a K-Means-based semantic mask restricting the optimization to the object. Combined with structure-preserving injection of debiased residual hidden features and first-frame key/value sharing, this yields a training-free pipeline that accepts depth, HED, canny, normal, segmentation, sketch, mesh, point cloud, and mixed inputs.

Load-bearing premise

The load-bearing premise is that the top principal components of the spatial self-attention query from the DDIM-inverted first frame remain temporally consistent and describe the object as one coherent entity, so aligning later frames to them yields the intended motion—a property the paper supports only with qualitative PCA visualizations on a few examples.

Editorial extensions

If this is right

  • Any modality that can be rendered as a single image—including ones with no paired training data, like meshes and point clouds—becomes a valid first-frame condition for a frozen video model, without retraining.
  • Mixed conditional inputs are supported in the same pass, so a depth map can anchor background structure while a sketch defines foreground detail.
  • LoRA and text prompts can edit the appearance of the conditional content during animation, adding style transfer and content editing without changing the motion-control mechanism.
  • Because the framework is training-free, porting it to a different text-to-video backbone requires no re-training; the authors demonstrate it on AnimateDiff, LaVie, and VideoCrafter2.
  • The method's own acknowledged limits are very large motion ranges and ambiguous occlusions, where spatial relationships can become unclear.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: If PCA-query temporal consistency is a general property of video diffusion U-Nets, the same alignment objective could be applied to interactive drag-style editing in videos and to other transformer-based video backbones, not just the three tested.
  • Inference: The semantic-mask clustering may fail when the object is heavily occluded or appears for the first time in later frames, because the mask is derived from first-frame cosine similarity; a testable extension is to refresh salient points per frame during optimization.
  • Inference: Because the query is aligned only in a PCA subspace, the method prioritizes global motion over fine articulation; a natural extension is multi-scale alignment on higher-order PCA components or a coarse-to-fine schedule for large displacements.
  • Inference: The framework turns any rendering (for example, 3D meshes or LiDAR point clouds) into animation, which could connect video diffusion with graphics pipelines, though that connection is not developed in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes AnyI2V, a training-free video generation framework built on AnimateDiff that accepts a single conditional image of arbitrary modality (depth, sketch, mesh, point cloud, etc.) and animates it along user-defined motion trajectories. The method performs DDIM inversion on the conditional image, injects debiased residual hidden features and attention queries from a chosen timestep into the generation U-Net, enforces cross-frame key/value consistency, and optimizes the latent so that PCA-reduced query features in user-provided bounding boxes match the first frame. A K-means-derived dynamic mask restricts the optimization to the target object. The authors report quantitative comparisons (Table 1), ablations, and generalization to other backbones.

Significance. The core idea of using PCA-reduced attention queries as a cross-frame alignment target is novel and potentially valuable: it yields a training-free method that accepts heterogeneous conditioning modalities and is portable across backbones. If the reported numbers were obtained under a clean protocol, the method would be a useful addition to controllable video generation. However, the evidence falls short of the 'superior performance' and 'any trajectory' claims, as detailed below.

major comments (5)
  1. [Table 1 / Abstract] The abstract's claim of 'superior performance' is contradicted by Table 1: the trained DragAnything baseline achieves better FID (95.83 vs. 104.53), FVD (556.09 vs. 569.89), and ObjMC (13.60 vs. 16.39) than AnyI2V. The claim should be revised to describe AnyI2V as competitive among training-free methods, or the comparison should be separated by training requirement with appropriate significance testing.
  2. [Fig. 2 caption / §4.2] The quantitative comparison is weakened by the protocol described in Fig. 2's caption, where the first frames for all comparison methods are generated by AnyI2V itself. Since DragAnything and DragNUWA are trained on real RGB first frames, evaluating them on AnyI2V-generated inputs introduces an uncontrolled confound. The paper also reports no error bars, number of evaluation videos, or random seeds, so the differences in Table 1 cannot be assessed for statistical significance.
  3. [Eq. (8) / §3.3] The loss in Eq. (8) multiplies M_i_1 and M_i_j elementwise, but §3.3 explicitly allows the bounding box size and position to differ per frame (H_i_j × W_i_j). No resizing, padding, or normalization step is described, so the product is undefined when the user specifies boxes of different sizes. Please specify how masks are aligned across frames.
  4. [Implementation Details / Table 1] Several key hyperparameters (t_alpha=201, M=64, p=4, the optimization schedule, and the choice of queries) appear to be selected using the same FID/FVD/ObjMC benchmark that is later used for the final comparison. This creates a circularity concern: the reported numbers may reflect tuning on the test set. A held-out validation set or a sensitivity analysis showing that results are stable across a range of hyperparameters would address this.
  5. [Limitations / §3.3] The manuscript's own Limitations section states that the method struggles with very large motion ranges and ambiguous occlusions, yet the title and abstract claim animation along arbitrary user-defined trajectories. Since the PCA-query alignment is the load-bearing mechanism for trajectory control and is validated only qualitatively (Fig. 5), the paper should quantify the failure regime, e.g., by reporting ObjMC binned by displacement magnitude or by the degree of occlusion, and adjust the 'any' claims accordingly.
minor comments (5)
  1. [Fig. 4 caption] The caption contains a typo: 'musuem' should be 'museum'.
  2. [§4.2] Please state the number of videos and trajectories in the evaluation set and whether ObjMC is averaged over all frames or only the final frame.
  3. [Eq. (5)] The notation F1[P k_i] does not clarify whether the same PCA transform is applied consistently to all frames; please specify that the projection matrix is computed once from the first-frame features and then applied to later frames.
  4. [Eq. (7)] The K-Means step uses random initialization; please specify the number of restarts or a deterministic initialization for reproducibility.
  5. [Figure 3] Figure 3 is difficult to read at the current resolution; please enlarge it or split it into separate subfigures.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the AnyI2V derivation is an independent feature-alignment optimization; evaluation-protocol caveats and the Limitations passage narrow the 'any trajectory' claim but do not make the derivation circular.

full rationale

The core derivation is self-contained rather than circular. AnyI2V's trajectory-control objective (Eq. 3, Eq. 4, Eq. 8) optimizes latents so that PCA-reduced spatial self-attention query features from later frames match a stop-gradient first-frame target inside user-defined bounding boxes and K-Means-derived semantic masks; this is a well-posed optimization objective, and none of these equations is defined in terms of the FID/FVD/ObjMC metrics used for evaluation. The choice of query features is justified by the qualitative PCA analysis in Fig. 5, which is an observation about intermediate diffusion features, not an input that already contains the output video. Hyperparameters such as t_alpha=201 and M=64 are selected in Sec. 4.3 using the same FID/FVD/ObjMC benchmark that Table 1 reports; while this is a test-set-tuning concern that weakens the quantitative claim, it is a validation-protocol flaw, not a derivation-level circularity, because the reported metrics are not by construction equal to the selection criterion. Similarly, Fig. 2's 'First Frame*' protocol, in which baseline condition images are 'generated using AnyI2V', is an unfair-comparison concern but does not make AnyI2V's output equivalent to its input. The Limitations section explicitly concedes that AnyI2V 'struggles with precise control of very large motion ranges and ambiguous occlusions', which narrows the 'any trajectory' claim but again is a scope limitation rather than a circular step. Self-citations ([3], [6], [35], [36]) are used for related work and metric support and are not load-bearing. One internal gap, the elementwise product M_i1 ⊙ M_i_j in Eq. 8, is undefined if per-frame bounding-box sizes differ as Sec. 3.3 permits, but this is an omitted implementation detail, not circularity. Overall, no step reduces an input to an output by definition, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central mechanism rests on several empirical assumptions about diffusion model feature semantics (Sec. 3.2, 3.3). No formal derivation or external validation of these assumptions is provided; they are supported by visualizations and ablations on the same benchmarks used for evaluation.

free parameters (5)
  • DDIM inversion feature timestep t_alpha = 201
    Chosen via ablation (Fig. 9) to balance structure and texture; directly affects first-frame layout control.
  • PCA reduction dimension M = 64
    Selected from ablation line chart (Fig. 8) that uses FID/FVD/ObjMC on the evaluation set; the alignment loss (Eq. 8) operates on M-component features.
  • Patch size p for patchified AdaIN = 4
    Set manually for debiasing residual hidden features (Eq. 1); no sensitivity analysis provided.
  • Latent optimization schedule = learning rate 0.01, every 5 steps for t' >= 20
    Hand-picked; no ablation or justification beyond implementation details.
  • Number of salient points K per bounding box = unspecified
    Used for semantic mask generation (Eq. 5); K is not given, so the mask generation is under-specified.
assumptions (4)
  • domain assumption DDIM inversion features at an intermediate timestep capture global layout while appearance is carried by noise or other features.
    Invoked in Sec. 3.2 to justify extracting features at t_alpha=201, based on prior observations [34,40] and Fig. 9 ablations.
  • domain assumption Attention query features are temporally consistent and entity-aware, unlike attention maps and residual hidden features.
    Established only by PCA visualizations in Fig. 5; this motivates aligning queries, not other features.
  • ad hoc to paper Replacing keys/values of subsequent frames with the first frame's K,V maintains cross-frame content coherence in spatial self-attention.
    Stated in Sec. 3.2 ('K2:f = K1 and V2:f = V1') without theoretical justification; validated implicitly by ablations.
  • domain assumption Gradient-based optimization of the latent with respect to the masked feature-matching loss (Eq. 8) produces valid video latents that follow the user trajectory.
    Standard practice from DragGAN/DragDiffusion, borrowed without proof; relies on the latent manifold assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AnyI2V: Animating Any Conditional Image with Motion Control." pith.science (2026). https://pith.science/paper/MKOJOG7P

@misc{pith2026250702857,
  author       = {Pith},
  title        = {Pith review of: AnyI2V: Animating Any Conditional Image with Motion Control},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MKOJOG7P}},
  note         = {Machine review of arXiv:2507.02857}
}
read the original abstract

Recent advancements in video generation, particularly in diffusion models, have driven notable progress in text-to-video (T2V) and image-to-video (I2V) synthesis. However, challenges remain in effectively integrating dynamic motion signals and flexible spatial constraints. Existing T2V methods typically rely on text prompts, which inherently lack precise control over the spatial layout of generated content. In contrast, I2V methods are limited by their dependence on real images, which restricts the editability of the synthesized content. Although some methods incorporate ControlNet to introduce image-based conditioning, they often lack explicit motion control and require computationally expensive training. To address these limitations, we propose AnyI2V, a training-free framework that animates any conditional images with user-defined motion trajectories. AnyI2V supports a broader range of modalities as the conditional image, including data types such as meshes and point clouds that are not supported by ControlNet, enabling more flexible and versatile video generation. Additionally, it supports mixed conditional inputs and enables style transfer and editing via LoRA and text prompts. Extensive experiments demonstrate that the proposed AnyI2V achieves superior performance and provides a new perspective in spatial- and motion-controlled video generation. Code is available at https://henghuiding.com/AnyI2V/.

Figures

Figures reproduced from arXiv: 2507.02857 by the authors.

Figure 1
Figure 1. The first frame conditional control of our Training-Free architecture AnyI2V. (a) AnyI2V supports diverse types of conditional [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison between AnyI2V and previous methods, DragAnything [ [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of Our Pipeline: Our pipeline begins by performing DDIM inversion on the conditional image. To do this, we remove the temporal module (i.e., temporal self-attention) from the 3D UNet and then extract features from its spatial blocks at timestep tα. Next, we optimize the latent representation by substituting the features from the first frame back into the U-Net. This optimization is constrained to a specific… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The study examines the influence of injecting different [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: This figure compares different PCA-reduced features in terms of their temporal consistency and entity representation. The results [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: The process of aligning the object across frames by [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: This picture demonstrates AnyI2V’s ability to control diverse conditions. AnyI2V can not only handle modalities that ControlNet [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: The line chart in the figure demonstrates the ablation study on the impact of different PCA reduction dimensions on FID, FVD, [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 28 canonical work pages

  1. [1]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 2, 3

    arXiv 2023

  2. [2]

    Align your latents: High-resolution video synthesis with latent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023. 3

    work page 2023

  3. [3]

    A unified 3d human motion synthesis model via conditional variational auto-encoder

    Yujun Cai, Yiwei Wang, Yiheng Zhu, Tat-Jen Cham, Jianfei Cai, Junsong Yuan, Jun Liu, Chuanxia Zheng, Sijie Yan, Henghui Ding, et al. A unified 3d human motion synthesis model via conditional variational auto-encoder. In ICCV,

    work page

  4. [4]

    Videocrafter1: Open diffusion models for high-quality video generation

    Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512, 2023. 2, 3

    arXiv

  5. [5]

    Videocrafter2: Overcoming data limitations for high-quality video diffusion models

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In CVPR, 2024. 8

    work page 2024

  6. [6]

    MeViS: A large-scale benchmark for video segmentation with motion expressions

    Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. MeViS: A large-scale benchmark for video segmentation with motion expressions. InICCV, 2023. 7

    work page 2023

  7. [7]

    Animatediff: Animate your personalized text- to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text- to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023. 2, 3, 6, 7

    arXiv 2023

  8. [8]

    Sparsectrl: Adding sparse controls to text-to-video diffusion models

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models. In ECCV, 2025. 2, 3, 7

    work page 2025

Show all 57 references
  1. [9]

    Cameractrl: Enabling camera control for text-to-video generation

    Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.02101, 2024. 3

    2024 arXiv

  2. [10]

    Latent video diffusion models for high-fidelity long video generation

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211.13221 ,

    arXiv

  3. [11]

    Prompt-to-prompt image editing with cross attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022. 3

    2022 arXiv

  4. [12]

    Video diffusion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. NeurIPS, 35, 2022. 2

    2022

  5. [13]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 1, 2

    2021 arXiv

  6. [14]

    Cocktail: Mixing multi-modality control for text-conditional image generation

    Minghui Hu, Jianbin Zheng, Daqing Liu, Chuanxia Zheng, Chaoyue Wang, Dacheng Tao, and Tat-Jen Cham. Cocktail: Mixing multi-modality control for text-conditional image generation. In NeurIPS, 2023. 3

    2023

  7. [15]

    Videocontrolnet: A motion-guided video-to-video translation framework by using diffusion model with controlnet

    Zhihao Hu and Dong Xu. Videocontrolnet: A motion-guided video-to-video translation framework by using diffusion model with controlnet. arXiv preprint arXiv:2307.14073 ,

    arXiv

  8. [16]

    Arbitrary style transfer in real-time with adaptive instance normalization

    Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV,

  9. [17]

    Cotracker: It is better to track together

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. In ECCV, 2024. 7

    2024

  10. [18]

    Text2video-zero: Text- to-image diffusion models are zero-shot video generators

    Levon Khachatryan, Andranik Movsisyan, Vahram Tade- vosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text- to-image diffusion models are zero-shot video generators. In ICCV, 2023. 2

    2023

  11. [19]

    Diffblender: Scalable and composable multimodal text-to-image diffusion models

    Sungnyun Kim, Junsoo Lee, Kibeom Hong, Daesik Kim, and Namhyuk Ahn. Diffblender: Scalable and composable multimodal text-to-image diffusion models. arXiv preprint arXiv:2305.15194, 2023. 3

    2023 arXiv

  12. [20]

    Compose and conquer: Diffusion-based 3d depth aware composable image synthe- sis

    Jonghyun Lee, Hansam Cho, Youngjoon Yoo, Seoung Bum Kim, and Yonghyun Jeong. Compose and conquer: Diffusion-based 3d depth aware composable image synthe- sis. arXiv preprint arXiv:2401.09048, 2024. 3

    2024 arXiv

  13. [21]

    Image conductor: Precision control for interactive video synthesis

    Yaowei Li, Xintao Wang, Zhaoyang Zhang, Zhouxia Wang, Ziyang Yuan, Liangbin Xie, Yuexian Zou, and Ying Shan. Image conductor: Precision control for interactive video synthesis. arXiv preprint arXiv:2406.15339, 2024. 6, 7

    2024 arXiv

  14. [22]

    Lovecon: Text-driven training-free long video editing with controlnet

    Zhenyi Liao and Zhijie Deng. Lovecon: Text-driven training-free long video editing with controlnet. arXiv preprint arXiv:2310.09711, 2023. 3

    2023 arXiv

  15. [23]

    Trailblazer: Trajectory control for diffusion-based video generation

    Wan-Duo Kurt Ma, John P Lewis, and W Bastiaan Kleijn. Trailblazer: Trajectory control for diffusion-based video generation. In SIGGRAPH Asia, 2024. 2, 6, 7

    2024

  16. [24]

    Some methods for classification and analysis of multivariate observations

    James MacQueen et al. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, 1967. 6

    1967

  17. [25]

    Large-scale video panoptic segmentation in the wild: A benchmark

    Jiaxu Miao, Xiaohan Wang, Yu Wu, Wei Li, Xu Zhang, Yunchao Wei, and Yi Yang. Large-scale video panoptic segmentation in the wild: A benchmark. In CVPR, 2022. 7

    2022

  18. [26]

    Freecontrol: Training-free spatial control of any text-to-image diffusion model with any condition

    Sicheng Mo, Fangzhou Mu, Kuan Heng Lin, Yanli Liu, Bochen Guan, Yin Li, and Bolei Zhou. Freecontrol: Training-free spatial control of any text-to-image diffusion model with any condition. In CVPR, 2024. 3

    2024

  19. [27]

    Sg-i2v: Self-guided trajectory control in image-to-video generation

    Koichi Namekata, Sherwin Bahmani, Ziyi Wu, Yash Kant, Igor Gilitschenski, and David B Lindell. Sg-i2v: Self-guided trajectory control in image-to-video generation. arXiv preprint arXiv:2411.04989, 2024. 3

    2024 arXiv

  20. [28]

    Mofa-video: Controllable image animation via generative motion field adaptions in frozen image-to-video diffusion model

    Muyao Niu, Xiaodong Cun, Xintao Wang, Yong Zhang, Ying Shan, and Yinqiang Zheng. Mofa-video: Controllable image animation via generative motion field adaptions in frozen image-to-video diffusion model. In ECCV, 2024. 3, 6, 7 9

    2024

  21. [29]

    Drag your gan: Interactive point-based manipulation on the generative image manifold

    Xingang Pan, Ayush Tewari, Thomas Leimk ¨uhler, Lingjie Liu, Abhimitra Meka, and Christian Theobalt. Drag your gan: Interactive point-based manipulation on the generative image manifold. In ACM SIGGRAPH, 2023. 3, 5

    2023

  22. [30]

    Unicontrol: A unified diffusion model for controllable visual generation in the wild

    Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, et al. Unicontrol: A unified diffusion model for controllable visual generation in the wild. arXiv preprint arXiv:2305.11147, 2023. 3

    2023 arXiv

  23. [31]

    Freetraj: Tuning-free trajectory control in video diffusion models

    Haonan Qiu, Zhaoxi Chen, Zhouxia Wang, Yingqing He, Menghan Xia, and Ziwei Liu. Freetraj: Tuning-free trajectory control in video diffusion models. arXiv preprint arXiv:2406.16863, 2024. 3, 6, 7

    2024 arXiv

  24. [32]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022. 2

    2022

  25. [33]

    Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling

    Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, et al. Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling. In ACM SIGGRAPH, 2024. 2, 3

    2024

  26. [34]

    Dragdiffusion: Harnessing diffusion models for interactive point-based image editing

    Yujun Shi, Chuhui Xue, Jun Hao Liew, Jiachun Pan, Hanshu Yan, Wenqing Zhang, Vincent YF Tan, and Song Bai. Dragdiffusion: Harnessing diffusion models for interactive point-based image editing. In CVPR, 2024. 3, 4, 5

    2024

  27. [35]

    A survey of multimodal-guided image editing with text-to-image diffusion models

    Xincheng Shuai, Henghui Ding, Xingjun Ma, Rongcheng Tu, Yu-Gang Jiang, and Dacheng Tao. A survey of multimodal-guided image editing with text-to-image diffusion models. arXiv preprint arXiv:2406.14555 , 2024. 2

    2024 arXiv

  28. [36]

    Free-form motion control: A synthetic video generation dataset with controllable camera and object motions

    Xincheng Shuai, Henghui Ding, Zhenyuan Qin, Hao Luo, Xingjun Ma, and Dacheng Tao. Free-form motion control: A synthetic video generation dataset with controllable camera and object motions. In ICCV, 2025. 3

    2025

  29. [37]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 ,

    arXiv

  30. [38]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 4, 5

    2010 arXiv

  31. [39]

    Anycontrol: create your artwork with versatile control on text-to-image generation

    Yanan Sun, Yanchen Liu, Yinhao Tang, Wenjie Pei, and Kai Chen. Anycontrol: create your artwork with versatile control on text-to-image generation. In ECCV, 2025. 3

    2025

  32. [40]

    Plug-and-play diffusion features for text-driven image-to-image translation

    Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In CVPR, 2023. 3, 4

    2023

  33. [41]

    Modelscope text-to-video technical report

    Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06571, 2023. 3

    2023 arXiv

  34. [42]

    Boximator: Generating rich and controllable motions for video synthesis

    Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guoqiang Wei, Liping Yuan, and Hang Li. Boximator: Generating rich and controllable motions for video synthesis. arXiv preprint arXiv:2402.01566, 2024. 2, 3

    2024 arXiv

  35. [43]

    Videocomposer: Compositional video synthesis with motion controllability

    Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. NeurIPS, 36, 2024. 3

    2024

  36. [44]

    Lavie: High-quality video generation with cascaded latent diffusion models

    Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. IJCV, 2024. 2, 3, 8

    2024

  37. [45]

    Objctrl-2.5 d: Training-free object control with camera poses

    Zhouxia Wang, Yushi Lan, Shangchen Zhou, and Chen Change Loy. Objctrl-2.5 d: Training-free object control with camera poses. arXiv preprint arXiv:2412.07721, 2024. 6, 7

    2024 arXiv

  38. [46]

    Motionctrl: A unified and flexible motion controller for video generation

    Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. In ACM SIGGRAPH, 2024. 2, 3

    2024

  39. [47]

    Principal component analysis

    Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1-3), 1987. 2

    1987

  40. [48]

    Motionbooth: Motion-aware customized text-to-video generation

    Jianzong Wu, Xiangtai Li, Yanhong Zeng, Jiangning Zhang, Qianyu Zhou, Yining Li, Yunhai Tong, and Kai Chen. Motionbooth: Motion-aware customized text-to-video generation. arXiv preprint arXiv:2406.17758, 2024. 2

    2024 arXiv

  41. [49]

    Draganything: Motion control for anything using entity representation

    Weijia Wu, Zhuang Li, Yuchao Gu, Rui Zhao, Yefei He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, and Di Zhang. Draganything: Motion control for anything using entity representation. In ECCV, 2024. 3, 6, 7

    2024

  42. [50]

    Video diffusion models are training-free motion interpreter and controller

    Zeqi Xiao, Yifan Zhou, Shuai Yang, and Xingang Pan. Video diffusion models are training-free motion interpreter and controller. arXiv preprint arXiv:2405.14864, 2024. 3

    2024 arXiv

  43. [51]

    Dynamicrafter: Animating open-domain images with video diffusion priors

    Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Dynamicrafter: Animating open-domain images with video diffusion priors. In ECCV,

  44. [52]

    Direct-a-video: Customized video generation with user- directed camera movement and object motion

    Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user- directed camera movement and object motion. In ACM SIGGRAPH, 2024. 2, 3

    2024

  45. [53]

    Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory

    Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308.08089, 2023. 2, 3, 6, 7

    2023 arXiv

  46. [54]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023. 2, 3

    2023

  47. [55]

    Controlvideo: Training-free controllable text-to-video generation

    Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Controlvideo: Training-free controllable text-to-video generation. arXiv preprint arXiv:2305.13077, 2023. 2, 3

    2023 arXiv

  48. [56]

    Tora: Trajectory-oriented diffusion transformer for video generation

    Zhenghao Zhang, Junchao Liao, Menghao Li, Zuozhuo Dai, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang. Tora: Trajectory-oriented diffusion transformer for video generation. arXiv preprint arXiv:2407.21705, 2024. 3

    2024 arXiv

  49. [57]

    Trackgo: A flexible and efficient method for controllable video generation

    Haitao Zhou, Chuang Wang, Rui Nie, Jinxiao Lin, Dongdong Yu, Qian Yu, and Changhu Wang. Trackgo: A flexible and efficient method for controllable video generation. arXiv preprint arXiv:2408.11475, 2024. 3 10

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.