Pith. sign in

REVIEW 4 major objections 5 minor 54 references

MotiF: Making Text Count in Image Animation with Motion Focal Loss

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MotiF reweights the diffusion loss by an optical-flow motion heatmap so the model concentrates on moving regions, and on a new 320-pair benchmark it beats nine open-sourced TI2V models with a 72% average human preference.

desk verdict A simple, well-tested loss reweighting for text-image-to-video that genuinely improves motion generation; the new benchmark is useful, but the headline human-preference evidence needs confidence intervals and a flow-blind follow-up. read the letter →

arxiv 2412.16153 v2 pith:KKOG6OJW submitted 2024-12-20 cs.CV cs.AI

Shijie Wang , Samaneh Azadi , Rohit Girdhar , Saketh Rambhatla , Chen Sun , Xi Yin This is my paper · ORCID
classification cs.CVcs.AI
keywords text-image-to-videogenerationimageanimationmotionfocallossopticalflowheatmapdiffusionmodeltrainingtextalignmenthumanevaluationbenchmarkTI2V-Bench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Text-guided image animation struggles because most of a video is static: in a typical training clip 97% of pixels do not move, so an equal-weight loss lets the model win by barely moving anything and ignoring the prompt. MotiF attacks this at the training objective: it computes an optical-flow heatmap of each video and multiplies the diffusion denoising error by that heatmap, forcing the model to spend its capacity on regions the text should be animating. On a new benchmark of 320 image-text pairs spanning 22 scenarios, with human A/B preference as the score, the approach beats nine open-sourced TI2V models with a 72% average preference, and the justifications show the gains are concentrated in exactly the target dimensions: text alignment and object motion. The paper also contributes TI2V-Bench and a justification-based human evaluation protocol for future comparisons.

What carries the argument

The motion focal loss: a motion heatmap $m\in[0,1]^{L\times H\times W}$ is built from RAFT optical flow between consecutive frames, normalized by a sigmoid $\sigma(x)=1/(1+e^{100(0.05-x)})$ to polarize high- and low-motion pixels, downsampled to $m'$ to match the video latents, and multiplied elementwise into the standard noise-prediction MSE of Eq. (1) to give Eq. (2). This reweighting is what carries the argument: it converts the imbalance between static background and moving foreground into a learning signal, without requiring any extra input at inference time. The other load-bearing design choice is image conditioning by pure concatenation (x-cat) rather than cross-attention, which the paper argues avoids competition between image and text features.

What would settle it

Curate a held-out set of prompts whose intended change is appearance-based rather than displacement-based (a light flashing, a balloon changing color, an object appearing), where ground-truth optical flow is near zero; if MotiF fails to improve text alignment over its baseline on that set, the flow-magnitude proxy is the failing link.

Watch

Extended reading notes

Core claim

The central claim is that a simple per-pixel reweighting of the diffusion loss, $L_{\text{motif}} = \mathbb{E}_{t,x,\epsilon}\|m' \cdot (\epsilon - \epsilon_\theta)\|_2^2$, where $m'$ is the optical-flow-derived motion heatmap downsampled to the latent grid, makes a TI2V model follow text-driven motion instead of collapsing toward the static first frame. Trained on top of a pretrained text-to-video model with the heatmap loss added at unit weight, the model wins 72% of pairwise human comparisons against nine open-sourced baselines on TI2V-Bench, and ablation against an inverse-motion reweighting shows that it is specifically the high-motion emphasis that helps. The paper further claims that concatenating the conditioning image with the noisy latent (x-cat) is the best image-injection choice, outperforming cross-attention alone and the combination, because cross-attention competes with text features.

Load-bearing premise

The method assumes that the pixels with the largest optical flow in a training video are precisely the pixels the text prompt wants animated, which fails for flashes, color changes, and newly appearing objects.

Editorial extensions

If this is right

  • If the claim holds, any TI2V model trained with the motion-weighted loss should show improved text alignment and object motion with no extra inference-time inputs.
  • The loss is complementary to methods that feed motion scores or masks into the model, so combining both is a direct next step the paper leaves open.
  • The ablation with the inverse-motion weight implies that weighting the loss toward static regions actively hurts text-driven animation, so the direction of the reweighting is essential, not just the reweighting itself.
  • The TI2V-Bench protocol, with multiple prompts per image and forced-choice human justification, gives the community a single TI2V score plus per-axis diagnostics for future model comparisons.
  • On automatic metrics, MotiF is comparable to prior models, and the paper's static-video baseline result shows those metrics reward stillness; human preference is therefore the metric that reveals the improvement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: replacing the optical-flow heatmap with a text-conditioned saliency map (for example, from attention or segmentation) could extend MotiF to appearance-only changes such as flashes or new objects, which are the failure modes the paper reports.
  • Editorial inference: because the loss operates on the objective rather than the inputs, it should stack with conditional-image weakening and motion-prior injection; the gains could compound beyond what the 72% single-model result shows.
  • Editorial inference: the benchmark's finding that a static video tops image-alignment and text-alignment automatic scores suggests a motion-aware automatic metric is needed; one could score text-relevant motion regions rather than whole-frame similarity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Motion Focal Loss (MotiF) for text-image-to-video (TI2V) generation. The method computes an optical-flow-derived motion heatmap from ground-truth training videos and reweights the standard diffusion denoising loss by this heatmap, so that the model concentrates on high-motion regions during training. The authors also introduce TI2V-Bench, a new benchmark of 320 image-text pairs across 22 scenarios, and a human evaluation protocol based on forced A-B preference with justification categories. They report that MotiF outperforms nine open-sourced TI2V methods with an average 72% human preference on TI2V-Bench, with the largest gains in text alignment and object motion.

Significance. If the claims hold, MotiF is a simple, training-time-only modification to the diffusion objective that is orthogonal to input-based motion priors and requires no extra conditioning at inference. The proposed benchmark and the justification-based human evaluation protocol are potentially useful resources for the community. The paper also provides a controlled ablation isolating the motion focal loss (Table 2) and discloses hyperparameter choices in the appendix. However, the significance depends on whether the optical-flow proxy for text-relevant motion is valid beyond flow-friendly scenarios, and on the statistical reliability of the reported human preferences.

major comments (4)
  1. [§3.2, Eq. (2), Appendix C.3, Fig. A3] The load-bearing premise of the method is that pixels with large optical flow are the pixels whose content is specified by the text prompt. This premise fails for appearance changes (e.g., 'the red light bulb flashing', Fig. A3d) and for new objects entering the scene (e.g., 'a dolphin blowing bubbles', Fig. A3c), where text-relevant change is photometric or involves no correspondence, and the flow-based heatmap assigns low weight to exactly those pixels. The paper itself lists these as failure cases in Appendix C.3, yet the headline 72% preference is aggregated over all prompts, including these difficult cases. To support the claim that MotiF improves text alignment, please report per-category or per-scenario preference results on TI2V-Bench (e.g., split by translation-type motion, appearance change, and new-object introduction), or evaluate on an additional benchmark enriched with such prompts, or extend the heatmap to incorporate photometric change (e.g., frame difference or segmentation masks). Without this, the central claim that the loss improves text-guided motion generation is not established for a substantial class of text-relevant changes.
  2. [§5.2, Figure 4] The human evaluation results are reported as preference percentages without confidence intervals, significance tests, or inter-annotator agreement statistics. With only 5 annotators per comparison and majority voting, the difference between MotiF and several baselines (e.g., 72% vs. 68.8% for VideoCrafter, and 72% vs. 72.5% for AnimateAnything) could be within noise. Please provide bootstrap confidence intervals, a paired significance test (e.g., Wilcoxon signed-rank on the per-item majority votes), or at least raw vote counts and agreement rates. This is essential because the central evidence for the method's effectiveness rests entirely on these human preference numbers.
  3. [Appendix A.2, Table A2] The motion focal loss weight λ was selected by evaluating on TI2V-Bench (Table A2), which is the same benchmark used for the headline comparisons in Figure 4. This creates a selection-bias risk: the reported 72% may be inflated by tuning λ on the test set. Please either fix λ a priori, or select it on a separate validation split and report the test results for that selection. At minimum, state explicitly how many hyperparameter configurations were tried and how the final λ=1 was chosen.
  4. [§5.2, Table 4, and Section 6] The paper claims that MotiF 'complements existing techniques that utilize motion priors as model inputs.' However, the experiments only compare MotiF against released checkpoints of prior methods used out-of-the-box, without fine-tuning any baseline on the same training data or with the same image-conditioning setup. The claim of complementarity would require a direct experiment, e.g., adding MotiF to a motion-prior-based method or fine-tuning a strong baseline under identical conditions (same data, same x-cat injection). Without such an experiment, the complementarity claim is not supported by the presented evidence.
minor comments (5)
  1. [Section 6] The phrase 'we hypothesis that' should be 'we hypothesize that'.
  2. [Abstract and Section 4.1] The phrase 'a dataset consists of 320 image-text pairs' is better as 'a dataset consisting of 320 image-text pairs' or 'a dataset that consists of 320 image-text pairs'.
  3. [Section 3.2, Motion Heatmaps] The sentence 'We then apply a sigmoid-like function fl to normalize the intensity map' is awkward; 'fl' is the intensity map, so please rephrase to 'apply a sigmoid-like function to the intensity map fl'.
  4. [Figure 4] The stacked bar chart is visually dense and difficult to compare across methods; consider a grouped bar chart or a table that separates the overall TI2V score from the justification percentages.
  5. [Section 5.1] The training data is described only as an 'internal licensed dataset of 1M video-text pairs that is similar to [46]'. Since the dataset is not public, improving reproducibility would benefit from a more detailed description of the data composition, filtering, or licensing constraints.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the MotiF objective is an explicit reweighting of the diffusion loss by ground-truth optical flow, and its central evidence is external human preference with controlled ablations.

full rationale

The derivation chain is self-contained. Equation (2) defines L_motif as the diffusion error weighted by a motion heatmap m' computed from RAFT optical flow between ground-truth frames (Section 3.2), and the paper never fits a parameter to the headline evaluation and then renames that fit a prediction. The central claim that MotiF improves text alignment and motion quality is supported by an external signal: human A/B preference judgments on TI2V-Bench (Section 5.2, Figure 4), plus ablations that hold all training settings constant (w/o MotiF and Inv-MotiF in Table 2). The flow-magnitude-as-text-motion assumption is an explicit modeling choice with acknowledged failure modes for new-object and multi-object prompts (Appendix C.3, Figure A3), which shows the mechanism is not tautologically successful. The choice of lambda on TI2V-Bench (Table A2) is a mild benchmark-selection risk, but it does not make the central result true by construction. The self-citation to Emu Video [15] for the JUICE-style evaluation protocol is methodological and not load-bearing. No equation or claim reduces to its own input, so there is no circular step.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central claim rests on a standard latent diffusion training pipeline, a pretrained T2V model (VideoCrafter2), an internal 1M video-text dataset, RAFT optical flow, and human annotations. The hand-tuned constants are lambda and the sigmoid normalization parameters; no new entities are postulated.

free parameters (3)
  • lambda (motion focal loss weight) = 1
    Set in Eq. (3) as L = L_diffusion + lambda * L_motif; lambda=1 chosen after ablations on TI2V-Bench (Table A2) and used for the headline 72% result.
  • Sigmoid threshold in heatmap normalization = 0.05
    sigma(x)=1/(1+e^{100(0.05-x)}) in Section 5.1; hand-chosen, not ablated, controls which flow magnitudes count as motion.
  • Sigmoid sharpness in heatmap normalization = 100
    Same formula; the large exponent makes the heatmap near-binary. Hand-chosen and not ablated.
assumptions (6)
  • standard math Denoising diffusion probabilistic models and the MSE training objective (Section 3.1, Eq. 1) are valid.
    Background assumption for the entire training pipeline.
  • standard math Latent video diffusion with a pretrained VAE is appropriate (Section 3.1).
    Standard architecture choice; the paper builds on VideoCrafter2.
  • domain assumption VideoCrafter2 is a strong, representative T2V prior for building TI2V models (Section 3.2).
    The method initializes from this pretrained model; if this prior is weak, results may not generalize.
  • domain assumption The internal 1M video-text dataset is similar to the one used in DynamiCrafter [46] and is diverse enough for training (Section 5.1).
    Training data is not released; the claim that the method works on open-domain TI2V rests on this dataset's representativeness.
  • domain assumption RAFT optical flow provides reliable motion intensity maps for training videos (Section 3.2).
    Errors in optical flow would misdirect the loss reweighting.
  • domain assumption Human annotators' forced-choice preference on TI2V-Bench is a valid measure of TI2V quality (Section 4.2).
    The headline result is a human preference metric; if the protocol is biased, the main evidence weakens.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MotiF: Making Text Count in Image Animation with Motion Focal Loss." pith.science (2026). https://pith.science/paper/KKOG6OJW

@misc{pith2026241216153,
  author       = {Pith},
  title        = {Pith review of: MotiF: Making Text Count in Image Animation with Motion Focal Loss},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KKOG6OJW}},
  note         = {Machine review of arXiv:2412.16153}
}
read the original abstract

Text-Image-to-Video (TI2V) generation aims to generate a video from an image following a text description, which is also referred to as text-guided image animation. Most existing methods struggle to generate videos that align well with the text prompts, particularly when motion is specified. To overcome this limitation, we introduce MotiF, a simple yet effective approach that directs the model's learning to the regions with more motion, thereby improving the text alignment and motion generation. We use optical flow to generate a motion heatmap and weight the loss according to the intensity of the motion. This modified objective leads to noticeable improvements and complements existing methods that utilize motion priors as model inputs. Additionally, due to the lack of a diverse benchmark for evaluating TI2V generation, we propose TI2V Bench, a dataset consists of 320 image-text pairs for robust evaluation. We present a human evaluation protocol that asks the annotators to select an overall preference between two videos followed by their justifications. Through a comprehensive evaluation on TI2V Bench, MotiF outperforms nine open-sourced models, achieving an average preference of 72%. The TI2V Bench and additional results are released in https://wang-sj16.github.io/motif/.

Figures

Figures reproduced from arXiv: 2412.16153 by the authors.

Figure 1
Figure 1. Motivation and results of MotiF. (a) Example video frames and the corresponding motion heatmaps calculated from optical flow. In this example, 97% of the pixels are static while only 3% has meaningful motion. (b) In standard TI2V training pipeline, the model may learn to over-rely on the conditional image to optimize the L2 loss. This issue has been identified in [53] and termed as conditional image leakage. We prop… view at source ↗
Figure 2
Figure 2. High-level comparisons of MotiF vs. prior works. Previous TI2V generation methods mainly focused on deriving additional motion signals (motion score and/or motion mask) as inputs for the model to leverage implicitly. On the contrary, we focus on the learning objective and propose to weight the diffusion loss based on the motion intensity, that is derived from optical flow. Our method is simple, effective, and does n… view at source ↗
Figure 3
Figure 3. Example image-text pairs in TI2V-Bench. For each scenario (column), we first think of a scene that could be potentially animated to generate different types of motion. We include challenging scenarios when there are multiple objects (yellow/blue/red balloon) in the initial image for fine-grained control or the text prompt describes a new object (frisbee, bubbles) to enter the scene. Then we come up with different pr… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Human evaluation results comparing MotiF to nine open-sourced models [7, 9, 12, 25, 28, 33, 46, 50, 53] on TI2V-Bench. We achieved considerable improvements across the board with an average preference of 72%. Through examining the justification choices, we found that o…
Figure 5
Figure 5. Figure 5: Qualitative comparison to prior works on TI2V-Bench. Sampled frames are ordered from left to right. prompts, even on challenging cases with multiple objects in the scene with fine-grained motion animation. I2VGen-XL and VideoCrafter generate videos that do not align we…
Figure 6
Figure 6. Figure 6: Loss comparison. We calculate the ratio of the average loss in the high motion region to the average overall loss on a hold￾out validation set with different timesteps. MotiF can effectively reduce the relative loss of the high motion regions. to mainly rely on human e…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 17 canonical work pages

  1. [1]

    Latent-shift: Latent diffu- sion with temporal shift for efficient text-to-video genera- tion

    Jie An, Songyang Zhang, Harry Yang, Sonal Gupta, Jia-Bin Huang, Jiebo Luo, and Xi Yin. Latent-shift: Latent diffu- sion with temporal shift for efficient text-to-video genera- tion. arXiv preprint arXiv:2304.08477, 2023. 2, 3

    arXiv 2023

  2. [2]

    Frozen in time: A joint video and image encoder for end-to-end retrieval

    Max Bain, Arsha Nagrani, G ¨ul Varol, and Andrew Zisser- man. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 1728–1738,

  3. [3]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 2, 3

    arXiv 2023

  4. [4]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023. 3

    2023

  5. [5]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators

  6. [6]

    Animat- ing general image with large visual motion model

    Dengsheng Chen, Xiaoming Wei, and Xiaolin Wei. Animat- ing general image with large visual motion model. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7131–7140, 2024. 2

    work page 2024

  7. [7]

    Videocrafter1: Open diffusion models for high-quality video generation

    Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512, 2023. 7

    arXiv

  8. [8]

    Videocrafter2: Overcoming data limitations for high-quality video diffusion models, 2024

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models, 2024. 4, 5, 6, 8

    work page 2024

Show all 54 references
  1. [9]

    Seine: Short-to-long video diffu- sion model for generative transition and prediction

    Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffu- sion model for generative transition and prediction. In The Twelfth International Conference on Learning Representa- ti...

    2023

  2. [10]

    Livephoto: Real image animation with text-guided motion control

    Xi Chen, Zhiheng Liu, Mengting Chen, Yutong Feng, Yu Liu, Yujun Shen, and Hengshuang Zhao. Livephoto: Real image animation with text-guided motion control. In Eu- ropean Conference on Computer Vision , pages 475–491. Springer, 2025. 2, 3

    2025

  3. [11]

    Emu: Enhanc- ing image generation models using photogenic needles in a haystack

    Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xi- aofang Wang, Abhimanyu Dubey, et al. Emu: Enhanc- ing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.15807, 2023. 2

    2023 arXiv

  4. [12]

    Animateanything: Fine- grained open domain image animation with motion guid- ance

    Zuozhuo Dai, Zhenghao Zhang, Yao Yao, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang. Animateanything: Fine- grained open domain image animation with motion guid- ance. arXiv e-prints, pages arXiv–2311, 2023. 3, 7, 8

    2023

  5. [13]

    Aigcbench: Comprehensive evaluation of image-to-video content generated by ai

    Fanda Fan, Chunjie Luo, Wanling Gao, and Jianfeng Zhan. Aigcbench: Comprehensive evaluation of image-to-video content generated by ai. arXiv preprint arXiv:2401.01651 ,

    arXiv

  6. [14]

    Preserve your own correlation: A noise prior for video diffusion models

    Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming- Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. In Proceedings of the IEEE/CVF International Conference on Comp...

    2023

  7. [15]

    Emu video: Factoriz- ing text-to-video generation by explicit image conditioning

    Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Du- val, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factoriz- ing text-to-video generation by explicit image conditioning. ECCV, 2024. 2, 3, 6

    2024

  8. [16]

    I2v-adapter: A general image-to-video adapter for diffusion models

    Xun Guo, Mingwu Zheng, Liang Hou, Yuan Gao, Yufan Deng, Pengfei Wan, Di Zhang, Yufan Liu, Weiming Hu, Zhengjun Zha, et al. I2v-adapter: A general image-to-video adapter for diffusion models. In ACM SIGGRAPH 2024 Conference Papers, pages 1–12, 2024. 3

    2024

  9. [17]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 2, 3

    2020

  10. [18]

    Video dif- fusion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video dif- fusion models. Advances in Neural Information Processing Systems, 35:8633–8646, 2022. 2

    2022

  11. [19]

    Make it move: Controllable image-to-video generation with text descrip- tions

    Yaosi Hu, Chong Luo, and Zhenzhong Chen. Make it move: Controllable image-to-video generation with text descrip- tions. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 18219– 18228, 2022. 2

    2022

  12. [20]

    VBench: Com- prehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Com- prehensive benchmark suite for video generative models....

    2024

  13. [21]

    Vbench: Comprehensive bench- mark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive bench- mark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and P...

    2024

  14. [22]

    Videogen: A reference-guided latent diffusion ap- proach for high definition text-to-video generation

    Xin Li, Wenqing Chu, Ye Wu, Weihang Yuan, Fanglong Liu, Qi Zhang, Fu Li, Haocheng Feng, Errui Ding, and Jingdong Wang. Videogen: A reference-guided latent diffusion ap- proach for high definition text-to-video generation. arXiv preprint arXiv:2309.00398, 2023. 2

    2023 arXiv

  15. [23]

    Physgen: Rigid-body physics-grounded image- to-video generation

    Shaowei Liu, Zhongzheng Ren, Saurabh Gupta, and Shen- long Wang. Physgen: Rigid-body physics-grounded image- to-video generation. In European Conference on Computer Vision, pages 360–378. Springer, 2025. 2

    2025

  16. [24]

    Evalcrafter: Benchmarking and eval- uating large video generation models

    Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and eval- uating large video generation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern R...

    2024

  17. [25]

    Cinemo: Consis- tent and controllable image animation with motion diffusion models

    Xin Ma, Yaohui Wang, Gengyu Jia, Xinyuan Chen, Yuan- Fang Li, Cunjian Chen, and Yu Qiao. Cinemo: Consis- tent and controllable image animation with motion diffusion models. arXiv preprint arXiv:2407.15642, 2024. 2, 3, 7, 8

    2024 arXiv

  18. [26]

    Follow-your-click: Open- domain regional image animation via short prompts

    Yue Ma, Yingqing He, Hongfa Wang, Andong Wang, Chenyang Qi, Chengfei Cai, Xiu Li, Zhifeng Li, Heung- Yeung Shum, Wei Liu, et al. Follow-your-click: Open- domain regional image animation via short prompts. arXiv preprint arXiv:2403.08268, 2024. 2, 3

    2024 arXiv

  19. [27]

    Sync-draw: Automatic video generation using deep recurrent attentive architectures

    Gaurav Mittal, Tanya Marwah, and Vineeth N Balasubrama- nian. Sync-draw: Automatic video generation using deep recurrent attentive architectures. In Proceedings of the 25th ACM international conference on Multimedia , pages 1096– 1104, 2017. 2

    2017

  20. [28]

    Ti2v-zero: Zero-shot image condition- ing for text-to-video diffusion models

    Haomiao Ni, Bernhard Egger, Suhas Lohit, Anoop Cherian, Ye Wang, Toshiaki Koike-Akino, Sharon X Huang, and Tim K Marks. Ti2v-zero: Zero-shot image condition- ing for text-to-video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2024

  21. [29]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  22. [30]

    Movie gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720,

    arXiv

  23. [31]

    Hier- archical spatio-temporal decoupling for text-to-video gener- ation

    Zhiwu Qing, Shiwei Zhang, Jiayu Wang, Xiang Wang, Yujie Wei, Yingya Zhang, Changxin Gao, and Nong Sang. Hier- archical spatio-temporal decoupling for text-to-video gener- ation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 6635–6645,

  24. [32]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junt- ing Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Doll´ar, and Christoph Feic...

    2024 arXiv

  25. [33]

    Consisti2v: Enhanc- ing visual consistency for image-to-video generation

    Weiming Ren, Harry Yang, Ge Zhang, Cong Wei, Xinrun Du, Stephen Huang, and Wenhu Chen. Consisti2v: Enhanc- ing visual consistency for image-to-video generation. arXiv preprint arXiv:2402.04324, 2024. 3, 5, 7, 8

    2024 arXiv

  26. [34]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 4

    2022

  27. [35]

    Focal loss for dense ob- ject detection

    T-YLPG Ross and GKHP Doll ´ar. Focal loss for dense ob- ject detection. In proceedings of the IEEE conference on computer vision and pattern recognition, pages 2980–2988,

  28. [36]

    Progressive distillation for fast sampling of diffusion models

    Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022. 6

    2022 arXiv

  29. [37]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 ,

    arXiv

  30. [38]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International confer- ence on machine learning, pages 2256–2265. PMLR, 2015. 3

    2015

  31. [39]

    Denois- ing diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois- ing diffusion implicit models. In International Conference on Learning Representations, 2021. 6

    2021

  32. [40]

    Score-based generative modeling through stochastic differential equa- tions

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equa- tions. arXiv preprint arXiv:2011.13456, 2020. 3

    2011 arXiv

  33. [41]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    K Soomro. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 ,

    arXiv

  34. [42]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Proceedings, Part II 16, pages 402–419. Springer,

    2020

  35. [43]

    Microcinema: A divide-and- conquer approach for text-to-video generation

    Yanhui Wang, Jianmin Bao, Wenming Weng, Ruoyu Feng, Dacheng Yin, Tao Yang, Jingxu Zhang, Qi Dai, Zhiyuan Zhao, Chunyu Wang, et al. Microcinema: A divide-and- conquer approach for text-to-video generation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Patte...

    2024

  36. [44]

    Dreamvideo: Composing your dream videos with customized subject and motion

    Yujie Wei, Shiwei Zhang, Zhiwu Qing, Hangjie Yuan, Zhi- heng Liu, Yu Liu, Yingya Zhang, Jingren Zhou, and Hong- ming Shan. Dreamvideo: Composing your dream videos with customized subject and motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog...

    2024

  37. [45]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference...

    2023

  38. [46]

    Dynamicrafter: Animating open-domain images with video diffusion priors

    Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Dynamicrafter: Animating open-domain images with video diffusion priors. In Eu- ropean Conference on Computer Vision , pages 399–417. Springer, 202...

    2025

  39. [47]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288–5296, 2016. 3, 5

    2016

  40. [48]

    Motion-conditioned image animation for video editing

    Wilson Yan, Andrew Brown, Pieter Abbeel, Rohit Girdhar, and Samaneh Azadi. Motion-conditioned image animation for video editing. arXiv preprint arXiv:2311.18827, 2023. 3

    2023 arXiv

  41. [49]

    Zero-shot controllable image-to-video animation via motion decomposition

    Shoubin Yu, Jacob Zhiyuan Fang, Skyler Zheng, Gunnar Sigurdsson, Vicente Ordonez, Robinson Piramuthu, and Mo- hit Bansal. Zero-shot controllable image-to-video animation via motion decomposition. 2024. 2

    2024

  42. [50]

    I2vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models

    Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. I2vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint arXiv:2311.04145, 2023. 2, 7, 8

    2023 arXiv

  43. [51]

    Pia: Your personalized image animator via plug-and-play modules in text-to-image models

    Yiming Zhang, Zhening Xing, Yanhong Zeng, Youqing Fang, and Kai Chen. Pia: Your personalized image animator via plug-and-play modules in text-to-image models. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7747–7756, 2024. 2, 3, 5, 7, 8

    2024

  44. [52]

    Benchmarking aigc video quality as- sessment: A dataset and unified model

    Zhichao Zhang, Xinyue Li, Wei Sun, Jun Jia, Xiongkuo Min, Zicheng Zhang, Chunyi Li, Zijian Chen, Puyi Wang, Zhongpeng Ji, et al. Benchmarking aigc video quality as- sessment: A dataset and unified model. arXiv preprint arXiv:2407.21408, 2024. 3

    2024 arXiv

  45. [53]

    Identifying and solving con- ditional image leakage in image-to-video diffusion model

    Min Zhao, Hongzhou Zhu, Chendong Xiang, Kaiwen Zheng, Chongxuan Li, and Jun Zhu. Identifying and solving con- ditional image leakage in image-to-video diffusion model. arXiv preprint arXiv:2406.15735, 2024. 1, 2, 3, 7, 8

    2024 arXiv

  46. [54]

    Magicvideo: Efficient video generation with latent diffusion models

    Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018, 2022. 2, 3 Appendix This appendix includes the following sections: • Additional Ablation Study (Se...

    2022 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.