Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Llama Learns to Direct: DirectorLLM for Human-Centric Video Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Training an LLM to output human-pose tokens as a video director yields more faithful human motion than end-to-end text-to-video models.

desk verdict Novel architecture worth a look, but the evaluation is too weak to support the headline claim of superiority over existing T2V models. read the letter →

arxiv 2412.14484 v3 pith:BUX77XZH submitted 2024-12-19 cs.CV

Kunpeng Song , Tingbo Hou , Zecheng He , Haoyu Ma , Jialiang Wang , Animesh Sinha , Sam Tsai , Yaqiao Luo
show 7 more authors
Xiaoliang Dai Li Chen Xide Xia Peizhao Zhang Peter Vajda Ahmed Elgammal Felix Juefei-Xu
This is my paper · ORCID
classification cs.CV
keywords text-to-videogenerationhumanmotionplanninglargelanguagemodeldirectorposetokenpredictionresidualVQ-VAEdiffusioninterpolationControlNetvideohuman-centric
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that text-to-video generation for people improves when the motion planning is handed to a trained large language model: the LLM reads the prompt and emits a discrete token sequence describing human poses at 1 frame per second, a lightweight diffusion model interpolates those poses to smooth 30 fps motion, and a pose-conditioned video generator renders the final frames. The central proposal is to take human-motion simulation out of the video diffusion model entirely, so the renderer can focus on drawing anatomically plausible frames instead of also deciding how bodies move. If the claim is right, prompts about running, dancing, yoga, and other dynamic actions will produce videos with fewer broken or extra limbs and better alignment to the requested activity than end-to-end text-to-video baselines. The paper supports the claim with automatic scores and blind A/B human judgments on 150 deliberately motion-heavy prompts, and with ablations showing that each of the three stages contributes to the final score.

What carries the argument

The load-bearing device is the tokenized pose channel: a residual VQ-VAE with six codebooks of 512 tokens converts 18-keypoint 2D human skeletons, sampled at 1 fps into 20 key frames, into 120 discrete tokens, turning human motion into a language-model prediction problem. The fine-tuned LLM autoregressively predicts those tokens from the prompt, effectively acting as a layout and motion planner that also outputs instance-level bounding boxes. A linear diffusion interpolator, built on a prior pose-diffusion architecture, densifies the decoded sparse poses to 200 frames and conditions on CLIP text embeddings; finally a ControlNet layer with zero convolutions guides a frozen UNet-based video generator. The residual structure of the VQ-VAE is what preserves pose fidelity across the discrete bottleneck, and the ablation shows that losing it degrades visual quality more than losing the other two components.

What would settle it

Run the same comparison on the full unfiltered prompt set, or on repeated seeds of the 150 dynamic prompts with confidence intervals, and check whether the VideoScore gaps and human-preference rates over VideoCrafter2 remain outside the noise; if they shrink to zero on stationary prompts or on resampled dynamic prompts, the paper's superiority claim fails.

Watch

Extended reading notes

Core claim

DirectorLLM is a three-stage architecture in which a Llama 3-based LLM is fine-tuned with next-token prediction to take the subject-level sentence of a caption and produce 120 vector-quantized pose tokens, representing 20 key-frame skeletons at 1 fps. The tokens are decoded back into poses, a linear diffusion model densifies them into 200 frames at 30 fps under CLIP text conditioning, and a ControlNet-augmented VideoCrafter2 renders the video from the dense pose sequence and the full caption. The paper's central claim is that this separation of labor lets the LLM absorb scene understanding and motion reasoning, so the system beats VideoCrafter, LaVie, and CogVideo on human motion fidelity, prompt faithfulness, and subject naturalness, especially for complex dynamics. The paper presents this as the first trained-LLM-as-video-director design, and its ablations show that removing the residual VQ-VAE hurts visual quality most, followed by removing pose diffusion and then by removing text conditioning in the interpolator.

Load-bearing premise

The method is judged on 150 prompts filtered to motion-heavy keywords, and the reported automatic-score margins over the strongest baseline are tiny, so the whole superiority claim rests on those small margins being real signal rather than noise, especially since the paper itself notes minimal gain on stationary prompts.

Editorial extensions

If this is right

  • Human-centric text-to-video becomes modular: because the DirectorLLM outputs ordinary pose frames, the same motion-planning module can be attached to different renderers (UNet or DiT) with minimal extra training.
  • The pose plan extends the effective video length: the LLM plans 200 frames of motion while the base renderer generates 16 frames at once, with FIFO-Diffusion stitching the clip together into a 7-second video.
  • The ablations imply that each stage is load-bearing: residual quantization preserves pose fidelity, the linear diffusion interpolator supplies smoothness, and text conditioning in the interpolator keeps motion aligned with the prompt.
  • The advertised gains are concentrated on dynamic actions (running, walking, dancing, yoga, jogging, jumping, cycling); the paper notes minimal performance gain on stationary prompts, so the method is a targeted improvement for motion-heavy human scenes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same director pattern could generalize beyond humans: any subject with a learnable discrete motion vocabulary (animals, vehicles, rigid bodies) could be planned by an LLM and rendered by a pose- or trajectory-conditioned generator, making the approach a general motion-planning interface.
  • Because the pose plan is text-generated and fully explicit, a natural next step the paper does not test is interactive editing: a user could revise the prompt and have the LLM re-emit only the changed pose tokens, giving fine-grained control over a single action without re-rendering the whole video.
  • The very small visual-quality gap over VideoCrafter2 (about 0.003 on VideoScore) suggests the honest advantage is in motion realism and prompt alignment, not pixel quality; a targeted evaluation measuring joint-angle plausibility or per-action success rates would show where the method actually wins.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents DirectorLLM, a three-stage system for human-centric text-to-video generation: (1) a Llama 3 8B LLM fine-tuned via next-token prediction to produce VQ-VAE-encoded pose tokens (20 keyframes at 1 FPS) from a text prompt; (2) a compact linear diffusion model that interpolates the sparse poses to 30 FPS, conditioned on CLIP text embeddings; and (3) a VideoCrafter2-based diffusion renderer augmented with a pose ControlNet. The authors evaluate on 150 dynamic human-centric prompts with VideoScore and a human A/B study, and ablate each component. They claim that this decoupling of motion planning from rendering yields improved human motion fidelity, prompt faithfulness, and subject naturalness over VideoCrafter2, LaVie, and CogVideo.

Significance. The modular design is a plausible and timely contribution: moving motion planning into an LLM token space is a clean way to inject explicit pose structure into text-to-video generation, and the component ablations in Table 2 (residual VQ, pose diffusion, text conditioning) show each stage contributes to the reported metrics. The approach is also renderer-agnostic in principle. However, the central claim of superiority over existing methods is not yet established because the quantitative evaluation uses a post-hoc-filtered prompt set and reports no uncertainty, while the human evaluation is summarized only as a figure. If the authors supply a less biased evaluation and proper statistics, the paper could be a useful contribution to LLM-guided video generation.

major comments (4)
  1. [Section 3.3 and Appendix A3, Table 1] The headline comparison is evaluated on a test set of 150 prompts that were explicitly filtered to contain dynamic-motion keywords ('walking', 'running', 'yoga', etc.), and Appendix A3 acknowledges that 'We notice minimal performance gain with stationary prompts.' This post-hoc selection makes the reported superiority conditional on the subset where the method is expected to help and does not support the unqualified conclusion in Section 5 that the model 'outperforms existing methods.' Please report results on an unfiltered or balanced random sample of human-centric prompts, or explicitly restrict the claim to dynamic-motion scenarios and prespecify the filtering criterion.
  2. [Table 1] The reported differences over the strongest baseline, VideoCrafter2, are extremely small on several dimensions (e.g., VC 2.5205 vs 2.5171, TC 2.4578 vs 2.4452), yet no variance, confidence intervals, or significance tests are reported. Since VideoScore is a learned metric, differences of this magnitude are plausibly within noise. Please provide per-prompt score distributions, standard errors, and a paired significance test (e.g., Wilcoxon signed-rank) for each metric, and interpret the effect sizes accordingly.
  3. [Section 3.3, Fig. 6, and Appendix D] The human A/B evaluation is presented only as a bar chart without raw preference percentages, counts per comparison, or statistical testing, despite Appendix D stating that over 5K ratings were collected. Please report the number of comparisons and raters per condition, the exact preference percentages, a binomial test or confidence intervals for each of the five questions, and inter-rater agreement for the majority-vote procedure.
  4. [Sections 3.1 and A3] The evaluation prompts are randomly sampled from the same Shutterstock caption distribution used to train the DirectorLLM, whereas the baselines have not seen this distribution. This gives the proposed method a distributional advantage and weakens the fairness of the comparison. For a more neutral assessment, evaluate on prompts drawn from a different source or from standard video-caption benchmarks, keeping the human-centric and dynamic filtering identical across methods.
minor comments (5)
  1. [Section 3.3] The text says 'We show the results in Fig. 1 for key evaluation dimensions', but the quantitative results are in Table 1; Fig. 1 is the system overview figure.
  2. [Section 3.3, Table 1] Section 3.3 lists 'VideoCrafter' among baselines, while Table 1 uses 'VideoCrafter2'; please make the naming consistent and specify the exact model version.
  3. [Appendix A2] There are several typos that should be corrected: 'dimentional' to 'dimensional', 'represening' to 'representing', 'Pytorch-Lightining' to 'PyTorch Lightning', and '1−5' to '1e-5'.
  4. [Section 2.2] The text refers to 'Fig. 8' for the structure of the DirectorLLM and the interpolator, but those figures appear only in the appendix; please renumber or add cross-references to the main-text figures (Figs. 2 and 3).
  5. [Appendix A1] Appendix A1 states that poses were extracted for 25K videos but does not state how many videos remained after the human-centric and single-person filtering; please report the final training set size.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the LLM-pose-render pipeline is not equivalent to its training inputs by construction, and the self-citations are methodological rather than load-bearing.

full rationale

This is an empirical systems paper rather than a derivation, and no equation-level circularity is exhibited. The three-stage pipeline (DirectorLLM next-token prediction of VQ-VAE pose tokens, linear diffusion interpolation, and ControlNet-conditioned VideoCrafter rendering) is trained with standard losses whose targets are external data: Equation (1) is a noise-prediction loss conditioned on poses and text, and Equation (2) is a next-token cross-entropy loss over pose tokens. Neither output is a refitted version of the metric it is compared against. The only citations to prior work by overlapping authors are methodological: the person-consistency data filtering is done 'following Polyak et al. (2024)', and the human evaluation setup was 'inspired by the video generation model Polyak et al. (2024)'. These citations do not supply the load-bearing evidence for the superiority claim, which rests on comparisons against external baselines using the external learned VideoScore metric and a blind human A/B study. Concerns about the filtered 150-prompt test set, small metric gaps such as 0.003 on Visual Quality, missing error bars, and the appendix admission of 'minimal performance gain with stationary prompts' are legitimate statistical and external-validity concerns, but they are correctness risks rather than circularity: the evaluation numbers are not forced by construction from the training objective or from a self-citation chain. No specific step reduces to its own input, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical entities, forces, dimensions, or conservation laws are postulated. DirectorLLM, the VQ-VAE, and the pose interpolator are trained system components, not invented entities with falsifiable handles outside the trained artifacts. The free parameters that matter for the central claim are the hand-chosen pose tokenization capacity, the 1 FPS key-frame rate, the 16-frame render window, and the evaluation filter that selects dynamic-motion prompts.

free parameters (4)
  • Pose VQ-VAE capacity (6 codebooks x 512 tokens) = 6 x 512 = 3072 codewords, 120 tokens per video
    Chosen by hand in Sec 3.2 to balance reconstruction fidelity against LLM sequence length; the ablation 'w/o Residual VQ' shows the design matters, but the exact capacity is not derived from any principle or external benchmark.
  • Key-frame sampling rate = 1 FPS (20 key frames over 200 frames)
    The entire pose interpolation stage exists to densify this sparse output; the choice of 1 FPS is a modeling decision, not a quantity forced by the data or a uniqueness theorem.
  • ControlNet render length = 16 frames per inference
    Limited by VideoCrafter2; long 200-frame outputs require FIFO-Diffusion queueing, adding temporal-consistency assumptions that are evaluated only qualitatively.
  • Evaluation prompt motion filter = Walking, hiking, running, yoga, jogging, workout, dance, jump, cycling
    Appendix A3 filters the 150 test prompts to these dynamic keywords and reports minimal gain on stationary prompts, so the headline comparison is scoped to this favorable subset.
assumptions (5)
  • domain assumption OpenPose 18-keypoint 2D poses, with face, hand, and foot keypoints removed, are sufficient to represent human motion for conditioning video generation.
    Sec 3.1 and Appendix A2 build the entire pipeline on flattened 36-dimensional pose vectors; if keypoints omit information the renderer needs, the central claim weakens.
  • domain assumption The video captioning model's structured prompts accurately describe the motion content of the Shutterstock videos.
    Sec 3.1 uses these captions as ground truth for the text-to-pose mapping, but the paper provides no caption accuracy validation.
  • domain assumption VideoScore is a reliable proxy for human judgment on the evaluated dimensions.
    Sec 3.3 rests the quantitative comparison entirely on VideoScore with no confidence intervals or calibration against the paper's own human study.
  • domain assumption Standard next-token prediction on pose tokens teaches an LLM to generate plausible, prompt-aligned human motion.
    Equation 2 is the only training objective for the director; there is no kinematic, physical, or temporal consistency loss.
  • domain assumption Single-person filtering (keeping only the largest detected person) covers the human-centric use cases claimed in the introduction.
    Appendix A1 discards multi-person interactions, although the introduction claims the model handles human interactions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Llama Learns to Direct: DirectorLLM for Human-Centric Video Generation." pith.science (2026). https://pith.science/paper/BUX77XZH

@misc{pith2026241214484,
  author       = {Pith},
  title        = {Pith review of: Llama Learns to Direct: DirectorLLM for Human-Centric Video Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BUX77XZH}},
  note         = {Machine review of arXiv:2412.14484}
}
read the original abstract

In this paper, we introduce DirectorLLM, a novel video generation model that employs a large language model (LLM) to orchestrate human poses within videos. As foundational text-to-video models rapidly evolve, the demand for high-quality human motion and interaction grows. To address this need and enhance the authenticity of human motions, we extend the LLM from a text generator to a video director and human motion simulator. Utilizing open-source resources from Llama 3, we train the DirectorLLM to generate detailed instructional signals, such as human poses, to guide video generation. This approach offloads the simulation of human motion from the video generator to the LLM, effectively creating informative outlines for human-centric scenes. These signals are used as conditions by the video renderer, facilitating more realistic and prompt-following video generation. As an independent LLM module, it can be applied to different video renderers, including UNet and DiT, with minimal effort. Experiments on automatic evaluation benchmarks and human evaluations show that our model outperforms existing ones in generating videos with higher human motion fidelity, improved prompt faithfulness, and enhanced rendered subject naturalness.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multi-human Interactive Talking Dataset

    cs.CV 2025-08 conditional novelty 6.0 of 10

    The paper contributes a 12-hour multi-person conversational video dataset with pose and speaking annotations, plus a baseline model for generating full-body talking videos of two to four people.

  2. DanceTogether! Identity-Preserving Multi-Person Interactive Video Generation

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A diffusion model fuses per-person masks with pose keypoints to generate identity-preserving, two-person interactive videos from a single reference image, outperforming prior single-person-animation pipelines.

Reference graph

Works this paper leans on

28 extracted references · 7 canonical work pages · cited by 2 Pith papers

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    arXiv

  2. [3]

    Palm 2 technical report.arXiv preprint arXiv:2305.10403 ,

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report.arXiv preprint arXiv:2305.10403 ,

    arXiv

  3. [4]

    ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers

    Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, et al. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 ,

    arXiv

  4. [6]

    Emu: Enhancing image generation models using photogenic needles in a haystack

    Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.15807 ,

    arXiv

  5. [7]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783 ,

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783 ,

    arXiv

  6. [8]

    subject-level prompt

    Each pose is represented by a sequence of 6 VQ tokens, enabling detailed and structured encoding. For each video, we evenly sample 20 key frames from a total of 200 frames of human poses. These key frames can be converted to 6 x 20 = 120tokens by VQ-VAE encoder. The model is compact, requiring about 2 hours of training on a single A100 GPU. DirectorLLM. W...

    work page 2024

  7. [9]

    Cameractrl: Enabling camera control for text-to-video generation.arXiv preprint arXiv:2404.02101 , 2024a

    Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation.arXiv preprint arXiv:2404.02101 , 2024a. Xuan He, Dongfu Jiang, Ge Zhang, Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, Ziyan Jiang, Aaran Arulraj, et al. Videoscore: Building automatic...

    arXiv

  8. [11]

    Fifo-diffusion: Generating infinite videos from text without training

    Jihwan Kim, Junoh Kang, Jinyoung Choi, and Bohyung Han. Fifo-diffusion: Generating infinite videos from text without training. arXiv preprint arXiv:2405.11473 ,

    arXiv

Show all 28 references
  1. [12]

    Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models.arXiv preprint arXiv:2305.13655 ,

    10 Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models.arXiv preprint arXiv:2305.13655 ,

    arXiv

  2. [13]

    Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning

    Han Lin, Abhay Zala, Jaemin Cho, and Mohit Bansal. Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning. arXiv preprint arXiv:2309.15091 ,

    arXiv

  3. [14]

    Trailblazer: Trajectory control for diffusion-based video generation

    Wan-Duo Kurt Ma, John P Lewis, and W Bastiaan Kleijn. Trailblazer: Trajectory control for diffusion-based video generation. arXiv preprint arXiv:2401.00896 ,

    arXiv

  4. [15]

    Accessed: 2024-10-31

    https://openai.com/index/sora/. Accessed: 2024-10-31. Quynh Phung, Songwei Ge, and Jia-Bin Huang. Grounded text-to-image synthesis with attention refocusing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7932–7942,

    2024

  5. [16]

    Sampson, Shikai Li, Simone Parmeggiani, Steve Fine, Tara Fowler, Vladan Petrovic, and Yuming Du

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, David Yan, Dhruv Choudhary, Dingkang Wang, Geet Sethi, Guan Pang, Haoyu Ma, Ishan Misra, Ji Hou, Jialiang Wang, Kiran Jagadeesh, Kunpeng Li, Lu...

    arXiv

  6. [18]

    Kandinsky: an improved text-to-image synthesis with image prior and latent diffusion.arXiv preprint arXiv:2310.03502 ,

    Anton Razzhigaev, Arseniy Shakhmatov, Anastasia Maltseva, Vladimir Arkhipkin, Igor Pavlov, Ilya Ryabov, Angelina Kuts, Alexander Panchenko, Andrey Kuznetsov, and Denis Dimitrov. Kandinsky: an improved text-to-image synthesis with image prior and latent diffusion.arXiv preprint...

    arXiv

  7. [19]

    Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling

    Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, et al. Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling. In ACM SIGGRAPH 2024 Conference Papers , p...

    2024

  8. [20]

    Shutterstock, 2024.https://www.shutterstock.com/

    Shutterstock. Shutterstock, 2024.https://www.shutterstock.com/. Accessed: 2024-10-31. Jonathan Tseng, Rodrigo Castellon, and Karen Liu. Edge: Editable dance generation from music. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 448–458,

    2024

  9. [21]

    Boximator: Generating rich and controllable motions for video synthesis.arXiv preprint arXiv:2402.01566 , 2024a

    11 Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guoqiang Wei, Liping Yuan, and Hang Li. Boximator: Generating rich and controllable motions for video synthesis.arXiv preprint arXiv:2402.01566 , 2024a. Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shi...

    2024 arXiv

  10. [23]

    Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory.arXiv preprint arXiv:2308.08089 ,

    Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory.arXiv preprint arXiv:2308.08089 ,

    arXiv

  11. [24]

    Motiondiffuse: Text-driven human motion generation with diffusion model.arXiv preprint arXiv:2208.15001 ,

    Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model.arXiv preprint arXiv:2208.15001 ,

    arXiv

  12. [25]

    Tora: Trajectory-oriented diffusion transformer for video generation.arXiv preprint arXiv:2407.21705 ,

    Zhenghao Zhang, Junchao Liao, Menghao Li, Long Qin, and Weizhi Wang. Tora: Trajectory-oriented diffusion transformer for video generation.arXiv preprint arXiv:2407.21705 ,

    arXiv

  13. [26]

    Motiondirector: Motion customization of text-to-video diffusion models.arXiv preprint arXiv:2310.08465 ,

    Rui Zhao, Yuchao Gu, Jay Zhangjie Wu, David Junhao Zhang, Jiawei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou. Motiondirector: Motion customization of text-to-video diffusion models.arXiv preprint arXiv:2310.08465 ,

    arXiv

  14. [27]

    Subject-Level Prompt

    12 Appendix A. Implementation Details A1. Pre-Processing Details Our video caption model generates highly structured prompts. Below is an example of full text prompt generated by video captioning model: The video shows a woman running through a grassy field. The woman has fair...

    2024

  15. [2017]

    Videocrafter1: Open diffusion models for high-quality video generation.arXiv preprint arXiv:2310.19512,

    Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafter1: Open diffusion models for high-quality video generation.arXiv preprint arXiv:2310.19512,

    arXiv

  16. [2021]

    Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125 , 1(2):3,

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125 , 1(2):3,

    arXiv

  17. [2022]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868 ,

    arXiv

  18. [2023]

    Unclip model documentation, 2024.https://github.com/Stability-AI/stablediffusion/blob/main/doc/ UNCLIP.MD

    Stability AI. Unclip model documentation, 2024.https://github.com/Stability-AI/stablediffusion/blob/main/doc/ UNCLIP.MD. Accessed: 2024-10-31. Simon Alexanderson, Rajmund Nagy, Jonas Beskow, and Gustav Eje Henter. Listen, denoise, action! audio-driven motion synthesis with dif...

    2024

  19. [2024]

    insightface, 2024.https://github.com/deepinsight/insightface/tree/master/examples/ person_detection

    Jia Guo and Jiankang Deng. insightface, 2024.https://github.com/deepinsight/insightface/tree/master/examples/ person_detection. Accessed: 2024-10-31. Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: ...

    2024 arXiv

  20. [2025]

    Direct-a-video: Customized video generation with user-directed camera movement and object motion

    Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user-directed camera movement and object motion. InACM SIGGRAPH 2024 Conference Papers , pages 1–12,

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.