Pith. sign in

REVIEW 2 major objections 4 minor 60 references

Towards motion from video diffusion models

T0 review · 2 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper shows that open-source text-to-video diffusion models, guided by score distillation sampling, can synthesize plausible SMPL-X human animations for common actions like running and walking, but fail for rare actions like…

desk verdict A clear proof of concept for SDS-driven SMPL-X animation, but the target-prior initialization in Section 5 confounds the central claim that video models are the bottleneck. read the letter →

arxiv 2411.12831 v1 pith:KXHOC3IX submitted 2024-11-19 cs.CV

Paul Janson , Tiberiu Popa , Eugene Belilovsky This is my paper · ORCID
classification cs.CV
keywords humanmotiongenerationscoredistillationsamplingtext-to-videodiffusionSMPL-XDistillmodelscharacteranimation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether open-source text-to-video diffusion models can serve as a guide for generating human body animation from a text prompt, without relying on motion capture data. It proposes MotionDistill, a pipeline that optimizes SMPL-X body poses frame by frame using score distillation sampling gradients from a video diffusion model. The central finding is that this works well for common activities such as running and walking, but degrades for uncommon actions such as punching and fails outright for cartwheels. By ablating the optimization directly in the video model's latent space, the authors argue that the limitation comes from the video diffusion model's bias toward common motions, not from the SMPL-X pose representation. A sympathetic reader would care because it charts a path toward extracting animation directly from video models rather than from scarce motion capture data.

What carries the argument

The central object is MotionDistill, a two-stage pipeline. Stage one defines a PoseField, a two-hidden-layer MLP $P_\alpha : \tau \to \theta_b$ that maps a frame index to SMPL-X body pose parameters; these parameters are fed through the SMPL-X layer $M(\beta,\theta,\psi)$ to produce a mesh, which is rasterized from a random camera on a circular trajectory to form an $F$-frame video. Stage two encodes that video into the latent space of a video diffusion model, adds noise, and computes score distillation sampling gradients $\nabla_\alpha L_{\mathrm{SDS-T}} = \lambda_{\mathrm{SDS}}^t \mathbb{E}_{\sigma,\epsilon}[w(\sigma)(\hat{\epsilon}_\phi(Z_{\alpha,\sigma,\epsilon}|y,\sigma)-\epsilon)\frac{\partial Z_\alpha}{\partial \alpha}]$, with an additional image-diffusion SDS term and a temporal smoothness regularization on consecutive poses. This machinery lets gradient information from a pretrained video diffusion model flow back into the pose parameters, so the text prompt alone is supposed to drive the motion.

What would settle it

Run MotionDistill on a 'running' prompt while initializing PoseField from the mean pose and covariance of a different action, such as sitting, and hold all other settings fixed; if the generated motion no longer looks like running or converges to the sitting prior, then the reported success for common actions depends on the target-motion-specific initialization rather than on the video diffusion model. Conversely, if running still emerges from a neutral initialization, the paper's attribution of failures to the video model is directly supported.

Watch

Extended reading notes

Core claim

The paper claims that a differentiable animation pipeline built from an MLP pose field, an SMPL-X body model, and a differentiable renderer can be steered by score distillation sampling from a text-to-video diffusion model to produce plausible human motion for common actions. Running and walking come out nearly correct, punching is only partially realized, and cartwheels fail even at the level of a plausible pose. The authors further claim, based on a latent-space ablation in which the rendered video is detached and the video diffusion latent is optimized directly, that these failures are attributable to the video diffusion models themselves and not to the body representation or the SDS formulation. They observe that newer video models such as VideoCrafter2 give markedly more realistic motion than older ones such as ModelScope and ZeroScope, which supports the conclusion that progress in the underlying text-to-video model is the key lever for better motion generation.

Load-bearing premise

The pipeline initializes PoseField at the mean pose of the target motion and bounds outputs to three times the standard deviation of that motion's pose parameters, so it assumes access to the statistical distribution of the exact motion it is supposed to generate from text alone.

Editorial extensions

If this is right

  • If the central claim is right, motion generation can bypass motion capture data and instead tap the much larger corpus of video data already embedded in text-to-video diffusion models.
  • The failure pattern across actions indicates that open-source video diffusion models have an internal bias toward frequently occurring human activities, so evaluating any text-to-motion method needs to separate common from rare actions.
  • Because the latent-space ablation reproduces the same failure modes, the pose representation and SDS formulation are probably not the bottleneck; improving the video foundation model should directly improve the generated animation.
  • Newer video diffusion models such as VideoCrafter2 already yield more realistic motion than older ones, so the method's quality is expected to scale upward as open-source text-to-video models continue to improve.
  • MotionDistill can serve as a proof of concept and a diagnostic tool: success and failure cases expose which actions a given video diffusion model actually understands.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct way to test the paper's attribution would be to initialize PoseField from a neutral or mismatched pose distribution and run the same prompts; if common actions still emerge while rare ones fail, the video model's learned bias is confirmed, whereas if performance collapses the mean-pose initialization was doing the work.
  • The paper implicitly assumes that the video diffusion model's gradient signal is informative about human motion, but the same pipeline could be applied to other articulated objects, such as animals or robots, to probe whether the model has an action-specific rather than human-specific motion prior.
  • The authors' latent-space ablation could be extended into a quantitative measure of action bias: for a fixed set of prompts, the variance of the optimized latents across frames could serve as a proxy for how confidently the video model represents each action.
  • Because the method optimizes only the 21 major body joints and keeps hands and face fixed, the punching failure may partly stem from under-represented hand motion; a version that also optimizes finger poses would separate a representation gap from a model gap in the semi-failure cases.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The manuscript proposes MotionDistill, a pipeline that generates SMPL-X human animation by optimizing a small MLP (PoseField) that maps frame indices to body-pose parameters, using score distillation sampling (SDS) gradients from open-source text-to-video diffusion models (ModelScope, ZeroScope, VideoCrafter). The method is evaluated visually on prompts such as 'running', 'walking', 'punching', and 'doing a cartwheel'. The authors report plausible results for common actions, partial failures for 'punching', and extreme failures for 'cartwheel'. A latent-space ablation optimizes video latents directly with the same SDS objective, and the paper concludes that the observed failures point to limitations of the video diffusion models rather than the body representation. The paper positions itself as an initial proof-of-concept study of whether open-source text-to-video diffusion models can guide human motion synthesis.

Significance. If the central attribution were established, the paper would be a useful early empirical probe of a fast-moving question: whether video diffusion priors can drive human motion generation without MoCap supervision. The choice of three open-source models, the SDS-based formulation, and the latent-space ablation are sensible exploratory steps. However, the key claim that failures stem from the video diffusion model is currently confounded by the target-motion prior used in initialization and clamping, and the evidence is entirely qualitative. The work is therefore a promising proof-of-concept whose main empirical conclusion needs substantially stronger support before it can bear the weight the manuscript places on it.

major comments (2)
  1. [Section 5] The implementation as written contradicts the text-to-motion framing of Section 4. The paper states: 'We initialized the PoseField to output the mean pose of the target motion and constrained its output within a range defined by three times the standard deviation of the target pose parameters.' This injects the statistical distribution of the exact motion to be generated. For 'running', the optimizer starts inside a known-good pose region and is clamped there, so success does not demonstrate that the video diffusion model can synthesize running from text alone; it may only show that SDS can refine local details around a provided prior. For 'cartwheel' or other rare actions, the target-motion statistics may be scarce, noisy, or mismatched, so the failure could reflect the prior or the SMPL-X parameterization rather than the video model. The latent-space ablation in Section 5 starts from a Stage-1 video rendered with the same prior-initialized PoseField, so it does not remove the confound. I see two possible remedies: (a) remove or substantially weaken the target-motion initialization/clamping and show the common/rare distinction persists; or (b) explicitly reframe the method as motion refinement conditioned on action-specific pose statistics, which would materially change the paper's claims. Without one of these, the paper's central attribution is not established.
  2. [Section 5] The evaluation is entirely visual, with no quantitative metrics, no error bars, no multiple seeds, and no comparison to any baseline. Statements such as 'the models generated more natural motion sequences for running' and 'VideoCrafter demonstrates a higher degree of realism' in Figure 4 are not backed by measurements. The common/rare distinction is the paper's central empirical finding, but it rests on the authors' subjective inspection of a handful of sequences. At minimum, the paper should report quantitative motion-quality metrics (e.g., per-frame pose plausibility, motion smoothness, text-motion alignment scores, or user-study ratings) and should include multiple random seeds per prompt and per model so that variability (or lack thereof) is visible. Without this, the conclusions in Sections 5 and 6 are not adequately supported.
minor comments (4)
  1. [Equation 5] The regularization term is written as a sum of differences between consecutive pose vectors without a norm or squared norm, so as written it is not a scalar loss. It should be something like ||θ_i+1 − θ_i||^2 or an absolute/norm-based form.
  2. [Equation 4] There is a missing parenthesis: the expectation is written as Eσ,ϵ[w(σ)ϵ̂(Zα,σ,ϵ|y, σ) − ϵ) ∂Zα/∂α], which does not balance; the closing parenthesis appears twice, and the intended (ϵ̂ − ϵ) factor should be written correctly.
  3. [Footnote 3] The supplementary results are referenced via a GitHub URL that the reader is told to visit; for a journal version, the repository should either contain the code and full videos with clear instructions or the supplementary material should be included in the submission package.
  4. [Notation] The paper alternates between 'SMPLx' and 'SMPL-X'; please use the official 'SMPL-X' consistently.

Circularity Check

2 steps flagged · score 6.0 of 10

Section 5's target-motion prior (mean pose initialization and 3-sigma clamp) injects the distribution of the motion to be generated, so the paper's attribution of common-action success and rare-action failure to the video diffusion model is confounded.

  1. fitted input called prediction [Section 5, Implementation details]
    "We initialized the PoseField to output the mean pose of the target motion and constrained its output within a range defined by three times the standard deviation of the target pose parameters."

    The method is presented as deriving joint rotations from a text prompt via SDS, with the problem stated as determining the optimal joint rotations given only a text prompt. However, the optimizer is initialized at the mean pose of the target motion and clamped to three standard deviations of the target pose parameters. If target motion refers to the action class to be generated, this prior already encodes the statistical distribution of the very motion the paper claims to synthesize from text. For common actions such as running and walking, the optimization starts inside a plausible pose region, so SDS only refines local details; for rare actions such as cartwheels, with scarce or mismatched statistics, failure may reflect the prior rather than the video diffusion model.

  2. fitted input called prediction [Section 5, Ablating MotionDistill]
    "We first render the initial video using Stage 1 of our pipeline. Then detach this rendering from the optimization process and directly optimize the latentsZ."

    This latent-space ablation is used to conclude that the issue lies in the video diffusion model rather than the motion representation. However, the initial video is rendered from the PoseField that was itself initialized and clamped using the target-motion prior. The latent optimization therefore starts from content already biased by the target-motion distribution, so the ablation does not remove the confound and cannot cleanly attribute common-action success or rare-action failure to the video diffusion model.

full rationale

The paper's central empirical claim is that open-source text-to-video diffusion models can generate plausible human animation for common actions such as running and walking but fail for rare actions such as cartwheels, and that this asymmetry reflects the video diffusion model rather than the body representation. The load-bearing implementation detail in Section 5 initializes PoseField to the mean pose of the target motion and clamps its output to three standard deviations of the target pose parameters. This injects the target motion's statistical distribution into the optimization. For common actions with well-defined motion-capture statistics, the initial pose is already a plausible instance of that action, so the SDS signal only refines a known-good starting point; for rare actions with scarce or mismatched statistics, the failure could stem from the prior or the SMPL-X parameterization rather than the video model. The latent ablation does not resolve the confound because its initial video is rendered from the same target-prior-initialized PoseField. There is no evidence of load-bearing self-citation or an imported uniqueness theorem; the SDS formulation is standard and externally sourced. The circularity is partial: the SDS gradient still contributes to the optimization, and the paper does compare multiple video models, but the specific attribute of common versus rare action competence is not cleanly separable from the target-motion prior. Score 6 reflects this partial reduction-by-construction of the central prediction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central result rests on a handful of modeling and optimization choices, most notably the undisclosed target-motion mean and standard deviation used to initialize and bound the pose network. That prior carries much of the work; SDS refines within a distribution that already encodes the target action.

free parameters (4)
  • lambda_reg = 1e-3
    Regularization weight in Equation 5, chosen by hand; affects smoothness of motion.
  • lambda_SDS_t = 1e-3
    SDS loss weight in Equation 4, chosen by hand; controls influence of diffusion guidance.
  • CFG scale = 100
    Classifier-free guidance scale used in all experiments; unusually high and likely to influence gradient magnitude.
  • target pose mean/std = not reported
    PoseField is initialized to the mean pose of the target motion and outputs are constrained to three standard deviations of the target pose parameters; these statistics are derived from the motion being generated and are not disclosed.
assumptions (4)
  • domain assumption SMPL-X body model with linear blend skinning is an adequate representation for human motion
    The method relies on SMPL-X to render the character; if its articulated model cannot represent the motions (e.g., finger details), the generated animation would be limited.
  • domain assumption Video diffusion models' latent spaces and denoising UNets provide gradients that align with human pose realism
    The SDS gradient in Equation 4 assumes the video model's noise prediction is sensitive to body pose; this is the core hypothesis being tested, and the paper's conclusion depends on it.
  • standard math Differentiable rendering via nvdiffrast gives usable gradients from pixels back to pose parameters
    The pipeline backpropagates through the renderer; this is a standard assumption in differentiable rendering.
  • ad hoc to paper Classifier-free guidance at scale 100 yields useful, not degenerate, gradients
    CFG=100 is far above typical values (7.5); the paper does not justify why this extremely high guidance works, yet the optimization and conclusions depend on it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards motion from video diffusion models." pith.science (2026). https://pith.science/paper/KXHOC3IX

@misc{pith2026241112831,
  author       = {Pith},
  title        = {Pith review of: Towards motion from video diffusion models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KXHOC3IX}},
  note         = {Machine review of arXiv:2411.12831}
}
read the original abstract

Text-conditioned video diffusion models have emerged as a powerful tool in the realm of video generation and editing. But their ability to capture the nuances of human movement remains under-explored. Indeed the ability of these models to faithfully model an array of text prompts can lead to a wide host of applications in human and character animation. In this work, we take initial steps to investigate whether these models can effectively guide the synthesis of realistic human body animations. Specifically we propose to synthesize human motion by deforming an SMPL-X body representation guided by Score distillation sampling (SDS) calculated using a video diffusion model. By analyzing the fidelity of the resulting animations, we gain insights into the extent to which we can obtain motion using publicly available text-to-video diffusion models using SDS. Our findings shed light on the potential and limitations of these models for generating diverse and plausible human motions, paving the way for further research in this exciting area.

Figures

Figures reproduced from arXiv: 2411.12831 by the authors.

Figure 1
Figure 1. Human motion sequence resembling running generated using text-to-video model. The figure illustrates that the current video models can generate realistic mo￾tion for commonly occuring human activity such as running achieve this, we utilize the widely adopted SMPL-X [32] digital human template model to render a character. This character is animated through an optimiza￾tion process that iteratively updates a multi-lay… view at source ↗
Figure 2
Figure 2. Our study consists of two stages. Stage:1 (top) Joint rotations required to animate the character are generated using PoseField. Passed through SMPLx Layer to get the final mesh which is then rasterized using a differentiable renderer. We use a random camera and a predetermined texture. This is repeated for F frames to obtain the video. Stage:2 (bottom) Rendered video is encoded to the latent space of the diffusion … view at source ↗
Figure 3
Figure 3. Our results for different motions: All results are obtained by using the model VideoCrafter [8]. (i) Walking motion is one of the best cases in addition to running (Fig:1). (ii) Punching is a semi-failure case (iii) Cartwheel is an extreme failure case. 5 Experiments and results Evaluating MotionDistill [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Visualization of the optimized latent when given the two actions as prompts. Generated videos in the top row of each video model denote the action "running". The bottom row of each denotes the action "punching". Clearly, the top rows of each model show a more natural m…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 45 canonical work pages

  1. [1]

    In: 2018 IEEE International Conference on Robotics and Automation (ICRA)

    Ahn, H., Ha, T., Choi, Y., Yoo, H., Oh, S.: Text2action: Generative adversarial synthesis from language to action. In: 2018 IEEE International Conference on Robotics and Automation (ICRA). pp. 5915–5920. IEEE (2018) 3

    work page 2018

  2. [2]

    In: 2019 International Conference on 3D Vision (3DV)

    Ahuja, C., Morency, L.P.: Language2pose: Natural language grounded pose fore- casting. In: 2019 International Conference on 3D Vision (3DV). pp. 719–728. IEEE (2019) 3

    work page 2019

  3. [3]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Bahmani, S., Skorokhodov, I., Rong, V., Wetzstein, G., Guibas, L., Wonka, P., Tulyakov, S., Park, J.J., Tagliasacchi, A., Lindell, D.B.: 4d-fy: Text-to-4d gener- ation using hybrid score distillation sampling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 7996–8006 (June 2024) 2, 5

    work page 2024

  4. [4]

    ArXiv (2021), https://crfm.stanford.edu/assets/report.pdf 2

    Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M.S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N.S., Chen, A.S., Creel, K.A., Davis, J., Dem- szky, D., Donahue, C., Doumbouya, M., Durmus, E., Ermon, S., Etchemendy, J., Ethayarajh, K., Fei-Fei, L., Finn, C....

    work page 2021

  5. [5]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Cao, Y., Cao, Y.P., Han, K., Shan, Y., Wong, K.Y.K.: Dreamavatar: Text-and- shape guided 3d human avatar generation via diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 958–968 (2024) 2

    work page 2024

  6. [6]

    co / cerspense / zeroscope_v2_576w 2, 7

    Cerspense: Zeroscope modelcard, https : / / huggingface . co / cerspense / zeroscope_v2_576w 2, 7

    work page

  7. [7]

    Chen, H., Xia, M., He, Y., Zhang, Y., Cun, X., Yang, S., Xing, J., Liu, Y., Chen, Q., Wang, X., Weng, C., Shan, Y.: Videocrafter1: Open diffusion models for high- quality video generation (2023) 2

    work page 2023

  8. [8]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Chen, H., Zhang, Y., Cun, X., Xia, M., Wang, X., Weng, C., Shan, Y.: Videocrafter2: Overcoming data limitations for high-quality video diffusion mod- els. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7310–7320 (2024) 1, 2, 6, 7

    work page 2024

Show all 60 references
  1. [9]

    Advances in neural information processing systems34, 8780–8794 (2021) 2

    Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. Advances in neural information processing systems34, 8780–8794 (2021) 2

    2021

  2. [10]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Gal, R., Vinker, Y., Alaluf, Y., Bermano, A., Cohen-Or, D., Shamir, A., Chechik, G.: Breathing life into sketches using text-to-video priors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4325– 4336 (2024) 2 Towards motion from vide...

    2024

  3. [11]

    Advances In Neural Information Processing Systems35, 31841–31854 (2022) 1, 2

    Gao, J., Shen, T., Wang, Z., Chen, W., Yin, K., Li, D., Litany, O., Gojcic, Z., Fidler, S.: Get3d: A generative model of high quality 3d textured shapes learned from images. Advances In Neural Information Processing Systems35, 31841–31854 (2022) 1, 2

    2022

  4. [12]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Ghosh, A., Cheema, N., Oguz, C., Theobalt, C., Slusallek, P.: Synthesis of compo- sitional animations from textual descriptions. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 1396–1406 (2021) 3

    2021

  5. [13]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Guo, C., Zou, S., Zuo, X., Wang, S., Ji, W., Li, X., Cheng, L.: Generating di- verse and natural 3d human motions from text. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5152–5161 (2022) 3

    2022

  6. [14]

    arXiv preprint arXiv:2210.02303 (2022) 1

    Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D.P., Poole, B., Norouzi, M., Fleet, D.J., et al.: Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 (2022) 1

    2022 arXiv

  7. [15]

    Advances in neural information processing systems33, 6840–6851 (2020) 3

    Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems33, 6840–6851 (2020) 3

    2020

  8. [16]

    In: NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications (2021) 7

    Ho, J., Salimans, T.: Classifier-free diffusion guidance. In: NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications (2021) 7

    2021

  9. [18]

    2024 IEEE/CVF Winter Conference on Applications of Com- puter Vision (WACV) pp

    Karthikeyan, A., Ren, R., Kant, Y., Gilitschenski, I.: Avatarone: Monocular 3d human animation. 2024 IEEE/CVF Winter Conference on Applications of Com- puter Vision (WACV) pp. 3635–3645 (2024),https://api.semanticscholar.org/ CorpusID:267751541 1, 2

    2024

  10. [19]

    ACM Trans

    Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph.42(4), 139–1 (2023) 2

    2023

  11. [20]

    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Int. Conf. Learn. Represent. (2014) 7

    2014

  12. [21]

    arXiv preprint arXiv:1312.6114 (2013) 3

    Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013) 3

    2013 arXiv

  13. [22]

    ACM Transactions on Graphics (ToG)39(6), 1–14 (2020) 7

    Laine, S., Hellsten, J., Karras, T., Seol, Y., Lehtinen, J., Aila, T.: Modular primi- tives for high-performance differentiable rendering. ACM Transactions on Graphics (ToG)39(6), 1–14 (2020) 7

    2020

  14. [23]

    In: 2024 International Conference on 3D Vision (3DV)

    Liao, T., Yi, H., Xiu, Y., Tang, J., Huang, Y., Thies, J., Black, M.J.: Tada! text to animatable digital avatars. In: 2024 International Conference on 3D Vision (3DV). pp. 1508–1519. IEEE (2024) 2

    2024

  15. [25]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Ling, H., Kim, S.W., Torralba, A., Fidler, S., Kreis, K.: Align your gaussians: Text- to-4d with dynamic 3d gaussians and composed diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8576–8588 (2024) 2, 5

    2024

  16. [26]

    ACM Transactions on Graphics (TOG)34(6), 1–16 (2015) 3 10 P

    Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: Smpl: a skinned multi-person linear model. ACM Transactions on Graphics (TOG)34(6), 1–16 (2015) 3 10 P. Janson et al

    2015

  17. [27]

    In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV)

    Mahmood,N.,Ghorbani,N.,Troje,N.F.,Pons-Moll,G.,Black,M.:Amass:Archive of motion capture as surface shapes. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE (Oct 2019).https://doi.org/10.1109/iccv. 2019.00554, http://dx.doi.org/10.1109/ICCV.2019.00554 3

    2019

  18. [28]

    In: Eu- ropean Conference on Computer Vision

    Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. In: Eu- ropean Conference on Computer Vision. pp. 405–421. Springer (2020) 2

    2020

  19. [29]

    In: SIGGRAPH Asia 2022 conference papers

    Mohammad Khalid, N., Xie, T., Belilovsky, E., Popa, T.: Clip-mesh: Generating textured meshes from text using pretrained image-text models. In: SIGGRAPH Asia 2022 conference papers. pp. 1–8 (2022) 1, 2

    2022

  20. [30]

    OpenAI: Video generation models as world simulators, https://openai.com/ index/video-generation-models-as-world-simulators/ 1

  21. [31]

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high- performance deep learning library. In: Adv. Neural Inform. Process. Syst. (2019) 7

    2019

  22. [32]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Pavlakos, G., Choutas, V., Ghorbani, N., Bolkart, T., Osman, A.A., Tzionas, D., Black, M.J.: Expressive body capture: 3d hands, face, and body from a single im- age. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10975–10985 (2019) 2, 3

    2019

  23. [33]

    2021 IEEE/CVF International Conference on Com- puter Vision (ICCV) pp

    Petrovich, M., Black, M.J., Varol, G.: Action-conditioned 3d human motion syn- thesis with transformer vae. 2021 IEEE/CVF International Conference on Com- puter Vision (ICCV) pp. 10965–10975 (2021), https://api.semanticscholar. org/CorpusID:233210075 1

    2021

  24. [34]

    In: European Conference on Computer Vision

    Petrovich, M., Black, M.J., Varol, G.: Temos: Generating diverse human motions from textual descriptions. In: European Conference on Computer Vision. pp. 480–

  25. [35]

    Big data 4(4), 236–252 (2016) 3

    Plappert, M., Mandery, C., Asfour, T.: The kit motion-language dataset. Big data 4(4), 236–252 (2016) 3

    2016

  26. [36]

    Robotics and Autonomous Systems109, 13–26 (2018) 3

    Plappert, M., Mandery, C., Asfour, T.: Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural net- works. Robotics and Autonomous Systems109, 13–26 (2018) 3

    2018

  27. [37]

    In: The Eleventh International Conference on Learning Representations (2023) 2, 3, 5

    Poole, B., Jain, A., Barron, J.T., Mildenhall, B.: Dreamfusion: Text-to-3d using 2d diffusion. In: The Eleventh International Conference on Learning Representations (2023) 2, 3, 5

    2023

  28. [38]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021) 2, 3

    2021

  29. [39]

    2022 ieee

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. 2022 ieee. In: CVF Conference on Computer Vision and Pattern Recognition (CVPR). vol. 1 (2021) 3, 4

    2021

  30. [40]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022) 2

    2022

  31. [41]

    arXiv preprint arXiv:2201.02610 (2022) 3

    Romero, J., Tzionas, D., Black, M.J.: Embodied hands: Modeling and capturing hands and bodies together. arXiv preprint arXiv:2201.02610 (2022) 3

    2022 arXiv

  32. [42]

    In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, Oc- tober 5-9, 2015, proceedings, part III 18

    Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomed- ical image segmentation. In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, Oc- tober 5-9, 2015, proceedings, part III 18....

    2015

  33. [43]

    Advances in neural information processing systems35, 36479–36494 (2022) 2

    Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E.L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al.: Photorealistic text- to-image diffusion models with deep language understanding. Advances in neural information processing systems35,...

    2022

  34. [45]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Sanyal, S., Bolkart, T., Feng, H., Black, M.J.: Learning to regress 3d face shape and expression from an image without 3d supervision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7763– 7772 (2019) 3

    2019

  35. [46]

    In: The Twelfth International Conference on Learning Represen- tations (2024) 1, 3

    Shafir, Y., Tevet, G., Kapon, R., Bermano, A.H.: Human motion diffusion as a generative prior. In: The Twelfth International Conference on Learning Represen- tations (2024) 1, 3

    2024

  36. [47]

    In: Advances in Neural Information Processing Systems (NeurIPS) (2021) 2

    Shen, T., Gao, J., Yin, K., Liu, M.Y., Fidler, S.: Deep marching tetrahedra: a hy- brid representation for high-resolution 3d shape synthesis. In: Advances in Neural Information Processing Systems (NeurIPS) (2021) 2

    2021

  37. [48]

    arXiv preprint arXiv:2209.14792 (2022) 1

    Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., et al.: Make-a-video: Text-to-video generation without text- video data. arXiv preprint arXiv:2209.14792 (2022) 1

    2022 arXiv

  38. [49]

    In: Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., Scar- lett, J

    Singer, U., Sheynin, S., Polyak, A., Ashual, O., Makarov, I., Kokkinos, F., Goyal, N., Vedaldi, A., Parikh, D., Johnson, J., Taigman, Y.: Text-to-4D dynamic scene generation. In: Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., Scar- lett, J. (eds.) Proceedings ...

    2023

  39. [50]

    arXiv preprint arXiv:2408.04803 (2024) 2

    Sivakumar, P., Janson, P., Rajasegaran, J., Ambegoda, T.: Fewshotnerf: Meta- learning-based novel view synthesis for rapid scene-specific adaptation. arXiv preprint arXiv:2408.04803 (2024) 2

    2024 arXiv

  40. [51]

    In: International conference on machine learning

    Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep unsuper- vised learning using nonequilibrium thermodynamics. In: International conference on machine learning. pp. 2256–2265. PMLR (2015) 2, 3, 4

    2015

  41. [52]

    In: Interna- tional Conference on Learning Representations (2021) 3

    Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score- based generative modeling through stochastic differential equations. In: Interna- tional Conference on Learning Representations (2021) 3

    2021

  42. [53]

    Tang, J., Ren, J., Zhou, H., Liu, Z., Zeng, G.: Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. In: Int. Conf. Learn. Represent. (2024) 2

    2024

  43. [54]

    In: European Conference on Com- puter Vision

    Tevet, G., Gordon, B., Hertz, A., Bermano, A.H., Cohen-Or, D.: Motionclip: Ex- posing human motion generation to clip space. In: European Conference on Com- puter Vision. pp. 358–374. Springer (2022) 1, 3

    2022

  44. [55]

    In: The Eleventh International Conference on Learning Representations (2023), https://openreview.net/forum?id=SJ1kSyO2jwu 1, 3

    Tevet, G., Raab, S., Gordon, B., Shafir, Y., Cohen-or, D., Bermano, A.H.: Human motion diffusion model. In: The Eleventh International Conference on Learning Representations (2023), https://openreview.net/forum?id=SJ1kSyO2jwu 1, 3

    2023

  45. [56]

    Advances in neural information pro- cessing systems30 (2017) 5 12 P

    Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information pro- cessing systems30 (2017) 5 12 P. Janson et al

    2017

  46. [57]

    In: International Conference on Learning Representations (2022) 1

    Villegas, R., Babaeizadeh, M., Kindermans, P.J., Moraldo, H., Zhang, H., Saffar, M.T., Castro, S., Kunze, J., Erhan, D.: Phenaki: Variable length video generation from open domain textual descriptions. In: International Conference on Learning Representations (2022) 1

    2022

  47. [58]

    In: 2022 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR)

    Wang, C., Chai, M., He, M., Chen, D., Liao, J.: Clip-nerf: Text-and-image driven manipulation of neural radiance fields. In: 2022 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR). IEEE (Jun 2022).https://doi. org/10.1109/cvpr52688.2022.00381, http://dx.d...

    2022

  48. [59]

    arXiv preprint arXiv:2308.06571 (2023) 1, 2, 7

    Wang, J., Yuan, H., Chen, D., Zhang, Y., Wang, X., Zhang, S.: Modelscope text- to-video technical report. arXiv preprint arXiv:2308.06571 (2023) 1, 2, 7

    2023 arXiv

  49. [60]

    Advances in Neural Information Processing Systems36 (2023) 1, 2

    Wang, Z., Lu, C., Wang, Y., Bao, F., Li, C., Su, H., Zhu, J.: Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems36 (2023) 1, 2

    2023

  50. [61]

    In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Xu, J., Wang, X., Cheng, W., Cao, Y.P., Shan, Y., Qie, X., Gao, S.: Dream3d: Zero- shot text-to-3d synthesis using 3d shape prior and text-to-image diffusion models. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (Jun 2023). https://doi.or...

    2023

  51. [62]

    CVPR Workshop on AI for Content Creation (2024) 2

    Zamani, A., Aghdam, A.G., Popa, T., Belilovsky, E.: Temporally consistent object editing in videos using extended attention. CVPR Workshop on AI for Content Creation (2024) 2

    2024

  52. [63]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Zheng, Y., Li, X., Nagano, K., Liu, S., Hilliges, O., De Mello, S.: A unified approach for text- and image-guided 4d scene generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 7300–7309 (June 2024) 2

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.