Pith. sign in

REVIEW 4 major objections 6 minor 69 references

Every Image Listens, Every Image Dances: Music-Driven Image Animation

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MuseDance claims that one reference image, one music track, and one text prompt are enough to generate a dance video that follows the music and the text, with no pose or depth sequence required.

desk verdict Plausible music+text image animation with a useful dataset, but the 'no pose guidance' claim is unverifiable as written and the 10-video test set can't carry it. read the letter →

arxiv 2501.18801 v1 pith:LZNRPQYR submitted 2025-01-30 cs.CV cs.AI

Zhikang Dong , Weituo Hao , Ju-Chiang Wang , Peng Zhang , Pawel Polak This is my paper · ORCID
classification cs.CVcs.AI
keywords music-drivenimageanimationdancevideogenerationdiffusionmodelsmusic-conditionedtext-guidedmotioncontrolbeatsynchronizationmultimodaldataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to establish that a static reference image, a piece of music, and a short text prompt are enough to generate a dance video in which the subject moves in time with the music and follows the described motion. The proposed model, MuseDance, is trained on a new dataset of 2,904 short dance clips paired with music and motion-focused captions, and it is designed to work without the pose or depth sequences that earlier dance-transfer systems require. If the claim holds, users could animate humans, animals, or arbitrary objects to dance to a chosen track using only an image and a sentence. The paper reports quantitative gains over two reimplemented baselines and gives ablations showing that its music, beat, and motion modules each contribute to quality and temporal consistency.

What carries the argument

The central mechanism is a two-stage latent diffusion pipeline. In the first stage, the model learns single-frame appearance and motion from random pairs of nearby frames, conditioning on a reference-image encoder and on a pose mask of the reference figure, while text cross-attention teaches it to separate motion semantics from appearance. In the second stage, those spatial modules are frozen and three new modules are added: a music module that injects audio embeddings through cross-attention, a beat module that turns detected beat locations into a per-frame one-hot vector and embeds it with a lookup table, and a motion module that runs self-attention over previously generated frames to keep the sequence temporally consistent. The beat one-hot encoding is the explicit mechanism that ties movement onsets to musical beats, and the frozen first stage plus temporally-aware second stage is how the paper tries to get both fidelity and synchronization.

What would settle it

Inspect the inference pipeline to see whether any pose or depth estimator runs on the reference image before generation; if it does, the no-pose claim is refuted. Separately, generate 50 clips of non-human objects and compare the timing of motion changes with detected beat onsets against a control whose beat onsets are shifted in time; if beat-aligned motion is indistinguishable from the control, the beat module fails its intended role.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that music and text can act as complete driving signals for image animation: the model listens to the audio, senses beat locations, reads the text instruction, and produces a multi-frame video of the reference figure dancing, with no per-frame pose, skeleton, depth, or style input. The authors argue that this removes the main barrier to non-expert use of dance video generation and, because the driving signal is not a human skeleton, extends animation to non-human subjects such as cartoon characters and animals. The work also contributes the first dataset built specifically for music-driven image animation, in which every video comes with its background music and a caption describing the motion while ignoring appearance and background.

Load-bearing premise

The load-bearing premise is that at inference the model works with music and text alone: the paper never states whether the pose mask used in Stage 1 is still computed for the user's reference image, and if it is, the claimed removal of pose guidance no longer holds.

Editorial extensions

If this is right

  • A user with no animation skill can generate a personalized 4-second dance clip by supplying one photo, a music file, and a sentence describing the desired motion.
  • Because the driving signal is music rather than a skeleton, the same pipeline can animate non-human subjects, which pose-based human dance transfer cannot do.
  • The model can continue generation from its previous frames, so output length can be extended beyond 4 seconds in principle, though the paper acknowledges that consistency degrades on long sequences.
  • The dataset of 2,904 video-music-caption clips gives the task a quantitative footing; future methods can be compared against the numbers reported here.
  • Ablations show each of the three added modules matters, with the motion module contributing the largest FVD gain, so the design can be pruned or extended module by module.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the no-pose inference claim survives inspection, the same architecture could be adapted to non-musical audio such as speech or sound effects, provided a training set with matching audio-video pairs exists.
  • The Stage 1 reliance on a human-pose mask suggests a testable modification: replace the mask with a class-agnostic segmentation map and measure whether non-human animation quality holds; if it does, the method would no longer be tied to human pose estimation at any point.
  • The text captions deliberately ignore appearance and background, so the model likely has weak control over scene style; adding appearance-aware captions would be a natural way to test how much of the motion-vs-appearance disentanglement is real.
  • Beat alignment is fed as a detected binary signal rather than learned from raw audio; allowing the model to predict beat salience itself could make it more robust to off-beat or rhythmically dense music.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces MuseDance, a diffusion-based framework that animates a single reference image into a dance video conditioned on music and a text prompt, aiming to eliminate pose/depth guidance at inference. The method is trained in two stages: an appearance-pretraining stage (Section 3.2) that uses DensePose masks and ReferenceNet to learn frame-level appearance and motion, and a second stage (Section 3.3) that adds music, beat, and motion modules while freezing spatial attention blocks. The authors also present a new dataset of 2,904 dance videos with music and GPT-4o-generated text captions, and they report quantitative comparisons against EDGE+DISCO and MM-Diffusion on image/video quality metrics, plus ablations. The central claims are that MuseDance achieves music-driven animation without pose or depth inputs and generalizes to non-human objects.

Significance. If the method works as claimed, it would open a useful direction in music-conditioned image animation, and the dataset could be a valuable resource for the community. The paper's attempt to replace pose/depth guidance with music and text is a reasonable goal, and the two-stage training scheme with frozen spatial attention is a plausible design. However, the current manuscript does not sufficiently validate the central claim of pose-free inference, and the quantitative evaluation is too weak to establish the superiority of the approach. The dataset contribution is potentially significant but needs clearer positioning relative to existing datasets such as AIST++.

major comments (4)
  1. [§3.2, Eq. (1); §3.3] The paper never states whether the DensePose mask Di is still computed and injected at inference time. Equation (1) adds Conv(Di) to the reference latent in the appearance-pretraining stage, and Section 3.3 says only that spatial attention blocks are frozen, with no mention of removing or zeroing the mask branch. If DensePose is still required at test time, the Abstract's claim that MuseDance 'eliminates the need for complex motion guidance inputs, such as pose or depth sequences' is false. Please state explicitly what inputs are used at inference, and provide an ablation that replaces the mask with zeros or removes the mask encoder entirely, including for non-human objects where DensePose is known to be unreliable.
  2. [§4.1, §4.3] The test-time text captions are generated by GPT-4o from frames sampled from the target videos themselves. Therefore the model is conditioned on a dense, explicit description of the exact motion it is asked to reproduce, and the reported PSNR/SSIM/LPIPS/FVD metrics largely measure reconstruction of a described motion rather than open-ended music-driven generation. Please evaluate using captions that are not derived from the target videos, or set up the evaluation so that the text condition is independent of the ground-truth motion, and report the corresponding metrics.
  3. [§4.3] The quantitative evaluation is based on only 10 test videos, with no error bars, confidence intervals, or significance tests. Moreover, no metric directly quantifies music-motion synchronization or beat alignment, even though synchronization with music is one of the two central claims of the paper. Please run the evaluation on a larger test set with statistical significance reporting, and add a beat-alignment metric or a human perceptual study that specifically measures whether the generated motion follows the music.
  4. [§4.1] The dataset is introduced as 'the first music-dance video dataset,' but Table 1 lists AIST++ as an existing music-dance video dataset with 1,408 videos and 60 songs. The novelty claim should be qualified, for example by stating that this is the first music-dance dataset with paired text descriptions of motion, or the first dataset of this scale with diverse human and non-human subjects.
minor comments (6)
  1. [§4.1] The text says the sub-videos 'vary in length from 10 to 30 seconds' and then states that 'each paired with background music and lasting approximately 4 seconds'; please clarify that the 4 seconds refers to the final preprocessed clips after splitting and manual editing.
  2. [§4.1] 'Human-object videos' should likely be 'human-subject videos' or 'human-dancer videos'; the current wording is confusing.
  3. [§3.3 and §4.2] 'Animatediff' should be spelled 'AnimateDiff' in both places for consistency with the reference [23].
  4. [§4.2] 'Librosa' should be written as 'librosa' (the library's canonical capitalization).
  5. [Table 2 and Table 3] The tables report FVD values, but the text does not explain how FVD is computed for these short clips or how many videos are used for the FVD estimate; please add this detail.
  6. [§6] The limitations section acknowledges that text descriptions lack explicit temporal information, which is a useful admission, but the paper does not discuss the potential failure mode of DensePose on non-human objects despite claiming generalization to such objects; this should be addressed.

Circularity Check

2 steps flagged · score 6.0 of 10

Partial circularity: the test text condition is generated from the target video the model is then scored against, and the 'no pose guidance' claim is assumed rather than derived because the DensePose mask branch from Eq. (1) is never said to be removed at inference.

  1. self definitional [Sec. 4.1 (Music Dance Dataset) and Sec. 4.3 (Quantitative Results)]
    "We leverage OpenAI's GPT-4o API to generate video captions, sampling each video every 10 frames and combining these samples with a text prompt for GPT-4o. ... Similar to the approach in [57], we randomly select 10 videos as the test split, including various figures, such as human and non-human ones."

    The test-time text prompt is generated from the same ground-truth video whose frames are used as the reconstruction target for SSIM/LPIPS/PSNR/FVD. The caption therefore already encodes the exact motion to be reproduced, so the semantic-preservation and similarity scores are partly guaranteed by a target-to-input caption channel rather than by the model's music-driven generalization. The evaluation measures reconstruction through a text description of the target, not a genuinely independent prediction.

  2. other [Abstract; Sec. 3.2 Eq. (1); Sec. 3.3 (Dynamic Trigger Video Generation)]
    "Unlike existing approaches, MuseDance eliminates the need for complex motion guidance inputs, such as pose or depth sequences. ... Using the DensePose [22] mask, we obtain a dense and robust pose signal of the dancer in the reference image, denoted as Di. ... z0 = E(Ii) +Conv(Di), (1)"

    The paper's central differentiator is that no pose guidance is needed at inference, but the only appearance-conditioning equation injects a DensePose mask into the reference latent, and Section 3.3 only says spatial attention blocks are frozen, never that the mask branch is removed or zeroed at test time. If DensePose is still computed, the no-pose claim is false; if it is removed, the architectural change is omitted. The claim is assumed rather than derived, and the non-human-object generalization rests on DensePose, a human pose estimator.

full rationale

The GPT-4o caption loop (Section 4.1, Figure 3) is the main partial circularity: captions are derived from the target videos and then used as the conditioning text for generating and scoring against those same videos. Additionally, the abstract's 'eliminates the need for complex motion guidance inputs' is not supported by the method description, since Eq. (1) conditions on a DensePose mask and the paper never states its inference-time fate; this is a missing-support gap that undermines the headline contribution. The overlap of only 454 unique music tracks across 2,904 clips is a data-contamination concern rather than a derivation-level circularity. Self-citations appear only as background references and are not load-bearing. The architecture itself contains genuinely trainable music, beat, and motion modules, and Table 3's ablations show they affect quality, so the work is not wholly circular; however, the measured 'text semantic preservation' and the no-pose claim both reduce, in part, to the paper's own construction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The model is a learned neural system; no explicit data-fitted constants appear in a derivation. The key assumptions are the transferability of pretrained components, the reliability of DensePose for arbitrary objects, the sufficiency of beat as a temporal signal, and the quality of GPT-4o captions. No new physical or conceptual entities are introduced; the music, beat, and motion modules are architectural components.

assumptions (4)
  • domain assumption Pretrained Stable Diffusion weights transfer to the dance animation task.
    The model is initialized from stable-diffusion-v1-5 (Section 4.2); the entire approach assumes these weights provide a good prior for video generation.
  • domain assumption DensePose provides a reliable dense pose signal for the reference dancer.
    Section 3.2 uses DensePose mask Di as a conditioning input in stage 1; for non-human objects this assumption is questionable.
  • domain assumption Music beat locations, extracted by Librosa, are a strong and sufficient temporal signal for dance alignment.
    Section 3.3 builds the beat alignment module on this premise; no evidence is given that beat alone suffices for synchronization.
  • domain assumption GPT-4o-generated captions accurately and completely describe the motion in each video without appearance information.
    Section 4.1 uses these captions as ground-truth text for both training and evaluation; errors in captions would directly bias the motion-text alignment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Every Image Listens, Every Image Dances: Music-Driven Image Animation." pith.science (2026). https://pith.science/paper/LZNRPQYR

@misc{pith2026250118801,
  author       = {Pith},
  title        = {Pith review of: Every Image Listens, Every Image Dances: Music-Driven Image Animation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LZNRPQYR}},
  note         = {Machine review of arXiv:2501.18801}
}
read the original abstract

Image animation has become a promising area in multimodal research, with a focus on generating videos from reference images. While prior work has largely emphasized generic video generation guided by text, music-driven dance video generation remains underexplored. In this paper, we introduce MuseDance, an innovative end-to-end model that animates reference images using both music and text inputs. This dual input enables MuseDance to generate personalized videos that follow text descriptions and synchronize character movements with the music. Unlike existing approaches, MuseDance eliminates the need for complex motion guidance inputs, such as pose or depth sequences, making flexible and creative video generation accessible to users of all expertise levels. To advance research in this field, we present a new multimodal dataset comprising 2,904 dance videos with corresponding background music and text descriptions. Our approach leverages diffusion-based methods to achieve robust generalization, precise control, and temporal consistency, setting a new baseline for the music-driven image animation task.

Figures

Figures reproduced from arXiv: 2501.18801 by the authors.

Figure 1
Figure 1. MuseDance generates a dancing video from a reference image, synchronizing movements to the provided music, aligning with [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. In the first training stage, we train the model to capture spatial information by generating individual frames, with reference and [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. An example of textual data generation, we provide a series of frames and a detailed prompt to instruct GPT-4o to generate motion [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Music driven dancing video generation on non-human objects. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Dance video generations with the same text prompt but [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 38 canonical work pages

  1. [1]

    Pose with style: Detail- preserving pose-guided image synthesis with conditional stylegan

    Badour Albahar, Jingwan Lu, Jimei Yang, Zhixin Shu, Eli Shechtman, and Jia-Bin Huang. Pose with style: Detail- preserving pose-guided image synthesis with conditional stylegan. ACM Transactions on Graphics (TOG) , 40(6):1– 11, 2021. 3

    2021

  2. [2]

    Stochastic variational video prediction

    Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. arXiv preprint arXiv:1710.11252, 2017. 2

    arXiv 2017

  3. [3]

    Feature-based image metamorphosis

    Thaddeus Beier and Shawn Neely. Feature-based image metamorphosis. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 529–536. 2023. 3

    work page 2023

  4. [4]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 2, 3

    arXiv 2023

  5. [5]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023. 2

    2023

  6. [6]

    Video rewrite: Driving visual speech with audio

    Christoph Bregler, Michele Covell, and Malcolm Slaney. Video rewrite: Driving visual speech with audio. In Sem- inal Graphics Papers: Pushing the Boundaries, Volume 2 , pages 715–722. 2023. 3

    work page 2023

  7. [7]

    Video generation models as world simu- lators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simu- lators. https://openai.com/research/video_ generation _ models _ as _ world _ simulators,

    work page

  8. [8]

    Lan- guage models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. Advances in neural in- formation processing systems, 33:1877–1901, 2020. 1

    1901

Show all 69 references
  1. [9]

    Im- proved conditional vrnns for video prediction

    Lluis Castrejon, Nicolas Ballas, and Aaron Courville. Im- proved conditional vrnns for video prediction. In Proceed- ings of the IEEE/CVF international conference on computer vision, pages 7608–7617, 2019. 2

    2019

  2. [10]

    Everybody dance now

    Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros. Everybody dance now. In Proceedings of the IEEE/CVF international conference on computer vision , pages 5933–5942, 2019. 3, 6

    2019

  3. [11]

    Magicpose: Realistic human poses and facial expressions retargeting with identity-aware diffusion

    Di Chang, Yichun Shi, Quankai Gao, Hongyi Xu, Jessica Fu, Guoxian Song, Qing Yan, Yizhe Zhu, Xiao Yang, and Mo- hammad Soleymani. Magicpose: Realistic human poses and facial expressions retargeting with identity-aware diffusion. In Forty-first International Conference on Machi...

    2023

  4. [12]

    Livephoto: Real image animation with text-guided motion control

    Xi Chen, Zhiheng Liu, Mengting Chen, Yutong Feng, Yu Liu, Yujun Shen, and Hengshuang Zhao. Livephoto: Real image animation with text-guided motion control. In Eu- ropean Conference on Computer Vision , pages 475–491. Springer, 2025. 1

    2025

  5. [13]

    Markerless human motion transfer

    German KM Cheung, Simon Baker, Jessica Hodgins, and Takeo Kanade. Markerless human motion transfer. In Pro- ceedings. 2nd International Symposium on 3D Data Process- ing, Visualization and Transmission, 2004. 3DPVT 2004. , pages 373–378. IEEE, 2004. 3

    2004

  6. [14]

    Stochastic video generation with a learned prior

    Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. In International conference on machine learning, pages 1174–1183. PMLR, 2018. 2

    2018

  7. [15]

    Cp-pinns: Changepoints de- tection in pdes using physics informed neural networks with total-variation penalty

    Zhikang Dong and Pawel Polak. Cp-pinns: Changepoints de- tection in pdes using physics informed neural networks with total-variation penalty. In Machine Learning and the Physi- cal Sciences Workshop, NeurIPS 2023, 2023. 1

    2023

  8. [16]

    Mamba fusion: Learning actions through questioning

    Zhikang Dong, Apoorva Beedu, Jason Sheinkopf, and Irfan Essa. Mamba fusion: Learning actions through questioning. arXiv preprint arXiv:2409.11513, 2024

    2024 arXiv

  9. [17]

    Mapping the in- visible: Face-gps for facial muscle dynamics in videos

    Zhikang Dong, Juni Kim, and Paweł Polak. Mapping the in- visible: Face-gps for facial muscle dynamics in videos. In 2024 IEEE First International Conference on Artificial In- telligence for Medicine, Health and Care (AIMHC) , pages 209–213. IEEE, 2024

    2024

  10. [18]

    Musechat: A conversational music recommen- dation system for videos

    Zhikang Dong, Xiulong Liu, Bin Chen, Pawel Polak, and Peng Zhang. Musechat: A conversational music recommen- dation system for videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12775–12785, 2024. 1

    2024

  11. [19]

    Recognizing action at a dis- tance

    Efros, Berg, Mori, and Malik. Recognizing action at a dis- tance. In Proceedings Ninth IEEE International Conference on Computer Vision, pages 726–733. IEEE, 2003. 3

    2003

  12. [20]

    Structure and content-guided video synthesis with diffusion models

    Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7346–7356, 2023. 3

    2023

  13. [21]

    Ast: Audio spectrogram transformer

    Yuan Gong, Yu-An Chung, and James Glass. Ast: Audio spectrogram transformer. arXiv preprint arXiv:2104.01778,

    arXiv

  14. [22]

    Densepose: Dense human pose estimation in the wild

    Rıza Alp G ¨uler, Natalia Neverova, and Iasonas Kokkinos. Densepose: Dense human pose estimation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7297–7306, 2018. 2, 4

    2018

  15. [23]

    Animatediff: Animate your personalized text- to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text- to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023. 2, 5, 6

    2023 arXiv

  16. [24]

    Image quality metrics: Psnr vs

    Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference on pattern recognition, pages 2366–2369. IEEE, 2010. 6

    2010

  17. [25]

    Animate anyone: Consistent and controllable image- to-video synthesis for character animation

    Li Hu. Animate anyone: Consistent and controllable image- to-video synthesis for character animation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8153–8163, 2024. 3, 4

    2024

  18. [26]

    Learning high fi- delity depths of dressed humans by watching social media dance videos

    Yasamin Jafarian and Hyun Soo Park. Learning high fi- delity depths of dressed humans by watching social media dance videos. In Proceedings of the IEEE/CVF Conference 9 on Computer Vision and Pattern Recognition, pages 12753– 12762, 2021. 6

    2021

  19. [27]

    Dreampose: Fashion image-to-video synthesis via stable diffusion

    Johanna Karras, Aleksander Holynski, Ting-Chun Wang, and Ira Kemelmacher-Shlizerman. Dreampose: Fashion image-to-video synthesis via stable diffusion. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 22623–22633. IEEE, 2023. 3

    2023

  20. [28]

    Analyzing and improv- ing the image quality of stylegan

    Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improv- ing the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8110–8119, 2020. 3

    2020

  21. [29]

    Kling video model

    Kuaishou. Kling video model. https : / / kling . kuaishou.com/en, 2024. 3

    2024

  22. [30]

    https://www.pika.art/

    Pika Labs. https://www.pika.art/. 3

  23. [31]

    Stochastic adversarial video prediction

    Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:1804.01523 , 2018. 2

    2018 arXiv

  24. [32]

    Ai choreographer: Music conditioned 3d dance generation with aist++

    Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision , pages 13401– 13412, 2021. 6

    2021

  25. [33]

    Video generation from text

    Yitong Li, Martin Min, Dinghan Shen, David Carlson, and Lawrence Carin. Video generation from text. InProceedings of the AAAI conference on artificial intelligence, 2018. 2

    2018

  26. [34]

    Tackling data bias in music-avqa: Crafting a balanced dataset for unbiased question-answering

    Xiulong Liu, Zhikang Dong, and Peng Zhang. Tackling data bias in music-avqa: Crafting a balanced dataset for unbiased question-answering. In Proceedings of the IEEE/CVF Win- ter Conference on Applications of Computer Vision , pages 4478–4487, 2024. 1

    2024

  27. [35]

    M2c: Concise mu- sic representation for 3d dance generation

    Matthew Marchellus and In Kyu Park. M2c: Concise mu- sic representation for 3d dance generation. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 3126–3135, 2023. 3

    2023

  28. [36]

    To create what you tell: Generating videos from cap- tions

    Yingwei Pan, Zhaofan Qiu, Ting Yao, Houqiang Li, and Tao Mei. To create what you tell: Generating videos from cap- tions. In Proceedings of the 25th ACM international confer- ence on Multimedia, pages 1789–1798, 2017. 2

    2017

  29. [37]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 1

    2023 arXiv

  30. [38]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  31. [39]

    Hierarchical text-conditional image gener- ation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gener- ation with clip latents. arXiv preprint arXiv:2204.06125, 1 (2):3, 2022. 3

    2022 arXiv

  32. [40]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1, 3

    2022

  33. [41]

    Mm-diffusion: Learning multi-modal diffusion mod- els for joint audio and video generation

    Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusion: Learning multi-modal diffusion mod- els for joint audio and video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern...

    2023

  34. [42]

    Tempo- ral generative adversarial nets with singular value clipping

    Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Tempo- ral generative adversarial nets with singular value clipping. In Proceedings of the IEEE international conference on com- puter vision, pages 2830–2839, 2017. 2

    2017

  35. [43]

    First order motion model for image animation

    Aliaksandr Siarohin, St ´ephane Lathuili`ere, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Advances in neural information processing systems, 32, 2019. 1

    2019

  36. [44]

    Motion representations for ar- ticulated animation

    Aliaksandr Siarohin, Oliver J Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion representations for ar- ticulated animation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 13653–13662, 2021. 1

    2021

  37. [45]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 ,

    arXiv

  38. [46]

    Bailando: 3d dance generation by actor-critic gpt with choreographic memory

    Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu. Bailando: 3d dance generation by actor-critic gpt with choreographic memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11050–...

    2022

  39. [47]

    Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2

    Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elho- seiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3626–3636, 2022. 2

    2022

  40. [48]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 3

    2010 arXiv

  41. [49]

    Lamda: Language models for dialog applications

    Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022. 1

    2022 arXiv

  42. [50]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 1

    2023 arXiv

  43. [51]

    Edge: Editable dance generation from music

    Jonathan Tseng, Rodrigo Castellon, and Karen Liu. Edge: Editable dance generation from music. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 448–458, 2023. 3, 7

    2023

  44. [52]

    Aist dance video database: Multi-genre, 10 multi-dancer, and multi-camera database for dance informa- tion processing

    Shuhei Tsuchida, Satoru Fukayama, Masahiro Hamasaki, and Masataka Goto. Aist dance video database: Multi-genre, 10 multi-dancer, and multi-camera database for dance informa- tion processing. In ISMIR, page 6, 2019. 6

    2019

  45. [53]

    Mocogan: Decomposing motion and content for video generation

    Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1526–1535,

  46. [54]

    To- wards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. 6

    2018 arXiv

  47. [55]

    Decomposing motion and con- tent for natural video sequence prediction

    Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and con- tent for natural video sequence prediction. arXiv preprint arXiv:1706.08033, 2017. 2

    2017 arXiv

  48. [56]

    Generating videos with scene dynamics

    Carl V ondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. Advances in neu- ral information processing systems, 29, 2016. 2

    2016

  49. [57]

    Disco: Disentangled control for realistic human dance generation

    Tan Wang, Linjie Li, Kevin Lin, Yuanhao Zhai, Chung- Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Disco: Disentangled control for realistic human dance generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ,...

    2024

  50. [58]

    Dance any beat: Blending beats with visuals in dance video generation

    Xuanchen Wang, Heng Wang, Dongnan Liu, and Weidong Cai. Dance any beat: Blending beats with visuals in dance video generation. arXiv preprint arXiv:2405.09266, 2024. 3

    2024 arXiv

  51. [59]

    Image quality assessment: from error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Si- moncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004. 6

    2004

  52. [60]

    Video-based characters: creating new human performances from a multi-view video database

    Feng Xu, Yebin Liu, Carsten Stoll, James Tompkin, Gau- rav Bharaj, Qionghai Dai, Hans-Peter Seidel, Jan Kautz, and Christian Theobalt. Video-based characters: creating new human performances from a multi-view video database. In ACM SIGGRAPH 2011 papers, pages 1–10. 2011. 3

    2011

  53. [61]

    Hallo: Hierarchical audio-driven vi- sual synthesis for portrait image animation

    Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Li- wei Zhang, Ce Liu, Jingdong Wang, Luc Van Gool, Yao Yao, and Siyu Zhu. Hallo: Hierarchical audio-driven vi- sual synthesis for portrait image animation. arXiv preprint arXiv:2406.08801, 2024. 4, 5

    2024 arXiv

  54. [62]

    Magicanimate: Temporally consistent human im- age animation using diffusion model

    Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human im- age animation using diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2024

  55. [63]

    Lm2d: Lyrics-and music-driven dance synthesis

    Wenjie Yin, Xuejiao Zhao, Yi Yu, Hang Yin, Danica Kragic, and M ˚arten Bj ¨orkman. Lm2d: Lyrics-and music-driven dance synthesis. arXiv preprint arXiv:2403.09407, 2024. 3

    2024 arXiv

  56. [64]

    Generating videos with dynamics-aware implicit generative adversarial net- works

    Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial net- works. arXiv preprint arXiv:2202.10571, 2022. 2

    2022 arXiv

  57. [65]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 3, 4

    2023

  58. [66]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 586–595, 2018. 6

    2018

  59. [67]

    Thin-plate spline motion model for image animation

    Jian Zhao and Hui Zhang. Thin-plate spline motion model for image animation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 3657–3666, 2022. 1

    2022

  60. [68]

    Hedge fund portfolio construction using polymodel theory and itransformer

    Siqiao Zhao, Zhikang Dong, Zeyu Cao, and Raphael Douady. Hedge fund portfolio construction using polymodel theory and itransformer. arXiv preprint arXiv:2408.03320 ,

    arXiv

  61. [2024]

    Accessed: 2024-08-22. 3

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.