Pith. sign in

REVIEW 3 major objections 5 minor 57 references

Semantics-Aware Human Motion Generation from Audio Instructions

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes that spoken audio instructions can replace text as the conditioning signal for 3D human motion generation, and reports near-parity with text-conditioned models while cutting inference latency by more than half.

desk verdict The central claim that audio can replace text for semantic motion conditioning is plausible but not yet proven, because the quantitative comparisons use TTS audio synthesized from the same text annotations given to text baselines. read the letter →

arxiv 2505.23465 v1 pith:JQUPLKQM submitted 2025-05-29 cs.SD cs.CV

Zi-An Wang , Shihao Zou , Shiyao Yu , Mingyuan Zhang , Chao Dong This is my paper
classification cs.SDcs.CV
keywords humanmotiongenerationaudio-conditionedmaskedgenerativetransformermemory-retrievalattentiontext-to-speechdataaugmentationsemanticaudioencodingmultimodallearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces the task of generating 3D human motion directly from spoken audio instructions, bypassing speech-to-text. It argues that audio can carry the same semantic content as text for motion generation, provided the audio is encoded into compact motion-relevant features rather than matched only to rhythm. To test this, the authors build an end-to-end masked-transformer model that conditions on WavLM audio features compressed by a memory-retrieval attention module, and they create oral-style versions of HumanML3D and KIT by rephrasing descriptions conversationally and synthesizing speech with varied voices. On standard benchmarks the audio-conditioned model lands close to the text-conditioned MoMask, and the direct pipeline generates motions over 50% faster than a speech-recognition-then-text pipeline.

What carries the argument

The load-bearing component is a memory-retrieval based attention module: a set of learnable memory tokens that act as key-value pairs, so that each audio feature sequence generates a query vector and attends over the memory to produce a compact, fixed-size condition vector $y \in \mathbb{R}^{d_y}$ regardless of the original audio length. This addresses the sparse and lengthy nature of WavLM-extracted audio features. The condition feeds a Masked Transformer that iteratively generates base motion tokens and a Residual Transformer that generates residual tokens layer by layer, following the RVQ-VAE motion latent representation from MoMask and the masked generative paradigm from MaskGIT.

What would settle it

Run the same evaluation on a test set of real human recordings of the same instructions, with and without added noise, and compare R-precision and FID against the synthetic-audio test set; a substantial drop in R-precision or a large FID degradation would show that the equivalence claim holds only for synthesized speech rather than for real microphone input.

Watch

Extended reading notes

Core claim

The central claim is that audio signals can serve as an effective alternative to text for semantic conditioning in human motion generation. Built as an end-to-end masked generative transformer, the model takes raw audio instructions, compresses them into a fixed-size condition through a memory-retrieval attention module, and produces motion through base and residual quantization codes. On the HumanML3D original test set the audio-conditioned model reaches R-Precision Top-1 0.519 against MoMask's 0.521 with text, while FID is 0.121 versus 0.045; the paper reads this as audio conveying semantics comparable to text. It also shows that models trained on the newly synthesized oral-style data outperform those trained on the original text-style data when tested on spoken instructions, supporting the value of conversational audio training data.

Load-bearing premise

The conclusion that audio matches text depends on the assumption that the Tortoise-synthesized speech used for training and testing is a faithful stand-in for real spoken instructions; the only real-audio evidence in the paper is three qualitative examples.

Editorial extensions

If this is right

  • Audio-conditioned motion generation can be built without a speech-to-text cascade, simplifying deployment and reducing inference latency by over 50%.
  • Spoken, conversational instructions become usable as conditioning inputs, and training on oral-style rephrased data better matches how users actually speak.
  • The memory-retrieval compression technique is a reusable pattern for conditioning transformers on long, sparse signals beyond audio, such as video or sensor streams.
  • Because audio and text conditions land close in quality, future systems can choose the input modality based on interface constraints rather than expected generation quality.
  • The near-parity result suggests that adding a speech recognizer before a text-to-motion model may be unnecessary when the audio encoder is trained end-to-end for the motion task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the TTS-to-real gap is closed by larger real-audio evaluation, this points toward voice-driven animation tools where a user narrates an action and sees motion nearly in real time; the latency result makes that plausible at interactive rates.
  • The same memory-retrieval conditioning could be applied to other long-horizon inputs where a fixed-size semantic bottleneck is needed, and a direct test would be to swap WavLM for a video or sensor encoder and measure whether the compression still preserves task-relevant semantics.
  • A testable extension is multilingual and multi-stage instructions, which the paper lists as untested limitations; the framework's performance there would reveal whether the semantic equivalence holds beyond short English single actions.
  • The finding that oral-trained models outperform original-trained models on oral-style tests suggests that collecting more natural spoken instruction data may contribute as much as architectural improvements to real-world usability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes an end-to-end framework for generating 3D human motion from audio instructions. The method encodes raw speech with WavLM, compresses variable-length features through a memory-retrieval attention module, and uses a masked generative transformer with a residual transformer to produce motion tokens under classifier-free guidance. To obtain training data, the authors rewrite existing text annotations (HumanML3D, KIT-ML) into conversational style with ChatGPT and synthesize corresponding audio with the Tortoise TTS model, yielding 'Original' and 'Oral' audio-motion datasets. Experiments compare the audio-conditioned model against text-conditioned baselines on those datasets, reporting near-parity with MoMask (e.g., R-Precision Top-1 0.519 vs 0.521 on HumanML3D) and over 50% lower inference latency than a Whisper-based cascaded pipeline. The paper also includes ablations on audio encoders, compression methods, and a small qualitative robustness test with real and noisy recordings.

Significance. If the central claim holds, the paper introduces a useful new task and a practical interface for motion generation, and the efficiency gain over an ASR cascade is a concrete, reproducible advantage. The work is also constructive in creating augmented audio-motion datasets, which could support future research. The paper ships a clear architecture description, quantitative comparisons following standard text-to-motion metrics, and an explicit efficiency measurement. The main value is conditional on the evaluation design: the reported parity between audio and text is established only for synthesized speech derived from the same text descriptions, so the significance for real voice input is not yet demonstrated.

major comments (3)
  1. [§4.1, §4.2, Tables 2–3] The central claim that 'audio signals can serve as an effective alternative to text for semantic conditioning' (Section 5.1) is evaluated exclusively on audio synthesized by Tortoise from the same text annotations that condition the text baselines. This design conflates direct audio-to-motion semantic extraction with the ability of WavLM/Tortoise to preserve text information in a noisy channel. The only real-audio evidence is the qualitative example in Fig. 7 (three recorded instructions, one SNR condition), which is not sufficient to support the headline conclusion for actual user voice input. The paper should either add quantitative evaluation on real recorded/perturbed speech (e.g., R-Precision, FID, or at least word-error-rate or ASR-fidelity statistics for the synthesized audio) or explicitly restrict the claim to synthesized audio instructions.
  2. [Table 6] The Mem-Retr(Ours) row reports MM Dist 1.221, whereas Table 3 lists the same model (Ours*) with MM Dist 2.955 and MultiModality 1.221 on the same HumanML3D Original setting. The column in Table 6 appears mislabeled, because 1.221 matches the MultiModality value in Table 3. This discrepancy prevents the reader from verifying the claimed improvement of the memory-retrieval module over the Transformer encoder (MM Dist 3.259), and it should be corrected or clarified before the ablation can be assessed.
  3. [§5.3, Table 4] The efficiency comparison against the cascaded method measures only speed, not generation quality of the cascade. The text says the cascaded approach is 'generally comparable' to text-to-motion, but no quantitative comparison is provided for the specific Whisper-to-MoMask pipeline used here. Since the paper's efficiency argument depends on the end-to-end method maintaining quality while being faster, reporting the cascade's R-Precision/FID on the same test split would make the comparison complete.
minor comments (5)
  1. [§3.3] Equation (2) contains a typo: 'finds the closet code' should read 'finds the closest code'.
  2. [§4.1] The text states that Tortoise does not support specifying speakers directly and then says 'we generate data using randomly selected voices'; this is slightly contradictory and should be rephrased to clarify how speaker variation is obtained.
  3. [§5.1, Fig. 7] The caption says three audio instructions were tested under three conditions, but the figure shows only one representative example; the paper should state explicitly that the other two examples are omitted for space or include them in supplementary material.
  4. [References] The reference list includes placeholder entries (e.g., '[1] The frobnicatable foo filter', '[2] Frobnication tutorial') that appear to be template leftovers and are not cited in the text; these should be removed.
  5. [§5.3, Table 6] The ablation on audio feature compression is reported only on HumanML3D; adding the same ablation on KIT-ML would strengthen the generality of the conclusion, though this is not essential for the main claim.

Circularity Check

1 steps flagged · score 6.0 of 10

The audio-vs-text semantic parity is an artifact of dataset construction: every quantitative comparison uses audio synthesized by Tortoise from the exact text annotations used to condition the text baselines, so the headline claim reduces to a property of the TTS pipeline rather than an independent property of audio instructions.

  1. self definitional [Sec. 4.1 (Original Dataset construction), Sec. 5.2 (Comparison to Text-Based Methods), Tables 2-3]
    "To address this gap, we leverage existing text-to-speech algorithms to synthesize an audio-motion paired dataset from the original text-motion datasets ... For a given text input, Tortoise synthesizes audio that not only matches human voices in timbre and quality but also captures natural features like pauses, emphasis, and tonal variation, reflecting the semantic intricacies of the text ..."

    The quantitative support for the headline claim (Sec. 5.2, Tables 2-3) uses audio generated by Tortoise from the exact text annotations that condition the text baselines (Sec. 4.1). The semantic content of the audio is therefore defined, by construction, as the semantic content of the text it is claimed to match. The observed 'parity' between audio and text conditions is thus a restatement of the dataset-generation choice, not an independent empirical discovery about audio semantics. The only non-TTS evidence is Fig. 7, which presents three qualitative real-audio samples with no R-Precision/FID numbers, so the central quantitative conclusion is self-definitional rather than independently established for real spoken input.

full rationale

The derivation of the generative model itself (WavLM features, memory-retrieval compression, masked transformer, residual transformer) is not circular: it is a standard architecture with independent ablations, and the efficiency comparison to a Whisper-based cascade is a legitimate engineering result. The circularity is confined to the semantic-parity claim. Because the audio in all quantitative tables (Original and Oral datasets) is synthesized by Tortoise from the same text annotations used to condition the text baselines, the statement that audio can serve as an equally effective condition as text is largely a restatement of the dataset-construction choice: the TTS model was explicitly selected to preserve the semantics of the source text. The conclusion would be independent only if the evaluation used natural speech with quantitative metrics; the paper provides only three qualitative real-audio samples (Fig. 7) and reports no word-error-rate or SNR sweep to quantify how much of the semantics survives TTS. This is partial, not total, circularity: the model still has to learn to decode acoustic realizations, and the architectural contributions have independent value. A separate verification concern, not treated as a circularity step, is that Table 6 lists MM Dist 1.221 for Mem-Retr(Ours) while Table 3 lists MultiModality 1.221 for Ours*, suggesting a mislabeled column that makes the ablation numbers difficult to audit without released code or data.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The ledger shows that the central claim rests on two empirical proxies (TTS and LLM rewriting) and on reusing pretrained models, rather than on new physical entities or fitted physical parameters. Most free parameters are standard training hyperparameters, but they are not derived from first principles.

free parameters (5)
  • Number of memory tokens = 256
    Hand-chosen hyperparameter in Section 3.2 and Implementation Details; controls audio compression capacity.
  • Memory token dimension = 512
    Dimension of learnable memory tokens, chosen by hand in Implementation Details.
  • Audio condition drop rate = 0.2
    Hand-chosen dropout for classifier-free guidance training, stated in Implementation Details.
  • CFG guidance scales = s=4,5 on HumanML3D; s=2,5 on KIT-ML
    Hand-tuned guidance scales for masked and residual transformers, stated in Implementation Details.
  • Fixed sequence length = 10
    Generation uses a fixed sequence length of 10 (units not specified), capping motion duration, stated in Implementation Details.
assumptions (5)
  • domain assumption WavLM final-layer features contain enough motion-relevant semantics.
    Section 3.1 selects WavLM because of generalization; no task-specific validation of its features for motion semantics.
  • domain assumption Tortoise TTS preserves the semantics of the source text.
    Section 4.1 synthesizes training and test audio from text; if TTS changes meaning, the central comparison fails.
  • domain assumption ChatGPT-3.5 rewrites preserve original action semantics.
    Section 4.2 uses LLM rewriting to create oral descriptions; semantic fidelity is only illustrated with examples, not measured.
  • domain assumption Text-motion evaluation metrics are valid when conditioning is audio.
    Section 5 applies R-precision, FID, and MM Dist from T2M and MotionDiffuse without specifying how audio features are used in retrieval.
  • domain assumption RVQ-VAE and masked transformer from MoMask transfer to audio-conditioned generation.
    Section 3.3 adopts MoMask components; no derivation of why audio conditions interact well with the residual quantization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Semantics-Aware Human Motion Generation from Audio Instructions." pith.science (2026). https://pith.science/paper/JQUPLKQM

@misc{pith2026250523465,
  author       = {Pith},
  title        = {Pith review of: Semantics-Aware Human Motion Generation from Audio Instructions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JQUPLKQM}},
  note         = {Machine review of arXiv:2505.23465}
}
read the original abstract

Recent advances in interactive technologies have highlighted the prominence of audio signals for semantic encoding. This paper explores a new task, where audio signals are used as conditioning inputs to generate motions that align with the semantics of the audio. Unlike text-based interactions, audio provides a more natural and intuitive communication method. However, existing methods typically focus on matching motions with music or speech rhythms, which often results in a weak connection between the semantics of the audio and generated motions. We propose an end-to-end framework using a masked generative transformer, enhanced by a memory-retrieval attention module to handle sparse and lengthy audio inputs. Additionally, we enrich existing datasets by converting descriptions into conversational style and generating corresponding audio with varied speaker identities. Experiments demonstrate the effectiveness and efficiency of the proposed framework, demonstrating that audio instructions can convey semantics similar to text while providing more practical and user-friendly interactions.

Figures

Figures reproduced from arXiv: 2505.23465 by the authors.

Figure 1
Figure 1. Overview of Our Work. Given an audio instruction as the conditional signal (with text included for reference purposes only), our generative model is able to produce high-quality human motion sequences that accurately align with the semantics of the audio input. tions. These compact features will serve as conditioning inputs for generating corresponding motions. In the area of generative models, diffusion-based gener… view at source ↗
Figure 2
Figure 2. Distribution of Audio Feature Lengths. WavLM [11] is utilized to extract audio features from the augmented Oral Datasets derived from Hu￾manML3D [21] and KIT [39]. A statistical analysis of these feature lengths re￾veals significant variability, with some features exhibiting notably long lengths. This variability presents challenges for processing conditional signals, as it complicates the integration of audio data … view at source ↗
Figure 3
Figure 3. Audio Conditions Processing Pipeline. The audio features extracted by WavLM [11] are processed through a memory-retrieval based module, which standardizes the varying lengths of the input audio conditions. This module ensures that all audio signals are converted into a consistent length, facilitating smooth integration with subsequent components in the pipeline. The processed features will be used as conditional sig… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Overview of Our Generative Framework in Training and Inference. The framework consists of two key components: The Masked Transformer is designed to model the relationship between audio conditions and the base motion codes, which capture the principal components of the …
Figure 5
Figure 5. Figure 5: Overview of the Audio-Motion Dataset Augmentation Process. The texts from existing datasets, KIT [39] and HumanML3D [21], are fed into the text2speech model, Tortoise [38], generating audio signals with random speaker identities to create the Original Dataset. Addition…
Figure 6
Figure 6. Figure 6: Qualitative Comparison between Models Trained on Original and Oral Datasets. As illustrated in the figure, a comparison of the performance of models trained on the Oral and Original datasets reveals that the model trained on the Oral dataset demonstrates greater robust…
Figure 7
Figure 7. Figure 7: Qualitative result with different audio instructions. Under conditions of synthetic audio, real user input, and noisy speech instruction, the proposed method is able to correctly generate actions that align with the command requirements, demonstrating the robustness of…
Figure 8
Figure 8. Figure 8: Comparison with Cascaded Method. Our end-to-end method can generate more than 50% faster than cascaded approach, while maintaining high generation quality. Audio Encoding FID↓ R-top1↑ R-top2↑ R-top3↑ MM Dist↓ AvgPool-8 0.733 0.376 0.539 0.627 4.244 AvgPool-32 1.258 0.4…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 43 canonical work pages

  1. [1]

    Face and Gesture submission ID 324

    Authors, The frobnicatable foo filter, 2012. Face and Gesture submission ID 324. Supplied as additional material fg324.pdf

    work page 2012

  2. [2]

    Supplied as additional material tr.pdf

    Authors, Frobnication tutorial, 2012. Supplied as additional material tr.pdf

    work page 2012

  3. [3]

    Alpher, Frobnication, Journal of Foo 12 (2002) 234–778

    A. Alpher, Frobnication, Journal of Foo 12 (2002) 234–778

    work page 2002

  4. [4]

    Alpher, J

    A. Alpher, J. P. N. Fotheringham-Smythe, Frobnication revisited, Journal of Foo 13 (2003) 234–778

    work page 2003

  5. [5]

    Alpher, J

    A. Alpher, J. P. N. Fotheringham-Smythe, G. Gamow, Can a machine frobnicate?, Journal of Foo 14 (2004) 234–778

    work page 2004

  6. [6]

    Goodfellow, J

    I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y . Bengio, Generative adversarial networks, Com- munications of the ACM 63 (2020) 139–144. Publisher: ACM New York, NY , USA

    work page 2020

  7. [7]

    D. P. Kingma, Auto-encoding variational bayes, arXiv preprint arXiv:1312.6114 (2013)

    arXiv 2013

  8. [8]

    Papamakarios, E

    G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed, B. Lakshmi- narayanan, Normalizing flows for probabilistic modeling and inference, Journal of Machine Learning Research 22 (2021) 1–64

    work page 2021

Show all 57 references
  1. [9]

    J. Ho, A. Jain, P. Abbeel, Denoising diffusion probabilistic models, Ad- vances in neural information processing systems 33 (2020) 6840–6851

    2020

  2. [10]

    Chang, H

    H. Chang, H. Zhang, L. Jiang, C. Liu, W. T. Freeman, Maskgit: Masked generative image transformer, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022, pp. 11315–11325

    2022

  3. [11]

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, et al., Wavlm: Large-scale self-supervised pre- training for full stack speech processing, IEEE Journal of Selected Topics in Signal Processing 16 (2022) 1505–1518

    2022

  4. [12]

    D ´efossez, J

    A. D ´efossez, J. Copet, G. Synnaeve, Y . Adi, High fidelity neural audio compression, arXiv preprint arXiv:2210.13438 (2022)

    2022 arXiv

  5. [13]

    Radford, J

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, I. Sutskever, Robust speech recognition via large-scale weak supervision, in: Interna- tional conference on machine learning, PMLR, 2023, pp. 28492–28518

    2023

  6. [14]

    Girdhar, A

    R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V . Alwala, A. Joulin, I. Misra, Imagebind: One embedding space to bind them all, in: Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, 2023, pp. 15180–15190

    2023

  7. [15]

    Ahuja, L.-P

    C. Ahuja, L.-P. Morency, Language2pose: Natural language grounded pose forecasting, in: 2019 International Conference on 3D Vision, IEEE, 2019, pp. 719–728

    2019

  8. [16]

    Ghosh, N

    A. Ghosh, N. Cheema, C. Oguz, C. Theobalt, P. Slusallek, Synthesis of compositional animations from textual descriptions, in: Proceedings of the IEEE international conference on computer vision, 2021, pp. 1396– 1406

    2021

  9. [17]

    A. S. Lin, L. Wu, R. Corona, K. Tai, Q. Huang, R. J. Mooney, Generating animated videos of human activities from natural language descriptions, Learning 1 (2018) 1

    2018

  10. [18]

    Tevet, B

    G. Tevet, B. Gordon, A. Hertz, A. H. Bermano, D. Cohen-Or, Motion- clip: Exposing human motion generation to clip space, in: European Conference on Computer Vision, Springer, 2022, pp. 358–374

    2022

  11. [19]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., Learning transferable visual models from natural language supervision, in: International con- ference on machine learning, PMLR, 2021, pp. 8748–8763

    2021

  12. [20]

    Petrovich, M

    M. Petrovich, M. J. Black, G. Varol, Temos: Generating diverse human motions from textual descriptions, in: European Conference on Computer Vision, Springer, 2022, pp. 480–497

    2022

  13. [21]

    C. Guo, S. Zou, X. Zuo, S. Wang, W. Ji, X. Li, L. Cheng, Generating diverse and natural 3d human motions from text, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022, pp. 5152–5161

    2022

  14. [22]

    Van Den Oord, O

    A. Van Den Oord, O. Vinyals, et al., Neural discrete representation learn- ing, Advances in neural information processing systems 30 (2017)

    2017

  15. [23]

    Zhang, Y

    J. Zhang, Y . Zhang, X. Cun, Y . Zhang, H. Zhao, H. Lu, X. Shen, Y . Shan, Generating human motion from textual descriptions with discrete repre- sentations, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2023, pp. 14730–14740

    2023

  16. [24]

    J. Kim, J. Kim, S. Choi, Flame: Free-form language-based motion syn- thesis & editing, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 2023, pp. 8255–8263

    2023

  17. [25]

    Y . Yuan, J. Song, U. Iqbal, A. Vahdat, J. Kautz, Physdiff: Physics-guided human motion diffusion model, in: Proceedings of the IEEE international conference on computer vision, 2023, pp. 16010–16021

    2023

  18. [26]

    Tevet, S

    G. Tevet, S. Raab, B. Gordon, Y . Shafir, D. Cohen-or, A. H. Bermano, Human motion diffusion model, in: The Eleventh International Confer- ence on Learning Representations, 2023

    2023

  19. [27]

    C. Guo, Y . Mu, M. G. Javed, S. Wang, L. Cheng, Momask: Gener- ative masked modeling of 3d human motions, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2024, pp. 1900–1910

    2024

  20. [28]

    T. Tang, J. Jia, H. Mao, Dance with melody: An lstm-autoencoder ap- proach to music-oriented dance synthesis, in: Proceedings of the 26th ACM international conference on Multimedia, 2018, pp. 1598–1606

    2018

  21. [29]

    Graves, A

    A. Graves, A. Graves, Long short-term memory, Supervised sequence labelling with recurrent neural networks (2012) 37–45

    2012

  22. [30]

    H.-Y . Lee, X. Yang, M.-Y . Liu, T.-C. Wang, Y .-D. Lu, M.-H. Yang, J. Kautz, Dancing to music, Advances in neural information process- ing systems 32 (2019)

    2019

  23. [31]

    K. Chen, Z. Tan, J. Lei, S.-H. Zhang, Y .-C. Guo, W. Zhang, S.-M. Hu, Choreomaster: choreography-oriented music-driven dance synthe- sis, ACM Transactions on Graphics 40 (2021) 1–13

    2021

  24. [32]

    Ginosar, A

    S. Ginosar, A. Bar, G. Kohavi, C. Chan, A. Owens, J. Malik, Learn- 12 ing individual styles of conversational gesture, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 3497–3506

    2019

  25. [33]

    Kucherenko, D

    T. Kucherenko, D. Hasegawa, G. E. Henter, N. Kaneko, H. Kjellstr ¨om, Analyzing input and output representations for speech-driven gesture gen- eration, in: Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents, 2019, pp. 97–104

    2019

  26. [34]

    J. Li, D. Kang, W. Pei, X. Zhe, Y . Zhang, Z. He, L. Bao, Audio2gestures: Generating diverse gestures from speech audio with conditional varia- tional autoencoders, in: Proceedings of the IEEE International Confer- ence on Computer Vision, 2021, pp. 11293–11302

    2021

  27. [35]

    W. Zhu, X. Ma, D. Ro, H. Ci, J. Zhang, J. Shi, F. Gao, Q. Tian, Y . Wang, Human motion generation: A survey, IEEE Transactions on Pattern Anal- ysis and Machine Intelligence (2023)

    2023

  28. [36]

    Borsos, R

    Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi, et al., Audiolm: a language modeling approach to audio generation, IEEE transactions on audio, speech, and language processing 31 (2023) 2523–2533

    2023

  29. [37]

    A. J. OpenAI, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Ale- man, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al., Gpt-4 technical report. 2023, URL: https://arxiv. org/abs/2303.08774 (2024)

    2024 arXiv

  30. [38]

    Betker, Better speech synthesis through scaling, arXiv preprint arXiv:2305.07243 (2023)

    J. Betker, Better speech synthesis through scaling, arXiv preprint arXiv:2305.07243 (2023)

    2023 arXiv

  31. [39]

    Plappert, C

    M. Plappert, C. Mandery, T. Asfour, The kit motion-language dataset, Big data 4 (2016) 236–252

    2016

  32. [40]

    Vaswani, Attention is all you need, Advances in Neural Information Processing Systems (2017)

    A. Vaswani, Attention is all you need, Advances in Neural Information Processing Systems (2017)

    2017

  33. [41]

    Jiang, C

    J. Jiang, C. Liang, J. Yang, G. Lin, T. Zhong, Y . Zheng, Loopy: Taming audio-driven portrait avatar with long-term motion dependency, arXiv preprint arXiv:2409.02634 (2024)

    2024 arXiv

  34. [42]

    Zhang, Z

    M. Zhang, Z. Cai, L. Pan, F. Hong, X. Guo, L. Yang, Z. Liu, Motiondif- fuse: Text-driven human motion generation with diffusion model, arXiv preprint arXiv:2208.15001 (2022)

    2022 arXiv

  35. [43]

    C. Guo, X. Zuo, S. Wang, L. Cheng, Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts, in: European Conference on Computer Vision, Springer, 2022, pp. 580– 597

    2022

  36. [44]

    X. Chen, B. Jiang, W. Liu, Z. Huang, B. Fu, T. Chen, G. Yu, Executing your commands via motion diffusion in latent space, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023, pp. 18000–18010

    2023

  37. [45]

    Zhang, X

    M. Zhang, X. Guo, L. Pan, Z. Cai, F. Hong, H. Li, L. Yang, Z. Liu, Remodiffuse: Retrieval-augmented motion diffusion model, in: Proceed- ings of the IEEE International Conference on Computer Vision, 2023, pp. 364–373

    2023

  38. [46]

    T. Ao, Z. Zhang, L. Liu, Gesturediffuclip: Gesture diffusion model with clip latents, ACM Transactions on Graphics 42 (2023) 1–18

    2023

  39. [47]

    T. Ao, Q. Gao, Y . Lou, B. Chen, L. Liu, Rhythmic gesticulator: Rhythm- aware co-speech gesture synthesis with hierarchical neural embeddings, ACM Transactions on Graphics 41 (2022) 1–19

    2022

  40. [48]

    Dabral, M

    R. Dabral, M. H. Mughal, V . Golyanik, C. Theobalt, Mofusion: A frame- work for denoising-diffusion-based motion synthesis, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2023, pp. 9760–9770

    2023

  41. [49]

    K. Gong, D. Lian, H. Chang, C. Guo, Z. Jiang, X. Zuo, M. B. Mi, X. Wang, Tm2d: Bimodality driven 3d dance generation via music-text integration, in: Proceedings of the IEEE International Conference on Computer Vision, 2023, pp. 9942–9952

    2023

  42. [50]

    R. Li, S. Yang, D. A. Ross, A. Kanazawa, Ai choreographer: Music conditioned 3d dance generation with aist++, in: Proceedings of the IEEE International Conference on Computer Vision, 2021, pp. 13401–13412

    2021

  43. [51]

    Zhang, T

    Z. Zhang, T. Ao, Y . Zhang, Q. Gao, C. Lin, B. Chen, L. Liu, Semantic gesticulator: Semantics-aware co-speech gesture synthesis, ACM Trans- actions on Graphics 43 (2024) 1–17

    2024

  44. [52]

    Devlin, Bert: Pre-training of deep bidirectional transformers for lan- guage understanding, arXiv preprint arXiv:1810.04805 (2018)

    J. Devlin, Bert: Pre-training of deep bidirectional transformers for lan- guage understanding, arXiv preprint arXiv:1810.04805 (2018)

    2018 arXiv

  45. [53]

    Chang, H

    H. Chang, H. Zhang, J. Barber, A. Maschinot, J. Lezama, L. Jiang, M.-H. Yang, K. Murphy, W. T. Freeman, M. Rubinstein, et al., Muse: Text- to-image generation via masked generative transformers, arXiv preprint arXiv:2301.00704 (2023)

    2023 arXiv

  46. [54]

    P. Liu, P. Zhang, H. Kim, P. Garrido, A. Shapiro, K. Olszewski, Contex- tual gesture: Co-speech gesture video generation through context-aware gesture representation, arXiv preprint arXiv:2502.07239 (2025)

    2025

  47. [55]

    Q. Zhou, M. Li, Q. Zeng, A. Aristidou, X. Zhang, L. Chen, C. Tu, Let’s all dance: Enhancing amateur dance motions, Computational Visual Media (2023)

    2023

  48. [56]

    X. Liu, Y . Xu, Q. Wu, H. Zhou, W. Wu, B. Zhou, Semantic-aware im- plicit neural audio-driven video portrait generation, in: Proceedings of the European Conference on Computer Vision (ECCV), 2022, pp. 112–129

    2022

  49. [57]

    M. Shi, W. Feng, L. Gao, D. Zhu, Generating diverse clothed 3d hu- man animations via a generative model, Computational Visual Media 10 (2024) 261–277. 13

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.