Pith. sign in

REVIEW 3 major objections 7 minor 63 references

Music-Aligned Holistic 3D Dance Generation via Hierarchical Motion Modeling

T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces SoulNet, a music-to-dance system that generates coordinated 3D body, hand, and facial motion together, and claims it surpasses prior methods on quality, synchronization, and emotional alignment.

desk verdict A genuinely useful new dataset and a plausible architecture, but the headline alignment claim rests on a metric trained by the same model, so those numbers are unverified. read the letter →

arxiv 2507.14915 v3 pith:RYAOLF6Z submitted 2025-07-20 cs.MM cs.SDeess.AS

Xiaojie Li , Ronghui Li , Shukai Fang , Shuzhao Xie , Xiaoyang Guo , Jiaqing Zhou , Junkun Peng , Zhi Wang This is my paper
classification cs.MMcs.SDeess.AS
keywords 3Ddancegenerationholisticmotionsynthesismusic-dancealignmenthierarchicalresidualvectorquantizationcross-modalretrievalfacialexpressioncapturedatasetSMPL-X
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that 3D dance generation should model the whole performer—body, hands, and face—and that holistic motion can be made to follow music by breaking it into hierarchical discrete codes and supervising generation with a learned music-dance alignment space. It introduces SoulDance, a 12.5-hour professional motion-capture dataset of music-dance pairs across 15 genres that includes body, hand, and facial motion, and SoulNet, a pipeline with three components: hierarchical residual vector quantization (HRVQ) for tokenizing holistic motion, a transformer generative model that composes the tokens, and a pretrained music-motion retrieval module (MMR) that acts as a cross-modal alignment prior during training. The paper reports that SoulNet outperforms prior methods (FACT, Bailando, EDGE, FineNet) on reconstruction error, FID, the proposed MMR-Matching Score, the proposed Emotion Alignment Score, and a user study. If the reported results hold, music could drive expressive virtual characters directly, with hand gestures and facial expressions synchronized to beat and emotional tone instead of body-only motion.

What carries the argument

The load-bearing mechanism is the body-hand-face chain in HRVQ: for each residual layer, the body quantization code $b^v$ is transformed with $T$ and fed into the hand quantization step, and the hand code $h^v$ similarly conditions the face step (Eqs. 2–3), with a joint decoder $D_{\mathrm{whole}}$ reconstructing body, hands, and face from the summed codes. The MMR module supplies the alignment signal: a Jukebox-based music encoder and a motion encoder are contrastively pretrained so matched music-dance pairs lie close in a shared latent space, and that embedding space is reused as the training losses $L_{\text{Align-body}}$ and $L_{\text{Align-whole}}$ and as the evaluation metric MMR-Matching Score.

What would settle it

Have human raters or an independent beat-detection method (for example, musical onset times versus motion acceleration peaks) score the same set of generated dances from SoulNet, Bailando, EDGE, and FineNet, without any metric computed from the MMR embedding; if SoulNet does not place first on that external music-dance alignment measure, the claim that MMR yields genuinely better synchronization is weakened.

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is that coordinated holistic dance can be represented by a body-hand-face chain of residual quantizations: at each residual level the body's quantized code $b^v$ is injected as a hint into the hand quantizer, and the hand code $h^v$ is injected into the face quantizer, so inter-part coordination is encoded in the token representation rather than left to the decoder. The Music-Aligned Generative Model predicts masked base tokens and residual tokens, while the pretrained MMR module contributes two contrastive alignment losses derived from Eq. (8): $L_{\text{Align-body}}$ for local beat-level body alignment and $L_{\text{Align-whole}}$ for global whole-body semantic alignment. The paper reports state-of-the-art FID (0.029 on SoulDance, versus 0.048 for its ablation without MMR), lower reconstruction error than plain VQ or RVQ, best scores on the new MMR-Matching Score and Emotion Alignment Score, and top user-study ratings on whole, body, hands, emotion, and alignment criteria.

Load-bearing premise

The load-bearing premise is that the MMR module's learned embedding genuinely measures how well a dance matches its music, since that same embedding is used both to train SoulNet's alignment losses and to compute the MMR-Matching Score that reports the alignment improvement.

Editorial extensions

If this is right

  • SoulNet's reported FID of 0.029 on SoulDance (against 0.048 for its no-MMR variant and 1.008 for FACT) implies that the combination of HRVQ tokenization and MMR supervision produces substantially more realistic holistic dance than existing body-only or body-hand methods.
  • HRVQ lowers reconstruction error at every tested codebook size (Table 2), so the body-hand-face conditioning chain is presented as a better motion representation than plain residual VQ for fine-grained body, hand, and face synthesis.
  • MMR supervision improves both local and global music-motion alignment: ablations show $L_{\text{Align-body}}$ helps FID and beat accuracy while $L_{\text{Align-whole}}$ improves the MMR-Matching Score.
  • The two new metrics, EmotionAlign Score and MMR-Matching Score, allow evaluating facial-expression-to-music emotional tone and fine-grained rhythmic synchronization, which prior genre-matching metrics could not capture.
  • SoulDance, at 12.5 hours with synchronized body, hands, and face in BVH, SMPL-X, and FLAME formats, is presented as the largest holistic music-dance dataset to date and is the resource that makes training and benchmarking such models possible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the MMR embedding serves as both the training objective and the MMR-Matching Score, the reported alignment improvement over methods not trained with this loss may partly reflect optimizing the scoring metric; an external beat-sync or human-rated alignment test would reveal how much of the gain transfers.
  • The body-hand-face conditioning chain is a general representation idea that could transfer to other holistic motion generation tasks, such as co-speech gesture, sign-language synthesis, or avatar animation, where hand-face coordination matters as much as in dance.
  • The result that more residual layers improve reconstruction but hurt generation beyond V=5 suggests a trade-off between motion fidelity and learnable composition; the paper fixes V=5, but the optimal balance may shift with larger datasets or longer sequences.
  • The SoulDance capture setup—marker-based body mocap plus a helmet-mounted iPhone for ARKit face blendshapes converted to FLAME—offers a lower-cost template for building expressive holistic motion datasets beyond dance.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces SoulDance, a 12.5-hour motion-captured music-dance dataset with synchronized body, hand, and facial motion across 15 genres, and presents SoulNet, a three-module framework (HRVQ tokenization, a masked-transformer music-conditioned generator, and a pretrained Music-Motion Retrieval module used as an alignment prior). The paper reports state-of-the-art reconstruction and generation results on SoulDance and AIST++, and introduces two new metrics, MMR-Matching Score and Emotion Alignment Score, to measure music-motion and music-emotion alignment. The central claim is that SoulNet significantly outperforms existing methods in generating high-quality, music-aligned, holistic 3D dance.

Significance. If the evaluation were fully credible, the contribution would be significant: SoulDance appears to be the first holistic mocap dataset with body, hands, and face, and HRVQ's reconstruction improvements in Table 2 are internally consistent and mechanistically plausible. The paper also includes extensive ablations, a user study, and a retrieval evaluation of the MMR module. However, the central 'music-aligned' claim currently rests on a self-referential learned metric, and the emotion-alignment metric is underspecified, so the significance is conditional on fixing the evaluation. The dataset itself is a potentially valuable resource regardless of the generation results.

major comments (3)
  1. [Section 4.3, Section 5.1, Eq. (9), Table 5] MMR-MS is computed with the same MMR module that provides the training losses L_Align-body and L_Align-whole in Eqs. (6)-(8). Because FACT, Bailando, EDGE, and FineNet are never trained with these losses, SoulNet's lower MMR-MS could reflect optimizing the evaluation metric itself rather than genuinely better music-dance alignment. Table 5 makes this concrete: adding MMR supervision improves MMR-MS from 0.418 to 0.369, exactly in the direction the loss minimizes. Appendix H further states that MMR_whole is trained on a subset of SoulDance, making the metric in-distribution for SoulNet but out-of-distribution for all baselines. This is load-bearing for the paper's headline claim. Please provide (i) human perceptual validation of MMR-MS on held-out music-dance pairs, (ii) results with an independent alignment metric not used in training (e.g., beat-alignment statistics or a frozen pretrained audio-motion model), and (iii) a precise statement of which MMR variant is used in Table 4 and whether the evaluation split is disjoint from MMR training.
  2. [Section 5.1, Emotion Alignment Score] The EAS definition is ambiguous and not a clear music-alignment measure. The text says the score is 'the accuracy of comparing predicted expressions with ground truth expressions,' but for generated dances there is no ground-truth expression per music clip unless music emotion labels are defined. The paper does not specify how the music emotion ground truth was obtained (annotators, automatic labeling, or the original dancer's expressions), how the facial expression recognition model is applied to FLAME parameters or rendered faces, or how accuracy is computed relative to chance (1/7). Without these details and without validation against human emotion judgments, the EAS cannot support the claim that SoulNet improves emotional alignment to music.
  3. [Tables 4-7] The paper reports no error bars, confidence intervals, or significance tests, although the abstract and text use the word 'significantly.' Several decisive comparisons are small (e.g., Table 6 FID 0.029 vs. 0.042; Table 4 MMR-MS differences of roughly 0.01 between some baselines), and the user study uses only 28 participants and 22 pairs. Please report variance across multiple training seeds for Tables 4-6 and a paired statistical test for the user study in Table 7. Without this, the claimed superiority over baselines is not established.
minor comments (7)
  1. [Table 2] In the AIST++ column, HRVQ-5 reports the same body MPJPE (24.246) as RVQ-5; this is likely a copy-paste error and should be corrected or explained.
  2. [Appendix G vs. Appendix H] The MMR training batch size is given as 32 in Appendix G and 128 in Appendix H; please reconcile the two numbers.
  3. [Appendix G and Section 4.1] The implementation details state that 'each vector quantizer consists of 6 layers,' while the main model is described with V=5 residual layers; the relationship between these layer counts should be clarified.
  4. [Table 4] The AIST++ FID values for Bailando (11.079) and EDGE (21.370) are dramatically worse than SoulNet (0.081); please describe whether the baselines were retrained on AIST++ under the same motion representation and whether the same FID feature extractor was used for all methods, since otherwise the comparison is not interpretable.
  5. [Abstract and Section 1] The claim that SoulDance is 'the largest to date' should be qualified as the largest holistic dataset with face, because Table 1 lists AIOZ-GDANCE with 16.7 hours and FineDance with 14.6 hours without facial motion.
  6. [References] SMPL-X is cited as entries [35], [39], and [40] with overlapping content; please consolidate duplicate references.
  7. [Section 5.1, Eq. (9)] Please state whether the latent vectors z_i and m_i are L2-normalized before computing Euclidean distances, and specify the temporal alignment of the 1-second segments used for the dynamic term.

Circularity Check

2 steps flagged · score 6.0 of 10

MMR-MS is computed with the same MMR embedding that supervises SoulNet, so the headline alignment advantage is self-referential; independent FID and user-study evidence partly mitigate.

  1. fitted input called prediction [Section 4.3 Eq. (8); Section 5.1 Eq. (9); Tables 4-6]
    "Both losses are derived from an alignment objective: LAlign = ... (8) ... During inference, the music signal C and motion sequences M are encoded into latent representations z and c via the MMR encoder. ... To quantify the alignment between music and the generated dance, we employ MMR encoders to project both modalities into latent space. The MMR-Matching Score (MMR-MS) is then computed as the Euclidean distance between these latent representations: MMR-MS = ... (9)"

    The same learned MMR latent space is used in two roles. Equations 6-7 train SoulNet with L_Align-body and L_Align-whole, both derived from the MMR contrastive objective (Eq. 8), and Eq. 9 then scores generated dances by the distance between music and motion in that same MMR latent space. SoulNet is therefore explicitly optimized to reduce the quantity that MMR-MS measures, while the baselines FACT, Bailando, EDGE, and FineNet never receive this supervision. Table 5 confirms the coupling: adding MMR supervision changes MMR-MS from 0.418 to 0.369, exactly the direction that the training loss optimizes. The claimed 'superior music-motion alignment' in Table 4 is thus, for the MMR-MS column, partly a restatement of SoulNet's training objective rather than an independent measurement.

  2. fitted input called prediction [Appendix H]
    "MMR whole is trained on a subset of SoulDance holistic motion data (including body, hands and face) with a 723-dimensional representation (Dm = 723)."

    The whole-body MMR module that supplies the alignment supervision and the MMR-MS evaluation is trained on a subset of SoulDance, which is also the dataset used for the main generation comparison. This makes the MMR-MS evaluation in-distribution for SoulNet while the comparison methods are scored with an embedding fitted to the target dataset. The metric is therefore not an independently trained, out-of-distribution judge of music-dance alignment for the central claim.

full rationale

The derivation chain for the music-alignment claim contains a concrete construction-level circularity: the MMR module's contrastive embedding is both the training objective for SoulNet (Eqs. 6-8) and the evaluation metric MMR-MS (Eq. 9). SoulNet is explicitly trained to minimize distances in this embedding, while the baselines are not, so the MMR-MS advantage in Table 4 is partly a measure of how well SoulNet optimized its own objective. The problem is evidenced internally by Table 5, where adding MMR supervision improves MMR-MS from 0.418 to 0.369, and by Table 6, where L_Align-whole specifically improves MMR-MS. Appendix H further shows that MMR_whole is trained on a SoulDance subset, making the metric in-distribution for the main evaluation. However, the paper is not wholly circular: FID is computed with the HumanML3D feature extractor, BAS, diversity, multimodality, and the user study all provide independent evidence, and the user study includes an alignment question answered by human raters. Those external signals support the high-level claim of quality and coordination, so the score is 6 rather than 8 or 10. No load-bearing self-citation chain or imported uniqueness theorem is present.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central claim rests on standard deep learning components (VQ, transformers, contrastive learning) rather than on new physical entities. The main burden is on learned embeddings: the MMR embedding is trained by the authors and used for both supervision and evaluation, and the FID feature extractor and emotion recognition models are external assumptions. Hyperparameters such as the MMR-MS weights and the number of residual layers are chosen by hand and directly shape the headline numbers.

free parameters (6)
  • MMR-MS metric weights mu and lambda = mu=0.7, lambda=0.3
    Hand-chosen weights in Eq. 9; they define the reported MMR-Matching Score, so the headline alignment comparisons are sensitive to this choice.
  • Alignment loss weights lambda_b and lambda_w = 0.5 and 0.5
    Set in Eqs. 6 and 7 without ablations; they balance mask and residual losses with the MMR alignment terms.
  • Number of residual VQ layers V = 5
    Selected based on Table 3 and Figure 4 as a tradeoff between reconstruction and generation quality; all main results use this value.
  • Codebook size and layer count in HRVQ = 512 codes per layer, 6 layers
    Model capacity choices in Appendix G; no sensitivity analysis is reported.
  • MMR latent dimension, temperature, InfoNCE weight, negative threshold = d=256, tau=0.1, w=0.1, threshold=0.8
    Training choices for the contrastive module (Appendix H) that determine the embedding space used for alignment losses and the MMR-MS metric.
  • Experimental subset of SoulDance = 6.3 hours (randomly selected)
    The full dataset is 12.5 hours, but experiments use a randomly selected subset; the choice of subset affects all reported numbers and is described without a seed.
assumptions (6)
  • domain assumption Pretrained Jukebox encoder provides a music representation rich enough for dance alignment.
    Invoked in Section 4.3 as the audio feature extractor for the MMR module; if this representation misses rhythm or affect, the alignment prior is weak.
  • domain assumption The MMR contrastive embedding space captures semantically meaningful music-dance alignment.
    Central to Eqs. 8 and 9 and Appendix H; the whole alignment evaluation rests on this assumption, and the embedding is learned rather than derived.
  • domain assumption The HumanML3D motion encoder is a valid feature extractor for holistic dance data including body, hands, and face.
    Used in Section 5.1 'Public benchmarks' to compute FID and diversity; this encoder was trained for text-motion retrieval on HumanML3D, not for holistic dance with facial expressions.
  • domain assumption The facial expression recognition models provide correct emotion labels for the Emotion Alignment Score.
    Section 5.1 'Emotion Alignment Score' uses these models to classify generated expressions; accuracy of the EAS depends on their reliability on rendered avatars.
  • domain assumption The ARKit-to-FLAME conversion via EMAGE preserves the expressive content of the captured facial motion.
    Appendix C describes fitting a transformation matrix; any lossy conversion would propagate into HRVQ training and evaluation.
  • domain assumption The motion capture and retargeting pipeline preserves authentic, physically plausible motion.
    Section 3 and Appendix B describe manual refinement of retargeted SMPL-X; ground truth quality is assumed to be high and consistent across sequences.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Music-Aligned Holistic 3D Dance Generation via Hierarchical Motion Modeling." pith.science (2026). https://pith.science/paper/RYAOLF6Z

@misc{pith2026250714915,
  author       = {Pith},
  title        = {Pith review of: Music-Aligned Holistic 3D Dance Generation via Hierarchical Motion Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RYAOLF6Z}},
  note         = {Machine review of arXiv:2507.14915}
}
read the original abstract

Well-coordinated, music-aligned holistic dance enhances emotional expressiveness and audience engagement. However, generating such dances remains challenging due to the scarcity of holistic 3D dance datasets, the difficulty of achieving cross-modal alignment between music and dance, and the complexity of modeling interdependent motion across the body, hands, and face. To address these challenges, we introduce SoulDance, a high-precision music-dance paired dataset captured via professional motion capture systems, featuring meticulously annotated holistic dance movements. Building on this dataset, we propose SoulNet, a framework designed to generate music-aligned, kinematically coordinated holistic dance sequences. SoulNet consists of three principal components: (1) Hierarchical Residual Vector Quantization, which models complex, fine-grained motion dependencies across the body, hands, and face; (2) Music-Aligned Generative Model, which composes these hierarchical motion units into expressive and coordinated holistic dance; (3) Music-Motion Retrieval Module, a pre-trained cross-modal model that functions as a music-dance alignment prior, ensuring temporal synchronization and semantic coherence between generated dance and input music throughout the generation process. Extensive experiments demonstrate that SoulNet significantly surpasses existing approaches in generating high-quality, music-coordinated, and well-aligned holistic 3D dance sequences.

Figures

Figures reproduced from arXiv: 2507.14915 by the authors.

Figure 1
Figure 1. We introduce SoulDance, a high-quality, comprehensive dance dataset that incorporates body motions, hand gestures, and facial expressions. The dataset processing pipeline (left) consists of separate steps for capturing facial expressions and body-hand movements. Moreover, we present SoulNet, the first framework able to generate expressive and holistic dance, as demonstrated in the results (right). Abstract Well-coor… view at source ↗
Figure 2
Figure 2. Hierarchical Residual Vector Quantization. For the body component, we apply RVQbv. In this process, each quan￾tized layer output b v serves as a body hint that is passed se￾quentially to the hands component, which is then quantized using HRVQhv. Similarly, the face component undergoes HRVQfv, using hands hints derived from the preceding quantization steps to guide the encoding process. 4. SoulNet Given a music C, ou… view at source ↗
Figure 3
Figure 3. An overview of our SoulNet framework. On the left, the Music-Aligned Generative Model (MAGM) Training and Inference consists of two stages: (1) transformer layers generate primary motion tokens in the base layer, and (2) residual layers refine motion through V − 1 quantization layers. On the right, the Music-Motion Retrieval Module (MMR) illustrates the motion-music alignment process during its pretraining. Once tra… view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: FID and MMR-Matching Score. Performance compar￾ison across varying numbers of residual layers for different quan￾tization methods. Public benchmarks. (1) FID score. Frechet inception dis- ´ tance (FID) is widely used to measure how close the distri￾bution of the genera…
Figure 5
Figure 5. Figure 5: Overview of the distribution of the SoulDance dataset. (a) shows the distribution of dance sequences by dancer count (1–5), with most sequences featuring solo performances. (b) depicts the proportion of dance duration for each dancer (A-E). (c) illustrates the distribu…
Figure 6
Figure 6. Figure 6: Comparison of Visualization Results. We visualize the dance generation results on the SoulDance dataset using different methods. The dashed line represents the motion trajectory along the direction of gravity, where smaller fluctuations indicate more stable generated d…
Figure 7
Figure 7. Figure 7: Visualizes dance motion reconstruction on the SoulDance dataset. From left to right, the columns represent the ground truth (GT), HRVQ, RVQ, and VQ results, respectively. H. Training: Music-Motion Retrieval Module Dataset. To establish robust dance-music alignment prio…
Figure 8
Figure 8. Figure 8: Showcase of various dance styles in the SoulDance dataset. The SoulDance dataset demonstrates high motion quality and diversity across multiple dance styles [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Qualitative generation result comparisons for a Rock song in the AIST++ dataset. FACT EDGE Bailando FineNet SoulNet (Ours) [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Qualitative generation result comparisons for a Pop song in the SoulDance dataset [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Diversity of generated dances. The SoulNet method demonstrates rich diversity under identical input music of the Chinese Style genre, encompassing variations in body movements, hand gestures, and facial expressions. Score=0.9506 Score=0.9716 Score=0.2820 Score=0.2690 …
Figure 12
Figure 12. Figure 12: Qualitative results of Music-Motion Retrieval. For the Pop music genre, higher similarity scores indicate greater correspon￾dence between the retrieved dance motions and the input music [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Screenshot of video page in the user study. The interface provides independent A/B video links, allowing users to view each corresponding video separately [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: User interface of our surveys. The interface presents a set of questions alongside two videos, A and B. Screenshots of the videos linked in the survey are shown in [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

63 extracted references · 50 canonical work pages

  1. [1]

    Interactive motion gen- eration from examples

    Okan Arikan and David A Forsyth. Interactive motion gen- eration from examples. ACM Transactions on Graphics (TOG), 21(3):483–490, 2002. 2, 3

    work page 2002

  2. [2]

    Chore- ograph: Music-conditioned automatic dance choreography over a style and tempo consistent dynamic graph

    Ho Yin Au, Jie Chen, Junkun Jiang, and Yike Guo. Chore- ograph: Music-conditioned automatic dance choreography over a style and tempo consistent dynamic graph. In Pro- ceedings of the 30th ACM International Conference on Mul- timedia, pages 3917–3925, 2022. 3

    work page 2022

  3. [3]

    Autodesk motionbuilder, 2023

    Autodesk, Inc. Autodesk motionbuilder, 2023. Avail- able at https://www.autodesk.com/products/ motionbuilder/overview. 1

    work page 2023

  4. [4]

    Advances in residual vector quantization: A re- view

    Christopher F Barnes, Syed A Rizvi, and Nasser M Nasrabadi. Advances in residual vector quantization: A re- view. IEEE transactions on image processing, 5(2):226–262,

    work page

  5. [5]

    Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data

    Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, et al. Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data. arXiv preprint arXiv:2111.08897, 2021. 3

    arXiv 2021

  6. [6]

    Audiolm: a language modeling approach to audio genera- tion

    Zal ´an Borsos, Rapha ¨el Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: a language modeling approach to audio genera- tion. IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, 2023. 3

    work page 2023

  7. [7]

    Choreomaster: choreography-oriented music-driven dance synthesis

    Kang Chen, Zhipeng Tan, Jin Lei, Song-Hai Zhang, Yuan- Chen Guo, Weidong Zhang, and Shi-Min Hu. Choreomaster: choreography-oriented music-driven dance synthesis. ACM Transactions on Graphics (TOG), 40(4):1–13, 2021. 2, 3, 4

    work page 2021

  8. [8]

    Jukebox: A gen- erative model for music

    Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A gen- erative model for music. arXiv preprint arXiv:2005.00341,

    arXiv 2005

Show all 63 references
  1. [9]

    Constants across cul- tures in the face and emotion

    Paul Ekman and Wallace V Friesen. Constants across cul- tures in the face and emotion. Journal of personality and social psychology, 17(2):124, 1971. 7

    1971

  2. [10]

    Unreal engine 5, 2022

    Epic Games. Unreal engine 5, 2022. Available at https: //www.unrealengine.com/. 1

    2022

  3. [11]

    Faceformer: Speech-driven 3d facial anima- tion with transformers

    Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, and Taku Komura. Faceformer: Speech-driven 3d facial anima- tion with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18770–18780, 2022. 2

    2022

  4. [12]

    Rmpe: Regional multi-person pose estimation

    Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu. Rmpe: Regional multi-person pose estimation. In Proceed- ings of the IEEE international conference on computer vi- sion, pages 2334–2343, 2017. 3

    2017

  5. [13]

    Evolution and functions of human dance

    Bernhard Fink, Bettina Bl ¨asing, Andrea Ravignani, and Todd K Shackelford. Evolution and functions of human dance. Evolution and Human Behavior , 42(4):351–360,

  6. [14]

    Dancemeld: Unraveling dance phrases with hierarchi- cal latent codes for music-to-dance synthesis

    Xin Gao, Li Hu, Peng Zhang, Bang Zhang, and Liefeng Bo. Dancemeld: Unraveling dance phrases with hierarchi- cal latent codes for music-to-dance synthesis. arXiv preprint arXiv:2401.10242, 2023. 3

    2023 arXiv

  7. [15]

    Gray and David L

    Robert M. Gray and David L. Neuhoff. Quantization. IEEE transactions on information theory, 44, 1998. 3

    1998

  8. [16]

    Ac- tion2motion: Conditioned generation of 3d human motions

    Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Ac- tion2motion: Conditioned generation of 3d human motions. In Proceedings of the 28th ACM International Conference on Multimedia, pages 2021–2029, 2020. 3

    2021

  9. [17]

    Generating diverse and natural 3d human motions from text

    Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5152–5161, 2022. 7, 1

    2022

  10. [18]

    Tm2t: Stochastic and tokenized modeling for the reciprocal genera- tion of 3d human motions and texts

    Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for the reciprocal genera- tion of 3d human motions and texts. InEuropean Conference on Computer Vision, pages 580–597. Springer, 2022. 3, 4, 7

    2022

  11. [19]

    Momask: Generative masked model- ing of 3d human motions

    Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Generative masked model- ing of 3d human motions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1900–1910, 2024. 3, 4, 7

    1900

  12. [20]

    Residual quan- tization with implicit neural codebooks

    Iris AM Huijben, Matthijs Douze, Matthew Muckley, Ruud JG Van Sloun, and Jakob Verbeek. Residual quan- tization with implicit neural codebooks. arXiv preprint arXiv:2401.14732, 2024. 2

    2024 arXiv

  13. [21]

    Categorical reparameterization with gumbel-softmax

    Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016. 5, 4

    2016 arXiv

  14. [22]

    Motiongpt: Human motion as a foreign language

    Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign language. arXiv preprint arXiv:2306.14795, 2023. 4

    2023 arXiv

  15. [23]

    Towards the automatic anime characters creation with generative adversarial net- works

    Yanghua Jin, Jiakai Zhang, Minjun Li, Yingtao Tian, Huachun Zhu, and Zhihao Fang. Towards the automatic anime characters creation with generative adversarial net- works. arXiv preprint arXiv:1708.05509, 2017. 5

    2017 arXiv

  16. [24]

    Digitization and visualization of folk dances in cultural heritage: A review

    Iris Kico, Nikos Grammalidis, Yiannis Christidis, and Fotis Liarokapis. Digitization and visualization of folk dances in cultural heritage: A review. Inventions, 3(4):72, 2018. 1

    2018

  17. [25]

    Flame: Free- form language-based motion synthesis & editing

    Jihoon Kim, Jiseob Kim, and Sungjoon Choi. Flame: Free- form language-based motion synthesis & editing. In Pro- ceedings of the AAAI Conference on Artificial Intelligence , pages 8255–8263, 2023. 1

    2023

  18. [26]

    Making them dance

    Jae Woo Kim, Hesham Fouad, and James K Hahn. Making them dance. In AAAI Fall Symposium: Aurally Informed Performance, page 2, 2006. 2, 3

    2006

  19. [27]

    Rhythmic-motion synthesis based on motion-beat analy- sis

    Tae-hoon Kim, Sang Il Park, and Sung Yong Shin. Rhythmic-motion synthesis based on motion-beat analy- sis. ACM Transactions on Graphics (TOG), 22(3):392–401,

  20. [28]

    Error bounds of projection models in weakly super- vised 3d human pose estimation

    Nikolas Klug, Moritz Einfalt, Stephan Brehm, and Rainer Lienhart. Error bounds of projection models in weakly super- vised 3d human pose estimation. In 2020 International Con- ference on 3D Vision (3DV), pages 898–907. IEEE, 2020. 2

    2020

  21. [29]

    Music-driven group choreography

    Nhat Le, Thang Pham, Tuong Do, Erman Tjiputra, Quang D Tran, and Anh Nguyen. Music-driven group choreography. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8673–8682, 2023. 2, 3, 4

    2023

  22. [30]

    Dance- former: Music conditioned 3d dance generation with para- metric motion transformer

    Buyu Li, Yongchi Zhao, Shi Zhelun, and Lu Sheng. Dance- former: Music conditioned 3d dance generation with para- metric motion transformer. In Proceedings of the AAAI Con- ference on Artificial Intelligence, pages 1272–1279, 2022. 3, 4

    2022

  23. [31]

    Ai choreographer: Music conditioned 3d dance generation with aist++

    Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision , pages 13401– 13412, 2021. 2, 3, 4, 6, 8

    2021

  24. [32]

    Finedance: A fine-grained choreography dataset for 3d full body dance generation

    Ronghui Li, Junfan Zhao, Yachao Zhang, Mingyang Su, Zeping Ren, Han Zhang, Yansong Tang, and Xiu Li. Finedance: A fine-grained choreography dataset for 3d full body dance generation. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision , pages 10234– 10...

    2023

  25. [33]

    Lodge: A coarse to fine diffusion network for long dance generation guided by the characteristic dance primitives

    Ronghui Li, YuXiang Zhang, Yachao Zhang, Hongwen Zhang, Jie Guo, Yan Zhang, Yebin Liu, and Xiu Li. Lodge: A coarse to fine diffusion network for long dance generation guided by the characteristic dance primitives. In Proceed- ings of the IEEE/CVF Conference on Computer Vision ...

    2024

  26. [34]

    Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero. Learning a model of facial shape and ex- pression from 4D scans. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 36(6), 2017. 2, 3

    2017

  27. [35]

    Learning a model of facial shape and expression from 4d scans

    Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4d scans. ACM Trans. Graph., 36(6):194–1, 2017. 1

    2017

  28. [36]

    Emage: Towards unified holis- tic co-speech gesture generation via expressive masked audio gesture modeling

    Haiyang Liu, Zihao Zhu, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Xuefei Zhe, Naoya Iwamoto, Bo Zheng, and Michael J Black. Emage: Towards unified holis- tic co-speech gesture generation via expressive masked audio gesture modeling. In Proceedings of the IEEE/CVF ...

    2024

  29. [37]

    Matthew Loper, Naureen Mahmood, Javier Romero, Ger- ard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6):248:1–248:16, 2015. 1

    2015

  30. [38]

    Humantomato: Text-aligned whole-body motion generation

    Shunlin Lu, Ling-Hao Chen, Ailing Zeng, Jing Lin, Ruimao Zhang, Lei Zhang, and Heung-Yeung Shum. Humantomato: Text-aligned whole-body motion generation. arXiv preprint arXiv:2310.12978, 2023. 3, 7, 1

    2023 arXiv

  31. [39]

    Expressive body capture: 3d hands, face, and body from a single image

    Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from a single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognitio...

    2019

  32. [40]

    Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3d hands, face, and body from a single image. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019. 2, 3

    2019

  33. [41]

    Temos: Generating diverse human motions from textual descriptions

    Mathis Petrovich, Michael J Black, and G ¨ul Varol. Temos: Generating diverse human motions from textual descriptions. In European Conference on Computer Vision , pages 480–

  34. [42]

    Tmr: Text-to-motion retrieval using contrastive 3d human motion synthesis

    Mathis Petrovich, Michael J Black, and G ¨ul Varol. Tmr: Text-to-motion retrieval using contrastive 3d human motion synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9488–9497, 2023. 4

    2023

  35. [43]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In International Conference on Machine Learning...

    2021

  36. [44]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022. 5

    2022

  37. [45]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information...

    2022

  38. [46]

    Facial expression recognition with adap- tive frame rate based on multiple testing correction

    Andrey Savchenko. Facial expression recognition with adap- tive frame rate based on multiple testing correction. In Pro- ceedings of the 40th International Conference on Machine Learning (ICML), pages 30119–30129. PMLR, 2023. 7

    2023

  39. [47]

    Chingmu, 2022

    SHANGHAI CHINGMU VISION TECHNOL- OGY . Chingmu, 2022. Available at https : //www.chingmu.com/. 3

    2022

  40. [48]

    Dancing-to-music character animation

    Takaaki Shiratori, Atsushi Nakazawa, and Katsushi Ikeuchi. Dancing-to-music character animation. In Computer Graph- ics Forum, pages 449–458. Wiley Online Library, 2006. 2, 3

    2006

  41. [49]

    Bailando: 3d dance generation by actor-critic gpt with choreographic memory

    Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu. Bailando: 3d dance generation by actor-critic gpt with choreographic memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11050–...

    2022

  42. [50]

    Dancetrack: Multi-object track- ing in uniform appearance and diverse motion

    Peize Sun, Jinkun Cao, Yi Jiang, Zehuan Yuan, Song Bai, Kris Kitani, and Ping Luo. Dancetrack: Multi-object track- ing in uniform appearance and diverse motion. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20993–21002, 2022. 3

    2022

  43. [51]

    Dance with melody: An lstm-autoencoder approach to music-oriented dance syn- thesis

    Taoran Tang, Jia Jia, and Hanyang Mao. Dance with melody: An lstm-autoencoder approach to music-oriented dance syn- thesis. In Proceedings of the 26th ACM international confer- ence on Multimedia, pages 1598–1606, 2018. 2, 3, 4

    2018

  44. [52]

    Edge: Editable dance generation from music

    Jonathan Tseng, Rodrigo Castellon, and Karen Liu. Edge: Editable dance generation from music. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 448–458, 2023. 2, 3, 6, 8, 4

    2023

  45. [53]

    Transflower: probabilistic autoregressive dance gen- eration with multimodal attention

    Guillermo Valle-P ´erez, Gustav Eje Henter, Jonas Beskow, Andre Holzapfel, Pierre-Yves Oudeyer, and Simon Alexan- derson. Transflower: probabilistic autoregressive dance gen- eration with multimodal attention. ACM Transactions on Graphics (TOG), 40(6):1–14, 2021. 2, 4

    2021

  46. [54]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in Neural Information Processing Systems, 30, 2017. 2, 3

    2017

  47. [55]

    Codetalker: Speech-driven 3d facial animation with discrete motion prior

    Jinbo Xing, Menghan Xia, Yuechen Zhang, Xiaodong Cun, Jue Wang, and Tien-Tsin Wong. Codetalker: Speech-driven 3d facial animation with discrete motion prior. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12780–12790, 2023. 2

    2023

  48. [56]

    Vision transformer with attentive pooling for robust facial expression recognition.IEEE Trans- actions on Affective Computing, 2022

    Fanglei Xue, Qiangchang Wang, Zichang Tan, Zhongsong Ma, and Guodong Guo. Vision transformer with attentive pooling for robust facial expression recognition.IEEE Trans- actions on Affective Computing, 2022. 7

    2022

  49. [57]

    Moconvq: Unified physics- based motion control via scalable discrete representations

    Heyuan Yao, Zhenhua Song, Yuyang Zhou, Tenglong Ao, Baoquan Chen, and Libin Liu. Moconvq: Unified physics- based motion control via scalable discrete representations. ACM Transactions on Graphics (TOG), 43(4):1–21, 2024. 4

    2024

  50. [58]

    Soundstream: An end- to-end neural audio codec

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end- to-end neural audio codec. IEEE/ACM Transactions on Au- dio, Speech, and Language Processing , 30:495–507, 2021. 3, 4

    2021

  51. [59]

    Bidirectional au- toregessive diffusion model for dance generation

    Canyu Zhang, Youbao Tang, Ning Zhang, Ruei-Sung Lin, Mei Han, Jing Xiao, and Song Wang. Bidirectional au- toregessive diffusion model for dance generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 687–696, 2024. 3

    2024

  52. [60]

    T2m-gpt: Generating human motion from textual descriptions with discrete representations

    Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. T2m-gpt: Generating human motion from textual descriptions with discrete representations. arXiv preprint arXiv:2301.06052, 2023. 3

    2023 arXiv

  53. [61]

    On the continuity of rotation representations in neural networks

    Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5745–5753,

  54. [62]

    Music2dance: Dancenet for music-driven dance generation

    Wenlin Zhuang, Congyi Wang, Jinxiang Chai, Yangang Wang, Ming Shao, and Siyu Xia. Music2dance: Dancenet for music-driven dance generation. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 18(2):1–21, 2022. 2, 4 Music-Aligned Holistic 3D Dance ...

    2022

  55. [497]

    Springer, 2022. 6, 4

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.