Pith. sign in

REVIEW 4 major objections 6 minor 58 references

Harmony-Aware Music-driven Motion Synthesis with Perceptual Constraint on UGC Datasets

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a perception-inspired harmony score, computed by saliency-weighted audio-visual beat alignment and used as a weakly supervised loss, improves rhythmic synchronization in music-driven motion synthesis on limited UGC…

desk verdict The harmony metric is a nice idea, but the training loss as written cannot be optimized by gradient descent, so the paper's central claim does not hold up as presented. read the letter →

arxiv 2506.07076 v1 pith:HBSFFJCQ submitted 2025-06-08 cs.MM

Xinyi Wu , Haohong Wang , Aggelos K. Katsaggelos This is my paper
classification cs.MM
keywords music-drivenmotionsynthesisaudio-visualrhythmicharmonybeatalignmentscoreweaklysupervisedperceptualconstraintgenerativeadversarialnetworkUGCdancedatasetmusic-to-motiongeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that rhythmic harmony in music-to-motion synthesis can be made explicit and trainable. It proposes a harmony score that detects audio beats from onset strength and visual beats from local extrema of joint velocity, weights them by saliency, and counts how many audio-visual beat pairs fall within a human perceptual delay. The score is then used as a weakly supervised loss inside a GAN. On a small, video-based user-generated content (UGC) dance set, the model reports higher beat-synchronization scores and hit rates than leading baselines, and a user study matched the score's judgments 81.5% of the time. If this is right, perceptually grounded beat alignment can substitute for large training data in keeping synthesized motion on the music's rhythm.

What carries the argument

The machinery is the harmony score $h$ of Equation (11), an F-score-inspired harmonic mean $h=(1+\beta^2) h_v h_a / (\beta^2 h_v + h_a)$ that the paper reduces to counting synchronized beat pairs. Three operations feed it: audio beats from onset strength, visual beats from sign changes of the joint-velocity sum $J$, and saliency masks $W_a$, $W_v$ built from standard deviations that discard low-attention beats. Synchronization is decided by whether the nearest visual beat to each audio beat lies within a perceptual tolerance $T_{delay}=0.25$ s. As a loss, the paper uses $L_{harmo}=-h \approx -h_s + |M' - N'|$, which penalizes both missed alignments and surplus visual beats, and it is this quantity that turns the perceptual score into a training signal.

What would settle it

Take a trained HarmoGAN, perturb a single joint trajectory by a small amount $\epsilon$, and measure how $L_{harmo}$ changes: if for many perturbations the loss is piecewise constant or discontinuous rather than smoothly varying, then gradient descent on $L_{harmo}$ cannot be the mechanism behind the reported $75.73\%$ hit rate, and the gains should instead be attributed to the other loss terms.

Watch

Extended reading notes

Core claim

The central claim is that a harmony measure built from refined cross-modal beat extraction and attention-based beat alignment can serve both as an evaluation metric and as a weakly supervised training regularizer for music-to-motion synthesis. The authors detect audio beats from onset strength and visual beats from local extrema of the joint-velocity sum, keep only salient beats via standard-deviation masks, and count beat pairs as synchronized when they fall within a $0.25$ s perceptual delay. The resulting harmony score $h$ is an F-score-style harmonic mean that the loss approximates as $-h_s + |M' - N'|$. Trained with this loss, HarmoGAN reports an average hit rate of $75.73\%$ and a harmony score of $0.3527$ on the Ballroom music test set, above the [28] and [18] baselines, and a user study agrees with the score's harmony judgments in $81.5\%$ of comparisons. The authors interpret these results as evidence that perceptual beat alignment can be learned even from a small UGC dance dataset.

Load-bearing premise

The load-bearing premise is that the harmony score, despite being built from discrete local-extrema detection and beat counting, is a trainable loss whose gradients push generated motions toward better alignment; if that is false, the reported improvements are mostly an artifact of optimizing and measuring with the same score.

Editorial extensions

If this is right

  • If the harmony score reflects perception, it can be used as an evaluation metric for other audio-visual generation tasks without running user studies.
  • The weakly supervised loss should let music-to-motion models train on smaller, noisier user-generated collections rather than large studio-captured motion datasets.
  • Because the score needs only beats and saliency, it can be computed online during generation, offering a real-time rhythm controller instead of post-processing warping.
  • The beat-based alignment should transfer to other cross-modal rhythm tasks such as speech-to-lip synchronization or music-driven video editing, as the paper's discussion suggests.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper leaves implicit is that the $|M' - N'|$ term should also control dance tempo by discouraging surplus visual beats, which could be checked by comparing generated beat frequency with audio tempo.
  • The $81.5\%$ user-study consistency was measured on pairs where one video was permuted or warped; an untested stronger claim is whether the score ranks natural generated dances by perceived harmony, which a rank-correlation study on HarmoGAN outputs could check.
  • If the beat-alignment count is the active mechanism, then simpler differentiable surrogates, such as cross-correlation between audio onset strength and joint speed, may match the reported gains at lower implementation cost, a comparison the paper does not run.
  • Because the compared models are all GAN-based, the paper does not establish whether the harmony loss gives the same advantage to transformer or diffusion generators, so applying $L_{harmo}$ to those architectures is the direct next test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes HarmoGAN, a GAN-based music-to-motion synthesis framework for UGC dance videos. The authors design a beat-based "harmony score" h (Eq. 11) that aligns audio beats with saliency-weighted visual beats under a perceptual delay, and they use a negative approximation of this score as a training loss Lharmo (Eq. 16). The model is trained on the Dance dataset with meter-segmented audio and dual discriminators, and evaluated against Lee et al. and Ren et al. on the Ballroom dataset using harmony score and hit rate, plus user studies. The paper claims significant quantitative and qualitative improvements in rhythmic harmony.

Significance. The paper addresses a relevant problem: explicit rhythmic-synchronization control in music-to-motion synthesis, with attention to human perception (delay, saliency). The proposed framework includes useful architectural elements (meter segmentation, depth lifting, dual discriminators). However, the central evidence is weakened by (i) the non-differentiability of Lharmo, (ii) the use of the same harmony metric for training and evaluation, and (iii) the questionable formulation of the F-score. If the loss is made differentiable and evaluated with independent metrics, the contribution could be significant; as it stands, the quantitative claims are not yet supported.

major comments (4)
  1. [IV.C, Eq. (16); III.B, Eqs. (3)-(11)] The harmony loss Lharmo is constructed from non-differentiable operations: Bv is defined by sign changes of Δj_t (Eq. 3), h_p^i is an indicator over nearest-neighbor beat distances (Eq. 9), and M', N' are discrete counts. Consequently Lharmo is piecewise constant and has zero gradient with respect to the generated joint positions almost everywhere. The paper does not describe any relaxation, straight-through estimator, or alternative optimization mechanism. The observed differences between HarmoGAN and HarmoGAN w/o Lharmo in Fig. 9 and Tables I-II therefore lack a stated mechanism. Please provide a differentiable surrogate or explicitly describe how gradient signal flows through Eq. (16).
  2. [V.B, V.E, Tables I-II] The primary evaluation metrics (harmony score h and hit rate) are computed with the same beat-extraction and alignment machinery used to define the training loss Lharmo, and the thresholds λ1 and λ2 in Eqs. (4) and (6) are tuned on the Dance ground truth (Section V.B). Reporting these metrics as evidence of improvement is therefore circular: HarmoGAN is trained to maximize this specific score, so higher values do not independently demonstrate better rhythmic synchronization. Please include an evaluation with an independent beat-alignment metric, report threshold sensitivity, and provide error bars or significance tests for all quantitative comparisons.
  3. [III.C, Eqs. (9)-(11)] The definitions of ha = hs/N' and hv = hs/M' are not bounded by 1: hs in Eq. (10) is a saliency-weighted sum over N' audio beats, and multiple audio beats can be paired with the same nearest visual beat, so hs may exceed M'. Applying the F-score harmonic mean in Eq. (11) to quantities that are not precision/recall values is therefore mathematically questionable. The paper should either enforce one-to-one pairing or normalize hs differently before the harmonic mean.
  4. [V.D, V.E, Figs. 11 and 13] The user-study validation of the harmony strategy in Section V.D uses ground-truth videos versus permuted/warped versions, not motions generated by the compared models, so it does not establish that the metric can distinguish HarmoGAN from Lee et al. or Ren et al. The preference study in Section V.E reports only average percentages over 10 clips and 12 participants, without statistical tests or confidence intervals, and the abstract's 'significantly outperforms' claim is not backed by any significance testing.
minor comments (6)
  1. [V.C, Fig. 9] The phrase 'wealy-supervised' in the discussion of Fig. 9 is a typo for 'weakly-supervised'.
  2. [Abstract] The abstract contains 'Live samples 15', which appears to be a typo; it should likely read 'Live samples'.
  3. [III.B, Eq. (5)] The initial value j in Eq. (5) is defined only in the surrounding prose; please define it explicitly at first use within the equation context.
  4. [IV.C, Eq. (16)] The approximation relating Lharmo to -hs + |M' - N'| is not derived; please explain how it follows from Eq. (11) and why the saliency weighting is dropped in this approximation.
  5. [IV.A, Eq. (12)] The phrase 'fails within' in the definition of et should be 'falls within'.
  6. [Tables I and II] Tables I and II report only point estimates; please include sample sizes and measures of variance (e.g., standard deviation or confidence intervals) for each value.

Circularity Check

2 steps flagged · score 6.0 of 10

The rhythmic-harmony improvement is measured by the metric that Lharmo directly minimizes (Eq. 16 = -h), so the central quantitative claim is partially self-confirming.

  1. self definitional [Section IV.C, Eq. (16); Section V.E, Tables I-II]
    "To maximize audio-visual harmony in our generated results, we minimize the negative harmony score −h, which is derived based on Equations (1) and (11) as: Lharmo = −h = −L(Fa(A), Fv(G(Af , Vf ))) ≈ −hs + |M′ − N′| ... The rhythmic harmony of our generated results is evaluated using two metrics: the harmony score h (see Equation (11) for details) and the hit rate [2], [28]."

    Training and evaluation use the same scalar: Lharmo is the negative of the harmony score h (Eq. 11) reported in Tables I-II and Fig. 12, and the hit rate counts exactly the synchronized audio-visual beat pairs (the h_p terms in Eq. 9) that -hs penalizes. Hence HarmoGAN's higher h/hit-rate is, under successful optimization, a direct consequence of minimizing Lharmo; it does not independently establish that the loss learned perceptual rhythmic synchronization. The 81.5% user-study evaluates the metric itself on permuted/warped ground-truth clips, and the 12-participant preference test supports overall quality, but none of these anchors breaks the loss/evaluation identity for the central quantitative comparison.

  2. self definitional [Section III.C, final paragraph]
    "As a heuristic evaluation strategy, we perform verification by introducing it as a regularizer during model training for the music-to-motion synthesis task. The validity of this harmony strategy can be confirmed if it successfully guides the motion generation process, producing high-quality cross-modal rhythmic consistency that aligns closely with subjective human perception."

    This states the paper's validation protocol for the harmony strategy: the strategy is deemed valid if, when inserted as a training regularizer, it improves rhythmic consistency. But the reported improvement is measured by the same harmony score h (Eq. 11) and the same beat-alignment-based hit rate, so the test is circular: optimizing the metric is then taken as evidence that the metric captures perception. The separate 81.5% user-study on permuted/warped ground-truth clips is the only non-circular validation, and that study is not used to score the generated motions in Tables I-II.

full rationale

The paper's headline claim is that the harmony-aware loss Lharmo significantly improves rhythmic harmony. The specific reduction is visible in the equations: Lharmo is defined as minus the harmony score h, and the primary quantitative evidence is that HarmoGAN has a higher h and hit rate. Since h is exactly the quantity minimized during training, the increase in h is, to first order, a restatement of the optimization objective rather than an independent confirmation of perceptual synchronization. This is a genuine but partial circularity: the paper also provides external anchors, including an 81.5% user-study consistency for the harmony metric on altered ground-truth clips and a small 12-participant preference study comparing overall video quality, and the motion-realism metrics (MSE, FID, LPIPS) are independent of the harmony score. No load-bearing self-citation chain or imported uniqueness theorem is present; references to the authors' own prior works are peripheral. The non-differentiable construction of h from discrete local-extrema beats and indicator comparisons further undermines the causal attribution of the reported Lharmo-vs-no-Lharmo gap, but the core circularity is the identity of the training objective and the evaluation metric. Overall, the central quantitative claim partially reduces to its own training objective, so a score of 6 is appropriate.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central empirical claims rest on a small number of hand-set or ground-truth-tuned constants, several domain assumptions about human perception and beat semantics, and no invented physical entities.

free parameters (4)
  • lambda1 (audio saliency threshold) = 0.1
    Set in Section V.B to achieve optimal cross-media beat alignment on ground-truth Dance data; controls which audio beats survive the saliency mask in Eq. (4).
  • lambda2 (visual saliency threshold) = 1
    Set in Section V.B to achieve optimal cross-media beat alignment on ground truth; controls visual beat selection in Eq. (6).
  • beta (audio-visual balance in harmony score) = 2
    Chosen by hand in Section V.B to balance audio and visual contributions in Eq. (11); no external or fitted justification is given.
  • lambda_harmo (harmony loss weight) = 1
    Hand-selected loss weight in Eq. (20); the entire rhythmic-harmony claim depends on this term being active at this strength.
assumptions (6)
  • domain assumption Human perceptual reaction delay of 0.25 seconds applies to audio-visual beat alignment in dance videos.
    Tdelay is borrowed from [22] without re-validation on this UGC dataset; it is load-bearing for Eq. (9).
  • domain assumption Local extrema of the joint velocity sum mark perceptually meaningful visual beats.
    Section III.B claims this based on Fig. 4 and movement-unit theory; no independent validation is provided for the Dance dataset.
  • domain assumption Inattentional blindness and deafness justify discarding non-salient beats via SD thresholds.
    The paper imports attention findings [39], [42] to justify Eqs. (4)-(6), but the specific thresholding scheme is not derived from psychophysical data.
  • ad hoc to paper The F-score harmonic mean with beta=2 is an appropriate summary of audio-visual harmony.
    Eq. (11) adopts the F-score form from [47] and selects beta=2 by hand; no experiments compare alternative aggregation schemes.
  • ad hoc to paper The harmony loss can be optimized by gradient descent through the discrete beat detector.
    Eq. (16) uses a non-differentiable pipeline (local extrema, counting, masks); the paper does not provide a differentiable relaxation or a straight-through estimator.
  • domain assumption Musical meter categories from reference [48] are sufficient for segmenting UGC audio.
    The meter mapping table in Section IV.A is described only by a reference and is not specified exactly.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Harmony-Aware Music-driven Motion Synthesis with Perceptual Constraint on UGC Datasets." pith.science (2026). https://pith.science/paper/HBSFFJCQ

@misc{pith2026250607076,
  author       = {Pith},
  title        = {Pith review of: Harmony-Aware Music-driven Motion Synthesis with Perceptual Constraint on UGC Datasets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HBSFFJCQ}},
  note         = {Machine review of arXiv:2506.07076}
}
read the original abstract

With the popularity of video-based user-generated content (UGC) on social media, harmony, as dictated by human perceptual principles, is critical in assessing the rhythmic consistency of audio-visual UGCs for better user engagement. In this work, we propose a novel harmony-aware GAN framework, following a specifically designed harmony evaluation strategy to enhance rhythmic synchronization in the automatic music-to-motion synthesis using a UGC dance dataset. This harmony strategy utilizes refined cross-modal beat detection to capture closely correlated audio and visual rhythms in an audio-visual pair. To mimic human attention mechanism, we introduce saliency-based beat weighting and interval-driven beat alignment, which ensures accurate harmony score estimation consistent with human perception. Building on this strategy, our model, employing efficient encoder-decoder and depth-lifting designs, is adversarially trained based on categorized musical meter segments to generate realistic and rhythmic 3D human motions. We further incorporate our harmony evaluation strategy as a weakly supervised perceptual constraint to flexibly guide the synchronized audio-visual rhythms during the generation process. Experimental results show that our proposed model significantly outperforms other leading music-to-motion methods in rhythmic harmony, both quantitatively and qualitatively, even with limited UGC training data. Live samples 15 can be watched at: https://youtu.be/tWwz7yq4aUs

Figures

Figures reproduced from arXiv: 2506.07076 by the authors.

Figure 1
Figure 1. Illustration of the overall workflow. The proposed generation flow [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Example of synchronized and unsynchronized audio-visual beat pairs, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Distortion of rhythmic harmony between onset-based audio beats and [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (11 more)
Figure 5
Figure 5. Figure 5: Improved beat synchronization between our extracted visual beats and [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 4
Figure 4. Figure 4: Relationship between joint velocity sum J and evolution of human motions. The changes of movement units generally accompany the local minimum and maximum in J. As demonstrated in [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: Mapping from audio to 4 types of meters based on beat analysis. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: The overview of the whole framework, where the training and testing process are demonstrated in details. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Detailed structural designs for all networks in our framework, where [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Comparison of motion quality between HarmoGAN and its other GAN variants based on multiple quantitative metrics. All the metrics are the lower [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 11
Figure 11. Figure 11: Percentage of consistency between human judgments and our [PITH_FULL_IMAGE:figures/full_fig_p010_11.png]
Figure 10
Figure 10. Figure 10: Qualitative comparison based on the Dance dataset to demonstrate [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 13
Figure 13. Figure 13: Average preference percentages across 10 groups of side-by-side [PITH_FULL_IMAGE:figures/full_fig_p011_13.png]
Figure 12
Figure 12. Figure 12: Comparison of cross-modal rhythmic harmony on the Dance dataset [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]
Figure 14
Figure 14. Figure 14: Qualitative comparison example of generated motion sequences with corresponding audio beats (highlighted in red) to illustrate cross-modal rhythmic [PITH_FULL_IMAGE:figures/full_fig_p012_14.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

58 extracted references · 52 canonical work pages

  1. [1]

    Visual rhythm and beat

    A. Davis and M. Agrawala, “Visual rhythm and beat.” ACM Trans. Graph., vol. 37, no. 4, pp. 122–1, 2018

    work page 2018

  2. [2]

    Deepdance: music-to-dance motion choreography with adversarial learning,

    G. Sun, Y . Wong, Z. Cheng, M. S. Kankanhalli, W. Geng, and X. Li, “Deepdance: music-to-dance motion choreography with adversarial learning,” IEEE Transactions on Multimedia, vol. 23, pp. 497–509, 2020

    work page 2020

  3. [3]

    Dance with melody: An lstm-autoencoder approach to music-oriented dance synthesis,

    T. Tang, J. Jia, and H. Mao, “Dance with melody: An lstm-autoencoder approach to music-oriented dance synthesis,” in Proceedings of the 26th ACM international conference on Multimedia , 2018, pp. 1598–1606

    work page 2018

  4. [4]

    Groupdancer: Music to multi-people dance synthesis with style collaboration,

    Z. Wang, J. Jia, H. Wu, J. Xing, J. Cai, F. Meng, G. Chen, and Y . Wang, “Groupdancer: Music to multi-people dance synthesis with style collaboration,” in Proceedings of the 30th ACM International Conference on Multimedia , 2022, pp. 1138–1146

    work page 2022

  5. [5]

    A brand new dance partner: Music-conditioned pluralistic dancing controlled by multiple dance genres,

    J. Kim, H. Oh, S. Kim, H. Tong, and S. Lee, “A brand new dance partner: Music-conditioned pluralistic dancing controlled by multiple dance genres,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 3490–3500

    work page 2022

  6. [6]

    Weakly-supervised deep recurrent neural networks for basic dance step generation,

    N. Yalta, S. Watanabe, K. Nakadai, and T. Ogata, “Weakly-supervised deep recurrent neural networks for basic dance step generation,” in 2019 International Joint Conference on Neural Networks (IJCNN) . IEEE, 2019, pp. 1–8

    work page 2019

  7. [7]

    Automatic translation of music-to-dance for in-game characters

    Y . Duan, T. Shi, Z. Hu, Z. Zou, C. Fan, Y . Yuan, and X. Li, “Automatic translation of music-to-dance for in-game characters.” in IJCAI, 2021, pp. 2344–2351

    work page 2021

  8. [8]

    Rhythm is a dancer: Music-driven motion syn- thesis with global structure,

    A. Aristidou, A. Yiannakidis, K. Aberman, D. Cohen-Or, A. Shamir, and Y . Chrysanthou, “Rhythm is a dancer: Music-driven motion syn- thesis with global structure,” IEEE Transactions on Visualization and Computer Graphics, pp. 1–1, 2022

    work page 2022

Show all 58 references
  1. [9]

    Dancing-to-music character animation,

    T. Shiratori, A. Nakazawa, and K. Ikeuchi, “Dancing-to-music character animation,” in Computer Graphics Forum, vol. 25, no. 3. Wiley Online Library, 2006, pp. 449–458

    2006

  2. [10]

    Music similarity-based approach to generating dance motion sequence,

    M. Lee, K. Lee, and J. Park, “Music similarity-based approach to generating dance motion sequence,” Multimedia tools and applications , vol. 62, no. 3, pp. 895–912, 2013

    2013

  3. [11]

    Multi-modal analysis of dance performances for music-driven choreography synthesis,

    F. Ofli, E. Erzin, Y . Yemez, and A. M. Tekalp, “Multi-modal analysis of dance performances for music-driven choreography synthesis,” in 2010 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2010, pp. 2466–2469

    2010

  4. [12]

    Learn2dance: Learning statistical music-to-dance mappings for choreography synthesis,

    ——, “Learn2dance: Learning statistical music-to-dance mappings for choreography synthesis,” IEEE Transactions on Multimedia , vol. 14, no. 3, pp. 747–759, 2011

    2011

  5. [13]

    Example-based automatic music-driven conventional dance motion synthesis,

    R. Fan, S. Xu, and W. Geng, “Example-based automatic music-driven conventional dance motion synthesis,” IEEE transactions on visualiza- tion and computer graphics , vol. 18, no. 3, pp. 501–515, 2011

    2011

  6. [14]

    Listen to dance: Music-driven choreog- raphy generation using autoregressive encoder-decoder network,

    J. Lee, S. Kim, and K. Lee, “Listen to dance: Music-driven choreog- raphy generation using autoregressive encoder-decoder network,” arXiv preprint arXiv:1811.00818, 2018

    2018 arXiv

  7. [15]

    Graph convolutional gru for music-oriented dance choreography generation,

    Y . Hou, H. Yao, and H. Li, “Graph convolutional gru for music-oriented dance choreography generation,” in 2023 IEEE International Conference on Multimedia and Expo (ICME) . IEEE, 2023, pp. 2111–2116

    2023

  8. [16]

    Combining recurrent neural networks and adversarial training for human motion synthesis and control,

    Z. Wang, J. Chai, and S. Xia, “Combining recurrent neural networks and adversarial training for human motion synthesis and control,” IEEE transactions on visualization and computer graphics , vol. 27, no. 1, pp. 14–28, 2019

    2019

  9. [17]

    Generative autoregressive networks for 3d dancing move synthesis from music,

    H. Ahn, J. Kim, K. Kim, and S. Oh, “Generative autoregressive networks for 3d dancing move synthesis from music,” IEEE Robotics and Automation Letters , vol. 5, no. 2, pp. 3500–3507, 2020

    2020

  10. [18]

    Self-supervised dance video synthesis conditioned on music,

    X. Ren, H. Li, Z. Huang, and Q. Chen, “Self-supervised dance video synthesis conditioned on music,” in Proceedings of the 28th ACM International Conference on Multimedia , 2020, pp. 46–54

    2020

  11. [19]

    Music-driven group choreography,

    N. Le, T. Pham, T. Do, E. Tjiputra, Q. D. Tran, and A. Nguyen, “Music-driven group choreography,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 8673–8682

    2023

  12. [20]

    Edge: Editable dance generation from music,

    J. Tseng, R. Castellon, and K. Liu, “Edge: Editable dance generation from music,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 448–458

    2023

  13. [21]

    Rhythm of motion extraction and rhythm- based cross-media alignment for dance videos,

    W.-T. Chu and S.-Y . Tsai, “Rhythm of motion extraction and rhythm- based cross-media alignment for dance videos,” IEEE Transactions on Multimedia, vol. 14, no. 1, pp. 129–141, 2011

    2011

  14. [22]

    Extraction and alignment evaluation of motion beats for street dance,

    C. Ho, W.-T. Tsai, K.-S. Lin, and H. H. Chen, “Extraction and alignment evaluation of motion beats for street dance,” in 2013 IEEE International Conference on Acoustics, Speech and Signal Processing . IEEE, 2013, pp. 2429–2433

    2013

  15. [23]

    Music to dance: Motion generation based on multi-feature fusion strategy,

    Y . Gao, W. Yu, X. Zhang, X. Deng, and Z. Zhang, “Music to dance: Motion generation based on multi-feature fusion strategy,” in 2022 IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, 2022, pp. 2735–2739

    2022

  16. [24]

    Ai choreographer: Music conditioned 3d dance generation with aist++,

    R. Li, S. Yang, D. A. Ross, and A. Kanazawa, “Ai choreographer: Music conditioned 3d dance generation with aist++,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 13 401–13 412

    2021

  17. [25]

    Transflower: probabilistic autoregressive dance generation with multimodal attention,

    G. Valle-P ´erez, G. E. Henter, J. Beskow, A. Holzapfel, P.-Y . Oudeyer, and S. Alexanderson, “Transflower: probabilistic autoregressive dance generation with multimodal attention,” ACM Transactions on Graphics (TOG), vol. 40, no. 6, pp. 1–14, 2021

    2021

  18. [26]

    Danceformer: Music con- ditioned 3d dance generation with parametric motion transformer,

    B. Li, Y . Zhao, S. Zhelun, and L. Sheng, “Danceformer: Music con- ditioned 3d dance generation with parametric motion transformer,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 36, no. 2, 2022, pp. 1272–1279

    2022

  19. [27]

    Dance to the beat: Synchro- nizing motion to audio,

    R. Bellini, Y . Kleiman, and D. Cohen-Or, “Dance to the beat: Synchro- nizing motion to audio,” Computational Visual Media, vol. 4, no. 3, pp. 197–208, 2018

    2018

  20. [28]

    Dancing to music,

    H.-Y . Lee, X. Yang, M.-Y . Liu, T.-C. Wang, Y .-D. Lu, M.-H. Yang, and J. Kautz, “Dancing to music,” in Advances in Neural Information Processing Systems, 2019, pp. 3586–3596

    2019

  21. [29]

    Choreograph: Music- conditioned automatic dance choreography over a style and tempo con- sistent dynamic graph,

    H. Y . Au, J. Chen, J. Jiang, and Y . Guo, “Choreograph: Music- conditioned automatic dance choreography over a style and tempo con- sistent dynamic graph,” in Proceedings of the 30th ACM International Conference on Multimedia , 2022, pp. 3917–3925

    2022

  22. [30]

    Dance revolution: Long-term dance generation with music via curriculum learning,

    R. Huang, H. Hu, W. Wu, K. Sawada, M. Zhang, and D. Jiang, “Dance revolution: Long-term dance generation with music via curriculum learning,” in International Conference on Learning Representations, 2021. [Online]. Available: https://openreview.net/ forum?id=xGZG2kS5bFk

    2021

  23. [31]

    Learning to dance: A graph convolutional adversarial network to generate realistic dance motions from audio,

    J. P. Ferreira, T. M. Coutinho, T. L. Gomes, J. F. Neto, R. Azevedo, R. Martins, and E. R. Nascimento, “Learning to dance: A graph convolutional adversarial network to generate realistic dance motions from audio,” Computers & Graphics , vol. 94, pp. 11–21, 2021

    2021

  24. [32]

    Deepphase: Periodic autoencoders for learning motion phase manifolds,

    S. Starke, I. Mason, and T. Komura, “Deepphase: Periodic autoencoders for learning motion phase manifolds,” ACM Transactions on Graphics (TOG), vol. 41, no. 4, pp. 1–13, 2022

    2022

  25. [33]

    Bailando: 3d dance generation by actor-critic gpt with choreographic memory,

    L. Siyao, W. Yu, T. Gu, C. Lin, Q. Wang, C. Qian, C. C. Loy, and Z. Liu, “Bailando: 3d dance generation by actor-critic gpt with choreographic memory,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 11 050–11 059

    2022

  26. [34]

    Discriminative cross-modality attention network for temporal inconsistent audio-visual event localization,

    H. Xuan, L. Luo, Z. Zhang, J. Yang, and Y . Yan, “Discriminative cross-modality attention network for temporal inconsistent audio-visual event localization,” IEEE Transactions on Image Processing , vol. 30, pp. 7878–7888, 2021

    2021

  27. [35]

    Sparse in space and time: Audio-visual synchronisation with trainable selectors,

    V . Iashin, W. Xie, E. Rahtu, and A. Zisserman, “Sparse in space and time: Audio-visual synchronisation with trainable selectors,” in 33rd British Machine Vision Conference 2022, BMVC 2022, London, UK, November 21-24, 2022 . BMV A Press, 2022. [Online]. Available: https://bmvc...

    2022

  28. [36]

    Audio-visual mismatch-aware video retrieval via association and adjustment,

    S. Lee, S. Park, and Y . M. Ro, “Audio-visual mismatch-aware video retrieval via association and adjustment,” in European Conference on Computer Vision. Springer, 2022, pp. 497–514

    2022

  29. [37]

    Choreomaster: choreography-oriented music-driven dance synthesis,

    K. Chen, Z. Tan, J. Lei, S.-H. Zhang, Y .-C. Guo, W. Zhang, and S.-M. Hu, “Choreomaster: choreography-oriented music-driven dance synthesis,” ACM Transactions on Graphics (TOG) , vol. 40, no. 4, pp. 1–13, 2021

    2021

  30. [38]

    Benward, Music in Theory and Practice Volume 1

    B. Benward, Music in Theory and Practice Volume 1 . McGraw-Hill Higher Education, 2014, vol. 1

    2014

  31. [39]

    A. Mack, I. Rock et al., Inattentional blindness. MIT press, 1998. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY , VOL. 14, NO. 8, AUGUST 2021 14

    1998

  32. [40]

    Tempo and beat analysis of acoustic musical signals,

    E. D. Scheirer, “Tempo and beat analysis of acoustic musical signals,” The Journal of the Acoustical Society of America , vol. 103, no. 1, pp. 588–601, 1998

    1998

  33. [41]

    Choreonet: Towards music to dance synthesis with choreographic action unit,

    Z. Ye, H. Wu, J. Jia, Y . Bu, W. Chen, F. Meng, and Y . Wang, “Choreonet: Towards music to dance synthesis with choreographic action unit,” in Proceedings of the 28th ACM International Conference on Multimedia , 2020, pp. 744–752

    2020

  34. [42]

    Failure to detect critical auditory alerts in the cockpit: Evidence for inattentional deafness,

    F. Dehais, M. Causse, F. Vachon, N. R ´egis, E. Menant, and S. Tremblay, “Failure to detect critical auditory alerts in the cockpit: Evidence for inattentional deafness,” Human factors , vol. 56, no. 4, pp. 631–644, 2014

    2014

  35. [43]

    Adaptive image sampling using deep learning and its application on x-ray fluorescence image reconstruction,

    Q. Dai, H. Chopp, E. Pouyet, O. Cossairt, M. Walton, and A. K. Katsaggelos, “Adaptive image sampling using deep learning and its application on x-ray fluorescence image reconstruction,” IEEE Trans- actions on Multimedia , vol. 22, no. 10, pp. 2564–2578, 2019

    2019

  36. [44]

    Saliency-aware texture smoothing,

    L. Zhu, X. Hu, C.-W. Fu, J. Qin, and P.-A. Heng, “Saliency-aware texture smoothing,” IEEE transactions on visualization and computer graphics , vol. 26, no. 7, pp. 2471–2484, 2018

    2018

  37. [45]

    Survey of fuzzy logic applications in brain-related researches,

    O. F. Bay and A. B. Usakli, “Survey of fuzzy logic applications in brain-related researches,” Journal of Medical Systems , vol. 27, no. 2, pp. 215–223, 2003

    2003

  38. [46]

    On-line detection of drowsi- ness using brain and visual information,

    A. Picot, S. Charbonnier, and A. Caplier, “On-line detection of drowsi- ness using brain and visual information,” IEEE Transactions on systems, man, and cybernetics-part A: systems and humans , vol. 42, no. 3, pp. 764–775, 2011

    2011

  39. [47]

    Beyond accuracy, f-score and roc: a family of discriminant measures for performance evaluation,

    M. Sokolova, N. Japkowicz, and S. Szpakowicz, “Beyond accuracy, f-score and roc: a family of discriminant measures for performance evaluation,” in Australasian joint conference on artificial intelligence . Springer, 2006, pp. 1015–1021

    2006

  40. [48]

    The sdn theory of music: Rudiments,

    J. W. Acuff and W. D. Evridge, “The sdn theory of music: Rudiments,” 1906

    1906

  41. [49]

    Lifting from the deep: Convolu- tional 3d pose estimation from a single image,

    D. Tome, C. Russell, and L. Agapito, “Lifting from the deep: Convolu- tional 3d pose estimation from a single image,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017, pp. 2500–2509

    2017

  42. [50]

    Consistent video depth estimation,

    X. Luo, J.-B. Huang, R. Szeliski, K. Matzen, and J. Kopf, “Consistent video depth estimation,” ACM Trans. Graph., vol. 39, no. 4, Jul. 2020. [Online]. Available: https://doi.org/10.1145/3386569.3392377

    2020

  43. [51]

    One shot 3d photography,

    J. Kopf, K. Matzen, S. Alsisan, O. Quigley, F. Ge, Y . Chong, J. Patterson, J.-M. Frahm, S. Wu, M. Yu et al. , “One shot 3d photography,” ACM Transactions on Graphics (TOG) , vol. 39, no. 4, pp. 76–1, 2020

    2020

  44. [52]

    Gen- erative adversarial networks and perceptual losses for video super- resolution,

    A. Lucas, S. Lopez-Tapia, R. Molina, and A. K. Katsaggelos, “Gen- erative adversarial networks and perceptual losses for video super- resolution,” IEEE Transactions on Image Processing , vol. 28, no. 7, pp. 3312–3327, 2019

    2019

  45. [53]

    Learning temporal coherence via self-supervision for gan-based video generation,

    M. Chu, Y . Xie, J. Mayer, L. Leal-Taix ´e, and N. Thuerey, “Learning temporal coherence via self-supervision for gan-based video generation,” ACM Transactions on Graphics (TOG) , vol. 39, no. 4, pp. 75–1, 2020

    2020

  46. [54]

    Very deep convolutional networks for large-scale image recognition,

    K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556 , 2014

    2014 arXiv

  47. [55]

    Spatial consistency constrained gan for human motion transfer,

    F. Ma, G. Xia, and Q. Liu, “Spatial consistency constrained gan for human motion transfer,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 2, pp. 730–742, 2021

    2021

  48. [56]

    An experimental comparison of audio tempo induction algorithms,

    F. Gouyon, A. Klapuri, S. Dixon, M. Alonso, G. Tzanetakis, C. Uhle, and P. Cano, “An experimental comparison of audio tempo induction algorithms,” IEEE Transactions on Audio, Speech, and Language Pro- cessing, vol. 14, no. 5, pp. 1832–1844, 2006

    2006

  49. [57]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , 2014

    2014 arXiv

  50. [58]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium,

    M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” arXiv preprint arXiv:1706.08500 , 2017

    2017 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.