{"id":"6863d1ec-9f25-497b-aa9e-1ee47830e64b","arxiv_id":"2507.19836","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A two-stage diffusion framework generates style-controllable dance videos from music and a reference image, using SMPL pose as a bridge between audio and video.","lead":"ChoreoMuse turns a piece of music and one photo of a person into a dance video, with the dance style chosen by the user. The system first generates a 3D dance sequence from the audio, then renders a realistic video of the photo's subject performing that sequence.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Style-adherence SOTA rests on unvalidated MSAS/CSAS; the human study in Sec. 4.4 is not a metric validation, and CSAS has no baseline comparison.","rationale":"The paper is well-structured and the two-stage architecture is plausible, with informative ablations. However, the central 'state-of-the-art' claim is only as strong as the evaluation. The reader correctly identifies the style metrics as the weakest assumption. I agree and emphasize two further points: (1) the human study is not a validation of MSAS/CSAS - it is a separate, unquantified observation about ChoreoMuse only; (2) the 'state-of-the-art' phrasing for style adherence cannot be supported by CSAS at all, since no baseline is evaluated on it. The concrete test of correlating metrics with human judgments would directly address the reader's condition. The video-quality comparisons also lack matched-resolution details for DabFusion and mix tasks by comparing on different datasets, but those are secondary to the metrics issue because they use standard metrics; the style metrics are bespoke and unvalidated. Given the reviewer's CONDITIONAL verdict, my stress-test does not change it: the same conditions (release code, report phi and alpha, validate metrics, add baselines) would resolve the concern. Overall, the paper merits a conditional accept, not a rejection, because the architecture is coherent and the limitations are addressable.","tokens_in":15250,"tokens_out":7223,"duration_ms":77630,"concrete_test":"Correlate per-clip MSAS and CSAS with the binary human judgments from the Sec. 4.4 user study on the same 100 clips (Spearman rho; require rho > 0.2 with p < 0.05 as a minimal bar). Separately, recompute CSAS on a fixed set of generated clips with alpha in {0.1, 1, 10} and with at least two alternative feature extractors (e.g., I3D action features and MotionCLIP embeddings) in place of the unspecified phi; if scores change by more than 0.05 or reorder any two methods, the metric is parameter-sensitive and the state-of-the-art claim is not robust. This settles whether the metrics track perception and whether the reported CSAS value is a stable quantity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"ChoreoMuse's abstract claims state-of-the-art style adherence, but the evidence for this rests entirely on two new metrics defined in Sec. 4.3. MSAS (Eqs. 15-16) uses a style classifier trained on AIST++ (Sec. 4.1), the same corpus used to train the dance generator; high MSAS could reflect that generated sequences are in-distribution for the classifier rather than that they match the intended style. CSAS (Eq. 18) requires an embedding phi(x) that is never specified and a decay parameter alpha that is never reported; the paper reports CSAS only for ChoreoMuse (Table 2) because prior methods lack style control, so no state-of-the-art comparison exists for this dimension. The user study in Sec. 4.4 reports 83.2% music-style and 76.8% choreography-style agreement, but it does not correlate those judgments with MSAS/CSAS scores, and details are deferred to an unavailable supplement. Thus the headline style-adherence claim is unsupported unless the metrics are shown to track human perception and are robust to parameter choices.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ChoreoMuse is a two-stage diffusion framework for music-to-dance video generation. The first stage generates 3D dance sequences in an SMPL-based 6D rotation representation from audio, using a contrastively trained music encoder (MotionTune), a music-style classifier, and a choreography style controller. The second stage renders a reference person into a video guided by the generated 3D sequence, using silhouette/keypoint shape alignment and multi-layer motion fusion. The paper introduces two style metrics (MSAS and CSAS) and reports experiments on AIST++ and TikTok datasets, claiming state-of-the-art video quality, beat alignment, diversity, and style adherence.","tokens_in":15470,"tokens_out":3518,"duration_ms":42662,"significance":"The system is a plausible engineering contribution: using SMPL parameters as an intermediate between music and video is a reasonable design choice that bypasses resolution limits, and MotionTune is a sensible mechanism for aligning audio and motion embeddings. If the style-control and quality claims were rigorously validated, the work would be useful to the computational choreography and human animation communities. However, the paper's headline claims currently outrun its evidence: the style metrics are not validated, the only music-conditioned video baseline is the authors' own DabFusion, and the TikTok comparisons are against pose-guided methods on a different task. The technical scaffolding is interesting, but the evaluation needs substantial strengthening before the state-of-the-art claim can be accepted.","major_comments":[{"comment":"The claim of state-of-the-art style adherence is unsupported. CSAS is reported only for ChoreoMuse in Table 2 (all baselines have '-'), so there is no comparison on this dimension. Moreover, Eq. (18) defines CSAS in terms of an unspecified feature embedding phi(x) and an unreported decay parameter alpha; without these choices and a sensitivity analysis, the metric is not reproducible and its value of 0.84 cannot be interpreted. The paper must specify phi and alpha, and either provide a baseline comparison on CSAS or restrict the claim to internal evaluation.","section":"Sec. 4.3, Eq. (18), Table 2"},{"comment":"MSAS is likely inflated by distributional overlap rather than style fidelity. The multi-class style classifier used in MSAS is trained on AIST++ (Sec. 4.1), the same dataset used to train the dance generator, so generated sequences are in-distribution for the classifier by construction. High MSAS may simply reflect that the generated dances resemble the training set, not that they match the intended musical style. To support the style-adherence claim, the authors should correlate MSAS with human perceptual judgments or evaluate with an independently trained style classifier on out-of-distribution data.","section":"Sec. 4.1, Sec. 4.3, Eq. (15)"},{"comment":"The video-quality comparison on TikTok is not an apples-to-apples evaluation of music-to-dance generation. DisCo, MagicAnimate, and Animate Anyone are pose-guided image animation baselines: they are conditioned on ground-truth poses rather than on music, so the task is different from ChoreoMuse's music-conditioned video generation. The reported margins on TikTok (e.g., PSNR 29.85 vs 29.49, FVD 165.4 vs 173.5) therefore do not establish superiority for the music-to-dance task. On AIST++, where the task matches, the only baseline is DabFusion, the authors' own prior system; this single comparison does not justify the phrase 'state-of-the-art across multiple dimensions' in the abstract.","section":"Sec. 4.2, Table 1"},{"comment":"The user study does not validate the proposed metrics as claimed. The text says the study was conducted 'to validate these metrics', but it only reports whether participants judged generated videos as matching music style (83.2%) and choreography style (76.8%). It never correlates these human judgments with MSAS or CSAS scores, so it cannot establish that either metric tracks human perception. In addition, the 80% success threshold is ad hoc and no confidence intervals or significance tests are supplied. The authors should either report a correlation analysis between human ratings and the proposed metrics or remove the validation claim.","section":"Sec. 4.4, Human Evaluations"},{"comment":"All quantitative results are reported without variance or significance testing. For example, Table 1 reports FVD values 176.3 (MagicAnimate) vs 165.4 (Ours) on TikTok, and Table 2 reports BAS 0.26 (EDGE) vs 0.28 (Ours), but without multiple runs, confidence intervals, or paired significance tests these differences may not be reliable. Given that several comparisons involve small margins, the paper should include error bars and statistical tests for the headline metrics (PSNR, SSIM, LPIPS, FVD, PFC, BAS, diversity, MSAS, CSAS).","section":"Tables 1–5"}],"minor_comments":[{"comment":"The symbol 'B' in the equation 'x_{t-1} B m⊙q(...) + ...' appears to be a typesetting error; it should probably be an equality or assignment symbol.","section":"Sec. 3.3, Eq. (11)"},{"comment":"The paper mentions that the model can generate a 7.5-second clip by constraining the first 2.5 seconds of the new sequence, but all training clips are 5 seconds; a brief description of how the model handles variable-length inference would help reproducibility.","section":"Sec. 3.3"},{"comment":"The sentence 'we propose our CSAS metric as a baseline for future style-controllable research' is confusing because a baseline is a comparison method, not a metric; rephrase to describe CSAS as a proposed evaluation protocol.","section":"Sec. 4.4"},{"comment":"The description of the MSAS classifier and the CSAS centroid computation would benefit from stating the dimensionality of the embeddings and the number of style classes, since these details affect the interpretation of the scores.","section":"Sec. 4.3"},{"comment":"The user study section states that 'Additional user study details are provided in the supplementary materials', but no supplement is included with the manuscript; if this is a journal submission, the supplementary material should be provided.","section":"Sec. 4.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a preprint-style submission with no supplementary material despite repeated references to one. The authors should be asked to supply the supplement and to substantially rework the evaluation before resubmission. In particular, the style-adherence claim needs either a validated metric (with human correlation and parameter robustness) or a more modest statement, and the video-quality comparison should be reframed with appropriate baselines and uncertainty quantification. The core system is interesting, so I do not recommend rejection at this stage, but the current evidence does not support the 'state-of-the-art' claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: ChoreoMuse is a competent two-stage system for music-to-dance video generation. The architecture is a clean integration of existing ideas—DDPM, SMPL/6D pose, contrastive audio-motion learning, Jukebox features, a music classifier, and Champ-style multi-layer fusion—plus a new MotionTune encoder. The ablations are informative: removing MotionTune hurts most metrics, and the 6D representation helps pose gen while original SMPL helps video gen. If you work on dance generation or human video animation, this is worth a careful read.\n\nWhat's actually new is the packaging: SMPL as the intermediate between music and video, style control via predicted music genre and text, and the two proposed metrics MSAS and CSAS. The silhouette-based shape alignment is a sensible addition.\n\nThe soft spots are real but mostly fixable. The headline claim of state-of-the-art style adherence rests on MSAS and CSAS, both introduced here. MSAS uses a style classifier trained on AIST++, the same dataset that trained the generator, so high MSAS may just mean the outputs are in-distribution for that classifier. CSAS depends on an embedding phi(x) that is never specified and a decay parameter alpha that is never reported, and it is only computed for ChoreoMuse—there is no comparison. The user study in Sec. 4.4 shows decent agreement rates, but it does not correlate human judgments with the proposed metrics, so it does not validate them. The rest of the evaluation also has no error bars or significance tests, and the direct music-to-dance baseline is only the authors' own DabFusion. The TikTok video-quality comparison is against pose-guided models like MagicAnimate and Animate Anyone, which are solving a different task. So \"state-of-the-art across multiple dimensions\" overshoots what the evidence supports.\n\nThese are correctable. Release code and hyperparameters, add error bars, correlate MSAS/CSAS with human judgments, and compare against a proper music-conditioned baseline. The architecture itself is coherent, and the ablations show the design choices matter. I'd send it to reviewers; it deserves referee time. I'd be hesitant to cite the style metrics until they're externally validated, but the system as a whole is a legitimate contribution to the subfield.","headline":"A well-built music-to-dance video system, but its style-adherence SOTA claim rests on unvalidated metrics and thin baselines.","tokens_in":16013,"tokens_out":2607,"would_cite":true,"duration_ms":29275,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ChoreoMuse is a two-stage diffusion framework that generates style-controlled dance videos from any music and reference image, using SMPL body parameters as the bridge between music and pixels.","keywords":["music-to-dance generation","image-to-video generation","diffusion model","SMPL parametric body model","beat alignment","style transfer","contrastive learning","dance video synthesis"],"falsifier":"Run a preregistered human study in which independent choreographers pick which generated clip best matches a target choreography style, and check whether the CSAS and MSAS rankings agree with their picks when the feature extractor in CSAS is replaced by a frozen, independently trained motion encoder; if the rankings flip, the style-alignment claim is an artifact of the metric.","tokens_in":15025,"feed_emoji":"💃","tokens_out":9407,"duration_ms":93990,"temperature":0.7,"pith_summary":"ChoreoMuse is a two-stage diffusion framework that generates dance videos from any piece of music and a single reference image, with user-selectable choreography style. The paper's central claim is that 3D body parameters in the SMPL format are a better intermediary between music and pixels than the optical-flow or keypoint representations used by earlier systems, because they are resolution-independent and carry enough structure to guide high-fidelity video. A second claim is that a dedicated music encoder, MotionTune, trained with contrastive audio-motion alignment, captures dance-relevant cues that generic audio features miss, producing stronger beat alignment and greater diversity. The paper also introduces two metrics, MSAS and CSAS, to measure alignment with musical and choreographic style, and reports that ChoreoMuse beats prior music-to-dance video and pose-guided animation methods on all reported benchmarks.","feed_headline":"ChoreoMuse dances to any song in any chosen style from one photo","feed_subtitle":"SMPL body parameters carry the beat from music to pixels, letting the user pick the dancer, the style, and the scene.","key_machinery":"The load-bearing object is the SMPL parametric body model, a low-dimensional representation of human shape and pose, used in two parameterizations: the original compact form for the video stage and a 6D-rotation form with foot-contact labels for the choreography stage. Stage one uses a DDPM conditioned on a fused embedding made from MotionTune's contrastively trained audio-motion representation, Jukebox features, and a style embedding produced by a music classifier and text encoder. Stage two renders the transferred SMPL mesh into depth, normal, segmentation, and foot-contact maps, fuses them with Multi-Layer Motion Fusion, and feeds them through cross-attention to a U-Net video diffusion model, with a silhouette-based shape-alignment step adjusting the SMPL body parameters to match the reference person's contours.","core_discovery":"The central claim is that decomposing music-to-dance video generation into two diffusion stages connected by an explicit SMPL-format 3D dance sequence removes the resolution and background constraints that plague direct music-to-video methods while preserving beat adherence and enabling style transfer. In the first stage, a denoising diffusion model generates a 3D dance sequence in a 6D-rotation variant of SMPL, conditioned on music embeddings and a classifier-driven choreography style; in the second, a video diffusion model animates the reference person from that sequence rendered into depth, normal, segmentation, and foot-contact maps. The authors report that this design outperforms the prior direct music-to-dance baseline and several pose-guided animation methods on video quality, and outperforms motion-generation baselines on beat alignment, dance diversity, and the two new style-alignment scores.","pith_inferences":["Because the SMPL intermediate decouples motion from appearance, the same trained modules could plausibly be retargeted to stylized avatars or non-human subjects, which the paper's toy, comic-character, and oil-painting examples already hint at.","A natural extension would let users supply reference dance clips instead of choosing a classifier-defined genre, so the style controller could imitate an arbitrary choreographic vocabulary rather than only the styles present in AIST++.","The silhouette-based shape alignment suggests a testable recipe: fit SMPL to a reference photo once, then reuse those fitted parameters across many music inputs, isolating choreography quality from identity-preservation quality in evaluation."],"forward_implications":["Music-to-dance video generation becomes resolution-independent: the output video can match the resolution and environment of any reference image rather than a fixed low-resolution canvas.","Users can choreograph the same music in different styles, and can reuse one choreography with different reference people or backgrounds, because the 3D dance sequence and the final video are generated in separate stages.","The contrastively trained MotionTune encoder should improve beat alignment and motion diversity relative to conditioning on generic audio features alone.","The MSAS and CSAS scores give later work a quantitative way to compare style adherence, extending evaluation beyond beat scores and physical-plausibility checks."],"supporting_citations":[{"why":"Supplies the SMPL parametric body model that the whole two-stage pipeline uses as the music-to-video bridge.","marker":"[36]"},{"why":"Provides the AIST++ dataset of paired music and 3D dance, the genre and style taxonomy, and the training data for MotionTune and the choreography stage.","marker":"[35]"},{"why":"Defines the DDPM noising and denoising processes and loss used in both the dance-sequence and video generation stages.","marker":"[20]"},{"why":"The prior direct music-to-dance video model that ChoreoMuse compares against on the AIST++ test set.","marker":"[56]"},{"why":"Contributes the auxiliary joint, velocity, and foot-contact losses and the PFC, Dist_k, and Dist_g evaluation metrics used for choreography quality.","marker":"[53]"},{"why":"Provides the Beat Alignment Score baseline and is one of the motion-generation methods compared.","marker":"[48]"},{"why":"Supplies Jukebox's general-purpose music representation, fused with MotionTune to condition dance generation.","marker":"[9]"},{"why":"Provides the human-video dataset and the Multi-Layer Motion Fusion mechanism used in the video generation stage.","marker":"[65]"}],"fun_headline_variants":["Dance video from any song, style, and single photo","One photo, any song, any style: ChoreoMuse dances","Style-swap dance videos from just one reference","Beat-perfect choreography from any track and photo","Turn any song and photo into a custom dance video"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The style-adherence claim rests on two new scores defined by the same authors: one uses a style classifier trained on the same AIST++ dataset used to train the model, and the other uses a feature extractor that is never specified plus a hand-chosen decay parameter, so the whole style-match result depends on these scores tracking what humans actually perceive as style.","fun_headline_variants_meta":{"raw":{"variants":["Dance video from any song, style, and single photo","One photo, any song, any style: ChoreoMuse dances","Style-swap dance videos from just one reference","Beat-perfect choreography from any track and photo","Turn any song and photo into a custom dance video"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000731,"raw_usage":{"total_tokens":3285,"prompt_tokens":973,"completion_tokens":2312,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":2244}},"tokens_in":589,"tokens_out":2312,"duration_ms":19761,"temperature":1.0,"reasoning_tokens":2244,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T13:58:52.266629+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a preregistered human study in which independent choreographers pick which generated clip best matches a target choreography style, and check whether the CSAS and MSAS rankings agree with their picks when the feature extractor in CSAS is replaced by a frozen, independently trained motion encoder; if the rankings flip, the style-alignment claim is an artifact of the metric.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SMPL parametric body model that the whole two-stage pipeline uses as the music-to-video bridge."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the AIST++ dataset of paired music and 3D dance, the genre and style taxonomy, and the training data for MotionTune and the choreography stage."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The prior direct music-to-dance video model that ChoreoMuse compares against on the AIST++ test set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the auxiliary joint, velocity, and foot-contact losses and the PFC, Dist_k, and Dist_g evaluation metrics used for choreography quality."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the human-video dataset and the Multi-Layer Motion Fusion mechanism used in the video generation stage."}],"review_version":1}