{"id":"e4851176-fbf6-44f9-9df8-560a2582e949","arxiv_id":"2505.18078","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A diffusion model fuses per-person masks with pose keypoints to generate identity-preserving, two-person interactive videos from a single reference image, outperforming prior single-person-animation pipelines.","lead":"DanceTogether generates two-person videos from one reference photo plus separate pose and mask tracks for each person, keeping each person's identity stable across hundreds of frames. It also contributes new figure-skating and robot-interaction datasets and a three-part benchmark for multi-person controllable video generation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HOTA/IDF1 on Track 1 measure track stability, not whether each generated actor matches its reference identity; a swapped or blended appearance can still score high, so the paper's strict identity-preservation claim is not actually tested.","rationale":"The reader's concern is that the automated SAMURAI/DWPose/OSNet pipeline may be a biased instrument because it generated the training annotations. I agree that this detector-legibility bias is real and can inflate the reported gains. My concern is more specific and even more load-bearing: the identity metrics used in Track 1 are the wrong instrument for the claim, independent of detector bias. HOTA and IDF1 measure temporal detection and association quality of automatically extracted tracks; they do not compare any generated actor's appearance to the reference identity. A global identity swap between the two actors, or a blending of their appearances into a common look, would not be penalized as long as tracks remain stable and aligned to the GT trajectories. Therefore the paper's central assertion of 'strictly preserving every identity' is not actually tested by the benchmark. This does not mean the method fails; the ablations and qualitative results suggest the mask-conditioned design is plausible and may well preserve identity. It means the headline claim currently rests on an untested assumption, so the appropriate verdict remains CONDITIONAL: independent face-identity verification and ideally an independent detector pass are needed before the claim can be accepted at face value.","tokens_in":26112,"tokens_out":6278,"duration_ms":59818,"concrete_test":"On DanceTogEval-100, for each generated video, detect face crops per frame with RetinaFace (not the training mask pipeline), extract embeddings with AdaFace (an off-the-shelf face recognizer not used in training), and compare each generated track's mean embedding to the two reference face embeddings. Report same-identity cosine similarity and cross-identity confusion (fraction of frames where the track assigned to GT person A is closer to reference B) for DanceTogether vs. StableAnimator w/Dataswing. If same-identity similarity is not clearly higher and confusion not clearly lower than the baseline, the HOTA/IDF1 gains do not establish identity preservation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Track 1's central metrics cannot certify the paper's headline claim. HOTA (Eq. 23) is sqrt(DetA * AssA), and IDF1 (Eq. 21) is a harmonic mean of ID precision/recall over pipeline-extracted tracks. Neither term references the input reference image or the appearance of the two identities. They only measure whether the automated detector/re-ID pipeline produces temporally stable, separable instances that align with ground-truth trajectory labels. A generated video with the two actors' identities swapped, or with both faces blended into a single average appearance, can still yield stable separate tracks and hence high HOTA/IDF1. Thus the identity-consistency track does not test 'strictly preserving every identity' (Abstract) or 'eliminating appearance bleeding' (Sec. 3.3), even if the detector pipeline were unbiased. Moreover, because DWPose/SAMURAI annotations generated by the same pipeline were used as training conditions (Secs. D.3, E), the model is optimized to be legible to these exact detectors, which can inflate HOTA, MPJPE2D, and masked FVD alike. No reference-to-generated face similarity or human study is reported, leaving the central claim supported mainly by qualitative stills.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DanceTogether, an end-to-end diffusion framework for generating two-person interaction videos from a single reference image plus per-person pose and mask sequences. The method introduces a MaskPoseAdapter that fuses tracking masks with pose heatmaps to bind identity to action, and a MultiFace Encoder that injects identity tokens into cross-attention. The authors also contribute PairFS-4K and HumanRob-300 datasets, and a three-track benchmark (TogetherVideoBench) with 100 curated test clips. They report large improvements over prior baselines on identity-consistency (HOTA 83.94 vs. 71.35), interaction-coherence (MPJPE2D 492 vs. 1555 px), and video quality (masked FVD 17.1 vs. 29.0).","tokens_in":26371,"tokens_out":6980,"duration_ms":54774,"significance":"If the reported gains reflect genuine identity preservation, the work is a useful step toward controllable multi-person video generation: the MaskPoseAdapter design and the ablations isolating the mask input are informative, and the datasets/benchmark are a potential community resource. The paper also includes a same-data comparison (DanceTog w. Dataswing vs. StableAnimator w. Dataswing) that partially controls for training data, and the module ablation in Table 6 clearly shows the mask input is critical. However, the central claim of 'strictly preserving every identity' is not substantiated by the chosen metrics, and the evaluation pipeline's reliance on the same detectors used to create training annotations is a serious correctness risk.","major_comments":[{"comment":"The Identity-Consistency track does not measure identity preservation relative to the reference image. HOTA (Eq. 23) is sqrt(DetA * AssA) and IDF1 (Eq. 21) is a harmonic mean of ID precision/recall; both are computed from per-frame detections, ReID associations, and ground-truth trajectory labels. They measure whether the automated pipeline yields temporally stable, separable tracks, not whether each actor's appearance matches the reference identity. A generated video with the two identities swapped, or with both faces blended into an average appearance, can still produce stable separate tracks and hence high HOTA/IDF1. Consequently, Table 2 and the Abstract's 'strictly preserving every identity' and Sec. 3.3's 'eliminating appearance bleeding' are not supported by the reported evidence. Please add a reference-to-generated appearance metric (e.g., ArcFace cosine similarity between the reference face crop and each generated per-person face crop averaged over frames) and ideally a human perception study.","section":"Sec. F.1, Eq. (21), Eq. (23)"},{"comment":"All three tracks rely on the same automated pipeline (DWPose, SAMURAI, OSNet) used to produce training annotations, which creates a circularity risk. MPJPE2D (Eq. 25) is computed from DWPose keypoints on generated frames; the model is trained to satisfy DWPose-derived pose conditions, so it may be optimized to be legible to DWPose rather than to produce accurate human pose. Similarly, Track-3 masked FVD/FID use SAMURAI masks on the generated video, and Track-1 uses OSNet ReID embeddings; all are the same detectors/embeddings used in Sec. D.3 to annotate training data. A model that learns to produce videos 'legible' to these exact detectors can score high even if human-visible identity and pose fidelity are worse. Please evaluate at least one independent detector/tracker (e.g., a different pose estimator and a different segmentation or ReID model) on a subset, or validate the automated scores against human ratings on a subset, and report the agreement.","section":"Sec. D.3, Sec. F.1, Eq. (25)"},{"comment":"The headline comparisons are asymmetric. In Sec. 4.2 only StableAnimator is fine-tuned, on the swing-dance subset for 40 epochs; all other baselines are used with released weights, while DanceTogether is trained on the full multi-person dataset (Tab. 1) including the new PairFS-4K. The only controlled comparison that isolates method from data is 'DanceTog w. Dataswing' versus 'StableAnimator w. Dataswing' in Tables 2–5, which still shows substantial gains. But the prose and Abstract quote the fully trained model's numbers. Please either fine-tune all baselines on the same training data, or restrict the 'SOTA' claims to the same-data row, and discuss the contribution of additional training data explicitly.","section":"Sec. 4.2, Tables 2–5"}],"minor_comments":[{"comment":"'Pre-trained weights from StableAnimator [5]' cites reference [5] (Stable Video Diffusion); the StableAnimator reference is [79]. This citation error should be corrected.","section":"Sec. 4.1"},{"comment":"The sentence 'Tevet et al. [77] generate high-quality action sequences but lack robust identity modeling' describes a human motion diffusion model, not a video generation method; please rephrase to avoid confusion.","section":"Sec. 2.2"},{"comment":"The tables report no confidence intervals or significance tests over generation seeds; given the stochastic nature of diffusion sampling and only 100 test clips, please report at least three seeds with error bars, or per-clip bootstrap intervals, for the main comparisons.","section":"Tables 2–5"},{"comment":"The text calls lambda a learnable weight and reports lambda approx 0.8, while s_p is set to 1.5 and alpha_res to 0.5. Please clarify which quantities are learned and which are fixed, and report the final learned values.","section":"Sec. 3.3, Eqs. (7) and (10)"},{"comment":"The detection model is referred to as YOLOv8x in Sec. 3.5 and D.2 but as YOLOX in Sec. E; please make the detector name consistent.","section":"Secs. 3.5, D.2, E"},{"comment":"Some table entries have formatting artifacts (e.g., 'DanceTog w.Data f ull+DataP airF S'); please clean the row labels for readability.","section":"Tables 2–5"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a strong 'first' claim and a 'strictly preserving identity' claim that are not fully supported by the current evaluation. The method, datasets, and benchmark are potentially valuable, and the requested revisions are achievable. The main risks are the circular detector-based evaluation and the lack of any appearance-level identity metric; these should be addressed before publication in a serious venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nDanceTogether is worth knowing about for its mechanism and resources, not because its headline numbers establish the claim. The MaskPoseAdapter idea—gating noisy pose heatmaps with stable tracking masks at every denoising step—is sensible, and the ablation evidence supports it: dropping the mask input collapses HOTA from 83.9 to 33.6. That is a real signal that the mask stream carries the identity–action binding the paper describes. The MultiFace Encoder is a modest but clean extension of StableAnimator to multiple identity token sets. PairFS-4K and TogetherVideoBench, if actually released, would be useful community assets.\n\nThe soft spots are in the evaluation, and they are load-bearing for the central claim. Track 1's HOTA, IDF1, and MOTA are multi-object tracking metrics computed on SAMURAI/DWPose/OSNet outputs. They measure whether the automated pipeline produces stable, separable tracks over time; they never compare the generated person's appearance to the reference identity. A video with the two identities swapped, or with both faces blended into a common average, can still score high on HOTA/IDF1. The abstract says “strictly preserving every identity,” but no metric in Track 1 tests reference-to-generated identity match, and no human study or face-similarity score is reported. That is not a minor gap; it is the paper's main claim.\n\nThe circularity compounds it. The same DWPose/SAMURAI/ReID pipeline that generated the training annotations (Secs. D.3, E) is used to score the generated videos (Sec. F.1). A model trained to be legible to those detectors will look better on those metrics even when human-visible identity consistency is weaker. The Limitations section acknowledges dependence on detector quality, but not this evaluation loop. Baseline asymmetry also weakens the comparison: only StableAnimator is fine-tuned on the swing-dance data; the other baselines appear to use released weights, which likely understates their multi-person capability. No error bars or significance tests are reported.\n\nNone of this makes the method worthless. The ablations and qualitative side-by-sides suggest the approach genuinely improves two-person identity consistency over long sequences. What is missing is a fair, independent evaluation of identity preservation: at minimum reference-to-generated face similarity (e.g., ArcFace cosine between the reference and each generated track) plus a human perceptual study, and consistently fine-tuned baselines.\n\nThis paper deserves a serious referee, but with heavy revision requested. The mechanism is worth publishing and the dataset/benchmark could help the field. I would send it to review with the clear instruction that the identity claim needs a metric that actually measures identity.","headline":"A plausible multi-person animation method with a real new mechanism and, if released, useful datasets, but the evaluation does not actually test the paper's headline claim of strict identity preservation.","tokens_in":26954,"tokens_out":2033,"would_cite":true,"duration_ms":17892,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DanceTogether claims to be the first end-to-end diffusion framework that turns a single reference image plus per-person pose and mask streams into long, photorealistic two-person videos while strictly preserving every identity, by binding…","keywords":["controllable video generation","identity preservation","multi-person interaction","diffusion model","pose-guided animation","mask-pose fusion","video generation benchmark","human-robot interaction"],"falsifier":"Take the DanceTogEval-100 outputs and have human raters, or an independent detector suite never used in training, judge identity switches and pose faithfulness; if human-visible identity drift or detector-free judgment diverges from the HOTA/IDF1/MPJPE2D numbers, the benchmark's margin is partly an artifact of the model learning to be legible to SAMURAI-DWPose-OSNet rather than a true gain in identity preservation.","tokens_in":25895,"feed_emoji":"💃","tokens_out":8455,"duration_ms":63272,"temperature":0.7,"pith_summary":"DanceTogether claims that multi-person controllable video generation fails because existing systems never bind identity to motion in a persistent way: a frame-wise pipeline lets who a person is drift apart from what they are doing. The paper's fix is to feed the diffusion model two control streams per actor — a pose map that says how the body moves and a tracking mask that says where this particular person is — fused at every denoising step so the two can never decouple. To make the claim testable, the authors build their own two-person figure-skating dataset (PairFS-4K), a one-hour robot-interaction set, and a three-track benchmark on 100 clips spanning dance, boxing, wrestling, yoga, and skating. On that benchmark the method beats the strongest prior model by a wide margin, and the paper shows that the gains come specifically from the mask-pose fusion and identity tokens rather than from more training data alone.","feed_headline":"One photo, two dancers, zero identity drift","feed_subtitle":"MaskPoseAdapter fuses tracking masks with pose maps so both actors keep their identities through position swaps.","key_machinery":"The MaskPoseAdapter is the load-bearing module: it encodes each person's RGB pose map with a shared PoseNet, runs binary tracking masks through a light 3-channel processor, gates both streams per-pixel with a learnable weight (λ≈0.8 favoring pose), enhances and LayerNorms the fused features, then applies temperature-scaled soft attention across the person dimension so each subject's identity and motion are bound in a single latent at every denoising step. It works alongside the MultiFace Encoder, which turns an ArcFace vector into K=4 Perceiver-refined identity tokens appended to the UNet's cross-attention keys/values, and a masked reconstruction loss that weights body regions 1× and face regions 2×. The paper's ablation shows that removing the adapter collapses HOTA from 83.94 to 48.95, which is the concrete evidence that the fusion — not just the extra data — carries the result.","core_discovery":"The central claim is that an explicit, persistent binding between identity and action is what lets a diffusion model generate long, photorealistic multi-actor videos from a single reference image. DanceTogether implements this with three coupled modules: a MultiFace Encoder that compresses an ArcFace identity embedding into four refined tokens injected into every cross-attention layer; a MaskPoseAdapter that fuses per-person tracking masks with semantically rich but noisy pose heat-maps through gated, attention-reordered integration, producing a single pose-mask latent; and a StableAnimator-style video diffusion backbone with a human-tracking masked reconstruction loss that upweights face and body regions. Trained on the paper's curated multi-person data, the pipeline lifts HOTA from 71.35 to 83.94, cuts MPJPE2D from 1555 px to 492 px, and improves masked-region FVD from 29.0 to 17.1 relative to the strongest baseline on the new TogetherVideoBench.","pith_inferences":["The benchmark's detectors (SAMURAI masks, DWPose keypoints, OSNet ReID) also produced the training annotations, so the reported HOTA/IDF1 gains may partly measure how legible the generated video is to those specific models rather than to human viewers; a human-rating study or a held-out detector suite would separate the two effects.","The gated-fusion recipe is portable: any control task pairing a semantically rich but noisy signal with a sparse but reliable one (depth plus keypoint tracks, text prompts plus object masks, robot joints plus silhouettes) could reuse the same MaskPoseAdapter-style design.","If identity-action binding is the true cause of the gains, the framework should transfer to non-human multi-instance generation (multiple objects, multiple camera viewpoints of a scene) without architectural rethinking — a directly testable prediction of the paper's framing.","The ablation ordering (framework +8.91 HOTA on swing data versus PairFS-4K adding only +2.15 HOTA on top of full data) suggests that once interaction types are represented, the fusion machinery rather than raw data volume is the binding constraint."],"forward_implications":["Identity drift and appearance bleeding in two-person generation become tractable: the same single-reference-image interface used for single-person animation now works for pairs, including position exchanges and occlusion-heavy interactions.","Because the conditioning is per-person, a user can choreograph two actors independently and the model composes them into one coherent video, realizing what the paper calls compositionally controllable multi-actor interaction.","A one-hour fine-tune on the HumanRob-300 robot set yields human-robot interaction videos that retain each agent's identity, pointing to cheap cross-domain transfer for embodied-AI and human-robot interaction work.","TogetherVideoBench gives the field three standard tracks (identity consistency, interaction coherence, video quality), and the reported margins set the bar that future multi-person controllable video generation systems must beat."],"supporting_citations":[{"why":"StableAnimator supplies the SVD-based UNet backbone, the PoseNet and face-encoder initialization, and the strongest baseline (fine-tuned on swing data) that DanceTogether must beat.","marker":"[79]"},{"why":"SAMURAI's zero-shot tracking masks supply the robust per-person signal fused in MaskPoseAdapter and generate the human-mask annotations used for training and identity-consistency evaluation.","marker":"[103]"},{"why":"DWPose produces the 133-point pose maps that serve as conditioning input during training and as the basis of MPJPE2D, OKS, and PoseSSIM scores.","marker":"[107]"},{"why":"OSNet ReID features assign and maintain subject IDs through the data curation pipeline, grounding the identity ground truth that the benchmark tracks.","marker":"[117, 118, 119]"},{"why":"ArcFace provides the frozen identity embedding that the MultiFace Encoder refines into per-person tokens for cross-attention injection.","marker":"[17]"},{"why":"Stable Video Diffusion is the pre-trained latent video model whose UNet and VAE form the frozen backbone of the video diffusion stream.","marker":"[5]"},{"why":"The Swing Dance dataset defines the strongest baseline's fine-tuning data and is a two-person training corpus; the swing-data comparison isolates the framework's contribution from the data's contribution.","marker":"[58]"},{"why":"HOTA defines the identity-consistency track's headline metric and motivates the association-quality design of MaskPoseAdapter.","marker":"[53]"}],"fun_headline_variants":["One image, many actors, all identities kept","Identity-action binding lifts multi-person video quality","DanceTogether: multi-actor video with no drift or bleeding","From one photo to interactive duets with intact identities"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported identity and motion scores assume that the automated detectors used to measure them are neutral, accurate instruments, even though the same mask tracker, pose estimator, and ReID model also produced the training annotations those scores reward.","fun_headline_variants_meta":{"raw":{"variants":["One image, many actors, all identities kept","Identity-action binding lifts multi-person video quality","DanceTogether: multi-actor video with no drift or bleeding","From one photo to interactive duets with intact identities"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000298,"raw_usage":{"total_tokens":1779,"prompt_tokens":1056,"completion_tokens":723,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":672,"completion_tokens_details":{"reasoning_tokens":660}},"tokens_in":672,"tokens_out":723,"duration_ms":6873,"temperature":1.0,"reasoning_tokens":660,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:35:06.669916+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the DanceTogEval-100 outputs and have human raters, or an independent detector suite never used in training, judge identity switches and pose faithfulness; if human-visible identity drift or detector-free judgment diverges from the HOTA/IDF1/MPJPE2D numbers, the benchmark's margin is partly an artifact of the model learning to be legible to SAMURAI-DWPose-OSNet rather than a true gain in identity preservation.","supporting_citations":[{"cited_title":"Effective whole-body pose estimation with two- stages distillation","cited_arxiv_id":null,"evidence_quote":"DWPose produces the 133-point pose maps that serve as conditioning input during training and as the basis of MPJPE2D, OKS, and PoseSSIM scores."}],"review_version":1}