{"id":"ad6b5fca-2467-49af-9123-dd3e0774547d","arxiv_id":"2411.13607","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"VioPose fuses audio and visual keypoints in a hierarchical network to estimate 3D violin-performance poses, outperforming visual-only baselines on a new synchronized dataset.","lead":"VioPose is a new computer vision system that estimates 3D violin-player poses over time by combining video-derived body keypoints with the audio being played. The paper also introduces VioDat, a synchronized video-audio-motion-capture dataset of 12 violinists, and reports improved pose accuracy, particularly for subtle motions such as vibrato.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported MPJVE/MPJAE gains are not a fair comparison: VioPose is trained with velocity and acceleration losses (Eq. 8) while SoTA baselines are not, so the subtle-motion claim is unsupported until baselines are retrained with the same loss.","rationale":"The central claim is empirical: VioPose outperforms SoTA on VioDat, especially for subtle motion. The largest reported advantages are on MPJVE and MPJAE, and the downstream vibrato result. These metrics are exactly what the extra loss terms in Eq. (8) optimize. The paper states baselines were retrained on VioDat for fairness, but does not control the training objective; SoTA methods are typically trained with MPJPE only. Therefore, the comparison conflates the architecture/audio with the loss function. This is more load-bearing than the audio-causality premise because even if the audio were perfectly causal, the empirical evidence for its benefit would still be confounded. A single ablation—training the best baseline with VioPose's loss—could settle it. The small test set (3 participants) compounds the issue but is a secondary concern; the loss mismatch alone is sufficient to undermine the 'subtle motion' claim. I therefore align with the reader's conditional verdict but not with the reader's stated weakest assumption.","tokens_in":17450,"tokens_out":10355,"duration_ms":96608,"concrete_test":"Retrain the strongest visual baseline (e.g., P-STMO or StridedFormer) on VioDat using the identical training setup as VioPose, including the loss in Eq. (8) with the same λv and λa, and report Table 2 metrics. If its MPJVE/MPJAE drop to VioPose's level (or its MPJPE changes), then the claimed subtle-motion advantage is attributable to the loss function, not the audiovisual hierarchy. Additionally, train a VioPose variant with only Lp to measure the loss's contribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"VioPose's central claim of superior 4D pose estimation, especially for subtle motions like vibrato, rests on the MPJVE and MPJAE columns of Table 2 (1.57 vs 1.90; 1.02 vs 1.69) and the vibrato analysis in Table 7. However, the training objective in §3.5 (Eq. 8) explicitly adds Lv and La, while the SoTA baselines are retrained with their original loss functions (typically MPJPE only) under 'fair comparison' (§4.3). Thus VioPose directly optimizes the metrics on which it is evaluated, placing baselines at a disadvantage. The paper provides no ablation removing Lv/La from VioPose, nor does it add these losses to the baselines. The MPJAE gain (39.64%) and MPJVE gain (17.37%) are therefore confounded with the loss function, and the vibrato detection (which uses acceleration local minima) inherits this bias. Even VioPose w/o audio (MPJAE 1.25, MPJVE 1.78) already beats all baselines on these metrics while using the same additional losses, confirming that the loss—not the audio signal—is the likely driver of the 'subtle motion' improvements. Without a controlled comparison that equalizes the training objective, the claim 'outperforms SoTA' for fine-grained motion is not established.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VioPose, a multimodal 3D pose estimation network for violin performance that takes 2D keypoints and raw audio as input and outputs 3D pose, velocity, and acceleration. It introduces VioDat, a synchronized violin dataset with 12 players, four cameras, four microphones, and MoCap ground truth. The method uses a hierarchical architecture with cascade summation and bidirectional mixing, trained with pose, velocity, and acceleration losses. Experiments on VioDat show VioPose outperforms several visual-only SoTA baselines on MPJPE and especially on MPJVE/MPJAE, and an ablation demonstrates gains from the audio module. A downstream violin-performance analysis shows improvements in bowing direction, straight bow, violin hold, and vibrato detection.","tokens_in":17645,"tokens_out":6042,"duration_ms":59658,"significance":"If the results hold, the dataset alone is a significant contribution, filling a gap in calibrated audiovisual performance data. The hierarchical fusion and bidirectional mixing are plausible design choices, and the ablations are extensive, particularly the audio on/off comparison that isolates the audio contribution. The paper also promises code and dataset release, which supports reproducibility. However, the reported gains over SoTA on velocity and acceleration metrics are confounded with the training loss, and the test set is very small, so the magnitude and generalizability of the claimed improvements are not yet established. The central MPJPE improvement is plausible, but the fine-motion claims need a fairer evaluation protocol.","major_comments":[{"comment":"The large MPJVE and MPJAE improvements over SoTA in Table 2 are confounded: VioPose is trained with velocity and acceleration losses L_v and L_a, while the SoTA baselines are retrained with their original loss functions, which typically only supervise pose. Because VioPose directly optimizes the metrics on which it is evaluated, the claimed 17.37% and 39.64% improvements do not isolate the proposed architecture or the audio fusion. This is supported by Table 2, where VioPose w/o audio, trained with the same additional losses, already beats every baseline on MPJVE and MPJAE. Please retrain the baselines with the same velocity and acceleration losses, or remove those losses from VioPose, and report the comparison; without this, the fine-motion and vibrato claims are not established.","section":"§3.5 and §4.3, Eq. (8)"},{"comment":"The test set contains only three participants (an advanced male adult, a novice female teenager, and a novice male child), yet the paper reports point estimates without error bars, confidence intervals, or per-subject breakdowns. With n=3, the 5.4% MPJPE improvement and the MPJVE/MPJAE margins may not be statistically reliable. Please report per-subject results and uncertainty estimates, and ideally include more test subjects or cross-validation, to support the generalization claims made in the abstract and Section 4.3.","section":"§4.1"}],"minor_comments":[{"comment":"The sentence 'the w/o Cascade model shows the worst result in MPJPE' is inconsistent with the table: w/o Mixing has MPJPE 47.87, which is the worst, while w/o Cascade has MPJPE 44.21. Please correct the description or the table.","section":"§4.4, Table 5"},{"comment":"The definitions of L_v and L_a are incomplete: as written, they sum the dot products of normalized vectors and do not clearly show the max operation or the normalization denominator that would define the max-cosine similarity. Please clarify the formula and its relation to the cited reference.","section":"§3.5, Eq. (8)"},{"comment":"In the sentence about the bottleneck layer, 'block box in Fig 2' should read 'black box in Fig 2'.","section":"§3.2"},{"comment":"The qualitative trajectory plots are helpful, but they show only a few examples; adding quantitative trajectory errors per motion type (e.g., bowing vs. vibrato) would strengthen the claim that VioPose captures fine motions.","section":"§4.5 and Figures 4-5"}],"recommendation":"major_revision","confidential_remarks":"The dataset contribution is valuable and the ablation study is a strength. The main obstacle to acceptance is the evaluation protocol: the loss-function confound and the very small test set. If the authors can provide loss-matched baselines and uncertainty estimates, the paper could become a solid contribution to the audiovisual pose estimation literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful thing here is VioDat: a calibrated, synchronized violin dataset with 12 players, four camera views, and mocap ground truth. That's a real resource for pose estimation and music performance analysis. The architecture—cascaded dynamics hierarchy with audio treated as a prior, plus bidirectional mixing—is a sensible new combination, and the ablations are informative. The downstream analysis tasks (bowing, vibrato) are a nice way to show the point.\n\nThe soft spot is the evaluation. VioPose is trained with velocity and acceleration losses (Eq. 8) while the SoTA baselines are retrained with their original losses, so the large MPJVE and MPJAE wins (17% and 39%) are not a fair comparison. The paper's own ablation row, VioPose w/o audio, beats every baseline on those metrics while using the same extra losses, which tells you the loss is likely driving the improvement, not the audio. The MPJPE gain (43.60 vs 46.09) is more plausible but still comes from a model that also optimizes the other losses. To claim superior 4D pose, especially on subtle motion, the baselines need to be retrained with the same loss terms, or at least an ablation without Lv and La should be reported.\n\nOther issues: the test set has three participants, with no error bars, so the ranking is fragile. There is no comparison to existing audiovisual pose or music-motion methods (e.g., Audeo-style or Audio to Body Dynamics), so the contribution of the audio modality is not well contextualized. The \"Bayesian update\" language is a loose analogy to additive feature fusion; it's cosmetic rather than a derivation.\n\nThese are fixable. The dataset alone warrants a serious look, and the architecture is worth reporting. I'd send it to peer review with a request for a controlled comparison, error bars, and retrained baselines. The reader's conditional verdict is about right.\n\nRecommendation: engage with this paper; with revisions it can be a solid contribution for the pose and music analysis communities.","headline":"VioDat is a genuinely useful resource and the architecture is a sensible new combination, but the headline velocity/acceleration gains are confounded because VioPose is trained with extra losses that the baselines don't get.","tokens_in":18291,"tokens_out":2482,"would_cite":true,"duration_ms":24178,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VioPose estimates 4D violin-playing poses by using the music itself as a prior, beating visual-only methods on subtle motions such as vibrato.","keywords":["4D human pose estimation","audiovisual inference","violin performance analysis","hierarchical motion dynamics","Bayesian prior fusion","vibrato tracking","multimodal dataset","2D-to-3D lifting"],"falsifier":"Retrain or test VioPose on VioDat with the input audio shifted by a fixed delay of about 100 ms relative to the video. If MPJPE, MPJVE, and MPJAE stay close to the synchronized results, the causal-audio prior is not the source of the improvements; a substantial degradation would confirm that the audio is doing the claimed work.","tokens_in":17143,"feed_emoji":"🎻","tokens_out":6525,"duration_ms":59127,"temperature":0.7,"pith_summary":"VioPose sets out to show that the sound a violinist produces can be used as a reliable prior for estimating their 3D body motion over time, even when cameras miss fast or tiny movements. The paper introduces a hierarchical network that takes noisy 2D keypoints and raw audio and lifts them to a 4D pose (3D joint positions over time) with explicit velocity and acceleration estimates. It also contributes VioDat, a calibrated violin-playing dataset of 12 players with synchronized video, audio, and motion-capture ground truth. The result is a measured improvement over visual-only baselines on pose error and a larger improvement on the velocity and acceleration errors that capture subtle motion.","feed_headline":"Violin audio lifts 4D pose tracking past video-only models","feed_subtitle":"On a newly collected violin dataset, a causal audio prior beats the best visual-only method on pose, velocity, and acceleration error.","key_machinery":"The core mechanism is a three-layer hierarchy that treats audio as a prior for acceleration, cascades high-level dynamics down to lower layers through a summation that mimics a log-Bayesian update ($\\log p(x|y) \\approx \\log p(y|x) + \\log p(x)$), and then reconciles the estimated acceleration, velocity, and pose through a bidirectional mixing module that alternates integration and differentiation. The network consumes off-the-shelf 2D keypoints and a 35-dim audio feature vector, encodes both with transformer blocks, and is trained with a position loss plus max-cosine-similarity losses on velocity and acceleration, which forces the model to match motion dynamics rather than only average joint positions.","core_discovery":"The paper claims that estimating motion dynamics hierarchically, with acceleration as the highest-level signal and pose as the lowest, and feeding audio into the highest level as a causal prior, yields 4D pose sequences that are both more accurate and smoother than visual-only methods. On VioDat, VioPose reports MPJPE of 43.60 mm versus 46.09 mm for the best retrained visual-only baseline, MPJVE of 1.57 versus 1.90, and MPJAE of 1.02 versus 1.69. The largest joint-level gains are in the upper limbs, such as the left index and right wrist, and the method tracks vibrato perturbations of roughly 10 mm that other methods flatten into straight lines.","pith_inferences":["Beyond the paper, the causal-audio prior should transfer to other motion-produced sounds such as piano, drums, or speech articulation whenever the audio track can be isolated from background noise.","Beyond the paper, the hierarchy's success suggests a general recipe for lifting methods: predict acceleration and velocity before position, and use the highest-derivative signal as the prior, which could improve fine-grained movement estimation in domains like typing or surgical motion.","Beyond the paper, a direct way to test the claimed mechanism is to shift the input audio by a known delay, such as 100 ms, and measure how much the pose, velocity, and acceleration errors degrade; the paper does not report this experiment."],"forward_implications":["Monocular pose estimation for instrument playing can be improved by treating the instrument's sound as a causal signal rather than a merely correlated one.","Velocity and acceleration errors, not just joint positions, are the metrics that expose whether a model captures subtle motion such as vibrato.","The same causal-audio prior should transfer to other instruments or activities where sound is physically produced by the body motion being estimated.","A calibrated audiovisual dataset with motion-capture ground truth is necessary to fairly train and compare such methods, because existing music datasets lack full 3D kinematic ground truth."],"supporting_citations":[{"why":"Supplies the off-the-shelf 2D keypoint estimates that form the visual input of the model.","marker":"[47]"},{"why":"Establishes the temporal 2D-to-3D lifting approach that the pose module builds on and is compared against.","marker":"[54]"},{"why":"Is the strongest visual-only baseline on MPJPE in the benchmark, the main accuracy target to beat.","marker":"[58]"},{"why":"Is the strongest visual-only baseline on velocity and acceleration error, the key comparison for the smoothness claims.","marker":"[43]"},{"why":"Is a frequency-domain transformer baseline retrained on VioDat and a core competitor in the reported comparison.","marker":"[76]"},{"why":"Is a spatio-temporal transformer baseline retrained on VioDat and a core competitor in the reported comparison.","marker":"[73]"},{"why":"Is a multi-hypothesis transformer baseline used for comparison, especially on trajectory jitter and vibrato.","marker":"[44]"},{"why":"Supplies the audio feature set (envelope, MFCC, chroma, one-hot peaks, RMS) used to encode the sound input.","marker":"[41]"},{"why":"Provides the concise music representation that informs the audio feature design and sampling choices.","marker":"[48]"}],"fun_headline_variants":["Audio cues sharpen 4D violin pose tracking","VioPose: Sound guides 4D pose past video limits","Hierarchical audio-video model outperforms visual-only pose","Violin audio boosts 4D pose accuracy and smoothness","Causal audio prior refines violin motion capture"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"In violin playing, the audio is caused by the same body motion that the pose estimator is trying to recover, so the sound can be trusted as a prior; if the audio track is desynchronized, contains unrelated sounds, or the player is miming, the reported gains should disappear.","fun_headline_variants_meta":{"raw":{"variants":["Audio cues sharpen 4D violin pose tracking","VioPose: Sound guides 4D pose past video limits","Hierarchical audio-video model outperforms visual-only pose","Violin audio boosts 4D pose accuracy and smoothness","Causal audio prior refines violin motion capture"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000187,"raw_usage":{"total_tokens":1324,"prompt_tokens":937,"completion_tokens":387,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":306}},"tokens_in":553,"tokens_out":387,"duration_ms":4261,"temperature":1.0,"reasoning_tokens":306,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:07:19.015262+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain or test VioPose on VioDat with the input audio shifted by a fixed delay of about 100 ms relative to the video. If MPJPE, MPJVE, and MPJAE stay close to the synchronized results, the causal-audio prior is not the source of the improvements; a substantial degradation would confirm that the audio is doing the claimed work.","supporting_citations":[{"cited_title":"MediaPipe: A Framework for Building Perception Pipelines,","cited_arxiv_id":null,"evidence_quote":"Supplies the off-the-shelf 2D keypoint estimates that form the visual input of the model."},{"cited_title":"3D Human Pose Estimation in Video With Temporal Convo- lutions and Semi-Supervised Training,","cited_arxiv_id":null,"evidence_quote":"Establishes the temporal 2D-to-3D lifting approach that the pose module builds on and is compared against."},{"cited_title":"P-STMO: Pre-trained Spatial Temporal Many-to-One Model for 3D Human Pose Estimation,","cited_arxiv_id":null,"evidence_quote":"Is the strongest visual-only baseline on MPJPE in the benchmark, the main accuracy target to beat."},{"cited_title":"Ex- ploiting Temporal Contexts With Strided Transformer for 3D Human Pose Estimation,","cited_arxiv_id":null,"evidence_quote":"Is the strongest visual-only baseline on velocity and acceleration error, the key comparison for the smoothness claims."},{"cited_title":"Pose- FormerV2: Exploring Frequency Domain for Efficient and Robust 3D Human Pose Estimation,","cited_arxiv_id":null,"evidence_quote":"Is a frequency-domain transformer baseline retrained on VioDat and a core competitor in the reported comparison."},{"cited_title":"MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Video,","cited_arxiv_id":null,"evidence_quote":"Is a spatio-temporal transformer baseline retrained on VioDat and a core competitor in the reported comparison."},{"cited_title":"MHFormer: Multi-Hypothesis Transformer for 3D Hu- man Pose Estimation,","cited_arxiv_id":null,"evidence_quote":"Is a multi-hypothesis transformer baseline used for comparison, especially on trajectory jitter and vibrato."},{"cited_title":"AI Chore- ographer: Music Conditioned 3D Dance Generation With AIST++,","cited_arxiv_id":null,"evidence_quote":"Supplies the audio feature set (envelope, MFCC, chroma, one-hot peaks, RMS) used to encode the sound input."},{"cited_title":"M2C: Concise Music Repre- sentation for 3D Dance Generation,","cited_arxiv_id":null,"evidence_quote":"Provides the concise music representation that informs the audio feature design and sampling choices."}],"review_version":1}