{"id":"6c5ffc17-d4f8-4a2d-9e0e-4a80c4655e4a","arxiv_id":"2502.06710","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Amuse fuses audio, video, and language early and adds rhythm and instrument-source supervision over time to reach state-of-the-art accuracy on Music AVQA and Music AVQA-v2.","lead":"This paper presents Amuse, a music-specific audio-visual question answering system that lets audio, video, and text attend to each other from early in the network and that adds rhythm and instrument-source annotations. Amuse reports state-of-the-art accuracy on the Music AVQA and Music AVQA-v2 benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pretraining of the frozen rhythm/source encoders may use test-set audio and visual features, potentially inflating the SOTA comparison.","rationale":"The reader's weakest assumption concerns the semantic validity of the automatic rhythm/source annotations. That is a plausible concern, but the single most load-bearing issue for the empirical claim is whether the two-stage training protocol leaks information from the test set. Even if the annotations are perfectly meaningful, using them to pretrain on test videos would give Amuse an unfair advantage over baselines. The paper's §4.1 description does not rule this out, and the large performance gaps make the issue consequential. This is a concrete, checkable methodological concern rather than a critique of the architecture or the annotation quality. The appropriate response is not rejection but an explicit condition: the authors must clarify the pretraining split and, if test videos were included, rerun with a train-only split or compare against baselines that receive the same unlabeled test-set pretraining. Since the verdict was already conditional, my recommendation is that it remain conditional, now with a more specific and testable condition attached.","tokens_in":12537,"tokens_out":4613,"duration_ms":46557,"concrete_test":"Inspect the released code and data splits at https://github.com/xid32/Amuse to determine whether the rhythm/source pretraining in §4.1 includes videos from the test split. If it does, rerun the full Amuse pipeline with the rhythm and source encoders pretrained only on training-split videos and recompute the v1 and v2 test accuracies, comparing against Tables 2 and 3. If the overall accuracy drops by more than the reported margin over the previous best, the SOTA claim is partly driven by test-set pretraining rather than by the proposed musical representations.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Amuse achieves state-of-the-art accuracy rests on a fair comparison with baselines. Amuse has two training stages: first, the rhythm and source predictors are pretrained with MSE loss against automatically generated annotations (Eqs. 4-6 in §3.2); second, these encoders are frozen and used as a Universal Encoder during finetuning (§4.1). The paper states that pretraining 'exclusively utilizes the rhythm and music source annotations,' but it never specifies whether the pretraining videos are restricted to the training split. Because the annotations are produced automatically from every clip's audio with no label supervision, a natural implementation would generate them for the full dataset and pretrain on all videos, including test videos. If that occurred, the frozen encoders would have seen the test videos' audio and visual frames before any supervised finetuning, giving Amuse an advantage that the compared baselines, trained only on labeled train splits, do not have. This concern is independent of annotation quality: even perfect rhythm/source labels would not eliminate it. The reported gains, e.g., +8.07 overall on Music AVQA v1 and +9.72 on v2 over LAST-Att, are large enough that a transductive pretraining advantage could materially contribute. The lack of reported error bars makes the effect harder to bound. If a train-only pretraining rerun reproduces the results, the claim stands; if it drops substantially, the comparison is unfair and the headline SOTA claim is weakened.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Amuse, a framework for audio-visual question answering (AVQA) on music performance videos. Amuse combines (i) three interactive transformers for vision, audio, and language with cross-modal adapters for early fusion, (ii) rhythm and music-source encoders pretrained with automatically generated temporal labels and then frozen, and (iii) a YOLOv8-based musical region-of-interest highway. The authors release rhythm/source annotations for the Music AVQA v1 and v2 datasets and report state-of-the-art overall accuracies of 83.52% and 85.16%, with ablations showing the contribution of each component.","tokens_in":12780,"tokens_out":5017,"duration_ms":46896,"significance":"If the reported results hold, Amuse would be a meaningful step for music-specific AVQA: the early-interaction encoder design is plausible, the temporal alignment of rhythm and source features is a sensible inductive bias for music, and the released annotations could support further research. The paper includes ablations for each proposed module and makes code available. The main significance is currently qualified by two concerns: the pretraining stage for the frozen encoders is not explicitly restricted to the training split, and all central numerical comparisons are reported without variance estimates or significance tests. These issues directly affect the state-of-the-art claim, so the significance can only be assessed after they are resolved.","major_comments":[{"comment":"The pretraining stage for the rhythm and source encoders is not stated to be restricted to the training split. Section 4.1 says only that pretraining 'exclusively utilizes the rhythm and music source annotations,' and Section 3.2 describes an annotation pipeline that is fully automatic and applicable to every clip in the dataset. If the frozen encoders were pretrained on audio and visual features from test videos, the comparison against baselines trained only on labeled training data would be transductive and potentially inflated. Because the reported gains over prior work (e.g., +2.33 overall on v1, +9.72 on v2) are large enough that this effect could matter, the authors must specify which split(s) were used for pretraining and, if the full dataset was used, rerun the pretraining on the training split only and report the resulting test accuracies.","section":"§4.1, §3.2"},{"comment":"All Amuse numbers are described as averages over three runs, but no standard deviations, confidence intervals, or significance tests are reported, and baseline numbers appear to be taken from prior papers without associated variance. Several margins are small: on Music AVQA v1, the overall gap over LSTTA is 2.33 points and Amuse is actually below LAST-Att on audio counting (84.61 vs. 85.71). Without error bars or a significance test, the claim that Amuse 'demonstrates impressive performance across all categories' is not statistically supported. Please report per-run results and variance for Amuse and, where possible, for rerun baselines, or at least state clearly which comparisons are not statistically distinguishable.","section":"§4.2, Tables 2 and 3"},{"comment":"The automatically generated rhythm and source annotations are not validated. The 25% BPM threshold and the 6-second segment length are chosen by hand, and the Universal Source Separation outputs are used without reporting agreement with human judgment or with the QA question content. Since these labels are the supervision signal for the pretrained encoders and the ablations attribute substantial gains to rhythm and source modules (Table 4), noisy or misaligned labels could either overstate or understate the contribution of these components. The authors should provide annotation-quality evidence, for example agreement with a small human-annotated subset or a comparison of pretraining on predicted vs. gold labels where feasible.","section":"§3.2, Eqs. (4)–(6)"}],"minor_comments":[{"comment":"The text says Amuse shows 'impressive performance across all categories,' but Table 2 shows LAST-Att is better on audio counting on v1 (85.71 vs. 84.61) and several baseline entries for average columns are missing. Please qualify the claim as 'overall average' and fill or footnote the missing entries.","section":"§4.2, Table 2"},{"comment":"The sentence 'its final feature mappings exhibit approximate linearity' is not supported by the attention-importance analysis presented in that section; either provide a derivation or remove the claim.","section":"§4.4"},{"comment":"The source annotation procedure says 'Counts track the occurrences' without specifying which instrument classes are tracked, how counts are aggregated into labels, or what prediction target is used for the source predictor. Please specify the exact vocabulary and labeling format, since the released annotations are a stated contribution.","section":"§3.2"},{"comment":"The importance scores in Figure 5 lack details on how attention outputs are aggregated and normalized across layers and runs. Please state the procedure and, ideally, report variance over the three runs.","section":"Figure 5"},{"comment":"There are several wording and formatting issues: 'Music A VQA' is inconsistently spaced, 'convolutedly correlate' in Section 3.2 is awkward, and 'approximately linearity' in Section 4.4 should be 'approximately linear.' Please copyedit.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The most important concern is the unspecified data split for the pretraining stage; if the authors confirm that pretraining used only the training split and supply variance information, this would likely be a publishable contribution. The current manuscript, however, does not rule out a transductive effect that could materially change the state-of-the-art comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest, useful read. The paper proposes Amuse, a music-specific AVQA architecture with three ideas that genuinely go beyond the cited AVQA baselines: early cross-modal interaction through adapters between Swin-V2, HTS-AT, and the language transformer; auxiliary rhythm and source labels derived automatically from BPM tracking and universal source separation, used to pretrain temporal encoders; and YOLOv8-based RoI features for instruments and performers. The ablation study shows each module contributes, and the gains on Music AVQA v1 and v2 are large. The writing is clear and the architecture is reproducible from the description. That is real value.\n\nSoft spots, in order of importance. First, the pretraining protocol for the rhythm/source encoders is underspecified. The paper says this stage 'exclusively utilizes the rhythm and music source annotations' but never states whether pretraining videos are restricted to the training split. Since these annotations are generated automatically for every clip without label supervision, the natural implementation would run the BPM tracker and source separator on the entire dataset and pretrain on all videos, including the test split. If that happened, the frozen encoders would have seen test-video audio and visual features before supervised finetuning, giving Amuse an advantage that the train-only baselines do not have. The reported gains are large enough that this could materially contribute. The authors need to state the split, and ideally re-run with train-only pretraining. This is a load-bearing ambiguity, not a minor detail.\n\nSecond, there are no error bars or significance tests on the three-run averages, and several baseline numbers are taken from prior papers without re-running. That is common in this literature but weakens confidence in the precise SOTA margins. Third, the automatic rhythm and source labels are not validated against human perception or QA content. The 25% BPM threshold and 6-second segments are hand choices. The ablation shows these modules help, but if the labels are noisy, the mechanism could be different than claimed.\n\nThe central idea is coherent and the paper is worth engaging with. But the pretraining split question must be resolved before the SOTA claim can be accepted. My recommendation: send to peer review, but with the explicit requirement that the authors clarify the data split and either report train-only pretraining results or release the exact configuration. If they cannot show the same gains without transductive pretraining, the headline claim collapses.","headline":"A well-engineered music AVQA model with a genuine architectural contribution, but an underspecified pretraining split could undermine the SOTA comparison and must be clarified.","tokens_in":13294,"tokens_out":3281,"would_cite":false,"duration_ms":28201,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Amuse, built around early cross-modal interaction and temporally aligned rhythm and source encoders, achieves the highest reported accuracy on both Music AVQA benchmarks.","keywords":["audio-visual question answering","music performance understanding","multimodal early fusion","cross-modal adapters","rhythm annotation","music source separation","temporal alignment","YOLO regions of interest"],"falsifier":"Shuffle or randomize the rhythm and source labels while keeping the model architecture and training protocol identical; if Music AVQA-v2 accuracy does not fall materially below the reported 85.16%, the claimed contribution of temporal musical annotation is not real. A second check is to have human annotators mark rhythm changes and instrument presence on a subset and measure agreement with the automatic labels.","tokens_in":12348,"feed_emoji":"🎵","tokens_out":4506,"duration_ms":37173,"temperature":0.7,"pith_summary":"The paper proposes Amuse, a system for answering questions about music performance videos, and claims it outperforms prior audio-visual QA methods on both Music AVQA and Music AVQA-v2, reaching 83.52% and 85.16% overall accuracy. The motivating problem is that music performances have dense, continuous audio and tightly coupled sound and image, which general-purpose audio-visual QA models handle poorly. Amuse's answer is threefold: let vision, audio, and question tokens interact from early layers through cross-modal adapters; make musical characteristics explicit by annotating rhythm and instrument sources along time and training predictors on them; and add a lightweight detector's region features for instruments and performers as a shortcut. A sympathetic reader would take the paper's central claim to be that these three music-specific additions, not a larger backbone, are what lift accuracy.","feed_headline":"Amuse answers music questions with 83.5% and 85.2% accuracy","feed_subtitle":"Early fusion, rhythm, and source timing lift audio-visual QA on both Music AVQA benchmarks.","key_machinery":"The load-bearing machinery is a multimodal interactive encoder: three transformers (Swin-V2 for video, HTS-AT for audio, and a language transformer for questions) connected by cross-modal adapters that compute attention between tokens of different modalities and inject the attended outputs into each other's next blocks. Around this backbone, Amuse adds per-modality rhythm and source encoders trained with MSE loss on automatically generated labels (beat-tracking BPM changes over six-second segments at a 25% threshold, and instrument-presence counts from Universal Source Separation), frozen after pretraining. A YOLOv8 detector supplies musical regions of interest as a local visual shortcut. The adapters enable early fusion, the temporal encoders tie musical characteristics to the time axis, and the RoI highway gives local visual evidence for visual and counting questions.","core_discovery":"On the paper's own terms, the discovery is that a QA model built specifically for music performance can beat general audio-visual QA baselines by treating rhythm, instrument source, and temporal alignment as first-class supervision signals. Concretely, Amuse combines a multimodal interactive encoder with a rhythm encoder, a music source encoder, and a YOLOv8-based musical regions-of-interest highway. The experiments report the highest average accuracy on both Music AVQA test sets, and the ablations attribute most of the gain to the multimodal interactive encoder (removing it drops overall accuracy by 28.79 percentage points), with smaller but consistent contributions from the rhythm, source, and RoI modules.","pith_inferences":["Editorial inference: if the temporal annotations are indeed what drive the gain, the same recipe should transfer to other dense-audio domains such as speech, sports, or ambient music by swapping the source-separation categories; the paper does not test this.","Editorial inference: the ablation suggests the audio branch of the source encoder matters most for temporal and audio-related questions, so a natural next experiment is to feed the model pretrained separated instrument stems rather than count labels.","Editorial inference: the 25% BPM-change threshold and six-second window are hand-picked; sweeping these hyperparameters against human rhythm judgments would reveal how much of the reported gain depends on label quality."],"forward_implications":["Audio-visual QA in music settings can be improved without larger models, by fusing modalities early and adding temporally aligned musical features.","Counting questions benefit substantially from the rhythm and source encoders, with ablation drops of up to 5.55% on audio-visual counting.","Visual-region features from a light detector are what drive visual and audio-visual existential and counting accuracy, with a 13.40% drop when removed.","The released rhythm and source annotations can serve as supervision for future music AVQA models and may be reused as pretraining signals."],"supporting_citations":[{"why":"Introduces the Music AVQA benchmark and the AVST baseline that Amuse must beat; supplies the v1 test set and QA categories.","marker":"Li et al. (2022)"},{"why":"Introduces the balanced Music AVQA-v2 dataset and the LAST-Att baseline; supplies the v2 test set and the main comparison competitor.","marker":"Liu et al. (2024)"},{"why":"Provides the Swin-V2 Transformer used as the visual backbone of Amuse and its ablations.","marker":"Liu et al. (2022)"},{"why":"Provides the HTS-Audio Transformer used as the audio backbone of Amuse.","marker":"Chen et al. (2022)"},{"why":"Provides Universal Source Separation, used to annotate instrument sources in each audio segment.","marker":"Kong et al. (2023)"},{"why":"Provides YOLOv8, used to extract musical regions of interest for the RoI highway.","marker":"Ultralytics (2023)"},{"why":"Supplies the attention mechanism and transformer architecture underlying the cross-modal adapters and language encoder.","marker":"Vaswani et al. (2017)"},{"why":"Provides the LSTTA baseline, which is the strongest prior competitor on several audio-visual question categories.","marker":"Liu et al. (2023)"}],"fun_headline_variants":["Amuse tunes music QA with rhythm and source cues","Rhythm-aware audio-visual model answers music performance questions","Multimodal rhythm and source cues sharpen music QA","Early fusion and rhythm boost music AVQA results","Audio-visual rhythm cues improve music performance QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the automatically generated rhythm and instrument-source labels being true enough to the music: if the 25% BPM-change threshold and the source-separation counts do not match what humans or the questions actually care about, the temporal encoders are supervising the model with noise.","fun_headline_variants_meta":{"raw":{"variants":["Amuse tunes music QA with rhythm and source cues","Rhythm-aware audio-visual model answers music performance questions","Multimodal rhythm and source cues sharpen music QA","Early fusion and rhythm boost music AVQA results","Audio-visual rhythm cues improve music performance QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00074,"raw_usage":{"total_tokens":3263,"prompt_tokens":865,"completion_tokens":2398,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":2323}},"tokens_in":481,"tokens_out":2398,"duration_ms":15812,"temperature":1.0,"reasoning_tokens":2323,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T14:34:24.790420+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Shuffle or randomize the rhythm and source labels while keeping the model architecture and training protocol identical; if Music AVQA-v2 accuracy does not fall materially below the reported 85.16%, the claimed contribution of temporal musical annotation is not real. A second check is to have human annotators mark rhythm changes and instrument presence on a subset and measure agreement with the automatic labels.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides YOLOv8, used to extract musical regions of interest for the RoI highway."}],"review_version":1}