{"id":"1523dc1e-102e-4363-b955-b8d16c4f672b","arxiv_id":"2412.05831","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A semi-supervised contrastive learning framework with an inference-time control dial improves and tunes video-to-music and music-to-video retrieval.","lead":"This paper introduces a system that learns to match video clips with music by combining two training signals: whether the audio and video come from the same clip, and whether they share a music genre label. A dial lets the user choose how much to rely on each signal when retrieving music for a video or a video for a music track.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Supervised signal may be audio-only: AudioSet genre labels are derived from audio content (Sec. III-A), so the SupCon branch (Eqs. 4-6) may learn audio-genre matching rather than visual-to-music alignment; no test shows visual information drives the supervised benefit.","rationale":"The reader's weakest assumption correctly identifies that AudioSet genre labels are audio-derived and that the paper lacks a label-visual correlation analysis. This is the most load-bearing concern because the entire contribution hinges on the supervised branch providing useful cross-modal supervision for visual-to-music alignment. If the supervision is audio-only, the method still works as an audio-genre-conditioned retrieval system, but the stated novelty and interpretation ('combines self-supervised and supervised objectives for learning an alignment between music and video') are not established. The concrete test of muted-video queries directly settles whether the supervised branch uses visual information. The paper has strengths: it includes a fully self-supervised baseline, a supervised baseline, and a semi-supervised baseline, and the controllability plots show monotonic trends. However, the lack of error bars and the selection of an optimal alpha on the validation set create uncertainty about the magnitude of the benefit, and the conclusion overstates the Table II results. The verdict remains CONDITIONAL pending the audio-removed experiment and uncertainty quantification.","tokens_in":8555,"tokens_out":6614,"duration_ms":70426,"concrete_test":"Run retrieval experiments with video queries whose audio is removed (e.g., replace all frames with their mean or zero out the audio track, keeping only CLIP visual features). If genre-supervised P@1 and self-supervised R@10 drop to near chance while audio-present queries perform as reported, the supervised branch relies on the query's audio rather than visual content, undermining the visual-to-music alignment claim. Additionally, report bootstrap confidence intervals or standard errors for the Table II metrics, especially the alpha=0.4 vs. alpha=0.0 self-supervised R@10 difference, to verify the claimed improvement is not within noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that combining self-supervised and supervised objectives yields a controllable and improved music-video alignment. The load-bearing premise is that the supervised branch actually supervises a visual-to-music correspondence. But Section III-A states that AudioSet annotations 'primarily focus on audio content and effectively ignore visual content,' and in Eq. 6 the positive set for an audio anchor includes any video whose genre label matches, which is the genre of that video's audio track. Thus the SupCon loss can be minimized by mapping all videos of a given genre to a single audio-genre centroid, without ever using visual content. The paper provides no evidence that the visual features of same-genre videos are more similar than those of different genres, nor that the supervised branch contributes to exact video-music matching beyond a genre prior. The observed improvement at optimal alpha (e.g., self-supervised R@10 10.42 vs. 8.90 for the self-supervised baseline, Table II) could be a regularization or genre-prior effect, not evidence of visual-audio semantic alignment. If the supervised branch is effectively audio-only, then the claimed 'video-to-music' genre-supervised retrieval is largely audio-to-audio retrieval with a video query that happens to contain the audio, and the conclusion that the model 'outperformed all baseline methods on self-supervised retrieval tasks' is also inconsistent with Table II (Wav2CLIP has higher R@1).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Control-MVR, a framework for video-to-music and music-to-video retrieval that combines a self-supervised cross-modal InfoNCE loss with a genre-supervised SupCon loss. Two parallel branches produce self-supervised and supervised embeddings, whose weighted combination is controlled at inference time by a scalar α. The authors evaluate on a subset of AudioSet music videos with 11 genre classes, compare against Wav2CLIP, AudioCLIP, MVPt, and several internal baselines, and report retrieval metrics for both self-supervised (exact video matching) and genre-supervised (genre matching) evaluation. They also demonstrate that varying α trades off between the two retrieval behaviors. The central claim is that this is the first framework to combine self-supervised and supervised objectives for music-video alignment, that the combination improves over self-supervised training alone, and that inference-time controllability is a useful practical feature.","tokens_in":8862,"tokens_out":2915,"duration_ms":30626,"significance":"If the claims hold, the paper makes a useful empirical contribution: it shows that genre labels can be combined with self-supervised audiovisual correspondence without sacrificing either objective, and it introduces a simple, interpretable control mechanism (α) for retrieval emphasis. The dual-branch design with separate q embeddings and a shared g network is a reasonable architecture. The evaluation covers two different retrieval tasks and an explicit controllability experiment, which is a strength. However, the significance is limited by the lack of statistical rigor (no error bars, one seed), by the fact that the supervised signal is derived from audio-centric genre labels, and by a conclusion that overstates the results. The paper is a credible empirical study but needs additional evidence to support its strongest claims.","major_comments":[{"comment":"The supervised branch may be learning an audio-to-audio genre mapping rather than a visual-to-music alignment. The paper states that AudioSet annotations 'primarily focus on audio content and effectively ignore visual content' (Section III-A), and the SupCon positives in Eq. (6) are defined by the genre label of each clip, which is the genre of the audio track. Because the video branch can be trained to map video features to audio-genre centroids without using any visual content, the observed benefit of the supervised branch (e.g., the peak at α=0.4 in Fig. 2) could come from a genre prior or from regularization, not from visual-audio semantic alignment. Please add a concrete test of whether genre labels correlate with visual content, such as genre classification from the video-only features, or an ablation that removes the video branch from the SupCon loss and measures the drop. Without such evidence, the claim that supervised labels improve video-to-music retrieval remains weak.","section":"Section III-A and Eq. (6)"},{"comment":"The conclusion states that Control-MVR 'outperformed all baseline methods on self-supervised retrieval tasks,' but Table II contradicts this: Wav2CLIP achieves higher R@1 than Control-MVR with optimal α for both Video→Music (2.08 vs. 1.65) and Music→Video (1.78 vs. 1.76). The results section itself acknowledges this for R@1, so the conclusion should be corrected to say that Control-MVR is competitive with Wav2CLIP on self-supervised retrieval, not strictly better. This is a factual inconsistency in a central claim and must be fixed.","section":"Table II and Section IV"},{"comment":"The experimental comparison lacks error bars, multiple seeds, and statistical significance tests. Many of the key differences are small (e.g., self-supervised R@10 of 10.42 vs. 9.84 for Wav2CLIP, and R@1 of 1.65 vs. 2.08), so it is unclear whether they are meaningful. Please report the mean and standard deviation over at least three training runs, or perform a significance test, for the main comparisons in Table II. This is essential for supporting the claims of 'noticeably outperforms' and 'SOTA or near-SOTA.'","section":"Section III-C and Table II"},{"comment":"The comparison to MVPt is weakened by the modification of the model: MVPt was retrained on 10 one-second segments instead of the original longer-form inputs, which may change its behavior. The paper discloses this, but the subsequent claim that Control-MVR outperforms MVPt is only about this modified variant, not the published MVPt. Please either justify that the modification is neutral for the comparison or soften the SOTA claim accordingly.","section":"Section III-C, Baseline Approaches"}],"minor_comments":[{"comment":"There is a typo in the second paragraph: 'allows the user to to dynamically prioritize' should read 'allows the user to dynamically prioritize.'","section":"Section I, Introduction"},{"comment":"Reference [5] is incomplete: 'A. Vaswani, Attention is all you need' should be 'A. Vaswani et al., Attention is all you need.'","section":"References"},{"comment":"The notation α is used for both the training-time combination weight and the inference-time control weight; the paper should clarify explicitly whether the same value is used in both phases. In the experiments, α is set to 0.5 during training and varied at inference, which is stated later, but clarifying it at the equation would avoid confusion.","section":"Section II-B, Eq. (2)"},{"comment":"The plots in Fig. 2 would benefit from showing individual data points or confidence intervals, since the lines alone do not convey the variability that the accompanying claims depend on.","section":"Figure 2"},{"comment":"The phrase 'all baselines except for Wav2CLIP' is imprecise because the baseline group includes internal baselines and MVPt; please specify exactly which baselines are included in the comparison for each claim.","section":"Section III-C, Results"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical contribution, but the central claim about the supervised branch improving video-to-music retrieval is not yet established because the labels are audio-derived. The missing variance statistics and the overstated conclusion in Section IV need to be addressed. This is within the scope of a major revision rather than a rejection, provided the authors can supply the requested analyses or substantially temper the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nShort version: this is a competent, incremental paper that combines two known contrastive losses—InfoNCE and SupCon—in a dual-branch architecture, and adds an inference-time alpha knob to blend self-supervised and genre-supervised embeddings for video-to-music retrieval. The alpha sweep is the most interesting part: it shows a genuine trade-off and even a small peak benefit from mixing the two. The writing is clear, the dataset is decently large (106k AudioSet clips), and they retrain MVPt on their data rather than just citing numbers.\n\nThe main thing to know: the stress-test note is right. AudioSet genre labels are audio-based, and the paper says so in Sec. III-A. So the SupCon branch is essentially teaching the visual (CLIP) features to predict the genre of the video's own audio track. That is a legitimate supervision signal for genre-conditional retrieval, but it is not evidence of visual-to-music 'semantic alignment' beyond a genre prior. The paper doesn't check whether same-genre videos are visually similar, or whether the supervised branch helps exact video-music matching beyond a genre smoothing/regularization effect. The observed R@10 gain from 8.90 to 10.42 over the self-supervised baseline is modest and could be a regularization artifact.\n\nOther soft spots: no error bars or multiple seeds, so the 1-2 point R@ differences are unquantified. Baselines are trained on different data, and Wav2CLIP still beats Control-MVR on self-supervised R@1 in video-to-music (2.08 vs 1.65). The paper's own text acknowledges this, so the claim in the conclusion that they 'outperformed all baseline methods' is overstated. Also, the 'first to combine self-supervised and supervised for music-video' claim is a narrow novelty; the components are all standard.\n\nThat said, the core result is probably correct: alpha gives you control, and mixing a bit of genre supervision doesn't hurt. The paper deserves a serious referee, mainly to push for error bars, an ablation that tests the visual-audio dependence, and more careful claims. I'd recommend acceptance after moderate revision, not a desk reject.","headline":"A competent, incremental combination of InfoNCE and SupCon with a useful inference-time control knob; the supervised branch likely learns audio-genre priors rather than visual-music alignment, so the interpretation needs tightening but the method deserves a serious referee.","tokens_in":9370,"tokens_out":3267,"would_cite":false,"duration_ms":31212,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training music-video retrieval with both self-supervised and genre-supervised contrastive losses yields a controllable joint embedding that beats single-objective baselines on both tasks.","keywords":["semi-supervised contrastive learning","cross-modal retrieval","video-to-music retrieval","music-video alignment","controllable retrieval","InfoNCE loss","SupCon loss","genre labels"],"falsifier":"Train the supervised branch on the same videos but replace the visual features with random noise or a constant frame; if genre-supervised retrieval precision remains essentially unchanged, then the genre-supervised branch is driven entirely by audio content and the video-to-music supervised result is not actually using the visuals. A complementary check is to measure how well the genre labels predict visual-only CLIP features on held-out data; near-chance prediction would indicate the labels carry no visual signal.","tokens_in":8364,"feed_emoji":"🎵","tokens_out":4950,"duration_ms":43259,"temperature":0.7,"pith_summary":"This paper proposes Control-MVR, a framework for retrieving a music clip that matches a given video, and vice versa. Its central claim is that training separate self-supervised and genre-supervised contrastive branches and combining their outputs with an inference-time weight α yields a single joint embedding that is strong at both exact music-video matching and genre-level matching, while letting the user dial between the two at retrieval time. The authors argue this is the first method to combine self-supervised and supervised objectives for music-video alignment, and they show that the combined model outperforms single-objective baselines on both evaluation types. If true, content creators could search music by the feel of a video or by a target genre with one adjustable control.","feed_headline":"One knob tunes video-to-music search between vibe and genre","feed_subtitle":"A semi-supervised contrastive model shifts retrieval from artistic vibe to genre match with a single weight.","key_machinery":"The load-bearing object is the α-weighted blend of two projected embeddings, each trained by a different contrastive loss. A shared $g$ network feeds two task-specific heads $h_{\\mathrm{ssl}}$ and $h_{\\mathrm{sup}}$, whose outputs $q_{\\mathrm{ssl}}$ and $q_{\\mathrm{sup}}$ are projected by $p_{\\mathrm{ssl}}$ and $p_{\\mathrm{sup}}$ and linearly combined with weight $\\alpha$ into the final embedding $z$. The self-supervised head is trained with cross-modal InfoNCE, which pulls together audio and video of the same music video; the supervised head is trained with cross-modal SupCon, which pulls together audio and video sharing a genre label. During training α is fixed at 0.5, and at inference it becomes a user-controlled dial; the experiments show that neither extreme is optimal for either retrieval task.","core_discovery":"The paper's central discovery is that self-supervised audiovisual correspondence and supervised genre information can be combined in one embedding space without one objective destroying the other. The model uses a dual-branch architecture: shared base networks produce representations, task-specific networks produce self-supervised q embeddings and supervised q embeddings, and projection layers map them to a shared output space as $z = (1-\\alpha) p_{\\mathrm{ssl}}(q_{\\mathrm{ssl}}) + \\alpha p_{\\mathrm{sup}}(q_{\\mathrm{sup}})$. Training optimizes cross-modal InfoNCE and cross-modal SupCon losses at both the task-specific and output levels. At inference time α can be swept from 0 to 1, giving a continuous trade-off: low α favors retrieval of the exact same music video, high α favors retrieval of clips with the same genre, and intermediate values peak on both tasks, indicating that the two signals complement each other.","pith_inferences":["Because AudioSet genre labels are derived from the audio track, the supervised branch may be effectively classifying music audio rather than aligning visuals; a visual-ablation experiment would settle whether the genre-supervised video-to-music gains are truly cross-modal.","The same α-blending recipe could be applied to other retrieval pairs, such as text-to-music or speech-to-music, or to balancing multiple supervised labels simultaneously, since the output space is just a convex combination of separately trained projections.","The fact that peak self-supervised recall occurs at α around 0.4 rather than 0 suggests genre information acts as a regularizer; a testable extension would be to anneal α during training to find a better operating point.","Because the base audio and video encoders are frozen, the method is cheap to train on large video corpora; scaling to diverse content beyond standard music videos could test whether the genre-supervised branch transfers across distributions."],"forward_implications":["With the same trained weights, a user can emphasize self-supervised artistic correspondence (low α) or genre match (high α) at query time, without retraining.","Adding genre supervision improves self-supervised retrieval over the self-supervised-only baseline, so labeled data and unlabeled audiovisual correspondence need not compete.","The framework is stated to generalize to any music annotation, such as emotion or instrument labels, by swapping the supervised labels.","The peak performance at intermediate α on both evaluations suggests a mixture of self-supervised and supervised signals is better than either extreme, giving a principled reason to set α between 0 and 1 rather than at an endpoint."],"supporting_citations":[{"why":"Supplies the cross-modal InfoNCE loss used in the self-supervised branch.","marker":"[16]"},{"why":"Supplies the SupCon supervised contrastive loss used in the supervised branch.","marker":"[17]"},{"why":"Provides the AudioSet music-video dataset and the genre labels used for supervision and evaluation.","marker":"[18]"},{"why":"Prior artistic-correspondence method whose retrieval evaluation protocol and dataset setup are followed.","marker":"[4]"},{"why":"Provides the frozen MERT audio encoder that produces the audio features.","marker":"[14]"},{"why":"Provides the frozen CLIP vision encoder that produces the video frame features.","marker":"[15]"},{"why":"Prior cross-modal SupCon approach that the paper adapts for the supervised music-video loss.","marker":"[11]"},{"why":"Wav2CLIP serves as a strong baseline trained on a larger dataset, used for comparison.","marker":"[21]"}],"fun_headline_variants":["One dial flips video-music retrieval from vibe to genre","Tunable weight balances vibe and genre in video-music search","Blend self-supervised and genre cues for video-music retrieval","Continuous control from vibe to genre in video-music retrieval","A single slider tunes video-to-music match between vibe and genre"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that AudioSet's music genre labels, which the paper itself says are assigned primarily from the audio track and effectively ignore visual content, provide a useful supervised signal for aligning video with music; if those labels carry no visual information, the supervised branch may be learning an audio-to-audio genre mapping and the claimed benefit for video-to-music retrieval under supervised control would not be established.","fun_headline_variants_meta":{"raw":{"variants":["One dial flips video-music retrieval from vibe to genre","Tunable weight balances vibe and genre in video-music search","Blend self-supervised and genre cues for video-music retrieval","Continuous control from vibe to genre in video-music retrieval","A single slider tunes video-to-music match between vibe and genre"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000514,"raw_usage":{"total_tokens":2497,"prompt_tokens":948,"completion_tokens":1549,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":1461}},"tokens_in":564,"tokens_out":1549,"duration_ms":13611,"temperature":1.0,"reasoning_tokens":1461,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:17:56.426704+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the supervised branch on the same videos but replace the visual features with random noise or a constant frame; if genre-supervised retrieval precision remains essentially unchanged, then the genre-supervised branch is driven entirely by audio content and the video-to-music supervised result is not actually using the visuals. A complementary check is to measure how well the genre labels predict visual-only CLIP features on held-out data; near-chance prediction would indicate the labels carry no visual signal.","supporting_citations":[{"cited_title":"Supervised contrastive learn- ing,","cited_arxiv_id":null,"evidence_quote":"Supplies the SupCon supervised contrastive loss used in the supervised branch."},{"cited_title":"Audio set: An ontology and human-labeled dataset for audio events,","cited_arxiv_id":null,"evidence_quote":"Provides the AudioSet music-video dataset and the genre labels used for supervision and evaluation."},{"cited_title":"It’s time for artistic correspondence in music and video,","cited_arxiv_id":null,"evidence_quote":"Prior artistic-correspondence method whose retrieval evaluation protocol and dataset setup are followed."},{"cited_title":"Mert: Acoustic music understanding model with large-scale self-supervised training,","cited_arxiv_id":null,"evidence_quote":"Provides the frozen MERT audio encoder that produces the audio features."},{"cited_title":"Learning transferable visual models from natural language supervision,","cited_arxiv_id":null,"evidence_quote":"Provides the frozen CLIP vision encoder that produces the video frame features."},{"cited_title":"Emotion-aligned contrastive learning between images and music,","cited_arxiv_id":null,"evidence_quote":"Prior cross-modal SupCon approach that the paper adapts for the supervised music-video loss."},{"cited_title":"Wav2clip: Learning robust audio representations from clip,","cited_arxiv_id":null,"evidence_quote":"Wav2CLIP serves as a strong baseline trained on a larger dataset, used for comparison."}],"review_version":1}