Pith. sign in

REVIEW 4 major objections 5 minor 24 references

Semi-Supervised Contrastive Learning for Controllable Video-to-Music Retrieval

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Training music-video retrieval with both self-supervised and genre-supervised contrastive losses yields a controllable joint embedding that beats single-objective baselines on both tasks.

desk verdict A competent, incremental combination of InfoNCE and SupCon with a useful inference-time control knob; the supervised branch likely learns audio-genre priors rather than visual-music alignment, so the interpretation needs tightening but the method deserves a serious referee. read the letter →

arxiv 2412.05831 v2 pith:WSCCH3MG submitted 2024-12-08 cs.MM cs.SDeess.AS

Shanti Stewart , Gouthaman KV , Lie Lu , Andrea Fanelli This is my paper · ORCID
classification cs.MMcs.SDeess.AS
keywords semi-supervisedcontrastivelearningcross-modalretrievalvideo-to-musicmusic-videoalignmentcontrollableInfoNCElossSupCongenrelabels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes Control-MVR, a framework for retrieving a music clip that matches a given video, and vice versa. Its central claim is that training separate self-supervised and genre-supervised contrastive branches and combining their outputs with an inference-time weight α yields a single joint embedding that is strong at both exact music-video matching and genre-level matching, while letting the user dial between the two at retrieval time. The authors argue this is the first method to combine self-supervised and supervised objectives for music-video alignment, and they show that the combined model outperforms single-objective baselines on both evaluation types. If true, content creators could search music by the feel of a video or by a target genre with one adjustable control.

What carries the argument

The load-bearing object is the α-weighted blend of two projected embeddings, each trained by a different contrastive loss. A shared $g$ network feeds two task-specific heads $h_{\mathrm{ssl}}$ and $h_{\mathrm{sup}}$, whose outputs $q_{\mathrm{ssl}}$ and $q_{\mathrm{sup}}$ are projected by $p_{\mathrm{ssl}}$ and $p_{\mathrm{sup}}$ and linearly combined with weight $\alpha$ into the final embedding $z$. The self-supervised head is trained with cross-modal InfoNCE, which pulls together audio and video of the same music video; the supervised head is trained with cross-modal SupCon, which pulls together audio and video sharing a genre label. During training α is fixed at 0.5, and at inference it becomes a user-controlled dial; the experiments show that neither extreme is optimal for either retrieval task.

What would settle it

Train the supervised branch on the same videos but replace the visual features with random noise or a constant frame; if genre-supervised retrieval precision remains essentially unchanged, then the genre-supervised branch is driven entirely by audio content and the video-to-music supervised result is not actually using the visuals. A complementary check is to measure how well the genre labels predict visual-only CLIP features on held-out data; near-chance prediction would indicate the labels carry no visual signal.

Watch

Extended reading notes

Core claim

The paper's central discovery is that self-supervised audiovisual correspondence and supervised genre information can be combined in one embedding space without one objective destroying the other. The model uses a dual-branch architecture: shared base networks produce representations, task-specific networks produce self-supervised q embeddings and supervised q embeddings, and projection layers map them to a shared output space as $z = (1-\alpha) p_{\mathrm{ssl}}(q_{\mathrm{ssl}}) + \alpha p_{\mathrm{sup}}(q_{\mathrm{sup}})$. Training optimizes cross-modal InfoNCE and cross-modal SupCon losses at both the task-specific and output levels. At inference time α can be swept from 0 to 1, giving a continuous trade-off: low α favors retrieval of the exact same music video, high α favors retrieval of clips with the same genre, and intermediate values peak on both tasks, indicating that the two signals complement each other.

Load-bearing premise

The load-bearing premise is that AudioSet's music genre labels, which the paper itself says are assigned primarily from the audio track and effectively ignore visual content, provide a useful supervised signal for aligning video with music; if those labels carry no visual information, the supervised branch may be learning an audio-to-audio genre mapping and the claimed benefit for video-to-music retrieval under supervised control would not be established.

Editorial extensions

If this is right

  • With the same trained weights, a user can emphasize self-supervised artistic correspondence (low α) or genre match (high α) at query time, without retraining.
  • Adding genre supervision improves self-supervised retrieval over the self-supervised-only baseline, so labeled data and unlabeled audiovisual correspondence need not compete.
  • The framework is stated to generalize to any music annotation, such as emotion or instrument labels, by swapping the supervised labels.
  • The peak performance at intermediate α on both evaluations suggests a mixture of self-supervised and supervised signals is better than either extreme, giving a principled reason to set α between 0 and 1 rather than at an endpoint.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because AudioSet genre labels are derived from the audio track, the supervised branch may be effectively classifying music audio rather than aligning visuals; a visual-ablation experiment would settle whether the genre-supervised video-to-music gains are truly cross-modal.
  • The same α-blending recipe could be applied to other retrieval pairs, such as text-to-music or speech-to-music, or to balancing multiple supervised labels simultaneously, since the output space is just a convex combination of separately trained projections.
  • The fact that peak self-supervised recall occurs at α around 0.4 rather than 0 suggests genre information acts as a regularizer; a testable extension would be to anneal α during training to find a better operating point.
  • Because the base audio and video encoders are frozen, the method is cheap to train on large video corpora; scaling to diverse content beyond standard music videos could test whether the genre-supervised branch transfers across distributions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Control-MVR, a framework for video-to-music and music-to-video retrieval that combines a self-supervised cross-modal InfoNCE loss with a genre-supervised SupCon loss. Two parallel branches produce self-supervised and supervised embeddings, whose weighted combination is controlled at inference time by a scalar α. The authors evaluate on a subset of AudioSet music videos with 11 genre classes, compare against Wav2CLIP, AudioCLIP, MVPt, and several internal baselines, and report retrieval metrics for both self-supervised (exact video matching) and genre-supervised (genre matching) evaluation. They also demonstrate that varying α trades off between the two retrieval behaviors. The central claim is that this is the first framework to combine self-supervised and supervised objectives for music-video alignment, that the combination improves over self-supervised training alone, and that inference-time controllability is a useful practical feature.

Significance. If the claims hold, the paper makes a useful empirical contribution: it shows that genre labels can be combined with self-supervised audiovisual correspondence without sacrificing either objective, and it introduces a simple, interpretable control mechanism (α) for retrieval emphasis. The dual-branch design with separate q embeddings and a shared g network is a reasonable architecture. The evaluation covers two different retrieval tasks and an explicit controllability experiment, which is a strength. However, the significance is limited by the lack of statistical rigor (no error bars, one seed), by the fact that the supervised signal is derived from audio-centric genre labels, and by a conclusion that overstates the results. The paper is a credible empirical study but needs additional evidence to support its strongest claims.

major comments (4)
  1. [Section III-A and Eq. (6)] The supervised branch may be learning an audio-to-audio genre mapping rather than a visual-to-music alignment. The paper states that AudioSet annotations 'primarily focus on audio content and effectively ignore visual content' (Section III-A), and the SupCon positives in Eq. (6) are defined by the genre label of each clip, which is the genre of the audio track. Because the video branch can be trained to map video features to audio-genre centroids without using any visual content, the observed benefit of the supervised branch (e.g., the peak at α=0.4 in Fig. 2) could come from a genre prior or from regularization, not from visual-audio semantic alignment. Please add a concrete test of whether genre labels correlate with visual content, such as genre classification from the video-only features, or an ablation that removes the video branch from the SupCon loss and measures the drop. Without such evidence, the claim that supervised labels improve video-to-music retrieval remains weak.
  2. [Table II and Section IV] The conclusion states that Control-MVR 'outperformed all baseline methods on self-supervised retrieval tasks,' but Table II contradicts this: Wav2CLIP achieves higher R@1 than Control-MVR with optimal α for both Video→Music (2.08 vs. 1.65) and Music→Video (1.78 vs. 1.76). The results section itself acknowledges this for R@1, so the conclusion should be corrected to say that Control-MVR is competitive with Wav2CLIP on self-supervised retrieval, not strictly better. This is a factual inconsistency in a central claim and must be fixed.
  3. [Section III-C and Table II] The experimental comparison lacks error bars, multiple seeds, and statistical significance tests. Many of the key differences are small (e.g., self-supervised R@10 of 10.42 vs. 9.84 for Wav2CLIP, and R@1 of 1.65 vs. 2.08), so it is unclear whether they are meaningful. Please report the mean and standard deviation over at least three training runs, or perform a significance test, for the main comparisons in Table II. This is essential for supporting the claims of 'noticeably outperforms' and 'SOTA or near-SOTA.'
  4. [Section III-C, Baseline Approaches] The comparison to MVPt is weakened by the modification of the model: MVPt was retrained on 10 one-second segments instead of the original longer-form inputs, which may change its behavior. The paper discloses this, but the subsequent claim that Control-MVR outperforms MVPt is only about this modified variant, not the published MVPt. Please either justify that the modification is neutral for the comparison or soften the SOTA claim accordingly.
minor comments (5)
  1. [Section I, Introduction] There is a typo in the second paragraph: 'allows the user to to dynamically prioritize' should read 'allows the user to dynamically prioritize.'
  2. [References] Reference [5] is incomplete: 'A. Vaswani, Attention is all you need' should be 'A. Vaswani et al., Attention is all you need.'
  3. [Section II-B, Eq. (2)] The notation α is used for both the training-time combination weight and the inference-time control weight; the paper should clarify explicitly whether the same value is used in both phases. In the experiments, α is set to 0.5 during training and varied at inference, which is stated later, but clarifying it at the equation would avoid confusion.
  4. [Figure 2] The plots in Fig. 2 would benefit from showing individual data points or confidence intervals, since the lines alone do not convey the variability that the accompanying claims depend on.
  5. [Section III-C, Results] The phrase 'all baselines except for Wav2CLIP' is imprecise because the baseline group includes internal baselines and MVPt; please specify exactly which baselines are included in the comparison for each claim.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; the paper's claims rest on external benchmarks and disclosed validation-set selection, with only a minor non-load-bearing self-citation.

full rationale

Control-MVR is an empirical method paper without a fitted theoretical derivation. Its central result is that combining a self-supervised cross-modal InfoNCE loss with a supervised SupCon loss, and blending the two branches at inference via alpha, improves or controls video-to-music retrieval. The potentially suspicious elements are: (1) the supervised positive set in Eq. 6 is defined by audio genre labels, and genre-supervised retrieval is also evaluated by matching those labels; and (2) reference [11] is a self-citation used for the cross-modal SupCon formulation. Neither reduces the paper's central claim to its own inputs by construction. The genre-label positive set and the genre-supervised retrieval metric are both defined by the same external AudioSet annotations; the labels are not generated by the model, so this is a task definition rather than a fitted-parameter-renamed-as-prediction. The reported 'optimal alpha' is explicitly selected on the validation set from a finite grid, which is standard hyperparameter selection and is disclosed, not a hidden fit. The self-citation to [11] only points to a prior application of SupCon to an image-music setting; the dual-branch architecture, the combination of InfoNCE and SupCon objectives, and the inference-time alpha blending are evaluated against external baselines (Wav2CLIP, AudioCLIP, MVPt) and internally trained self-supervised, supervised, and semi-supervised baselines. The AudioSet annotation limitation stated in Section III-A, that labels primarily focus on audio content and effectively ignore visual content, is a substantive validity concern for the video-to-music claim, but it is a correctness/threats-to-validity issue, not circularity. Overall, no load-bearing derivation step reduces by definition or through a self-citation chain to its own output.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The framework rests entirely on established contrastive losses and frozen pretrained encoders; there are no new theoretical axioms. The two domain assumptions that matter are that co-occurring video-audio pairs form positive pairs for self-supervision and that AudioSet genre labels, which the authors note are audio-centric, are a useful supervised signal for visual-audio alignment. All remaining choices are hyperparameters, with temperature and genre taxonomy the most consequential.

free parameters (5)
  • contrastive temperature tau = 0.1
    Reported as optimal for all contrastive losses, but no sweep or sensitivity analysis is shown. Retrieval metrics depend on this value.
  • training-time combination weight alpha = 0.5
    Chosen to equally balance the self-supervised and supervised branches during training. The paper does not compare alternative training-time alpha values.
  • genre taxonomy = 11 merged classes from AudioSet Music Genre
    The hand-designed grouping of AudioSet's original genre classes into 11 categories sets the ceiling for genre-supervised retrieval; different groupings would change results.
  • joint embedding dimension = 256
    Dimension of all joint embedding spaces, chosen without a stated ablation.
  • dropout probability = 0.4
    Dropout for all MLP blocks, chosen without a stated ablation.
assumptions (4)
  • standard math InfoNCE and SupCon losses are valid objectives for learning cross-modal embeddings.
    The losses are taken from [16] and [17]; no new mathematical claims are made.
  • domain assumption MERT audio features and CLIP video features are frozen and contain enough information for music-video alignment.
    The paper relies entirely on these pretrained features and does not train or adapt the encoders; if the features discard relevant temporal or visual cues, the method cannot recover them.
  • domain assumption A video and its own music track form the correct positive pair for self-supervised learning.
    Used in Eq. 3; assumes artistic correspondence is captured by co-occurrence in sourced music videos.
  • domain assumption AudioSet genre labels are a meaningful supervised signal for this task.
    Used in Eq. 4-6; the paper itself notes the labels are audio-centric and effectively ignore visual content, which weakens this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Semi-Supervised Contrastive Learning for Controllable Video-to-Music Retrieval." pith.science (2026). https://pith.science/paper/WSCCH3MG

@misc{pith2026241205831,
  author       = {Pith},
  title        = {Pith review of: Semi-Supervised Contrastive Learning for Controllable Video-to-Music Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WSCCH3MG}},
  note         = {Machine review of arXiv:2412.05831}
}
read the original abstract

Content creators often use music to enhance their videos, from soundtracks in movies to background music in video blogs and social media content. However, identifying the best music for a video can be a difficult and time-consuming task. To address this challenge, we propose a novel framework for automatically retrieving a matching music clip for a given video, and vice versa. Our approach leverages annotated music labels, as well as the inherent artistic correspondence between visual and music elements. Distinct from previous cross-modal music retrieval works, our method combines both self-supervised and supervised training objectives. We use self-supervised and label-supervised contrastive learning to train a joint embedding space between music and video. We show the effectiveness of our approach by using music genre labels for the supervised training component, and our framework can be generalized to other music annotations (e.g., emotion, instrument, etc.). Furthermore, our method enables fine-grained control over how much the retrieval process focuses on self-supervised vs. label information at inference time. We evaluate the learned embeddings through a variety of video-to-music and music-to-video retrieval tasks. Our experiments show that the proposed approach successfully combines self-supervised and supervised objectives and is effective for controllable music-video retrieval.

Figures

Figures reproduced from arXiv: 2412.05831 by the authors.

Figure 1
Figure 1. Overview of the semi-supervised Control-MVR framework. A dual￾branch architecture separately processes music and video, using frozen pre￾trained models as well a series of trainable networks. Self-supervised and supervised cross-modal contrastive losses operate on different points in the model architecture. A user-defined weight parameter α provides explicit control of the output embeddings z A and z V , which are u… view at source ↗
Figure 2
Figure 2. Control-MVR enables explicit control over the retrieval process at [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 21 canonical work pages

  1. [1]

    Cbvmr: Content-based video-music retrieval using soft intra-modal structure constraint,

    S. Hong, W. Im, and H. S.Yang, “Cbvmr: Content-based video-music retrieval using soft intra-modal structure constraint,” in Proceedings of the ACM International Conference on Multimedia Retrieval , 2018

    work page 2018

  2. [2]

    Cross-modal music-video recommendation: A study of design choices,

    L. Pr ´etet, G. Richard, and G. Peeters, “Cross-modal music-video recommendation: A study of design choices,” in Proceedings of the International Joint Conference on Neural Networks (IJCNN) , 2021

    work page 2021

  3. [3]

    Is there a

    ——, “Is there a ”language of music-video clips”? a qualitative and quantitative study,” in Proceedings of the International Society for Music Information Retrieval Conference (ISMIR) , 2021

    work page 2021

  4. [4]

    It’s time for artistic correspondence in music and video,

    D. Sur ´ıs, C. V ondrick, B. Russell, and J. Salamon, “It’s time for artistic correspondence in music and video,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022

    work page 2022

  5. [5]

    Attention is all you need,

    A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems, 2017

    2017

  6. [6]

    Language-guided music recommendation for video via prompt analogies,

    D. McKee, J. Salamon, J. Sivic, and B. Russell, “Language-guided music recommendation for video via prompt analogies,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    work page 2023

  7. [7]

    Mulan: A joint embedding of music audio and natural language,

    Q. Huang, A. Jansen, J. Lee, R. Ganti, J. Y . L, and D. P. W. Ellis, “Mulan: A joint embedding of music audio and natural language,” in Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), 2022

    work page 2022

  8. [8]

    Contrastive audio- language learning for music,

    I. Manco, E. Benetos, E. Quinton, and G. Fazekas, “Contrastive audio- language learning for music,” in Proceedings of the International Society for Music Information Retrieval Conference (ISMIR) , 2022

    work page 2022

Show all 24 references
  1. [9]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,

    Y . Wu, K. Chen, T. Zhang, Y . Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” in Proceedings of the Interna- tional Conference on Acoustics, Speech and Signal Processing (IC...

    2023

  2. [10]

    Textless speech-to-music retrieval using emotion similarity,

    S. Doh, M. Won, K. Choi, and J. Nam, “Textless speech-to-music retrieval using emotion similarity,” in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2023

    2023

  3. [11]

    Emotion-aligned contrastive learning between images and music,

    S. Stewart, K. Avramidis, T. Feng, and S. Narayanan, “Emotion-aligned contrastive learning between images and music,” in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024

    2024

  4. [12]

    Bridging high-quality audio and video via language for sound effects retrieval from visual queries,

    J. Wilkins, J. Salamon, M. Fuentes, J. P. Bello, and O. Nieto, “Bridging high-quality audio and video via language for sound effects retrieval from visual queries,” in WASPAA, 2023

    2023

  5. [13]

    Query by video: Cross-modal music retrieval,

    B. Li and A. Kumar, “Query by video: Cross-modal music retrieval,” in Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), 2019

    2019

  6. [14]

    Mert: Acoustic music understanding model with large-scale self-supervised training,

    Y . Li, R. Yuan, G. Zhang, Y . Ma, X. Chen, H. Yin, C. Xiao, C. Lin, A. Ragni, E. Benetos, N. Gyenge, R. Dannenberg, R. Liu, W. Chen, G. Xia, Y . Shi, W. Huang, Z. Wang, Y . Guo, and J. Fu, “Mert: Acoustic music understanding model with large-scale self-supervised training,” i...

    2024

  7. [15]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in Proceedings of the International Conference on Machine Learni...

    2021

  8. [16]

    Representation learning with contrastive predictive coding,

    A. van den Oord, Y . Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748 , 2018

    2018 arXiv

  9. [17]

    Supervised contrastive learn- ing,

    P. Khosla, P. Teterwak, C. Wang, Y . T. Aaron Sarna, P. Isola, A. Maschinot, C. Liu, and D. Krishnan, “Supervised contrastive learn- ing,” in Advances in Neural Information Processing Systems , 2020

    2020

  10. [18]

    Audio set: An ontology and human-labeled dataset for audio events,

    J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017

    2017

  11. [19]

    Youtube-8m: A large-scale video classi- fication benchmark,

    S. Abu-El-Haija, N. Kothari, J. Lee, P. Natsev, G. Toderici, B. Varadara- jan, and S. Vijayanarasimhan, “Youtube-8m: A large-scale video classi- fication benchmark,” arXiv preprint arXiv:1609.08675 , 2016

    2016 arXiv

  12. [20]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Proceedings of the International Conference on Learning Represen- tations (ICLR) , 2019

    2019

  13. [21]

    Wav2clip: Learning robust audio representations from clip,

    H.-H. Wu, P. Seetharaman, K. Kumar, and J. P. Bello, “Wav2clip: Learning robust audio representations from clip,” in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022

    2022

  14. [22]

    Audioclip: Extending clip to image, text and audio,

    A. Guzhov, F. Raue, J. Hees, and A. Dengel, “Audioclip: Extending clip to image, text and audio,” in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2022

    2022

  15. [23]

    Emotion embedding spaces for matching music to stories,

    M. Won, J. Salamon, N. J. Bryan, G. J. Mysore, and X. Serra, “Emotion embedding spaces for matching music to stories,” in Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), 2021

    2021

  16. [24]

    Vggsound: A large- scale audio-visual dataset,

    H. Chen, W. Xie, A. Vedaldi, and A. Zisserman, “Vggsound: A large- scale audio-visual dataset,” in Proceedings of the International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP) , 2020

    2020

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.