Pith. sign in

REVIEW 4 major objections 6 minor 27 references

Audio-Visual Embedding for Cross-Modal MusicVideo Retrieval through Supervised Deep CCA

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read Supervised deep CCA lets a short audio clip retrieve the matching full-length music video.

desk verdict Plausible method and a genuinely useful dataset, but the evaluation's ground truth comes from the same audio clusters used for training, so the reported MAP numbers do not yet demonstrate emotion-based cross-modal retrieval. read the letter →

arxiv 1908.03744 v1 pith:CZ7QGOXZ submitted 2019-08-10 cs.MM cs.IR

classification cs.MMcs.IR
keywords cross-modalretrievalmusicvideodeepcanonicalcorrelationanalysisaudio-visualembeddingemotionsimilarityattentionmechanismLSTMk-meansclustering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a supervised variant of deep canonical correlation analysis, S-DCCA, can embed music audio and music-video visuals into a shared space organized by emotion, so that an audio snippet of arbitrary length can retrieve a full-length music video. The key move is to augment the usual one-to-one audio-video training pairs with pairs drawn from different videos that fall in the same emotion cluster, injecting cluster labels into the CCA objective. The paper also proposes selecting a small set of representative audio chunks with an attention-based bidirectional LSTM, rather than using the whole audio. On a 10,000-video music dataset built from a large public video collection, it reports that S-DCCA reaches a mean average precision of 21.76, beating CCA, DCCA, and Cluster-CCA.

What carries the argument

The central mechanism is the supervised deep CCA objective. Two neural branches map audio features $X$ and visual features $Y$ into a common space, maximizing the correlation $\mathrm{corr}(f_x(X,\theta_x), f_y(Y,\theta_y))$ under the constraints $W_x^T \Sigma_{xx} W_x = I$ and $W_y^T \Sigma_{yy} W_y = I$, while the training-pair set is expanded to include audio and visual content from different videos in the same emotion cluster. A second mechanism is the attention-based bidirectional LSTM that scores each 3-second audio chunk; the top $k$ chunks form the query. A third is k-means clustering of audio emotion features into 10 classes, which supplies the cluster labels used for forming supervised pairs and for defining relevance at evaluation time.

What would settle it

Collect human relevance judgments for a random sample of query audios and compare them with the k-means cluster labels used as ground truth; if humans frequently disagree with the cluster assignments, then the reported MAP values overstate emotion-based retrieval quality and the embedding should be re-evaluated on human-annotated relevance.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that emotion cluster labels can be turned into cross-modal supervision: by training the two network branches not only on matching audio-video pairs but also on pairs from different videos sharing a k-means emotion cluster, the learned embedding makes audio and video comparable by emotion similarity. The attention-LSTM chunk selection then gives a compact audio query that retains the temporally local, emotion-dense parts of the signal. The paper claims that this combination outperforms existing CCA-family methods and that using a few selected chunks performs about as well as using the full-length audio average, which supports the idea that the selected chunks carry the emotion content.

Load-bearing premise

The evaluation assumes that the k-means clusters computed from audio emotion features are a correct ground truth for which music videos are relevant to an audio query; if those clusters do not match human emotion similarity, the reported retrieval scores measure cluster membership rather than emotion-based retrieval.

Editorial extensions

If this is right

  • Audio-to-video retrieval works from a short query: a snippet of a few seconds can stand in for the full track without a large drop in reported mean average precision.
  • Using same-cluster cross-instance pairs during training improves retrieval, and the reported MAP increases as the pair set grows from 8,000 to 1.5 million pairs.
  • The attention-based chunk selector identifies emotion-dense parts of an audio, since selecting 3 of 9 chunks matches or exceeds the full-audio average as a query.
  • The same supervised-CCA architecture can be reversed to retrieve a full audio from a short video query, which the paper names as future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's experiments, the same cluster-pair construction could transfer to other emotion-labeled media pairs, such as audio with lyrics or album art, wherever cluster labels are available.
  • The reported MAP is measured against the same k-means clusters used in training; if those clusters are noisy relative to human emotion judgments, the true emotion-retrieval performance would be lower. A human-labeled relevance study on a sample of queries would settle this.
  • A testable extension the paper does not run is an ablation that varies the number of selected chunks while fixing the total audio length; the attention scores predict that retrieval degrades smoothly as fewer chunks are kept, which can be checked directly.
  • Since relevance at test time is defined by audio-feature clusters, the ranking might be improved by fusing visual features into the clustering step, a variant the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a Supervised Deep Canonical Correlation Analysis (S-DCCA) model for cross-modal music-video retrieval, in which an audio segment of variable length retrieves a full-length music video on the basis of emotion similarity. The audio branch first selects top-k representative chunks using an attention-based bidirectional LSTM emotion model; both audio and video features (from YouTube-8M) are then projected into a shared space via a CCA-style objective that is augmented with audio-visual pairs drawn from videos sharing the same k-means cluster. Experiments on a constructed 10K music-video subset compare S-DCCA with CCA, KCCA, DCCA, C-CCA, and multi-view baselines, reporting the best MAP (21.76 for S-DCCA vs. 19.72 for C-CCA and 18.40 for DCCA in Table II). The paper also analyzes precision-recall behavior under different numbers of audio chunks and selected chunks.

Significance. If the empirical results were valid, the paper would offer a practically relevant contribution: an audio-to-video retrieval formulation for music videos, a constructed 10K dataset from YouTube-8M, and an end-to-end architecture combining attention-based chunk selection with supervised CCA. The use of public frame-level features and the direct comparison with several CCA variants are strengths. However, the evaluation ground truth is derived from the same k-means clustering of audio emotion features that also supplies the supervision signal, so the reported MAP numbers do not yet establish emotion-based cross-modal retrieval. The missing covariance equations and the incorrect k-means objective also hamper reproducibility. The underlying idea of supervised deep CCA with cluster-conditioned pair augmentation is worth pursuing, but the current evidence is not sufficient to support the paper's central claim.

major comments (4)
  1. [Section IV.A.2 and Section III.C] The test relevance labels are defined as 'all the music videos that have the same cluster label as the query audio,' where those cluster labels are produced by the k-means procedure in Section III.C applied to audio emotion features. Since the same clusters also determine the audio-visual training pairs for S-DCCA, the evaluation is partially self-referential. A model could achieve high MAP on this metric by learning to reproduce the audio-derived k-means partition, without using any visual emotion information from the video frames. The reported MAP advantage over DCCA and C-CCA in Table II therefore does not by itself demonstrate emotion-based cross-modal retrieval. Please re-evaluate with relevance labels that are independent of the training supervision, for example human emotion annotations of video content, or at least provide evidence that the k-means clusters correspond to human-judged emotion similarity.
  2. [Section III.B and Eq. (10)] The formal definition of S-DCCA is incomplete. The text states that the covariance matrices Σxx, Σxy, and Σyy 'are computed as' and then leaves the definition blank, and the pair-construction rule and the optimization objective are referenced only as '( ?? )'. Without these equations the central method cannot be reproduced or compared rigorously. Please supply the missing definitions and equation numbers.
  3. [Section III.C, Eq. (11)] The k-means objective in Eq. (11) is written as an arg max over within-cluster squared distances to the cluster means. k-means minimizes this quantity; as written, the objective would drive points away from cluster centers. This is a technical error that should be corrected to arg min.
  4. [Section IV.D and Section III.C] It is unclear whether the k-means clustering is applied to the full 10,000-video dataset before splitting into cross-validation folds. If the clustering includes test queries and database videos, then the test relevance labels are derived from a procedure that has already seen the test data, which is a form of leakage. Please clarify the data flow and, if necessary, recompute clusters using only the training fold.
minor comments (6)
  1. [Section III.B, Eq. (8)] The notation 'X ∈ R^k, Y ∈ R^l' is inconsistent with the use of W_x^T X and W_y^T Y in Eq. (8); please define all dimensions and state the sizes of W_x and W_y.
  2. [Section III.D and Section IV.A] The paper says a pre-trained VGG16 model extracts frame-level audio features in Section III.D, while Section IV.A states that a VGG-like model described in [8] is used. Please make the feature extraction description consistent.
  3. [Table II] The column headers 'k/chunks 1/3 2/6 3/9 mean' are ambiguous. Please define k and c explicitly and specify what 'mean' refers to (e.g., average of all chunk-level features).
  4. [Figs. 5-9] The notation k/c and the meaning of 'mean' in the figures should be defined in the captions rather than only in the body text, to make the figures self-contained.
  5. [Section III.A, Eqs. (1)-(3)] The input gate is denoted s_t in Eq. (1) but i_t in Eq. (3). Please use a single consistent symbol.
  6. [Abstract and Section II] There are several typos and formatting artifacts, including 'Y ouTube' in the abstract and 'SVM hmm' in the related-work paragraph for [18]. A careful proofread is needed.

Circularity Check

1 steps flagged · score 6.0 of 10

Evaluation is circular: k-means clusters of audio emotion features define both the S-DCCA training pairs and the test relevance labels, so the reported MAP may only reflect reproducibility of the audio clustering, not emotion-based cross-modal retrieval.

  1. fitted input called prediction [Section III.B–III.C (K-means clustering; S-DCCA supervised pairs) and Section IV.A.2 (Evaluation Metrics)]
    "we allocate 3 annotated audios for each 10 pre-defined categories (angry, tender, bitter, cheerful, fun, bright, happy, anxious, calm and warm) to compute the initiated mean u0. We use the k-means method to cluster all audios into 10 semantic classes based on the emotion features. ... During testing, we assume all the music videos that have the same cluster label as the query audio are relevant."

    The same k-means partition of audio emotion features provides both the supervision signal and the test ground truth. Section III.B forms 'new relevant pairs' from audio and visual contents of videos 'with the same class label,' and Section IV.A.2 defines relevance as 'the same cluster label as the query audio.' Thus a model can score well by learning, from the visual branch, the audio-derived cluster label of each database video; no visual emotion content is needed. The MAP advantage over DCCA and C-CCA therefore shows internal consistency with the audio clustering, not an independently validated emotion-based cross-modal retrieval result.

full rationale

The paper's architecture and optimization are not themselves circular: S-DCCA extends DCCA with supervised same-cluster pairs, and the attention-based chunk selection is a legitimate technical contribution. However, the central empirical claim—that S-DCCA enables emotion-based music video retrieval and outperforms baselines—rests on an evaluation whose relevance labels are constructed from the same k-means clusters used to create the training pairs. Section III.C clusters all audios into ten semantic classes 'based on the emotion features,' initialized with three annotated audios per class. Section IV.A.2 then states, 'During testing, we assume all the music videos that have the same cluster label as the query audio are relevant,' and Section IV.D confirms that 'the correct retrieved video in the rank list which has the same category as query.' Since videos receive their cluster label from their own audio track, the ground truth is a function of the audio features only, and the supervised training objective already forces audio and video from the same audio cluster to be close. The evaluation therefore measures how well the embedding reproduces the audio-derived k-means partition, rather than how well it captures human-judged emotion similarity across modalities. A model that ignored visual content and simply retrieved videos whose audio cluster matched the query would be perfectly correct under this metric. This is a partial but load-bearing circularity in the evaluation, so the score is 6 rather than 0. No additional circularity from self-citation or imported uniqueness theorems was found.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard statistical machinery (CCA, DCCA, k-means, LSTM) plus several domain assumptions about emotion being encoded in pre-trained audio and visual features. The most fragile is the use of k-means clusters on audio emotion features as ground truth labels for video relevance, which makes the evaluation self-referential. No new physical or conceptual entities are introduced.

free parameters (4)
  • number of semantic clusters = 10
    K-means is run with 10 pre-defined emotion categories (angry, tender, bitter, cheerful, fun, bright, happy, anxious, calm, warm); chosen by hand.
  • number of CCA components = 30
    Reported as a network parameter in Section IV.B.
  • audio chunk length = 3 seconds
    Each audio is divided into 3-second chunks; 216 seconds gives 72 chunks. This affects all results.
  • number of selected audio chunks k = 1, 2, or 3
    The attention model selects top k chunks; the paper tests different k values.
assumptions (4)
  • standard math CCA and DCCA objectives maximize correlation between linear or nonlinear projections of paired modalities
    The method is built on the standard CCA and DCCA formulations in Eqs. (8) and (9).
  • domain assumption Audio emotion features from a pre-trained LSTM attention model trained on MER31K capture emotion in arbitrary audio chunks
    The chunk selection and clustering rely on emotion scores from the model of Huang et al. [10]; no validation on MV-10K is provided.
  • domain assumption Visual features extracted from an ImageNet-trained Inception network encode emotion-relevant semantics
    The visual branch uses these pre-trained features without any emotion-specific adaptation.
  • ad hoc to paper K-means clusters of audio features correspond to meaningful semantic classes for retrieval relevance
    These clusters are used both as training supervision and as test ground truth, making the evaluation depend on this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Audio-Visual Embedding for Cross-Modal MusicVideo Retrieval through Supervised Deep CCA." pith.science (2026). https://pith.science/paper/CZ7QGOXZ

@misc{pith2026190803744,
  author       = {Pith},
  title        = {Pith review of: Audio-Visual Embedding for Cross-Modal MusicVideo Retrieval through Supervised Deep CCA},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CZ7QGOXZ}},
  note         = {Machine review of arXiv:1908.03744}
}
read the original abstract

Deep learning has successfully shown excellent performance in learning joint representations between different data modalities. Unfortunately, little research focuses on cross-modal correlation learning where temporal structures of different data modalities, such as audio and video, should be taken into account. Music video retrieval by given musical audio is a natural way to search and interact with music contents. In this work, we study cross-modal music video retrieval in terms of emotion similarity. Particularly, audio of an arbitrary length is used to retrieve a longer or full-length music video. To this end, we propose a novel audio-visual embedding algorithm by Supervised Deep CanonicalCorrelation Analysis (S-DCCA) that projects audio and video into a shared space to bridge the semantic gap between audio and video. This also preserves the similarity between audio and visual contents from different videos with the same class label and the temporal structure. The contribution of our approach is mainly manifested in the two aspects: i) We propose to select top k audio chunks by attention-based Long Short-Term Memory (LSTM)model, which can represent good audio summarization with local properties. ii) We propose an end-to-end deep model for cross-modal audio-visual learning where S-DCCA is trained to learn the semantic correlation between audio and visual modalities. Due to the lack of music video dataset, we construct 10K music video dataset from YouTube 8M dataset. Some promising results such as MAP and precision-recall show that our proposed model can be applied to music video retrieval.

Figures

Figures reproduced from arXiv: 1908.03744 by the authors.

Figure 1
Figure 1. Overview of music video retrieval: Select one or more representative [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Emotion learning model for evaluating the contribution of each chunk [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Audio-visual embedding architecture through S-DCCA. (left) During the training process, the model learns the correlation between audio and visual [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Precision-recall curve with the number of chunks set to 3, where [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 7
Figure 7. Figure 7: Precision-recall curve with the chunks=9, where “mean” denotes using [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Precision-recall curve, achieved by changing the number of output, [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Mean average precision when using different numbers of audio chunks [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 20 canonical work pages

  1. [24]

    Category- based deep cca for fine-grained venue discovery from multimodal data

    Yi Yu, Suhua Tang, Kiyoharu Aizawa, and Akiko Aizawa. Category- based deep cca for fine-grained venue discovery from multimodal data. IEEE Transactions on Neural Networks and Learning Systems , pages 1–9, 2018

  2. [1]

    Youtube-8m: A large-scale video classification benchmark

    Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. Youtube-8m: A large-scale video classification benchmark. arXiv preprint arXiv:1609.08675, 2016

  3. [2]

    Understanding affective content of music videos through learned representations

    Esra Acar, Frank Hopfgartner, and Sahin Albayrak. Understanding affective content of music videos through learned representations. In International Conference on Multimedia Modeling , pages 303–314. Springer, 2014

  4. [3]

    Deep canonical correlation analysis

    Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In International Conference on Machine Learning, pages 1247–1255, 2013

  5. [4]

    The sound of an album cover: Probabilistic multimedia and information retrieval

    Eric Brochu, Nando De Freitas, and Kejie Bao. The sound of an album cover: Probabilistic multimedia and information retrieval. In Artificial Intelligence and Statistics (AISTATS) , 2003

  6. [5]

    An introduction to support vector machines and other kernel-based learning methods

    Nello Cristianini, John Shawe-Taylor, et al. An introduction to support vector machines and other kernel-based learning methods . Cambridge university press, 2000

  7. [6]

    Cross-modal retrieval with correspondence autoencoder

    Fangxiang Feng, Xiaojie Wang, and Ruifan Li. Cross-modal retrieval with correspondence autoencoder. In Proceedings of the 22nd ACM international conference on Multimedia , pages 7–16. ACM, 2014

  8. [7]

    On the correlation of automatic audio and visual segmentations of music videos

    Olivier Gillet, Slim Essid, and Gal Richard. On the correlation of automatic audio and visual segmentations of music videos. IEEE Transactions on Circuits and Systems for Video Technology , 17(3):347– 355, 2007

Show all 27 references
  1. [8]

    Cnn architectures for large-scale audio classification

    Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cnn architectures for large-scale audio classification. In Acoustics, Speech and Signal Processing (ICASSP), 2017 IE...

  2. [9]

    Long short-term memory

    Sepp Hochreiter and J ¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997

  3. [10]

    Music thumbnail- ing via neural attention modeling of music emotion

    Yu-Siang Huang, Szu-Yu Chou, and Yi-Hsuan Yang. Music thumbnail- ing via neural attention modeling of music emotion. In Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), 2017 , pages 347–350. IEEE, 2017

  4. [11]

    Deep cross-modal hashing

    Qing-Yuan Jiang and Wu-Jun Li. Deep cross-modal hashing. CoRR, 2016

  5. [12]

    An empirical evaluation of doc2vec with practical insights into document embedding generation

    Jey Han Lau and Timothy Baldwin. An empirical evaluation of doc2vec with practical insights into document embedding generation. arXiv preprint arXiv:1607.05368, 2016

  6. [13]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014

  7. [14]

    Analysing the similarity of album art with self-organising maps

    Rudolf Mayer. Analysing the similarity of album art with self-organising maps. In International Workshop on Self-Organizing Maps , pages 357–

  8. [15]

    Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation

    David Martin Powers. Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation. 2011

  9. [16]

    A new approach to cross-modal multimedia retrieval

    Nikhil Rasiwasia, Jose Costa Pereira, Emanuele Coviello, Gabriel Doyle, Gert RG Lanckriet, Roger Levy, and Nuno Vasconcelos. A new approach to cross-modal multimedia retrieval. In Proceedings of the 18th ACM international conference on Multimedia , pages 251–260. ACM, 2010

  10. [17]

    Cluster canonical correlation analysis

    Nikhil Rasiwasia, Dhruv Mahajan, Vijay Mahadevan, and Gaurav Ag- garwal. Cluster canonical correlation analysis. In Artificial Intelligence and Statistics, pages 823–831, 2014

  11. [18]

    Advisor: Person- alized video soundtrack recommendation by late fusion with heuristic rankings

    Rajiv Ratn Shah, Yi Yu, and Roger Zimmermann. Advisor: Person- alized video soundtrack recommendation by late fusion with heuristic rankings. In Proceedings of the 22nd ACM international conference on Multimedia, pages 607–616. ACM, 2014

  12. [19]

    Very deep convolu- tional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolu- tional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014

  13. [20]

    Canonical correlation analysis

    Bruce Thompson. Canonical correlation analysis. Encyclopedia of statistics in behavioral science , 2005

  14. [21]

    Learning deep structure- preserving image-text embeddings

    Liwei Wang, Yin Li, and Svetlana Lazebnik. Learning deep structure- preserving image-text embeddings. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 5005–5013, 2016

  15. [22]

    Automatic music sound- track generation for outdoor videos from contextual sensor information

    Yi Yu, Zhijie Shen, and Roger Zimmermann. Automatic music sound- track generation for outdoor videos from contextual sensor information. In Proceedings of the 20th ACM international conference on Multimedia, pages 1377–1378. ACM, 2012

  16. [23]

    Venuenet: Fine-grained venue discovery by deep correlation learning

    Yi Yu, Suhua Tang, Kiyoharu Aizawa, and Akiko Aizawa. Venuenet: Fine-grained venue discovery by deep correlation learning. In Multi- media (ISM), 2017 IEEE International Symposium on , pages 288–291. IEEE, 2017

  17. [25]

    Deep cross- modal correlation learning for audio and lyrics in music retrieval

    Yi Yu, Suhua Tang, Francisco Raposo, and Lei Chen. Deep cross- modal correlation learning for audio and lyrics in music retrieval. ACM Transaction on Multimedia Computing Communication and Applica- tions, 2017

  18. [26]

    Video captioning and retrieval models with semantic attention

    Youngjae Yu, Hyungjin Ko, Jongwook Choi, and Gunhee Kim. Video captioning and retrieval models with semantic attention. arxiv preprint. arXiv preprint arXiv:1610.02947 , 2, 2016

  19. [27]

    Multi-view learning overview: Recent progress and new challenges

    Jing Zhao, Xijiong Xie, Xin Xu, and Shiliang Sun. Multi-view learning overview: Recent progress and new challenges. Information Fusion , 38:43–54, 2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.