Learning Joint Embedding for Cross-Modal Retrieval

Donghuo Zeng

REVIEW 3 major objections 5 minor 12 references

Learning Joint Embedding for Cross-Modal Retrieval

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Triplet-enhanced supervised CCA lifts audio-visual retrieval to 72.69% mAP on VEGAS.

desk verdict A plausible 2.4-point mAP gain on VEGAS from adding triplet loss to S-DCCA, but the missing train/test split makes the headline result uninterpretable as reported. read the letter →

arxiv 1908.07673 v1 pith:X4UECAV2 submitted 2019-08-21 cs.IR cs.MM

Donghuo Zeng This is my paper · ORCID

classification cs.IRcs.MM

keywords cross-modalretrievaljointembeddingcanonicalcorrelationanalysistripletneuralnetworksaudio-visualdeeplearningVEGASdataset

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that adding triplet neural networks (TNN) to a supervised deep canonical correlation analysis (S-DCCA) architecture improves cross-modal retrieval between audio and video. On the VEGAS dataset, the TNN model reaches 72.69% mean average precision for audio-to-visual retrieval and 71.84% for visual-to-audio, outperforming S-DCCA and other baselines by about 2.4 points. The authors argue that explicitly pulling similar pairs together and pushing dissimilar pairs apart in the shared embedding space enhances the correlation learned by CCA. They further claim the approach generalizes to other paired multimedia data, such as image-text and video-text retrieval.

What carries the argument

The central machinery is a two-branch triplet neural network attached to a supervised deep CCA (S-DCCA) embedding. Raw audio is represented with VGGish features (128-dimensional per second) and raw video with Inception features (1024-dimensional per second); a mean layer collapses the time dimension before correlation learning. The CCA branch learns linear projections that maximize audio-visual correlation in a shared space, while the triplet branch uses anchor-positive and anchor-negative samples to pull matching pairs together and push non-matching pairs apart. This combined objective generates the final embeddings used for similarity matrix computation and ranking.

What would settle it

Run the TNN and S-DCCA models on a dataset of longer, temporally varying videos (e.g., full-length movie scenes or unedited event videos) where the per-second features are not aligned by concept; if the mAP gain over S-DCCA disappears or reverses, the temporal-alignment assumption is the cause.

Watch

Extended reading notes

Core claim

The central claim is that a triplet neural network, operating on top of a CCA-based joint embedding, yields a better cross-modal retrieval model than supervised CCA alone. The authors show that when audio and visual features are first mapped into a common space by maximizing correlation, adding triplet loss terms—which enforce that matching audio-visual pairs are closer than mismatched ones—improves retrieval accuracy. On the VEGAS dataset, their TNN model achieves 72.69% mAP for audio-to-visual and 71.84% for visual-to-audio, compared to 70.34% and 69.27% for S-DCCA, and much lower scores for plain CCA and DCCA. The paper presents this as evidence that incorporating category-level similarity and dissimilarity information into correlation learning strengthens the joint embedding.

Load-bearing premise

The method assumes that per-second high-level features (VGGish/Inception) collapsed by a mean layer are enough to make audio and visual sequences comparable in a shared CCA space, without any temporal alignment.

Editorial extensions

If this is right

If the TNN improvement holds beyond VEGAS, cross-modal retrieval systems for audio, video, image, and text could adopt triplet-enhanced supervised CCA as a straightforward upgrade to plain CCA-based joint embeddings.
The reported gap between supervised (VGGish/Inception) and unsupervised (SoundNet/I3D) feature representations suggests that the choice of per-second high-level features is a major driver of retrieval performance, more so than the correlation learning method itself.
The architecture's reliance on one-to-one concept correspondences indicates it could be applied to other paired datasets that have category labels, such as video-text captioning pairs, though temporal alignment is not explicitly handled.
The authors' plan to combine generative adversarial networks with the triplet discriminative model points toward a future direction where generated features are used to enrich training data for cross-modal retrieval.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The mean layer that collapses time likely works because VEGAS clips are short and conceptually homogeneous; on longer videos where events shift over time, the fixed mean may wash out the temporal structure that links specific audio moments to specific visual moments.
A testable extension would be to replace the mean layer with temporal attention or a sequence encoder and measure whether retrieval on VEGAS or a longer-video dataset improves; this would directly test the paper's implicit assumption that temporal alignment is unnecessary.
Because the triplet objective only contrasts whole-video embeddings, it may underuse fine-grained correspondences; a frame-level or segment-level triplet loss could be a stronger constraint, but it would require temporally aligned positive pairs, which the current dataset and features do not provide.
The reported improvements are small relative to the gain from switching feature extractors, suggesting that future work should prioritize better audio and visual representations over tweaks to the correlation objective.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is a short, honest work-in-progress note. What is actually new is a small increment on the author's S-DCCA line: add a triplet loss to the supervised CCA objective and evaluate on VEGAS across different feature extractors. Table I gives a useful internal consistency check (VGGish/Inception beats SoundNet/I3D by a wide margin), and the external VEGAS benchmark is the right kind of evaluation. The gain direction is consistent for both audio-visual and visual-audio, which is at least a sign that the effect is not entirely arbitrary. That should be credited.

The soft spots, in order of severity. First, the stress-test concern is real and load-bearing: the paper does not state whether the VEGAS videos used to train the triplet network are disjoint from the queries used for retrieval. Section II-B constructs correspondences from the dataset using concept labels, and Section III reports retrieval results with no train/test split, held-out query set, or cross-validation. VEGAS is small and category-structured; if training and querying share videos, the 2.4-point TNN gain could reflect concept memorization rather than retrieval generalization. This affects the reported numbers themselves, not just transfer to other datasets.

Second, there are no error bars or variance estimates, so we cannot tell whether 2.4 points is above noise. Third, no training details, data splits, hyperparameters, or code are provided, so the experiments are not reproducible. Fourth, Table II compares only against CCA and DCCA, and DCCA at 14.09% being far below linear CCA at 32.43% is odd enough that the setup needs explanation. Fifth, the relation to the author's reference [12], which already announces deep triplet networks with cluster-CCA, is not clarified; the incremental novelty here seems limited to feature representation and a slightly different loss application. The temporal-alignment assumption in Section II-B (mean layer over per-second features) is a real limitation but secondary to the split problem.

I would not publish this as is, but I would not desk reject it. A referee could ask for a clear split, error bars, and a direct comparison to [12], and if the gain survives on held-out queries this becomes a legitimate incremental result. I would send it to peer review with a conditional recommendation: fix the evaluation protocol before acceptance.

Referee Report

3 major / 5 minor

Summary. This short forum paper proposes a triplet neural network (TNN) enhancement over a supervised deep canonical correlation analysis (S-DCCA) architecture for audio-visual cross-modal retrieval. The method extracts high-level features (VGGish/Inception), applies a mean layer, learns a CCA-based joint embedding, and then uses triplet ranking to incorporate dissimilar pairs into the training objective. Preliminary results on the VEGAS dataset are reported in two tables: Table I compares S-DCCA under different feature combinations, and Table II compares CCA, DCCA, S-DCCA, and TNN, with TNN achieving 72.69% mAP for audio-visual and 71.84% for visual-audio, about 2.4 points above S-DCCA. The paper concludes that the proposed architecture achieves the best result when supervised data representations are used.

Significance. If the reported gains are reproducible and statistically reliable, the paper demonstrates a simple, potentially generalizable way to improve supervised CCA for cross-modal retrieval by adding triplet-based ranking. The evaluation uses the external VEGAS benchmark, so the central claim is not normalized by construction and is in principle falsifiable. However, the contribution is incremental, and the paper's value depends entirely on the credibility of the empirical comparison, which is currently not established because key experimental details are missing. The architecture is simple enough that the work could be a useful baseline if fully specified, but as written it does not support the 'state-of-the-art' claim.

major comments (3)

[Section III and Table II] The evaluation protocol is unspecified: the paper never states how the VEGAS data is split into training and test sets, whether queries are disjoint from the videos used to train the triplet network, or whether any cross-validation or held-out set is used. Section II-B says correspondences are built 'based on their concepts' and Section III reports retrieval results over the dataset, but no split is described. If the triplets are trained on the same videos later used as queries, the reported mAP gain could reflect memorization of concept-level correspondences rather than retrieval generalization. The authors must specify the train/test protocol, report results on a disjoint test set, and provide error bars or variance estimates so the 2.4-point difference can be assessed.
[Section II-B and III] The TNN architecture and training procedure are not specified in enough detail to reproduce the method. The paper does not give the triplet loss formulation, margin, sampling strategy, number of negative samples per query, optimizer, learning rate, number of epochs, or regularization. The only architectural details are a mean layer and an output dimension of 10. Without these details, the comparison in Table II is not interpretable as a fair evaluation of a well-defined method, and the claim that 'TNN model outperforms S-DCCA' cannot be independently verified.
[Table II and Section III] The comparison to 'state-of-the-art' methods is incomplete and partly self-referential. Table II includes only CCA, DCCA, S-DCCA, and TNN, where S-DCCA is the authors' own prior method and TNN is described as 'our proposed architecture [11], [12],' citing two previous papers by the same group. The relation between the current TNN and the earlier deep triplet network [12] is not clarified, so it is unclear what new contribution this paper makes beyond those earlier works. A stronger comparison should include recent cross-modal retrieval methods that have published results on VEGAS or on comparable audio-visual benchmarks, and the text should state explicitly what is new relative to [11] and [12].

minor comments (5)

[Section II-B] The phrase 'based on their concepts' is vague; the authors should state how many concepts VEGAS contains, how one-to-one correspondences are constructed from concept labels, and whether every video is paired with all other videos in the same concept or only with its own pair.
[Section III] Table I reports mAP for S-DCCA under four feature combinations, but it is not stated whether these features were taken from the official VEGAS release or re-extracted by the authors. This matters because feature extraction details can affect the reported numbers.
[Section III] The sentence 'Once The similarity matrix is generated' contains a capitalization error; it should read 'Once the similarity matrix is generated.'
[Section I and III] The paper refers to the proposed architecture as 'our proposed architecture [11], [12]' but does not specify which components are inherited from the prior works and which are newly introduced here. Adding one sentence to delimit the novelty would clarify the contribution.
[Section II-A] The notation 'ai ∈ R^{128×L×M}' is used, but L is later described as 'the number of extracted frames for video i' and M as 'the number of videos.' Since i indexes the video, the notation should make it clear whether L and M are global constants or vary per video; a brief clarification would help.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation; the TNN result is an empirical comparison on the external VEGAS benchmark, with only minor non-load-bearing self-citation.

full rationale

The paper's central claim is an empirical result: Table II reports mAP values for CCA, DCCA, S-DCCA, and TNN on the external VEGAS benchmark, so the TNN-vs-S-DCCA comparison is not forced by construction or by an internal equation. The method is described directly in Section II-B: data representations are extracted by pretrained VGGish/Inception models, passed through a mean layer, and used to learn a CCA-based joint embedding, after which a triplet neural network pushes dissimilar pairs apart and pulls similar pairs together. No fitted parameter is renamed as a prediction, and no uniqueness theorem or ansatz is imported from the author's prior work. The only self-citation appears in Section III, where the author writes 'Our proposed architecture [11], [12] is evaluated on the audio-visual dataset: VEGAS.' Those citations are to the author's earlier S-DCCA and triplet-network work, but the numerical comparison in Table II is self-contained in this paper and is measured against an external dataset, so the self-citation is not load-bearing. Concerns about the lack of a stated train/test split and missing error bars are evaluation-protocol or statistical-correctness issues, not circularity, and are therefore not counted as circular steps here.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The reported result rests on pretrained feature extractors, one-to-one concept correspondences, and an unspecified CCA-plus-triplet training procedure. The only explicit hyperparameter is the 10-dimensional output embedding. No invented entities are introduced.

free parameters (1)

Output embedding dimension = 10
Set to 10 for each audio and visual branch in the experiments. No sensitivity analysis is provided, so the choice could affect the reported mAP.

assumptions (3)

domain assumption High-level per-second feature representations from VGGish, Inception, SoundNet, and I3D preserve enough temporal structure for cross-modal correlation learning.
Invoked in Section II-B and Section III. No explicit temporal alignment is learned, so the pretrained features are assumed to bridge the temporal gap.
domain assumption Every audio-visual pair in VEGAS is aligned by concept, so one-to-one correspondences can be built from category labels.
Training set construction in Section II-B relies on category-based one-to-one pairings. If labels are noisy or concepts overlap, the triplet supervision would be affected.
domain assumption Combining a CCA-based objective with a triplet loss improves retrieval ranking in the shared embedding space.
Stated in Section II-B without derivation, convergence analysis, or ablation showing that the triplet loss, rather than the supervised representation, causes the gain.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Learning Joint Embedding for Cross-Modal Retrieval." pith.science (2026). https://pith.science/paper/X4UECAV2

@misc{pith2026190807673,
  author       = {Pith},
  title        = {Pith review of: Learning Joint Embedding for Cross-Modal Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X4UECAV2}},
  note         = {Machine review of arXiv:1908.07673}
}

read the original abstract

A cross-modal retrieval process is to use a query in one modality to obtain relevant data in another modality. The challenging issue of cross-modal retrieval lies in bridging the heterogeneous gap for similarity computation, which has been broadly discussed in image-text, audio-text, and video-text cross-modal multimedia data mining and retrieval. However, the gap in temporal structures of different data modalities is not well addressed due to the lack of alignment relationship between temporal cross-modal structures. Our research focuses on learning the correlation between different modalities for the task of cross-modal retrieval. We have proposed an architecture: Supervised-Deep Canonical Correlation Analysis (S-DCCA), for cross-modal retrieval. In this forum paper, we will talk about how to exploit triplet neural networks (TNN) to enhance the correlation learning for cross-modal retrieval. The experimental result shows the proposed TNN-based supervised correlation learning architecture can get the best result when the data representation extracted by supervised learning.

Figures

Figures reproduced from arXiv: 1908.07673 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 12 canonical work pages

[12]

Deep Triplet Neural Networks with Cluster-CCA for Audio-Visual Cross-modal Retrieval

Z. Donghuo, Y . Yi, and O. Keizo, “Deep triplet neural networks with cluster-cca for audio-visual cross-modal retrieval,”arXiv preprint arXiv:1908.03737, 2019

work page Pith review arXiv 1908
[11]

Audio-visual embedding for cross- modal music video retrieval through supervised deep cca

Z. Donghuo, Y . Yu, and O. Keizo, “Audio-visual embedding for cross- modal music video retrieval through supervised deep cca.” IEEE, 2018, pp. 143–150

work page 2018
[1]

Automatic music soundtrack generation for outdoor videos from contextual sensor information,

Y . Yu, Z. Shen, and R. Zimmermann, “Automatic music soundtrack generation for outdoor videos from contextual sensor information,” 2012, pp. 1377–1378

work page 2012
[2]

Learning from between- class examples for deep sound recognition,

Y . Tokozume, Y . Ushiku, and T. Harada, “Learning from between- class examples for deep sound recognition,” in International Conference on Learning Representations , 2018. [Online]. Available: https://openreview.net/forum?id=B1Gi6LeRZ

work page 2018
[3]

Audio set: An ontology and human-labeled dataset for audio events,

J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2017, pp. 776–780

work page 2017
[4]

Learnable pooling with context gating for video classiﬁcation,

A. Miech, I. Laptev, and J. Sivic, “Learnable pooling with context gating for video classiﬁcation,” 2017

work page 2017
[5]

Canonical corre- lation analysis: An overview with application to learning methods,

D. R. Hardoon, S. Szedmak, and J. Shawe-Taylor, “Canonical corre- lation analysis: An overview with application to learning methods,” Neural computation, vol. 16, no. 12, pp. 2639–2664, 2004

work page 2004
[6]

Deep cross-modal correlation learning for audio and lyrics in music retrieval,

Y . Yu, S. Tang, F. Raposo, and L. Chen, “Deep cross-modal correlation learning for audio and lyrics in music retrieval,” vol. 15, no. 1, article

work page

Show all 12 references

[7]

Deep canonical cor- relation analysis,

G. Andrew, R. Arora, J. Bilmes, and K. Livescu, “Deep canonical cor- relation analysis,” in International Conference on Machine Learning , 2013, pp. 1247–1255

2013
[8]

Cluster canonical correlation analysis,

N. Rasiwasia, D. Mahajan, V . Mahadevan, and G. Aggarwal, “Cluster canonical correlation analysis,” in Artiﬁcial Intelligence and Statistics, 2014, pp. 823–831

2014
[9]

Category-based deep cca for ﬁne-grained venue discovery from multimodal data,

Y . Yu, S. Tang, K. Aizawa, and A. Aizawa, “Category-based deep cca for ﬁne-grained venue discovery from multimodal data,” vol. 30, no. 4, 2019, pp. 1250–1258

2019
[10]

Visual to sound: Generating natural sound for videos in the wild,

Y . Zhou, Z. Wang, C. Fang, T. Bui, and T. L. Berg, “Visual to sound: Generating natural sound for videos in the wild,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 3550–3558

2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [12]

Deep Triplet Neural Networks with Cluster-CCA for Audio-Visual Cross-modal Retrieval

Z. Donghuo, Y . Yi, and O. Keizo, “Deep triplet neural networks with cluster-cca for audio-visual cross-modal retrieval,”arXiv preprint arXiv:1908.03737, 2019

work page Pith review arXiv 1908

[2] [11]

Audio-visual embedding for cross- modal music video retrieval through supervised deep cca

Z. Donghuo, Y . Yu, and O. Keizo, “Audio-visual embedding for cross- modal music video retrieval through supervised deep cca.” IEEE, 2018, pp. 143–150

work page 2018

[3] [1]

Automatic music soundtrack generation for outdoor videos from contextual sensor information,

Y . Yu, Z. Shen, and R. Zimmermann, “Automatic music soundtrack generation for outdoor videos from contextual sensor information,” 2012, pp. 1377–1378

work page 2012

[4] [2]

Learning from between- class examples for deep sound recognition,

Y . Tokozume, Y . Ushiku, and T. Harada, “Learning from between- class examples for deep sound recognition,” in International Conference on Learning Representations , 2018. [Online]. Available: https://openreview.net/forum?id=B1Gi6LeRZ

work page 2018

[5] [3]

Audio set: An ontology and human-labeled dataset for audio events,

J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2017, pp. 776–780

work page 2017

[6] [4]

Learnable pooling with context gating for video classiﬁcation,

A. Miech, I. Laptev, and J. Sivic, “Learnable pooling with context gating for video classiﬁcation,” 2017

work page 2017

[7] [5]

Canonical corre- lation analysis: An overview with application to learning methods,

D. R. Hardoon, S. Szedmak, and J. Shawe-Taylor, “Canonical corre- lation analysis: An overview with application to learning methods,” Neural computation, vol. 16, no. 12, pp. 2639–2664, 2004

work page 2004

[8] [6]

Deep cross-modal correlation learning for audio and lyrics in music retrieval,

Y . Yu, S. Tang, F. Raposo, and L. Chen, “Deep cross-modal correlation learning for audio and lyrics in music retrieval,” vol. 15, no. 1, article

work page