Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Deep Triplet Neural Networks with Cluster-CCA for Audio-Visual Cross-modal Retrieval

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Audio-visual retrieval improves when a deep triplet network refines Cluster-CCA embeddings by pulling same-category cross-modal pairs together in cosine space.

desk verdict A plausible stacking of Cluster-CCA with a cosine triplet network for audio-visual retrieval, but the headline MAP gains are unreliable because hyperparameters were tuned on the test set and baseline tuning is undocumented. read the letter →

arxiv 1908.03737 v3 pith:EQI4KFJ4 submitted 2019-08-10 cs.MM cs.IR

Donghuo Zeng , Yi Yu , Keizo Oyama This is my paper · ORCID
classification cs.MMcs.IR
keywords audio-visualcross-modalretrievaldeeptripletneuralnetworkclustercanonicalcorrelationanalysislossjointembeddinglearningmeanaverageprecision
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to improve cross-modal retrieval between audio and video, a setting with little prior work, by learning a shared embedding where a sound and a video from the same semantic category are close while pairs from different categories are pushed apart. It proposes TNN-C-CCA, a supervised two-branch architecture: Cluster-CCA first projects VGGish audio features and Inception V3 visual features into a common low-dimensional space, and a deep triplet neural network then refines that space with triplet loss over cosine distance. On the VEGAS and MV-10K datasets the authors report that this combination beats six CCA-based baselines and four recent cross-modal retrieval methods on mean average precision, with the largest gains on VEGAS (74.66% audio-to-visual and 73.77% visual-to-audio). A sympathetic reader would care because the method is a concrete recipe for category-supervised audio-visual retrieval and because the triplet stage directly attacks the pairwise-only limitation of standard correlation learning.

What carries the argument

The architecture chains two stages. Cluster-CCA projects the two modalities into a shared subspace by treating every cross-modal pair inside a semantic category as a matching pair and applying canonical correlation analysis, which segregates categories while maximizing correlation. A deep triplet neural network then refines those embeddings with loss $d_{\cos}(T(i),S(j)) + \alpha < d_{\cos}(T(i),S(k))$, where $T(i)$ is the anchor audio embedding, $S(j)$ a same-category visual embedding, $S(k)$ a different-category visual embedding, $d_{\cos}$ is cosine distance, and $\alpha=0.5$; in-batch triplet selection (batch all, batch hard, batch semi-hard) controls which negatives are used. The triplet fine-tuning stage is what turns pairwise and category-level correlation into a space where same-category cross-modal pairs are closer than different-category pairs.

What would settle it

Retrain every baseline on the same VEGAS and MV-10K folds with per-method hyperparameter search, especially C-DCCA's beta and the regularization of the hashing and adversarial baselines, and check whether TNN-C-CCA's MAP margins survive; a modest grid search that closes the 4.28-point gap on VEGAS would falsify the outperformance claim.

Watch

Extended reading notes

Core claim

The paper claims that adding a deep triplet neural network on top of Cluster-CCA produces more discriminative audio-visual embeddings than either component alone or than existing CCA variants. The architecture maps audio and visual features to 10-dimensional embeddings via Cluster-CCA, then trains three fully connected branches (audio anchor, visual positive, visual negative, with the two visual branches sharing weights) using triplet loss with cosine distance and a margin of 0.5. In 5-fold cross-validation, TNN-C-CCA reaches MAP 74.66% and 73.77% for audio-to-visual and visual-to-audio retrieval on VEGAS, and 23.34% and 21.32% on MV-10K, surpassing the strongest CCA baseline C-DCCA by 4.28 and 4.5 percentage points on VEGAS. Ablation results show cosine distance beats Euclidean distance, batch-all triplet selection beats batch-hard and batch semi-hard, and the learned embedding clusters are more separated than Cluster-CCA's.

Load-bearing premise

The reported lead over baselines assumes that all comparison methods were implemented and tuned well enough that their hyperparameters are near-optimal, so the score gaps reflect method quality rather than undertuned competitors.

Editorial extensions

If this is right

  • Audio-to-visual and visual-to-audio retrieval improve together, so the learned shared space is usable in both query directions.
  • Explicitly training on negative cross-modal pairs, rather than only on positive/category pairs, is the component the paper credits for the margin over C-DCCA.
  • The method's gains shrink on the noisily labeled MV-10K dataset, indicating that label quality is a binding constraint on the approach.
  • The reported batch-size experiments imply a practical cost curve: reaching the top MAP requires enough negatives per batch, and the paper reports training times up to 32 hours for the best configuration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The two-stage recipe should transfer to other labeled modality pairs such as image-text or video-text, since the architecture does not depend on audio-specific machinery beyond the feature extractors.
  • Because performance drops sharply on MV-10K's automatically assigned labels, a testable extension is to corrupt a known fraction of VEGAS labels and measure how quickly the MAP advantage over C-DCCA erodes.
  • The paper's own batch-size table suggests that an external reader could check whether hard-negative mining at small batch sizes recovers most of the reported gain at a fraction of the 32-hour cost.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript presents TNN-C-CCA, an end-to-end two-branch architecture for audio-visual cross-modal retrieval. Audio clips are embedded with VGGish and visual frames with Inception-v3 plus PCA; Cluster-CCA projects the two views into a shared 10-dimensional space, and a triplet network with cosine distance and in-batch mining refines the embeddings using same-category positive pairs and different-category negative pairs. The method is evaluated on VEGAS and the authors' MV-10K dataset with MAP and PRC, and compared with six CCA-based methods and four state-of-the-art cross-modal retrieval methods. The paper claims consistent improvements, with the largest being 74.66% versus 70.34% audio-to-visual MAP over C-DCCA on VEGAS. Ablation studies cover triplet selection strategies, Euclidean versus cosine distance, and sensitivity to margin, batch size, and correlation components.

Significance. The audio-visual retrieval setting is underexplored, and the combination of cluster-level correlation learning with triplet ranking is a reasonable and clearly motivated idea. The paper's strengths include an end-to-end architecture, five-fold cross-validation, a broad comparison set, and an ablation study that probes several design choices. If the reported gains survive a statistically fair comparison, the method would be a useful contribution to multimedia retrieval. However, the quantitative claim is currently not supported by the evaluation protocol, because the proposed method's hyperparameters are selected on the test folds and the baselines are not shown to be comparably tuned. The contribution is incremental rather than conceptual, but it could be acceptable after a rigorous re-evaluation.

major comments (4)
  1. [Section 4.2(3) and Section 4.5.4] The margin (0.5), batch number (400), and correlation component count (10) are chosen by running the model on the VEGAS dataset and selecting the values with the highest MAP. Figure 10 plots MAP against the number of correlation components without any indicated validation split. Since the same folds are used for both selection and evaluation, the headline MAPs in Table 2 are optimistically biased. The authors should use a held-out validation split (or nested cross-validation) and report test MAP for the configuration selected without access to test labels.
  2. [Section 4.3] The baseline comparison is not sufficient to support the claimed improvement. The paper states that all baselines were implemented "with the same dimension of outputs and the same parameters," which does not address hyperparameter tuning. C-DCCA's beta is explicitly described as "very hard to set," yet no search is reported; the VEGAS advantage over C-DCCA is only 4.3 percentage points. To make the comparison interpretable, the authors should report validation-based hyperparameter selection for every baseline or use the numbers from the original publications.
  3. [Tables 5 and 6] The reported numbers are internally inconsistent. Table 5, which is presented as the margin study with "batch_num is 500," gives audio-visual MAP 75.31 and visual-audio MAP 73.26 at margin 0.5. Table 6, the batch-size study at margin 0.5, reports at 500 batches 74.51 and 72.98. The text states that the best configuration is 400 batches, yet the same MAP values appear in Table 5 for 500 batches. These inconsistencies must be corrected and explained.
  4. [Section 4.2(4) and Table 2] No measure of variability is reported. The MV-10K improvements over C-DCCA are 1.55 and 1.24 percentage points, which may be within fold-to-fold noise. The authors should report per-fold results with standard deviations and perform a paired significance test across the five folds.
minor comments (6)
  1. [Section 3.1, Eq. (1)] Equation (1) omits the squares in the denominator: the cosine distance should be 1 - sum(x_k y_k) / (sqrt(sum x_k^2) sqrt(sum y_k^2)).
  2. [Table 2] The MV-10K visual-to-audio value for LSTM_C_CCA is written as "18,89" with a comma; it should be 18.89.
  3. [Section 4.1] The list of VEGAS categories includes "rail transport" twice, although ten categories are claimed.
  4. [Section 4.3 and Table 2] The baseline LSTM_C_CCA appears in Table 2 and is mentioned in Section 4.3, but no reference, architecture description, or training procedure is given.
  5. [Section 4.5.4] There is an arithmetic slip: 6*63*70 is 26,460, not 264,640 as stated.
  6. [Section 3.4, Eqs. (7)-(8)] The indices j and k are not defined under the summation; the triplet condition should be written with explicit (i,j,k) triples.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation; only minor non-load-bearing self-citations in dataset and baseline choices.

full rationale

TNN-C-CCA is presented as an empirical architecture: Cluster-CCA produces 10-D embeddings that are then refined by a deep triplet network with cosine triplet loss (Eq. 8), and performance is measured by MAP/PRC under 5-fold cross-validation on VEGAS and MV-10K. The headline numbers in Table 2 come from held-out test folds, so the central claim is not equivalent to any fitted input by construction. The paper's only self-references are the MV-10K dataset ("MV-10K Dataset refers to our previous work [57]"), the C-DCCA comparator [55] by co-author Yi Yu, and the LSTM_C_CCA baseline appearing in Table 2; these are benchmarks or comparators, not load-bearing derivations, and no equation in Section 3 defines the proposed output in terms of the reported MAP. The Section 4.5.4 parameter sweeps over margin, batch number, and correlation components are reported directly on VEGAS and may inflate the proposed method's MAP relative to baselines that receive no comparable sweep, but that is an evaluation-protocol concern rather than an equation-level reduction of a prediction to its input; under the hard rules in the reviewing instructions it is not classified as circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the chosen feature extractors, the category structure of the datasets, and the assumption that the linear 10-dimensional Cluster-CCA projection preserves enough information for the triplet network to refine. No new entities such as particles, forces, or dimensions are introduced; the method is a supervised pipeline over published features and CCA variants.

free parameters (3)
  • Correlation component (Cluster-CCA output dimension) = 10
    Selected because Fig. 10 shows best MAP at 10; this is a hyperparameter chosen on the test set.
  • Triplet loss margin alpha = 0.5
    Chosen from a sweep over 0.1 to 1.1 in Table 5 based on test MAP; the claimed performance depends on this value.
  • Number of batches (inverse batch size) = 400 (batch size about 55)
    Chosen from a sweep over 300 to 900 in Table 6 based on test MAP; Table 6 reports this as best, though Table 5 attributes the same MAP to 500 batches.
assumptions (4)
  • domain assumption Pre-trained VGGish and Inception V3 features, averaged over frames, are sufficient semantic representations of audio and video.
    Section 3.2 fixes these as inputs; if they lack discriminative audio-visual alignment, the downstream pipeline cannot succeed.
  • domain assumption The ten semantic categories in VEGAS and MV-10K are consistently aligned between audio and visual content.
    Section 4.1 defines relevance by category labels; noisy labels in MV-10K are acknowledged by the authors and weaken this assumption.
  • domain assumption Cluster-CCA's expansion to all within-category cross-modal pairs preserves the category structure needed for retrieval.
    Section 3.3 relies on [25]; the one-to-one correspondence construction defines the embedding that the triplet network refines.
  • domain assumption Triplet loss with cosine distance and a fixed margin produces a ranking that transfers to retrieval MAP.
    Section 3.4 adopts the FaceNet-style loss; this is a heuristic, not a theorem, and no derivation links it to MAP.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Deep Triplet Neural Networks with Cluster-CCA for Audio-Visual Cross-modal Retrieval." pith.science (2026). https://pith.science/paper/EQI4KFJ4

@misc{pith2026190803737,
  author       = {Pith},
  title        = {Pith review of: Deep Triplet Neural Networks with Cluster-CCA for Audio-Visual Cross-modal Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EQI4KFJ4}},
  note         = {Machine review of arXiv:1908.03737}
}
read the original abstract

Cross-modal retrieval aims to retrieve data in one modality by a query in another modality, which has been a very interesting research issue in the field of multimedia, information retrieval, and computer vision, and database. Most existing works focus on cross-modal retrieval between text-image, text-video, and lyrics-audio.Little research addresses cross-modal retrieval between audio and video due to limited audio-video paired datasets and semantic information. The main challenge of audio-visual cross-modal retrieval task focuses on learning joint embeddings from a shared subspace for computing the similarity across different modalities, where generating new representations is to maximize the correlation between audio and visual modalities space. In this work, we propose a novel deep triplet neural network with cluster canonical correlation analysis(TNN-C-CCA), which is an end-to-end supervised learning architecture with audio branch and video branch.We not only consider the matching pairs in the common space but also compute the mismatching pairs when maximizing the correlation. In particular, two significant contributions are made: i) a better representation by constructing deep triplet neural network with triplet loss for optimal projections can be generated to maximize correlation in the shared subspace. ii) positive examples and negative examples are used in the learning stage to improve the capability of embedding learning between audio and video. Our experiment is run over 5-fold cross-validation, where average performance is applied to demonstrate the performance of audio-video cross-modal retrieval. The experimental results achieved on two different audio-visual datasets show the proposed learning architecture with two branches outperforms existing six CCA-based methods and four state-of-the-art based cross-modal retrieval methods.

Figures

Figures reproduced from arXiv: 1908.03737 by the authors.

Figure 1
Figure 1. The overview of audio-to-visual cross-modal retrieval. Given an audio to find similar visual contents, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The overall framework of our TNN-C-CCA model. It consists of two parts: feature extraction and [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. (a) and (b) show the traditional triplet loss minimizes the Euclidean distance between (anchor, positive) [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: The PRC achieved on the VEGAS dataset with our TNN-C-CCA model and other eight different [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: The PRC achieved on the MV-10K dataset with nine different models. The left figure is for audio-to [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Given an Anchor-Positive pair with its angle <A, P>, those negative samples having the same modality [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: The visualization of the two learned subspace with the t-SNE plot, shows audio, visual and audio [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: The visualized audio-visual retrieval results of our proposed method and other three best methods, [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: The left figure is the MAP curve of TNN-C-CCA and C-DCCA on batch number range from 300 to 700 [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: The MAP curve of the correlation component changes from 10 to 50, the corner point in the curve [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Learning Joint Embedding for Cross-Modal Retrieval

    cs.IR 2019-08 conditional novelty 4.0 of 10

    Adding triplet neural networks to supervised deep CCA raises audio-visual retrieval mean average precision from 70.34% to 72.69% on VEGAS.

Reference graph

Works this paper leans on

62 extracted references · 43 canonical work pages · cited by 1 Pith paper

  1. [1]

    Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Apostol (Paul) Natsev, George Toderici, Balakrishnan Varadara- jan, and Sudheendra Vijayanarasimhan. 2016. YouTube-8M: A Large-Scale Video Classification Benchmark. In arXiv:1609.08675. https://arxiv.org/pdf/1609.08675v1.pdf

    arXiv 2016

  2. [2]

    Bilmes, and Karen Livescu

    Galen Andrew, Raman Arora, Jeff A. Bilmes, and Karen Livescu. 2013. Deep Canonical Correlation Analysis. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013 . pp.1247–1255. https://doi.org/10.4324/9781315755649-3

    work page doi:10.4324/9781315755649-3 2013

  3. [3]

    Henriques, Andrea Vedaldi, and Philip H

    Luca Bertinetto, Jack Valmadre, João F. Henriques, Andrea Vedaldi, and Philip H. S. Torr. 2016. Fully-Convolutional Siamese Networks for Object Tracking. In Computer Vision - ECCV 2016 Workshops - Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part II . pp.850–865. https://doi.org/10.1007/978-3-319-48881-3_56

    doi:10.1007/978-3-319-48881-3_56 2016

  4. [4]

    Corinna Cortes and Vladimir Vapnik. 1995. Support-Vector Networks.Machine Learning. Vol.20, no.3 (1995), pp.273–297. https://doi.org/10.1007/BF00994018

    doi:10.1007/bf00994018 1995

  5. [5]

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. 2009. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA . pp.248–255. https://doi.org/10.1109/CVPRW.2009.5206848

    arXiv 2009

  6. [6]

    Fangxiang Feng, Xiaojie Wang, Ruifan Li, and Ibrar Ahmad. 2015. Correspondence Autoencoders for Cross-Modal Retrieval. TOMCCAP Vol.12, no.1s (2015), pp.26:1–26:22. https://doi.org/10.1145/2808205

    work page doi:10.1145/2808205 2015

  7. [7]

    Wen Gu, Xiaoyan Gu, Jingzi Gu, Bo Li, Zhi Xiong, and Weiping Wang. 2019. Adversary Guided Asymmetric Hashing for Cross-Modal Retrieval. In Proceedings of the 2019 on International Conference on Multimedia Retrieval, ICMR 2019, Ottawa, ON, Canada, June 10-13, 2019. 159–167. https://doi.org/10.1145/3323873.3325045

    arXiv 2019

  8. [8]

    Hardoon, Sándor Szedmák, and John Shawe-Taylor

    David R. Hardoon, Sándor Szedmák, and John Shawe-Taylor. 2004. Canonical Correlation Analysis: An Overview with Application to Learning Methods. Neural Computation. Vol.16, no.12 (2004), pp.2639–2664. https://doi.org/10.1162/ 0899766042321814

    work page 2004

Show all 62 references
  1. [9]

    Li He, Xing Xu, Huimin Lu, Yang Yang, Fumin Shen, and Heng Tao Shen. 2017. Unsupervised cross-modal retrieval through adversarial learning. In 2017 IEEE International Conference on Multimedia and Expo, ICME 2017, Hong Kong, China, July 10-14, 2017 . pp.1153–1158. https://doi.o...

    2017

  2. [10]

    Alexander Hermans, Lucas Beyer, and Bastian Leibe. 2017. In Defense of the Triplet Loss for Person Re-Identification. CoRR abs/1703.07737 (2017). arXiv:1703.07737 http://arxiv.org/abs/1703.07737

    2017 arXiv

  3. [11]

    Mengqiu Hu, Yang Yang, Fumin Shen, Ning Xie, Richang Hong, and Heng Tao Shen. 2019. Collective Reconstructive Embeddings for Cross-Modal Hashing. IEEE Trans. Image Processing Vol.28, 6 (2019), pp.2770–2784. https://doi.org/10. 1109/TIP.2018.2890144

    2019

  4. [12]

    Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015 . pp.448–456

    2015

  5. [13]

    Go Irie, Hiroyuki Arai, and Yukinobu Taniguchi. 2015. Alternating Co-Quantization for Cross-Modal Hashing. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015 . pp.1886–1894. https://doi.org/10.1109/ICCV.2015.219

    2015 doi

  6. [14]

    Yanli Ji, Yuhan Hu, Yang Yang, Fumin Shen, and Heng Tao Shen. 2019. Cross-domain facial expression recognition via an intra-category common feature and inter-category Distinction feature fusion network. Neurocomputing Vol.333 ACM Trans. Multimedia Comput. Commun. Appl., Vol. 1...

    2019 doi

  7. [15]

    Andrej Karpathy, Armand Joulin, and Fei-Fei Li. 2014. Deep Fragment Embeddings for Bidirectional Image Sentence Mapping. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Queb...

    2014

  8. [16]

    Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. 2015. Siamese Neural Networks for One-shot Image Recognition. In ICML Deep Learning Workshop, Vol. 7, no.11. pp.956–963. https://doi.org/10.1017/CBO9781107415324.004

    2015 doi

  9. [17]

    Pei Ling Lai and Colin Fyfe. 2000. Kernel and Nonlinear Canonical Correlation Analysis. Int. J. Neural Syst. Vol.10, no.5 (2000), pp.365–377. https://doi.org/10.1142/S012906570000034X

    2000 doi

  10. [18]

    Dongge Li, Nevenka Dimitrova, Mingkun Li, and Ishwar K. Sethi. 2003. Multimedia content processing through cross-modal association. In Proceedings of the Eleventh ACM International Conference on Multimedia, Berkeley, CA, USA, November 2-8, 2003. pp.604–611. https://doi.org/10....

    2003

  11. [19]

    Manmatha, Chao-Yuan Wu, Alexander J

    R. Manmatha, Chao-Yuan Wu, Alexander J. Smola, and Philipp Krähenbühl. 2017. Sampling Matters in Deep Embedding Learning. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017. pp.2859–2867. https://doi.org/10.1109/ICCV.2017.309

    2017 doi

  12. [20]

    Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y. Ng. 2011. Multimodal Deep Learning. In Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, June 28 - July 2, 2011 . pp.689–696

    2011

  13. [21]

    Yuxin Peng, Xin Huang, and Jinwei Qi. 2016. Cross-Media Shared Representation by Hierarchical Learning with Multiple Deep Networks. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016 . pp....

    2016

  14. [22]

    Yuxin Peng and Jinwei Qi. 2019. CM-GANs: cross-modal generative adversarial networks for common representation learning. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) Vol.15, 1 (2019), pp.22

    2019

  15. [23]

    Yuxin Peng, Jinwei Qi, and Yuxin Yuan. 2018. Modality-specific cross-modal similarity measurement with recurrent attention network. IEEE Transactions on Image Processing Vol.27, 11 (2018), pp.5585–5599

    2018

  16. [24]

    Viresh Ranjan, Nikhil Rasiwasia, and C. V. Jawahar. 2015. Multi-label Cross-Modal Retrieval. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015 . pp.4094–4102. https://doi.org/10.1109/ ICCV.2015.466

    2015

  17. [25]

    Nikhil Rasiwasia, Dhruv Mahajan, Vijay Mahadevan, and Gaurav Aggarwal. 2014. Cluster Canonical Correlation Analysis. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, AISTATS 2014, Reykjavik, Iceland, April 22-25, 2014 . pp.8...

    2014 doi

  18. [26]

    Nikhil Rasiwasia, Jose Costa Pereira, Emanuele Coviello, Gabriel Doyle, Gert R. G. Lanckriet, Roger Levy, and Nuno Vasconcelos. 2010. A new approach to cross-modal multimedia retrieval. In Proceedings of the 18th International Conference on Multimedia 2010, Firenze, Italy, Oct...

    2010

  19. [27]

    Mohammad Rastegari, Jonghyun Choi, Shobeir Fakhraei, Hal Daumé III, and Larry S. Davis. 2013. Predictable Dual- View Hashing. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013. 1328–1336. http://proceedings.ml...

    2013

  20. [28]

    Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. FaceNet: A unified embedding for face recognition and clustering. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12,

    2015

  21. [29]

    Rajiv Ratn Shah, Yi Yu, and Roger Zimmermann. 2014. ADVISOR: Personalized Video Soundtrack Recommendation by Late Fusion with Heuristic Rankings. In Proceedings of the ACM International Conference on Multimedia, MM ’14, Orlando, FL, USA, November 03-07, 2014 . pp.607–616. http...

    2014

  22. [30]

    Abhishek Sharma, Abhishek Kumar, Hal Daume, and David W Jacobs. 2012. Generalized multiview analysis: A discriminative latent space. In 2012 IEEE Conference on Computer Vision and Pattern Recognition . IEEE, pp.2160–2167

    2012

  23. [31]

    Abhishek Sharma, Abhishek Kumar, Hal Daumé III, and David W. Jacobs. 2012. Generalized Multiview Analysis: A discriminative latent space. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, June 16-21, 2012. pp.2160–2167. https://doi.org/10...

    2012

  24. [32]

    Blake Shaw, Bert Huang, and Tony Jebara. 2011. Learning a Distance Metric from a Network. In Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12-14 December 2011, Granada, ...

    2011

  25. [33]

    Chen Shen, Zhongming Jin, Wenqing Chu, Rongxin Jiang, Yaowu Chen, Guo-Jun Qi, and Xian-Sheng Hua. 2019. Multi-level Similarity Perception Network for Person Re-identification. TOMM 15, 2 (2019), 32:1–32:19. https: //doi.org/10.1145/3309881

    2019 doi

  26. [34]

    Xiaoxiao Shi and Philip S. Yu. 2012. Dimensionality Reduction on Heterogeneous Feature Space. In 12th IEEE International Conference on Data Mining, ICDM 2012, Brussels, Belgium, December 10-13, 2012 . pp.635–644. https: //doi.org/10.1109/ICDM.2012.30 ACM Trans. Multimedia Comp...

    2012 doi

  27. [35]

    Josef Sivic and Andrew Zisserman. 2003. Video Google: A Text Retrieval Approach to Object Matching in Videos. In 9th IEEE International Conference on Computer Vision (ICCV 2003), 14-17 October 2003, Nice, France . pp.1470–1477. https://doi.org/10.1109/ICCV.2003.1238663

    2003 arXiv

  28. [36]

    Richard Socher and Fei-Fei Li. 2010. Connecting modalities: Semi-supervised segmentation and annotation of images using unaligned text corpora. In The Twenty-Third IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2010, San Francisco, CA, USA, 13-18 June 2010 . ...

    2010

  29. [37]

    Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A. Alemi. 2017. Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco...

    2017

  30. [38]

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2016. Rethinking the Inception Architecture for Computer Vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 . pp.28...

    2016 doi

  31. [39]

    Bokun Wang, Yang Yang, Xing Xu, Alan Hanjalic, and Heng Tao Shen. 2017. Adversarial Cross-Modal Retrieval. In Proceedings of the 2017 ACM on Multimedia Conference, MM 2017, Mountain View, CA, USA, October 23-27, 2017 . pp.154–162. https://doi.org/10.1145/3123266.3123326

    2017

  32. [40]

    Cheng Wang, Haojin Yang, and Christoph Meinel. 2015. Deep Semantic Mapping for Cross-Modal Retrieval. In 27th IEEE International Conference on Tools with Artificial Intelligence, ICTAI 2015, Vietri sul Mare, Italy, November 9-11, 2015 . pp.234–241. https://doi.org/10.1109/ICTA...

    2015 doi

  33. [41]

    Di Wang, Xinbo Gao, Xiumei Wang, Lihuo He, and Bo Yuan. 2016. Multimodal discriminative binary embedding for large-scale cross-modal retrieval. IEEE Transactions on Image Processing Vol.25, 10 (2016), pp.4540–4554

    2016

  34. [42]

    Jian Wang, Yonghao He, Cuicui Kang, Shiming Xiang, and Chunhong Pan. 2015. Image-Text Cross-Modal Retrieval via Modality-Specific Feature Learning. In Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, Shanghai, China, June 23-26, 2015 . pp.347–354...

    2015

  35. [43]

    Kaiye Wang, Ran He, Wei Wang, Liang Wang, and Tieniu Tan. 2013. Learning Coupled Feature Spaces for Cross-Modal Matching. In IEEE International Conference on Computer Vision, ICCV 2013, Sydney, Australia, December 1-8, 2013 . pp.2088–2095. https://doi.org/10.1109/ICCV.2013.261

    2013 doi

  36. [44]

    Liwei Wang, Yin Li, and Svetlana Lazebnik. 2016. Learning Deep Structure-Preserving Image-Text Embeddings. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 . pp.5005–5013. https://doi.org/10.1109/CVPR.2016.541

    2016 doi

  37. [45]

    Zhangcheng Wang, Ya Li, Richang Hong, and Xinmei Tian. 2019. Eigenvector-Based Distance Metric Learning for Image Classification and Retrieval. TOMM 15, 3 (2019), 84:1–84:19. https://doi.org/10.1145/3340262

    2019 doi

  38. [46]

    Chang Xu, Dacheng Tao, and Chao Xu. 2013. A survey on multi-view learning. Neural Comput. Appl. Vol.23 (2013), pp.2031–2038

    2013

  39. [47]

    Xing Xu, Li He, Huimin Lu, Lianli Gao, and Yanli Ji. 2019. Deep adversarial metric learning for cross-modal retrieval. World Wide Web Vol.22, 2 (2019), pp.657–672

    2019

  40. [48]

    Xing Xu, Li He, Atsushi Shimada, Rin-ichiro Taniguchi, and Huimin Lu. 2016. Learning unified binary codes for cross-modal retrieval via latent semantic hashing. Neurocomputing Vol.213 (2016), pp.191–203. https://doi.org/10.1016/ j.neucom.2015.11.133

    2016

  41. [49]

    X. Xu, H. Lu, J. Song, Y. Yang, H. T. Shen, and X. Li. 2019. Ternary Adversarial Networks With Self-Supervision for Zero-Shot Cross-Modal Retrieval. IEEE Transactions on Cybernetics Vol.49, 7 (2019), pp.1–14. https://doi.org/10.1109/ TCYB.2019.2928180

    2019

  42. [50]

    Xing Xu, Fumin Shen, Yang Yang, Heng Tao Shen, and Xuelong Li. 2017. Learning discriminative binary codes for large-scale cross-modal retrieval. IEEE Transactions on Image Processing Vol.26, 5 (2017), pp.2494–2507

    2017

  43. [51]

    Fei Yan and Krystian Mikolajczyk. 2015. Deep correlation for matching images and text. InIEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015 . pp.3441–3450. https://doi.org/10.1109/ CVPR.2015.7298966

    2015

  44. [52]

    Yan Yan, Feiping Nie, Wen Li, Chenqiang Gao, Yi Yang, and Dong Xu. 2016. Image Classification by Cross-Media Active Learning With Privileged Information. IEEE Trans. Multimedia Vol.18, 12 (2016), pp.2494–2502. https://doi.org/ 10.1109/TMM.2016.2602938

    2016

  45. [53]

    Yi Yang, Feiping Nie, Dong Xu, Jiebo Luo, Yueting Zhuang, and Yunhe Pan. 2012. A Multimedia Retrieval Framework Based on Semi-Supervised Ranking and Relevance Feedback. IEEE Trans. Pattern Anal. Mach. Intell. Vol.34, 4 (2012), pp.723–742. https://doi.org/10.1109/10.1109/TPAMI.2011.170

    2012 doi

  46. [54]

    Ting Yao, Tao Mei, and Chong-Wah Ngo. 2015. Learning query and image similarities with ranking canonical correlation analysis. In Proceedings of the IEEE International Conference on Computer Vision . pp.28–36

    2015

  47. [55]

    Yi Yu, Suhua Tang, Kiyoharu Aizawa, and Akiko Aizawa. 2018. Category-based deep CCA for fine-grained venue discovery from multimodal data. IEEE transactions on neural networks and learning systems. Vol.30, no.99 (2018), pp.1–9. ACM Trans. Multimedia Comput. Commun. Appl., Vol....

    2018

  48. [56]

    Yi Yu, Suhua Tang, Francisco Raposo, and Lei Chen. 2019. Deep Cross-Modal Correlation Learning for Audio and Lyrics in Music Retrieval. TOMCCAP. Vol.15, no.1 (2019), pp.20:1–20:16. https://doi.org/10.1145/3281746

    2019 doi

  49. [57]

    Donghuo Zeng, Yi Yu, and Keizo Oyama. 2018. Audio-Visual Embedding for Cross-Modal Music Video Retrieval through Supervised Deep CCA. In 2018 IEEE International Symposium on Multimedia, ISM 2018, Taichung, Taiwan, December 10-12, 2018. pp.143–150. https://doi.org/10.1109/ISM.2...

    2018 doi

  50. [58]

    Jian Zhang, Yuxin Peng, and Mingkuan Yuan. 2018. Unsupervised Generative Adversarial Cross-Modal Hashing. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8...

    2018

  51. [59]

    Yin Zheng, Yu-Jin Zhang, and Hugo Larochelle. 2014. Topic Modeling of Multimodal Data: An Autoregressive Approach. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014 . pp.1370–1377. https://doi.org/10.1109/CVPR.2014.178

    2014 doi

  52. [60]

    Yipin Zhou, Zhaowen Wang, Chen Fang, Trung Bui, and Tamara L. Berg. 2018. Visual to Sound: Generating Natural Sound for Videos in the Wild. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 . pp.3550–3558. ...

    2018

  53. [61]

    Hauptmann

    Linchao Zhu, Zhongwen Xu, Yi Yang, and Alexander G. Hauptmann. 2017. Uncovering the Temporal Context for Video Question Answering. International Journal of Computer Vision Vol.124, 3 (01 Sep 2017), pp.409–421. https: //doi.org/10.1007/s11263-017-1033-7 Received Oct 2019; revis...

    2017 doi

  54. [2015]

    https://doi.org/10.1109/CVPR.2015.7298682

    pp.815–823. https://doi.org/10.1109/CVPR.2015.7298682

    2015

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.