Do Cross Modal Systems Leverage Semantic Relationships?

Alessandro Calefati; Arif Mahmood; Faisal Shafait; Ignazio Gallo; Muhammad Kamran Janjua; Shah Nawaz

REVIEW 4 major objections 5 minor 57 references

Do Cross Modal Systems Leverage Semantic Relationships?

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Cross-modal retrieval systems should be judged by semantic similarity, not just paired captions.

desk verdict SemanticMap is trivially gameable by center-loss collapse, so the paper's central evaluation claim fails; the text-as-image single-stream trick and the motivation about R@K are worth a look. read the letter →

arxiv 1909.01976 v1 pith:LYUBM57L submitted 2019-09-03 cs.CV

Shah Nawaz , Muhammad Kamran Janjua , Ignazio Gallo , Arif Mahmood , Alessandro Calefati , Faisal Shafait This is my paper · ORCID

classification cs.CV

keywords cross-modalretrievalsemanticsimilarityRecall@KMapsingle-streamnetworkcenterlosstextencodingasimagesimage-text

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the standard Recall-at-K ($R@K$) metric misreports how well cross-modal retrieval systems work, because it only rewards hits against manually annotated image-caption pairs and ignores retrievals that are semantically related but appear in different pairs. To correct this, it proposes SemanticMap ($\lambda@K$), a measure that averages the cosine similarity between query and retrieved representations in the latent embedding space. The paper also introduces a single-stream network that encodes text captions as images and trains with an extended center loss, so one network handles both image-to-text and text-to-image retrieval. Evaluated on MSCOCO and Flickr30K, the system scores lower than the state of the art on $R@K$ but reports higher SemanticMap scores than the best $R@K$ method on the MSCOCO non-pair evaluation, with comparable results on Flickr30K. The paper concludes that retrieval systems should be compared by how well they bridge semantic relationships, not by whether they hit ground-truth pairs.

What carries the argument

The central object is SemanticMap, built from the cosine similarity between an image representation $X^l$ and a text representation $Y^m$ in the latent space, $\lambda(X^l,Y^m) = \frac{\sum_i x_i^l y_i^m}{\sqrt{\sum_i (x_i^l)^2}\sqrt{\sum_i (y_i^m)^2}}$, aggregated over the top-$K$ retrieved items as $\lambda@K = \frac{1}{NK}\sum_{l=1}^{c}\sum_{m=1}^{K}\lambda(X^l,Y^m)$. This aggregated similarity carries the paper's evaluative claim: it gives credit to any semantically close retrieval, not only the ground-truth pair. The retrieval system is carried by three supporting mechanisms: encoding each caption as an image from word-vector embeddings, running both images and encoded captions through a single deep convolutional network, and training with an extended center loss that minimizes the distance of each feature vector from its class center, so semantically related images and captions land near each other in the embedding.

What would settle it

Compute SemanticMap for a deliberately degenerate model that maps every image and every caption to the same constant vector; if $\lambda@K$ approaches 1 while $R@K$ stays near 0, the metric cannot separate semantic alignment from representation collapse.

Watch

Extended reading notes

Core claim

The core claim is that the standard Recall-at-K ($R@K$) evaluation is inadequate for cross-modal retrieval because it treats only manually marked image-text pairs as correct and therefore does not reward semantically related retrievals. The paper introduces SemanticMap ($\lambda@K$), which averages the cosine similarity between a query representation and the top-$K$ retrieved representations in the shared latent space, and argues that this measures semantic relationship rather than pair membership. It further proposes a single-stream network in which captions are encoded as images using word-vector embeddings, allowing one deep network to embed both modalities, trained with an extended center loss that pulls image and caption features toward shared class centers. In the authors' comparison, the proposed system has lower $R@K$ than Structure Preserving, the strongest existing method on that metric, but under $\lambda@K$ it achieves higher scores on the MSCOCO evaluation, especially when pairs are excluded, which the authors take as evidence of better semantic generalization. The paper also reports a human baseline indicating that many retrievals counted as misses by $R@K$ are judged semantically related by human annotators.

Load-bearing premise

The load-bearing premise is that cosine similarity in the learned embedding space is a faithful measure of semantic relatedness, even though the same space is trained by pulling every image and caption of a class toward a single point, so a degenerate model that collapses classes could receive high SemanticMap scores without understanding meaning.

Editorial extensions

If this is right

Adopting SemanticMap would change comparative rankings: a model with lower pair-recall can be judged better at semantic retrieval, as the paper demonstrates against the strongest $R@K$ baseline on MSCOCO.
Retrieval systems could be evaluated on open-domain data without reliable ground-truth pairs, using embedding proximity as the correctness signal.
Encoding text descriptions as images lets existing image-classification networks be reused for text, reducing the memory footprint of cross-modal systems.
The non-pair $\lambda@K$ comparison indicates that networks trained only on ground-truth pairs generalize less well to semantically similar but unpaired queries.
Human expert ratings of $R@K$-missed queries provide evidence that a substantial portion of retrieval misses are semantically reasonable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves implicit that $\lambda@K$ needs a degeneracy check: because center loss pulls every member of a class toward one point, a model that collapses all features could inflate $\lambda@K$ without any semantic understanding, so the metric should be reported alongside a trivial-embedding baseline.
The same text-as-image encoding and center-loss recipe could be carried to other modality pairs, such as audio-to-image or video-to-text, where paired annotations are even noisier.
A combined evaluation that reports both $R@K$ and $\lambda@K$ would separate exact pair matching from semantic proximity, giving practitioners two complementary views of a retrieval system.
Applying $\lambda@K$ to already-published embeddings might re-order the state of the art, since systems optimized for pair recall are not necessarily optimized for embedding-space semantic proximity.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the paper is a good example of a real problem—R@K ignores semantically related retrieval hits—but the proposed fix, SemanticMap, is not a valid measure, and the model's R@K results are not actually comparable to the state of the art.

What's new: they encode text as a Word2Vec image and feed it to a single InceptionResNet stream, which is a neat extension of Gallo et al. The motivation is fair: pairwise ground truth misses near-misses that a human would accept. The human baseline idea is also worth credit, even if the execution is thin.

The soft spots are structural. λ@K is the mean cosine similarity between query and top-K neighbors in the learned space (Eq. 1–2). There is no chance-level calibration; a model that outputs a constant vector for every input scores 1.0. And their own training objective, extended center loss (Eq. 3–4), pulls all members of a class toward a single center, which directly inflates those cosine similarities. So the MSCOCO advantage over Structure Preserving in Table 2 is largely the training loss doing what it was designed to do. The opposite happens on Flickr30K, where their λ@K is worse than Structure Preserving's, yet Section 5.4 claims better performance without qualification. That inconsistency should have been caught by the authors.

The R@K story is also overstated. Their best MSCOCO numbers (cfg-3: 40.0 i2t R@1, 30.9 t2i) are well below Structure Preserving's 50.1/39.6, so 'comparable' is generous. No error bars, no code, no data—so the comparisons are hard to trust. The human baseline reports R@10 for their own model and hit rates from three annotators, but never computes λ@K on those same queries, so it doesn't validate the metric as advertised.

Bottom line: the paper has a plausible motivation and a cute architectural trick, but the central evaluation metric is gameable and the model is not competitive on the existing measure. I would not cite the SemanticMap result, but I would not desk reject the paper either: the question of how to evaluate semantic relatedness in retrieval is important, and a serious referee can articulate exactly why the current formulation fails and what a calibrated version would need (e.g., null-model normalization, correlation with human judgments). Expect the paper to be rejected, but the conversation is worth having.

Referee Report

4 major / 5 minor

Summary. The paper argues that the standard R@K metric for cross-modal retrieval is inadequate because it strictly follows manually marked image-text pairs and does not exploit semantic relationships. It proposes a new metric, SemanticMap (λ@K), defined in Eqs. (1)-(2) as the mean cosine similarity between a query and its top-K retrieved items in the learned embedding space. The paper also proposes a single-stream network that encodes text descriptions as images and trains with an extended center loss (Eqs. (3)-(4)). The system is evaluated on MSCOCO and Flickr30K, with R@K results in Table 1, λ@K comparisons against Structure Preserving in Table 2, a restricted λ@K evaluation in Table 3, a human baseline in Section 6, and an ablation study in Section 7.

Significance. If the proposed metric were valid, it would address a genuine limitation of R@K, since semantically related but not manually paired retrievals are currently scored as errors. The single-stream architecture with text encoded as images is also an interesting memory-saving direction. However, as presented, the metric is not calibrated against chance, it is circular with respect to the center-loss training objective, and the paper's empirical claims are internally inconsistent: the reported λ@K superiority holds only on one dataset and contradicts the unqualified statement in Section 5.4, while the headline R@K results are far below the cited state of the art. The human baseline does not validate the metric. No error bars or null-model comparisons are reported, so the central evaluation-metric contribution and the model-superiority claim are unsupported.

major comments (4)

[Section 3.1, Eqs. (1)-(2)] The proposed SemanticMap metric is not calibrated against chance. Since λ(X^l, Y^m) is a cosine similarity, a degenerate model that maps every image and every text to the same fixed vector achieves λ@K = 1.0 on any retrieval list. The paper provides no null-model baseline, no random-embedding reference value, and no analysis of the range of λ@K under non-semantic models. Therefore, high λ@K values do not by themselves indicate semantic understanding, and the central claim that SemanticMap is a better evaluation measure rests on an unvalidated definition.
[Section 4.2, Eqs. (3)-(4); Table 2] The center loss in Eqs. (3)-(4) explicitly minimizes intra-class distances, pulling all image and text features of a class toward a single center. λ@K, as defined in Eq. (2), measures exactly the average cosine similarity in that same learned space. Consequently, the proposed model's higher λ@K on MSCOCO in Table 2 is an expected consequence of the training objective rather than independent evidence of better semantic bridging. This circularity is confirmed by Table 2, where on Flickr30K the proposed model is worse than Structure Preserving on every λ@K entry (e.g., 49.13 vs. 62.30 for image-to-text λ@1), directly contradicting the unqualified statement in Section 5.4 that 'our model achieves better performance than the current best approach using λ@K measure.' No error bars or significance tests are reported, so the MSCOCO advantage cannot be evaluated.
[Table 1; Abstract; Conclusion] The abstract and conclusion state that the proposed system gives 'comparable results' to state-of-the-art methods, but Table 1 shows large gaps under the paper's own baseline metric: on MSCOCO cfg-3, image-to-text R@1 is 40.0 vs. 50.1 for Structure Preserving and R@10 is 76.7 vs. 89.2; text-to-image R@1 is 30.9 vs. 39.6. Section 5.3 itself acknowledges that 'compared to the current state-of-the-art, our method performance is comparatively low.' The claim of comparable results is therefore internally inconsistent with the reported numbers.
[Section 6] The human baseline experiment does not validate SemanticMap. The procedure takes the model's top-10 retrievals, asks three human annotators whether the retrieved items are semantically relevant, and reports averaged hit rates (85.40, 85.00, 83.40) alongside the model's R@10 of 84.60. It never computes λ@K on the human-judged queries and provides no correlation between λ@K scores and human semantic judgments. Thus the claimed advantage of SemanticMap over R@K in capturing semantic similarity is not supported by the presented human baseline.

minor comments (5)

[Section 5.1] The naming of configurations is inconsistent: the text says 'conﬁg-standard' but Table 1 uses 'cfg-std', and the 'cfg-2' and 'cfg-3' names are introduced without a clear mapping to the described augmentation schemes.
[Section 5.2] The description of 'conﬁg-2' says it uses 'ten different encoded text descriptions' per image, but it is not explained how ten descriptions are derived from the five captions; please specify the augmentation procedure precisely.
[Equation (2)] The notation in Eq. (2) is unclear: the summation over classes c and the definition of N should be stated precisely, because the numerator and denominator are not dimensionally matched as written.
[Table 4] The ablation table reports only R@10 for the ablation model without the corresponding cfg-3 R@10 in the same table, forcing the reader to cross-reference Table 1; no variance or repeated-run statistics are provided.
[Figure 2 caption] The caption states that R@5 = 0 but shows five retrieved texts; it should explicitly clarify that the ground-truth paired caption is not among the five, since otherwise the R@5 = 0 claim is confusing.

Circularity Check

1 steps flagged · score 6.0 of 10

SemanticMap λ@K measures the same intra-class similarity that the proposed center-loss training is designed to maximize, so the claimed semantic advantage is substantially forced by the training objective.

fitted input called prediction [Section 3.1, Eq. (1)-(2); Section 4.2, Eq. (3)-(4); Section 5.4, Table 2]
"λ@K = 1/N·K ∑_{l=1}^c ∑_{m=1}^K λ(X^l,Y^m) (2) ... d(f_c) = ∑_{i=1}^{n_t+n_i} ‖f_i^c − 1/(n_t+n_i) ∑_{j=1}^{n_t+n_i} f_j^c‖_2^2 (3) ... L(mini batch) = 1/2 ∑_{c=1}^m d(f_c) (4) ... we show that our model achieves better performance than the current best approach using λ@K measure."

λ@K (Eq. 2) averages the cosine similarity (Eq. 1) between a query embedding and its top-K retrieved embeddings. The proposed training loss (Eqs. 3-4) is a center loss that explicitly minimizes the distance of each image/text feature from its class center, collapsing same-class embeddings and inflating pairwise cosine similarities. Thus the model is trained to maximize the very quantity λ@K evaluates. The Section 5.4 claim of better performance than Structure Preserving under λ@K is therefore not independent: higher MSCOCO λ@K scores are the expected product of the center-loss objective, and a constant-embedding model would attain λ@K=1.0. No null-model baseline, error bars, or human correlation for λ@K is provided; the Section 6 human study reports R@10 hit rates, not λ@K.

full rationale

The network architecture and R@K results are reported in a self-contained way, so the system derivation itself is not formally circular. The circularity is confined to the evaluation claim. SemanticMap is defined as mean cosine similarity in the embedding space (Eq. 1-2), and the proposed extended center loss (Eq. 3-4) is explicitly designed to pull image and text features of a class toward a common center. A model trained this way will have artificially high cosine similarities for same-class and semantically related items, making its high λ@K scores and the Section 5.4 superiority claim substantially forced by the training objective rather than by an independent semantic benchmark. The paper supplies no null-model baseline, no error bars, and no human-judgment validation of λ@K; the human experiment in Section 6 evaluates R@10 hits, not λ@K. Moreover, Table 2 shows the proposed model below Structure Preserving under λ@K on Flickr30K, contradicting the unqualified Section 5.4 claim. Because the training loss (Euclidean center loss) and the metric (pairwise cosine similarity) are not formally identical, the circularity is partial, not total; hence score 6 rather than 8.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the domain assumption that cosine similarity in a learned space captures semantics, and on underspecified training hyperparameters (center loss weight and text encoding dimension). No new physical or conceptual entities are introduced.

free parameters (2)

center loss weight = not reported
The total loss is a weighted combination of center loss and softmax loss (Sections 4.2, 5.1), but the weighting factor is never specified, which is required to reproduce the training.
Word2Vec vector length for text encoding = 15
Figure 4 states the encoding uses vector length 15; this hand-chosen dimension determines the text-image input size and is not justified or swept.

assumptions (3)

domain assumption Cosine similarity in the learned embedding space is a valid measure of semantic relatedness between an image and a text description.
The entire SemanticMap metric rests on this; introduced in Eq. (1)-(2) in Section 3.1.
domain assumption Encoding text descriptions as images via Word2Vec preserves semantic relationships and makes them usable by CNN image networks.
Section 4.1 builds on Gallo et al. [13], but the paper does not validate that this encoding preserves the semantics needed for cross-modal retrieval.
domain assumption The center loss, combined with softmax, learns an embedding where class centers are meaningful and intra-class compactness corresponds to semantic similarity.
This is the foundation of the proposed training objective in Section 4.2; the paper does not demonstrate that compactness in feature space aligns with human semantic judgments.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Do Cross Modal Systems Leverage Semantic Relationships?." pith.science (2026). https://pith.science/paper/LYUBM57L

@misc{pith2026190901976,
  author       = {Pith},
  title        = {Pith review of: Do Cross Modal Systems Leverage Semantic Relationships?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LYUBM57L}},
  note         = {Machine review of arXiv:1909.01976}
}

read the original abstract

Current cross-modal retrieval systems are evaluated using R@K measure which does not leverage semantic relationships rather strictly follows the manually marked image text query pairs. Therefore, current systems do not generalize well for the unseen data in the wild. To handle this, we propose a new measure, SemanticMap, to evaluate the performance of cross-modal systems. Our proposed measure evaluates the semantic similarity between the image and text representations in the latent embedding space. We also propose a novel cross-modal retrieval system using a single stream network for bidirectional retrieval. The proposed system is based on a deep neural network trained using extended center loss, minimizing the distance of image and text descriptions in the latent space from the class centers. In our system, the text descriptions are also encoded as images which enabled us to use a single stream network for both text and images. To the best of our knowledge, our work is the first of its kind in terms of employing a single stream network for cross-modal retrieval systems. The proposed system is evaluated on two publicly available datasets including MSCOCO and Flickr30K and has shown comparable results to the current state-of-the-art methods.

Figures

Figures reproduced from arXiv: 1909.01976 by the authors.

**Figure 2.** Image to text retrieval result (top 5) from the MSCOCO dataset. Query and retrieved text descriptions are semantically related but not in pairs and thus R@5 = 0. In the last column the proposed SemanticMap (λ) scores are reported resulting λ@5 = 0.79 showing high semantic similarity. some distance measure is minimized between the learned representations in the latent space. Once these representations are obtained, … view at source ↗

**Figure 3.** An example of text to image retrieval result (top [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: The word “tablet” is encoded into an image using Word2Vec encoding with vector length 15. Consecutive words in the text descriptions are encoded as image preserving relative position of each word. Note that words that occur in similar context will have similar embedd…

**Figure 5.** Figure 5: In the first phase, the proposed cross modal retrieval system transforms word embedding into encoded text which [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

**Figure 6.** Figure 6: Embedding of MSCOCO test set in latent space visualized using t-SNE [47]. The semantically similar text feature [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 52 canonical work pages

[1]

Antol, A

S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Lawrence Zitnick, and D. Parikh. Vqa: Visual question answering. pages 2425–2433, 2015

work page 2015
[2]

Arandjelovic and A

R. Arandjelovic and A. Zisserman. Look, listen and learn. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 609–617, 2017

work page 2017
[3]

Bromley, I

J. Bromley, I. Guyon, Y . LeCun, E. S¨ackinger, and R. Shah. Signature veriﬁcation using a” siamese” time delay neural network. In Advances in neural information processing sys- tems, pages 737–744, 1994

work page 1994
[4]

Chrupała, L

G. Chrupała, L. Gelderloos, and A. Alishahi. Representa- tions of language in a model of visually grounded speech signal. Association for Computational Linguistics, 2017

work page 2017
[5]

Chung, C

J. Chung, C. Gulcehre, K. Cho, and Y . Bengio. Gated feed- back recurrent neural networks. In International Conference on Machine Learning, pages 2067–2075, 2015

work page 2015
[6]

J. S. Chung, A. Nagrani, and A. Zisserman. V oxceleb2: Deep speaker recognition. pages 1086–1090, 2018

work page 2018
[7]

Learning Deep Representations of Medical Images using Siamese CNNs with Application to Content-Based Image Retrieval

Y .-A. Chung and W.-H. Weng. Learning deep represen- tations of medical images using siamese cnns with appli- cation to content-based image retrieval. arXiv preprint arXiv:1711.08490, 2017

work page Pith review arXiv 2017
[8]

S. Dey, A. Dutta, S. K. Ghosh, E. Valveny, J. Llad ´os, and U. Pal. Learning cross-modal deep embeddings for multi- object image retrieval using text and sketch. In 2018 24th International Conference on Pattern Recognition (ICPR) , pages 916–921, 2018

work page 2018

Show all 57 references

[9]

Donahue, L

J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Dar- rell. Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2...

2015
[10]

Eisenschtat and L

A. Eisenschtat and L. Wolf. Linking image and text with 2-way nets. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4601–4611, 2017

2017
[11]

H. Fang, S. Gupta, F. Iandola, R. K. Srivastava, L. Deng, P. Doll´ar, J. Gao, X. He, M. Mitchell, J. C. Platt, et al. From captions to visual concepts and back. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 1473–1482, 2015

2015
[12]

Frome, G

A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, T. Mikolov, et al. Devise: A deep visual-semantic embed- ding model. In Advances in neural information processing systems, pages 2121–2129, 2013

2013
[13]

Gallo, S

I. Gallo, S. Nawaz, and A. Calefati. Semantic text encoding for text classiﬁcation using convolutional neural networks. In Document Analysis and Recognition (ICDAR), 2017 14th IAPR International Conference on, volume 5, pages 16–21, 2017

2017
[14]

H. Gao, J. Mao, J. Zhou, Z. Huang, L. Wang, and W. Xu. Are you talking to a machine? dataset and methods for mul- tilingual image question. In Advances in neural information processing systems, pages 2296–2304, 2015

2015
[15]

Y . Gong, L. Wang, M. Hodosh, J. Hockenmaier, and S. Lazebnik. Improving image-sentence embeddings using large weakly annotated photo collections. In European Con- ference on Computer Vision, pages 529–545, 2014

2014
[16]

Hadsell, S

R. Hadsell, S. Chopra, and Y . LeCun. Dimensionality reduc- tion by learning an invariant mapping. In IEEE Computer Society Conference on Computer Vision and Pattern Recog- nition, pages 1735–1742, 2006

2006
[17]

X. Han, T. Leung, Y . Jia, R. Sukthankar, and A. C. Berg. Matchnet: Unifying feature and metric learning for patch- based matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3279– 3286, 2015

2015
[18]

D. R. Hardoon, S. Szedmak, and J. Shawe-Taylor. Canonical correlation analysis: An overview with application to learn- ing methods. Neural computation, 16(12):2639–2664, 2004

2004
[19]

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learn- ing for image recognition. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 770–778, 2016

2016
[20]

Hochreiter and J

S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997

1997
[21]

Hoffer and N

E. Hoffer and N. Ailon. Deep metric learning using triplet network. In International Workshop on Similarity-Based Pattern Recognition, pages 84–92, 2015

2015
[22]

J. Hu, J. Lu, and Y .-P. Tan. Discriminative deep metric learn- ing for face veriﬁcation in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 1875–1882, 2014

2014
[23]

Huang, W

Y . Huang, W. Wang, and L. Wang. Instance-aware image and sentence matching with selective multimodal lstm. 2017

2017
[24]

Jabri, A

A. Jabri, A. Joulin, and L. van der Maaten. Revisiting visual question answering baselines. In European conference on computer vision, pages 727–739, 2016

2016
[25]

Karpathy and L

A. Karpathy and L. Fei-Fei. Deep visual-semantic align- ments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 3128–3137, 2015

2015
[26]

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. International Conference on Learning Repre- sentations, 2015

2015
[27]

Klein, G

B. Klein, G. Lev, G. Sadeh, and L. Wolf. Associating neu- ral word embeddings with deep image representations us- ing ﬁsher vectors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4437– 4446, 2015

2015
[28]

Klein, G

B. Klein, G. Lev, G. Sadeh, and L. Wolf. Fisher vectors derived from hybrid gaussian-laplacian mixture models for image annotation. Proceedings of the IEEE conference on computer vision and pattern recognition, 2015

2015
[29]

Lei Ba, K

J. Lei Ba, K. Swersky, S. Fidler, et al. Predicting deep zero- shot convolutional neural networks using textual descrip- tions. In Proceedings of the IEEE International Conference on Computer Vision, pages 4247–4255, 2015

2015
[30]

T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ra- manan, P. Doll´ar, and C. L. Zitnick. Microsoft coco: Com- mon objects in context. InEuropean conference on computer vision, pages 740–755, 2014

2014
[31]

Y . Liu, L. Liu, Y . Guo, and M. S. Lew. Learning visual and textual representations for multimodal matching and classi- ﬁcation. Pattern Recognition, 84:51–67, 2018

2018
[32]

L. Ma, Z. Lu, L. Shang, and H. Li. Multimodal convolu- tional neural networks for matching image and sentence. In Proceedings of the IEEE international conference on com- puter vision, pages 2623–2631, 2015

2015
[33]

J. Mao, W. Xu, Y . Yang, J. Wang, Z. Huang, and A. Yuille. Deep captioning with multimodal recurrent neural networks (m-rnn). International Conference on Learning Representa- tions, 2015

2015
[34]

Mensink, J

T. Mensink, J. Verbeek, F. Perronnin, and G. Csurka. Metric learning for large scale image classiﬁcation: Generalizing to new classes at near-zero cost. In Computer Vision–ECCV 2012, pages 488–501. 2012

2012
[35]

Mikolov, I

T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural informa- tion processing systems, pages 3111–3119, 2013

2013
[36]

Nagrani, S

A. Nagrani, S. Albanie, and A. Zisserman. Seeing voices and hearing faces: Cross-modal biometric matching. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, 2018

2018
[37]

Z. Niu, M. Zhou, L. Wang, X. Gao, and G. Hua. Hierarchi- cal multimodal lstm for dense visual-semantic embedding. In Computer Vision (ICCV), 2017 IEEE International Con- ference on, pages 1899–1907, 2017

2017
[38]

Park and W

G. Park and W. Im. Image-text multi-modal representa- tion learning by adversarial backpropagation. arXiv preprint arXiv:1612.08354, 2016

2016 arXiv
[39]

B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik. Flickr30k entities: Col- lecting region-to-phrase correspondences for richer image- to-sentence models. In Proceedings of the IEEE interna- tional conference on computer vision , pages 264...

2015
[40]

B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik. Flickr30k entities: Col- lecting region-to-phrase correspondences for richer image- to-sentence models. In IJCV, pages 123(1):74–93, 2017

2017
[41]

R. Qiao, L. Liu, C. Shen, and A. v. d. Hengel. Visually aligned word embeddings for improving zero-shot learning. arXiv preprint arXiv:1707.05427, 2017

2017 arXiv
[42]

Rohrbach, M

A. Rohrbach, M. Rohrbach, R. Hu, T. Darrell, and B. Schiele. Grounding of textual phrases in images by re- construction. In European Conference on Computer Vision, pages 817–834, 2016

2016
[43]

Schroff, D

F. Schroff, D. Kalenichenko, and J. Philbin. Facenet: A uniﬁed embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815–823, 2015

2015
[44]

B. Shaw, B. Huang, and T. Jebara. Learning a distance metric from a network. In Advances in Neural Information Process- ing Systems, pages 1899–1907, 2011

1907
[45]

K. Sohn, W. Shang, and H. Lee. Improved multimodal deep learning with variation of information. In Advances in Neural Information Processing Systems , pages 2141–2149, 2014

2014
[46]

Srivastava and R

N. Srivastava and R. R. Salakhutdinov. Multimodal learn- ing with deep boltzmann machines. In Advances in neural information processing systems, pages 2222–2230, 2012

2012
[47]

Van Der Maaten

L. Van Der Maaten. Accelerating t-sne using tree-based algorithms. The Journal of Machine Learning Research , 15(1):3221–3245, 2014

2014
[48]

Vendrov, R

I. Vendrov, R. Kiros, S. Fidler, and R. Urtasun. Order- embeddings of images and language. 2016

2016
[49]

Vinyals, A

O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 3156–3164, 2015

2015
[50]

J. Wang, Y . Song, T. Leung, C. Rosenberg, J. Wang, J. Philbin, B. Chen, and Y . Wu. Learning ﬁne-grained image similarity with deep ranking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1386–1393, 2014

2014
[51]

L. Wang, Y . Li, and S. Lazebnik. Learning deep structure- preserving image-text embeddings. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 5005–5013, 2016

2016
[52]

L. Wang, Y . Li, and S. Lazebnik. Learning two-branch neu- ral networks for image-text matching tasks. arXiv preprint arXiv:1704.03470, 2017

2017 arXiv
[53]

Y . Wen, K. Zhang, Z. Li, and Y . Qiao. A discriminative fea- ture learning approach for deep face recognition. In Euro- pean Conference on Computer Vision, pages 499–515, 2016

2016
[54]

Weston, S

J. Weston, S. Bengio, and N. Usunier. Wsabie: Scaling up to large vocabulary image annotation. In International Joint Conference on Artiﬁcial Intelligence, 2011

2011
[55]

K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudi- nov, R. Zemel, and Y . Bengio. Show, attend and tell: Neural image caption generation with visual attention. In Interna- tional conference on machine learning , pages 2048–2057, 2015

2015
[56]

Zhang, B

Y . Zhang, B. Gong, and M. Shah. Fast zero-shot image tag- ging. Proceedings of the IEEE conference on computer vi- sion and pattern recognition, 2016

2016
[57]

Zheng, L

Z. Zheng, L. Zheng, M. Garrett, Y . Yang, and Y .-D. Shen. Dual-path convolutional image-text embedding. arXiv preprint arXiv:1711.05535, 2017

2017 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Antol, A

S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Lawrence Zitnick, and D. Parikh. Vqa: Visual question answering. pages 2425–2433, 2015

work page 2015

[2] [2]

Arandjelovic and A

R. Arandjelovic and A. Zisserman. Look, listen and learn. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 609–617, 2017

work page 2017

[3] [3]

Bromley, I

J. Bromley, I. Guyon, Y . LeCun, E. S¨ackinger, and R. Shah. Signature veriﬁcation using a” siamese” time delay neural network. In Advances in neural information processing sys- tems, pages 737–744, 1994

work page 1994

[4] [4]

Chrupała, L

G. Chrupała, L. Gelderloos, and A. Alishahi. Representa- tions of language in a model of visually grounded speech signal. Association for Computational Linguistics, 2017

work page 2017

[5] [5]

Chung, C

J. Chung, C. Gulcehre, K. Cho, and Y . Bengio. Gated feed- back recurrent neural networks. In International Conference on Machine Learning, pages 2067–2075, 2015

work page 2015

[6] [6]

J. S. Chung, A. Nagrani, and A. Zisserman. V oxceleb2: Deep speaker recognition. pages 1086–1090, 2018

work page 2018

[7] [7]

Learning Deep Representations of Medical Images using Siamese CNNs with Application to Content-Based Image Retrieval

Y .-A. Chung and W.-H. Weng. Learning deep represen- tations of medical images using siamese cnns with appli- cation to content-based image retrieval. arXiv preprint arXiv:1711.08490, 2017

work page Pith review arXiv 2017

[8] [8]

S. Dey, A. Dutta, S. K. Ghosh, E. Valveny, J. Llad ´os, and U. Pal. Learning cross-modal deep embeddings for multi- object image retrieval using text and sketch. In 2018 24th International Conference on Pattern Recognition (ICPR) , pages 916–921, 2018

work page 2018