Pith. sign in

REVIEW 5 major objections 5 minor 68 references

Multimodal Representation Alignment for Cross-modal Information Retrieval

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read For contrastively trained vision-language models, cosine similarity is the best metric for cross-modal retrieval.

desk verdict A useful empirical comparison with a solid negative result on MLP aligners, but the arXiv abstract promises more than the experiments deliver; the body is noticeably more honest than the abstract. read the letter →

arxiv 2506.08774 v2 pith:NKVSO4L6 submitted 2025-06-10 cs.IR cs.AI

Fan Xu , Luis A. Leiva This is my paper
classification cs.IRcs.AI
keywords multimodalrepresentationalignmentcross-modalretrievalcosinesimilaritymodalitygapcontrastivelossvision-languagemodelsPrecision@K
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks how to match an image embedding to a text embedding when the two come from separately trained encoders, and which similarity metric does that best. It argues that for vision-language models trained with a contrastive objective, cosine similarity consistently outperforms Euclidean, Manhattan, Chi-square, and learned MLP-based metrics at top-K retrieval. It also argues that Wasserstein-2 distance is a complementary way to measure the gap between image and text distributions, although a smaller gap does not by itself guarantee better retrieval. Finally, it argues that MLPs trained to mimic cosine similarity cannot align representations from unimodal encoders, so contrastively trained multimodal models remain the practical route for cross-modal retrieval.

What carries the argument

The load-bearing mechanism is the shared embedding space produced by contrastive training, which creates an anisotropic geometry in which matched image-text pairs have high cosine similarity but negatives are only weakly pushed apart. The paper evaluates four standard metrics (Euclidean, Manhattan, Chi-square, cosine) plus two learned metrics: an MSE-trained MLP and an MLP trained with a custom contrastive loss of the form $L(\hat{x}_i,\hat{y}_j) = I_{ij}\frac{1}{2}(1-\hat{d}_{ij})^2 + (1-I_{ij})\frac{1}{2}[\hat{d}_{ij}+1]^2$, with $\hat{d}_{ij}$ the MLP output on concatenated embeddings and $I_{ij}$ the matching indicator. All alignments are scored by Precision@K, and Wasserstein-2 distance is computed between the image and text feature distributions as a distribution-level measure of the modality gap.

What would settle it

Rerun the retrieval comparisons on Flickr30K or MS-COCO using all five captions as positive matches instead of only the first, and check whether cosine similarity still beats Euclidean, Manhattan, and Chi-square at Precision@K; if it does not, the central ranking claim fails. A second check: compare the custom contrastive loss against MSE on a transformer-based scorer rather than an MLP, since the paper's framing predicts an advantage there that the reported MLP tables do not reveal.

Watch

Extended reading notes

Core claim

The central discovery the paper advances is that the metric used during contrastive training, rather than the geometry of the embedding space, determines which similarity measure works for retrieval. CLIP and BLIP, both trained with InfoNCE-style contrastive objectives, produce embeddings that are best ranked by cosine similarity, and this holds for text-to-image and image-to-text across IMDB, Flickr30K, and MS-COCO. A second claim is that the modality gap, traditionally measured by Euclidean distance between cluster centroids, can also be quantified by Wasserstein-2 distance, a transport cost between the image and text feature distributions, with BLIP showing the smallest gap. A third claim is negative: MLPs, even with a purpose-built contrastive loss that pulls matched pairs toward 1 and pushes unmatched pairs toward -1, fail to learn the similarity score from concatenated features, so combining unimodal encoders does not close the alignment gap.

Load-bearing premise

The experiments assume every text has exactly one correct image and vice versa, and when a dataset offers five captions per image they keep only the first; if real retrieval includes all captions, the metric rankings that support the main claim could change.

Editorial extensions

If this is right

  • For retrieval over embeddings from contrastively trained models such as CLIP and BLIP, cosine similarity should be the default ranking metric; the distance-based and learned alternatives tested here are not competitive.
  • Wasserstein-2 distance can be reported alongside centroid distance as a diagnostic of the modality gap, though it should not be read as a predictor of retrieval quality.
  • Concatenating image and text features and training an MLP to score similarity is not an effective alignment strategy; joint contrastive pretraining appears necessary.
  • Dataset annotation structure matters: with five captions per image, image-to-text Precision@K has a higher ceiling than text-to-image, so the two retrieval directions are not directly comparable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the cosine advantage is a by-product of the narrow-cone anisotropy induced by contrastive training, models trained with other contrastive objectives (for instance sigmoid-based ones) may not show the same ranking, and this can be tested directly.
  • Editorial inference: using all five captions instead of the first would change the Precision@K ceilings and may reorder the metric rankings; a recall-oriented evaluation would be a more demanding test of the paper's claim.
  • Editorial inference: the custom contrastive loss, which anchors matched pairs at 1 and unmatched pairs at -1, could be plugged into a transformer-based alignment head; the paper positions the loss as useful for both MLPs and transformers, so this is a direct next experiment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper studies cross-modal representation alignment for text-to-image and image-to-text retrieval. It compares four standard similarity metrics (Euclidean, Manhattan, Chi-square, cosine) and two learned metrics (MLPs trained with MSE and a custom contrastive loss) using features from three vision-language models (CLIP, BLIP, Meta-Transformer) and three combined unimodal models (ResNet/BERT, ConViT/RoBERTa, ConvNeXt/XLNet) on IMDB, Flickr30K, and MS-COCO. It also analyzes embedding geometry via t-SNE plots, centroid distances, and Wasserstein-2 distances. The central claims are that cosine similarity consistently outperforms other metrics, Wasserstein distance quantifies the modality gap, MLPs are insufficient for learning alignment, and the custom contrastive loss improves over MSE. The body of the paper, however, supports these claims only partially.

Significance. If the qualified findings held, the paper would provide a useful reference point for practitioners choosing retrieval metrics for contrastively trained VLMs, and the Wasserstein analysis adds a distribution-level view of the modality gap. The paper is systematic in covering multiple models, datasets, and metrics and makes its code publicly available, which supports reproducibility. The main limitation is that the headline claims overreach the evidence: the metric ranking results are only clear for CLIP and BLIP, where cosine is the training objective, and the learned-alignment experiments use only MLPs, so the claimed universal cosine superiority and the contrastive-loss advantage for transformer-based models are not established.

major comments (5)
  1. [Abstract; Tables 3 and 4] The abstract states that 'cosine similarity consistently outperforms all the investigated metrics' and that the custom contrastive loss 'is advantageous over the MSE loss ... for both multilayer perceptrons and transformer-based models.' These statements are not supported by the reported experiments. In Tables 3 and 4, cosine is the best metric only for CLIP and BLIP; for Meta-Transformer and UM1-UM3 all metrics are near chance (e.g., Table 3, MS-COCO, Meta: P@1 = 0.0 for cosine and Euclidean; UM1: P@1 = 0.4 for cosine vs 0.1 for Euclidean), and the learned metrics are always MLPs (Section 4.5), with no transformer-based alignment learner anywhere in the experiments. The body abstract and conclusions use more qualified language, but the abstract of record promises more than the data deliver.
  2. [Section 4.5.1, paragraph 2; Section 4.5.2, paragraph 2] The paper itself notes that 'retrieval only performs best when we use the same metric as the one used during training, as MML models are optimized for that specific metric.' Since CLIP and BLIP are trained with cosine-based InfoNCE, the observed cosine superiority for these models is expected by construction; it does not establish a general property of cosine for alignment tasks. To support a general claim, the paper would need models trained with other objectives or an explicit statement that the finding is limited to contrastive-trained VLMs.
  3. [Section 4.5.2; Tables 3 and 4] The claim that the custom contrastive loss is advantageous over MSE is not supported by the tables. In Table 3, MS-COCO, CLIP: MLP(MSE) achieves P@1 = 0.6% while MLP(contr) achieves 0.0%; in IMDB, CLIP: MSE = 0.5% vs contrast = 0.3%. Similar reversals appear in Table 4. No significance test or repeated-run variability is reported for the loss comparison; the only statistical tests in Section 4.5.2 compare models, not metrics or losses. The paper should either provide proper statistical evidence for the loss comparison or remove the claim.
  4. [Section 4.1; Tables 3 and 4] The decision to keep only the first of five captions per image for Flickr30K and MS-COCO changes the retrieval ground truth, and Tables 3 and 4 are computed on this truncated setup. Table 5 shows that adding the full caption set substantially changes precision values (e.g., CLIP image-to-text P@1 rises from 70.1 in Table 4 to 79.3 in Table 5). Since the metric ranking is the central result, the paper should verify that the ranking is stable when using all captions, or at least justify why the first-caption subset is representative.
  5. [Section 4.3, paragraph 4; Abstract] Section 4.3 states: 'we can conclude that (1) the alignment among different modalities can be measured neither by spatial distances (e.g., Euclidean) nor by distribution differences (e.g., Wasserstein distance).' This directly contradicts the abstract's claim that 'Wasserstein distance can serve as an informative measure of the modality gap' and the contribution bullet that 'a distribution distance like W2 can also quantify this.' The authors need to decide whether W2 is informative about the modality gap or not, and adjust the text accordingly.
minor comments (5)
  1. [Table 4, IMDB, UM2 row] The row reports P@5 = 1.1% and P@10 = 0.1%, which is inconsistent because P@10 cannot be lower than P@5; please check and correct the typo.
  2. [Section 3, paragraph 1] 'we maps all features' should be 'we map all features'.
  3. [Section 4.4, paragraph 1] 'Meta-Transformer has consistently yields cosine similarity' should be 'Meta-Transformer consistently yields cosine similarity'.
  4. [Tables 1 and 3 headings] The headings 'T able 1' and 'T able 3' contain extra spaces and should be 'Table 1' and 'Table 3'.
  5. [Section 4.5.2, final paragraph] The sentence 'No matter if it is image-to-text or text-to-image, cosine similarity performs best for CLIP and BLIP' should explicitly say 'among the four standard metrics' and should acknowledge that for the other models all metrics are at chance level.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: results are empirical measurements on held-out test splits; the only self-citation is corroborative, and the abstract's overclaims are correctness issues rather than derivations that reduce to their inputs.

full rationale

The paper makes no derived "prediction" that is defined by its own inputs. Cosine superiority for CLIP and BLIP is measured on held-out retrieval test sets, not fitted; the paper explicitly acknowledges the confound that these models are trained with cosine-based InfoNCE ("retrieval only performs best when we use the same metric as the one used during training"), which is an interpretation caveat rather than a circular derivation. The learned MLP metrics are trained on 80/10/10 splits and tested on the held-out 10%, so no fitted parameter is renamed as a prediction. The Wasserstein-2 measurements are independent distributional statistics. The one self-citation ([44], Yilma and Leiva) merely corroborates a secondary observation about modality gap and downstream performance and is not load-bearing for the central claims. The abstract's phrases "consistently outperforms all the investigated metrics" and "for both multilayer perceptrons and transformer-based models" exceed what Tables 3-5 support: for Meta-Transformer and UM1-UM3, cosine P@1 is at chance, and no transformer-based alignment learner is tested. These are overgeneralizations or support gaps, not circularity. Similarly, the contrastive-versus-MSE comparison is reported without error bars or significance tests; that is a statistical rigor issue, not a reduction of the claim to its own definition.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

No new theoretical entities are introduced; the paper is an empirical comparison. The free parameters are the tuning choices and subset sizes that affect the MLP and retrieval numbers. The axioms are the data and representation assumptions that turn the raw comparison into the paper's conclusions.

free parameters (3)
  • MLP hidden architecture = Optuna-searched: 1-5 layers, 100-1100 neurons; concrete per-task architectures not reported
    Used to learn similarity scores; the exact 72 architectures are not enumerated in the preprint, affecting reproduction of the MLP results.
  • Learning rate schedule = 5e-5 initial, decay (1+2^0.5 c)^-1
    Hand-tuned; a footnote says this decay 'proved useful in preliminary experiments' in Section 3.
  • Random subset sizes = 1K evaluation pairs, 20K training pairs, 5K W2 batches
    Efficiency choices that affect P@K variance and the Wasserstein-2 estimates, and no error bars are attached to them.
assumptions (6)
  • domain assumption One-to-one modality mapping; use first caption for Flickr30K and MS-COCO, each with five captions per image.
    Section 4.1 states this assumption and applies it to all retrieval experiments; it changes the task's ground truth and upper bounds.
  • domain assumption Feature vectors from paired unimodal models are compatible in dimension for direct metric computation.
    Section 4.2 says models were intentionally selected so composed models have compatible image and text features; this constrains the model choices and the generality of the unimodal comparison.
  • domain assumption Penultimate-layer outputs are the learned feature representations.
    Section 4.2: 'We extract representations from the layer before the final output layer from each model.' The modality-gap and retrieval analyses depend on this choice.
  • domain assumption t-SNE projections and centroid distances in the original dimension reveal the modality gap.
    Section 4.3 and Figure 2 use this to characterize geometry; Section 5 concedes t-SNE 'reveal feature-space structures induced by the model, rather than faithfully representing the intrinsic structure of the raw data.'
  • domain assumption A learned similarity score that approximates the cosine target can be used for ranking.
    Section 4.5 trains MLPs to output a cosine-like scalar and uses that scalar for P@K; the validity of this proxy for retrieval is assumed and the negative result is interpreted as MLP insufficiency.
  • domain assumption Random 1K evaluation and 20K training subsets approximate full-dataset retrieval characteristics.
    Section 4.5 uses these subsets for efficiency; P@K is reported without error bars, so subset effects are unquantified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multimodal Representation Alignment for Cross-modal Information Retrieval." pith.science (2026). https://pith.science/paper/NKVSO4L6

@misc{pith2026250608774,
  author       = {Pith},
  title        = {Pith review of: Multimodal Representation Alignment for Cross-modal Information Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NKVSO4L6}},
  note         = {Machine review of arXiv:2506.08774}
}
read the original abstract

Different machine learning models can represent the same underlying concept in different ways. This variability is particularly valuable for in-the-wild multimodal retrieval, where the objective is to identify the corresponding representation in one modality given another modality as input. This challenge can be effectively framed as a representation alignment problem. For example, given a sentence encoded by a language model, retrieve the most semantically aligned image based on representations produced by an image encoder, or vice versa. To gain insights into the performance impact of different metrics, embedding spaces, and representation alignment for retrieval tasks, we first empirically investigate the geometric relationships between visual and textual embeddings derived from both vision-language models and combined unimodal models. We then align these representations using four standard similarity metrics as well as two learned ones, implemented via neural networks of different architectures with varying losses across multiple benchmarks. Our experimental findings indicate that cosine similarity consistently outperforms all the investigated metrics in representation alignment tasks, and that Wasserstein distance provides a complementary perspective on cross-modal distributional differences. We also observe that our proposed custom contrastive loss is advantageous over the MSE loss for aligning image and text representations, for both multilayer perceptrons and transformer-based models. Taken together, our findings offer novel insights and practical considerations for researchers working in multimodal information retrieval, particularly in real-world, cross-modal applications. Our code is publicly available.

Figures

Figures reproduced from arXiv: 2506.08774 by the authors.

Figure 1
Figure 1. illustrates our comprehensive experimental pipeline. Initially, we employ both multimodal and unimodal architectures as feature extrac￾tors to derive representations from three distinct datasets. Subsequently, we examine the relationship between retrieval performance and the geometric structures of these representations. We quantitatively evaluate cross-modal retrieval efficacy using the aforementioned metrics, asse… view at source ↗
Figure 2
Figure 2. Embedding spaces of multimodal models and combined unimodal models on different datasets through t-SNE. Image features are in blue while text features are in orange. Dashed lines ( ) denote cluster contours, and yellow dots (•) denote centroids of clusters, for which the modality gap (MG) is shown on the top-right part of each plot. smallest one. No significant differences between cluster contours were ob￾served. Am… view at source ↗
Figure 3
Figure 3. Cosine similarity (↑) matrix of a random selection of 10 <text,image> pairs. In each cell, cosine similarity is computed based on text and image features from the dataset showed on the left, extracted by models listed on the top. In each heatmap, the X-axis denotes text features and the Y-axis denotes the image features. Red cells have larger cosine while blue cells are smaller. As observed, both CLIP and BLIP have … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Image retrieval examples given randomly selected texts as input queries using CLIP. Correct results marked in green. CLIP successfully retrieved the corresponding image on all datasets, although for MS-COCO it was at position 4. The examples of retrieval results for BL…
Figure 5
Figure 5. Figure 5: Text retrieval examples given randomly selected images as input queries using CLIP. Correct results marked in green. CLIP successfully retrieved the corresponding text on IMDB and Flickr30K datasets, whereas it failed on MS-COCO. cross-modality retrieval tasks. To the …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

68 extracted references · 53 canonical work pages

  1. [1]

    LeCun, B

    Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hub- bard, L. D. Jackel, Backpropagation applied to handwritten zip code recognition, Neural Computation (1989)

    work page 1989

  2. [2]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, I. Polosukhin, Attention is all you need, in: Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc., 2017

    work page 2017

  3. [3]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby, An image is worth 16x16 words: Transform- ers for image recognition at scale, in: 9th International Conference on Learning Representations (ICLR), 2021

    work page 2021

  4. [4]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, in: Pro- ceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies (NAACL), 2019

    work page 2019

  5. [5]

    Radford, K

    A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, Improving language understanding by generative pre-training, 2018. https: //cdn.openai.com/research-covers/language-unsupervised/ language_understanding_paper.pdf

    work page 2018

  6. [6]

    Radford, J

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, Language models are unsupervised multitask learners, OpenAI 25 (2019). URL: https://cdn.openai.com/better-language-models/ language_models_are_unsupervised_multitask_learners.pdf

    work page 2019

  7. [7]

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, D. Amodei,...

    work page 2020

  8. [8]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, I. Sutskever, Learning transferable visual models from natural language supervision, in: Proceedings of the 38th International Conference on Machine Learn- ing (ICML), Proceedings of Machine Learning Research, PMLR, 2021

    work page 2021

Show all 68 references
  1. [9]

    J. Li, D. Li, C. Xiong, S. Hoi, BLIP: Bootstrapping language-image pre- training for unified vision-language understanding and generation, in: Proceedings of the 39th International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, PMLR, 2022

    2022

  2. [10]

    V. W. Liang, Y. Zhang, Y. Kwon, S. Yeung, J. Y. Zou, Mind the gap: Understanding the modality gap in multi-modal contrastive representa- tion learning, in: Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  3. [11]

    Hornik, Approximation capabilities of multilayer feedforward networks, Neural Networks (1991)

    K. Hornik, Approximation capabilities of multilayer feedforward networks, Neural Networks (1991). doi: https://doi.org/10.1016/ 0893-6080(91)90009-T

    1991

  4. [12]

    Z. Li, F. Liu, W. Yang, S. Peng, J. Zhou, A survey of convolutional neu- ral networks: Analysis, applications, and prospects, IEEE Transactions on Neural Networks and Learning Systems (2022)

    2022

  5. [13]

    K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 26

    2016

  6. [14]

    L.Leavitt, A

    S.d’Ascoli, H.Touvron, M. L.Leavitt, A. S.Morcos, G.Biroli, L. Sagun, ConViT: improving vision transformers with soft convolutional inductive biases, JournalofStatisticalMechanics: TheoryandExperiment(2022)

    2022

  7. [15]

    Y. Li, H. Mao, R. Girshick, K. He, Exploring plain vision transformer backbones for object detection, in: European conference on computer vision (ECCV), Springer, 2022, pp. 280–296

    2022

  8. [16]

    Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, S. Xie, A convnet for the 2020s, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), 2022, pp. 11976– 11986

    2022

  9. [17]

    Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, B. Guo, Swin transformer: Hierarchical vision transformer using shifted windows, in: Proceedings of the IEEE/CVF international conference on computer vision (ICCV), 2021

    2021

  10. [18]

    Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized BERT pretraining approach, CoRR arXiv: abs/1907.11692 (2019)

    2019 arXiv

  11. [19]

    P. He, X. Liu, J. Gao, W. Chen, Deberta: Decoding-enhanced bert with disentangled attention, CoRR arXiv: abs/2006.03654 (2020)

    2020 arXiv

  12. [20]

    Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, Q. V. Le, Xlnet: Generalized autoregressive pretraining for language understand- ing, in: Advances in Neural Information Processing Systems (NeurIPS), volume 32, Curran Associates, Inc., 2019

    2019

  13. [21]

    Touvron, T

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, G. Lample, LLaMA: Open and efficient foundation language models, CoRR ArXiv:abs/2302.13971 (2023)

    2023 arXiv

  14. [22]

    BigScience Workshop, BLOOM: A 176b-parameter open-access multi- lingual language model, 2023

    2023

  15. [23]

    Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, J. Tang, GLM: Gen- eral language model pretraining with autoregressive blank infilling, in: 27 Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (ACL), 2022

    2022

  16. [24]

    J. Li, D. Li, S. Savarese, S. Hoi, BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, in: Proceedings of the 40th International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, PMLR, 2023

    2023

  17. [25]

    Zhang, K

    Y. Zhang, K. Gong, K. Zhang, H. Li, Y. Qiao, W. Ouyang, X. Yue, Meta-transformer: A unified framework for multimodal learning, CoRR arXiv:abs/2307.10802 (2023)

    2023 arXiv

  18. [26]

    Z. Tang, Z. Yang, C. Zhu, M. Zeng, M. Bansal, Any-to-any generation viacomposablediffusion, in: AdvancesinNeuralInformationProcessing Systems (NeurIPS), 2024

    2024

  19. [27]

    Mensch, K

    J.-B.Alayrac, J.Donahue, P.Luc, A.Miech, I.Barr, Y.Hasson, K.Lenc, A. Mensch, K. Millican, M. Reynolds, et al., Flamingo: a visual lan- guage model for few-shot learning, in: Proceedings of the 36th Interna- tional Conference on Advances in neural information processing system...

    2022

  20. [28]

    L. Yuan, D. Chen, Y.-L. Chen, N. Codella, X. Dai, J. Gao, H. Hu, X. Huang, B. Li, C. Li, et al., Florence: A new foundation model for computer vision, CoRR arXiv:abs/2111.11432 (2021)

    2021 arXiv

  21. [29]

    B. Xiao, H. Wu, W. Xu, X. Dai, H. Hu, Y. Lu, M. Zeng, C. Liu, L. Yuan, Florence-2: Advancing a unified representation for a variety of vision tasks, in: Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), 2024

    2024

  22. [30]

    C. Wu, S. Yin, W. Qi, X. Wang, Z. Tang, N. Duan, Visual chatgpt: Talking, drawing and editing with visual foundation models, CoRR arXiv:abs/2303.04671 (2023)

    2023 arXiv

  23. [31]

    W. Wang, Q. Lv, W. Yu, W. Hong, J. Qi, Y. Wang, J. Ji, Z. Yang, L. Zhao, X. Song, et al., Cogvlm: Visual expert for pretrained language models, CoRR arXiv:abs/2311.03079 (2023). 28

    2023 arXiv

  24. [32]

    F. Chen, M. Han, H. Zhao, Q. Zhang, J. Shi, S. Xu, B. Xu, X-llm: Boot- strapping advanced large language models by treating multi-modalities as foreign languages, CoRR arXiv:abs/2305.04160 (2023)

    2023 arXiv

  25. [33]

    L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, K.-W. Chang, Visual- bert: A simple and performant baseline for vision and language, CoRR arXiv:abs/1908.03557 (2019)

    2019 arXiv

  26. [34]

    Zhang, S

    D. Zhang, S. Li, X. Zhang, J. Zhan, P. Wang, Y. Zhou, X. Qiu, SpeechGPT: Empowering large language models with intrinsic cross- modal conversational abilities, in: Findings of the Association for Com- putational Linguistics (EMNLP), Association for Computational Lin- guistics, 2023

    2023

  27. [35]

    S. Wu, H. Fei, L. Qu, W. Ji, T.-S. Chua, Next-gpt: Any-to-any multi- modal llm, CoRR arXiv:abs/2309.05519 (2023)

    2023 arXiv

  28. [36]

    Z. Li, X. Wu, H. Du, F. Liu, H. Nghiem, G. Shi, A survey of state of the art large vision language models: Alignment, benchmark, evaluations and challenges, CoRR ArXiv (2025).arXiv:abs/2501.02189

    2025 arXiv

  29. [37]

    Hadsell, S

    R. Hadsell, S. Chopra, Y. LeCun, Dimensionality reduction by learning an invariant mapping, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), 2006

    2006

  30. [38]

    van den Oord, Y

    A. van den Oord, Y. Li, O. Vinyals, Representation learning with con- trastive predictive coding, CoRR abs/1807.03748 (2018)

    2018 arXiv

  31. [39]

    K. He, H. Fan, Y. Wu, S. Xie, R. Girshick, Momentum contrast for unsupervised visual representation learning, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020

    2020

  32. [40]

    T. Chen, S. Kornblith, M. Norouzi, G. Hinton, A simple framework for contrastive learning of visual representations, in: Proceedings of the 37th International Conference on Machine Learning (ICML), Proceed- ings of Machine Learning Research, 2020

    2020

  33. [41]

    Grill, F

    J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E.Buchatskaya, C.Doersch, B.AvilaPires, Z.Guo, M.GheshlaghiAzar, 29 B. Piot, k. kavukcuoglu, R. Munos, M. Valko, Bootstrap your own la- tent - a new approach to self-supervised learning, in: Advances in Neural Informat...

    2020

  34. [42]

    C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, T. Duerig, Scaling up visual and vision-language represen- tation learning with noisy text supervision, in: Proceedings of the 38th International Conference on Machine Learning (ICML), Proceedin...

    2021

  35. [43]

    X. Zhai, B. Mustafa, A. Kolesnikov, L. Beyer, Sigmoid loss for language image pre-training, in: Proceedings of the IEEE/CVF international conference on computer vision (ICCV), 2023

    2023

  36. [44]

    on User Modeling, Adaptation and Personalization (UMAP), 2023

    B.A.Yilma, L.A.Leiva, Togetheryetapart: Multimodalrepresentation learning for personalised visual art recommendation, in: Proceedings of the ACM Conf. on User Modeling, Adaptation and Personalization (UMAP), 2023

    2023

  37. [45]

    K. Ethayarajh, How contextual are contextualized word representa- tions? Comparing the geometry of BERT, ELMo, and GPT-2 embed- dings, in: Proceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th International Joint Conference on Nat...

    2019 doi

  38. [46]

    W. Wang, X. Yang, B. C. Ooi, D. Zhang, Y. Zhuang, Effective deep learning-based multi-modal retrieval, The VLDB Journal (2016)

    2016

  39. [47]

    J. Luo, Y. Shen, X. Ao, Z. Zhao, M. Yang, Cross-modal image-text retrieval with multitask learning, in: Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM), Association for Computing Machinery, 2019

    2019

  40. [48]

    L. Zhen, P. Hu, X. Peng, R. S. M. Goh, J. T. Zhou, Deep multimodal transfer learning for cross-modal retrieval, IEEE Transactions on Neural Networks and Learning Systems (2022)

    2022

  41. [49]

    Kumar, T

    K. Kumar, T. Arici, T. Neiman, J. Yang, S. Sam, Y. Xu, H. Ferhatos- manoglu, I. Tutar, Unsupervised multi-modal representation learning 30 for high quality retrieval of similar products at e-commerce scale, in: Proceedings of the 32nd ACM International Conference on Informatio...

    2023

  42. [50]

    G. Luo, L. Dunlap, D. H. Park, A. Holynski, T. Darrell, Diffusion hyperfeatures: Searching through time and space for semantic corre- spondence, in: Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc., 2023

    2023

  43. [51]

    L. V. B. Beltrán, J. C. Caicedo, N. Journet, M. Coustaty, F. Lecellier, A. Doucet, Deep multimodal learning for cross-modal retrieval: One model for all tasks, Pattern Recognition Letters (2021)

    2021

  44. [52]

    T. Yu, Y. Yang, H. Fei, Y. Li, X. Chen, P. Li, Assorted attention network for cross-lingual language-to-vision retrieval, in: Proceedings of the 30th ACM International Conference on Information & Knowledge Management (CIKM), Association for Computing Machinery, 2021

    2021

  45. [53]

    van der Maaten, G

    L. van der Maaten, G. Hinton, Visualizing data using t-SNE, Journal of Machine Learning Research (2008)

    2008

  46. [54]

    McInnes, J

    L. McInnes, J. Healy, N. Saul, L. Großberger, Umap: Uniform manifold approximationandprojection, JournalofOpenSourceSoftware3(2018) 861

    2018

  47. [55]

    S. Zhu, L. Liu, Y. Wang, Information retrieval using hellinger distance and sqrt-cos similarity, in: 2012 7th International Conference on Com- puter Science & Education (ICCSE), 2012

    2012

  48. [56]

    Acree, E

    B. Acree, E. Hansen, J. Jansa, K. Shoub, Comparing and evaluating cosine similarity scores, weighted cosine similarity scores and substring matching, Working Paper (2016)

    2016

  49. [57]

    S. Tan, Z. Zhou, Z. Xu, P. Li, On efficient retrieval of top similarity vectors, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-IJCNLP), Association f...

    2019

  50. [58]

    H. Gong, T. Sakakini, S. Bhat, J. Xiong, Document similarity for texts of varying lengths via hidden topics, in: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), 2018

    2018

  51. [59]

    Zhang, IMDB Vision and NLP, 2022

    X. Zhang, IMDB Vision and NLP, 2022. doi: 10.34740/KAGGLE/DS/ 1937446

    2022 doi

  52. [60]

    Young, A

    P. Young, A. Lai, M. Hodosh, J. Hockenmaier, From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions, Transactions of the Association for Computational Linguistics (ACL) (2014)

    2014

  53. [61]

    T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, C. L. Zitnick, Microsoft coco: Common objects in context, in: The 13th European Conference on Computer Vision (ECCV), Springer, 2014

    2014

  54. [62]

    I. Bica, A. Ilic, M. Bauer, G. Erdogan, M. Bošnjak, C. Kaplanis, A. A. Gritsenko, M. Minderer, C. Blundell, R. Pascanu, J. Mitrovic, Improving fine-grained understanding in image-text pre-training, in: R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, ...

    2024

  55. [63]

    I. T. Jolliffe, J. Cadima, Principal component analysis: a review and recent developments, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences (2016)

    2016

  56. [64]

    Hearst, S

    M. Hearst, S. Dumais, E. Osuna, J. Platt, B. Scholkopf, Support vector machines, IEEE Intelligent Systems and their Applications (1998)

    1998

  57. [65]

    Wiles, S

    O. Wiles, S. Gowal, F. Stimberg, S.-A. Rebuffi, I. Ktena, K. D. Dvi- jotham, A. T. Cemgil, A fine-grained analysis on distribution shift, in: International Conference on Learning Representations (ICLR), 2022

    2022

  58. [66]

    Robey, H

    A. Robey, H. Hassani, G. J. Pappas, Model-based robust deep learning, CoRR arXiv:abs/2005.10247 (2020)

    2020 arXiv

  59. [67]

    Iscen, M

    A. Iscen, M. Caron, A. Fathi, C. Schmid, Retrieval-enhanced contrastive vision-text models, CoRR arXiv:abs/2306.07196 (2023). 32

    2023 arXiv

  60. [68]

    Wendler, V

    C. Wendler, V. Veselovsky, G. Monea, R. West, Do llamas work in En- glish? on the latent language of multilingual transformers, in: Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), Association for Computational Linguistics, 2024....

    2005

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.