Pith. sign in

REVIEW 4 major objections 6 minor 37 references

Closing the Modality Gap for Mixed Modality Search

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GR-CLIP removes CLIP's modality gap and lifts mixed-modality NDCG@10 by up to 26 points.

desk verdict Useful benchmark and a simple method with real gains, but the missing ablation on the query-mean term leaves the modality-gap attribution unproven. read the letter →

arxiv 2507.19054 v1 pith:4TMM6YDT submitted 2025-07-25 cs.CV cs.AIcs.CLcs.IRcs.LG

Binxu Li , Yuhui Zhang , Xiaohan Wang , Weixin Liang , Ludwig Schmidt , Serena Yeung-Levy This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.IRcs.LG
keywords mixedmodalitysearchgapCLIPembeddingcalibrationretrievalNDCGmultimodalfusionVLM2Vec
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that CLIP-style contrastive models are poorly suited to mixed modality search—retrieval over a corpus that mixes text-only, image-only, and image-text documents—because their embeddings carry a modality gap: image and text features cluster separately, so same-modality items are ranked too high and fused image-text embeddings land in weak regions. It proposes GR-CLIP, a post-hoc calibration that subtracts each modality's mean embedding before computing cosine similarity, which cancels the constant offset between modalities. On MixBench, a new benchmark built from four image-text datasets, GR-CLIP improves NDCG@10 by up to 26 percentage points over CLIP and outperforms the generative embedding model VLM2Vec by about 4 points while using roughly 75× less compute. The practical stakes are that a nearly free, training-free preprocessing step may capture most of the benefit that expensive generative embedding models provide for heterogeneous retrieval.

What carries the argument

The key machinery is the modality-gap vector and its cancellation by mean-shift calibration. Prior work has characterized the gap as a constant vector $c_\perp$ such that $e^T_i - e^I_i \approx c_\perp$ for paired items, orthogonal to the shared image/text subspace. GR-CLIP estimates $c_\perp$ as the difference between dataset-level mean text and image embeddings, subtracts each modality's mean from its own embeddings—using a separate mean for short text queries versus long text documents—and then ranks by cosine similarity. The subtraction makes $e'^T_i - e'^I_i \approx 0$, so cross-modal similarities become comparable to intra-modal ones, and the same calibration is applied inside the linear fusion used for image-text documents.

What would settle it

Compute residuals $r_i = (e^T_i - e^I_i) - (\bar e^T - \bar e^I)$ on a held-out corpus; if the residual variance is large or the residuals correlate with semantic content, the constant-offset model fails. Alternatively, estimate means on one document distribution and evaluate on another; if NDCG@10 gains vanish or reverse, the gap is not a domain-independent constant.

Watch

Extended reading notes

Core claim

The central discovery is that the modality gap—a near-constant offset vector $c_\perp$ between paired image and text embeddings, approximately orthogonal to the semantic subspace—is the main obstacle preventing CLIP from ranking across mixed-modality corpora. The paper shows that CLIP's retrieval performance traces a U-shaped curve as text documents are replaced by screenshots: performance collapses when most documents are in the other modality, then recovers when all are, a pattern reproduced by a simulation that pushes cross-modal items to the bottom. Removing the gap by subtracting modality-specific means flattens the curve, makes linear fusion of image and text features useful at intermediate fusion weights, and on MixBench raises NDCG@10 by up to 26 points across five CLIP variants, surpassing the VLM2Vec baseline by about 4 points with 75× less compute; the exception is MSCOCO, which VLM2Vec was trained on.

Load-bearing premise

The load-bearing premise is that the modality gap is a single constant vector shared by all paired image-text items, so subtracting dataset-level means removes it; if the gap varies with content or has a component inside the semantic subspace, mean subtraction only partially closes it and the claimed gains may fail to transfer to new corpora or domains.

Editorial extensions

If this is right

  • On MixBench, GR-CLIP raises NDCG@10 by up to 26 percentage points over the corresponding CLIP variant, with gains on all four subsets and across all five CLIP, OpenCLIP, and SigLIP variants tested.
  • GR-CLIP outperforms the VLM2Vec generative-embedding baseline by about 4 NDCG@10 points on average while using roughly 75× less inference compute; the MSCOCO subset is the exception because VLM2Vec was trained on it.
  • The U-shaped retrieval curve observed as text documents are replaced by screenshots is explained by modality-gap ranking bias and is flattened after mean-shift calibration.
  • After calibration, linear fusion of image and text embeddings peaks at intermediate fusion weights and beats both unimodal endpoints, so multimodal documents contribute positively rather than degrading retrieval.
  • The same pattern holds under NDCG@100 and Recall@1, and for text-to-video and text-to-audio retrieval.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the constant-offset characterization holds in other domains, the same mean-subtraction calibration should be applied to every contrastive embedding system (audio, video, and future modalities) before attributing retrieval gains to heavier models.
  • Because the paper estimates means on training splits and applies them to test corpora, a natural extension is to estimate means from the unlabeled target corpus itself and measure whether that improves transfer under domain shift.
  • These results suggest retrieval benchmark reports should include a calibrated contrastive baseline; otherwise, some reported advantages of generative embedding models over CLIP may simply reflect an uncalibrated modality gap.
  • A direct follow-up is to check whether generative embedding models like VLM2Vec have their own modality gap and whether the same mean subtraction shrinks their gap and their compute disadvantage.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies mixed-modality retrieval, in which queries and corpus documents can be text, image, or multimodal. It observes a modality gap in CLIP-style embeddings and proposes GR-CLIP, a post-hoc calibration that subtracts modality-specific mean vectors from embeddings before cosine-similarity ranking. It introduces MixBench, constructed from Google-WIT, MSCOCO, OVEN, and VisualNews, and reports that GR-CLIP improves NDCG@10 over the corresponding CLIP models across five CLIP variants, outperforms VLM2Vec on most subsets, and generalizes to video-text and audio-text settings. The main causal story is that removing the modality gap resolves intra-modal ranking bias and fusion failure.

Significance. If the results hold, the paper makes a useful empirical and practical contribution: it demonstrates that a few mean subtractions applied to frozen CLIP embeddings can produce large gains on heterogeneous retrieval, and it provides a benchmark (MixBench) plus code and data for future work. The internal consistency across five CLIP variants, four datasets, three metrics, and additional audio/video pairs is a genuine strength, as is the reproducible artifact release. The significance would be higher once the attribution of the gains to modality-gap removal rather than query-style normalization is established by ablation, and once the headline numbers are reconciled with the tables.

major comments (4)
  1. [§2.3; Appendix B, Algorithm 1] The main text and the derivation in §2.3 justify GR-CLIP by subtracting only the document text mean and the document image mean, based on the model eT_i - eI_i ≈ c_perp. Algorithm 1 in Appendix B additionally subtracts a query mean e_q (and a separate OVEN-specific query mean), with the stated purpose of accounting for 'structural and semantic differences' between short queries and long documents. The reported MixBench results therefore evaluate a three-mean procedure, not solely the modality-gap removal described in the main text, so the claim that the gains come from closing the modality gap is not directly supported. Please provide an ablation on MixBench comparing (i) image/text document mean subtraction only, (ii) query-mean subtraction only, and (iii) the full procedure; this is needed to attribute the observed improvement.
  2. [Abstract; Table 1] The abstract and §5.2 state an 'up to 26 percentage points' gain, but Table 1 contains larger gaps, e.g., SigLIP-400m on VisualNews goes from 0.385 to 0.769 NDCG@10 (+0.384) and on MSCOCO from 0.327 to 0.696 (+0.369). Please correct the headline, specify the exact subset/condition to which 26 points refers, and state whether the reported values are raw NDCG differences rather than relative improvements.
  3. [§4.1; Appendix E; Appendix B Algorithm 1] Table 2 reports OVEN queries as T+I (image plus text), and Appendix E states that the OVEN query remains unchanged, but Algorithm 1 and §2.3 describe a pipeline in which queries are text-only and encoded with f_T only. The manuscript never specifies how the image component of an OVEN query is handled for CLIP, GR-CLIP, or VLM2Vec. This omission makes the OVEN rows in Table 1 unreproducible and leaves open the possibility that the comparison treats the two methods asymmetrically. Please clarify the exact query encoding used for OVEN (e.g., text-only, image-only, or fused) for every method.
  4. [§5.2; Figure 4b; §2.3] The fusion weight α for multimodal documents in the final MixBench results is not reported. Section 2.3 defines the interpolation e = α e_T + (1−α) e_I, and Section 4 shows that performance varies strongly with α (including a U-shape for original CLIP), but Figure 4b and Table 1 do not state which α was used per dataset/model or whether it was tuned. Please report the α values used and any selection criterion, since without this the headline benchmark numbers cannot be reproduced or compared across methods.
minor comments (6)
  1. [§3.2] The push-down simulation assigns zero similarity to all screenshots, which by construction imposes the modality-bias being tested; please label it as an illustrative mechanism rather than an independent confirmation, or add a less extreme penalty variant.
  2. [§2.1] The second bullet point in the definition of the two distinguishing properties is labeled 'b)' while the first is labeled '1)'; renumber for consistency.
  3. [Figure 1 caption] The caption contains the leftover text 'Reviewer #2: This is a very interesting paper! I really like it!' in the top-left panel; this appears to be a template artifact and should be removed.
  4. [Code/Data Availability] The manuscript calls the GitHub and Hugging Face links 'anonymous', but the URLs contain 'yuhui-zh15', which reveals an author identity; if this is a double-blind submission, the links should be anonymized.
  5. [Figure 7 caption] The label 'SigLIP-ViusalNews' is a typo for 'VisualNews'.
  6. [Tables 1 and Figures 4-7] The tables do not report error bars or significance tests; since the evaluation appears deterministic given the fixed query/document sets, state this explicitly and mention whether any randomness (e.g., caption sampling or GPT generation) was involved in the reported numbers.

Circularity Check

2 steps flagged · score 4.0 of 10

Benchmark gains are externally measured, but the U-shape 'confirmation' is built from its own assumption and the modality-gap theory is a self-citation chain.

  1. self definitional [Section 3.2, 'Push-down simulation confirms the hypothesis' (Figure 2b)]
    "To verify this explanation, we simulate a modality-induced ranking bias by assigning a fixed similarity score of zero to all screenshots, effectively pushing them to the bottom of the ranked list. The resulting performance curve (Figure 2b) closely matches the actual CLIP curve, validating our hypothesis that the U-shape arises from modality gap–induced ranking distortion."

    The simulation encodes the hypothesis as its input: it forces every screenshot's similarity score to zero, which is exactly the cross-modal penalty that the modality-gap account is supposed to explain. The reproduced U-shape is therefore a logical consequence of the construction, not an independent confirmation. Any mechanism that penalized screenshots would produce the same curve, so the match cannot discriminate the modality-gap explanation from other ranking-bias mechanisms.

  2. self citation load bearing [Section 1 (method introduction); Section 2.3]
    "Prior work [35, 36] has shown that the modality gap in CLIP-like models can be approximated by a constant vector that is orthogonal to the image and text embedding subspaces. Based on this theory, we compute the mean embeddings of all image and text data, use their difference to estimate the modality gap, and subtract this vector from all embeddings before performing retrieval."

    The theoretical premise that licenses the mean-subtraction recipe is cited to [35, 36] (and elsewhere [17]), all of which share authors with the present paper (Yuhui Zhang, Weixin Liang, Serena Yeung-Levy). The derivation e'T - e'I = (eT - eI) - (E[eT] - E[eI]) ≈ c_perp - c_perp = 0 is a direct restatement of that self-cited constant-vector model. This is not full circularity because the NDCG gains are measured against external baselines, but the causal attribution 'removing the modality gap causes the gains' rests on a self-citation chain rather than an independent check of the constant-vector assumption on the MixBench distributions.

full rationale

The central benchmark claim is not circular: GR-CLIP's NDCG improvements are measured against CLIP, VLM2Vec, and other baselines on MixBench, so the headline result is externally anchored. The circularity burden is confined to two secondary elements. First, the push-down simulation is a by-construction consistency check presented as confirmation of the U-shape explanation: zeroing all screenshot similarities builds the hypothesized bias into the simulation, so the resulting curve cannot independently validate the modality-gap account. Second, the theoretical justification for subtracting modality means is imported from the authors' own prior work ([17], [35], [36]); while peer-reviewed, it is a self-citation chain rather than an external or machine-checked result, and it is load-bearing for the paper's causal narrative. The Appendix additionally reveals that Algorithm 1 subtracts a separate query mean that is not part of the Section 2.3 derivation; this is an extra fitted component and a confound for attributing all gains to modality-gap removal, but it is not itself a circular step. Overall, the empirical comparison is independent, so the paper is only partially circular.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim depends on domain assumptions about the embedding space and about benchmark relevance construction. No new physical or formal entities are introduced; GR-CLIP is a calibration procedure and MixBench is a dataset suite.

free parameters (2)
  • fusion weight alpha = not specified
    Linear interpolation weight between text and image embeddings for multimodal documents. Section 4.2 sweeps alpha in [0,1], but Section 5.2 does not state how alpha is selected for the reported MixBench NDCG values.
  • OVEN-specific query mean = 2000 OVEN training queries
    A separate query mean is introduced for OVEN because its queries are short, as described in Appendix B. This is a per-dataset calibration choice not made for other datasets.
assumptions (5)
  • domain assumption The modality gap is a constant vector c_perp approximately orthogonal to the shared embedding subspace, so eT_i - eI_i ≈ c_perp for all paired items.
    This is the entire mechanism of GR-CLIP in Section 2.3. It is cited from the authors' own prior work [36]. If the gap varies per item, subtracting means is only a partial correction.
  • domain assumption A separate query mean and a separate document text mean are both needed because queries and documents are structurally different.
    Appendix B introduces e_q separately from e_T. This is a two-mean design choice; if one text mean sufficed, GR-CLIP would have one fewer parameter.
  • domain assumption A weighted linear sum of image and text features adequately represents a multimodal document for retrieval.
    Section 2.3 defines the fused embedding as alpha * f_T + (1-alpha) * f_I, and the Limitations admit CLIP lacks fine-grained modality interaction. The fusion evaluation depends entirely on this linearity.
  • domain assumption Mean vectors computed from training splits of MSCOCO, Google WIT, NFCorpus, VisualNews, and OVEN remain valid calibration offsets for the test splits of all four MixBench datasets.
    Appendix B explicitly excludes test sets when computing means, so it assumes cross-dataset transfer of the estimated gap. If dataset-specific gaps differ, GR-CLIP could undercorrect or overcorrect.
  • domain assumption GPT-4o-generated long captions that preserve the sampled query caption content are valid ground-truth documents.
    Appendix E describes this construction for MSCOCO and VisualNews, which can make text-text relevance near-duplicate rather than semantic and may inflate absolute NDCG scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Closing the Modality Gap for Mixed Modality Search." pith.science (2026). https://pith.science/paper/4TMM6YDT

@misc{pith2026250719054,
  author       = {Pith},
  title        = {Pith review of: Closing the Modality Gap for Mixed Modality Search},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4TMM6YDT}},
  note         = {Machine review of arXiv:2507.19054}
}
read the original abstract

Mixed modality search -- retrieving information across a heterogeneous corpus composed of images, texts, and multimodal documents -- is an important yet underexplored real-world application. In this work, we investigate how contrastive vision-language models, such as CLIP, perform on the mixed modality search task. Our analysis reveals a critical limitation: these models exhibit a pronounced modality gap in the embedding space, where image and text embeddings form distinct clusters, leading to intra-modal ranking bias and inter-modal fusion failure. To address this issue, we propose GR-CLIP, a lightweight post-hoc calibration method that removes the modality gap in CLIP's embedding space. Evaluated on MixBench -- the first benchmark specifically designed for mixed modality search -- GR-CLIP improves NDCG@10 by up to 26 percentage points over CLIP, surpasses recent vision-language generative embedding models by 4 percentage points, while using 75x less compute.

Figures

Figures reproduced from arXiv: 2507.19054 by the authors.

Figure 1
Figure 1. Overview of mixed modality search. (a) Problem Formulation: Mixed modality search aims to retrieve relevant information from a heterogeneous corpus containing multimodal documents. This is achieved by embedding both the query and documents, followed by similarity-based retrieval. (b) Embedding Method: Unimodal documents are embedded using CLIP’s modality-specific encoder, while multimodal documents are embedded via … view at source ↗
Figure 2
Figure 2. Retrieval with a heterogeneous corpus. (a) Dataset Construction: We construct a heterogeneous corpus by randomly replacing text documents with either screenshot renderings of the text or paired images with probability p. Since the semantic content remains unchanged, a retrieval system with perfect cross-modal alignment should maintain the same performance regardless of p. (b) Initial Results & Simulation: Surprising… view at source ↗
Figure 3
Figure 3. Retrieval with multimodal documents. (a) Dataset Construction: Each document contains both image and text, and embeddings are obtained by fusing modality-specific features. We vary the fusion coefficient α to evaluate the model’s ability to integrate multimodal information. (b) Results: GR-CLIP consistently outperforms CLIP across three model variants and four datasets, demonstrating that the modality gap hinders ef… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Mixed modality search. (a) Dataset Construction: We introduce MixBench, a benchmark where the corpus is heterogeneous and includes multimodal documents, reflecting the most realistic setting for search engines. (b) Results: Across four MixBench subsets and five CLIP va…
Figure 5
Figure 5. Figure 5: Generalization across modalities. GR-CLIP consistently mitigates the U-shaped curve caused by the modality gap and significantly improves performance, demonstrating strong generaliz￾ability across diverse modality pairs. A.2 Generalization across Metrics In the main pa…
Figure 6
Figure 6. Figure 6: Reproduction of [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Reproduction of [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 28 canonical work pages

  1. [1]

    J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-vl: A versatile vision- language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  2. [2]

    BehnamGhader, V

    P. BehnamGhader, V . Adlakha, M. Mosbach, D. Bahdanau, N. Chapados, and S. Reddy. Llm2vec: Large language models are secretly powerful text encoders. arXiv preprint arXiv:2404.05961, 2024

    arXiv 2024

  3. [3]

    Boteva, D

    V . Boteva, D. Gholipour, A. Sokolov, and S. Riezler. A full-text learning to rank dataset for medical information retrieval. 2016

    work page 2016

  4. [4]

    Chen and W

    D. Chen and W. Dolan. Collecting highly parallel data for paraphrase evaluation. In ACL, 2011

    work page 2011

  5. [5]

    Y .-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y . Cheng, and J. Liu. Uniter: Universal image-text representation learning. In ECCV, 2020

    work page 2020

  6. [6]

    Drossos, S

    K. Drossos, S. Lipping, and T. Virtanen. Clotho: An audio captioning dataset. In ICASSP, pages 736–740. IEEE, 2020

    work page 2020

  7. [7]

    Faysse, H

    M. Faysse, H. Sibille, T. Wu, B. Omrani, G. Viaud, C. HUDELOT, and P. Colombo. Colpali: Efficient document retrieval with vision language models. In ICLR, 2025

    work page 2025

  8. [8]

    S. Fu, N. Y . Tamir, S. Sundaram, L. Chai, R. Zhang, T. Dekel, and P. Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. In NeurIPS, 2023

    work page 2023

Show all 37 references
  1. [9]

    Girdhar, A

    R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V . Alwala, A. Joulin, and I. Misra. Imagebind: One embedding space to bind them all. In CVPR, 2023

    2023

  2. [10]

    H. Hu, Y . Luan, Y . Chen, U. Khandelwal, M. Joshi, K. Lee, K. Toutanova, and M.-W. Chang. Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities. In ICCV, 2023

    2023

  3. [11]

    Järvelin and J

    K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of ir techniques. TOIS, 2002

    2002

  4. [12]

    Jiang, R

    Z. Jiang, R. Meng, X. Yang, S. Yavuz, Y . Zhou, and W. Chen. VLM2vec: Training vision-language models for massive multimodal embedding tasks. In ICLR, 2025

    2025

  5. [13]

    Karpukhin, B

    V . Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih. Dense passage retrieval for open-domain question answering. In EMNLP, 2020

    2020

  6. [14]

    Khattab and M

    O. Khattab and M. Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In SIGIR, 2020

    2020

  7. [15]

    K.-H. Lee, X. Chen, G. Hua, H. Hu, and X. He. Stacked cross attention for image-text matching. In ECCV, 2018

    2018

  8. [16]

    L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019

    1908 arXiv

  9. [17]

    V . W. Liang, Y . Zhang, Y . Kwon, S. Yeung, and J. Y . Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In NeurIPS, 2022

    2022

  10. [18]

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014

    2014

  11. [19]

    F. Liu, Y . Wang, T. Wang, and V . Ordonez. Visual news: Benchmark and challenges in news image captioning. In NeurIPS, 2021

    2021

  12. [20]

    H. Liu, C. Li, Y . Li, B. Li, Y . Zhang, S. Shen, and Y . J. Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    2024

  13. [21]

    H. Liu, C. Li, Q. Wu, and Y . J. Lee. Visual instruction tuning. In NeurIPS, 2023. 10

    2023

  14. [22]

    J. Lu, D. Batra, D. Parikh, and S. Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In NeurIPS, 2019

    2019

  15. [23]

    Muennighoff, S

    N. Muennighoff, S. Hongjin, L. Wang, N. Yang, F. Wei, T. Yu, A. Singh, and D. Kiela. Generative representational instruction tuning. In ICLR 2024 Workshop, 2024

    2024

  16. [24]

    Ngiam, A

    J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, A. Y . Ng, et al. Multimodal deep learning. In ICML, 2011

    2011

  17. [25]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021

    2021

  18. [26]

    Robertson, H

    S. Robertson, H. Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. F oundations and Trends® in Information Retrieval, 2009

    2009

  19. [27]

    Srinivasan, K

    K. Srinivasan, K. Raman, J. Chen, M. Bendersky, and M. Najork. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In SIGIR, 2021

    2021

  20. [28]

    Srivastava and R

    N. Srivastava and R. R. Salakhutdinov. Multimodal learning with deep boltzmann machines. In NIPS, 2012

    2012

  21. [29]

    voyage-multimodal-3: all-in-one embedding model for interleaved text, images, and screen- shots

    V oyage AI. voyage-multimodal-3: all-in-one embedding model for interleaved text, images, and screen- shots. Blog post, Nov. 2024

    2024

  22. [30]

    Wadden, S

    D. Wadden, S. Lin, K. Lo, L. L. Wang, M. van Zuylen, A. Cohan, and H. Hajishirzi. Fact or fiction: Verifying scientific claims. In EMNLP, 2020

    2020

  23. [31]

    Y . Wang, K. Li, Y . Li, Y . He, B. Huang, Z. Zhao, H. Zhang, J. Xu, Y . Liu, Z. Wang, et al. Internvideo: Gen- eral video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191, 2022

    2022 arXiv

  24. [32]

    Y . Wu*, K. Chen*, T. Zhang*, Y . Hui*, T. Berg-Kirkpatrick, and S. Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP, 2023

    2023

  25. [33]

    H. Xu, S. Xie, X. Tan, P.-Y . Huang, R. Howes, V . Sharma, S.-W. Li, G. Ghosh, L. Zettlemoyer, and C. Feichtenhofer. Demystifying CLIP data. In ICLR, 2024

    2024

  26. [34]

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023

    2023

  27. [35]

    Zhang, J

    Y . Zhang, J. Z. HaoChen, S.-C. Huang, K.-C. Wang, J. Zou, and S. Yeung. Diagnosing and rectifying vision models using language. In ICLR, 2023

    2023

  28. [36]

    Retrieve a relevant item that represents: {Query}\n

    Y . Zhang, E. Sui, and S. Yeung-Levy. Connect, collapse, corrupt: Learning cross-modal tasks with uni-modal data. In ICLR, 2024. 11 Limitations While our work demonstrates that removing the modality gap enables GR-CLIP to achieve substantial performance gains in the mixed moda...

    2024

  29. [2008]

    and Henry Howard, features American fresco mural paintings by 25 different onsite artists

    The unpainted reinforced concrete tower, designed by Arthur Brown, Jr. and Henry Howard, features American fresco mural paintings by 25 different onsite artists... Rank No.5, Cosine Similarity = 0.5252, Modality = multimodal Ilinden (Memorial). Also known as Makedonium, Ilinde...

    1974

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.