Pith. sign in

REVIEW 2 major objections 5 minor 49 references

CLaMR: Contextualized Late-Interaction for Multimodal Content Retrieval

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single vision-language model that jointly encodes video frames, speech, OCR, and metadata, then scores each modality with token-level late interaction, can learn which modality actually answers a query; the authors report large…

desk verdict A credible multimodal video retriever with a useful new synthetic dataset, but the abstract's headline gain overstates the result by omitting the strongest same-backbone baseline. read the letter →

arxiv 2506.06144 v1 pith:XQFUKERC submitted 2025-06-06 cs.CV cs.CLcs.IR

David Wan , Han Wang , Elias Stengel-Eskin , Jaemin Cho , Mohit Bansal This is my paper
classification cs.CVcs.CLcs.IR
keywords multimodalretrievalvideolateinteractionmodalityselectionvision-languagemodelsynthetictrainingdatacontrastivelearninglong-videoQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that retrieval over web video should treat the video, the speech, the on-screen text, and the metadata as one contextualized document, and should let the model decide which source answers a given query. It presents CLaMR, a retriever that encodes all four modalities together in a single vision-language backbone and scores each query with modality-wise late interaction, so a query matched against the audio track can win even when the visual frames are irrelevant. To train this behavior, the paper contributes MultiVENT 2.0++, a synthetic dataset of 371,644 modality-targeted queries, and a modality-aware contrastive loss. On the MultiVENT 2.0++ test set, CLaMR reaches 58.5 nDCG@10, beating the best single-modality retriever by 25.6 points and the best multi-modality retriever by 35.4 points; on MSR-VTT it reaches 62.4 nDCG@10. The same retriever also improves frame selection for long-video QA on Video-MME and LongVideoBench, which is why a reader should care: it suggests one model can replace per-modality encoders and hand-built fusion rules for multimodal search.

What carries the argument

The load-bearing object is the modality-wise late-interaction score $LI_{mw}(q,d)=\max_{m\in M}\sum_{i=1}^{N_q}\max_{j=1}^{N_{d,m}}\langle E_q^{(i)},E_{d,m}^{(j)}\rangle$, computed from token and patch embeddings produced by a single vision-language backbone (Qwen2.5-VL-3B) that jointly encodes video frames, ASR transcripts, OCR text, and metadata in one sequence. The score is plugged into an InfoNCE contrastive loss, so the model is trained both to rank the correct document above in-batch negatives and to make the correct modality's token-level match win. The other load-bearing component is the training data: MultiVENT 2.0++, built by prompting Gemma-3-27b-it to generate one base query and one query per modality for 91,000 previously unannotated MultiVENT 2.0 videos, yielding 371,644 query-document pairs that teach the model which modality answers a query. Together these components are what let the retriever dynamically select the relevant modality without explicit routers or fusion heuristics.

What would settle it

Give a human annotator a random sample of 500 MultiVENT 2.0++ training queries with the four modality sources of their videos, and ask whether the answer is uniquely available in the intended modality; if a large fraction are answerable from another modality or contain hallucinated content, the modality-selection supervision is weaker than assumed. A second check is to train CLaMR on the 1,504 human-written MultiVENT 2.0 queries alone and compare test performance; if it collapses, the synthetic set is doing the work rather than the architecture.

Watch

Extended reading notes

Core claim

The central claim is that dynamic modality selection can be learned end-to-end rather than engineered with routers or score-averaging. CLaMR concatenates video frames, ASR transcripts, OCR text, and metadata into one sequence, encodes them jointly with a vision-language model, and then computes a separate late-interaction similarity for each modality, taking the maximum over modalities as the query–document score. The training loss combines a standard InfoNCE objective with that modality-wise score, and the synthetic queries in MultiVENT 2.0++ are generated so that each one is answerable primarily from a designated modality. The paper reports that this design consistently outperforms both single-modality retrievers and multi-modality retrievers that average or route their scores, and that the improvement is not just a ranking artifact: the same model retrieves more useful frames for long-video QA.

Load-bearing premise

The gains depend on the assumption that the 371,644 synthetic queries generated by Gemma-3-27b-it are fluent, modality-grounded, and representative of real human retrieval queries; if these queries are noisy, hallucinated, or systematically different from human queries, the model could be learning to fit the synthetic distribution rather than general retrieval.

Editorial extensions

If this is right

  • On the MultiVENT 2.0++ test set, CLaMR improves nDCG@10 by 25.6 points over the best single-modality retriever and by 35.4 points over the best multi-modality retriever, reaching 58.5 nDCG@10.
  • On MSR-VTT, CLaMR reaches 62.4 nDCG@10, outperforming all unimodal and multimodal baselines it was compared with.
  • Ablations show that joint encoding of all modalities is responsible for a large share of the gain: removing contextualization drops nDCG@10 by about 14 points.
  • The modality-accuracy analysis reports 76.4% average accuracy at retrieving from the intended modality, versus 30.9% for a routing baseline, indicating the model learns modality selection without explicit routers.
  • In long-video QA, retrieving frames with CLaMR improves accuracy by 3.50% over LanguageBind on Video-MME (with subtitles) and by 1.42% over the no-sampling baseline on LongVideoBench.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would be to train CLaMR on human-authored modality-targeted queries only, or on a mix of human and synthetic queries, and see whether the MultiVENT 2.0++ gains persist; that would separate the value of the architecture from the value of the synthetic query distribution.
  • The modality-aware loss suggests a general recipe for any retrieval domain where evidence can live in several parallel streams (for example, documents with figures, tables, and footnotes): score each stream separately with late interaction and train with stream-targeted queries. The paper does not claim this generality.
  • The per-modality analysis in Section 6.3 uses LLM-generated and LLM-filtered queries for scaling, with human verification applied only to a small subset; a larger human-verified set would strengthen the claim that the model truly routes to the intended modality.
  • Because the synthetic queries were generated from the same ASR, OCR, and metadata used at training time, a risk is that the model learns dataset-specific text patterns rather than general video understanding; evaluating on a different distribution of queries, such as a search engine log, would test transfer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes CLaMR, a late-interaction retriever that jointly encodes video frames, ASR transcripts, OCR text, and metadata in a single Qwen-VL-2.5 backbone and scores query-document similarity as the maximum over per-modality late-interaction sums. To train it, the authors introduce MultiVENT 2.0++, a synthetic dataset of 371,644 modality-targeted query-document pairs generated by Gemma-3-27b-it from MultiVENT 2.0 videos, and a modality-aware contrastive objective. On MultiVENT 2.0++ and MSR-VTT, CLaMR reports nDCG@10 of 58.5 and 62.4, respectively, outperforming single- and multi-modality baselines; the retrieved frames also improve long-video QA accuracy on Video-MME and LongVideoBench compared with uniform sampling and LanguageBind.

Significance. If the results hold, CLaMR would be a strong example of multimodal late-interaction retrieval with learned dynamic modality selection, and MultiVENT 2.0++ would be a useful resource for training such retrievers. The evaluation is non-circular in an important respect: the MultiVENT 2.0 test videos are excluded from synthetic query generation, and the MSR-VTT evaluation is on a held-out standard split. The paper also releases code and data. However, the magnitude of the claimed improvement over 'the best multi-modality retriever' is overstated, and the fidelity of the synthetic training queries is not independently validated, so the generalizable modality-selection claim remains a risk.

major comments (2)
  1. [Abstract; Table 1] The abstract's claim that CLaMR 'improves nDCG@10 by ... 35.4 over the best multi-modality retriever' is not supported by Table 1. The best multi-modality baseline in that table is Qwen-VL-2.5 pooled with nDCG@10 = 52.2, giving a 6.3-point gap to CLaMR's 58.5; the 35.4-point figure is obtained only by comparing against the mCLIP (avg.) baseline at 23.0, which is not the best. This misreporting inflates the paper's central claim and must be corrected. The abstract's '25.6' should also be reconciled with the 25.7-point difference against ImageBind (Vision) in Table 1.
  2. [§4; Fig. 5; §6.3] The load-bearing premise that the 371,644 Gemma-3-27b-it synthetic queries are faithful training signals is never audited. Because the generator is conditioned on the exact ASR, OCR, and metadata text that CLaMR later indexes, the synthetic queries can share content words with the target modality, so the modality-wise max in Eq. (3) might be minimized by lexical overlap rather than by semantic relevance or true modality selection. The only direct evidence for modality selection, Table 3 in §6.3, is built from LLM-generated and LLM-filtered queries using the same style of prompt, and no sample size, filtering rate, or inter-annotator agreement is reported. The authors should provide a human audit of a random sample of training queries, report the filtering statistics, and evaluate on human-written MultiVENT 2.0 queries split by modality, to rule out the lexical-copy alternative.
minor comments (5)
  1. [General] The method name is written inconsistently as 'CLaMR' in the abstract and 'CLAMR' throughout the main text; please unify the spelling.
  2. [§3.3] The paper does not explicitly state whether the modality-wise late-interaction score LImw of Eq. (3) is used at inference or only during training; Figure 1 suggests it is used at inference, but this should be stated clearly in the text.
  3. [§4] The relationship between '91k unannotated videos' and '371,644 query-document pairs' is not explained, since 91,000 times four query types gives approximately 364,000 pairs; please clarify the source of the extra pairs.
  4. [§6.3; Table 3] The router baseline in Table 3 is described only as 'selects a modality per query based on similarity to query type embeddings,' and the mCLIP video accuracy of 0.0 is unexplained; please provide implementation details and a sanity check.
  5. [Table 1] The CLAMR (VLM) row lacks spaces between the MSR-VTT numbers, the Qwen-VL-2.5 pooled row is not flagged for statistical significance, and the abstract's 25.6 differs from the 25.7 computed from the table; please correct these presentation issues.

Circularity Check

1 steps flagged · score 2.0 of 10

Central retrieval results are not circular: headline nDCG numbers come from human-labeled MultiVENT 2.0 and MSRVTT, external to the synthetic training set. The only protocol overlap is the secondary Section 6.3 modality-accuracy analysis, which reuses the same LLM query-generation distribution as training.

  1. fitted input called prediction [Section 6.3 ('Query-Specific Analysis'), subsections 'Generating Synthetic Modality-Specific Queries' and 'Results and Accuracy Breakdown']
    "To scale this analysis, we generate new queries using an LLM prompted with four modality-specific documents (video, ASR, OCR, and metadata) and instructed to produce a query answerable only by one target modality. We then reapply our filtering step to verify that no other modality could answer the generated query. ... CLAMR dramatically outperforms the router and mCLIP baseline across all modalities, achieving an average accuracy of 76.4% versus 30.9%."

    This is not the central retrieval claim, but as evidence for 'query-specific modality selection' it is partly circular. The MultiVENT 2.0++ training queries are produced by the same recipe: an LLM conditioned on the video title, ASR, OCR, and description is asked to emit a query answerable 'primarily' from a designated modality (Sec. 4, Fig. 3/5). CLAMR's objective (Eq. 3) is trained to maximize the per-modality late-interaction score for exactly these labels. Section 6.3 then measures 'modality accuracy' on a fresh batch of queries generated with the same LLM-conditioned-on-modalities protocol and filtered by the same style of LLM judge.

full rationale

The main derivation chain is: build synthetic modality-targeted queries (Sec. 4), train the Eq. 3 contrastive objective, then evaluate on the public MultiVENT 2.0 human-labeled test split and on MSRVTT. The paper explicitly states that the videos behind the 1,504 MultiVENT 2.0 test queries were not used in the synthetic generation process, and MSRVTT is a standard independent benchmark, so the headline retrieval improvements are external and falsifiable rather than a restatement of the training signal. The ablations in Table 2 also compare against the same backbone with a pooled representation and without contextualization, providing internal controls. The self-citations that appear ([6], [38]) are not load-bearing: [6] is a related-work mention and [38] supports a qualitative claim about distracting full-frame inputs; neither is invoked as a uniqueness theorem or as the source of CLAMR's design. The one notable protocol overlap is Section 6.3, where the modality-accuracy evaluation is generated with the same LLM-conditioned-on-modalities prompt family as the training data, making that specific analysis an in-distribution check rather than independent evidence of generalization to human paraphrases. This limits the evidentiary value of the modality-accuracy table but does not contaminate the main nDCG results, which are evaluated on human-labeled data. Overall circularity is low.

Assumptions & free parameters 10 free parameters · 7 assumptions · 0 invented entities

No new physical or theoretical entities are postulated. MultiVENT 2.0++ is a new dataset, not an invented entity; it is paired with released data and external held-out evaluation, giving independent evidence. The main epistemic load is carried by synthetic query generation and pre-extracted modality features.

free parameters (10)
  • Learnable InfoNCE temperature tau = not reported
    Learned scalar in Eq. 2; standard in InfoNCE but counted because it is a fitted parameter.
  • Projection dimension D = 128
    Linear projection following ColPali; not swept.
  • LoRA rank r = 128
    Set for all experiments; no sweep reported.
  • LoRA alpha = 128
    Set to match rank; no sweep reported.
  • Learning rate = 1e-5
    Default chosen for all experiments; no sweep.
  • Training epochs = 1 (MultiVENT 2.0++), 5 (MSRVTT)
    Chosen per dataset; no ablation on epochs.
  • Batch size = 16
    Limited by GPU memory; not tuned.
  • Maximum query length = 64 tokens
    Set based on query lengths; no sweep.
  • Document modality token cap = 256 tokens per text modality; 10 frames; audio 30s/750 tokens
    Capacity choices for VLM input.
  • Number of placeholder query padding tokens = 5
    Added following prior late-interaction work to re-weight query terms; not tuned.
assumptions (7)
  • domain assumption MultiVENT 2.0 and MSRVTT human relevance labels are accurate and complete.
    Evaluation metrics rely entirely on these labels; incorrect labels would change all reported numbers.
  • domain assumption Pre-extracted keyframes, Whisper ASR, OCR, and metadata from MultiVENT 2.0 are correctly aligned and sufficient for retrieval.
    The document representation is fixed by these extractions; errors or misalignment would propagate to retrieval and QA.
  • domain assumption Concatenating all modality tokens into a single VLM sequence preserves enough modality identity for dynamic selection.
    The joint encoder relies on this to produce contextualized but separable modality representations.
  • domain assumption LLM-generated MultiVENT 2.0++ queries are grounded in the intended modality and transfer to human queries.
    The training signal is synthetic; no human validation of training-query grounding is reported.
  • ad hoc to paper The 1,504 MultiVENT 2.0 test videos were not used in synthetic query generation.
    Authors assert this in Section 4; if false, the test set would be contaminated by training distribution overlap.
  • standard math InfoNCE with in-batch negatives provides a valid learning signal for retrieval.
    Standard contrastive learning, but batch size 16 makes the negative set small.
  • domain assumption Max-over-token cosine similarity is a suitable relevance function for multimodal documents.
    Late interaction assumes a single strong token match can indicate relevance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CLaMR: Contextualized Late-Interaction for Multimodal Content Retrieval." pith.science (2026). https://pith.science/paper/XQFUKERC

@misc{pith2026250606144,
  author       = {Pith},
  title        = {Pith review of: CLaMR: Contextualized Late-Interaction for Multimodal Content Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XQFUKERC}},
  note         = {Machine review of arXiv:2506.06144}
}
read the original abstract

Online video web content is richly multimodal: a single video blends vision, speech, ambient audio, and on-screen text. Retrieval systems typically treat these modalities as independent retrieval sources, which can lead to noisy and subpar retrieval. We explore multimodal video content retrieval, where relevance can be scored from one particular modality or jointly across multiple modalities simultaneously. Consequently, an effective retriever must dynamically choose which modality (or set of modalities) best addresses the query. We introduce CLaMR, a multimodal, late-interaction retriever that jointly indexes 4 modalities: video frames, transcribed speech, on-screen text, and metadata. CLaMR jointly encodes all modalities with a unified multimodal backbone for improved contextualization and is trained to enhance dynamic modality selection via two key innovations. First, given the lack of training data for multimodal retrieval, we introduce MultiVENT 2.0++, a large-scale synthetic training dataset built on MultiVENT 2.0 (event-centric videos in various languages paired with queries) with modality-targeted queries. Next, we propose a modality-aware loss that jointly trains according to a standard contrastive objective alongside an objective for learning correct modality usage. On the test sets of MultiVENT 2.0++ and MSRVTT, conventional aggregation strategies, such as averaging similarities for baseline retrievers, degrade performance by introducing noise from irrelevant modalities. In contrast, CLaMR consistently outperforms existing retrievers: on MultiVENT 2.0++, CLaMR improves nDCG@10 by 25.6 over the best single-modality retriever and by 35.4 over the best multi-modality retriever. We illustrate CLaMR's downstream utility on long-video QA, retrieving relevant frames and obtaining a 3.50% boost over LanguageBind on Video-MME and 1.42% over dense sampling on LongVideoBench.

Figures

Figures reproduced from arXiv: 2506.06144 by the authors.

Figure 1
Figure 1. Illustration of multimodal video content retrieval task with standard retrieval and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. CLAMR with modality-wise late interaction for multimodal contrastive learning. A text query and multimodal video document consisting of video’s visual, audio, OCR, and metadata signals are encoded by the model. Then, token-level late interaction yields a similarity score for each modality; the highest of these scores becomes the query-document similarity. Similarities for the positive pair and in-batch negatives are… view at source ↗
Figure 3
Figure 3. Illustration of deriving modality-specific queries from multimodal video content. An LLM [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Prompts used for filtering relevant modality and generating synthetic modality-specific [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]
Figure 5
Figure 5. Figure 5: Prompt structure for synthetic query generation for [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Prompt for router with GPT-4.1. C.1 Safeguards for MULTIVENT 2.0++ The videos utilized are from the MULTIVENT 2.0 dataset. We rely on the safeguarding measures implemented by the original authors for this content and do not redistribute the videos. For our syntheticall…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

49 extracted references · 19 canonical work pages

  1. [1]

    Qwen2.5-vl technical report,

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report,

  2. [2]

    Vlmo: Unified vision- language pre-training with mixture-of-modality-experts

    Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vision- language pre-training with mixture-of-modality-experts. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors,Advances in Neural Infor- mation Processing Systems, volume 35, pages 32897–3...

    work page

  3. [3]

    Valor: Vision-audio-language omni-perception pretraining model and dataset, 2023

    Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, Jinhui Tang, and Jing Liu. Valor: Vision-audio-language omni-perception pretraining model and dataset, 2023

    work page 2023

  4. [4]

    V AST: A vision-audio-subtitle-text omni-modality foundation model and dataset

    Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. V AST: A vision-audio-subtitle-text omni-modality foundation model and dataset. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: //openreview.net/forum?id=scYa9DYUAy

    work page 2023

  5. [5]

    PaLI: A jointly-scaled multilingual language-image model

    Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish V Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Ca...

    work page 2023

  6. [6]

    M3docrag: Multi- modal retrieval is what you need for multi-page multi-document understanding, 2024

    Jaemin Cho, Debanjan Mahata, Ozan Irsoy, Yujie He, and Mohit Bansal. M3docrag: Multi- modal retrieval is what you need for multi-page multi-document understanding, 2024

    work page 2024

  7. [7]

    Jacolbertv2.5: Optimising multi-vector retrievers to create state-of-the-art japanese retrievers with constrained resources, 2024

    Benjamin Clavié. Jacolbertv2.5: Optimising multi-vector retrievers to create state-of-the-art japanese retrievers with constrained resources, 2024. URL https://arxiv.org/abs/2407. 20750

    work page 2024

  8. [8]

    Cormack, Charles L A Clarke, and Stefan Buettcher

    Gordon V . Cormack, Charles L A Clarke, and Stefan Buettcher. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. InProceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’09, page 758–759, New York, NY , USA, 2009. Association for Computing Machinery. ISBN 9781...

    arXiv 2009

Show all 49 references
  1. [9]

    Qlora: Efficient finetuning of quantized llms.arXiv preprint arXiv:2305.14314, 2023

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms.arXiv preprint arXiv:2305.14314, 2023

    2023 arXiv

  2. [10]

    Tibshirani.An Introduction to the Bootstrap

    Bradley Efron and Robert J. Tibshirani.An Introduction to the Bootstrap. Number 57 in Monographs on Statistics and Applied Probability. Chapman & Hall/CRC, Boca Raton, Florida, USA, 1993. 11

    1993

  3. [11]

    A hybrid model for multilingual ocr

    David Etter, Cameron Carpenter, and Nolan King. A hybrid model for multilingual ocr. In Document Analysis and Recognition - ICDAR 2023: 17th International Conference, San José, CA, USA, August 21–26, 2023, Proceedings, Part I, page 467–483, Berlin, Heidelberg, 2023. Springer-V...

    2023 doi

  4. [12]

    Colpali: Efficient document retrieval with vision language models

    Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, CELINE HUDELOT, and Pierre Colombo. Colpali: Efficient document retrieval with vision language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openreview.net...

    2025

  5. [13]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis, 2024

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-mme: The first-ever compre...

    2024 arXiv

  6. [14]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. InCVPR, 2023

    2023

  7. [15]

    Cumulated gain-based evaluation of ir techniques.ACM Trans

    Kalervo Järvelin and Jaana Kekäläinen. Cumulated gain-based evaluation of ir techniques.ACM Trans. Inf. Syst., 20(4):422–446, October 2002. ISSN 1046-8188. doi: 10.1145/582415.582418. URLhttps://doi.org/10.1145/582415.582418

    2002

  8. [16]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Marina Meila and Tong Zhang, editors,Proceedings of the 38th I...

    2021

  9. [17]

    Vlm2vec: Training vision-language models for massive multimodal embedding tasks, 2025

    Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. Vlm2vec: Training vision-language models for massive multimodal embedding tasks, 2025. URL https: //arxiv.org/abs/2410.05160

    2025 arXiv

  10. [18]

    Dense passage retrieval for open-domain question answering

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors,Proceedings of the 2020 Conference on Em...

    2020 doi

  11. [19]

    Colbert: Efficient and effective passage search via con- textualized late interaction over bert

    Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via con- textualized late interaction over bert. InProceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’20, page 39–48, New York, ...

    2020

  12. [20]

    Multivent 2.0: A massive multilingual benchmark for event-centric video retrieval, 2025

    Reno Kriz, Kate Sanders, David Etter, Kenton Murray, Cameron Carpenter, Kelly Van Ochten, Hannah Recknor, Jimena Guallar-Blasco, Alexander Martin, Ronald Colaianni, Nolan King, Eugene Yang, and Benjamin Van Durme. Multivent 2.0: A massive multilingual benchmark for event-centr...

    2025 arXiv

  13. [21]

    Learning to rank for information retrieval.Found

    Tie-Yan Liu. Learning to rank for information retrieval.Found. Trends Inf. Retr., 3(3):225–331, March 2009. ISSN 1554-0669. URLhttps://doi.org/10.1561/1500000016

    2009 doi

  14. [22]

    Clip4clip: An empirical study of clip for end to end video clip retrieval, 2021

    Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval, 2021. URL https://arxiv. org/abs/2104.08860

    2021 arXiv

  15. [23]

    Handwritten optical character recognition (ocr): A comprehensive systematic literature review (slr).IEEE Access, 8: 142642–142668, 2020

    Jamshed Memon, Maira Sami, Rizwan Ahmed Khan, and Mueen Uddin. Handwritten optical character recognition (ocr): A comprehensive systematic literature review (slr).IEEE Access, 8: 142642–142668, 2020. doi: 10.1109/ACCESS.2020.3012542. 12

    2020

  16. [24]

    Vladva: Discriminative fine-tuning of lvlms,

    Yassine Ouali, Adrian Bulat, Alexandros Xenos, Anestis Zaganidis, Ioannis Maniadis Metaxas, Brais Martinez, and Georgios Tzimiropoulos. Vladva: Discriminative fine-tuning of lvlms,

  17. [25]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila an...

    2021

  18. [27]

    URLhttps://arxiv.org/abs/2412.04378

    arXiv

  19. [28]

    de Melo, Benjamin Van Durme, and Rama Chellappa

    Arun Reddy, Alexander Martin, Eugene Yang, Andrew Yates, Kate Sanders, Kenton Murray, Reno Kriz, Celso M. de Melo, Benjamin Van Durme, and Rama Chellappa. Video-colbert: Contextualized late interaction for text-to-video retrieval, 2025. URL https://arxiv.org/ abs/2503.19009

    2025 arXiv

  20. [29]

    Sentence-bert: Sentence embeddings using siamese bert- networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL http://arxiv.org/ abs/1908.10084

    2019 arXiv

  21. [30]

    Robust speech recognition via large-scale weak supervision, 2022

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision, 2022. URL https: //arxiv.org/abs/2212.04356

    2022 arXiv

  22. [31]

    ColBERTv2: Effective and efficient retrieval via lightweight late interaction

    Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. ColBERTv2: Effective and efficient retrieval via lightweight late interaction. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz, editors,Proceedings of the 2022 Co...

    2022

  23. [32]

    R. Smith. An overview of the tesseract ocr engine. InNinth International Conference on Document Analysis and Recognition (ICDAR 2007), volume 2, pages 629–633, 2007. doi: 10.1109/ICDAR.2007.4376991

    2007

  24. [33]

    Mmmorrf: Multimodal multilingual modularized reciprocal rank fusion, 2025

    Saron Samuel, Dan DeGenaro, Jimena Guallar-Blasco, Kate Sanders, Oluwaseun Eisape, Arun Reddy, Alexander Martin, Andrew Yates, Eugene Yang, Cameron Carpenter, David Etter, Efsun Kayi, Matthew Wiesner, Kenton Murray, and Reno Kriz. Mmmorrf: Multimodal multilingual modularized r...

    2025 arXiv

  25. [34]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, ...

    2025 arXiv

  26. [35]

    BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models

    Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Ro...

    2021

  27. [36]

    Generative multimodal models are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),...

    2024

  28. [37]

    Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors,Advances in Neural Information Pr...

    2020

  29. [38]

    Videotree: Adaptive tree-based video representation for llm reasoning on long videos.arXiv preprint arXiv:2405.19209, 2024

    Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Videotree: Adaptive tree-based video representation for llm reasoning on long videos.arXiv preprint arXiv:2405.19209, 2024

    2024 arXiv

  30. [39]

    Representation learning with contrastive predictive coding, 2019

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2019. URLhttps://arxiv.org/abs/1807.03748

    2019 arXiv

  31. [40]

    Longvideobench: A benchmark for long-context interleaved video-language understanding

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors,Advances in Neural Information Processing System...

  32. [41]

    Qwen2.5-omni technical report, 2025

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-omni technical report, 2025. URLhttps://arxiv.org/abs/2503.20215

    2025 arXiv

  33. [42]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020

  34. [43]

    Uni- versalrag: Retrieval-augmented generation over multiple corpora with diverse modalities and granularities, 2025

    Woongyeong Yeo, Kangsan Kim, Soyeong Jeong, Jinheon Baek, and Sung Ju Hwang. Uni- versalrag: Retrieval-augmented generation over multiple corpora with diverse modalities and granularities, 2025. URLhttps://arxiv.org/abs/2504.20734

    2025 arXiv

  35. [44]

    CREAM: Coarse-to-fine retrieval and multi- modal efficient tuning for document VQA

    Jinxu Zhang, Yongqi Yu, and Yu Zhang. CREAM: Coarse-to-fine retrieval and multi- modal efficient tuning for document VQA. InACM Multimedia 2024, 2024. URL https: //openreview.net/forum?id=uxxdE9HFGI

    2024

  36. [45]

    Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment,

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, Wancai Zhang, Zhifeng Li, Wei Liu, and Li Yuan. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment,

  37. [46]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016

    2016

  38. [50]

    URLhttps://arxiv.org/abs/2310.01852. A Additional Experiments A.1 Exploration of Different Contrastive Loss Formulations We investigated two alternative formulations of the contrastive objective, each designed to progres- sively enforce the contribution of the single, most rel...

    arXiv

  39. [2022]

    URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ d46662aa53e78a62afd980a29e0c37ed-Paper-Conference.pdf

    2022

  40. [2024]

    URL https://proceedings.neurips.cc/paper_files/paper/2024/file/ 329ad516cf7a6ac306f29882e9c77558-Paper-Datasets_and_Benchmarks_Track. pdf. 14

    2024

  41. [2025]

    URLhttps://arxiv.org/abs/2502.13923

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.