{"id":"fe59421a-7306-448d-91f3-5265f09c5177","arxiv_id":"2506.06144","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":10,"one_line_summary":"CLaMR jointly encodes four video modalities in a vision-language model and uses token-level, per-modality matching to retrieve the right video and the right modality for a query.","lead":"CLaMR is a video search model that reads frames, spoken words, on-screen text, and metadata together, then matches a query to whichever part of the video contains the answer. It beats earlier retrieval methods on two video search benchmarks and helps question-answering systems pick better frames from long videos.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Synthetic query fidelity is the load-bearing premise: neither training queries nor the §6.3 modality-selection evaluation independently validate that CLaMR learns generalizable modality selection rather than lexical overlap.","rationale":"The reader's conditional verdict is appropriate, and the reader's weakest assumption identifies the same load-bearing premise. The paper has real independent support: released code and data, a same-backbone pooled baseline, ablations isolating contextualization and modality-wise late interaction, bootstrap significance, and downstream QA results that are consistent with better retrieval. None of these, however, validates the distributional assumption on which training and the modality-selection analysis rest. The §6.3 filtering pipeline is not an independent check because it uses the same LLM-generation/filtering style as the training data. The proposed paraphrase-controlled retraining holds architecture and objective fixed and varies only the lexical coupling between synthetic queries and documents; it therefore isolates the mechanism the paper claims to be the cause of its gains. If the result is robust to paraphrase, the concern is resolved in the authors' favor; if not, the headline numbers should be read as in-distribution evidence and the dynamic-modality-selection claim should be treated as unverified. A secondary issue flagged in the text is the absence of VideoColBERT [28] from Table 1; I do not make it the primary attack because even a strong VideoColBERT result would not resolve whether the synthetic training signal is faithful. This concern sharpens the reader's condition but does not move the verdict.","tokens_in":18906,"tokens_out":10202,"duration_ms":107724,"concrete_test":"Re-generate 20k MultiVENT 2.0++ training queries with the same Gemma-3 prompt plus an explicit paraphrase instruction that caps unigram overlap between each query and its source modality text at 40%; measure overlap before and after. Fine-tune CLaMR from the same Qwen-2.5-VL-3B checkpoint on this paraphrased set with identical epochs/batch/learning rate, and evaluate on the MultiVENT 2.0 public test queries. If nDCG@10 falls by more than ~3 points from 58.5, the original signal exploits lexical overlap and the dynamic-modality-selection claim is not demonstrated; if performance is stable, the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CLaMR learns dynamic modality selection and therefore consistently outperforms existing retrievers. The load-bearing premise, introduced in §4 and operationalized by the prompt in Appendix C (Fig. 5), is that the 371,644 Gemma-3-27b-it synthetic queries are faithful training signals: fluent, grounded in the intended modality, and representative of human MultiVENT 2.0 queries. That premise is never audited. The generator is conditioned on the exact video title, ASR transcript, OCR text, and description that CLaMR later indexes, and is asked to emit a 'concise and specific query' derived from that text. Such queries will share content words and n-grams with the target modality. Because the training/inference score in Eq. (3) is a max over per-modality late-interaction sums, and identical tokens receive near-maximum cosine similarity, InfoNCE can be minimized by lexical string matching rather than by semantic relevance or true modality selection. The only modality-fidelity evidence (§6.3) is built from LLM-generated and LLM-filtered queries using the same style of prompt; human annotators verify surviving examples, but no sample size, filtering rate, or inter-annotator agreement is reported. Thus training and the modality-selection evaluation share the same potential lexical-copy bias. If human queries are more paraphrastic, the reported nDCG@10 deltas (25.6/35.4) would not establish a generalizable modality-selection retriever.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CLaMR, a late-interaction retriever that jointly encodes video frames, ASR transcripts, OCR text, and metadata in a single Qwen-VL-2.5 backbone and scores query-document similarity as the maximum over per-modality late-interaction sums. To train it, the authors introduce MultiVENT 2.0++, a synthetic dataset of 371,644 modality-targeted query-document pairs generated by Gemma-3-27b-it from MultiVENT 2.0 videos, and a modality-aware contrastive objective. On MultiVENT 2.0++ and MSR-VTT, CLaMR reports nDCG@10 of 58.5 and 62.4, respectively, outperforming single- and multi-modality baselines; the retrieved frames also improve long-video QA accuracy on Video-MME and LongVideoBench compared with uniform sampling and LanguageBind.","tokens_in":19250,"tokens_out":8488,"duration_ms":72518,"significance":"If the results hold, CLaMR would be a strong example of multimodal late-interaction retrieval with learned dynamic modality selection, and MultiVENT 2.0++ would be a useful resource for training such retrievers. The evaluation is non-circular in an important respect: the MultiVENT 2.0 test videos are excluded from synthetic query generation, and the MSR-VTT evaluation is on a held-out standard split. The paper also releases code and data. However, the magnitude of the claimed improvement over 'the best multi-modality retriever' is overstated, and the fidelity of the synthetic training queries is not independently validated, so the generalizable modality-selection claim remains a risk.","major_comments":[{"comment":"The abstract's claim that CLaMR 'improves nDCG@10 by ... 35.4 over the best multi-modality retriever' is not supported by Table 1. The best multi-modality baseline in that table is Qwen-VL-2.5 pooled with nDCG@10 = 52.2, giving a 6.3-point gap to CLaMR's 58.5; the 35.4-point figure is obtained only by comparing against the mCLIP (avg.) baseline at 23.0, which is not the best. This misreporting inflates the paper's central claim and must be corrected. The abstract's '25.6' should also be reconciled with the 25.7-point difference against ImageBind (Vision) in Table 1.","section":"Abstract; Table 1"},{"comment":"The load-bearing premise that the 371,644 Gemma-3-27b-it synthetic queries are faithful training signals is never audited. Because the generator is conditioned on the exact ASR, OCR, and metadata text that CLaMR later indexes, the synthetic queries can share content words with the target modality, so the modality-wise max in Eq. (3) might be minimized by lexical overlap rather than by semantic relevance or true modality selection. The only direct evidence for modality selection, Table 3 in §6.3, is built from LLM-generated and LLM-filtered queries using the same style of prompt, and no sample size, filtering rate, or inter-annotator agreement is reported. The authors should provide a human audit of a random sample of training queries, report the filtering statistics, and evaluate on human-written MultiVENT 2.0 queries split by modality, to rule out the lexical-copy alternative.","section":"§4; Fig. 5; §6.3"}],"minor_comments":[{"comment":"The method name is written inconsistently as 'CLaMR' in the abstract and 'CLAMR' throughout the main text; please unify the spelling.","section":"General"},{"comment":"The paper does not explicitly state whether the modality-wise late-interaction score LImw of Eq. (3) is used at inference or only during training; Figure 1 suggests it is used at inference, but this should be stated clearly in the text.","section":"§3.3"},{"comment":"The relationship between '91k unannotated videos' and '371,644 query-document pairs' is not explained, since 91,000 times four query types gives approximately 364,000 pairs; please clarify the source of the extra pairs.","section":"§4"},{"comment":"The router baseline in Table 3 is described only as 'selects a modality per query based on similarity to query type embeddings,' and the mCLIP video accuracy of 0.0 is unexplained; please provide implementation details and a sanity check.","section":"§6.3; Table 3"},{"comment":"The CLAMR (VLM) row lacks spaces between the MSR-VTT numbers, the Qwen-VL-2.5 pooled row is not flagged for statistical significance, and the abstract's 25.6 differs from the 25.7 computed from the table; please correct these presentation issues.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper has a genuinely interesting core idea and the held-out evaluation is a strength. However, the abstract's headline numbers are misleading and the synthetic query fidelity issue is unresolved. I would ask the authors to correct the abstract and add the validation suggested in major comment 2; if those are satisfied, I would support publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"One thing to know: CLaMR is a credible late-interaction retriever for multimodal video, and the MultiVENT 2.0++ synthetic dataset is a real resource. But the abstract's claim of '35.4 nDCG@10 over the best multi-modality retriever' is misleading — the strongest multimodal baseline, a same-backbone Qwen-VL pooled model, sits in Table 1 at 52.2 nDCG@10, and the actual gain over that model is 6.3 points. The 35.4 number comes from comparing against much weaker fusion baselines like mCLIP averaging.\n\nWhat's actually new: the paper combines joint VLM encoding with modality-wise late interaction and a max-over-modalities contrastive loss. That is a sensible extension of ColPali-style late interaction to four-modality video retrieval, and it works: on MultiVENT 2.0++ test and MSRVTT they beat the pooled Qwen-VL baseline by 6.3 and 8.6 nDCG@10 respectively, which is a real same-backbone improvement. The synthetic data generation is careful — they build on MultiVENT 2.0, exclude test videos, and generate modality-targeted queries from ASR, OCR, and metadata. The long-video QA results are a nice extra, though not the main contribution.\n\nSoft spots. First, the reporting: the abstract's comparison is selective, and the paper should have led with the Qwen-VL pooled baseline. Second, no error bars or multiple seeds are reported; the paired bootstrap significance is a start, but single runs on a dataset this size are thin. Third, the load-bearing concern is the fidelity of the synthetic training queries. The generator is conditioned on the exact ASR, OCR, and metadata text that the retriever later indexes, so the queries share lexical content with the target modalities. The modality-accuracy analysis in §6.3 uses LLM-generated and filtered queries of the same style, with no human validation of the training queries themselves. This means the 'dynamic modality selection' story could partly be lexical-string matching rather than semantic generalization. That said, the evaluation on human-annotated MultiVENT 2.0 queries and MSRVTT is external to the synthetic training set, so the core retrieval claim isn't circular; it's the modality-selection interpretation that's less secure.\n\nOverall, this is a solid subfield contribution that deserves a serious referee. The method is coherent, the dataset is a genuine resource, and the central retrieval result holds up. The paper needs revision — fix the abstract, add variance information, and do a human audit of a sample of synthetic queries (or a paraphrase-based test) to back the modality-selection claim. I'd send it to review.","headline":"A credible multimodal video retriever with a useful new synthetic dataset, but the abstract's headline gain overstates the result by omitting the strongest same-backbone baseline.","tokens_in":19799,"tokens_out":3283,"would_cite":true,"duration_ms":28291,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single vision-language model that jointly encodes video frames, speech, OCR, and metadata, then scores each modality with token-level late interaction, can learn which modality actually answers a query; the authors report large…","keywords":["multimodal retrieval","video retrieval","late interaction","modality selection","vision-language model","synthetic training data","contrastive learning","long-video QA"],"falsifier":"Give a human annotator a random sample of 500 MultiVENT 2.0++ training queries with the four modality sources of their videos, and ask whether the answer is uniquely available in the intended modality; if a large fraction are answerable from another modality or contain hallucinated content, the modality-selection supervision is weaker than assumed. A second check is to train CLaMR on the 1,504 human-written MultiVENT 2.0 queries alone and compare test performance; if it collapses, the synthetic set is doing the work rather than the architecture.","tokens_in":18686,"feed_emoji":"🎬","tokens_out":7900,"duration_ms":64438,"temperature":0.7,"pith_summary":"The paper argues that retrieval over web video should treat the video, the speech, the on-screen text, and the metadata as one contextualized document, and should let the model decide which source answers a given query. It presents CLaMR, a retriever that encodes all four modalities together in a single vision-language backbone and scores each query with modality-wise late interaction, so a query matched against the audio track can win even when the visual frames are irrelevant. To train this behavior, the paper contributes MultiVENT 2.0++, a synthetic dataset of 371,644 modality-targeted queries, and a modality-aware contrastive loss. On the MultiVENT 2.0++ test set, CLaMR reaches 58.5 nDCG@10, beating the best single-modality retriever by 25.6 points and the best multi-modality retriever by 35.4 points; on MSR-VTT it reaches 62.4 nDCG@10. The same retriever also improves frame selection for long-video QA on Video-MME and LongVideoBench, which is why a reader should care: it suggests one model can replace per-modality encoders and hand-built fusion rules for multimodal search.","feed_headline":"CLaMR tops single-modality video retrievers by 25.6 points","feed_subtitle":"Late interaction over frames, speech, OCR, and metadata learns which source answers each query.","key_machinery":"The load-bearing object is the modality-wise late-interaction score $LI_{mw}(q,d)=\\max_{m\\in M}\\sum_{i=1}^{N_q}\\max_{j=1}^{N_{d,m}}\\langle E_q^{(i)},E_{d,m}^{(j)}\\rangle$, computed from token and patch embeddings produced by a single vision-language backbone (Qwen2.5-VL-3B) that jointly encodes video frames, ASR transcripts, OCR text, and metadata in one sequence. The score is plugged into an InfoNCE contrastive loss, so the model is trained both to rank the correct document above in-batch negatives and to make the correct modality's token-level match win. The other load-bearing component is the training data: MultiVENT 2.0++, built by prompting Gemma-3-27b-it to generate one base query and one query per modality for 91,000 previously unannotated MultiVENT 2.0 videos, yielding 371,644 query-document pairs that teach the model which modality answers a query. Together these components are what let the retriever dynamically select the relevant modality without explicit routers or fusion heuristics.","core_discovery":"The central claim is that dynamic modality selection can be learned end-to-end rather than engineered with routers or score-averaging. CLaMR concatenates video frames, ASR transcripts, OCR text, and metadata into one sequence, encodes them jointly with a vision-language model, and then computes a separate late-interaction similarity for each modality, taking the maximum over modalities as the query–document score. The training loss combines a standard InfoNCE objective with that modality-wise score, and the synthetic queries in MultiVENT 2.0++ are generated so that each one is answerable primarily from a designated modality. The paper reports that this design consistently outperforms both single-modality retrievers and multi-modality retrievers that average or route their scores, and that the improvement is not just a ranking artifact: the same model retrieves more useful frames for long-video QA.","pith_inferences":["A testable extension would be to train CLaMR on human-authored modality-targeted queries only, or on a mix of human and synthetic queries, and see whether the MultiVENT 2.0++ gains persist; that would separate the value of the architecture from the value of the synthetic query distribution.","The modality-aware loss suggests a general recipe for any retrieval domain where evidence can live in several parallel streams (for example, documents with figures, tables, and footnotes): score each stream separately with late interaction and train with stream-targeted queries. The paper does not claim this generality.","The per-modality analysis in Section 6.3 uses LLM-generated and LLM-filtered queries for scaling, with human verification applied only to a small subset; a larger human-verified set would strengthen the claim that the model truly routes to the intended modality.","Because the synthetic queries were generated from the same ASR, OCR, and metadata used at training time, a risk is that the model learns dataset-specific text patterns rather than general video understanding; evaluating on a different distribution of queries, such as a search engine log, would test transfer."],"forward_implications":["On the MultiVENT 2.0++ test set, CLaMR improves nDCG@10 by 25.6 points over the best single-modality retriever and by 35.4 points over the best multi-modality retriever, reaching 58.5 nDCG@10.","On MSR-VTT, CLaMR reaches 62.4 nDCG@10, outperforming all unimodal and multimodal baselines it was compared with.","Ablations show that joint encoding of all modalities is responsible for a large share of the gain: removing contextualization drops nDCG@10 by about 14 points.","The modality-accuracy analysis reports 76.4% average accuracy at retrieving from the intended modality, versus 30.9% for a routing baseline, indicating the model learns modality selection without explicit routers.","In long-video QA, retrieving frames with CLaMR improves accuracy by 3.50% over LanguageBind on Video-MME (with subtitles) and by 1.42% over the no-sampling baseline on LongVideoBench."],"supporting_citations":[{"why":"Supplies the MultiVENT 2.0 dataset of event-centric videos and the 1,504 human-judged test queries that are the basis for synthetic expansion and evaluation.","marker":"[20]"},{"why":"Introduces ColBERT's late interaction, the token-level max-similarity scoring mechanism that CLaMR adapts to a per-modality formulation.","marker":"[19]"},{"why":"Provides ColPali's precedent for applying late interaction to vision-language models and the projection-layer design CLaMR follows.","marker":"[12]"},{"why":"Gemma-3-27b-it is the LLM used to generate the 371,644 synthetic modality-targeted queries in MultiVENT 2.0++.","marker":"[34]"},{"why":"Qwen2.5-VL-3B is the unified vision-language backbone that jointly encodes all four modalities in CLaMR.","marker":"[1]"},{"why":"InfoNCE is the base contrastive objective that the modality-aware loss extends.","marker":"[36]"},{"why":"LanguageBind serves as a strong multi-modality baseline for retrieval and as the comparison retriever in long-video QA.","marker":"[45]"},{"why":"MSR-VTT is the standard text-video retrieval benchmark used for the second evaluation set.","marker":"[42]"},{"why":"Video-MME is the long-video QA benchmark used to show downstream utility of CLaMR's frame retrieval.","marker":"[13]"},{"why":"LongVideoBench is the second long-video QA benchmark used to evaluate the retrieved-frame pipeline.","marker":"[40]"}],"fun_headline_variants":["CLaMR picks the right modality per query, boosting retrieval by 25.6 nDCG@10","Late-interaction retriever learns when to use video, speech, or OCR text","CLaMR adapts modality choice per query, beating aggregate retrievers","Dynamic modality selection lifts video retrieval by 25.6 nDCG@10"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gains depend on the assumption that the 371,644 synthetic queries generated by Gemma-3-27b-it are fluent, modality-grounded, and representative of real human retrieval queries; if these queries are noisy, hallucinated, or systematically different from human queries, the model could be learning to fit the synthetic distribution rather than general retrieval.","fun_headline_variants_meta":{"raw":{"variants":["CLaMR picks the right modality per query, boosting retrieval by 25.6 nDCG@10","Late-interaction retriever learns when to use video, speech, or OCR text","CLaMR adapts modality choice per query, beating aggregate retrievers","Dynamic modality selection lifts video retrieval by 25.6 nDCG@10"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000786,"raw_usage":{"total_tokens":3530,"prompt_tokens":1072,"completion_tokens":2458,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":688,"completion_tokens_details":{"reasoning_tokens":2366}},"tokens_in":688,"tokens_out":2458,"duration_ms":15734,"temperature":1.0,"reasoning_tokens":2366,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:59:35.337008+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give a human annotator a random sample of 500 MultiVENT 2.0++ training queries with the four modality sources of their videos, and ask whether the answer is uniquely available in the intended modality; if a large fraction are answerable from another modality or contain hallucinated content, the modality-selection supervision is weaker than assumed. A second check is to train CLaMR on the 1,504 human-written MultiVENT 2.0 queries alone and compare test performance; if it collapses, the synthetic set is doing the work rather than the architecture.","supporting_citations":[{"cited_title":"Colpali: Efficient document retrieval with vision language models","cited_arxiv_id":null,"evidence_quote":"Provides ColPali's precedent for applying late interaction to vision-language models and the projection-layer design CLaMR follows."},{"cited_title":"Generative multimodal models are in-context learners","cited_arxiv_id":null,"evidence_quote":"InfoNCE is the base contrastive objective that the modality-aware loss extends."},{"cited_title":"Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment,","cited_arxiv_id":null,"evidence_quote":"LanguageBind serves as a strong multi-modality baseline for retrieval and as the comparison retriever in long-video QA."},{"cited_title":"Longvideobench: A benchmark for long-context interleaved video-language understanding","cited_arxiv_id":null,"evidence_quote":"LongVideoBench is the second long-video QA benchmark used to evaluate the retrieved-frame pipeline."}],"review_version":1}