{"id":"0757b237-2e29-40da-8f1e-991af7a1a93e","arxiv_id":"2505.19650","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"UNITE combines curated multimodal training data and a modality-masked contrastive loss to achieve strong retrieval performance across text, image, and video tasks.","lead":"This paper introduces UNITE, a multimodal embedding model trained on curated text, image, and video data plus a modality-aware contrastive loss. It reports strong results on several multimodal retrieval benchmarks and analyzes how data composition affects retrieval performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core 'modality curation' recipe is derived from 600K-sample, 2B-model ablations but is never validated at the full 6.4M/7B scale, leaving the attribution of SOTA performance to this curation unsupported.","rationale":"The reader's weakest assumption identifies precisely the load-bearing gap: the data-composition conclusions from 600K/2B are used to justify the final 6.4M/7B recipe without a scaling check. This is the most important concern because the paper's title and main contribution are about modality curation, and the causal claim 'largely due to data curation and MAMCL' depends on the curation recipe actually mattering at the scale where SOTA is reported. The SOTA numbers themselves are directly measured and impressive, so the concern is not about fabrication but about attribution: the paper does not provide a counterfactual showing that an alternative composition at full scale would be worse. The concrete tests I propose would settle whether the curation insight transfers; if it does not, the paper remains a strong empirical recipe but its central scientific contribution is materially weakened. I agree with the reader's CONDITIONAL verdict, so no verdict change is needed.","tokens_in":31036,"tokens_out":11370,"duration_ms":117433,"concrete_test":"Run the Section 5 composition ablation on Qwen2-VL-7B with the same 600K subsets as in Tables 18-19 (TT-only, TI-only, TV-only, and the three pair/three-way mixtures), then compare the ordering of configurations on the same coarse/fine/instruction-based metrics. If TV-only no longer dominates TI-only for image-text retrieval, or TT+TI no longer dominates TV for instruction-following, the curation rationale fails to transfer even at constant data size. A stronger test: train the full 6.4M/7B model with a deliberately different composition (e.g., a TV-heavy mix suggested by Table 9, or an equal TT/TI/TV mix) and identical stage-2 instruction tuning, then compare against UNITE on MMEB, WebVid-CoVR, and CaReBench; if the alternative matches or beats UNITE, the claim that the curated composition is responsible for SOTA performance is falsified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"UNITE's central claim is that data curation and MAMCL are largely responsible for state-of-the-art results. The curation decisions in Section 5 and Appendix B.3 are based on experiments with 600K samples on Qwen2-VL-2B (Tables 9, 18, 19). The paper then builds the final 6.4M-sample, 7B training run using the composition implied by these small-scale results, without ever testing whether the composition ordering (e.g., TV-only outperforming TI-only for image-text retrieval; TT+TI required for instruction following) persists at scale. This is not a trivial extrapolation: Table 9 shows that the full TT+TI+TV mix is often worse than TV-only on cross-modal retrieval (e.g., Flickr30K T→I 58.1 vs 60.2; CaRe V→T 61.2 vs 68.7) at 2B/600K, yet the final recipe includes all three modalities. The paper assumes the larger data volume and model capacity reverse this ordering, but provides no evidence. Furthermore, the small-scale differences between configurations are frequently under 1 point (e.g., MMEB overall 61.1 vs 62.1; CoVR R@1 64.8 vs 65.6) with no reported variance or multiple seeds, so even the small-scale conclusions may not be stable. If the data-composition findings do not transfer to the final scale, the 'modality curation' contribution is unsupported, and the SOTA results could be driven by scale, backbone, or training details rather than the proposed recipe.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents UNITE, a unified multimodal embedder built on Qwen2-VL (2B/7B) with a two-stage training scheme: retrieval adaptation on 6.4M curated text-text, image-text, and video-text pairs, followed by instruction tuning on 1.3M samples from MMEB and WebVid-CoVR. The authors propose Modal-Aware Masked Contrastive Learning (MAMCL), which restricts contrastive comparisons to candidates sharing the same target modality, to reduce inter-modal interference. They report extensive evaluations on fine-grained, coarse-grained, and instruction-based retrieval benchmarks, claiming state-of-the-art results on 40+ tasks. A systematic study in Section 5 examines how data composition (TT, TI, TV, and mixtures) affects downstream performance, and the final recipe is justified by those findings.","tokens_in":31397,"tokens_out":6109,"duration_ms":57515,"significance":"If substantiated, the paper's main contribution is an empirical recipe for curating multimodal training data and a modality-aware contrastive loss that generalizes to arbitrary modalities. The paper is valuable for its breadth of evaluation (MMEB, WebVid-CoVR, CaReBench, Flickr30K, etc.), transparent hyperparameters (LoRA rank, temperature, batch sizes), and the inclusion of OOD splits within MMEB, which provides some independent grounding. The data-composition analysis is a useful starting point, though the conclusions are drawn from small-scale ablations. The MAMCL idea is simple and plausible, but the measured gains are modest. Overall the work is potentially a useful engineering contribution, but the central causal claims are stronger than the current evidence.","major_comments":[{"comment":"The data-composition conclusions central to the 'modality curation' claim are drawn entirely from 600K-sample ablations on Qwen2-VL-2B. The final 6.4M-sample, 7B runs use a different composition (Figure 5: TT 21.6%, TI 39.2%, TV 36.1%) and never test whether the configuration ordering from Table 9 persists at that scale. For example, Table 9 shows TV-only outperforming TI-only on Flickr30K T→I (60.2 vs 55.4) and the full TT+TI+TV mix underperforming TV-only on CaRe V→T (61.2 vs 68.7), yet the final recipe includes all three modalities. The differences between some configurations are under one point (e.g., MMEB overall 61.1 vs 62.1; CoVR R@1 65.6 vs 64.8) with no variance or multiple seeds. Consequently, the attribution of UNITE's SOTA results to the curated composition is not directly supported; the results could be driven by scale, backbone, or other training details. This should be addressed by either validating the composition at the target scale (e.g., a 7B/6.4M ablation of at least the key configurations) or substantially softening the causal claim.","section":"§5, Appendix B.3, Tables 9/18/19, Figure 5"},{"comment":"The claim of 'state-of-the-art in 40+ different tasks' is broader than the evidence. On the MMEB meta-task level, UNITE 7B's Grounding score (84.8) is below CAFe 7B (91.2), mmE5 11B (89.7), and IDMR 26B (88.6). On individual datasets in Table 20, UNITE 7B is not the best on ImageNet-1K (80.2 vs IDMR 80.6), VOC2007 (84.9 vs IDMR 92.7), N24News (80.3 vs mmE5 81.7), or RefCOCO (89.2 vs mmE5 92.7), among others. The overall MMEB average (70.3) is within 0.5 points of mmE5 and CAFe (69.8). Thus the superiority claim should be qualified to specific benchmarks/tasks, and the 40+ count should be verified against the per-dataset tables.","section":"§4.1, Tables 4 and 20, Abstract"},{"comment":"The MAMCL ablation shows modest gains: for the 7B model, adding MAMCL increases MMEB overall by +0.3 and the MMEB/CoVR average by +0.7; for 2B, +0.5 and +1.1. No standard deviations or multiple-seed results are reported, and OOD changes are within ±0.5. The qualitative claim that MAMCL 'mitigates competitive relationships' is therefore not strongly quantified. If MAMCL is a core contribution, the authors should either provide statistical significance/variance estimates or present a more measured interpretation.","section":"§4.2, Table 7"}],"minor_comments":[{"comment":"There is a typo: 'repsentation space' should be 'representation space'.","section":"§3.1"},{"comment":"There is a typo: 'instrction-based' should be 'instruction-based'.","section":"§5"},{"comment":"The author list contains 'Victoria W.' without a full surname; this appears to be a placeholder and should be corrected.","section":"Author list"},{"comment":"The Limitations paragraph discusses audio as a future modality but does not acknowledge that the data-composition conclusions in Section 5 are based on 600K/2B experiments; given that the core claim rests on these conclusions, this limitation should be stated explicitly.","section":"Limitations"},{"comment":"Table 4 mixes zero-shot and fine-tuned results in one table; the 'Finetuning Setting' rows are not directly comparable to the 'Zero-shot Setting' rows, and the caption should clarify which baselines are fine-tuned on which data.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong empirical contribution with transparent experimental details, but the central claims need additional support: the data-composition recipe should be validated at the target scale, and the SOTA/40+ claims should be reconciled with the per-task tables. The MAMCL gains are small enough that the authors should either add significance tests or moderate the causal language. I do not see a fundamental correctness error; the issues are addressable within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a solid, practical empirical paper, not a breakthrough. The genuinely new pieces are MAMCL, a modality-masked contrastive loss, and a systematic data-composition study for retrieval. The finding that video-text pairs alone beat image-text pairs for image-text retrieval at the 2B scale is interesting and easy to test elsewhere.\n\nWhat it does well: the evaluation is broad (36 MMEB tasks plus fine-grained and coarse-grained benchmarks), the ablations are mostly clean, and the 2B/7B comparison gives a useful scaling picture. The OOD numbers on MMEB are not cherry-picked, and the paper is honest about fluctuations. The recipe (6.4M samples, two-stage training, LoRA, temperature 0.03) is specified in enough detail that replication is plausible.\n\nThe soft spots are real. The biggest one is that the curation recipe comes from 600K-sample, 2B-parameter runs, and the paper then builds the 6.4M/7B model using that composition without checking whether the ordering holds at scale. That is not a trivial extrapolation. Table 9 shows the full TT+TI+TV mix is often worse than TV-only on cross-modal retrieval (e.g., CaRe V→T 61.2 vs 68.7), yet the final recipe includes all three. The paper attributes the SOTA results largely to curation, but scale or backbone could be doing the work. At minimum the authors need a scaled-up composition check, or a caveat that the recipe is heuristic at scale.\n\nSecond, MAMCL gains are modest and mostly on in-distribution data: on MMEB the 7B gain is +0.3, and OOD sometimes goes slightly down. It is plausible as a regularizer, but the claim that it is central to the SOTA results is stronger than the evidence.\n\nThird, the 'first comprehensive analysis' claim overstates. MM1.5 and Data Metabolism already did systematic data-composition studies for LMMs, though not specifically for retrieval. Also, no code or models are released on the project page, which hurts reproducibility; the authors should ship artifacts.\n\nThe concerns are proportionate. The central methodology is coherent, the benchmarks are standard, and the empirical work is extensive. I would not desk reject this. Send it to a serious referee, but the referee should ask for scale-transfer evidence or a revised attribution claim, and for code.\n\nWho this is for: anyone building universal multimodal embedders, and people comparing training recipes for LMM-based retrieval. I'd bring it to a reading group and I'd cite the data-composition findings. My recommendation: engage with it, with the missing scale-check and artifacts.","headline":"Solid empirical recipe for universal multimodal embedders, but the load-bearing data-curation claims rest on small-scale ablations that are never validated at the final 7B/6.4M scale.","tokens_in":31918,"tokens_out":3182,"would_cite":true,"duration_ms":30463,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single 7B model can outperform larger specialized retrieval models across over 40 multimodal tasks through careful data curation and a modality-aware contrastive loss.","keywords":["multimodal retrieval","universal embeddings","data curation","modality-aware masked contrastive learning","MAMCL","vision-language models","video-text retrieval","instruction-based retrieval"],"falsifier":"Train the final 7B model on a 6.4M-sample dataset in which the text-video fraction is replaced by text-image data while keeping total size and hyperparameters fixed; if image-text retrieval does not drop relative to UNITE, the paper's claim that video-text pairs are superior training data for image-text retrieval is contradicted.","tokens_in":30865,"feed_emoji":"🔍","tokens_out":6861,"duration_ms":54287,"temperature":0.7,"pith_summary":"UNITE is a training framework for a single multimodal embedding model that handles text, image, video, and fused queries. The paper's central claim is that with carefully curated modality data and a modality-aware contrastive loss, a 7B model can outperform larger specialized retrieval models across more than 40 tasks spanning coarse-grained, fine-grained, and instruction-based retrieval. The authors report that text-video pairs are the most effective training data for general cross-modal retrieval, while text-text and text-image pairs are essential for instruction following. They also find that fine-grained video-caption pairs should be added directly to retrieval adaptation rather than used in a separate alignment stage. The proposed loss, MAMCL, restricts contrastive comparisons to candidates with the same target modality, and this mask is what lets the model balance text, image, and video in one embedding space.","feed_headline":"A 7B model beats 11B and 26B rivals on 40+ retrieval tasks","feed_subtitle":"Data curation and a modality-aware contrastive mask let one model master text, image, and video retrieval at once.","key_machinery":"The central mechanism is Modal-Aware Masked Contrastive Learning (MAMCL), a masked variant of the InfoNCE loss. For each query, the similarity matrix is multiplied by a modality mask $\\mathbf{M}$ whose entry is 1 only when the candidate's target modality combination matches that of the positive candidate; mismatched entries are set to $-\\infty$, so the query is contrasted only against candidates of the same target modality. This removes the competitive interference the paper observes when text-only, image-only, video-only, and fused embeddings are contrasted jointly. The second pillar is the data curation recipe derived from 600K-sample composition experiments: roughly 22% text-text, 39% image-text, and 36% video-text pairs in retrieval adaptation, followed by instruction tuning on MMEB plus WebVid-CoVR.","core_discovery":"The paper claims that the modality composition of training data is a first-order determinant of performance in multimodal retrieval, and that a single language-multimodal model can reach state-of-the-art results across task families if that composition is chosen correctly. The empirical discoveries are: text-video pairs alone beat text-image pairs even on image-text retrieval; text-text and text-image pairs are needed for instruction-based retrieval; and direct inclusion of fine-grained video captions in retrieval adaptation is both more effective and cheaper than a separate fine-grained alignment stage. Built on Qwen2-VL and trained on 6.4M retrieval-adaptation samples plus 1.3M instruction-tuning samples, UNITE reports leading Recall@1 numbers on CaReBench, ShareGPT4V, Urban1K, DOCCI, WebVid-CoVR, and the 36-task MMEB benchmark, surpassing models with 11B and 26B parameters.","pith_inferences":["The data-composition conclusions come from 600K-sample runs on a 2B backbone; the paper's final recipe assumes these conclusions transfer to 6.4M samples and a 7B model, a scaling step the paper does not directly validate.","The TV-over-TI result for image-text retrieval may reflect differences in caption quality or dataset difficulty between InternVid and CapsFusion rather than modality per se; an ablation that matches caption length and cleanliness would isolate the cause.","MAMCL could be extended to mask by fine-grained subtype, such as spatial versus temporal video aspects, or by task family, which the paper does not explore and which might sharpen instruction-based retrieval further.","If the recipe transfers, the field's emphasis on ever-larger retrieval models would shift toward data composition and loss design, making the paper's contribution a blueprint for adding audio or other modalities."],"forward_implications":["A single 7B embedder can replace separate text-image, text-video, and composed-retrieval models, so downstream systems can serve many retrieval tasks with one index and one model.","Multimodal data recipes should assign a substantial share to video-text pairs even when the target benchmark is image-text, because the paper finds TV-only training outperforms TI-only training on that task.","Fine-grained video caption data should be folded into retrieval adaptation rather than staged as a separate alignment pass; the paper reports this is both more accurate and about four times cheaper per 500K instances.","MAMCL's modality mask is a general plug-in: any new modality can be added by defining its mask from input prior knowledge, extending the framework beyond text, image, and video.","Model scaling helps most on temporal video retrieval, where the 7B model shows its largest relative gains over the 2B model, suggesting temporal reasoning is a scale-hungry capability."],"supporting_citations":[{"why":"Supplies the Qwen2-VL backbone that UNITE adapts for retrieval.","marker":"[82]"},{"why":"Provides the MMEB benchmark and its 20 training datasets used in instruction tuning, and a primary baseline to beat.","marker":"[32]"},{"why":"Supplies the WebVid-CoVR composed video retrieval dataset used in both instruction tuning and evaluation.","marker":"[79]"},{"why":"Provides CaReBench, the fine-grained video-text retrieval benchmark, and the fine-grained alignment idea the paper extends.","marker":"[96]"},{"why":"Supplies the Tarsier2-Recap fine-grained video-caption pairs used in retrieval adaptation and in the fine-grained alignment analysis.","marker":"[103]"},{"why":"Supplies InternVid video-text pairs used in retrieval adaptation and in the data-composition experiments.","marker":"[85]"},{"why":"Supplies CapsFusion image-text pairs used in retrieval adaptation and in composition ablations.","marker":"[102]"},{"why":"Defines the InfoNCE contrastive loss that MAMCL modifies with a modality mask.","marker":"[65]"}],"fun_headline_variants":["UNITE: Curation beats scale in multimodal retrieval","Data mix decides: 7B model tops larger rivals","Modality-aware masking lifts retrieval across tasks","How one 7B model outdoes 11B and 26B in MIR","UNITE framework: Smarter data, not bigger models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the data-composition conclusions from 600K-sample experiments on a 2B model still hold for the final 6.4M-sample, 7B training run, a transfer the paper assumes without scaling validation.","fun_headline_variants_meta":{"raw":{"variants":["UNITE: Curation beats scale in multimodal retrieval","Data mix decides: 7B model tops larger rivals","Modality-aware masking lifts retrieval across tasks","How one 7B model outdoes 11B and 26B in MIR","UNITE framework: Smarter data, not bigger models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000211,"raw_usage":{"total_tokens":1408,"prompt_tokens":932,"completion_tokens":476,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":392}},"tokens_in":548,"tokens_out":476,"duration_ms":53410,"temperature":1.0,"reasoning_tokens":392,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:08:44.961635+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the final 7B model on a 6.4M-sample dataset in which the text-video fraction is replaced by text-image data while keeping total size and hyperparameters fixed; if image-text retrieval does not drop relative to UNITE, the paper's claim that video-text pairs are superior training data for image-text retrieval is contradicted.","supporting_citations":[{"cited_title":"Covr: Learning composed video retrieval from web video captions","cited_arxiv_id":null,"evidence_quote":"Supplies the WebVid-CoVR composed video retrieval dataset used in both instruction tuning and evaluation."},{"cited_title":"Internvid: A large-scale video-text dataset for multimodal understanding and generation","cited_arxiv_id":null,"evidence_quote":"Supplies InternVid video-text pairs used in retrieval adaptation and in the data-composition experiments."},{"cited_title":"Capsfusion: Rethinking image-text data at scale","cited_arxiv_id":null,"evidence_quote":"Supplies CapsFusion image-text pairs used in retrieval adaptation and in composition ablations."}],"review_version":1}