Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Current multimodal embeddings cannot reliably follow modality instructions, according to a new 190-task benchmark and its controlled diagnostic set.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

On 190 tasks and a 12-direction cross-modal diagnostic, seven embedding models frequently fail to honor explicit target-modality instructions: retrieval is biased toward the query modality and instruction-induced shifts are misaligned.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection Useful benchmark extension with a genuinely good diagnostic idea; the headline asymmetry numbers are partly construction artifacts, but the central failure finding survives on unconfounded directions. the 4 major comments →

arxiv 2604.23321 v2 pith:L6UVHNQF submitted 2026-04-25 cs.IR

MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models

Haohang Huang , Xuan Lu , Mingyi Su , Xuan Zhang , Ziyan Jiang , Ping Nie , Kai Zou , Tomas Pfister
show 4 more authors
Wenhu Chen Wei Zhang Xiaoyu Shen Rui Meng
This is my paper
classification cs.IR
keywords multimodal embeddingsbenchmarkmodality instruction followingcross-modal retrievalembedding evaluationagent retrievalsemantic equivalence tuplesmodality bias
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MMEB-V3 is a 190-task benchmark that evaluates embedding models across text, image, video, audio, and agent-centric settings, and it carries a diagnostic component, OmniSET, that pairs semantically identical content across all four modalities. By controlling semantics while varying modality, OmniSET lets the authors ask a precise question: when an instruction names a target modality, does the retrieved instance actually come from that modality? Their answer is no. On 12 directed cross-modal retrieval tasks, three representative models score near zero in most directions, and the modality that dominates the top-10 results tracks the query's own modality far more often than the instructed target. Instruction-augmented queries do shift in embedding space—sometimes substantially—but the shifts rarely reduce distance to the target and often increase it. The paper concludes that today's omni-embedding models treat modality as a byproduct of semantics rather than as an enforceable instruction constraint, which matters because agents and retrieval-augmented systems increasingly depend on retrieving the right sensory format, not just the right content.

Core claim

On OmniSET, ~100 queries with hard negatives exist in all four modalities, giving 12 directed retrieval tasks against a shared mixed-modality pool that keeps the source as a distractor. Across three models, Hit@1 is 0.0 in most directions (text→image, text→audio, video→text); only generation-linked directions (image→video, audio→text) succeed. Top-10 dominant modality follows the query, not the target: Nemotron retrieves 82.7% text for text queries even when images are requested; WAVE retrieves video 99.9% of the time. Instruction augmentation shifts queries up to 0.4 cosine distance, but distance to target improves in only a few directions (all <0.09). The claim: current omni-modality embed

What carries the argument

OmniSET (Omni-modality Semantic Equivalence Tuples) — aligned quadruples {text, image, video, audio} with the same semantics, each query expanded into 12 directed cross-modal retrieval tasks sharing one mixed-modality candidate pool; source instance kept as a distractor, same-modality pairs excluded, 15–20 human-verified hard negatives per query. It isolates the modality dimension by holding semantics fixed, and the 12-direction design exposes directionality and query-modality bias. Its acknowledged limitation is that video and audio are synthesized from images and captions, which may tie certain pairs together and partly drive the I→V and A→T near-perfect scores.

Load-bearing premise

The findings rest on the assumption that the pipeline that generates videos from images and speech from captions yields faithful semantic equivalents, so that cross-modal scores measure instruction-following rather than the fidelity of the generated candidates—an assumption the paper itself flags as a possible source of modality bias.

What would settle it

Take a set of human-verified quadruples where the video is a real recording of the scene in the image and the audio is a genuine spoken version of the caption, so no generation artifacts tie any modality pair. If, on these, all 12 directed retrieval directions show Hit@1 well above chance—or if the query-modality bias disappears—then the paper's central claim of systematic instruction-failure is falsified; if the same near-zero and asymmetric pattern persists, the claim is confirmed.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Retrieval pipelines relying on omni-modality embeddings for modality-specified queries will silently return wrong-format results: agents asking for an audio clip or a video will often get text or the query's own modality instead.
  • Because the failure is consistent across very different models, it points to a shared training-objective gap—contrastive alignment over semantic similarity does not teach modality-conditioning—rather than to a defect of one architecture.
  • The instruction-shift analysis implies that naive prompt augmentation (appending 'retrieve a video' to a query) is not sufficient: embedding shifts must be oriented toward the target modality, not merely increased in magnitude.
  • OmniSET provides a reusable diagnostic: future embedding models can be tested on the same 12 directed tasks with a shared candidate pool, making modality-instruction following directly measurable.
  • Benchmark scores on standard cross-modal retrieval may overstate capability, because retrieval of the correct semantic content is not the same as retrieval of content in the instructed modality—a distinction MMEB-V3 makes visible.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because OmniSET's video and audio are generated (Veo-3.1 from images, Gemini-2.5-Flash-TTS from captions), the paper's headline asymmetry may partly reflect generation fidelity: I→V and A→T look easy precisely because the generated targets are near-copies of their sources. A follow-up with naturally occurring quadruples—real video of the same event as the image, real speech of the caption—would se
  • The dominant-modality statistic (e.g., 99.9% video for WAVE) suggests query-modality bias is not soft preference but a broken retrieval policy: for a text→video query, a model that retrieves only video would have Hit@1 = 0; the embedding geometry appears clustered by modality rather than aligned by content, which would explain why semantic equivalence across modalities is invisible to these models
  • A testable extension: train or fine-tune an omni-embedding model with an explicit loss that penalizes retrieving an instance whose modality differs from the instruction, then re-run the OmniSET directions. If modality-instruction following improves sharply, the gap is a training objective problem; if it does not, it is an architectural or capacity constraint.
  • For agent applications, the practical implication the paper leaves implicit is that modality must be enforced downstream—e.g., by reranking candidates with a modality classifier or by constraining the candidate set—since the embedding alone cannot be trusted to honor the instruction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces MMEB-V3, an extension of the MMEB-V2 benchmark to 190 tasks spanning text, image, video, audio, visual documents, and agent-centric retrieval. It also constructs OmniSET, a diagnostic dataset of approximately 100 'semantic equivalence tuples' in which the same content is rendered as text, image, video, and audio; videos are synthesized from images with Veo-3.1 and audio from captions with Gemini TTS. Using OmniSET, the paper reports three headline findings: (1) explicit modality instructions often fail (Hit@1 near zero in most cross-modal directions), (2) cross-modal retrieval is asymmetric and dominated by query-modality bias, and (3) instruction-induced embedding shifts do not consistently move queries toward the target modality. The conclusion is that current omni-modality embeddings cannot reliably enforce modality constraints. The main benchmark evaluation uses standard public datasets and task-appropriate metrics; the diagnostic analysis uses the newly constructed OmniSET.

Significance. The benchmark resource itself is a genuine contribution: it assembles a large, heterogeneous collection of tasks with clear protocols, reports detailed per-task results for seven models, and extends coverage to audio and agent scenarios that previous MMEB versions lack. The OmniSET design is also valuable in principle: controlled cross-modal tuples can expose modality bias in a way that standard retrieval benchmarks cannot. If the three findings survive the confounding concerns below, they would be an important signal for the multimodal embedding community. However, the abstract and conclusion generalize from a diagnostic set whose construction is confounded with the very modality asymmetries the paper reports; until that is addressed, the headline claims are not yet established. I also credit the authors for explicitly discussing the synthetic-data limitation in §A.2.2; the problem is that the main text does not carry that caveat into the abstract and conclusions.

major comments (4)
  1. [§A.2.1, §A.2.2, Table 4] The near-perfect directions in Table 4 are exactly the generation edges: I→V (video is Veo-generated from the query image) and A→T (audio is TTS-generated from the caption). Conversely, the headline failures (T→A, V→T) are directions where the target is a generated artifact whose semantic fidelity to the query is unverified. The paper itself concedes in §A.2.2 that the construction 'may introduce a form of modality preference' and that I→V/A→T scores 'may partially reflect dataset construction effects.' Without human or automatic verification that the generated video/audio are semantically equivalent to their source and to the other tuple members, Hit@1 in these directions measures generation fidelity plus model behavior, not modality-instruction following. The abstract and §5.1 nevertheless present asymmetry and instruction failure as general model limitations. Please provide equivalenc
  2. [§A.2, Table 4] OmniSET is built from roughly 100 hand-curated queries (Table 1 reports 1.2K directed query instances, §A.2 says 100 base queries). Table 4 reports Hit@1/MRR with no confidence intervals, significance tests, or per-query variability. With 100 base queries, the difference between Hit@1=0.0 and Hit@1=3.0 is three successes; a single query changes scores by 1 percentage point. This is particularly problematic for the small differences in §5.3 ('all below 0.09') and for the WAVE T2V/A2V values. Please report bootstrap confidence intervals or per-query breakdowns and state the effective sample size for each directional score. The current presentation makes it impossible to know which asymmetries are robust.
  3. [§5.3 vs. Figure 10] §5.3 describes T→V as 'a small improvement (+0.041)' and V→T as 'degradation (−0.158)', implying positive = better. The caption of Figure 10 states 'Negative values indicate that the instruction-augmented query moves closer to the target modality, while positive values indicate increased distance', i.e., negative = better. If Figure 10's convention is correct, the two examples in §5.3 are reversed and the qualitative claim about which directions improve is inverted. If §5.3's convention is correct, Figure 4a and Figure 10 captions are wrong. Please make the sign convention consistent across the text and figures and recompute the affected statements.
  4. [§A.2.2] The mitigation says the key phenomena are 'consistent across multiple models and modality directions, including those not directly affected by synthetic generation.' This is not supported by the data. Every direction involving V or A is affected because both V and A are synthetic; only T→I and I→T use the original MSCOCO image/text pair exclusively. Table 4 shows T→I Hit@1=0.0 and I→T=0.0 for all models, but those are the only unaffected directions, and they too come from the same 100-query instrument. The claimed consistency across 'unaffected' directions therefore cannot be checked from the reported results. Either list which directions are considered unaffected and report them separately, or drop this mitigation.
minor comments (5)
  1. [§3.1, Table 3] The 'All' column treats missing audio as 0 for Qwen/VLM2Vec/GME, which conflates lack of modality support with poor performance. All* (average over available tasks) is more interpretable; recommend reporting it as primary or adding a footnote with per-modality coverage.
  2. [§A.2, Table 1] The number of OmniSET queries is not consistent: 'approximately 100' high-quality samples vs. 1.2K query count in Table 1. Clarify that 1.2K = 100 base tuples × 12 directed tasks (or state the actual base count).
  3. [Table 4] Hit@1 values with two decimals (e.g., 68.32) are confusing for count-based metrics; report as fractions (e.g., 820/1200) or state the sample size so readers can interpret.
  4. [§5.2, Figure 3] 'Sensitivity' measured as cosine distance is a magnitude, not necessarily 'responsiveness' in terms of instruction following; consider renaming or clarifying to avoid implying effectiveness.
  5. [Throughout] Typos and formatting: 'T uples' should be 'Tuples' in the abstract and §3.1; Figure 4a's heatmap has no colorbar or units. A final proofread would help.

Circularity Check

1 steps flagged

OmniSET's equivalence labels are generated from the source modality (V from I, A from T), so the headline I→V and A→T successes and the asymmetry findings are partially baked into the construction, not independently measured.

specific steps
  1. self definitional [§A.2.1 (construction of OmniSET), §A.2.2 (impact of synthetic data), §5.1 (interpretation of Table 4), Abstract]
    "motion videos are generated from images using Google Veo-3.1, and speech audio is generated from captions using Gemini-2.5-Flash-TTS. This results in semantically aligned multi-modal tuples with consistent content across modalities. (§A.2.1) ... Video samples are directly derived from images, and audio samples are derived from text. As a result, modality pairs such as image–video and text–audio may exhibit higher intrinsic similarity than other cross-modal pairs. (§A.2.2)"

    OmniSET defines 'semantically equivalent instances' by generating the target modality from the source modality: x_V is generated from x_I and x_A is generated from x_T. The benchmark then reports as a finding that I→V and A→T are near-perfect while reverse directions fail (Table 4). By construction, in exactly those two directions the target is a deterministic function of the query item, so high scores can be achieved by low-level artifact matching (visual frame similarity, speech-to-text overlap) without any instruction-controlled modality selection. The observed asymmetry is therefore co-produced by the generation graph rather than independently measuring modality-instruction following. The paper's own §A.2.2 concession—that these scores 'may partially reflect dataset construction effect

full rationale

MMEB-V3 is a benchmark paper, not a derivation: there are no fitted parameters, no closed-form predictions, and no load-bearing self-citation chains. The non-OmniSET portions (MSCOCO, MSR-VTT, BRIGHT, Tool-REX, etc.) are externally grounded tasks and do not raise circularity concerns. The circularity issue is confined to the diagnostic OmniSET component, which carries the paper's three headline findings. OmniSET's ground-truth equivalence relation is defined by a synthetic generation pipeline—video from image, audio from caption—so the 'semantically equivalent' tuples are constructed along exactly the directions that later show near-perfect retrieval (I→V, A→T). This makes those scores and the accompanying asymmetry partially self-referential: the input and the 'predicted' target are linked by construction. The authors explicitly concede this in §A.2.2 and in the Table 4 discussion, which is an honest limitation statement, but the concession does not remove the construct-validity problem; it documents it. The claim that models 'fail to reliably enforce modality constraints' therefore rests, in its strongest form, on a diagnostic whose ground truth is partly generated from the query. Because the broader MMEB-V3 leaderboard results are independent and the OmniSET analysis does contain model-behavior information beyond the generation graph, the circularity is partial rather than total, warranting a 6 rather than a higher score. No separate self-citation circularity is present: self-citations to VLM2Vec, VLM2Vec-V2, and MMEB are background references, not load-bearing evidence for the central findings.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The central diagnostic claims rest on: (a) the fidelity of the synthetic-modality equivalence assumption (video/audio generated from image/caption are semantically equivalent to the source — the paper's own caveat in §A.2.2); (b) the standard retrieval-stack assumptions (cosine similarity in a shared space ranks by relevance; MSCOCO caption–image pairs are ground-truth equivalents); (c) reporting conventions (Hit@1 vs NDCG@5, missing-as-0 averaging) that are design choices rather than fitted numbers. There are no fitted parameters in the mathematical sense; the closest analogues are hand-chosen design quantities: ~100 OmniSET queries, 15–20 hard negatives per query, ≤1K-query/≤10K-candidate subsampling caps, and the (unspecified) instruction-augmentation templates. The one invented artifact is OmniSET itself, whose independent evidentiary value is limited until the instances are released.

free parameters (4)
  • OmniSET query set (≈100 hand-curated queries) = 100 queries (≈1.2K directional tasks)
    Chosen by hand through license filtering, object-category ranking, and manual inspection (§A.2.1); no power analysis; all diagnostic claims in §5 rest on this sample.
  • Cross-modal instruction templates = not specified in text
    The magnitude of "instruction-induced shifts" (§5.2–5.3) depends entirely on the wording of the modality instructions; templates are not given in the main text or appendix, so the measured shifts are conditional on an unstated design choice.
  • Audio subsampling caps = ≤1,000 queries; ≤10,000 candidates
    NSynth, SpeechCommands, SoundDescs, and SpeechCOCO subsampled "to ensure computational tractability" (§A.2); the caps change task difficulty relative to full-set evaluations and affect the reported audio scores.
  • Aggregation convention: All column treats missing modalities as 0 = 0 for missing modalities
    Table 3 'All' averages over 190 tasks with missing audio scores as 0, while 'All*' averages over available tasks; this hand-chosen reporting rule changes relative model rankings (e.g., WAVE All=26.3).
axioms (4)
  • ad hoc to paper Generated video/audio instances are semantically equivalent to their source image/caption and to each other across the four modalities
    Load-bearing premise of OmniSET (§A.2.1); the paper's own §A.2.2 flags that it introduces generation dependencies (I→V, A→T artificially easy). If false, findings 1 and 2 are confounded.
  • domain assumption Cosine similarity between embeddings is a valid retrieval score in a shared space
    Standard for embedding benchmarks; used throughout MMEB-V3 evaluation and the §5 shift analyses without questioning the shared-space premise.
  • domain assumption MSCOCO caption–image pairs are semantically equivalent ground truth
    Inherited from prior dataset work; OmniSET is "built upon the widely used MSCOCO dataset" (§A.2.1).
  • domain assumption Averaging Hit@1 and NDCG@5 across heterogeneous tasks yields a comparable 'All' score
    Tables 2–3 mix metrics per task type; cross-task comparability is assumed without normalization analysis.
invented entities (1)
  • OmniSET (Omni-modality Semantic Equivalence Tuples) independent evidence
    purpose: Diagnostic construct of semantically equivalent {xT, xI, xV, xA} tuples used to disentangle semantic content from modality effects in instruction-conditioned retrieval
    It is a measurement instrument, not an explanatory entity; its instances are falsifiable once released, but the text provides no hosting link — the only URL points to the VLM2Vec repo. Independence is limited because the tuples are synthesized by proprietary generators (Veo-3.1, Gemini-2.5-Flash-TTS) controlled by the authors.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models." pith.science (2026). https://pith.science/paper/L6UVHNQF

@misc{pith2026260423321,
  author       = {Pith},
  title        = {Pith review of: MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L6UVHNQF}},
  note         = {Machine review of arXiv:2604.23321}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Multimodal embedding models aim to map heterogeneous inputs, such as text, images, videos, and audio, into a shared semantic space. However, existing methods and benchmarks remain largely limited to partial modality coverage, making it difficult to systematically evaluate full-modality representation learning. In this work, we take a step toward the full-modality setting. We introduce MMEB-V3, a comprehensive benchmark that evaluates embeddings across text, image, video, audio, as well as agent-centric scenarios. To enable more fine-grained diagnosis, we further construct OmniSET (Omni-modality Semantic Equivalence Tuples), where semantically equivalent instances are represented across modalities, allowing us to disentangle semantic similarity from modality effects. Through experiments on MMEB-V3, we conduct a systematic analysis of full-modality embeddings and identify three key findings: (1) models often fail to retrieve the intended target modality; (2) cross-modal retrieval is highly asymmetric and dominated by query-modality bias; and (3) instruction-induced shifts are either insufficient or misaligned with the target modality, and therefore do not reliably improve retrieval. These results indicate that current multimodal embeddings are not yet capable of reliably enforcing modality constraints specified by instructions, and consequently fail to exhibit consistent modality-aware retrieval behavior. We hope MMEB-V3 provides a useful benchmark for understanding and diagnosing these limitations, and for guiding future research on full-modality embeddings.

Figures

Figures reproduced from arXiv: 2604.23321 by Haohang Huang, Kai Zou, Mingyi Su, Ping Nie, Rui Meng, Tomas Pfister, Wei Zhang, Wenhu Chen, Xiaoyu Shen, Xuan Lu, Xuan Zhang, Ziyan Jiang.

Figure 1
Figure 1. Figure 1: MMEB-V3 Overview: New Additions—Agent Tasks, Complex Text Retrieval, view at source ↗
Figure 2
Figure 2. Figure 2: Diversity of modalities, tasks, and instruction patterns in MMEB-V3 view at source ↗
Figure 3
Figure 3. Figure 3: Model sensitivity to modality-constrained instructions, measured by the mean cosine distance between raw and instruction-augmented queries. Larger values indicate greater embedding shifts after instruction augmentation. the target modality (99.9% across all directions). Qwen3-VL shows a similar pattern, where retrieved modalities align more with the query modality than with the instructed target. 5.2 Model… view at source ↗
Figure 4
Figure 4. Figure 4: Instruction-induced changes in query–target distance and embedding shifts. (a) Change in cosine distance to the target modality after instruction augmentation, measured relative to the raw query. (b,c) t-SNE visualizations showing how instruction-augmented queries move in the embedding space for text and image queries. In (b,c), raw queries are shown as circles (•), instruction-augmented queries as downwar… view at source ↗
Figure 5
Figure 5. Figure 5: Construction pipeline of OmniSET. The hard negative construction process is performed as follows: 15 view at source ↗
Figure 6
Figure 6. Figure 6: Prompt templates for video and audio generation. view at source ↗
Figure 7
Figure 7. Figure 7: Example Hard Negative Set 18046 from OmniSET. view at source ↗
Figure 8
Figure 8. Figure 8: Example Hard Negative Set 80130 from OmniSET. view at source ↗
Figure 9
Figure 9. Figure 9: Example Hard Negative Set 3067 from OmniSET. view at source ↗
Figure 10
Figure 10. Figure 10: Instruction-induced changes in query–target distance across models. Each cell shows the change in cosine distance to the target modality after instruction augmentation, computed relative to the raw query. Negative values indicate that the instruction-augmented query moves closer to the target modality, while positive values indicate increased distance. 20 view at source ↗
Figure 11
Figure 11. Figure 11: Embedding space geometry across models. t-SNE visualizations of embeddings from different modalities for three representative models. 23 view at source ↗
Figure 12
Figure 12. Figure 12: Instruction-induced shifts in the embedding space for different source query view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. CompRank: Efficient LLM Reranking via Token-Level Compression and Decoding-Free Scoring

    cs.IR 2026-06 unverdicted novelty 7.0

    CompRank maintains near-identical NDCG@10 (39.2 vs 39.7) using 10.2% of document tokens and delivers 4.9x-9.5x speedup over generation-based rerankers on seven BEIR datasets.

  2. Geo-Embed: Towards Unified Multimodal Embeddings for Urban Understanding

    cs.CV 2026-08 conditional novelty 5.0

    GeoMEB unifies 45 urban embedding tasks into a ranking protocol, and Geo-Embed, an instruction-conditioned vision-language embedder fine-tuned on it, tops the leaderboard.

Reference graph

Works this paper leans on

47 extracted references · 19 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Peerqa: A scientific question answering dataset from peer reviews, 2025

    Tim Baumgärtner, Ted Briscoe, and Iryna Gurevych. Peerqa: A scientific question answering dataset from peer reviews, 2025. URL https://arxiv.org/abs/2502.13668

    Pith/arXiv arXiv 2025

  2. [2]

    Crema-d: Crowd-sourced emotional multimodal actors dataset

    Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5 0 (4): 0 377--390, 2014

    2014

  3. [3]

    Clotho: An audio captioning dataset

    Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 736--740. IEEE, 2020

    2020

  4. [4]

    Neural audio synthesis of musical notes with wavenet autoencoders, 2017

    Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Douglas Eck, Karen Simonyan, and Mohammad Norouzi. Neural audio synthesis of musical notes with wavenet autoencoders, 2017

    2017

  5. [5]

    Finevideo

    Miquel Farré, Andi Marafioti, Lewis Tunstall, Leandro Von Werra, and Thomas Wolf. Finevideo. https://huggingface.co/datasets/HuggingFaceFV/finevideo, 2024

    2024

  6. [6]

    Speech-coco: 600k visually grounded spoken captions aligned to mscoco data set

    William Havard, Laurent Besacier, and Olivier Rosec. Speech-coco: 600k visually grounded spoken captions aligned to mscoco data set. arXiv preprint arXiv:1707.08435, 2017

    Pith/arXiv arXiv 2017

  7. [7]

    Omniret: Efficient and high-fidelity omni modality retrieval, 2026

    Chuong Huynh, Manh Luong, and Abhinav Shrivastava. Omniret: Efficient and high-fidelity omni modality retrieval, 2026. URL https://arxiv.org/abs/2603.02098

    arXiv 2026

  8. [8]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp.\ 4904--4916. PMLR, 2021

    2021

  9. [10]

    Vlm2vec: Training vision-language models for massive multimodal embedding tasks, 2025

    Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. Vlm2vec: Training vision-language models for massive multimodal embedding tasks, 2025. URL https://arxiv.org/abs/2410.05160

    Pith/arXiv arXiv 2025

  10. [11]

    Sophia Koepke, Andreea-Maria Oncescu, João F

    A. Sophia Koepke, Andreea-Maria Oncescu, João F. Henriques, Zeynep Akata, and Samuel Albanie. Audio retrieval with natural language queries: A benchmark study. IEEE Transactions on Multimedia, 25: 0 2675–2685, 2023. ISSN 1941-0077. doi:10.1109/tmm.2022.3149712. URL http://dx.doi.org/10.1109/TMM.2022.3149712

    arXiv 2023

  11. [12]

    Realtalk: A 21-day real-world dataset for long-term conversation, 2025

    Dong-Ho Lee, Adyasha Maharana, Jay Pujara, Xiang Ren, and Francesco Barbieri. Realtalk: A 21-day real-world dataset for long-term conversation, 2025. URL https://arxiv.org/abs/2502.13270

    Pith/arXiv arXiv 2025

  12. [13]

    Berg, and Mohit Bansal

    Jie Lei, Tamara L. Berg, and Mohit Bansal. Qvhighlights: Detecting moments and highlights in videos via natural language queries, 2021. URL https://arxiv.org/abs/2107.09609

    Pith/arXiv arXiv 2021

  13. [14]

    R2med: A benchmark for reasoning-driven medical retrieval, 2025

    Lei Li, Xiao Zhou, and Zheng Liu. R2med: A benchmark for reasoning-driven medical retrieval, 2025. URL https://arxiv.org/abs/2505.14558

    Pith/arXiv arXiv 2025

  14. [15]

    Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of-the-art multimodal retrieval and ranking

    Mingxin Li, Yanzhao Zhang, Dingkun Long, Chen Keqin, Sibo Song, Shuai Bai, Zhibo Yang, Pengjun Xie, An Yang, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of-the-art multimodal retrieval and ranking. arXiv preprint arXiv:2601.04720, 2026

    Pith/arXiv arXiv 2026

  15. [16]

    Mm-embed: Universal multimodal retrieval with multimodal llms, 2025

    Sheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. Mm-embed: Universal multimodal retrieval with multimodal llms, 2025. URL https://arxiv.org/abs/2411.02571

    Pith/arXiv arXiv 2025

  16. [17]

    Lawrence Zitnick, and Piotr Dollár

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015. URL https://arxiv.org/abs/1405.0312

    Pith/arXiv arXiv 2015

  17. [18]

    M ulti C on IR : Towards multi-condition information retrieval

    Xuan Lu, Sifan Liu, Bochao Yin, Yongqi Li, Xinghao Chen, Hui Su, Yaohui Jin, Wenjun Zeng, and Xiaoyu Shen. M ulti C on IR : Towards multi-condition information retrieval. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Findings of the Association for Computational Linguistics: EMNLP 2025, pp.\ 13471--13494, Suzhou...

    doi:10.18653/v1/2025.findings-emnlp.726 2025

  18. [19]

    Tools are under-documented: Simple document expansion boosts tool retrieval

    Xuan Lu, Haohang Huang, Rui Meng, Yaohui Jin, Wenjun Zeng, and Xiaoyu Shen. Tools are under-documented: Simple document expansion boosts tool retrieval. In The Fourteenth International Conference on Learning Representations, 2026 a . URL https://openreview.net/forum?id=g9D9MgG7iW

    2026

  19. [20]

    Rethinking reasoning in document ranking: Why chain-of-thought falls short

    Xuan Lu, Haohang Huang, Rui Meng, Yaohui Jin, Wenjun Zeng, and Xiaoyu Shen. Rethinking reasoning in document ranking: Why chain-of-thought falls short. In The Fourteenth International Conference on Learning Representations, 2026 b . URL https://openreview.net/forum?id=txmqENuRcc

    2026

  20. [21]

    Beyond global similarity: Towards fine-grained, multi-condition multimodal retrieval, 2026 c

    Xuan Lu, Kangle Li, Haohang Huang, Rui Meng, Wenjun Zeng, and Xiaoyu Shen. Beyond global similarity: Towards fine-grained, multi-condition multimodal retrieval, 2026 c . URL https://arxiv.org/abs/2603.01082

    arXiv 2026

  21. [22]

    Vidore benchmark v2: Raising the bar for visual retrieval, 2025

    Quentin Macé, António Loison, and Manuel Faysse. Vidore benchmark v2: Raising the bar for visual retrieval, 2025. URL https://arxiv.org/abs/2505.17166

    arXiv 2025

  22. [23]

    Vlm2vec-v2: Advancing multimodal embedding for videos, images, and visual documents, 2025

    Rui Meng, Ziyan Jiang, Ye Liu, Mingyi Su, Xinyi Yang, Yuepeng Fu, Can Qin, Zeyuan Chen, Ran Xu, Caiming Xiong, Yingbo Zhou, Wenhu Chen, and Semih Yavuz. Vlm2vec-v2: Advancing multimodal embedding for videos, images, and visual documents, 2025. URL https://arxiv.org/abs/2507.04590

    Pith/arXiv arXiv 2025

  23. [24]

    Tut database for acoustic scene classification and sound event detection

    Annamaria Mesaros, Toni Heittola, and Tuomas Virtanen. Tut database for acoustic scene classification and sound event detection. In 2016 24th European signal processing conference (EUSIPCO), pp.\ 1128--1132. IEEE, 2016

    2016

  24. [25]

    Mteb: Massive text embedding benchmark, 2023

    Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. Mteb: Massive text embedding benchmark, 2023. URL https://arxiv.org/abs/2210.07316

    Pith/arXiv arXiv 2023

  25. [26]

    Esc: Dataset for environmental sound classification

    Karol J Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pp.\ 1015--1018, 2015

    2015

  26. [27]

    Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp.\ 2641--2649, 2015

    2015

  27. [28]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL https://arxiv.org/abs/2103.00020

    Pith/arXiv arXiv 2021

  28. [29]

    A dataset and taxonomy for urban sound research

    Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. A dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM international conference on Multimedia, pp.\ 1041--1044, 2014

    2014

  29. [30]

    BRIGHT : A realistic and challenging benchmark for reasoning-intensive retrieval

    Hongjin SU, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han yu Wang, Liu Haisu, Quan Shi, Zachary S Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan O Arik, Danqi Chen, and Tao Yu. BRIGHT : A realistic and challenging benchmark for reasoning-intensive retrieval. In The Thirteenth International Conference on Learning Representations, 202...

    2025

  30. [31]

    Wave: Learning unified & versatile audio-visual embeddings with multimodal llm, 2025

    Changli Tang, Qinfan Xiao, Ke Mei, Tianyi Wang, Fengyun Rao, and Chao Zhang. Wave: Learning unified & versatile audio-visual embeddings with multimodal llm, 2025. URL https://arxiv.org/abs/2509.21990

    arXiv 2025

  31. [32]

    Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models, 2021

    Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models, 2021. URL https://arxiv.org/abs/2104.08663

    Pith/arXiv arXiv 2021

  32. [33]

    Audio-visual event localization in unconstrained videos, 2018

    Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos, 2018. URL https://arxiv.org/abs/1803.08842

    Pith/arXiv arXiv 2018

  33. [34]

    Speech commands: A dataset for limited-vocabulary speech recognition, 2018

    Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition, 2018. URL https://arxiv.org/abs/1804.03209

    Pith/arXiv arXiv 2018

  34. [35]

    Uniir: Training and benchmarking universal multimodal information retrievers

    Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers. arXiv preprint arXiv:2311.17136, 2023

    Pith/arXiv arXiv 2023

  35. [36]

    Followir: Evaluating and teaching information retrieval models to follow instructions

    Orion Weller, Benjamin Chang, Sean MacAvaney, Kyle Lo, Arman Cohan, Benjamin Van Durme, Dawn Lawrie, and Luca Soldaini. Followir: Evaluating and teaching information retrieval models to follow instructions. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technolo...

    2025

  36. [37]

    Knowme-bench: Benchmarking person understanding for lifelong digital companions, 2026

    Tingyu Wu, Zhisheng Chen, Ziyan Weng, Shuhe Wang, Chenglong Li, Shuo Zhang, Sen Hu, Silin Wu, Qizhen Lan, Huacan Wang, and Ronghao Chen. Knowme-bench: Benchmarking person understanding for lifelong digital companions, 2026. URL https://arxiv.org/abs/2601.04745

    Pith/arXiv arXiv 2026

  37. [38]

    Omni-embed-nemotron: A unified multimodal retrieval model for text, image, audio, and video, 2025

    Mengyao Xu, Wenfei Zhou, Yauhen Babakhin, Gabriel Moreira, Ronay Ak, Radek Osmulski, Bo Liu, Even Oldridge, and Benedikt Schifferer. Omni-embed-nemotron: A unified multimodal retrieval model for text, image, audio, and video, 2025. URL https://arxiv.org/abs/2510.03458

    arXiv 2025

  38. [39]

    Gme: Improving universal multimodal retrieval by multimodal llms, 2025 a

    Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. Gme: Improving universal multimodal retrieval by multimodal llms, 2025 a . URL https://arxiv.org/abs/2412.16855

    Pith/arXiv arXiv 2025

  39. [40]

    Universal retrieval for multimodal trajectory modeling

    Xuan Zhang, Ziyan Jiang, Rui Meng, Yifei Leng, Zhenbang Xiao, Zora Zhiruo Wang, Yanyi Shang, and Dehan Kong. Universal retrieval for multimodal trajectory modeling. In ICML 2025 Workshop on Computer Use Agents, 2025 b

    2025

  40. [41]

    Deepplanning: Benchmarking long-horizon agentic planning with verifiable constraints, 2026

    Yinger Zhang, Shutong Jiang, Renhao Li, Jianhong Tu, Yang Su, Lianghao Deng, Xudong Guo, Chenxu Lv, and Junyang Lin. Deepplanning: Benchmarking long-horizon agentic planning with verifiable constraints, 2026. URL https://arxiv.org/abs/2601.18137

    arXiv 2026

  41. [42]

    Lmeb: Long-horizon memory embedding benchmark, 2026

    Xinping Zhao, Xinshuo Hu, Jiaxin Xu, Danyu Tang, Xin Zhang, Mengjia Zhou, Yan Zhong, Yao Zhou, Zifei Shan, Meishan Zhang, Baotian Hu, and Min Zhang. Lmeb: Long-horizon memory embedding benchmark, 2026. URL https://arxiv.org/abs/2603.12572

    Pith/arXiv arXiv 2026

  42. [43]

    Beyond content relevance: Evaluating instruction following in retrieval models

    Jianqun Zhou, Yuanlei Zheng, Wei Chen, Qianqian Zheng, Shang Zeyuan, Wei Zhang, Rui Meng, and Xiaoyu Shen. Beyond content relevance: Evaluating instruction following in retrieval models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=OlRjxSuSwl

    2025

  43. [44]

    Longembed: Extending embedding models for long context retrieval

    Dawei Zhu, Liang Wang, Nan Yang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. Longembed: Extending embedding models for long context retrieval. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.\ 802--816, 2024

    2024

  44. [45]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  45. [46]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  46. [47]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  47. [48]

    J]m | 3M6SL1 `֭N; ޤ Q4 t<묳q >d DF6I>T[ f v-֫ ?xj fڵ] bպFފ 8 Z

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.