Pith. sign in

REVIEW 2 major objections 4 minor 47 references

Forcing a single end-of-sequence token to rebuild masked text compresses the whole input into one embedding, letting small models rival far larger contrastive pre-training runs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 19:36 UTC pith:ESZE7VKG

load-bearing objection CoCoA is a plausible and honestly-ablated three-stage recipe for MLLM-based multimodal embeddings, with a genuinely new EOS-bridged attention-truncation mechanism, but the paper never isolates that mechanism from the extra reconstruction data, so its central causal claim is not yet proven. the 2 major comments →

arxiv 2603.01471 v3 pith:ESZE7VKG submitted 2026-03-02 cs.IR cs.LG

Reconstructing Content with Collaborative Attention for Universal Multimodal Representation Learning

Jiahan Chen , Da Li , Hengran Zhang , Yinqiong Cai , Lixin Su , Jiafeng Guo , Daiting Shi , Dawei Yin
show 1 more author
Keping Bi
This is my paper
classification cs.IR cs.LG
keywords multimodal embeddingcontent reconstructionattention truncationEOS token compressioncontrastive learningbidirectional attentionmasked next token predictionpre-training data efficiency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

CoCoA is a pre-training recipe that turns an MLLM into a multimodal embedding model by teaching it to reconstruct its own input from a single token. The pipeline first switches the model from causal to bidirectional attention and warms that up with masked image and text reconstruction. It then splits every input into a compression block and a reconstruction block separated by an end-of-sequence (EOS) token, forbids direct attention between the two blocks, masks most of the reconstruction text, and trains the model to generate the masked words from the EOS embedding alone. Because the EOS is the only information bridge, it is forced to summarize the whole input; that token then becomes the embedding used for contrastive learning. The paper reports that this staged recipe scores best among small models in its comparison and matches larger models that were pre-trained on tens of billions of tokens, using only a few hundred thousand pre-training pairs.

Core claim

The paper's central claim is that an engineered information bottleneck, not data volume, is what makes a multimodal embedding compact. By formatting training instances as Block A (image, or image plus question) followed by an EOS token and Block B (caption or answer), and truncating attention so that Block A and Block B can communicate only through the EOS token, the model must pack Block A's semantics into the EOS hidden state to succeed at masked next-token prediction on Block B. With 70% of Block B masked (all of it when short), correct reconstruction is possible only if the EOS carries the essential visual and textual content. After this reconstruction stage, the EOS embedding is aligned

What carries the argument

The carrying mechanism is 'EOS-Bridged reconstruction via attention truncation.' Inputs are arranged as Block A, EOS token, Block B. Bidirectional attention operates inside each block and between each block and the EOS, but an attention mask prohibits any direct attention between Block A and Block B. With most of Block B masked, the only way to predict the missing tokens is to read them out of the EOS embedding, which in turn can only hold information that Block A's tokens have written into it. The masked-token reconstruction loss therefore exerts pressure on the EOS to become a dense, information-rich summary of the multimodal input. This same EOS embedding is later extracted for contrastiv

Load-bearing premise

The load-bearing assumption is that the EOS token is truly the only path for information to flow from Block A to Block B; if residual connections, positional embeddings, or the unmasked portion of Block B leak content around the attention mask, the model may reconstruct Block B without ever compressing anything into the EOS.

What would settle it

Replace the EOS hidden state with a fixed random vector at the reconstruction head during inference (or detach its gradient during training) while keeping the same truncated attention and masking; if masked-token accuracy stays high, information is reaching Block B by some route other than the EOS, and the claimed compression mechanism is not what drives the gains. A cleaner variant: train a linear probe to predict Block B words from intermediate hidden states of Block A; strong leakage would contradict the 'EOS as sole bridge' claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Multimodal embedding models can be trained to a competitive level with roughly 300K original plus 200K synthetic pre-training pairs, far less than the multi-billion-token corpora used by some baselines.
  • Adding the reconstruction pre-training stage before contrastive learning improves downstream retrieval and classification scores over contrastive-only training with the same backbone.
  • Raising the masking ratio on the reconstruction block from 20% to 70% yields steady gains, indicating that the difficulty of the reconstruction task controls how much compression the EOS token must perform.
  • Adding more pre-training data from the same task distribution eventually saturates and can even hurt out-of-domain scores, while adding semantically diverse synthetic data keeps improving both in-domain and out-of-domain performance.
  • The EOS token trained as the compression bridge serves directly as the embedding for contrastive learning, so the compression behavior carries through to the final representation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the EOS bottleneck is real, the same attention-truncation trick could be applied to long-document retrieval: forcing a document to be reconstructed from a single token may produce denser query-document embeddings than mean pooling, which tends to dilute rare but important information.
  • The paper's own case study shows that one token captures only one aspect of multi-object images; a multi-embedding variant where several tokens compress different regions or questions (which the authors mention as future work) is a natural way to escape that ceiling.
  • Since data diversity rather than volume drives the gains, generating reconstruction targets in other languages or specialized domains could yield multilingual or transferable embedding models without collecting new paired data.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes CoCoA, a three-stage training pipeline for turning MLLMs into multimodal embedding encoders. Stage 1 warms up bidirectional attention via masked next-token prediction (MNTP) on text and masked autoencoding (MAE) on images. Stage 2 introduces an EOS-bridged reconstruction task: the input is split into Block A and Block B separated by an EOS token, attention is truncated so that Block A and Block B cannot attend directly, and the model must reconstruct aggressively masked Block B tokens from the EOS representation, forcing compression of Block A semantics into EOS. Stage 3 performs standard contrastive learning on the EOS embeddings. On MMEB-V1, CoCoA built on Qwen2-VL/Qwen2.5-VL improves over contrastive-only baselines and approaches MoCa at 3B, while using substantially less pretraining data.

Significance. The core idea is attractive and timely: replacing some of the massive contrastive data with a self-supervised reconstruction objective that explicitly compresses input semantics into the embedding token could make MLLM-based embedding training more data-efficient. The paper provides a clean staged design, helpful ablations (Tables 3–4), and a scaling analysis (Figures 4–5) showing diminishing returns for in-distribution data and gains from synthetic diversity. If the EOS-bottleneck mechanism is validated by a proper control, this would be a useful contribution to the multimodal embedding literature. The current evidence, however, does not isolate the proposed attention-truncation mechanism from the added reconstruction supervision, so the central claim remains to be established.

major comments (2)
  1. [§4.2 and Table 3] The ablation 'w/o EOS-Bridged Reconstruction' removes the entire Stage 2, which changes both the attention mask (truncation) and the training objective (MNTP on Block B). Consequently, the drop from 62.9 to 60.7 does not establish that the EOS bottleneck is the active ingredient; it may reflect the benefit of additional reconstruction supervision on the same data. Please add a control that trains on the same Stage 2 data and masking ratios but with full bidirectional attention between Block A and Block B (no EOS bridge). If this control matches the 62.9 result, the compression mechanism is not necessary for the reported gains.
  2. [§4.2] The claim that 'any cross-block information must be transmitted exclusively through the ⟨EOS⟩' requires a formal specification of the attention mask for every layer, including handling of residual connections, positional encodings, and the 30% unmasked Block B tokens. Without a mask-matrix definition and a probe (e.g., reconstructing with the EOS token removed, or measuring the mutual information between Block B hidden states and Block A inputs), the exclusivity assertion is unverified. The mask-ratio trend in Figure 3 is consistent with EOS compression but does not rule out shortcuts through unmasked local context.
minor comments (4)
  1. [§6 and Table 2] The phrase 'state-of-the-art performance' for models ≤3B is overstated: at 3B CoCoA ties MoCa at 67.5, and at 7B it trails MoCa on overall (70.6 vs 71.5) and OOD (64.3 vs 67.6). Recommend 'competitive' or 'matching SOTA' to reflect the numbers.
  2. [§7.1 and Table 3] The ablation uses 100K pretraining samples for all variants, but the full pipeline in Table 2 uses 300K original + 200K synthetic for Stage 2. The relationship between the 62.9 in Table 3 and the 66.0 in Table 2 should be clarified, since the contrastive data also differs (1M vs. full MMEB-V1 train).
  3. [Figure 3] The bar chart would be easier to interpret with numeric labels on each bar and preferably error bars or significance tests. As printed, the differences between 20%/50% are small (e.g., Classification 59.6 vs 60.4 vs 60.6).
  4. [Throughout] There are several typos and formatting issues: 'picbecue' in the case study, 'CoCoAworks' in §7.1, and inconsistent spacing in 'CoCoA' headings. Also, the paper does not specify whether the attention mask is applied to all layers identically; consider a short pseudo-code or diagram.

Circularity Check

0 steps flagged

No significant circularity: CoCoA is an empirical training-method paper evaluated on an external benchmark; its central claims are not equivalent to fitted inputs or self-citation chains.

full rationale

The paper makes no equation-level derivation that reduces to its own inputs. The core claim is empirical: a three-stage pretraining pipeline (bidirectional warm-up, EOS-bridged reconstruction, contrastive learning) improves MMEB-V1 scores over strong baselines. The EOS compression mechanism is a training objective, not a prediction: the model is optimized to reconstruct masked Block B tokens from the EOS under an explicit attention-truncation mask, and the paper then measures downstream retrieval/classification metrics on MMEB-V1 eval. The statement that cross-block information passes only through the EOS is a designed property of the attention mask, not a fitted result. The main ablation removes the entire EOS-bridged stage, so it does not isolate the mask from the extra reconstruction data, but this is an experimental-control limitation, not circularity. The self-citations ([19] 'Compressing then Matching', [39] 'Unleashing the Power of LLMs') appear in related work and motivation, while the method's load-bearing components are also anchored to external work (LLM2Vec, MAE, MoCa) and to direct benchmark comparisons; no uniqueness theorem or prior conclusion is invoked to force the result. The paper's own case study acknowledges a limitation—single-EOS compression can bias representations for complex images—which further indicates the claims are empirical rather than definitional. Training on MMEB-V1 train and evaluating on MMEB-V1 eval is standard benchmark practice and does not constitute a fitted-input-called-prediction pattern.

Axiom & Free-Parameter Ledger

9 free parameters · 6 axioms · 0 invented entities

The method relies on standard contrastive/autoencoding objectives, benchmark-specific data assumptions, and the unverified architectural guarantee that EOS is the sole cross-block information bottleneck. The free parameters are mostly training hyperparameters selected by hand or by performance on MMEB-V1. No new physical entities are introduced.

free parameters (9)
  • Stage-1 text mask ratio = 20%
    MNTP masking fraction chosen without a reported sweep; affects warm-up quality.
  • Stage-1 image patch mask ratio = 50%
    MAE masking fraction inherited from MAE defaults; chosen by hand.
  • MAE loss weight w = 0.5
    Weight in Eq. (3); no ablation shown for this value.
  • Stage-2 Block B mask ratio = 70% (100% if <4 tokens)
    Masking probability; authors ablate 20/50/70 and select 70 based on MMEB-V1 performance.
  • Contrastive temperature tau = 0.02
    Temperature in InfoNCE; no sensitivity analysis reported.
  • LoRA rank r = 16
    Rank for parameter-efficient fine-tuning; chosen by hand.
  • Learning rate = 5e-5
    Used across all stages; no per-stage schedule or warm-up described.
  • Effective batch size = 1024
    Batch size for contrastive learning; chosen by hand.
  • Pretraining data volumes = 100K stage1; 300K real + 200K synthetic stage2
    Data sizes chosen empirically; scaling curves in Figure 4 show saturation at 300K real samples.
axioms (6)
  • domain assumption MMEB-V1 train query-target pairs are semantically aligned and representative of multimodal embedding tasks
    All pretraining and contrastive data come from MMEB-V1's train split (Sections 5.1, 5.2); if pairs are noisy, the reconstruction signal weakens.
  • domain assumption Qwen2.5-VL-7B synthetic captions/questions/answers are high-quality and do not leak the eval distribution
    200K synthetic pairs generated from MMEB-V1 train images are added to pretraining (Section 4.2); no contamination or quality analysis is provided.
  • ad hoc to paper The attention truncation blocks all non-EOS cross-block information flow in every layer
    The central claim in Section 4.2 assumes EOS is the sole bridge; residual connections, positional embeddings, or an incomplete mask implementation could violate this.
  • domain assumption MNTP on masked text and MAE on image patches provide a valid warm-up for bidirectional attention
    Borrowed from LLM2Vec/MoCa; only ablative support on MMEB-V1 is given (Tables 3-4).
  • standard math InfoNCE with in-batch negatives is a valid contrastive objective for the target tasks
    Assumes sampled in-batch negatives are true negatives; standard in the embedding literature.
  • domain assumption MMEB-V1 eval set is a valid proxy for universal multimodal embedding quality
    All conclusions are drawn from this single benchmark; OOD subsets mitigate but do not eliminate the limitation.

pith-pipeline@v1.3.0-alltime-deepseek · 16938 in / 17132 out tokens · 164758 ms · 2026-08-02T19:36:50.002245+00:00 · methodology

0 comments
read the original abstract

Multimodal embedding models, rooted in multimodal large language models (MLLMs), have yielded significant performance improvements across diverse tasks such as retrieval and classification. However, most existing approaches rely heavily on large-scale contrastive learning, with limited exploration of how the architectural and training paradigms of MLLMs affect embedding quality. While effective for generation, the causal attention and next-token prediction paradigm of MLLMs does not explicitly encourage the formation of globally compact representations, limiting their effectiveness as multimodal embedding backbones. To address this, we propose CoCoA, a Content reconstruction pre-training paradigm based on Collaborative Attention for multimodal embedding optimization. Specifically, we restructure the attention flow and introduce an EOS-based reconstruction task, encouraging the model to reconstruct input from the corresponding <EOS> embeddings. This drives the multimodal model to compress the semantic information of the input into the <EOS> token, laying the foundations for subsequent contrastive learning. Extensive experiments on MMEB-V1 demonstrate that CoCoA built upon Qwen2-VL and Qwen2.5-VL significantly improves embedding quality. Results validate that content reconstruction serves as an effective strategy to maximize the value of existing data, enabling multimodal embedding models generate compact and informative representations, raising their performance ceiling.

Figures

Figures reproduced from arXiv: 2603.01471 by Daiting Shi, Da Li, Dawei Yin, Hengran Zhang, Jiafeng Guo, Jiahan Chen, Keping Bi, Lixin Su, Yinqiong Cai.

Figure 1
Figure 1. Figure 1: Overview of CoCoA. Stage 1: Bidirectional attention warm-up using MAE and MNTP for different modalities; Stage 2: [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: An example of synthetic data. The fine-grained [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance under different mask ratios. We adopt [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: The effect of pretraining data scale on In-Domain [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 4
Figure 4. Figure 4: Performance under different pre-training data vol [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Case study of visual information reconstruction [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 37 linked inside Pith

  1. [1]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski,...

    Pith/arXiv arXiv 2022

  2. [2]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. Qwen2.5-VL Technical Rep...

    Pith/arXiv arXiv 2025

  3. [3]

    Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders. arXiv:2404.05961 [cs.CL] https://arxiv.org/abs/ 2404.05961

    Pith/arXiv arXiv 2024

  4. [4]

    Ali Borji. 2020. ObjectNet Dataset: Reanalysis and Correction. arXiv:2004.02042 [cs.CV] https://arxiv.org/abs/2004.02042

    Pith/arXiv arXiv 2020

  5. [5]

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 2014. Food-101–mining discriminative components with random forests. InEuropean conference on com- puter vision. Springer, 446–461

    2014

  6. [6]

    Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. 2022. WebQA: Multihop and Multimodal QA. arXiv:2109.00590 [cs.CL] https://arxiv.org/abs/2109.00590

    Pith/arXiv arXiv 2022

  7. [7]

    Haonan Chen, Hong Liu, Yuping Luo, Liang Wang, Nan Yang, Furu Wei, and Zhicheng Dou. 2025. MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings. arXiv:2506.23115 [cs.CV] https: //arxiv.org/abs/2506.23115

    Pith/arXiv arXiv 2025

  8. [8]

    Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, and Zhicheng Dou. 2025. mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data. arXiv:2502.08468 [cs.CV] https://arxiv.org/abs/ 2502.08468

    Pith/arXiv arXiv 2025

  9. [9]

    Jiahan Chen, Da Li, and Keping Bi. 2025. LifeIR at the NTCIR-18 Lifelog-6 Task. arXiv:2505.20987 [cs.IR] https://arxiv.org/abs/2505.20987

    Pith/arXiv arXiv 2025

  10. [10]

    Yan Fang, Jingtao Zhan, Qingyao Ai, Jiaxin Mao, Weihang Su, Jia Chen, and Yiqun Liu. 2024. Scaling Laws For Dense Retrieval. arXiv:2403.18684 [cs.IR] https://arxiv.org/abs/2403.18684

    Pith/arXiv arXiv 2024

  11. [11]

    Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. 2025. ColPali: Efficient Document Retrieval with Vision Language Models. arXiv:2407.01449 [cs.IR] https://arxiv.org/abs/2407. 01449

    Pith/arXiv arXiv 2025

  12. [12]

    Tiancheng Gu, Kaicheng Yang, Ziyong Feng, Xingjun Wang, Yanzhao Zhang, Dingkun Long, Yingda Chen, Weidong Cai, and Jiankang Deng. 2025. Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs. arXiv:2504.17432 [cs.CV] https://arxiv.org/abs/2504.17432

    arXiv 2025

  13. [13]

    Tiancheng Gu, Kaicheng Yang, Kaichen Zhang, Xiang An, Ziyong Feng, Yueyi Zhang, Weidong Cai, Jiankang Deng, and Lidong Bing. 2025. UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning. arXiv:2510.13515 [cs.CV] https://arxiv.org/abs/2510.13515

    arXiv 2025

  14. [14]

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2021. Masked Autoencoders Are Scalable Vision Learners. arXiv:2111.06377 [cs.CV] https://arxiv.org/abs/2111.06377

    Pith/arXiv arXiv 2021

  15. [15]

    Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, and Fuzhen Zhuang

  16. [16]

    Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. 2024. E5-V: Universal Embeddings with Multimodal Large Language Models. arXiv:2407.12580 [cs.CL] https://arxiv. org/abs/2407.12580

    Pith/arXiv arXiv 2024

  17. [17]

    Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen

  18. [18]

    Zhibin Lan, Liqiang Niu, Fandong Meng, Jie Zhou, and Jinsong Su. 2025. LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Con- trastive Learning. arXiv:2503.04812 [cs.CV] https://arxiv.org/abs/2503.04812

    arXiv 2025

  19. [19]

    Da Li, Yuxiao Luo, Keping Bi, Jiafeng Guo, Wei Yuan, Biao Yang, Yan Wang, Fan Yang, Tingting Gao, and Guorui Zhou. 2026. Compressing then Matching: An Effi- cient Pre-training Paradigm for Multimodal Embedding. arXiv:2511.08480 [cs.CV] https://arxiv.org/abs/2511.08480

    Pith/arXiv arXiv 2026

  20. [20]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. arXiv:2201.12086 [cs.CV] https://arxiv.org/abs/2201.12086

    Pith/arXiv arXiv 2022

  21. [21]

    Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi

    Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. 2021. Align before Fuse: Vision and Language Representation Learning with Momentum Distillation. arXiv:2107.07651 [cs.CV] https://arxiv.org/abs/2107.07651

    Pith/arXiv arXiv 2021

  22. [22]

    Mingxin Li, Yanzhao Zhang, Dingkun Long, Keqin Chen, Sibo Song, Shuai Bai, Zhibo Yang, Pengjun Xie, An Yang, Dayiheng Liu, Jingren Zhou, and Junyang Lin

  23. [23]

    Lawrence Zitnick, and Piotr Dollár

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár

  24. [24]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruc- tion Tuning. arXiv:2304.08485 [cs.CV] https://arxiv.org/abs/2304.08485

    Pith/arXiv arXiv 2023

  25. [25]

    V Jawahar

    Minesh Mathew, Viraj Bagal, Rubèn Pérez Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V Jawahar. 2021. InfographicVQA. arXiv:2104.12756 [cs.CV] https://arxiv.org/abs/2104.12756

    Pith/arXiv arXiv 2021

  26. [26]

    Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. 2021. DocVQA: A Dataset for VQA on Document Images. arXiv:2007.00398 [cs.CV] https://arxiv. org/abs/2007.00398

    Pith/arXiv arXiv 2021

  27. [27]

    Rui Meng, Ziyan Jiang, Ye Liu, Mingyi Su, Xinyi Yang, Yuepeng Fu, Can Qin, Zeyuan Chen, Ran Xu, Caiming Xiong, Yingbo Zhou, Wenhu Chen, and Semih Yavuz. 2025. VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents. arXiv:2507.04590 [cs.CV] https://arxiv.org/abs/2507. 04590

    Pith/arXiv arXiv 2025

  28. [28]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.00020 [cs.CV] https://arxiv.org/ abs/2103.00020

    Pith/arXiv arXiv 2021

  29. [29]

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki

  30. [30]

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-OKVQA: A Benchmark for Visual Question Answer- ing using World Knowledge. arXiv:2206.01718 [cs.CV] https://arxiv.org/abs/ 2206.01718

    Pith/arXiv arXiv 2022

  31. [31]

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. 2025. SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featur...

    Pith/arXiv arXiv 2025

  32. [32]

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2019. Representation Learning with Contrastive Predictive Coding. arXiv:1807.03748 [cs.LG] https://arxiv.org/ abs/1807.03748

    Pith/arXiv arXiv 2019

  33. [33]

    Zhen Wang, Xu Shan, Xiangxie Zhang, and Jie Yang. 2022. N24News: A New Dataset for Multimodal News Classification. arXiv:2108.13327 [cs.CL] https: //arxiv.org/abs/2108.13327

    Pith/arXiv arXiv 2022

  34. [34]

    Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. 2023. UniIR: Training and Benchmarking Universal Multimodal Information Retrievers. arXiv:2311.17136 [cs.CV] https://arxiv.org/abs/2311. 17136

    Pith/arXiv arXiv 2023

  35. [35]

    Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogerio Feris. 2020. Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback. arXiv:1905.12794 [cs.CV] https://arxiv. org/abs/1905.12794

    Pith/arXiv arXiv 2020

  36. [36]

    Chunyu Xie, Bin Wang, Fanjing Kong, Jincheng Li, Dawei Liang, Gengshen Zhang, Dawei Leng, and Yuhui Yin. 2025. FG-CLIP: Fine-Grained Visual and Textual Alignment. arXiv:2505.05071 [cs.CV] https://arxiv.org/abs/2505.05071

    Pith/arXiv arXiv 2025

  37. [37]

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xi- aodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. 2021. FILIP: Fine- grained Interactive Language-Image Pre-Training. arXiv:2111.07783 [cs.CV] https://arxiv.org/abs/2111.07783

    Pith/arXiv arXiv 2021

  38. [38]

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. 2022. CoCa: Contrastive Captioners are Image-Text Foundation Models. arXiv:2205.01917 [cs.CV] https://arxiv.org/abs/2205.01917

    Pith/arXiv arXiv 2022

  39. [39]

    Hengran Zhang, Keping Bi, Jiafeng Guo, Xiaojie Sun, Shihao Liu, Daiting Shi, Dawei Yin, and Xueqi Cheng. 2025. Unleashing the Power of LLMs in Dense Retrieval with Query Likelihood Modeling. arXiv:2504.05216 [cs.IR] https: //arxiv.org/abs/2504.05216

    Pith/arXiv arXiv 2025

  40. [40]

    Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. 2025. GME: Improving Universal Multimodal Retrieval by Multimodal LLMs. arXiv:2412.16855 [cs.CL] https://arxiv.org/abs/2412.16855

    Pith/arXiv arXiv 2025

  41. [41]

    Junjie Zhou, Zheng Liu, Shitao Xiao, Bo Zhao, and Yongping Xiong. 2024. VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval. arXiv:2406.04292 [cs.IR] https://arxiv.org/abs/2406.04292

    Pith/arXiv arXiv 2024

  42. [42]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. arXiv:2304.10592 [cs.CV] https://arxiv.org/abs/2304.10592

    Pith/arXiv arXiv 2023

  43. [2015]

    Microsoft COCO: Common Objects in Context. arXiv:1405.0312 [cs.CV] https://arxiv.org/abs/1405.0312 Reconstructing Content with Collaborative Attention for Universal Multimodal Representation Learning SIGIR ’26, July 20–24, 2026, Melbourne, VIC, Australia

    Pith/arXiv arXiv 2026

  44. [2021]

    arXiv:2111.02114 [cs.CV] https://arxiv.org/abs/2111.02114

    LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs. arXiv:2111.02114 [cs.CV] https://arxiv.org/abs/2111.02114

    Pith/arXiv arXiv

  45. [2024]

    InFindings of the Association for Computational Linguistics: EMNLP 2024, Yaser Al-Onaizan, Mo- hit Bansal, and Yun-Nung Chen (Eds.)

    Scaling Sentence Embeddings with Large Language Models. InFindings of the Association for Computational Linguistics: EMNLP 2024, Yaser Al-Onaizan, Mo- hit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 3182–3196. doi:10.18653/v1/2024.findings-emnlp.181

    doi:10.18653/v1/2024.findings-emnlp.181 2024

  46. [2025]

    arXiv:2410.05160 [cs.CV] https://arxiv.org/abs/2410.05160

    VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks. arXiv:2410.05160 [cs.CV] https://arxiv.org/abs/2410.05160

    Pith/arXiv arXiv

  47. [2026]

    arXiv:2601.04720 [cs.CL] https://arxiv.org/abs/2601.04720

    Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking. arXiv:2601.04720 [cs.CL] https://arxiv.org/abs/2601.04720

    Pith/arXiv arXiv