Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

ReT-2 claims that one recurrent fused token can represent multimodal queries and documents, setting state-of-the-art results on M2KR and M-BEIR at lower cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A single multimodal retriever with a gated recurrent Transformer cell, one-token output, and three-layer sampling achieves strong results on M2KR and M-BEIR and improves RAG-VQA accuracy.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Solid architectural extension of ReT, but the headline M2KR SOTA claim is not yet supported because the PreFLMR/FLMR baselines were not re-run on the image-augmented splits. the 3 major comments →

arxiv 2509.08897 v1 pith:ACRLGGDI submitted 2025-09-10 cs.CV cs.AIcs.CLcs.MM

Recurrence Meets Transformers for Universal Multimodal Retrieval

Davide Caffagni , Sara Sarto , Marcella Cornia , Lorenzo Baraldi , Rita Cucchiara This is my paper
classification cs.CV cs.AIcs.CLcs.MM
keywords universal multimodal retrievalrecurrent TransformerLSTM gatingmulti-layer feature fusionsingle-token embeddingM2KR benchmarkM-BEIR benchmarkretrieval-augmented generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that universal multimodal retrieval—matching queries and documents that mix images and text—can be done with a compact, single-token embedding produced by a recurrence-augmented Transformer. ReT-2 fuses features from early, middle, and late layers of a vision and a text encoder using LSTM-style gates, then adds global backbone features to the final token. If correct, it would show that heavy late-interaction machinery and massive MLLM-based retrievers are unnecessary for state-of-the-art multimodal search, and that retrieval quality directly improves downstream question answering. The paper backs this with results on M2KR and M-BEIR local across eight task configurations, plus RAG-VQA gains with off-the-shelf MLLMs.

Core claim

ReT-2 is a unified retrieval model that encodes a multimodal query or document as a single token by running a recurrent Transformer cell over three sampled layers of frozen (or fine-tuned) visual and textual backbones. The cell applies an LSTM-inspired forget gate and separate input gates for text and vision, so the model can decide how much of each modality and each layer to keep. The paper reports that this design—with shared query/document weights, layer pruning, and global feature injection—outperforms the previous 32-token ReT and existing baselines on M2KR and M-BEIR, while reducing inference time and memory. It also shows that ReT-2 as a retrieval backbone improves accuracy of LLaVA-M

What carries the argument

The load-bearing component is the recurrent fusion cell: a Transformer block whose hidden state evolves over backbone layers under three gates (forget, text-input, vision-input), with cross-attention into the normalized state from layer-specific visual and textual features. The final state is a single k×d token (k=1) that sums global pooler features from both backbones; this token replaces the 32-token late-interaction matrix of ReT and enables a plain dot-product InfoNCE loss.

Load-bearing premise

The M2KR state-of-the-art comparison assumes that the PreFLMR and FLMR numbers cited in Tables III and IV were produced under the same document-image-augmented splits used for ReT-2 (Section V-A augments OVEN, InfoSeek, Encyclopedic-VQA, and OKVQA with document images), but those baseline rows carry no dagger marking as re-runs, so if they come from the original text-only-document benchmark, ReT-2 may be receiving extra visual evidence the baselines did not get.

What would settle it

Re-run PreFLMR and FLMR on the image-augmented M2KR splits for OVEN, InfoSeek, Encyclopedic-VQA, and OKVQA (i.e., provide the same document images to their document encoders or mark them as text-only), and compare with ReT-2; if either baseline surpasses ReT-2 under identical document inputs, the claimed M2KR state-of-the-art does not stand. Also, evaluating ReT-2 on the M-BEIR global setting would settle whether the single-token design truly generalizes to the full 5.6M-document pool.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Universal multimodal retrieval is achievable without MLLM-scale models: ReT-2's best variant (SigLIP2 ViT-L backbone) matches or exceeds several 7B-parameter MLLM-based retrievers on M-BEIR local while using far fewer parameters and half the inference time.
  • The single-token representation eliminates the need for fine-grained late-interaction scoring (the max-over-tokens operator), replacing it with a standard dot-product and cutting memory and latency.
  • Layer pruning to three representative layers (early, middle, late) is sufficient: it preserves accuracy on M2KR and improves average score by +0.1 over using all layers, and enables the same recurrent cell to work across backbones of different depths.
  • Retrieval quality is a decisive factor in knowledge-intensive VQA: using ReT-2 to retrieve top-3 documents raises the accuracy of off-the-shelf MLLMs on Encyclopedic-VQA and InfoSeek compared to using CLIP/SigLIP2, UniIR, ReT, or PreFLMR as the retriever.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the single-token result holds beyond these benchmarks, the 32-token late-interaction design shared by several retrievers may be over-parameterized; testing ReT-2 against a rank-regularized variant of ReT would isolate whether the gain comes from token compression or from the recurrent gating itself.
  • The gate activations shown in Figure 5 provide a per-layer, per-modality relevance profile; one could use those statistics as a cheap diagnostic to predict which datasets benefit most from visual versus textual context, or to guide automatic layer selection for new backbones.
  • The paper evaluates only the local M-BEIR setting; extending ReT-2 to the global 5.6M-document pool would test whether the single fused token retains discriminative power at billion-scale candidate sets, where late-interaction methods often shine or fail.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces ReT-2, a unified multimodal retrieval model that extends the authors' earlier ReT architecture. ReT-2 shares query/document encoder weights, uses a single fused token instead of 32 tokens, samples only three backbone layers (early, middle, late) for the recurrent fusion cell, and injects global pooler features. The authors evaluate on M2KR (with document-image-augmented splits for four datasets) and M-BEIR local, and additionally test the retriever in retrieval-augmented VQA with off-the-shelf MLLMs. They report consistent gains over ReT, PreFLMR, FLMR, and UniIR, along with faster inference than late-interaction baselines. Ablations in Table II support the main architectural choices, and code and models are released.

Significance. If the claims hold, ReT-2 is a useful, efficient alternative to late-interaction multimodal retrievers and to much larger MLLM-based retrievers, achieving near-SOTA on M-BEIR and strong RAG-VQA results at a fraction of the inference cost. The paper's strengths are its systematic ablations, the reproduced UniIR baseline on M-BEIR, the efficiency analysis, and the public release of code and models. However, the main M2KR state-of-the-art claim currently rests on an unfair protocol comparison: the PreFLMR/FLMR rows in Tables III and IV are cited from the original text-only-document benchmark while ReT-2 is evaluated on the document-image-augmented splits described in Section V-A. In addition, the method's handling of missing modalities, which is central to the 'universal' claim and to several M-BEIR tasks, is not specified, and the layer-pruning decision is based on test-split activations. These issues are fixable but require substantial revision.

major comments (3)
  1. [Section V-A, Tables III and IV] The M2KR SOTA comparison is not apples-to-apples. Section V-A states that ReT-2 augments the OVEN, InfoSeek, Encyclopedic-VQA, and OKVQA splits by attaching images to reference documents. However, the PreFLMR and FLMR rows in Tables III and IV that lack the † marker are not re-run under this protocol; the dagger is reserved for the authors' own reproductions. Thus ReT-2 sees an extra visual signal on four of the eight M2KR datasets, while the cited baseline numbers come from the original text-only-document benchmark. The large gaps (e.g., 67.9 vs. 57.4 for CLIP ViT-L in Table III) could be partly due to this protocol difference. The authors must either re-run PreFLMR/FLMR on the augmented splits or clearly separate the augmented-split comparison from the original-benchmark comparison before claiming state-of-the-art on M2KR.
  2. [Section IV-A, Eqs. (4)-(9); Section V-A] The paper claims that ReT-2 'can also handle scenarios with missing modalities' and evaluates on M-BEIR tasks that are unimodal on one side (e.g., q_T→d_V, q_T→(d_V,d_T)). However, the method description defines the recurrent cell, gating, and global feature injection always in terms of both a visual representation E_V and a textual representation E_T. There is no description of how a missing image or missing text is represented (zero vector, masked cross-attention, separate token, or something else). This is a reproducibility gap for a central claim of the paper: without this detail, the M-BEIR results in Table V cannot be reconstructed from the text. Please specify the missing-modality protocol explicitly.
  3. [Section V-C, Fig. 5] The layer-pruning selection uses test-split information. In the Layer Pruning subsection, the authors state that the choice of three representative layers is 'further supported by an empirical analysis of the average gate activations of ReT, conducted on the InfoSeek and Encyclopedic-VQA test splits of M2KR.' The selected layer indices are then used as a fixed component of ReT-2 and evaluated on the same test splits. This constitutes tuning a hyperparameter on the test set, which can inflate the reported numbers. The selection should be made on validation splits, or the authors should argue that the layer choice is architecture-agnostic and not fitted to the test data.
minor comments (5)
  1. [Abstract and Table V] The abstract claims 'consistently achieves state-of-the-art performance across diverse settings,' but on M-BEIR local, LamRA achieves an average of 56.6 vs. ReT-2 SigLIP2's 56.4 (Table V). The paper acknowledges this in Section V-D, so the abstract should be qualified (e.g., 'state-of-the-art among non-MLLM methods').
  2. [Section IV-A, Eqs. (8)-(9)] The dimension notation for W_final is inconsistent. Eq. (8) writes W_final ∈ R^{d×d}, but Section V-B says its dimension is 'set to match d with the dimension of the global features.' Since the recurrent hidden size (1,024 according to Section V-B) can differ from the backbone pooler size (e.g., 768 for CLIP ViT-L), Eq. (9)'s addition h_L + E_V + E_T is dimensionally unclear. Please clarify whether W_final projects to the global feature dimension.
  3. [Table VII] The row label 'PreFLMR [2]' should be 'PreFLMR [18]'; [2] is the authors' own ReT paper. The same typo appears in the qualitative results section.
  4. [Section V-A] Minor language issue: 'enabling a more effectively evaluation' should be 'enabling a more effective evaluation.'
  5. [Table VI] The column headers ('Training Info', 'Inference Time (ms)', 'All↓', '#Tokens') are hard to parse. Please use clearer subheadings (e.g., 'GPUs', 'Hours', 'Forward (ms)', 'Retrieval (ms)', 'Total (ms)').

Circularity Check

0 steps flagged

No significant circularity: the paper is an empirical benchmark evaluation, and its few self-citations are disclosed and non-load-bearing.

full rationale

The paper does not claim to derive a theoretical result or to validate a fitted parameter against data from which it was fit. ReT-2 is a new architecture trained and evaluated on held-out splits of the M2KR and M-BEIR benchmarks, and the reported numbers are standard empirical results. The main self-referential element is the use of the authors' earlier ReT model as an architectural starting point and as a baseline; this is disclosed explicitly ('This work is an extended and improved version of our earlier conference paper [2]') and is not used to justify the outcome. No uniqueness theorem or load-bearing result is imported from the authors' prior work. The claimed 'predictions' are benchmark retrieval scores, not quantities that are equal by construction to training targets. The protocol concern raised in the skeptical note (Section V-A augments some M2KR splits with document images, while some baseline rows in Tables III and IV are cited from papers that used text-only documents) is a validity/fairness issue, not a circularity issue: it does not make ReT-2's numbers an algebraic consequence of its inputs. Under the review rules, such a concern belongs to correctness risk, not to the circularity score. No circular step can be quoted and exhibited.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claim is an empirical system result, not a theoretical derivation. The main hand-chosen quantities are the three fused layers per backbone (selected using test-split gate activations) and the single-token output (selected by ablation). No new physical or conceptual entities are introduced.

free parameters (4)
  • Selected backbone layer indices (e.g., 3,7,11 for CLIP ViT-B; 3,18,23 for ViT-L/SigLIP2; 4,25,31 for OpenCLIP ViT-H) = Table I
    Chosen by hand and from gate activation analysis on InfoSeek and Encyclopedic-VQA test splits (Fig. 5); assumed to transfer to all other datasets.
  • Number of fused layers per backbone = 3
    Fixed by design after efficiency analysis and ablation; not derived from first principles.
  • Output token count k = 1
    Selected after rank-collapse analysis and ablation over 32, 16, 8, 4, and 1 tokens (Table II).
  • Hidden dimension d = 1024
    Carried over from ReT; no analysis justifying this value is provided.
axioms (4)
  • domain assumption Pre-trained CLIP/SigLIP2/ColBERTv2 backbones provide usable layer-wise features for both modalities
    Relied on throughout; the recurrent cell operates on backbone activations E^V_l and E^T_l (Eq. 4).
  • domain assumption Benchmark metrics (R@K, pseudo-recall, exact-match accuracy) reflect retrieval quality
    All SOTA claims are defined by these metrics.
  • domain assumption InfoNCE loss with single-token dot product is an adequate training objective for retrieval
    Training procedure in Section IV-B.
  • domain assumption Document-image augmentation of M2KR splits is a valid benchmark modification
    Section V-A introduces augmented documents; the paper assumes comparison with non-augmented baselines remains fair.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Recurrence Meets Transformers for Universal Multimodal Retrieval." pith.science (2026). https://pith.science/paper/ACRLGGDI

@misc{pith2026250908897,
  author       = {Pith},
  title        = {Pith review of: Recurrence Meets Transformers for Universal Multimodal Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ACRLGGDI}},
  note         = {Machine review of arXiv:2509.08897}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

With the rapid advancement of multimodal retrieval and its application in LLMs and multimodal LLMs, increasingly complex retrieval tasks have emerged. Existing methods predominantly rely on task-specific fine-tuning of vision-language models and are limited to single-modality queries or documents. In this paper, we propose ReT-2, a unified retrieval model that supports multimodal queries, composed of both images and text, and searches across multimodal document collections where text and images coexist. ReT-2 leverages multi-layer representations and a recurrent Transformer architecture with LSTM-inspired gating mechanisms to dynamically integrate information across layers and modalities, capturing fine-grained visual and textual details. We evaluate ReT-2 on the challenging M2KR and M-BEIR benchmarks across different retrieval configurations. Results demonstrate that ReT-2 consistently achieves state-of-the-art performance across diverse settings, while offering faster inference and reduced memory usage compared to prior approaches. When integrated into retrieval-augmented generation pipelines, ReT-2 also improves downstream performance on Encyclopedic-VQA and InfoSeek datasets. Our source code and trained models are publicly available at: https://github.com/aimagelab/ReT-2

Figures

Figures reproduced from arXiv: 2509.08897 by Davide Caffagni, Lorenzo Baraldi, Marcella Cornia, Rita Cucchiara, Sara Sarto.

Figure 1
Figure 1. Figure 1: In this work, we present Recurrence-enhanced Transformer (ReT-2) , a novel retrieval approach supporting different tasks and data configurations, from cross-modal image-to-text retrieval – i.e., q V → d T , to multimodal text–image-to-text–image retrieval – i.e., (q T , qV ) → (d T , dV ). The plot shows average results on the M-BEIR benchmark tasks [1], highlighting the performance gains of the proposed m… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the proposed Recurrence-enhanced Transformer (ReT-2) for universal multimodal retrieval. IV. PROPOSED METHOD In this section, we introduce an enhanced variant of ReT, referred to as ReT-2, which is specifically designed to address the limitations identified in the original model. ReT-2 aims to improve retrieval effectiveness and efficiency when dealing with heterogeneous data sources in large-s… view at source ↗
Figure 3
Figure 3. Figure 3: Graphical illustration of the proposed recurrent cell for multimodal [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Visualization of the differences between the previous method ( [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Analysis of average gate activation over 2k examples from the InfoSeek and Encyclopedic-VQA test split of the M2KR benchmark. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative results on the M2KR benchmark [18], for datasets that do not include document images. [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative results on the M2KR benchmark [18], for datasets that include document images. We highlight the reference answer in bold font whenever [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Qualitative results on the Task #3 of the M-BEIR benchmark [1], using an example from the EDIS subset [61]. Ground-truth image-text documents [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Qualitative results on the Task #8 of the M-BEIR benchmark [1], using an example from the InfoSeek subset [19]. Ground-truth image-text documents [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Sample results for the knowledge-intensive VQA task on the validation split of InfoSeek, augmenting Qwen2.5-VL with context retrieved by different [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Sample results for the knowledge-intensive VQA task on the test split of Encyclopedic-VQA, augmenting Qwen2.5-VL with context retrieved by [PITH_FULL_IMAGE:figures/full_fig_p019_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. TEMA: Anchor the Image, Follow the Text for Multi-Modification Composed Image Retrieval

    cs.CV 2026-04 unverdicted novelty 7.0

    TEMA is the first framework for multi-modification composed image retrieval, using entity mapping to improve accuracy on both new complex datasets and existing benchmarks while balancing efficiency.

Reference graph

Works this paper leans on

80 extracted references · 12 linked inside Pith · cited by 1 Pith paper

  1. [1]

    UniIR: Training and Benchmarking Universal Multimodal Information Retrievers,

    C. Wei, Y . Chen, H. Chen, H. Hu, G. Zhang, J. Fu, A. Ritter, and W. Chen, “UniIR: Training and Benchmarking Universal Multimodal Information Retrievers,” inECCV, 2024

    2024

  2. [2]

    Recurrence-Enhanced Vision-and-Language Transformers for Robust Multimodal Document Retrieval,

    D. Caffagni, S. Sarto, M. Cornia, L. Baraldi, and R. Cucchiara, “Recurrence-Enhanced Vision-and-Language Transformers for Robust Multimodal Document Retrieval,” inCVPR, 2025

    2025

  3. [3]

    Unsupervised Dense Information Retrieval with Con- trastive Learning,

    G. Izacard, M. Caron, L. Hosseini, S. Riedel, P. Bojanowski, A. Joulin, and E. Grave, “Unsupervised Dense Information Retrieval with Con- trastive Learning,”arXiv preprint arXiv:2112.09118, 2021

    Pith/arXiv arXiv 2021

  4. [4]

    SIFT meets CNN: A decade survey of instance retrieval,

    L. Zheng, Y . Yang, and Q. Tian, “SIFT meets CNN: A decade survey of instance retrieval,”IEEE Trans. PAMI, vol. 40, no. 5, pp. 1224–1244, 2017

    2017

  5. [5]

    Large-scale image retrieval with attentive deep local features,

    H. Noh, A. Araujo, J. Sim, T. Weyand, and B. Han, “Large-scale image retrieval with attentive deep local features,” inCVPR, 2017

    2017

  6. [6]

    Microsoft COCO: Common Objects in Context,

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft COCO: Common Objects in Context,” inECCV, 2014

    2014

  7. [7]

    Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning,

    P. Sharma, N. Ding, S. Goodman, and R. Soricut, “Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning,” inACL, 2018

    2018

  8. [8]

    LAION-5B: An open large-scale dataset for training next generation image-text models,

    C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsmanet al., “LAION-5B: An open large-scale dataset for training next generation image-text models,” inNeurIPS, 2022

    2022

  9. [9]

    Learning Transferable Visual Models From Natural Language Super- vision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning Transferable Visual Models From Natural Language Super- vision,” inICML, 2021

    2021

  10. [10]

    Scaling up visual and vision-language representation learning with noisy text supervision,

    C. Jia, Y . Yang, Y . Xia, Y .-T. Chen, Z. Parekh, H. Pham, Q. Le, Y .- H. Sung, Z. Li, and T. Duerig, “Scaling up visual and vision-language representation learning with noisy text supervision,” inICML, 2021

    2021

  11. [11]

    Reproducible scaling laws for contrastive language-image Learning,

    M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev, “Reproducible scaling laws for contrastive language-image Learning,” inCVPR, 2023

    2023

  12. [12]

    Sigmoid Loss for Language Image Pre-Training,

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid Loss for Language Image Pre-Training,” inICCV, 2023

    2023

  13. [13]

    SigLIP 2: Multilingual Vision-Language Encoders with Improved Se- mantic Understanding, Localization, and Dense Features,

    M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Alabdulmohsin, N. Parthasarathy, T. Evans, L. Beyer, Y . Xia, B. Mustafaet al., “SigLIP 2: Multilingual Vision-Language Encoders with Improved Se- mantic Understanding, Localization, and Dense Features,”arXiv preprint arXiv:2502.14786, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    The Revolution of Multimodal Large Language Models: A Survey,

    D. Caffagni, F. Cocchi, L. Barsellotti, N. Moratelli, S. Sarto, L. Baraldi, M. Cornia, and R. Cucchiara, “The Revolution of Multimodal Large Language Models: A Survey,” inACL Findings, 2024

    2024

  15. [15]

    Improved Baselines with Visual Instruction Tuning,

    H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved Baselines with Visual Instruction Tuning,” inCVPR, 2024

    2024

  16. [16]

    LLaV A-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning,

    F. Cocchi, N. Moratelli, D. Caffagni, S. Sarto, L. Baraldi, M. Cornia, and R. Cucchiara, “LLaV A-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning,” inICCV Workshops, 2025

    2025

  17. [17]

    Qwen2.5-VL Technical Report,

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tanget al., “Qwen2.5-VL Technical Report,”arXiv preprint arXiv:2502.13923, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    PreFLMR: Scaling Up Fine- Grained Late-Interaction Multi-modal Retrievers,

    W. Lin, J. Mei, J. Chen, and B. Byrne, “PreFLMR: Scaling Up Fine- Grained Late-Interaction Multi-modal Retrievers,” inACL, 2024

    2024

  19. [19]

    Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?

    Y . Chen, H. Hu, Y . Luan, H. Sun, S. Changpinyo, A. Ritter, and M.-W. Chang, “Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?” inEMNLP, 2023

    2023

  20. [20]

    Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories,

    T. Mensink, J. Uijlings, L. Castrejon, A. Goel, F. Cadar, H. Zhou, F. Sha, A. Araujo, and V . Ferrari, “Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories,” inICCV, 2023

    2023

  21. [21]

    Long Short-Term Memory,

    S. Hochreiter and J. Schmidhuber, “Long Short-Term Memory,”Neural Computation, vol. 9, 1997

    1997

  22. [22]

    Open-domain Visual Entity Recog- nition: Towards Recognizing Millions of Wikipedia Entities,

    H. Hu, Y . Luan, Y . Chen, U. Khandelwal, M. Joshi, K. Lee, K. Toutanova, and M.-W. Chang, “Open-domain Visual Entity Recog- nition: Towards Recognizing Millions of Wikipedia Entities,” inCVPR, 2023

    2023

  23. [23]

    OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge,

    K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi, “OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge,” inCVPR, 2019

    2019

  24. [24]

    EV A-CLIP: Improved Training Techniques for CLIP at Scale,

    Q. Sun, Y . Fang, L. Wu, X. Wang, and Y . Cao, “EV A-CLIP: Improved Training Techniques for CLIP at Scale,”arXiv preprint arXiv:2303.15389, 2023

    Pith/arXiv arXiv 2023

  25. [25]

    Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models,

    B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hocken- maier, and S. Lazebnik, “Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models,” inICCV, 2015

    2015

  26. [26]

    Image Retrieval on Real-Life Images With Pre-Trained Vision-and-Language Models,

    Z. Liu, C. Rodriguez-Opazo, D. Teney, and S. Gould, “Image Retrieval on Real-Life Images With Pre-Trained Vision-and-Language Models,” inICCV, 2021

    2021

  27. [27]

    Long-CLIP: Unlocking the Long-Text Capability of CLIP,

    B. Zhang, P. Zhang, X. Dong, Y . Zang, and J. Wang, “Long-CLIP: Unlocking the Long-Text Capability of CLIP,” inECCV, 2024

    2024

  28. [28]

    Thinking Fast and Slow: Efficient Text-to-Visual Retrieval With Transformers,

    A. Miech, J.-B. Alayrac, I. Laptev, J. Sivic, and A. Zisserman, “Thinking Fast and Slow: Efficient Text-to-Visual Retrieval With Transformers,” in CVPR, 2021

    2021

  29. [29]

    Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrieval,

    A. Brown, W. Xie, V . Kalogeiton, and A. Zisserman, “Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrieval,” inECCV, 2020

    2020

  30. [30]

    Conditioned and Composed Image Retrieval Combining and Partially Fine-Tuning CLIP-Based Features,

    A. Baldrati, M. Bertini, T. Uricchio, and A. Del Bimbo, “Conditioned and Composed Image Retrieval Combining and Partially Fine-Tuning CLIP-Based Features,” inCVPR, 2022

    2022

  31. [31]

    BLIP: Bootstrapping Language- Image Pre-training for Unified Vision-Language Understanding and Generation,

    J. Li, D. Li, C. Xiong, and S. Hoi, “BLIP: Bootstrapping Language- Image Pre-training for Unified Vision-Language Understanding and Generation,” inICML, 2022

    2022

  32. [32]

    GENIUS: A Generative Framework for Universal Multimodal Search,

    S. Kim, X. Zhu, X. Lin, M. Bastan, D. Gray, and S. Kwak, “GENIUS: A Generative Framework for Universal Multimodal Search,” inCVPR, 2025

    2025

  33. [33]

    Fine-grained Late- interaction Multi-modal Retrieval for Retrieval Augmented Visual Ques- tion Answering,

    W. Lin, J. Chen, J. Mei, A. Coca, and B. Byrne, “Fine-grained Late- interaction Multi-modal Retrieval for Retrieval Augmented Visual Ques- tion Answering,” inNeurIPS, 2023

    2023

  34. [34]

    ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT,

    O. Khattab and M. Zaharia, “ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT,” inACM SIGIR, 2020

    2020

  35. [35]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askellet al., “Language models are few-shot learners,” inNeurIPS, 2020

    2020

  36. [36]

    The Llama 3 Herd of Models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fanet al., “The Llama 3 Herd of Models,”arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  37. [37]

    LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant,

    Y . Liu, Y . Zhang, J. Cai, X. Jiang, Y . Hu, J. Yao, Y . Wang, and W. Xie, “LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant,” inCVPR, 2025

    2025

  38. [38]

    MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs,

    S.-C. Lin, C. Lee, M. Shoeybi, J. Lin, B. Catanzaro, and W. Ping, “MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs,” inICLR, 2025

    2025

  39. [39]

    PUMA: Layer- Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning,

    Y . Lyu, R. Shao, G. Chen, Y . Zhu, W. Guan, and L. Nie, “PUMA: Layer- Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning,”ACM Multimedia, 2025

    2025

  40. [40]

    Joint Fusion and Encoding: Advancing Multimodal Retrieval from the Ground Up,

    L. Huang, Q. Wu, Z. Miao, and T. Yamasaki, “Joint Fusion and Encoding: Advancing Multimodal Retrieval from the Ground Up,”arXiv preprint arXiv:2502.20008, 2025

    Pith/arXiv arXiv 2025

  41. [41]

    Attention Is All You Need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention Is All You Need,” inNeurIPS, 2017

    2017

  42. [42]

    Transformers: State- of-the-Art Natural Language Processing,

    T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowiczet al., “Transformers: State- of-the-Art Natural Language Processing,” inEMNLP, 2020

    2020

  43. [43]

    LLaMA: Open and Efficient Foundation Language Models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi`ere, N. Goyal, E. Hambro, F. Azharet al., “LLaMA: Open and Efficient Foundation Language Models,”arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  44. [44]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” inICLR, 2021

    2021

  45. [45]

    Training data-efficient image transformers & distillation through attention,

    H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. J ´egou, “Training data-efficient image transformers & distillation through attention,” inICML, 2021

    2021

  46. [46]

    Scaling Vision Transformers,

    X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer, “Scaling Vision Transformers,” inCVPR, 2022

    2022

  47. [47]

    Transformers in Vision: A Survey,

    S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in Vision: A Survey,”ACM CSUR, vol. 54, no. 10s, pp. 1–41, 2022

    2022

  48. [48]

    When attention meets fast recurrence: Training language models with reduced compute,

    T. Lei, “When attention meets fast recurrence: Training language models with reduced compute,” inEMNLP, 2021

    2021

  49. [49]

    Simple recurrent units for highly parallelizable recurrence,

    T. Lei, Y . Zhang, S. I. Wang, H. Dai, and Y . Artzi, “Simple recurrent units for highly parallelizable recurrence,” inEMNLP, 2017

    2017

  50. [50]

    The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation,

    A. Bapna, G. Foster, L. Jones, M. Hughes, M. Johnson, M. Chen, M. Schuster, N. J. Parmaret al., “The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation,” inACL, 2018. 13

    2018

  51. [51]

    R-Transformer: Recurrent Neu- ral Network Enhanced Transformer,

    Z. Wang, Y . Ma, Z. Liu, and J. Tang, “R-Transformer: Recurrent Neu- ral Network Enhanced Transformer,”arXiv preprint arXiv:1907.05572, 2019

    Pith/arXiv arXiv 1907

  52. [52]

    Block- Recurrent Transformers,

    D. Hutchins, I. Schlag, Y . Wu, E. Dyer, and B. Neyshabur, “Block- Recurrent Transformers,” inNeurIPS, 2022

    2022

  53. [53]

    ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction,

    K. Santhanam, O. Khattab, J. Saad-Falcon, C. Potts, and M. Zaharia, “ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction,” inNAACL, 2022

    2022

  54. [54]

    Lambda-Skip Connections: the Architectural Component that Prevents Rank Collapse,

    F. A. Joseph, J. Sieber, M. Zeilinger, and C. A. Alonso, “Lambda-Skip Connections: the Architectural Component that Prevents Rank Collapse,” inICLR, 2025

    2025

  55. [55]

    Layer Normalization,

    J. Lei Ba, J. R. Kiros, and G. E. Hinton, “Layer Normalization,”arXiv preprint arXiv:1607.06450, 2016

    Pith/arXiv arXiv 2016

  56. [56]

    WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning,

    K. Srinivasan, K. Raman, J. Chen, M. Bendersky, and M. Najork, “WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning,” inACM SIGIR, 2021

    2021

  57. [57]

    IGLUE: A Benchmark for Transfer Learning Across Modalities, Tasks, and Languages,

    E. Bugliarello, F. Liu, J. Pfeiffer, S. Reddy, D. Elliott, E. M. Ponti, and I. Vuli ´c, “IGLUE: A Benchmark for Transfer Learning Across Modalities, Tasks, and Languages,” inICML, 2022

    2022

  58. [58]

    KVQA: Knowledge- Aware Visual Question Answering,

    S. Shah, A. Mishra, N. Yadati, and P. P. Talukdar, “KVQA: Knowledge- Aware Visual Question Answering,” inAAAI, 2019

    2019

  59. [59]

    MS MARCO: A Human Generated MAchine Reading COmprehension Dataset,

    T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, and L. Deng, “MS MARCO: A Human Generated MAchine Reading COmprehension Dataset,” inNeurIPS, 2016

    2016

  60. [60]

    Visual Instruction Tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual Instruction Tuning,” NeurIPS, 2024

    2024

  61. [61]

    EDIS: Entity- Driven Image Search over Multimodal Web Content,

    S. Liu, W. Feng, T.-j. Fu, W. Chen, and W. Y . Wang, “EDIS: Entity- Driven Image Search over Multimodal Web Content,” inEMNLP, 2023

    2023

  62. [62]

    Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback,

    H. Wu, Y . Gao, X. Guo, Z. Al-Halah, S. Rennie, K. Grauman, and R. Feris, “Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback,” inCVPR, 2021

    2021

  63. [63]

    Automatic Spatially-Aware Fashion Concept Discovery,

    X. Han, Z. Wu, P. X. Huang, X. Zhang, M. Zhu, Y . Li, Y . Zhao, and L. S. Davis, “Automatic Spatially-Aware Fashion Concept Discovery,” inICCV, 2017

    2017

  64. [64]

    Visual News: Benchmark and Challenges in News Image Captioning,

    F. Liu, Y . Wang, T. Wang, and V . Ordonez, “Visual News: Benchmark and Challenges in News Image Captioning,” inEMNLP, 2021

    2021

  65. [65]

    DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data,

    S. Fu, N. Tamir, S. Sundaram, L. Chai, R. Zhang, T. Dekel, and P. Isola, “DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data,” inNeurIPS, 2023

    2023

  66. [66]

    ADAM: a Method for Stochastic Optimiza- tion,

    D. P. Kingma and J. L. Ba, “ADAM: a Method for Stochastic Optimiza- tion,” inICML, 2015

    2015

  67. [67]

    Billion-Scale Similarity Search with GPUs,

    J. Johnson, M. Douze, and H. J ´egou, “Billion-Scale Similarity Search with GPUs,”IEEE Trans. on Big Data, vol. 7, no. 3, pp. 535–547, 2019

    2019

  68. [68]

    Multi- Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practices,

    J. Lin, H. Chen, Y . Fan, Y . Fan, X. Jin, H. Su, J. Fu, and X. Shen, “Multi- Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practices,” inCVPR, 2025

    2025

  69. [69]

    Wiki-LLaV A: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs,

    D. Caffagni, F. Cocchi, N. Moratelli, S. Sarto, M. Cornia, L. Baraldi, and R. Cucchiara, “Wiki-LLaV A: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs,” inCVPR Workshops, 2024

    2024

  70. [70]

    Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Ques- tion Answering Evaluation,

    J. Bulian, C. Buck, W. Gajewski, B. B ¨orschinger, and T. Schuster, “Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Ques- tion Answering Evaluation,” inEMNLP, 2022

    2022

  71. [71]

    Aug- menting Multimodal LLMs with Self-Reflective Tokens for Knowledge- based Visual Question Answering,

    F. Cocchi, N. Moratelli, M. Cornia, L. Baraldi, and R. Cucchiara, “Aug- menting Multimodal LLMs with Self-Reflective Tokens for Knowledge- based Visual Question Answering,” inCVPR, 2025

    2025

  72. [72]

    RoRA-VLM: Robust Retrieval-Augmented Vision Language Models,

    J. Qi, Z. Xu, R. Shao, Y . Chen, J. Di, Y . Cheng, Q. Wang, and L. Huang, “RoRA-VLM: Robust Retrieval-Augmented Vision Language Models,” arXiv preprint arXiv:2410.08876, 2024

    Pith/arXiv arXiv 2024

  73. [73]

    EchoSight: Advancing Visual-Language Models with Wiki Knowledge,

    Y . Yan and W. Xie, “EchoSight: Advancing Visual-Language Models with Wiki Knowledge,” inEMNLP Findings, 2024

    2024

  74. [74]

    Towards Gen- eral Continuous Memory for Vision-Language Models,

    W. Wu, Z. Song, K. Zhou, Y . Shao, Z. Hu, and B. Huang, “Towards Gen- eral Continuous Memory for Vision-Language Models,”arXiv preprint arXiv:2505.17670, 2025

    Pith/arXiv arXiv 2025

  75. [75]

    mR 2AG: Multimodal Retrieval-Reflection- Augmented Generation for Knowledge-Based VQA,

    T. Zhang, Z. Zhang, Z. Ma, Y . Chen, Z. Qi, C. Yuan, B. Li, J. Pu, Y . Zhao, Z. Xieet al., “mR 2AG: Multimodal Retrieval-Reflection- Augmented Generation for Knowledge-Based VQA,”arXiv preprint arXiv:2411.15041, 2024

    Pith/arXiv arXiv 2024

  76. [76]

    BLIP-2: Bootstrapping Language- Image Pre-training with Frozen Image Encoders and Large Language Models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “BLIP-2: Bootstrapping Language- Image Pre-training with Frozen Image Encoders and Large Language Models,” inICML, 2023

    2023

  77. [77]

    InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning,

    W. Dai, J. Li, D. Li, A. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. Hoi, “InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning,” inNeurIPS, 2023

    2023

  78. [78]

    WebQA: Multihop and Multimodal QA,

    Y . Chang, M. Narang, H. Suzuki, G. Cao, J. Gao, and Y . Bisk, “WebQA: Multihop and Multimodal QA,” inCVPR, 2022. Davide Caffagnireceived the M.Sc. degree in Com- puter Engineering cum laude from the University of Modena and Reggio Emilia in 2023. He is currently pursuing a PhD in Information and Communication Technologies (ICT) at the University of Moden...

    2022

  79. [1992]

    The Eleven

    She is currently a Full Professor of Computer Engineering and the elected Rector of the University of Modena and Reggio Emilia, where she also heads the AImageLab Laboratory. She has authored or coauthored more than 600 papers in journals and international proceedings, and has been a coordi- nator of several projects in computer vision and pattern recogni...

    1907

  80. [2016]

    Eu- phorbia pulcherrima

    T˜onis Lukas went to [...] PreFLMR [18]: The dog in the im- age is white. PreFLMR [18]: The train shown in the image is a passenger train. ReT [2]: This is an image of Brie at the 2009 Los Angeles Film Festival. Alison Brie went to California Insti- tute of the Arts, Royal [...] ReT [2]: This is an image of Jean-Luc Warsmann (2016). Jean- Luc Warsmann wen...

    2009

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.