Pith. sign in

REVIEW 2 major objections 5 minor 3 cited by

Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering

T0 review · 2 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Reflective tokens let an MLLM decide when to retrieve and what to trust

desk verdict A solid adaptation of Self-RAG tokens to multimodal knowledge-based VQA with real reported gains and honest ablations; the main caveat is that relevance supervision flows through GPT-4 judging captions plus the gold answer, so the visual grounding of those tokens is not fully proven. read the letter →

arxiv 2411.16863 v2 pith:2GSDZXB5 submitted 2024-11-25 cs.CV cs.AIcs.CLcs.MM

Federico Cocchi , Nicholas Moratelli , Marcella Cornia , Lorenzo Baraldi , Rita Cucchiara This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.MM
keywords reflectivetokensretrieval-augmentedgenerationmultimodallargelanguagemodelsknowledge-basedvisualquestionansweringWikipediaretrievalrelevancepredictiontwo-stagetrainingEncyclopedic-VQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a multimodal large language model can be taught to manage external knowledge by emitting four special tokens: two that decide whether retrieval is necessary and two that judge whether each retrieved passage is relevant. The proposed model, ReflectiVA, adds these tokens to a pretrained vision-language model and trains them in two stages, first on passages within the same Wikipedia article and then on a broader mixture that includes passages from other articles and samples that need no retrieval. The result is a retrieval-augmented VQA system that outperforms prior methods on Encyclopedic-VQA and InfoSeek benchmarks while retaining strong performance on standard multimodal benchmarks. The authors argue this shows that explicit self-reflective decisions about when to retrieve and what to use are more effective than simply feeding all retrieved text to the model.

What carries the argument

The central object is the set of four reflective tokens added to the model's vocabulary — <RET>, <NORET>, <REL>, <NOREL> — which turn retrieval decisions into ordinary next-token predictions. They work with a coarse-grained CLIP-based retriever that proposes top-k Wikipedia documents from the query image, and the tokens act as a trained gate: one pair decides whether to enter the retrieval pipeline at all, the other filters the candidate passages before answer generation. The two-stage, two-model training recipe is the load-bearing mechanism: an in-article model first learns relevance from automatically annotated passages, and its predictions are used to generate the training labels for the final model, which is trained jointly on retrieval-needed, retrieval-not-needed, and soft/hard negative samples.

What would settle it

Take a set of Encyclopedic-VQA or InfoSeek questions whose answers depend on visual details that standard captioners usually miss, such as fine-grained color, texture, or small objects, collect human relevance labels for the same passages, and compare ReflectiVA's <REL>/<NOREL> accuracy on those cases against cases where the caption captures the needed detail. If accuracy is near chance on the caption-miss cases, the central claim that the model judges passage relevance from the image-question pair would be undermined.

Watch

Extended reading notes

Core claim

ReflectiVA extends the vocabulary of a pretrained multimodal LLM with four reflective tokens and trains the model to use them as a decision layer over retrieval. Given an image and question, the model first samples <RET> or <NORET>; if retrieval is chosen, it scores each candidate passage with <REL> or <NOREL> and only then generates the answer from the accepted passages. The two-stage recipe first trains an in-article discriminator on automatically annotated positive and negative passages, then uses that model's predictions to label a larger mixed dataset with soft and hard negatives, along with instruction-following samples that require no retrieval. Experiments report gains over recent retrieval-augmented competitors, for example 35.5 versus 26.4 single-hop accuracy on Encyclopedic-VQA under the same visual retrieval mode, and similar margins on InfoSeek, with ablations showing that removing the relevance tokens causes a large accuracy drop. The paper also reports that the model retains most of its performance on standard MLLM benchmarks, which it attributes to the <NORET> pathway.

Load-bearing premise

The relevance labels that train the reflective tokens are produced by GPT-4 from image captions, not from the image pixels themselves; if the captions omit the visual detail needed to judge a question, the labels and the learned retrieval decisions will be systematically wrong.

Editorial extensions

If this is right

  • If the central claim is correct, retrieval-augmented MLLMs can avoid injecting irrelevant context: the model only reads passages it has marked <REL>.
  • The approach can be applied jointly to multiple knowledge-based datasets in a single fine-tuned model rather than training a separate model per dataset.
  • Zero-shot transfer to other knowledge-based VQA datasets improves over the base model and over prior retrieval-augmented systems, suggesting the token decisions generalize beyond the training benchmarks.
  • The <NORET> pathway preserves performance on standard MLLM benchmarks, so adding a retrieval mechanism does not necessarily degrade general visual abilities.
  • Built-in re-ranking via the log-probability difference of <REL> and <NOREL> can further boost accuracy, reaching 37.8 single-hop accuracy on Encyclopedic-VQA with k=50 and kp=2.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A test that separates image-only cues from caption-only cues would reveal how much of the learned relevance decision depends on pixel content versus textual captions; if the latter dominates, stronger captioning or text-only inputs could replace the visual branch.
  • The same token-gating recipe could be applied to other modalities or structured knowledge sources where retrieval decisions are currently made by external pipeline components rather than by the generator itself.
  • Because the two-stage training labels come from captions, the ceiling of the approach is set by caption quality; a human-labeled subset could quantify this ceiling and guide better label generation.
  • The built-in re-ranking result suggests the <REL>/<NOREL> distribution is a usable relevance score, which could replace separate re-ranking modules in other retrieval-augmented systems.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper proposes ReflectiVA, a retrieval-augmented multimodal LLM that augments LLaVA-v1.5 with four special tokens (<RET>, <NORET>, <REL>, <NOREL>) to decide whether external knowledge is needed and to filter retrieved Wikipedia passages. Training uses a two-stage, two-model procedure: an in-article reflective model is trained on GPT-4-generated positive/negative passage labels (derived from image captions and the ground-truth answer), and its predictions are then used to annotate a larger corpus for training the final model. Experiments on Encyclopedic-VQA and InfoSeek show consistent gains over reproduced Wiki-LLaVA and EchoSight baselines, and ablations attribute most of the gain to the relevance tokens. The paper also reports zero-shot results on ViQuAE and S3VQA, performance preservation on standard MLLM benchmarks, and an analysis of token prediction accuracy.

Significance. The strengths of the paper are the clarity of the two-stage recipe, the fairness of the baseline comparisons (same LLM, same knowledge base via reproductions), the public release of code and annotations, and the systematic ablations in Table 5 that isolate the relevance tokens. If the results hold, the reflective-token mechanism is a practical and simple alternative to dedicated re-rankers. The main caveat is that the relevance supervision originates from GPT-4 judging text (captions and question-answer strings) rather than from the image content; this dependency is acknowledged only implicitly and is not stress-tested.

major comments (2)
  1. [Sec. 3.2, Sec. 3.3, Table 5] The relevance supervision for the in-article model is generated by prompting GPT-4 with the question, the ground-truth answer, and captions of the image produced by three captioning models, rather than with the image itself. As stated in Sec. 3.2, the prompt asks the LLM to judge whether a passage answers the question 'given the textual description of I'. For fine-grained questions on Encyclopedic-VQA and InfoSeek, the captions frequently omit the specific visual entity needed to link the image to its Wikipedia article, and the presence of the ground-truth answer in the prompt allows GPT-4 to label a passage as positive merely because it contains the answer string. The second stage (Sec. 3.3) then distills the in-article model's predictions into the final training set, so this label noise propagates. Because Table 5 attributes a drop of 11.9 points on E-VQA single-hop (35.5 to 23.6) and 8.7 points on InfoSeek (40.1 to 31.4) to the removal of the <REL>/<NOREL> tokens, the claim that these tokens implement multimodal relevance determination is not fully supported. I ask the authors to (i) report the agreement between the GPT-4-generated labels and human judgments on a random sample, (ii) include an ablation in which the in-article model is trained without the image (text-only) to measure the contribution of visual grounding, and (iii) state this limitation explicitly in the main paper.
  2. [Table 5 (top), Table 6] The difference between the full model and the 'always <RET>' variant is 0.2 points on Encyclopedic-VQA (35.5 vs. 35.3) and 0.1 points on InfoSeek (40.1 vs. 40.0). The paper does not report variance, confidence intervals, or multiple seeds for any of the experiments, so these differences are within the expected run-to-run noise of fine-tuning an 8B model. The 'always <RET>' row is the main evidence for the <RET>/<NORET> token's usefulness on the two target datasets, and the text in Sec. 4.3 describes it as 'only a minor performance degradation', which is consistent with the numbers but also with no effect. Please report standard deviations over at least three training runs for Table 5, and consider evaluating the need token on a dataset (or an artificially mixed split) where a substantial fraction of questions do not require retrieval, where the token decision has a measurable effect.
minor comments (5)
  1. [Sec. 4.1 and Supplementary Material] The exact GPT-4 prompt and the few-shot examples used for passage annotation are not provided, despite the text saying they are in the supplementary; please include the full prompt template and examples.
  2. [Table 1] The InfoSeek row for 'EVA-CLIP-8B Textual (T+S)' has a formatting error in the R@20 column ('29.456.1'), and similar broken-number issues appear in Table 9; please correct the table formatting throughout.
  3. [Table 9] The green percentages indicating the fraction of samples for which the model emits <RET> are not explained in the caption; specify the denominator (e.g., percentage of all ViQuAE/S3VQA samples).
  4. [Sec. 3.2] The phrase 'proprietary LLM' is vague; since GPT-4 is named later in Sec. 4.1, name it at first occurrence for reproducibility.
  5. [Sec. 5] The conclusion would benefit from a short limitations paragraph in the main text, particularly regarding the caption-based annotation dependency; the supplementary failure-case section is welcome but does not address this issue.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the reflective-token gains are measured on held-out external benchmarks and isolated by ablations, and the minor self-citations are not load-bearing.

full rationale

The paper's derivation chain is a supervised training pipeline, not a logical reduction. GPT-4 is used only to generate relevance labels for training passages from the ground-truth Wikipedia article; the final ReflectiVA model does not receive GPT-4's labels or the ground-truth answer at inference, so its <REL>/<NOREL>/<RET>/<NORET> predictions are not equivalent to the training inputs by construction. The main accuracy gains (Table 2) are reported on the Encyclopedic-VQA test set and the InfoSeek validation set, which are not used for pseudo-label generation (only the training splits are used), and the zero-shot ViQuAE/S3VQA results and Table 5 ablations (removing relevance tokens drops E-VQA single-hop from 35.5 to 23.6) provide independent evidence that the relevance-selection mechanism, rather than a fitted constant, drives the improvement. The use of GPT-4 with the ground-truth answer in the labeling prompt is a legitimate data-quality and potential answer-leakage concern, but it does not make the reported predictions equal to the labels by construction. The self-citations to Wiki-LLaVA [9] and LLaVA-MORE [15] are to baselines and a base initialization, and are not used to justify the central claim. No circular step can be quoted or reduced to an equation of the paper.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the quality of automatically generated relevance labels, the retrieval index, and the hyperparameter choice k=5; no new physical or external entities are introduced.

free parameters (1)
  • number of retrieved documents k = 5
    Chosen based on validation performance across the two datasets; Table 5 bottom shows k=5 as the best trade-off.
assumptions (3)
  • domain assumption The external knowledge base contains a passage that answers each knowledge-based question.
    Used throughout; if the knowledge base lacks the answer, retrieval cannot help. Stated implicitly in the task definition in Sec 3.
  • domain assumption The CLIP/EVA-CLIP retriever provides sufficient recall at k=5 for the correct document.
    Coarse-grained retrieval is the first step of the pipeline, and Table 1 shows recall limits, especially for CLIP ViT-L/14.
  • domain assumption GPT-4 annotations of passage relevance are accurate enough to train the reflective tokens.
    Sec 3.2 describes generating positive and negative passages with GPT-4 using image captions; the whole two-stage training depends on these labels.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering." pith.science (2026). https://pith.science/paper/2GSDZXB5

@misc{pith2026241116863,
  author       = {Pith},
  title        = {Pith review of: Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2GSDZXB5}},
  note         = {Machine review of arXiv:2411.16863}
}
read the original abstract

Multimodal LLMs (MLLMs) are the natural extension of large language models to handle multimodal inputs, combining text and image data. They have recently garnered attention due to their capability to address complex tasks involving both modalities. However, their effectiveness is limited to the knowledge acquired during training, which restricts their practical utility. In this work, we introduce a novel method to enhance the adaptability of MLLMs by integrating external knowledge sources. Our proposed model, Reflective LLaVA (ReflectiVA), utilizes reflective tokens to dynamically determine the need for external knowledge and predict the relevance of information retrieved from an external database. Tokens are trained following a two-stage two-model training recipe. This ultimately enables the MLLM to manage external knowledge while preserving fluency and performance on tasks where external knowledge is not needed. Through our experiments, we demonstrate the efficacy of ReflectiVA for knowledge-based visual question answering, highlighting its superior performance compared to existing methods. Source code and trained models are publicly available at https://aimagelab.github.io/ReflectiVA.

Figures

Figures reproduced from arXiv: 2411.16863 by the authors.

Figure 1
Figure 1. Overview of ReflectiVA, which employs reflective tokens [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Training approach for ReflectiVA. An in-article model is trained to predict the relevance of passages extracted from the ground-truth [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Sample qualitative results on image-question pairs from Encyclopedic-VQA (top row) and InfoSeek (bottom row), where we [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Re-ranking performance on the single-hop question split [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Additional qualitative results on image-question pairs from Encyclopedic-VQA, where we compare the answers provided by [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Additional qualitative results on image-question pairs from InfoSeek, where we compare the answers provided by ReflectiVA with [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Examples of failure cases on image-question pairs from Encyclopedic-VQA (top row) and InfoSeek (bottom row). [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ExpStar: Towards Automatic Commentary Generation for Multi-discipline Scientific Experiments

    cs.CV 2025-07 conditional novelty 6.0 of 10

    ExpStar, with a new 7,714-sample ExpInstruct dataset, generates step-level scientific experiment commentary including procedures, principles, and safety guidelines.

  2. Towards General Continuous Memory for Vision-Language Models

    cs.LG 2025-05 conditional novelty 5.0 of 10

    A vision-language model can act as its own continuous memory encoder, compressing external multimodal knowledge into eight embeddings that improve reasoning when prepended to the frozen model.

  3. OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrieval

    cs.IR 2025-05 conditional novelty 5.0 of 10

    A coarse-to-fine multimodal RAG pipeline, OMGM, aligns a query image with entity summaries, reranks with a trained multimodal late-interaction model, and selects the best section, achieving state-of-the-art retrieval ...

Reference graph

Works this paper leans on

75 extracted references · 55 canonical work pages · cited by 3 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 6, 15

    arXiv 2023

  2. [2]

    Flamingo: a Visual Language Model for Few-Shot Learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a Visual Language Model for Few-Shot Learning. In NeurIPS,

    work page

  3. [3]

    Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Han- naneh Hajishirzi. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. In ICLR, 2024. 2, 15

    work page 2024

  4. [4]

    OpenFlamingo: An Open- Source Framework for Training Large Autoregressive Vision- Language Models

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. OpenFlamingo: An Open- Source Framework for Training Large Autoregressive Vision- Language Models. arXiv preprint arXiv:2308.01390, 2023. 2

    arXiv 2023

  5. [5]

    Improving Language Models by Retrieving from Trillions of Tokens

    Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving Language Models by Retrieving from Trillions of Tokens. In ICML, 2022. 2

    work page 2022

  6. [6]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020. 1

    work page 2020

  7. [7]

    Tomayto, Tomahto

    Jannis Bulian, Christian Buck, Wojciech Gajewski, Benjamin Boerschinger, and Tal Schuster. Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation. arXiv preprint arXiv:2202.07654, 2022. 5

    arXiv 2022

  8. [8]

    The Revolution of Multi- modal Large Language Models: A Survey

    Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Lorenzo Baraldi, Mar- cella Cornia, and Rita Cucchiara. The Revolution of Multi- modal Large Language Models: A Survey. In ACL Findings,

    work page

Show all 75 references
  1. [9]

    Wiki-LLaV A: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs

    Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Wiki-LLaV A: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs. In CVPR Workshops, 2024. 2, 5, 6, 7, 8, 12, 14, 16, 17

    2024

  2. [10]

    CLAIR: Evaluating Image Captions with Large Language Models

    David Chan, Suzanne Petryk, Joseph E Gonzalez, Trevor Darrell, and John Canny. CLAIR: Evaluating Image Captions with Large Language Models. In EMNLP, 2023. 13

    2023

  3. [11]

    Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts. In CVPR, 2021. 2

    2021

  4. [12]

    Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions? In EMNLP, 2023

    Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming-Wei Chang. Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions? In EMNLP, 2023. 1, 2, 4, 5, 12

    2023

  5. [13]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality, 2023. 2, 6

    2023

  6. [14]

    Scaling Instruction- Finetuned Language Models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling Instruction- Finetuned Language Models. JMLR, 25(70):1–53, 2024. 1

    2024

  7. [15]

    LLaV A-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning

    Federico Cocchi, Nicholas Moratelli, Davide Caffagni, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. LLaV A-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning. arXiv preprint arXiv:2503.15621, 2025. 2, 5

    2025 arXiv

  8. [16]

    InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. arXiv preprint arXiv:2305.06500, 2023. 2, 4, 6

    2023 arXiv

  9. [17]

    Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models. arXiv preprint arXiv:2409.17146, 2024. 2

    2024 arXiv

  10. [18]

    The Llama 3 Herd of Models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783,

    arXiv

  11. [19]

    MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Meng- dan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv preprint arXiv:2306.13394, 2023. 2, 13

    2023 arXiv

  12. [20]

    Data- Comp: In search of the next generation of multimodal datasets

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Data- Comp: In search of the next generation of multimodal datasets. In NeurIPS, 2023. 2

    2023

  13. [21]

    Making the v in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answer- ing. In CVPR, 2017. 5

    2017

  14. [22]

    KAT: A Knowledge Aug- mented Transformer for Vision-and-Language

    Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, and Jianfeng Gao. KAT: A Knowledge Aug- mented Transformer for Vision-and-Language. In NAACL,

  15. [23]

    Retrieval Augmented Language Model Pre- Training

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval Augmented Language Model Pre- Training. In ICML, 2020. 2

    2020

  16. [24]

    OneLLM: One Framework to Align All Modalities with Language

    Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. OneLLM: One Framework to Align All Modalities with Language. In CVPR, 2024. 1

    2024

  17. [25]

    REVEAL: Retrieval-Augmented Visual-Language Pre-Training With Multi-Source Multi- modal Knowledge Memory

    Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai- Wei Chang, Yizhou Sun, Cordelia Schmid, David A Ross, and Alireza Fathi. REVEAL: Retrieval-Augmented Visual-Language Pre-Training With Multi-Source Multi- modal Knowledge Memory. In CVPR, 2023. 12

    2023

  18. [26]

    GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering

    Drew A Hudson and Christopher D Manning. GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering. In CVPR, 2019. 2, 13

    2019

  19. [27]

    Unsupervised Dense Information Retrieval with Contrastive Learning

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised Dense Information Retrieval with Contrastive Learning. arXiv preprint arXiv:2112.09118, 2021. 4, 7, 8, 14

    2021 arXiv

  20. [28]

    Atlas: Few- shot Learning with Retrieval Augmented Language Models

    Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hos- seini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few- shot Learning with Retrieval Augmented Language Models. JMLR, 24(251):1–43, 2023. 2

    2023

  21. [29]

    Se- lect, Substitute, Search: A New Benchmark for Knowledge- Augmented Visual Question Answering

    Aman Jain, Mayank Kothyari, Vishwajeet Kumar, Preethi Jyothi, Ganesh Ramakrishnan, and Soumen Chakrabarti. Se- lect, Substitute, Search: A New Benchmark for Knowledge- Augmented Visual Question Answering. In SIGIR, 2021. 2, 13, 14

    2021

  22. [30]

    Mistral 7B

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B. arXiv preprint arXiv:2310.06825,

    arXiv

  23. [31]

    Mixtral of Experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Deven- dra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of Experts. arXiv preprint arXiv:2401.04088, 2024. 1

    2024 arXiv

  24. [32]

    Active Retrieval Augmented Generation

    Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Gra- ham Neubig. Active Retrieval Augmented Generation. In EMNLP, 2023. 2

    2023

  25. [33]

    A Diagram is Worth a Dozen Images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A Diagram is Worth a Dozen Images. In ECCV, 2016. 13

    2016

  26. [34]

    OBELICS: An Open Web-Scale Filtered Dataset of Inter- leaved Image-Text Documents

    Hugo Laurenc ¸on, Lucile Saulnier, L´eo Tronchon, Stas Bek- man, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Sid- dharth Karamcheti, Alexander Rush, Douwe Kiela, et al. OBELICS: An Open Web-Scale Filtered Dataset of Inter- leaved Image-Text Documents. In NeurIPS, 2023. 1, 2

    2023

  27. [35]

    What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024

    Hugo Laurenc ¸on, L´eo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024. 2

    2024 arXiv

  28. [36]

    ViQuAE, a dataset for knowledge-based visual question answering about named entities

    Paul Lerner, Olivier Ferret, Camille Guinaudeau, Herv ´e Le Borgne, Romaric Besan c ¸on, Jos´e G Moreno, and Jes ´us Lov´on Melgarejo. ViQuAE, a dataset for knowledge-based visual question answering about named entities. In SIGIR,

  29. [37]

    Cross- modal Retrieval for Knowledge-based Visual Question An- swering

    Paul Lerner, Olivier Ferret, and Camille Guinaudeau. Cross- modal Retrieval for Knowledge-based Visual Question An- swering. In ECIR, 2024. 2, 6

    2024

  30. [38]

    SEED-Bench: Benchmarking Multi- modal LLMs with Generative Comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. SEED-Bench: Benchmarking Multi- modal LLMs with Generative Comprehension. arXiv preprint arXiv:2307.16125, 2023. 13

    2023 arXiv

  31. [39]

    LLaV A-OneVision: Easy Visual Task Transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chun- yuan Li. LLaV A-OneVision: Easy Visual Task Transfer.arXiv preprint arXiv:2408.03326, 2024. 2

    2024 arXiv

  32. [40]

    BLIP- 2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP- 2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. In ICML, 2023. 1, 2, 4, 6, 13, 14, 15

    2023

  33. [41]

    Evaluating Object Hallucination in Large Vision-Language Models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating Object Hallucination in Large Vision-Language Models. In EMNLP, 2023. 2, 13

    2023

  34. [42]

    Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answer- ing

    Weizhe Lin, Jinghong Chen, Jingbiao Mei, Alexandru Coca, and Bill Byrne. Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answer- ing. In NeurIPS, 2023. 12

    2023

  35. [43]

    PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi- modal Retrievers

    Weizhe Lin, Jingbiao Mei, Jinghong Chen, and Bill Byrne. PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi- modal Retrievers. In ACL, 2024. 2

    2024

  36. [44]

    REVIVE: Regional Visual Rep- resentation Matters in Knowledge-Based Visual Question Answering

    Yuanze Lin, Yujia Xie, Dongdong Chen, Yichong Xu, Chen- guang Zhu, and Lu Yuan. REVIVE: Regional Visual Rep- resentation Matters in Knowledge-Based Visual Question Answering. In NeurIPS, 2022. 12

    2022

  37. [45]

    Visual Instruction Tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. In NeurIPS, 2023. 2

    2023

  38. [46]

    Improved Baselines with Visual Instruction Tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved Baselines with Visual Instruction Tuning. InCVPR,

  39. [47]

    MMBench: Is Your Multi-modal Model an All-around Player? arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. MMBench: Is Your Multi-modal Model an All-around Player? arXiv preprint arXiv:2307.06281, 2023. 13

    2023 arXiv

  40. [48]

    Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering. InNeurIPS,

  41. [49]

    Ok-VQA: A Visual Question Answering Benchmark Requiring External Knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-VQA: A Visual Question Answering Benchmark Requiring External Knowledge. In CVPR, 2019. 2, 12

    2019

  42. [50]

    MM1: Methods, Analy- sis & Insights from Multimodal LLM Pre-training

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. MM1: Methods, Analy- sis & Insights from Multimodal LLM Pre-training. In ECCV,

  43. [51]

    Encyclopedic VQA: Visual Questions About Detailed Properties of Fine-Grained Categories

    Thomas Mensink, Jasper Uijlings, Lluis Castrejon, Arushi Goel, Felipe Cadar, Howard Zhou, Fei Sha, Andr ´e Araujo, and Vittorio Ferrari. Encyclopedic VQA: Visual Questions About Detailed Properties of Fine-Grained Categories. In ICCV, 2023. 1, 2, 4, 5, 12

    2023

  44. [52]

    PlotQA: Reasoning over Scientific Plots

    Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. PlotQA: Reasoning over Scientific Plots. In WACV, 2020. 5

    2020

  45. [53]

    Revisiting Image Cap- tioning Training Paradigm via Direct CLIP-based Optimiza- tion

    Nicholas Moratelli, Davide Caffagni, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Revisiting Image Cap- tioning Training Paradigm via Direct CLIP-based Optimiza- tion. In BMVC, 2024. 13

    2024

  46. [54]

    Training Language Models to Follow Instructions with Human Feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training Language Models to Follow Instructions with Human Feedback. In NeurIPS, 2022. 1

    2022

  47. [55]

    RoRA-VLM: Ro- bust Retrieval-Augmented Vision Language Models

    Jingyuan Qi, Zhiyang Xu, Rulin Shao, Yang Chen, Jing Di, Yu Cheng, Qifan Wang, and Lifu Huang. RoRA-VLM: Ro- bust Retrieval-Augmented Vision Language Models. arXiv preprint arXiv:2410.08876, 2024. 2, 6, 12

    2024 arXiv

  48. [56]

    Learning Transferable Visual Models from Natural Language Supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models from Natural Language Supervi- sion. In ICML, 2021. 2, 4, 5

    2021

  49. [57]

    Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR, 21(1):5485–5551,

  50. [58]

    In- Context Retrieval-Augmented Language Models

    Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Am- non Shashua, Kevin Leyton-Brown, and Yoav Shoham. In- Context Retrieval-Augmented Language Models. TACL, 11: 1316–1331, 2023. 2, 7

    2023

  51. [59]

    A-OKVQA: A Benchmark for Visual Question Answering Using World Knowledge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-OKVQA: A Benchmark for Visual Question Answering Using World Knowledge. In ECCV, 2022. 2, 12

    2022

  52. [60]

    KVQA: Knowledge-aware Visual Question Answering

    Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar. KVQA: Knowledge-aware Visual Question Answering. In AAAI, 2019. 2

    2019

  53. [61]

    Towards VQA Models That Can Read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA Models That Can Read. In CVPR, 2019. 2, 13

    2019

  54. [62]

    WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning

    Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning. In ACM SIGIR, 2021. 12

    2021

  55. [63]

    EV A-CLIP- 18B: Scaling CLIP to 18 Billion Parameters

    Quan Sun, Jinsheng Wang, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, and Xinlong Wang. EV A-CLIP- 18B: Scaling CLIP to 18 Billion Parameters. arXiv preprint arXiv:2402.04252, 2024. 2, 4, 5

    2024 arXiv

  56. [64]

    Emu: Generative Pretraining in Multimodality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative Pretraining in Multimodality. In ICLR, 2024. 1, 2

    2024

  57. [65]

    Stanford Alpaca: An Instruction-Following LLaMA Model, 2023

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford Alpaca: An Instruction-Following LLaMA Model, 2023. 2

    2023

  58. [66]

    LLaMA: Open and Efficient Foundation Language Models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Mar- tinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302.13971, 2023. 1, 2

    2023 arXiv

  59. [67]

    InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining

    Boxin Wang, Wei Ping, Lawrence McAfee, Peng Xu, Bo Li, Mohammad Shoeybi, and Bryan Catanzaro. InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining. In ICML, 2024. 2

    2024

  60. [68]

    UniIR: Training and Benchmarking Universal Multimodal Information Retrievers

    Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. UniIR: Training and Benchmarking Universal Multimodal Information Retrievers. In ECCV, 2024. 2

    2024

  61. [69]

    CLAIR-A: Leveraging Large Lan- guage Models to Judge Audio Captions

    Tsung-Han Wu, Joseph E Gonzalez, Trevor Darrell, and David M Chan. CLAIR-A: Leveraging Large Lan- guage Models to Judge Audio Captions. arXiv preprint arXiv:2409.12962, 2024. 13

    2024 arXiv

  62. [70]

    Grounding Language Models for Visual Entity Recognition

    Zilin Xiao, Ming Gong, Paola Cascante-Bonilla, Xingyao Zhang, Jie Wu, and Vicente Ordonez. Grounding Language Models for Visual Entity Recognition. In ECCV, 2024. 2

    2024

  63. [71]

    EchoSight: Advancing Visual- Language Models with Wiki Knowledge

    Yibin Yan and Weidi Xie. EchoSight: Advancing Visual- Language Models with Wiki Knowledge. arXiv preprint arXiv:2407.12735, 2024. 2, 5, 6, 7, 8, 12, 14, 15, 16, 17

    2024 arXiv

  64. [72]

    mPLUG- Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mPLUG- Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration. In CVPR, 2024. 1

    2024

  65. [73]

    RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs

    Yue Yu, Wei Ping, Zihan Liu, Boxin Wang, Jiax- uan You, Chao Zhang, Mohammad Shoeybi, and Bryan Catanzaro. RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs. arXiv preprint arXiv:2407.02485, 2024. 2, 7

    2024 arXiv

  66. [74]

    Give a short answer

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. In CVPR, 2024. 2, 13 Augmenting Multimodal LLM...

    2024

  67. [2024]

    1, 2, 4, 5, 6, 7, 8, 14

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.