CRAG-MM-Diagnostics: Enabling Stage-Wise Analysis of Knowledge-Intensive VQA

Benno Krojer; Hanseok Oh; Hyunji Lee; Parishad BehnamGhader; Paul Liang; Siva Reddy; Verna Dankers

REVIEW 3 major objections 5 minor 59 references

CRAG-MM-Diagnostics: Enabling Stage-Wise Analysis of Knowledge-Intensive VQA

T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Providing the correct entity name resolves only about one-fifth of a strong model's errors on knowledge-intensive visual question answering, pointing to retrieval and reasoning—not visual grounding—as the primary bottleneck.

desk verdict Useful stage-wise KI-VQA benchmark, but the headline grounding gains and the primary-bottleneck claim are both weaker than the abstract implies. read the letter →

arxiv 2607.21155 v1 pith:YJESQQA6 submitted 2026-07-23 cs.CV cs.AI

Hanseok Oh , Parishad BehnamGhader , Benno Krojer , Hyunji Lee , Paul Liang , Siva Reddy , Verna Dankers This is my paper · ORCID

classification cs.CVcs.AI

keywords KI-VQAvisualquestionansweringgroundingobjectidentificationretrieval-augmentedgenerationdiagnosticbenchmarkreferringexpressionambiguityvision-languagemodels

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the biggest obstacle in knowledge-intensive visual question answering (KI-VQA) is not seeing or naming the object in the image, but finding and reasoning over the external knowledge needed to answer. To show this, the authors introduce CRAG-MM-Diagnostics, a benchmark built on CRAG-MM that adds stage-specific annotations—target bounding boxes, entity names, disambiguated questions, and text-only questions—so each step of the pipeline can be measured separately. Across a range of vision-language models, they find that inserting the correct entity name into the question (which removes the need to localize and identify the object) still leaves the majority of errors intact; for GPT-5 only 20.9% of errors are resolved. The paper also shows that cropping the target object before image retrieval improves retrieval-augmented generation, boosting GPT-5 and Qwen accuracy by 13.3 and 8.5 percentage points. If right, the field's effort should shift toward better retrieval and multi-hop reasoning rather than further improving object recognition or grounding.

What carries the argument

The key mechanism is the benchmark's stage-isolating metadata, especially the 'text-only question' that inserts the gold entity name into the question, which is intended to decouple stages 1–2 from stage 3 without changing task difficulty. The visual grounding module (Grounding-DINO) that crops the target region before retrieval is the mechanism behind the proposed grounded bimodal RAG pipeline. The benchmark also provides referring-expression types (salient, in-image cue, knowledge-intensive cue, ambiguous) and visual-saliency scores (size, centrality, crowdedness) used to attribute failures.

What would settle it

Run the same text-only questions (with gold entity names) through a strong text-only language model with no image input, and compare to the VLMs' text-only performance. If the text-only LLM performs at or above the VLM level, the residual errors indeed lie in stage 3; if it performs substantially better, the VLM failures on text-only questions are not purely knowledge retrieval/reasoning but also reflect visual interference or modality interaction, weakening the isolation.

Watch

Extended reading notes

Core claim

We introduce CRAG-MM-Diagnostics, a diagnostic extension of CRAG-MM that annotates each KI-VQA sample with target ROIs, entity names, referring-expression types, disambiguated questions, and text-only questions, enabling stage-wise evaluation of (1) language-based visual grounding, (2) object identification, and (3) knowledge retrieval and reasoning. Across fully parametric and retrieval-augmented VLMs, our central finding is that knowledge retrieval and reasoning is the primary bottleneck: even when the ground-truth target entity name is inserted into the question, removing the need for visual grounding and object identification, the majority of errors persist (on average an 8.7-point gain,

Load-bearing premise

The text-only question—created by inserting the gold entity name into the question—is assumed to remove localization and identification errors without otherwise changing the task; if naming the entity also makes retrieval easier or alters the reasoning needed, the claim that retrieval/reasoning is the dominant bottleneck is overstated.

Editorial extensions

If this is right

Improving knowledge retrieval quality and multi-hop reasoning over retrieved evidence should be a higher priority than further improving visual grounding or object identification for KI-VQA end-task accuracy.
Benchmarks should report stage-wise metrics (grounding, identification, retrieval/reasoning) rather than only end-task accuracy.
Query ambiguity is a structural problem: disambiguating referring expressions yields consistent gains across all stages and should be addressed (e.g., via iterative query reformulation).
'Identify first, then retrieve' is an effective design principle: using the target entity name in the text retrieval query gives the largest performance gains.
Cropping the target object before image retrieval (grounded bimodal RAG) is a cheap way to boost retrieval-augmented VQA accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The text-only ablation may overestimate the retrieval/reasoning bottleneck if naming the entity also simplifies reasoning or changes the query's information needs; a control with a text-only LLM or with a misleading image alongside the entity-named question would test this.
The finding that multimodal retrievers (e.g., VLM2Vec, Qwen-3-VL-Embedding) perform worse when given text suggests current joint embedding training does not align textual cues with region-level visual retrieval; better cross-modal grounding in embedding models could unlock further RAG gains.
The paper's own generalizability check on EVQA shows grounding helps object-centric subsets (iNaturalist) but hurts landmark categories, implying a routing mechanism that decides when to crop could make grounded RAG more universally beneficial.
The strong popularity effect on VLM object identification suggests that synthetic or augmented training with balanced entity distributions, or better long-tail recognition, may be complementary to retrieval improvements.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The benchmark is the contribution. The stage-wise annotations on CRAG-MM — ROIs, entity names, referring-expression types, text-only and disambiguated questions — are carefully done and should be useful to anyone building or evaluating KI-VQA systems. The subjectivity check on the annotations is more than most benchmark papers do, and the error analysis in the appendix is genuinely informative. The empirical decomposition of grounding, identification, and retrieval/reasoning failure modes is a real step forward.

The headline empirical finding is oversold. The abstract attributes a 13.3/8.5 pp boost to the grounded RAG pipeline, but the ablations in Table 7b show the grounding module itself adds less than one point over image+text RAG for both GPT-5 and Qwen. The big gains come from adding retrieval, not from cropping. That framing should be fixed.

The bigger soft spot is the primary-bottleneck claim. The text-only condition is not a clean isolation of stage 3. Replacing the image with a rephrased question that names the target entity removes stages 1–2 but also changes the retrieval query and, for some questions, removes visual evidence the question actually needs. So the 20.9% error-resolution figure is not a firm estimate of the stage 1–2 contribution. An image-present control with the gold entity name inserted into the question would settle this. That said, the qualitative conclusion that retrieval/reasoning is a major bottleneck still seems plausible — the RAG ablations show large gains from explicit entity names in text retrieval, and even GT region cropping does not close the gap to text-only retrieval. The magnitude claim needs softening, not a reversal.

The Grounding-DINO evaluation is partly circular since it proposed the candidate ROIs, but human verification and the fact that other models are scored against the same boxes limit the damage. Worth a sentence in the paper, not a death blow.

Also, there are no error bars or significance tests anywhere. The differences driving the headline are small, so this matters.

Who this is for: people working on multimodal RAG, VQA diagnostics, and benchmark design. It deserves a serious referee. The revisions I'd request: fix the framing of the grounded-RAG gains, add the image-present control, and report error bars.

Referee Report

3 major / 5 minor

Summary. The paper introduces CRAG-MM-Diagnostics, a benchmark that augments CRAG-MM with stage-wise annotations — target ROIs, entity names, referring-expression types, disambiguated questions, text-only questions, and visual-complexity metadata — to decompose KI-VQA into (1) language-based visual grounding, (2) object identification, and (3) knowledge retrieval and reasoning. The authors evaluate generalized VLMs, specialized grounding models, and retrieval models, and report that the dominant bottleneck is knowledge retrieval and reasoning, with visual grounding and object identification secondary. They also propose a grounded bimodal RAG pipeline that crops the target with Grounding-DINO before image retrieval, reporting gains of 13.3 and 8.5 percentage points for GPT-5 and Qwen over a zero-shot no-RAG baseline.

Significance. If the central bottleneck claim is correct, the paper makes a valuable and actionable point: stage-aware diagnostics can redirect KI-VQA research toward retrieval and reasoning rather than only visual grounding. The benchmark itself — with human-validated annotations, public code/data linkages, and multiple new metadata dimensions — is a useful community resource. The paper is also honest in its ablations: the grounded-RAG gains over non-grounded RAG are small, and the appendix reports that grounding does not universally transfer to EVQA. However, the primary-bottleneck conclusion rests on a confounded text-only control, and the abstract overstates the grounding module's contribution relative to what the ablations show.

major comments (3)

[§6.2, metadata #4 (Section 3)] The primary-bottleneck conclusion is grounded in the text-only condition: providing the gold entity name in a text-only question resolves only 20.9% of GPT-5's errors. This condition changes three things at once: the image is removed, the question is rephrased to include the gold entity, and the query fed to parametric/retrieval knowledge is altered. It therefore does not isolate stage 3. For example, a question may require instance-specific visual attributes (color, condition, layout) that the entity name does not convey; conversely, the gold entity name may make text retrieval easier. The paper should add an image-present control where the gold entity name is inserted into the original question (e.g., 'Is the grey car, the Lexus GX 470, sold in South America?') and report the error-resolution rate for that condition. Without this control, the Section 7 lesson 3 statement — that retriev
[Abstract and Figure 7b] The abstract attributes the +13.3/+8.5 point gains to the grounded bimodal RAG pipeline and its visual grounding module. In Figure 7b, these gains are relative to the no-RAG zero-shot baseline (GPT-5: 48.9 → 62.2; Qwen: 26.2 → 34.7). The marginal contribution of the grounding module over the non-grounded bimodal RAG condition (✗✓✓ vs ✓✓✓) is only +0.7 for GPT-5 and +0.3 for Qwen. The current wording implies the grounding component is responsible for the full boost. Please report both contrasts explicitly and rephrase the abstract, the introduction, and the stage-3 takeaways accordingly.
[§A.4, §4.1, §6.1] Grounding-DINO is used to generate preliminary ROIs that human annotators verify, and the same model is then evaluated for stage-1 grounding and used as the grounding module in the proposed RAG pipeline. This creates a potential correlation between the benchmark's ROI geometry and G-DINO's output distribution, which could inflate the apparent benefit of G-DINO in the grounded RAG pipeline. The concern is reduced by human verification, but it should be quantified: report the overlap rate between G-DINO proposals and the final human-verified ROIs, or re-run the Section 6 grounded-RAG experiment with an independent off-the-shelf detector that was not used in annotation. This does not affect the retrieval/reasoning bottleneck conclusion, but it does affect the claim that the proposed grounding module is beneficial.

minor comments (5)

[Abstract] Please define the baseline for the 13.3/8.5 point gains (zero-shot no-RAG) so the reader does not infer they are due solely to the grounding module.
[Figure 3] The x-axis labels for distance-to-center bins appear garbled ('0 0.1 / 0.1 0.1 / ...'); the boundaries should be monotonic and clearly formatted.
[Figure 7b] Add a legend for the ✗/✓ matrix and explain the hyphens in cells. It is otherwise hard to tell which modules are active in each row.
[§3, footnote 7] The main text says annotations are 'manually annotated' while footnote 7 says some metadata was model-assisted. Clarify that target ROIs and entity names are human-verified rather than fully manual, and describe the verification process in the main text.
[Figure 4b] The caption does not specify whether the 'disambig.' delta is computed only on the ambiguous subset or all examples. Please state this explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the primary-bottleneck claim is an empirical ablation, and the only potential contamination (Grounding-DINO in annotation) is human-verified and cross-benchmark checked.

full rationale

The central claim that knowledge retrieval and reasoning is the primary KI-VQA bottleneck is supported by the text-only ablation (Section 6.2, Figure 7a), not by a fitted parameter. The text-only metadata (#4, Section 3) is an operationalization of 'perfect stages 1-2,' and the observed residual error is measured, not constructed; inserting the gold entity name does not by definition make the remaining error equal the stage-3 error, since the model must still retrieve and reason. A cleaner control (gold entity inserted while keeping the image) would strengthen the inference, but the absence of that control is a validity/correctness limitation, not a circular reduction. The only apparent entanglement—Grounding-DINO proposing boxes that are later used to evaluate Grounding-DINO—is broken by the stated human verification ('Bounding box annotations are obtained via... Grounding-DINO-Base... Yet, note that this annotation is only preliminary and that the human annotator should verify and potentially modify...', Appendix A.4), and the grounded-RAG gain is cross-checked on EVQA (Appendix B), where grounding sometimes hurts, showing the result is not forced. No load-bearing self-citation or uniqueness theorem is used. Score 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest chiefly on: (a) the text-only question ablation isolating retrieval/reasoning, (b) the correctness of the human annotations, (c) the retrieval corpus coverage, and (d) the LLM judge. The saliency metric and G-DINO thresholds are hand-chosen analysis parameters. No fundamentally new physical or theoretical entities are introduced; the only new artifact is the annotated dataset itself.

free parameters (3)

visual saliency weights α, β, γ = α=β=γ=1.0 (hand-set)
Eq. 8 defines saliency with weights all set to 1.0; the saliency bins used in Stage 1 and Stage 2 analyses depend on this hand choice, and no sensitivity analysis is reported.
IoU success threshold = 0.5
Grounding accuracy in Section 4 counts a prediction correct if IoU ≥ 0.5, following prior work; headline grounding scores would shift with this threshold.
Grounding-DINO thresholds = 0.4 box / 0.3 text for evaluation; 0.1 for annotation
G-DINO is used with different thresholds in annotation (Appendix A.4) and evaluation (Appendix C); thresholds affect which boxes/crops are produced and thus the grounded RAG results.

assumptions (5)

domain assumption LLM-as-a-judge (gpt-4o-mini) correctly determines QA accuracy.
All Stage 3 accuracy numbers use this judge; validity is inherited from CRAG-MM's reported 99.1% human agreement, with no new human spot-check on CRAG-MM-Diagnostics outputs.
domain assumption Human-verified ROIs and entity names are accurate ground truth.
The benchmark's stage-wise scores assume the bounding boxes and Wikipedia entity links are correct; inter-annotator agreement on referring-expression category is moderate (κ=0.453), and the ROI boxes were initially proposed by Grounding-DINO.
domain assumption CRAG-MM's image knowledge graph and web index constitute a complete retrieval corpus.
Retrieval recall and RAG gains depend on this corpus; no analysis shows whether failures are due to missing corpus entries.
domain assumption Wikipedia page views are a valid popularity proxy.
Used to split entities into popularity bins in Section 5.2; page views may not reflect visual familiarity.
domain assumption Egocentric CRAG-MM images are representative of real-world KI-VQA.
Generalizations about 'fundamental limitations' assume this curated egocentric setting is representative of broader KI-VQA.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CRAG-MM-Diagnostics: Enabling Stage-Wise Analysis of Knowledge-Intensive VQA." pith.science (2026). https://pith.science/paper/YJESQQA6

@misc{pith2026260721155,
  author       = {Pith},
  title        = {Pith review of: CRAG-MM-Diagnostics: Enabling Stage-Wise Analysis of Knowledge-Intensive VQA},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YJESQQA6}},
  note         = {Machine review of arXiv:2607.21155}
}

read the original abstract

Knowledge-Intensive Visual Question Answering (KI-VQA) benchmarks evaluate Vision-Language Models (VLMs) as multimodal knowledge assistants by requiring external information beyond a provided image to answer questions. KI-VQA involves multiple sub-problems -referring expression understanding, visual grounding, object recognition, knowledge retrieval, and reasoning-yet existing benchmarks typically report only end-task accuracy, obscuring where failures arise. To analyze the full KI-VQA pipeline, we introduce CRAG-MM-Diagnostics, a diagnostic benchmark with stage-wise data annotations that isolate 1) language-based visual grounding, 2) object identification, and 3) knowledge retrieval and reasoning. We evaluate fully parametric and retrieval-augmented VLMs, providing fine-grained analyses using newly collected metadata, such as target ROIs, entity names, and visual complexity scores. Our results point to knowledge retrieval and reasoning as the primary bottleneck, but also highlight issues in the other parts of the KI-VQA pipeline, such as the fact that VLMs struggle with target object identification or that image retrievers struggle to integrate textual cues. These findings expose fundamental limitations in current KI-VQA systems and motivate stage-aware evaluation. We, lastly, leverage these findings to propose a grounded bimodal RAG pipeline that integrates a visual grounding module to crop targets before image retrieval, boosting GPT-5 and Qwen's respective accuracies by 13.3 and 8.5 percentage points.

Figures

Figures reproduced from arXiv: 2607.21155 by the authors.

**Figure 1.** Illustrative summary of the stage-wise evaluation enabled by CRAG-MMDiagnostics, thanks to new types of metadata we add. The three stages evaluated are (1) language-based visual grounding, (2) object identification, and (3) knowledge extraction & reasoning, and are further detailed in the figure. for these assistants is the ability to interpret visual scenes and provide informative responses, typically evaluated t… view at source ↗

**Figure 2.** Grounding accuracy per referring expression type (horizontal lines: mean; hatched: disambiguated query). It shows that: (1) Drops on non-salient and ambiguous targets reveal high sensitivity to expression type. (2) Gains from disambiguation suggest many failures arise from query ambiguity rather than visual processing deficits. 4 Stage 1 Language-based Visual Grounding We use CRAG-MM-Diagnostics to systematically e… view at source ↗

**Figure 3.** Visual grounding scores along four dimensions of visual complexity (black squares indicate model mean). These plots show that low target saliency is the primary bottleneck, acutely compounding performance drops caused by scene density, off-center placement, and small target size. 4.2 Analyses and Findings [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (12 more)

**Figure 4.** Figure 4: Performance breakdown of generalized models on the stage 2 object identification task. (b) Accuracy changes under input manipulations: ∆ ‘cropping’ marks the accuracy change after cropping the target object; ∆ ‘question’ marks the change after simplifying the question…

**Figure 5.** Figure 5: Stage 2 object identification results for specialized models. (b) Accuracy change under input manipulations: ∆ ‘cropping’ marks the change after cropping the target object; ∆ ‘text’ adds the question for the multimodal models. (c) Recall@10 across visual saliency bins.…

**Figure 6.** Figure 6: Grounded bimodal RAG: In stage 3 , we study a modeling pipeline that combines VLMs with RAG and visual grounding. Adding the grounding module is meant to make the image retriever more effective. an analysis that isolates it from previous stages using ‘text-only query’…

**Figure 7.** Figure 7: Comparison of (a) generalized models and (b) RAG pipeline ablations on KIVQA accuracy. All reports performance over all instances; Subset evaluates ambiguous (ambig.) vs. disambiguated (dis.) queries. Columns ground., img ret., and txt ret. denote the activation of g…

**Figure 8.** Figure 8: CRAG-MM-Diagnostics statistics based on scene crowdedness, the target object’s distance to center, the target’s size, and the referring expression types. interval grouping and using similar subgroup sizes for each metric across the dataset, with each subgroup comprisin…

**Figure 9.** Figure 9: Human annotation interface. Human Annotation Procedure and Tool To refine and verify the preliminary annotations as well as to generate additional metadata, we employ an annotation tool, as illustrated in [PITH_FULL_IMAGE:figures/full_fig_p026_9.png]

**Figure 10.** Figure 10: Prompt used for Visual Grounding Inference. Experimental Setup. We implemented both generalized VLMs and specialized zero-shot object detection models, including Grounding-DINO and OWL-ViT. The infrastructure is built on Python 3.10 using PyTorch and the Hugging Face …

**Figure 11.** Figure 11: Prompt used for Object Identification Inference Experimental Setup. We follow the same experimental setup as stage 1, visual grounding task overall (see Appendix C). Aside of that, we further elaborate different specialized models in this stage. These specialized mod…

**Figure 12.** Figure 12: Generalized model (VLM) object identification accuracy with different saliency interval. D.2 Specialized Models (Image Retriever) In this section, we evaluate the performance of various image retrieval models across different input modalities [PITH_FULL_IMAGE:figures…

**Figure 14.** Figure 14: Specialized image retriever object identification results with different entity popularity interval. For comparison, all retrievers utilize image only input and ground truth target region cropped version. Black rectangle means model mean [PITH_FULL_IMAGE:figures/fu…

**Figure 15.** Figure 15: Prompt used for Knowledge Extraction Inference Experimental Setup. We maintain an experimental configuration consistent with the stage 1 visual grounding and stage 2 object identification tasks (see Sections C and D). For the additional text retrieval component, we e…

**Figure 16.** Figure 16: Prompt used for LLM-as-a-judge [PITH_FULL_IMAGE:figures/full_fig_p036_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 1 canonical work pages

[1]

In: Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV)

Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D.: VQA: Visual question answering. In: Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV). p. 2425–2433. ICCV ’15, IEEE Computer Society, USA (2015).https://doi.org/10.1109/ICCV.2015.279,https://doi.org/ 10.1109/ICCV.2015.2793

arXiv 2015
[2]

arXiv preprint (2025),https://arxiv.org/ abs/2502.139231, 6, 7

Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., Lin, J.: Qwen2.5-vl technical report. arXiv preprint (2025),https://arxiv.org/ abs/2502.139231, 6, 7

arXiv 2025
[3]

Bhattacharya, N., Li, Q., Gurari, D.: Why does a visual question have different answers? In: Proceedings of the IEEE/CVF International Conference on Com- puter Vision (ICCV) (October 2019),https://openaccess.thecvf.com/content_ ICCV_2019/papers/Bhattacharya_Why_Does_a_Visual_Question_Have_Different_ Answers_ICCV_2019_paper.pdf22

2019
[4]

(eds.) Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing

Chen, Y., Hu, H., Luan, Y., Sun, H., Changpinyo, S., Ritter, A., Chang, M.W.: Can pre-trained vision and language models answer visual information-seeking ques- tions? In: Bouamor, H., Pino, J., Bali, K. (eds.) Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing. pp. 14948–14968. Association for Computational Linguisti...

doi:10.18653/v1/2023.emnlp-main.925 2023
[5]

trychroma.com/31

Chroma: Chroma: The AI-native open-source embedding database.,https://www. trychroma.com/31
[6]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Deitke, M., Clark, C., Lee, S., Tripathi, R., Yang, Y., Park, J.S., Salehi, M., Muen- nighoff,N.,Lo,K.,Soldaini,L.,Lu,J.,Anderson,T.,Bransom,E.,Ehsani,K.,Ngo, H., Chen, Y., Patel, A., Yatskar, M., Callison-Burch, C., Head, A., Hendrix, R., Bastani, F., VanderBilt, E., Lambert, N., Chou, Y., Chheda, A., Sparks, J., Skjons- berg, S., Schmitz, M., Sarnat, A....

2025
[7]

In: Larson, K

Ding, Y., Ren, K., Huang, J., Luo, S., Han, S.C.: MMVQA: A comprehensive dataset for investigating multipage multimodal information retrieval in pdf-based visual question answering. In: Larson, K. (ed.) Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24. pp. 6243–6251. International Joint Conferences on Art...

work page doi:10.24963/ijcai.2024/690 2024
[8]

In: Proceedings of the IEEE/CVF Conference on Computer CRAG-MM-Diagnostics17 Vision and Pattern Recognition (CVPR)

Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., Yacoob, Y., Manocha, D., Zhou, T.: Hallusionbench: An advanced diagnos- tic suite for entangled language hallucination and visual illusion in large vision- language models. In: Proceedings of the IEEE/CVF Conference on Computer CRAG-MM-Diagnostics17 Vision and Pattern Rec...

2024

Show all 59 references

[9]

5281/zenodo.121230323

Honnibal, M., Montani, I., Van Landeghem, S., Boyd, A.: spaCy: Industrial- strength Natural Language Processing in Python (2020).https://doi.org/10. 5281/zenodo.121230323

2020
[10]

In: Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV)

Hu, H., Luan, Y., Chen, Y., Khandelwal, U., Joshi, M., Lee, K., Toutanova, K., Chang, M.W.: Open-domain visual entity recognition: Towards recognizing mil- lions of wikipedia entities. In: Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV). pp. 12...

2023
[11]

In: Yue, Y., Garg, A., Peng, N., Sha, F., Yu, R

Hu, W., Gu, J.C., Dou, Z.Y., Fayyaz, M., Lu, P., Chang, K.W., Peng, N.V.: MRAG-Bench: Vision-centric evaluation for retrieval-augmented multi- modal models. In: Yue, Y., Garg, A., Peng, N., Sha, F., Yu, R. (eds.) In- ternational Conference on Learning Representations. vol. 202...

2025
[12]

In: Al-Onaizan, Y., Bansal, M., Chen, Y.N

Jian, P., Yu, D., Zhang, J.: Large language models know what is key visual en- tity: An LLM-assisted multimodal retrieval for VQA. In: Al-Onaizan, Y., Bansal, M., Chen, Y.N. (eds.) Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing. pp. 10...

2024
[13]

arXiv preprint (2024),https://api

Kabir, R., Haque, N., Islam, M.S., et al.: A comprehensive survey on visual ques- tion answering datasets and algorithms. arXiv preprint (2024),https://api. semanticscholar.org/CorpusID:2741318622

2024
[14]

arXiv preprint (2026),https://arxiv.org/abs/2601.1906012

Kim, J., Tao, R., Sharma, S., Wang, J., Sun, K., Lin, Z., Moon, S., Mathias, L., Ku- mar, A., Ji, H., et al.: Pixel-grounded retrieval for knowledgeable large multimodal models. arXiv preprint (2026),https://arxiv.org/abs/2601.1906012

2026
[15]

Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C.H., Gonzalez, J., Zhang, H., Stoica, I.: Efficient memory management for large language model serving with PagedAttention. In: Proceedings of the 29th Symposium on Operating Systems Principles.p.611–626.SOSP’23,Associat...

2023
[16]

In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

Lerner, P., Ferret, O., Guinaudeau, C., Le Borgne, H., Besançon, R., Moreno, J.G., Lovón Melgarejo, J.: ViQuAE, a dataset for knowledge-based visual ques- tion answering about named entities. In: Proceedings of the 45th International ACM SIGIR Conference on Research and Develo...

2022
[17]

arXiv preprint (2026), https://arxiv.org/abs/2601.047209, 33 18 H

Li, M., Zhang, Y., Long, D., Chen, K., Song, S., Bai, S., Yang, Z., Xie, P., Yang, A., Liu, D., et al.: Qwen3-VL-Embedding and Qwen3-VL-Reranker: A unified frame- work for state-of-the-art multimodal retrieval and ranking. arXiv preprint (2026), https://arxiv.org/abs/2601.0472...

2026 arXiv
[18]

In: The Thirteenth International Conference on Learning Representations (2025), https://openreview.net/forum?id=VvDEuyVXkG3, 22

Li, Y., Li, Y., Wang, X., Jiang, Y., Zhang, Z., Zheng, X., Wang, H., Zheng, H.T., Huang, F., Zhou, J., Yu, P.S.: Benchmarking multimodal retrieval aug- mented generation with dynamic VQA dataset and self-adaptive planning agent. In: The Thirteenth International Conference on L...

2025
[19]

In: Oh, A.H., Agarwal, A., Belgrave, D., Cho, K

Lin, L., Xie, Y., Chen, D., Xu, Y., Zhu, C., Yuan, L.: REVIVE: Regional vi- sual representation matters in knowledge-based visual question answering. In: Oh, A.H., Agarwal, A., Belgrave, D., Cho, K. (eds.) Advances in Neural Information Processing Systems (2022),https://openre...

2022
[20]

In: Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part XLVII

Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q., Li, C., Yang, J., Su, H., Zhu, J., Zhang, L.: Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. In: Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, Se...

2024 doi
[21]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR)

Ma, T., Bai, B., Lin, H., Wang, H., Wang, Y., Luo, L., Fang, L.: When visual grounding meets gigapixel-level large-scale scenes: Benchmark and approach. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR). pp. 22119–22128 (June 2024),...

2024
[22]

arXiv preprint (2025),https://arxiv.org/abs/2503

Ma, X., Ding, Z., Luo, Z., Chen, C., Guo, Z., Wong, D.F., Feng, X., Sun, M.: Deep- Perception:Advancingr1-likecognitivevisualperceptioninMLLMsforknowledge- intensive visual grounding. arXiv preprint (2025),https://arxiv.org/abs/2503. 127974

2025
[23]

In: Rogers, A., Boyd-Graber, J., Okazaki, N

Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., Hajishirzi, H.: When not to trust language models: Investigating effectiveness of parametric and non- parametric memories. In: Rogers, A., Boyd-Graber, J., Okazaki, N. (eds.) Pro- ceedings of the 61st Annual Meeting of th...

2023 doi
[24]

Marino, K., Rastegari, M., Farhadi, A., Mottaghi, R.: OK-VQA: A visual ques- tion answering benchmark requiring external knowledge. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019),https://openaccess.thecvf.com/content_CV...

2019
[25]

Transactions on Machine Learning Research (2026),https://openreview.net/ forum?id=TpU38jbKIJ9, 33

Meng, R., Jiang, Z., Liu, Y., Su, M., Yang, X., Fu, Y., Qin, C., Thirukovalluru, R., Zhang, X., Chen, Z., Xu, R., Xiong, C., Zhou, Y., Chen, W., Yavuz, S.: VLM2vec- v2: Advancing multimodal embedding for videos, images, and visual documents. Transactions on Machine Learning Re...

2026
[26]

In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

Mensink, T., Uijlings, J., Castrejon, L., Goel, A., Cadar, F., Zhou, H., Sha, F., Araujo, A., Ferrari, V.: Encyclopedic VQA: Visual questions about detailed prop- erties of fine-grained categories. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (IC...

2023
[27]

Meta, A.: Llama 3.2: Revolutionizing edge AI and vision with open, customiz- able models (2024),https://ai.meta.com/blog/llama-3-2-connect-2024-vision- edge-mobile-devices/6, 23

2024
[28]

In: Computer Vi- sion – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part X

Minderer, M., Gritsenko, A., Stone, A., Neumann, M., Weissenborn, D., Dosovit- skiy, A., Mahendran, A., Arnab, A., Dehghani, M., Shen, Z., Wang, X., Zhai, X., Kipf, T., Houlsby, N.: Simple open-vocabulary object detection. In: Computer Vi- sion – ECCV 2022: 17th European Confe...

2022 doi
[29]

arXiv preprint (2025),https://arxiv.org/abs/2510.12801 1, 2

Narayan, K., Xu, Y., Cao, T., Nerella, K., Patel, V.M., Shiee, N., Grasch, P., Jia, C., Yang, Y., Gan, Z.: DeepMMSearch-R1: Empowering multimodal llms in multimodal web search. arXiv preprint (2025),https://arxiv.org/abs/2510.12801 1, 2

2025
[30]

In: The Thir- teenth International Conference on Learning Representations (2025),https:// openreview.net/forum?id=v9CDpLpjiE22

Ni, M., Fan, Y., Zhang, L., Zuo, W.: Visual-o1: Understanding ambiguous in- structions via multi-modal multi-turn chain-of-thoughts reasoning. In: The Thir- teenth International Conference on Learning Representations (2025),https:// openreview.net/forum?id=v9CDpLpjiE22

2025
[31]

In: Proceedings of the 33rd ACM International Conference on Multimedia

Qiang,C.,Wei,Z.,Han,X.,Wang,Z.,Li,S.,Lan,X.,Jiao,J.,Han,Z.:VER-Bench: Evaluating MLLMs on reasoning with fine-grained visual evidence. In: Proceedings of the 33rd ACM International Conference on Multimedia. p. 12698–12705. MM ’25, Association for Computing Machinery, New York,...

2025
[32]

In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024),https://openreview.net/forum?id=qLnXPVvwLx4, 22

Qiao, Y., Duan, H., Fang, X., Yang, J., Chen, L., Zhang, S., Wang, J., Lin, D., Chen, K.: Prism: A framework for decoupling and assessing the capabilities of VLMs. In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024),https://openreview.net/fo...

2024
[33]

In: Al-Onaizan, Y., Bansal, M., Chen, Y.N

Qiu, J., Madotto, A., Lin, Z., Crook, P.A., Xu, Y.E., Damavandi, B., Dong, X.L., Faloutsos,C.,Li,L.,Moon,S.:SnapNTell:Enhancingentity-centricvisualquestion answering with retrieval augmented multimodal LLM. In: Al-Onaizan, Y., Bansal, M., Chen, Y.N. (eds.) Findings of the Asso...

2024 doi
[34]

In: Meila, M., Zhang, T

Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th Int...
[35]

Redmon,J.,Divvala,S.,Girshick,R.,Farhadi,A.:Youonlylookonce:Unified,real- time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2016),https://www.cv-foundation.org/ openaccess/content_cvpr_2016/papers/Redmon_You_O...

2016
[36]

In: Computer Vi- 20 H

Schwenk, D., Khandelwal, A., Clark, C., Marino, K., Mottaghi, R.: A-OKVQA: A benchmark for visual question answering using world knowledge. In: Computer Vi- 20 H. Oh et al. sion – ECCV 2022: 17th European Conference, Proceedings, Part VIII. p. 146–162. Springer-Verlag, Berlin,...

2022 doi
[37]

arXiv preprint (2015),https:// arxiv.org/abs/1503.039579

Singh, P., Dhawan, S., Agarwal, S., Thakur, N.: Implementation of an efficient fuzzy logic based information retrieval system. arXiv preprint (2015),https:// arxiv.org/abs/1503.039579

2015 arXiv
[38]

In: Rogers, A., Boyd-Graber, J., Okazaki, N

Stengel-Eskin, E., Guallar-Blasco, J., Zhou, Y., Van Durme, B.: Why did the chicken cross the road? Rephrasing and analyzing ambiguous questions in VQA. In: Rogers, A., Boyd-Graber, J., Okazaki, N. (eds.) Proceedings of the 61st Annual Meeting of the Association for Computatio...

2023 doi
[39]

In: Proceedings of the IEEE/CVF International Conference on Computer Vi- sion (ICCV)

Tran, D.T., Tran, T.K., Hauswirth, M., Le Phuoc, D.: ReasonVQA: A multi- hop reasoning benchmark with structural knowledge for visual question answer- ing. In: Proceedings of the IEEE/CVF International Conference on Computer Vi- sion (ICCV). pp. 18793–18803 (October 2025),http...

2025
[40]

arXiv preprint (2025),https://arxiv.org/ abs/2510.261601, 2, 3, 4, 9, 12, 13, 22, 28

Wang, J., Yang, X., Sun, K., Suresh, P., Sharma, S., Czyzewski, A., Andersen, D., Appini, S., Banerjee, A., Choudhary, S., et al.: CRAG-MM: Multi-modal multi- turn comprehensive rag benchmark. arXiv preprint (2025),https://arxiv.org/ abs/2510.261601, 2, 3, 4, 9, 12, 13, 22, 28

2025
[41]

In: Proceedings of the 26th International Joint Conference on Artificial Intelligence

Wang,P.,Wu,Q.,Shen,C.,Dick,A.,VanDenHenge,A.:Explicitknowledge-based reasoning for visual question answering. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence. p. 1290–1296. IJCAI’17, AAAI Press (2017),https://dl.acm.org/doi/10.5555/3171642...

2017
[42]

arXiv preprint (2025),https: //arxiv.org/abs/2503.064924, 22, 27, 29

Wang, Y., Zhao, Y., Chen, X., Guo, S., Liu, L., Li, H., Xiao, Y., Zhang, J., Li, Q., Xu, K.: VisualSimpleQA: A benchmark for decoupled evaluation of large vision- language models in fact-seeking question answering. arXiv preprint (2025),https: //arxiv.org/abs/2503.064924, 22, 27, 29

2025 arXiv
[43]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Wu, P., Xie, S.: V?: Guided visual search as a core mechanism in multimodal LLMs. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 13084–13094 (June 2024),https://openaccess. thecvf.com/content/CVPR2024/papers/Wu_V_Guided_Visual...

2024
[44]

Xiao, L., Yang, X., Lan, X., Wang, Y., Xu, C.: Toward visual grounding: A survey. IEEE Transactions on Pattern Analysis & Machine Intelligence48(03), 2749–2771 (Mar 2026).https://doi.org/10.1109/TPAMI.2025.3630635,https: //doi.ieeecomputersociety.org/10.1109/TPAMI.2025.36306356

2026
[45]

In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval

Xiao, S., Liu, Z., Zhang, P., Muennighoff, N., Lian, D., Nie, J.Y.: C-pack: Packed resources for general Chinese embeddings. In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. p. 641–649. SIGIR ’24, Association ...

2024
[46]

In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

Xu, Y., Zhu, L., Yang, Y.: MC-Bench: A benchmark for multi-context visual grounding in the era of MLLMs. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 17675–17687 (October 2025),https: //openaccess.thecvf.com/content/ICCV2025/papers/Xu...

2025
[47]

In: Findings of the Association for Computational Linguistics: EMNLP 2024

Yan, Y., Xie, W.: Echosight: Advancing visual-language models with wiki knowl- edge. In: Findings of the Association for Computational Linguistics: EMNLP 2024. pp. 1538–1551 (2024),https://aclanthology.org/2024.findings-emnlp.83.pdf 29

2024
[48]

In: Proceedings of the 41st International Conference on Machine Learning

Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., Wang, L.: MM-Vet: evaluating large multimodal models for integrated capabilities. In: Proceedings of the 41st International Conference on Machine Learning. ICML’24, JMLR.org (2024),https://dl.acm.org/doi/10.5555/3...

2024
[49]

id" "675653e6-4477-4865-b4e1-3f2db72f891d

Zhang, Y., Zhang, H., Tian, H., Fu, C., Zhang, S., Wu, J., Li, F., Wang, K., Wen, Q., Zhang, Z., Wang, L., Jin, R.: MME-realworld: Could your multimodal LLM challenge high-resolution real-world scenarios that are difficult for humans? In: The Thirteenth International Conferenc...

2025
[51]

landmark

annotating the target region of interest (ROI) for each multimodal input (i.e., query and image), 2) mapping the target object to its corresponding Wikipedia entity, including the associated Wikipedia URL; and 3) checking whether the question is ambiguous or lacks sufficient c...

2025
[52]

If the prediction indicates uncertainty or refusal to answer, output json {'accuracy': False}
[53]

If the prediction exactly matches the ground truth, output json {'accuracy': True}
[54]

If the ground truth is a number 3.1 If the prediction gives a number that almost exactly matches the ground truth, output json {'accuracy': True} 3.2 If the prediction gives a number that is not the same as the ground truth, output json {'accuracy': False}
[55]

If the prediction is self-contradictory, output json {'accuracy': False}
[56]

If the prediction is not answering the question, output json {'accuracy': False}
[57]

If ground truth contains a set of objects, 6.1 if the prediction contains exactly same objects as the ground truth, output json {'accuracy': True} 6.2 if the prediction contains different objects from the ground truth, output json {'accuracy': False} 6.3 if the prediction is a...
[58]

If the prediction is grounded by the ground truth, output json {'accuracy': True}
[59]

answer: True

If the prediction is unrelated or contradictory to the ground truth, output json {'accuracy': False} ## Additional Guidelines - Take it as granted that the ground truth is always correct. - If the prediction gives extra information that is not in the ground truth, it is still ...

2025
[2021]

Proceedings of Machine Learning Research, vol. 139, pp. 8748–8763. PMLR (2021),http://proceedings.mlr.press/v139/radford21a.html9, 12, 33

2021

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [1]

In: Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV)

Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D.: VQA: Visual question answering. In: Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV). p. 2425–2433. ICCV ’15, IEEE Computer Society, USA (2015).https://doi.org/10.1109/ICCV.2015.279,https://doi.org/ 10.1109/ICCV.2015.2793

arXiv 2015

[2] [2]

arXiv preprint (2025),https://arxiv.org/ abs/2502.139231, 6, 7

Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., Lin, J.: Qwen2.5-vl technical report. arXiv preprint (2025),https://arxiv.org/ abs/2502.139231, 6, 7

arXiv 2025

[3] [3]

Bhattacharya, N., Li, Q., Gurari, D.: Why does a visual question have different answers? In: Proceedings of the IEEE/CVF International Conference on Com- puter Vision (ICCV) (October 2019),https://openaccess.thecvf.com/content_ ICCV_2019/papers/Bhattacharya_Why_Does_a_Visual_Question_Have_Different_ Answers_ICCV_2019_paper.pdf22

2019

[4] [4]

(eds.) Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing

Chen, Y., Hu, H., Luan, Y., Sun, H., Changpinyo, S., Ritter, A., Chang, M.W.: Can pre-trained vision and language models answer visual information-seeking ques- tions? In: Bouamor, H., Pino, J., Bali, K. (eds.) Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing. pp. 14948–14968. Association for Computational Linguisti...

doi:10.18653/v1/2023.emnlp-main.925 2023

[5] [5]

trychroma.com/31

Chroma: Chroma: The AI-native open-source embedding database.,https://www. trychroma.com/31

[6] [6]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Deitke, M., Clark, C., Lee, S., Tripathi, R., Yang, Y., Park, J.S., Salehi, M., Muen- nighoff,N.,Lo,K.,Soldaini,L.,Lu,J.,Anderson,T.,Bransom,E.,Ehsani,K.,Ngo, H., Chen, Y., Patel, A., Yatskar, M., Callison-Burch, C., Head, A., Hendrix, R., Bastani, F., VanderBilt, E., Lambert, N., Chou, Y., Chheda, A., Sparks, J., Skjons- berg, S., Schmitz, M., Sarnat, A....

2025

[7] [7]

In: Larson, K

Ding, Y., Ren, K., Huang, J., Luo, S., Han, S.C.: MMVQA: A comprehensive dataset for investigating multipage multimodal information retrieval in pdf-based visual question answering. In: Larson, K. (ed.) Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24. pp. 6243–6251. International Joint Conferences on Art...

work page doi:10.24963/ijcai.2024/690 2024

[8] [8]

In: Proceedings of the IEEE/CVF Conference on Computer CRAG-MM-Diagnostics17 Vision and Pattern Recognition (CVPR)

Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., Yacoob, Y., Manocha, D., Zhou, T.: Hallusionbench: An advanced diagnos- tic suite for entangled language hallucination and visual illusion in large vision- language models. In: Proceedings of the IEEE/CVF Conference on Computer CRAG-MM-Diagnostics17 Vision and Pattern Rec...

2024