Multimodal Graph RAG for Long-range Visually Rich Document Understanding

Chu-Song Chen; Yi-Cheng Wang

arxiv: 2606.28780 · v1 · pith:Y3WTEY7Jnew · submitted 2026-06-27 · 💻 cs.IR · cs.CV

Multimodal Graph RAG for Long-range Visually Rich Document Understanding

Yi-Cheng Wang , Chu-Song Chen This is my paper

Pith reviewed 2026-06-30 09:05 UTC · model grok-4.3

classification 💻 cs.IR cs.CV

keywords multimodal graph RAGdocument-level VQAknowledge graphvisually rich documentsDLVQA benchmarkmultimodal knowledge graphlong document understanding

0 comments

The pith

Multimodal graph RAG builds knowledge graphs from visually rich documents to support holistic long-range VQA.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a multimodal graph-based retrieval-augmented generation method to handle visual question answering on long documents where standard multimodal large language models fail due to context limits. Page-level retrieval approaches like MMRAG address length but fall short on questions needing the full document context. The solution constructs multimodal knowledge graphs that integrate visual and textual elements to summarize global document knowledge. A new DLVQA benchmark supplies reference summaries and supporting facts to test document-level comprehension. Experiments demonstrate better results than existing MMRAG and knowledge-graph baselines on multi-hop QA/VQA tasks and the new benchmark.

Core claim

The central claim is that extending LLM-based knowledge graph construction to multimodal data from visually rich documents produces a graph RAG system that outperforms page-retrieval and text-only graph methods on tasks requiring holistic document understanding.

What carries the argument

The multimodal knowledge graph (MMKG) that summarizes global document knowledge by linking visual and textual information across pages, retrieved within a graph-based RAG pipeline.

Load-bearing premise

LLM-based methods for building knowledge graphs can be extended to multimodal data from visually rich documents while preserving the ability to support holistic document-level VQA.

What would settle it

A controlled test on DLVQA-style questions where the multimodal graph RAG system yields lower accuracy than MMRAG baselines on holistic queries would show the central claim does not hold.

Figures

Figures reproduced from arXiv: 2606.28780 by Chu-Song Chen, Yi-Cheng Wang.

**Figure 1.** Figure 1: (a) MMRAG is effective for local VQA where the answer appears on a page. (b) Text-only Graph-based RAG supports query-focused summarization across dispersed content. (c) Our KG4VD integrates text, figures, and layout into a unified MMKG that supports long-range VQA over documents. Rot (Hong et al., 2025). Recent MMRAG methods (Zhang et al., 2025; Faysse et al., 2025; Yu et al., 2025) mitigate this issue… view at source ↗

**Figure 2.** Figure 2: Overall pipeline of KG4VD. KG4VD constructs a reusable multimodal knowledge graph (MMKG) for visually rich documents and uses it to support query-time retrieval and answer generation. Numbered circles indicate the major processing steps in the pipeline. 2024) builds text KGs and organizes them with hierarchical community detection; LightRAG (Guo et al., 2025b) improves scalability through direct entity-re… view at source ↗

**Figure 3.** Figure 3: Representative pages from the four DLVQA [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

**Figure 4.** Figure 4: Example reflector output used to guide the next extraction round. [PITH_FULL_IMAGE:figures/full_fig_p028_4.png] view at source ↗

**Figure 5.** Figure 5: Entity and relation counts of graphs constructed by GraphRAG, RAGAnything, MegaRAG, and KG4VD [PITH_FULL_IMAGE:figures/full_fig_p029_5.png] view at source ↗

**Figure 6.** Figure 6: Add and revise operations across adaptive extraction rounds on four DLVQA subsets. Add operations [PITH_FULL_IMAGE:figures/full_fig_p031_6.png] view at source ↗

**Figure 7.** Figure 7: Per-page extraction-round distribution across [PITH_FULL_IMAGE:figures/full_fig_p032_7.png] view at source ↗

**Figure 8.** Figure 8: Qualitative example of adaptive extraction across rounds. The top row shows annotated page images, [PITH_FULL_IMAGE:figures/full_fig_p033_8.png] view at source ↗

**Figure 9.** Figure 9: Build-time prompt for adaptive page-level entity and relation extraction. [PITH_FULL_IMAGE:figures/full_fig_p036_9.png] view at source ↗

**Figure 10.** Figure 10: Build-time prompt for reflection, coverage diagnosis, masking guidance, and stop-signal prediction. [PITH_FULL_IMAGE:figures/full_fig_p037_10.png] view at source ↗

**Figure 11.** Figure 11: Build-time prompt for cross-page connection before canonicalization. [PITH_FULL_IMAGE:figures/full_fig_p038_11.png] view at source ↗

**Figure 12.** Figure 12: Query-time prompt for selecting the query-adaptive PPR expansion mode. [PITH_FULL_IMAGE:figures/full_fig_p039_12.png] view at source ↗

**Figure 13.** Figure 13: Query-time prompt for organizing retrieved entities and relations into a preliminary answer. [PITH_FULL_IMAGE:figures/full_fig_p039_13.png] view at source ↗

**Figure 14.** Figure 14: Query-time prompt for combining the preliminary answer with retrieved page images. [PITH_FULL_IMAGE:figures/full_fig_p040_14.png] view at source ↗

read the original abstract

Multimodal large language models (MLLMs) are widely applied to visual document understanding. However, comprehending long documents remains an issue by the limited context window. Though recent multimodal retrieval-augmented generation (MMRAG) can address this challenge by retrieving relevant pages. It still struggles with the visual question answering (VQA) requiring holistic comprehension of a document. To cope with this, knowledge graph (KG) that summarizes global knowledge of a document can provide an effective solution. However, most existing LLM-based KG construction methods handle only the language modality, leaving the automatic creation of multimodal KGs (MMKGs) for visually rich documents largely unexplored. In this paper, we introduce a multimodal graph-based RAG approach to tackle this problem. Existing LLM-based KG methods evaluate the QA performance relying on indirect evidence such as comprehensiveness, diversity, empowerment, and so on. The lack of annotated datasets for comprehensive document-level VQA poses a significant challenge to effective model evaluation. To overcome this limitation, we also introduce a new benchmark, DLVQA (document-level VQA), which provides reference summaries and corresponding supporting facts for global document-level questions. Experimental results show that our approach outperforms existing MMRAG or KG-based approaches on multi-hop QA/VQA benchmarks and DLVQA.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper extends graph RAG to multimodal document data and ships a new DLVQA benchmark with reference summaries; the central outperformance claim looks intact once the full experiments are in view.

read the letter

The two things to know are that this work builds a multimodal knowledge graph RAG pipeline aimed at holistic document VQA and introduces DLVQA, a benchmark that supplies reference summaries plus supporting facts for global questions.

What is new is the adaptation of LLM-driven KG construction to handle visual elements and layout in long documents, an area the abstract correctly flags as mostly unexplored. The benchmark itself is also presented as fresh, with explicit reference material that lets evaluators check whether answers actually draw on the whole document rather than isolated pages. That combination directly targets the context-window problem in MLLMs when page-level retrieval alone is not enough.

The paper does a clean job of stating the limitation of existing MMRAG methods on multi-hop and global questions. The stress-test note indicates the full manuscript shows no load-bearing gaps in the argument, no obvious circularity, and no unstated assumptions that would invalidate the reported gains over baselines. That is worth noting because many RAG papers stop at indirect metrics.

The soft spots are modest. The abstract itself gives no numbers, dataset sizes, or ablation tables, so any referee will still need to verify effect sizes and baseline strength in the results section. The precise schema used for visual entities and relations in the MMKG could also use extra detail to make the construction reproducible. Neither issue looks fatal from the available evidence.

This is for people working on retrieval-augmented document understanding and long-context VQA. A reader who needs a concrete benchmark with reference facts or wants to try a graph layer on top of MLLMs will find usable material here.

I would send it for peer review. The problem is practical, the extension is straightforward, and the new benchmark has clear utility even if later work refines the graph construction.

Referee Report

2 major / 2 minor

Summary. The paper introduces a multimodal graph-based RAG framework for long-range understanding of visually rich documents. It argues that standard MMRAG struggles with holistic VQA requiring global document comprehension and that existing LLM-based KG construction is text-only, leaving automatic MMKG creation for multimodal documents unexplored. The authors propose a multimodal graph RAG method, introduce the DLVQA benchmark providing reference summaries and supporting facts for document-level questions, and report that their approach outperforms MMRAG and KG baselines on multi-hop QA/VQA tasks and DLVQA.

Significance. If the empirical gains hold under rigorous controls, the work would provide a concrete path to extend graph-augmented retrieval to multimodal document settings, addressing context-window limits while preserving global structure. The DLVQA benchmark itself would be a useful contribution for evaluating holistic VQA, independent of the specific method.

major comments (2)

[§4] §4 (Experimental Setup): the abstract and introduction claim outperformance on multi-hop QA/VQA benchmarks and DLVQA, yet the provided text supplies no dataset sizes, number of documents, error bars, or statistical significance tests. Without these, it is impossible to assess whether the reported gains are load-bearing or within noise.
[§3.2] §3.2 (MMKG Construction): the central claim rests on successful automatic construction of multimodal KGs from visually rich documents. The manuscript must explicitly define the node/edge schema for visual elements (e.g., how figures, tables, and layout are encoded) and the prompting strategy used; absence of these details makes the extension from text-only KG methods unverifiable.

minor comments (2)

[Abstract] The abstract states that existing methods rely on 'indirect evidence such as comprehensiveness, diversity, empowerment'; a citation or short definition of these metrics would improve clarity.
[Figures/Tables] Figure captions and table headers should explicitly state whether results are averaged over multiple runs or single-shot.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address the major comments point by point below and will revise the manuscript to incorporate the requested details.

read point-by-point responses

Referee: [§4] §4 (Experimental Setup): the abstract and introduction claim outperformance on multi-hop QA/VQA benchmarks and DLVQA, yet the provided text supplies no dataset sizes, number of documents, error bars, or statistical significance tests. Without these, it is impossible to assess whether the reported gains are load-bearing or within noise.

Authors: We agree that the reviewed version omitted key experimental details. In the revision we will add: the exact number of documents and questions per benchmark (including DLVQA), the number of documents used for MMKG construction, error bars from multiple runs where applicable, and statistical significance tests (e.g., paired t-tests or Wilcoxon tests with p-values) comparing our method against baselines. These additions will be placed in §4 and the corresponding result tables. revision: yes
Referee: [§3.2] §3.2 (MMKG Construction): the central claim rests on successful automatic construction of multimodal KGs from visually rich documents. The manuscript must explicitly define the node/edge schema for visual elements (e.g., how figures, tables, and layout are encoded) and the prompting strategy used; absence of these details makes the extension from text-only KG methods unverifiable.

Authors: We acknowledge the need for explicit specification. The revised §3.2 will include: (1) the full node/edge schema, defining node types for text blocks, figures, tables, and layout regions together with their attributes (e.g., bounding-box coordinates, visual type, OCR content); (2) edge types capturing spatial, semantic, and referential relations between visual and textual elements; and (3) the complete prompting templates and few-shot examples used to instruct the LLM for MMKG extraction. These additions will make the construction process fully reproducible. revision: yes

Circularity Check

0 steps flagged

No significant circularity identified

full rationale

The paper introduces a multimodal graph RAG method and a new DLVQA benchmark, with central claims resting on empirical outperformance versus MMRAG and KG baselines. No equations, fitted parameters, predictions, or derivation chains are described in the abstract or reader summary. No self-citations are invoked as load-bearing premises, and the evaluation relies on external benchmarks rather than self-referential definitions or renamings. The derivation chain is therefore self-contained against external data.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 1 invented entities

Abstract-only review; no explicit free parameters, axioms, or invented entities are detailed beyond the high-level proposal of multimodal KGs.

invented entities (1)

multimodal knowledge graph (MMKG) no independent evidence
purpose: to summarize global knowledge of a visually rich document for holistic VQA
Introduced to overcome limitations of page-level retrieval in long documents.

pith-pipeline@v0.9.1-grok · 5761 in / 974 out tokens · 25561 ms · 2026-06-30T09:05:22.311922+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

19 extracted references · 4 canonical work pages · 2 internal anchors

[1]

Multi-hop question answering under temporal knowledge editing. InCOLM. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Mar- cel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic c...

work page internal anchor Pith review Pith/arXiv arXiv 2025
[2]

Colpali: Efficient document retrieval with vision language models. InICLR. Hanning Gao, Lingfei Wu, Po Hu, Zhihua Wei, Fangli Xu, and Bo Long. 2022. Graph-augmented learning to rank for querying large-scale knowledge graph. In AACL. Zirui Guo, Xubin Ren, Lingrui Xu, Jiahao Zhang, and Chao Huang. 2025a. Rag-anything: All-in-one rag framework.arXiv preprint...

work page arXiv 2022
[3]

Think-on-graph 2.0: Deep and faithful large language model reasoning with knowledge-guided retrieval augmented generation. InICLR. Xueguang Ma, Sheng-Chieh Lin, Minghan Li, Wenhu Chen, and Jimmy Lin. 2024a. Unifying multimodal retrieval via document screenshot embedding. In EMNLP. Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyu...

2023
[4]

hdbscan: Hierarchical density based clustering. J. Open Source Softw. Hatem Mousselly-Sergieh, Teresa Botschen, Iryna Gurevych, and Stefan Roth. 2018. A multimodal translation-based approach for knowledge graph rep- resentation learning. InSEM. OpenAI. 2025. Gpt-5 system card. Technical report, OpenAI. Tyler Thomas Procko and Omar Ochoa. 2024. Graph retri...

work page arXiv 2018
[5]

Taming the untamed: Graph-based knowledge retrieval and reasoning for mllms to conquer the un- known. InICCV. Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen
[6]

Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V

Uniir: Training and benchmarking universal multimodal information retrievers. InECCV. Ruobing Xie, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2017. Image-embodied knowledge representa- tion learning. InIJCAI. Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chun- yuan Li, and Jianfeng Gao. 2023. Set-of-mark prompting unleashes extraordinary visual grounding ...

work page internal anchor Pith review Pith/arXiv arXiv 2017
[9]

Current page graph G_i^(t): {current_graph}
[10]

Propose graph edits Delta_i^(t) for the current page

Previous reflector feedback B_i^(t), if available: {reflector_feedback} TASK. Propose graph edits Delta_i^(t) for the current page. On the initial round, extract candidate entities and relations from the annotated page image and component manifest. On later rounds, use the current page graph and reflector feedback to revise the previous extraction. Allowe...
[11]

Cite only component IDs that appear in the component manifest
[12]

Ground each node or edge by citing source layout components.,→

Do not generate free-form coordinates. Ground each node or edge by citing source layout components.,→
[13]

Use add operations for new entities or relations, replace operations to correct attributes, descriptions, or grounding, and delete operations to remove unsupported or duplicate items.,→
[14]

(a) Page-level extractorExt

Avoid duplicating entities or relations that are already present in the current page graph unless the operation explicitly replaces them.,→ OUTPUT a JSON object {"ops": [...]}, no markdown. (a) Page-level extractorExt. Figure 9: Build-time prompt for adaptive page-level entity and relation extraction. 36 KG4VD Page-level Reflector Prompt You are a reflect...
[15]

Annotated page image I_hat_i^(t)
[16]

Component manifest C_i: {component_manifest}
[17]

per_component

Updated page graph G_i^(t+1): {updated_page_graph} TASK. Produce a component-level coverage report. For each component, classify whether it is covered, partially_covered, or uncovered. Identify unresolved regions, missing entities, missing relations, duplicate items, unsupported items, and nodes or edges that should be revised in the next extractor round....
[18]

Compare types, descriptions, local context, and visual crops when available.,→

Do not decide from entity names alone. Compare types, descriptions, local context, and visual crops when available.,→
[19]

Different surface forms can still refer to the same entity when the descriptions and visual evidence support alignment.,→
[20]

If A and B are related but not identical, do not mark them as same_entity
[21]

verdict": ...,

If the evidence is insufficient, choose unrelated or related_not_same with low confidence rather than forcing a same-entity link.,→ OUTPUT JSON: {"verdict": ..., "rationale": ..., "confidence": 0.0--1.0} (a) Cross-page entity judge. Figure 11: Build-time prompt for cross-page connection before canonicalization. 38 KG4VD Query Analyzer Prompt Classify the ...

[1] [1]

Multi-hop question answering under temporal knowledge editing. InCOLM. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Mar- cel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic c...

work page internal anchor Pith review Pith/arXiv arXiv 2025

[2] [2]

Colpali: Efficient document retrieval with vision language models. InICLR. Hanning Gao, Lingfei Wu, Po Hu, Zhihua Wei, Fangli Xu, and Bo Long. 2022. Graph-augmented learning to rank for querying large-scale knowledge graph. In AACL. Zirui Guo, Xubin Ren, Lingrui Xu, Jiahao Zhang, and Chao Huang. 2025a. Rag-anything: All-in-one rag framework.arXiv preprint...

work page arXiv 2022

[3] [3]

Think-on-graph 2.0: Deep and faithful large language model reasoning with knowledge-guided retrieval augmented generation. InICLR. Xueguang Ma, Sheng-Chieh Lin, Minghan Li, Wenhu Chen, and Jimmy Lin. 2024a. Unifying multimodal retrieval via document screenshot embedding. In EMNLP. Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyu...

2023

[4] [4]

hdbscan: Hierarchical density based clustering. J. Open Source Softw. Hatem Mousselly-Sergieh, Teresa Botschen, Iryna Gurevych, and Stefan Roth. 2018. A multimodal translation-based approach for knowledge graph rep- resentation learning. InSEM. OpenAI. 2025. Gpt-5 system card. Technical report, OpenAI. Tyler Thomas Procko and Omar Ochoa. 2024. Graph retri...

work page arXiv 2018

[5] [5]

Taming the untamed: Graph-based knowledge retrieval and reasoning for mllms to conquer the un- known. InICCV. Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen

[6] [6]

Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V

Uniir: Training and benchmarking universal multimodal information retrievers. InECCV. Ruobing Xie, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2017. Image-embodied knowledge representa- tion learning. InIJCAI. Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chun- yuan Li, and Jianfeng Gao. 2023. Set-of-mark prompting unleashes extraordinary visual grounding ...

work page internal anchor Pith review Pith/arXiv arXiv 2017

[7] [9]

Current page graph G_i^(t): {current_graph}

[8] [10]

Propose graph edits Delta_i^(t) for the current page

Previous reflector feedback B_i^(t), if available: {reflector_feedback} TASK. Propose graph edits Delta_i^(t) for the current page. On the initial round, extract candidate entities and relations from the annotated page image and component manifest. On later rounds, use the current page graph and reflector feedback to revise the previous extraction. Allowe...

[9] [11]

Cite only component IDs that appear in the component manifest

[10] [12]

Ground each node or edge by citing source layout components.,→

Do not generate free-form coordinates. Ground each node or edge by citing source layout components.,→

[11] [13]

Use add operations for new entities or relations, replace operations to correct attributes, descriptions, or grounding, and delete operations to remove unsupported or duplicate items.,→

[12] [14]

(a) Page-level extractorExt

Avoid duplicating entities or relations that are already present in the current page graph unless the operation explicitly replaces them.,→ OUTPUT a JSON object {"ops": [...]}, no markdown. (a) Page-level extractorExt. Figure 9: Build-time prompt for adaptive page-level entity and relation extraction. 36 KG4VD Page-level Reflector Prompt You are a reflect...

[13] [15]

Annotated page image I_hat_i^(t)

[14] [16]

Component manifest C_i: {component_manifest}

[15] [17]

per_component

Updated page graph G_i^(t+1): {updated_page_graph} TASK. Produce a component-level coverage report. For each component, classify whether it is covered, partially_covered, or uncovered. Identify unresolved regions, missing entities, missing relations, duplicate items, unsupported items, and nodes or edges that should be revised in the next extractor round....

[16] [18]

Compare types, descriptions, local context, and visual crops when available.,→

Do not decide from entity names alone. Compare types, descriptions, local context, and visual crops when available.,→

[17] [19]

Different surface forms can still refer to the same entity when the descriptions and visual evidence support alignment.,→

[18] [20]

If A and B are related but not identical, do not mark them as same_entity

[19] [21]

verdict": ...,

If the evidence is insufficient, choose unrelated or related_not_same with low confidence rather than forcing a same-entity link.,→ OUTPUT JSON: {"verdict": ..., "rationale": ..., "confidence": 0.0--1.0} (a) Cross-page entity judge. Figure 11: Build-time prompt for cross-page connection before canonicalization. 38 KG4VD Query Analyzer Prompt Classify the ...