Pith. sign in

REVIEW 3 major objections 5 minor 12 cited by

MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphs

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper's central claim is that turning images into scene-graph entities and linking them to a textual knowledge graph with spectral clustering makes multimodal document QA more accurate and more robust against unanswerable questions.

desk verdict Solid engineering with a useful new benchmark, but the hallucination-suppression claims outrun the current evidence. read the letter →

arxiv 2507.20804 v3 pith:7CNBEQIU submitted 2025-07-28 cs.AI

Xueyao Wan , Hang Yu This is my paper
classification cs.AI
keywords MultimodalknowledgegraphRAGCross-modalentitylinkingSceneSpectralclusteringDocumentquestionansweringHallucinationmitigationRetrieval-augmentedgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MMGraphRAG sets out to prove that the right way to bring images into retrieval-augmented generation is not captions, shared embedding spaces, or joint extraction, but explicit graphs: turn each image into a scene graph, build a knowledge graph from the text, and fuse the two by linking visual entities to textual entities. The fusion is carried out by SpecLink, a candidate-generation method that runs spectral clustering on a matrix encoding both embedding similarity and graph-relation weight, then lets an LLM pick the best match. On the DocBench document-QA benchmark the full system reaches 76.8% overall accuracy (88.7% on multimodal questions), beating NaiveRAG and GraphRAG, and on MMLongBench it reaches 38.8% accuracy. The paper also reports a sharp gain in recognizing unanswerable questions (35.1% vs. 5.8% for M3DocRAG), which it reads as evidence that explicit graph structure suppresses hallucination. If these results hold, structured cross-modal entity linking rather than semantic flattening becomes the central design choice for multimodal RAG.

What carries the argument

MMGraphRAG's load-bearing object is the multimodal knowledge graph itself: an image is converted by Img2Graph into a scene graph (YOLO segmentation, MLLM descriptions, entity and relation extraction, plus a global image entity), and a text-side KG is built from document chunks. The fusion mechanism is SpecLink, which builds a weighted adjacency matrix $A_{pq} = \mathrm{sim}(\mathbf{v}_p, \mathbf{v}_q) \cdot \mathrm{weight}(r_{pq})$, forms a graph Laplacian, takes the smallest $m$ eigenvectors, clusters the resulting row space with DBSCAN, selects the cluster closest to each visual entity, and hands the candidates to an LLM for final alignment. SpecLink's job is to keep candidate sets accurate so the final alignment and downstream retrieval operate on clean cross-modal links rather than noisy embeddings.

What would settle it

Independently re-annotate a random sample of CMEL instances with human annotators who see only the original document, not the LLM-generated labels, then re-run SpecLink and DBSCAN on those labels; if the reported roughly 15-point micro-accuracy lead shrinks to statistical noise, the claimed advantage of spectral candidate generation would not survive.

Watch

Extended reading notes

Core claim

The paper's central claim is that a node-based multimodal knowledge graph—one in which visual content is represented as first-class entities and relations, not as attributes or captions—can be constructed zero-shot from raw documents and then used for structure-aware retrieval and generation. It claims that the key bottleneck, cross-modal entity linking, is best handled by SpecLink: spectral clustering over a weighted adjacency matrix whose entries multiply embedding cosine similarity by an LLM-assessed relation weight, followed by LLM-based alignment within the selected cluster. It further claims this design outperforms text-only RAG, image-captioning RAG, and shared-embedding multimodal RAG on DocBench and MMLongBench, and that it is markedly better at declining to answer unanswerable questions, which the authors attribute to the explicit reasoning paths in the MMKG.

Load-bearing premise

The load-bearing premise is that the CMEL gold alignments are correct: they are generated by the authors' own LLM entity-fusion pipeline, with only 20% of the data manually inspected, and every SpecLink-vs-baseline comparison is scored against those labels.

Editorial extensions

If this is right

  • Multimodal RAG no longer needs to flatten images into captions or vectors; images can contribute entities and relations that participate in retrieval directly.
  • Systems built this way should be able to answer questions that require joint evidence from an image and a passage, because the retrieved context includes explicit cross-modal reasoning paths.
  • Unanswerable questions can be handled more safely: the graph exposes missing or inconsistent evidence, which is what the reported 35.1% vs. 5.8% unanswerable accuracy points to.
  • The same MMKG can be indexed once and reused for many queries, making the approach scalable relative to query-time graph construction.
  • The CMEL benchmark gives entity-linking researchers a real-document testbed with 1,114 alignment instances across news, academia, and novels.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported advantage of spectral clustering over simpler clustering baselines likely depends on the quality of the MLLM-generated scene graphs and entity descriptions; if those descriptions are noisy, the spectral structure may not add much beyond embedding similarity.
  • Because the CMEL gold labels were themselves generated by an LLM fusion pipeline, the relative ranking of SpecLink and its baselines should be re-checked against independently human-annotated labels before treating the 51.8% micro-accuracy as a stable property of the method.
  • The same node-based MMKG construction could be extended to other modalities—audio, video, tables—since the framework only requires a scene-graph-like structure and a cross-modal linker; that extension is not tested in the paper.
  • A testable implication is that SpecLink's benefit should grow with graph density: on documents where entities are densely related, the relation-weight term matters more, while on sparse pages it should converge to plain embedding clustering.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes MMGraphRAG, a framework that constructs a multimodal knowledge graph by converting text into a text-based KG and images into scene-graph-based image KGs, then linking entities across modalities with a proposed method, SpecLink, which combines spectral clustering with LLM-based candidate selection and alignment. The paper also introduces the CMEL dataset for fine-grained cross-modal entity linking and reports experiments on CMEL, DocBench, and MMLongBench, claiming state-of-the-art multimodal document QA accuracy and improved robustness in identifying unanswerable questions.

Significance. If the claims are supported, the framework is a useful contribution: it extends GraphRAG-style reasoning to images with explicit visual nodes, provides interpretable reasoning paths, and introduces a new benchmark for cross-modal entity linking. The paper also includes a human-LLM judge consistency study (96.0% and 94.6% agreement with Cohen's kappa around 0.88) and reports a paired bootstrap test between SpecLink and the strongest CMEL baseline (p<0.01). These are strengths. However, the evaluation's load-bearing comparisons currently rest on internally reimplemented baselines, and the new benchmark's gold labels were generated with a pipeline that resembles the method being evaluated. These issues need to be addressed before the headline claims of 'superior performance and stronger robustness' can be accepted.

major comments (3)
  1. [Section A.2, Steps 4-5, and Table 5] The CMEL ground-truth alignments are generated by an LLM prompt ('Prompt for Entity Fusion') that merges image and text entities, and only 20% of the data is manually verified. Since SpecLink also uses an LLM-based final alignment step in Section 3.3.1 with a similar prompting style, the CMEL evaluation is potentially self-referential: the method and the label-generation process may share a common LLM bias, which could inflate both the reported absolute accuracy and the gap over embedding-only baselines. Please provide independent human annotation of the full CMEL test set or evaluate on an existing external benchmark such as MATE before using CMEL to support the SpecLink contribution.
  2. [Section 4.2.3 and Tables 2-3, 7-8] The comparisons against NaiveRAG and GraphRAG use internally reimplemented versions (e.g., top-10 chunks, GraphRAG with community detection removed), and no published DocBench or MMLongBench state-of-the-art numbers are reported. The abstract's 'state-of-the-art' claim and Section 4.2's 'superior performance' claim are therefore not externally anchored; a reader cannot determine whether the gains reflect the method or the particular reimplementation choices. Please compare against official published baselines or report unmodified reference implementations, and clearly state that DocBench metadata questions are excluded from the reported overall accuracy, so the 76.8% figure may not be comparable to official DocBench scores that include metadata.
  3. [Section 4.3.2 and Table 4 with Table 8] The sixfold unanswerable-question improvement (35.1% vs 5.8%) is computed against one internally evaluated M3DocRAG configuration using 7B backbone models, not necessarily the official M3DocRAG system, so the gap may overstate MMGraphRAG's robustness. Moreover, the complete MMLongBench results in Table 8 show text-only GraphRAG rows with Unanswerable accuracy between 77.1% and 86.5%, while every MMGraphRAG row is between 51.6% and 63.2%. The paper does not reconcile this with the claim that explicit MMKG reasoning suppresses hallucination; without an error analysis distinguishing false refusals from hallucinated answers, and without a faithful official M3DocRAG reproduction, the robustness claim is not supported.
minor comments (5)
  1. [Section 3.3.1, Eq. (1)] When no relation exists between two entities, the paper sets weight(r_pq)=1, making A_pq a plain cosine similarity and the adjacency matrix dense. This weakens the graph-structure signal in the spectral clustering; please clarify how the KG topology is still exploited and how this differs from clustering the embedding space directly.
  2. [Section 4.2.2 and Figure 5] The human-LLM consistency study samples only 50 DocBench and 25 MMLongBench documents and evaluates only the best configuration; please state the total number of human judgments and, if feasible, verify the judge against multiple methods to ensure the same judge does not systematically favor one family of outputs.
  3. [Figure 6] Please add axis labels and numeric values to Figure 6; the text cites gains such as +4.9%, +21.5%, -9.7%, and +18.2%, but the figure alone does not clearly convey these quantities.
  4. [References] References [14] and [15] are the same GraphRAG paper, and [27] and [28] are the same Lewis et al. paper; please unify the duplicate entries.
  5. [Tables 2 and 3] The abbreviations in the table captions (L, Q, Qv, Iv, MGR, NR, GR) are introduced in the text but not restated in the captions; a short legend in each caption would improve readability.

Circularity Check

1 steps flagged · score 4.0 of 10

CMEL benchmark validation is partly self-referential because its gold alignments are LLM-generated by the same fusion/alignment operation SpecLink uses; external DocBench/MMLongBench results remain independent.

  1. self definitional [Section 4.1.1, Appendix A.2 Steps 4-5, and Section 3.3.1]
    ""Gold alignments are constructed through a semi-automatic entity fusion pipeline followed by manual verification." ... "In this step, LLM is used to generate the final alignment results, i.e., the alignment between image entities and text entities." ... "we almost do not distinguish between the differences of fusion and alignment." ... SpecLink: "Finally, we perform entity alignment using LLM-based inference.""

    The CMEL gold standard is not an independently established set of cross-modal alignments: Step 4 of the construction pipeline uses an LLM to generate the alignment labels, and Step 5 manually checks only 20%. The evaluated method's final stage is itself 'entity alignment using LLM-based inference' with an analogous fusion/alignment objective, and the authors state they 'almost do not distinguish between the differences of fusion and alignment.' Thus CMEL accuracy measures, for the majority of unverified instances, how often SpecLink's LLM alignment agrees with the label-generating LLM's fusion output, rather than agreement with an external ground truth.

full rationale

No derivation in the paper reduces to its inputs by equation, and there is no load-bearing self-citation chain: the authors do not invoke a prior 'uniqueness theorem' and the DocBench/MMLongBench comparisons use external benchmarks with an LLM-judge protocol that is checked against human labels. The one genuine circularity concern is the CMEL benchmark itself: the gold alignments are produced by the authors' LLM-based entity fusion pipeline, and SpecLink's final alignment is also LLM-based, with the paper explicitly conflating fusion and alignment. Since only 20% of labels receive manual inspection, the majority of CMEL accuracy scores measure LLM self-agreement rather than external correctness. This makes the CMEL validation of SpecLink partially self-referential, though it does not undermine the central external QA claims; the internal M3DocRAG reimplementation and absence of published SOTA numbers are fairness or reproducibility risks, not circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The ledger shows the method rests on four modeling assumptions and five hand or LLM-chosen parameters. None of these are fitted to external targets in a classical sense, but the LLM-produced relation weights and the CMEL dataset assumptions are load-bearing for the main claims.

free parameters (5)
  • spectral clustering eigenvector count m
    The number of smallest eigenvectors used to form Q depends on the number of textual entities and is not specified in detail (Section 3.3.1).
  • DBSCAN parameters (eps, min_samples)
    Clustering on the row space of Q uses DBSCAN, but its density parameters are not reported (Section 3.3.1).
  • Relation importance weight weight(r_pq)
    Each edge weight in the adjacency matrix is assigned by LLM judgment; this is an ad hoc, per-edge scalar that strongly shapes the spectral clustering (Equation 1).
  • Retrieval top-k and token limits = 10 entities, 4000 tokens, 10 chunks
    Chosen by hand for GraphRAG and MMGraphRAG retrieval (Section 4.2.3).
  • Naive fusion cosine threshold = 0.7
    Selected by grid search on a held-out development split of the academia subset (Section 4.3.1).
assumptions (4)
  • domain assumption MLLM-based scene graph extraction (YOLO segmentation plus InternVL/Qwen descriptions) captures sufficiently complete and accurate visual entities and relations.
    The entire Img2Graph pipeline (Section 3.2) assumes the MLLM outputs are reliable enough to build the image KG.
  • domain assumption LLM-based text KG extraction produces entities and relations that preserve the document's content for QA.
    Text2Graph (Section 3.1) relies on LLM extraction; the paper does not validate extraction quality beyond downstream accuracy.
  • domain assumption The CMEL dataset gold alignments, generated by a semi-automatic LLM pipeline and 20% human verification, are correct.
    Section A.2 uses LLM entity fusion plus manual inspection of 20% of data; the remaining 80% is unverified.
  • standard math Spectral clustering on a semantic plus structural adjacency matrix yields clusters that contain the correct cross-modal matches.
    The method follows standard spectral clustering theory (Section 3.3.1), but the effectiveness depends on the engineered A matrix.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphs." pith.science (2026). https://pith.science/paper/7CNBEQIU

@misc{pith2026250720804,
  author       = {Pith},
  title        = {Pith review of: MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7CNBEQIU}},
  note         = {Machine review of arXiv:2507.20804}
}
read the original abstract

Large Language Models (LLMs) suffer from hallucinations due to their static parametric knowledge. Retrieval-Augmented Generation (RAG) and GraphRAG mitigate this issue by incorporating external knowledge and structured reasoning over knowledge graphs (KGs). However, existing approaches remain largely text-centric, as constructing fine-grained multimodal knowledge graphs (MMKGs) with explicit cross-modal semantics remains challenging. In this paper, we propose MMGraphRAG, a framework for building interpretable MMKGs that unify textual and visual knowledge. Our approach represents visual content as structured scene graphs and integrates them with textual KGs through a novel cross-modal entity linking method, SpecLink, which leverages spectral clustering to jointly model semantic similarity and graph structure. This design preserves explicit entities, relations, and reasoning paths across modalities, enabling structure-aware retrieval and generation. To support evaluation, we introduce the CMEL dataset, a benchmark for fine-grained cross-modal entity alignment. Experimental results on CMEL demonstrate improved entity linking accuracy, while evaluations on DocBench and MMLongBench show that MMGraphRAG achieves superior performance and stronger robustness, particularly in complex multimodal reasoning scenarios.

Figures

Figures reproduced from arXiv: 2507.20804 by the authors.

Figure 1
Figure 1. Comparison of Image-Text Fusion Methods. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. MMGraphRAG Framework Overview. The framework begins by parsing sources like novels and webpages, creating [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. An Example of the Img2Graph Module in Action The Img2Graph module maps images into KGs through a five￾step pipeline. First, semantic segmentation is performed using YOLO [55] to divide the image into semantically independent re￾gions, referred to as image feature blocks. Second, MLLMs generate textual descriptions for each feature block. Third, entities and their relations are extracted from the image. Fourth, MLLMs… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Entity Distribution Across Document Domains. The figure illustrates how entity types and quantities vary among [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Confusion matrices comparing human majority [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Impact of Cross-Modal Fusion Module by Metric [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: The entire dataset is constructed based on the number of [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: The Dursleys’ photo wall. From Chapter 1 of "Harry [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. DualG-MRAG: Decoupling Macro-Reasoning and Micro-Matching for Multimodal Retrieval-Augmented Generation

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Decoupling macro topological routing from micro visual matching, plus query-driven GNN path decoding, improves multimodal multi-hop retrieval and QA over strong MM-RAG baselines.

  2. DeCoRAG: Cognitive Decoupling and Semantic-Aware Cropping for Complex Document Understanding

    cs.IR 2026-07 conditional novelty 6.0 of 10

    Decoupling semantic anchoring from high-res cropping fixes VLM visual attention sinks in document Graph RAG, gaining up to 12.5 SPR points and 40.8% fewer prompt tokens.

  3. Constrained Dominant Sets for Multimodal Document Question Answering

    cs.IR 2026-06 unverdicted novelty 6.0 of 10

    Constrained Dominant Sets on query-augmented graphs select complementary evidence for long multimodal document QA, claiming new SOTA of 66.99 on VisDoMBench and gains of 37.1 and 4.8 points over baselines.

  4. FLOWREADER: Min-Cost Flow Optimization for Multi-Modal Long Document Q&A

    cs.IR 2026-06 unverdicted novelty 6.0 of 10

    FLOWREADER reframes multimodal long-document evidence assembly as min-cost flow optimization and reports competitive results on VisDoMBench subsets with fragmented evidence.

  5. Task-Adaptive Retrieval over Agentic Multi-Modal Web Histories via Learned Graph Memory

    cs.IR 2026-04 unverdicted novelty 6.0 of 10

    ACGM learns task-adaptive sparse graphs over multi-modal agent histories via policy-gradient optimization, reaching 82.7 nDCG@10 and 89.2% Precision@10 on WebShop, VisualWebArena, and Mind2Web while outperforming 19 b...

  6. MG$^2$-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation

    cs.IR 2026-04 unverdicted novelty 6.0 of 10

    MG²-RAG proposes a multi-granularity graph RAG framework that constructs hierarchical multimodal nodes via entity-driven visual grounding and performs structured retrieval, delivering SOTA results on four multimodal t...

  7. MG$^2$-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation

    cs.IR 2026-04 conditional novelty 6.0 of 10

    A lightweight hierarchical multimodal graph RAG that fuses entity-grounded visual objects with text nodes and propagates relevance via multi-granularity PPR, delivering SOTA multimodal task performance at far lower co...

  8. KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question Answering

    cs.CV 2026-01 conditional novelty 6.0 of 10

    KG-ViP answers visual questions by merging an image scene graph with a commonsense knowledge graph, reporting 7.8-11.3 point LLM-J gains over prior retrieval baselines.

  9. KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question Answering

    cs.CV 2026-01 unverdicted novelty 6.0 of 10

    KG-ViP fuses scene graphs and commonsense graphs via a query-based retrieval-and-fusion pipeline to improve multi-modal LLM performance on visual question answering.

  10. Structured and Abstractive Reasoning on Multi-modal Relational Knowledge Images

    cs.CV 2025-10 unverdicted novelty 6.0 of 10

    Authors build a synthetic data generator and two-stage training pipeline for structured abstractive reasoning on multi-modal relational knowledge images, releasing STAR-64K and showing 3B/7B models outperforming GPT-4o.

  11. HVM-GraphRAG: Holistic-View Multimodal Graph Retrieval-Augmented Generation on Complex Document

    cs.IR 2026-07 conditional novelty 5.0 of 10

    A holistic-view multimodal GraphRAG that conflict-resolves concept indices and retrieves via compact concept anchors plus modality-grouped evidence, beating strong baselines on three complex-document QA sets.

  12. Towards integrated sensors for optimized OCT with undetected photons

    quant-ph 2025-08 unverdicted novelty 5.0 of 10

    The authors show that an induced-coherence scheme outperforms the standard SU(1,1) scheme for integrated OCT sensors with undetected photons, achieving 28 micrometer axial resolution.

Reference graph

Works this paper leans on

85 extracted references · 36 canonical work pages · cited by 10 Pith papers

  1. [1]

    Tabrizi, Azadeh Shakery, Masoud Asadpour, Maziar Abbasi, and Mo- hammad Ali Tavallaie

    Shayan A. Tabrizi, Azadeh Shakery, Masoud Asadpour, Maziar Abbasi, and Mo- hammad Ali Tavallaie. 2013. Personalized PageRank Clustering: A graph cluster- ing algorithm based on random walks.Physica A: Statistical Mechanics and its Applications392, 22 (2013), 5772–5785. doi:10.1016/j.physa.2013.07.021

    work page doi:10.1016/j.physa.2013.07.021 2013

  2. [2]

    Meta AI. 2024. LLaMA-3.1-70B-Instruct. Available at https://huggingface.co/meta- llama/Llama-3.1-70B-Instruct

    2024

  3. [3]

    Mistral AI. 2024. Mistral-Large-Instruct-2411. Available at https://huggingface. co/mistralai/Mistral-Large-Instruct-2411

    2024

  4. [4]

    Stability AI. 2024. stella-en-1.5B-v5. https://huggingface.co/stabilityai/stella-en- 1.5B-v5. Open-weight English language model

    2024

  5. [5]

    Iñigo Alonso, Gorka Azkune, Ander Salaberria, Jeremy Barnes, and Oier Lopez de Lacalle. 2025. Vision-Language Models Struggle to Align Entities across Modalities. arXiv:2503.03854 [cs.CL] https://arxiv.org/abs/2503.03854

    arXiv 2025

  6. [6]

    Chenyang Bu, Guojie Chang, Zihao Chen, Cunyuan Dang, Zhize Wu, Yi He, and Xindong Wu. 2025. Query-Driven Multimodal GraphRAG: Dynamic Local Knowl- edge Graph Construction for Online Reasoning. InFindings of the Association for Computational Linguistics: ACL 2025. 21360–21380

    2025

  7. [7]

    Guikun Chen, Jin Li, and Wenguan Wang. 2024. Scene Graph Genera- tion with Role-Playing Large Language Models. InAdvances in Neural In- formation Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37. Curran Associates, Inc., 9 Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Xueyao Wa...

    2024

  8. [8]

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation.arXiv preprint arXiv:2402.03216 (2024)

    arXiv 2024

Show all 85 references
  1. [9]

    Yixiang Chen, Penglei Sun, Xiang Li, and Xiaowen Chu. 2025. MRD-RAG: En- hancing Medical Diagnosis with Multi-Round Retrieval-Augmented Generation. arXiv:2504.07724 [cs.CL] https://arxiv.org/abs/2504.07724

    2025 arXiv

  2. [10]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al . 2024. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on compu...

    2024

  3. [11]

    Jaemin Cho, Debanjan Mahata, Ozan Irsoy, Yujie He, and Mohit Bansal. 2024. M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi- document Understanding.CoRRabs/2411.04952 (2024). https://doi.org/10.48550/ arXiv.2411.04952

    2024 doi arXiv

  4. [12]

    Yun-Wei Chu, Kai Zhang, Christopher Malon, and Martin Renqiang Min. 2025. Reducing Hallucinations of Medical Multimodal Large Language Models with Visual Retrieval-Augmented Generation. InWorkshop on Large Language Models and Generative AI for Health at AAAI 2025. https://open...

    2025

  5. [13]

    Dingsheng Deng. 2020. DBSCAN Clustering Algorithm Based on Density. In 2020 7th International Forum on Electrical Engineering and Automation (IFEEA). 949–953. doi:10.1109/IFEEA51475.2020.00199

    2020

  6. [15]

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. 2025. From Local to Global: A Graph RAG Approach to Query-Focused Summarization. arXiv:2404.16130 [cs.CL] https://arxiv....

    2025 arXiv

  7. [16]

    Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, CELINE HUDELOT, and Pierre Colombo. 2025. ColPali: Efficient Document Retrieval with Vision Language Models. InThe Thirteenth International Conference on Learning Representations. https://openreview.net/forum...

    2025

  8. [17]

    Jingru Gan, Jinchang Luo, Haiwei Wang, Shuhui Wang, Wei He, and Qingming Huang. 2021. Multimodal Entity Linking: A New Dataset and A Baseline. In Proceedings of the 29th ACM International Conference on Multimedia(Virtual Event, China)(MM ’21). Association for Computing Machine...

    2021

  9. [18]

    Gongde Guo, Hui Wang, David Bell, Yaxin Bi, and Kieran Greer. 2003. KNN model-based approach in classification. InOn The Move to Meaningful Internet Systems 2003: CoopIS, DOA, and ODBASE: OTM Confederated International Confer- ences, CoopIS, DOA, and ODBASE 2003, Catania, Sici...

    2003

  10. [19]

    Zirui Guo, Lianghao Xia, Yanhua Yu, Tu Ao, and Chao Huang. 2024. Lightrag: Simple and fast retrieval-augmented generation.arXiv preprint arXiv:2410.05779 (2024)

    2024 arXiv

  11. [20]

    Mahd Hindi, Linda Mohammed, Ommama Maaz, and Abdulmalik Alwarafy. 2025. Enhancing the Precision and Interpretability of Retrieval-Augmented Generation (RAG) in Legal Technology: A Survey.IEEE Access13 (2025), 46171–46189. doi:10.1109/ACCESS.2025.3550145

    2025

  12. [21]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions.ACM Trans. Inf. Sy...

    2025 doi

  13. [22]

    Hongjie Jia, Shifei Ding, Xinzheng Xu, and Ru Nie. 2014. The latest research progress on spectral clustering.Neural Computing and Applications24 (2014), 1477–1486

    2014

  14. [23]

    Glenn Jocher, Ayush Chaurasia, and Jing Qiu. 2023. Ultralytics YOLO v8. Available at https://github.com/ultralytics/ultralytics

    2023

  15. [24]

    Sarasvady

    Kamran Khan, Saif Ur Rehman, Kamran Aziz, Simon Fong, and S. Sarasvady

  16. [25]

    Trupti M Kodinariya, Prashant R Makwana, et al. 2013. Review on determining number of Cluster in K-Means Clustering.International Journal1, 6 (2013), 90–95

    2013

  17. [26]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Princip...

    2023

  18. [27]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing s...

    2020

  19. [28]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. InAdvances in N...

    2020

  20. [29]

    Zhuoqun Li, Xuanang Chen, Haiyang Yu, Hongyu Lin, Yaojie Lu, Qiaoyu Tang, Fei Huang, Xianpei Han, Le Sun, and Yongbin Li. 2025. StructRAG: Boosting Knowledge Intensive Reasoning of LLMs via Inference-time Hybrid Information Structurization. InThe Thirteenth International Confe...

    2025

  21. [30]

    Wanying Liang, Pasquale De Meo, Yong Tang, and Jia Zhu. 2024. A Survey of Multi-modal Knowledge Graphs: Technologies and Trends.ACM Comput. Surv. 56, 11, Article 273 (June 2024), 41 pages. doi:10.1145/3656579

    2024 doi

  22. [31]

    Aristidis Likas, Nikos Vlassis, and Jakob J. Verbeek. 2003. The global k-means clustering algorithm.Pattern Recognition36, 2 (2003), 451–461. doi:10.1016/S0031- 3203(02)00060-2 Biometrics

    2003 doi

  23. [32]

    Weizhe Lin, Jinghong Chen, Jingbiao Mei, Alexandru Coca, and Bill Byrne. 2023. Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answering. InAdvances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko...

    2023

  24. [33]

    Xin Lin, Changxing Ding, Jinquan Zeng, and Dacheng Tao. 2020. GPS-Net: Graph Property Sensing Network for Scene Graph Generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2020

  25. [34]

    Zihan Ling, Zhiyao Guo, Yixuan Huang, Yi An, Shuai Xiao, Jinsong Lan, Xi- aoyong Zhu, and Bo Zheng. 2025. MMKB-RAG: A Multi-Modal Knowledge- Based Retrieval-Augmented Generation Framework. arXiv:2504.10074 [cs.AI] https://arxiv.org/abs/2504.10074

    2025 arXiv

  26. [35]

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. 2024. A Survey on Hallucination in Large Vision-Language Models.CoRRabs/2402.00253 (2024). https://doi.org/10.48550/ arXiv.2402.00253

    2024 doi arXiv

  27. [36]

    Junming Liu, Siyuan Meng, Yanting Gao, Song Mao, Pinlong Cai, Guohang Yan, Yirong Chen, Zilin Bian, Botian Shi, and Ding Wang. 2025. Aligning Vision to Language: Text-Free Multimodal Knowledge Graph Construction for Enhanced LLMs Reasoning. arXiv:2503.12972 [cs.CV] https://arx...

    2025

  28. [37]

    Pei Liu, Xin Liu, Ruoyu Yao, Junming Liu, Siyuan Meng, Ding Wang, and Jun Ma. 2025. HM-RAG: Hierarchical Multi-Agent Multimodal Retrieval Augmented Generation. arXiv:2504.12330 [cs.CL] https://arxiv.org/abs/2504.12330

    2025 arXiv

  29. [38]

    Qi Liu, Yongyi He, Tong Xu, Defu Lian, Che Liu, Zhi Zheng, and Enhong Chen

  30. [39]

    Xukai Liu, Ye Liu, Kai Zhang, Kehang Wang, Qi Liu, and Enhong Chen. 2024. OneNet: A Fine-Tuning Free Framework for Few-Shot Entity Linking via Large Language Model Prompting.arXiv preprint arXiv:2410.07549(2024)

    2024 arXiv

  31. [40]

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255(2023)

    2023 arXiv

  32. [41]

    Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. 2024. Ovis: Structural embedding alignment for multimodal large language model.arXiv preprint arXiv:2405.20797(2024)

    2024 arXiv

  33. [42]

    Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, et al. 2025. Mmlongbench-doc: Benchmarking long-context document understanding with visualizations.Advances in Neural Information Processing Systems37 (2025), 95963–96010

    2025

  34. [43]

    Lang Mei, Siyu Mo, Zhihan Yang, and Chong Chen. 2025. A Survey of Multimodal Retrieval-Augmented Generation. arXiv:2504.08748 [cs.IR] https://arxiv.org/abs/ 2504.08748

    2025 arXiv

  35. [44]

    Boci Peng, Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, and Siliang Tang. 2024. Graph retrieval-augmented generation: A survey. arXiv preprint arXiv:2408.08921(2024)

    2024 arXiv

  36. [45]

    Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2020. Recipes for building an open-domain chatbot.CoRRabs/2004.13637 (2020). https://arxiv.org/abs/2004.13637

    2020 arXiv

  37. [46]

    Wei Shen, Yuhan Li, Yinan Liu, Jiawei Han, Jianyong Wang, and Xiaojie Yuan

  38. [47]

    Wei Shen, Jianyong Wang, and Jiawei Han. 2015. Entity Linking with a Knowledge Base: Issues, Techniques, and Solutions.IEEE Transactions on Knowledge and Data Engineering27, 2 (2015), 443–460. doi:10.1109/TKDE.2014.2327028

    2015

  39. [48]

    Sinaga and Miin-Shen Yang

    Kristina P. Sinaga and Miin-Shen Yang. 2020. Unsupervised K-Means Clustering Algorithm.IEEE Access8 (2020), 80716–80727. doi:10.1109/ACCESS.2020.2988796

    2020

  40. [49]

    Shezheng Song, Shan Zhao, ChengYu Wang, Tianwei Yan, Shasha Li, Xiaoguang Mao, and Meng Wang. 2024. A Dual-Way Enhanced Framework from Text Matching Point of View for Multimodal Entity Linking.Proceedings of the AAAI Conference on Artificial Intelligence38, 17 (Mar. 2024), 190...

    2024

  41. [50]

    Mohammed Suhail, Abhay Mittal, Behjat Siddiquie, Chris Broaddus, Jayan Ele- dath, Gerard Medioni, and Leonid Sigal. 2021. Energy-Based Learning for Scene Graph Generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 13936–13945

    2021

  42. [51]

    Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. 2020. Unbiased Scene Graph Generation From Biased Training. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2020

  43. [52]

    Qwen Team. 2024. Qwen2.5: A Party of Foundation Models. https://qwenlm. github.io/blog/qwen2.5/. Includes instruction -tuned Qwen2.5-72B-Instruct via Hugging Face

    2024

  44. [53]

    Vincent A Traag, Ludo Waltman, and Nees Jan Van Eck. 2019. From Louvain to Leiden: guaranteeing well-connected communities.Scientific reports9, 1 (2019), 1–12

    2019

  45. [54]

    Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov

    Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Multimodal Transformer for Unaligned Multimodal Language Sequences. InProceedings of the 57th Annual Meeting of the Association for Computational Linguisti...

    2019 doi

  46. [55]

    Ultralytics. 2023. Ultralytics YOLOv8: Cutting-Edge Object Detection Models. https://github.com/ultralytics/ultralytics. Accessed: 2025-07-14

    2023

  47. [56]

    Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In2015 IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR). 3156–3164. doi:10.1109/CVPR.2015. 7298935

    2015 doi

  48. [57]

    Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, et al. 2024. Mineru: An open-source solution for precise document content extraction.arXiv preprint arXiv:2409.18839 (2024)

    2024 arXiv

  49. [58]

    Jingyi Wang, Jianzhong Ju, Jian Luan, and Zhidong Deng. 2025. LLaVA-SG: Leveraging Scene Graphs as Visual Semantic Expression in Vision-Language Models. InICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 1–5. doi:10.1109/ICA...

    2025

  50. [59]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191(2024)

    2024 arXiv

  51. [60]

    Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, Jifeng Dai, and et al. 2024. InternVL2.5 -MPO: Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization. arXiv pre...

    2024 arXiv

  52. [61]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al

  53. [62]

    Peng Xia, Kangyu Zhu, Haoran Li, Tianze Wang, Weijia Shi, Sheng Wang, Linjun Zhang, James Zou, and Huaxiu Yao. 2025. MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models. InThe Thirteenth Interna- tional Conference on Learning Representations. https://o...

    2025

  54. [63]

    Choy, and Li Fei-Fei

    Danfei Xu, Yuke Zhu, Christopher B. Choy, and Li Fei-Fei. 2017. Scene Graph Generation by Iterative Message Passing. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2017

  55. [64]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al . 2024. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115(2024)

    2024 arXiv

  56. [65]

    Barry Menglong Yao, Yu Chen, Qifan Wang, Sijia Wang, Minqian Liu, Zhiyang Xu, Licheng Yu, and Lifu Huang. 2023. AMELI: Enhancing Multimodal Entity Linking with Fine-Grained Attributes.CoRRabs/2305.14725 (2023). https: //doi.org/10.48550/arXiv.2305.14725

    2023 doi arXiv

  57. [66]

    Same Semantics, Different Structure

    Chen Yin and Zixuan Zhang. 2024. A Study of Sentence Similarity Based on the All-minilm-l6-v2 Model With “Same Semantics, Different Structure” After Fine Tuning. In2024 2nd International Conference on Image, Algorithms and Artificial Intelligence (ICIAAI 2024). Atlantis Press, 677–684

    2024

  58. [67]

    Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, and Maosong Sun. 2024. VisRAG: Vision- based Retrieval-augmented Generation on Multi-modality Documents.CoRR abs/2410.10594 (2024). https://doi.org/10.48550/arXiv.2410.10594

    2024 doi arXiv

  59. [68]

    Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. 2018. Neural Motifs: Scene Graph Parsing With Global Context. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2018

  60. [69]

    Dun Zhang, Jiacheng Li, Ziyang Zeng, and Fulong Wang. 2024. Jasper and Stella: distillation of SOTA embedding models.arXiv preprint arXiv:2412.19048(2024)

    2024 arXiv

  61. [70]

    Fengzhi Zhao, Zhezhou Yu, Tao Wang, and Yi Lv. 2024. Image Captioning Based on Semantic Scenes.Entropy26, 10 (2024). doi:10.3390/e26100876

    2024 doi

  62. [71]

    Chaofan Zheng, Xinyu Lyu, Lianli Gao, Bo Dai, and Jingkuan Song. 2023. Prototype-Based Embedding Network for Scene Graph Generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 22783–22792

    2023

  63. [72]

    Shuyan Zhou, Shruti Rijhwani, John Wieting, Jaime Carbonell, and Graham Neubig. 2020. Improving candidate generation for low-resource cross-lingual entity linking.Transactions of the Association for Computational Linguistics8 (2020), 109–124

    2020

  64. [73]

    Harry Potter and the Sorcerer’s Stone

    Anni Zou, Wenhao Yu, Hongming Zhang, Kaixin Ma, Deng Cai, Zhuosheng Zhang, Hai Zhao, and Dong Yu. 2024. Docbench: A benchmark for evaluating llm-based document reading systems.arXiv preprint arXiv:2407.10701(2024). 11 Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Xue...

    2024 arXiv

  65. [78]

    Modify and enhance the entity_name if necessary

  66. [79]

    Expand the description by integrating relevant details and insights from the chunk_text

  67. [80]

    no match

    Include an original_name field to capture the original entity name before enhancement. Ensure the final output is in valid JSON format, only including the list of enhanced entities without any additional text. After generating candidate entities, LLM is used to align visual en...

    2018

  68. [81]

    Extract all entities from the image. For each identified entity, extract the following information: - Entity Name: The name of the entity - Entity Type: Can be one of the following types: [{entity_types}] - Entity Description: A comprehensive description of the entity’s at- tr...

  69. [82]

    relationship

    From the entities identified in Step 1, identify all pairs of (Source Entity, Target Entity) where the entities are clearly related. For each related pair of entities, extract the following information: - Source Entity: The name of the source entity, as identified in Step 1 - ...

  70. [83]

    Use {record_delimiter} as the list separator

    Return the output as a list including all entities and relationships identified in Steps 1 and 2. Use {record_delimiter} as the list separator

  71. [84]

    entity

    Upon completion, output {completion_delimiter} The examples contained within the prompt are excessively lengthy. For illustrative purposes, only a small excerpt is presented here to demonstrate the format, as follows: ( " entity " { tuple_delimiter } " Girl " { tuple_delimiter...

    2018

  72. [85]

    Based on the provided entity-description pairs, determine the entity corresponding to the image feature block and output the following information: - Entity Name: The name of the entity corresponding to the image feature block

  73. [86]

    The image feature block Image Feature Block Name is a picture of Entity Name

    Output the relationship between the image feature block and the corresponding entity, and extract the following information: - Image Feature Block Name: The name of the input image feature block - Relationship Description: Describe the relationship between the entity and the i...

    2007

  74. [2014]

    InThe Fifth International Conference on the Applications of Digital Information and Web Technologies (ICADIWT 2014)

    DBSCAN: Past, present and future. InThe Fifth International Conference on the Applications of Digital Information and Web Technologies (ICADIWT 2014). 232–238. doi:10.1109/ICADIWT.2014.6814687

    2014

  75. [2020]

    InProceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations

    Transformers: State-of-the-art natural language processing. InProceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations. 38–45

    2020

  76. [2023]

    Entity Linking Meets Deep Learning: Techniques and Solutions.IEEE Transactions on Knowledge and Data Engineering35, 3 (2023), 2556–2578. doi:10. 1109/TKDE.2021.3117715 10 MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphs Conference acronym...

    2023

  77. [2024]

    InProceedings of the 33rd ACM International Conference on Information and Knowledge Management(Boise, ID, USA)(CIKM ’24)

    UniMEL: A Unified Framework for Multimodal Entity Linking with Large Language Models. InProceedings of the 33rd ACM International Conference on Information and Knowledge Management(Boise, ID, USA)(CIKM ’24). Association for Computing Machinery, New York, NY, USA, 1909–1919. do...

    1909 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.