Pith. sign in

REVIEW 4 major objections 5 minor 64 references

KG-ViP claims that fusing scene graphs with commonsense graphs through a query-bridged retrieval pipeline lets multi-modal LLMs answer knowledge-based visual questions more accurately than retrieval-augmented baselines.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 10:39 UTC pith:XODZ7P7I

load-bearing objection Good graph-fusion system, but the headline multi-hop gains rest on a benchmark generated from the same knowledge graph the method retrieves from. the 4 major comments →

arxiv 2601.11632 v3 pith:XODZ7P7I submitted 2026-01-14 cs.CV

KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question Answering

Zhiyang Li , Ao Ke , Yukun Cao , Xike Xie This is my paper
classification cs.CV
keywords visual question answeringmulti-modal large language modelsscene graphcommonsense knowledge graphgraph retrieval-augmented generationmulti-hop reasoningknowledge groundingvisual perception
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper is trying to establish that the two main failure modes of multi-modal LLMs in visual question answering—hallucinating external facts and missing fine-grained visual relations—can be addressed together by combining two structured representations: a commonsense graph supplying domain knowledge and a scene graph supplying the image's objects and interactions. The proposed framework, KG-ViP, builds a multi-modal commonsense graph, generates a scene graph from the query image, then uses the question as a semantic bridge to retrieve a query-aware subgraph from each and fuse them into one unified graph that the MLLM reads before answering. On FVQA 2.0+ and a new multi-hop benchmark called MVQA, the authors report consistent gains over prior retrieval-augmented and scene-graph methods, with the largest improvements on multi-hop questions. If the reported results hold, the practical upshot is that knowledge grounding and visual perception are not competing strategies but two halves of a single reasoning context.

Core claim

KG-ViP's central claim is that scene graphs and commonsense graphs are complementary in exactly the way MLLMs need: scene graphs anchor abstract knowledge to concrete visible objects, while commonsense graphs supply the identities, relationships, and background facts that the image alone cannot provide. The paper shows that by constructing both graphs in multi-modal form, retrieving a text-guided commonsense subgraph first, using it to prune the scene graph, then retrieving commonsense nodes via direct vision-to-vision matching, and finally fusing the two subgraphs into one unified graph, the MLLM can answer questions that neither graph alone supports. The authors report that this pipeline o

What carries the argument

The central mechanism is a retrieval-and-fusion pipeline with three stages. First, a multi-modal commonsense graph (a knowledge graph whose entities carry images as well as text) is built for the domain, and a scene graph (objects in the image with relations like holding or next to) is generated from the query image. Second, two-stage retrieval uses the question to retrieve a text-guided commonsense subgraph, uses that subgraph to prune the noisy scene graph into a query-aware scene subgraph, and then uses the refined visual entities to retrieve commonsense nodes by direct vision-to-vision matching. Third, the two subgraphs are fused by cross-modal entity alignment—combining image-to-text an

Load-bearing premise

The load-bearing assumption is that the MVQA benchmark is a valid independent test of multi-hop VQA—but because its questions were generated using the same commonsense graph that KG-ViP retrieves from, the large MVQA gains could simply reflect retrieving the exact facts used to write the questions, leaving FVQA 2.0+ as the only fully independent support.

What would settle it

Build a multi-hop benchmark whose questions are authored from one commonsense graph but evaluated with retrieval from a disjoint graph, or from the same graph with all answer-defining facts removed; if KG-ViP's margin over baselines disappears, the claimed multi-hop gains are an artifact of benchmark construction rather than a general reasoning improvement.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A VQA system no longer has to choose between external knowledge and fine-grained perception; both can be supplied as one structured context before the model answers.
  • Scene graphs can act as visual anchors for external knowledge, so direct vision-to-vision retrieval avoids the ambiguity of matching generic textual descriptions to specific visual entities.
  • The pipeline is model-agnostic: the same framework improves accuracy across MLLMs from 3B to 106B parameters, including models not used during development.
  • Graph fusion and refinement are load-bearing components, not decoration: removing either one lowers accuracy on both benchmarks, with fusion removal costing 5.64 points on MVQA.
  • The construction pipeline is modular, so domains with an existing textual knowledge base and per-image scene graphs can adopt the same recipe.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the MVQA benchmark may overstate the method's advantage, because its QA pairs are authored by an MLLM conditioned on the same commonsense graph that KG-ViP retrieves from; a fairer evaluation would generate questions from a held-out graph.
  • Editorial inference: if the fusion mechanism is genuinely general, it should transfer to other paired graph structures—for example, temporal event graphs with video scene graphs—where one graph grounds perception and the other supplies background knowledge.
  • Editorial inference: because the paper identifies scene graph quality as the upstream bottleneck, gains should grow as scene graph generators improve, while severe occlusion and small objects remain likely failure modes.
  • Editorial inference: replacing the offline commonsense graph with a dynamic, updateable graph is a natural stress test; if query-bridged fusion still works under frequent updates, the method can serve evolving domains.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper proposes KG-ViP, a framework that improves multi-modal LLM visual question answering by fusing scene graphs (capturing fine-grained visual relations) with commonsense graphs (providing external knowledge). The method constructs a multimodal commonsense graph, generates a scene graph from the input image, performs text-guided and vision-guided retrieval to obtain query-aware subgraphs, and fuses them into a unified structured context for the MLLM. Experiments are reported on FVQA 2.0+ (an upgraded version of FVQA 2.0) and on a new benchmark MVQA constructed from MovieBench. The paper claims that KG-ViP significantly outperforms existing VQA methods, with the largest margins on MVQA.

Significance. The core idea—jointly integrating scene and commonsense graphs for MLLM reasoning—is timely and the pipeline is clearly presented. The ablations (Tables 2–4) usefully isolate the contributions of fusion, refinement, retrieval mode, and pruning, and Table 5 shows gains across several MLLM scales. The FVQA 2.0+ result, being based on an external benchmark, provides some independent evidence that the approach helps. However, the MVQA benchmark, which supplies the largest reported gains, was generated by conditioning the QA creation on the same commonsense graph that KG-ViP retrieves from; ground-truth answers are effectively facts stored in the retrieval index, and the baselines do not have access to this graph. The MVQA evidence is therefore not an independent validation of the multi-hop claims. The paper also does not release code or the generated QA pairs, and several key hyperparameters are unspecified. If the MVQA issues are resolved and the FVQA 2.0+ result is confirmed with proper statistical testing, the contribution would be significant; at present the independent support is limited.

major comments (4)
  1. [Appendix A.2/A.4, Table 1] The MVQA benchmark is not an independent evaluation. The QA generation prompt in A.4 supplies 'Reference facts' retrieved from the commonsense graph and instructs the model to involve attributes 'mentioned in the reference facts.' The ground-truth answer for each question is therefore one of the facts stored in KG-ViP's retrieval index. KG-ViP is asked to retrieve the exact fact used to write the question, while baselines (LLM-RA, FilterRAG, CCoT) cannot access this graph. The largest reported improvements are on MVQA (LLM-J 35.44 vs. 24.10; SAS 23.25 vs. 17.22 in Table 1). Without external validation, human evaluation of the generated QAs, or a fairer protocol that gives baselines the same graph, these margins cannot support the central 'multi-hop outperformance' claim. The FVQA 2.0+ benchmark is external and less suspect, but the MVQA claim needs substantial rework.
  2. [Section 3.2, Eq. (1), Section 4.2] Several load-bearing hyperparameters are not reported: the fusion weight alpha in Eq. (1); the top-n entities selected in Stage 1; the top-m entities selected in Step B; the order k of the k-th order subgraph extraction; and the number of graph-refinement steps (only the maximum t is given, and Section 4.2 fixes t=1). Without these values, the experiments are not reproducible, and the sensitivity of the method to these choices is unknown. Please report all hyperparameter values and, ideally, a sensitivity analysis for alpha, n, m, and k.
  3. [Section 4.2, Tables 1–5] No error bars, confidence intervals, or significance tests are reported. All comparisons are single runs. The FVQA 2.0+ improvements over LLM-RA are modest on some metrics (e.g., METEOR +0.87 in Table 1). The abstract and conclusion state that KG-ViP 'significantly outperforms' existing methods, but without variance estimates or statistical tests this is not established. I recommend reporting results over multiple seeds or bootstrapped confidence intervals, especially given the use of LLM-as-a-judge, which can be noisy.
  4. [Section 2, Table 1] The related work section mentions several strong recent systems (EchoSight, Wiki-LLaVA, M3COT, MMCD, MKG-RAG, graph-RAG methods), but Table 1 compares only against CCoT, FilterRAG, LLM-RA, and NaiveRAG. The claim that KG-ViP 'significantly outperforms existing VQA methods' is broader than the evidence. Please add the most relevant recent baselines, or explicitly narrow the claim to the evaluated subset and justify the omission.
minor comments (5)
  1. [Section 4.3] The text says 'SAS by 2.93%' but Table 1 shows a 3.93-point increase (50.75 vs. 46.82). Please correct the arithmetic and clarify whether differences are percentage points or relative percentages.
  2. [Table 1] The header 'V anilla' appears to be a typo for 'Vanilla'.
  3. [Section 3.3, Eq. (1)] The notation phi and psi is used without explicit definition in the main text; clarify that phi is a visual encoder and psi a text encoder.
  4. [Appendix A.1] The LoRA scale factor is also called alpha, which could be confused with the fusion weight alpha in Eq. (1). Consider renaming one of them.
  5. [Figure 3] The bar chart would benefit from error bars and axis labels; currently it shows only point values.

Circularity Check

1 steps flagged

MVQA benchmark circularity: QA pairs are generated from the same commonsense graph that KG-ViP retrieves from, making the headline multi-hop gains partly a fact-retrieval exercise; FVQA 2.0+ remains an independent check.

specific steps
  1. self definitional [Section 4.1 (MVQA construction; QA Generation Pipeline) and Appendix A.4 (VQA Data Generator prompt)]
    "we incorporate an accessible commonsense graph as an external source of background knowledge ... In implementation, we retrieve local knowledge fragments associated with candidate entities from the graph and provide them as auxiliary context to the MLLM. ... The question should not be fully answerable from the Image alone, and is expected to benefit from the provided reference facts (retrieved from a commonsense graph). ... Try to involve at least one aspect that is typically not directly visible in the Image, such as name/identity ... or other personal attributes mentioned in the reference fa"

    MVQA, the benchmark producing KG-ViP's largest margins (LLM-J 35.44 vs 24.10), is generated by giving an MLLM 'reference facts' retrieved from the same cinematic commonsense graph that Section 4.1 builds and that KG-ViP uses as its retrieval index. The instructions force the QA writer to include attributes 'mentioned in the reference facts,' so the ground-truth answer is, by construction, a fact in KG-ViP's retrieval index. The evaluation task reduces to retrieving the exact fact used to write the question, while baselines without access to this graph are at an asymmetric disadvantage. The paper's multi-hop claim therefore rests on a self-referential benchmark; only FVQA 2.0+ is independent.

full rationale

The framework itself is not a fitted equation and its retrieval/fusion pipeline is not derived from the benchmark numbers, so there is no circularity in the method's construction. FVQA 2.0+ uses the original 2,820 external QA pairs from FVQA 2.0; the fact that the authors enrich the knowledge base with images does not make the questions or answers definitional to KG-ViP's index, so those results are genuine evidence. However, MVQA is a self-constructed benchmark whose QA pairs are generated from the same commonsense graph that supplies KG-ViP's retrieval context (Appendix A.2 and A.4). The generated answers are explicitly drawn from 'reference facts' in that graph, so KG-ViP's MVQA performance is in part answer-fact retrieval rather than a test of general multi-hop fusion. Because this circular benchmark supplies the larger and more striking margins, the central claim of significant multi-hop improvement is partially reduced by construction. No load-bearing self-citation or imported uniqueness theorem appears elsewhere; citations to the authors' prior GraphRAG work are contextual only. Score 6: one benchmark's predictions reduce to the graph input by construction, while FVQA 2.0+ remains independent.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The central claim inherits four domain assumptions and five unreported hyperparameters. The MVQA generation assumption is the most consequential because the benchmark is constructed from the same graph used for retrieval; if that assumption fails, the larger performance margin on MVQA is not interpretable. No invented entities are introduced.

free parameters (5)
  • fusion weight alpha in Eq. (1) = not reported
    Balances cross-modal text alignment against visual similarity during graph fusion; no value or search range is given, suggesting it was tuned on the benchmarks.
  • text-guided subgraph size n (top-n entities after RWR/PPR) = not reported
    Controls pruning and retrieval scope in Stage 1; threshold is not stated.
  • vision-guided subgraph size m (top-m entities) = not reported
    Controls how many commonsense nodes survive visual retrieval.
  • k-th order subgraph extraction order = not reported
    Determines the local neighborhood size around query entities in the commonsense graph.
  • graph refinement step budget t = 1
    Fixed to 1; the number of Expand/Prune iterations is a model-selection choice.
axioms (4)
  • domain assumption The automatically generated MVQA questions are unbiased, unambiguous, and require multi-hop reasoning
    Appendix A.2 generates QA pairs with an MLLM conditioned on scene descriptions and reference facts from the same commonsense graph used for evaluation; no human validation is reported.
  • domain assumption LLM-as-a-Judge scores correlate with human answer quality
    Section 4.2 uses Qwen2.5-7B and DeepSeek-V3.2 as judges without a human agreement study.
  • domain assumption The MLLM-generated scene graph covers the objects and relations needed to answer the query
    Section 3.1.2 uses Qwen2.5-VL-7B for scene graph extraction; the Limitations section concedes visual parsing quality bounds downstream reasoning.
  • domain assumption Text-guided commonsense subgraph is a sound filter for scene graph pruning
    Section 3.2 Step A prunes scene entities based on relevance to G_c,T; if text retrieval misses an entity, the scene graph loses it and the final answer cannot recover it.

pith-pipeline@v1.3.0-alltime-deepseek · 16576 in / 13593 out tokens · 122697 ms · 2026-08-03T10:39:13.059220+00:00 · methodology

0 comments
read the original abstract

Multi-modal Large Language Models (MLLMs) for Visual Question Answering (VQA) often suffer from dual limitations: knowledge hallucination and insufficient fine-grained visual perception. Crucially, we identify that commonsense graphs and scene graphs provide precisely complementary solutions to these respective deficiencies by providing rich external knowledge and capturing fine-grained visual details. However, prior works typically treat them in isolation, overlooking their synergistic potential. To bridge this gap, we propose KG-ViP, a unified framework that empowers MLLMs by fusing scene graphs and commonsense graphs. The core of the KG-ViP framework is a novel retrieval-and-fusion pipeline that utilizes the query as a semantic bridge to progressively integrate both graphs, synthesizing a unified structured context that facilitates reliable multi-modal reasoning. Extensive experiments on FVQA 2.0+ and MVQA benchmarks demonstrate that KG-ViP significantly outperforms existing VQA methods.

Figures

Figures reproduced from arXiv: 2601.11632 by Ao Ke, Xike Xie, Yukun Cao, Zhiyang Li.

Figure 1
Figure 1. Figure 1: Complementary roles of Scene Graphs and Commonsense Graphs. Using a scene from Inception as an example: Given the query asking “Who is the man holding the cup?”, the scene graph explicitly captures visual relations (e.g., holding) yet fails to recognize character identities (Cobb, Robert). Conversely, the commonsense graph offers semantic facts (e.g., Cobb targets Robert) but suffers from grounding ambigui… view at source ↗
Figure 2
Figure 2. Figure 2: The Overview of KG-ViP Framework. KG-ViP operates in three stages. First, we prepare a domain Commonsense Graph Gc and derive a Scene Graph Gs from the input image Iq. Second, the query Q retrieves a text-guided subgraph Gc,T from Gc, which is used to filter Gs into a refined Gˆs. Subsequently, Gˆs guides the retrieval of visual knowledge Gc,I to form the final commonsense subgraph G˜c. Finally, Gˆs and G˜… view at source ↗
Figure 3
Figure 3. Figure 3: Impact of vision encoder selection on KG-ViP. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative comparison of VQA results. The [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

64 extracted references · 1 canonical work pages

  1. [1]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. https://openreview.net/forum?id=hSyW5go0v8 Self- RAG : Learning to retrieve, generate, and critique through self-reflection . In The Twelfth International Conference on Learning Representations

    2024

  2. [2]

    Satanjeev Banerjee and Alon Lavie. 2005. https://aclanthology.org/W05-0909/ METEOR : An automatic metric for MT evaluation with improved correlation with human judgments . In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , pages 65--72, Ann Arbor, Michigan. Association for Compu...

    2005

  3. [3]

    Tobias Braun, Mark Rothermel, Marcus Rohrbach, and Anna Rohrbach. 2025. https://arxiv.org/abs/2412.10510 DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts . In Proceedings of the 42nd International Conference on Machine Learning

    Pith/arXiv arXiv 2025

  4. [4]

    Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. 2024. https://arxiv.org/abs/2404.15406 Wiki-llava: Hierarchical retrieval-augmented generation for multimodal llms . Preprint, arXiv:2404.15406

    Pith/arXiv arXiv 2024

  5. [5]

    Kevin Zhou, and Jianliang Xu

    Yukun Cao, Zengyi Gao, Zhiyang Li, Xike Xie, S. Kevin Zhou, and Jianliang Xu. 2025. https://doi.org/10.14778/3748191.3748194 Lego-graphrag: Modularizing graph-based retrieval-augmented generation for design space exploration . Proc. VLDB Endow., 18(10):3269–3283

    arXiv 2025

  6. [6]

    Eun Chang, Zhuangqun Huang, Yiwei Liao, Sagar Ravi Bhavsar, Amogh Param, Tammy Stark, Adel Ahmadyan, Xiao Yang, Jiaqi Wang, Ahsan Abdullah, Giang Nguyen, Akil Iyer, David Hall, Elissa Li, Shane Moon, Nicolas Scheffer, Kirmani Ahmed, Babak Damavandi, Rakesh Wanga, and 3 others. 2025. https://arxiv.org/abs/2511.22154 Wearvqa: A visual question answering ben...

    arXiv 2025

  7. [7]

    Xiaojun Chang, Pengzhen Ren, Pengfei Xu, Zhihui Li, Xiaojiang Chen, and Alex Hauptmann. 2021. A comprehensive survey of scene graphs: Generation and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):1--26

    2021

  8. [8]

    Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming-Wei Chang. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.925 Can pre-trained vision and language models answer visual information-seeking questions? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14948--14968, Singap...

    doi:10.18653/v1/2023.emnlp-main.925 2023

  9. [9]

    Zuyao Chen, Jinlin Wu, Zhen Lei, Zhaoxiang Zhang, and Changwen Chen. 2024. https://arxiv.org/abs/2312.04314 Gpt4sgg: Synthesizing scene graphs from holistic and region-specific narratives . Preprint, arXiv:2312.04314

    Pith/arXiv arXiv 2024

  10. [10]

    Changin Choi, Wonseok Lee, Jungmin Ko, and Wonjong Rhee. 2025. https://arxiv.org/abs/2509.00798 Multimodal iterative rag for knowledge-intensive visual question answering . Preprint, arXiv:2509.00798

    Pith/arXiv arXiv 2025

  11. [11]

    Maruf, Ismini Lourentzou, Arka Daw, and Anuj Karpatne

    Amartya Dutta, Kazi Sajeed Mehrab, Medha Sawhney, Abhilash Neog, Mridul Khurana, Sepideh Fatemi, Aanish Pradhan, M. Maruf, Ismini Lourentzou, Arka Daw, and Anuj Karpatne. 2025. https://arxiv.org/abs/2506.08189 Open world scene graph generation using vision language models . Preprint, arXiv:2506.08189

    Pith/arXiv arXiv 2025

  12. [12]

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. 2024. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130

    Pith/arXiv arXiv 2024

  13. [13]

    Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pages 6491--6501

    2024

  14. [14]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. https://arxiv.org/abs/2411.15594 A survey on llm-as-a-judge . Preprint, arXiv:2411.15594

    Pith/arXiv arXiv 2025

  15. [15]

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. 2024. Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vi...

    2024

  16. [16]

    Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, and Jianfeng Gao. 2022. https://arxiv.org/abs/2112.08614 Kat: A knowledge augmented transformer for vision-and-language . Preprint, arXiv:2112.08614

    Pith/arXiv arXiv 2022

  17. [17]

    Zirui Guo, Lianghao Xia, Yanhua Yu, Tu Ao, and Chao Huang. 2024. https://arxiv.org/abs/2410.05779 Lightrag: Simple and fast retrieval-augmented generation

    Pith/arXiv arXiv 2024

  18. [18]

    Roei Herzig, Alon Mendelson, Leonid Karlinsky, Assaf Arbelle, Rogerio Feris, Trevor Darrell, and Amir Globerson. 2023. https://arxiv.org/abs/2305.06343 Incorporating structured representations into pretrained vision & language models using scene graphs . Preprint, arXiv:2305.06343

    Pith/arXiv arXiv 2023

  19. [19]

    Jinbae Im, JeongYeon Nam, Nokyung Park, Hyungmin Lee, and Seunghyun Park. 2024. https://arxiv.org/abs/2404.02072 Egtr: Extracting graph from transformer for scene graph generation . Preprint, arXiv:2404.02072

    Pith/arXiv arXiv 2024

  20. [20]

    Pu Jian, Donglei Yu, and Jiajun Zhang. 2024. Large language models know what is key visual entity: An llm-assisted multimodal retrieval for vqa. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10939--10956

    2024

  21. [21]

    Uku Kangur, Krish Agrawal, Yashashvi Singh, Ahmed Sabir, and Rajesh Sharma. 2025. Multireflect: Multimodal self-reflective rag-based automated fact-checking. In Proceedings of the 1st Workshop on Multimodal Augmented Generation via Multimodal Retrieval (MAGMaR 2025), pages 1--17

    2025

  22. [22]

    Muhammad Junaid Khan, Adil Masood Siddiqui, Hamid Saeed Khan, and Jaleed Khan. 2025. Enhancing visual question answering with common sense knowledge: a data-driven neurosymbolic graph routing approach. International Journal of Data Science and Analytics, pages 1--16

    2025

  23. [23]

    Siddhesh Khandelwal and Leonid Sigal. 2022. https://arxiv.org/abs/2207.13440 Iterative scene graph generation . Preprint, arXiv:2207.13440

    Pith/arXiv arXiv 2022

  24. [24]

    Kibum Kim, Kanghoon Yoon, Jaehyeong Jeon, Yeonjun In, Jinyoung Moon, Donghyun Kim, and Chanyoung Park. 2024. Llm4sgg: Large language models for weakly supervised scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 28306--28316

    2024

  25. [25]

    Insu Lee, Wooje Park, Jaeyun Jang, Minyoung Noh, Kyuhong Shim, and Byonghyo Shim. 2025. https://arxiv.org/abs/2505.21955 Towards comprehensive scene understanding: Integrating first and third-person views for lvlms . Preprint, arXiv:2505.21955

    arXiv 2025

  26. [26]

    Junlin Lee, Yequan Wang, Jing Li, and Min Zhang. 2024. Multimodal reasoning with multimodal knowledge graph. arXiv preprint arXiv:2406.02030

    Pith/arXiv arXiv 2024

  27. [27]

    Hongsheng Li, Guangming Zhu, Liang Zhang, Youliang Jiang, Yixuan Dang, Haoran Hou, Peiyi Shen, Xia Zhao, Syed Afaq Ali Shah, and Mohammed Bennamoun. 2024. Scene graph generation: A comprehensive survey. Neurocomputing, 566:127052

    2024

  28. [28]

    Lin Li, Chuhan Zhang, Dong Zhang, Chong Sun, Chen Li, and Long Chen. 2025 a . https://arxiv.org/abs/2511.05935 Interaction-centric knowledge infusion and transfer for open-vocabulary scene graph generation . Preprint, arXiv:2511.05935

    arXiv 2025

  29. [29]

    Xinwei Li, Li Lin, Shuai Wang, and Hanqian Wu. 2025 b . https://doi.org/10.1145/3726302.3730081 Seeing beyond hallucinations: Llm-based compositional information extraction for multimodal reasoning . In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '25, page 1000–1010, New York, NY, ...

    arXiv 2025

  30. [30]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. https://arxiv.org/abs/2305.10355 Evaluating object hallucination in large vision-language models . Preprint, arXiv:2305.10355

    Pith/arXiv arXiv 2023

  31. [31]

    Fake Lin, Xi Zhu, Ziwei Zhao, Deqiang Huang, Yu Yu, Xueying Li, Zhi Zheng, Tong Xu, and Enhong Chen. 2024. Knowledge graph pruning for recommendation. arXiv preprint arXiv:2405.11531

    Pith/arXiv arXiv 2024

  32. [32]

    Weizhe Lin and Bill Byrne. 2022. https://arxiv.org/abs/2210.03809 Retrieval augmented visual question answering with outside knowledge . Preprint, arXiv:2210.03809

    Pith/arXiv arXiv 2022

  33. [33]

    Weizhe Lin, Jinghong Chen, Jingbiao Mei, Alexandru Coca, and Bill Byrne. 2023 a . https://arxiv.org/abs/2309.17133 Fine-grained late-interaction multi-modal retrieval for retrieval augmented visual question answering . Preprint, arXiv:2309.17133

    Pith/arXiv arXiv 2023

  34. [34]

    Weizhe Lin, Zhilin Wang, and Bill Byrne. 2023 b . https://aclanthology.org/2023.findings-eacl.11 FVQA 2.0: Introducing adversarial samples into fact-based visual question answering . In Findings of the Association for Computational Linguistics: EACL 2023, pages 149--157, Dubrovnik, Croatia. Association for Computational Linguistics

    2023

  35. [35]

    Ye Liu, Hui Li, Alberto Garcia-Duran, Mathias Niepert, Daniel Onoro-Rubio, and David S Rosenblum. 2019. Mmkg: multi-modal knowledge graphs. In European Semantic Web Conference, pages 459--474. Springer

    2019

  36. [36]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024. https://arxiv.org/abs/2307.06281 Mmbench: Is your multi-modal model an all-around player? Preprint, arXiv:2307.06281

    Pith/arXiv arXiv 2024

  37. [37]

    Maria Lomaeva and Nitisha Jain. 2022. Relation canonicalization in open knowledge graphs: a quantitative analysis. In European Semantic Web Conference, pages 21--25. Springer

    2022

  38. [38]

    Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. 2023. Crepe: Can vision-language foundation models reason compositionally? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10910--10921

    2023

  39. [39]

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Conference on Computer Vision and Pattern Recognition (CVPR)

    2019

  40. [40]

    Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. 2024. Compositional chain of thought prompting for large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2024

  41. [41]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748--8763. PmLR

    2021

  42. [42]

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. 2024. https://arxiv.org/abs/2401.14159 Grounded sam: Assembling open-world models for diverse visual tasks . Preprint, arXiv:2401.14159

    Pith/arXiv arXiv 2024

  43. [43]

    Julian Risch, Timo M \"o ller, Julian Gutsch, and Malte Pietsch. 2021. Semantic answer similarity for evaluating question answering models. arXiv preprint arXiv:2108.06130

    Pith/arXiv arXiv 2021

  44. [44]

    Nobin Sarwar. 2025. Filterrag: zero-shot informed retrieval-augmented generation to mitigate hallucinations in vqa. arXiv preprint arXiv:2502.18536

    arXiv 2025

  45. [45]

    Qingyi Si, Yuchen Mo, Zheng Lin, Huishan Ji, and Weiping Wang. 2023. https://doi.org/10.18653/v1/2023.acl-long.614 Combo of thinking and observing for outside-knowledge VQA . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10959--10975, Toronto, Canada. Association for Computational...

    work page doi:10.18653/v1/2023.acl-long.614 2023

  46. [46]

    Ni, Heung-Yeung Shum, and Jian Guo

    Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel M. Ni, Heung-Yeung Shum, and Jian Guo. 2024. https://arxiv.org/abs/2307.07697 Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph . Preprint, arXiv:2307.07697

    Pith/arXiv arXiv 2024

  47. [47]

    Zequn Sun, Qingheng Zhang, Wei Hu, Chengming Wang, Muhao Chen, Farahnaz Akrami, and Chengkai Li. 2020. http://www.vldb.org/pvldb/vol13/p2326-sun.pdf A benchmarking study of embedding-based entity alignment for knowledge graphs . Proceedings of the VLDB Endowment, 13(11):2326--2340

    2020

  48. [48]

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. 2024. https://arxiv.org/abs/2406.16860 Cambrian-1: A fully open, vision-centric exploration of multimodal llms

    Pith/arXiv arXiv 2024

  49. [49]

    Xueyao Wan and Hang Yu. 2025. https://arxiv.org/abs/2507.20804 Mmgraphrag: Bridging vision and language with interpretable multimodal knowledge graphs . Preprint, arXiv:2507.20804

    Pith/arXiv arXiv 2025

  50. [50]

    Jingyi Wang, Jianzhong Ju, Jian Luan, and Zhidong Deng. 2024. https://arxiv.org/abs/2408.16224 Llava-sg: Leveraging scene graphs as visual semantic expression in vision-language models . Preprint, arXiv:2408.16224

    Pith/arXiv arXiv 2024

  51. [51]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. https://arxiv.org/abs/2201.11903 Chain-of-thought prompting elicits reasoning in large language models . Preprint, arXiv:2201.11903

    Pith/arXiv arXiv 2023

  52. [52]

    Weijia Wu, Mingyu Liu, Zeyu Zhu, Xi Xia, Haoen Feng, Wen Wang, Kevin Qinghong Lin, Chunhua Shen, and Mike Zheng Shou. 2024. https://arxiv.org/abs/2411.15262 Moviebench: A hierarchical movie level dataset for long video generation . Preprint, arXiv:2411.15262

    Pith/arXiv arXiv 2024

  53. [53]

    Yubao Wu, Ruoming Jin, and Xiang Zhang. 2014. https://doi.org/10.1145/2588555.2610500 Fast and unified local search for random walk based k-nearest-neighbor query in large graphs . In Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data, SIGMOD '14, page 1139–1150, New York, NY, USA. Association for Computing Machinery

    arXiv 2014

  54. [54]

    Peng Xia, Kangyu Zhu, Haoran Li, Tianze Wang, Weijia Shi, Sheng Wang, Linjun Zhang, James Zou, and Huaxiu Yao. 2024. Mmed-rag: Versatile multimodal rag system for medical vision language models. arXiv preprint arXiv:2410.13085

    Pith/arXiv arXiv 2024

  55. [55]

    Dexuan Xu, Yanyuan Chen, Jieyi Wang, Yue Huang, Hanpin Wang, Zhi Jin, Hongxing Wang, Weihua Yue, Jing He, Hang Li, and Yu Huang. 2024. https://doi.org/10.18653/v1/2024.findings-acl.296 ML e VLM : Improve multi-level progressive capabilities based on multimodal large language model for medical visual question answering . In Findings of the Association for ...

    doi:10.18653/v1/2024.findings-acl.296 2024

  56. [56]

    Yibin Yan and Weidi Xie. 2024. https://aclanthology.org/2024.findings-emnlp.83 E cho S ight: Advancing visual-language models with W iki knowledge . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1538--1551, Miami, Florida, USA. Association for Computational Linguistics

    2024

  57. [57]

    Dongil Yang, Minjin Kim, Sunghwan Kim, Beong woo Kwak, Minjun Park, Jinseok Hong, Woontack Woo, and Jinyoung Yeo. 2025. https://arxiv.org/abs/2505.19510 Llm meets scene graph: Can large language models understand and generate scene graphs? a benchmark and empirical study . Preprint, arXiv:2505.19510

    Pith/arXiv arXiv 2025

  58. [58]

    Mingji Yang, Hanzhi Wang, Zhewei Wei, Sibo Wang, and Ji-Rong Wen. 2024. https://doi.org/10.1109/tkde.2024.3376000 Efficient algorithms for personalized pagerank computation: A survey . IEEE Transactions on Knowledge and Data Engineering, 36(9):4582–4602

    arXiv 2024

  59. [59]

    Xu Yuan, Liangbo Ning, Wenqi Fan, and Qing Li. 2025. https://arxiv.org/abs/2508.05318 mkg-rag: Multimodal knowledge graph-enhanced rag for visual question answering . Preprint, arXiv:2508.05318

    Pith/arXiv arXiv 2025

  60. [60]

    Wenjia Zhai. 2024. https://arxiv.org/abs/2410.11321 Self-adaptive multimodal retrieval-augmented generation . Preprint, arXiv:2410.11321

    Pith/arXiv arXiv 2024

  61. [61]

    Bowen Zhang and Harold Soh. 2024. Extract, define, canonicalize: An llm-based framework for knowledge graph construction. arXiv preprint arXiv:2404.03868

    Pith/arXiv arXiv 2024

  62. [62]

    Xiangru Zhu, Zhixu Li, Xiaodan Wang, Xueyao Jiang, Penglei Sun, Xuwu Wang, Yanghua Xiao, and Nicholas Jing Yuan. 2022. Multi-modal knowledge graph construction and application: A survey. IEEE Transactions on Knowledge and Data Engineering, 36(2):715--735

    2022

  63. [63]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  64. [64]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...