{"id":"34a943af-58b8-45bc-b4ff-2dfc0eb87f06","arxiv_id":"2601.11632","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"KG-ViP answers visual questions by merging an image scene graph with a commonsense knowledge graph, reporting 7.8-11.3 point LLM-J gains over prior retrieval baselines.","lead":"This paper proposes KG-ViP, a visual-question-answering system that fuses a scene graph of the image with a commonsense knowledge graph before a multimodal language model answers. It reports large gains on a standard benchmark and on a new movie benchmark, but the new benchmark was generated using the same knowledge graph the system retrieves from.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MVQA benchmark circularity: QA pairs are generated from the same commonsense graph KG-ViP retrieves from, so the large MVQA gains may reflect answer-fact retrieval rather than graph-fusion reasoning.","rationale":"The reader's weakest assumption is exactly the load-bearing issue: MVQA's construction is circular with respect to KG-ViP's retrieval index. The paper uses MVQA as the primary evidence for multi-hop gains, but the QA generation prompt in Appendix A.4 conditions on the same commonsense graph used during inference, and the reference facts are supplied as context to the generator. This means the benchmark is better described as testing retrieval of answer-bearing facts than as an external test of joint visual-knowledge reasoning. The FVQA 2.0+ results are real and show a consistent improvement, but they are single-hop and smaller, so they do not independently support the multi-hop claim. The paper also lacks error bars, code/data release, and human validation of the new benchmark, all of which would be needed to rule out the circularity concern. This does not mean the method is without merit; the ablations, robustness across MLLMs, and the FVQA 2.0+ results are positive signals. But the central claim currently rests on the MVQA benchmark, which is the least secure component. The proposed oracle-retrieval baseline would directly test whether the MVQA margin is due to graph access rather than the fusion pipeline. Since the reader already flagged this and reached CONDITIONAL, my independent assessment does not change the verdict.","tokens_in":16890,"tokens_out":5842,"duration_ms":59905,"concrete_test":"Re-run the MVQA comparison with an oracle-retrieval baseline: give LLM-RA and FilterRAG the same reference facts from the MVQA commonsense graph as retrieval context (or the exact fact used to generate each question). If these baselines reach or exceed KG-ViP's scores, the MVQA gains are due to graph access rather than the proposed fusion. To interpret the result, also check a random sample of MVQA questions to see whether the answer is directly stated in the supplied reference facts without needing the image; if so, the benchmark is a fact-retrieval test, not a multi-hop VQA test.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The weakest point is the validity of MVQA as an independent benchmark. Appendix A.2 describes an automated QA generation pipeline that conditions question creation on the same commonsense graph that KG-ViP retrieves from; the prompt in A.4 ('VQA Data Generator') explicitly supplies 'Reference facts' retrieved from the graph and instructs the model to involve attributes 'mentioned in the reference facts.' Thus the ground-truth answer for each MVQA question is one of the facts stored in KG-ViP's retrieval index. Baselines (LLM-RA, FilterRAG, CCoT) do not have access to this graph, so the comparison is asymmetric: KG-ViP is essentially being asked to retrieve the exact fact used to write the question, while baselines must find it elsewhere. The MVQA gains are the largest evidence for the paper's multi-hop claim (LLM-J 35.44 vs 24.10; SAS 23.25 vs 17.22). If MVQA is circular, only FVQA 2.0+ remains, and there the gains are smaller and single-hop (LLM-J +7.76, METEOR +0.87). The central claim of 'significant outperformance' for multi-hop reasoning therefore rests on an unvalidated, self-constructed benchmark. The paper itself does not report human validation of MVQA questions, nor does it release the generated QA pairs or code.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes KG-ViP, a framework that improves multi-modal LLM visual question answering by fusing scene graphs (capturing fine-grained visual relations) with commonsense graphs (providing external knowledge). The method constructs a multimodal commonsense graph, generates a scene graph from the input image, performs text-guided and vision-guided retrieval to obtain query-aware subgraphs, and fuses them into a unified structured context for the MLLM. Experiments are reported on FVQA 2.0+ (an upgraded version of FVQA 2.0) and on a new benchmark MVQA constructed from MovieBench. The paper claims that KG-ViP significantly outperforms existing VQA methods, with the largest margins on MVQA.","tokens_in":17265,"tokens_out":5312,"duration_ms":53730,"significance":"The core idea—jointly integrating scene and commonsense graphs for MLLM reasoning—is timely and the pipeline is clearly presented. The ablations (Tables 2–4) usefully isolate the contributions of fusion, refinement, retrieval mode, and pruning, and Table 5 shows gains across several MLLM scales. The FVQA 2.0+ result, being based on an external benchmark, provides some independent evidence that the approach helps. However, the MVQA benchmark, which supplies the largest reported gains, was generated by conditioning the QA creation on the same commonsense graph that KG-ViP retrieves from; ground-truth answers are effectively facts stored in the retrieval index, and the baselines do not have access to this graph. The MVQA evidence is therefore not an independent validation of the multi-hop claims. The paper also does not release code or the generated QA pairs, and several key hyperparameters are unspecified. If the MVQA issues are resolved and the FVQA 2.0+ result is confirmed with proper statistical testing, the contribution would be significant; at present the independent support is limited.","major_comments":[{"comment":"The MVQA benchmark is not an independent evaluation. The QA generation prompt in A.4 supplies 'Reference facts' retrieved from the commonsense graph and instructs the model to involve attributes 'mentioned in the reference facts.' The ground-truth answer for each question is therefore one of the facts stored in KG-ViP's retrieval index. KG-ViP is asked to retrieve the exact fact used to write the question, while baselines (LLM-RA, FilterRAG, CCoT) cannot access this graph. The largest reported improvements are on MVQA (LLM-J 35.44 vs. 24.10; SAS 23.25 vs. 17.22 in Table 1). Without external validation, human evaluation of the generated QAs, or a fairer protocol that gives baselines the same graph, these margins cannot support the central 'multi-hop outperformance' claim. The FVQA 2.0+ benchmark is external and less suspect, but the MVQA claim needs substantial rework.","section":"Appendix A.2/A.4, Table 1"},{"comment":"Several load-bearing hyperparameters are not reported: the fusion weight alpha in Eq. (1); the top-n entities selected in Stage 1; the top-m entities selected in Step B; the order k of the k-th order subgraph extraction; and the number of graph-refinement steps (only the maximum t is given, and Section 4.2 fixes t=1). Without these values, the experiments are not reproducible, and the sensitivity of the method to these choices is unknown. Please report all hyperparameter values and, ideally, a sensitivity analysis for alpha, n, m, and k.","section":"Section 3.2, Eq. (1), Section 4.2"},{"comment":"No error bars, confidence intervals, or significance tests are reported. All comparisons are single runs. The FVQA 2.0+ improvements over LLM-RA are modest on some metrics (e.g., METEOR +0.87 in Table 1). The abstract and conclusion state that KG-ViP 'significantly outperforms' existing methods, but without variance estimates or statistical tests this is not established. I recommend reporting results over multiple seeds or bootstrapped confidence intervals, especially given the use of LLM-as-a-judge, which can be noisy.","section":"Section 4.2, Tables 1–5"},{"comment":"The related work section mentions several strong recent systems (EchoSight, Wiki-LLaVA, M3COT, MMCD, MKG-RAG, graph-RAG methods), but Table 1 compares only against CCoT, FilterRAG, LLM-RA, and NaiveRAG. The claim that KG-ViP 'significantly outperforms existing VQA methods' is broader than the evidence. Please add the most relevant recent baselines, or explicitly narrow the claim to the evaluated subset and justify the omission.","section":"Section 2, Table 1"}],"minor_comments":[{"comment":"The text says 'SAS by 2.93%' but Table 1 shows a 3.93-point increase (50.75 vs. 46.82). Please correct the arithmetic and clarify whether differences are percentage points or relative percentages.","section":"Section 4.3"},{"comment":"The header 'V anilla' appears to be a typo for 'Vanilla'.","section":"Table 1"},{"comment":"The notation phi and psi is used without explicit definition in the main text; clarify that phi is a visual encoder and psi a text encoder.","section":"Section 3.3, Eq. (1)"},{"comment":"The LoRA scale factor is also called alpha, which could be confused with the fusion weight alpha in Eq. (1). Consider renaming one of them.","section":"Appendix A.1"},{"comment":"The bar chart would benefit from error bars and axis labels; currently it shows only point values.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The MVQA circularity is the central concern. If the authors cannot provide external multi-hop validation or a non-circular version of MVQA, the paper's main contribution would rest only on FVQA 2.0+ and should be judged accordingly. I also recommend the editor verify that the benchmark QA pairs are released with the revision, since the current supplement does not include them."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the thing to know: KG-ViP is a genuinely assembled system — scene graph + commonsense graph, two-stage retrieval, fusion and refinement — and that integration is new. The FVQA 2.0+ results on an existing benchmark are consistent, and the ablations show each component pulling its weight. But the new MVQA benchmark is a serious problem. The QA pairs are generated by an MLLM that is given 'reference facts' retrieved from the same commonsense graph KG-ViP retrieves from, and the generator is told to involve attributes mentioned in those facts. So the large MVQA gains (LLM-J 35.44 vs 24.10 for LLM-RA) may simply reflect KG-ViP retrieving the exact fact used to write the question. Baselines don't have access to that graph, so the comparison is asymmetric. That weakens the paper's strongest evidence for the multi-hop reasoning claim.\n\nWhat's good: the two-stage text-then-vision retrieval is sensible. Starting with text retrieval to prune the scene graph, then using the pruned scene graph for vision-based retrieval into the knowledge graph, is a clean way to reduce noisy entities and anchor knowledge. The cross-modal fusion equation (Eq. 1) is straightforward. The authors report ablations for fusion, refinement, retrieval mode, pruning, vision encoder, and MLLM robustness across four architectures. The FVQA 2.0+ benchmark is external, and the improvement over LLM-RA (LLM-J +7.76, METEOR +0.87, SAS +2.93) is consistent. The limitations section is honest about scene graph quality bounding downstream performance.\n\nSoft spots, proportionally: MVQA construction is the load-bearing one. There is no human validation of the generated questions, no release of the QA pairs or code, and the graph itself is built by the same pipeline used for the method. That's circular for the retrieval component. Also missing: error bars or significance tests anywhere, and several hyperparameters (alpha, n, m, k-th order) are undefined in the main text. A few recent graph-based VQA baselines (e.g., MMGraphRAG, mkg-rag) are cited in related work but not compared against. The FVQA 2.0+ gains, while positive, are modest; without error bars it's hard to know if the 7.76 point gap is robust.\n\nWho should read it: anyone building graph-augmented MLLM systems, and anyone designing VQA benchmarks. The system idea is worth engaging. But the evaluation needs real work — either an MVQA constructed from held-out facts or human-validated questions, a data/code release, and error bars — before the multi-hop claim can be taken at face value. I'd send it to peer review: the core idea and FVQA evidence justify referee time, and a sharp reviewer can push for the benchmark fix. Not a desk reject.","headline":"Good graph-fusion system, but the headline multi-hop gains rest on a benchmark generated from the same knowledge graph the method retrieves from.","tokens_in":17725,"tokens_out":2962,"would_cite":true,"duration_ms":29777,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"KG-ViP claims that fusing scene graphs with commonsense graphs through a query-bridged retrieval pipeline lets multi-modal LLMs answer knowledge-based visual questions more accurately than retrieval-augmented baselines.","keywords":["visual question answering","multi-modal large language models","scene graph","commonsense knowledge graph","graph retrieval-augmented generation","multi-hop reasoning","knowledge grounding","visual perception"],"falsifier":"Build a multi-hop benchmark whose questions are authored from one commonsense graph but evaluated with retrieval from a disjoint graph, or from the same graph with all answer-defining facts removed; if KG-ViP's margin over baselines disappears, the claimed multi-hop gains are an artifact of benchmark construction rather than a general reasoning improvement.","tokens_in":16773,"feed_emoji":"🧠","tokens_out":10314,"duration_ms":82647,"temperature":0.7,"pith_summary":"The paper is trying to establish that the two main failure modes of multi-modal LLMs in visual question answering—hallucinating external facts and missing fine-grained visual relations—can be addressed together by combining two structured representations: a commonsense graph supplying domain knowledge and a scene graph supplying the image's objects and interactions. The proposed framework, KG-ViP, builds a multi-modal commonsense graph, generates a scene graph from the query image, then uses the question as a semantic bridge to retrieve a query-aware subgraph from each and fuse them into one unified graph that the MLLM reads before answering. On FVQA 2.0+ and a new multi-hop benchmark called MVQA, the authors report consistent gains over prior retrieval-augmented and scene-graph methods, with the largest improvements on multi-hop questions. If the reported results hold, the practical upshot is that knowledge grounding and visual perception are not competing strategies but two halves of a single reasoning context.","feed_headline":"Fusing two graph types lifts visual question answering accuracy","feed_subtitle":"Multi-hop visual questions that need both image details and outside facts gain 8–11 points over retrieval baselines.","key_machinery":"The central mechanism is a retrieval-and-fusion pipeline with three stages. First, a multi-modal commonsense graph (a knowledge graph whose entities carry images as well as text) is built for the domain, and a scene graph (objects in the image with relations like holding or next to) is generated from the query image. Second, two-stage retrieval uses the question to retrieve a text-guided commonsense subgraph, uses that subgraph to prune the noisy scene graph into a query-aware scene subgraph, and then uses the refined visual entities to retrieve commonsense nodes by direct vision-to-vision matching. Third, the two subgraphs are fused by cross-modal entity alignment—combining image-to-text an","core_discovery":"KG-ViP's central claim is that scene graphs and commonsense graphs are complementary in exactly the way MLLMs need: scene graphs anchor abstract knowledge to concrete visible objects, while commonsense graphs supply the identities, relationships, and background facts that the image alone cannot provide. The paper shows that by constructing both graphs in multi-modal form, retrieving a text-guided commonsense subgraph first, using it to prune the scene graph, then retrieving commonsense nodes via direct vision-to-vision matching, and finally fusing the two subgraphs into one unified graph, the MLLM can answer questions that neither graph alone supports. The authors report that this pipeline o","pith_inferences":["Editorial inference: the MVQA benchmark may overstate the method's advantage, because its QA pairs are authored by an MLLM conditioned on the same commonsense graph that KG-ViP retrieves from; a fairer evaluation would generate questions from a held-out graph.","Editorial inference: if the fusion mechanism is genuinely general, it should transfer to other paired graph structures—for example, temporal event graphs with video scene graphs—where one graph grounds perception and the other supplies background knowledge.","Editorial inference: because the paper identifies scene graph quality as the upstream bottleneck, gains should grow as scene graph generators improve, while severe occlusion and small objects remain likely failure modes.","Editorial inference: replacing the offline commonsense graph with a dynamic, updateable graph is a natural stress test; if query-bridged fusion still works under frequent updates, the method can serve evolving domains."],"forward_implications":["A VQA system no longer has to choose between external knowledge and fine-grained perception; both can be supplied as one structured context before the model answers.","Scene graphs can act as visual anchors for external knowledge, so direct vision-to-vision retrieval avoids the ambiguity of matching generic textual descriptions to specific visual entities.","The pipeline is model-agnostic: the same framework improves accuracy across MLLMs from 3B to 106B parameters, including models not used during development.","Graph fusion and refinement are load-bearing components, not decoration: removing either one lowers accuracy on both benchmarks, with fusion removal costing 5.64 points on MVQA.","The construction pipeline is modular, so domains with an existing textual knowledge base and per-image scene graphs can adopt the same recipe."],"fun_headline_variants":["Scene + commonsense graphs: the VQA accuracy boost","Merging graph types lifts VQA by up to 11 points","KG-ViP: Two graphs, one answer—better VQA","Fusing graphs to cut VQA hallucinations and errors","Unified graph fusion: the missing VQA ingredient"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing assumption is that the MVQA benchmark is a valid independent test of multi-hop VQA—but because its questions were generated using the same commonsense graph that KG-ViP retrieves from, the large MVQA gains could simply reflect retrieving the exact facts used to write the questions, leaving FVQA 2.0+ as the only fully independent support.","fun_headline_variants_meta":{"raw":{"variants":["Scene + commonsense graphs: the VQA accuracy boost","Merging graph types lifts VQA by up to 11 points","KG-ViP: Two graphs, one answer—better VQA","Fusing graphs to cut VQA hallucinations and errors","Unified graph fusion: the missing VQA ingredient"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000576,"raw_usage":{"total_tokens":2525,"prompt_tokens":685,"completion_tokens":1840,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":429,"completion_tokens_details":{"reasoning_tokens":1755}},"tokens_in":429,"tokens_out":1840,"duration_ms":14655,"temperature":1.0,"reasoning_tokens":1755,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T10:39:13.059220+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a multi-hop benchmark whose questions are authored from one commonsense graph but evaluated with retrieval from a disjoint graph, or from the same graph with all answer-defining facts removed; if KG-ViP's margin over baselines disappears, the claimed multi-hop gains are an artifact of benchmark construction rather than a general reasoning improvement.","supporting_citations":[],"review_version":2}