{"id":"f3a5813f-2bbe-4b0c-accc-32635531a30e","arxiv_id":"2507.16826","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"QMKGF builds multi-path knowledge graph subgraphs from LLM-extracted entities, fuses the highest-scoring subgraph with query-relevant triples, and expands the query to improve RAG answer quality.","lead":"This paper introduces QMKGF, a retrieval pipeline that builds a knowledge graph from documents, selects and fuses query-relevant subgraphs, and uses them to expand the user query before answering. The authors report consistent gains over standard RAG baselines across five question answering benchmarks, most sharply on multi-hop datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core fusion step is underspecified: Eq. (12) and Algorithm line 16 use a threshold tau that is never defined or linked to r in Eq. (10), so the headline gains cannot be reproduced or attributed to the method as written.","rationale":"The reader's CONDITIONAL verdict is sound. I read the central claim as an empirical one: QMKGF, including its RM-based multi-path fusion, outperforms rerank-only RAG baselines. For that claim to be verifiable, the fusion rule must be executable. It is not, because tau is undefined. The most direct test is to pin down tau. I do not think this should flip the verdict to ACCEPT or REJECT; it is an addressable specification gap, so UNCHANGED (still CONDITIONAL) is appropriate. I considered other candidate concerns: (a) baselines may lack embedding fine-tuning, but the paper's own w/o-fintune ablation still beats BGE-rerank on HotpotQA by about 6 ROUGE-1 points, so that is not sufficient to overturn the comparison; (b) reward-model LLM labels are unvalidated, but the w/o-attention and fusion-strategy ablations provide indirect evidence that the scoring/fusion module contributes. The undefined tau is more load-bearing than either because it makes the reported result non-reproducible as written. If the authors clarify tau and its selection procedure, the central empirical claim can be properly tested.","tokens_in":17790,"tokens_out":7030,"duration_ms":75573,"concrete_test":"Obtain the authors' run configuration or, failing that, re-run the method with tau swept over a grid (e.g., 0.0 to 1.0 in steps of 0.05, plus tau = r from Eq. (10)) while keeping all other settings fixed, and compare the resulting HotpotQA ROUGE-1 to the reported 64.98. If 64.98 is reproduced at tau = r or at a documented value, the concern is resolved; if it is only reached at an unexplained tau (or not at all), the headline gain is not attributable to the described fusion rule and the verdict should remain conditional pending a specification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest empirical claim is that QMKGF's RM-based subgraph fusion produces large, consistent gains (e.g., HotpotQA ROUGE-1 64.98 vs. BGE-rerank 55.26). The fusion mechanism that distinguishes the method is not fully determined in the manuscript. Section 3.3 says the similarity between the highest-scoring subgraph and the query is 'set as a threshold r' and Eq. (10) defines r = cos(KGS_max, q), but the retention rule Eq. (12) and Algorithm line 16 filter triples by sim((h,r,t),q) >= tau. No value or definition of tau is given, and no relationship between tau and r is stated. This is not cosmetic: Section 5.8 shows the fusion strategy materially changes results (All fusion 64.07 vs. RM fusion 64.98 ROUGE-1 on HotpotQA), so the threshold controls how much of the lower-scoring subgraphs enters G_fusion. As submitted, an independent implementation cannot decide which triples to fuse, and the 9.72-point headline gain may depend on an undisclosed threshold value. The reward-model training labels are also only described as LLM scores, adding a second reproducibility gap, but the undefined tau alone is sufficient to block verification of the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes QMKGF, a retrieval-augmented generation framework that constructs a knowledge graph from unstructured documents via LLM-based entity and relation extraction, builds three types of query-anchored subgraphs (one-hop, multi-hop, and personalized-PageRank importance-based), scores these subgraphs with a query-aware attention reward model, fuses high-relevance triples from lower-scoring subgraphs into the highest-scoring one, and uses the fused subgraph to expand the query before reranking retrieved chunks. The method is evaluated on SQuAD, IIRC, Cultour, HotpotQA, and MuSiQue against several baselines. The central empirical claim is that QMKGF consistently outperforms all baselines across all datasets, with the headline result being a HotpotQA ROUGE-1 score of 64.98% versus 55.26% for BGE-Rerank. The paper also reports ablations on the attention mechanism, embedding fine-tuning, fusion strategy, subgraph size, reward-model backbone, and number of attention heads.","tokens_in":18066,"tokens_out":3335,"duration_ms":39281,"significance":"If the reported results are reproducible and statistically robust, the contribution is useful: the multi-path subgraph construction with reward-model-guided fusion is a nontrivial extension of graph-based RAG, and the evaluation covers five datasets with several non-tautological lexical metrics (ROUGE, BLEU, METEOR). The ablations in Tables 4 and 7 are directionally consistent with the claimed contributions. However, reproducibility is currently blocked by an undefined fusion threshold in Eq. (12) and by underspecified reward-model training data, and the absence of error bars or significance tests leaves the headline improvement of 9.72 points statistically ungrounded. These are load-bearing issues for the paper's central claim, but they appear fixable within the manuscript's scope.","major_comments":[{"comment":"The fusion rule is not fully specified: Eq. (10) defines r = cos(KGS_max, q), but Eq. (12) and Algorithm line 16 filter triples by sim((h,r,t), q) >= tau, and tau is never defined or related to r. The value of tau controls which triples from lower-scoring subgraphs enter G_fusion, and Table 7 shows that the fusion strategy materially changes results (RM fusion 64.98 vs. All fusion 64.07 ROUGE-1 on HotpotQA). As written, an independent implementation cannot decide which triples to retain, and the headline gain may depend on an undisclosed threshold. The authors must specify the value or selection procedure for tau, state whether it was tuned on a validation split, and report its chosen value.","section":"Section 3.3, Eqs. (10)-(12), and Algorithm line 16"},{"comment":"The reward model is central to selecting KGS_max and to the fusion step, but the training data are described only as 'scores assigned by LLMs' in terms of KG richness, question relevance, and connectivity. The manuscript does not report the dataset size, the prompt template, the number of LLMs, the score scale, or any agreement/quality checks on these labels. This is a reproducibility gap for a load-bearing component. The authors should also provide the final number of attention heads h used in the RM, since Fig. 6 varies h but no single final value is stated in Section 4.3.","section":"Section 3.3, reward model training"},{"comment":"Hyperparameters K and the RM attention-head count appear to be selected on the test sets: Fig. 5 reports optimal K=10 and Fig. 6 reports performance for different head counts, apparently on the evaluation subsets. With no validation split, no error bars, and no significance tests in Tables 2-3, the claim that QMKGF 'consistently outperforms all baseline models' is not statistically supported. The authors should report multiple runs with standard deviations, significance tests where appropriate, and a hyperparameter-selection protocol that does not use the test labels.","section":"Section 5.6-5.7 and Tables 2-3"},{"comment":"The ablation text states that removing personalized PageRank has a notable negative impact on English datasets, but Table 4 contains no row for 'w/o PageRank' — only w/o-attention and w/o-fintune. Either the missing ablation must be added or the claim removed, since the current text makes an assertion that the reported experiments do not support.","section":"Section 5.2 and Table 4"}],"minor_comments":[{"comment":"The text refers to 'Fig. 1' for the prompt templates and the subgraph-construction process, but the actual figures are Fig. 3 and Fig. 2 respectively; these cross-references should be corrected.","section":"Section 3.1 and Section 3.2"},{"comment":"The caption of Fig. 7 reads 'Effect of subgraph node count,' but Section 5.9 describes it as analyzing the effect of entity quantity in queries; the caption should match the content.","section":"Figure 7"},{"comment":"The dataset is called 'Culture' in the abstract and conclusion but 'Cultour' in Section 4.1 and Tables 2 and 5; one consistent name should be used throughout.","section":"Abstract and Section 4.1"},{"comment":"The sentence 'the improvement is less pronounced compared to IIRC' is confusing because Table 4 reports only HotpotQA and Cultour; it should probably refer to Cultour.","section":"Section 5.2"},{"comment":"There are several typos and formatting issues, including 'docment' in Eq. (1), the stray '♪' in reference [26], and inconsistent capitalization of 'BGE-rerank' vs. 'bge-rerank'; these should be cleaned up.","section":"Throughout"},{"comment":"The manuscript does not include a code or data availability statement, which would be valuable for a method whose reproducibility currently depends on several unspecified implementation details.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is reasonable and the experimental design is broad, but the undefined threshold tau and the underspecified reward-model training are genuine reproducibility blockers rather than minor omissions. I recommend major revision rather than rejection because the issues are fixable by reporting the missing values and protocols. If the authors cannot supply tau or the RM training details, the empirical claims should be treated as unverified. The paper may also benefit from a stronger statistical analysis before being considered for a journal-level acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a reasonable engineering paper with a load-bearing reproducibility gap. The multi-path subgraph construction (one-hop, multi-hop, PageRank) plus a query-aware reward-model fusion is a sensible combination, and the ablations suggest each component earns its keep. But the fusion step that distinguishes the method is underspecified, and the experimental reporting lacks the controls needed to trust the 9.72-point HotpotQA gain.\n\nWhat is genuinely new: I don't know prior work that combines the three subgraph paths with a query-aware attention reward model for triple-level fusion. The ablation in Table 4 shows both the attention mechanism and the embedding fine-tuning matter, and Table 7 shows RM fusion beats both all-fusion and top-5 fusion, which supports the central design choice. The consistent gains across five datasets, including two multi-hop QA sets, make the effect look real if the method can be pinned down.\n\nThe soft spots, in order:\n\n1. The tau problem is real and not cosmetic. Eq. (10) defines r = cos(KGS_max, q) as a threshold, but Eq. (12) and Algorithm line 16 filter triples by sim((h,r,t),q) >= tau, with no definition of tau and no stated relationship to r. Since Table 7 shows the fusion strategy materially changes results, an independent implementation literally cannot decide which triples to fuse. This blocks verification of the central claim.\n\n2. The reward model training data is described only as 'scores assigned by LLMs' for KG richness, relevance, and connectivity. No dataset size, prompt template, or examples are given, and there is no validation that these LLM judgments correlate with downstream answer quality. That is a second, independent reproducibility gap.\n\n3. No error bars or significance tests anywhere. On HotpotQA the gap over KG2RAG is about 6 points, which is large, but single-run numbers on a 1,000-sample subset are not enough to know if the ordering is stable.\n\n4. Test-set tuning is visible: K in Fig. 5 and attention head counts in Fig. 6 are chosen on the test data, which inflates reported gains. Also, Section 5.3 shows embedding fine-tuning gives large retrieval improvements, but the baselines are not given the same fine-tuning, so part of QMKGF's advantage over BGE-rerank may come from an asymmetry in the setup.\n\nThese are all fixable in a revision, and none of them kills the core idea. This paper deserves a serious referee. I would send it to review, asking specifically for a defined tau, RM training details, error bars, and a baseline-fairness check. It is also a decent reading-group case study in what reproducibility means for RAG systems.","headline":"A plausible engineering contribution with a real reproducibility hole: the fusion threshold tau is never defined, so the headline gains cannot be verified as written.","tokens_in":18678,"tokens_out":1641,"would_cite":false,"duration_ms":21013,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A query-aware knowledge-graph fusion method for RAG reports the best scores on all five QA benchmarks it tests, including a 9.72-point gain over reranking on HotpotQA.","keywords":["Retrieval Augmented Generation","Knowledge Graph","Query-Aware Attention","Reward Model","Subgraph Fusion","Query Expansion","Multi-Hop Question Answering","Large Language Models"],"falsifier":"On HotpotQA, replace the reward model with a random subgraph selector and run the identical pipeline; if ROUGE-1 stays near 64.98%, the query-aware fusion is not responsible for the reported gain.","tokens_in":17547,"feed_emoji":"🔗","tokens_out":10057,"duration_ms":94557,"temperature":0.7,"pith_summary":"The paper proposes QMKGF, a retrieval-augmented generation method that builds a knowledge graph from the document collection and expands the user query with entities, relations, and triples drawn from three kinds of graph subgraphs: one-hop neighbors, multi-hop paths, and importance-ranked nodes. The central claim is that fusing these subgraphs under the guidance of a query-aware reward model produces more relevant retrieved passages and better answers than standard embedding retrieval, reranking, single-path graph methods, or unfiltered graph fusion. On HotpotQA the method reports a ROUGE-1 score of 64.98%, a gain of 9.72 percentage points over the reranking baseline, and it reports the best scores among the compared methods across all five datasets. The underlying insight is that connections between retrieved pieces of evidence carry information that isolated text chunks miss, and that a query-sensitive selection step can keep those connections while filtering noise.","feed_headline":"Knowledge graph fusion beats reranking RAG by 9.7 points","feed_subtitle":"A query-aware reward model filters and merges three kinds of knowledge-graph subgraphs before retrieval.","key_machinery":"The load-bearing object is the multi-path subgraph construction and the query-aware attention reward model that scores and fuses the subgraphs. The reward model is a small trained head on a pretrained encoder that takes query and subgraph representations, applies multi-head attention with the query as the query vector, and outputs a scalar score; training data come from large-language-model judgments of each subgraph's KG richness, question relevance, and connectivity. Fusion uses the cosine similarity between the top subgraph and the query as a reference threshold and keeps triples from lower-scoring subgraphs whose similarity to the query meets that threshold, producing $G_{fusion}$. This mechanism converts graph structure into query-specific context before any document retrieval happens.","core_discovery":"QMKGF's central claim is that answer quality in RAG improves when retrieval is driven by a fused knowledge-graph subgraph rather than by isolated chunks or any single graph path. Entities and relations are extracted from the document collection with prompts to a large language model, producing a KG and an entity vector database. For a query, mapped entities anchor three subgraphs: top-$K$ one-hop neighbours, expansions through the two most relevant neighbours to second hop, and top-$K$ nodes by personalized PageRank. A query-aware attention reward model scores each subgraph's triples by semantic relevance to the query; the highest-scoring subgraph forms the backbone, and triples from the other subgraphs whose similarity to the query clears a threshold are added. The fused subgraph's entities, relations, and triples are concatenated with the original query to retrieve and rerank document chunks, which are then passed to the LLM. The paper claims this consistently outperforms all baselines on SQuAD, IIRC, Cultour, HotpotQA, and MuSiQue, with the largest gain on HotpotQA (64.98% ROUGE-1 versus 55.26% for the reranking baseline).","pith_inferences":["If the large-language-model judgments used to train the reward model are valid proxies for answer quality, the same query-aware fusion could be applied at sentence or paragraph level inside a document, not just on KG triples, to reduce noise in long-context retrieval.","A testable extension is to make the fusion threshold in Eq. (12) adaptive to the query instead of fixed, since the reported method does not specify how the threshold is set.","The approach implies the extracted KG's quality is the ceiling: on noisy or poorly extracted relations, the reward model's filtering becomes more influential, so robustness to extraction error is a natural stress test.","The method might reduce the context-window pressure in long-document QA by replacing many raw chunks with a compact, query-aligned subgraph; this is not tested in the paper."],"forward_implications":["Multi-hop reasoning benefits most: the reported gains on HotpotQA, IIRC, and MuSiQue are larger than on single-hop SQuAD, suggesting fused multi-path context helps cross-document inference.","Query expansion with KG triples improves retrieval recall: the fine-tuned embedding experiments and the 99.4% hit@10 on Cultour indicate that the expanded query retrieves a wider, more relevant document set.","Reward-model fusion beats both fusing all triples and fusing a fixed top-5, so query-aware filtering appears to remove noise rather than merely add more context.","Subgraph size matters: $K=10$ nodes outperforms $K=5$ and $K=20$, indicating a balance between semantic coverage and redundancy."],"supporting_citations":[{"why":"Supplies HotpotQA, the dataset on which the paper reports its headline 9.72-point ROUGE-1 gain over the reranking baseline.","marker":"[36]"},{"why":"Provides the reranking baseline and general-purpose embedding resources that QMKGF compares against and fine-tunes.","marker":"[33]"},{"why":"Serves as the graph-rerank baseline that QMKGF extends and outperforms across the benchmark sets.","marker":"[28]"},{"why":"Provides the KG-guided RAG baseline that QMKGF surpasses by more than 6 ROUGE-1 points on HotpotQA.","marker":"[41]"},{"why":"Supplies the query-expansion idea that the paper's KG-based query expansion builds on.","marker":"[27]"},{"why":"Provides the pretrained encoder used as the reward-model backbone in the main experiments.","marker":"[24]"},{"why":"Supplies the Cultour dataset used for Chinese-domain evaluation and for the subgraph-size and reward-model ablations.","marker":"[29]"},{"why":"Supplies the IIRC incomplete-information dataset, one of the five benchmarks in the main comparison.","marker":"[6]"}],"fun_headline_variants":["Query-aware KG fusion boosts RAG by 9.7 points","Fused KG subgraphs beat reranking in RAG by 9.7","Multi-path KG fusion lifts RAG, plus 9.7 ROUGE","Query-aware subgraph fusion improves RAG, +9.7","KG fusion tops reranking RAG with 9.7-point gain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method assumes that the reward model, trained on large-language-model judgments of graph richness, question relevance, and connectivity, correctly identifies which subgraphs and triples improve the final generated answer; the paper does not test that correlation.","fun_headline_variants_meta":{"raw":{"variants":["Query-aware KG fusion boosts RAG by 9.7 points","Fused KG subgraphs beat reranking in RAG by 9.7","Multi-path KG fusion lifts RAG, plus 9.7 ROUGE","Query-aware subgraph fusion improves RAG, +9.7","KG fusion tops reranking RAG with 9.7-point gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000421,"raw_usage":{"total_tokens":2254,"prompt_tokens":1128,"completion_tokens":1126,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":744,"completion_tokens_details":{"reasoning_tokens":1028}},"tokens_in":744,"tokens_out":1126,"duration_ms":11080,"temperature":1.0,"reasoning_tokens":1028,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:43:57.791849+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On HotpotQA, replace the reward model with a random subgraph selector and run the identical pipeline; if ROUGE-1 stays near 64.98%, the query-aware fusion is not responsible for the reported gain.","supporting_citations":[{"cited_title":"QCG-Rerank: Chunks Graph Rerank with Query Expansion in Retrieval-Augmented LLMs for Tourism Domain","cited_arxiv_id":"2411.08724","evidence_quote":"Serves as the graph-rerank baseline that QMKGF extends and outperforms across the benchmark sets."},{"cited_title":"IIRC: A Dataset of Incomplete Information Reading Comprehension Questions","cited_arxiv_id":"2011.07127","evidence_quote":"Supplies the IIRC incomplete-information dataset, one of the five benchmarks in the main comparison."}],"review_version":1}