{"id":"9e7b101f-59c9-4793-bdcb-77cc5ca413ed","arxiv_id":"2505.19410","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SRP combines reference retrieval, relation checking, and iterative self-reflection to improve LLM question answering over knowledge graphs, reporting gains over Readi on WebQSP, CWQ, and GrailQA.","lead":"A new LLM framework, Self-Reflective Planning (SRP), plans and corrects knowledge graph search paths for question answering by first retrieving similar solved questions and then reflecting on whether retrieved facts answer the question. It reports higher accuracy than several comparable methods on three knowledge graph QA benchmarks, though the gains are modest and some comparisons use cited rather than rerun baselines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never measures the accuracy of the sequence judge in §4.5, and a false 'have answer' stops reflection with no recovery; this unmeasured stopping rule is load-bearing for the claimed reliability advantage over Readi.","rationale":"The reader's conditional verdict and weakest-assumption analysis point to the sequence judge's 'have answer' decision as the fragile premise, and my reading agrees. Section 4.5 makes that decision the sole termination gate for the reflection loop, and the paper never evaluates the judge against ground truth. The claimed reliability improvement over Readi depends on this gate being trustworthy; a premature 'have answer' yields a wrong or incomplete answer, while the answering prompt's permission to fall back on parametric knowledge can mask retrieval failures and confound the reported reliable answering rate in Section 5.4. The concrete test I propose would settle whether this concern lands by measuring judge precision/recall and comparing SRP to an oracle-stop version. If the oracle version does not improve Hits@1 appreciably, the judge is not the bottleneck and the conditional acceptance can remain. If it does improve, the paper needs to report judge diagnostics or add a verification step before claiming reliability. This does not move the overall verdict because the reader already conditioned acceptance on precisely this gap; it reinforces the condition rather than discovering a new fatal flaw. I am not treating the absence of error bars or the GrailQA subset choice as the primary objection, though both are real secondary concerns; the judge reliability issue is more directly connected to the paper's central reliability claim and to the mechanism that differentiates SRP from Readi.","tokens_in":1858,"tokens_out":1059,"duration_ms":68754,"concrete_test":"Take a random sample of 200 test questions (100 WebQSP, 100 CWQ). Instrument SRP to log, for each question, every triplet sequence S the sequence judge evaluates and the judge's <HAVE_ANSWER>/<NO_ANSWER> output. For the stopping round, have a human or SPARQL oracle label whether S actually contains enough triples to derive the gold answer; compute judge precision and recall. Then rerun SRP with an oracle stop rule that never accepts on a false-positive round and continues editing until the gold answer is present (with a hard round cap). If oracle-stop Hits@1 exceeds the reported SRP Hits@1 by more than about 2 points, premature judge acceptance is a real reliability bottleneck; if not, the judge is adequate for the headline claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central reliability claim rests on the sequence judge in §4.5: when it outputs 'have answer', path editing stops and SRP answers from the current pruned sequence. No experiment in the paper measures this judge's precision, recall, or agreement with an oracle label of whether the retrieved triplet sequence actually entails the gold answer, and no maximum reflection round is specified. A false 'have answer' on a path like the Figure 1 failure would terminate reflection at a pruned sequence that lacks the answer. The answering prompt (Appendix C, Table 10) explicitly allows the model to use its own knowledge when the triples are insufficient, so the final answer could be supplied from parametric memory even when the retrieval did not support it. This same prompt threatens the 'reliable answering rate' in §5.4 and Figure 3, whose computation is not defined and could count answers not grounded in triples. The ablation in Table 3 shows removing self-reflection costs roughly 1.6–3.2 points, so the mechanism is not decorative; without judge-reliability data, the central reliability gain over Readi is unsupported as stated. This is an empirical gap, not an internal contradiction, but it is the weakest load-bearing step of the argument.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Self-Reflective Planning (SRP), a prompting pipeline for KGQA over Freebase. Given a question and a topic entity, SRP retrieves similar reference cases consisting of questions, reasoning paths, and answers from the training set; checks the candidate 1-hop relations of the topic entity; generates an initial multi-hop reasoning path; instantiates it as a triplet sequence via BM25/Contriever retrieval; and then iteratively judges and edits the path until the sequence judge emits a \"have answer\" verdict, after which a chain-of-thought answer is generated from the retained triples. Experiments on WebQSP, CWQ, and a GrailQA subset compare SRP with fine-tuned, LLM-only, and prompting baselines; the paper reports state-of-the-art Hits@1 on WebQSP and CWQ, second-best overall on GrailQA, and higher searching-success and reliable-answering rates than Readi. Ablation studies show that removing relation check, self-reflection, or references lowers accuracy, and a case study illustrates path editing behavior.","tokens_in":16335,"tokens_out":4716,"duration_ms":44482,"significance":"If the reported results are robust, SRP is a useful empirical contribution to KGQA prompting. The relation-check step and the reflection loop are clearly motivated, and the ablation in Table 3 shows that each component contributes non-trivially, so the framework is not merely a wrapper around the baseline. Comparing Readi on the same GPT-3.5 and GPT-4.1-mini backbones is a fairer comparison than many prior works. The paper is empirical rather than theoretical: it provides no machine-checked proofs and the code is only an anonymous repository link, so the value rests on the reliability of the measured gains and on the operational definitions of the proposed reliability metrics. The central claims are plausible but currently rest on several unquantified design choices, most notably the sequence judge's stopping rule and the definition of the reliable answering rate.","major_comments":[{"comment":"The \"reliable answering rate\" is not defined precisely enough to support the reliability claim. The text says it is \"the proportion of correct answers supported by factual triples from the KG,\" but it does not specify how \"supported\" was determined, who annotated it, or how the values in Figure 3 were computed. This matters because the answering prompt in Appendix C (Table 10) explicitly instructs the model to use its own knowledge when the triples are insufficient; a correct answer can therefore be produced without any grounding in the retrieved triples. Please provide an operational definition, an annotation procedure, and preferably per-case examples and error analysis showing that the reported reliable answering rate excludes answers generated from parametric memory.","section":"§5.4, Figure 3"},{"comment":"The sequence judge's \"have answer\" verdict stops reflection with no recovery path, and the accuracy of this judge is never measured. A false \"have answer\" on a pruned sequence that lacks the true answer terminates the loop prematurely and can lead to a wrong or incomplete final answer. The Table 3 ablation shows that removing self-reflection costs about 1.6–3.2 points on WebQSP and CWQ and 2.5 points on GrailQA, so the mechanism is material to the reported gains. The paper should report judge precision and recall against an oracle label (e.g., whether the pruned triplet sequence entails the gold answer), the distribution of reflection rounds, and a maximum-round policy, or otherwise demonstrate that premature stopping is not a dominant failure mode.","section":"§4.5, Reflection and Reasoning"},{"comment":"The reference base is constructed from the training set, and each reference includes the question, the reasoning path, and the correct answer. Because these references are injected into the relation-check, sequence-judge, and path-edit prompts, the inference-time guidance includes gold-answer information for similar training questions. The paper does not quantify how much of the gain comes from this answer leakage rather than from the proposed planning and reflection mechanisms. Please ablate references with the answers removed (paths only) or with the answers withheld, and clarify whether this constitutes a form of supervision at inference time rather than a purely training-free prompting method.","section":"§4.2, Reference Searching and Appendix C prompts"},{"comment":"GrailQA results are reported on \"the same test samples established in ToG,\" but several baseline numbers in Table 2, such as TIARA and Flexkbqa, are cited from their original publications and may use the full GrailQA test set or a different subset. If the baseline values are not all computed on the same 1,000-question subset, the claim of \"second-highest overall performance on GrailQA\" is not supported. Please either recompute all baselines on the same subset or clearly flag which comparisons are not directly comparable.","section":"§5.1, Table 2 and Appendix B"},{"comment":"All main results appear to come from single runs at temperature 0.3, with no standard errors, repeated seeds, or significance tests. The reported gains over Readi on WebQSP and CWQ are 2.7–8.8 points, and without variance information it is hard to judge whether the state-of-the-art claim is stable across sampling variability. Please report multiple runs or statistical significance measures, and disclose the API call dates and model versions used.","section":"§5.2, Table 2"}],"minor_comments":[{"comment":"There are several typos: \"dataets\" in the Table 2 header, \"GraliQA\" in Table 3 and Figure 3, and \"theree datasets\" in the Figure 3 caption.","section":"Table 2, Table 3, Figure 3"},{"comment":"\"Contriver\" should be \"Contriever\" in both the Knowledge Retrieval description and the Implementation Details paragraph.","section":"§4.4 and §5.1"},{"comment":"The expected output tag appears as \"<HA VE_ANSWER>\" with a space in the prompt text. If the actual prompt contains this token, the sequence judge may never emit the intended string; please correct this typo.","section":"Appendix C, Table 8"},{"comment":"The sentence \"we extract 100 questions and corresponding references from each dataset\" should specify how the 100 questions are selected from the clusters and should explicitly confirm that they come from the training split and are disjoint from the test questions.","section":"§5.1"},{"comment":"The statement \"Because we use data from training set, this limitation doesn't influence the performance of SRP\" is too strong; using training-set-derived references can still overfit to the benchmark distribution. Please soften this and discuss how the method would behave under domain shift.","section":"Limitations"},{"comment":"The \"searching success rate\" is described as \"how efficiently each approach retrieves answer of question from the KG,\" but the computation is not defined. Please specify whether success requires exact match of the gold answer entity, containment in the retrieved sequence, or some other criterion.","section":"§5.4, Table 4"}],"recommendation":"major_revision","confidential_remarks":"This is a straightforward empirical systems paper. The main scientific risk is the unmeasured sequence-judge reliability and the undefined reliable-answering metric; the requested analyses are feasible within the manuscript's scope. I also see a disclosure concern: the reference base includes gold answers from the training set and is injected into the inference prompts, which is a form of inference-time supervision. The GrailQA comparison needs a common test subset before the second-best claim can be taken at face value. I recommend major revision rather than rejection because the central direction is sound and the deficits are empirical gaps that the authors can address with additional experiments and clarifications."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: the accuracy numbers are probably real, but the 'reliability' claim is softer than it looks. The SRP pipeline—KNN reference retrieval, relation check, iterative path reflection—does beat Readi on WebQSP and CWQ with both GPT-3.5 and GPT-4.1-mini, and the ablations show each piece pulls its weight. That is a solid incremental contribution.\n\nWhat's new is the specific combination. None of the components is novel by itself: reference retrieval resembles in-context learning, relation check is reminiscent of ToG's relation pruning, and self-reflection is Readi/Reflexion territory. But the paper shows the combination works across two backbones and three datasets, and the code is available. The case studies are actually informative.\n\nThe soft spots are the ones the reader flagged. The sequence judge in Section 4.5 is load-bearing: if it says 'have answer' when it shouldn't, reflection stops and the model answers from a pruned sequence that lacks the answer. The paper never measures judge precision or recall, doesn't specify a maximum number of reflection rounds, and the answering prompt in Table 10 explicitly lets the model fall back on its own parametric knowledge. That makes the 'reliable answering rate' in Section 5.4 hard to interpret—it is never precisely defined, and it could count answers that came from the model's memory rather than from the triples. The ablation shows removing self-reflection costs 1.6–3.2 points, so the mechanism matters; without judge-reliability data, the reliability advantage over Readi is not fully established.\n\nOther issues are minor but worth noting: no error bars or repeated runs, GrailQA uses ToG's test subset while some baselines come from earlier papers, and there's no same-backbone ToG comparison with GPT-4.1-mini. These don't sink the paper; they just bound how strongly you can read the comparisons.\n\nOverall, this is a competent empirical paper with a modest but useful contribution. I'd send it to review, but ask for three things: a definition and measurement of the judge's accuracy, a clearer grounding metric that separates triple-grounded answers from parametric ones, and repeated runs or confidence intervals. If those come back okay, the reliability claim will actually be supported.","headline":"A plausible incremental KGQA pipeline whose accuracy gains are credible, but the reliability headline rests on an unmeasured stopping rule and a vague grounding metric.","tokens_in":16810,"tokens_out":2079,"would_cite":true,"duration_ms":19438,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that letting an LLM judge and edit its own knowledge-graph reasoning paths until the answer is found makes question answering more accurate and more factually grounded.","keywords":["knowledge graph question answering","self-reflection","self-reflective planning","LLM reasoning","reasoning path planning","multi-hop reasoning","retrieval-augmented generation","self-correction"],"falsifier":"Measure the sequence judge's precision on the three datasets: run SRP on a sample of questions, record each 'have answer' verdict, and check whether the pruned triples actually contain the gold answer. If a substantial share of 'have answer' stops occur on wrong or incomplete triples, the reliability gain over Readi would shrink or disappear when the loop is forced to continue.","tokens_in":15887,"feed_emoji":"🔁","tokens_out":10058,"duration_ms":79774,"temperature":0.7,"pith_summary":"Large language models asked to answer questions over a knowledge graph often generate reasoning paths that look plausible but miss the relation the question actually needs. This paper claims that adding a self-reflection loop—where the model judges the triples it retrieved, prunes them, and edits its path when the answer is absent—makes the reasoning reliable enough to beat strong baselines on three benchmark datasets. The proposed framework, Self-Reflective Planning (SRP), also checks the topic entity's one-hop relations before planning and pulls in similar solved examples as references. If the claim holds, a training-free prompting method can rival or exceed fine-tuned KGQA systems while keeping answers traceable to graph triples.","feed_headline":"Self-reflection boosts knowledge-graph QA by up to 8.8 points","feed_subtitle":"SRP has the model judge and edit its own graph-search paths until answers are grounded in verified triples.","key_machinery":"The load-bearing mechanism is the reflection loop: a sequence judge that inspects the retrieved triplet sequence, prunes irrelevant triples, and emits a 'have answer' or 'no answer' verdict, paired with a path editor that rewrites the reasoning path when the verdict is negative. The loop is seeded by two planning aids: a reference search that retrieves similar solved questions with their reasoning paths, and a relation check that scores the topic entity's one-hop relations so the initial path starts from relations that actually exist in the graph. The loop stops only when the judge says the answer is present, and then the model answers from the pruned sequence.","core_discovery":"SRP's central claim is that reliability in KGQA comes from treating the reasoning path as editable state, not a one-shot prediction. The model first retrieves reference cases, scores the 1-hop relations of the topic entity to seed the path, generates a path, and instantiates it into triplet sequences via semantic retrieval. A sequence judge then decides whether the answer is present; if not, a path editor replaces the offending relation using candidate relations from the graph, and the loop repeats. On WebQSP and CWQ, SRP with GPT-4.1-mini reports Hits@1 of 83.6 and 69.0, respectively, surpassing the Readi baseline by 2.7 and 8.8 points, and on GrailQA it reports 78.8 overall, second only to ToG-GPT4 among the compared methods. The paper also reports higher searching success rates and reliable answering rates than Readi, arguing that SRP's correct answers are more often grounded in actual KG triples.","pith_inferences":["An extension the paper does not run is to measure the sequence judge's precision and recall directly; if the judge often says 'have answer' before the true answer is present, the loop's stopping rule is the weakest point.","The same judge-edit loop could transfer to other structured reasoning environments such as SQL databases or code execution, where a path is a query or a program and the judge checks intermediate results.","Because the reference base is drawn from each dataset's training set, domain-shift robustness is an open question; on out-of-distribution questions the benefit of reference search may shrink or require adaptive reference selection.","SRP's own stated limitation—higher LLM call counts from iterative reflection—points to a testable cost-benefit question: whether the accuracy gain over Readi persists when measured per API call or per unit of latency."],"forward_implications":["If SRP's reported numbers hold, a prompting-only LLM can beat fine-tuned KGQA models such as TIARA and Flexkbqa on WebQSP and CWQ without retraining.","The relation check and the reference search are load-bearing: removing them drops WebQSP accuracy from 78.5 to 76.9 and 75.7, respectively.","The reflection loop is especially valuable on complex multi-hop questions: on CWQ, removing self-reflection lowers Hits@1 from 58.7 to 55.5 for GPT-3.5.","SRP's reliable answering rate on WebQSP reaches 92.2% with GPT-4.1-mini, versus 70.5% for Readi, implying correct answers are more frequently supported by graph triples."],"supporting_citations":[{"why":"Supplies the retrieval-by-semantic-similarity method that SRP uses to instantiate reasoning paths, and defines the Readi baseline SRP must beat.","marker":"(Cheng et al., 2024)"},{"why":"Supplies the ToG prompting baseline, the GrailQA test-sample setup, and the relation-check prompt demonstration SRP adapts.","marker":"(Sun et al., 2023)"},{"why":"Supplies the chain-of-thought prompting style used in SRP's answering step.","marker":"(Wei et al., 2022)"},{"why":"Supplies the sentence embedder used to construct and search the reference base.","marker":"(Reimers and Gurevych, 2019)"},{"why":"Supplies the dense retriever used with BM25 to match predicted relations to Freebase relations.","marker":"(Izacard et al., 2022)"},{"why":"Defines the WebQSP benchmark where SRP reports its largest absolute gains.","marker":"(Yih et al., 2016)"},{"why":"Defines the CWQ benchmark whose multi-hop questions show the biggest relative improvement for SRP.","marker":"(Talmor and Berant, 2018)"},{"why":"Defines the GrailQA benchmark and its I.I.D., compositional, and zero-shot generalization splits.","marker":"(Gu et al., 2021)"},{"why":"Supplies Freebase, the knowledge graph that all methods in the comparison retrieve from.","marker":"(Bollacker et al., 2008)"},{"why":"Supplies the reported ToG-GPT3.5 and ToG-GPT4 numbers that SRP compares against.","marker":"(Chen et al., 2024)"}],"fun_headline_variants":["Self-reflective paths lift KGQA up to 8.8 points","LLMs edit their own graph paths for better QA","Iterative reflection grounds LLM answers in KG triples","SRP: model judges and fixes its reasoning paths"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method depends on the model's own 'have answer' verdict being reliable enough to stop the reflection loop; the paper does not measure how often the judge stops early on a path that lacks the true answer.","fun_headline_variants_meta":{"raw":{"variants":["Self-reflective paths lift KGQA up to 8.8 points","LLMs edit their own graph paths for better QA","Iterative reflection grounds LLM answers in KG triples","SRP: model judges and fixes its reasoning paths"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000154,"raw_usage":{"total_tokens":1196,"prompt_tokens":917,"completion_tokens":279,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":211}},"tokens_in":533,"tokens_out":279,"duration_ms":3375,"temperature":1.0,"reasoning_tokens":211,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:14:26.764315+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the sequence judge's precision on the three datasets: run SRP on a sample of questions, record each 'have answer' verdict, and check whether the pruned triples actually contain the gold answer. If a substantial share of 'have answer' stops occur on wrong or incomplete triples, the reliability gain over Readi would shrink or disappear when the loop is forced to continue.","supporting_citations":[],"review_version":1}