{"id":"ce144b81-ee8d-4bca-a977-0cb04874c287","arxiv_id":"2412.05453","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Using LLM-generated knowledge graphs to guide question decomposition modestly improves GPT-4's success rate on 100 high-school physics questions, but the evidence is informal and the dataset is not released.","lead":"The paper tests a question-answering pipeline that turns a physics problem into a knowledge graph, then uses the graph to generate sub-questions before producing a final answer. It reports small gains over plain prompting in a 100-question human evaluation and a five-student preference survey.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains in Table 1 are within sampling error: with n=54 and n=46, the 6.8- and 6.0-percentage-point improvements are not statistically significant, so the abstract's 'significantly improved' claim is unsupported.","rationale":"The reader's weakest assumption targets the representativeness of the 100-question subset and the reliability of human judgments. My stress-test found a more specific, load-bearing flaw: even if the subset is perfectly representative and the judgments are noise-free, the reported success rates cannot support the word 'significantly' because the effect sizes are within the sampling error of paired proportions. For the numerical category, 77.62% vs 84.45% on 54 paired questions gives a difference of about 6.8 percentage points, with a standard error of roughly 7 percentage points; the conceptual comparison is similar. The correct statistical test for paired binary outcomes (McNemar's test) is absent, as are confidence intervals and per-question data. The paper's own Limitations section explicitly disclaims a quantitative metric, saying only 'significant qualitative improvements' were noticed. Thus the central claim of the abstract and conclusion is not established by the evidence as presented, independently of any concern about question selection or annotator subjectivity. This supports, rather than changes, the reader's REJECT verdict. I marked agreement as 'partial' because my concern is not identical to the reader's: it focuses on sampling error of the observed paired differences rather than on selection bias or judgment noise, though both point to the same missing statistical support.","tokens_in":10608,"tokens_out":4034,"duration_ms":40243,"concrete_test":"Request the per-question success/failure labels for the 100 questions under the three prompting methods, or re-run the evaluation with a pre-registered rubric and at least two independent annotators. Then compute McNemar's exact test on the paired 2x2 table comparing 'standard prompting' with 'decomposition with KG', separately for the 54 numerical and 46 conceptual questions, and report 95% Wilson confidence intervals for the success rates. If the two-sided p-value is above 0.05 in either category, or the confidence interval for the difference includes zero, the abstract's 'significantly improved' claim is unsupported; if p < 0.05 in both categories, the quantitative claim would be supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim, that KG-guided decomposition yields 'significantly improved fidelity' (Abstract), relies entirely on the human evaluation in Section 5.1. The design is paired: the same GPT-4 model answers the same 100 hand-selected questions under three prompts, and success is judged manually with no rubric, no inter-annotator agreement, and no blinding. The decisive problem is quantitative, not qualitative: the observed gains are within sampling error. Numerical success rises from 77.62% to 84.45% (roughly 42/54 to 46/54) and conceptual from 68.53% to 74.58% (roughly 32/46 to 34/46); the gains are 6.8 and 6.0 percentage points. For paired binomial outcomes at these sample sizes, the standard error of the difference is roughly 5-8 percentage points, so the improvements are about one standard error, and the difference between KG decomposition and decomposition-without-KG is even smaller (+3.1/+3.3 pp). A McNemar test on the paired 2x2 tables would be the appropriate assessment; the paper reports none, nor confidence intervals, nor per-question outcomes. Section 7 concedes that no quantitative metric was used: 'Quantitatively evaluating such a pipeline is quite different in the absence of large amounts of pre-existing data' and only 'significant qualitative improvements' were noticed. Therefore the headline result is not distinguishable from chance on the numbers as presented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a pipeline for high-school physics question answering in which an LLM first constructs a knowledge graph from a question, uses that graph to generate sub-questions, answers the sub-questions, and synthesizes a final answer. The authors evaluate the approach against standard prompting and against decomposition without a knowledge graph on 100 hand-selected questions from an 8,000-question bank, using GPT-4 and human grading, and they supplement the results with a five-student preference survey. They report higher success rates for KG-guided decomposition (84.45% vs 77.62% for numerical questions; 74.58% vs 68.53% for conceptual questions) and claim that the generated sub-questions exhibit \"significantly improved fidelity\" to the original question's logic. The paper also contributes the augmented dataset of knowledge graphs and subqueries.","tokens_in":10818,"tokens_out":5639,"duration_ms":55512,"significance":"If the reported effect were real, KG-guided decomposition would be a cheap and prompt-level improvement for high-school physics QA, and the dataset of knowledge graphs and subqueries could be a useful resource for fine-tuning open-source models. The pipeline is clearly described, the control condition (decomposition without KG) is appropriate, and the worked example illustrates the intended behavior. However, the central quantitative claim is not statistically established: the evaluation is small, the selection and grading procedures are under-specified, and the claim about sub-question fidelity is not directly measured by the reported experiments. The paper's own limitations section explicitly concedes the absence of a well-framed quantitative evaluation, which is in tension with the abstract's claim of significant improvement.","major_comments":[{"comment":"Table 1 reports improvements of 6.8 and 6.0 percentage points on 54 numerical and 46 conceptual questions, but no confidence intervals, significance tests, or per-question outcomes are reported. With paired binomial data at these sample sizes, the standard error of the difference is on the order of 5–8 percentage points; the observed gains are therefore not distinguishable from chance without a McNemar test, which is not provided. The word \"significantly\" in the Abstract, Section 8, and Section 9 is thus unsupported by the quantitative evidence as presented.","section":"§5.1, Table 1"},{"comment":"The headline claim concerns \"fidelity to the original question's logic\" of the generated sub-questions, but the human evaluation in Section 5.1 measures only whether the final numerical or conceptual answer is judged correct; no rubric or score for sub-question fidelity is described. The student survey in Section 6 rates clarity, consistency, and helpfulness, which is a preference measure rather than a fidelity measure. The principal claim is therefore not directly evaluated by the reported quantitative results.","section":"§5.1 vs. Abstract"},{"comment":"Section 7 states that \"Quantitatively evaluating such a pipeline is quite different in the absence of large amounts of pre-existing data\" and that only \"significant qualitative improvements\" were noticed. This statement directly contradicts the quantitative significance asserted in the Abstract, Table 1, and Section 8. The authors should either supply the missing statistical analysis or present the finding as qualitative and exploratory.","section":"§7"},{"comment":"The human evaluation is described as a \"manual qualitative evaluation\" without a rubric, blinding, multiple annotators, or inter-annotator agreement, and the student survey uses only five students with no statistical treatment. These design features make it impossible to assess measurement reliability or to rule out experimenter bias. At minimum, the authors should provide annotation instructions, the number and background of annotators, agreement statistics, and a predefined success criterion.","section":"§5.1 and §6"}],"minor_comments":[{"comment":"The question bank size is described as \"approximately 8,000\" in Section 3.1 and as \"8000\" in Section 3.2; please make these numbers consistent.","section":"§3.1 vs. §3.2"},{"comment":"Several references are incomplete or misattributed (for example, \"Michael Petrochuk et al. 2022. n/a\" and a Dasgupta et al. entry whose title does not match the cited arXiv paper); these entries should be corrected before publication.","section":"References"},{"comment":"Please clarify whether the knowledge graph is generated by Gemini Pro (as in the dataset construction in Section 3.2) or by GPT-4 (as in the human evaluation in Section 5.1), since the two models could produce graphs of different quality and this affects reproducibility.","section":"§3.2 vs. §5.1"},{"comment":"The student survey reports average ratings without standard deviations, per-question breakdowns, or any indication of how the surveyed questions were selected; with n=5, the averages are highly sensitive to individual responses and should be presented as anecdotal.","section":"§6"},{"comment":"Figure 3 is not referenced in the main text; please add a citation where the survey results are discussed.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript relies heavily on self-citations to prior work from the same group; this is not a defect per se, but the related-work section would read better with more independent citations. Given the small-scale evaluation and the mismatch between the claim and the measured outcome, the paper may be better suited to a workshop or an applied-education venue than to a general CL journal in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The paper describes a clean, plausible pipeline: use GPT-4 to build a knowledge graph from a physics question, generate subquestions from that graph, answer them, and synthesize. That specific combination is mildly new, and the authors contribute a dataset of 8,000 physics questions with KG/subquery annotations. The worked example in Section 5.1.3 shows the intended mechanism, and the KG-guided version gives the right answer where standard prompting and decomposition-without-KG hallucinate. Credit where due: the idea is easy to understand and the dataset could be useful for fine-tuning or benchmarking.\n\nThe soft spots are real and they are in the empirical core. The headline claim of 'significantly improved fidelity' rests on a 100-question hand-selected set, judged by humans with no rubric, no blinding, and no inter-annotator agreement. No confidence intervals, no significance tests, no per-question outcomes. The stress-test calculation is right: with n=54 and n=46, gains of 6.8 and 6.0 points are about one standard error; you cannot distinguish them from chance. The comparison against decomposition without KG is even smaller (+3.1/+3.3 pp). The student survey is five people. The Limitations section concedes that no quantitative metric was used. So as presented, the abstract's 'significantly' is not supported.\n\nThere are also smaller issues: the dataset and code are not released, which makes the dataset contribution hard to verify. The citation pattern is heavily self-referential, though the cited works are at least relevant to the pipeline.\n\nWho is this for? People working on prompt-level decomposition in educational NLP. The idea is worth a proper test, but the evidence in this version doesn't establish the gain. I would not cite it myself in its current form.\n\nFor peer review: I'd send it out, because the dataset and pipeline are concrete enough that a serious referee could push the authors to rerun with a larger sample, report McNemar tests, and release the data. If they can show a stable effect with proper statistics, this becomes a decent workshop or short-paper contribution. As-is, the empirical claim fails, so the review should be a major revision at best.\n\nRecommendation: don't desk reject, but the referee should make significance testing and data release a condition of acceptance.","headline":"A plausible pipeline paper whose headline claim of 'significantly improved fidelity' is not supported by the reported numbers—100 hand-selected questions, no significance tests, gains within sampling error.","tokens_in":11459,"tokens_out":2580,"would_cite":false,"duration_ms":26151,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that sub-questions generated from an LLM-built knowledge graph stay more faithful to the original question and that this raises GPT-4's success on high-school physics problems by about 7 points in a human evaluation…","keywords":["knowledge graph","question decomposition","physics question answering","LLM reasoning","GPT-4","sub-question generation","high school physics","prompt engineering"],"falsifier":"Conduct the same comparison on a randomly drawn, larger sample of questions (e.g., 500) with multiple blind annotators and pre-registered success criteria; if KG-guided decomposition no longer beats decomposition without a KG by a margin beyond chance, the central claim fails. A simpler check: measure inter-annotator agreement on the original 100 questions; low agreement would show the reported gains are not reliably measurable.","tokens_in":10325,"feed_emoji":"🧲","tokens_out":3675,"duration_ms":31334,"temperature":0.7,"pith_summary":"This paper tries to establish that asking an LLM to build a knowledge graph of a high-school physics question before splitting it into sub-questions produces sub-questions that stay truer to the original question's logic, and that this leads to better final answers. In a human evaluation of 100 GPT-4 responses, success on numerical problems rose from 77.62% with standard prompting to 84.45% with knowledge-graph-guided decomposition, and from 68.53% to 74.58% on conceptual problems. If right, the method is a cheap, prompt-only upgrade for physics QA that needs no fine-tuning or external retrieval. The paper also introduces a dataset of 8,000 physics questions with associated knowledge graphs and sub-queries, intended for fine-tuning open-source models.","feed_headline":"Knowledge graphs lift GPT-4 physics answers by 7 points","feed_subtitle":"Asking GPT-4 to draw the question's logic before decomposing it beats both baselines on a 100-question physics set.","key_machinery":"The pipeline's load-bearing object is the knowledge graph constructed by the LLM from the question's text. Nodes capture entities such as objects, materials, and given values; edges capture relationships such as 'has_bulk_modulus'. This graph is then used to prompt the model to generate sub-questions that follow the graph's structure, each sub-question is answered in a few-shot setting, and a synthesis module combines the sub-answers with the original question to produce the final response. The knowledge graph acts as a structural scaffold that keeps decomposition aligned with the question's internal logic, which the authors argue is what standard decomposition lacks.","core_discovery":"The central claim is that sub-questions derived from an LLM-generated knowledge graph exhibit significantly improved fidelity to the original question's logic, and that this fidelity translates into higher answer accuracy. The paper argues this by comparing three reasoning methods on GPT-4: standard prompting, decomposition without a knowledge graph, and decomposition guided by a knowledge graph. The measured success rates on a hand-picked set of 54 numerical and 46 conceptual questions were 84.45% and 74.58% respectively for the KG-guided method, beating both baselines. The authors attribute the gain to the knowledge graph anchoring the model to the relevant concepts and entities, preventing it from drifting into wrong formulas or hallucinations, especially on conceptual problems.","pith_inferences":["The reported gains could partly reflect the extra prompt structure itself rather than the knowledge graph per se; an ablation that uses an equivalent structured scaffold (e.g., a generic entity-relation list) would isolate the graph's specific contribution.","Because the knowledge graph is generated by the same model that later answers, errors in graph construction may propagate; testing with a fixed, human-verified graph versus an LLM-generated one would separate graph quality from decomposition quality.","The approach is a natural fit for retrieval-augmented educational tools: a KG-guided decomposition could name missing quantities or concepts and trigger targeted retrieval for conceptual sub-questions, where the paper reports the smallest gains.","If the effect replicates on larger and randomly sampled benchmarks, KG-guided decomposition could become a standard pre-processing step in LLM-based tutoring systems, since it requires no architectural changes."],"forward_implications":["The method can be applied to any LLM without fine-tuning, since it relies only on prompt construction around a generated knowledge graph.","Improvements are larger for numerical than conceptual questions, suggesting the graph helps most when the solution path follows distinct entities and formulas.","The released dataset of 8,000 physics questions with knowledge graphs and sub-queries enables fine-tuning of open-source models to emulate the decomposition behavior of larger models like GPT-4 and Gemini Pro.","The approach may transfer to other structured disciplines such as mathematics, where questions also decompose into named entities and relations.","Student survey ratings (4.4-4.8 out of 5 for KG-guided sub-questions) suggest the method also improves perceived clarity and helpfulness for learners, albeit on a very small sample."],"supporting_citations":[{"why":"Supplies the physics question bank of roughly 8,000 high-school questions that the dataset and all experiments are built on.","marker":"(Anand et al., 2023a)"},{"why":"Defines chain-of-thought prompting, the baseline reasoning technique this paper positions its decomposition approach against.","marker":"(Wei et al., 2022)"},{"why":"One of the traditional question-decomposition approaches that the paper claims produces less logically consistent sub-questions.","marker":"(Perez et al., 2020)"},{"why":"Another decomposition baseline for compositional reading comprehension that the paper compares its KG-guided method to.","marker":"(Xu et al., 2022)"},{"why":"A representative method for integrating knowledge graphs into language models, which the paper extends to question decomposition.","marker":"(Zhang et al., 2022b)"}],"fun_headline_variants":["KG-guided reasoning boosts GPT-4 physics scores by 7 pts","Knowledge graphs add 7 points to GPT-4 physics answers","LLM-built knowledge graphs improve physics QA by 7 points","How knowledge graphs sharpen GPT-4 on physics questions","KG-anchored GPT-4 beats baselines on physics problems"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hand-selected 100-question subset and the human judgments made on it provide a reliable estimate of the pipeline's effect, since the paper gives no random sampling, inter-annotator agreement, confidence intervals, or significance tests.","fun_headline_variants_meta":{"raw":{"variants":["KG-guided reasoning boosts GPT-4 physics scores by 7 pts","Knowledge graphs add 7 points to GPT-4 physics answers","LLM-built knowledge graphs improve physics QA by 7 points","How knowledge graphs sharpen GPT-4 on physics questions","KG-anchored GPT-4 beats baselines on physics problems"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000548,"raw_usage":{"total_tokens":2560,"prompt_tokens":832,"completion_tokens":1728,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":448,"completion_tokens_details":{"reasoning_tokens":1642}},"tokens_in":448,"tokens_out":1728,"duration_ms":11146,"temperature":1.0,"reasoning_tokens":1642,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:42:48.258582+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Conduct the same comparison on a randomly drawn, larger sample of questions (e.g., 500) with multiple blind annotators and pre-registered success criteria; if KG-guided decomposition no longer beats decomposition without a KG by a margin beyond chance, the central claim fails. A simpler check: measure inter-annotator agreement on the original 100 questions; low agreement would show the reported gains are not reliably measurable.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the traditional question-decomposition approaches that the paper claims produces less logically consistent sub-questions."}],"review_version":1}