{"id":"f9f6b758-b38d-41db-9265-40cd255f130d","arxiv_id":"2607.22652","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"KG2Code-QA represents knowledge-graph subgraphs as executable Python/NetworkX code and trains LLMs to complete the code, improving KGQA accuracy and cross-KG transfer under oracle retrieval.","lead":"This paper turns question-relevant knowledge-graph subgraphs into executable Python code and asks LLMs to complete that code to answer questions. On four KGQA benchmarks the trained code-writing models beat RAG, agent, and SPARQL baselines, though the retrieval setup uses gold query subgraphs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation uses oracle subgraphs from gold SPARQL queries, so the claimed KGQA advantage is not demonstrated for realistic retrieval; this is the paper's largest unaddressed assumption.","rationale":"The reader's weakest assumption matches this concern: the retrieval protocol in §4.1.5 is an oracle. I agree this is the single most load-bearing issue because the abstract and introduction make unqualified claims about outperforming existing KGQA methods, while all experiments condition on gold subgraphs. The error analysis independently confirms that subgraph coverage is the largest error source, so the reasoning interface cannot compensate for missing facts. The retrieval concern is more fundamental than other issues (e.g., missing prior citation, hard/soft answer analysis) because it affects external validity of the central claim. Verdict remains CONDITIONAL: the paper should either add an end-to-end retrieval evaluation or explicitly scope the claims. Since the reader already assigned CONDITIONAL for this reason, no verdict change is needed.","tokens_in":29526,"tokens_out":5699,"duration_ms":57272,"concrete_test":"Replace §4.1.5's oracle subgraphs with an automatic retriever (e.g., Falcon/ReFinED entity linking + BFS or PageRank subgraph extraction capped at 100 triples) and rerun KG2Code-QA and DirectPrompt on WWQ. Compute the F1 gap between the two methods under oracle and realistic retrieval. If the KG2Code advantage (≈9.4 F1 for Llama on WWQ) drops below half or reverses, the central claim is unsupported outside the oracle setting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1.5 states that ground-truth subgraphs are extracted by parsing gold SPARQL queries, then expanded with 1-hop noise up to 100 triples. Every headline comparison (Tables 2, 4, 5, 6) and the transfer experiments inherit this oracle setting; no automatic entity-linking/retrieval component is evaluated. The paper's own error analysis (Section 4.10) attributes 40% of errors to subgraph coverage, making retrieval the dominant error source. If a practical retriever returns incomplete or noisy subgraphs, the code-based reasoning advantage may shrink or vanish, because RAG baselines are also operating on the same oracle subgraph and the comparison isolates reasoning rather than end-to-end KGQA. The abstract and Section 1 claim 'KG2Code-QA significantly outperforms existing KG-enhanced LLM methods for KGQA' without the qualification 'given gold subgraphs.' This is the load-bearing premise: the central contribution is a reasoning interface, not a complete KGQA system, yet the headline claims system-level superiority.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes KG2Code, a representation that rewrites KG subgraphs as executable Python/NetworkX code, and KG2Code-QA, a KGQA framework that asks an LLM to complete a QA function over this code representation. The framework alternates natural-language comments with executable statements (ReAct-style), obtains hard answers by executing the generated code, and falls back to soft answers parsed from comments when execution fails. An automated pipeline constructs a large (over 200k examples) code-based training corpus from Wikidata, and the authors instruction-tune Llama-3.1-8B-Instruct and Qwen2.5-Coder-7B-Instruct with LoRA. Experiments on WikiWebQuestions, LC-QuAD 2.0, and transfer to WebQSP and GrailQA compare against SPARQL, RAG, agent, GNN, and CoT baselines, reporting consistent improvements in Precision, F1, and EM. The authors claim this is the first conversion of KGs into executable code for KGQA and that KG2Code-QA significantly outperforms existing KG-enhanced LLM methods while generalizing to unseen KGs.","tokens_in":29816,"tokens_out":3147,"duration_ms":35005,"significance":"If the result holds, the paper makes a useful contribution: a faithful, interpretable, code-grounded interface between KGs and LLMs, with a practical distillation pipeline that lets small open-source models outperform their closed-source teacher in the tested setting. The corpus construction with execution-based filtering, the use of an independent training corpus, and the transfer evaluation are all strengths. The central claim, however, is stated as an end-to-end KGQA result, while the evaluation protocol isolates the reasoning component by supplying oracle subgraphs derived from gold SPARQL queries. The contribution is therefore best understood as a strong reasoning/interface method, not yet a demonstrated complete KGQA system. With proper qualification and additional retrieval evaluation, this would be a solid paper.","major_comments":[{"comment":"The retrieval setting uses ground-truth subgraphs extracted by parsing gold SPARQL queries, expanded with 1-hop noise up to 100 triples. All headline comparisons in Tables 2, 4, 5, and 6, as well as the transfer experiments, inherit this oracle setting. No automatic entity linking or retrieval component is evaluated. The paper's own error analysis in §4.10 attributes 40% of errors to subgraph retrieval issues, making retrieval the dominant error source. Under a realistic retriever that returns noisier or incomplete subgraphs, the advantage of KG2Code-QA may shrink or vanish, since the RAG baselines receive the same oracle subgraphs and the comparison isolates reasoning rather than end-to-end KGQA. The abstract and Section 1 claim that 'KG2Code-QA significantly outperforms existing KG-enhanced LLM methods for KGQA' without this qualification. The authors should either evaluate with a prac","section":"§4.1.5 and §1"},{"comment":"For LC-QuAD 2.0, the test answers are obtained by executing the provided Wikidata SPARQL queries, and questions with no returned answers are excluded, leaving 4,932 of 6,046 test questions. This self-computed and filtered test set is used for all LC-QuAD comparisons. The filtering step may interact with model behavior (e.g., questions that produce empty SPARQL results may also be hard for KG2Code-QA), and the answer computation is not independently verified. The authors should report the unfiltered test size, provide the exact filtering criteria, and ideally release the computed test answers for reproducibility. Without this, the LC-QuAD results are difficult to compare against other published work.","section":"§4.1.1"},{"comment":"The paper repeatedly states that KG2Code-QA 'significantly outperforms' baselines, but no variance, confidence intervals, or statistical significance tests are reported. All results appear to be from a single evaluation run. Given that several differences in Tables 2 and 5 are small (e.g., WWQ EM: 64.08 vs 61.01 for CoT-Tuning; WebQSP Precision: 85.98 vs 85.68 for CoT-Tuning), the claimed significance is not supported by the evidence presented. The authors should add multiple seeds or bootstrap resampling and report significance tests, or soften the language to 'outperforms in the reported runs.'","section":"Tables 2, 4, 5"}],"minor_comments":[{"comment":"The running example is inconsistent: Figure 1 states 'Answer: 1' for the question about screenwriters who died in Rome, while Figure 2's code computes two screenwriters died in Rome and returns 2. Please reconcile the example.","section":"Figures 1–2"},{"comment":"In the KG2Code-QA code response, the second loop contains a duplicated 'for u, v, data in graph.edges(data=True):' line, which appears to be a typo. Also the response includes 'return fields' instead of returning the final answer, but the claimed answer is correct; clarify.","section":"Table 10"},{"comment":"Tables B.12, B.13, and B.14 are referenced in the text but appear as empty placeholders in the provided manuscript. Please include the actual example code blocks.","section":"Appendix B"},{"comment":"The corpus generation section states 'over 200,000' training examples, but no exact number is given. Please provide the exact corpus size and the number of examples per question type, as this affects the reproducibility of the instruction-tuning results.","section":"§3.5"},{"comment":"The ToG, GoG, and DoG baselines are all run on the same retrieved subgraphs as KG2Code-QA. This is a reasonable setting for isolating reasoning, but it may disadvantage agent methods that are designed to explore the full KG. Please state this explicitly as a deliberate choice and discuss its possible effect on the comparison.","section":"§4.1.3"},{"comment":"There are numerous typos and formatting issues (e.g., missing spaces, inconsistent capitalization of 'LLaMA' vs 'Llama', and garbled Table 1). A careful proofreading pass is needed.","section":"Global"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is novel and the experimental work is extensive, but the oracle-subgraph evaluation is a serious gap between the claims and what is demonstrated. I would like to see either an evaluation with a real retriever or a revised set of claims that explicitly scope the contribution to reasoning over provided subgraphs. The statistical-significance issue is also important for a journal. I do not see a fatal flaw in the methodology itself; the issues are addressable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the paper has two real contributions — the KG-to-Python/NetworkX representation and the automated corpus pipeline — but the central evaluation claim is not yet supported. Every headline comparison uses subgraphs extracted from gold SPARQL queries (§4.1.5), expanded with up to 100 triples of 1-hop noise. That is an oracle, not a realistic retriever. The paper calls it a 'realistic retrieval scenario' but it starts from ground truth. Given the paper's own error analysis attributes 40% of errors to subgraph coverage, you cannot know whether the code-based reasoning advantage survives with noisier or incomplete retrieval. The abstract and introduction say 'significantly outperforms existing KG-enhanced LLM methods' with no qualification. That is the load-bearing soft spot.\n\nCredit where due: the code representation is new to the KGQA literature as far as the paper's citations go, and the idea of grounding reasoning in executable graph operations is more verifiable than linearized triples or agent-style exploration. The corpus pipeline — generate SPARQL templates, use an LLM to produce questions, generate code, keep only examples where hard and soft answers match the gold answer — is a concrete recipe that others can reuse. The fact that the trained Llama/Qwen models beat GPT-5-Mini on these benchmarks is a genuine sign the distillation works. The transfer results to WebQSP and GrailQA are also suggestive, though they inherit the same oracle setting.\n\nOther soft spots are smaller. LC-QuAD test answers are self-computed and filtered to questions that return non-empty answers, which changes the benchmark. There is no comparison to any prior code-generation KBQA method, only to RAG/agent/SPARQL baselines. No variance or significance is reported. These are fixable in revision.\n\nWho gets value: KGQA researchers who care about representations and training data for grounded reasoning. This is a serious paper that deserves a referee, but the referee should ask for a realistic retriever experiment (even a simple entity-linking plus subgraph expansion pipeline) and for the claims to be qualified accordingly. If the authors can show the advantage holds outside the oracle, it's a useful contribution.","headline":"A solid code-based KGQA representation and corpus recipe, but the headline claim of superiority rests on gold-SPARQL subgraphs; the evaluation needs a realistic retriever.","tokens_in":30292,"tokens_out":3801,"would_cite":true,"duration_ms":35252,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Representing knowledge-graph subgraphs as executable Python code lets LLMs answer questions more faithfully and transferably than RAG, agent-based, or SPARQL pipelines.","keywords":["KG2Code","knowledge graph question answering","code generation","executable code","large language models","retrieval-augmented generation","faithful reasoning","graph-structured representation"],"falsifier":"Run the same trained model on subgraphs produced by an actual retrieval system (for example, semantic search over triples) using the same 100-triple budget, and compare F1 and exact match against numbers reported with gold-query subgraphs; if the gap over RAG baselines collapses or reverses, the claim that code-based reasoning is inherently more faithful depends on the curated retrieval setting.","tokens_in":29450,"feed_emoji":"🐍","tokens_out":4512,"duration_ms":45216,"temperature":0.7,"pith_summary":"The paper tries to establish that a knowledge graph's question-relevant subgraph can be rewritten as executable Python code, and that a language model asked to complete a function over that code will reason more faithfully than when the same facts are linearized into text or translated into SPARQL. It claims this 'code as bridge' formulation preserves graph topology, grounds every reasoning step in graph operations, and aligns with LLMs' pretraining on code, thereby reducing hallucination. The paper further claims that an automated pipeline can distill over 200,000 verified code-question pairs to train open-source LLMs, which then answer KGQA in a zero-shot setting and generalize to unseen knowledge graphs. A sympathetic reader would care because the approach offers a verifiable, interpretable alternative to the three dominant KG-enhanced LLM paradigms, with reported gains on benchmarks and a pathway for small models to acquire the skill.","feed_headline":"Python-code knowledge graphs beat RAG and SPARQL for KG QA","feed_subtitle":"Executable code keeps reasoning tied to the graph, and an automated corpus lets open LLMs learn the trick.","key_machinery":"KG2Code is the rewriting method: it turns a retrieved KG subgraph into executable Python that constructs a directed labeled graph, with nodes for entities and edges carrying relation types, so the graph's topology is explicitly preserved. KG2Code-QA then frames KGQA as code completion: the model fills in a function that interleaves natural-language comments (reasoning intentions) with executable statements (graph queries and postprocessing), and the answer is obtained by running the function. A third piece is the automated corpus pipeline: it builds subgraphs from SPARQL patterns, asks a teacher model to generate questions and code with in-context examples, and filters examples by executing","core_discovery":"The central claim is that the question-relevant portion of a knowledge graph can be serialized as Python code that builds a graph object with nodes and directed labeled edges, and that a language model completing a `KGQA` function over that code will produce answers traceable to explicit graph operations. The completed code is executed to yield a 'hard' answer; if execution fails, the model's ending comment is parsed as a 'soft' answer. The paper reports that this method outperforms RAG-based, agent-based, and SPARQL-based baselines on two Wikidata-based benchmarks, and that transfer experiments on Freebase-based benchmarks demonstrate strong generalization without KG-specific training.","pith_inferences":["If the representation generalizes, the same rewriting could drop into Graph-RAG pipelines that build a knowledge graph from text or tables, turning retrieved evidence into executable code for any reasoning task, not just KGQA.","The paper's hard/soft split suggests a self-improvement loop: examples that produce soft answers (failed execution) could be automatically regenerated or repaired, potentially shrinking the 4% hallucination error category in future training cycles.","The 40% subgraph-coverage error share implies the headline gains are partially a statement about retrieval quality; pairing code-based reasoning with a learned retriever is a natural next test, since the paper simulates retrieval from gold SPARQL subgraphs.","A reader could test whether the approach's advantage persists under a deployed retriever that returns noisier or incomplete subgraphs, because the paper's retrieval simulation caps at 100 triples with 1-hop expansion, which may be cleaner than real-world retrieval."],"forward_implications":["A code-based representation can replace triple linearization in KGQA, preserving multi-hop structure and relational constraints that plain-text contexts lose.","Framing KGQA as code generation yields verifiable reasoning traces, so execution itself acts as a hallucination check and answers are traceable to graph operations.","Open-source LLMs trained on the distilled corpus can perform zero-shot KGQA without benchmark-specific fine-tuning, and transfer results suggest generalization to unseen knowledge graphs and schemas.","Code's looser syntax and stronger presence in LLM pretraining offer a more flexible alternative to SPARQL, which the paper shows suffers high no-answer rates even after fine-tuning.","The hard/soft answer fallback ensures the system always returns an answer, preferring executable graph-grounded results but preserving robustness when code generation fails."],"fun_headline_variants":["Turn knowledge graphs into Python code for better QA","KG2Code: Executable graphs outperform RAG and SPARQL","Code-based KG representation beats RAG, agents, SPARQL","Why code beats SPARQL for graph question answering","LLMs answer KG questions via executable code"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that subgraphs extracted from gold SPARQL queries, expanded with a little 1-hop noise and capped at 100 triples, are a valid stand-in for what a real retriever would return; if practical retrieval is substantially noisier or more incomplete, the reported advantage over RAG and agent baselines could shrink.","fun_headline_variants_meta":{"raw":{"variants":["Turn knowledge graphs into Python code for better QA","KG2Code: Executable graphs outperform RAG and SPARQL","Code-based KG representation beats RAG, agents, SPARQL","Why code beats SPARQL for graph question answering","LLMs answer KG questions via executable code"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00021,"raw_usage":{"total_tokens":1266,"prompt_tokens":782,"completion_tokens":484,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":404}},"tokens_in":526,"tokens_out":484,"duration_ms":5387,"temperature":1.0,"reasoning_tokens":404,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T09:51:20.597317+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same trained model on subgraphs produced by an actual retrieval system (for example, semantic search over triples) using the same 100-triple budget, and compare F1 and exact match against numbers reported with gold-query subgraphs; if the gap over RAG baselines collapses or reverses, the claim that code-based reasoning is inherently more faithful depends on the curated retrieval setting.","supporting_citations":[],"review_version":1}