{"id":"d843127b-581c-4af1-8277-2c46c11afec3","arxiv_id":"2412.10654","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Representing knowledge graph triples as Python code improved LLM multi-hop reasoning accuracy over text and JSON in this study, though the effect is modest and possibly due to explicit inference steps in the code.","lead":"This paper tests whether representing knowledge graph facts as Python code helps large language models answer multi-step reasoning questions. The Python format outperformed plain text and JSON in the experiments, but the gains were small and the comparison was not fully controlled.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Python-representation advantage is confounded with the presence of explicit inference code; no ablation isolates code format from reasoning steps, so the paper's central comparison is not yet demonstrated.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the Python representation is entangled with explicit inference steps, and no ablation separates the format from the reasoning content. This concern is central because it targets the causal claim of the paper, not merely the magnitude of an effect. The abstract and Section 4.7.1 assert that Python representations outperform other representations, and Section 4.8 attributes this to the 'controlled and unambiguous' nature of programming code. But the prompts in Appendix A.3 and A.4 contain executable lookup operations absent from the JSON and natural-language prompts, so the comparison is not a clean test of representation format. A positive result from the proposed ablation would strengthen the claim; a null result would reduce the contribution to 'explicit inference steps improve reasoning,' which is already well established in the chain-of-thought literature. The concern does not require rejecting the paper; it requires additional experiments before the central claim can be accepted. Since the reader already issued a conditional verdict, my read does not change that verdict, so no adjustment is needed.","tokens_in":14799,"tokens_out":4314,"duration_ms":40441,"concrete_test":"Run a 2x2 ablation on Dataset 1 with Llama-3.1-8B, one-shot prompting: (1) Python dictionary with data only and no infer statements; (2) Python dictionary with the current executable infer code; (3) natural language with explicit 'Step 1/Step 2' lookup statements; (4) JSON with an added 'inference_steps' field containing the same two lookups. Match answer format and approximate prompt length. Repeat for LoRA fine-tuning by including or excluding the inference code in the training targets. Report conditional and final accuracy with McNemar tests or bootstrap confidence intervals. If condition (2) ties (4) and (1) ties the data-only variants, the active ingredient is explicit inference steps, not Python syntax; if (2) beats (4) while (1) also remains best, the format claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Python KG representations outperform natural language and JSON representations in one-shot prompting and fine-tuning. The experiments cannot support this attribution because the Python condition bundles two variables: syntax and explicit inference steps. The one-shot prompts in Appendix A.3 and A.4 include executable lookups such as 'e2 = relationships[r1][e1]', 'e3 = relationships[r2][e2]', and 'kb.infer(e1, r1, r2)', while the JSON prompt contains only data and the natural language prompt contains a prose explanation but no executable code. Thus the Python treatment differs from JSON in both representation format and the presence of step-by-step lookup instructions. The paper's own discussion in Section 4.8 states that Python representations 'store both structured data and the inference process,' which explicitly bundles the candidate mechanism with the treatment. If the observed benefit comes from showing the model explicit inference steps, the headline attribution to programming-language format is unsupported. The fine-tuning condition is equally ambiguous because the paper does not show whether the training targets include the same inference code or only the data structures. No ablation, significance test, or confidence interval is provided, so the reported differences could reflect the confound rather than the representational format.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes representing knowledge-graph triples as Python code (a static dictionary variant and a dynamic KnowledgeBase-class variant) to improve multi-hop reasoning in LLMs. It compares these representations against natural-language and JSON representations under one-shot prompting and LoRA fine-tuning of Llama-3.1-8B-Instruct, using two two-hop Wikidata-derived datasets, a three-hop generalization set, and a context-augmented setting. The central claim is that Python representations of KGs outperform natural-language and JSON representations in complex reasoning tasks, and that the authors are the first to represent KGs with programming language and to fine-tune pretrained LLMs with such representations.","tokens_in":15038,"tokens_out":6330,"duration_ms":53173,"significance":"If the central comparison were cleanly established, the paper would offer a practically useful and simple idea: encoding KG triples as Python data structures and inference code, capitalizing on LLMs' familiarity with code from pretraining. The study covers several relevant settings (one-shot prompting, fine-tuning, three-hop generalization, and context-augmented inference) and reports a hop-conditioned accuracy metric, which is a reasonable way to isolate reasoning ability from factual knowledge. However, the load-bearing comparison is currently confounded: the Python prompts include explicit step-by-step inference code that the JSON and natural-language prompts do not contain, and no error bars or significance tests are reported. The lack of a released implementation or prompt templates also makes the results hard to verify. The underlying idea is plausible and worth further investigation, but the paper as written does not yet demonstrate that the programming-language format itself is the cause of the observed improvements.","major_comments":[{"comment":"The Python one-shot conditions contain explicit executable inference steps (\"e2 = relationships[r1][e1]\", \"e3 = relationships[r2][e2]\", \"kb.infer(e1, r1, r2)\"), while the JSON prompt in A.2 contains only data and the natural-language prompt in A.1 contains a prose explanation but no such stepwise lookup. Thus the Python treatment bundles the programming-language syntax with the presence of explicit inference instructions. The paper's own discussion in §4.8 states that Python representations \"store both structured data and the inference process,\" which acknowledges this entanglement. Without an ablation that varies the format and the presence of inference steps independently (for example, JSON with explicit lookup steps, or Python code without the inference code), the headline attribution to programming-language representation is not supported.","section":"§4.5 and Appendix A.2–A.4"},{"comment":"No error bars, confidence intervals, or significance tests are reported for any accuracy figure. For instance, in Table 5 the fine-tuned natural-language model achieves 25.6% and the fine-tuned Python-static model achieves 26.2%; with roughly ten thousand test questions the standard error of that difference is on the order of 0.6 percentage points, so the gap is within noise. The one-shot results in Table 6 show similarly small differences (e.g., Python Static 4.98% vs. Python Dynamic 4.12%). The paper should report uncertainty intervals or significance tests before claiming that Python representations are significantly better.","section":"Tables 5 and 6"},{"comment":"The three-hop generalization experiment is heavily overlapped with the training relation pairs. Table 3 shows that 973 of the 1,007 test rows share the same (r1, r2) pair as Dataset 1 training rows, and the paper itself notes the significant overlap. Consequently, the three-hop improvements in Table 7 may reflect reuse of familiar relation pairs rather than generalization to longer unseen reasoning chains. The generalization claim would need an evaluation on a held-out split with disjoint relation pairs, or at least a separate reporting of the non-overlapping subset.","section":"§4.1.3 and §4.7.2 (RQ2)"},{"comment":"The fine-tuning condition is underspecified. The paper says the final training data include one-hop and two-hop prompts and responses, but it does not state whether the Python training targets contain the same inference code shown in the one-shot prompts (Appendix A.3 and A.4) or only the data structures plus the answer. This matters because if the targets include the inference code, the fine-tuning benefit may come from learning to emit lookup operations rather than from the representational format. The authors should specify the exact training templates and, ideally, ablate the inference-code component.","section":"§4.1.3 and §4.4.1"}],"minor_comments":[{"comment":"There is a typo: \"impove\" should be \"improve\".","section":"§2"},{"comment":"The JSON representation row has a malformed key (\"“Explanation”” :\") and a duplicated phrase \"generate explanation and provide answer to the question\"; the table layout should be cleaned up.","section":"Table 4"},{"comment":"The code snippet uses `self.facts = {{}}`, which is not valid Python; presumably `self.facts = {}` is intended.","section":"Appendix A.4"},{"comment":"The sentence reporting \"approximately 78% higher\" performance appears to refer to the final-accuracy column (67.9% vs. 38.2% in Table 5) rather than to the conditioned metric shown as \"Prompt Accuracy\" (19.1% vs. 16.3%). Please clarify which metric is being compared, since the two give very different impressions.","section":"§4.7.1"},{"comment":"The repeated column headers \"1st hop correct 2nd hop correct\" make the tables hard to read; please define each column with a clear label, for example distinguishing counts where the final answer is correct from counts where it is incorrect.","section":"Tables 5–8"},{"comment":"The comparisons between fine-tuned 8B models and the zero-shot 70B model are not apples-to-apples because the larger model is not fine-tuned; the text should note this asymmetry explicitly when stating that small fine-tuned models \"surpass\" the larger baseline.","section":"§4.2 and §4.7.3"}],"recommendation":"major_revision","confidential_remarks":"The confound between code format and explicit inference steps is real and fixable with targeted ablations, so I would not reject the paper outright. The authors should also add uncertainty quantification and clarify the fine-tuning targets. If the ablations confirm that the inference-steps component, rather than the code syntax, drives the gains, the paper would still be a useful empirical study, but its framing should be adjusted accordingly. The novelty claim relative to related code-style prompting work should be softened or delimited more precisely."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is simple and probably useful: represent KG triples as Python code, write the multi-hop inference chain explicitly, and use that as a prompt or fine-tuning target. The paper shows this beats JSON and, in most conditions, natural language, on two synthetic Wikidata two-hop datasets. The dynamic Python representation with a KnowledgeBase class and infer() function is the most interesting piece; it makes the reasoning path explicit and executable. To the authors' credit, they test one-shot prompting, fine-tuning, 3-hop generalization, and a RAG-style context condition. The observation that a fine-tuned 8B model can beat a zero-shot 70B model on latent accuracy is worth reporting.\n\nThe main soft spot is the missing ablation, though it is less severe than a strict reading of the confound suggests. Both the NL and Python one-shot prompts contain step-by-step inference: for NL this is the 'Explanation' field like 'r1 of e1 is e2. r2 of e2 is e3'; for Python it is the executable lookup code. So the Python-vs-NL difference is mostly format, not presence of inference steps. The JSON condition lacks those steps entirely, which probably explains why it lags. What is missing is a control that gives Python-style inference steps in a non-code format or code without explicit steps, to isolate the format effect. That is a fair referee request, not a fatal flaw.\n\nMore minor issues: no significance tests or error bars; several reported differences are small (e.g., 25.6% vs 26.2% fine-tuned NL vs Python static on Dataset 1). The 'first' claim is overstated given Nie et al. did code-style ICL for KBQA; the paper differentiates itself but 'first to represent KGs with programming language' is too strong. Fine-tuning details are thin (LoRA rank, learning rate, etc.). No code or data release.\n\nThe paper is honest about its limitations: it only studies compositional two-hop and three-hop reasoning on synthetic data. It is a modest, applied contribution, and as long as the claims are reined in and the ablation added, it deserves a serious referee and probably a workshop or applied NLP venue.","headline":"A useful, modest empirical study showing Python-encoded KG triples help LLM multi-hop reasoning, but the key ablation separating code format from explicit inference steps is missing.","tokens_in":15558,"tokens_out":2557,"would_cite":false,"duration_ms":23569,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Representing knowledge graphs as Python code improves how well LLMs chain facts together, the paper argues.","keywords":["knowledge graphs","multi-hop reasoning","programming language representation","LLM reasoning","fine-tuning","one-shot prompting","Python","structured data"],"falsifier":"Run the one-shot comparison again with a natural-language prompt that spells out the same inference steps (for example, first look up the composer of X, then look up the spouse of the result) and with a JSON prompt that includes an ordered list of lookup operations; if either matches the Python accuracy, the claimed format advantage is not specific to Python.","tokens_in":14605,"feed_emoji":"🐍","tokens_out":8067,"duration_ms":64577,"temperature":0.7,"pith_summary":"This paper argues that the format used to feed a knowledge graph to a large language model changes how well the model can chain facts together. The authors claim that representing the graph's triples as Python code—a dictionary of entities and relations plus a small inference function—outperforms natural-language sentences and JSON data, both in one-shot prompting and in fine-tuning. On a two-hop Wikidata reasoning task, the best Python one-shot prompt raised the accuracy of completing the second hop given a correct first hop from 38.2% to 67.9%, and fine-tuned Python versions reach the low nineties on that metric. Across datasets, fine-tuned small models with Python representations match or exceed a much larger zero-shot model. The paper presents this as evidence that grounding LLM reasoning in structured knowledge can be largely a representation choice.","feed_headline":"Python-encoded knowledge graphs lift LLM multi-hop reasoning","feed_subtitle":"One-shot prompts and fine-tuning with Python-formatted triples improve two-hop and three-hop reasoning accuracy.","key_machinery":"The central object is the Python representation of a knowledge graph: a `relationships` dictionary whose keys are relation names and whose values map a start entity to an end entity, optionally wrapped in a `KnowledgeBase` class with an `infer` function that composes any number of relations. The argument turns on bundling the static triples with an explicit, executable inference process, so the prompt shows the LLM the exact dictionary lookups required to answer the multi-hop question. This is contrasted with JSON, which stores the same data but no lookup sequence, and with natural-language descriptions, which state the facts but do not lay out the retrieval steps.","core_discovery":"The paper's central claim is that encoding a knowledge graph as executable Python code improves LLM multi-hop reasoning compared to encoding the same triples as natural language or JSON. The static form is a `relationships` dictionary with explicit lookup steps; the dynamic form wraps the dictionary in a `KnowledgeBase` class whose `infer` function walks through an arbitrary chain of relations. In one-shot prompting, the dynamic Python representation raises the conditioned two-hop accuracy from 38.2% to 67.9% on Dataset 1, while JSON one-shot prompting stays below zero-shot. After LoRA fine-tuning, Python-tuned models reach the highest conditioned accuracies on both datasets and also show the best transfer to three-hop questions. The authors state they are the first to represent knowledge graphs with programming language and to fine-tune pretrained LLMs with those representations.","pith_inferences":["The non-Python prompts in the comparison lack the explicit step-by-step lookups that the Python prompts include, so the active ingredient may be the worked inference path rather than the code format itself.","The same representation idea could be applied to other structured reasoning inputs, such as tables or provenance traces, wherever the model must follow a fixed traversal path.","A follow-up that swaps Python for pseudocode, or adds a natural-language chain of thought with identical steps, would separate the syntax effect from the step-explicitness effect.","The reported gains are on compositional chains; whether the Python advantage holds for non-compositional or cyclic graph queries is not addressed."],"forward_implications":["Python-encoded knowledge graphs can be dropped into existing one-shot prompts or fine-tuning datasets without changing the model architecture.","Fine-tuning on two-hop Python triples improves three-hop reasoning, so the learned benefit is not tied to the trained hop count.","A small fine-tuned model with Python representations can outperform a much larger zero-shot model on the same multi-hop questions.","Supplying RAG-style context as Python code that spells out inference steps yields higher answer accuracy than supplying the same facts as plain text."],"supporting_citations":[{"why":"supplies Dataset 1, the two-hop reasoning query set, and the finding that LLMs struggle on later hops.","marker":"[3]"},{"why":"supplies Dataset 2, a multi-hop QA dataset with compositional relationships.","marker":"[16]"},{"why":"provides the base 8B and 70B model checkpoints that are prompted and fine-tuned.","marker":"[1]"},{"why":"provides the LoRA parameter-efficient fine-tuning method used in tuning.","marker":"[17]"},{"why":"provides the public knowledge base from which both datasets are extracted.","marker":"[27]"},{"why":"is the code-style in-context learning approach whose representation the paper extends.","marker":"[24]"},{"why":"established that LLMs perform latent multi-hop reasoning, the capability the paper strengthens.","marker":"[33]"}],"fun_headline_variants":["Python-coded knowledge graphs strengthen LLM multi-hop reasoning","Encoding KGs as Python code lifts LLM reasoning accuracy","LLMs reason better when knowledge graph triples become Python","First KG-to-Python fine-tune improves LLM two-hop and three-hop"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes the gain comes from the Python format itself, but the Python prompts also contain explicit step-by-step lookup instructions that the JSON and natural-language prompts do not, so the format effect is entangled with the presence of a worked inference path.","fun_headline_variants_meta":{"raw":{"variants":["Python-coded knowledge graphs strengthen LLM multi-hop reasoning","Encoding KGs as Python code lifts LLM reasoning accuracy","LLMs reason better when knowledge graph triples become Python","First KG-to-Python fine-tune improves LLM two-hop and three-hop"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000643,"raw_usage":{"total_tokens":2913,"prompt_tokens":860,"completion_tokens":2053,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":1981}},"tokens_in":476,"tokens_out":2053,"duration_ms":13753,"temperature":1.0,"reasoning_tokens":1981,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:43:59.913127+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the one-shot comparison again with a natural-language prompt that spells out the same inference steps (for example, first look up the composer of X, then look up the spouse of the result) and with a JSON prompt that includes an ordered list of lookup operations; if either matches the Python accuracy, the claimed format advantage is not specific to Python.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies Dataset 1, the two-hop reasoning query set, and the finding that LLMs struggle on later hops."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the LoRA parameter-efficient fine-tuning method used in tuning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the public knowledge base from which both datasets are extracted."}],"review_version":1}