Pith. sign in

REVIEW 3 major objections 6 minor 58 references

Representing knowledge-graph subgraphs as executable Python code lets LLMs answer questions more faithfully and transferably than RAG, agent-based, or SPARQL pipelines.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 09:51 UTC pith:KJPQVIM3

load-bearing objection A solid code-based KGQA representation and corpus recipe, but the headline claim of superiority rests on gold-SPARQL subgraphs; the evaluation needs a realistic retriever. the 3 major comments →

arxiv 2607.22652 v1 pith:KJPQVIM3 submitted 2026-06-26 cs.AI

KG2Code: Bridging Knowledge Graphs and Large Language Models via Executable Code for Question Answering

Yike Wu , Nan Hu , Guilin Qi , Guohui Xiao , Chen Jiang , Xinchun Zou , Yuchen Lu , Songlin Zhai
show 6 more authors
Yongrui Chen Yuyang Zhang Xiaoguang Li Lifeng Shang Jiaoyan Chen Jeff Z. Pan
This is my paper
classification cs.AI
keywords KG2Codeknowledge graph question answeringcode generationexecutable codelarge language modelsretrieval-augmented generationfaithful reasoninggraph-structured representation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that a knowledge graph's question-relevant subgraph can be rewritten as executable Python code, and that a language model asked to complete a function over that code will reason more faithfully than when the same facts are linearized into text or translated into SPARQL. It claims this 'code as bridge' formulation preserves graph topology, grounds every reasoning step in graph operations, and aligns with LLMs' pretraining on code, thereby reducing hallucination. The paper further claims that an automated pipeline can distill over 200,000 verified code-question pairs to train open-source LLMs, which then answer KGQA in a zero-shot setting and generalize to unseen knowledge graphs. A sympathetic reader would care because the approach offers a verifiable, interpretable alternative to the three dominant KG-enhanced LLM paradigms, with reported gains on benchmarks and a pathway for small models to acquire the skill.

Core claim

The central claim is that the question-relevant portion of a knowledge graph can be serialized as Python code that builds a graph object with nodes and directed labeled edges, and that a language model completing a `KGQA` function over that code will produce answers traceable to explicit graph operations. The completed code is executed to yield a 'hard' answer; if execution fails, the model's ending comment is parsed as a 'soft' answer. The paper reports that this method outperforms RAG-based, agent-based, and SPARQL-based baselines on two Wikidata-based benchmarks, and that transfer experiments on Freebase-based benchmarks demonstrate strong generalization without KG-specific training.

What carries the argument

KG2Code is the rewriting method: it turns a retrieved KG subgraph into executable Python that constructs a directed labeled graph, with nodes for entities and edges carrying relation types, so the graph's topology is explicitly preserved. KG2Code-QA then frames KGQA as code completion: the model fills in a function that interleaves natural-language comments (reasoning intentions) with executable statements (graph queries and postprocessing), and the answer is obtained by running the function. A third piece is the automated corpus pipeline: it builds subgraphs from SPARQL patterns, asks a teacher model to generate questions and code with in-context examples, and filters examples by executing

Load-bearing premise

The load-bearing premise is that subgraphs extracted from gold SPARQL queries, expanded with a little 1-hop noise and capped at 100 triples, are a valid stand-in for what a real retriever would return; if practical retrieval is substantially noisier or more incomplete, the reported advantage over RAG and agent baselines could shrink.

What would settle it

Run the same trained model on subgraphs produced by an actual retrieval system (for example, semantic search over triples) using the same 100-triple budget, and compare F1 and exact match against numbers reported with gold-query subgraphs; if the gap over RAG baselines collapses or reverses, the claim that code-based reasoning is inherently more faithful depends on the curated retrieval setting.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A code-based representation can replace triple linearization in KGQA, preserving multi-hop structure and relational constraints that plain-text contexts lose.
  • Framing KGQA as code generation yields verifiable reasoning traces, so execution itself acts as a hallucination check and answers are traceable to graph operations.
  • Open-source LLMs trained on the distilled corpus can perform zero-shot KGQA without benchmark-specific fine-tuning, and transfer results suggest generalization to unseen knowledge graphs and schemas.
  • Code's looser syntax and stronger presence in LLM pretraining offer a more flexible alternative to SPARQL, which the paper shows suffers high no-answer rates even after fine-tuning.
  • The hard/soft answer fallback ensures the system always returns an answer, preferring executable graph-grounded results but preserving robustness when code generation fails.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the representation generalizes, the same rewriting could drop into Graph-RAG pipelines that build a knowledge graph from text or tables, turning retrieved evidence into executable code for any reasoning task, not just KGQA.
  • The paper's hard/soft split suggests a self-improvement loop: examples that produce soft answers (failed execution) could be automatically regenerated or repaired, potentially shrinking the 4% hallucination error category in future training cycles.
  • The 40% subgraph-coverage error share implies the headline gains are partially a statement about retrieval quality; pairing code-based reasoning with a learned retriever is a natural next test, since the paper simulates retrieval from gold SPARQL subgraphs.
  • A reader could test whether the approach's advantage persists under a deployed retriever that returns noisier or incomplete subgraphs, because the paper's retrieval simulation caps at 100 triples with 1-hop expansion, which may be cleaner than real-world retrieval.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes KG2Code, a representation that rewrites KG subgraphs as executable Python/NetworkX code, and KG2Code-QA, a KGQA framework that asks an LLM to complete a QA function over this code representation. The framework alternates natural-language comments with executable statements (ReAct-style), obtains hard answers by executing the generated code, and falls back to soft answers parsed from comments when execution fails. An automated pipeline constructs a large (over 200k examples) code-based training corpus from Wikidata, and the authors instruction-tune Llama-3.1-8B-Instruct and Qwen2.5-Coder-7B-Instruct with LoRA. Experiments on WikiWebQuestions, LC-QuAD 2.0, and transfer to WebQSP and GrailQA compare against SPARQL, RAG, agent, GNN, and CoT baselines, reporting consistent improvements in Precision, F1, and EM. The authors claim this is the first conversion of KGs into executable code for KGQA and that KG2Code-QA significantly outperforms existing KG-enhanced LLM methods while generalizing to unseen KGs.

Significance. If the result holds, the paper makes a useful contribution: a faithful, interpretable, code-grounded interface between KGs and LLMs, with a practical distillation pipeline that lets small open-source models outperform their closed-source teacher in the tested setting. The corpus construction with execution-based filtering, the use of an independent training corpus, and the transfer evaluation are all strengths. The central claim, however, is stated as an end-to-end KGQA result, while the evaluation protocol isolates the reasoning component by supplying oracle subgraphs derived from gold SPARQL queries. The contribution is therefore best understood as a strong reasoning/interface method, not yet a demonstrated complete KGQA system. With proper qualification and additional retrieval evaluation, this would be a solid paper.

major comments (3)
  1. [§4.1.5 and §1] The retrieval setting uses ground-truth subgraphs extracted by parsing gold SPARQL queries, expanded with 1-hop noise up to 100 triples. All headline comparisons in Tables 2, 4, 5, and 6, as well as the transfer experiments, inherit this oracle setting. No automatic entity linking or retrieval component is evaluated. The paper's own error analysis in §4.10 attributes 40% of errors to subgraph retrieval issues, making retrieval the dominant error source. Under a realistic retriever that returns noisier or incomplete subgraphs, the advantage of KG2Code-QA may shrink or vanish, since the RAG baselines receive the same oracle subgraphs and the comparison isolates reasoning rather than end-to-end KGQA. The abstract and Section 1 claim that 'KG2Code-QA significantly outperforms existing KG-enhanced LLM methods for KGQA' without this qualification. The authors should either evaluate with a prac
  2. [§4.1.1] For LC-QuAD 2.0, the test answers are obtained by executing the provided Wikidata SPARQL queries, and questions with no returned answers are excluded, leaving 4,932 of 6,046 test questions. This self-computed and filtered test set is used for all LC-QuAD comparisons. The filtering step may interact with model behavior (e.g., questions that produce empty SPARQL results may also be hard for KG2Code-QA), and the answer computation is not independently verified. The authors should report the unfiltered test size, provide the exact filtering criteria, and ideally release the computed test answers for reproducibility. Without this, the LC-QuAD results are difficult to compare against other published work.
  3. [Tables 2, 4, 5] The paper repeatedly states that KG2Code-QA 'significantly outperforms' baselines, but no variance, confidence intervals, or statistical significance tests are reported. All results appear to be from a single evaluation run. Given that several differences in Tables 2 and 5 are small (e.g., WWQ EM: 64.08 vs 61.01 for CoT-Tuning; WebQSP Precision: 85.98 vs 85.68 for CoT-Tuning), the claimed significance is not supported by the evidence presented. The authors should add multiple seeds or bootstrap resampling and report significance tests, or soften the language to 'outperforms in the reported runs.'
minor comments (6)
  1. [Figures 1–2] The running example is inconsistent: Figure 1 states 'Answer: 1' for the question about screenwriters who died in Rome, while Figure 2's code computes two screenwriters died in Rome and returns 2. Please reconcile the example.
  2. [Table 10] In the KG2Code-QA code response, the second loop contains a duplicated 'for u, v, data in graph.edges(data=True):' line, which appears to be a typo. Also the response includes 'return fields' instead of returning the final answer, but the claimed answer is correct; clarify.
  3. [Appendix B] Tables B.12, B.13, and B.14 are referenced in the text but appear as empty placeholders in the provided manuscript. Please include the actual example code blocks.
  4. [§3.5] The corpus generation section states 'over 200,000' training examples, but no exact number is given. Please provide the exact corpus size and the number of examples per question type, as this affects the reproducibility of the instruction-tuning results.
  5. [§4.1.3] The ToG, GoG, and DoG baselines are all run on the same retrieved subgraphs as KG2Code-QA. This is a reasonable setting for isolating reasoning, but it may disadvantage agent methods that are designed to explore the full KG. Please state this explicitly as a deliberate choice and discuss its possible effect on the comparison.
  6. [Global] There are numerous typos and formatting issues (e.g., missing spaces, inconsistent capitalization of 'LLaMA' vs 'Llama', and garbled Table 1). A careful proofreading pass is needed.

Circularity Check

0 steps flagged

No circular derivation: claims rest on held-out evaluations; oracle-subgraph retrieval is an external-validity limitation, not a circular step.

full rationale

No significant circularity. The paper is an empirical systems paper, not a formal derivation. Its central comparisons are measured on held-out benchmarks (WWQ, LC-QuAD 2.0, WebQSP, GrailQA), and the open-source models are explicitly trained on a separately constructed corpus, not on the benchmark training splits: Section 4.1.4 states 'The proposed method is trained exclusively on the constructed training data, rather than on the training split of the benchmark.' The corpus filtering step (hard and soft answers must match the gold answer) is a quality filter, not a fitted parameter later relabeled as a prediction; there is no equation in which the claimed result equals its input by construction. The only self-citations are to the authors' earlier KGQA work (e.g., [13] and [47]) and are used as related work or baseline construction, not as load-bearing evidence; no uniqueness theorem or ansatz is imported from them. The oracle-subgraph retrieval protocol in Section 4.1.5, 'the ground-truth subgraphs are first extracted by parsing the gold SPARQL queries,' is a genuine external-validity limitation: the headlines claim end-to-end KGQA superiority without evaluating a practical retriever, and the error analysis attributes 40% of errors to subgraph coverage. However, this is an evaluation-scope problem, not a circular derivation: predicted answers are not constructed from the gold answers. Score 2 reflects minor self-citation overlap and the slight inwardness that the teacher model used to build the corpus is also one evaluation condition, but no circular step meets the quoted-reduction bar.

Axiom & Free-Parameter Ledger

2 free parameters · 7 axioms · 0 invented entities

The paper introduces a representation (KG2Code code snippets) and a framework, but no new physical or mathematical entity; the representation is defined entirely by the paper's own code templates. The main assumptions are the oracle retrieval proxy, trust in Wikidata SPARQL gold answers, and the sufficiency of the hand-designed question-type taxonomy.

free parameters (2)
  • Retrieved subgraph size cap = 100 triples
    Hand-set in §4.1.5; limits the input provided to all methods and defines the evaluation regime.
  • Noise expansion per non-answer entity = 1-hop, up to 10 triples
    Hand-set in §4.1.5; controls how much irrelevant context is added to gold subgraphs.
axioms (7)
  • domain assumption Ground-truth subgraphs extracted from gold SPARQL queries at test time are a valid proxy for a realistic retrieval component.
    Section 4.1.5 states this retrieval setting; the central comparison depends on it.
  • domain assumption Executing Wikidata SPARQL queries yields correct gold answers for LC-QuAD 2.0, and filtering out questions with no returned answers does not bias evaluation.
    Section 4.1.1; dataset answers are self-computed, not official.
  • domain assumption The LLM-generated code corpus, filtered by execution and gold-answer agreement, teaches faithful code-based KGQA.
    Section 3.5 corpus filtering; assumes high-quality supervision from an LLM teacher and Wikidata SPARQL answers.
  • domain assumption Surface names of entities and relations in code are sufficient and unambiguous for the model to perform graph operations.
    KG2Code uses add_node/add_edge with surface names (§3.3, §3.5), avoiding KG-specific IDs; entity linking and relation disambiguation are left to the LLM.
  • ad hoc to paper The hand-defined six question types and eight subgraph patterns cover the distribution of benchmark questions.
    Appendix A and Table 1; corpus generation and postprocessing are built on these taxonomies.
  • standard math Python/NetworkX execution semantics are the ground truth for faithful answers.
    The hard-answer mechanism assumes generated code, once executed, faithfully computes the answer from the provided subgraph (§3.4.2).
  • domain assumption Comments embedded in generated code can serve as soft answers when code fails to execute.
    Soft-answer extraction in §3.4.2 assumes parseable comments contain the correct answer; error analysis shows 4% non-executable code cases rely on potentially incorrect soft answers.

pith-pipeline@v1.3.0-alltime-deepseek · 29277 in / 16073 out tokens · 149207 ms · 2026-08-02T09:51:20.597317+00:00 · methodology

0 comments
read the original abstract

Recent research has explored the integration of knowledge graphs (KGs) with large language models (LLMs) to enhance their performance on downstream knowledge-intensive tasks, particularly knowledge graph question answering (KGQA). Existing approaches primarily combine LLMs with KGs through retrieval-augmented generation (RAG)-based, agent-based, and SPARQL-based methods. Although these methods have achieved notable success, they still suffer from several limitations, including structural information loss, unfaithful reasoning, and limited flexibility and generalization. To address these challenges, this paper proposes KG2Code, a novel approach that transforms knowledge graphs into a code-based representation, preserving structural semantics while naturally aligning with the code-aware pretraining of modern LLMs. Based on KG2Code, KG2Code-QA is further introduced as a KGQA framework that formulates KGQA as a code generation task. This formulation enables the generation of verifiable reasoning traces and executable code, thereby substantially mitigating the impact of hallucinations. In addition, an automated pipeline is developed to construct a large-scale, high-quality code corpus for effectively training open-source LLMs on KG2Code-QA. After training, LLMs are able to perform KGQA in zero-shot scenarios. Extensive experiments demonstrate that the proposed approach significantly outperforms existing KG-enhanced LLM methods for KGQA, while exhibiting strong generalization to unseen KGs. The code and data are available at Github.

Figures

Figures reproduced from arXiv: 2607.22652 by Chen Jiang, Guilin Qi, Guohui Xiao, Jeff Z. Pan, Jiaoyan Chen, Lifeng Shang, Nan Hu, Songlin Zhai, Xiaoguang Li, Xinchun Zou, Yike Wu, Yongrui Chen, Yuchen Lu, Yuyang Zhang.

Figure 1
Figure 1. Figure 1: Comparison of existing KG-enhanced LLMs methods for KGQA. [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: llustration of our framework, KG2Code-QA, highlighting the code-style repre [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The corpus generation framework. • Subgraph Construction. To ensure the timeliness and accuracy of knowledge, Wikidata [54], which is continuously updated, is adopted as the underlying knowledge source. Question-relevant subgraphs are constructed through two complementary strategies: pattern extraction and SPARQL querying. In the pattern extraction method, eight types of subgraph struc￾tures are predefined… view at source ↗
Figure 4
Figure 4. Figure 4: Proportion of hard and soft answers produced by KG2Code-QA across four [PITH_FULL_IMAGE:figures/full_fig_p032_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Win–Tie–Lose diagrams comparing the use of hard answers versus soft answers [PITH_FULL_IMAGE:figures/full_fig_p034_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The results of scalability experiments on WikiWebQuestions and LC-QuAD 2.0. [PITH_FULL_IMAGE:figures/full_fig_p036_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

58 extracted references · 1 canonical work pages

  1. [1]

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, et al., Lan- guage models are few-shot learners, in: H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, H. Lin (Eds.), Advances in Neural Informa- tion Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020

    2020

  2. [2]

    V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, et al., Multitask prompted training enables zero-shot task generalization, in: The Tenth International Conference on Learning Representations, ICLR 2022, Vir- tual Event, April 25-29, 2022, OpenReview.net, 2022

    2022

  3. [3]

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, et al., A survey of large language models, CoRR abs/2303.18223 (2023). arXiv:2303.18223, doi:10.48550/ARXIV.2303.18223

    Pith/arXiv arXiv doi:10.48550/arxiv.2303.18223 2023

  4. [4]

    C. Wang, J. Yang, Y. Zhou, X. Yue, Cookie: commonsense knowledge- guided mixture-of-experts framework for fine-grained visual question an- swering, Inf. Sci. 695 (2025) 121742. doi:10.1016/J.INS.2024.121742

    arXiv 2025

  5. [5]

    P. Qiu, S. Zhou, E. Ferrara, Information suppression in large lan- guage models: Auditing, quantifying, and characterizing censorship in deepseek, Inf. Sci. 724 (2026) 122702. doi:10.1016/J.INS.2025.122702

    arXiv 2026

  6. [6]

    S. Feng, Z. Lang, J. He, H. Zhang, W. Chen, et al., A group recom- mendation method based on automatically integrating members’ pref- erences via taking advantages of LLM, Inf. Sci. 709 (2025) 122067. doi:10.1016/J.INS.2025.122067

    arXiv 2025

  7. [7]

    H. Tan, S. Zhan, F. Jia, H. Zheng, W. K. V. Chan, A hierarchical framework for measuring scientific paper innovation via large language models, Inf. Sci. 728 (2026) 122787. doi:10.1016/J.INS.2025.122787

    arXiv 2026

  8. [8]

    arXiv:2303.08774, doi:10.48550/ARXIV.2303.08774

    OpenAI, GPT-4 technical report, CoRR abs/2303.08774 (2023). arXiv:2303.08774, doi:10.48550/ARXIV.2303.08774

    Pith/arXiv arXiv doi:10.48550/arxiv.2303.08774 2023

  9. [9]

    Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, et al., Survey of hallucination in natural language generation, ACM Comput. Surv. 55 (12) (2023) 248:1–248:38. 45

    2023

  10. [10]

    Maynez, S

    J. Maynez, S. Narayan, B. Bohnet, R. T. McDonald, On faithfulness and factuality in abstractive summarization, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, 2020, pp. 1906–1919

    2020

  11. [11]

    N. Hu, Y. Wu, G. Qi, D. Min, J. Chen, et al., An empirical study of pre- trained language models in simple knowledge graph question answering, World Wide Web (WWW) 26 (5) (2023) 2855–2886

    2023

  12. [12]

    Y. Tan, D. Min, Y. Li, W. Li, N. Hu, et al., Can chatgpt replace tra- ditional KBQA models? an in-depth analysis of the question answering performance of the GPT LLM family, in: The Semantic Web - ISWC 2023, Vol. 14265, 2023, pp. 348–367

    2023

  13. [13]

    Y. Wu, N. Hu, S. Bi, G. Qi, J. Ren, et al., Retrieve-rewrite-answer: A kg-to-text enhanced llms framework for knowledge graph question answering, CoRR abs/2309.11206 (2023)

    Pith/arXiv arXiv 2023

  14. [14]

    J. Baek, A. F. Aji, A. Saffari, Knowledge-augmented language model prompting for zero-shot knowledge graph question answering, in: Pro- ceedingsofthe1stWorkshoponNaturalLanguageReasoningandStruc- tured Explanations (NLRSE), 2023, pp. 78–106

    2023

  15. [15]

    P. Sen, S. Mavadia, A. Saffari, Knowledge graph-augmented language models for complex question answering, in: Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE), 2023, pp. 1–8

    2023

  16. [16]

    Mavromatis, S

    C. Mavromatis, S. Adeshina, V. N. Ioannidis, Z. Han, Q. Zhu, et al., BYOKG-RAG: Multi-strategy graph retrieval for knowledge graph ques- tion answering, in: Proceedings of the 2025 Conference on Empiri- cal Methods in Natural Language Processing, EMNLP 2025, 2025, pp. 27881–27898

    2025

  17. [17]

    S. Tian, S. Xing, X. Li, Y. Luo, C. Yuan, et al., A systematic exploration of knowledge graph alignment with large language models in retrieval augmented generation, in: AAAI 2025, 2025, pp. 25291–25299

    2025

  18. [18]

    Korhonen, D

    Z.Zhang, X.Han, Z.Liu, X.Jiang, M.Sun, etal., ERNIE:enhancedlan- guage representation with informative entities, in: A. Korhonen, D. R. 46 Traum, L. Màrquez (Eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, Association for Com- putational ...

    doi:10.18653/v1/p19-1139 2019

  19. [19]

    X. Wang, T. Gao, Z. Zhu, Z. Zhang, Z. Liu, et al., KEPLER: A unified model for knowledge embedding and pre-trained language representation, Trans. Assoc. Comput. Linguistics 9 (2021) 176–194. doi:10.1162/TACL_A_00360

    doi:10.1162/tacl_a_00360 2021

  20. [20]

    W. Liu, P. Zhou, Z. Zhao, Z. Wang, Q. Ju, et al., K-BERT: enabling language representation with knowledge graph, in: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educa- tional Advances in Artificial Intelligen...

    doi:10.1609/aaai.v34i03.5681 2020

  21. [21]

    Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, et al., Retrieval-augmented generation for large language models: A survey, CoRR abs/2312.10997 (2023). arXiv:2312.10997, doi:10.48550/ARXIV.2312.10997

    Pith/arXiv arXiv doi:10.48550/arxiv.2312.10997 2023

  22. [22]

    Lewis, E

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, et al., Retrieval-augmented generation for knowledge-intensive NLP tasks, in: H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, H. Lin (Eds.), Ad- vancesinNeuralInformationProcessingSystems33: AnnualConference on Neural Information Processing Systems 2020, NeurIPS 2020, Decem- ber 6-12, 2020, virtual, 2020

    2020

  23. [23]

    H. Luo, H. E, Z. Tang, S. Peng, Y. Guo, et al., Chatkbqa: A generate- then-retrieve framework for knowledge base question answering with fine-tuned large language models, in: L. Ku, A. Martins, V. Sriku- mar (Eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, Findings of A...

    doi:10.18653/v1/2024.findings-acl.122 2024

  24. [24]

    J. Sun, C. Xu, L. Tang, S. Wang, C. Lin, et al., Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph, 47 in: The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, OpenReview.net, 2024

    2024

  25. [25]

    C. Niu, Y. Wu, J. Zhu, S. Xu, K. Shum, et al., Ragtruth: A halluci- nation corpus for developing trustworthy retrieval-augmented language models, in: L. Ku, A. Martins, V. Srikumar (Eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, Associ...

    doi:10.18653/v1/2024.acl-long.585 2024

  26. [26]

    Z. Sun, X. Zang, K. Zheng, J. Xu, X. Zhang, et al., Redeep: Detecting hallucination in retrieval-augmented generation via mechanistic inter- pretability, in: The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, OpenRe- view.net, 2025

    2025

  27. [27]

    Y. Xu, S. He, J. Chen, Z. Wang, Y. Song, et al., Generate-on-graph: Treat LLM as both agent and KG in incomplete knowledge graph question answering, CoRR abs/2404.14741 (2024). arXiv:2404.14741, doi:10.48550/ARXIV.2404.14741

    Pith/arXiv arXiv doi:10.48550/arxiv.2404.14741 2024

  28. [28]

    X. He, Y. Tian, Y. Sun, N. V. Chawla, T. Laurent, et al., G-retriever: Retrieval-augmented generation for textual graph understanding and question answering, in: A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomczak, C. Zhang (Eds.), Advances in Neural Infor- mation Processing Systems 38: Annual Conference on Neural Informa- tion Process...

    2024

  29. [29]

    Mavromatis, G

    C. Mavromatis, G. Karypis, GNN-RAG: graph neural retrieval for effi- cient large language model reasoning on knowledge graphs, in: W. Che, J. Nabende, E. Shutova, M. T. Pilehvar (Eds.), Findings of the Associ- ation for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025, Findings of ACL, Association for Computational Linguistic...

    2025

  30. [30]

    Scarselli, M

    F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, G. Monfardini, The graph neural network model, IEEE Trans. Neural Networks 20 (1) (2009) 61–80. 48

    2009

  31. [31]

    T. Feng, L. He, RGR-KBQA: generating logical forms for question answering using knowledge-graph-enhanced large language model, in: O. Rambow, L. Wanner, M. Apidianaki, H. Al-Khalifa, B. D. Eugenio, S. Schockaert (Eds.), Proceedings of the 31st International Conference on Computational Linguistics, COLING 2025, Abu Dhabi, UAE, Jan- uary 19-24, 2025, Associ...

    2025

  32. [32]

    P. A. K. K. Diallo, S. Reyd, A. Zouaq, A comprehen- sive evaluation of neural SPARQL query generation from nat- ural language questions, IEEE Access 12 (2024) 125057–125078. doi:10.1109/ACCESS.2024.3453215

    arXiv 2024

  33. [33]

    Mecharnia, M

    T. Mecharnia, M. d’Aquin, Performance and limitations of fine-tuned llms in SPARQL query generation, in: Proceedings of the 31st Interna- tional Conference on Computational Linguistics, COLING 2025 - Work- shops, Abu Dhabi, UAE, January 19-24, 2025, Association for Compu- tational Linguistics, 2025, pp. 69–77

    2025

  34. [34]

    Aryabumi, Y

    V. Aryabumi, Y. Su, R. Ma, A. Morisot, I. Zhang, et al., To code or not to code? exploring impact of code in pre-training, in: The Thirteenth International Conference on Learning Representations, ICLR 2025, Sin- gapore, April 24-28, 2025, OpenReview.net, 2025

    2025

  35. [35]

    Chowdhery, S

    A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, et al., Palm: Scaling language modeling with pathways, J. Mach. Learn. Res. 24 (2023) 240:1–240:113

    2023

  36. [36]

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, et al., Deepseek-coder: When the large language model meets programming - the rise of code intelligence, CoRR abs/2401.14196 (2024). arXiv:2401.14196, doi:10.48550/ARXIV.2401.14196

    Pith/arXiv arXiv doi:10.48550/arxiv.2401.14196 2024

  37. [37]

    Z. Li, Y. Zeng, Y. Zuo, W. Ren, W. Liu, et al., Knowcoder: Cod- ing structured knowledge into llms for universal information extraction, in: L. Ku, A. Martins, V. Srikumar (Eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, Association f...

    work page doi:10.18653/v1/2024.acl-long.475 2024

  38. [38]

    Q. Zeng, Y. Bai, Z. Tan, Z. Wu, S. Feng, et al., Codetaxo: Enhancing taxonomy expansion with limited examples via code language prompts, in: W. Che, J. Nabende, E. Shutova, M. T. Pilehvar (Eds.), Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025, Findings of ACL, Association for Computational Li...

    2025

  39. [39]

    Z. Bi, J. Chen, Y. Jiang, F. Xiong, W. Guo, et al., Codekgc: Code language model for generative knowledge graph construction, ACM Trans. Asian Low Resour. Lang. Inf. Process. 23 (3) (2024) 45. doi:10.1145/3641850

    doi:10.1145/3641850 2024

  40. [40]

    Touvron, T

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, et al., Llama: Open and efficient foundation language models, CoRR abs/2302.13971 (2023). arXiv:2302.13971, doi:10.48550/ARXIV.2302.13971

    Pith/arXiv arXiv doi:10.48550/arxiv.2302.13971 2023

  41. [41]

    Rozière, J

    B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, et al., Code llama: Open foundation models for code, CoRR abs/2308.12950 (2023). arXiv:2308.12950, doi:10.48550/ARXIV.2308.12950

    Pith/arXiv arXiv doi:10.48550/arxiv.2308.12950 2023

  42. [42]

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, et al., React: Synergizing reasoning and acting in language models, in: The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, OpenReview.net, 2023

    2023

  43. [43]

    S. Xu, S. Liu, T. Culhane, E. Pertseva, M. Wu, et al., Fine-tuned llms know more, hallucinate less with few-shot sequence-to-sequence seman- tic parsing over wikidata, in: Proceedings of the 2023 Conference on Em- pirical Methods in Natural Language Processing, EMNLP 2023, 2023, pp. 5778–5791

    2023

  44. [44]

    11779, 2019, pp

    M.Dubey, D.Banerjee, A.Abdelkawi, J.Lehmann, Lc-quad2.0: Alarge dataset for complex question answering over wikidata and dbpedia, in: The Semantic Web - ISWC 2019, Vol. 11779, 2019, pp. 69–78

    2019

  45. [45]

    W.Yih, M.Richardson, C.Meek, M.Chang, J.Suh, Thevalueofseman- tic parse labeling for knowledge base question answering, in: Proceed- ings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, 2016. 50

    2016

  46. [46]

    Y. Gu, S. Kase, M. Vanni, B. M. Sadler, P. Liang, et al., Beyond I.I.D.: three levels of generalization for question answering on knowledge bases, in: WWW ’21: The Web Conference 2021, 2021, pp. 3477–3488

    2021

  47. [47]

    Y. Wu, Y. Huang, N. Hu, Y. Hua, G. Qi, et al., Cotkr: Chain-of-thought enhancedknowledgerewritingforcomplexknowledgegraphquestionan- swering, in: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, 2024, pp. 3501–3520

    2024

  48. [48]

    S. Ko, H. Cho, H. Chae, J. Yeo, D. Lee, Evidence-focused fact sum- marization for knowledge-augmented zero-shot question answering, in: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, 2024, pp. 10636–10651

    2024

  49. [49]

    J. Ma, Z. Gao, Q. Chai, W. Sun, P. Wang, et al., Debate on graph: A flexible and reliable reasoning framework for large language models, in: T. Walsh, J. Shah, Z. Kolter (Eds.), Thirty-Ninth AAAI Confer- ence on Artificial Intelligence, Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence, Fifteenth Symposium on Educa- tional Ad...

    2025

  50. [50]

    D. Yu, S. Zhang, P. Ng, H. Zhu, A. H. Li, et al., Decaf: Joint decoding of answers and logical forms for question answering over knowledge bases, in: The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, OpenReview.net, 2023

    2023

  51. [51]

    Q. Dong, L. Li, D. Dai, C. Zheng, J. Ma, et al., A survey on in- context learning, in: Y. Al-Onaizan, M. Bansal, Y. Chen (Eds.), Pro- ceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, Association for Computational Linguistics, 2024, pp. 1107–1128. doi:10.18653/V1/2024.E...

    doi:10.18653/v1/2024.emnlp-main.64 2024

  52. [52]

    Y. Guo, Z. Li, X. Jin, Y. Liu, Y. Zeng, et al., Retrieval-augmented code generation for universal information extraction, in: Natural Language Processing and Chinese Computing - NLPCC 2024, Vol. 15360, 2024, pp. 30–42. 51

    2024

  53. [53]

    P. Li, T. Sun, Q. Tang, H. Yan, Y. Wu, et al., Codeie: Large code gener- ation models are better few-shot information extractors, in: Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics, ACL 2023, 2023, pp. 15339–15353

    2023

  54. [54]

    Vrandecic, M

    D. Vrandecic, M. Krötzsch, Wikidata: a free collaborative knowledge- base, Commun. ACM 57 (10) (2014) 78–85

    2014

  55. [55]

    Y. Xu, S. He, J. Chen, Z. Wang, Y. Song, et al., Generate-on-graph: Treat LLM as both agent and KG for incomplete knowledge graph question answering, in: Proceedings of the 2024 Conference on Em- pirical Methods in Natural Language Processing, EMNLP 2024, 2024, pp. 18410–18430

    2024

  56. [56]

    E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, et al., Lora: Low- rank adaptation of large language models, in: The Tenth International Conference on Learning Representations, ICLR 2022, 2022

    2022

  57. [57]

    Kaplan, S

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, et al., Scaling laws for neural language models, CoRR abs/2001.08361 (2020). arXiv:2001.08361

    Pith/arXiv arXiv 2001

  58. [58]

    D. Heim, L. Meyer, M. Schröder, J. Frey, A. Dengel, How do scaling laws applytoknowledgegraphengineeringtasks? theimpactofmodelsizeon large language model performance, in: M. Sabou, A. Harth, P. Lisena, E. Curry, B. Zhang, R. Alharbi, Y. He, G. Rehm, S. Schimmler, S. Di- etze, N.Manola, N.Fornara, V.Rodríguez-Doncel, J.Domingue, A.Ret- tinger, D. Trilling...

    2025