REVIEW 4 major objections 5 minor 31 references
Knowledge Graph Based Repository-Level Code Generation
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A knowledge-graph context, built from AST parsing and a two-hop dependency subgraph, lifts repository-level code generation to a 36.36 percent pass@1 on EvoCodeBench, the paper argues.
desk verdict The paper carefully describes a knowledge-graph retrieval pipeline but never runs the retrieval in its evaluation, so the headline pass@1 numbers support a different claim than the one made. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the repository knowledge graph, built by AST parsing and organised by a schema with node types File, Class, Method, Function, Attribute, and Generated Description, and relation types defines class, defines function, has method, used in, has attribute, and has description. Retrieval is hybrid: an LLM first identifies schema-relevant entities in the user query for full-text lookup, and a sentence encoder embeds the query for vector search over documentation and descriptions; the matched nodes are then expanded through $n$-hop graph traversal into a subgraph that a semantic reranker prunes to the top-$k$ most relevant nodes. In the experiments the expansion is a fixed two-hop traversal from the target function, and this pruned subgraph, with both code snippets and their relationship edges, is the context handed to the LLM for generation.
What would settle it
Run the full pipeline on EvoCodeBench using only the natural-language task description as input to the hybrid retrieval step, without revealing the target function, and measure pass@1; if the score drops to the no-context baseline level, the reported gains come from the known target rather than from the retrieval system.
Extended reading notes
Core claim
The paper's central claim is that a code repository, encoded as a knowledge graph with nodes for files, classes, methods, functions, attributes, and LLM-generated descriptions and edges for defining, containing, and using relationships, supplies the context that LLMs lack when generating code in an unfamiliar codebase. The pipeline parses the repository with ASTs, aligns elements to a fixed schema, ingests them into a graph database with full-text and vector indexes, and, in the evaluation, retrieves a two-hop subgraph starting from the known target function, prunes it by semantic similarity, and passes it to an LLM with a prompt that tells it to respect the captured relationships and dependencies. On EvoCodeBench, the paper reports pass@1 of 36.36 percent for its best model, 32.00 and 33.45 percent for two other LLMs, 7.27 to 20.73 percent for no-context and file-local baselines, and 36.02 percent for the strongest prior graph-based system. The paper also proposes a hybrid retrieval step for genuine user queries, combining entity identification by an LLM with full-text and vector search followed by $n$-hop expansion and semantic pruning, although the reported experiments use the target-anchored subgraph rather than exercising that query path.
Load-bearing premise
The evaluation assumes that expanding a two-hop subgraph from the already-known target function faithfully tests the proposed hybrid retrieval system, even though no natural-language query is processed and the target's location is given in advance.
Editorial extensions
If this is right
- Repository-level generation can move beyond file-local context: dependency and usage edges give the LLM a picture of how the target function fits into the rest of the codebase.
- If the gain transfers to query-driven retrieval, graph-grounded retrieval augmented generation becomes a credible upgrade over lexical or similarity-only code search.
- The cost of subgraph retrieval and semantic filtering grows with repository size, so practical deployments will need to trade context richness against latency.
- Extending the schema to decorators, variable types, and auxiliary files is a direct route to better context for complex queries.
- Because the schema is language-agnostic, porting the pipeline to non-Python repositories mainly requires a new parser and schema alignment.
Reading between the lines
- The reported edge over the strongest graph-based baseline is under half a percentage point, so a repeated-run comparison with multiple seeds would be needed to confirm the improvement is not sampling noise.
- Because the evaluation reveals the target node, the result is evidence that dependency-rich context helps generation, not yet evidence that the hybrid query retrieval works; running the full query pipeline on natural-language descriptions is the direct next experiment.
- The two-hop subgraph is guaranteed relevant by construction, so the make-or-break question for a real system is whether query-driven retrieval returns subgraphs of comparable relevance; retrieval precision, not generation, is the likely bottleneck.
- If the benefit really comes from dependency edges, a simpler static dependency extractor might capture much of the gain, leaving the knowledge graph's distinctive value in flexible query-time retrieval.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a knowledge-graph-based retrieval-augmented generation system for repository-level code generation. The methodology has three stages: parse a Python repository with ASTs and build a Neo4j knowledge graph with nodes for classes, methods, functions, attributes, files, and LLM-generated descriptions; retrieve an n-hop subgraph using a hybrid of full-text search, vector search, LLM-based entity identification, and semantic reranking; and generate a function or method body by passing the subgraph as context to an LLM. The approach is evaluated on EvoCodeBench with pass@1, reporting 36.36% with Claude 3.5 Sonnet, which the authors compare with CodeXGraph's 36.02% on GPT-4o and with EvoCodeBench baselines. The central claim is that the proposed hybrid retrieval improves repository-level code generation quality.
Significance. If the claims were supported, the paper would make a useful contribution: it provides a concrete three-stage pipeline for representing repository structure as a knowledge graph, combining several retrieval signals, and using the retrieved subgraph as context for code generation. The schema definition, the hybrid retrieval design, and the use of a public benchmark are strengths. However, the evaluation as reported does not exercise the retrieval system at all, so the central attribution of the improvements to the proposed retrieval approach is not established. The paper also compares against results taken from other papers under different evaluation conditions, so the headline margin over CodeXGraph is not reliable. The idea and the pipeline are plausible, but the current evidence does not support the paper's main conclusion.
major comments (4)
- [Section IV-B] The evaluation does not exercise the proposed retrieval system. The text states: 'We eliminate some steps from the original pipeline as we are not dealing with an actual natural language query. We already know our target and hence we only perform the graph-based retrieval for evaluation.' The subgraph is expanded from the known target function whose body is to be generated, so the context is anchored at the answer node by construction. This is an oracle condition: the relevant code is guaranteed to be in the context, and the full-text search, semantic search, entity identification, and reranking modules in Section III-B are never tested. Consequently, the pass@1 scores in Table I cannot support the central claim that the proposed hybrid retrieval improves repository-level code generation; they measure only the quality of context provided when the target is given.
- [Table I / Section V] The comparison against CodeXGraph and EvoCodeBench baselines is not controlled. CodeXGraph's 36.02% is on 212/275 samples while the proposed approach is on 275/275; the baselines are copied from prior papers rather than re-run under identical prompting and sampling conditions; and no confidence intervals, standard errors, or repeated-run variance are reported. The 0.34 percentage-point margin between 36.36% and 36.02% is therefore not established as a real difference.
- [Section IV-B / Section III-B] EvoCodeBench samples include a natural-language description of the expected functionality, as stated in Section IV-A ('expected functionality'). The paper's assertion that there is 'no actual natural language query' is therefore not justified, and the full hybrid retrieval pipeline could have been run using the provided descriptions as queries. The decision to substitute an oracle subgraph expansion is unexplained and makes the reported results incommensurable with the system described in the abstract and conclusion.
- [Section IV-B] The evaluation protocol for pass@1 is ambiguous. The text states that for each sample there are predefined test cases and 'the system's final performance is reported as the percentage of test cases passed across all samples,' which is a test-case-level aggregate, not a sample-level pass@1 metric. If samples differ in the number of test cases, the aggregate is not comparable to the pass@1 numbers from EvoCodeBench or CodeXGraph, and Table I does not make clear whether it reports the fraction of samples with all tests passing or the fraction of individual test cases passing.
minor comments (5)
- [Abstract] The abstract claims the method 'significantly outperforms the baseline approach,' but no statistical significance test is reported, and the margin over CodeXGraph is very small.
- [Section III-B] The values of the hyperparameters top_k_initial, n_hops, top_k_filter, and score_threshold are never given; they should be reported for reproducibility.
- [Section III-A] The embedding model is written as 'all-Mini-LM V6' but the standard model name is 'all-MiniLM-L6-v2'; correcting this would aid reproducibility.
- [Table I] The table title uses 'EvoCode Bench' while the text uses 'EvoCodeBench'; the names should be consistent.
- [Section V] No confidence intervals or variance are reported for any pass@1 score, which is particularly important for the close comparison with CodeXGraph.
Circularity Check
Evaluation never runs hybrid retrieval: the context is a two-hop subgraph anchored on the known target function, so the reported pass@1 cannot support the claim that hybrid retrieval improves code generation.
-
self definitional
[Section IV-B (Evaluation Methodology), with the credit assigned in Section V and Section VI]
"We eliminate some steps from the original pipeline as we are not dealing with an actual natural language query. We already know our target and hence we only perform the graph-based retrieval for evaluation. ... we retrieve a two-hop sub-graph starting from the target node that we want to generate the body for."
The paper's central contribution is the hybrid retrieval system of Section III-B (LLM entity identification, full-text search, vector search, n-hop expansion, semantic reranking). The evaluation, however, does not run that system: it starts from the known target function whose body is to be generated and expands a two-hop subgraph from it. Thus the retrieved context is relevant by construction, not because the retrieval method found it. Table I reports these results under 'Graph-based Retrieval,' and Section VI concludes that 'the code context retrieved by this hybrid system ...
full rationale
The paper contains no self-citation chains or imported uniqueness theorems. The single load-bearing circular step is the evaluation-to-claim mapping: the proposed hybrid retrieval system is nowhere tested, because Section IV-B explicitly replaces the query-based pipeline with an oracle that already knows the target node. The scores in Table I therefore measure graph-anchored context around the known answer, not retrieval quality, and the Section VI attribution of the improvement to the hybrid retrieval system is unsupported by the experiment. This is partial circularity in the central claim: the result is forced by the evaluation construction rather than by the proposed retrieval mechanism.
Assumptions & free parameters
free parameters (4)
- top_k_initial =
not reported
- n_hops =
2 (for evaluation)
- top_k_filter =
not reported
- score_threshold =
not reported
assumptions (5)
- domain assumption AST parsing of Python files captures all relevant code elements and their relations.
- domain assumption Embeddings of documentation and LLM-generated descriptions in all-MiniLM-V6 capture code semantics for retrieval.
- ad hoc to paper The two-hop subgraph around the target node provides sufficient and non-leaking context for generating the function body.
- domain assumption EvoCodeBench test cases correctly and independently judge the generated code.
- ad hoc to paper The evaluation setup is equivalent to the proposed system.
Cite this review
Pith. "Pith review of Knowledge Graph Based Repository-Level Code Generation." pith.science (2026). https://pith.science/paper/E5OWIOKR
@misc{pith2026250514394,
author = {Pith},
title = {Pith review of: Knowledge Graph Based Repository-Level Code Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/E5OWIOKR}},
note = {Machine review of arXiv:2505.14394}
}
read the original abstract
Recent advancements in Large Language Models (LLMs) have transformed code generation from natural language queries. However, despite their extensive knowledge and ability to produce high-quality code, LLMs often struggle with contextual accuracy, particularly in evolving codebases. Current code search and retrieval methods frequently lack robustness in both the quality and contextual relevance of retrieved results, leading to suboptimal code generation. This paper introduces a novel knowledge graph-based approach to improve code search and retrieval leading to better quality of code generation in the context of repository-level tasks. The proposed approach represents code repositories as graphs, capturing structural and relational information for enhanced context-aware code generation. Our framework employs a hybrid approach for code retrieval to improve contextual relevance, track inter-file modular dependencies, generate more robust code and ensure consistency with the existing codebase. We benchmark the proposed approach on the Evolutionary Code Benchmark (EvoCodeBench) dataset, a repository-level code generation benchmark, and demonstrate that our method significantly outperforms the baseline approach. These findings suggest that knowledge graph based code generation could advance robust, context-sensitive coding assistance tools.
Figures
Reference graph
Works this paper leans on
-
[1]
OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji, V . Balcom, P. Baltescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berner, L. Bogdonoff, O. Boiko, M. Boyd, A.-L. Brakman, G. Brockman, T. Brooks, M. Brunda...
arXiv 2024
-
[2]
Evaluating large language models trained on code,
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Her...
arXiv 2021
-
[3]
Application of large language models to software engi- neering tasks: Opportunities, risks, and implications,
I. Ozkaya, “Application of large language models to software engi- neering tasks: Opportunities, risks, and implications,” IEEE Software , vol. 40, no. 3, pp. 4–8, 2023
2023
-
[4]
What’s wrong with your code generated by large language models? an extensive study,
S. Dou, H. Jia, S. Wu, H. Zheng, W. Zhou, M. Wu, M. Chai, J. Fan, C. Huang, Y . Tao, Y . Liu, E. Zhou, M. Zhang, Y . Zhou, Y . Wu, R. Zheng, M. Wen, R. Weng, J. Wang, X. Cai, T. Gui, X. Qiu, Q. Zhang, and X. Huang, “What’s wrong with your code generated by large language models? an extensive study,” 2024. [Online]. Available: https://arxiv.org/abs/2407.06153
arXiv 2024
-
[5]
Large language models for software engineering: A systematic literature review,
X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review,” 2024. [Online]. Available: https://arxiv.org/abs/2308.10620
arXiv 2024
-
[6]
Beyond functional correctness: Investigating coding style inconsistencies in large language models,
Y . Wang, T. Jiang, M. Liu, J. Chen, and Z. Zheng, “Beyond functional correctness: Investigating coding style inconsistencies in large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2407.00456
arXiv 2024
-
[7]
Code search: A survey of techniques for finding code,
L. Di Grazia and M. Pradel, “Code search: A survey of techniques for finding code,” ACM Computing Surveys , vol. 55, no. 11, p. 1–31, Feb
-
[8]
Prompt-based Code Completion via Multi-Retrieval Augmented Generation
H. Tan, Q. Luo, L. Jiang, Z. Zhan, J. Li, H. Zhang, and Y . Zhang, “Prompt-based code completion via multi-retrieval augmented generation,” 2024. [Online]. Available: https://arxiv.org/abs/2405.07530
work page Pith review arXiv 2024
Show all 31 references
-
[9]
Retrieval-augmented generation for knowledge- intensive nlp tasks,
P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt ¨aschel, S. Riedel, and D. Kiela, “Retrieval-augmented generation for knowledge- intensive nlp tasks,” in Advances in Neural Information Processing Systems, H. Laroc...
2020
-
[10]
A review of prominent paradigms for llm-based agents: Tool use (including rag), planning, and feedback learning,
X. Li, “A review of prominent paradigms for llm-based agents: Tool use (including rag), planning, and feedback learning,” 2024. [Online]. Available: https://arxiv.org/abs/2406.05804
2024 arXiv
-
[11]
Agilecoder: Dynamic collaborative agents for software development based on agile methodology,
M. H. Nguyen, T. P. Chau, P. X. Nguyen, and N. D. Q. Bui, “Agilecoder: Dynamic collaborative agents for software development based on agile methodology,” 2024. [Online]. Available: https: //arxiv.org/abs/2406.11912
2024 arXiv
-
[12]
Evocodebench: An evolving code generation benchmark aligned with real-world code repositories,
J. Li, G. Li, X. Zhang, Y . Dong, and Z. Jin, “Evocodebench: An evolving code generation benchmark aligned with real-world code repositories,” 2024. [Online]. Available: https://arxiv.org/abs/2404.00599
2024 arXiv
-
[13]
Finetuned language models are zero-shot learners,
J. Wei, M. Bosma, V . Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V . Le, “Finetuned language models are zero-shot learners,” in International Conference on Learning Representations , 2022. [Online]. Available: https://openreview.net/forum?id=gEZrGCozdqR
2022
-
[14]
Scaling instruction-finetuned language models,
H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y . Tay, W. Fedus, Y . Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. S. Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, A. Castro-Ros, M. Pellat, K. Robinson, D. Valter, S. Narang, G. Mishra, A. Yu, V . Zhao, Y . Huang, A. Dai, H. Y...
2022 arXiv
-
[15]
A survey on large language models for code generation,
J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim, “A survey on large language models for code generation,” 2024. [Online]. Available: https://arxiv.org/abs/2406.00515
2024 arXiv
-
[16]
Enhancing repository-level code generation with integrated contextual information,
Z. Pan, X. Hu, X. Xia, and X. Yang, “Enhancing repository-level code generation with integrated contextual information,” 2024. [Online]. Available: https://arxiv.org/abs/2406.03283
2024
-
[17]
Repocoder: Repository-level code completion through iterative retrieval and generation,
F. Zhang, B. Chen, Y . Zhang, J. Keung, J. Liu, D. Zan, Y . Mao, J.-G. Lou, and W. Chen, “Repocoder: Repository-level code completion through iterative retrieval and generation,” 2023. [Online]. Available: https://arxiv.org/abs/2303.12570
2023 arXiv
-
[18]
Repofusion: Training code models to understand your repository,
D. Shrivastava, D. Kocetkov, H. de Vries, D. Bahdanau, and T. Scholak, “Repofusion: Training code models to understand your repository,”
-
[19]
Leveraging passage retrieval with generative models for open domain question answering,
G. Izacard and E. Grave, “Leveraging passage retrieval with generative models for open domain question answering,” in Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main V olume , P. Merlo, J. Tiedemann, and R. Tsar...
2021
-
[20]
A probabilistic model of information retrieval: development and comparative experiments - part 1,
K. S. Jones, S. Walker, and S. E. Robertson, “A probabilistic model of information retrieval: development and comparative experiments - part 1,” Inf. Process. Manag. , vol. 36, pp. 779–808, 2000. [Online]. Available: https://api.semanticscholar.org/CorpusID:1965284
2000
-
[21]
Available: https://arxiv.org/abs/2306.10998
[Online]. Available: https://arxiv.org/abs/2306.10998
-
[22]
The rise and potential of large language model based agents: A survey,
Z. Xi, W. Chen, X. Guo, W. He, Y . Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou, R. Zheng, X. Fan, X. Wang, L. Xiong, Y . Zhou, W. Wang, C. Jiang, Y . Zou, X. Liu, Z. Yin, S. Dou, R. Weng, W. Cheng, Q. Zhang, W. Qin, Y . Zheng, X. Qiu, X. Huang, and T. Gui, “The rise and ...
2023 arXiv
-
[23]
Large language model based multi-agents: A survey of progress and challenges,
T. Guo, X. Chen, Y . Wang, R. Chang, S. Pei, N. V . Chawla, O. Wiest, and X. Zhang, “Large language model based multi-agents: A survey of progress and challenges,” 2024. [Online]. Available: https://arxiv.org/abs/2402.01680
2024 arXiv
-
[24]
Coderag-bench: Can retrieval augment code generation?
Z. Z. Wang, A. Asai, X. V . Yu, F. F. Xu, Y . Xie, G. Neubig, and D. Fried, “Coderag-bench: Can retrieval augment code generation?”
-
[25]
Graphcoder: Enhancing repository-level code completion via code context graph-based retrieval and language model,
W. Liu, A. Yu, D. Zan, B. Shen, W. Zhang, H. Zhao, Z. Jin, and Q. Wang, “Graphcoder: Enhancing repository-level code completion via code context graph-based retrieval and language model,” ArXiv, vol. abs/2406.07003, 2024. [Online]. Available: https://api.semanticscholar. org/C...
2024 arXiv
-
[26]
Codexgraph: Bridging large language models and code repositories via code graph databases,
X. Liu, B. Lan, Z. Hu, Y . Liu, Z. Zhang, F. Wang, M. Shieh, and W. Zhou, “Codexgraph: Bridging large language models and code repositories via code graph databases,” 2024. [Online]. Available: https://arxiv.org/abs/2408.03910
2024 arXiv
-
[27]
Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers,
W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou, “Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers,” 2020. [Online]. Available: https://arxiv. org/abs/2002.10957
2020 arXiv
-
[28]
Agentcoder: Multi-agent-based code generation with iterative testing and optimisation,
D. Huang, J. M. Zhang, M. Luck, Q. Bu, Y . Qing, and H. Cui, “Agentcoder: Multi-agent-based code generation with iterative testing and optimisation,” 2024. [Online]. Available: https://arxiv.org/abs/2312. 13010
2024
-
[2023]
Available: http://dx.doi.org/10.1145/3565971
[Online]. Available: http://dx.doi.org/10.1145/3565971
-
[2024]
Available: https://arxiv.org/abs/2406.14497
[Online]. Available: https://arxiv.org/abs/2406.14497
-
[9474]
Available: https://proceedings.neurips.cc/paper files/ paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf
[Online]. Available: https://proceedings.neurips.cc/paper files/ paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf
2020
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.