Pith. sign in

REVIEW 4 major objections 5 minor 70 references

Repository-level code reasoning is a distinct, largely unsolved ability for LLMs: even with oracle context—the exact files a test touches—the best evaluated model scores only 69.1% on output prediction.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 00:52 UTC pith:2P7RRTFG

load-bearing objection Useful benchmark with a clean oracle design; the absolute ceiling numbers rest on a dynamic tracer the paper does not validate, so treat them as conditional. the 4 major comments →

arxiv 2607.25996 v1 pith:2P7RRTFG submitted 2026-07-28 cs.SE

RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models

Yanlin Wang , Suiquan Wang , Yanli Wang , Bowen Zhang , Daya Guo , Jiachi Chen , Zibin Zheng This is my paper
classification cs.SE
keywords repository-level code reasoninglarge language modelsbenchmarkoutput predictioncall chain predictiondynamic tracingmemorization vs reasoninglong-context
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

RepoReasoner asks whether large language models can reason about code the way developers do: across file boundaries, tracking state through a whole repository, not just inside one function. The paper builds a benchmark from real Python projects by dynamically tracing pytest runs to capture ground-truth file-level call chains, then challenges models to predict a masked assertion's value (Output Prediction) and to identify which files a test actually invokes from a noisy pool (Call Chain Prediction). Even with perfect oracle context, the best evaluated model answers only 69.1% of Output Prediction tasks correctly, and no model reconstructs full call chains reliably—precision is high, recall low. Models also lose accuracy when the same logic is rewritten with different inputs and outputs, revealing partial reliance on memorization, and longer contexts sometimes hurt more than they help. If these findings hold, cross-file inference is the bottleneck, not retrieval or context length.

Core claim

The paper's central claim is that contemporary LLMs cannot reliably perform repository-level code reasoning even when the exact files a test touches are supplied. Under oracle context with 10k tokens of ground-truth source code, the best evaluated model reaches only 69.1% Pass@1 on Output Prediction, and on Call Chain Prediction the oracle setting's best F1 is 66.3% with 21.3% exact match. The high-precision/low-recall pattern is interpreted as evidence that models recognize direct imports but lose multi-hop, runtime-dependent dependencies; the universal drop on I/O-rewritten data is interpreted as evidence of partial memorization; and the mixed effect of longer contexts indicates that noise

What carries the argument

The load-bearing machinery is the benchmark construction pipeline. A custom pytest plugin built on the hunter tracing library records every function call during test runs, capturing each call's origin file, destination file, and the sequence of file-level transitions; this dynamic trace provides both the ground-truth call chains and the oracle context for evaluation. An LLM-assisted rewriting stage alters input values and expected outputs while preserving logic, producing instances less likely to be memorized, and assertion masking turns real tests into Output Prediction tasks while file pools mix signal files with distractors for Call Chain Prediction. Evaluation then measures Pass@k for ou

Load-bearing premise

Section 3.2.2 asserts that the tracer 'records every function call during the test runs,' but no completeness check is reported; if calls through C extensions, dynamic imports, eval/exec, or pytest fixtures are missed, the oracle context omits needed files and the call-chain labels are wrong, making both the 69.1% ceiling and the low recall numbers artificially low.

What would settle it

Run a sample of RepoReasoner test suites under a second, independent tracing method—for example, a sys.settrace-based profiler or static call-graph analysis—and compare the resulting file-level call chains. If the two tracers disagree on a nontrivial fraction of instances, or if model recall rises sharply when the oracle context is replaced by a broader static-analysis-reachable file set, the reported performance ceiling and recall figures are artifacts of the tracer rather than genuine model limits.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A perfect-context ceiling near 69% on Output Prediction implies that repository-level reasoning is a separate capability from retrieval, and current models have not mastered it.
  • High precision but low recall on Call Chain Prediction means models tend to confirm direct dependencies but miss the deeper structural paths that define real architectures, so architectural understanding remains incomplete.
  • The consistent drop on I/O-rewritten data implies benchmarks that use only original tests overstate genuine reasoning; test-driven rewriting should become a standard part of code-reasoning evaluation.
  • Increasing context from 10k to 30k tokens did not reliably improve results and sometimes lowered them, so future long-context systems need noise-robust attention, not just larger windows.
  • The automated pipeline—Docker environment setup, dynamic tracing, rewriting, and ground-truth packaging—is designed to extend to larger repositories and other languages, making the benchmark scaffold reusable.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The tracer's completeness is the key unverified link: calls made through C extensions, dynamic imports, eval/exec, or pytest fixture machinery may be invisible to the hunter-based trace. If so, oracle contexts would omit files models actually need, making the 69.1% ceiling and low recall artificially pessimistic.
  • The high-precision/low-recall signature is exactly what a model that reasons over import statements rather than runtime dispatch would produce. A natural next experiment is to compare RepoReasoner results against a pure static import-graph baseline to separate 'tracing architecture' from 'recalling imports.'
  • Since the rewriting stage changes only inputs and expected outputs, it tests robustness to surface variation, not to algorithmic novelty. Refactoring function structure or reordering call chains—an extension the paper explicitly leaves for future work—would likely widen the memorization gap.
  • Because the tracing approach is Python-specific (pytest and hunter), extension to other languages will require a different dynamic-analysis backend, so cross-language generalization is a real constraint on the benchmark's reach.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces RepoReasoner, a benchmark for repository-level code reasoning with two tasks: Output Prediction, where a model fills a masked assertion by simulating cross-file execution, and Call Chain Prediction, where a model identifies the set/sequence of files invoked by a test from a noisy file pool. Ground truth is produced by a custom pytest plugin plus the hunter tracing library inside Dockerized repositories, augmented by LLM-based I/O rewriting intended to reduce memorization. Seven LLMs are evaluated under oracle and retrieval contexts. The central findings are that even with oracle context the best model reaches only 69.1% Pass@1 on Output Prediction, and that models show high precision but low recall on Call Chain Prediction, with universal performance drops on rewritten data and mixed effects from longer contexts.

Significance. If the benchmark is valid, it addresses a real gap: existing code-reasoning benchmarks largely operate at function level, while repository-level reasoning is both practically important and under-evaluated. The design has genuine strengths: ground truth comes from real execution rather than static heuristics; the oracle-context setup cleanly isolates reasoning from retrieval; and the I/O rewriting is a sensible first attempt to detect memorization. The error analysis (RQ5) is also useful. However, the benchmark's load-bearing assumption is that the dynamic tracer captures every relevant file-level call, and the paper does not validate this. Several additional issues — oracle context truncation, a confounded memorization comparison, and missing statistical uncertainty — affect the strength of the headline claims. These are correctable, so the contribution is potentially solid but not yet established as presented.

major comments (4)
  1. [§3.2.2, Fig. 1, §6.1] The completeness of the dynamic tracer is load-bearing for both tasks. The paper states the tracer 'records every function call' and that the oracle context is 'complete', but no validation is reported. The Fig. 1 example uses joblib Parallel with n_jobs=2; joblib's loky backend runs in child processes, and a Python-level tracer installed only in the parent process would miss those calls. C extensions, eval/exec, and dynamic imports are also plausible blind spots. If any file is missed, the RQ1 oracle context is not perfect (lowering the measured ceiling) and the RQ2 ground-truth call chains are incomplete (lowering recall). Section 6.1 concedes that pipeline bugs 'could introduce errors' and defers 'more thorough validation' to future work. Please add a concrete completeness audit — e.g., compare tracer output against static call graphs, run known tests with subprocesses, or manually tr
  2. [§4.3.3, §5.1] The oracle context is described as containing 'exactly the ground-truth files' and 'minimal yet complete', but RQ1 uses '10k tokens containing exactly the ground-truth files'. The paper does not state what happens when the ground-truth call-chain files exceed 10k tokens. If the context is truncated, the oracle is not complete and the 69.1% Pass@1 ceiling is an underestimate of model capability. If instances with oversized contexts are excluded, the benchmark may be biased toward simpler tests. Please specify the handling (truncation, exclusion, or per-instance token budget) and report how many instances are affected.
  3. [§3.2.3, Table 3, RQ3] The memorization conclusion is confounded with task difficulty. The rewriting pipeline changes input values and expected outputs while preserving code logic, so the rewritten instances may simply be harder for any reasoner — e.g., more unusual constants, edge-case types, or less familiar value distributions — rather than exposing memorization. The paper provides no control for inherent difficulty (such as measuring a non-memorizing baseline or matching value distributions between original and rewritten sets). Without such a control, the universal performance drop in Table 3 does not isolate memorization. Please add a difficulty-matching analysis or a control condition that separates reasoning difficulty from training-data familiarity.
  4. [§4.2, Eq. (1), Table 1] Pass@1 is computed as the correctness of the first of five stochastic samples (Eq. (1) with k=1), and the paper reports no confidence intervals or significance tests. The headline ceiling of 69.1% is thus a single-draw point estimate with variance, and model comparisons such as 'R1-Distill-Qwen-14B performs worse than its base model' could easily be noise. Since the paper makes strong claims about ceilings and rankings, please report confidence intervals (e.g., bootstrap) or use the standard unbiased Pass@k estimator over the five samples, and indicate whether Pass@1 is actually the first-sample criterion.
minor comments (5)
  1. [§3.3] Figure 3's call-chain section lists 'initial.py, domain.py, filed.py' — 'filed.py' appears to be a typo for 'field.py'. Please check filenames.
  2. [§4.3.2] The definition of EM for Call Chain Prediction says 'identical to the ground-truth sequence', but the task description says 'list of files'. Clarify whether the order of files matters in the ground truth and in the EM metric.
  3. [§5.3] Table 3's I/O-Rewritten columns do not report whether the rewritten instances were also evaluated under the same oracle context length and whether the same token truncation rules applied.
  4. [§6.1] The sentence 'we rigorously tested these tools across diverse codebases' is vague. If a validation was performed, it should be described in detail or moved to an appendix; otherwise the claim is unverifiable.
  5. [Abstract] The phrase 'scientific repositories' and 'scientific domains' in §3.2.1 is used but the benchmark could benefit from a breakdown of the 14 included repositories and their domains.

Circularity Check

0 steps flagged

No significant circularity: ground truth is execution-derived, not fitted to the models being evaluated.

full rationale

RepoReasoner's central claims (RQ1's 69.1% Pass@1 ceiling under oracle context, RQ2's high-precision/low-recall call-chain pattern, RQ3's memorization drop) are empirical measurements over a benchmark whose ground truth is produced by dynamic tracing of real pytest executions and by re-executing LLM-rewritten tests. No parameter is fitted to model outputs and then renamed as a prediction; the oracle context is assembled from the execution trace, not from the model's answers. The paper's use of GPT-4 to generate I/O-rewritten instances while evaluating GPT-4.1-Mini and other models is a distributional confound, not a circular derivation, because the rewritten ground-truth answers come from the execution trace, not from the generating model. Related-work citations include the authors' own prior papers, but none is load-bearing in the sense of replacing an argument with a self-citation: no uniqueness theorem or ansatz is imported to force the benchmark design. Section 6.1's 'Benchmark Correctness' caveat that dynamic-tracer bugs could affect ground truth is a validity threat—potentially harming the completeness of the oracle context—but it is an external evidence-quality limitation rather than a reduction of the results to their own inputs. The benchmark is self-contained against the evaluated models: the tasks, labels, and contexts derive from separate runtime instrumentation and assertion execution, so no circularity score above 0 is warranted.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The benchmark's validity rests on four domain assumptions: the tracer's completeness, the I/O rewriting equivalence, the exact-match evaluation, and the representativeness of the curated repositories. There are no fitted parameters and no invented entities; ground truth comes from execution, not from the models being measured.

axioms (4)
  • domain assumption Dynamic tracing via hunter captures the complete file-level call chain for each pytest test.
    Section 3.2.2 uses it to build Oracle context and call-chain ground truth; misses would systematically lower measured performance.
  • domain assumption LLM-based I/O rewriting preserves logical equivalence and only removes memorization, not changing problem difficulty.
    Section 3.2.3 relies on this to attribute performance drops in RQ3 to memorization; the rewritten inputs introduce edge cases and different data types, which may instead increase difficulty.
  • domain assumption Exact-match evaluation against a small answer set is a fair measure of output correctness.
    Sections 4.3.1 and 6.1; semantically correct but differently formatted answers are counted as errors, and the authors acknowledge this threat.
  • domain assumption The 14 selected Python repositories with passing tests are representative of repository-level coding tasks.
    Section 3.2 selection criteria; acknowledged in Section 6.2 that results may not generalize beyond Python/scientific libraries.

pith-pipeline@v1.3.0-alltime-deepseek · 21817 in / 12511 out tokens · 114067 ms · 2026-08-01T00:52:35.493218+00:00 · methodology

0 comments
read the original abstract

Recent large language models (LLMs) have shown strong performance on software engineering tasks, yet most existing benchmarks evaluate code reasoning at the function level, where all relevant information is localized. This setting fails to reflect real-world development, which requires reasoning across multiple files and complex dependency structures. We introduce RepoReasoner, a benchmark for evaluating repository-level code reasoning. It assesses two complementary abilities: Output Prediction, which measures fine-grained, stateful execution reasoning across files, and Call Chain Prediction, which evaluates high-level architectural dependency understanding under noisy context. Our benchmark is constructed through a multi-stage pipeline that leverages dynamic tracing of pytest executions to obtain ground-truth call chains, along with LLM-based I/O rewriting to reduce memorization effects. We evaluate seven state-of-the-art LLMs. Even under oracle context, the best-performing model achieves only 69.1% Pass@1 on Output Prediction, indicating that cross-file reasoning remains a major challenge. In Call Chain Prediction, models exhibit high precision but low recall, suggesting limited multi-hop dependency understanding. Furthermore, performance drops on rewritten data reveal partial reliance on memorization, and longer contexts do not consistently improve results due to noise. These findings highlight fundamental limitations in current LLMs' repository-level reasoning and motivate future work on structured architectural understanding and cross-file inference.

Figures

Figures reproduced from arXiv: 2607.25996 by Bowen Zhang, Daya Guo, Jiachi Chen, Suiquan Wang, Yanlin Wang, Yanli Wang, Zibin Zheng.

Figure 1
Figure 1. Figure 1: An illustration of the two tasks in RepoReasoner. The left panel shows an example of the [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: An overview of RepoReasoner construction pipeline. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Representative example. LLMs need to understand the code through the call chain order of files. 4 Experimental Design This section outlines the experimental methodology we employed to evaluate the repository-level code reasoning capabilities of LLMs. We detail the model selection, the strategies for providing long-context information, the core research questions guiding our investigation, and the specific … view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of F1 Scores (Bar Chart, Left Y-axis) and EM Scores (Line Chart, Right Y-axis) Across [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Case of reasoning influenced by memorization. [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: presents the error distribution across all evaluated models under 10k context with original data. 0 20 40 60 80 100 Percentage of Total Errors (%) Qwen-2.5-14B Qwen-2.5-Coder-14B R1-Distill-Qwen-14B GPT-4.1-Mini Gemini-2.5-Flash Qwen3-235B DeepSeek-R1 35.3 14.7 50.0 5.2 14.4 80.4 21.8 19.1 59.1 2.6 14.5 82.9 30.3 16.9 52.8 27.3 17.1 55.7 7.5 11.9 80.6 Output Prediction Error Distribution (Original, 10k Con… view at source ↗
Figure 7
Figure 7. Figure 7: Example of Formatting/Extraction Error: The model correctly understands that the function returns [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Example of Identifier Confusion: The model correctly identifies the value of a specific internal variable [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Example of Wrong Value Error: The model predicts [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Distribution of error types in Call Chain Prediction task. The three categories are: Unparseable/Empty, Invalid Path Style, and Wrong Answer.1 signal), while larger models maintain structured outputs (3.3–16.1%), indicating that model scale governs robustness to noise in instruction-following. Invalid Path Style errors remain consistently low (≤8%) across all models and settings, confirming that path-form… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

70 extracted references · 1 canonical work pages

  1. [1]

    2025. Docker. https://www.docker.com. (2025)

    2025

  2. [2]

    2025. hunter. https://github.com/ionelmc/python-hunter. (2025)

    2025

  3. [3]

    Georgy Andryushchenko, Vladimir Ivanov, Vladimir Makharev, Elizaveta Tukhtina, and Aidar Valeev. 2024. Leveraging Large Language Models in Code Question Answering: Baselines and Issues. arXiv:2411.03012 [cs.CL] https://arxiv. org/abs/2411.03012

    Pith/arXiv arXiv 2024

  4. [4]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models.arXiv preprint arXiv:2108.07732 (2021)

    Pith/arXiv arXiv 2021

  5. [5]

    Ramakrishna Bairi, Atharva Sonwane, Aditya Kanade, Vageesh Iyer, Suresh Parthasarathy, Sriram Rajamani, Nikitha B., Rahul Ashok, Shashank Shet, Akash Jain, et al. 2023. CodePlan: Repository-level Coding using LLMs and Planning. arXiv preprint arXiv:2309.12499(2023). https://arxiv.org/abs/2309.12499

    Pith/arXiv arXiv 2023

  6. [6]

    Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. 2024. Reasoning runtime behavior of a program with llm: How far are we?arXiv preprint arXiv:2403.16437(2024)

    Pith/arXiv arXiv 2024

  7. [7]

    Jialiang Chen, Kaifa Zhao, Jie Liu, Chao Peng, Jierui Liu, Hang Zhu, Pengfei Gao, Ping Yang, and Shuiguang Deng

  8. [8]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374(2021)

    Pith/arXiv arXiv 2021

  9. [9]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems.arXiv preprint arXiv:2110.14168(2021)

    Pith/arXiv arXiv 2021

  10. [10]

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261(2025)

    Pith/arXiv arXiv 2025

  11. [11]

    Yangruibo Ding, Zijian Wang, Wasi Ahmad, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2024. CoCoMIC: Code Completion by Jointly Modeling In-file and Cross-file Context. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Ni...

    2024

  12. [12]

    Aryaz Eghbali and Michael Pradel. 2024. De-hallucinator: Iterative Grounding for LLM-based Code Completion.arXiv preprint arXiv:2403.04642(2024)

    Pith/arXiv arXiv 2024

  13. [13]

    Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. 2023. Large language models for software engineering: Survey and open problems. (2023), 31–53

    2023

  14. [14]

    Lingyue Fu, Huacan Chai, Shuang Luo, Kounianhua Du, Weiming Zhang, Longteng Fan, Jiayi Lei, Renting Rui, Jianghao Lin, Yuchen Fang, et al. 2023. Codeapex: A bilingual programming evaluation benchmark for large language models. arXiv preprint arXiv:2309.01940(2023)

    Pith/arXiv arXiv 2023

  15. [15]

    Rao Fu, Ziyang Luo, Hongzhan Lin, Zhen Ye, and Jing Ma. 2024. ScratchEval: Are GPT-4o Smarter than My Child? Evaluating Large Multimodal Models with Visual Programming Challenges.arXiv preprint arXiv:2411.18932(2024)

    Pith/arXiv arXiv 2024

  16. [16]

    Jing Gong, Yanghui Wu, Linxi Liang, Yanlin Wang, Jiachi Chen, Mingwei Liu, and Zibin Zheng. 2026. CoSQA+: Enhancing Code Search Evaluation With a Multi-Choice Benchmark and Test-Driven Agents.IEEE Transactions on Software Engineering52, 1 (2026), 206–220. doi:10.1109/TSE.2025.3631886

    arXiv 2026

  17. [17]

    Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. 2024. Cruxeval: A benchmark for code reasoning, understanding and execution.arXiv preprint arXiv:2401.03065(2024)

    Pith/arXiv arXiv 2024

  18. [18]

    Wenchao Gu, Juntao Chen, Yanlin Wang, Tianyue Jiang, Xingzhe Li, Mingwei Liu, Xilin Liu, Yuchi Ma, and Zibin Zheng. 2025. What to retrieve for effective retrieval-augmented code generation? an empirical study and beyond. arXiv preprint arXiv:2503.20589(2025)

    Pith/arXiv arXiv 2025

  19. [19]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948(2025)

    Pith/arXiv arXiv 2025

  20. [20]

    Lianghong Guo, Wei Tao, Runhan Jiang, Yanlin Wang, Jiachi Chen, Xilin Liu, Yuchi Ma, Mingzhi Mao, Hongyu Zhang, and Zibin Zheng. 2025. OmniGIRL: A Multilingual and Multimodal Benchmark for GitHub Issue Resolution.Proc. ACM Softw. Eng.2, ISSTA, Article ISSTA002 (June 2025), 23 pages. doi:10.1145/3728871

    work page doi:10.1145/3728871 2025

  21. [21]

    Shirley Anugrah Hayati, Raphael Olivier, Pravalika Avvaru, Pengcheng Yin, Anthony Tomasic, and Graham Neubig

  22. [22]

    Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. 2024. MetaGPT: Meta programming for a multi-agent collaborative framework. International Conference on Learning Representations, ICLR

    2024

  23. [23]

    Ruida Hu, Chao Peng, Jingyi Ren, Bo Jiang, Xiangxin Meng, Qinyun Wu, Pengfei Gao, Xinchen Wang, and Cuiyun Gao. 2024. CodeRepoQA: A Large-scale Benchmark for Software Engineering Question Answering.arXiv preprint arXiv:2412.14764(2024)

    Pith/arXiv arXiv 2024

  24. [24]

    Dong Huang, Qingwen Jia, Jianbo Fan, Anh Tuan Luu, and Shuicheng Li. 2023. AgentCoder: Multi-agent-based Code Generation with Iterative Testing and Optimisation.arXiv preprint arXiv:2312.13010(2023)

    Pith/arXiv arXiv 2023

  25. [25]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical report.arXiv preprint arXiv:2409.12186(2024)

    Pith/arXiv arXiv 2024

  26. [27]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues? (2024). https://openreview.net/forum?id= VTF8yNQM66

    2024

  27. [28]

    Changyoon Lee, Yeon Seonwoo, and Alice Oh. 2022. CS1QA: A Dataset for Assisting Code-based Question Answering in an Introductory Programming Course. InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2022). 2017–2028. https:// aclanthology.org/2022.naacl-m...

    2022

  28. [29]

    Jia Li, Xuyuan Guo, Lei Li, Kechi Zhang, Ge Li, Zhengwei Tao, Fang Liu, Chongyang Tao, Yuqi Zhu, and Zhi Jin

  29. [30]

    Jia Li, Ge Li, Zhuo Shi, Zhi Zhang, and Zhi Jin. 2023. ACECoder: Utilizing Existing Code to Enhance Code Generation. arXiv preprint arXiv:2303.17780(2023). https://arxiv.org/abs/2303.17780

    Pith/arXiv arXiv 2023

  30. [31]

    Jia Li, Yongmin Li, Ge Li, Xing Hu, Xin Xia, and Zhi Jin. 2021. EditSum: A Retrieve-and-Edit Framework for Source Code Summarization. In2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 155–166. doi:10.1109/ASE51524.2021.9678724

    arXiv 2021

  31. [32]

    LONGCODEU: Benchmarking Long-Context Language Models on Long Code Understanding.arXiv preprint arXiv:2503.04359(2025)

    arXiv 2025

  32. [33]

    Dianshu Liao, Shidong Pan, Qing Huang, Xiaoxue Ren, Zhenchang Xing, Huan Jin, and Qinying Li. 2023. Context-aware code generation framework for code repositories: Local, global, and third-party library awareness.CoRR(2023)

    2023

  33. [34]

    Changshu Liu, Yang Chen, and Reyhaneh Jabbarvand. 2024. CodeMind: Evaluating Large Language Models for Code Reasoning.arXiv preprint arXiv:2402.09664(2024)

    Pith/arXiv arXiv 2024

  34. [35]

    Ming Liang, Xiaohong Chen, Jing Liu, Hongyu Yu, and Dong Liang. 2024. RepoFuse: Repository-level Code Completion with Fused Dual Context.arXiv preprint arXiv:2402.14323(2024)

    Pith/arXiv arXiv 2024

  35. [36]

    Jiawei Liu, Jia Le Tian, Vijay Daita, Yuxiang Wei, Yifeng Ding, Yuhan Katherine Wang, Jun Yang, and Lingming Zhang

  36. [37]

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2022. ReACC: A Retrieval-Augmented Code Completion Framework. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 6227–6240

    2022

  37. [38]

    Chenxiao Liu and Xiaojun Wan. 2021. CodeQA: A Question Answering Dataset for Source Code Comprehension. Findings of the Association for Computational Linguistics: EMNLP 2021(2021), 1481–1492. https://aclanthology.org/ 2021.findings-emnlp.223.pdf

    2021

  38. [39]

    Dung Nguyen Manh, Thang Phan Chau, Nam Le Hai, Thong T Doan, Nam V Nguyen, Quang Pham, and Nghi DQ Bui. 2024. CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs.arXiv preprint arXiv:2410.01999(2024)

    Pith/arXiv arXiv 2024

  39. [40]

    OpenAI. 2025. GPT-4.1: A Multimodal Large Language Model. https://openai.com/index/gpt-4-1/ Accessed: 2025-09-09

    2025

  40. [41]

    Guangsheng Ou, Mingwei Liu, Yuxuan Chen, Yanlin Wang, Xin Peng, and Zibin Zheng. 2025. RustRepoTrans: Repository-level Context Code Translation Benchmark Targeting Rust. (2025), 610–622. doi:10.1109/ASE63991.2025. 00057

    arXiv 2025

  41. [42]

    Lezhi Ma, Shangqing Liu, Lei Bu, Shangru Li, Yida Wang, and Yang Liu. 2024. Speceval: Evaluating code comprehension in large language models via program specifications.arXiv preprint arXiv:2409.12866(2024)

    Pith/arXiv arXiv 2024

  42. [43]

    Nguyen, and Nghi D

    Huy Duc Phan, Phong X. Nguyen, and Nghi D. Q. Le. 2024. RepoHyper: Better Context Retrieval Is All You Need for Repository-Level Code Completion.arXiv preprint arXiv:2403.06095(2024)

    Pith/arXiv arXiv 2024

  43. [44]

    Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. 2023. Communicative agents for software development.arXiv preprint arXiv:2307.079246, 3 (2023), 1. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE124. Publication date: July 2026. RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Co...

    Pith/arXiv arXiv 2023

  44. [45]

    Monoshi Kumar Roy, Simin Chen, Benjamin Steenhoek, Jinjun Peng, Gail Kaiser, Baishakhi Ray, and Wei Le. 2025. CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoning.arXiv preprint arXiv:2506.00750(2025)

    arXiv 2025

  45. [46]

    Md Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Retrieval augmented code generation and summarization.arXiv preprint arXiv:2108.11601(2021)

    Pith/arXiv arXiv 2021

  46. [47]

    Yanlin Wang, Yanli Wang, Daya Guo, Jiachi Chen, Ruikai Zhang, Yuchi Ma, and Zibin Zheng. 2024. Rlcoder: Reinforce- ment learning for repository-level code completion.arXiv preprint arXiv:2407.19487(2024)

    Pith/arXiv arXiv 2024

  47. [48]

    Yanli Wang, Yanlin Wang, Suiquan Wang, Daya Guo, Jiachi Chen, John Grundy, Xilin Liu, Yuchi Ma, Mingzhi Mao, Hongyu Zhang, and Zibin Zheng. 2024. RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation.arXiv e-prints, Article arXiv:2412.17744 (Dec. 2024), arXiv:2412.17744 pages. arXiv:2412.17744 [cs.SE] doi:10.48550/arXiv...

    doi:10.48550/arxiv.2412.17744 2024

  48. [49]

    Yanlin Wang, Ziyao Zhang, Chong Wang, Xinyi Xu, Mingwei Liu, Yong Wang, Jiachi Chen, and Zibin Zheng

  49. [50]

    Chong Wang, Jian Liu, Yangyang Luo, Yiling Wang, Yufei Zhang, Lei Li, Yingfei Zeng, and Shengyu Xu. 2024. Teaching Code LLMs to Use Autocompletion Tools in Repository-level Code Generation.arXiv preprint arXiv:2401.06391(2024)

    Pith/arXiv arXiv 2024

  50. [51]

    Di Wu, Wasi Uddin Ahmad Liu, Dejiao Xu, and Xiaofei Chen. 2024. RepoFormer: Selective Retrieval for Repository-level Code Completion.arXiv preprint arXiv:2403.10059(2024)

    Pith/arXiv arXiv 2024

  51. [52]

    Danning Xie, Mingwei Zheng, Xuwei Liu, Jiannan Wang, Chengpeng Wang, Lin Tan, and Xiangyu Zhang. 2025. CORE: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks.arXiv preprint arXiv:2507.05269 (2025)

    arXiv 2025

  52. [53]

    Ruiyang Xu, Jialun Cao, Yaojie Lu, Ming Wen, Hongyu Lin, Xianpei Han, Ben He, Shing-Chi Cheung, and Le Sun. 2025. CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL 2025). https://aclanthology.org/2025.acl- long.1158.pdf

    2025

  53. [54]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report.arXiv preprint arXiv:2505.09388(2025)

    Pith/arXiv arXiv 2025

  54. [55]

    Anjiang Wei, Jiannan Cao, Ran Li, Hongyu Chen, Yuhui Zhang, Ziheng Wang, Yuan Liu, Thiago SFX Teixeira, Diyi Yang, Ke Wang, et al. 2025. EquiBench: Benchmarking Large Language Models’ Understanding of Program Semantics via Equivalence Checking.arXiv preprint arXiv:2502.12466(2025)

    arXiv 2025

  55. [56]

    Daoguang Zan, Bei Chen, Zeqi Lin, Bei Guan, Yongji Wang, and Jian-Guang Lou. 2022. When language model meets private library.arXiv preprint arXiv:2210.17236(2022)

    Pith/arXiv arXiv 2022

  56. [57]

    Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Wang Yongji, and Jian-Guang Lou. 2023. Large Language Models Meet NL2Code: A Survey. (July 2023), 7443–7464. doi:10.18653/v1/2023.acl-long.411

    doi:10.18653/v1/2023.acl-long.411 2023

  57. [58]

    Chunyan Zhang, Zhao Li, Yanlin Wang, Ge Peng, and Yu Zhang. 2023. Syntax-aware Retrieval Augmented Code Generation.arXiv preprint arXiv:2309.17057(2023). https://arxiv.org/abs/2309.17057

    Pith/arXiv arXiv 2023

  58. [59]

    Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen

  59. [60]

    Chi Yu, Sheng Huang, Qihao Wang, Zhuo Shi, and Zhi Jin. 2022. BashExplainer: Retrieval-Augmented Bash Code Comment Generation based on Fine-tuned CodeBERT. In2022 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 82–93. doi:10.1109/ICSME55016.2022.00014

    arXiv 2022

  60. [61]

    Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, and Xudong Liu. 2020. Retrieval-based neural source code summarization. InProceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 1385–1397

    2020

  61. [62]

    Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. 2024. CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges.arXiv preprint arXiv:2401.07339(2024)

    Pith/arXiv arXiv 2024

  62. [63]

    Ziyin Zhang, Zhaokun Jiang, Lizhen Xu, Hongkun Hao, and Rui Wang. 2024. Multiple-choice questions are efficient and robust llm evaluators.arXiv preprint arXiv:2405.11966(2024)

    Pith/arXiv arXiv 2024

  63. [64]

    Dewu Zheng, Yanlin Wang, Ensheng Shi, Xilin Liu, Yuchi Ma, Hongyu Zhang, and Zibin Zheng. 2025. Top General Performance = Top Domain Performance? DomainCodeBench: A Multi-domain Code Generation Benchmark. (2025). arXiv:2412.18573 [cs.SE] https://arxiv.org/abs/2412.18573

    Pith/arXiv arXiv 2025

  64. [65]

    Shuyan Zhou, Uri Alon, Frank F Xu, Zhiruo Wang, Zhengbao Jiang, and Graham Neubig. 2022. Docprompting: Generating code by retrieving the docs.arXiv preprint arXiv: 2207.05987(2022). Received 2026-02-23; accepted 2026-03-24 Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE124. Publication date: July 2026

    Pith/arXiv arXiv 2022

  65. [66]

    Jing Zhang, Lianghong Guo, Yanlin Wang, Mingwei Liu, Jiachi Chen, Yuchi Ma, Ensheng Shi, Terry Yue Zhuo, Hongyu Zhang, and Zibin Zheng. 2025. SimpleDevQA: Benchmarking Large Language Models on Development Knowledge QA. (2025). arXiv:2512.08867 [cs.SE] https://arxiv.org/abs/2512.08867

    arXiv 2025

  66. [2018]

    Retrieval-based neural code generation.arXiv preprint arXiv:1808.10025(2018). Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE124. Publication date: July 2026. FSE124:22 Yanlin Wang, Suiquan Wang, Yanli Wang, Bowen Zhang, Daya Guo, Jiachi Chen, and Zibin Zheng

    Pith/arXiv arXiv 2018

  67. [2023]

    InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.)

    RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 2471–2484. doi:10.18653/v1/2023.emnlp-main.151

    doi:10.18653/v1/2023.emnlp-main.151 2023

  68. [2024]

    Repoqa: Evaluating long context code understanding.arXiv preprint arXiv:2406.06025(2024)

    Pith/arXiv arXiv 2024

  69. [2025]

    CoreQA: uncovering potentials of language models in code repository question answering.arXiv preprint arXiv:2501.03447(2025)

    Pith/arXiv arXiv 2025

  70. [2026]

    RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories. (2026). arXiv:2601.22706 [cs.CR] https://arxiv.org/abs/2601.22706

    arXiv 2026