Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

MRG-Bench: Evaluating and Exploring the Requirements of Context for Repository-Level Code Generation

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MRG-Bench, a multi-language benchmark with runnable tests, finds that code-generation models fail mainly because they cannot understand what the user wants built, not because they lack implementation skill.

desk verdict MRG-Bench is a genuinely useful multi-language executable benchmark, but the headline 'What vs How' failure diagnosis rests on unvalidated LLM self-annotation and should be treated as preliminary. read the letter →

arxiv 2508.02998 v1 pith:4DIB7OBD submitted 2025-08-05 cs.SE

Haiyang Li This is my paper
classification cs.SE
keywords repository-levelcodegenerationLLMevaluationbenchmarkmulti-languagerunnabletestcasesretrieval-augmentedfailurecauseanalysiswhatvshowannotationPythonJavaGo
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces MRG-Bench, a benchmark of 383 function-level code-generation tasks drawn from real Python, Java, and Go repositories, each paired with runnable project-level tests. Across six general models, three reasoning models, long-context setups, and four retrieval-augmented baselines, the best approach still passes under 40% of tasks. The paper's central claim is that the dominant failure mode is not generating correct code but first understanding what functionality the user's request describes in the repository context: more than 68% of failed generations were labeled as missing "what-to-do" information. The authors also find that the context that helps differs sharply by language, with Python behaving differently from Java and Go. If right, MRG-Bench shifts attention from implementation tricks to requirement comprehension and language-specific context design.

What carries the argument

The central object is the MRG-Bench dataset itself, built with per-language function-call-graph analyzers that map test functions to target functions, alongside the What/How failure taxonomy: a two-way split of missing information into what functionality to implement versus how to implement it, annotated through a prompt that asks model judges to classify wrong generations. These two carry the argument: the dataset makes real-world multi-language evaluation with runnable tests possible, and the taxonomy converts observed failures into an explanation of why models fail. A supporting device is the reconstruction-error metric over docstring embeddings, used to argue that MRG-Bench matches real code semantics better than prior repository-level benchmarks.

What would settle it

Have human experts independently classify a random sample of Claude-3.5-Sonnet's failures on MRG-Bench using the same What/How rubric; if the human-assigned 'What' share falls below 50%, the central claim loses its support. A second check: if adding explicit 'What' context such as a README or feature description to the prompt does not raise Pass@1, the dominant-error conclusion predicts the wrong remedy.

Watch

Extended reading notes

Core claim

MRG-Bench establishes that current repository-level code generation is bottlenecked by requirement understanding rather than implementation ability. The paper builds a dataset where each sample is a function with its real annotation, signature, repository file context, called functions, and executable tests, then evaluates open and closed models, long-context models, reasoning models, and RAG pipelines. Even Claude3.5-Sonnet, the strongest general model, reaches 32.5% average Pass@1 with in-file context; reasoning models stay below 40%. To explain failures, the paper labels each incorrect generation as missing either "What" information (what the code should do) or "How" information (how to implement it), using votes from five LLMs. The label distribution shows over 68% of failures are "What" errors, and the effect of repository context on this error type varies across Python, Java, and Go.

Load-bearing premise

The paper's headline conclusion rests on LLM-based annotations: five models vote on whether each failed generation was missing 'What' or 'How' information, only unanimous or 4:1 votes are kept, and no human check on those labels is reported.

Editorial extensions

If this is right

  • Python-only benchmarks overstate model capability; all models here score lower on Java and Go, and the language gap narrows only partially once context is added.
  • RAG methods that retrieve code blocks are not enough: BM25, embedding, and hybrid retrieval all underperform simply placing the target file's content in the prompt.
  • The fastest route to better repository-level generation is improving models' understanding of user intent, so prompts and datasets should supply semantic context such as feature descriptions and usage scenarios alongside code.
  • Context needs are language-specific; a single retrieval strategy will not serve Python, Java, and Go equally, so future systems should tune context per language.
  • A benchmark of 383 real, runnable, multi-language tasks is usable as a shared testbed, and the released call-graph framework lets other researchers build private equivalents for their own codebases.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the What/How split is taken seriously, retrieval should be re-targeted at documents that describe behavior, such as READMEs, docstrings, and issue threads, rather than at code fragments alone; the paper hints at READMEs and feature descriptions but does not test them.
  • The Python anomaly suggests a testable hypothesis the paper does not fully explore: Python function semantics are less tied to repository context, so for Python, pre-generation strategies that infer intent from the target function may outperform context expansion.
  • Human-validated labels would turn the annotation into a scalable quality signal; if a small human-labeled set matches the five-model votes on MRG-Bench, the same voting pipeline could audit failures on other benchmarks.
  • The sub-40% ceiling is a natural stress test for agent-style coding systems: an agent that cannot reliably understand what to build will compound errors at each step, and MRG-Bench offers a generation-oriented setting where such agents can be measured.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces MRG-Bench, a repository-level code generation benchmark with 383 samples from 22 real-world Python, Java, and Go projects, each with a function-level task, developer-written annotations, and runnable test cases. The authors evaluate six LLMs, several reasoning models, long-context models, and RAG-style retrievers, reporting low Pass@1 scores even for the best models. They then propose a What/How failure taxonomy and use five LLMs to annotate failure cases of Claude-3.5-Sonnet, concluding that over 68% of failures stem from missing What information and that different languages require different contextual strategies. The paper also reports data-leakage detection via CDD and argues that MRG-Bench is closer to the real-world semantic distribution than EvoCodeBench and CoderEval-Java.

Significance. If the central diagnosis is supported, MRG-Bench is a useful contribution: it provides a multi-language, executable, repository-level benchmark, releases data and code, and reports leakage detection with a concrete method. The counterintuitive RAG finding and the language-specific context effects are falsifiable and practically relevant. The benchmark's raw Pass@1 numbers are credible because they come from executable tests. However, the paper's main qualitative claim, that the primary failure cause is requirement understanding rather than implementation, rests on unvalidated LLM self-annotation and a non-operational taxonomy. The stress-test concern lands: the benchmark and its accuracy numbers remain valid, but the headline diagnosis may be an artifact of the annotation procedure.

major comments (4)
  1. [§5.1.1, Eq. (1), Figure 3b] The claim that MRG-Bench is significantly closer to the real-world distribution rests on a single reconstruction-error value per dataset with no significance testing or confidence intervals. The metric in Eq. (1) is a mean nearest-neighbor distance and is sensitive to the embedding model, the sample sizes, and the size mismatch between the 383-sample MRG-Bench and the 10,000-sample real set; the t-SNE panels are illustrative only. I ask for bootstrap confidence intervals, a paired or permutation test against the random-sampling baseline, and a statement of how many samples from each language are used in this analysis. Without this, RQ1.1 is not established at the strength claimed.
  2. [§5.3, Figure 4] The headline that over 68% of failures are due to missing What information is produced entirely by five LLMs voting on the failures of a single model (Claude-3.5-Sonnet). No human labels, inter-annotator agreement, prompt-order counterbalancing, or validation on another model's failures are reported; 3:2 disagreements are discarded, and the paper does not state how many cases were 3:2 or what those cases look like. Because the annotation prompt defines What first and in terms of functionality, the voting could reflect a shared prompt-induced label bias rather than the true failure cause. I request human-validated annotations on a stratified sample, agreement statistics such as Cohen's kappa, and annotation of failures from at least one open-source model before generalizing to the majority of methods in the abstract. Without this, the central diagnosis of RQ3 may be an artifact.
  3. [§5.3, Figure 4] The What/How taxonomy is not operational because the two categories are not mutually exclusive. The paper states that a missing callable function belongs to How, but omitting or misusing a callable function also changes what the generated code does; conversely, misunderstanding the requirement also affects implementation. The prompt's examples do not provide a decision procedure for such cases. With a binary forced-choice prompt in which What is described first and tied to functionality, the default label for any behavioral mismatch is likely What. I ask for a mutually exclusive rubric with a decision tree, worked examples, and a label-reliability test on cases where the missing information is known by construction.
  4. [Tables 3-6, §5.2] The experiments report Pass@1 and Pass@3 percentages for 383 samples with no confidence intervals or significance tests. Many differences that motivate takeaways are small, and context-condition differences are not tested; with 124-163 samples per language a single changed sample moves Pass@1 by roughly 0.6-0.8 percentage points. I ask for binomial confidence intervals or bootstrap intervals and, where contexts are compared on the same samples, paired tests. Without these, RQ2's ranking and the claimed significant disparities across languages and contexts are fragile.
minor comments (4)
  1. [References] The reference list is duplicated in full, and the entry for Codex cites a ground-based instrumentation paper rather than the OpenAI Codex model; both should be corrected.
  2. [§5.1.1] The dataset name is written as EvolBench in one place and EvoCodeBench elsewhere; please use a single consistent name throughout.
  3. [Figure 2 caption] The caption contains the typo 'One Samle'; it should read 'One Sample'.
  4. [§3.2 and §6] The text says 1,000 repositories were screened in §3.2 but §6 says 1,400 projects; the numbers should be reconciled, as should the statement that seven languages were initially screened.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the benchmark and its headline numbers are grounded in executable tests and real repositories; the What/How annotation is an unvalidated but non-circular empirical measurement.

full rationale

I examined the paper's claimed derivation chain. The benchmark construction (Sections 2-3) is externally grounded: samples come from real GitHub repositories, and Pass@k is computed by executing generated code against project tests in Docker (Section 3.5). The representativeness argument (RQ1.1) compares embedding reconstruction errors against a separately sampled set of 10,000 real functions; this is an external comparison, not a fitted input. The RQ2 context effects are measured Pass@1 values, not predicted from a fitted model. The central qualitative claim (RQ3, 'over 68% of failures stem from missing What information') rests on LLM-based annotation with no human agreement (Figure 4, Section 5.3); this is a validity threat, and the paper over-generalizes from Claude-3.5-Sonnet to 'the majority of methods.' However, the annotation label is not forced by construction: the prompt asks annotators to choose between two defined categories, and the 68% figure is an empirical vote outcome, not an equation or a fitted parameter renamed as a prediction. There is one self-citation (CoderUJB [28] in Related Work) that is not load-bearing. No circular step satisfies the quoted-reduction standard, so the correct finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The paper's contributions are a dataset, an evaluation, and a labeling framework (What/How), none of which are postulated entities.

assumptions (5)
  • domain assumption Repositories created after January 2023 are sufficiently unfamiliar to modern LLMs to reduce memorization effects.
    Section 3.2: used to mitigate data leakage. If this fails, low model scores could be due to memorization rather than difficulty.
  • domain assumption Functions with developer comments, corresponding tests, and 100% line coverage represent the real-world distribution of repository-level code generation tasks.
    Sections 3.3-3.4: 383 samples from 22 projects after filtering. This selection may systematically exclude complex or poorly documented code, undermining the representativeness claim.
  • domain assumption NV-Embedding-2 embeddings and nearest-neighbor reconstruction error accurately measure semantic similarity of code comments across languages.
    Section 5.1.1: used to show MRG-Bench is closer to real-world data than EvoCodeBench and CoderEval. No validation of the metric or significance tests.
  • ad hoc to paper LLM-based annotation with 5-model voting identifies the true cause of model failures.
    Section 5.3: the What/How taxonomy and the annotation prompt are introduced in this paper. No human-annotated ground truth or inter-annotator agreement is reported.
  • domain assumption The function call graph analyzers correctly map test functions to target functions using Tree-sitter and naming heuristics.
    Section 3.3: if test-to-function linking is inaccurate, test-based evaluation may not measure the intended function's correctness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MRG-Bench: Evaluating and Exploring the Requirements of Context for Repository-Level Code Generation." pith.science (2026). https://pith.science/paper/4DIB7OBD

@misc{pith2026250802998,
  author       = {Pith},
  title        = {Pith review of: MRG-Bench: Evaluating and Exploring the Requirements of Context for Repository-Level Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4DIB7OBD}},
  note         = {Machine review of arXiv:2508.02998}
}
read the original abstract

Large Language Models (LLMs) have demonstrated impressive capabilities in code generation. However, current evaluation datasets suffer from issues such as the lack of runnable test cases, deviation from the distribution of real-world code, and the ability to evaluate only the Python language. These limitations undermine the credibility of the evaluation results. To address these limitations, we introduce \textbf{MRG-Bench} (Multi-language Repository-level Code Generation Benchmark), a novel dataset that provides a more accurate evaluation of LLMs in practical repository-level code generation tasks. MRG-Bench has three main features: (1) practical data sourced from real-world code repositories that align to the practical distribution, (2) multiple programming languages support, including Python, Java, and Go, and (3) project-level runnable test cases to assess the quality of the generated code. Based on MRG-Bench, we conducted extensive experiments including large language models, long-context models, and RAG-related methods. These evaluation results demonstrate that \textbf{current repository-level code generation techniques suffer from significant performance deficiencies}. To further investigate why models fail, we designed novel experiments to annotate the underlying causes of generation errors. The results explicitly show that the majority of methods suffer from "\textbf{difficulty in understanding user requirements}," failing to comprehend their assigned tasks accurately. Moreover, the impact of different repository-level contexts on this issue exhibits significant disparities across different programming languages, suggesting that, in practice, specialized contextual information needs to be designed for different languages.

Figures

Figures reproduced from arXiv: 2508.02998 by the authors.

Figure 2
Figure 2. All Information in One Sample of MRG-Bench [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Prompt for label fail cases to How and What. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Annotation result of fail reason. information require further exploration. Notably, RepoCoder demon￾strates exceptional capability in eliminating language bias, which warrants deeper investigation. 5.3 RQ3:Why do current methods fails on MRG-Bench? From the previous two RQs, we observe several intriguing phe￾nomena: on MRG-Bench, RAG algorithms show no significant ad￾vantages, and providing available functions for c… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Repository-Aware Metamorphic Relation Generation for Augmented Reality Applications using Large Language Models

    cs.SE 2026-07 conditional novelty 6.0 of 10

    A context-aware LLM pipeline generates metamorphic relations for 142 AR repositories; hierarchical context plus agentic deliberation yields 3,760 refined MRs that human raters judge mostly valid and testable, with 5 c...

  2. DocRetriever: A Plug-and-Play Framework for Multimodal Document Retrieval with Comprehensive Benchmark

    cs.CV 2026-05 unverdicted novelty 5.0 of 10

    DocRetriever introduces a framework using layout-aware sparse embeddings for hybrid encoding without OCR and a generalizable reasoning-augmented reranker for few-shot settings, plus the MultiDocR benchmark for evaluation.

Reference graph

Works this paper leans on

31 extracted references · 7 canonical work pages · cited by 2 Pith papers

  1. [1]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models.arXiv preprint arXiv:2108.07732(2021)

    arXiv 2021

  2. [2]

    Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug.arXiv preprint arXiv:2304.05128(2023)

    arXiv 2023

  3. [3]

    DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948 [cs.CL] https://arxiv.org/abs/2501. 12948

    arXiv 2025

  4. [4]

    Yangruibo Ding, Zijian Wang, Wasi Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, et al. 2024. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion.Advances in Neural Information Processing Systems36 (2024)

    work page 2024

  5. [5]

    Yihong Dong, Jiazheng Ding, Xue Jiang, Zhuo Li, Ge Li, and Zhi Jin

    work page

  6. [6]

    Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. 2024. Generalization or Memorization: Data Contamination and Trustwor- thy Evaluation for Large Language Models. InFindings of the Association for Computational Linguistics: ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics...

    doi:10.18653/v1/2024.findings-acl.716 2024

  7. [7]

    Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2023. Classeval: A manually- crafted benchmark for evaluating llms on class-level code generation.arXiv preprint arXiv:2308.01861(2023)

    arXiv 2023

  8. [8]

    Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2023. InCoder: A Generative Model for Code Infilling and Synthesis. arXiv:2204.05999 (April 2023). doi:10.48550/arXiv.2204.05999 arXiv:2204.05999 [cs]

    arXiv doi:10.48550/arxiv.2204.05999 2023

Show all 31 references
  1. [9]

    2024.Copilot

    GitHub. 2024.Copilot. https://github.com/features/copilot

    2024

  2. [10]

    Wenhan Xiong Grattafiori, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Syn- naeve. 2023. Code Llama: Open Foundation Models for Code.arXiv preprint arXiv:2308.12950(2023)

    2023 arXiv

  3. [11]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence.arXiv preprint arXiv:2401.14196(2024)

    2024 arXiv

  4. [12]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. 2021. Mea- suring coding challenge competence with apps.arXiv preprint arXiv:2105.09938 (2021)

    2021 arXiv

  5. [13]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770(2023)

    2023 arXiv

  6. [14]

    Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. 2023. xcodeeval: A large scale multilin- gual multitask benchmark for code understanding, generation, translation and retrieval.arXiv preprint arXiv:2303.03004(2023)

    2023 arXiv

  7. [15]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Mem- ory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating...

    2023

  8. [16]

    Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models.arXiv preprint arXiv:2405.17428 (2024)

    2024 arXiv

  9. [17]

    Jia Li, Ge Li, Xuanming Zhang, Yihong Dong, and Zhi Jin. 2024. EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories. arXiv:2404.00599 (March 2024). http://arxiv.org/abs/2404.00599 arXiv:2404.00599 [cs]

    2024 arXiv

  10. [18]

    Tianyang Liu, Canwen Xu, and Julian McAuley. 2023. Repobench: Benchmarking repository-level code auto-completion systems.arXiv preprint arXiv:2306.03091 (2023)

    2023 arXiv

  11. [19]

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy- Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. 2024. Starcoder 2 and the stack v2: The next generation.arXiv preprint arXiv:2402.19173(2024)

    2024 arXiv

  12. [20]

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambro- sio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al . 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation.arXiv preprint arXiv:2102.04664(2021)

    2021 arXiv

  13. [21]

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis.arXiv preprint arXiv:2203.13474 (2022)

    2022 arXiv

  14. [22]

    2024.ChatGPT

    OpenAI. 2024.ChatGPT. https://chatgpt.com/

    2024

  15. [23]

    Luca Pasquini, Stefano Cristiani, Ramón García López, Martin Haehnelt, Michel Mayor, Jochen Liske, Antonio Manescau, Gerardo Avila, Hans Dekker, Olaf Iwert, et al. 2010. Codex. InGround-based and Airborne Instrumentation for Astronomy III, Vol. 7735. SPIE, 957–968

    2010

  16. [24]

    Sijie Shen, Xiang Zhu, Yihong Dong, Qizhi Guo, Yankun Zhen, and Ge Li. 2022. Incorporating domain knowledge through task augmentation for front-end javascript code generation. InProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Fou...

    2022

  17. [25]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  18. [26]

    Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. 2024. Codereval: A benchmark of prag- matic code generation with generative pre-trained models. InProceedings of the 46th IEEE/ACM International Conference on So...

    2024

  19. [27]

    Daoguang Zan, Bei Chen, Dejian Yang, Zeqi Lin, Minsu Kim, Bei Guan, Yongji Wang, Weizhu Chen, and Jian-Guang Lou. 2022. CERT: Continual Pre-training on Sketches for Library-oriented Code Generation. InProceedings of the Thirty-First International Joint Conference on Artificial...

    2022 doi

  20. [28]

    Zhengran Zeng, Yidong Wang, Rui Xie, Wei Ye, and Shikun Zhang. 2024. CoderUJB: An Executable and Unified Java Benchmark for Practical Programming Scenarios.arXiv preprint arXiv:2403.19287(2024)

    2024 arXiv

  21. [29]

    Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. Repocoder: Repository-level code com- pletion through iterative retrieval and generation.arXiv preprint arXiv:2303.12570 (2023)

    2023 arXiv

  22. [30]

    Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. 2024. CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges. arXiv:2401.07339 (Jan. 2024). http://arxiv.org/abs/2401.07339 arXiv:2401.07339 [cs]. Received 20 Feb...

    2024 arXiv

  23. [2023]

    arXiv:2301.09043 (Jan

    CodeScore: Evaluating Code Generation by Learning Code Execution. arXiv:2301.09043 (Jan. 2023). doi:10.48550/arXiv.2301.09043 arXiv:2301.09043 [cs]. Conference acronym, Date, Venue Haiyang Li, Qing Gao, and Shikun Zhang

    2023 doi arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.