The reviewed record of science sign in
Pith

arxiv: 2405.19856 · v1 · pith:YUR3O2VF · submitted 2024-05-30 · cs.CL · cs.SE

DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories

Jia Li , Ge Li , Yunfei Zhao , Yongmin Li , Huanyu Liu , Hao Zhu , Lecheng Wang , Kaibo Liu
show 10 more authors
Zheng Fang Lanshen Wang Jiazheng Ding Xuanming Zhang Yuqi Zhu Yihong Dong Zhi Jin Binhua Li Fei Huang Yongbin Li
This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:YUR3O2VFrecord.jsonopen to challenge →

classification cs.CL cs.SE
keywords devevalcodellmsrepositoriesreal-worldabilitiescodingevaluate
0
0 comments X
read the original abstract

How to evaluate the coding abilities of Large Language Models (LLMs) remains an open question. We find that existing benchmarks are poorly aligned with real-world code repositories and are insufficient to evaluate the coding abilities of LLMs. To address the knowledge gap, we propose a new benchmark named DevEval, which has three advances. (1) DevEval aligns with real-world repositories in multiple dimensions, e.g., code distributions and dependency distributions. (2) DevEval is annotated by 13 developers and contains comprehensive annotations (e.g., requirements, original repositories, reference code, and reference dependencies). (3) DevEval comprises 1,874 testing samples from 117 repositories, covering 10 popular domains (e.g., Internet, Database). Based on DevEval, we propose repository-level code generation and evaluate 8 popular LLMs on DevEval (e.g., gpt-4, gpt-3.5, StarCoder 2, DeepSeek Coder, CodeLLaMa). Our experiments reveal these LLMs' coding abilities in real-world code repositories. For example, in our experiments, the highest Pass@1 of gpt-4-turbo is only 53.04%. We also analyze LLMs' failed cases and summarize their shortcomings. We hope DevEval can facilitate the development of LLMs in real code repositories. DevEval, prompts, and LLMs' predictions have been released.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ClassEval-Pro: A Cross-Domain Benchmark for Class-Level Code Generation

    cs.SE 2026-04 unverdicted novelty 7.0

    ClassEval-Pro benchmark shows frontier LLMs achieve at most 45.6% Pass@1 on class-level code tasks, with logic errors (56%) and dependency errors (38%) as dominant failure modes.

  2. Evaluating the Formal Reasoning Capabilities of Large Language Models through Chomsky Hierarchy

    cs.CL 2026-04 unverdicted novelty 7.0

    LLMs display clear performance stratification on formal language tasks aligned with Chomsky hierarchy complexity levels, limited by severe efficiency barriers rather than absolute capability.

  3. Can Large Language Models Generate Observability-Aware Code?

    cs.SE 2026-07 conditional novelty 6.0

    AI coding agents produce microservice systems where only 4.95–13.99% of injected faults generate explicit fault signals in logs, revealing a semantic observability gap that lightweight guidance only partially addresses.

  4. SetupX: Can LLM Agents Learn from Past Failures in Functionality-Correct Code Repository Setup?

    cs.SE 2026-05 unverdicted novelty 6.0

    SetupX presents an experiential learning framework for LLM agents that reaches 92% pass rate on functionality-correct repository setup by transferring verified fixes across repositories via XPU representations, LIFO D...

  5. AdaDec: A Uncertainty-Guided Lookahead Decoding Framework for LLM-Based Code Generation

    cs.SE 2025-06 unverdicted novelty 5.0

    AdaDec improves Pass@1 accuracy of LLM code generation by up to 20.9% over greedy decoding by triggering lookahead reranking only at high-uncertainty steps on HumanEval+, MBPP+, and DevEval.