Pith. sign in

REVIEW 4 cited by

DevEval: Evaluating Code Generation in Practical Software Projects

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.06401 v4 pith:DRT7NPV7 submitted 2024-01-12 cs.SE cs.AIcs.CL

Jia Li , Ge Li , Yunfei Zhao , Yongmin Li , Zhi Jin , Hao Zhu , Huanyu Liu , Kaibo Liu
show 9 more authors
Lecheng Wang Zheng Fang Lanshen Wang Jiazheng Ding Xuanming Zhang Yihong Dong Yuqi Zhu Bin Gu Mengfei Yang
This is my paper · ORCID
classification cs.SEcs.AIcs.CL
keywords practicalprojectsdevevalcodegenerationllmsbenchmarkscontexts
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

How to evaluate Large Language Models (LLMs) in code generation is an open question. Many benchmarks have been proposed but are inconsistent with practical software projects, e.g., unreal program distributions, insufficient dependencies, and small-scale project contexts. Thus, the capabilities of LLMs in practical projects are still unclear. In this paper, we propose a new benchmark named DevEval, aligned with Developers' experiences in practical projects. DevEval is collected through a rigorous pipeline, containing 2,690 samples from 119 practical projects and covering 10 domains. Compared to previous benchmarks, DevEval aligns to practical projects in multiple dimensions, e.g., real program distributions, sufficient dependencies, and enough-scale project contexts. We assess five popular LLMs on DevEval (e.g., gpt-4, gpt-3.5-turbo, CodeLLaMa, and StarCoder) and reveal their actual abilities in code generation. For instance, the highest Pass@1 of gpt-3.5-turbo only is 42 in our experiments. We also discuss the challenges and future directions of code generation in practical projects. We open-source DevEval and hope it can facilitate the development of code generation in practical projects.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Enhancing Project-Specific Code Completion by Inferring Internal API Information

    cs.SE 2025-07 conditional novelty 6.0 of 10

    A retrieval-augmented code completion method that infers project-internal APIs from a rough draft and a static knowledge base, beating existing repo-level baselines.

  2. Unseen Horizons: Unveiling the Real Capability of LLM Code Generation Beyond the Familiar

    cs.SE 2024-12 conditional novelty 6.0 of 10

    A code-obfuscation benchmark shows LLM code generation pass rates fall sharply when descriptions, code, and dependencies are rewritten to remove training-data familiarity.

  3. SVA-ICL: Improving LLM-based Software Vulnerability Assessment via In-Context Learning and Information Fusion

    cs.SE 2025-05 conditional novelty 5.0 of 10

    An in-context learning approach that retrieves similar vulnerability examples by fusing code and description similarities improves LLM-based severity assessment over prior baselines.

  4. BinMetric: A Comprehensive Binary Analysis Benchmark for Large Language Models

    cs.SE 2025-05 conditional novelty 5.0 of 10

    BinMetric is a new 1,000-question, six-task benchmark that measures LLM ability across the binary analysis lifecycle, with an empirical study of 12 models showing strong semantic understanding but weak low-level lifti...

Pith tools