Pith. sign in

REVIEW 16 cited by

EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.00599 v1 pith:RIALZA7B submitted 2024-03-31 cs.CL cs.AIcs.SE

Jia Li , Ge Li , Xuanming Zhang , Yihong Dong , Zhi Jin This is my paper · ORCID
classification cs.CLcs.AIcs.SE
keywords evocodebenchcodellmsrepositoriesreal-worldbenchmarkevaluategeneration
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

How to evaluate Large Language Models (LLMs) in code generation is an open question. Existing benchmarks demonstrate poor alignment with real-world code repositories and are insufficient to evaluate the coding abilities of LLMs. This paper proposes a new benchmark - EvoCodeBench to address the preceding problems, which has three primary advances. (1) EvoCodeBench aligns with real-world repositories in multiple dimensions, e.g., code distributions and dependency distributions. (2) EvoCodeBench offers comprehensive annotations (e.g., requirements, reference code, and reference dependencies), and robust evaluation metrics (e.g., Pass@k and Recall@k). (3) EvoCodeBench is an evolving benchmark to avoid data leakage. We build an automatic pipeline to update EvoCodeBench from the latest repositories. We release the first version - EvoCodeBench-2403, containing 275 samples from 25 real-world repositories. Based on EvoCodeBench, we propose repository-level code generation and evaluate 10 popular LLMs (e.g., gpt-4, gpt-3.5, DeepSeek Coder, StarCoder 2, CodeLLaMa, Gemma, and Qwen 1.5). Our experiments reveal the coding abilities of these LLMs in real-world repositories. For example, the highest Pass@1 of gpt-4 only is 20.73% in our experiments. We also analyze failed cases and summarize the shortcomings of existing LLMs in EvoCodeBench. We release EvoCodeBench, all prompts, and LLMs' completions for further community analysis.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale

    cs.CL 2025-01 conditional novelty 7.0 of 10

    A new large-scale benchmark shows that current large language models pass under half of repository execution tests when asked to infer dependencies.

  2. From Execution to Education: A Bloom-Aligned Framework for Measuring Educational Control in LLMs

    cs.CL 2026-07 conditional novelty 6.5 of 10

    On 2,520 programming tasks, matched Qwen general and coder models reliably raise Bloom cognitive demand but fail to lower it, so execution skill does not imply educational control.

  3. MRG-Bench: Evaluating and Exploring the Requirements of Context for Repository-Level Code Generation

    cs.SE 2025-08 conditional novelty 6.0 of 10

    MRG-Bench is a multi-language, executable repository-level code generation benchmark on which current LLMs and RAG methods perform poorly, with the dominant failure mode being poor understanding of user requirements.

  4. CoCoTen: Detecting Adversarial Inputs to Large Language Models through Latent Space Features of Contextual Co-occurrence Tensors

    cs.CL 2025-08 conditional novelty 6.0 of 10

    MRG-Bench is a multi-language, runnable-test benchmark showing that LLMs' main failure in repository-level code generation is understanding user requirements.

  5. FrontendBench: A Benchmark for Evaluating LLMs on Front-End Development via Automatic Evaluation

    cs.SE 2025-06 conditional novelty 6.0 of 10

    A new benchmark adds 148 interactive front-end development tasks with automated sandbox tests, reporting a 90.54% agreement rate with human evaluation across four LLMs.

  6. PersonaFeedback: A Large-scale Human-annotated Benchmark For Personalization

    cs.CL 2025-06 conditional novelty 6.0 of 10

    PersonaFeedback provides a human-labeled benchmark showing current LLMs, including strong reasoners, score only about 65-70 percent on hard personalization choices, and explicit persona information helps more than retrieval.

  7. Safe: Enhancing Mathematical Reasoning in Large Language Models via Retrospective Step-aware Formal Verification

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Safe uses step-level formal verification in Lean 4, aggregated by a small LSTM and combined with process reward scores, to improve best-of-n accuracy for LLM mathematical reasoning.

  8. Code Copycat Conundrum: Demystifying Repetition in LLM-based Code Generation

    cs.SE 2025-04 conditional novelty 6.0 of 10

    LLM-generated code frequently contains repetitive patterns at character, statement, and block levels, and a rule-based pruner named DeRep reduces this repetition and improves functional correctness on HumanEval and MBPP.

  9. LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks

    cs.SE 2025-02 conditional novelty 6.0 of 10

    Across 83 SE benchmarks, average leakage into StarCoder's pretraining data is 4.8% (Python), 2.8% (Java), and 0.7% (C/C++), but QuixBugs and BigCloneBench are 100% and 55.7% leaked.

  10. Unseen Horizons: Unveiling the Real Capability of LLM Code Generation Beyond the Familiar

    cs.SE 2024-12 conditional novelty 6.0 of 10

    A code-obfuscation benchmark shows LLM code generation pass rates fall sharply when descriptions, code, and dependencies are rewritten to remove training-data familiarity.

  11. SIMCOPILOT: Evaluating Large Language Models for Copilot-Style Code Generation

    cs.LG 2025-05 conditional novelty 5.0 of 10

    A new benchmark for copilot-style code completion and infilling shows that top models differ by up to 50 points on realistic tasks despite near-identical HumanEval scores.

  12. Software Engineering for Large Language Models: Research Status, Challenges and the Road Ahead

    cs.SE 2025-06 conditional novelty 4.0 of 10

    A literature review organizes LLM development into a six-phase software engineering lifecycle and identifies challenges and research directions for each phase.

  13. GeoPandas-AI: A Smart Class Bringing LLM as Stateful AI Code Assistant

    cs.HC 2025-06 conditional novelty 4.0 of 10

    A new open-source Python class, GeoDataFrameAI, adds a stateful LLM chat interface directly to GeoPandas data frames for geospatial code generation and analysis.

  14. Knowledge Graph Based Repository-Level Code Generation

    cs.AI 2025-05 reject novelty 4.0 of 10

    A knowledge graph code retrieval pipeline is described, but its headline results come from an evaluation that skips the retrieval step and anchors context on the known target function.

  15. ACECode: A Reinforcement Learning Framework for Aligning Code Efficiency and Correctness in Code Language Models

    cs.SE 2024-12 conditional novelty 4.0 of 10

    Fine-tuning code LLMs with PPO and an execution-based reward that combines test results with runtime relative to a reference solution improves both correctness and efficiency on EffiBench+.

  16. Hallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challenges

    cs.SE 2025-04 conditional novelty 3.0 of 10

    This survey organizes code-LLM hallucinations into four types, reviews 52 related papers, and identifies open challenges in detection and mitigation.

Pith tools