Pith. sign in

REVIEW 28 cited by

ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2308.01861 v2 pith:2UF75TJC submitted 2023-08-03 cs.CL cs.AI

Xueying Du , Mingwei Liu , Kaixin Wang , Hanlin Wang , Junwei Liu , Yixuan Chen , Jiayi Feng , Chaofeng Sha
show 2 more authors
Xin Peng Yiling Lou
This is my paper · ORCID
classification cs.CLcs.AI
keywords generationcodeclass-levelllmsfirstabilityfindbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

In this work, we make the first attempt to evaluate LLMs in a more challenging code generation scenario, i.e. class-level code generation. We first manually construct the first class-level code generation benchmark ClassEval of 100 class-level Python code generation tasks with approximately 500 person-hours. Based on it, we then perform the first study of 11 state-of-the-art LLMs on class-level code generation. Based on our results, we have the following main findings. First, we find that all existing LLMs show much worse performance on class-level code generation compared to on standalone method-level code generation benchmarks like HumanEval; and the method-level coding ability cannot equivalently reflect the class-level coding ability among LLMs. Second, we find that GPT-4 and GPT-3.5 still exhibit dominate superior than other LLMs on class-level code generation, and the second-tier models includes Instruct-Starcoder, Instruct-Codegen, and Wizardcoder with very similar performance. Third, we find that generating the entire class all at once (i.e. holistic generation strategy) is the best generation strategy only for GPT-4 and GPT-3.5, while method-by-method generation (i.e. incremental and compositional) is better strategies for the other models with limited ability of understanding long instructions and utilizing the middle information. Lastly, we find the limited model ability of generating method-dependent code and discuss the frequent error types in generated classes. Our benchmark is available at https://github.com/FudanSELab/ClassEval.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 28 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Oilbird: Training-Free Speculative Decoding with Keys the Verifier Already Computes

    cs.AI 2026-08 conditional novelty 8.0 of 10

    A training-free speculative decoder using the verifier's hidden states as a semantic key retrieves correct continuations unreachable by exact suffix matching, lifting accepted length 24-29%.

  2. DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale

    cs.CL 2025-01 conditional novelty 7.0 of 10

    A new large-scale benchmark shows that current large language models pass under half of repository execution tests when asked to infer dependencies.

  3. WebDesignIter: Co-Evolving Design Knowledge for Repository-Level Front-End Code Generation

    cs.SE 2026-07 conditional novelty 6.0 of 10

    A co-evolving design-and-code knowledge graph improves multi-iteration front-end repository generation by ~9.5 Pass@2 points and cuts input tokens 25–30× versus agent baselines.

  4. Rethinking Code Performance Benchmarks for LLMs

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Re-evaluating four LLM code-efficiency benchmarks with 30-run statistical testing shows 93.89% of 'performant' implementations are indistinguishable from baselines; a multi-agent test-generation framework reveals hidd...

  5. SWE-IF: Aligning Code Evaluation with Human Preference

    cs.CL 2025-10 conditional novelty 6.0 of 10

    A blend of functional correctness and instruction-following predicts human preference for code LLMs better than pass@k alone.

  6. ImportSnare: Directed "Code Manual" Hijacking in Retrieval-Augmented Code Generation

    cs.CR 2025-09 conditional novelty 6.0 of 10

    Documentation poisoning with hidden ranking and suggestion sequences can make RAG-based code generators confidently recommend malicious dependencies, even at 0.01% poisoning ratios.

  7. MRG-Bench: Evaluating and Exploring the Requirements of Context for Repository-Level Code Generation

    cs.SE 2025-08 conditional novelty 6.0 of 10

    MRG-Bench is a multi-language, executable repository-level code generation benchmark on which current LLMs and RAG methods perform poorly, with the dominant failure mode being poor understanding of user requirements.

  8. CoCoTen: Detecting Adversarial Inputs to Large Language Models through Latent Space Features of Contextual Co-occurrence Tensors

    cs.CL 2025-08 conditional novelty 6.0 of 10

    MRG-Bench is a multi-language, runnable-test benchmark showing that LLMs' main failure in repository-level code generation is understanding user requirements.

  9. A Hierarchical and Evolvable Benchmark for Fine-Grained Code Instruction Following with Multi-Turn Feedback

    cs.SE 2025-07 conditional novelty 6.0 of 10

    MultiCodeIF introduces a 2,021-task, 14-language benchmark with 27 constraint types to evaluate code instruction following, finding that multi-level constraints sharply reduce model success and iterative feedback subs...

  10. SwiftEval: Developing a Language-Specific Benchmark for LLM-generated Code Evaluation

    cs.LG 2025-05 conditional novelty 6.0 of 10

    SwiftEval, a 28-problem hand-crafted Swift benchmark, evaluates 44 code LLMs and shows large performance drops on Swift tasks, especially for smaller models.

  11. Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generation

    cs.SE 2025-05 conditional novelty 6.0 of 10

    The paper builds a 1000-task test-driven development benchmark for LLMs and concludes that instruction following and in-context learning matter more than general coding proficiency.

  12. LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks

    cs.SE 2025-02 conditional novelty 6.0 of 10

    Across 83 SE benchmarks, average leakage into StarCoder's pretraining data is 4.8% (Python), 2.8% (Java), and 0.7% (C/C++), but QuixBugs and BigCloneBench are 100% and 55.7% leaked.

  13. Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering

    cs.SE 2025-02 conditional novelty 6.0 of 10

    Output-based LLM-as-a-judge methods using large LLMs achieve near-human correlation with human scores for code translation and generation, but not for code summarization or pairwise comparisons.

  14. Large Language Model Critics for Execution-Free Evaluation of Code Changes

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Reference-aware, test-centric LLM critics predict per-test outcomes of code patches and aggregate them into execution-free build status predictions on SWE-bench-Lite.

  15. CodeV: Issue Resolving with Visual Data

    cs.SE 2024-12 conditional novelty 6.0 of 10

    CodeV converts images and videos in GitHub issues into detailed text and structured summaries, then feeds them to a text-only LLM to generate patches, improving resolution rates on the new Visual SWE-bench benchmark.

  16. Unseen Horizons: Unveiling the Real Capability of LLM Code Generation Beyond the Familiar

    cs.SE 2024-12 conditional novelty 6.0 of 10

    A code-obfuscation benchmark shows LLM code generation pass rates fall sharply when descriptions, code, and dependencies are rewritten to remove training-data familiarity.

  17. PBE Meets LLM: When Few Examples Aren't Few-Shot Enough

    cs.DB 2025-07 conditional novelty 5.0 of 10

    GPT-4o with multi-turn verification plus a Foofah fallback reaches 86.3% weighted accuracy on tabular PBE benchmarks, beating Foofah (57.1%) and Prose (47.3%).

  18. Do Generative AI Tools Ensure Green Code? An Investigative Study

    cs.SE 2025-06 conditional novelty 5.0 of 10

    An early study of three AI coding tools found that generated code frequently fails to follow sustainable coding practices, especially for GitHub Copilot.

  19. SIMCOPILOT: Evaluating Large Language Models for Copilot-Style Code Generation

    cs.LG 2025-05 conditional novelty 5.0 of 10

    A new benchmark for copilot-style code completion and infilling shows that top models differ by up to 50 points on realistic tasks despite near-identical HumanEval scores.

  20. BinMetric: A Comprehensive Binary Analysis Benchmark for Large Language Models

    cs.SE 2025-05 conditional novelty 5.0 of 10

    BinMetric is a new 1,000-question, six-task benchmark that measures LLM ability across the binary analysis lifecycle, with an empirical study of 12 models showing strong semantic understanding but weak low-level lifti...

  21. Large Language Models for Code Generation: The Practitioners Perspective

    cs.SE 2025-01 reject novelty 5.0 of 10

    In a practitioner survey with a reported 60 respondents, GPT-4o ranked as the best model for code generation and GPT-3.5 Turbo as the worst.

  22. ContextModule: Improving Code Completion via Repository-level Contextual Information

    cs.SE 2024-12 conditional novelty 5.0 of 10

    Feeding code-completion models a developer's recent file history, similar code, and key symbol definitions raised completion acceptance rates in internal ByteDance deployments by an estimated 15 percent.

  23. Software Engineering for Large Language Models: Research Status, Challenges and the Road Ahead

    cs.SE 2025-06 conditional novelty 4.0 of 10

    A literature review organizes LLM development into a six-phase software engineering lifecycle and identifies challenges and research directions for each phase.

  24. Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A survey that classifies AI agent evaluation benchmarks along environment and capability axes, and proposes five traits that distinguish agents from chatbots.

  25. CoCo-Bench: A Comprehensive Code Benchmark For Multi-task Large Language Model Evaluation

    cs.SE 2025-04 reject novelty 4.0 of 10

    CoCo-Bench is a new multi-language benchmark of 705 manually reviewed samples that evaluates code LLMs across understanding, generation, modification, and review tasks, with difficulty-weighted scoring.

  26. The importance of visual modelling languages in generative software engineering

    cs.SE 2024-11 conditional novelty 4.0 of 10

    Multimodal GPTs can turn UML class, sequence, and hand-drawn activity diagrams into working Python code, and can reverse engineer code back into diagrams, demonstrated across several software engineering tasks.

  27. A Preliminary Study of Multilingual Code Language Models for Code Generation Task Using Translated Benchmarks

    cs.SE 2024-11 reject novelty 4.0 of 10

    PolyCoder's pass@1 performance on translated HumanEval benchmarks is inconsistent across MultiPL-E and HumanEvalSynthesize, reversing the relative ranking of Java and Rust.

  28. Hallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challenges

    cs.SE 2025-04 conditional novelty 3.0 of 10

    This survey organizes code-LLM hallucinations into four types, reviews 52 related papers, and identifies open challenges in detection and mitigation.

Pith tools