REVIEW 21 cited by
A Survey on Evaluating Large Language Models in Code Generation Tasks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
This paper provides a comprehensive review of the current methods and metrics used to evaluate the performance of Large Language Models (LLMs) in code generation tasks. With the rapid growth in demand for automated software development, LLMs have demonstrated significant potential in the field of code generation. The paper begins by reviewing the historical development of LLMs and their applications in code generation. Next, it details various methods and metrics for assessing the code generation capabilities of LLMs, including code correctness, efficiency, readability, and evaluation methods based on expert review and user experience. The paper also evaluates the widely used benchmark datasets, identifying their limitations and proposing directions for future improvements. Specifically, the paper analyzes the performance of code generation models across different tasks by combining multiple evaluation metrics, such as code compilation/interpretation success rates, unit test pass rates, and performance and efficiency metrics, to comprehensively assess the practical application of LLMs in code generation. Finally, the paper discusses the challenges faced in evaluating LLMs in code generation, particularly how to ensure the comprehensiveness and accuracy of evaluation methods and how to adapt to the evolving practices of software development. These analyses and discussions provide valuable insights for further optimizing and improving the application of LLMs in code generation tasks.
Forward citations
Cited by 21 Pith papers
-
Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios
A new benchmark for 0-to-1 CLI tool generation shows state-of-the-art LLMs achieve under 43% success rate with black-box equivalence testing against real oracles.
-
From Execution to Education: A Bloom-Aligned Framework for Measuring Educational Control in LLMs
On 2,520 programming tasks, matched Qwen general and coder models reliably raise Bloom cognitive demand but fail to lower it, so execution skill does not imply educational control.
-
Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality
Prompt language affects LLM code generation, but English is not consistently best: Chinese prompts improve Python correctness on CoderEval, while quality and lexicon effects vary by model and programming language.
-
Effectiveness of LLM-based Software Diversity for Reliability Improvement -- an Empirical Study
LLM-generated programs, especially when paired across languages or with human code, produce measurable 1-out-of-2 reliability gains under classical software-diversity analysis on three programming-contest specifications.
-
Rethinking Technology Stack Selection with AI Coding Proficiency
AI coding proficiency, a new measure of how effectively LLMs use a given library, reveals large quality gaps between competing Python libraries, with winners varying by model.
-
A Hierarchical and Evolvable Benchmark for Fine-Grained Code Instruction Following with Multi-Turn Feedback
MultiCodeIF introduces a 2,021-task, 14-language benchmark with 27 constraint types to evaluate code instruction following, finding that multi-level constraints sharply reduce model success and iterative feedback subs...
-
LLM-based Satisfiability Checking of String Requirements by Consistent Data and Checker Generation
An LLM-based system that generates SMT and Python checkers from natural-language string requirements and uses them with feedback to verify satisfiability improves accuracy over direct LLM generation.
-
WebUIBench: A Comprehensive Benchmark for Evaluating Multimodal Large Language Models in WebUI-to-Code
WebUIBench is a 21,793-question benchmark that splits WebUI-to-Code into perception, HTML programming, and cross-modal understanding, and it ranks 29 multimodal LLMs on each sub-skill.
-
MARCO: Meta-Reflection with Cross-Referencing for Code Reasoning
MARCO combines cross-problem knowledge accumulation with cross-agent lesson sharing to improve LLM code reasoning at inference time.
-
AutoGEEval: A Multimodal and Automated Framework for Geospatial Code Generation on GEE with Large Language Models
An automated, execution-based benchmark of 1,325 Google Earth Engine unit tests shows 18 LLMs scoring between 31.40% and 71.55% pass@1, with parameter-knowledge errors the dominant failure mode.
-
LLM-Supported Natural Language to Bash Translation
New 600-pair verified NL2SH test set and an execution-plus-LLM equivalence checker improve Bash-command translation evaluation; parsing and in-context learning give the largest gains for small models.
-
Studying How Configurations Impact Code Generation in LLMs: the Case of ChatGPT
A large empirical study of GPT-4o shows that top-p affects Java method generation more than temperature, and that low temperature reduces the number of methods for which a correct implementation is found across repeat...
-
CIgrate: Automating CI Service Migration with Large Language Models
The abstract reports that a fine-tuned Gemma 3 12B outperforms the rule-based CIMig tool on Travis-to-GitHub Actions CI migration, but the manuscript body contains only a planned study with no experimental results.
-
Improving Existing Optimization Algorithms with LLMs
A GPT-4o-proposed age-aware heuristic for the CMSA metaheuristic outperforms the expert degree-based heuristic on Maximum Independent Set instances across three graph families.
-
Large Language Models for Code Generation: The Practitioners Perspective
In a practitioner survey with a reported 60 respondents, GPT-4o ranked as the best model for code generation and GPT-3.5 Turbo as the worst.
-
Automated Code Review Using Large Language Models at Ericsson: An Experience Report
A lightweight LLM-based code review tool using enclosing-method context from static analysis received encouraging but mixed feedback from Ericsson developers.
-
Evaluation of Large Language Model-Driven AutoML in Data and Model Management from Human-Centered Perspective
A 15-participant within-subjects study claims LLM-driven AutoML outperforms traditional AutoML on speed, accuracy, and usability, though supporting data are not provided and several numbers conflict.
-
Assessing and Enhancing the Robustness of LLM-based Multi-Agent Systems Through Chaos Engineering
The paper proposes, but does not implement or validate, a chaos engineering framework for improving the robustness of LLM-based multi-agent systems.
-
The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs
A structured survey of LLM safety evaluation that proposes a why/what/where/how taxonomy and catalogs metrics, datasets, benchmarks, evaluators, and frameworks.
-
Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI
A qualitative taxonomy positions vibe coding and agentic coding as complementary paradigms rather than rivals in AI-assisted software development.
-
Engineering Trustworthy Software: A Mission for LLMs
A position paper that calls for integrating LLMs across the software lifecycle to build trustworthy software and outlines the challenges that must be solved first.
Discussion (0). Continue with ORCID to comment.