REVIEW 7 cited by
NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Prompts
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large language models (LLMs) have manifested strong ability to generate codes for productive activities. However, current benchmarks for code synthesis, such as HumanEval, MBPP, and DS-1000, are predominantly oriented towards introductory tasks on algorithm and data science, insufficiently satisfying challenging requirements prevalent in real-world coding. To fill this gap, we propose NaturalCodeBench (NCB), a challenging code benchmark designed to mirror the complexity and variety of scenarios in real coding tasks. NCB comprises 402 high-quality problems in Python and Java, meticulously selected from natural user queries from online coding services, covering 6 different domains. Noting the extraordinary difficulty in creating testing cases for real-world queries, we also introduce a semi-automated pipeline to enhance the efficiency of test case construction. Comparing with manual solutions, it achieves an efficiency increase of more than 4 times. Our systematic experiments on 39 LLMs find that performance gaps on NCB between models with close HumanEval scores could still be significant, indicating a lack of focus on practical code synthesis scenarios or over-specified optimization on HumanEval. On the other hand, even the best-performing GPT-4 is still far from satisfying on NCB. The evaluation toolkit and development set are available at https://github.com/THUDM/NaturalCodeBench.
Forward citations
Cited by 7 Pith papers
-
WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch
WebGen-Bench provides 101 website-building tasks with 647 atomic test cases, and a trained Qwen2.5-Coder-32B model reaches 38.2% accuracy, outperforming DeepSeek-R1's 27.8%.
-
AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators
AutoCodeBench is an LLM-generated, sandbox-verified code benchmark with 3,920 problems across 20 languages, where top models reach only 52.4% pass@1.
-
IFEvalCode: Controlled Code Generation
A 1,620-sample, 8-language, Chinese/English benchmark separates code correctness from instruction-following and shows instruction compliance is far lower than correctness across 40+ LLMs.
-
Seed-Coder: Let the Code Model Curate Data for Itself
Seed-Coder shows that an LLM-trained quality scorer can filter 6T tokens of code data and yield 8B models that outperform similar-size open code models.
-
FullStack Bench: Evaluating LLMs as Full Stack Coders
FullStack Bench and SandboxFusion offer a broader, execution-based benchmark for code LLMs with 3,374 problems in 16 languages and 11 domains.
-
CoCo-Bench: A Comprehensive Code Benchmark For Multi-task Large Language Model Evaluation
CoCo-Bench is a new multi-language benchmark of 705 manually reviewed samples that evaluates code LLMs across understanding, generation, modification, and review tasks, with difficulty-weighted scoring.
-
Programming with AI: Evaluating ChatGPT, Gemini, AlphaCode, and GitHub Copilot for Programmers
By recompiling published benchmark scores, the paper names ChatGPT GPT-4-Turbo-0125 the most accurate coding assistant, with 87.2% pass@1 on HumanEval.
Discussion (0). Continue with ORCID to comment.