REVIEW 7 cited by
CodeEditorBench: Evaluating Code Editing Capability of Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs) for code are rapidly evolving, with code editing emerging as a critical capability. We introduce CodeEditorBench, an evaluation framework designed to rigorously assess the performance of LLMs in code editing tasks, including debugging, translating, polishing, and requirement switching. Unlike existing benchmarks focusing solely on code generation, CodeEditorBench emphasizes real-world scenarios and practical aspects of software development. We curate diverse coding challenges and scenarios from five sources, covering various programming languages, complexity levels, and editing tasks. Evaluation of 19 LLMs reveals that closed-source models (particularly Gemini-Ultra and GPT-4), outperform open-source models in CodeEditorBench, highlighting differences in model performance based on problem types and prompt sensitivities. CodeEditorBench aims to catalyze advancements in LLMs by providing a robust platform for assessing code editing capabilities. We will release all prompts and datasets to enable the community to expand the dataset and benchmark emerging LLMs. By introducing CodeEditorBench, we contribute to the advancement of LLMs in code editing and provide a valuable resource for researchers and practitioners.
Forward citations
Cited by 7 Pith papers
-
Obey, Diverge, Collapse: Blind Obedience to Incorrect Instructions Drives Code LLMs to Irrecoverable Code Semantic Collapse
Code LLMs correctly label incorrect repair instructions as wrong, then follow them anyway, creating compounding Ghost Errors that self-guided iterative repair usually cannot reverse.
-
SynH-Rank: Quality-Aware Code Search via Diverse Data Synthesis and Hierarchical Ranking Training
A quality-aware code-search reranking framework with LLM-synthesized quality variants and a hierarchical ranking loss beats relevance-only training on a new QPA/MCA benchmark.
-
RepoDebug: Repository-Level Multi-Task and Multi-Language Debugging Evaluation of Large Language Models
RepoDebug is a new multi-language, multi-task benchmark for repository-level code debugging on which current LLMs, including the best model Claude 3.5 Sonnet, perform poorly.
-
AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators
AutoCodeBench is an LLM-generated, sandbox-verified code benchmark with 3,920 problems across 20 languages, where top models reach only 52.4% pass@1.
-
SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories?
A repository-level benchmark shows LLMs and agents currently achieve far smaller, statistically verified speedups than expert human patches on 140 real-world Python performance tasks.
-
Coding Triangle: How Does Large Language Model Understand Code?
The Coding Triangle framework evaluates LLMs on editorials, code, and test cases, revealing that models are self-consistent yet lack diversity and that model mixtures improve robustness.
-
Code Monitor Red Teaming for Public-Test-Passing Code
Weak LLM verifiers miss most hidden bugs in public-test-passing code, and adversarial pressure makes that monitoring worse.
Discussion (0). Sign in to comment.