Pith. sign in

REVIEW 15 cited by

CodeEditorBench: Evaluating Code Editing Capability of Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.03543 v3 pith:TEJKS4WZ submitted 2024-04-04 cs.SE cs.AIcs.CLcs.LG

Jiawei Guo , Ziming Li , Xueling Liu , Kaijing Ma , Tianyu Zheng , Zhouliang Yu , Ding Pan , Yizhi LI
show 8 more authors
Ruibo Liu Yue Wang Shuyue Guo Xingwei Qu Xiang Yue Ge Zhang Wenhu Chen Jie Fu
This is my paper · ORCID
classification cs.SEcs.AIcs.CLcs.LG
keywords codecodeeditorbencheditingllmsmodelscapabilityemergingevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Large Language Models (LLMs) for code are rapidly evolving, with code editing emerging as a critical capability. We introduce CodeEditorBench, an evaluation framework designed to rigorously assess the performance of LLMs in code editing tasks, including debugging, translating, polishing, and requirement switching. Unlike existing benchmarks focusing solely on code generation, CodeEditorBench emphasizes real-world scenarios and practical aspects of software development. We curate diverse coding challenges and scenarios from five sources, covering various programming languages, complexity levels, and editing tasks. Evaluation of 19 LLMs reveals that closed-source models (particularly Gemini-Ultra and GPT-4), outperform open-source models in CodeEditorBench, highlighting differences in model performance based on problem types and prompt sensitivities. CodeEditorBench aims to catalyze advancements in LLMs by providing a robust platform for assessing code editing capabilities. We will release all prompts and datasets to enable the community to expand the dataset and benchmark emerging LLMs. By introducing CodeEditorBench, we contribute to the advancement of LLMs in code editing and provide a valuable resource for researchers and practitioners.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Obey, Diverge, Collapse: Blind Obedience to Incorrect Instructions Drives Code LLMs to Irrecoverable Code Semantic Collapse

    cs.SE 2026-07 conditional novelty 6.5 of 10

    Code LLMs correctly label incorrect repair instructions as wrong, then follow them anyway, creating compounding Ghost Errors that self-guided iterative repair usually cannot reverse.

  2. SynH-Rank: Quality-Aware Code Search via Diverse Data Synthesis and Hierarchical Ranking Training

    cs.SE 2026-07 conditional novelty 6.0 of 10

    A quality-aware code-search reranking framework with LLM-synthesized quality variants and a hierarchical ranking loss beats relevance-only training on a new QPA/MCA benchmark.

  3. RepoDebug: Repository-Level Multi-Task and Multi-Language Debugging Evaluation of Large Language Models

    cs.SE 2025-09 conditional novelty 6.0 of 10

    RepoDebug is a new multi-language, multi-task benchmark for repository-level code debugging on which current LLMs, including the best model Claude 3.5 Sonnet, perform poorly.

  4. WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generation

    cs.CV 2025-08 conditional novelty 6.0 of 10

    WebMMU introduces a multilingual, three-task benchmark for website understanding and code generation, and finds current MLLMs underperform on reasoning, grounding, and functional code editing.

  5. AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators

    cs.CL 2025-08 conditional novelty 6.0 of 10

    AutoCodeBench is an LLM-generated, sandbox-verified code benchmark with 3,920 problems across 20 languages, where top models reach only 52.4% pass@1.

  6. SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories?

    cs.SE 2025-07 conditional novelty 6.0 of 10

    A repository-level benchmark shows LLMs and agents currently achieve far smaller, statistically verified speedups than expert human patches on 140 real-world Python performance tasks.

  7. Coding Triangle: How Does Large Language Model Understand Code?

    cs.CL 2025-07 conditional novelty 6.0 of 10

    The Coding Triangle framework evaluates LLMs on editorials, code, and test cases, revealing that models are self-consistent yet lack diversity and that model mixtures improve robustness.

  8. LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks

    cs.SE 2025-02 conditional novelty 6.0 of 10

    Across 83 SE benchmarks, average leakage into StarCoder's pretraining data is 4.8% (Python), 2.8% (Java), and 0.7% (C/C++), but QuixBugs and BigCloneBench are 100% and 55.7% leaked.

  9. Evaluating and Aligning CodeLLMs on Human Preference

    cs.CL 2024-12 conditional novelty 6.0 of 10

    A new 397-sample, 44-language code Q&A benchmark with LLM-judged human preference reveals gaps between open and proprietary code models and a 20B-token synthetic instruction corpus.

  10. Code Monitor Red Teaming for Public-Test-Passing Code

    cs.AI 2026-07 conditional novelty 5.0 of 10

    Weak LLM verifiers miss most hidden bugs in public-test-passing code, and adversarial pressure makes that monitoring worse.

  11. BinMetric: A Comprehensive Binary Analysis Benchmark for Large Language Models

    cs.SE 2025-05 conditional novelty 5.0 of 10

    BinMetric is a new 1,000-question, six-task benchmark that measures LLM ability across the binary analysis lifecycle, with an empirical study of 12 models showing strong semantic understanding but weak low-level lifti...

  12. ExecRepoBench: Multi-level Executable Code Completion Evaluation

    cs.CL 2024-12 reject novelty 5.0 of 10

    ExecRepoBench is an executable, repository-level code completion benchmark with unit tests, and Qwen2.5-Coder-Instruct-C is a 7B model fine-tuned on the accompanying Repo-Instruct corpus.

  13. PLD+: Accelerating LLM inference by leveraging Language Model Artifacts

    cs.CL 2024-12 conditional novelty 5.0 of 10

    PLD+ accelerates LLM inference on input-guided tasks by ranking prompt-derived draft spans with hidden states or attention heads, beating tuning-free baselines and often surpassing the tuned EAGLE method.

  14. CoCo-Bench: A Comprehensive Code Benchmark For Multi-task Large Language Model Evaluation

    cs.SE 2025-04 reject novelty 4.0 of 10

    CoCo-Bench is a new multi-language benchmark of 705 manually reviewed samples that evaluates code LLMs across understanding, generation, modification, and review tasks, with difficulty-weighted scoring.

  15. Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks

    cs.CL 2025-04 conditional novelty 4.0 of 10

    A survey that frames the core problem of LLM evaluation as 'evaluation generalization': finite test sets cannot scale with unbounded model capabilities, and proposes two transitions in evaluation design.

Pith tools