Pith. sign in

REVIEW 3 major objections 4 minor 93 references

Language models that repair code systematically retain lines their edit requires removing: even on tasks all five solve, deletion recall tops out at 71.7% and 29% of passing patches guard the obsolete code.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 01:23 UTC pith:62P2KX6J

load-bearing objection A careful, well-scoped paper that makes a real measurement contribution—deletion avoidance in LLM code edits—and despite the developer-patch-as-ground-truth caveat, the central claim survives because the benchmark and the paired retrofit are honest about what they test. the 3 major comments →

arxiv 2607.28887 v1 pith:62P2KX6J submitted 2026-07-30 cs.SE cs.AIcs.LG

To Add Is Machine, To Delete Is Human: Measuring and Mitigating Deletion Avoidance in LLM Code Editing

Amir M. Ebrahimi , Mohammed Mehedi Hasan , Aaditya Bhatia , Gopi Krishnan Rajbahadur , Ahmed E. Hassan This is my paper
classification cs.SE cs.AIcs.LG
keywords deletion avoidancecode repair evaluationSWE-bench VerifiedGuard-and-GoCanItDeletesubtractive code editspost-trainingmaintainability of AI-generated code
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper identifies and measures a concrete failure mode in LLM code repair: deletion avoidance, the systematic tendency to keep code that an intended edit requires removing. Even on SWE-bench Verified tasks that all five leading models solve, their patches delete at most 71.7% of the lines a human developer's patch removes — they reach the right file for over 92% of required deletions but cut the exact line in under 52% of cases — and 29% of passing patches preserve the targeted code inside a new guard or fallback ('Guard-and-Go'). Such patches pass because the original tests rarely check that code was removed; retrofitting 34 tasks with deletion-sensitive tests drops resolution from 63.2% to 41.9% across four frontier models. In a purpose-built deletion-only benchmark (CanItDelete), the best model still fails one task in five, and giving models the exact lines to delete trades retention for over-deletion rather than fixing the edit. A small deletion-focused post-training intervention (0.7% of tokens) cuts the behavior and improves general repair benchmarks, indicating deletion is undertrained rather than beyond models' reach.

Core claim

Models consistently edit the right location yet fail to finish the removal. On the 197 SWE-bench Verified tasks all five evaluated models solve, deletion recall against the developer patch ranges from 65.2% to 71.7%, and the dominant failure is not locating the code but deleting it: models modify the file containing a required deletion for over 92% of targets and the enclosing scope for 68–74%, yet remove the exact line only 44.6–51.6% of the time. The paper shows this is an evaluation blind spot rather than a stylistic preference: source-level tests that fail when a validated deletion target remains cut resolution by 21.3 percentage points, with 29 of 86 previously passing attempts retainin

What carries the argument

The argument is carried by three interlocking instruments. First, deletion recall — the fraction of a human developer patch's removed lines that a model also removes, matched by source location rather than text — turns 'test-passing' from a boolean into a measure of edit fidelity. Second, the Guard-and-Go taxonomy names the dominant substitute behavior (retaining the removed logic behind a newly added condition or fallback), and source-level retrofitted tests quantify how often that substitute survives behavioral grading. Third, CanItDelete — 200 deletion-only tasks mined from real commits, scored by a deterministic occurrence-aware evaluator, with a four-mode diagnostic ladder (vanilla, exp

Load-bearing premise

The entire measurement treats the human developer patch's deleted lines as ground truth for what an edit requires removing; the paper explicitly allows that a retained target may belong to an adequate alternative repair, and the retrofitted checks are built from those same deletions by design, so if developer-patch deletions are not genuinely required, the recall figures and the 21.3-point drop overstate the deficiency.

What would settle it

Take tasks where deletion is truly required, verified independently of any human patch — for example, a test that fails whenever the targeted code is present for some input — and count how many model patches that retain the target still pass. If most retained-target patches pass such independent tests, deletion avoidance is largely a stylistic divergence; if they fail, the paper's central gap is a genuine correctness failure.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Benchmark pass rates overstate repair quality: on deletion-heavy SWE-bench Verified tasks, roughly a third of patches accepted by the original test suites retain a validated deletion target.
  • Test-passing and merge-ready behavior diverge: Guard-and-Go patches exceed the developer patch in size in 61.1% of pairs (median 1.67×) and keep obsolete logic executable, which the paper ties to maintainers' reported rejection of AI-generated pull requests.
  • Deletion completion and scope preservation are distinct capabilities: exact spans nearly eliminate incomplete deletion for four of five models, yet up to 26.0% of attempts still fail by deleting beyond the boundary or adding code.
  • Deletion avoidance is trainable: a 0.7% deletion-focused addition to a code post-training mixture reduced incomplete deletion by 13.9 points and transferred to SWE-bench Verified (+5.3) and CanItEdit (+1.4) without regressing EditBench.
  • Test design should encode removal requirements: deletion-sensitive checks, not just behavioral fail-to-pass tests, are needed before a patch can be certified as actually performing the subtractive part of a change.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the 92%-file / 52%-line gap implies the real bottleneck is token-level knowledge of 'where the edit ends', not 'where the problem is' — which predicts that localization-based agents will keep underperforming until boundary supervision, not search, is built into training.
  • Beyond the paper: because the retrofitted checks are constructed from the developer patch's deletions by design (Appendix C), the 21.3-point drop measures agreement with the human edit rather than independently verified necessity; an independent ground truth built from behavior alone would separate style from correctness.
  • Beyond the paper: the same additive bias should appear in other generative editing domains — configs, documentation, data pipelines — where a 'working' output rarely requires absence; Guard-and-Go is a testable general hypothesis beyond code.
  • Beyond the paper: deletion recall against a human diff is a cheap, concrete maintainability lint — flagging model patches that retain lines a parallel human edit removed could catch bloat before merge.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper identifies and measures 'deletion avoidance' in LLM code editing: the tendency of models to retain source lines that a developer patch removes. Using official SWE-bench Verified submissions for five models, it reports deletion recall against the developer patch (65.2–71.7% on tasks all five solve), a Guard-and-Go pattern in 29.0% of passing patches, and a 21.3 percentage-point drop when 34 deletion-heavy tasks are retrofitted with source-level checks requiring the developer-removed target to be absent. It then introduces CanItDelete, a 200-task deletion-only benchmark mined from real commits, with a deterministic occurrence-aware evaluator and a four-mode diagnostic ladder; best vanilla success is 79.0%, and incomplete deletion dominates failures. A pilot adds 0.7% deletion examples to a 7B post-training mixture and reports reduced incomplete deletion with transfer gains on SWE-bench Verified and CanItEdit.

Significance. If the measurements hold, the paper makes an important, timely contribution. The central observation—that test-passing patches often preserve code a developer removed, and that current benchmarks do not penalize this—is plausible and is supported by three convergent designs: in-the-wild patch analysis, a retrofitted deletion-sensitive evaluation, and a deletion-only benchmark with deterministic scoring. The paper's explicit limitations and the deterministic evaluator are strengths. However, the headline claim that test-based benchmarks 'overstate repair quality' rests on equating developer-patch deletions with behaviorally required removals. The paper hedges this in prose (§2 opening; §3) and in the limitations, but the abstract and findings repeatedly use 'required deletion,' which is not independently established. This gap is load-bearing for the overstatement conclusion, though it is fixable by tempering the claims or adding independent validation. The pilot is also a single-model, three-run result without variance, so the mitigation claim is preliminary.

major comments (3)
  1. [§2.1, §3, Appendix C; abstract] The central metric, deletion recall R_t,m = |G_t∩M_t,m|/|G_t|, counts exact-location agreement with the developer patch, and every headline quantity inherits that choice: the 29.0% Guard-and-Go rate is defined relative to developer-removed logic, and the retrofitted checks are constructed from the developer patch's deletions by design (Appendix C). The paper correctly states that a retained target may belong to an adequate alternative repair, but this nuance is not reflected in the abstract's 'required deletions' or in Finding 3's 'explicit removal requirement.' The 21.3-pp retrofit drop therefore measures performance under a developer-derived removal requirement, not under an independently justified necessity. To support 'benchmarks overstate repair quality,' the paper must either temper the claim to 'divergence from the developer patch' or provide independent evidence—e.g., maintainer
  2. [§4.1, Appendix D.2, §7] CanItDelete instructions are drafted by GPT-5.6 Sol, itself one of the evaluated models, using the pre-edit file and reference diff from the 100 most-starred repositories. The paper acknowledges in §7 that post-edit files may appear in training data, but the benchmark is presented as a diagnostic stress test for all twelve models. If GPT-5.6 Sol (or other models) memorized post-edit files, its 74.0% vanilla-mode success could be inflated, and the instruction-generation model is not independent of the evaluation. Please provide a contamination analysis—e.g., a temporal split using commits after model knowledge cutoffs, or a memorization probe—or at least present results separately for tasks likely to be in training data. Without this, the benchmark's difficulty for frontier models is hard to interpret.
  3. [§5, Table 4, abstract] The pilot reports three-run means without variance, on a single 7B model, and the transfer claim rests on +5.30 points on SWE-bench Verified and +1.40 on CanItEdit while EditBench is −0.19. These differences are small relative to likely run-to-run variance, and no confidence intervals or per-run values are reported. The abstract's statement that deletion post-training 'improves broader code-editing performance' is therefore not supported by the evidence as presented. The paper itself calls the study a proof of concept, but the title-level framing 'Mitigating Deletion Avoidance' and the abstract go further. Please add variance estimates or multiple models, and restrict the claim to 'reduces incomplete deletion on CanItDelete and does not regress the three benchmarks tested.'
minor comments (4)
  1. [§1 and reference list] Citation 'AlAwad and Ivanov 2026' in the introduction appears as 'Al Awad, M. N.; and Ivanov, S.' in the references; please standardize the name formatting.
  2. [Appendix A.4] The heading 'Exploring the Deletion Avoidance from Localization point of view' has inconsistent capitalization; please revise.
  3. [§1, 'Contributions'] The paper says all measurement code, the benchmark, and prompts are in the supplementary material 'withheld from public release for anonymity.' For a measurement-heavy paper, independent verification requires at least reviewer access to these artifacts; please clarify the review-time availability and the release plan.
  4. [§2.2] The sentence 'Localization explains the decline from file to scope' is slightly ambiguous on first reading; the intended contrast is that localization explains the file→scope drop, not the scope→line drop. Consider rewording for clarity.

Circularity Check

1 steps flagged

No significant circularity; one acknowledged definitional boundary where 'required deletion' is operationalized as developer-patch deletion, plus a non-load-bearing self-citation.

specific steps
  1. self definitional [§2.1 (Metrics); abstract; §3 retrofit checks]
    "A reference deletion is a source location removed by the developer patch from a non-test Python file. ... Deletion recall is the share of the developer's deletions that the model also performs: R_{t,m} = |G_t ∩ M_{t,m}| / |G_t|."

    The abstract defines the phenomenon as retaining 'code that an intended edit requires removing.' The operational definition of 'requires removing' is 'removed by the developer patch' (G_t), so deletion recall, the 28.3–34.8% retention figures, the 29.0% Guard-and-Go share, and the §3 deletion-sensitive checks all measure divergence from the developer patch by construction. The §3 retrofit checks are explicitly derived from 'a target removed by the developer patch,' so the 21.3-pp drop and 33.7% retention rate cannot independently establish that the retained code is required to be removed. The paper acknowledges this limitation (§2: 'a mismatch alone does not establish an error'; §3: 'rather than establishing that deletion is the only behaviorally valid repair'), making the assumption visib

full rationale

The load-bearing derivation chain is otherwise self-contained. Deletion recall and precision are computed against external Git diffs and developer patches; the Guard-and-Go taxonomy is an open-coding exercise on observed patches; the diagnostic ladder uses a deterministic occurrence-aware evaluator with no LLM judge; and the post-training pilot transfers are measured on independent benchmarks (SWE-bench Verified, CanItEdit, EditBench), with CanItDelete evaluation problems excluded from training. Self-citations (Ebrahimi & Rajbahadur 2026 for Edit-Bench coverage; Hassan 2009 and Kamei et al. 2013 for change-complexity dimensions) support auxiliary facts, not the central inference. The only reduction-by-construction step is the identification of 'required deletion' with 'developer-patch deletion': every headline quantity is a divergence measure from G_t, so the paper's strong wording ('retain code that an intended edit requires removing') partly restates the metric. This is explicitly hedged in the text, and CanItDelete independently demonstrates deletion difficulty in settings where deletion is the entire specified edit, so the central claim retains independent content. Score 2 reflects this single acknowledged definitional boundary, not a forced self-citation chain or a fitted-parameter prediction.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The paper's central quantities are measured, not derived; the main ledger items are measurement assumptions and hand-chosen benchmark thresholds. No new physical or mathematical entities are postulated.

free parameters (3)
  • CanItDelete selection thresholds (top-200, ≥3 hunks, ≤100 KB, 100-starred repos)
    Hand-chosen thresholds in Section 4.1 and Appendix D.1 set benchmark difficulty. Not fitted to model performance, but they determine the observed 18-79% success spread.
  • Equal-weight complexity index coefficients = pL/3, pC/3, pH/3 (Eq. 2)
    Equal weights are chosen to avoid an exchange rate among LOC, deleted lines, and hunks; arbitrary but disclosed and independent of model outputs.
  • Retrofit selection threshold: deletions ≥25% of changed lines
    Appendix C selects 69 tasks using this threshold, and 34 survive validation. The threshold may bias toward larger, more detectable deletions.
axioms (4)
  • domain assumption Developer-patch deletions are the ground truth for 'required deletion'.
    Sections 2.1 and 3 define reference deletions from the human patch and derive deletion-sensitive tests from them. The paper concedes alternative repairs may validly retain the target.
  • domain assumption Source-level absence checks in the retrofitted F2P tests correctly encode behavioral correctness.
    Appendix C tests only whether target source lines are absent; runtime equivalence of retaining the target under a guard is not tested, so the 21.3 pp drop depends on this encoding.
  • domain assumption LLM-classifier strategy labels (Guard-and-Go etc.) are accurate.
    Section 2.3 and Appendix B use MiniMax-M2.7 to label 2,358 pairs with cited evidence. Validation is a single author checking 50 of 550 typed pairs; no inter-annotator agreement is reported.
  • domain assumption CanItDelete tasks remain valid despite generated instructions and possible training-data overlap.
    Appendix D.2: instructions are drafted by GPT-5.6 Sol (itself evaluated) from most-starred repos whose post-edit files may be in training data; acknowledged in Section 7.

pith-pipeline@v1.3.0-alltime-deepseek · 20434 in / 12504 out tokens · 136702 ms · 2026-08-03T01:23:22.700373+00:00 · methodology

0 comments
read the original abstract

Large language models increasingly write and repair production code, yet evidence is mounting that their test-passing patches leave codebases harder to maintain. We identify one concrete source: deletion avoidance, the systematic tendency to retain code that an intended edit requires removing. Across the five leading models on the official SWE-bench Verified leaderboard, deletion recall against the developer patch reaches at most 71.7% even on tasks all five solve, and models reach the right file for over 92% of required deletions but cut the exact line in under 52% of cases. Instead, 29.0% of passing patches wrap the targeted code in a guard or fallback, a pattern we call Guard-and-Go. Such patches pass because the original tests rarely check removal: when we retrofit 34 Verified tasks with tests that fail if the targeted code remains, four frontier models spanning closed and open weights fall from 63.2% to 41.9%. Because real repairs mix removal with addition, we curate CanItDelete, a benchmark of 200 tasks mined from real commits whose entire required edit is deletion. Even with the addition work gone, the best model still fails one task in five, and smaller open models fall to 18.0%. We then ablate GPT-5.6 Sol under four cumulative prompts; success moves little until we supply the exact lines, which nearly eliminate incomplete deletion yet raise success only to 80.5% because the model then deletes beyond the spans or adds code instead. Finally, through a pilot study we show one potential fix: teaching deletion during post-training reduces deletion avoidance and improves broader code-editing performance, suggesting the behavior is undertrained rather than beyond reach.

Figures

Figures reproduced from arXiv: 2607.28887 by Aaditya Bhatia, Ahmed E. Hassan, Amir M. Ebrahimi, Gopi Krishnan Rajbahadur, Mohammed Mehedi Hasan.

Figure 1
Figure 1. Figure 1: An example of deletion avoidance in a solved SWE [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of CanItDelete benchmark construction, cumulative diagnostic modes, and the structural outcome taxonomy. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Vanilla-mode success and failure composition [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Diagnostic-ladder outcomes under increasingly precise deletion guidance. Four models use 200 tasks; Claude Opus [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: File-, scope-, and exact-line overlap across all re [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: summarizes the benchmark. Median (IQR) values are 1,551 (1,266–1,800) pre-edit LOC, 34 (29–39) deleted lines, 4 (3–5) deletion hunks, and 8 (6–10) instruction words. D.2 Instruction Generation and Task Validation After structural selection, GPT-5.6 Sol receives the pre-edit file, reference diff, file path, and commit message as weak context. It generates a short, present-tense request covering the complete… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

93 extracted references · 14 linked inside Pith

  1. [1]

    Copilot Arena: A Platform for Code

    Wayne Chi and Valerie Chen and Anastasios Nikolas Angelopoulos and Wei-Lin Chiang and Aditya Mittal and Naman Jain and Tianjun Zhang and Ion Stoica and Chris Donahue and Ameet Talwalkar , booktitle =. Copilot Arena: A Platform for Code. 2025 , url =

    2025

  2. [2]

    Hassan , journal =

    Hao Li and Haoxiang Zhang and Ahmed E. Hassan , journal =. The Rise of. 2025 , url =

    2025

  3. [3]

    2025 , howpublished =

    Octoverse: A New Developer Joins. 2025 , howpublished =

    2025

  4. [4]

    2025 , url =

    Wayne Chi and Valerie Chen and Ryan Shar and Aditya Mittal and Jenny Liang and Wei-Lin Chiang and Anastasios Nikolas Angelopoulos and Ion Stoica and Graham Neubig and Ameet Talwalkar and Chris Donahue , booktitle =. 2025 , url =

    2025

  5. [5]

    Is Your Code Generated by

    Liu, Jiawei and Xia, Chunqiu Steven and Wang, Yuyao and Zhang, Lingming , booktitle =. Is Your Code Generated by. 2023 , url =

    2023

  6. [6]

    Evaluating the Test Adequacy of Benchmarks for

    Liu, Xiangyue and Sun, Xiaobing and Bo, Lili and Hu, Yufei and Liu, Xinwei and Ye, Zhenlei , journal =. Evaluating the Test Adequacy of Benchmarks for. 2025 , doi =

    2025

  7. [7]

    and Cheung, Shing-Chi , year =

    Cao, Jialun and Chan, Yuk-Kit and Ling, Zixuan and Wang, Wenxuan and Li, Shuqing and Liu, Mingwei and Qiao, Ruixi and Han, Yuting and Wang, Chaozheng and Yu, Boxi and He, Pinjia and Wang, Shuai and Zheng, Zibin and Lyu, Michael R. and Cheung, Shing-Chi , year =. Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility (. 2501.10711 , arch...

    Pith/arXiv arXiv

  8. [8]

    Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (ESEC/FSE) , pages =

    How Do Fixes Become Bugs? , author =. Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (ESEC/FSE) , pages =. 2011 , doi =

    2011

  9. [9]

    2022 , doi =

    Guo, Daya and Lu, Shuai and Duan, Nan and Wang, Yanlin and Zhou, Ming and Yin, Jian , booktitle =. 2022 , doi =

    2022

  10. [10]

    arXiv preprint arXiv:2511.18538 , year =

    From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence , author =. arXiv preprint arXiv:2511.18538 , year =

    arXiv

  11. [11]

    Information and Software Technology , year =

    Agnia Sergeyuk and Yaroslav Golubev and Timofey Bryksin and Iftekhar Ahmed , title =. Information and Software Technology , year =

  12. [12]

    2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR) , pages=

    Can LLMs replace manual annotation of software engineering artifacts? , author=. 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR) , pages=. 2025 , organization=

    2025

  13. [13]

    Understanding and Supporting How Developers Prompt for

    Nam, Daye and Omran, Ahmed and Murillo, Ambar and Thakur, Saksham and Araujo, Abner and Blistein, Marcel and Fr. Understanding and Supporting How Developers Prompt for. Proceedings of the ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE) , year =

  14. [14]

    2025 , howpublished =

    State of. 2025 , howpublished =

    2025

  15. [15]

    2025 2nd IEEE/ACM International Conference on AI-powered Software (AIware) , pages=

    Where Do LLMs Still Struggle? An In-Depth Analysis of Code Generation Benchmarks , author=. 2025 2nd IEEE/ACM International Conference on AI-powered Software (AIware) , pages=. 2025 , organization=

    2025

  16. [16]

    and Rajbahadur, Gopi Krishnan and Bhatia, Aaditya and Zhang, Haoxiang and Chen, Yihao and Chen, Zhilong and Leung, Arthur and Lin, Dayi and Chen, Boyuan and Hassan, Ahmed E

    Oliva, Gustavo A. and Rajbahadur, Gopi Krishnan and Bhatia, Aaditya and Zhang, Haoxiang and Chen, Yihao and Chen, Zhilong and Leung, Arthur and Lin, Dayi and Chen, Boyuan and Hassan, Ahmed E. , year =. SPICE: An Automated SWE-Bench Labeling Pipeline for Issue Clarity, Test Coverage, and Effort Estimation , url =. doi:10.1109/ase63991.2025.00192 , booktitle =

    arXiv 2025

  17. [17]

    Octoverse: A New Developer Joins GitHub Every Second as AI Leads TypeScript to \#1 , year =

  18. [18]

    The 3rd DL4C Workshop: Emergent Possibilities and Challenges in Deep Learning for Code, at ICLR 2025 , year =

    Jiawei Guo and Ziming Li and Xueling Liu and Kaijing Ma and Tianyu Zheng and Zhouliang Yu and Ding Pan and Yizhi Li and Ruibo Liu and Yue Wang and Shuyue Guo and Xingwei Qu and Xiang Yue and Ge Zhang and Wenhu Chen and Jie Fu , title =. The 3rd DL4C Workshop: Emergent Possibilities and Challenges in Deep Learning for Code, at ICLR 2025 , year =

    2025

  19. [19]

    arXiv preprint arXiv:2509.25203 , year=

    Generating High-Quality Datasets for Code Editing via Open-Source Language Models , author=. arXiv preprint arXiv:2509.25203 , year=

    arXiv

  20. [20]

    Advances in Neural Information Processing Systems , volume=

    Selfcodealign: Self-alignment for code generation , author=. Advances in Neural Information Processing Systems , volume=

  21. [21]

    arXiv preprint arXiv:2402.19173 , year=

    Starcoder 2 and the stack v2: The next generation , author=. arXiv preprint arXiv:2402.19173 , year=

    Pith/arXiv arXiv

  22. [22]

    arXiv e-prints , pages=

    LLMs Love Python: A Study of LLMs' Bias for Programming Languages and Libraries , author=. arXiv e-prints , pages=

  23. [23]

    2026 , howpublished =

    Why. 2026 , howpublished =

    2026

  24. [24]

    arXiv preprint arXiv:2604.05100 , year =

    Edit, But Verify: An Empirical Audit of Instructed Code-Editing Benchmarks , author =. arXiv preprint arXiv:2604.05100 , year =

    Pith/arXiv arXiv

  25. [25]

    Chong, Chun Jie and Ahmed, Muyeed and Yao, Zhihao and Neamtiu, Iulian , booktitle =. Can. 2026 , url =

    2026

  26. [26]

    An Empirical Study of Python Library Migration Using Large Language Models , year =

  27. [27]

    2025 , howpublished =

    Submit to. 2025 , howpublished =

    2025

  28. [28]

    Garg, Spandan and Steenhoek, Benjamin and Huang, Yufan , journal =. Saving. 2025 , note =

    2025

  29. [29]

    2025 , url =

    Li, Wei and Zhang, Xin and Guo, Zhongxin and Mao, Shaoguang and Luo, Wen and Peng, Guangyue and Huang, Yangyu and Wang, Houfeng and Li, Scarlett , booktitle =. 2025 , url =

    2025

  30. [30]

    arXiv preprint arXiv:2509.22237 , year =

    FeatBench: Evaluating Coding Agents on Feature Implementation for Vibe Coding , author =. arXiv preprint arXiv:2509.22237 , year =

    arXiv

  31. [31]

    arXiv preprint arXiv:2602.10975 , year =

    FeatureBench: Benchmarking Agentic Coding for Complex Feature Development , author =. arXiv preprint arXiv:2602.10975 , year =

    arXiv

  32. [32]

    Harbor: A Framework for Evaluating and Improving Agents , year =

  33. [33]

    Liang, Shanchao and Garg, Spandan and Zilouchian Moghaddam, Roshanak , journal =. The

  34. [34]

    Advances in Neural Information Processing Systems , volume =

    Establishing Best Practices in Building Rigorous Agentic Benchmarks , author =. Advances in Neural Information Processing Systems , volume =

  35. [35]

    and Nadgir, Nitya and Narayanan, Arvind , journal =

    Kapoor, Sayash and Stroebl, Benedikt and Siegel, Zachary S. and Nadgir, Nitya and Narayanan, Arvind , journal =

  36. [36]

    and Fitz, Stephen and Hendrycks, Dan , booktitle =

    Ren, Richard and Basart, Steven and Khoja, Adam and Gatti, Alice and Phan, Long and Yin, Xuwang and Mazeika, Mantas and Pan, Alexander and Mukobi, Gabriel and Kim, Ryan H. and Fitz, Stephen and Hendrycks, Dan , booktitle =. Safetywashing: Do. 2024 , url =

    2024

  37. [37]

    , booktitle =

    Reuel, Anka and Hardy, Amelia and Smith, Chandler and Lamparth, Max and Hardy, Malcolm and Kochenderfer, Mykel J. , booktitle =. BetterBench: Assessing. 2024 , url =

    2024

  38. [38]

    and Paullada, Amandalynne and Denton, Emily and Hanna, Alex , booktitle =

    Raji, Inioluwa Deborah and Bender, Emily M. and Paullada, Amandalynne and Denton, Emily and Hanna, Alex , booktitle =. 2021 , url =

    2021

  39. [39]

    Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT) , pages =

    Measurement and Fairness , author =. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT) , pages =. 2021 , doi =

    2021

  40. [40]

    Concrete Problems in

    Amodei, Dario and Olah, Chris and Steinhardt, Jacob and Christiano, Paul and Schulman, John and Man. Concrete Problems in. arXiv preprint arXiv:1606.06565 , year =

    Pith/arXiv arXiv

  41. [41]

    2020 , howpublished =

    Krakovna, Victoria and Uesato, Jonathan and Mikulik, Vladimir and Rahtz, Matthew and Everitt, Tom and Kumar, Ramana and Kenton, Zac and Leike, Jan and Legg, Shane , title =. 2020 , howpublished =

    2020

  42. [42]

    arXiv preprint arXiv:1803.04585 , year =

    Categorizing Variants of Goodhart's Law , author =. arXiv preprint arXiv:1803.04585 , year =

    Pith/arXiv arXiv

  43. [43]

    arXiv preprint arXiv:2411.15114 , year=

    Re-bench: Evaluating frontier ai r&d capabilities of language model agents against human experts , author=. arXiv preprint arXiv:2411.15114 , year=

    Pith/arXiv arXiv

  44. [44]

    We Need a Science of Evals , year =

  45. [45]

    Responsible Scaling Policy , year =

  46. [46]

    Preparedness Framework, Version 2 , year =

  47. [47]

    Frontier Safety Framework , year =

  48. [48]

    2026 , howpublished =

    Shayanfar, Radin and Gallaba, Keheliya , title =. 2026 , howpublished =

    2026

  49. [49]

    2026 , month = mar, howpublished =

    Opus Finds the Problems. 2026 , month = mar, howpublished =

    2026

  50. [50]

    Proceedings of the 27th International Conference on Software Engineering (ICSE) , pages =

    Use of Relative Code Churn Measures to Predict System Defect Density , author =. Proceedings of the 27th International Conference on Software Engineering (ICSE) , pages =. 2005 , doi =

    2005

  51. [51]

    Proceedings of the 31st International Conference on Software Engineering (ICSE) , pages =

    Predicting Faults Using the Complexity of Code Changes , author =. Proceedings of the 31st International Conference on Software Engineering (ICSE) , pages =. 2009 , doi =

    2009

  52. [52]

    IEEE Transactions on Software Engineering , volume =

    A Large-Scale Empirical Study of Just-in-Time Quality Assurance , author =. IEEE Transactions on Software Engineering , volume =. 2013 , doi =

    2013

  53. [53]

    Proceedings of the 30th International Conference on Software Engineering (ICSE) , pages =

    A Comparative Analysis of the Efficiency of Change Metrics and Static Code Attributes for Defect Prediction , author =. Proceedings of the 30th International Conference on Software Engineering (ICSE) , pages =. 2008 , doi =

    2008

  54. [54]

    Ni, Ziyi and Wang, Huacan and Zhang, Shuo and Lu, Shuo and He, Ziyang and Tang, Zhenheng and Hu, Sen and Li, Bo and Hu, Chen and Jiao, Binxing and others , booktitle=

  55. [55]

    Ouyang, Shuyin and Huang, Dong and Guo, Jingwen and Sun, Zeyu and Zhu, Qihao and Zhang, Jie M , booktitle=

  56. [56]

    arXiv preprint arXiv:2408.13204 , year=

    Domaineval: An auto-constructed benchmark for multi-domain code generation , author=. arXiv preprint arXiv:2408.13204 , year=

    Pith/arXiv arXiv

  57. [57]

    2024 , url =

    Li, Yucheng and Guerin, Frank and Lin, Chenghua , booktitle =. 2024 , url =

    2024

  58. [58]

    2026 , howpublished =

    Prompting Guidance for. 2026 , howpublished =

    2026

  59. [59]

    arXiv preprint arXiv:2605.07769 , year =

    Coding Agents Don't Know When to Act , author =. arXiv preprint arXiv:2605.07769 , year =

    Pith/arXiv arXiv

  60. [60]

    Empirical Software Engineering , volume =

    A Comprehensive Study of Code-removal Patches in Automated Program Repair , author =. Empirical Software Engineering , volume =. 2022 , doi =

    2022

  61. [61]

    2025 , url =

    Chen, Minyu and Li, Guoqiang and Wu, Ling-I and Liu, Ruibang , booktitle =. 2025 , url =

    2025

  62. [62]

    On the Role of Fault Localization Context for

    Sepidband, Melika and Pham, Hung Viet and Hemmati, Hadi , journal =. On the Role of Fault Localization Context for. 2026 , url =

    2026

  63. [63]

    arXiv preprint arXiv:2606.28430 , year =

    Building to the Test: Coding Agents Deliver What You Check, Not What You Requested , author =. arXiv preprint arXiv:2606.28430 , year =

    Pith/arXiv arXiv

  64. [64]

    A Long Way to Go: Investigating Length Correlations in

    Singhal, Prasann and Goyal, Tanya and Xu, Jiacheng and Durrett, Greg , journal =. A Long Way to Go: Investigating Length Correlations in. 2023 , url =

    2023

  65. [65]

    NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following , year =

    Verbosity Bias in Preference Labeling by Large Language Models , author =. NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following , year =

    2023

  66. [66]

    2025 , month = nov, howpublished =

    Myers, Cashius , title =. 2025 , month = nov, howpublished =

    2025

  67. [67]

    arXiv preprint arXiv:2604.17338 , year =

    Precise Debugging Benchmark: Is Your Model Debugging or Regenerating? , author =. arXiv preprint arXiv:2604.17338 , year =

    Pith/arXiv arXiv

  68. [68]

    arXiv preprint arXiv:2603.24631 , year =

    Coherence Collapse: Diagnosing Why Code Agents Fail After Reaching the Right Code , author =. arXiv preprint arXiv:2603.24631 , year =

    Pith/arXiv arXiv

  69. [69]

    Whitfill, Parker and Wu, Cheryl and Becker, Joel and Rush, Nate , howpublished =. Many. 2026 , month =

    2026

  70. [70]

    2026 , month =

    Introducing. 2026 , month =

    2026

  71. [71]

    Proceedings of the 23rd International Conference on Mining Software Repositories (MSR) , year =

    What to Cut? Predicting Unnecessary Methods in Agentic Code Generation , author =. Proceedings of the 23rd International Conference on Mining Software Repositories (MSR) , year =

  72. [72]

    Code Provenance: Use of

  73. [73]

    Understanding the Rejection of Fixes Generated by Agentic Pull Requests: Insights from the

    Abujadallah, Mahmoud and Arabat, Ali and Sayagh, Mohammed , journal =. Understanding the Rejection of Fixes Generated by Agentic Pull Requests: Insights from the

  74. [74]

    ArXiv , year=

    Why Are AI Agent Involved Pull Requests (Fix-Related) Remain Unmerged? An Empirical Study , author=. ArXiv , year=

  75. [75]

    arXiv preprint arXiv:2605.22534 , year =

    Why Are Agentic Pull Requests Merged or Rejected? An Empirical Study , author =. arXiv preprint arXiv:2605.22534 , year =

    Pith/arXiv arXiv

  76. [76]

    Loc2Repair: A Framework for Evaluating the Impact of File-Level Issue Localization in Repo-Level

    Al Awad, Mohammad Nour and Ivanov, Sergey , journal =. Loc2Repair: A Framework for Evaluating the Impact of File-Level Issue Localization in Repo-Level

  77. [77]

    On the Role of Fault Localization Context for

    Sepidband, Milad and Pham, Hung Viet and Hemmati, Hadi , journal =. On the Role of Fault Localization Context for

  78. [78]

    2026 , note =

    The Maintainability Gap:. 2026 , note =

    2026

  79. [79]

    Debt Behind the

    Liu, Yue and Widyasari, Ratnadira and Zhao, Yanjie and Irsan, Ivana Clairine and Chen, Junkai and Lo, David , journal =. Debt Behind the

  80. [80]

    Nature , volume =

    People Systematically Overlook Subtractive Changes , author =. Nature , volume =. 2021 , doi =

    2021

Showing first 80 references.