Pith. sign in

REVIEW 2 major objections 5 minor 94 references

Coding agents that pass functional tests still systematically miss behavior-preserving code-quality improvements measured by the new SWE-NFI benchmark.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 07:50 UTC pith:7ANK5EOE

load-bearing objection A solid, useful benchmark for evaluating coding agents on non-functional improvements, but the headline agent-vs-human gap is computed over non-identical task sets and should not be taken at face value. the 2 major comments →

arxiv 2607.27409 v1 pith:7ANK5EOE submitted 2026-07-29 cs.SE cs.AI

SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvements

Pengyu Xue , He Yang Yuan , Xin Wang , Junkai Chen , Haonan Zhang , Boyuan Chen , Zishuo Ding , Zhenhao Li
show 1 more author
Weiyi Shang
This is my paper
classification cs.SE cs.AI
keywords SWE-NFInon-functional improvementscoding agentsbenchmarkcode qualityfunctional correctnesssoftware maintenancedeterministic evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

SWE-NFI asks whether coding agents can do what human developers do during maintenance: improve documentation, error handling, library usage, coding style, and logic patterns without changing observable behavior. The benchmark turns those five aspects into 92 executable rules and scores each agent output as the difference in rule satisfaction before and after, but only counts outputs that first pass hand-written functional tests. Across 188 tasks built from real merged pull requests, the best agent passes functional tests 70% of the time, yet every agent falls short of the human reference on structural improvements (logic-pattern scores of 0.0–1.3 versus 1.5). The paper's central claim is that functional-correctness benchmarks overstate agent usefulness: today's agents can preserve behavior but rarely improve its maintainability the way developers do.

Core claim

The paper introduces SWE-NFI, a benchmark of 188 tasks drawn from real merged Python pull requests, and claims it measures a real gap. Its central finding: on outputs that pass functional tests, all eight evaluated agent configurations score below the human reference on logic-pattern improvements, and multi-file tasks widen the gap, while documentation and error handling are the aspects agents handle best. The benchmark operationalizes developer-oriented non-functional improvements as 92 deterministic and threshold-based rules across five aspects, so the evaluation is reproducible and does not depend on an LLM judge.

What carries the argument

The core mechanism is the per-aspect improvement score I_p = S_p(code_after) − S_p(code_before), where S_p is the fraction of the aspect's rules satisfied by the code. Functional correctness acts as a gate: NFI scoring is applied only to outputs that pass the task's hand-written tests. This combination — a deterministic rule delta over a functionally validated output — is what turns 'non-functional improvement' from a subjective judgment into a reproducible benchmark number.

Load-bearing premise

The manually written unit tests for each of the 188 tasks are comprehensive enough that any output passing them is accepted as behavior-preserving, and the paper itself notes the tests may not cover all possible cases.

What would settle it

Collect the agent outputs that pass the benchmark's tests and earned positive NFI scores, then run additional differential or property-based tests on them against the original code (random inputs, boundary values, and the modified functions' edge cases). If a meaningful fraction of those outputs diverge, the NFI score is being awarded to code that changed observable behavior, which would undercut the benchmark's central measurement.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Correctness-only benchmarks give an incomplete picture; a 70% functional pass rate can coexist with systematic shortfalls in maintainability-oriented quality.
  • Logic-pattern improvements, the structural changes developers make, are the clearest frontier: no evaluated agent reached the human reference on that aspect.
  • Multi-file NFI tasks are harder than single-file ones, especially for coordinated error-handling and logic-pattern changes, so agent progress on localized edits does not transfer to repository-level maintenance.
  • Deterministic rule-based scoring enables reproducible comparison across agents and repeated runs, with aggregate NFI scores stable across five executions per task.
  • Higher compute and token budgets do not proportionally improve NFI quality, so cost must be evaluated alongside validity, correctness, and improvement.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the 92 rules are public and deterministic, an agent could be tuned to satisfy the rules without producing genuinely more maintainable code; a human-preference validation study would reveal whether high rule scores track perceived quality.
  • The benchmark's behavior-preservation guarantee is only as strong as its hand-written tests; adding differential testing against the original code on inputs beyond the test suite would test whether the measured NFI gap persists when behavior is verified more strictly.
  • The measured gap may be Python-specific; porting the rule framework to other languages with different lint and documentation cultures could shift both absolute scores and the size of the agent–human gap.
  • A direct extension would be to let agents see the rule checklist and measure how much scores rise, separating 'can follow explicit quality rules' from 'spontaneously improves structure the way developers do.'

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. SWE-NFI introduces a benchmark of 188 tasks constructed from merged pull requests in popular Python repositories, operationalizing five aspects of non-functional improvements into 92 executable rules and pairing each task with functional tests plus deterministic rule-based NFI evaluation. The paper evaluates eight coding-agent configurations (Codex, Claude Code Agent, and Aider and mini-SWE-agent with three backbone models), repeating each configuration five times, and compares agent NFI improvement scores with human-reference improvements computed from the original PRs. The headline findings are that agents lag human developers in NFI capability, especially on logic patterns; multi-file tasks are more challenging; NFI scores are relatively stable across repeated runs; and higher computational cost does not yield proportionally better NFI outcomes.

Significance. The benchmark is valuable and largely well constructed: tasks are derived from real merged PRs, the rule set is grounded in multiple sources (prior studies, GitHub issues, and Python guidelines), human validation used two independent annotators with reported inter-rater agreement, and evaluation is deterministic rather than LLM-as-judge. The human reference implementation is never exposed to agents, and the replication package is promised. If the headline result is supported, the paper provides a reproducible foundation for measuring a real gap in coding agents' ability to make behavior-preserving quality improvements. The main obstacle is that the central human-agent comparison is computed over non-identical task sets, which must be fixed before the headline claim is fully supported.

major comments (2)
  1. [§4.3, Table 3] The central comparison of agent NFI with the human reference is not like-for-like. The protocol restricts agent NFI scores to valid and functionally correct runs and removes subsets with no eligible outputs, whereas the Human row is averaged over all 188 tasks because the reference passes all tests. For the best agents, only 59–70% of runs are functionally correct, so each agent column is a conditional mean over an agent-dependent subset. If tasks that an agent fails have systematically higher or lower human NFI improvements, the reported logic-pattern gap (0.0–1.3 vs. 1.5) and the abstract's overall claim can be biased in either direction. No matched-subset comparison is reported. Please compute, for each agent and aspect, the human-reference improvement restricted to exactly the tasks in the agent's evaluated subset, and/or report paired per-task differences. For low-pass agents (e.g.,
  2. [§3.4, §7] The behavior-preservation gate is the manually written unit-test suite. The paper validates this suite only by executing it on the human reference, and it explicitly concedes in Section 7 that tests 'may not cover all possible cases for each task.' If a test suite misses behavior, an agent output that changes observable behavior can still be classified as functionally correct, and its NFI score would be computed on code whose behavior is not preserved. This directly affects the meaning of the NFI improvement metric in Eq. (2). Please provide quantitative evidence of test adequacy (e.g., line/branch coverage, mutation testing, or comparison with the original PR test suite), or analyze the likely direction of bias; at minimum, the benchmark release should document per-task coverage.
minor comments (5)
  1. [§3.2] There is a duplicated paragraph: 'Rules Summary' and the following 'NFI Rules Overview' repeat the same content. Please merge.
  2. [Table 3] No confidence intervals or statistical tests are reported for the aspect-level averages. Small differences such as Documentation 8.1 vs. 8.0 for Aider + GPT-4.1 are annotated as above the human reference without uncertainty. Add bootstrap confidence intervals or at least avoid overinterpreting point estimates.
  3. [Abstract, §5.1] The abstract's phrase 'overall NFI capability' is not directly reported as a single aggregate for both agents and humans; Table 3 is aspect-level only. Clarify what 'overall' means or report an explicit overall measure.
  4. [Figure 3] '21.05Documentationrule-score improvement' is missing spaces and should clarify the unit (percentage points).
  5. [§4.2] Clarify how multi-file tasks are presented to agents: is the repository context provided as a bundle of files, a patch, or individual paths? This affects reproducibility.

Circularity Check

0 steps flagged

No significant circularity: the benchmark's metric, task selection, and human reference are external to the agent results, and the central agent-vs-human gap is an empirical measurement, not a construction.

full rationale

I walked the paper's derivation chain. The NFI rule set is derived from external sources (PEP standards, prior studies, GitHub issues, coding guidelines), not from the benchmark's agent outputs or from the authors' own prior results. The improvement metric in Eq. (1)-(3) is a rule-satisfaction difference, with no fitted parameters; thresholds are adopted from public guidelines. Tasks are selected by requiring I_i,p(h) > 0, i.e., a real human PR improved under the rules, but this does not force any agent's score to be lower — the gap reported in Table 3 is an empirical outcome. The human reference is computed from merged PRs and is never exposed to agents. There is no uniqueness theorem, ansatz, or 'prediction' that reduces to a fit. The paper's self-citations (e.g., [10], [69], [80]) are methodological or refer to prior work by some co-authors but are not load-bearing for the central claim. The different-subset averaging noted in Sec. 4.3 (agents averaged over functionally correct outputs, human over all tasks) is a legitimate statistical-validity concern, but it is not a definitional circularity: it does not make the agent scores equal to the inputs by construction. Overall, the benchmark is self-contained and the central comparison is not forced.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The benchmark's conclusions rest on the validity of its rule set, thresholds, and test suites rather than on free parameters fitted to data. The main choices are the 21 threshold values and the five-aspect taxonomy, neither of which is independently validated against human quality judgments. The I_p metric itself is a defined quantity, not an invented entity.

free parameters (1)
  • Threshold values for 21 threshold-based NFI rules = e.g., line length ≤100, cyclomatic complexity ≤10, docstring coverage ≥80%, nesting depth ≤3, exception handlers ≤3, mag
    These thresholds determine whether a rule is satisfied and therefore directly set I_p scores. They are chosen by the authors from coding guidelines rather than fitted, but alternative thresholds could change agent/human rankings.
axioms (4)
  • domain assumption The five NFI aspects from SWE-IF (documentation, error handling, library constraints, coding style, logic patterns) are a representative and useful taxonomy of developer-oriented non-functional improvements.
    Section 3.2: the entire benchmark is organized around these five aspects. If this taxonomy is incomplete, the benchmark misses relevant NFIs.
  • domain assumption Rule-based static analysis (Ruff, Radon, AST, custom scripts) can measure non-functional improvement without human judgment or LLM-as-a-judge, and higher rule scores correspond to better code quality.
    Section 3.2/3.4: the central metric I_p is defined as a difference in rule-satisfaction counts; no evidence correlates rule scores with human quality ratings.
  • domain assumption The hand-written functional tests for each task are sufficient to verify that the revised code preserves the intended observable behavior of the original code.
    Section 3.4: tests are manually implemented and validated on the human reference; Section 7 admits they may not cover all cases, making behavior-preservation only as strong as the tests.
  • domain assumption The merged pull request (the human reference code_after) represents a 'good' non-functional improvement that agents should be compared against.
    Used throughout Section 5 as the human reference baseline. A PR author's change is not necessarily optimal, so 'falling short of human' is relative to a possibly average human change.

pith-pipeline@v1.3.0-daily-deepseek · 20603 in / 15217 out tokens · 145554 ms · 2026-08-01T07:50:08.685435+00:00 · methodology

0 comments
read the original abstract

Although coding agents have achieved impressive performance on correctness-oriented benchmarks, their ability to make behavior-preserving non-functional improvements (NFIs) remains underexplored. In real-world software development, developers continuously improve software quality without changing observable behavior, yet existing benchmarks primarily evaluate functional correctness and provide limited support for assessing these non-functional improvements. In this paper, we present SWE-NFI, a benchmark for evaluating coding agents on NFIs beyond functional correctness. Our benchmark contains 188 tasks constructed from real merged pull requests in open-source Python projects. We operationalize developer-oriented NFIs into 92 executable rules and develop a comprehensive evaluation suite that combines functional correctness testing with rule-based NFI evaluation. We evaluate state-of-the-art commercial and open-source coding agents. Although the best-performing agent achieves a 70.0\% functional correctness rate, all evaluated agents generally fall short of human developers in overall NFI capability. The gap is particularly evident for structural code improvements, where agents' NFI scores range from 0.0 to 1.3, compared with 1.5 for the human reference. Our benchmark and findings provide a reproducible foundation for evaluating and advancing coding agents beyond functional correctness.

Figures

Figures reproduced from arXiv: 2607.27409 by Boyuan Chen, Haonan Zhang, He Yang Yuan, Junkai Chen, Pengyu Xue, Weiyi Shang, Xin Wang, Zhenhao Li, Zishuo Ding.

Figure 1
Figure 1. Figure 1: Overview of SWE-NFI. the reasoning ability of LLMs to retrieve code, localize relevant ele￾ments, and generate patches in an iterative manner. SWE-agent [74] proposes the idea of an Agent-Computer Interface, which trans￾forms common computer operations into agent-friendly instruc￾tions to enable effective iteration between action and feedback. Building on SWE-agent, Abramovich et al. [2] design interactive… view at source ↗
Figure 2
Figure 2. Figure 2: Execution time and token usage across agents [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of different Agent outputs 6 Discussion 6.1 Case Study [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

94 extracted references · 1 canonical work pages

  1. [1]

    Replication Package

    2026. Replication Package. https://figshare.com/s/f14692e227368f6c392b. Last accessed June 2026

    2026

  2. [2]

    Talor Abramovich, Meet Udeshi, Minghao Shao, Kilian Lieret, Haoran Xi, Kim- berly Milner, Sofija Jancheska, John Yang, Carlos E Jimenez, Farshad Khor- rami, Prashanth Krishnamurthy, Brendan Dolan-Gavitt, Muhammad Shafique, Karthik R Narasimhan, Ramesh Karri, and Ofir Press. 2025. EnIGMA: Inter- active Tools Substantially Assist LM Agents in Finding Securi...

    2025

  3. [3]

    Aider. 2026. Aider: An ai pair programming tool. https://aider.chat/

    2026

  4. [4]

    Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ra- manathan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Su...

    2023

  5. [5]

    Cai, Michael Terry, Quoc V

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXiv preprint arXiv:2108.07732(2021)

    Pith/arXiv arXiv 2021

  6. [6]

    Amiangshu Bosu, Michaela Greiler, and Christian Bird. 2015. Characteristics of Useful Code Reviews: An Empirical Study at Microsoft. InIEEE/ACM Working Conference on Mining Software Repositories (MSR)

    2015

  7. [7]

    Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2025. Repairagent: An autonomous, llm-based agent for program repair. In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 2188–2200

    2025

  8. [8]

    Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps- Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al . 2023. Multipl-e: A scalable and polyglot approach to benchmarking neural code generation.IEEE Transactions on Software Engineering 49, 7 (2023), 3675–3691

    2023

  9. [9]

    Junkai Chen, Xing Hu, Zhenhao Li, Cuiyun Gao, Xin Xia, and David Lo. 2024. Code search is all you need? improving code suggestions with code search. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE). 1–13. doi:10.1145/3597503.3639085

    arXiv 2024

  10. [10]

    Junkai Chen, Huihui Huang, Yunbo Lyu, Junwen An, Jieke Shi, Chengran Yang, Ting Zhang, Haoye Tian, Yikun Li, Zhenhao Li, Xin Zhou, Xing Hu, and David Lo. 2026. SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios. InProceedings of the 64th Annual Meeting of the Association for Computational L...

    2026

  11. [11]

    Junkai Chen, Zhenhao Li, Xing Hu, and Xin Xia. 2026. Nlperturbator: Studying the robustness of code llms to natural language variations.ACM Transactions on Software Engineering and Methodology35, 4 (2026), 1–20

    2026

  12. [12]

    Junkai Chen, Zhenhao Li, Qiheng Mao, Xing Hu, Kui Liu, and Xin Xia. 2025. Understanding practitioners’ expectations on clear code review comments. Proceedings of the ACM on Software Engineering2, ISSTA (2025), 1257–1279. doi:10.1145/3728931

    work page doi:10.1145/3728931 2025

  13. [13]

    Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. 2025. Reasoning Runtime Behavior of a Program with LLM: How Far Are We?. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering (ICSE)

    2025

  14. [14]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, et al. 2021. Evaluating Large Language Models Trained on Code.arXiv preprint arXiv:2107.03374(2021)

    Pith/arXiv arXiv 2021

  15. [15]

    Claude Code. 2026. Claude Code Docs. https://docs.anthropic.com/en/docs/ claude-code. Last Access: 2026

    2026

  16. [16]

    Deepseek. 2026. DeepSeek. https://www.deepseek.com/en/. Last Access: 2026

    2026

  17. [17]

    Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, et al. 2025. Swe-bench pro: Can ai agents solve long-horizon software engineering tasks?arXiv preprint arXiv:2509.16941(2025)

    Pith/arXiv arXiv 2025

  18. [18]

    Jingzhe Ding, Shengda Long, Changxin Pu, Huan Zhou, Hongwan Gao, Xiang Gao, Chao He, Yue Hou, Fei Hu, Zhaojian Li, et al . 2026. NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents. arXiv preprint arXiv:2512.12730(2026)

    arXiv 2026

  19. [19]

    Zishuo Ding, Yiming Tang, Yang Li, Heng Li, and Weiyi Shang. 2023. On the temporal relations between logging and code. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 843–854

    2023

  20. [20]

    Mingzhe Du, Luu A Tuan, Bin Ji, Qian Liu, and See-Kiong Ng. 2024. Mercury: A code efficiency benchmark for code large language models.Advances in Neural Information Processing Systems37 (2024), 16601–16622

    2024

  21. [21]

    Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2024. Evaluating large language models in class-level code generation. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13

    2024

  22. [22]

    Yongkang Du, Jen-tse Huang, Jieyu Zhao, and Lu Lin. 2025. Faircoder: Evaluating social bias of llms in code generation.arXiv preprint arXiv:2501.05396(2025)

    Pith/arXiv arXiv 2025

  23. [23]

    Ryo Fujii, Makoto Morishita, Kazuki Yano, and Jun Suzuki. 2026. TimeMachine- bench: A Benchmark for Evaluating Model Capabilities in Repository-Level Migration Tasks.arXiv preprint arXiv:2601.22597(2026)

    Pith/arXiv arXiv 2026

  24. [24]

    David Goodger and Rossum Guido. 2001. PEP 257 - Docstring Conventions. https://peps.python.org/pep-0257/

    2001

  25. [25]

    Google. 2026. Gemini API. https://ai.google.dev/gemini-api/docs/models. Last Access: 2026

    2026

  26. [26]

    Xinyi He, Qian Liu, Mingzhe Du, Lin Yan, Zhijie Fan, Yiming Huang, Zejian Yuan, and Zejun Ma. 2025. Swe-perf: Can language models optimize code performance on real-world repositories?arXiv preprint arXiv:2507.12415(2025)

    Pith/arXiv arXiv 2025

  27. [27]

    Ruida Hu, Xinchen Wang, Chao Peng, Cuiyun Gao, and David Lo. 2026. Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios.arXiv preprint arXiv:2604.06742(2026)

    Pith/arXiv arXiv 2026

  28. [28]

    Xing Hu, Feifei Niu, Junkai Chen, Xin Zhou, Junwei Zhang, Junda He, Xin Xia, and David Lo. 2025. Assessing and advancing benchmarks for evaluating large language models in software engineering tasks.ACM Transactions on Software Engineering and Methodology(2025)

    2025

  29. [29]

    Xing Hu, Xin Xia, et al. 2022. Practitioners’ Expectations on Automated Code Comment Generation. InIEEE/ACM International Conference on Software Engi- neering (ICSE)

    2022

  30. [30]

    Dong Huang, Yuhao Qing, Weiyi Shang, Heming Cui, and Jie M. Zhang. 2025. Ef- fibench: Benchmarking the efficiency of automatically generated code.Advances in Neural Information Processing Systems37 (2025), 11506–11544

    2025

  31. [31]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world GitHub Issues?arXiv preprint arXiv:2310.06770(2024)

    Pith/arXiv arXiv 2024

  32. [32]

    Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettle- moyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. DS-1000: A natural and reliable benchmark for data science code generation. InInternational Conference on Machine Learning. PMLR, 18319–18345

    2023

  33. [33]

    Lukasz Langa. 2019. PEP 585 – Type Hinting Generics In Standard Collections. https://peps.python.org/pep-0585/

    2019

  34. [34]

    Wei Li, Xin Zhang, Zhongxin Guo, Shaoguang Mao, Wen Luo, Guangyue Peng, Yangyu Huang, Houfeng Wang, and Scarlett Li. 2025. Fea-bench: A benchmark for evaluating repository-level code generation for feature implementation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 17160–17176

    2025

  35. [35]

    Zhenhao Li, An Ran Chen, Xing Hu, Xin Xia, Tse-Hsun Chen, and Weiyi Shang

  36. [36]

    Zhenhao Li, Tse-Hsun Chen, and Weiyi Shang. 2020. Where Shall We Log? Studying and Suggesting Logging Locations in Code Blocks. In35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020. 361–372. doi:10.1145/3324884.3416636

    arXiv 2020

  37. [37]

    Zhenhao Li, Tse-Hsun Chen, Jinqiu Yang, and Weiyi Shang. 2019. DLFinder: Characterizing and Detecting Duplicate Logging Code Smells. In2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). doi:10.1109/ICSE. 2019.00032

    arXiv 2019

  38. [38]

    Zhenhao Li, Heng Li, Tse-Hsun Chen, and Weiyi Shang. 2021. Deeplv: Sug- gesting log levels using ordinal based neural networks. In2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 1461–1472. doi:10.1109/ICSE43902.2021.00131

    arXiv 2021

  39. [39]

    Zhengquan Li, Zhenhao Li, and Zishuo Ding. 2025. From Feedback to Fail- ure: Automated Android Performance Issue Reproduction.arXiv preprint arXiv:2508.11147(2025)

    arXiv 2025

  40. [40]

    Zhenhao Li, Chuan Luo, Tse-Hsun Chen, Weiyi Shang, Shilin He, Qingwei Lin, and Dongmei Zhang. 2023. Did we miss something important? Studying and exploring variable-aware log abstraction. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 830–842

    2023

  41. [41]

    Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. 2025. Large language model-based agents for software en- gineering: A survey.ACM Transactions on Software Engineering and Methodology (2025)

    2025

  42. [42]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Lan- guage Models for Code Generation. InAdvances in Neural Information Processing Systems (NeurIPS)

    2023

  43. [43]

    Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang. 2024. Evaluating language models for efficient code generation.arXiv preprint arXiv:2408.06450(2024). ArXiv, 2026, Online Pengyu Xue, He Yang Yuan, Xin Wang, Junkai Chen, Haonan Zhang, Boyuan Chen, Zishuo Ding, Zhenhao Li, and Weiyi Shang

    Pith/arXiv arXiv 2024

  44. [44]

    Jeffrey Jian Ma, Milad Hashemi, Amir Yazdanbakhsh, Kevin Swersky, Ofir Press, Enhui Li, Vijay Janapa Reddi, and Parthasarathy Ranganathan. 2025. SWE- fficiency: Can Language Models Optimize Real-World Repositories on Real Work- loads?arXiv preprint arXiv:2511.06090(2025)

    Pith/arXiv arXiv 2025

  45. [45]

    Qiheng Mao, Zhenhao Li, Xing Hu, Kui Liu, Xin Xia, and Jianling Sun. 2025. Towards explainable vulnerability detection with large language models.IEEE Transactions on Software Engineering(2025)

    2025

  46. [46]

    Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E. Hassan. 2016. An Empirical Study of the Impact of Modern Code Review Practices on Software Quality. InEmpirical Software Engineering, Vol. 21. 2146–2189

    2016

  47. [47]

    Tam Nguyen, Phong Vu, and Tung Nguyen. 2020. Code Recommendation for Exception Handling. InACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE)

    2020

  48. [48]

    OpenAI. 2025. Codex CLI: A Lightweight Coding Agent That Runs in Your Terminal. https://github.com/openai/codex. Accessed: 2026-06-28

    2025

  49. [49]

    OpenAI. 2026. OpenAI Developer. https://developers.openai.com/api/docs/ models. Last Access: 2026

    2026

  50. [50]

    Khouloud Oueslati, Maxime Lamothe, and Foutse Khomh. 2026. RefAgent: A Multi-agent LLM-based Framework for Automatic Software Refactoring. In IEEE/ACM International Conference on Software Engineering (ICSE)

    2026

  51. [51]

    Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2025. Asleep at the keyboard? assessing the security of github copilot’s code contributions.Commun. ACM68, 2 (2025), 96–105

    2025

  52. [52]

    Norman Peitek, Sven Apel, Chris Parnin, André Brechmann, and Janet Siegmund

  53. [53]

    Jinjun Peng, Leyi Cui, Kele Huang, Junfeng Yang, and Baishakhi Ray. 2025. Cwe- val: Outcome-driven evaluation on functionality and security of llm code gener- ation. In2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code). IEEE, 33–40

    2025

  54. [54]

    Yun Peng, Jun Wan, Yichen Li, and Xiaoxue Ren. 2025. Coffe: A code efficiency benchmark for code generation.Proceedings of the ACM on Software Engineering 2, FSE (2025), 242–265

    2025

  55. [55]

    Veronica Pimenova, Sarah Fakhoury, Christian Bird, Margaret-Anne Storey, and Madeline Endres. 2025. Good Vibrations? A Qualitative Study of Co-Creation, Communication, Flow, and Trust in Vibe Coding. arXiv:2509.12491 [cs.SE]

    Pith/arXiv arXiv 2025

  56. [56]

    Yuhao Qing, Boyu Zhu, Mingzhe Du, Zhijiang Guo, Terry Yue Zhuo, Qianru Zhang, Jie Zhang, Heming Cui, Siu Ming Yiu, Dong Huang, et al. 2026. Effibench- x: A multi-language benchmark for measuring efficiency of llm-generated code. Advances in Neural Information Processing Systems38 (2026)

    2026

  57. [57]

    Muhammad Shihab Rashid, Christian Bock, Yuan Zhuang, Alexander Buch- holz, Tim Esler, Simon Valentin, Luca Franceschi, Martin Wistuba, Prabhu Teja Sivaprasad, Woo Jung Kim, et al . 2026. Swe-polybench: A multi-language benchmark for repository level evaluation of coding agents.arXiv preprint arXiv:2504.08703(2026)

    Pith/arXiv arXiv 2026

  58. [58]

    Guido Rossum, Jukka Lehtosalo, and Lukasz Langa. 2001. PEP 484 - Type Hints. https://peps.python.org/pep-0484/

    2001

  59. [59]

    SciTools. 2026. Understanding McCabe Cyclomatic Complexity. https://support.scitools.com/support/solutions/articles/70000582297- understanding-mccabe-cyclomatic-complexity

    arXiv 2026

  60. [60]

    Chihao Shen, Connor Dilgren, Purva Chiniya, Luke Griffith, Yu Ding, and Yizheng Chen. 2026. SecRepoBench: Benchmarking Code Agents for Secure Code Com- pletion in Real-World Repositories.arXiv preprint arXiv:2504.21205(2026)

    arXiv 2026

  61. [61]

    Xinyu Shi, Zhenhao Li, and An Ran Chen. 2025. Enhancing LLM-based Fault Localization with a Functionality-Aware Retrieval-Augmented Generation Frame- work.arXiv preprint arXiv:2509.20552(2025)

    arXiv 2025

  62. [62]

    Anthony Sottile. 2021. pyupgrade Documentation. https://github.com/asottile/ pyupgrade. Last Access: 2026

    2021

  63. [63]

    SWE-agent Team. 2026. mini-SWE-Agent. https://mini-swe-agent.com/latest/

    2026

  64. [64]

    Tufts University. 2026. CS 40 Coding Standards. https://www.cs.tufts.edu/comp/ 40/docs/coding-standards.html

    2026

  65. [65]

    Guido Van Rossum, Barry Warsaw, and Nick Coghlan. 2001. PEP 8 - Style Guide for Python Code. https://peps.python.org/pep-0008/

    2001

  66. [66]

    Chong Wang, Kaifeng Huang, Jian Zhang, Yebo Feng, Lyuye Zhang, Yang Liu, and Xin Peng. 2025. LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-based Code Completion. InIEEE/ACM International Conference on Software Engineering (ICSE). arXiv:2406.09834

    Pith/arXiv arXiv 2025

  67. [67]

    Xin Wang, Yang Feng, Jiaoxiao Qian, Yang Zhang, Zhenhao Li, and Zishuo Ding

  68. [68]

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. 2025. Openhands: An open platform for ai software developers as generalist agents. InInternational Conference on Learning Representations, Vol. 2025. 65882–65919

    2025

  69. [69]

    Xin Wang, Zhenhao Li, and Zishuo Ding. 2025. Defects4Log: Benchmarking LLMs for Logging Code Defect Detection and Reasoning. In40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025. IEEE, 1931–1942. doi:10.1109/ASE63991.2025.00161

    arXiv 2025

  70. [70]

    Xin Wang, Zhenhao Li, and Zishuo Ding. 2026. LLM4Perf: Large Language Models Are Effective Samplers for Multi-Objective Performance Modeling. In Proceedings of the IEEE/ACM 48th International Conference on Software Engineering (ICSE)

    2026

  71. [71]

    Tongtong Wu, Weigang Wu, Xingyu Wang, Kang Xu, Suyu Ma, Bo Jiang, Ping Yang, Zhenchang Xing, Yuan-Fang Li, and Gholamreza Haffari. 2024. Versicode: Towards version-controllable code generation.arXiv preprint arXiv:2406.07411 (2024)

    Pith/arXiv arXiv 2024

  72. [72]

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agent- less: Demystifying LLM-based Software Engineering Agents.arXiv preprint arXiv:2407.01489(2024)

    Pith/arXiv arXiv 2024

  73. [73]

    Weiwei Xu, Kai Gao, Hao He, and Minghui Zhou. 2025. Licoeval: Evaluating llms on license compliance in code generation. In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 1665–1677

    2025

  74. [74]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer Inter- faces Enable Automated Software Engineering. InAdvances in Neural Information Processing Systems (NeurIPS)

    2024

  75. [75]

    John Yang, Carlos E Jimenez, Alex L Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R Narasimhan, et al

  76. [76]

    John Yang, Kilian Lieret, Jeffrey Ma, Parth Thakkar, Dmitrii Pedchenko, Sten Sootla, Emily McMilin, Pengcheng Yin, Rui Hou, Gabriel Synnaeve, et al. 2026. ProgramBench: Can Language Models Rebuild Programs From Scratch?arXiv preprint arXiv:2605.03546(2026)

    Pith/arXiv arXiv 2026

  77. [77]

    Lanxin Yang, Jinwei Xu, Yifan Zhang, He Zhang, and Alberto Bacchelli. 2023. Evacrc: Evaluating code review comments. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 275–287

    2023

  78. [78]

    Ka-Ping Yee. 2005. PEP 3134 – Exception Chaining and Embedded Tracebacks. https://peps.python.org/pep-3134/

    2005

  79. [79]

    Boxi Yu, Yuxuan Zhu, Pinjia He, and Daniel Kang. 2025. Utboost: Rigorous evaluation of coding agents on swe-bench. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 3762–3774

    2025

  80. [80]

    He Yang Yuan, Xin Wang, Kundi Yao, An Ran Chen, Zishuo Ding, and Zhenhao Li. 2026. Towards Secure Logging: Characterizing and Benchmarking Logging Code Security Issues with LLMs.Proceedings of the ACM on Software Engineering 3, FSE (2026), 745–765

    2026

Showing first 80 references.