Pith. sign in

REVIEW 3 major objections 5 minor 34 references

Prefactory claims that LLM-synthesized executable detectors can find and apply library-adoption refactorings at scale, detecting 75 of 100 real-world opportunities and validating 40 rewrites.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 18:40 UTC pith:G42CEDKY

load-bearing objection Prefactory's detector-synthesis idea is genuinely new and beats agentic search on detection; the '40 validated refactorings' number is softer than it looks because the tests share the model's blind spots. the 3 major comments →

arxiv 2607.17211 v1 pith:G42CEDKY submitted 2026-07-19 cs.SE

Prefactory: Automated Discovery and Application of Library-Adoption Refactorings

Islem Bouzenia , Michael Pradel This is my paper
classification cs.SE
keywords library-adoption refactoringdetector synthesisLLM-generated detectorsPython refactoringdifferential testingAST matchinglexical searchPrefactoryBench
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper attempts to establish that library-adoption refactoring—replacing hand-written Python code with calls to existing libraries—can be automated by converting an LLM's semantic knowledge into reusable executable detectors. Prefactory generates lexical regular expressions and AST-based structural matchers for a target library, runs them over a project without any LLM calls to find candidate functions, then asks an LLM to rewrite only the top-ranked candidates and validates each edit with differential tests and project tests. On a benchmark of 100 real developer refactorings, the approach detects the correct file in 75 cases and the correct function in 56, and produces 40 test-validated rewrites, outperforming all baselines. The reframing—detector synthesis rather than repeated prompting—makes the search cheap, deterministic, and reproducible, which is what would be needed to apply such refactorings systematically across many projects.

Core claim

Prefactory's central claim is that the hard part of library-adoption refactoring is detection, and that detection can be solved by synthesizing executable search heuristics once per target library rather than by repeatedly querying an LLM over a codebase. The paper shows that lexical detectors (regular expressions over source text) and structural detectors (AST matchers) are complementary: each alone finds roughly 30 of 100 reference functions, and together they find 56. After ranking and a bounded LLM rewrite step, 40 rewrites survive validation against differential tests and the project test suite. The paper interprets the gap between the 75 detected files and the 40 validated refactorings

What carries the argument

The load-bearing mechanism is detector synthesis: an LLM is prompted once per project-library pair with library metadata, usage examples, and mined refactoring hints to emit two families of executable search programs—regular expressions and AST matchers—which are filtered for syntax, over-broad matching, and positive/negative examples. These detectors are then run deterministically over the project in a scan phase that requires no LLM calls, and the ranked output feeds a second LLM stage that rewrites the top candidates and validates them with differential tests (original implementation as oracle) plus project tests. The detector synthesis is what converts the LLM's semantic knowledge into a

Load-bearing premise

The central claim rests on the assumption that the tests used for validation—differential tests generated by the same model family that writes the refactorings, plus project tests that cover only a median of 4% of the refactored region—are sufficient to certify that a rewrite preserves behavior on untested inputs.

What would settle it

Run the 40 validated refactorings against a held-out test suite written by a different model family or by human experts, or apply property-based random testing with an independent oracle; if any rewritten function diverges from the original on inputs outside Prefactory's generated differential tests, the validation claim is weakened. A simpler check: re-run Prefactory with differential tests generated by a different model and count how many of the 40 survive.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Library-adoption refactoring can be automated end-to-end at low cost: after one-time detector generation, scanning a project takes about 1.3 seconds and no LLM calls, and end-to-end per-instance cost stays below $0.05.
  • Detection improves over rule-based tools and over direct LLM/agentic search by a wide margin—75 vs 11 file-level detections for linters and 75 vs 35 for the strongest agentic baseline—while using the same underlying model as that baseline.
  • Because acceptance requires passing differential and project tests, the pipeline rejects rewrites that would drop project-specific behavior, as demonstrated in a distributed-training example where a tempting API replacement was refused.
  • The two detector families are complementary: combining them nearly doubles function-level detection compared to either alone, suggesting that both lexical cues and code-shape cues carry independent evidence.
  • The approach surfaces actionable refactorings on current project versions: two of five pull requests submitted to open-source maintainers were accepted, including a 17-line recursive Cartesian-product routine replaced by a single library call.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural next step, which the paper does not pursue, is to generate detectors per API family or submodule for broad libraries like numpy and pandas; the paper's own failure analysis shows that treating such libraries as a single target accounts for many missed detections.
  • The same detector-synthesis pattern may transfer to other code-search tasks where the target is defined semantically rather than syntactically—for example, finding dead code, insecure patterns, or non-idiomatic constructs—since the bottleneck is turning LLM knowledge into a deterministic, reusable scan.
  • The validation pipeline is the place where the strongest measurement assumption lives: the differential tests are written by the same model family that writes the rewrites, and project tests cover only a median of 4% of the refactored region, so an independent oracle would sharpen the count of truly behavior-preserving refactorings.
  • The fixed ranking weights (2:1 structural-to-lexical) and budget (top five files, top ten functions) are engineering choices; the paper's sensitivity analysis shows recall saturates around these budgets, but a learned ranker or adaptive budget might recover some of the 35 instances lost to ranking.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Prefactory, a four-stage pipeline for library-adoption refactoring in Python: (1) an LLM synthesizes lexical and structural detectors from target-library metadata and project vocabulary; (2) these detectors scan the project deterministically with no LLM calls; (3) candidates are ranked and filtered under fixed budgets; (4) an LLM rewrites the top functions and the edits are validated with LLM-generated differential tests followed by project tests. The approach is evaluated on PrefactoryBench, a new benchmark of 100 real-world refactorings from 61 projects and 18 libraries. The headline results are 75 file-level and 56 function-level detections versus 35 and 32 for Codex CLI, the strongest baseline, and 40 test-validated refactorings. The paper also reports low scanning cost, an ablation study, and an open-source study with two accepted pull requests.

Significance. The central idea of converting LLM semantic knowledge into reusable executable detectors that scan a repository without repeated LLM prompting is novel and well motivated. The benchmark is large, manually checked, and paired with execution environments, which is a substantial resource for the community. The evaluation is internally consistent: it includes multiple baselines, ablations, cost measurements, failure analysis, and an external pull-request study, with two acceptances providing real-world grounding. The paper is generally well written and the pipeline is described in sufficient detail to replicate. If the validation concerns below are addressed, this would be a strong contribution to automated refactoring and to the broader study of LLM-based software engineering tasks.

major comments (3)
  1. [III-A3, Table I, Table II] The 'validated refactoring' metric in Section III-A3 requires passing the differential test, the project tests, and the benchmark's LLM-generated held-out tests. This is the basis for the headline claim of '40 test-validated refactorings.' However, Table I reports median project-test line coverage of the refactored region of only 4% (average 3%), and the held-out tests are generated with gpt-5-mini, the same model family used to generate the refactorings (Section III-A4). The paper itself states in Section IV that 'tests cannot prove semantic equivalence.' The nine refactorings that pass differential tests but fail project tests (Table III) show that project tests catch real issues, but they do not establish that the 40 accepted rewrites preserve behavior on untested inputs. This does not invalidate the detection claim (75/56), but it is load-bearing for the transformation claim. I recom
  2. [III-B1, Fig. 5] Figure 5 shows that without repair only 21 of 49 proposed refactorings pass the differential test; after up to three repair attempts all 49 pass. The repair loop feeds failing inputs back to the same gpt-5-mini model that wrote both the refactoring and the differential tests. There is a risk of overfitting to the model's own test generator rather than to behavioral equivalence: a repaired edit is acceptable if it satisfies tests that share the model's blind spots. The paper should report how many of the 49 pass without repair, whether repaired edits were independently reviewed, and ideally evaluate the repaired set with an oracle outside the gpt family. At minimum, the text should maintain the distinction between 'passes generated tests' and 'preserves behavior' throughout the abstract, results, and conclusion.
  3. [III-A1, III-B5] Section III-A1 describes mining PrefactoryBench from public GitHub commit histories from 2016 to 2026. Section III-B5 acknowledges that 'historical commits may appear in model training data' only in the context of the open-source study. The same concern applies to PrefactoryBench itself: if gpt-5-mini has seen the benchmark instances, the absolute detection and validation rates (75/56/40) may be inflated. The comparison against Codex CLI remains informative because the same model is used for both, but the paper should either establish that the benchmark instances postdate the model's training cutoff, test on a set of held-out recent commits, or explicitly state this as a threat to validity for the absolute numbers.
minor comments (5)
  1. [II-A] The detector-generation prompts are said to be included in the supplementary material, but the paper does not provide an artifact URL or an availability statement. Please add one for reproducibility.
  2. [Table VI] The 'Gen. cost' column uses a relative scale ('1x', '2x') without defining the base. State the base cost or remove the column.
  3. [III-A2(a)] The scoring description for linters is unclear: 'function-level detection is scored by mapping each matched line to its enclosing function' - linters typically report lines rather than functions, so please clarify the mapping procedure.
  4. [III-B2] The sentence 'none of its 23 function-level detections translate to validated refactorings because the approach has no refactoring phase' is confusing, since Semgrep+LLM is evaluated for detection only. Rephrase to avoid implying a missing phase in a detection-only baseline.
  5. [II-C] The ranking tuple uses -S(c), so smaller candidates are preferred in lexicographic comparison. This is fine, but the text could explain the minus sign in one sentence for readers unfamiliar with lexicographic ordering.

Circularity Check

0 steps flagged

No definitional circularity: detection and validation are empirically grounded against external baselines and real commits; the main weakness is a disclosed LLM-generated-test blind-spot threat, not a circular derivation.

full rationale

The paper's central claims are empirical rather than definitional. Detectors are synthesized from library metadata, project vocabulary, and mined examples drawn from projects that the paper explicitly states do not overlap with the benchmark projects (Section II-A: 'These examples are drawn from projects that do not overlap with the benchmark projects.'). Benchmark labels are not used to fit the detectors or the fixed ranking weights, and detection is scored against external baselines on held-out commits. The self-citation [31] appears only in related work as an example of an LLM agent that executes tests; it is not load-bearing for the main derivation. The most serious validity concern is that Prefactory's internal differential tests and the benchmark's held-out tests are LLM-generated and share the model family with the refactoring generator, so they could share blind spots. The paper acknowledges this directly in Section IV: 'tests cannot prove semantic equivalence,' and reports median project-test coverage of only 4% of the refactored region (Table I). This is a measurement assumption about behavioral preservation, not a circular step in the detection or generation pipeline. The two accepted pull requests in RQ5 provide independent external grounding. No equation-level or definitional reduction was found, so the circularity score is low.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

No new physical or conceptual entities are postulated. The free parameters are engineering choices (weights, budgets, thresholds) that affect the reported numbers; the axioms are assumptions about the benchmark's validity and the reliability of LLM-generated test oracles.

free parameters (5)
  • Ranking weights (structural:lexical = 2:1) = 2:1
    Chosen by the authors to favor structural detectors; fixed across experiments but not fitted. Section III-A (detector-match strength D(c)).
  • Ranking budget: top-5 files / top-10 functions = 5 files, 10 functions
    Fixed budget to bound LLM cost; ablation in RQ4 shows recall as a function of budget, indicating the choice affects results.
  • Lexical detector noise threshold = 20% of project files
    Detectors matching >20% of files are discarded; threshold chosen by hand in Section II-A.
  • Repair budget = up to 3 additional attempts
    LLM repair attempts capped at 3 in Stage 4; ablation shows monotonic improvement in differential-test pass rate.
  • API hint cap = 100 APIs
    If target library exposes >100 importable functions/classes, prompt selects up to 100; Section II-D.
axioms (4)
  • domain assumption The original implementation is a correct behavioral oracle for differential tests.
    Section II-D uses original function as oracle; if the original has bugs, refactorings preserving the bug are accepted. Also Section IV acknowledges tests cannot prove equivalence.
  • domain assumption LLM-generated differential tests and held-out tests exercise behavior beyond the 3-4% median project-test coverage.
    Benchmark relies on held-out tests to raise coverage to 94% average (Section III-A1), but these tests are generated by the same model family as the refactorer.
  • domain assumption PrefactoryBench instances are genuine, representative library-adoption refactorings.
    Instances are mined and manually audited, but selection criteria (projects ≥100 stars, 2016-2026 commits, capped at 500 recent commits) may bias toward certain patterns. Section III-A1.
  • domain assumption Single-run evaluation with gpt-5-mini is representative.
    Codex CLI runs once with default temperature; no confidence intervals or multi-seed analysis (Section III-A4).

pith-pipeline@v1.3.0-alltime-deepseek · 15904 in / 11402 out tokens · 94858 ms · 2026-08-01T18:40:16.737023+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Prefactory: Automated Discovery and Application of Library-Adoption Refactorings." pith.science (2026). https://pith.science/paper/G42CEDKY

@misc{pith2026260717211,
  author       = {Pith},
  title        = {Pith review of: Prefactory: Automated Discovery and Application of Library-Adoption Refactorings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G42CEDKY}},
  note         = {Machine review of arXiv:2607.17211}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Replacing hand-written code with library API calls is a common refactoring that can reduce code size, make code more idiomatic, and reuse well-tested implementations. Yet many library-adoption opportunities are hard to find automatically: the original code often does not mention the target library and may resemble the library API only in behavior, with little syntactic overlap. Existing tools, such as linters and static modernizers, cover only a small set of manually specified patterns. LLMs and LLM-based agents, on the other hand, can generalize to more patterns, but they are costly, difficult to reproduce and to apply systematically at scale. This paper introduces Prefactory, an automated approach for library-adoption refactoring in Python. The key idea is to use an LLM to synthesize executable search heuristics rather than relying on repeated LLM prompting over a codebase. Given a target project and a target library name, Prefactory collects library metadata and project vocabulary, then generates lexical and structural detectors. Prefactory executes the detectors during a scan phase to find candidate functions. It then heuristically ranks the candidate functions, generates refactorings for the highest-ranked ones using an LLM, and validates the results with project tests and newly generated differential tests. We evaluate Prefactory on PrefactoryBench, a benchmark of 100 real-world library-adoption refactorings from 61 open-source Python projects and 18 libraries. Prefactory detects 75 instances at the file level and 56 at the function level, compared with 35 and 32 for the strongest baseline (Codex CLI). From the 56 detected functions, Prefactory produces 40 test-validated refactorings.

Figures

Figures reproduced from arXiv: 2607.17211 by Islem Bouzenia, Michael Pradel.

Figure 1
Figure 1. Figure 1: Motivating library-adoption refactoring from [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the Prefactory four-stage pipeline. Stages 1 and 4 invoke an LLM; Stages 2 and 3 are deterministic and require no LLM calls. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Detector examples. Top: a lexical detector for [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Representative failure examples. happen when other functions in the same file are plausible library-adoption opportunities and rank above the benchmark target. In a manual inspection of 10 higher-ranked non￾reference candidates sampled from ranking failures, six were genuine library-adoption opportunities, just not the developer￾refactored instances recorded in the benchmark. The remaining failures happen … view at source ↗
Figure 5
Figure 5. Figure 5: Effect of repair budget on differential-test outcomes (out of 49 [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Accepted pull request: recursive Cartesian-product enumeration [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 1 linked inside Pith

  1. [1]

    Ruff: An extremely fast python linter and code formatter, written in Rust,

    C. Marsh and Astral Software Inc., “Ruff: An extremely fast python linter and code formatter, written in Rust,” https://github.com/astral-sh/ ruff, 2022, accessed 2026-06-27

    2022

  2. [2]

    refurb: A tool for refurbishing and modernizing python code- bases,

    dosisod, “refurb: A tool for refurbishing and modernizing python code- bases,” https://github.com/dosisod/refurb, 2022, accessed 2026-06-27

    2022

  3. [3]

    Pylint: A static code analyser for Python,

    Pylint contributors, “Pylint: A static code analyser for Python,” https: //github.com/pylint-dev/pylint, 2003, accessed 2026-06-27

    2003

  4. [4]

    Don’t reinvent the wheel: Towards automatic replacement of custom implementations with APIs,

    R. Tufano, E. Aghajani, and G. Bavota, “Don’t reinvent the wheel: Towards automatic replacement of custom implementations with APIs,” in2022 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2022, pp. 394–398

    2022

  5. [5]

    From custom logic to APIs: Understanding and recommending API replacement refactorings,

    B. Nyirongo, Y . Jiang, Y . Zhang, and H. Liu, “From custom logic to APIs: Understanding and recommending API replacement refactorings,” 2026

    2026

  6. [6]

    Differential testing for software,

    W. M. McKeeman, “Differential testing for software,”Digital Technical Journal, vol. 10, no. 1, pp. 100–107, 1998

    1998

  7. [7]

    The oracle problem in software testing: A survey,

    E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The oracle problem in software testing: A survey,”IEEE Transactions on Software Engineering, vol. 41, no. 5, pp. 507–525, 2015

    2015

  8. [8]

    Conversational automated program repair,

    C. S. Xia and L. Zhang, “Conversational automated program repair,” arXiv preprint arXiv:2301.13246, 2023

    Pith/arXiv arXiv 2023

  9. [9]

    Making python code idiomatic by automatic refactoring non-idiomatic python code with pythonic idioms,

    Z. Zhang, Z. Xing, X. Xia, X. Xu, and L. Zhu, “Making python code idiomatic by automatic refactoring non-idiomatic python code with pythonic idioms,” inProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2022. ACM, 2022, pp. 696–708

    2022

  10. [10]

    Swe-refactor: A repository-level benchmark for real-world llm-based code refactoring,

    Y . Xu, J. Yang, and T.-H. Chen, “Swe-refactor: A repository-level benchmark for real-world llm-based code refactoring,”arXiv preprint arXiv:2602.03712, 2026

    arXiv 2026

  11. [11]

    PyDriller: Python frame- work for mining software repositories,

    D. Spadini, M. F. Aniche, and A. Bacchelli, “PyDriller: Python frame- work for mining software repositories,” inProceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), 2018, pp. 908–911

    2018

  12. [12]

    Semgrep: Lightweight static analysis for many lan- guages,

    Semgrep, Inc., “Semgrep: Lightweight static analysis for many lan- guages,” https://github.com/semgrep/semgrep, 2020

    2020

  13. [13]

    Accurate and efficient refactoring detection in commit history,

    N. Tsantalis, M. Mansouri, L. M. Eshkevari, D. Mazinanian, and D. Dig, “Accurate and efficient refactoring detection in commit history,” inPro- ceedings of the 40th International Conference on Software Engineering (ICSE), 2018, pp. 483–494

    2018

  14. [14]

    RefactoringMiner 2.0,

    N. Tsantalis, A. Ketkar, and D. Dig, “RefactoringMiner 2.0,”IEEE Transactions on Software Engineering, vol. 48, no. 3, pp. 930–950, 2022

    2022

  15. [15]

    Fowler,Refactoring: Improving the Design of Existing Code, 2nd ed

    M. Fowler,Refactoring: Improving the Design of Existing Code, 2nd ed. Boston, MA: Addison-Wesley Professional, 2018

    2018

  16. [16]

    Why we refactor? confessions of GitHub contributors,

    D. Silva, N. Tsantalis, and M. T. Valente, “Why we refactor? confessions of GitHub contributors,” inProceedings of the 24th ACM SIGSOFT International Symposium on the Foundations of Software Engineering (FSE), 2016, pp. 858–870

    2016

  17. [17]

    How we refactor, and how we know it,

    E. R. Murphy-Hill, C. Parnin, and A. P. Black, “How we refactor, and how we know it,”IEEE Transactions on Software Engineering, vol. 38, no. 1, pp. 5–18, 2012

    2012

  18. [18]

    Mining library migration graphs,

    C. Teyton, J.-R. Falleri, and X. Blanc, “Mining library migration graphs,” in19th Working Conference on Reverse Engineering (WCRE). IEEE Computer Society, 2012, pp. 289–298

    2012

  19. [19]

    A3: Assisting Android API migrations using code examples,

    M. Lamothe, W. Shang, and T.-H. P. Chen, “A3: Assisting Android API migrations using code examples,”IEEE Transactions on Software Engineering, vol. 48, no. 2, pp. 417–431, 2022

    2022

  20. [20]

    MAPO: Mining and recommending API usage patterns,

    H. Zhong, T. Xie, L. Zhang, J. Pei, and H. Mei, “MAPO: Mining and recommending API usage patterns,” inEuropean Conference on Object- Oriented Programming (ECOOP), ser. Lecture Notes in Computer Science, vol. 5653. Springer, 2009, pp. 318–343

    2009

  21. [21]

    Deep API learning,

    X. Gu, H. Zhang, D. Zhang, and S. Kim, “Deep API learning,” in Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE). ACM, 2016, pp. 631– 642

    2016

  22. [22]

    When LLMs meet API documentation: Can retrieval augmentation aid code generation just as it helps developers?

    J. Chen, S. Chen, J. Cao, J. Shen, and S.-C. Cheung, “When LLMs meet API documentation: Can retrieval augmentation aid code generation just as it helps developers?” 2025

    2025

  23. [23]

    pyupgrade: A tool to automatically upgrade syntax for newer versions of the language,

    A. Sottile, “pyupgrade: A tool to automatically upgrade syntax for newer versions of the language,” https://github.com/asottile/pyupgrade, 2017, accessed 2026-06-27

    2017

  24. [24]

    flake8: Your tool for style guide enforcement,

    PyCQA, “flake8: Your tool for style guide enforcement,” https://github. com/PyCQA/flake8, 2010, accessed 2026-06-27

    2010

  25. [25]

    Making Python code idiomatic by automatic refactoring non-idiomatic Python code with Pythonic idioms,

    Z. Zhang, Z. Xing, X. Xia, X. Xu, and L. Zhu, “Making Python code idiomatic by automatic refactoring non-idiomatic Python code with Pythonic idioms,” 2022

    2022

  26. [26]

    Documenting and automating collateral evolutions in Linux device drivers,

    Y . Padioleau, J. Lawall, R. R. Hansen, and G. Muller, “Documenting and automating collateral evolutions in Linux device drivers,” inProceedings of the 3rd ACM SIGOPS/EuroSys European Conference on Computer Systems (EuroSys). ACM, 2008, pp. 247–260

    2008

  27. [27]

    Tricorder: Building a program analysis ecosystem,

    C. Sadowski, J. van Gogh, C. Jaspan, E. Söderberg, and C. Winter, “Tricorder: Building a program analysis ecosystem,” in37th IEEE/ACM International Conference on Software Engineering (ICSE), Volume 1. IEEE Computer Society, 2015, pp. 598–608

    2015

  28. [28]

    Lessons from building static analysis tools at Google,

    C. Sadowski, E. Aftandilian, A. Eagle, L. Miller-Cushon, and C. Jaspan, “Lessons from building static analysis tools at Google,”Communications of the ACM, vol. 61, no. 4, pp. 58–66, 2018

    2018

  29. [29]

    SWE-agent: Agent-computer interfaces enable automated software engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “SWE-agent: Agent-computer interfaces enable automated software engineering,” inAdvances in Neural Information Processing Systems (NeurIPS), 2024. [Online]. Available: https://openreview.net/ forum?id=mXpq6ut8J3

    2024

  30. [30]

    SWE-bench: Can language models resolve real-world GitHub issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “SWE-bench: Can language models resolve real-world GitHub issues?” inThe Twelfth International Conference on Learning Representations (ICLR), 2024. [Online]. Available: https://openreview.net/forum?id=VTF8yNQM66

    2024

  31. [31]

    You name it, i run it: An llm agent to execute tests of arbitrary projects,

    I. Bouzenia and M. Pradel, “You name it, i run it: An llm agent to execute tests of arbitrary projects,”Proceedings of the ACM on Software Engineering, vol. 2, no. ISSTA, pp. 1054–1076, 2025

    2025

  32. [32]

    Aider: Ai pair programming in your terminal,

    P. Gauthier, “Aider: Ai pair programming in your terminal,” https: //github.com/Aider-AI/aider, 2023, accessed 2026-06-27

    2023

  33. [33]

    Claude code: Anthropic’s agentic coding system,

    Anthropic, “Claude code: Anthropic’s agentic coding system,” https:// www.anthropic.com/product/claude-code, 2025, accessed 2026-06-27

    2025

  34. [34]

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” inAdvances in Neural Information Processing Systems 36 (NeurIPS), Datasets and Benchmarks Track, 2023

    2023