Pith. sign in

REVIEW 4 major objections 6 minor 4 cited by

Agentic Program Repair from Test Failures at Scale: A Neuro-symbolic approach with static analysis and test execution feedback

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An agentic repair system combining a reasoning loop with static analysis and test-failure feedback solved 42.3% of curated production failures offline and landed 25.5% of fixes it generated in a three-month production run.

desk verdict Real industrial deployment with a land-rate number that needs per-diff attribution before it can be trusted; the offline ablations are the strongest part. read the letter →

arxiv 2507.18755 v1 pith:OOHRE3OU submitted 2025-07-24 cs.SE cs.AIcs.PL

Chandra Maddila , Adam Tait , Claire Chang , Daniel Cheng , Nauman Ahmad , Vijayaraghavan Murali , Marshall Roch , Arnaud Avondet
show 16 more authors
Aaron Meltzer Victor Montalvao Michael Hopko Chris Waterson Parth Thakkar Renuka Fernandez Kristian Kristensen Sivan Barzily Sherry Chen Rui Abreu Nachiappan Nagappan Payam Shodjai Killian Murphy James Everingham Aparna Ramani Peter C. Rigby
This is my paper
classification cs.SEcs.AIcs.PL
keywords automatedprogramrepairagenticAIsystemsreasoning-actionagentloopneuro-symbolicfeedbackstaticanalysistestexecutionLLM-as-a-Judgesearch-replacepatchformat
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that an autonomous coding agent can repair real production test failures end-to-end inside a large industrial codebase, not just on curated open-source bugs. The authors built an Engineering Agent that starts from a triaged failing test, runs a reasoning-and-acting loop with fifteen tools, and gets deterministic feedback from static analyzers and from actually running the tests; on a 123-task offline benchmark the balanced configuration solves 42.3% of tasks in one run, with the deterministic feedback raising solve rate from 28.5%. In a three-month production deployment the agent generated 1,589 fixes, 80% of which received human review and 25.5% of which were landed, which the authors read as evidence that the validation pipeline plus an LLM-as-a-Judge blocks most low-quality patches before engineers see them. A sympathetic reader would care because these numbers show that LLM agents have moved past feasibility demonstrations and already absorb a non-trivial slice of real repair work, while the gap between solve rate and land rate shows where human judgment still matters.

What carries the argument

The load-bearing mechanism is a neuro-symbolic verification loop. A ReAct-style agent alternates thoughts and actions drawn from fifteen tools (reading files, searching code, running tests, editing, and so on); after an edit, deterministic static-analysis output and the oracle test run produce observations that are fed back into the loop, so the model is corrected by symbolic signals rather than left to guess. Patch generation is a separate sub-agent that emits search-replace blocks—a unique "search" snippet paired with a "replace" snippet—because the paper finds this format far more natural for LLMs than unified diff. Before reaching a human reviewer, a patch passes an LLM-as-a-Judge, a many-shot classifier calibrated on 244 human-labeled patches to predict whether a human engineer would find the change acceptable. These components together implement what the paper calls Oracle-Guided Patch Generation.

What would settle it

Run the same agent on the production test-failure tasks that the benchmark curation excludes—those fixed by changing test files or generated code, those with no human fix within 90 days, and those later deemed flaky—and compare solve rates; if the excluded slice solves at rates far below 42.3% while the curated slice stays near 42.3%, the central claim is an artifact of benchmark construction rather than a property of production failures.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that Oracle-Guided Patch Generation—working from a triplet of specification, golden patch, and test oracle—can be operationalized inside a large monorepo through a ReAct loop with fifteen tool actions, symbolic validation feedback, and a learned judge. The patcher writes changes in a search-replace block format rather than unified diff, which the paper finds improves solve rate by up to 23 percentage points and lets a fine-tuned 70B model nearly match a much larger vanilla 405B model. In the offline ablation, plain ReAct solves 28.5% of the benchmark tasks; adding only static-analysis feedback reaches 34.1%, adding only test-execution feedback reaches 43.9%, and the deployed combination of both reaches 42.3% with a lower 0.2% error rate and 11.8 average iterations, while five repeated runs reach 61.0% with test-execution feedback. In production, the funnel was 1,589 published diffs, 1,285 reviewed, and 405 landed over three months, corresponding to 25.5% of generated fixes and 31.5% of reviewed fixes.

Load-bearing premise

The load-bearing premise is that the 123-task offline benchmark, which keeps only failures that a human actually fixed within 90 days by changing source code, fairly represents the full production test-failure stream; if the real stream is dominated by flaky tests, test-file fixes, generated-code changes, and unsolved problems, the 42.3% solve rate and the conclusions of the ablation may not transfer.

Editorial extensions

If this is right

  • A single agent run can be used as a cheap first pass: 42.3% of benchmarked failures are solved within about 12 feedback iterations, and allowing five repeated runs lifts the solve rate to 61.0% when test-execution feedback is used.
  • Deterministic feedback is a first-class contributor, not a small addition: removing both static-analysis and test-execution feedback drops single-run solve rate from 42.3% to 28.5%.
  • Patch format is a low-cost lever: switching from unified diff to search-replace blocks improves the large model's solve rate from 30% to 53%, which means existing LLM repair pipelines can gain accuracy without changing the model.
  • The production funnel implies that roughly one in four generated fixes lands without further human code changes; the validation pipeline, judge, and human review together reject or defer the other three.
  • The judge's calibration trades recall for precision: 86.7% of patches it accepts are human-acceptable and it catches 80.7% of unacceptable patches, but it also rejects 61.5% of acceptable patches, so its main effect is reducing reviewer load rather than maximizing landed fixes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The offline benchmark's curation—human-fixed within 90 days, source-only changes, oracle tests that fail without the fix—almost certainly makes 42.3% an upper bound for the full production stream, which also contains test-file fixes, generated-code changes, flaky failures, and unsolved tasks; the 25.5% land rate is the more conservative measure of real-world performance.
  • The paper's own rollouts suggest a testable extension: it enabled the agent to edit test files, so a before/after comparison of land rates would show whether lifting the source-only restriction increases production impact without increasing bad fixes.
  • The qualitative feedback that partially correct solutions are useful starting points implies that binary solve rate undercounts value: offline runs judged as failures may still save engineer time by localizing the bug or sketching the right approach.
  • Because the search-replace advantage was observed across two models and three formats in a non-agentic patch-generation setting, the result likely transfers to other LLM-based repair and editing pipelines, where it can be adopted without retraining.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper describes the design, offline evaluation, and three-month production deployment of an agentic program repair system at Meta. The agent consumes test-failure tasks from a rule-based Test Failure Management Bot, uses a ReAct-style harness with 15 tools, receives feedback from static analysis and test execution, and generates search-replace patches using an internal fine-tuned Llama-70B model. The paper reports offline benchmark results (42.3% solve rate at one attempt for the balanced agent configuration on a 123-task benchmark), an ablation study showing that test-execution feedback improves solve rate over the pure neural ReAct agent, an LLM-as-a-judge pipeline to filter unacceptable patches, and a production outcome of 1,589 published diffs of which 405 landed (25.5% of published, 31.5% of reviewed). Qualitative feedback from engineers is analyzed with open coding, and the paper describes system changes made in response to that feedback.

Significance. If the production land-rate claim holds, the paper provides one of the first large-scale industrial evidence points for end-to-end agentic program repair, with a concrete pipeline, deployment numbers, and engineer feedback. The paper is honest about several limitations, including benchmark size and generalizability, and it makes a useful contribution by demonstrating that symbolic feedback (static analysis and test execution traces) improves a purely neural ReAct agent. However, the central production metric currently has a measurement-validity problem: the paper does not establish that the 405 landed diffs are the agent's own published revisions rather than human fixes attached to the same task. The judge metric is also reported inconsistently with its own table. The offline benchmarks are small and lack uncertainty quantification, and the curated benchmark distribution excludes several categories that the production feedback shows to be common.

major comments (4)
  1. [Section 9.1, Table 5, Section 9.2 (Rules Engine)] The headline production result, 405 landed diffs (25.5% of 1,589 published), is not demonstrated to measure agent-authored landings. The description of the later rules engine -- 'if the original task had any attached diffs, we would not publish the agent-generated change' -- indicates that during the three-month measurement window a single TFMB task could have both a human-authored diff and an agent-published diff. The paper does not state whether the 405-landed numerator is restricted to diffs whose landed revision is the agent's published revision, or whether the 1,589 denominator excludes tasks where a human diff already landed. As written, the land-rate metric conflates task-level resolution with agent-diff acceptance. Please report per-diff attribution: for each agent-published diff, did the exact published revision (or a clearly derived revision of it) land? Also report how many of the 1,285 reviewed diffs were superseded by attached human diffs.
  2. [Section 8.3, Table 4, Section 1 (Contribution 4)] The text and table are inconsistent in reporting the judge's operating point. Table 4 reports Class-0 (unacceptable) precision 0.286 and Class-1 (acceptable) precision 0.867, yet Section 8.3 claims 'we optimized for a higher Class-1 precision' and then states that 'we landed on a prompt that yielded a high precision for Class-0 .86'; the Introduction claims '.867 precision' for low-quality samples. These statements cannot all be true with Table 4. Please reconcile: if the deployed judge uses the Table 4 operating point, it removes 80.7% of unacceptable patches but also discards 61.5% of acceptable patches (Class-1 recall 0.385), so the production impact of the judge must be recomputed. If a different calibrated prompt was deployed, report its benchmark metrics and explain the discrepancy.
  3. [Section 4.1, Section 8.2, Section 10.3, Section 9.2] The offline benchmarks are small (n=123 for the agent ablation, n=210 for PatchGen, n=244 for the judge) and are reported without confidence intervals or significance tests. Table 3's SR@1 differences (28.5% vs 34.1% vs 43.9% vs 42.3%) may be within sampling noise; a sign test or bootstrap confidence interval is needed to support the claim that symbolic feedback improves the neural model. Additionally, the curation filters in Section 4.1 (landed within 90 days, source-only changes, oracle passing with the golden fix) exclude categories that Section 9.2 shows to be common in production, notably 'Test already fixed' (n=47), 'Identifying source vs. test changes' (n=6), and flaky tests. This undermines the extrapolation from the 42.3% offline solve rate and the ablation conclusions to the full production distribution.
  4. [Section 7.2 vs Section 8.2, Table 3] The sampling protocol is inconsistent. Section 7.2 states that the agent is run repeatedly with temperature 0.8 and that 'we consider a solution to be correct if any of the 10 solutions generated by the agent fixes a test failure' (pass@10), while Section 8.2 and Table 3 report SR@5 and describe 'running the agent multiple times, up to 5'. Please state whether the maximum number of runs is 5 or 10, and label the metric consistently. The reported best-model figure of 61.0% depends on this choice.
minor comments (6)
  1. [Section 12] The text says 'RaAct-based agentic harness'; this should be 'ReAct-based'.
  2. [Section 6.2] The phrase 'logged for suture analyses' appears to be a typo for 'future analyses' or 'further analyses'.
  3. [Section 8.3] The sentence 'When we evaluated out LLM-as-a-Judge' should read 'our LLM-as-a-Judge'.
  4. [Section 5] The sentence 'The final patch format is based on the search-replace format proposed.1 This intuitive format...' has a misplaced footnote marker; it should read 'proposed [reference].'
  5. [Table 5 and Section 9.1] The table reports 1,285 reviewed diffs out of 1,589 published (80.9%), while the prose says '80%'; please use the exact percentage or state the rounding convention.
  6. [References] References [8] and [12] appear to be the same work, 'Evaluating agent-based program repair at google'; one should be removed or cross-referenced.

Circularity Check

1 steps flagged · score 3.0 of 10

Central claims are empirical and non-circular; only the LLM-judge validation is circular because its prompt was calibrated on the same 244 labeled patches used to report precision.

  1. fitted input called prediction [Section 6.2 (LLM-as-a-Judge benchmark) and Section 8.3 (LLM-as-a-Judge Benchmark Results)]
    "To evaluate the efficacy of the LLM-as-a-Judge approach, we created a benchmark of 244 data points. We asked human engineers to review these 244 Engineering Agent-generated patches and classify them into classes: acceptable, unacceptable. Human engineers labeled 57 as acceptable and 187 as unacceptable. Then, we tested our pre-calibrated LLM on this data set to calculate metrics such as precision and recall. ... After going through multiple calibration optimizations and iterations, we landed on a prompt that yielded a high precision for Class-0 .86 allowing us to use this judge in production."

    The same 244 human-labeled patches were used first to calibrate the judge prompt through 'multiple calibration optimizations and iterations' and then to compute the reported precision and recall. The .86 precision is therefore an in-sample goodness-of-fit value, not an independent prediction about held-out patches. The judge's reported validation metrics reduce to the labels that were already used to select the prompt. This circularity is confined to the judge-validation claim; the production land rate is not affected because final accept/reject decisions were made by human reviewers.

full rationale

The paper's central chain — TFMB triage, agentic loop, oracle tests, human review, and landed diffs — is not definitionally circular. Offline solve rates are measured against oracle tests that are checked to fail without the human fix and pass with it, and the PatchGen benchmark is explicitly artificial because instructions are backtranslated from the golden diff, so it is a diagnostic comparison rather than a production prediction. The production land rate is decided by human reviewers rather than by the agent or judge, and the 47/100 'Test already fixed' comments point to an attribution/measurement-validity concern, not a circular derivation. The one concrete circular step is the LLM-as-a-Judge evaluation: the same 244 human labels were used to calibrate the prompt through iterative optimization and then to compute the reported precision/recall, making the .86 figure an in-sample fit. There is also an internal reporting inconsistency — Section 8.3 claims high Class-0 precision of .86 while Table 4 lists Class-0 precision as 0.286 and Class-1 precision as 0.867 — but that is a correctness/reporting issue, not circularity. No load-bearing self-citation or imported uniqueness theorem appears; the self-citations to iCodeLlama describe the internally fine-tuned base model and do not substitute for the measured results. Overall, the principal empirical claims remain independently grounded, so the circularity score is modest.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper's empirical claims rest on internal benchmarks and internal telemetry, with several unstated representativeness assumptions. No new physical or formal entities are introduced; the main load-bearing assumptions are about test-passing as a proxy for correctness and about benchmark curation.

free parameters (3)
  • LLM sampling temperature = 0.8
    Temperature for repeated runs; affects SR@5 but the effect is not isolated from the number of runs.
  • Benchmark curation window = 90 days
    TFMB tasks created in the last 90 days; temporal scope determines benchmark difficulty and composition.
  • Source-only fix filter = source files only
    Excludes human fixes that touched tests, build, config, or generated files; induces selection bias in the benchmark.
assumptions (5)
  • domain assumption Passing the broken oracle tests is a sufficient signal that a patch is correct.
    Section 2 defines the Oracle and calls the method Oracle-Guided Patch Generation; the solve rate is test-pass based and does not check semantic equivalence.
  • domain assumption Human-landed fixes are valid ground truth for benchmark tasks.
    Section 4.1 uses the human solution as the golden patch and validates that oracle tests pass with it; assumes the human fix itself was correct and representative.
  • domain assumption TFMB stack traces and bisection provide enough specification for the agent to locate and fix the fault.
    Section 3 feeds TFMB output as the task specification; if triage is wrong or incomplete, the agent starts from inadequate input.
  • domain assumption Reviewer acceptance and landing indicate fix quality in production.
    Section 7.3 tracks review and land rates; Section 10.2 acknowledges that 20% never received review and that passing tests does not guarantee correct code.
  • domain assumption LLM judge labels, calibrated on 244 human-annotated patches, reflect Meta engineering standards.
    Section 6.2 uses 244 patches labeled by engineers; small sample and potential label subjectivity bound the judge claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Agentic Program Repair from Test Failures at Scale: A Neuro-symbolic approach with static analysis and test execution feedback." pith.science (2026). https://pith.science/paper/OOHRE3OU

@misc{pith2026250718755,
  author       = {Pith},
  title        = {Pith review of: Agentic Program Repair from Test Failures at Scale: A Neuro-symbolic approach with static analysis and test execution feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OOHRE3OU}},
  note         = {Machine review of arXiv:2507.18755}
}
read the original abstract

Aim: With the advent of LLMs, sophisticated agentic program repair has become viable at large organizations with large codebases. In this work, we develop an Engineering Agent that fixes the source code based on test failures at scale across diverse software offerings internally. Method: Using Llama as the base, we employ the ReAct harness to develop an agent. We start with a test failure that was triaged by a rule-based test failure bot. We then set up an agentic harness and allow the agent to reason and run a set of 15 actions from reading a file to generating a patch. We provide feedback to the agent through static analysis and test failures so it can refine its solution. We leverage an LLM-as-a-Judge to ensure that the patch conforms to the standards followed by a human review to land fixes. Benchmark Findings: We curated offline benchmarks for our patch generator, the Engineering Agent loop, and the LLM-as-a-Judge. In offline evaluations we found that a specialized 70B model is highly competitive with the much larger but vanilla Llama-405B. In an ablation study, we found that the ReAct harness (neural model) benefited from the symbolic information from static analysis tools and test execution traces. A model that strikes a balance between the solve rate and error rate vs the cost and latency has a benchmark solve rate of 42.3% using an average 11.8 feedback iterations. Production Findings: In a three month period, 80% of the generated fixes were reviewed, of which 31.5% were landed (25.5% of the total number of generated fixes). Feedback from Engineers: We used open coding to extract qualitative themes from engineers' feedback. We saw positive feedback in the form of quick approvals, gratitude, and surprise. We also found mixed feedback when the Engineering Agent's solution was partially correct and it served as a good starting point.

Figures

Figures reproduced from arXiv: 2507.18755 by the authors.

Figure 1
Figure 1. Engineering Agent Flow. We begin with a test failure that is triaged by the Test Failure Management Bot. The [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 1
Figure 1. 1. inside the ReAct loop 2. at the end of the ReAct [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. An example of the trajectory of the agent fixing a broken test. This information is also provided to the reviewer so [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. AuditRepairBench: A Paired-Execution Trace Corpus for Evaluator-Channel Ranking Instability in Agent Repair

    cs.AI 2026-05 conditional novelty 7.0 of 10

    AuditRepairBench isolates and measures evaluator-to-selector coupling in agent-repair systems, releasing a 12,000-cell Lite corpus and showing targeted blinding cuts rank displacement by 55–74%.

  2. AuditRepairBench: A Paired-Execution Trace Corpus for Evaluator-Channel Ranking Instability in Agent Repair

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    AuditRepairBench supplies a large trace corpus and four screening methods that reduce evaluator-channel ranking instability in agent repair leaderboards by a mean of 62%.

  3. Hallucination Inspector: A Fact-Checking Judge for API Migration

    cs.SE 2026-04 unverdicted novelty 6.0 of 10

    Hallucination Inspector verifies symbols in LLM-generated API migration code against a documentation-derived knowledge base using AST extraction, identifying scaffolding hallucinations and cutting false positives vers...

  4. LLM-Based Automated Diagnosis Of Integration Test Failures At Google

    cs.SE 2026-04 unverdicted novelty 4.0 of 10

    Auto-Diagnose applies LLMs to summarize and diagnose root causes of integration test failures, reporting 90.14% accuracy on 71 manual cases and positive adoption after Google-wide rollout.

Reference graph

Works this paper leans on

36 extracted references · 12 canonical work pages · cited by 3 Pith papers

  1. [1]

    Fixminer: Mining relevant fix patterns for automated program repair,

    A. Koyuncu, K. Liu, T. F. Bissyand ´e, D. Kim, J. Klein, M. Monperrus, and Y. Le Traon, “Fixminer: Mining relevant fix patterns for automated program repair,” Empirical Software Engineering, vol. 25, no. 3, p. 1980–2024, Mar. 2020. [Online]. Available: http://dx.doi.org/10.1007/s10664-019-09780-z

    doi:10.1007/s10664-019-09780-z 1980

  2. [2]

    A survey on automated program repair techniques,

    K. Huang, Z. Xu, S. Yang, H. Sun, X. Li, Z. Yan, and Y. Zhang, “A survey on automated program repair techniques,” 2023. [Online]. Available: https://arxiv.org/abs/2303.18184

    arXiv 2023

  3. [3]

    Automated program repair,

    C. Le Goues, M. Pradel, and A. Roychoudhury, “Automated program repair,” Commun. ACM , vol. 62, no. 12, p. 56–65, Nov

    work page

  4. [4]

    A systematic literature review on large language models for automated program repair,

    Q. Zhang, C. Fang, Y. Xie, Y. Ma, W. Sun, Y. Yang, and Z. Chen, “A systematic literature review on large language models for automated program repair,” 2024. [Online]. Available: https://arxiv.org/abs/2405.01466

    arXiv 2024

  5. [5]

    Enhancing automated program repair with deductive verification,

    X.-B. D. Le, Q. L. Le, D. Lo, and C. Le Goues, “Enhancing automated program repair with deductive verification,” in 2016 IEEE Inter- national Conference on Software Maintenance and Evolution (ICSME) , 2016, pp. 428–432

    work page 2016

  6. [6]

    Automatic program repair using formal verification and expression templates,

    T.-T. Nguyen, Q.-T. Ta, and W.-N. Chin, “Automatic program repair using formal verification and expression templates,” in Verification, Model Checking, and Abstract Interpretation, C. Enea and R. Piskac, Eds. Cham: Springer International Publishing, 2019, pp. 70–91

    work page 2019

  7. [7]

    Frenkel, O

    H. Frenkel, O. Grumberg, B.-C. Rothenberg, and S. Sheinvald, Automated Program Repair Using Formal Verification Techniques . Cham: Springer Nature Switzerland, 2022, pp. 511–534. [Online]. Available: https://doi.org/10.1007/978-3-031-22337-2 25

    doi:10.1007/978-3-031-22337-2 2022

  8. [9]

    Agentic bug reproduction for effective automated program repair at google,

    R. Cheng, M. Tufano, J. Cito, J. Cambronero, P . Rondon, R. Wei, A. Sun, and S. Chandra, “Agentic bug reproduction for effective automated program repair at google,” 2025. [Online]. Available: https://arxiv.org/abs/2502.01821

    arXiv 2025

Show all 36 references
  1. [10]

    Orca: Differential bug localization in Large-Scale services,

    R. Bhagwan, R. Kumar, C. S. Maddila, and A. A. Philip, “Orca: Differential bug localization in Large-Scale services,” in 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18) . Carlsbad, CA: USENIX Association, Oct. 2018, pp. 493–509. [Online]. Availab...

    2018

  2. [11]

    React: synergizing reasoning and acting in language models (2022),

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “React: synergizing reasoning and acting in language models (2022),” arXiv preprint arXiv:2210.03629, 2023

    2022 arXiv

  3. [12]

    Evaluating agent-based program repair at google,

    “Evaluating agent-based program repair at google,” 2025. [Online]. Available: https://arxiv.org/pdf/2501.07531

    2025 arXiv

  4. [13]

    How do you test your tests?

    “How do you test your tests?” 2020. [Online]. Available: https://engineering.fb.com/2020/12/10/developer-tools/proba bilistic-flakiness/ TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. X, NO. Y , JUL Y 2025 14

    2020

  5. [14]

    Neurosymbolic repair for low-code formula languages,

    R. Bavishi, H. Joshi, J. Cambronero, A. Fariha, S. Gulwani, V . Le, I. Radi ˇcek, and A. Tiwari, “Neurosymbolic repair for low-code formula languages,” Proc. ACM Program. Lang. , vol. 6, no. OOPSLA2, Oct. 2022. [Online]. Available: https: //doi.org/10.1145/3563327

    2022 doi

  6. [15]

    Ai-assisted code authoring at scale: Fine-tuning, deploying, and mixed methods evaluation,

    V . Murali, C. Maddila, I. Ahmad, M. Bolin, D. Cheng, N. Ghorbani, R. Fernandez, N. Nagappan, and P . C. Rigby, “Ai-assisted code authoring at scale: Fine-tuning, deploying, and mixed methods evaluation,” Proc. ACM Softw. Eng. , vol. 1, no. FSE, Jul. 2024. [Online]. Available:...

    2024 doi

  7. [16]

    Multi-line ai-assisted code authoring,

    O. Dunay, D. Cheng, A. Tait, P . Thakkar, P . C. Rigby, A. Chiu, I. Ahmad, A. Ganesan, C. Maddila, V . Murali, A. Tayyebi, and N. Nagappan, “Multi-line ai-assisted code authoring,” in Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software...

    2024

  8. [17]

    Ai- assisted sql authoring at industry scale,

    C. Maddila, N. Ghorbani, K. Jabre, V . Murali, E. Kim, P . Thakkar, N. P . Laptev, O. Harman, D. Hsu, R. Abreu, and P . C. Rigby, “Ai- assisted sql authoring at industry scale,” in Foundations of Software Engineering Industry Track, 2025

    2025

  9. [18]

    Large language monkeys: Scaling inference compute with repeated sampling,

    B. Brown, J. Juravsky, R. Ehrlich, R. Clark, Q. V . Le, C. R ´e, and A. Mirhoseini, “Large language monkeys: Scaling inference compute with repeated sampling,” 2024. [Online]. Available: https://arxiv.org/abs/2407.21787

    2024 arXiv

  10. [19]

    Building knowledge through families of experiments,

    V . Basili, F. Shull, and F. Lanubile, “Building knowledge through families of experiments,” IEEE Transactions on Software Engineering, vol. 25, no. 4, pp. 456–473, 1999

    1999

  11. [20]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” 2024. [Online]. Available: https://arxiv.org/abs/2310.06770

    2024 arXiv

  12. [21]

    (Accessed 2025) Introducing swe-bench verified. OpenAI. [Online]. Available: https://openai.com/index/introducing-swe-bench-v erified/

    2025

  13. [22]

    Swe-smith: Scaling data for software engineering agents,

    J. Yang, K. Leret, C. E. Jimenez, A. Wettig, K. Khandpur, Y. Zhang, B. Hui, O. Press, L. Schmidt, and D. Yang, “Swe-smith: Scaling data for software engineering agents,” 2025. [Online]. Available: https://arxiv.org/abs/2504.21798

    2025 arXiv

  14. [23]

    Swe- lancer: Can frontier llms earn $1 million from real-world freelance software engineering?

    S. Miserendino, M. Wang, T. Patwardhan, and J. Heidecke, “Swe- lancer: Can frontier llms earn $1 million from real-world freelance software engineering?” arXiv preprint arXiv:2502.12115, 2025

    2025 arXiv

  15. [24]

    From llm reasoning to autonomous ai agents: A comprehensive review,

    M. A. Ferrag, N. Tihanyi, and M. Debbah, “From llm reasoning to autonomous ai agents: A comprehensive review,” arXiv preprint arXiv:2504.19678, 2025

    2025 arXiv

  16. [25]

    RepairAgent: An au- tonomous, llm-based agent for program repair,

    I. Bouzenia, P . Devanbu, and M. Pradel, “RepairAgent: An au- tonomous, llm-based agent for program repair,” arXiv preprint arXiv:2403.17134, 2024

    2024 arXiv

  17. [26]

    Autocoderover: Autonomous program improvement,

    Y. Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” 2024. [Online]. Available: https://arxiv.org/abs/2404.05427

    2024 arXiv

  18. [27]

    SpecRover: Code intent extraction via LLMs,

    H. Ruan, Y. Zhang, and A. Roychoudhury, “SpecRover: Code intent extraction via LLMs,” arXiv preprint arXiv:2408.02232, 2024

    2024 arXiv

  19. [28]

    Coder: Issue resolving with multi- agent and task graphs,

    D. Chen, S. Lin, M. Zeng, D. Zan, J.-G. Wang, A. Cheshkov, J. Sun, H. Yu, G. Dong, A. Aliev et al., “Coder: Issue resolving with multi- agent and task graphs,” arXiv preprint arXiv:2406.01304, 2024

    2024 arXiv

  20. [29]

    Marscode agent: Ai-native automated bug fixing,

    Y. Liu, P . Gao, X. Wang, J. Liu, Y. Shi, Z. Zhang, and C. Peng, “Marscode agent: Ai-native automated bug fixing,” arXiv preprint arXiv:2409.00899, 2024

    2024 arXiv

  21. [31]

    Openhands: An open platform for ai software developers as generalist agents,

    X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, H. H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, Y. Shao, N. Muennighoff, Y. Zhang, B. Hui, J. Lin, R. Brennan, H. Peng, H. Ji, and G. Neubig, “Openhands: An open platform for ai software devel...

    2025 arXiv

  22. [32]

    Swe-agent: Agent-computer interfaces enable automated software engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated software engineering,” 2024. [Online]. Available: https://arxiv.org/abs/2405.15793

    2024 arXiv

  23. [33]

    Automated program repair in the era of large pre-trained language models,

    C. S. Xia, Y. Wei, and L. Zhang, “Automated program repair in the era of large pre-trained language models,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2023, pp. 1482–1494

    2023

  24. [34]

    Less training, more repairing please: revisiting automated program repair via zero-shot learning,

    C. S. Xia and L. Zhang, “Less training, more repairing please: revisiting automated program repair via zero-shot learning,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2022, pp. 959–971

    2022

  25. [35]

    The plas- tic surgery hypothesis,

    E. T. Barr, Y. Brun, P . Devanbu, M. Harman, and F. Sarro, “The plas- tic surgery hypothesis,” in Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering , 2014, pp. 306–317

    2014

  26. [36]

    Automated program repair via conversa- tion: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT,

    C. S. Xia and L. Zhang, “Automated program repair via conversa- tion: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024, pp. 819–831

    2024

  27. [37]

    Agentless: Demys- tifying llm-based software engineering agents,

    C. S. Xia, Y. Deng, S. Dunn, and L. Zhang, “Agentless: Demys- tifying llm-based software engineering agents,” arXiv preprint arXiv:2407.01489, 2024

    2024 arXiv

  28. [2019]

    Available: https://doi.org/10.1145/3318162

    [Online]. Available: https://doi.org/10.1145/3318162

    doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.