Pith. sign in

REVIEW 3 major objections 6 minor 5 cited by

Evaluating Agent-based Program Repair at Google

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A simple agent patches 73% of machine-reported Google bugs in an enterprise test.

desk verdict A credible first industrial benchmark and baseline for agentic APR, but the 73% machine-reported plausible rate is probably inflated by the lenient TOD oracle, which the paper itself flags. read the letter →

arxiv 2501.07531 v1 pith:FQM6HW7B submitted 2025-01-13 cs.SE cs.AI

Pat Rondon , Renyao Wei , José Cambronero , Jürgen Cito , Aaron Sun , Siddhant Sanyam , Michele Tufano , Satish Chandra This is my paper · ORCID
classification cs.SEcs.AI
keywords automatedprogramrepairagenticLLMagentsSWE-Benchenterprisemonorepobugbenchmarkcurationtest-orderdependencesanitizerbugs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that agentic automated program repair can work in a real industrial monorepo, not just on the open-source SWE-Bench benchmark. The authors curate 178 bugs from Google's issue tracker, build a deliberately simple SWE-Agent-style repair agent called Passerine, and measure how often its 20 sampled repair trajectories produce a patch that passes the bug-reproducing tests (plausible) and, on manual review, matches the ground-truth fix (valid). The headline numbers are 73% plausible and 43% valid for machine-reported bugs, and 25.6% plausible and 17.9% valid for human-reported bugs. If these numbers hold, they imply that a small, minimally engineered agent can take on a meaningful share of real enterprise repair work, and that improving bug report quality may be as important as improving the agent.

What carries the argument

The load-bearing object is Passerine, a fully dynamic ReAct-style repair agent with no prespecified control flow and a five-command tool set (cat, code search, edit, bazel test, finish) that interacts with Google's internal code search, build, and containerization APIs. The companion machinery is GITS-Eval, a 178-bug benchmark curated through a four-phase filtering funnel (fix association, testability, tractability, and execution-based test adequacy) that separates human-reported from machine-reported bugs.

What would settle it

Run Passerine with the same 20-trajectory protocol on a random sample of bugs from Google's issue tracker that were fixed in the same period but were excluded by the curation funnel (e.g., bugs with multimedia, patches over 150 lines, magic-constant fixes, or flaky tests) and measure the plausible-patch rate; if it drops sharply below 73% for machine-reported and 25.6% for human-reported bugs, the claim that these numbers represent enterprise APR viability is undermined.

Watch

Extended reading notes

Core claim

The paper's central claim is that Passerine, a deliberately minimal agentic repair system inspired by SWE-Agent and running inside Google's internal development environment, produces a patch that passes the bug-reproducing tests for 73% of machine-reported bugs and 25.6% of human-reported bugs in a curated 178-bug benchmark (GITS-Eval), using 20 trajectory samples per bug with Gemini 1.5 Pro. Manual annotation by the authors found at least one patch semantically equivalent to the ground-truth fix for 43% of machine-reported bugs and 17.9% of human-reported bugs. The paper further claims that GITS bugs differ from SWE-Bench bugs along dimensions such as language diversity, patch size, and spatial spread, so performance on one benchmark should not be assumed to transfer to the other.

Load-bearing premise

The reported success rates are computed on a curated benchmark that filters out bugs with multimedia, large patches, magic-constant fixes, flaky tests, and hard-to-test file types, and that judges human-bug fixes against the ground-truth test file; if this curation and these test oracles do not reflect the real distribution of enterprise bugs, the rates will not generalize to an unfiltered bug pool.

Editorial extensions

If this is right

  • A simple agent with a five-command interface can serve as a reproducible baseline for agentic repair in an enterprise monorepo, giving future systems concrete numbers to beat.
  • Machine-reported bugs (from sanitizers and a test-order dependency analyzer) are far more tractable for agents than human-reported bugs, suggesting that automated bug reporting already produces agent-friendly inputs.
  • Because GITS and SWE-Bench bugs differ in language, patch size, and spatial spread, SWE-Bench results should not be treated as predictive of industrial performance.
  • Trajectory analysis reveals that the agent adapts its strategy to bug type and that identifiable trajectory smells (no test runs, repeated reads, consecutive searches) correlate with failure, pointing to concrete optimization opportunities.
  • Rich bug reports, such as those from machine analyzers, improve both repair success and file-level localization, implying that bug report design is a leverage point for agentic APR.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported plausible rates are likely upper bounds for an unfiltered production bug pool, because the benchmark excludes exactly the hard cases (multimedia, large patches, magic constants, flaky tests) that would be common in practice; deployment would need abstention or triage.
  • If richer bug reports are the main driver of repair success, human bug-reporting platforms could be redesigned to nudge reporters toward including reproduction steps and expected behavior; the paper suggests this but does not test it.
  • The large gap between plausible and valid rates for TOD bugs suggests the test-order-dependence oracle is lenient; trying additional test orderings could make the benchmark stricter and lower the reported numbers.
  • The manual validity annotation could be supplemented by an automated semantic-equivalence check, but the paper does not propose one, so the validity numbers remain dependent on human judgment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper curates GITS-Eval, a benchmark of 178 fixed bugs drawn from Google's internal issue tracker (78 human-reported bugs and 100 machine-reported bugs, split into 50 sanitizer (SAN) and 50 test-order-dependence (TOD) bugs), using a multi-stage funnel that filters for testability, patch size, flakiness, and manually excludes 'magic constant' fixes. It then introduces Passerine, a minimal SWE-Agent-inspired ReAct agent with five commands, and evaluates it with 20 independent trajectories per bug using Gemini 1.5 Pro. The headline results are that Passerine produces at least one plausible patch for 73% of machine-reported bugs (68% TOD, 78% SAN) and 25.6% of human-reported bugs, and, after manual annotation, a semantically valid patch for 43% of machine-reported bugs (24% TOD, 62% SAN) and 17.9% of human-reported bugs. The paper also compares GITS bugs with SWE-Bench along localization and editing dimensions, reports trajectory-level observations about command usage and 'smells', and explicitly disclaims any claims about SWE-Bench performance.

Significance. If the results hold, this is a valuable first systematic study of agentic APR in a large industrial monorepo, and GITS-Eval is a useful benchmark for the community. The paper's strengths include direct measurement on a fixed benchmark, an explicit and detailed description of curation filters, author-acknowledged limitations, and a clear disclaimer that no SWE-Bench claims are made. The trajectory analysis and the observation that Passerine adapts its strategy to bug-report richness are credible and interesting. The main risks to the central quantitative claims are the lenient TOD plausibility oracle and the fact that agent modifications to test files are not evaluated; both are acknowledged in the text and can be addressed with additional experiments or sensitivity analyses.

major comments (3)
  1. [V-A and III-C] The TOD plausibility criterion is judged only under the single test ordering specified in the bug report, and Section V-A concedes that this criterion 'may be too lenient' because additional orderings are not tried. TOD bugs constitute 50 of the 100 machine-reported bugs, so an unknown fraction of TOD 'plausible' patches may only pass that one ordering. The paper's own TOD valid rate of 24% shows that many TOD-plausible patches are not true fixes. Please provide a sensitivity analysis: either re-run the evaluation for TOD bugs under multiple orderings, or report the plausible rate under the assumption that TOD's true plausible rate is close to its valid rate, with the implied impact on the combined machine-reported rate.
  2. [III-C] The evaluation framework does not evaluate agent modifications to test files. For machine-reported bugs, the bug-reproducing tests are extracted from the issue content rather than from a pristine ground-truth file, so the agent has access to these test files and can edit them. A patch that modifies the test itself (e.g., removing or reordering the order-sensitive test, or masking the failure) would be scored as plausible under the current oracle. Please report the frequency with which Passerine modifies the test files used for machine-reported bugs, and re-run the plausibility evaluation with those test files reverted to the issue-provided versions, or otherwise demonstrate that such edits do not inflate the reported rates.
  3. [IV and VII] The valid-patch rates (43% machine, 17.9% human) rest entirely on manual annotation by the authors, but Section IV reports only that 'one reviewer analyzed each patch, consulting with the other two authors for assessment of complex cases' and Section VII notes the usual subjectivity without providing reliability statistics. Given that these rates are a central headline claim, please provide an inter-rater reliability measure (e.g., Cohen's kappa) on a random, independently annotated sample of patches, and describe the annotation protocol in enough detail to be reproducible.
minor comments (6)
  1. [V, Table IV] Please add confidence intervals (e.g., Wilson intervals) to the rates in Table IV and Figure 7, and report the raw numerators and denominators, since several cells have n=50 and the differences between subgroup rates may be within sampling noise.
  2. [II and V] The text refers to 'Phase III bugs' in Section V, but the curation process is described as Phases 0 through 3; please use 'Phase 3' consistently.
  3. [III-B and Figure 8] The test-running command is named 'bazel' in the command list and Section III-B, but Figure 8's legend uses 'blaze'; please unify the command name across the text and figures.
  4. [I] In the Introduction, the sentence 'Passerine adapts its behavior based based on bug type' contains a duplicated word 'based'.
  5. [VIII-C] The Related Work section contains the typo 'Futhermore' instead of 'Furthermore'.
  6. [III-C] For human-reported bugs the framework uses the ground-truth test file to evaluate plausibility, but the paper does not state whether the same test-file-reverting policy is applied to machine-reported bugs; please clarify this in the evaluation description.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper reports measurements on a fixed benchmark, and its central rates do not reduce by construction to any fitted parameter, definition, or load-bearing self-citation.

full rationale

The paper's central claim is empirical: Passerine, run with 20 sampled trajectories on the curated GITS-Eval set, produces a patch that passes the bug-reproducing tests for 73% of machine-reported and 25.6% of human-reported bugs, with manually annotated validity rates of 43% and 17.9%. There is no derivation chain in which an output is defined in terms of an input, and no fitted parameter is later renamed as a prediction. The benchmark curation funnel in Section II is a selection procedure, not a definition of the success metric: the filters exclude bugs with multimedia, large patches, magic constants, and flaky or non-executable tests, but the reported plausible and valid rates are measured against the ground-truth test behavior and manual semantic-equivalence annotation, neither of which is constructed from Passerine's outputs. The paper explicitly compares GITS to SWE-Bench at Phase 1, before later curation, to avoid confounding the distribution comparison with Passerine's current limitations, which further indicates that the curation did not serve as an evaluation oracle. The acknowledged TOD oracle weakness in Section V-A, where plausibility is judged under a single test ordering and the authors state the criterion 'may be too lenient,' is a measurement-validity limitation rather than circularity: it does not make the plausible rate equal to an input by definition, and the authors openly report the corresponding valid rate (24%) that reveals the gap. The note in Section III-C that 'we do not evaluate agent modifications to test files' is similarly an oracle limitation, not a circular construction. Self-citations are present but not load-bearing: AutoDev [6] and other in-house or coauthored work are cited for context or inspiration, while Passerine's design is explicitly credited to the external SWE-Agent work [2], and no uniqueness theorem or prior result from the same authors is invoked to force the benchmark or the agent's design. No parameters are fitted to the evaluation data; the LLM is used out-of-the-box with fixed sampling settings, and the authors explicitly disclaim any claim about SWE-Bench. A skeptical reader could question the leniency of the TOD plausibility oracle or the representativeness of the curated benchmark, but those are empirical-threat concerns, not circularity under the definitions used here.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central empirical claim rests primarily on experimental settings and curation assumptions, not on a mathematical derivation. The free parameters are evaluation hyperparameters and benchmark filters chosen by hand; none are fitted to make the repair rates come out. The axioms are domain assumptions about test adequacy, manual validity judgment, and representativeness of the filtered dataset. No new physical or theoretical entities are introduced.

free parameters (6)
  • Number of trajectory samples per bug = 20
    The reported plausible/valid rates are at-least-one-out-of-20 rates; changing this number changes the headline percentages (Figure 7 shows saturation behavior).
  • Maximum trajectory steps = 25
    Agents are cut off after 25 ReAct steps; this bounds how much search can occur and affects success rates.
  • LLM sampling temperature = 0.2
    Fixed sampling temperature for Gemini 1.5 Pro in all runs.
  • Top-p sampling = 0.95
    Fixed nucleus sampling parameter for all runs.
  • Code search result limit = 5
    Code search returns at most 5 file matches per query; affects localization ability.
  • Patch size limit in curation = 150 lines
    Phase 2 filter excludes ground-truth patches larger than the 90th percentile (<150 lines), making the benchmark more tractable and biasing success rates upward.
assumptions (5)
  • domain assumption The bug-reproducing test suites used for plausibility correctly capture the bug and validate the fix.
    Phase 3 execution-based test adequacy check and the Section III-C framework rely on these tests; if a test is wrong or too lenient (e.g., TOD order-dependence test), the plausibility numbers are wrong. The paper itself notes TOD plausibility may be too lenient (Section V-A).
  • domain assumption Manual judgment by three authors that a patch is semantically equivalent to the ground truth is a reliable ground truth.
    Valid patch rates (17.9% to 43%) rest entirely on subjective human annotation; no inter-rater reliability statistic is reported (Section IV, Section VII).
  • domain assumption GITS-Eval, after filtering, is a useful proxy for industrial bug distributions at Google.
    The curation funnel in Section II deliberately removes hard-to-test and multimedia bugs; the paper acknowledges Google-specific threats in Section VII but uses the filtered set to draw conclusions about agentic APR viability.
  • domain assumption Google's monorepo and development environment are representative of large enterprise settings.
    Section VII states other industrial settings may differ, but the broader claim about enterprise context assumes some transferability.
  • domain assumption Machine-reported bug repro commands extracted by regex from issue content are correct reproductions.
    Section III-C says 'our framework automatically extracts bug-reproducing tests from the associated issue content using regular expressions'; incorrect extraction would break the reproduction and plausibility evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Agent-based Program Repair at Google." pith.science (2026). https://pith.science/paper/FQM6HW7B

@misc{pith2026250107531,
  author       = {Pith},
  title        = {Pith review of: Evaluating Agent-based Program Repair at Google},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FQM6HW7B}},
  note         = {Machine review of arXiv:2501.07531}
}
read the original abstract

Agent-based program repair offers to automatically resolve complex bugs end-to-end by combining the planning, tool use, and code generation abilities of modern LLMs. Recent work has explored the use of agent-based repair approaches on the popular open-source SWE-Bench, a collection of bugs from highly-rated GitHub Python projects. In addition, various agentic approaches such as SWE-Agent have been proposed to solve bugs in this benchmark. This paper explores the viability of using an agentic approach to address bugs in an enterprise context. To investigate this, we curate an evaluation set of 178 bugs drawn from Google's issue tracking system. This dataset spans both human-reported (78) and machine-reported bugs (100). To establish a repair performance baseline on this benchmark, we implement Passerine, an agent similar in spirit to SWE-Agent that can work within Google's development environment. We show that with 20 trajectory samples and Gemini 1.5 Pro, Passerine can produce a patch that passes bug tests (i.e., plausible) for 73% of machine-reported and 25.6% of human-reported bugs in our evaluation set. After manual examination, we found that 43% of machine-reported bugs and 17.9% of human-reported bugs have at least one patch that is semantically equivalent to the ground-truth patch. These results establish a baseline on an industrially relevant benchmark, which as we show, contains bugs drawn from a different distribution -- in terms of language diversity, size, and spread of changes, etc. -- compared to those in the popular SWE-Bench dataset.

Figures

Figures reproduced from arXiv: 2501.07531 by the authors.

Figure 1
Figure 1. An overview of the different phases of filtering we go through to collect a GITS evaluation set internal environment (e.g. logging, which is sensitive business data and must be stored accordingly). GITS, Google’s internal issue tracking system, houses a vast and diverse collection of bugs spanning a wide array of projects. This presents both an opportunity and a challenge for automated program repair (APR) systems. … view at source ↗
Figure 2
Figure 2. Machine-reported bug reports, which typically contain rich￾ness such as reproduction information. reflect the entirety of GitHub, it is a widely-used benchmark which sets the standard for evaluating automatic program repair systems. Thus, to create a comparable test set for GITS, we draw a random sample of 2,000 bug-fixing patches from Phase 1 of our bug filtering phases which comprise both human- and machine-report… view at source ↗
Figure 3
Figure 3. Comparing GITS Phase 1 bugs to SWE-Bench. Changes found in ground-truth patches solving GITS bugs tend to (a) have fewer likely code identifier tokens, which in turn require more sophisticated code search abilities and bug knowledge to successfully localize the original fault. These patches modify substantially more files (b), which are further apart in the codebase (c), and produce changes with many separate hunks … view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: GITS-Eval machine-reported bugs are comparable to SWE-Bench-Lite, while human-reported GITS-Eval bugs are more complex and resemble differences in our GITS and SWE-Bench comparison. LLM-Based ReAct-Style “Driver” “Think”: Plain-text “plan” Act: Unix-Style Command Obser…
Figure 5
Figure 5. Figure 5: A high-level overview of Passerine’s ReAct-style dynamic loop and the Unix-style (adapted to Google) commands exposed as tools for the agent to use. B. Commands Passerine takes a minimalist approach by providing a set of only 5 commands (and 1 additional alias commonly…
Figure 6
Figure 6. Figure 6: System and command documentation template for Passerine those that fail before and pass after the ground-truth patch has been applied. After the bug has been confirmed, the framework reverts the repository state to reflect only information available before the ground-t…
Figure 7
Figure 7. Figure 7: provides further detail on patch plausibility and validity as a function of samples. We observe that the gap between plausibility and validity varies by bug type, as a result of nuances in their testing behavior. TOD bugs, which have the largest gap, are judged to be p…
Figure 8
Figure 8. Figure 8: Passerine’s trajectory distributions differ substantially when applied to bugs of different types, demonstrating both the distributional differences in the information provided by each bug type and the agent’s ability to adapt its behavior to the available information.…
Figure 9
Figure 9. Figure 9: Average file-system distance, defined as distance between two nodes (files) in an n-ary tree (file-system), between the agent edited files and the ground-truth files for trajectories where Passerine does not produce a fail-to-pass (i.e., not plausible). Passerine can m…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. NoCode-bench: A Benchmark for Evaluating Natural Language-Driven Feature Addition

    cs.SE 2025-07 conditional novelty 7.0 of 10

    NoCode-bench provides 634 documentation-change-to-code feature addition tasks, and the best evaluated LLM succeeds on only 28.07% of the human-verified subset.

  2. MultiFixer: A Coordinator-Proposer Based Multi-Agent Framework For Fixing Multi-Hunk Bugs

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Coordinator-proposer multi-agent repair schedules hunks, proposes candidate patches in parallel, and selects/refines them, fixing 326/835 Defects4J bugs with GPT-3.5 and 420 with Claude-3.5-Sonnet.

  3. Agentic Program Repair from Test Failures at Scale: A Neuro-symbolic approach with static analysis and test execution feedback

    cs.SE 2025-07 conditional novelty 6.0 of 10

    Meta's Engineering Agent solved 42.3% of curated internal test-failure tasks offline and landed 25.5% of generated fixes in production over three months.

  4. Can LLMs Hack Enterprise Networks? Autonomous Assumed Breach Penetration-Testing Active Directory Networks

    cs.CR 2025-02 conditional novelty 6.0 of 10

    An autonomous LLM-driven agent can compromise accounts in a realistic Active Directory testbed, with reasoning models outperforming non-reasoning ones at competitive cost.

  5. Beyond Accuracy: Behavioral Dynamics of Agentic Multi-Hunk Repair

    cs.SE 2025-11 conditional novelty 5.0 of 10

    On 372 multi-hunk bugs, coding agents repair between 26% and 93% of defects, accuracy falls as edits become more divergent and dispersed, and failed repairs consume up to 343% more tokens.

Reference graph

Works this paper leans on

32 extracted references · 20 canonical work pages · cited by 5 Pith papers

  1. [1]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. [Online]. Available: https: //openreview.net/forum?id=VTF8yNQM66

    2024

  2. [2]

    Swe-agent: Agent-computer interfaces enable automated software engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated software engineering,” arXiv preprint arXiv:2405.15793 , 2024

    arXiv 2024

  3. [3]

    Autocoderover: Autonomous program improvement,

    Y . Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis , 2024, pp. 1592–1604

    2024

  4. [4]

    Repairagent: An autonomous, llm-based agent for program repair,

    I. Bouzenia, P. Devanbu, and M. Pradel, “Repairagent: An autonomous, llm-based agent for program repair,” arXiv preprint arXiv:2403.17134 , 2024

    arXiv 2024

  5. [5]

    Coder: Issue resolving with multi-agent and task graphs,

    D. Chen, S. Lin, M. Zeng, D. Zan, J.-G. Wang, A. Cheshkov, J. Sun, H. Yu, G. Dong, A. Alievet al., “Coder: Issue resolving with multi-agent and task graphs,” arXiv preprint arXiv:2406.01304 , 2024

    arXiv 2024

  6. [6]

    Autodev: Automated ai-driven development,

    M. Tufano, A. Agarwal, J. Jang, R. Z. Moghaddam, and N. Sun- daresan, “Autodev: Automated ai-driven development,” arXiv preprint arXiv:2403.08299, 2024

    arXiv 2024

  7. [7]

    Opendevin: An open platform for ai software developers as generalist agents,

    X. Wang, B. Li, Y . Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y . Song, B. Li, J. Singh et al. , “Opendevin: An open platform for ai software developers as generalist agents,” arXiv preprint arXiv:2407.16741, 2024

    arXiv 2024

  8. [8]

    Specrover: Code intent extraction via llms,

    H. Ruan, Y . Zhang, and A. Roychoudhury, “Specrover: Code intent extraction via llms,” arXiv preprint arXiv:2408.02232 , 2024

    arXiv 2024

Show all 32 references
  1. [9]

    Marscode agent: Ai- native automated bug fixing,

    Y . Liu, P. Gao, X. Wang, C. Peng, and Z. Zhang, “Marscode agent: Ai- native automated bug fixing,” arXiv preprint arXiv:2409.00899 , 2024

    2024 arXiv

  2. [10]

    Why google stores billions of lines of code in a single repository,

    R. Potvin and J. Levenberg, “Why google stores billions of lines of code in a single repository,” Communications of the ACM , vol. 59, no. 7, pp. 78–87, 2016

    2016

  3. [11]

    Distributed Builds,

    Bazel, “Distributed Builds,” https://bazel.build/basics/distributed-builds, 2024, [Online; accessed October 2024]

    2024

  4. [12]

    How developers search for code: a case study,

    C. Sadowski, K. T. Stolee, and S. Elbaum, “How developers search for code: a case study,” in Proceedings of the 2015 10th joint meeting on foundations of software engineering , 2015, pp. 191–201

    2015

  5. [13]

    Harnessing evolution for multi-hunk program repair,

    S. Saha et al., “Harnessing evolution for multi-hunk program repair,” in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 2019, pp. 13–24

    2019

  6. [14]

    Dissection of a bug dataset: Anatomy of 395 patches from defects4j,

    V . Sobreira, T. Durieux, F. Madeiral, M. Monperrus, and M. de Almeida Maia, “Dissection of a bug dataset: Anatomy of 395 patches from defects4j,” in 2018 IEEE 25th international conference on software analysis, evolution and reengineering (SANER) . IEEE, 2018, pp. 130–140

    2018

  7. [15]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,” in The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. [Onlin...

    2023

  8. [16]

    Executable code actions elicit better llm agents,

    X. Wang, Y . Chen, L. Yuan, Y . Zhang, Y . Li, H. Peng, and H. Ji, “Executable code actions elicit better llm agents,” in Proceedings of the 41st International Conference on Machine Learning , ser. ICML’24. JMLR.org, 2025

    2025

  9. [17]

    Building a distributed build system at google scale,

    A. Greenberg, “Building a distributed build system at google scale,” https://gotocon.com/dl/goto-chicago-2016/slides/AysyluGreenberg BuildingADistributedBuildSystemAtGoogleScale.pdf, 2016, [Online; accessed October 2024]

    2016

  10. [18]

    Smart build targets batching service at Google,

    K. Wang, D. Rall, G. Tener, V . Gullapalli, X. Huang, and A. Gad, “Smart build targets batching service at Google,” in 2021 IEEE/ACM 43rd In- ternational Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) . IEEE, 2021, pp. 160–169

    2021

  11. [19]

    An analysis of patch plausibility and correctness for generate-and-validate patch generation systems,

    Z. Qi, F. Long, S. Achour, and M. Rinard, “An analysis of patch plausibility and correctness for generate-and-validate patch generation systems,” in Proceedings of the 2015 International Symposium on Software Testing and Analysis , 2015, pp. 24–36

    2015

  12. [20]

    Identifying patch correctness in test-based program repair,

    Y . Xiong, X. Liu, M. Zeng, L. Zhang, and G. Huang, “Identifying patch correctness in test-based program repair,” in Proceedings of the 40th international conference on software engineering , 2018, pp. 789–799

    2018

  13. [21]

    Exploring the differences between plausible and correct patches at fine-grained level,

    B. Yang and J. Yang, “Exploring the differences between plausible and correct patches at fine-grained level,” in 2020 IEEE 2nd International Workshop on Intelligent Bug Fixing (IBF) . IEEE, 2020, pp. 1–8

    2020

  14. [22]

    On the efficiency of test suite based program repair: A systematic assessment of 16 automated repair systems for java programs,

    K. Liu, S. Wang, A. Koyuncu, K. Kim, T. F. Bissyand ´e, D. Kim, P. Wu, J. Klein, X. Mao, and Y . L. Traon, “On the efficiency of test suite based program repair: A systematic assessment of 16 automated repair systems for java programs,” in Proceedings of the ACM/IEEE 42nd Inte...

    2020

  15. [23]

    A large-scale empirical review of patch correctness checking approaches,

    J. Yang, Y . Wang, Y . Lou, M. Wen, and L. Zhang, “A large-scale empirical review of patch correctness checking approaches,” in Proceed- ings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2023, pp. 1203–1215

    2023

  16. [24]

    Fowler, Refactoring: improving the design of existing code

    M. Fowler, Refactoring: improving the design of existing code . Addison-Wesley Professional, 2018

    2018

  17. [25]

    Gemini 1.5: Our next-generation model, now available for private preview in google ai studio,

    J. Konzelmann and M. Li, “Gemini 1.5: Our next-generation model, now available for private preview in google ai studio,” 2024, [Online; accessed October 2024]. [Online]. Available: ”https://developers.googl eblog.com/en/gemini-15-pro-now-available-in-180-countries-with-nat ive...

    2024

  18. [26]

    Automatic software repair: A bibliography,

    M. Monperrus, “Automatic software repair: A bibliography,” ACM Computing Surveys (CSUR) , vol. 51, no. 1, pp. 1–24, 2018

    2018

  19. [27]

    Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using chatgpt,

    C. S. Xia and L. Zhang, “Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using chatgpt,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024, pp. 819–831

    2024

  20. [28]

    An empirical study on real bug fixes,

    H. Zhong and Z. Su, “An empirical study on real bug fixes,” in 2015 IEEE/ACM 37th IEEE International Conference on Software Engineer- ing, vol. 1. IEEE, 2015, pp. 913–923

    2015

  21. [29]

    Introducing Devin, the first AI software engineer,

    Cognition, “Introducing Devin, the first AI software engineer,” https: //www.cognition.ai/blog/introducing-devin, 2024, [Online; accessed October 2024]

    2024

  22. [30]

    Less training, more repairing please: revisiting automated program repair via zero-shot learning,

    C. S. Xia and L. Zhang, “Less training, more repairing please: revisiting automated program repair via zero-shot learning,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022, pp. 959– 971

    2022

  23. [31]

    Agentless: De- mystifying llm-based software engineering agents,

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Agentless: De- mystifying llm-based software engineering agents,” arXiv preprint arXiv:2407.01489, 2024

    2024 arXiv

  24. [32]

    Large language models for test-free fault localization,

    A. Z. Yang, C. Le Goues, R. Martins, and V . Hellendoorn, “Large language models for test-free fault localization,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering , 2024, pp. 1–12. 12

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.