Pith. sign in

REVIEW 4 major objections 5 minor 72 references

Bug reports for AI repair agents succeed when they point to the fault and a fix, not when they only narrate the failure.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 02:11 UTC pith:HRDZKGKU

load-bearing objection Solid empirical SE paper: localization and suggested fixes help mini-swe-agent more than classic human report elements; ablation n=65 is the softest load-bearing piece, not a reason to ignore the work. the 4 major comments →

arxiv 2607.09553 v1 pith:HRDZKGKU submitted 2026-07-10 cs.SE

Writing Bug Reports for Software Repair Agents: What Information Matters Most?

Vincenzo Luigi Bruno , Alessandro Giagnorio , Daniele Bifolco , Leon Wienges , Massimiliano Di Penta , Gabriele Bavota This is my paper
classification cs.SE
keywords agentic-first issuesbug report qualitysoftware repair agentslocalization cuessuggested fixesSWE-bench Verifiedablation studybinomial regression
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

As software teams assign bugs first to AI repair agents rather than human developers, the bug report becomes the agent's task brief. This paper asks which pieces of that brief actually raise the chance of a correct automated fix. The authors annotate hundreds of real repository bug reports by information type, run a repair agent with three language-model backbones, and model which labels predict success while holding issue length and fix difficulty fixed. They find that localization cues and suggested fixes, whether written as code or in plain language, are most strongly tied to success. Classic human-oriented content such as reproduction steps and expected behavior matter less once those operational cues are present. An ablation that strips selected sentence types from the same reports confirms the pattern: removing both localization and suggested fixes together hurts most. The practical claim is that agent-ready reports should be written as repair specifications, not exhaustive failure narratives.

Core claim

Agentic-first bug reports raise repair success when they narrow the agent's search and repair space. Mentions of affected lines or functions, and suggested fixes in code or natural language, show the strongest positive associations with a passing patch; removing both localization and suggestions together produces the largest drop in success, while dropping reproduction steps or expected behavior alone does not.

What carries the argument

Character-level information-type annotation of bug reports, combined with mixed-effect binomial regression of agent pass/fail and a controlled sentence-level ablation on information-complete issues, isolates the incremental association of each content type with repair success.

Load-bearing premise

That binary presence of each information type in the original issue body, measured with one agent architecture and three specific models on one benchmark's bugs, is enough to say what makes reports agent-ready in general.

What would settle it

Rerun the same annotation and ablation protocol with a different agent scaffold or a broader set of repositories and models; if localization and suggested-fix presence no longer predict pass rate once issue difficulty is controlled, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies which information types in bug reports are associated with success of LLM-based software repair agents. From SWE-bench Verified, the authors retain 441 bug reports, annotate them at character level with a literature-derived taxonomy (observed/expected behavior, reproduction steps, localization at several granularities, suggested fixes, etc.), and run mini-swe-agent with three LLM backbones (GPT-5-mini, MiniMax M2.5, Gemini 3 Flash). A mixed-effect binomial model with issue-level controls and repository random effects estimates associations between binary information-type indicators and pass@test success. A complementary ablation on 65 information-complete issues removes macro-categories of content. The main claim is that agent-ready reports benefit most from localization cues and suggested fixes (code or natural language), which narrow the search/repair space, while classic human-oriented elements such as reproduction steps are less decisive when operational cues remain.

Significance. If the result holds, it is a timely and practically useful contribution for agentic-first development: issue templates and reporting guidelines would need to shift from exhaustive human narratives toward repair specifications that expose where to look and how to fix. Strengths include dual independent annotation with third-author conflict resolution, character-level labeling, a literature-grounded taxonomy, careful mixed-effect modeling with diagnostics (no singular fit, no overdispersion, VIF < 3), both pooled and per-LLM analyses, a complementary ablation design, and a public replication package. The work cleanly separates human-oriented bug-report quality from agent-oriented task specification and gives concrete, falsifiable guidance for templates and multi-stage agent designs.

major comments (4)
  1. [Abstract; §III-B Tables V–VI; §VII] Abstract and §VII state that the ablation “confirms” agents benefit less from human-oriented cues and more from localization/suggested fixes. In Tables V–VI, single removals of localization or suggestions are non-significant or only marginal (e.g., no suggestions OR=0.78, p=0.093); only joint removal of both is clearly significant (OR=0.60, p<0.001). The stronger causal-sounding claim therefore rests almost entirely on the joint-removal treatment over n=65 complete issues. Soften abstract/conclusion language to match this nuance, and report power or sensitivity of the joint-removal result to the complete-issue filter and macro-category definitions.
  2. [§III-A Tables III–IV; §III-B; §IV] The observational model (Tables III–IV) finds significant positive ORs for individual localization and suggested-fix indicators, while the ablation finds that removing either alone does not significantly hurt success when the other remains. §III-B discusses this with a Django example, but the Implications (§IV) and conclusion still treat “localization and suggested fixes” as the decisive pair without reconciling substitutability. Clarify whether the practical recommendation is “provide at least one of {localization, suggested fix}” versus “provide both,” and state how that reading is supported by each analysis.
  3. [§II-C1; Table I; §VI] All predictors of interest are binary presence indicators (§II-C1; Threats §VI). Quality, completeness, and precision of a localization cue or suggested fix are not modeled, yet the central question is “what information matters most.” Binary encoding is a load-bearing modeling choice: a vague “somewhere in the parser” and a precise line reference both count as localization. Either add a coarse quality/granularity analysis (e.g., using the existing line|function|file|generic levels more fully in the ablation) or substantially strengthen the limitation and restrict claims to presence, not informativeness.
  4. [§II-B; §IV; §VI] External validity for “software repair agents” rests on one scaffold (mini-swe-agent, bash-only ReAct, max 75 steps, medium reasoning) and three backbones on SWE-bench Verified (§II-B, §VI). The invented framing “agentic-first issue report” and template recommendations in §IV generalize beyond this setup. Either run a second agent architecture on a subset, or reframe claims as specific to this class of agents and make the transfer assumption explicit as a primary threat rather than a brief external-validity note.
minor comments (5)
  1. [Table II; §III] Table II shows extreme imbalance (summary 100%, observed behavior 97.96%, file-level localization 4.99%). The text notes limited power for rare labels; consider marking under-powered predictors more visibly in Tables III–IV (e.g., sample counts or a footnote).
  2. [§II-A3] Cohen’s κ=0.57 for issue-level label presence is moderate; character-level agreement is 78.6%. A short per-label κ or confusion summary in the replication package or appendix would help readers judge which information types are hardest to annotate.
  3. [§III-A; §III-B] Issue length flips sign between observational (OR=0.73) and ablation (OR=1.13). The explanation in §III-B is plausible but easy to miss; flag it earlier when introducing controls so readers do not treat length as a stable effect.
  4. [§II-C2] Macro-category grouping (Bug location, Possible fix, Behavioral context) is reasonable but ad hoc (§II-C2). A one-sentence justification for grouping expected behavior with reproduction steps (rather than with observed behavior) would help.
  5. [Introduction; §II] Minor wording: “odd of success” should be “odds of success” (Introduction); “Gemini Flash 3” vs “Gemini 3 Flash” is inconsistent across sections.

Circularity Check

0 steps flagged

No circularity: associations and ablations are measured against external test-suite pass/fail, not defined by the information-type labels.

full rationale

This is an observational-plus-ablation empirical study, not a derivation. Information-type labels are taken from prior human bug-report literature and applied by independent manual annotation of issue text; agent success is defined by whether mini-swe-agent patches pass the SWE-bench Verified task-specific test suites. The binomial mixed-effect models estimate incremental associations of binary presence indicators (and of ablation treatments) with that external binary outcome, controlling for issue length, gold-patch size, difficulty, code mention, LLM backbone, and repository random effects. Nothing in the chain equates a fitted coefficient or a removed span to the success label by construction; there is no uniqueness theorem, no ansatz imported via self-citation, and no renaming of a known pattern as a first-principles result. The framing term “agentic-first” is definitional packaging only and does not force the reported odds ratios. Score 0 is therefore the correct finding.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 1 invented entities

The central claim rests on standard empirical-SE assumptions and experimental choices, not on free physical constants or invented particles. Load-bearing premises are: test-suite pass equals repair success; gold patches and human difficulty labels are valid controls; binary presence of literature-derived information types is a meaningful predictor; mini-swe-agent with three named LLMs represents agentic repair well enough for the claim; and SWE-bench Verified issues are a suitable population. No new physical entities; the ‘agentic-first issue report’ is a framing concept, not an unobserved mechanism required for the statistics to hold.

free parameters (4)
  • max_agent_steps
    Hard cap of 75 steps (following prior work) truncates long trajectories; 9.5% of runs hit the cap, so success rates depend on this choice.
  • reasoning_effort=medium
    Model reasoning setting chosen to contain cost; not swept, so associations could shift under different effort settings.
  • significance_threshold_0.05
    Standard α used to declare associations; several borderline effects (e.g., file-level location, suggestions-only ablation) sit near this cutoff.
  • macro_category_grouping_for_ablation
    Localization levels and fix modalities are collapsed into macro-categories; alternative groupings would change which treatments are tested.
axioms (6)
  • domain assumption Passing the SWE-bench Verified task-specific test suite is a valid measure of successful repair for the study’s dependent variable.
    Used throughout §II-C and results; standard for the benchmark but imperfect (tests may miss behavioral correctness).
  • domain assumption The nine information types adapted from prior human bug-report literature (Table I) are the right inventory for agent-relevant content.
    Annotation schema in §II-A2; excluded types (severity, assignee, etc.) are assumed irrelevant to agents.
  • ad hoc to paper Binary indicators for presence of each information type, plus log-length and log gold-patch size controls, adequately capture issue content for mixed-effect binomial modeling.
    Modeling choice in §II-C1; quality/detail of each span is not modeled (acknowledged in Threats).
  • domain assumption Repository-level random intercepts and the four issue-level controls sufficiently adjust for confounding between information content and success.
    §II-C1 and diagnostics; residual confounding remains possible (Threats §VI).
  • ad hoc to paper mini-swe-agent’s bash-only ReAct-style loop with the three chosen LLM backbones is representative enough to support claims about ‘software repair agents’.
    §II-B; external validity threat if other scaffolds or models use issue text differently.
  • standard math Standard mixed-effect binomial / odds-ratio inference (Wald CIs, VIF, overdispersion checks) is appropriate for the repeated-run design.
    §II-D; conventional GLM methodology.
invented entities (1)
  • agentic-first issue report no independent evidence
    purpose: Names the paper’s framing: issues written as task specs for repair agents, not only for human maintainers.
    Conceptual label introduced in the abstract/intro and contributions; not a latent physical mechanism. Independent evidence is the empirical association study itself, not an external measurement of a new object.

pith-pipeline@v1.1.0-grok45 · 27091 in / 3725 out tokens · 43988 ms · 2026-07-13T02:11:04.582776+00:00 · methodology

0 comments
read the original abstract

Software development is increasingly moving toward agentic-first workflows. This includes AI agents responsible for generating initial fixes for submitted issue reports. In this setting, issue reports are no longer merely documentation for human maintainers; they become the primary task specification for the agent. However, little is known about how such reports should be written to maximize the agent's chances of producing a correct fix. We study what makes a bug report agent-ready. Starting from the SWE-bench Verified benchmark (i.e., a collection of 500 real repository issues with human-written gold patches and test suites for evaluating generated fixes) we manually classify each issue by change type (e.g., bug fix vs refactoring) and annotate each sentence with its information type, such as observed behavior, expected behavior, reproduction steps, localization cues, and suggested fixes. We focus on the 441 issues representing bug reports, and we run on them mini-swe-agent using three LLM backbones (i.e., GPT-5-mini, MiniMax M2.5, and Gemini 3 Flash). We then fit a binomial regression model to estimate the incremental association between each information type and agent success, controlling for confounding factors. Our results suggest that agentic-first reports benefit most from information that narrows the agent's search and repair space. Localization cues, such as references to affected code areas, are positively associated with successful repairs, while suggested fixes, expressed either in code or natural language, show some of the strongest positive associations with pass probability. An ablation study removing selected information types confirms that agents benefit less from information traditionally useful to humans, such as reproduction steps, and more from sentences that expose a repair direction, either through bug localization or a suggested fix.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

72 extracted references · 7 canonical work pages

  1. [1]

    Autocoderover: Autonomous program improvement,

    Y . Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2024. New York, NY , USA: Association for Computing Machinery, 2024, p. 1592–1604. [Online]. Available: https://doi.org/10.1145/3650212.3680384

    doi:10.1145/3650212.3680384 2024

  2. [2]

    Specrover: Code intent extraction via llms,

    H. Ruan, Y . Zhang, and A. Roychoudhury, “Specrover: Code intent extraction via llms,” inProceedings of the IEEE/ACM 47th International Conference on Software Engineering, ser. ICSE ’25. IEEE Press, 2025, p. 963–974. [Online]. Available: https: //doi.org/10.1109/ICSE55347.2025.00080

    doi:10.1109/icse55347.2025.00080 2025

  3. [3]

    Alibaba lingmaagent: Improving automated issue resolution via comprehensive repository exploration,

    Y . Ma, Q. Yang, R. Cao, B. Li, F. Huang, and Y . Li, “Alibaba lingmaagent: Improving automated issue resolution via comprehensive repository exploration,” inProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, ser. FSE Companion ’25. New York, NY , USA: Association for Computing Machinery, 2025, p. 238–249. [On...

    doi:10.1145/3696630.3728549 2025

  4. [4]

    What makes a good bug report?

    T. Zimmermann, R. Premraj, N. Bettenburg, S. Just, A. Schroter, and C. Weiss, “What makes a good bug report?”IEEE Trans. Softw. Eng., vol. 36, no. 5, p. 618–643, Sep. 2010. [Online]. Available: https://doi.org/10.1109/TSE.2010.63

    doi:10.1109/tse.2010.63 2010

  5. [5]

    SWE-bench: Can language models resolve real- world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “SWE-bench: Can language models resolve real- world github issues?” inThe Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https: //openreview.net/forum?id=VTF8yNQM66

    2024

  6. [6]

    The significance of bug report elements,

    M. Soltani, F. Hermans, and T. B ¨ack, “The significance of bug report elements,”Empirical Softw. Engg., vol. 25, no. 6, p. 5255–5294, Nov

  7. [7]

    Available: https://doi.org/10.1007/s10664-020-09882-z

    [Online]. Available: https://doi.org/10.1007/s10664-020-09882-z

    doi:10.1007/s10664-020-09882-z

  8. [8]

    To Follow or Not to Follow: Understanding Issue/Pull-Request Templates on GitHub ,

    Z. Li, Y . Yu, T. Wang, Y . Lei, Y . Wang, and H. Wang, “ To Follow or Not to Follow: Understanding Issue/Pull-Request Templates on GitHub ,”IEEE Transactions on Software Engineering, vol. 49, no. 04, pp. 2530–2544, Apr. 2023. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/TSE.2022.3224053

    doi:10.1109/tse.2022.3224053 2023

  9. [9]

    A first look at bug report templates on github,

    H. Li, M. Yan, W. Sun, X. Liu, and Y . Wu, “A first look at bug report templates on github,”J. Syst. Softw., vol. 202, no. C, Aug. 2023. [Online]. Available: https://doi.org/10.1016/j.jss.2023.111709

    work page doi:10.1016/j.jss.2023.111709 2023

  10. [10]

    An empirical analysis of issue templates usage in large-scale projects on github,

    E. S ¨ul¨un, M. Sac ¸akc ¸ı, and E. T ¨uz¨un, “An empirical analysis of issue templates usage in large-scale projects on github,”ACM Trans. Softw. Eng. Methodol., vol. 33, no. 5, Jun. 2024. [Online]. Available: https://doi.org/10.1145/3643673

    work page doi:10.1145/3643673 2024

  11. [11]

    mini-swe-agent,

    “mini-swe-agent,” https://github.com/SWE-agent/mini-swe-agent, [n.d.]

  12. [12]

    Gpt-5-mini,

    “Gpt-5-mini,” https://developers.openai.com/api/docs/models/ gpt-5-mini

  13. [13]

    Minimax m2.5,

    “Minimax m2.5,” https://huggingface.co/MiniMaxAI/MiniMax-M2.5

  14. [14]

    Gemini 3 flash,

    “Gemini 3 flash,” https://ai.google.dev/gemini-api/docs/models/ gemini-3-flash-preview

  15. [15]

    McCullagh and J

    P. McCullagh and J. A. Nelder,Generalized Linear Models, 2nd ed. Chapman and Hall, 1989

    1989

  16. [16]

    Survey reproduction of defect reporting in industrial software development,

    E. I. Laukkanen and M. V . Mantyla, “Survey reproduction of defect reporting in industrial software development,” in2011 International Symposium on Empirical Software Engineering and Measurement, 2011, pp. 197–206

    2011

  17. [17]

    Replication package,

    “Replication package,” https://doi.org/10.5281/zenodo.21035233, [n.d.]

    work page doi:10.5281/zenodo.21035233

  18. [18]

    A coefficient of agreement for nominal scales,

    J. Cohen, “A coefficient of agreement for nominal scales,”Educ Psychol Meas., 1960

    1960

  19. [19]

    Yet even less is even better for agentic, reasoning, and coding llms,

    CodeArts Model Team, Y . Ye, J. Tan, T. Jiang, R. Ye, Q. He, J. Yang, J. Dong, S. Liang, C. Yueet al., “Yet even less is even better for agentic, reasoning, and coding llms,”arXiv preprint arXiv:2604.00824, 2026

    Pith/arXiv arXiv 2026

  20. [20]

    Artisan: Agentic artifact evaluation,

    D. Baek and M. Pradel, “Artisan: Agentic artifact evaluation,”arXiv preprint arXiv:2602.10046, 2026

    arXiv 2026

  21. [21]

    Live-swe-agent: Can software engineering agents self-evolve on the fly?

    C. S. Xia, Z. Wang, Y . Yang, Y . Wei, and L. Zhang, “Live-swe-agent: Can software engineering agents self-evolve on the fly?”arXiv preprint arXiv:2511.13646, 2025

    arXiv 2025

  22. [22]

    Swe-bench leaderboard,

    “Swe-bench leaderboard,” https://www.swebench.com/, [n.d.]

  23. [23]

    SWE-smith: Scaling data for software engineering agents,

    J. Yang, K. Lieret, C. E. Jimenez, A. Wettig, K. Khandpur, Y . Zhang, B. Hui, O. Press, L. Schmidt, and D. Yang, “SWE-smith: Scaling data for software engineering agents,” inAdvances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2025, spotlight. [Online]. Available: https://arxiv.org/abs/2504.21798

    Pith/arXiv arXiv 2025

  24. [24]

    F. E. Harrell Jr,Hmisc: Harrell Miscellaneous, 2023, r package version 5.1-0. [Online]. Available: https://CRAN.R-project.org/package=Hmisc

    2023

  25. [25]

    R Core Team,R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria,

  26. [26]

    Available: https://www.R-project.org/

    [Online]. Available: https://www.R-project.org/

  27. [27]

    Tests of statistical hypotheses concerning several parameters when the number of observations is large,

    A. Wald, “Tests of statistical hypotheses concerning several parameters when the number of observations is large,”Transactions of the American Mathematical Society, vol. 54, no. 3, pp. 426–482, 1943

    1943

  28. [28]

    Using static analysis to find bugs,

    N. Ayewah, W. Pugh, D. Hovemeyer, J. D. Morgenthaler, and J. Penix, “Using static analysis to find bugs,”IEEE Software, vol. 25, no. 5, pp. 22–29, 2008

    2008

  29. [29]

    A few billion lines of code later: using static analysis to find bugs in the real world,

    A. Bessey, K. Block, B. Chelf, A. Chou, B. Fulton, S. Hallem, C. Henri-Gros, A. Kamsky, S. McPeak, and D. Engler, “A few billion lines of code later: using static analysis to find bugs in the real world,”Commun. ACM, vol. 53, no. 2, p. 66–75, Feb. 2010. [Online]. Available: https://doi.org/10.1145/1646353.1646374

    doi:10.1145/1646353.1646374 2010

  30. [30]

    Where should the bugs be fixed? - more accurate information retrieval-based bug localization based on bug reports,

    J. Zhou, H. Zhang, and D. Lo, “Where should the bugs be fixed? - more accurate information retrieval-based bug localization based on bug reports,” inProceedings of the 34th International Conference on Software Engineering, ser. ICSE ’12. IEEE Press, 2012, p. 14–24

    2012

  31. [31]

    Improving bug localization using structured information retrieval,

    R. K. Saha, M. Lease, S. Khurshid, and D. E. Perry, “Improving bug localization using structured information retrieval,” inProceedings of the 28th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’13. IEEE Press, 2013, p. 345–355. [Online]. Available: https://doi.org/10.1109/ASE.2013.6693093

    doi:10.1109/ase.2013.6693093 2013

  32. [32]

    Learning to rank relevant files for bug reports using domain knowledge,

    X. Ye, R. Bunescu, and C. Liu, “Learning to rank relevant files for bug reports using domain knowledge,” inProceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, ser. FSE 2014. New York, NY , USA: Association for Computing Machinery, 2014, p. 689–699. [Online]. Available: https://doi.org/10.1145/2635868.2635874

    doi:10.1145/2635868.2635874 2014

  33. [33]

    Empirical evaluation of the tarantula automatic fault-localization technique,

    J. A. Jones and M. J. Harrold, “Empirical evaluation of the tarantula automatic fault-localization technique,” inProceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’05. New York, NY , USA: Association for Computing Machinery, 2005, p. 273–282. [Online]. Available: https://doi.org/10.1145/1101908.1101949 11

    Pith/arXiv arXiv doi:10.1145/1101908.1101949 2005

  34. [34]

    A practical evaluation of spectrum-based fault localization,

    R. Abreu, P. Zoeteweij, R. Golsteijn, and A. J. C. van Gemund, “A practical evaluation of spectrum-based fault localization,”J. Syst. Softw., vol. 82, no. 11, p. 1780–1792, Nov. 2009. [Online]. Available: https://doi.org/10.1016/j.jss.2009.06.035

    doi:10.1016/j.jss.2009.06.035 2009

  35. [35]

    Crashlocator: locating crashing faults based on crash stacks,

    R. Wu, H. Zhang, S.-C. Cheung, and S. Kim, “Crashlocator: locating crashing faults based on crash stacks,” inProceedings of the 2014 International Symposium on Software Testing and Analysis, ser. ISSTA 2014. New York, NY , USA: Association for Computing Machinery, 2014, p. 204–214. [Online]. Available: https://doi.org/10.1145/2610384.2610386

    doi:10.1145/2610384.2610386 2014

  36. [36]

    On the use of stack traces to improve text retrieval-based bug localization,

    L. Moreno, J. J. Treadway, A. Marcus, and W. Shen, “On the use of stack traces to improve text retrieval-based bug localization,” in2014 IEEE International Conference on Software Maintenance and Evolution, 2014, pp. 151–160

    2014

  37. [37]

    Demystifying llm-based software engineering agents,

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Demystifying llm-based software engineering agents,”Proc. ACM Softw. Eng., vol. 2, no. FSE, Jun. 2025. [Online]. Available: https://doi.org/10.1145/3715754

    doi:10.1145/3715754 2025

  38. [38]

    Flexfl: Flexible and effective fault localization with open-source large language models,

    C. Xu, Z. Liu, X. Ren, G. Zhang, M. Liang, and D. Lo, “Flexfl: Flexible and effective fault localization with open-source large language models,”IEEE Trans. Softw. Eng., vol. 51, no. 5, p. 1455–1471, May

  39. [39]

    Available: https://doi.org/10.1109/TSE.2025.3553363

    [Online]. Available: https://doi.org/10.1109/TSE.2025.3553363

    doi:10.1109/tse.2025.3553363 2025

  40. [40]

    S¡sc¿oap¡/sc¿fl: A standard operating procedure for llm- based method-level fault localization,

    Y . Qin, S. Wang, Y . Lou, J. Dong, K. Wang, X. Li, and X. Mao, “S¡sc¿oap¡/sc¿fl: A standard operating procedure for llm- based method-level fault localization,”IEEE Trans. Softw. Eng., vol. 51, no. 4, p. 1173–1187, Apr. 2025. [Online]. Available: https://doi.org/10.1109/TSE.2025.3543187

    doi:10.1109/tse.2025.3543187 2025

  41. [41]

    What makes a satisficing bug report?

    T. Dal Sasso, A. Mocci, and M. Lanza, “What makes a satisficing bug report?” in2016 IEEE International Conference on Software Quality, Reliability and Security (QRS), 2016, pp. 164–174

    2016

  42. [42]

    What’s in a bug report?

    S. Davies and M. Roper, “What’s in a bug report?” inProceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ser. ESEM ’14. New York, NY , USA: Association for Computing Machinery, 2014. [Online]. Available: https://doi.org/10.1145/2652524.2652541

    work page doi:10.1145/2652524.2652541 2014

  43. [43]

    An empirical study of the effects of expert knowledge on bug reports,

    D. Huo, T. Ding, C. McMillan, and M. Gethers, “An empirical study of the effects of expert knowledge on bug reports,” inProceedings of the 2014 IEEE International Conference on Software Maintenance and Evolution, ser. ICSME ’14. USA: IEEE Computer Society, 2014, p. 1–10. [Online]. Available: https://doi.org/10.1109/ICSME.2014.22

    work page doi:10.1109/icsme.2014.22 2014

  44. [44]

    Works for me! characterizing non-reproducible bug reports,

    M. Erfani Joorabchi, M. Mirzaaghaei, and A. Mesbah, “Works for me! characterizing non-reproducible bug reports,” inProceedings of the 11th Working Conference on Mining Software Repositories, ser. MSR 2014. New York, NY , USA: Association for Computing Machinery, 2014, p. 62–71. [Online]. Available: https://doi.org/10.1145/2597073.2597098

    doi:10.1145/2597073.2597098 2014

  45. [45]

    Detecting missing information in bug descriptions,

    O. Chaparro, J. Lu, F. Zampetti, L. Moreno, M. Di Penta, A. Marcus, G. Bavota, and V . Ng, “Detecting missing information in bug descriptions,” inProceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ser. ESEC/FSE 2017. New York, NY , USA: Association for Computing Machinery, 2017, p. 396–407. [Online]. Available: https://doi....

    doi:10.1145/3106237.3106285 2017

  46. [46]

    Assessing the quality of the steps to reproduce in bug reports,

    O. Chaparro, C. Bernal-C ´ardenas, J. Lu, K. Moran, A. Marcus, M. Di Penta, D. Poshyvanyk, and V . Ng, “Assessing the quality of the steps to reproduce in bug reports,” inProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2019. New York, NY , US...

    doi:10.1145/3338906.3338947 2019

  47. [47]

    Bee: a tool for structuring and analyzing bug reports,

    Y . Song and O. Chaparro, “Bee: a tool for structuring and analyzing bug reports,” inProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2020. New York, NY , USA: Association for Computing Machinery, 2020, p. 1551–1555. [Online]. Available: https://do...

    work page doi:10.1145/3368089.3417928 2020

  48. [48]

    Auto-completing bug reports for android applications,

    K. Moran, M. Linares-V ´asquez, C. Bernal-C ´ardenas, and D. Poshyvanyk, “Auto-completing bug reports for android applications,” inProceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ser. ESEC/FSE 2015. New York, NY , USA: Association for Computing Machinery, 2015, p. 673–686. [Online]. Available: https://doi.org/10.1145/278...

    doi:10.1145/2786805.2786857 2015

  49. [49]

    Burt: A chatbot for interactive bug reporting,

    Y . Song, J. Mahmud, N. De Silva, Y . Zhou, O. Chaparro, K. Moran, A. Marcus, and D. Poshyvanyk, “Burt: A chatbot for interactive bug reporting,” inProceedings of the 45th International Conference on Software Engineering: Companion Proceedings, ser. ICSE ’23. IEEE Press, 2023, p. 170–174. [Online]. Available: https://doi.org/10.1109/ICSE-Companion58688.2023.00048

    work page doi:10.1109/icse-companion58688.2023.00048 2023

  50. [50]

    Chatbr: Automated assessment and improvement of bug report quality using chatgpt,

    L. Bo, W. Ji, X. Sun, T. Zhang, X. Wu, and Y . Wei, “Chatbr: Automated assessment and improvement of bug report quality using chatgpt,” inProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’24. New York, NY , USA: Association for Computing Machinery, 2024, p. 1472–1483. [Online]. Available: https://doi.or...

    doi:10.1145/3691620.3695518 2024

  51. [51]

    Prompting is all you need: Automated android bug replay with large language models,

    S. Feng and C. Chen, “Prompting is all you need: Automated android bug replay with large language models,” inProceedings of the IEEE/ACM 46th International Conference on Software Engineering, ser. ICSE ’24. New York, NY , USA: Association for Computing Machinery,

  52. [52]

    Available: https://doi.org/10.1145/3597503.3608137

    [Online]. Available: https://doi.org/10.1145/3597503.3608137

    doi:10.1145/3597503.3608137

  53. [53]

    Combining language and app ui analysis for the automated assessment of bug reproduction steps,

    J. Mahmud, A. Saha, O. Chaparro, K. Moran, and A. Marcus, “Combining language and app ui analysis for the automated assessment of bug reproduction steps,” in2025 IEEE/ACM 33rd International Conference on Program Comprehension (ICPC). IEEE Press, 2025, p. 1–12. [Online]. Available: https://doi.org/10.1109/ICPC66645.2025. 00010

    doi:10.1109/icpc66645.2025 2025

  54. [54]

    ImproBR: Bug report improver using LLMs,

    E. F. Akyol, M. Dedeler, and E. T ¨uz¨un, “ImproBR: Bug report improver using LLMs,” inProceedings of the 30th International Conference on Evaluation and Assessment in Software Engineering (EASE), 2026

    2026

  55. [55]

    A survey of learning-based automated program repair,

    Q. Zhang, C. Fang, Y . Ma, W. Sun, and Z. Chen, “A survey of learning-based automated program repair,”ACM Trans. Softw. Eng. Methodol., vol. 33, no. 2, Dec. 2023. [Online]. Available: https://doi.org/10.1145/3631974

    doi:10.1145/3631974 2023

  56. [56]

    A systematic literature review on large language models for automated program repair,

    Q. Zhang, C. Fang, Y . Xie, Y . Ma, W. Sun, Y . Yang, and Z. Chen, “A systematic literature review on large language models for automated program repair,”ACM Trans. Softw. Eng. Methodol., Mar. 2026, just Accepted. [Online]. Available: https://doi.org/10.1145/3799693

    doi:10.1145/3799693 2026

  57. [57]

    Inside the scaffold: A source-code taxonomy of coding agent architectures,

    B. Rombaut, “Inside the scaffold: A source-code taxonomy of coding agent architectures,” 2026. [Online]. Available: https://arxiv.org/abs/ 2604.03515

    Pith/arXiv arXiv 2026

  58. [58]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,” in International Conference on Learning Representations (ICLR), 2023

    2023

  59. [59]

    SWE-agent: Agent-computer interfaces enable automated software engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. R. Narasimhan, and O. Press, “SWE-agent: Agent-computer interfaces enable automated software engineering,” inThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [Online]. Available: https://arxiv.org/abs/2405.15793

    Pith/arXiv arXiv 2024

  60. [60]

    Openhands: An open platform for AI software developers as generalist agents,

    X. Wang, B. Li, Y . Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y . Song, B. Li, J. Singh, H. H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, Y . Shao, N. Muennighoff, Y . Zhang, B. Hui, J. Lin, R. Brennan, H. Peng, H. Ji, and G. Neubig, “Openhands: An open platform for AI software developers as generalist agents,” inThe Thirteenth International Conference on Le...

    2025

  61. [61]

    Codex CLI,

    OpenAI, “Codex CLI,” https://github.com/openai/codex, 2025

    2025

  62. [62]

    Gemini CLI,

    Google, “Gemini CLI,” https://github.com/google-gemini/gemini-cli, 2025

    2025

  63. [63]

    OpenCode,

    Anomaly Innovations, “OpenCode,” https://github.com/sst/opencode, 2025

    2025

  64. [64]

    Swe-search: Enhancing software agents with monte carlo tree search and iterative refinement,

    A. Antoniades, A. ¨Orwall, K. Zhang, Y . Xie, A. Goyal, and W. Wang, “Swe-search: Enhancing software agents with monte carlo tree search and iterative refinement,” inInternational Conference on Learning Representations, Y . Yue, A. Garg, N. Peng, F. Sha, and R. Yu, Eds., vol. 2025, 2025, pp. 64 485–64 515. [Online]. Available: https://proceedings.iclr.cc/...

    2025

  65. [65]

    DARS: Dynamic action re-sampling to enhance coding agent performance by adaptive tree traversal,

    V . Aggarwal, O. Kamal, A. Japesh, Z. Jin, and B. Sch ¨olkopf, “DARS: Dynamic action re-sampling to enhance coding agent performance by adaptive tree traversal,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, Eds. Vienna, Austria: Associ...

    2025

  66. [66]

    Trae agent: An llm-based agent for software engineering with test-time scaling,

    T. R. Team, P. Gao, Z. Tian, X. Meng, X. Wang, R. Hu, Y . Xiao, Y . Liu, Z. Zhang, J. Chen, C. Gao, Y . Lin, Y . Xiong, C. Peng, and X. Liu, “Trae agent: An llm-based agent for software engineering with test-time scaling,” 2025. [Online]. Available: https://arxiv.org/abs/2507.23370

    Pith/arXiv arXiv 2025

  67. [67]

    Magis: Llm-based multi-agent framework for github issue resolution,

    W. Tao, Y . Zhou, Y . Wang, W. Zhang, H. Zhang, and Y . Cheng, “Magis: Llm-based multi-agent framework for github issue resolution,” inPro- ceedings of the 38th International Conference on Neural Information 12 Processing Systems, ser. NIPS ’24. Red Hook, NY , USA: Curran Associates Inc., 2024

    2024

  68. [68]

    Coder: Issue resolving with multi-agent and task graphs,

    D. Chen, S. Lin, M. Zeng, D. Zan, J.-G. Wang, A. Cheshkov, J. Sun, H. Yu, G. Dong, A. Aliev, J. Wang, X. Cheng, G. Liang, Y . Ma, P. Bian, T. Xie, and Q. Wang, “Coder: Issue resolving with multi-agent and task graphs,” 2024. [Online]. Available: https://arxiv.org/abs/2406.01304

    Pith/arXiv arXiv 2024

  69. [69]

    MASAI: Modular architecture for software-engineering AI agents,

    D. Arora, A. Sonwane, N. Wadhwa, A. Mehrotra, S. Utpala, R. Bairi, A. Kanade, and N. Natarajan, “MASAI: Modular architecture for software-engineering AI agents,” inNeurIPS 2024 Workshop on Open-World Agents, 2024. [Online]. Available: https://openreview.net/ forum?id=NSINt8lLYB

    2024

  70. [70]

    Swe-exp: Experience-driven software issue resolution,

    S. Chen, S. Lin, Y . Shi, H. Lian, X. Gu, L. Yun, D. Chen, L. Cao, J. Liu, N. Xia, and Q. Wang, “Swe-exp: Experience-driven software issue resolution,” 2026. [Online]. Available: https://arxiv.org/abs/2507.23361

    arXiv 2026

  71. [71]

    Experepair: Dual-memory enhanced llm-based repository-level program repair,

    F. Mu, J. Wang, L. Shi, S. Wang, S. Li, and Q. Wang, “Experepair: Dual-memory enhanced llm-based repository-level program repair,”

  72. [72]

    Available: https://arxiv.org/abs/2506.10484 13

    [Online]. Available: https://arxiv.org/abs/2506.10484 13

    Pith/arXiv arXiv