Pith. sign in

REVIEW 3 major objections 5 minor 50 references

Agent-generated patch explanations can be untrustworthy even when the patches themselves work, and the paper argues this makes explanation quality an independent axis that benchmarks must measure separately.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 15:32 UTC pith:XLVOGR74

load-bearing objection First automated benchmark for agent explanation quality, with a real finding that explanation quality and patch efficacy diverge; the main risk is that the LLM-scored questions may reward code-context leakage more than the explanation itself. the 3 major comments →

arxiv 2607.26451 v1 pith:XLVOGR74 submitted 2026-07-29 cs.SE

ExplainBench: Evaluating Code Explanations from Agents

Zhiyuan Pan , Sungmin Kang , Imam Nur Bani Yusuf , Abhik Roychoudhury This is my paper
classification cs.SE
keywords LLM agentscode explanationexplainability benchmarkexplanation trustworthinesspatch correctnessSWE-benchmultiple-choice evaluationsoftware engineering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that the quality of a coding agent's natural-language explanation is not the same thing as the quality of the patch it produces, and that this can be measured automatically. It introduces ExplainBench, which turns each patch explanation into a multiple-choice questionnaire about the bug's intended behavior and the patch's actual effect. On five agents, the explanation rankings diverge from the standard patch-resolution ranking, and a striking 79.30% of non-passing patches are still described in a way that lets the question-answering model predict they pass. The paper argues this overconfidence is a real risk to developer trust, and shows that an auditing agent which runs tests can make all evaluated agents' explanations more trustworthy.

Core claim

The central claim is that explanation quality is a distinct, independently measurable axis of agent evaluation. ExplainBench operationalizes quality as 'does the explanation allow an LLM to answer four grounded questions?': end-to-end intent, end-to-end effect, local intent, and local effect. Across five coding agents, the agent with the top patch-resolution rate placed fourth in explanation score, and the agent with the top explanation score placed fourth in resolution, so the two rankings do not coincide. The paper further finds that agent explanations are mostly informative about global intent but weak at local function-level reasoning, and are overconfident: when a patch does not actuall

What carries the argument

The carrying mechanism is the LLM questionnaire: each agent's final-tool-call explanation is fed to a weaker LLM along with grounded context (a property-based test, or the affected function and divergence point), and the model must pick the correct multiple-choice option. The questions are built from a Python execution tracer that finds the first observable behavioral difference between buggy and patched programs, with candidate expressions validated by execution and distractors selected to be plausible. The explanation score is simply the proportion of correctly answered questions, with an explicit 'explanation insufficient to answer' option to separate vagueness from misinformation.

Load-bearing premise

The benchmark's central premise is that a question-answering model's correct answers come from the explanation's content, not from the surrounding code or the model's own training knowledge; if that attribution fails, the scores and rankings do not measure explanation quality.

What would settle it

Swap each agent explanation for a fluent but unrelated explanation of similar length and run ExplainBench again; if scores stay well above the blank-explanation control (0), the questionnaire measures context or probing ability rather than explanation content. A more direct check: remove the code context and give the QA model the explanation alone—if the ranking changes, context was doing the work.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Agent rankings that only count resolved bugs will miss large differences in how trustworthy the accompanying explanations are.
  • Frequent overconfidence in patch correctness means developers cannot rely on an agent's summary that 'the issue is fixed'; automated checks that treat such claims as hypotheses are needed.
  • The benchmark's question-answer format gives researchers a scalar to optimize when designing agents that explain their work.
  • An independent audit agent that executes tests against the explanation can improve explanation scores for all agents evaluated, at roughly $0.05 per explanation.
  • Explanation scores should be reported alongside resolution rates as a second standard metric.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the QA model sees the buggy function and test context alongside the explanation, high scores may partly reflect information in the context rather than the explanation itself; a control that swaps in a wrong but plausible explanation would clarify whether the score isolates explanation content.
  • The 79% overconfidence result suggests a deployable 'explanation calibration' service: if a test can be run, no explanation should be trusted to assert a patch's success until differential testing confirms it.
  • The same questionnaire approach might transfer to non-code natural-language explanations about changes, such as commit messages or design docs, by generating grounded questions from before/after artifacts.
  • The benchmark's reliance on property-based tests and function-level tracing ties it to Python; extending it to other languages would test whether the independence result generalizes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces ExplainBench, an automatic benchmark for evaluating natural-language patch explanations produced by LLM coding agents. It constructs four MCQ types from 297 SWE-bench Verified instances: end-to-end intent, end-to-end effect, local intent, and local effect. Explanations are scored by whether GPT-5-mini can answer the MCQs when given the explanation together with code context. The main empirical claim is that explanation quality is an independent axis of agent evaluation: Table 3 ranks OpenHands first on explanation score but fourth on SWE-bench efficacy, while trae-agent is first on efficacy but fourth on explanation score. The paper also reports failure-mode analyses (uninformative vs. misaligned explanations; 79.30% overconfidence among non-passing patches) and introduces ExplanationAuditAgent, which improves explanation scores for all five agents.

Significance. If the QA-proxy relation is valid, this is a timely and useful benchmark: it is the first to target explanation trustworthiness for coding agents, it grounds answers in execution traces and developer patches rather than fitted constants, it ships a replication package, and it reports stability checks (SEM < 0.01, same ranking with GPT-5-nano) and a human-LLM agreement study (Kappa 0.7). The central finding—that the best-fixing agent is not the best-explaining agent—would have practical implications for agent design and evaluation. However, the significance is conditional on the validity of the proxy, because the ranking claim rests on interpreting accuracy on the LLM questionnaire as measuring information provided by the explanation.

major comments (3)
  1. [§4.4, §5.4, Table 3] The Table 3 ranking can only be read as measuring explanation content if the QA score is driven by the explanation. Every question supplies rich code context (masked PBT for end-to-end, function/line/inputs for local, §4.4), and the blank-explanation control in §5.4 only shows that an empty string makes the LLM pick 'insufficient to answer'; it does not show that any non-empty explanation is the source of correct answers. With public SWE-bench instances, the QA LLM can answer from context or pretraining even when the explanation is vacuous. The 40-question human agreement study uses the same context-plus-explanation prompts, so it does not isolate the explanation either. Please add (i) a context-only baseline, (ii) a non-empty vacuous explanation condition, and (iii) a check on questions whose context alone supports the correct answer; otherwise both Table 3 and Table 5 are confounded.
  2. [§7.3.2, Table 2] ExplainBench excludes 203/500 SWE-bench Verified instances (40.6%). The reasons in Table 2 are not random with respect to explanation quality: 'fragile program states' (67), 'delta behavior in deeply nested objects' (30), 'expression generation failure' (23), and tracing limits could systematically remove cases where explanations are especially hard or easy to evaluate. The manuscript states that the remaining 297 instances do not differ significantly in project composition, difficulty, or patch size, but gives no details or effect sizes and no sensitivity analysis. Because Table 3 recomputes efficacy on the same 297 instances, the relative ranking is internally consistent, but the generalization from 'explanation quality is distinct' to agents in general is weakened. Please report the excluded instances, per-category statistics, and a robustness check (e.g., leave-one-category-out or re
  3. [§5.2–§5.3, §7.3.3, §5.4] The question-generation and evaluation pipeline is self-referential: GPT-5.2 generates PBTs and candidate expressions (§5.2.1, §5.3.3), GPT-5-mini answers the questions (§7.3.3), and GPT-5-mini also powers ExplanationAuditAgent. Execution-based validation anchors the ground truth, so this is not circular in the fitted-parameter sense, but the LLM-as-judge step could systematically favor explanations whose phrasing matches the generation model's style. The cross-check with GPT-5-nano (§8.1) is a same-family comparison. A stronger check would include a different model family and a per-question-type human agreement breakdown; the current Kappa=0.7 on 40 questions is a useful but small anchor. Please also release the full set of generated questions and per-item scores to permit independent audit.
minor comments (5)
  1. [§5.2.3] Typo: 'N distractor distractor options' — duplicate word.
  2. [Fig. 3(b) and §4.5] End-to-end effect questions require a pair of letters as the answer, but Fig. 3(b) shows single-choice options A–E without indicating how pairs are formed; the prompt and figure should make this explicit.
  3. [§5.4] The 'blank explanation yields a score of 0' check should report whether this holds for every question type and every agent explanation slot, since the 'insufficient to answer' option is the last resort; a single global statement is ambiguous.
  4. [Table 2] The row 'SWE-bench harness failing with developer patches' should cite the Anthropic report mentioned in §7.3.2, or provide a link in the references.
  5. [§8.2, Table 5] The overconfidence denominator counts non-passing patches; please state whether 'pass' corresponds to the QA LLM's single best answer or a majority vote over the five repeated generations, and how ties are handled.

Circularity Check

0 steps flagged

No significant circularity: the benchmark score is an explicit operationalization, not a derived prediction; self-citations are tooling references, not load-bearing justification.

full rationale

ExplainBench defines explanation score as the proportion of multiple-choice questions that a QA LLM answers correctly when given the agent explanation (§4.5, Eq. 1), and the paper explicitly states the underlying relation as an assumption rather than as a derived theorem: 'Our core assumption to solve this problem is that informative explanations would help LLMs accurately answer questions about the bug' (§3). The ground-truth answers are anchored to execution traces, developer patches, and agent patches through validated PBTs and delta-behavior extraction (§5.2, §5.3), and all generated distractors are validated by execution. The central ranking claim (§8.1, Table 3) is an empirical comparison between this defined metric and SWE-bench Verified efficacy on the same subset; it does not reduce to a fitted parameter, a definitional identity, or a self-citation. Self-references such as SpecRover [32] and the tracer approach [7] are implementation/tooling citations, and the generated artifacts are independently validated by running the tests, so they are not load-bearing in a circular sense. The blank-explanation control scoring 0 (§5.4) corroborates the operationalization. A construct-validity concern—that the QA LLM might answer from code context or pretraining rather than from the explanation—is a potential validity threat, not a circular derivation, and the paper itself acknowledges the limited scope of the metric in §10. Under the requirement to exhibit a specific reduction, no circular step is present.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

No numeric constants are fitted to the target outcome; the central claims are empirical. The ledger records the modeling choices the benchmark rests on: LLM-as-judge, developer patch as intent oracle, validated PBTs, and hand-set hyperparameters for distractor selection.

free parameters (4)
  • MMR lambda = 0.7
    Hand-selected in Eq. 3 to trade relevance/diversity of distractor expressions (§5.3.4); affects question difficulty and thus scores, but is not fitted to explanation outcomes.
  • Equal weights across four question types = 1/4 each
    Explanation score in Eq. 1 weights end-to-end intent, end-to-end effect, local intent, and local effect equally; different weights could change the agent ranking.
  • Number of repeated QA generations = 5
    Score is averaged over five LLM answers at temperature 1.0 to reduce stochasticity (§7.3.3).
  • Candidate expression count N_candidate = 10
    LLM generates 10 changed and unchanged candidate expressions for local questions (§5.3.3); affects the distractor pool and question difficulty.
axioms (6)
  • domain assumption Informative explanations are sufficient for an LLM to answer MCQs correctly
    Stated as the core assumption in §3; the entire scoring metric in §4.5 depends on this proxy relation.
  • domain assumption Developer patch is an oracle for intended behavior
    §4.3 and §5.2.2: intent questions are grounded in developer-patch behavior; alternative valid fixes that differ from the developer patch can be marked wrong.
  • domain assumption Generated property-based tests reproduce the bug and pass on the fixed version
    §5.2.1: PBTs are validated automatically and 2% are written manually; if a PBT is wrong, end-to-end ground truth is wrong.
  • domain assumption LLM-generated candidate expressions and distractors are representative once validated
    §5.3.3: candidates from GPT-5.2 are validated by execution; expression generation failures led to 23 question exclusions (Table 2).
  • domain assumption GPT-5-mini is a reliable, non-contaminated QA judge
    §7.3.3: the weaker model is chosen so answers depend on the explanation rather than prior knowledge; a GPT-5-nano check is done, but contamination cannot be fully ruled out.
  • domain assumption Execution tracing and delta-behavior extraction faithfully capture patch semantics
    §5.3.1-5.3.2: a sys.settrace-based tracer and Algorithm 1 define the first observable difference; tracing limits caused 35 exclusions.

pith-pipeline@v1.3.0-daily-deepseek · 19845 in / 15057 out tokens · 126900 ms · 2026-08-01T15:32:11.734115+00:00 · methodology

0 comments
read the original abstract

Large Language Model (LLM) agents have seen rapid adoption in software engineering. As agents take a greater role in the actual generation of code, they are making larger changes, spanning tens to hundreds of lines. This makes manual review of agent results increasingly infeasible, leading developers to turn to explanations to understand enacted changes. Despite this, there are no benchmarks that evaluate the trustworthiness of agent-generated explanations. To bridge this gap, we propose ExplainBench, a benchmark to automatically evaluate explanations from coding agents. ExplainBench is based on the intuition that informative explanations should enable an LLM to correctly answer questions, allowing quantitative comparison of explanation quality between agents. With this observation, we construct a suite of questions that evaluates whether explanations accurately describe (1) the intended behavior of buggy code and (2) the effect of applying the agent patch itself. Experiments first reveal that explanation quality is a distinct axis of agent evaluation: ExplainBench ranks agents differently from the widely-used SWE-bench Verified benchmark. A deeper breakdown of explanation quality in agents shows frequent problems in explanations, such that explanations often claim that a patch is correct when it is not. Based on this insight, we implement and evaluate an explanation audit agent which runs additional tests to validate and refine explanations. This agent improved the explanations of all evaluated agents, demonstrating agent explanations can be automatically made more trustworthy.

Figures

Figures reproduced from arXiv: 2607.26451 by Abhik Roychoudhury, Imam Nur Bani Yusuf, Sungmin Kang, Zhiyuan Pan.

Figure 1
Figure 1. Figure 1: Example explanation and submitted patch from the [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of ExplainBench. Masked Test: @given(...) def test_396c8f15(array): table = Table([array], names=['col1']) col = table['col1'] assert [[MASKED]] Choices: A. hasattr(col, 'mask') B. isinstance(col, Column) C. isinstance(col, NdarrayMixin) D. not isinstance(col, Column) E. Explanation insufficient to answer. Q: Based on the explanation, what should go in [[MASKED]]? (a) End-to-end intent question. T… view at source ↗
Figure 3
Figure 3. Figure 3: Example questions. Edited for brevity. yet desired from LLM-generated explanations: Xiao et al. [40] find that one of the common manual edits to LLM-generated pull request descriptions was to add the intent of the change. Prior work on commit messages and pull requests [21, 36] also notes the need to describe the expected behavior of the code. Meanwhile, the explanation needs to accurately describe the act… view at source ↗
Figure 4
Figure 4. Figure 4: Example of a resolved human-labeler disagreement [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Question-answering prompt template [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Example of explanation misalignment [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

50 extracted references · 2 canonical work pages

  1. [1]

    Emad Aghajani, Csaba Nagy, Olga Lucero Vega-Márquez, Mario Linares-Vásquez, Laura Moreno, Gabriele Bavota, and Michele Lanza. 2019. Software documenta- tion issues unveiled. In2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 1199–1210. doi:10.1109/ICSE.2019.00122

    arXiv 2019

  2. [2]

    Toufique Ahmed, Jatin Ganhotra, Rangeet Pan, Avraham Shinnar, Saurabh Sinha, and Martin Hirzel. 2025. Otter: Generating Tests from Issues to Vali- date SWE Patches. InForty-second International Conference on Machine Learning. doi:10.48550/arXiv.2502.05368

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.05368 2025

  3. [3]

    Applis, Y

    L. Applis, Y. Zhang, S. Liang, N. Jiang, L. Tan, and A. Roychoudhury. 2026. Unified Software Engineering Agent as AI Software Engineer. InInternational Conference on Software Engineering (ICSE). doi:10.48550/arXiv.2506.14683

    doi:10.48550/arxiv.2506.14683 2026

  4. [4]

    Islem Bouzenia and Michael Pradel. 2025. You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects.Proc. ACM Softw. Eng.2, ISSTA, Article ISSTA047 (June 2025), 23 pages. doi:10.1145/3728922

    doi:10.1145/3728922 2025

  5. [5]

    2022.Functions-When Developers Use Them and Why

    Alexey Braver. 2022.Functions-When Developers Use Them and Why. Ph. D. Dissertation. Hebrew University of Jerusalem

    2022

  6. [6]

    Jaime Carbonell and Jade Goldstein. 1998. The use of MMR, diversity-based reranking for reordering documents and producing summaries. InProceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval. 335–336. doi:10.1145/290941.291025

    arXiv 1998

  7. [7]

    Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. 2025. Rea- soning Runtime Behavior of a Program with LLM: How Far are We?. In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). 1869–

    2025

  8. [8]

    Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry

    Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry. 2024. Introducing SWE-bench Verified. Retrieved 27 March 2026 from https://openai.com/index/introducing-swe-bench-verified/

    2024

  9. [9]

    Koen Claessen and John Hughes. 2000. QuickCheck: a lightweight tool for random testing of Haskell programs. InProceedings of the Fifth ACM SIGPLAN International Conference on Functional Programming (ICFP ’00). Association for Computing Machinery, New York, NY, USA, 268–279. doi:10.1145/351240.351266

    arXiv 2000

  10. [10]

    Jacob Cohen. 1960. A coefficient of agreement for nominal scales.Educational and psychological measurement20, 1 (1960), 37–46. doi:10.1177/001316446002000104

    doi:10.1177/001316446002000104 1960

  11. [11]

    2025.How AI Is Transforming Work at An- thropic

    Saffron Huang, Bryan Seethor, Esin Durmus, Kunal Handa, Miles McCain, Michael Stern, and Deep Ganguli. 2025.How AI Is Transforming Work at An- thropic. Retrieved 27 March 2026 from https://anthropic.com/research/how-ai- is-transforming-work-at-anthropic/

    2025

  12. [12]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. InThe Twelfth International Conference on Learning Representations. doi:10.48550/arXiv.2310.06770

    Pith/arXiv arXiv doi:10.48550/arxiv.2310.06770 2024

  13. [13]

    Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. 2025. DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?. InThe Thirteenth In- ternational Conference on Learning Representations. doi:10.48550/arXiv.2409.07703

    Pith/arXiv arXiv doi:10.48550/arxiv.2409.07703 2025

  14. [14]

    Thomas Joshi, Shayan Chowdhury, and Fatih Uysal. 2025. SWE-Bench-CL: Continual Learning for Coding Agents.arXiv preprint arXiv:2507.00014(2025). doi:10.48550/ARXIV.2507.00014

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.00014 2025

  15. [15]

    Sungmin Kang, Bei Chen, Shin Yoo, and Jian-Guang Lou. 2025. Explainable auto- mated debugging via large language model-driven scientific debugging.Empirical Software Engineering30, 2 (2025), 1–28. doi:10.1007/S10664-024-10594-X

    work page doi:10.1007/s10664-024-10594-x 2025

  16. [16]

    Pavneet Singh Kochhar, Xin Xia, David Lo, and Shanping Li. 2016. Practitioners’ Expectations on Automated Fault Localization. InProceedings of the 25th Interna- tional Symposium on Software Testing and Analysis (ISSTA 2016). Association for Computing Machinery, 165–176. doi:10.1145/2931037.2931051

    arXiv 2016

  17. [17]

    J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data.biometrics(1977), 159–174. doi:10.2307/2529310

    doi:10.2307/2529310 1977

  18. [18]

    Xuan-Bach D Le, Lingfeng Bao, David Lo, Xin Xia, Shanping Li, and Corina Pasare- anu. 2019. On reliability of patch correctness assessment. In2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 524–535. doi:10.1109/ICSE.2019.00064

    arXiv 2019

  19. [19]

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. 2024. From Crowdsourced Data to High- Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline.arXiv preprint arXiv:2406.11939(2024). doi:10.48550/arXiv.2406.11939

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.11939 2024

  20. [20]

    Gon- zalez, and Ion Stoica

    Tianle Li, Evan Frick Wei-Lin Chiang, Lisa Dunlap, Banghua Zhu, Joseph E. Gon- zalez, and Ion Stoica. 2024. From Live Data to High-Quality Benchmarks: The Arena-Hard Pipeline. https://lmsys.org/blog/2024-04-19-arena-hard/

    2024

  21. [21]

    Zhixing Li, Yue Yu, Tao Wang, Yan Lei, Ying Wang, and Huaimin Wang. 2022. To follow or not to follow: Understanding issue/pull-request templates on github.IEEE Transactions on Software Engineering49, 4 (2022), 2530–2544. doi:10.1109/TSE.2022.3224053

    arXiv 2022

  22. [22]

    Jingjing Liang, Yaozong Hou, Shurui Zhou, Junjie Chen, Yingfei Xiong, and Gang Huang. 2019. How to explain a patch: An empirical study of patch explanations in open source projects. In2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 58–69. doi:10.1109/ISSRE.2019.00016

    arXiv 2019

  23. [23]

    Qiheng Mao, Zhenhao Li, Xing Hu, Kui Liu, Xin Xia, and Jianling Sun. 2025. Towards Explainable Vulnerability Detection With Large Language Models.IEEE Transactions on Software Engineering51, 10 (2025), 2957–2971. doi:10.1109/TSE. 2025.3605442

    arXiv 2025

  24. [24]

    Niels Mündler, Mark Niklas Mueller, Jingxuan He, and Martin Vechev. 2024. SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. doi:10.48550/arXiv.2406.12952

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.12952 2024

  25. [25]

    Ziyi Ni, Huacan Wang, Shuo Zhang, Shuo Lu, Ziyang He, Wang You, Zhenheng Tang, Yuntao Du, Bill Sun, Hongzhang Liu, Sen Hu, Ronghao Chen, Bo Li, Xin Li, Chen Hu, Binxing Jiao, Daxin Jiang, and Pin Lyu. 2025. GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging. arXiv:2508.18993 [cs.SE] doi:10.48550/arXiv.2508.18993

    doi:10.48550/arxiv.2508.18993 2025

  26. [26]

    Yannic Noller, Ridwan Shariffdeen, Xiang Gao, and Abhik Roychoudhury. 2022. Trust enhancement issues in program repair. InProceedings of the 44th Inter- national Conference on Software Engineering(Pittsburgh, Pennsylvania)(ICSE ’22). Association for Computing Machinery, New York, NY, USA, 2228–2240. doi:10.1145/3510003.3510040

    arXiv 2022

  27. [27]

    OpenAI. 2025. GPT-5 mini Model. Retrieved 27 March 2026 from https: //developers.openai.com/api/docs/models/gpt-5-mini

    2025

  28. [28]

    OpenAI. 2025. GPT-5.2 Model. Retrieved 27 March 2026 from https://developers. openai.com/api/docs/models/gpt-5.2

    2025

  29. [29]

    ExplainBench: Evaluating Code Expla- nations from Agents

    Zhiyuan Pan, Sungmin Kang, Imam Nur Bani Yusuf, and Abhik ROYCHOUD- HURY. 2026.Replication Package for Paper "ExplainBench: Evaluating Code Expla- nations from Agents". doi:10.5281/zenodo.19230708

    work page doi:10.5281/zenodo.19230708 2026

  30. [30]

    Muhammad Shihab Rashid, Christian Bock, Yuan Zhuang, Alexander Buch- holz, Tim Esler, Simon Valentin, Luca Franceschi, Martin Wistuba, Prabhu Teja Sivaprasad, Woo Jung Kim, et al . 2025. SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents.arXiv preprint arXiv:2504.08703(2025). doi:10.48550/arXiv.2504.08703

    Pith/arXiv arXiv doi:10.48550/arxiv.2504.08703 2025

  31. [31]

    Niklas Risse, Jing Liu, and Marcel Böhme. 2025. Top score on the wrong exam: On benchmarking in machine learning for vulnerability detection.Proceedings of the ACM on Software Engineering2, ISSTA (2025), 388–410. doi:10.1145/3728887

    doi:10.1145/3728887 2025

  32. [32]

    Haifeng Ruan, Yuntong Zhang, and Abhik Roychoudhury. 2025. SpecRover: Code Intent Extraction via LLMs. InProceedings of the IEEE/ACM 47th International Conference on Software Engineering(Ottawa, Ontario, Canada)(ICSE ’25). IEEE Press, 963–974. doi:10.1109/ICSE55347.2025.00080

    arXiv 2025

  33. [33]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. (2024). arXiv:2403.05530 [cs.CL] https://arxiv.org/abs/2403.05530

    Pith/arXiv arXiv 2024

  34. [34]

    Trae Research Team, Pengfei Gao, Zhao Tian, Xiangxin Meng, Xinchen Wang, Ruida Hu, Yuanan Xiao, Yizhou Liu, Zhao Zhang, Junjie Chen, Cuiyun Gao, Yun Lin, Yingfei Xiong, Chao Peng, and Xia Liu. 2025. Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling. arXiv:2507.23370 [cs.SE] doi:10.48550/arXiv.2507.23370

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.23370 2025

  35. [35]

    The Terminal-Bench Team. 2025. Terminal-Bench: A Benchmark for AI Agents in Terminal Environments. Retrieved 27 March 2026 from https://github.com/laude- institute/terminal-bench

    2025

  36. [36]

    Yingchen Tian, Yuxia Zhang, Klaas-Jan Stol, Lin Jiang, and Hui Liu. 2022. What makes a good commit message?. InProceedings of the 44th International Conference on Software Engineering. ACM, 2389–2401. doi:10.1145/3510003.3510205

    arXiv 2022

  37. [37]

    Shangwen Wang, Ming Wen, Bo Lin, Hongjun Wu, Yihao Qin, Deqing Zou, Xiaoguang Mao, and Hai Jin. 2020. Automated patch correctness assessment: How far are we?. InProceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering. 968–980. doi:10.1145/3324884.3416590

    arXiv 2020

  38. [38]

    Xu, Xiangru Tang, and Mingchen Zhuge et al

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, and Mingchen Zhuge et al. 2025. OpenHands: An Open Platform for AI Software Developers as Generalist Agents. InThe Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=OJd3ayDDoF

    2025

  39. [39]

    Ratnadira Widyasari, Jia Wei Ang, Truong Giang Nguyen, Neil Sharma, and David Lo. 2024. Demystifying Faulty Code: Step-by-Step Reasoning for Explainable Fault Localization. In2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). 568–579. doi:10.1109/SANER60148.2024.00064

    arXiv 2024

  40. [40]

    Tao Xiao, Hideaki Hata, Christoph Treude, and Kenichi Matsumoto. 2024. Gener- ative AI for pull request descriptions: Adoption, impact, and developer interven- tions.Proceedings of the ACM on Software Engineering1, FSE (2024), 1043–1065. doi:10.1145/3643773

    doi:10.1145/3643773 2024

  41. [41]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information Processing Systems37, 50528–50652. doi:10.48550/arXiv.2405.15793

    Pith/arXiv arXiv doi:10.48550/arxiv.2405.15793 2024

  42. [42]

    Xu Yang, Jiayuan Zhou, Michael Pacheco, Wenhan Zhu, Pengfei He, Shaowei Wang, Kui Liu, and Ruiqi Pan. 2025. Lingxi: Repository-Level Issue Res- olution Framework Enhanced by Procedural Knowledge Guided Scaling. arXiv:2510.11838 [cs.SE] https://arxiv.org/abs/2510.11838 ExplainBench: Evaluating Code Explanations from Agents ASE ’26, October 12–16, 2026, Mun...

    arXiv 2025

  43. [43]

    Yanming Yang, Xin Xia, David Lo, Tingting Bi, John Grundy, and Xiaohu Yang

  44. [44]

    Li Yujian and Liu Bo. 2007. A normalized Levenshtein distance metric.IEEE transactions on pattern analysis and machine intelligence29, 6 (2007), 1091–1095. doi:10.1109/TPAMI.2007.1078

    arXiv 2007

  45. [45]

    Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Shulin Xin, Linhao Zhang, Qi Liu, Aoyan Li, Lu Chen, Xiaojian Zhong, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, RUI LONG, Ming Ding, and liang xiang. 2025. Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving. InThe Thirty-ninth Annual Conference on Neural Informa...

    2025

  46. [46]

    Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, Elsie Nalli- pogu, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, and Dongmei Zhang

  47. [47]

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Au- toCodeRover: Autonomous Program Improvement. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis(Vienna, Austria)(ISSTA 2024). Association for Computing Machinery, New York, NY, USA, 1592–1604. doi:10.1145/3650212.3680384

    arXiv 2024

  48. [48]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT- Bench and Chatbot Arena. InNeurIPS. Received 2026-03-26; accepted 2026-06-18

    2023

  49. [2022]

    ACM Transactions on Software Engineering and Methodology (TOSEM)31, 3 (2022), 1–72

    Predictive models in software engineering: Challenges and opportunities. ACM Transactions on Software Engineering and Methodology (TOSEM)31, 3 (2022), 1–72. doi:10.1145/3503509

    doi:10.1145/3503509 2022

  50. [2025]

    InThe Thirty-ninth Annual Conference on Neu- ral Information Processing Systems Datasets and Benchmarks Track

    SWE-bench Goes Live!. InThe Thirty-ninth Annual Conference on Neu- ral Information Processing Systems Datasets and Benchmarks Track. https: //openreview.net/forum?id=OGWkr7gXka