Pith. sign in

REVIEW 3 major objections 6 minor 50 references

This paper introduces a method to audit what passing tests in LLM repair runs actually prove, and shows that nearly half of positive validation events carry no bug-discriminating information.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 01:26 UTC pith:BBA7ZKQ4

load-bearing objection A useful new measurement layer for agent validation evidence, but the replay-equivalence blind spot (interface changes scored as bug discrimination) needs an audit before the headline rates are trusted. the 3 major comments →

arxiv 2607.28871 v1 pith:BBA7ZKQ4 submitted 2026-07-30 cs.SE cs.AI

Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug?

Xiaonan Xu , Wenjing Wu This is my paper
classification cs.SE cs.AI
keywords program repair agentsvalidation evidencetest adequacylarge language modelsevidence rolesbug-contrast feedbackevidence-inadequate closuresoftware quality
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks how often a passing test run by a repair agent actually provides evidence about the reported defect. It introduces BSG-VA, which captures each validation command during a repair trajectory, isolates the test-only changes, and replays them on the original buggy code, the candidate patch, and the developer's gold fix. Across 3,730 events, 46.0% of positive comparable events were regression-only or misleading, and 23.8% of baseline rollouts closed with no discriminating evidence at all. A three-arm experiment shows that feeding the buggy-state replay result back to the agent reduces such evidence-poor closures by 7.8 percentage points, though this falls below the pre-specified 10-percentage-point threshold of practical interest.

Core claim

The central discovery is a prevalence estimate with a method behind it: mid-trajectory passing tests in LLM repair agents frequently do not discriminate the reported bug. BSG-VA assigns each validation event an evidence role by replaying the extracted test-only patch on buggy (B), candidate (S), and gold-fix (G) states. Events are gold-aligned bug-discriminating if they fail on B and pass on both S and G; candidate-specific if they fail on B and pass on S but fail on G; regression-only if they pass on B; and misleading if they pass on B but fail on G. The method yields 46.0% non-discriminating positives among positive comparable events and 23.8% baseline evidence-inadequate closure at the ro

What carries the argument

The central object is BSG-VA (buggy-state/candidate-state/gold-fix validation analysis), which captures every validation command the agent executes, snapshots the working tree, separates a test-only patch from concurrent production edits, and replays the same command in a pinned container on three code states: the original buggy code (B), the candidate state (S), and the developer gold fix (G). The captured outcome plus the three replay outcomes map each event into one of seven exhaustive, mutually exclusive evidence roles, anchored by whether the check fails on B and whether it passes on G. This replay machinery is what turns 'the agent ran a test and it passed' into an auditable statement

Load-bearing premise

The load-bearing premise is that replaying the extracted test-only patch on the buggy, candidate, and gold states in a fresh pinned container faithfully reproduces what the agent's validation command actually proved at the original working tree; the paper itself flags that it does not independently audit this replay equivalence, so if a test fails on B because of an undeclared dependency on the agent's production edits rather than because it detects the bug, the role assignme

What would settle it

Manually re-run a random sample of events classified as regression-only with the full working tree (production edits included, not just the test-only patch) on the original buggy state in the original container; if a substantial share then fail, the test-only decomposition is unfaithful and the 46.0% estimate is not about the bug's detectability but about the replay construction.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If nearly half of positive validation evidence is non-discriminating, benchmark pass rates substantially overstate how often agents have actually verified the reported defect.
  • Evidence-inadequate closure quantifies a new failure mode: an agent can submit a patch on the basis of positive but irrelevant checks; integrators can now audit trajectories for this before trusting a patch.
  • Returning the buggy-state replay result to the agent is a practical, low-cost intervention (median 11 seconds per rollout) that reduces evidence-poor closures, and a generic reminder does part of the work.
  • Evidence quality carries predictive signal for official resolution beyond task-level covariates, so validation-evidence roles could serve as an online process signal for model selection or early stopping.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 46.0% figure likely underestimates the trajectory-wide rate, because the measurement starts only after the first production edit; pre-edit diagnostic checks are excluded.
  • The method generalizes beyond the specific models and benchmarks studied: any replayable trajectory that preserves code states and a pinned environment can be audited, so the same audit could be applied to different scaffolds, languages, or agent families.
  • A natural extension is to use evidence-role labels as a reward or filtering signal in reinforcement learning, training agents to maximize bug-discriminating evidence rather than raw pass counts.
  • The finding that the reminder alone accounts for roughly a third of the effect suggests a cheap deployment lever; whether the B-replay increment is worth its infrastructure cost depends on the base rate of evidence-inadequate closure in the target setting.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces BSG-VA, a replay-based method that captures each validation command a repair agent executes, extracts a test-only patch, and replays the command on the original buggy state (B), the candidate state (S), and the gold-fix state (G). Evidence roles are assigned from the captured outcome and replay pattern. In a controlled experiment with gpt-5.6-sol on 110 SWE-bench Verified / SWE-rebench tasks, the paper reports that 46.0% of 2,548 positive comparable validation events carry no bug-discriminating information, and that 23.8% of baseline rollouts close with evidence-inadequate closure. A three-arm experiment (baseline, static reminder, bug-contrast feedback) finds that BCF reduces evidence-inadequate closure by 7.8 pp relative to the reminder (p=0.0029) and raises bug-discriminating evidence by 7.4 pp (p=0.011), with both estimates below the prespecified 10 pp SESOI. Exploratory replications vary the scaffold and model. The central claims are that the proposed taxonomy provides an auditable measure of validation-evidence quality, and that the non-discriminating-evidence phenomenon is prevalent and partially addressable by feedback.

Significance. If the measurement method is faithful, this is a valuable contribution: it moves beyond trajectory-level counts and coarse pass/fail agreement to an event-level, executable audit of what an agent's own validation commands actually establish. The study design is unusually careful for this area: a prespecified analysis plan, a smallest effect size of interest, a fixed-sequence confirmatory family, an attention-matched control arm, leave-one-repository-out and wild-bootstrap robustness checks, balanced cell loss, and honest reporting that the confirmatory point estimates fall below the SESOI. The released dataset is a useful asset. The main weakness is that the replay-faithfulness assumption — that a B-replay failure means the test detects the reported bug rather than merely depending on the agent's production edits — is load-bearing and is not independently audited. The paper would be strengthened substantially by a failure-mode analysis of B-replay failures and by public release of the replay/capture code, not just the aggregate data.

major comments (3)
  1. [§3.3–3.4, Table 3] The B-fail criterion conflates genuine bug detection with interface breakage. Section 3.2 excludes only import errors caused by newly introduced production symbols; failures caused by changed signatures, renamed methods, or added required arguments on existing symbols are counted as B-fail. 1,377 B-fail/S-pass events (Table 3) therefore include failures of tests written against the agent's modified API, which would fail on B for reasons unrelated to the reported defect. This directly inflates the gold-aligned and candidate-specific counts and, in turn, the 46.0% non-discriminating share and the BCF effect. The manuscript's own NOT COMPARABLE flag is too narrow to address this. Please stratify all B-fail events by failure signature (e.g., TypeError/AttributeError/ImportError vs. assertion failure) and report the central estimates restricted to failures that do not reference symbols modifi
  2. [§3.2–3.3] The replay-faithfulness assumption is asserted, not demonstrated. Section 3.3 checks S-replay agreement with the captured outcome, but this only verifies that the candidate state reproduces what the agent saw; it does not validate that a B-replay failure is semantically a bug-detecting assertion. The paper reports no audit of a sample of B-replay failures to distinguish defect-related failures from interface-mismatch failures or other environment artifacts. Given that the taxonomy, the EIC construct, and the primary outcome are all defined by these replay outcomes, an independent audit of a stratified sample (or a reproducible extractions/replay pipeline) is needed to establish construct validity. The paper's release does not include the capture/replay code, so this cannot currently be checked externally.
  3. [§4.2, Table 1; §5.3] The BCF arm's feedback rule rewards any B-fail: the message states that the check 'distinguishes the two states.' Under this rule, an agent whose test fails on B because of a changed interface receives the same positive signal as an agent whose test fails because it detects the defect. This creates a differential incentive across arms: BCF may increase B-fail events by encouraging tests coupled to the agent's production edits, mechanically raising bug-discriminating evidence and reducing EIC without improving genuine evidence quality. The paper's robustness checks do not address this differential misclassification. A sensitivity analysis restricting the outcome to events whose B-fail is not attributable to modified interfaces is necessary to support the claim that the intervention changes evidence quality rather than the measurement artifact.
minor comments (6)
  1. [§1] Typographical issues: 'hereaftersol' should read 'hereafter sol' and 'hereafterthe' should read 'hereafter the'.
  2. [§3.4] Minor formatting: 'fromGOLD ALIGNED' should read 'from GOLD ALIGNED'.
  3. [§5.3, Table 4] p-values such as 0.00018 and 0.00058 are better reported as p<0.001 in the text and table for readability.
  4. [§4.2 / References] Unicode/TeX issues in author names (e.g., 'M¨ undler', '¨Orwall') may render as combining diacritics; please fix the LaTeX/Unicode encoding.
  5. [Code availability] The statement that the code is 'not publicly released' is a reproducibility limitation for a method whose central contribution is a deterministic replay pipeline. At minimum, the replay/harness code should be released alongside the data, or a replication package should be promised as a condition of publication.
  6. [§6.3] The acknowledged limitation about pre-edit commands is good, but it is not quantified. A descriptive table of events excluded because they occur before the first production edit would help readers gauge the scope of the prevalence claim.

Circularity Check

0 steps flagged

No significant circularity: the central claims are empirical measurements and a randomized comparison, not derivations that reduce to their inputs.

full rationale

The paper's load-bearing assertions — 46.0% non-discriminating positive events, 23.8% evidence-inadequate closure, and the 7.8 pp BCF effect — are empirical measurements and a controlled experiment. The evidence-role taxonomy is defined by replay outcomes (B-fail, S-pass, G-pass), so statements like '46.0% of positive comparable events are regression-only or misleading' are direct counts under an explicit operational definition, not derivations that assume the conclusion. No parameter is fitted to produce these estimates; they are simple aggregates of recorded replay results. The BCF intervention is a within-task randomized comparison with a prespecified analysis plan, and the outcome (EIC) is not an input to the intervention, even though both share the BSG-VA replay infrastructure. The paper explicitly acknowledges construct-validity threats (gold-fix reference standard, event-definition sensitivity, truncation at first production edit) and reports robustness checks. Self-citations are limited to the data release and are not load-bearing. No uniqueness theorem, fitted prediction, or ansatz is invoked from prior work. The skeptic's concern about interface-mismatch failures inflating B-fail counts is a validity threat, not a circularity: it questions whether the measurement faithfully captures bug discrimination, but the measurement itself is not defined in terms of the conclusions drawn from it. Overall, the derivation chain is self-contained and non-circular.

Axiom & Free-Parameter Ledger

1 free parameters · 6 axioms · 0 invented entities

The paper introduces no new physical or mathematical entities. The central measurement is empirical and is built on domain assumptions about replay fidelity, the gold-fix reference standard, artifact decomposition, classifier accuracy, flakiness filtering, and benchmark outcomes. The only hand-chosen numerical parameter is the prespecified SESOI. No fitted constants are used to derive the headline estimates.

free parameters (1)
  • Smallest effect size of interest (SESOI) = 10 percentage points
    Prespecified threshold chosen by design judgment (Section 4.3), not fitted to data, but it anchors the claim that the observed effects are below practical magnitude.
axioms (6)
  • domain assumption Replaying the extracted test-only patch on B, S, and G in a fresh pinned container faithfully reproduces the agent's validation command at its original working-tree state.
    Sections 3.2-3.3; the entire evidence-role taxonomy and EIC depend on these replay outcomes being faithful. The paper flags import-error cases but does not independently validate extraction/replay equivalence.
  • domain assumption The developer gold fix G is an adequate reference for whether a check discriminates the reported defect.
    Sections 3.4 and 6.3; roles such as candidate-specific (B-fail, S-pass, G-fail) and misleading (B-pass, G-fail) rely on G as ground truth; the paper acknowledges G may be incomplete or one of many valid fixes.
  • domain assumption The working-tree decomposition into production patch and test-only patch correctly separates validation artifacts from candidate changes.
    Sections 3.1-3.2; uses test-directory conventions and agent-created file sets; a misclassification would change B/S/G replay outcomes and role assignments.
  • domain assumption The online classifier of validation commands (project test run, scripted oracle, observational probe, non-validation) is accurate.
    Section 3.1; audited on 111 events with 92.8% agreement and corrections adopted; errors would alter the event population and prevalence estimates.
  • domain assumption Stable S-replay agreement with the captured outcome is a valid reproducibility filter; flaky events can be excluded without biasing the constructs.
    Sections 3.3-3.4; flaky events (15 of 3,730) are excluded; if flakiness correlates with evidence quality, estimates could shift.
  • domain assumption Official benchmark resolution (held-out tests on submitted production patch) is a valid measure of repair success.
    Section 4.1; used to check that BCF does not cost repair success; any benchmark noise affects that comparison.

pith-pipeline@v1.3.0-alltime-deepseek · 16660 in / 15343 out tokens · 161659 ms · 2026-08-03T01:26:09.545082+00:00 · methodology

0 comments
read the original abstract

When a repair agent runs a test and sees it pass, the result is treated as evidence about the reported defect. We measure how often that treatment is warranted. BSG-VA (buggy-state/candidate-state/gold-fix validation analysis) captures each validation command at its exact working-tree state, extracts a test-only patch, and replays the command on the original buggy code (B), the candidate state (S), and the developer gold fix (G). The captured outcome and the replay results assign every event an evidence role, from gold-aligned bug-discriminating through regression-only to misleading. Across 3,730 events in 643 rollouts on 110 tasks, 46.0% of positive comparable events carry no bug-discriminating information; 23.8% of baseline rollouts, with no feedback injected, close with a patch whose entire positive evidence base is of this kind. A three-arm experiment tests whether returning the B-replay outcome to the agent changes this pattern. Bug-contrast feedback reduces evidence-inadequate closure by 7.8 percentage points relative to an attention-matched reminder (p = 0.0029) and raises bug-discriminating evidence by 7.4 points (p = 0.011), with no detectable cost to repair success. Both estimates fall below the prespecified 10-percentage-point smallest effect size of interest, so practical magnitude remains uncertain. Roughly a third of the improvement traces to the reminder alone; across two exploratory replications, varying the scaffold and the model, the B-replay content adds a detectable increment only with gpt-5.6-sol under the unconstrained tool-use loop. BSG-VA applies post hoc to any replayable repair trajectory that preserves the required code states and execution environment. Keywords: program repair agents, validation evidence, test adequacy, large language models, software quality, controlled experiment.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

50 extracted references · 3 canonical work pages

  1. [1]

    , Ganhotra, J

    OTTER APACrefauthors Ahmed, T. , Ganhotra, J. , Pan, R. , Shinnar, A. , Sinha, S. Hirzel, M. APACrefauthors \ 2025 . Otter: Generating Tests from Issues to Validate SWE Patches Otter: Generating Tests from Issues to Validate SWE Patches . International Conference on Machine Learning ( ICML ). International conference on machine learning ( ICML ). APACrefU...

    Pith/arXiv arXiv 2025

  2. [2]

    , Ganhotra, J

    EVOOTTER APACrefauthors Ahmed, T. , Ganhotra, J. , Shinnar, A. Hirzel, M. APACrefauthors \ 2026 . EvoOtter: Evolutionary Reproduction Test Generator . EvoOtter: Evolutionary Reproduction Test Generator . APACrefURL https://arxiv.org/abs/2607.02854 APACrefURL arXiv preprint

    Pith/arXiv arXiv 2026

  3. [3]

    , Golubev, A

    SWEREBENCH APACrefauthors Badertdinov, I. , Golubev, A. , Nekrashevich, M. , Shevtsov, A. , Karasik, S. , Andriushchenko, A. Yangel, B. APACrefauthors \ 2025 . SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents . SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evalu...

    arXiv 2025

  4. [4]

    , Chowdhury, K

    SMOKE APACrefauthors Banik, D. , Chowdhury, K. Shamim, S.I. APACrefauthors \ 2026 . All Smoke, No Alarm: Oracle Signals in Agent-Authored Test Code . All Smoke, No Alarm: Oracle Signals in Agent-Authored Test Code . APACrefURL https://arxiv.org/abs/2606.18168 APACrefURL IEEE AITest 2026 accepted

    Pith/arXiv arXiv 2026

  5. [5]

    \ Pradel, M

    TAR_TRAJ APACrefauthors Bouzenia, I. \ Pradel, M. APACrefauthors \ 2025 . Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories . 2025 40th IEEE/ACM International Conference on Automated Software Engineering ( ASE ) 2025 40th IEEE/AC...

    2025

  6. [6]

    , Gelbach, J.B

    CGM2008 APACrefauthors Cameron, A.C. , Gelbach, J.B. Miller, D.L. APACrefauthors \ 2008 . Bootstrap-Based Improvements for Inference with Clustered Errors Bootstrap-Based Improvements for Inference with Clustered Errors . Review of Economics and Statistics 90 3 414--427, APACrefDOI doi:10.1162/rest.90.3.414 APACrefDOI

    doi:10.1162/rest.90.3.414 2008

  7. [7]

    , Mitchell, H

    TRACEABILITY APACrefauthors Ceka, I. , Mitchell, H. , Pujar, S. , Buratti, L. , Ramji, S. , Yang, J. Ray, B. APACrefauthors \ 2026 . Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study . Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study . APACrefURL https://arx...

    Pith/arXiv arXiv 2026

  8. [8]

    , Lin, S

    CODER APACrefauthors Chen, D. , Lin, S. , Zeng, M. , Zan, D. , Wang, J G. , Cheshkov, A. Wang, Q. APACrefauthors \ 2024 . CodeR: Issue Resolving with Multi-Agent and Task Graphs . CodeR: Issue Resolving with Multi-Agent and Task Graphs . APACrefURL https://arxiv.org/abs/2406.01304 APACrefURL

    Pith/arXiv arXiv 2024

  9. [9]

    , Lin, M

    CHEN2023SELFDEBUG APACrefauthors Chen, X. , Lin, M. , Schärli, N. Zhou, D. APACrefauthors \ 2023 . Teaching Large Language Models to Self-Debug . Teaching Large Language Models to Self-Debug . APACrefURL https://arxiv.org/abs/2304.05128 APACrefURL

    Pith/arXiv arXiv 2023

  10. [10]

    , Sun, Z

    RETHINK APACrefauthors Chen, Z. , Sun, Z. , Shi, Y. , Peng, C. , Gu, X. , Lo, D. Jiang, L. APACrefauthors \ 2026 . Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents . Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents . APACrefURL https://arxiv.org/abs/2602.07900 APACrefURL arXiv v...

    Pith/arXiv arXiv 2026

  11. [11]

    , Tufano, M

    DYNAMIC_BRT APACrefauthors Cheng, R. , Tufano, M. , Cambronero, J. , Wei, R. , Shi, S. , Uy, G. Ivančić, F. APACrefauthors \ 2026 . Dynamic Cogeneration of Bug Reproduction Test in Agentic Program Repair . Dynamic Cogeneration of Bug Reproduction Test in Agentic Program Repair . APACrefURL https://arxiv.org/abs/2601.19066 APACrefURL arXiv preprint

    arXiv 2026

  12. [12]

    , Pan, Y

    ECHO APACrefauthors Fei, Z. , Pan, Y. , Sarro, F. , Ge, J. , Liu, M. , Ng, V. Ye, H. APACrefauthors \ 2026 . Echo: Graph-Enhanced Retrieval and Execution Feedback for Issue Reproduction Test Generation . Echo: Graph-Enhanced Retrieval and Execution Feedback for Issue Reproduction Test Generation . APACrefURL https://arxiv.org/abs/2603.07326 APACrefURL arX...

    arXiv 2026

  13. [13]

    , Liu, Y

    SWEDOCTOR APACrefauthors Guo, Y. , Liu, Y. , Zhang, J.M. , Ma, Y. , Lou, Y. Chen, Z. APACrefauthors \ 2026 . SWE-Doctor: Guiding Software Engineering Agents with Runtime Diagnosis from Multi-Faceted Bug Reproduction Tests . SWE-Doctor: Guiding Software Engineering Agents with Runtime Diagnosis from Multi-Faceted Bug Reproduction Tests . APACrefURL https:/...

    Pith/arXiv arXiv 2026

  14. [14]

    \ Robbes, R

    OVERMOCK APACrefauthors Hora, A. \ Robbes, R. APACrefauthors \ 2026 . Are Coding Agents Generating Over-Mocked Tests? An Empirical Study . Are Coding Agents Generating Over-Mocked Tests? An Empirical Study . APACrefURL https://arxiv.org/abs/2602.00409 APACrefURL MSR 2026

    arXiv 2026

  15. [15]

    , Zhang, J.M

    INCORRECT_CODE APACrefauthors Huang, D. , Zhang, J.M. , Harman, M. , Du, M. Cui, H. APACrefauthors \ 2026 . Measuring the Influence of Incorrect Code on Test Generation . Measuring the Influence of Incorrect Code on Test Generation . APACrefURL https://conf.researchr.org/details/icse-2026/icse-2026-research-track/15/Measuring-the-Influence-of-Incorrect-Co...

    2026

  16. [16]

    , Liu, K

    JIANG2023 APACrefauthors Jiang, N. , Liu, K. , Lutellier, T. Tan, L. APACrefauthors \ 2023 . Impact of Code Language Models on Automated Program Repair Impact of Code Language Models on Automated Program Repair . 2023 IEEE/ACM 45th International Conference on Software Engineering ( ICSE ) 2023 IEEE/ACM 45th international conference on software engineering...

    2023

  17. [17]

    , Yang, J

    SWEBENCH APACrefauthors Jimenez, C.E. , Yang, J. , Wettig, A. , Yao, S. , Pei, K. , Press, O. Narasimhan, K. APACrefauthors \ 2024 . SWE-bench: Can Language Models Resolve Real-World GitHub Issues? SWE-bench: Can Language Models Resolve Real-World GitHub Issues? International Conference on Learning Representations ( ICLR ). International conference on lea...

    Pith/arXiv arXiv 2024

  18. [18]

    , Yoon, J

    LIBRO APACrefauthors Kang, S. , Yoon, J. Yoo, S. APACrefauthors \ 2023 . Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction . 2023 IEEE/ACM 45th International Conference on Software Engineering ( ICSE ) 2023 IEEE/ACM 45th internat...

    2023

  19. [19]

    , Mathews, N.S

    ASSERTFLIP APACrefauthors Khatib, L. , Mathews, N.S. Nagappan, M. APACrefauthors \ 2025 . AssertFlip: Reproducing Bugs via Inversion of LLM-Generated Passing Tests . AssertFlip: Reproducing Bugs via Inversion of LLM-Generated Passing Tests . APACrefURL https://arxiv.org/abs/2507.17542 APACrefURL ICSE 2026 Research Track

    arXiv 2025

  20. [20]

    , Tambon, F

    CODING_BEFORE_TEST APACrefauthors Konstantinou, M. , Tambon, F. Papadakis, M. APACrefauthors \ 2026 . On the risk of coding before testing: An empirical study on LLM-based test generation workflow . On the risk of coding before testing: An empirical study on LLM-based test generation workflow . APACrefURL https://arxiv.org/abs/2607.05139 APACrefURL arXiv preprint

    Pith/arXiv arXiv 2026

  21. [21]

    , Scheel, A.M

    LAKENS2018 APACrefauthors Lakens, D. , Scheel, A.M. Isager, P.M. APACrefauthors \ 2018 . Equivalence Testing for Psychological Research: A Tutorial Equivalence Testing for Psychological Research: A Tutorial . Advances in Methods and Practices in Psychological Science 1 2 259--269, APACrefDOI doi:10.1177/2515245918770963 APACrefDOI

    doi:10.1177/2515245918770963 2018

  22. [22]

    , Hanuska, L

    TESTMAP APACrefauthors Leary, H. , Hanuska, L. Brown, C. APACrefauthors \ 2026 . TestMap: Evidence Infrastructure for Foundation-Model-Assisted Test Generation . TestMap: Evidence Infrastructure for Foundation-Model-Assisted Test Generation . APACrefURL https://arxiv.org/abs/2606.10211 APACrefURL AIWare 2026 arXiv Track

    Pith/arXiv arXiv 2026

  23. [23]

    , Dewey-Vogt, M

    GOUES2012 APACrefauthors Le Goues, C. , Dewey-Vogt, M. , Forrest, S. Weimer, W. APACrefauthors \ 2012 . A systematic study of automated program repair: Fixing 55 out of 105 bugs for \ 8 each A systematic study of automated program repair: Fixing 55 out of 105 bugs for \ 8 each . 2012 34th International Conference on Software Engineering ( ICSE ) 2012 34th...

    2012

  24. [24]

    , Zhu, J

    TRNR APACrefauthors Lin, Z. , Zhu, J. , Zhou, M. , Wang, X. , Sun, Z. , Yang, R. Li, L. APACrefauthors \ 2026 . To Run or Not to Run: Analyzing the Cost-Effectiveness of Code Execution in LLM-Based Program Repair . To Run or Not to Run: Analyzing the Cost-Effectiveness of Code Execution in LLM-Based Program Repair . APACrefURL https://arxiv.org/abs/2606.2...

    Pith/arXiv arXiv 2026

  25. [25]

    , Chen, Z

    SAME_SIGNAL APACrefauthors Ma, W. , Chen, Z. , Gu, J. , Li, T. , Liu, S. Jiang, L. APACrefauthors \ 2026 . Same Signal, Different Semantics: A Cross-Framework Behavioral Analysis of Software Engineering Agents . Same Signal, Different Semantics: A Cross-Framework Behavioral Analysis of Software Engineering Agents . APACrefURL https://arxiv.org/abs/2605.18...

    Pith/arXiv arXiv 2026

  26. [26]

    , Kereopa-Yorke, B

    B2T APACrefauthors Ma, Y. , Kereopa-Yorke, B. Schultz, B. APACrefauthors \ 2026 . Building to the Test: Coding Agents Deliver What You Check, Not What You Requested . Building to the Test: Coding Agents Deliver What You Check, Not What You Requested . APACrefURL https://arxiv.org/abs/2606.28430 APACrefURL arXiv preprint / Microsoft Research

    Pith/arXiv arXiv 2026

  27. [27]

    \ Madiraju, M.S.P

    RIGORBENCH APACrefauthors Madiraju, M.B. \ Madiraju, M.S.P. APACrefauthors \ 2026 . RigorBench: Benchmarking Engineering Process Discipline in Autonomous AI Coding Agents . RigorBench: Benchmarking Engineering Process Discipline in Autonomous AI Coding Agents . APACrefURL https://arxiv.org/abs/2606.22678 APACrefURL arXiv preprint

    Pith/arXiv arXiv 2026

  28. [28]

    , Zhang, X

    EVIACT APACrefauthors Meng, Q. , Zhang, X. , Ren, Z. Visser, J. APACrefauthors \ 2026 . EviACT: An Evidence-to-Action Framework for Agentic Program Repair . EviACT: An Evidence-to-Action Framework for Agentic Program Repair . APACrefURL https://arxiv.org/abs/2605.27238 APACrefURL arXiv preprint

    Pith/arXiv arXiv 2026

  29. [29]

    , Müller, M

    SWT APACrefauthors Mündler, N. , Müller, M. , He, J. Vechev, M. APACrefauthors \ 2024 . SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents . Advances in Neural Information Processing Systems ( NeurIPS ). Advances in neural information processing systems ( NeurIPS ...

    2024

  30. [30]

    , Bouzenia, I

    ISSUE2TEST APACrefauthors Nashid, N. , Bouzenia, I. , Pradel, M. Mesbah, A. APACrefauthors \ 2026 . Issue2Test: Generating Reproducing Test Cases from Issue Reports . Issue2Test: Generating Reproducing Test Cases from Issue Reports . APACrefURL https://conf.researchr.org/details/icse-2026/icse-2026-research-track/87/Issue2Test-Generating-Reproducing-Test-...

    2026

  31. [31]

    Introducing SWE-bench Verified

    SWEBENCH_VERIFIED APACrefauthors OpenAI APACrefauthors \ 2024 . Introducing SWE-bench Verified . Introducing SWE-bench Verified . APACrefURL https://openai.com/index/introducing-swe-bench-verified/ APACrefURL Human-validated subset of SWE-bench

    2024

  32. [32]

    APACrefauthors \ 2024

    MOATLESS APACrefauthors \"O rwall, A. APACrefauthors \ 2024 . Moatless Tools . Moatless Tools . APACrefURL https://github.com/aorwall/moatless-tools APACrefURL Software repository

    2024

  33. [33]

    , Lomshakov, V

    AGENTLENS_PROD APACrefauthors Podivilov, A. , Lomshakov, V. , Savin, S. , Startsev, M. , Pozharskiy, R. , Parshin, M. Nikolenko, S. APACrefauthors \ 2026 . AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation . AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation . APACrefURL https://arxiv.org/abs/2607.066...

    Pith/arXiv arXiv 2026

  34. [34]

    , Mittal, G

    AGENTLENS_LUCKY APACrefauthors Sahoo, P. , Mittal, G. , Li, X. , Ma, S. , Steenhoek, B. , Lin, P. Hu, Y. APACrefauthors \ 2026 . AgentLens: Revealing the Lucky Pass Problem in SWE-Agent Evaluation . AgentLens: Revealing the Lucky Pass Problem in SWE-Agent Evaluation . APACrefURL https://arxiv.org/abs/2605.12925 APACrefURL arXiv v3 preprint

    Pith/arXiv arXiv 2026

  35. [35]

    , Nguyen, T

    TRACEVIEW APACrefauthors Sajadi, A. , Nguyen, T. , Huynh, K. , Parra, E. Chatterjee, P. APACrefauthors \ 2026 . TraceView: Interactive Visualization of Agentic Program Repair Trajectories . TraceView: Interactive Visualization of Agentic Program Repair Trajectories . APACrefURL https://arxiv.org/abs/2606.22110 APACrefURL arXiv preprint

    Pith/arXiv arXiv 2026

  36. [36]

    APACrefauthors \ 1946

    SATTERTHWAITE1946 APACrefauthors Satterthwaite, F.E. APACrefauthors \ 1946 . An Approximate Distribution of Estimates of Variance Components An Approximate Distribution of Estimates of Variance Components . Biometrics Bulletin 2 6 110--114, APACrefDOI doi:10.2307/3002019 APACrefDOI

    doi:10.2307/3002019 1946

  37. [37]

    , Chong, C.Y

    TRACEPROBE APACrefauthors Shu, R. , Chong, C.Y. , Zhou, X. , Peng, Y. , Wu, Z. , Han, X. Wang, Y. APACrefauthors \ 2026 . What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents . What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents . APACrefURL https://arxiv.org/abs/2607.06184 APACrefURL arXiv preprint

    Pith/arXiv arXiv 2026

  38. [38]

    , Carver, J.C

    SHULL2008 APACrefauthors Shull, F.J. , Carver, J.C. , Vegas, S. Juristo, N. APACrefauthors \ 2008 . The role of replications in Empirical Software Engineering The role of replications in Empirical Software Engineering . Empirical Software Engineering 13 2 211--218, APACrefDOI doi:10.1007/s10664-008-9060-1 APACrefDOI

    work page doi:10.1007/s10664-008-9060-1 2008

  39. [39]

    , Barr, E.T

    SMITH2015 APACrefauthors Smith, E.K. , Barr, E.T. , Le Goues, C. Brun, Y. APACrefauthors \ 2015 . Is the cure worse than the disease? overfitting in automated program repair Is the cure worse than the disease? overfitting in automated program repair . Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering Proceedings of the 2015...

    2015

  40. [40]

    , Zhao, Y

    SWE_MUTATION APACrefauthors Sun, Y. , Zhao, Y. , Wang, Y. , Du, Y. , Ma, Z. , Wang, J. Huang, Z. APACrefauthors \ 2026 . SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering? SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering? APACrefURL https://arxiv.org/abs/2605.22175 APACrefURL Findings of ACL 2026

    Pith/arXiv arXiv 2026

  41. [41]

    , Zhang, C

    VERIFY_HORIZON APACrefauthors Wang, B. , Zhang, C. , Liu, D. , Zhang, J. , Chen, J. , Li, M. Cui, Z. APACrefauthors \ 2026 . The Verification Horizon: No Silver Bullet for Coding Agent Rewards . The Verification Horizon: No Silver Bullet for Coding Agent Rewards . APACrefURL https://arxiv.org/abs/2606.26300 APACrefURL arXiv preprint

    Pith/arXiv arXiv 2026

  42. [42]

    , Nguyen, T

    WEIMER2009 APACrefauthors Weimer, W. , Nguyen, T. , Le Goues, C. Forrest, S. APACrefauthors \ 2009 . Automatically finding patches using genetic programming Automatically finding patches using genetic programming . 2009 IEEE 31st International Conference on Software Engineering 2009 IEEE 31st international conference on software engineering \ ( \ 364--374)

    2009

  43. [43]

    APACrefauthors \ 1947

    WELCH1947 APACrefauthors Welch, B.L. APACrefauthors \ 1947 . The Generalization of `Student's' Problem when Several Different Population Variances are Involved The Generalization of `Student's' Problem when Several Different Population Variances are Involved . Biometrika 34 1--2 28--35, APACrefDOI doi:10.1093/biomet/34.1-2.28 APACrefDOI

    doi:10.1093/biomet/34.1-2.28 1947

  44. [44]

    \ Krishen, A

    WK2001 APACrefauthors Westfall, P.H. \ Krishen, A. APACrefauthors \ 2001 . Optimally weighted, fixed sequence and gatekeeper multiple testing procedures Optimally weighted, fixed sequence and gatekeeper multiple testing procedures . Journal of Statistical Planning and Inference 99 1 25--40, APACrefDOI doi:10.1016/s0378-3758(01)00077-5 APACrefDOI

    work page doi:10.1016/s0378-3758(01)00077-5 2001

  45. [45]

    , Deng, Y

    AGENTLESS APACrefauthors Xia, C.S. , Deng, Y. , Dunn, S. Zhang, L. APACrefauthors \ 2025 . Demystifying LLM-Based Software Engineering Agents Demystifying LLM-Based Software Engineering Agents . Proceedings of the ACM on Software Engineering 2 FSE 801--824, APACrefDOI doi:10.1145/3715754 APACrefDOI

    doi:10.1145/3715754 2025

  46. [46]

    , Wei, Y

    XIA2023 APACrefauthors Xia, C.S. , Wei, Y. Zhang, L. APACrefauthors \ 2023 . Automated Program Repair in the Era of Large Pre-trained Language Models Automated Program Repair in the Era of Large Pre-trained Language Models . 2023 IEEE/ACM 45th International Conference on Software Engineering ( ICSE ) 2023 IEEE/ACM 45th international conference on software...

    2023

  47. [47]

    \ Reiss, S.P

    DIFFTGEN APACrefauthors Xin, Q. \ Reiss, S.P. APACrefauthors \ 2017 . Identifying test-suite-overfitted patches through test case generation Identifying test-suite-overfitted patches through test case generation . Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis Proceedings of the 26th ACM SIGSOFT international ...

    2017

  48. [48]

    xu2026validation APACrefauthors Xu, X. \ Wu, W. APACrefauthors \ 2026 07 . Validation evidence in LLM repair agents: event-level, rollout-level, and task-level data. Validation evidence in LLM repair agents: event-level, rollout-level, and task-level data. Dataset, Zenodo . APACrefURL https://doi.org/10.5281/zenodo.21642576 APACrefURL

    work page doi:10.5281/zenodo.21642576 2026

  49. [49]

    , Jimenez, C.E

    SWEAGENT APACrefauthors Yang, J. , Jimenez, C.E. , Wettig, A. , Lieret, K. , Yao, S. , Narasimhan, K. Press, O. APACrefauthors \ 2024 . SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering . Advances in Neural Information Processing Systems ( NeurIPS ). Advanc...

    Pith/arXiv arXiv 2024

  50. [50]

    , Ruan, H

    AUTOCODEROVER APACrefauthors Zhang, Y. , Ruan, H. , Fan, Z. Roychoudhury, A. APACrefauthors \ 2024 . AutoCodeRover: Autonomous Program Improvement AutoCodeRover: Autonomous Program Improvement . Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis Proceedings of the 33rd ACM SIGSOFT international symposium on softwa...

    2024