Adversarial Test-Hardening for AI-Written Code: An Instrument Autopsy and a Pre-Registered Causal Estimate of the Critic Loop

Arizona State University); Jeff Otterson (W. P. Carey School of Business

REVIEW 1 major objections 3 minor 13 references

A pre-registered test-hardening experiment isolates a critic loop's effect and traces a false cross-model result to a silent output cap.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 03:52 UTC pith:FW7UYH4E

load-bearing objection The instrument autopsy is the durable contribution; the 78% causal estimate is credible but narrow, and the oracle-gaming vector deserves a receipt audit before the number is cited. the 1 major comments →

arxiv 2607.23002 v1 pith:FW7UYH4E submitted 2026-07-25 cs.SE cs.AI

Adversarial Test-Hardening for AI-Written Code: An Instrument Autopsy and a Pre-Registered Causal Estimate of the Critic Loop

Jeff Otterson (W. P. Carey School of Business , Arizona State University) This is my paper

classification cs.SE cs.AI

keywords mutation testingLLM test generationadversarial critic loopmechanical oraclepre-registrationinstrument artifactcross-model evaluationcausal estimate

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that an adversarial test-hardening loop, where a second LLM writes tests aimed at the exact injected defects an initial suite missed, substantially improves fault detection under a mechanical oracle. It reports a pre-registered Experiment 2 in which the same-lineage critic loop killed a mean 78% of surviving mutants left by a frozen initial suite, a within-replicate causal estimate. It also presents an instrument autopsy: an earlier apparently overwhelming cross-model effect (p = 9.5e-66) was manufactured by an undetected output cap that silently truncated one model's verbose replies. The paper argues that closing the model-judgment channel is necessary but not sufficient for neutral cross-model comparison, because harness asymmetries can still bias results. A sympathetic reader should care because coverage, the usual test-suite metric, records what ran rather than what was verified, and this loop directly targets the gaps a coverage-based gate would miss.

Core claim

On its own terms, this paper claims that a guarded, add-only critic loop—where a second model writes tests aimed at the exact injected defects the initial suite missed—produces a large, measurable hardening effect, and that the effect can be isolated by freezing and sharing the initial suite. Under that pre-registered frozen-round-0 design, same-lineage critic rounds killed a mean 78% of the survivors left by the frozen initial suite (0.783, 95% cluster-bootstrap interval [0.592, 0.935]). The cross-provider variant showed a positive pilot gap (0.178, [0.039, 0.347]) at lower cost, but the paper is explicit that this compares two bundled model-provider-harness configurations and that most of

What carries the argument

The central object is the adversarial test-hardening loop under a mechanical mutation-kill oracle: a Tester model writes an initial suite, mutation testing identifies the surviving injected defects by name, and a Critic model writes tests to kill exactly those, with every verdict decided mechanically by whether a test passes or fails on the mutated code. The load-bearing design mechanism is the frozen-shared-round-0 protocol, which commits one initial suite per replicate so all continuations start from byte-identical state, making the loop's incremental effect a within-replicate paired quantity. The load-bearing instrument is the fail-closed truncation detector that catches replies cut off b

Load-bearing premise

The 95% interval around the 78% estimate is built by cluster-bootstrapping only four subject means; the paper itself states in Section 6.4 that with four clusters percentile coverage is unreliable and discrete, so if that coverage is materially off the headline interval overstates certainty.

What would settle it

Re-run the same frozen-round-0 protocol on a substantially larger sample of independently chosen modules and check whether a subject-cluster bootstrap interval for the incremental kill rate stays above roughly 0.5; and separately, deliberately induce a truncation on the provider that declares no output cap to see whether the instrument detects it.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

When the initial suite is frozen and shared, adding critic rounds against named survivors kills, on average, 78% of the remaining mutants—a large incremental improvement over one-shot generation.
Cross-model comparisons under a mechanical oracle are not automatically neutral; any harness asymmetry correlated with the outcome can manufacture a false difference even when no model judges another model's output.
Fail-closed instrumentation that detects truncation, timeouts, and rejected rounds turns recurrent operational failures into scored, visible data rather than laundered bias.
Per-outcome cost accounting shows that one configuration's higher cost came from output verbosity and rejected rounds, not per-token price, so cost-per-kill is a more interpretable comparison base than cost per run.
The frozen-seed, pre-registered design removes the initial-suite resampling confound and yields paired causal estimates, while the paper's own sensitivity analyses show the pilot gap's magnitude is fragile even though its direction is stable.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the 78% incremental effect replicates on more subjects, critic loops with mechanical oracles could become a cheap, standard hardening step in test pipelines, with cost-per-kill as the natural optimization metric.
The residual detection asymmetry—truncation is checked only where a provider declares an output cap—suggests that other cross-model benchmark harnesses may hide similar artifacts; adding server-side truncation checks to all providers is a direct testable extension.
Because the pilot gap is dominated by one receipted operational failure, future confirmatory work should stratify by available headroom and report capability versus deployability separately rather than a single pooled gap.
A practical consequence not developed in the paper: the same fail-closed principle could be applied to other silent failure modes in LLM evaluation pipelines, such as retry loops that mask crashes or parsers that quietly discard one model's formatting.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

The instrument autopsy is the durable contribution; the 78% causal estimate is credible but narrow, and the oracle-gaming vector deserves a receipt audit before the number is cited.

read the letter

This is the rare empirical LLM paper that treats its own instrument as the object of study, and that is what makes it worth reading. The durable contribution is the autopsy: an output cap silently truncating one arm's replies manufactured a p=9.5e-66 cross-provider effect, and the authors caught it only through adversarial review of the completed analysis. That lesson generalizes, and the paper walks the walk by disclosing the recurrences, the invalidated cells, and the residual asymmetries.

The headline result—same-lineage critic rounds killing a mean 78% of frozen round-0 survivors—is a real empirical data point, but it is exactly as narrow as the authors say: conditional on four subjects, five replicates each, one tester model, and specific stopping rules. The frozen-shared-round-0 design is a clear improvement over Experiment 1 and correctly removes initial-draw variance. Per-replicate tables and leave-one-out sensitivities are published; cost per kill is a useful normalization.

Soft spots. The pooled interval cluster-bootstraps four subject means, and the paper admits percentile coverage is unreliable with four clusters. The two-stage bootstrap narrows the concern but doesn't fix the four-subject dependence; treat the interval as descriptive, not inferential. The truncation-detection asymmetry is structural—only the provider that declares a cap is checked—and the authors say so. Finally, the stress-test concern about oracle gaming deserves a serious look: the Critic is handed survivor diffs, so a test that inspects source text (e.g., asserts the original line is present) would pass on pristine and kill every mutant it targets. The paper discloses no syscall sandbox but never audits the public receipts for this pattern. Given the paper's own fail-loud philosophy, that audit should have been part of the protocol. Without it, the 0.783 estimate could overstate behavioral test-hardening if any meaningful share of kills are source-inspecting. I don't think this is fatal—most LLM-generated tests are behavioral—but it is a genuine hole in construct validity, and it happens to be exactly the kind of silent artifact the rest of the paper is about.

Verdict: send to peer review. The authors should be asked to audit the receipts for source-inspecting or environment-sniffing tests and to report the fraction of kills from such tests. The paper deserves a serious referee; it will be a useful cite for anyone working on mutation-guided test generation or cross-model evaluation.

Referee Report

1 major / 3 minor

Summary. The paper reports two pre-registered experiments and an instrument autopsy for an adversarial test-hardening loop under a mechanical mutation-kill oracle. Experiment 1 compares one-shot, same-lineage, and cross-lineage critic configurations on five Python subjects; H1 is supported descriptively, H2 is an underpowered null, and a previously reported highly significant H2 effect (p = 9.5e-66) is shown to be an artifact of a provider-correlated output-token truncation. Experiment 2 uses a frozen-shared-round-0 design with five replicates on four subjects, yielding a within-replicate estimate that the same-lineage critic kills 78% of survivors (rho_same = 0.783, 95% cluster-bootstrap interval [0.592, 0.935]) and a positive cross-configuration pilot gap (rho_gap = 0.178, 95% interval [0.039, 0.347]) dominated by one replicate. The paper releases protocols, receipts, and analysis code.

Significance. If the central estimate holds, the paper is a model of disciplined, pre-registered, receipted instrumentation in LLM-based testing. The autopsy of the provider-correlated output-cap artifact is a durable methodological contribution, and the public artifacts enable independent verification. The causal estimate is appropriately scoped to the frozen seeds, subjects, models, and stopping rules, and the per-replicate/per-subject transparency is exemplary. However, the construct validity of the mutation-kill oracle is not fully secured against source-inspecting tests, which is a load-bearing threat to the headline estimate; this issue is fixable with the published receipts.

major comments (1)

[§3.2, §3.4, §6.1] The Critic prompt includes the exact surviving mutants and their diffs (Section 3.2). A generated test can therefore kill a mutant by asserting that the original source text is present (e.g., via inspect.getsource or by reading the module file), which passes on the pristine module and fails on the mutated module. Such tests are not pruned by the pass-on-pristine check (Section 3.4) and are not behavioral fault-detection tests. The paper discloses the absence of a syscall sandbox (Section 6.1) but does not audit the receipts for this specific pattern. Because rho_same = 0.783 is the paper's central causal estimate, this is a load-bearing construct-validity concern. The authors should audit their released receipts for tests that read source files or assert on source text, and either exclude such tests or bound how much of the incremental kill rate they contribute. This is feasible with the

minor comments (3)

[Abstract] The headline interval [0.592, 0.935] is a four-subject cluster-bootstrap interval whose coverage the authors themselves describe as unreliable and discrete in Section 6.4. Please present the two-stage bootstrap [0.583, 0.964] as primary or add a prominent caveat in the abstract.
[§6.4] The statement that 'bootstrap interval' is used rather than 'confidence interval' is good; consider also applying the same caution to the Experiment 1 pooled McNemar p-values, which the text already labels descriptive.
[§4.8] The phrase 'direction is stable under every sensitivity we ran, never crossing zero' is correct given the reported intervals, but the lower bound of the two-stage interval after excluding the truncation replicate is 0.004. Consider adding an explicit sentence that the magnitude should be interpreted with this closeness to zero in mind.

Circularity Check

0 steps flagged

No significant circularity: the headline estimates are measured under a mechanical oracle from frozen, pre-registered inputs, with no fitted-parameter-as-prediction and no load-bearing self-citation.

full rationale

This is an empirical, pre-registered measurement study rather than a derived-from-first-principles prediction chain. The central claim — Experiment 2's mean incremental kill rate rho_same = 0.783 — is defined as a measured ratio Delta/|S| over a frozen, committed round-0 survivor set: "the incremental mutation kills produced by this guarded, add-only Critic continuation, conditional on these frozen seeds, subjects, models, and stopping rules." The quantity is the estimand; it is not fitted to one subset and then "predicted" on another, and no equation in the paper reduces rho_same to the prompt or to the survivor diffs by construction. The survivor-feedback mechanism (Critic is handed named survivors and their diffs) is the intervention under test, not a hidden input that makes the outcome definitional: killing a named mutant is the mechanical oracle's verdict, and the paper explicitly frames the loop as a replication of MuTAP/AdverTest rather than a novel derivation. The cross-configuration pilot is likewise a measured within-replicate gap, reported with per-replicate values, leave-one-out sensitivities, and disclosure that part of the gap is one configuration's receipted operational failures; that is a validity caveat, not a circular reduction. No author-overlap citation chain is load-bearing: the compared methods (MuTAP, AdverTest, Refute-or-Promote) are external prior work, and the paper claims no uniqueness theorem or authoritative self-citation to force its choices. The closest concern — that generated tests could game the oracle by inspecting source text — is a construct-validity threat (partially disclosed in Section 6.1 as the absence of a syscall sandbox), not a circularity: it does not make the measured kill rate equivalent to an input by definition. The paper's own stated statistical limitations (four-subject cluster bootstrap, discrete percentile coverage, ceiling effects) are honest uncertainty bounds, not evidence that results were assumed into existence. Overall, no step in the derivation chain reduces to its own inputs, so the appropriate circularity score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

No entities beyond the experimental framing; the Critic loop is prior art (MuTAP/AdverTest). Free parameters are methodological thresholds chosen by hand and disclosed; the central claims are conditional on them. The axioms are standard SE/statistical assumptions the paper itself identifies in its Threats to Validity section.

free parameters (4)

Loop stopping rules (max_rounds=4, dry_rounds=2) = 4 and 2
Hand-chosen without a power analysis (disclosed §6.4); the headline rho=0.783 is conditional on these stopping rules and would be larger with more rounds.
Flake check (two consecutive passes on pristine code) = 2 runs
Hand-chosen threshold; the paper notes a ~1/3-flaky test still passes ~4/9 of the time, and one such marginal test was caught later by Experiment 2's reproduction check.
Output token cap (16,000 in Experiment 1; 32,000 after the repair) = 16,000 → 32,000
The artifact source; the value is arbitrary and provider-side. The cross-provider arm sends no cap and is never truncation-checked (disclosed asymmetry, §5.2).
Replicate and seed-draw budget (5 replicates/subject; 3 draws/replicate cap; 7 draws/subject cap) = 5, 3, 7
Fixed in advance, not adapted to data; buys unequal precision across subjects because round-0 variance is strongly subject-dependent (disclosed §6.4).

axioms (5)

domain assumption Mutation kills (mutmut, rule-based mutants) proxy fault-detection ability.
Invoked throughout §3.4; the paper cites the classic construct-validity caveat and performs no equivalent-mutant detection, so all kill rates are lower bounds.
domain assumption The mechanical oracle produces ground-truth verdicts (a test kills a mutant or it does not under the runner).
§3.1; this premise closes the self-preference channel. The autopsy shows the surrounding apparatus can still be asymmetric, but the verdict itself is treated as ground truth.
domain assumption The four subject modules are exchangeable clusters for bootstrap inference.
§3.10 and §6.4; the pooled 95% intervals cluster-bootstrap four subject means, and the paper admits coverage is unreliable with four clusters.
ad hoc to paper The cross-lineage provider never truncates replies server-side.
§5.2/§6.2: only the same-provider wrapper declares a cap and is truncation-checked; if the other provider truncated, it would be laundered exactly as the original defect was.
domain assumption Generated tests cannot detect and exploit the mutation harness (no syscall sandbox).
§6.1; admitted as a residual construct threat, reduced but not closed by add-only tests, pass-on-pristine validation, flake check, and wrong-oracle pruning.

pith-pipeline@v1.3.0-alltime-deepseek · 21519 in / 12309 out tokens · 122736 ms · 2026-08-01T03:52:05.361927+00:00 · methodology

0 comments

read the original abstract

Large language models increasingly write both code and the tests meant to check it; coverage records what ran, not what was verified. We study an adversarial test-hardening loop under a mechanical oracle: a Tester model writes tests, mutation testing names surviving injected defects, and a Critic model writes tests to kill exactly those, with every verdict decided mechanically, so no model judges another's output. In Experiment 1, on five Python subjects (one same-lineage-loop cell could not be scored), the loop killed 105 mutants that one-shot generation missed and lost none, and the cross-lineage-Critic question returned a pre-declared null. The central finding was an autopsy: an earlier analysis reported a cross-lineage effect at p = 9.5e-66 that was an instrument artifact, an output cap silently truncating the verbose model, caught only by adversarial review of the completed analysis. Review then found a further confound, each arm resampling its own initial suite; Experiment 2 removes it. Under a pre-registered frozen-shared-round-0 design (five replicates on each of four subjects, seeds committed in advance), same-lineage Critic rounds killed 78% of the survivors the frozen initial suite left standing (mean incremental kill rate 0.783, 95% cluster-bootstrap interval [0.592, 0.935]), a within-replicate causal estimate; the cross-provider configuration showed a positive pilot difference (rate gap 0.178, 95% interval [0.039, 0.347]; magnitude dominated by a single replicate) at 5.5x lower arm cost. This compares two named model-provider-harness configurations, not an isolated lineage effect: part of the gap is one configuration's receipted operational failures, including truncation recurrences, now detected and scored rather than laundered. Cross-model comparisons can inherit the asymmetries of the harness that runs them. We release both protocols, all receipts, and the analysis code.

discussion (0)

Reference graph

Works this paper leans on

13 extracted references · 12 linked inside Pith

[3]

Automated Unit Test Improvement using Large Language Models at Meta

arXiv:2408.11324. • TestGen-LLM: N. Alshahwan et al. “Automated Unit Test Improvement using Large Language Models at Meta. ” FSE 2024 (Industry). arXiv:2402.09171; DOI 10.1145/3663529.3663839. • MuTAP: A. M. Dakhel, A. Nikanjam, F. Khomh, M. C. Desmarais, H. Washizaki. arXiv:2308.16557; Information and Software Technology, 2024, DOI 10.1016/j.infsof.2024....

Pith/arXiv arXiv 2024
[4]

Improving Factuality and Rea- soning in Language Models through Multiagent Debate

arXiv:2404.13076. • Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, I. Mordatch. “Improving Factuality and Rea- soning in Language Models through Multiagent Debate. ” ICML

Pith/arXiv arXiv
[5]

Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?

arXiv:2305.14325. • Self-MoA: W. Li, Y. Lin, M. Xia, C. Jin. “Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?”

Pith/arXiv arXiv
[8]

TestForge: Feedback-Driven, Agentic Test Suite Gen- eration

arXiv:2502.04313. • TestForge: K. Jain, C. Le Goues. “TestForge: Feedback-Driven, Agentic Test Suite Gen- eration. ”

Pith/arXiv arXiv
[9]

EvoSuite: Automatic Test Suite Generation for Object- Oriented Software

arXiv:2503.14713. • EvoSuite: G. Fraser, A. Arcuri. “EvoSuite: Automatic Test Suite Generation for Object- Oriented Software. ” ESEC/FSE 2011, pp. 416-419. DOI 10.1145/2025113.2025179. • Registered Reports in Software Engineering: N. A. Ernst, M. T. Baldassarre. Empirical Software Engineering,

Pith/arXiv arXiv 2011
[10]

Watts and Debts of Agentic Frameworks: An Empirical Study (Registered Report)

arXiv:2302.03649. • Watts and Debts: A. S. Shany, S. Chandrasekar, K. Vaidhyanathan. “Watts and Debts of Agentic Frameworks: An Empirical Study (Registered Report). ” ESEM 2026 (Registered Reports Track). arXiv:2606.10702. • Oracle Problem: E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, S. Yoo. “The Oracle Problem in Software Testing: A Survey. ” IEEE Tran...

Pith/arXiv arXiv 2026
[126]

An Empirical Analysis of Flaky Tests

arXiv:2308.04748; DOI 10.1145/3597503.3639121. • Flaky Tests: Q. Luo, F. Hariri, L. Eloussi, D. Marinov. “An Empirical Analysis of Flaky Tests. ” FSE 2014, pp. 643-653. DOI 10.1145/2635868.2635920. 26

Pith/arXiv arXiv 2014
[2015]

Mutation Testing Advances: An Analysis and Survey

DOI 10.1109/TSE.2014.2372785. • Mutation Survey: M. Papadakis, M. Kintis, J. Zhang, Y. Jia, Y. Le Traon, M. Harman. “Mutation Testing Advances: An Analysis and Survey. ” Advances in Computers, vol. 112, pp. 275-378,

arXiv 2014
[2019]

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

DOI 10.1016/bs.adcom.2018.03.015. • LLM-as-judge: L. Zheng et al. “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. ” NeurIPS 2023 (Datasets and Benchmarks). arXiv:2306.05685. • Defects4J: R. Just, D. Jalali, M. D. Ernst. “Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs. ” ISSTA 2014, pp. 437-440. DOI...

Pith/arXiv arXiv 2018
[2023]

Pynguin: Automated Unit Test Genera- tion for Python

• Pynguin: S. Lukasczyk, G. Fraser. “Pynguin: Automated Unit Test Genera- tion for Python. ” ICSE-Companion 2022, pp. 168-172. arXiv:2202.05218; DOI 10.1145/3510454.3516829. • HITS: Z. Wang, K. Liu, G. Li, Z. Jin. “HITS: High-coverage LLM-based Unit Test Gen- eration via Method Slicing. ” ASE

Pith/arXiv arXiv 2022
[2024]

No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation

arXiv:2302.06527; DOI 10.1109/TSE.2023.3334955. • ChatTester: Z. Yuan, Y. Lou, M. Liu, S. Ding, K. Wang, Y. Chen, X. Peng. “No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation. ” arXiv:2305.04207. • ChatUniTest: Y. Chen et al. “ChatUniTest: A Framework for LLM-Based Test Genera- tion. ” arXiv:2305.04764. • CoverUp: J. A. Pizzor...

Pith/arXiv arXiv 2023
[2025]

Refute-or-Promote: An Adversarial Stage-Gated Multi- Agent Review Methodology for High-Precision LLM-Assisted Defect Discovery

arXiv:2502.00674. • Refute-or-Promote: A. Agarwal. “Refute-or-Promote: An Adversarial Stage-Gated Multi- Agent Review Methodology for High-Precision LLM-Assisted Defect Discovery. ”

Pith/arXiv arXiv
[2026]

Great Models Think Alike and this Undermines AI Oversight

arXiv:2604.19049. • Great Models Think Alike: S. Goel, J. Struber, I. A. Auzina, K. K. Chandra, P. Ku- maraguru, D. Kiela, A. Prabhu, M. Bethge, J. Geiping. “Great Models Think Alike and this Undermines AI Oversight. ”

Pith/arXiv arXiv

[1] [3]

Automated Unit Test Improvement using Large Language Models at Meta

arXiv:2408.11324. • TestGen-LLM: N. Alshahwan et al. “Automated Unit Test Improvement using Large Language Models at Meta. ” FSE 2024 (Industry). arXiv:2402.09171; DOI 10.1145/3663529.3663839. • MuTAP: A. M. Dakhel, A. Nikanjam, F. Khomh, M. C. Desmarais, H. Washizaki. arXiv:2308.16557; Information and Software Technology, 2024, DOI 10.1016/j.infsof.2024....

Pith/arXiv arXiv 2024

[2] [4]

Improving Factuality and Rea- soning in Language Models through Multiagent Debate

arXiv:2404.13076. • Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, I. Mordatch. “Improving Factuality and Rea- soning in Language Models through Multiagent Debate. ” ICML

Pith/arXiv arXiv

[3] [5]

Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?

arXiv:2305.14325. • Self-MoA: W. Li, Y. Lin, M. Xia, C. Jin. “Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?”

Pith/arXiv arXiv

[4] [8]

TestForge: Feedback-Driven, Agentic Test Suite Gen- eration

arXiv:2502.04313. • TestForge: K. Jain, C. Le Goues. “TestForge: Feedback-Driven, Agentic Test Suite Gen- eration. ”

Pith/arXiv arXiv

[5] [9]

EvoSuite: Automatic Test Suite Generation for Object- Oriented Software

arXiv:2503.14713. • EvoSuite: G. Fraser, A. Arcuri. “EvoSuite: Automatic Test Suite Generation for Object- Oriented Software. ” ESEC/FSE 2011, pp. 416-419. DOI 10.1145/2025113.2025179. • Registered Reports in Software Engineering: N. A. Ernst, M. T. Baldassarre. Empirical Software Engineering,

Pith/arXiv arXiv 2011

[6] [10]

Watts and Debts of Agentic Frameworks: An Empirical Study (Registered Report)

arXiv:2302.03649. • Watts and Debts: A. S. Shany, S. Chandrasekar, K. Vaidhyanathan. “Watts and Debts of Agentic Frameworks: An Empirical Study (Registered Report). ” ESEM 2026 (Registered Reports Track). arXiv:2606.10702. • Oracle Problem: E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, S. Yoo. “The Oracle Problem in Software Testing: A Survey. ” IEEE Tran...

Pith/arXiv arXiv 2026

[7] [126]

An Empirical Analysis of Flaky Tests

arXiv:2308.04748; DOI 10.1145/3597503.3639121. • Flaky Tests: Q. Luo, F. Hariri, L. Eloussi, D. Marinov. “An Empirical Analysis of Flaky Tests. ” FSE 2014, pp. 643-653. DOI 10.1145/2635868.2635920. 26

Pith/arXiv arXiv 2014

[8] [2015]

Mutation Testing Advances: An Analysis and Survey

DOI 10.1109/TSE.2014.2372785. • Mutation Survey: M. Papadakis, M. Kintis, J. Zhang, Y. Jia, Y. Le Traon, M. Harman. “Mutation Testing Advances: An Analysis and Survey. ” Advances in Computers, vol. 112, pp. 275-378,

arXiv 2014

[9] [2019]

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

DOI 10.1016/bs.adcom.2018.03.015. • LLM-as-judge: L. Zheng et al. “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. ” NeurIPS 2023 (Datasets and Benchmarks). arXiv:2306.05685. • Defects4J: R. Just, D. Jalali, M. D. Ernst. “Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs. ” ISSTA 2014, pp. 437-440. DOI...

Pith/arXiv arXiv 2018

[10] [2023]

Pynguin: Automated Unit Test Genera- tion for Python

• Pynguin: S. Lukasczyk, G. Fraser. “Pynguin: Automated Unit Test Genera- tion for Python. ” ICSE-Companion 2022, pp. 168-172. arXiv:2202.05218; DOI 10.1145/3510454.3516829. • HITS: Z. Wang, K. Liu, G. Li, Z. Jin. “HITS: High-coverage LLM-based Unit Test Gen- eration via Method Slicing. ” ASE

Pith/arXiv arXiv 2022

[11] [2024]

No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation

arXiv:2302.06527; DOI 10.1109/TSE.2023.3334955. • ChatTester: Z. Yuan, Y. Lou, M. Liu, S. Ding, K. Wang, Y. Chen, X. Peng. “No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation. ” arXiv:2305.04207. • ChatUniTest: Y. Chen et al. “ChatUniTest: A Framework for LLM-Based Test Genera- tion. ” arXiv:2305.04764. • CoverUp: J. A. Pizzor...

Pith/arXiv arXiv 2023

[12] [2025]

Refute-or-Promote: An Adversarial Stage-Gated Multi- Agent Review Methodology for High-Precision LLM-Assisted Defect Discovery

arXiv:2502.00674. • Refute-or-Promote: A. Agarwal. “Refute-or-Promote: An Adversarial Stage-Gated Multi- Agent Review Methodology for High-Precision LLM-Assisted Defect Discovery. ”

Pith/arXiv arXiv

[13] [2026]

Great Models Think Alike and this Undermines AI Oversight

arXiv:2604.19049. • Great Models Think Alike: S. Goel, J. Struber, I. A. Auzina, K. K. Chandra, P. Ku- maraguru, D. Kiela, A. Prabhu, M. Bethge, J. Geiping. “Great Models Think Alike and this Undermines AI Oversight. ”

Pith/arXiv arXiv