{"id":"f82ef796-060e-4b2e-bfbb-ffe6737d7db2","arxiv_id":"2607.28871","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"By replaying each validation command on buggy, candidate, and gold-fix code, this study finds 46% of positive checks do not discriminate the bug, and a reminder alone explains part of the feedback benefit.","lead":"A new replay-based analysis of LLM repair agents finds that nearly half of the positive test results agents collect mid-trajectory carry no information about the reported bug. The paper also tests a cheap feedback intervention and shows it reduces such evidence-inadequate closures, though the effect is below the preset threshold of practical importance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"B-replay failures from changed production APIs may be misread as bug discrimination, inflating bug-discriminating evidence and the BCF effect.","rationale":"The reader's weakest assumption identifies replay faithfulness as the key vulnerability. My concern is a specific, unhandled violation of that assumption: the test-only patch is replayed on B with the original code, but the test may depend on the agent's production patch not only via new symbols (which are flagged as NOT COMPARABLE) but also via changed interfaces to existing symbols. The latter produce B-failures that are interpreted as bug detection. This directly affects the central prevalence estimate and, more importantly, the confirmatory intervention effect: BCF tells agents whether their test passes on B, so agents in BCF can deliberately adjust their production code and tests to make the B-replay fail, even if the failing property is unrelated to the reported bug. The effect on EIC might therefore be partly a measurement artifact rather than a genuine improvement in evidence quality. The paper's own robustness checks (Appendix A) do not stratify by B-failure type, and the NOT COMPARABLE category only covers import errors. The proposed audit—manually classifying a stratified sample of B-fail events—would settle whether this artifact is substantial. If the reclassification changes the headline estimates materially, the paper's conclusions would need to be weakened; if not, the current CONDITIONAL verdict stands. I keep the verdict UNCHANGED because the reader already required conditional acceptance, and this concern reinforces that condition without moving the verdict category.","tokens_in":17045,"tokens_out":7669,"duration_ms":89303,"concrete_test":"Sample ~150 events classified as gold-aligned bug-discriminating or candidate-specific, stratified by arm (BCF, Static Reminder, Baseline). For each, inspect the B-replay execution trace and the agent's production diff to determine whether the test's failure on B is caused by an interface change (TypeError/AttributeError/NameError on a call or attribute, changed signature, removed symbol) rather than by an assertion failing because the buggy behavior is exercised. Recompute the role distribution and the BCF-vs-Reminder EIC contrast after reclassifying all interface-mismatch failures as NOT COMPARABLE. If the reclassified EIC effect shrinks by >3 pp or the 46.0% prevalence moves by >5 points, the central conclusions depend on this artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The BSG-VA taxonomy treats any B-replay failure as evidence of bug discrimination, except for import errors caused by newly introduced production symbols (Sections 3.2, 3.4). It does not exclude failures caused by changes to the interface of existing symbols. An agent may edit an existing function's signature, add a required argument, or rename a method; a test written against the new interface will raise TypeError/AttributeError when replayed on B. That failure is recorded as 'B fails' and counted as bug-discriminating (gold-aligned or candidate-specific), even though the test never asserts a behavior related to the defect. This is not hypothetical: BCF explicitly rewards tests that fail on B, so agents in the BCF arm are incentivized to write tests coupled to their own production edits, mechanically producing B-failures. The headline 46.0% non-discriminating share and the 7.8 pp EIC reduction could both be artifacts if a nontrivial fraction of the 1,377 B-fail/S-pass events (Table 3) are interface-mismatch failures rather than bug-detecting assertions. The paper's robustness checks do not stratify by failure mode, and the NOT COMPARABLE flag is too narrow.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BSG-VA, a replay-based method that captures each validation command a repair agent executes, extracts a test-only patch, and replays the command on the original buggy state (B), the candidate state (S), and the gold-fix state (G). Evidence roles are assigned from the captured outcome and replay pattern. In a controlled experiment with gpt-5.6-sol on 110 SWE-bench Verified / SWE-rebench tasks, the paper reports that 46.0% of 2,548 positive comparable validation events carry no bug-discriminating information, and that 23.8% of baseline rollouts close with evidence-inadequate closure. A three-arm experiment (baseline, static reminder, bug-contrast feedback) finds that BCF reduces evidence-inadequate closure by 7.8 pp relative to the reminder (p=0.0029) and raises bug-discriminating evidence by 7.4 pp (p=0.011), with both estimates below the prespecified 10 pp SESOI. Exploratory replications vary the scaffold and model. The central claims are that the proposed taxonomy provides an auditable measure of validation-evidence quality, and that the non-discriminating-evidence phenomenon is prevalent and partially addressable by feedback.","tokens_in":17290,"tokens_out":5095,"duration_ms":69218,"significance":"If the measurement method is faithful, this is a valuable contribution: it moves beyond trajectory-level counts and coarse pass/fail agreement to an event-level, executable audit of what an agent's own validation commands actually establish. The study design is unusually careful for this area: a prespecified analysis plan, a smallest effect size of interest, a fixed-sequence confirmatory family, an attention-matched control arm, leave-one-repository-out and wild-bootstrap robustness checks, balanced cell loss, and honest reporting that the confirmatory point estimates fall below the SESOI. The released dataset is a useful asset. The main weakness is that the replay-faithfulness assumption — that a B-replay failure means the test detects the reported bug rather than merely depending on the agent's production edits — is load-bearing and is not independently audited. The paper would be strengthened substantially by a failure-mode analysis of B-replay failures and by public release of the replay/capture code, not just the aggregate data.","major_comments":[{"comment":"The B-fail criterion conflates genuine bug detection with interface breakage. Section 3.2 excludes only import errors caused by newly introduced production symbols; failures caused by changed signatures, renamed methods, or added required arguments on existing symbols are counted as B-fail. 1,377 B-fail/S-pass events (Table 3) therefore include failures of tests written against the agent's modified API, which would fail on B for reasons unrelated to the reported defect. This directly inflates the gold-aligned and candidate-specific counts and, in turn, the 46.0% non-discriminating share and the BCF effect. The manuscript's own NOT COMPARABLE flag is too narrow to address this. Please stratify all B-fail events by failure signature (e.g., TypeError/AttributeError/ImportError vs. assertion failure) and report the central estimates restricted to failures that do not reference symbols modifi","section":"§3.3–3.4, Table 3"},{"comment":"The replay-faithfulness assumption is asserted, not demonstrated. Section 3.3 checks S-replay agreement with the captured outcome, but this only verifies that the candidate state reproduces what the agent saw; it does not validate that a B-replay failure is semantically a bug-detecting assertion. The paper reports no audit of a sample of B-replay failures to distinguish defect-related failures from interface-mismatch failures or other environment artifacts. Given that the taxonomy, the EIC construct, and the primary outcome are all defined by these replay outcomes, an independent audit of a stratified sample (or a reproducible extractions/replay pipeline) is needed to establish construct validity. The paper's release does not include the capture/replay code, so this cannot currently be checked externally.","section":"§3.2–3.3"},{"comment":"The BCF arm's feedback rule rewards any B-fail: the message states that the check 'distinguishes the two states.' Under this rule, an agent whose test fails on B because of a changed interface receives the same positive signal as an agent whose test fails because it detects the defect. This creates a differential incentive across arms: BCF may increase B-fail events by encouraging tests coupled to the agent's production edits, mechanically raising bug-discriminating evidence and reducing EIC without improving genuine evidence quality. The paper's robustness checks do not address this differential misclassification. A sensitivity analysis restricting the outcome to events whose B-fail is not attributable to modified interfaces is necessary to support the claim that the intervention changes evidence quality rather than the measurement artifact.","section":"§4.2, Table 1; §5.3"}],"minor_comments":[{"comment":"Typographical issues: 'hereaftersol' should read 'hereafter sol' and 'hereafterthe' should read 'hereafter the'.","section":"§1"},{"comment":"Minor formatting: 'fromGOLD ALIGNED' should read 'from GOLD ALIGNED'.","section":"§3.4"},{"comment":"p-values such as 0.00018 and 0.00058 are better reported as p<0.001 in the text and table for readability.","section":"§5.3, Table 4"},{"comment":"Unicode/TeX issues in author names (e.g., 'M¨ undler', '¨Orwall') may render as combining diacritics; please fix the LaTeX/Unicode encoding.","section":"§4.2 / References"},{"comment":"The statement that the code is 'not publicly released' is a reproducibility limitation for a method whose central contribution is a deterministic replay pipeline. At minimum, the replay/harness code should be released alongside the data, or a replication package should be promised as a condition of publication.","section":"Code availability"},{"comment":"The acknowledged limitation about pre-edit commands is good, but it is not quantified. A descriptive table of events excluded because they occur before the first production edit would help readers gauge the scope of the prevalence claim.","section":"§6.3"}],"recommendation":"major_revision","confidential_remarks":"This is a well-designed and unusually honest empirical study. The central claims, however, rest on the replay classification, and the current treatment of B-fail events is not sufficiently cautious: interface-mismatch failures are not distinguished from bug-detecting failures, and the BCF arm's message explicitly labels any B-fail as distinguishing the two states. I would want to see a failure-mode stratification and a sensitivity analysis excluding interface-mismatch events before this is publishable as the definitive prevalence estimate. If the results survive that analysis, this could be a strong contribution. I also note that the code is not released, which weakens the otherwise exemplary data-availability story."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper you'd want to know about: BSG-VA, a way to replay each test an agent runs during repair against the buggy, candidate, and gold-fixed states, and classify what that test actually proves. The headline finding — 46% of positive validation events don't discriminate the bug — is directionally believable, and the measurement method is genuinely new. Prior work counted tests or generated external fail-to-pass tests; nobody had audited the agent's own spontaneous validation commands this way.\n\nWhat's good: the experimental design is unusually careful. Prespecified analysis plan, SESOI set before execution, three-arm design with an attention-matched reminder, robustness checks (wild bootstrap, sign test, leave-one-repository-out), and honest reporting that effects fall below the SESOI. The active-control decomposition is a nice idea, and the exploratory replications are transparently labeled. They also released the event-level data, which is real credit.\n\nNow the soft spots, in proportion. The load-bearing assumption is that replaying the test-only patch on B faithfully reproduces what the agent proved. The paper flags import errors from newly introduced production symbols as NOT COMPARABLE, but it does not handle changes to the interface of existing symbols — a renamed method, an added argument, a changed signature. A test written against the new interface will fail on B with a TypeError, and that failure gets scored as bug-discriminating even though it says nothing about the reported defect. This is not hypothetical: BCF explicitly rewards tests that fail on B, so agents in that arm have an incentive to couple their tests to their own production edits. The paper's robustness checks don't stratify B-failures by failure mode. If a nontrivial fraction of the 1,377 B-fail/S-pass events are interface mismatches, the 46% non-discriminating share is an underestimate and the BCF effect is inflated. The direction of the conclusion probably survives, but the precision doesn't. This should be audited before the numbers are used as prevalence estimates.\n\nTwo smaller issues: the analysis code isn't released, which limits independent reproduction of the exact statistics despite the data release. And the 46% event-level figure is pooled across arms, even though the intervention changes the outcome; a baseline-stratified table would be cleaner. Both are minor relative to the replay-equivalence gap.\n\nWho this is for: anyone working on LLM repair agents or test generation. Worth a serious referee — the method is novel and the evidence quality is above typical bar for this area. But the authors should be asked to audit B-replay failure modes and release the code. As is, I'd cite it for the method, with a caveat on the estimates.\n\nRecommended: send to peer review with a request for that audit.","headline":"A useful new measurement layer for agent validation evidence, but the replay-equivalence blind spot (interface changes scored as bug discrimination) needs an audit before the headline rates are trusted.","tokens_in":17782,"tokens_out":3165,"would_cite":true,"duration_ms":35778,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces a method to audit what passing tests in LLM repair runs actually prove, and shows that nearly half of positive validation events carry no bug-discriminating information.","keywords":["program repair agents","validation evidence","test adequacy","large language models","evidence roles","bug-contrast feedback","evidence-inadequate closure","software quality"],"falsifier":"Manually re-run a random sample of events classified as regression-only with the full working tree (production edits included, not just the test-only patch) on the original buggy state in the original container; if a substantial share then fail, the test-only decomposition is unfaithful and the 46.0% estimate is not about the bug's detectability but about the replay construction.","tokens_in":16910,"feed_emoji":"🐛","tokens_out":4865,"duration_ms":45219,"temperature":0.7,"pith_summary":"The paper asks how often a passing test run by a repair agent actually provides evidence about the reported defect. It introduces BSG-VA, which captures each validation command during a repair trajectory, isolates the test-only changes, and replays them on the original buggy code, the candidate patch, and the developer's gold fix. Across 3,730 events, 46.0% of positive comparable events were regression-only or misleading, and 23.8% of baseline rollouts closed with no discriminating evidence at all. A three-arm experiment shows that feeding the buggy-state replay result back to the agent reduces such evidence-poor closures by 7.8 percentage points, though this falls below the pre-specified 10-percentage-point threshold of practical interest.","feed_headline":"46% of agents' positive test checks don't test the bug","feed_subtitle":"A quarter of agent submissions close on such evidence; replaying tests on the buggy base cuts that by 7.8 points.","key_machinery":"The central object is BSG-VA (buggy-state/candidate-state/gold-fix validation analysis), which captures every validation command the agent executes, snapshots the working tree, separates a test-only patch from concurrent production edits, and replays the same command in a pinned container on three code states: the original buggy code (B), the candidate state (S), and the developer gold fix (G). The captured outcome plus the three replay outcomes map each event into one of seven exhaustive, mutually exclusive evidence roles, anchored by whether the check fails on B and whether it passes on G. This replay machinery is what turns 'the agent ran a test and it passed' into an auditable statement","core_discovery":"The central discovery is a prevalence estimate with a method behind it: mid-trajectory passing tests in LLM repair agents frequently do not discriminate the reported bug. BSG-VA assigns each validation event an evidence role by replaying the extracted test-only patch on buggy (B), candidate (S), and gold-fix (G) states. Events are gold-aligned bug-discriminating if they fail on B and pass on both S and G; candidate-specific if they fail on B and pass on S but fail on G; regression-only if they pass on B; and misleading if they pass on B but fail on G. The method yields 46.0% non-discriminating positives among positive comparable events and 23.8% baseline evidence-inadequate closure at the ro","pith_inferences":["The 46.0% figure likely underestimates the trajectory-wide rate, because the measurement starts only after the first production edit; pre-edit diagnostic checks are excluded.","The method generalizes beyond the specific models and benchmarks studied: any replayable trajectory that preserves code states and a pinned environment can be audited, so the same audit could be applied to different scaffolds, languages, or agent families.","A natural extension is to use evidence-role labels as a reward or filtering signal in reinforcement learning, training agents to maximize bug-discriminating evidence rather than raw pass counts.","The finding that the reminder alone accounts for roughly a third of the effect suggests a cheap deployment lever; whether the B-replay increment is worth its infrastructure cost depends on the base rate of evidence-inadequate closure in the target setting."],"forward_implications":["If nearly half of positive validation evidence is non-discriminating, benchmark pass rates substantially overstate how often agents have actually verified the reported defect.","Evidence-inadequate closure quantifies a new failure mode: an agent can submit a patch on the basis of positive but irrelevant checks; integrators can now audit trajectories for this before trusting a patch.","Returning the buggy-state replay result to the agent is a practical, low-cost intervention (median 11 seconds per rollout) that reduces evidence-poor closures, and a generic reminder does part of the work.","Evidence quality carries predictive signal for official resolution beyond task-level covariates, so validation-evidence roles could serve as an online process signal for model selection or early stopping."],"fun_headline_variants":["46% of passing tests in agent repairs don't test the bug","One in four agent fixes close on tests that never check the bug","Nearly half of passing tests in agent repairs ignore the actual bug","A quarter of agent fixes rely on tests that miss the bug","Passing tests often miss the bug in LLM repair agents"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that replaying the extracted test-only patch on the buggy, candidate, and gold states in a fresh pinned container faithfully reproduces what the agent's validation command actually proved at the original working tree; the paper itself flags that it does not independently audit this replay equivalence, so if a test fails on B because of an undeclared dependency on the agent's production edits rather than because it detects the bug, the role assignme","fun_headline_variants_meta":{"raw":{"variants":["46% of passing tests in agent repairs don't test the bug","One in four agent fixes close on tests that never check the bug","Nearly half of passing tests in agent repairs ignore the actual bug","A quarter of agent fixes rely on tests that miss the bug","Passing tests often miss the bug in LLM repair agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001004,"raw_usage":{"total_tokens":4170,"prompt_tokens":916,"completion_tokens":3254,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":660,"completion_tokens_details":{"reasoning_tokens":3180}},"tokens_in":660,"tokens_out":3254,"duration_ms":23899,"temperature":1.0,"reasoning_tokens":3180,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T01:26:09.545082+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually re-run a random sample of events classified as regression-only with the full working tree (production edits included, not just the test-only patch) on the original buggy state in the original container; if a substantial share then fail, the test-only decomposition is unfaithful and the 46.0% estimate is not about the bug's detectability but about the replay construction.","supporting_citations":[],"review_version":1}