{"id":"167a6d91-a9fd-47bb-8c45-3c6e5d046065","arxiv_id":"2501.07531","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A SWE-Agent-style AI repaired 73% of machine-reported and 25.6% of human-reported bugs in a curated 178-bug Google-internal benchmark, with 43% and 17.9% matching the human fix.","lead":"Google researchers tested an AI coding agent, Passerine, on 178 real bugs from Google's internal issue tracker. The agent produced passing patches for 73% of machine-reported bugs and 25.6% of human-reported ones, and the paper argues enterprise bugs differ from the public SWE-Bench dataset.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TOD plausibility is measured under a single test ordering and agent test-file edits are not evaluated, so the 73% machine-reported 'plausible' rate may be inflated; re-running with multiple orderings and reverted tests would settle it.","rationale":"The paper's central contribution is an empirical baseline: Passerine produces plausible patches for 73% of machine-reported and 25.6% of human-reported bugs in GITS-Eval, with lower valid rates after manual annotation. The most load-bearing assumption behind the machine-reported plausible number is not the representativeness of the curation funnel broadly, but the correctness of the test oracle used to declare a patch plausible. For TOD bugs, that oracle is explicitly acknowledged to be too lenient: it checks only one test ordering, while the bug class is defined by order dependence across orderings. Because TOD constitutes 28% of the benchmark and its plausible rate (68%) far exceeds its valid rate (24%), this single weakness can move the combined machine-reported plausible rate by many percentage points. The additional statement that agent modifications to test files are not evaluated creates a plausible mechanism for even more inflation: a TOD agent could neutralize the reported ordering by editing the test rather than fixing the underlying shared-state bug. The paper reports both plausible and valid rates, which is honest and partially mitigates the concern, but the abstract and conclusion lead with the plausible numbers. A concrete re-evaluation under multiple orderings and with reverted test files would settle whether the concern lands. This is not an attack on the authors' integrity; it is a targeted check of an admitted methodological soft spot. The reader's weakest assumption identified the general area of oracle assumptions; this response sharpens that to a specific, testable failure mode, hence partial agreement. The verdict remains conditional pending that check, so no change to the reader's verdict is recommended.","tokens_in":17410,"tokens_out":5968,"duration_ms":63024,"concrete_test":"Take the 50 TOD bugs and all 20 Passerine trajectories. Re-run every trajectory's final workspace state under two conditions: (1) revert all agent modifications to test files before evaluating, restoring the bug-reproducing test extracted from the issue; and (2) execute the reproduction test under several additional orderings (e.g., reverse order and 3-5 random permutations), not just the single order in the report. Recompute the TOD plausible rate and the combined machine plausible rate. If TOD plausible drops from 68% toward the 24% valid rate, the headline 73% machine-reported plausible overstates the agent's repair capability; if it remains near 68%, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline plausible rate for machine-reported bugs (73%) is the average of SAN (78%) and TOD (68%). Section V-A explicitly states that TOD plausibility is judged by whether the order-dependence disappears under the single ordering specified in the bug report, and concedes that this criterion 'may be too lenient' because additional test orderings are not tried. TOD is 50 of the 178 bugs, so if many TOD-plausible patches only pass that one ordering, the combined machine rate is materially inflated; in the extreme, if TOD's true plausible rate were near its 24% valid rate, the machine rate would fall from 73% to roughly 51%. Section III-C also notes 'we do not evaluate agent modifications to test files'; for machine bugs, where reproduction tests are extracted from the issue rather than a pristine ground-truth file, an agent that edits the test to eliminate the order dependence in the specified order could be scored plausible without fixing the underlying shared-state problem. The paper's own TOD valid rate (24%) indicates the gap is real, not merely hypothetical. This is an internal, author-acknowledged oracle weakness, distinct from external generalizability concerns about curation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper curates GITS-Eval, a benchmark of 178 fixed bugs drawn from Google's internal issue tracker (78 human-reported bugs and 100 machine-reported bugs, split into 50 sanitizer (SAN) and 50 test-order-dependence (TOD) bugs), using a multi-stage funnel that filters for testability, patch size, flakiness, and manually excludes 'magic constant' fixes. It then introduces Passerine, a minimal SWE-Agent-inspired ReAct agent with five commands, and evaluates it with 20 independent trajectories per bug using Gemini 1.5 Pro. The headline results are that Passerine produces at least one plausible patch for 73% of machine-reported bugs (68% TOD, 78% SAN) and 25.6% of human-reported bugs, and, after manual annotation, a semantically valid patch for 43% of machine-reported bugs (24% TOD, 62% SAN) and 17.9% of human-reported bugs. The paper also compares GITS bugs with SWE-Bench along localization and editing dimensions, reports trajectory-level observations about command usage and 'smells', and explicitly disclaims any claims about SWE-Bench performance.","tokens_in":17701,"tokens_out":6653,"duration_ms":62864,"significance":"If the results hold, this is a valuable first systematic study of agentic APR in a large industrial monorepo, and GITS-Eval is a useful benchmark for the community. The paper's strengths include direct measurement on a fixed benchmark, an explicit and detailed description of curation filters, author-acknowledged limitations, and a clear disclaimer that no SWE-Bench claims are made. The trajectory analysis and the observation that Passerine adapts its strategy to bug-report richness are credible and interesting. The main risks to the central quantitative claims are the lenient TOD plausibility oracle and the fact that agent modifications to test files are not evaluated; both are acknowledged in the text and can be addressed with additional experiments or sensitivity analyses.","major_comments":[{"comment":"The TOD plausibility criterion is judged only under the single test ordering specified in the bug report, and Section V-A concedes that this criterion 'may be too lenient' because additional orderings are not tried. TOD bugs constitute 50 of the 100 machine-reported bugs, so an unknown fraction of TOD 'plausible' patches may only pass that one ordering. The paper's own TOD valid rate of 24% shows that many TOD-plausible patches are not true fixes. Please provide a sensitivity analysis: either re-run the evaluation for TOD bugs under multiple orderings, or report the plausible rate under the assumption that TOD's true plausible rate is close to its valid rate, with the implied impact on the combined machine-reported rate.","section":"V-A and III-C"},{"comment":"The evaluation framework does not evaluate agent modifications to test files. For machine-reported bugs, the bug-reproducing tests are extracted from the issue content rather than from a pristine ground-truth file, so the agent has access to these test files and can edit them. A patch that modifies the test itself (e.g., removing or reordering the order-sensitive test, or masking the failure) would be scored as plausible under the current oracle. Please report the frequency with which Passerine modifies the test files used for machine-reported bugs, and re-run the plausibility evaluation with those test files reverted to the issue-provided versions, or otherwise demonstrate that such edits do not inflate the reported rates.","section":"III-C"},{"comment":"The valid-patch rates (43% machine, 17.9% human) rest entirely on manual annotation by the authors, but Section IV reports only that 'one reviewer analyzed each patch, consulting with the other two authors for assessment of complex cases' and Section VII notes the usual subjectivity without providing reliability statistics. Given that these rates are a central headline claim, please provide an inter-rater reliability measure (e.g., Cohen's kappa) on a random, independently annotated sample of patches, and describe the annotation protocol in enough detail to be reproducible.","section":"IV and VII"}],"minor_comments":[{"comment":"Please add confidence intervals (e.g., Wilson intervals) to the rates in Table IV and Figure 7, and report the raw numerators and denominators, since several cells have n=50 and the differences between subgroup rates may be within sampling noise.","section":"V, Table IV"},{"comment":"The text refers to 'Phase III bugs' in Section V, but the curation process is described as Phases 0 through 3; please use 'Phase 3' consistently.","section":"II and V"},{"comment":"The test-running command is named 'bazel' in the command list and Section III-B, but Figure 8's legend uses 'blaze'; please unify the command name across the text and figures.","section":"III-B and Figure 8"},{"comment":"In the Introduction, the sentence 'Passerine adapts its behavior based based on bug type' contains a duplicated word 'based'.","section":"I"},{"comment":"The Related Work section contains the typo 'Futhermore' instead of 'Furthermore'.","section":"VIII-C"},{"comment":"For human-reported bugs the framework uses the ground-truth test file to evaluate plausibility, but the paper does not state whether the same test-file-reverting policy is applied to machine-reported bugs; please clarify this in the evaluation description.","section":"III-C"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical contribution that is transparent about many of its limitations. The two main technical concerns (TOD plausibility under a single ordering and unassessed test-file edits) are author-acknowledged and can be addressed with additional experiments or sensitivity analyses, so the paper is a plausible candidate for acceptance after revision. No concerns about citation practices or novelty disclosure. The benchmark artifact, even in its current curated form, is likely to be of interest to the software engineering and LLM-agent communities."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Rondon et al. give the SWE-Bench community something it has been missing: a systematic look at agentic repair inside an actual industrial monorepo. GITS-Eval (178 bugs, split into human-, sanitizer-, and test-order-dependency-reported) is new, and the distributional comparison with SWE-Bench is well done and genuinely useful. Passerine is deliberately simple, and the paper does not oversell it; the explicit refusal to claim SWE-Bench numbers is a sign of discipline.\n\nWhat the paper does well is mostly at the level of measurement and transparency. The success rates are directly measured, the trajectory analysis is concrete, and the threats section names most of the real problems. That transparency is what makes the main soft spot survivable.\n\nThe soft spot is the TOD oracle. The gap between plausible (68%) and valid (24%) for TOD bugs is large, and the paper itself admits the plausibility criterion 'may be too lenient' because only the single test ordering from the bug report is tried. The stress-test concern is real: for machine-reported bugs the reproduction tests are extracted from the issue, not from a pristine ground-truth test file, and the authors do not evaluate agent edits to test files. So a TOD 'plausible' patch might be one that edits the test to remove the order dependence under the specified ordering without fixing the shared-state bug. That alone could drag the combined machine-reported rate well below 73%. The reader is right to call this load-bearing for the headline claim, but the paper is also right to flag it; the 24% valid rate is the honest number for TOD, and the authors give it equal billing in the table.\n\nTwo smaller concerns: there are no confidence intervals on any of the rates, and the manual validity annotation is done by three of the authors with no inter-rater reliability statistics. Given that some of the validity calls are subtle, precision is not fully established. The curation funnel also means the raw rates do not generalize to an unfiltered enterprise bug pool, but the authors never claim they do.\n\nOverall: the central qualitative finding—that a simple agent can fix a meaningful fraction of machine-reported bugs in a large industrial codebase, and that bug report richness matters—holds up. The 73% number should be treated as an upper bound. The paper deserves serious peer review, and the authors should be pushed to report the TOD plausible rate under multiple orderings and to make the benchmark or at least the oracle details externally checkable.","headline":"A credible first industrial benchmark and baseline for agentic APR, but the 73% machine-reported plausible rate is probably inflated by the lenient TOD oracle, which the paper itself flags.","tokens_in":18197,"tokens_out":1626,"would_cite":true,"duration_ms":19240,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A simple agent patches 73% of machine-reported Google bugs in an enterprise test.","keywords":["automated program repair","agentic repair","LLM agents","SWE-Bench","enterprise monorepo","bug benchmark curation","test-order dependence","sanitizer bugs"],"falsifier":"Run Passerine with the same 20-trajectory protocol on a random sample of bugs from Google's issue tracker that were fixed in the same period but were excluded by the curation funnel (e.g., bugs with multimedia, patches over 150 lines, magic-constant fixes, or flaky tests) and measure the plausible-patch rate; if it drops sharply below 73% for machine-reported and 25.6% for human-reported bugs, the claim that these numbers represent enterprise APR viability is undermined.","tokens_in":17276,"feed_emoji":"🐛","tokens_out":5163,"duration_ms":44243,"temperature":0.7,"pith_summary":"The paper tries to establish that agentic automated program repair can work in a real industrial monorepo, not just on the open-source SWE-Bench benchmark. The authors curate 178 bugs from Google's issue tracker, build a deliberately simple SWE-Agent-style repair agent called Passerine, and measure how often its 20 sampled repair trajectories produce a patch that passes the bug-reproducing tests (plausible) and, on manual review, matches the ground-truth fix (valid). The headline numbers are 73% plausible and 43% valid for machine-reported bugs, and 25.6% plausible and 17.9% valid for human-reported bugs. If these numbers hold, they imply that a small, minimally engineered agent can take on a meaningful share of real enterprise repair work, and that improving bug report quality may be as important as improving the agent.","feed_headline":"A simple agent patches 73% of machine-reported Google bugs","feed_subtitle":"On 178 real bugs from Google's tracker, it also patches a quarter of human-reported ones, setting an enterprise repair baseline.","key_machinery":"The load-bearing object is Passerine, a fully dynamic ReAct-style repair agent with no prespecified control flow and a five-command tool set (cat, code search, edit, bazel test, finish) that interacts with Google's internal code search, build, and containerization APIs. The companion machinery is GITS-Eval, a 178-bug benchmark curated through a four-phase filtering funnel (fix association, testability, tractability, and execution-based test adequacy) that separates human-reported from machine-reported bugs.","core_discovery":"The paper's central claim is that Passerine, a deliberately minimal agentic repair system inspired by SWE-Agent and running inside Google's internal development environment, produces a patch that passes the bug-reproducing tests for 73% of machine-reported bugs and 25.6% of human-reported bugs in a curated 178-bug benchmark (GITS-Eval), using 20 trajectory samples per bug with Gemini 1.5 Pro. Manual annotation by the authors found at least one patch semantically equivalent to the ground-truth fix for 43% of machine-reported bugs and 17.9% of human-reported bugs. The paper further claims that GITS bugs differ from SWE-Bench bugs along dimensions such as language diversity, patch size, and spatial spread, so performance on one benchmark should not be assumed to transfer to the other.","pith_inferences":["The reported plausible rates are likely upper bounds for an unfiltered production bug pool, because the benchmark excludes exactly the hard cases (multimedia, large patches, magic constants, flaky tests) that would be common in practice; deployment would need abstention or triage.","If richer bug reports are the main driver of repair success, human bug-reporting platforms could be redesigned to nudge reporters toward including reproduction steps and expected behavior; the paper suggests this but does not test it.","The large gap between plausible and valid rates for TOD bugs suggests the test-order-dependence oracle is lenient; trying additional test orderings could make the benchmark stricter and lower the reported numbers.","The manual validity annotation could be supplemented by an automated semantic-equivalence check, but the paper does not propose one, so the validity numbers remain dependent on human judgment."],"forward_implications":["A simple agent with a five-command interface can serve as a reproducible baseline for agentic repair in an enterprise monorepo, giving future systems concrete numbers to beat.","Machine-reported bugs (from sanitizers and a test-order dependency analyzer) are far more tractable for agents than human-reported bugs, suggesting that automated bug reporting already produces agent-friendly inputs.","Because GITS and SWE-Bench bugs differ in language, patch size, and spatial spread, SWE-Bench results should not be treated as predictive of industrial performance.","Trajectory analysis reveals that the agent adapts its strategy to bug type and that identifiable trajectory smells (no test runs, repeated reads, consecutive searches) correlate with failure, pointing to concrete optimization opportunities.","Rich bug reports, such as those from machine analyzers, improve both repair success and file-level localization, implying that bug report design is a leverage point for agentic APR."],"supporting_citations":[{"why":"Provides the open-source SWE-Bench benchmark standard that GITS-Eval is compared against.","marker":"[1]"},{"why":"Supplies the SWE-Agent design and the agent-computer-interface approach that Passerine adapts.","marker":"[2]"},{"why":"Defines the monorepo and containerization infrastructure that Passerine operates within.","marker":"[10]"},{"why":"Motivates the code-search command used for localization inside the monorepo.","marker":"[12]"},{"why":"Provides the ReAct reasoning-and-acting loop that drives Passerine's dynamic control flow.","marker":"[15]"},{"why":"Establishes the plausible-versus-correct patch distinction used for the paper's metrics.","marker":"[19]"}],"fun_headline_variants":["Agent patches 73% of machine-reported Google bugs","Simple agent fixes 73% of auto-reported bugs at Google","Passerine passes tests for 73% of Google machine bugs","Enterprise repair: agent fixes 73% of machine bugs","73% of Google machine bugs fixed by agentic repair"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported success rates are computed on a curated benchmark that filters out bugs with multimedia, large patches, magic-constant fixes, flaky tests, and hard-to-test file types, and that judges human-bug fixes against the ground-truth test file; if this curation and these test oracles do not reflect the real distribution of enterprise bugs, the rates will not generalize to an unfiltered bug pool.","fun_headline_variants_meta":{"raw":{"variants":["Agent patches 73% of machine-reported Google bugs","Simple agent fixes 73% of auto-reported bugs at Google","Passerine passes tests for 73% of Google machine bugs","Enterprise repair: agent fixes 73% of machine bugs","73% of Google machine bugs fixed by agentic repair"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000233,"raw_usage":{"total_tokens":1525,"prompt_tokens":1009,"completion_tokens":516,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":433}},"tokens_in":625,"tokens_out":516,"duration_ms":5143,"temperature":1.0,"reasoning_tokens":433,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:38:31.247769+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Passerine with the same 20-trajectory protocol on a random sample of bugs from Google's issue tracker that were fixed in the same period but were excluded by the curation funnel (e.g., bugs with multimedia, patches over 150 lines, magic-constant fixes, or flaky tests) and measure the plausible-patch rate; if it drops sharply below 73% for machine-reported and 25.6% for human-reported bugs, the claim that these numbers represent enterprise APR viability is undermined.","supporting_citations":[{"cited_title":"Why google stores billions of lines of code in a single repository,","cited_arxiv_id":null,"evidence_quote":"Defines the monorepo and containerization infrastructure that Passerine operates within."},{"cited_title":"How developers search for code: a case study,","cited_arxiv_id":null,"evidence_quote":"Motivates the code-search command used for localization inside the monorepo."},{"cited_title":"React: Synergizing reasoning and acting in language models,","cited_arxiv_id":null,"evidence_quote":"Provides the ReAct reasoning-and-acting loop that drives Passerine's dynamic control flow."},{"cited_title":"An analysis of patch plausibility and correctness for generate-and-validate patch generation systems,","cited_arxiv_id":null,"evidence_quote":"Establishes the plausible-versus-correct patch distinction used for the paper's metrics."}],"review_version":1}