{"id":"14bce96c-56b9-48d6-86a1-c15ed9978109","arxiv_id":"2608.04682","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Introduces Active-SWE, a benchmark for proactive multi-bug fixing without issue reports, and shows state-of-the-art coding agents resolve few such bugs.","lead":"Active-SWE is a new benchmark that tests AI coding agents on finding and fixing bugs in code without any issue report to guide them. It covers 1,663 tasks in eight languages and shows that today's best agents fall short on proactive repair.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No feasibility check that recorded bugs are discoverable from code alone; low resolved rates may reflect under-specified tasks rather than agent capability.","rationale":"The paper proposes a genuinely useful evaluation direction, and the qualitative observation that coding agents struggle without issue reports is plausible. However, the central quantitative claim rests on the assumption that the mined recorded bugs are recognizable from code inspection alone once the issue text is discarded. The pipeline in §3.2 uses issue reports to annotate and filter PRs, and it never verifies that the retained bugs are discoverable without that information. Since the gold tests are derived from the issue-linked PRs, they encode intended behavior that may be invisible in the code snapshot. This makes the low Resolved rate ambiguous: it could measure agent capability or task feasibility. The reader's weakest-assumption analysis identified the same point, and the proposed human-oracle test would directly settle it. I do not think this requires changing the CONDITIONAL verdict, but the acceptance condition should explicitly require a discoverability validation of this kind before the 20.0% figure is cited as evidence about agent capability.","tokens_in":17256,"tokens_out":3733,"duration_ms":45175,"concrete_test":"Sample 50 simple tasks stratified across the six bug categories. Give each to 3 senior software engineers with only the parent-commit snapshot, the file list F_i, and the complete §C taxonomy (exactly the agent input; no issue text, gold patch, or gold tests). Ask each engineer to independently list all high-impact bugs and write one fail-to-pass test per identified bug. Define a task as 'discoverable' if at least 2 of 3 engineers identify the gold bug with a test that fails on the pre-patch snapshot and passes on the gold fix. If discoverability is below 70%, the low Resolved rates in Table 2 are partly an artifact of under-specified tasks and the central claim must be re-qualified; if discoverability is high, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's central quantitative claim—that SOTA agents resolve only 20.0% of recorded bugs without issue reports—requires that each task be well-posed and solvable from the repository snapshot, the file list F_i, and the bug taxonomy alone. The construction pipeline never validates this. In §3.2, taxonomy curation takes the issue report s_i as input (Eq. 3) and only filters PRs whose bug falls inside the taxonomy; it does not filter for bugs that are discoverable from code inspection. §3.3 then discards the issue text but retains gold tests t_i that were written against the issue's intended behavior. For bugs like 'DateTime.diff should support quarters' (Fig. 13) or 'zero-row label distributions should not produce NaN' (Fig. 14), the intended behavior is supplied by the issue, not derivable from the code alone, so an agent's failure to reproduce the gold fix may reflect under-specification rather than incapability. With no upper-bound or oracle study on the same inputs, the 20% Resolved number conflates task feasibility with agent capability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Active-SWE, a benchmark for evaluating coding agents on proactive bug fixing, i.e., discovering and repairing bugs in a repository without an issue report. The benchmark contains 1,663 tasks across six bug categories and eight programming languages, with a curated 400-task subset used for the main experiments. The construction pipeline mines GitHub PRs, uses LLM annotators to filter for taxonomy-valid bug fixes, sets up executability, and then discards issue text to create simple (single-bug) and hard (multi-bug) instances. Evaluation has two tracks: recorded bugs, scored by localization recall/precision and a Resolved indicator against human-written reference patches and tests, and potential bugs, scored by generated-test validity and an LLM judge's Revealed metric. Experiments on 17 state-of-the-art LLMs report that the best model, Claude Opus 4.8, resolves only 20.0% of recorded bugs, with most models well below that, and the paper concludes that current coding agents struggle substantially in proactive settings.","tokens_in":17494,"tokens_out":7006,"duration_ms":78454,"significance":"If the benchmark is valid, Active-SWE fills a genuine gap: existing SWE benchmarks assume issue reports are available, whereas real-world bug discovery is often proactive and report-free. The benchmark's scale, multilingual coverage, and dual-track evaluation (recorded plus potential bugs) are useful contributions, and the decoupled localization and repair metrics are a sensible way to attribute failure modes. The headline result—20.0% Resolved for the best model—would be an important indicator that today's agents are far from autonomous bug discovery and repair. The main weaknesses are evaluation validity: no feasibility check that recorded bugs are discoverable from code alone, a potential-bug track whose judge is an evaluated model without human validation, and single-run results without variance estimates. These issues need to be addressed before the quantitative claims can be taken at face value.","major_comments":[{"comment":"The construction pipeline never validates that recorded bugs are discoverable from the repository snapshot and the file list alone, despite the task requiring exactly that. Eq. (3) labels PRs using the issue report s_i, and Eq. (5) extracts the review scope from the reference patch; the issue text is then discarded in §3.3. The case studies in Figs. 13 and 14 show bugs whose intended behavior (quarters support in DateTime.diff; zero-row label-distribution normalization) is supplied by the issue report, not derivable from the code alone. Without an oracle or upper-bound study on the same instances, or a per-instance discoverability filter, the headline 20.0% Resolved in Table 2 conflates task feasibility with agent capability.","section":"§3.2–§3.3"},{"comment":"The potential-bug track is self-validating and lacks human validation. In Eq. (9), the same evaluated LLM L that produced the patch also generates the reproducing tests, and in Eq. (10) the judge J is instantiated as Qwen3.5-397B (§4.1), which is itself one of the evaluated baselines in Table 2. No human-annotated subset, inter-annotator agreement, or independent-judge comparison is reported. For the Qwen3.5-397B row, the model is effectively judging its own patch-test pairs. This can inflate TV/Revealed and should be addressed by a human-validated sample or by removing the judge model from the baseline set.","section":"§3.4.2"},{"comment":"All reported scores come from a single run with no variance estimates. On the 400-instance curated subset used in Table 2, the difference between the top Resolved values (20.0% vs. 18.5%) is within a few percentage points of binomial standard error, and the scaffold comparison in Table 8 shows differences of up to 48 points (e.g., Claude Opus 4.8 Revealed 73.0 under Claude Code vs. 25.0 under SWE-agent). Without multiple seeds, bootstrap confidence intervals, or standard errors, the cross-model and cross-scaffold rankings are not statistically grounded. Please report variance or at least per-model standard errors.","section":"Tables 2, 8"},{"comment":"The Test Validity metric TV = I[|t_i|=|f_i|] requires the number of generated tests to equal the number of fail-to-pass tests, so any extra or duplicate test that does not exhibit fail-to-pass behavior makes TV=0 and, because Revealed is built on TV=1, also makes Revealed=0. This penalizes correct bug identification whenever the agent writes a redundant test, conflating test-set parsimony with bug validity. A partial-credit metric (e.g., the fraction of generated tests that are fail-to-pass, or Revealed conditioned on the existence of at least one matching F2P test per bug) would better separate coverage from over-generation.","section":"Eq. (11)"},{"comment":"Hard-instance integration is validated only through the condition f_i = ∪_j f_j, but the union of two PR patches applied to the earliest snapshot may fail to apply cleanly, conflict, or produce a repository state that was never tested. No conflict rate or environment-rebuild statistics are reported, and the window size is fixed to 2 (§4.1) without ablating M. Since 100 of the 400 evaluation tasks are hard instances, a nontrivial integration failure rate would directly affect the hard-setting numbers in Table 7.","section":"§3.3, Eq. (6)"}],"minor_comments":[{"comment":"The number of annotators K and the voting rule V(·) are unspecified; the implementation mentions three models, but it is unclear whether the final label requires unanimity or a majority.","section":"§3.2, Eq. (4)"},{"comment":"The hunk matching relation ∼ is not defined; please specify the matching criterion (e.g., exact context match or a similarity threshold).","section":"Eq. (7)"},{"comment":"The symbol f_i is reused: in §3.2 it denotes the reference fail-to-pass tests, while in Eq. (11) it denotes the subset of generated tests exhibiting fail-to-pass behavior. Rename one of them to avoid confusion.","section":"Eq. (11)"},{"comment":"The hard setting says the review scope involves at least M bugs, but M is never explicitly stated; §4.1 says the window size is 2, yet the threshold M should be defined in the formulation.","section":"§3.3"},{"comment":"The 'Located if LR≥0.5' threshold is introduced without justification; please state how this threshold was chosen and whether the qualitative conclusions are sensitive to it.","section":"Fig. 5"},{"comment":"The column 'Sub Res.' is undefined; add a caption or footnote explaining what it measures.","section":"Table 7"},{"comment":"The abstract states 1,663 tasks while the main experiments use a curated 400-task subset; the distinction between Active-SWE and Active-SWE-Extend should be clarified in the main text, not only in the appendix.","section":"Abstract / §4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for cs.SE and addresses a worthwhile problem. The central contribution is a benchmark resource, and the dual-track framework is a reasonable starting point. However, the lack of a feasibility check for code-only discoverability and the self-validating potential-bug judge are load-bearing for the headline claims. I would not recommend rejection, because the issues are fixable within the manuscript's scope, but they require substantive additional experiments and analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things up front. First, Active-SWE attacks a real gap: existing coding benchmarks assume a detailed issue report, while proactive discovery of bugs is what you actually want from an agent. The dual-track setup, distinguishing recorded-bug repair from potential-bug discovery, and the hard multi-bug setting built from temporally adjacent PRs are thoughtful pieces of benchmark design. The paper is also honest about the low numbers and does a reasonable job of presenting category-level and language-level breakdowns.\n\nThe catch is that the benchmark never checks whether the recorded bugs are actually discoverable from code inspection. The curation pipeline uses the issue text when labeling PRs into the taxonomy, then discards that text. That leaves tasks where the intended behavior is only in the issue, not in the code. The case studies make this concrete: the `DateTime.diff` quarters bug is a missing registry entry; the zero-row normalization bug in label propagation is about a guard that only makes sense if you know distributions should not be NaN. An agent looking at the code alone has no way to infer those expectations. The 20% Resolved rate therefore conflates task infeasibility with agent incapability. Without an oracle study (e.g., giving the same agents the issue text, or having humans verify that the code alone suffices), the headline number doesn't support the conclusion.\n\nThe secondary issues are more routine but still worth flagging. The potential-bug track depends on an LLM judge with no human validation, and the Test Validity condition in Eq. 11 is coarse; it treats any duplicate test as invalid, which mixes up redundancy with validity. There are no error bars or multiple runs across the tables, and the paper does not ship code or data with a commit hash, which hurts reproducibility. These are fixable.\n\nSo: the core idea is good and the benchmark could become a useful community asset, but the current version needs serious revision. I'd send it to review, not desk-reject, because the task is important and the flaws are repairable. I just wouldn't trust the current numbers, especially the Resolved rates, until the feasibility question is settled.","headline":"A genuinely new proactive-fixing benchmark, but the headline 20% resolved rate conflates task infeasibility with agent incapability until the authors validate that bugs are actually discoverable from code.","tokens_in":18026,"tokens_out":2970,"would_cite":false,"duration_ms":32263,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The strongest coding agents fix only 20% of bugs they must discover without issue reports.","keywords":["proactive bug fixing","coding agents","LLM benchmark","bug localization","test-driven evaluation","issue-free debugging","GitHub pull requests","software engineering"],"falsifier":"Have a sample of the recorded-bug tasks attempted by human engineers who see only the files pending review and the bug-taxonomy template, with the issue report withheld; if humans cannot locate a substantial fraction of the recorded bugs, then those tasks are not discoverable from code and the benchmark's resolved rates reflect infeasibility rather than agent capability.","tokens_in":17079,"feed_emoji":"🐛","tokens_out":7991,"duration_ms":108818,"temperature":0.7,"pith_summary":"The paper claims that today's coding agents cannot yet be trusted to find and fix bugs on their own: when the human-written issue report is stripped away, the best agent tested resolves only 20.0% of recorded bugs. To make this measurable, it introduces Active-SWE, a benchmark of 1,663 bug-fixing tasks mined from real GitHub pull requests, spanning six bug categories and eight languages, with tasks reformulated so agents must inspect code, locate problems, and patch them without any report guidance. The paper also defines harder tasks that contain multiple recorded bugs at once, plus a 'potential bug' track that requires agents to validate newly discovered bugs with fail-to-pass tests. The result is evidence that current agents are far from autonomous bug discovery and repair, and the paper locates the bottleneck in bug localization: correctly found bugs are much more likely to be fixed. The benchmark matters because real-world bugs are usually discovered only after they have caused damage, and report-free repair is what would make coding agents genuinely proactive.","feed_headline":"Best coding agents fix just 20% of unreported bugs","feed_subtitle":"A 1,663-task benchmark strips issue reports away, exposing a steep gap between reactive and proactive repair.","key_machinery":"The machinery that carries the argument is the benchmark construction pipeline and its dual-track evaluation protocol. Each task removes the issue report and instead supplies a unified bug-fixing template plus the set of files pending review, extracted from the reference patch of a GitHub pull request; simple tasks take one recorded bug, and hard tasks merge temporally adjacent pull requests so one snapshot contains multiple recorded bugs. The recorded-bug track scores localization through hunk-level recall and precision against the reference patch and scores repair through a test-driven Resolved indicator; the potential-bug track asks the agent to generate tests reproducing each bug it claims to have fixed, then checks fail-to-pass behavior and semantic association with a judge. A taxonomy-driven curation step with multi-LLM consensus filters out non-bug pull requests, and an automated Docker setup agent makes each task executable and reproducible.","core_discovery":"On its own terms, the paper's discovery is that proactive bug fixing – locating and repairing bugs from code inspection alone – is a distinct and largely unsolved capability for state-of-the-art coding agents. Across 400 curated tasks, the strongest model resolves 20.0% of recorded bugs, and the reactive-versus-proactive comparison on a shared subset shows the same models falling from around 59% to 26% resolved when the issue text is removed. Agents that locate the bug are far more likely to fix it, yet localization recall stays below 30% even for the best model, so the paper concludes that accurate localization is the critical precondition for successful repair. In the hard multi-bug setting, full resolution is almost never achieved (0.4% for the best open-source model reported), and agents often fix only a subset of the recorded bugs. On the potential-bug track, agents can surface and validate additional bugs with generated tests at higher rates (up to 75.0% revealed), but invalid test generation and partial coverage keep these results from translating into full repair capability.","pith_inferences":["I would predict that injecting minimal hints – such as file names mentioned in commit messages or recently changed tests – will close much of the reactive-to-proactive gap, because the paper shows localization is the bottleneck rather than repair.","The potential-bug track could transfer directly to security-oriented code auditing, since it needs no gold patch and only requires a valid test that reproduces the discovered flaw.","The hard-setting design may understate genuine multi-bug capability: some temporally adjacent pull requests fix regressions introduced by intervening code, so the earlier snapshot may not contain a discoverable fault.","A human baseline (expert engineers attempting the same report-free tasks) would separate 'benchmark infeasibility' from 'agent weakness' – the paper does not report one."],"forward_implications":["Bug localization, not patch generation, is the binding constraint; improving how agents explore repositories should raise resolved rates more than improving code editors.","Reactive benchmarks overstate real-world agent capability by roughly a factor of two, so issue-report-based leaderboards should be read with that gap in mind.","Multi-bug scenarios are a qualitatively harder regime; evaluation that only tests single-bug fixes misses the failure mode where agents stop after the first fix.","Test generation is a workable validation mechanism for agent-discovered bugs, but only if tests are fail-to-pass and cover every claimed bug; both failure modes occur often.","Agent scaffold and model size both materially change proactive performance, so the harness is part of the capability, not just the model."],"supporting_citations":[{"why":"Defines the reactive issue-to-patch benchmark paradigm that Active-SWE contrasts against and builds from.","marker":"(Jimenez et al., 2024)"},{"why":"Provides the longer-horizon reactive benchmark whose reliance on issue reports motivates the proactive setting.","marker":"(Deng et al., 2025)"},{"why":"Supplies the evidence that curated issue-report benchmarks contain infeasible tasks, motivating report-free evaluation.","marker":"(OpenAI, 2026a)"},{"why":"ReAct loop is the basis of the setup agent that builds executable Docker environments for benchmark tasks.","marker":"(Yao et al., 2022)"},{"why":"Defines the agent-computer interface paradigm and a scaffold used in the paper's agent evaluations.","marker":"(Yang et al., 2024a)"},{"why":"Introduces test-driven validation of bug-fix patches, the template for the potential-bug evaluation track.","marker":"(Mündler et al., 2024)"}],"fun_headline_variants":["Coding agents fix only 20% of bugs with no issue reports","Proactive bug fixing: top agents resolve just a fifth","AI coders drop from 59% to 26% without issue text","Localization recall under 30%: key barrier to auto-fix","Multi-bug repairs nearly impossible for open-source agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's central bet is that the bugs it mines from GitHub pull requests are discoverable from the code alone once the issue text is removed; if a bug's existence is only knowable through external intent or the issue report itself, the task is infeasible and the low resolved rates would measure data quality, not agent ability.","fun_headline_variants_meta":{"raw":{"variants":["Coding agents fix only 20% of bugs with no issue reports","Proactive bug fixing: top agents resolve just a fifth","AI coders drop from 59% to 26% without issue text","Localization recall under 30%: key barrier to auto-fix","Multi-bug repairs nearly impossible for open-source agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000723,"raw_usage":{"total_tokens":3253,"prompt_tokens":966,"completion_tokens":2287,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":2197}},"tokens_in":582,"tokens_out":2287,"duration_ms":15292,"temperature":1.0,"reasoning_tokens":2197,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:57:03.989533+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a sample of the recorded-bug tasks attempted by human engineers who see only the files pending review and the bug-taxonomy template, with the issue report withheld; if humans cannot locate a substantial fraction of the recorded bugs, then those tasks are not discoverable from code and the benchmark's resolved rates reflect infeasibility rather than agent capability.","supporting_citations":[],"review_version":1}