{"id":"1f308a19-9cf2-46f3-8447-d687c0aa5155","arxiv_id":"2607.07593","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":6,"one_line_summary":"AI repair agents solve bugs more reliably when reports include executable reproduction scripts, file-level localization cues, and clear structure, while longer prose reports and human-oriented steps to reproduce show no benefit or hurt.","lead":"This paper identifies which bug-report features help AI coding agents fix bugs, finding that executable scripts, file localization, and structure matter more than prose steps or readability. It matters because bug-report templates and triage workflows may need redesigning as AI agents take on more repair work.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The claim that NL steps to reproduce 'contribute little' for agents is contradicted by the paper's own ablation: removing them causes 25–38.5 pp drops in Study 2, which the paper does not reconcile with its headline framing.","rationale":"The reader identified generalizability (2 open-weight models, minimal harness, 10-instance Claude check) as the weakest assumption. This is a valid external validity concern, but it is secondary to an internal inconsistency that the reader did not flag: the paper's own two studies disagree on whether NL steps to reproduce help agents. Study 1 finds no association; Study 2 finds substantial causal drops when they are removed. This directly challenges the paper's sharpest comparative claim (agents need scripts, not NL steps). The generalizability concern affects how far the causal findings travel, but the steps-to-reproduce tension affects whether the central comparative claim is internally coherent in the first place. That said, the paper's broader argument has multiple supporting legs: localization is supported by both studies (OR=2.33* in Study 1, 29–40 pp drops in Study 2), fix suggestions show the largest OR in Study 1, and structural mutations show independent effects in Study 2. The concern narrows the claim rather than overturning it: the paper should qualify that NL steps to reproduce do help agents causally (per Study 2) even if they show no independent observational association (per Study 1), likely due to confounding with report length or other features. The CONDITIONAL verdict remains appropriate because the concern is addressable through the proposed targeted ablation and does not invalidate the paper's other findings. The small sample sizes for content mutations (n=8–13) that the reader flagged compound this concern: the steps-to-reproduce drops rest on n=13 (Qwen) and n=12 (Gemma), making the precise magnitude unreliable even if the direction is clear.","tokens_in":18311,"tokens_out":5618,"duration_ms":311150,"concrete_test":"Run a targeted ablation on the Study 2 eligible set that directly tests the form contrast: (a) take instances with NL steps to reproduce but no executable script, and add a generated reproduction script; (b) take instances with an executable script and replace it with equivalent NL steps. If (a) improves solve rates and (b) decreases them, the paper's claim that agents need executable scripts rather than NL steps is causally supported. If both conditions perform similarly, the information content matters more than the form, and the headline contrast should be softened to 'agents benefit from reproduction information in any form, but executable scripts are more common in successful reports.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's clearest example of how agent and human needs diverge is the contrast between NL steps to reproduce (supposedly unhelpful for agents, OR=0.83 ns in Table II) and executable reproduction scripts (helpful, OR=2.52**). The headline claim that NL steps 'contribute little' rests on this observational null. But Study 2's ablation directly contradicts that characterization: removing steps to reproduce from report content causes drops of 38.5 pp (Qwen, n=13) and 25.0 pp (Gemma, n=12) — among the largest drops in Table III. The paper reports both findings side by side in Table IV but never reconciles them. The most likely explanation is residual confounding in the observational model: reports containing NL steps to reproduce may also be longer (length OR=0.49***) or lack other helpful features like reproduction scripts, masking the benefit of the steps themselves. The mixed-effects model adjusts for measured covariates and bug difficulty, but cannot rule out this confounding. Since the NL-steps-vs-scripts contrast is the paper's strongest evidence that 'a good bug report for an agent is not identical to a good report for a human,' the internal inconsistency between the two studies on this specific point weakens the central claim's sharpest edge. The other divergence examples (length, readability) are supported by Study 1 only and lack causal confirmation from Study 2.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"This paper investigates which bug-report features are associated with successful resolution by AI repair agents, combining (1) an observational mixed-effects regression over 433 SWE-bench Verified issues attempted by 87 agents and (2) a controlled ablation over 2 open-weight models and 17 problem-statement mutations on SWE-bench Pro. The central claim is that a good bug report for an agent overlaps with but is not identical to a good report for a human: agents benefit most from concrete, executable, and well-localized information (fix suggestions, reproduction scripts, repository code, file-level localization), whereas some qualities emphasized for human readers (natural-language steps to reproduce, readability, shorter technical prose) contribute little or even correlate with lower success. The mixed-effects model is well-specified with crossed random intercepts, VIF-based collinearity checks, and prevalence-based feature exclusion. The ablation design is careful, with eligible-set construction, 3-run repetition, and matched baselines. Replication materials are publicly available.","tokens_in":19218,"tokens_out":1495,"duration_ms":215208,"significance":"The paper addresses a timely and practically important question: as AI repair agents enter developer workflows, bug-report templates optimized for human readers may not serve agents equally well. The two-study design (observational + causal ablation) is a genuine strength, providing both ecological validity and controlled causal evidence. The finding that structural formatting changes alone (flattening lists, removing headers) can reduce solve rates without any content removal is novel and actionable. The comparison with prior human-centric findings (Table IV) is well-constructed and frames the contribution clearly. The public replication package and pre-registered-style ablation protocol add credibility. The paper ships falsifiable predictions (specific feature effects that can be tested on other models/benchmarks) and a parameter-free observational model (no fitted constants carried into the ablation design).","major_comments":[{"comment":"§VI.A and Table IV: The headline claim that natural-language steps to reproduce 'contribute little' for agents (OR=0.83, n.s. in Table II) is directly contradicted by the Study 2 ablation: removing steps to reproduce from report content causes drops of 38.5 pp (Qwen, n=13) and 25.0 pp (Gemma, n=12) in Table III — among the largest content-mutation drops. Table IV reports both findings side by side but the paper never reconciles them. The most likely explanation is residual confounding in the observational model (reports with NL steps may also be longer, OR=0.49***, or lack executable scripts), but the paper should explicitly acknowledge this tension rather than presenting the observational null as the characterization and the ablation result as a separate finding. Since the NL-steps-vs-scripts contrast is the paper's sharpest example of agent-human divergence, this unreconciled internal ","section":null},{"comment":"§V.F–V.H, Table III: The content-mutation conditions (RC−Steps to Reproduce, RC−Observed Behavior, etc.) are evaluated on very small eligible sets (n=8–13 for Qwen, n=12–26 for Gemma). With only 3 runs per instance, the solve@3 metric on 8–13 instances has substantial variance. The paper reports point estimates (e.g., 61.5% for Qwen RC−Steps to Reproduce) without any uncertainty quantification. Given that these small-n results are used to support the claim that 'no single part of [report content] is safely removable' (§V.H), confidence intervals or at least an explicit acknowledgment of the statistical power limitation would strengthen the claim. The structural and spec-composition mutations (n=79/83) are more robustly powered.","section":null},{"comment":"§VII, External Validity: The generalizability of the ablation results rests on two open-weight models running through a minimal bash-loop harness (mini-SWE-agent). The 10-instance Claude Sonnet 4.6 check is too small to establish that findings generalize across model families, scaffolding architectures, or proprietary systems. The paper acknowledges this, but the Discussion (§VI.B) presents prescriptive recommendations ('Point the agent to the fix location,' 'Give the agent an executable reproduction script,' 'Keep the report structured') without hedging. These recommendations should be framed as hypothesis-generating for broader model/harness configurations rather than general guidelines, or the validation set should be expanded.","section":null}],"minor_comments":[{"comment":"§IV.E: The VIF threshold of 5 and prevalence range of [10%, 90%] are stated without justification. A brief citation or rationale for these specific cutoffs would help readers assess sensitivity to them.","section":null},{"comment":"§IV.F: The readability finding (higher SMOG → higher success, OR=1.55**) is counterintuitive and the paper attributes it to 'more technical detail.' This is plausible but speculative. A brief check of whether high-SMOG reports correlate with presence of technical features (stack traces, code snippets) would support the interpretation.","section":null},{"comment":"Table I: 'has itemization' has prevalence 10.6%, which is just outside the [10%, 90%] exclusion range, yet it is retained in the model. The boundary case should be noted.","section":null},{"comment":"§V.C: The decomposition of SWE-bench Pro instances into five components is clear, but a brief example showing how a real instance maps to these components would aid reproducibility.","section":null},{"comment":"§V.H: The Jaccard similarity (0.26–0.34) for unsolved bugs across spec-composition deletions is mentioned in prose but not tabulated. A small table or figure would strengthen the point that different deletions lose different instances.","section":null},{"comment":"Table III: The 'Only Interface' row has n=35/36, notably smaller than the other spec-composition conditions (n=79/83). This is because the interface is optional. The paper notes this in §V.F, but a footnote on the table itself would improve readability.","section":null},{"comment":"§VI.B, 'Give the agent an executable reproduction script': This recommendation is supported by Study 1 (OR=2.52**) but not directly tested in Study 2 ('Not tested' in Table IV). The recommendation should note that the causal evidence is from the observational study only.","section":null},{"comment":"The abstract states '17 problem-statement mutations' but Table III lists more conditions when counting both models. Clarify whether 17 refers to distinct mutation types or model-mutation pairs.","section":null}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about the NL-steps-to-reproduce contradiction between Study 1 and Study 2 is well-founded and is the primary reason for the major_revision recommendation. The paper's framing in the abstract and Discussion leans heavily on the observational null (OR=0.83) to characterize NL steps as unhelpful, while the ablation shows substantial causal effects of removal. This is not a fatal flaw — the confounding explanation is plausible and the paper could resolve it by reframing — but it does need explicit reconciliation before publication. The two-study design is otherwise sound and the paper makes a genuine contribution to an under-studied question."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive review. The referee raises three major points: (1) an unreconciled tension between the observational null for NL steps-to-reproduce and the ablation drops when steps are removed, (2) absence of uncertainty quantification for small-n content-mutation conditions, and (3) insufficient hedging on prescriptive recommendations given limited external validation. We agree with all three points and will revise accordingly.","responses":[{"response":"The referee is correct that this tension exists and that we failed to reconcile it in the manuscript. We will revise to address it directly. The most likely explanation is indeed residual confounding in the observational model: reports with NL steps-to-reproduce may also be longer (OR=0.49*** for length) or may lack executable reproduction scripts, and the observational model cannot fully separate these correlated features. The ablation, by contrast, holds everything else fixed and removes only the steps, so its drop reflects the causal effect of that content in isolation. We will add an explicit paragraph in §VI.A reconciling the two findings: the observational null means that, after adjusting for co-occurring features, the marginal association of NL steps with success is not detectable; the ablation drop means that when steps are the only thing removed, agents do lose something. These are not contradictory—they answer different questions. We will also soften the headline framing in the abstract and Table IV to make clear that the agent-human divergence on this dimension is about the form of reproduction information (executable vs. prose), not about whether steps carry useful content. The claim that NL steps 'contribute little' will be revised to 'show no independent association in the observational model, though their causal removal in the ablation does reduce solve rates,' with the confounding explanation stated explicitly.","revision_made":"yes","referee_comment":"§VI.A and Table IV: The headline claim that NL steps to reproduce 'contribute little' for agents (OR=0.83, n.s.) is contradicted by the Study 2 ablation showing 25–38.5 pp drops when steps are removed. Table IV presents both side by side without reconciling them. The paper should explicitly acknowledge this tension."},{"response":"We agree. The content-mutation conditions are underpowered for strong inferential claims, and we should not present point estimates without acknowledging the uncertainty. We will add Wilson score confidence intervals for each content-mutation solve@3 proportion in Table III, and we will add an explicit caveat in §V.H noting that the content-mutation conditions have small eligible sets (n=8–26) and that the point estimates carry wide uncertainty. We will also revise the claim 'no single part of [report content] is safely removable' to note that this is supported most robustly by the Qwen results (where all four content deletions produced drops of 35–40 pp) and that the Gemma results, while directionally consistent, are based on small samples and should be interpreted with caution. The structural and spec-composition mutations (n=79/83) are more robustly powered and we will note this distinction explicitly.","revision_made":"yes","referee_comment":"§V.F–V.H, Table III: Content-mutation conditions are evaluated on very small eligible sets (n=8–13 for Qwen, n=12–26 for Gemma) with only 3 runs per instance. Point estimates are reported without uncertainty quantification. Given these small-n results support the claim that 'no single part of [report content] is safely removable,' confidence intervals or explicit acknowledgment of statistical power limitations would strengthen the claim."},{"response":"We agree that the prescriptive recommendations in §VI.B are stated more strongly than the evidence warrants given the limited model and harness coverage. We will revise §VI.B to frame each recommendation as a hypothesis supported by our two-model, single-harness evaluation, explicitly noting that generalization to proprietary models, alternative scaffolding architectures, and other harness configurations has not been established. We will also strengthen the external validity discussion in §VII to state plainly that the 10-instance Claude Sonnet check is a sanity check, not a validation, and that broader evaluation across model families and scaffolding types is needed before these recommendations can be treated as general guidelines. We considered expanding the validation set but cannot do so within the revision timeframe due to API cost constraints; we will note this as future work.","revision_made":"yes","referee_comment":"§VII, External Validity: The ablation results rest on two open-weight models through a minimal bash-loop harness. The 10-instance Claude Sonnet check is too small to establish generalization. The Discussion (§VI.B) presents prescriptive recommendations without hedging. These should be framed as hypothesis-generating rather than general guidelines, or the validation set should be expanded."}],"tokens_in":18351,"tokens_out":1000,"duration_ms":137025,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"Short version: this is a well-executed empirical paper with a genuine new result — the first systematic comparison of what bug-report features matter for LLM repair agents versus human developers. It deserves a serious referee. There is one real internal inconsistency the authors need to address, but it does not sink the paper. The stress-test note flags that the headline claim (NL steps to reproduce “contribute little” for agents) is contradicted by the paper's own ablation: removing steps to reproduce causes 25–38.5 pp drops in Study 2. Having read the paper carefully, I think the stress-test is partially right but overstates the problem. The observational null (OR=0.83, n.s.) and the ablation drop are not strictly contradictory — the observational model measures association after adjusting for other features including report length (OR=0.49***), and reports with NL steps may also be longer or lack executable scripts, which could mask the benefit. The paper's mistake is not in the data but in the framing. The abstract and discussion say NL steps “contribute little,” but Table IV itself shows a 25–39 pp drop when they are removed. The authors should soften that claim to match their own evidence: NL steps show no independent association in the observational model but do cause large drops when removed in the controlled ablation. That is a nuance the paper already has the data to support. What the paper does well: the mixed-effects model is properly specified — crossed random intercepts, VIF checks, prevalence-based exclusion, odds ratios with CIs. The ablation design is careful: eligible-set construction, 3-run repetition, matched baselines, structural mutations that change formatting without removing content. The finding that flattening lists or removing headers hurts solve rates without removing any information is genuinely novel and useful. The divergence between Qwen (over-searching, exhausting turn budget) and Gemma (over-committing to early interpretations) is a real empirical observation worth reporting. Soft spots in proportion: the 2-model, minimal-harness scope is a real limitation but the paper acknowledges it honestly. The 10-instance Claude check is too small to establish generalizability — the authors say this themselves. Several content-mutation conditions have small applicable sets (n=8–13), which limits statistical reliability for those specific drops. The causal language (“features that help”) creeps in despite the observational design, though Study 2 does provide causal evidence for the ablated features. These are addressable concerns, not load-bearing flaws. Who this is for: software engineering researchers working on APR agents, and practitioners designing bug-report templates or triage workflows for agent-in-the-loop development. The paper earns a serious review. The referee should push the authors to reconcile the NL-steps framing with their own ablation data and to tone down causal language in the observational sections, but the core contribution holds up.","headline":"Solid two-study design comparing bug-report feature importance for AI agents vs. humans; one internal inconsistency the authors don't reconcile.","tokens_in":19291,"tokens_out":657,"would_cite":true,"duration_ms":85887,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Bug reports written for humans fail AI repair agents","keywords":["automated program repair","bug report quality","LLM agents","fault localization","SWE-bench","software engineering","mixed-effects regression","controlled ablation"],"falsifier":"If a proprietary model with a sophisticated agent harness (tool-calling, multi-turn planning, clarification-seeking) showed no solve-rate drop when file references, section headers, or expected behavior were removed from bug reports, the paper's central claim that agents need concrete, well-localized, well-structured information would not generalize beyond the minimal-harness, open-weight setting tested.","tokens_in":18368,"feed_emoji":"🐛","tokens_out":1432,"duration_ms":73327,"temperature":0.7,"pith_summary":"This paper argues that the bug-report conventions optimized for human developers do not transfer cleanly to LLM-based automated program repair agents. Through an observational study of 87 agents across 433 real GitHub issues and a controlled ablation across 2 models and 17 report mutations, the authors find that agents benefit most from concrete, executable, and well-localized information: fix suggestions, reproduction scripts, repository source code, and file-level fault localization cues. Conversely, qualities long emphasized for human readers — natural-language steps to reproduce, readability, and shorter technical prose — show no independent benefit or even correlate with lower success rates. Longer reports are associated with roughly half the odds of resolution. Structural formatting (section headers, numbered lists) matters independently of content: removing headers or flattening lists into prose reduces solve rates even when no information is deleted, because agents confuse observed and expected behavior or implement only part of the specification. The two tested models diverge in how they handle missing information: Qwen searches more broadly and can exhaust its turn budget, while Gemma commits to a plausible interpretation early and patches on it. The central mechanism the paper identifies is that agents, unlike human developers, cannot ask follow-up questions; whatever the report omits, the agent must infer or do without, and any inference it acts on becomes a committed guess that shapes the rest of the repair.","feed_headline":"Bug reports written for humans fail AI repair agents","feed_subtitle":"Agents need executable scripts and file paths, not prose steps to reproduce — and formatting alone can make or break a fix.","key_machinery":"The argument is carried by two complementary mechanisms. The first is a mixed-effects logistic regression model with crossed random intercepts for instance and agent, which isolates the association between 27 bug-report features and resolution probability while controlling for bug difficulty and agent capability. The second is a controlled ablation design that holds the underlying task fixed while mutating the problem statement — removing components, deleting localization cues, stripping sections, and reformatting structure — so that each change in solve rate can be attributed to the specific information or formatting removed. Together, the observational model supplies ecological validity (t","core_discovery":"The paper's central discovery is that the features helping AI repair agents succeed are not the features long identified as helping human developers. Fix suggestions (odds ratio 3.61), repository source code (2.82), executable reproduction scripts (2.52), and file-level localization (2.33) are the strongest positive predictors of resolution, while report length carries a negative association (odds ratio 0.49 per standard deviation increase). Controlled ablation confirms causality for several of these: removing file references drops solve rates by 29–40 percentage points, removing expected behavior drops them by 35–42 percentage points, and purely structural changes — flattening lists orstriw","pith_inferences":["If structural formatting matters independently of content, then auto-generated or templated reports that standardize headers and lists could outperform free-form human-written reports for agent consumption, even when the content is identical.","The negative association between report length and success, combined with the positive association for harder-to-read prose (higher SMOG), suggests that agents benefit from dense technical content but are harmed by verbose natural-language explanation — implying a trade-off between human-friendly and agent-friendly report styles that may not be bridgeable by a single document.","The model-specific divergence in handling missing information raises the possibility that report-completeness requirements could serve as a diagnostic for agent robustness: models that fail more gracefully when information is removed may be better suited for real-world deployment where reports are inevitably incomplete."],"forward_implications":["Bug-tracking templates and reporting guidelines may need to be redesigned with a separate 'agent-readable' section emphasizing file paths, executable reproduction scripts, and fix suggestions rather than prose-oriented steps to reproduce.","Report length acting as a negative predictor suggests that verbose natural-language context may actively mislead agents by injecting noise into the context window, which has implications for how much context to include in agent prompts generally.","The finding that structural formatting matters independently of content implies that LLM-based agents are sensitive to document layout as a parsing signal, not just to the text itself — a property that could be exploited or broken by report-generation tools.","The divergence between Qwen and Gemma in handling missing information suggests that optimal report content may be model-dependent, complicating the goal of a single universal bug-report standard for agents."],"fun_headline_variants":["Bug reports optimized for humans underperform with AI repair agents","AI repair agents prefer executable bug reports over human-readable ones","What helps humans fix bugs can hinder AI repair agents","File paths and repro scripts beat prose for AI repair agents","Bug report formatting alone shifts AI repair success rates"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The ablation results rest on two open-weight models running through a minimal bash-loop harness, and the paper acknowledges that findings may differ for proprietary models or more complex agent scaffolds. A 10-instance check with one proprietary model is too small to establish that the structural and content-mutation findings generalize across model families, scaffolding architectures, or the proprietary systems that dominate real-world deployments.","fun_headline_variants_meta":{"raw":{"variants":["Bug reports optimized for humans underperform with AI repair agents","AI repair agents prefer executable bug reports over human-readable ones","What helps humans fix bugs can hinder AI repair agents","File paths and repro scripts beat prose for AI repair agents","Bug report formatting alone shifts AI repair success rates","Longer bug reports reduce AI repair agent success odds","AI agents need different bug reports than human developers do","Executable and localized bug reports outperform prose for AI agents","Removing section headers can break AI repair agent solve rates","Bug report features that help humans don't transfer to LLM agents"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":814,"prompt_tokens":636,"completion_tokens":178,"prompt_tokens_details":null},"tokens_in":636,"tokens_out":178,"duration_ms":7888,"temperature":1.0,"reasoning_tokens":39,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T05:42:44.909550+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If a proprietary model with a sophisticated agent harness (tool-calling, multi-turn planning, clarification-seeking) showed no solve-rate drop when file references, section headers, or expected behavior were removed from bug reports, the paper's central claim that agents need concrete, well-localized, well-structured information would not generalize beyond the minimal-harness, open-weight setting tested.","supporting_citations":[],"review_version":1}