{"id":"3da51889-58d8-4979-a0ec-747dc53eca66","arxiv_id":"2607.11111","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"QA-driven pre-repair knowledge acquisition (Questioner + Answerer, then Resolver) raises SWE-bench Verified Pass@1 by up to 4.4 points over Mini-SWE-Agent and other pre-repair methods.","lead":"ACQUIRE is a two-stage coding-agent framework that first asks and answers targeted repository questions, then uses that QA knowledge to patch GitHub issues. On SWE-bench Verified it lifts Pass@1 by up to 4.4 points over a shared base agent at modest extra cost, beating several pre-repair baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The 4.4pp Pass@1 gain is a single-run point estimate without uncertainty; net +22 instances could be noise or backbone-specific.","rationale":"The reader correctly flags taxonomy derivation, static injection, and 5/22 misleading regressions as real limitations, and CONDITIONAL is the right overall stance. Those issues matter for generality and residual risk, but they are secondary to whether the reported Pass@1 lift is a reliable effect size. The paper’s own evidence (single-run Table I, net +22, no error bars) leaves the strongest claim under-supported statistically even if the QA pipeline is sound. My concern therefore sharpens the same CONDITIONAL verdict rather than changing it: accept-shaped contribution pending multi-run significance (and the broader validity items the reader already listed). I disagree that the hand-derived four-category template is the single most load-bearing soft spot for the central empirical claim; it is load-bearing for design novelty, but the claim as stated stands or falls first on whether 4.4pp is real. Public code makes the proposed re-run feasible.","tokens_in":26686,"tokens_out":637,"duration_ms":6879,"concrete_test":"Re-run Mini-SWE-Agent and ACQUIRE (N=2) on all 500 SWE-bench Verified instances for ≥3 independent seeds (or temperature 0.2) with DeepSeek-V3.2; report mean±std Pass@1 and a paired McNemar test on per-instance outcomes. If the mean ACQUIRE−baseline gap falls below ~2pp or loses significance (p>0.05), the strongest claim weakens.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that ACQUIRE raises Pass@1 by up to 4.4pp over Mini-SWE-Agent and beats LocAgent/CoSIL/LingmaAgent/SWE-Debate on SWE-bench Verified (Table I: 70.8% vs 66.4% DeepSeek-V3.2; 62.2% vs 58.4% GPT-5-mini). That delta rests on one deterministic trajectory per instance (temperature 0.0 for Answerer/Resolver; §IV.E) with no multi-seed runs, bootstrap CIs, or McNemar/paired tests. Net gain is only +22 instances (44 Fail\to Pass − 22 Pass\to Fail; §V-B1). With N=500 and a shared stochastic agent scaffold, a few-point swing is within typical agent variance; the paper never shows the improvement is stable under re-sampling of trajectories or seeds. Ablations and the 99.1% QA audit support mechanism, but they do not replace uncertainty on the headline metric. If the true effect is near zero under modest re-runs, the “consistently outperforms … up to 4.4pp” claim does not hold as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes ACQUIRE, a two-stage agent framework that decouples repository knowledge acquisition from patch generation for automated issue resolution. A Questioner, guided by a four-category taxonomy (Mechanism & Behavior, Design & Usage, Locating & Structure, Ecosystem & Standards), emits targeted questions; parallel read-only Answerers produce evidence-grounded QA pairs; a Resolver then repairs with static pre-injection of that knowledge. On SWE-bench Verified (500 instances), with Mini-SWE-Agent as the shared repair backbone, ACQUIRE reports Pass@1 gains of +3.8 pp (GPT-5-mini: 62.2% vs 58.4%) and +4.4 pp (DeepSeek-V3.2: 70.8% vs 66.4%) over the bare agent, and outperforms LocAgent, CoSIL, LingmaAgent, and SWE-Debate at modest extra cost/time (Table I). Supporting analyses include a human audit of 232 QA pairs (99.1% Supported), trajectory stage composition on Fail→Pass cases, ablations of proposal-style injection and free (non-category) questions, N-sensitivity peaking at N=2, and an honest Pass→Fail regression breakdown (5/22 misleading).","tokens_in":27000,"tokens_out":1603,"duration_ms":29418,"significance":"If the reported gains hold under modest re-evaluation, the work is a clear contribution to agentic software engineering: it reframes pre-repair exploration as explicit knowledge-gap identification via repository QA rather than keyword- or graph-driven localization, and is, to the authors’ claim, the first integration of repository-level QA into issue resolution. Strengths that should be credited include a controlled shared-backbone design across two model families, stage-wise cost/time, mechanism-focused ablations, a substantial human factual audit, trajectory-stage evidence that speedup concentrates in Locating/Fixing, public code and data, and unusually transparent regression analysis. The practical message—that a small number of targeted, evidence-grounded QA pairs can improve both Pass@1 and repair efficiency—is actionable for the community even if absolute deltas remain modest.","major_comments":[{"comment":"Table I / §V-A: The headline claim that ACQUIRE “consistently outperforms” baselines and raises Pass@1 by “up to 4.4 percentage points” rests on single deterministic trajectories (Answerer/Resolver temperature 0.0; §IV-E) with no multi-seed runs, bootstrap intervals, or paired significance tests. The net gain is only +22 instances (44 Fail→Pass − 22 Pass→Fail; §V-B1). For N=500 agent runs this is within the range of typical scaffold variance. At minimum, report McNemar (or equivalent paired) tests on the Mini-SWE-Agent vs ACQUIRE outcome pairs already available, and either multi-seed Pass@1 on a stratified subset or explicit softening of “consistently / up to 4.4 pp” language. Without this, the central quantitative claim is under-supported relative to how it is stated.","section":"Table I, §V-A, §V-B1"},{"comment":"§II–III.B.1: The four-category taxonomy is hand-derived from 116 oracle questions on Mini-SWE-Agent failures under DeepSeek-V3.2, then the main evaluation reuses Mini-SWE-Agent on SWE-bench Verified (including DeepSeek). FreeQ ablation (§V-C2, Table II–III) shows categories help, but does not fully address whether the taxonomy is overfit to this agent’s failure modes or transfers to other repair scaffolds. Please (i) state the derivation sample’s overlap with the evaluation set more explicitly, (ii) report category distributions on held-out or non-Mini-SWE failures if available, and (iii) discuss transfer risk when the Resolver is not Mini-SWE-Agent.","section":"§II, §III.B.1, §V-C2"},{"comment":"§V-B3: The paper correctly finds that 5/22 Pass→Fail regressions are driven by plausible but wrong QA framing rather than ungrounded facts, and that the bottleneck is Resolver utilization. This is load-bearing for the “factually reliable understanding → more accurate resolution” narrative. The current discussion treats misuse as future work; the manuscript should either (a) quantify how often QA is consulted vs overridden in successful vs failed trajectories beyond the 44 Fail→Pass ring chart, or (b) add a controlled experiment (e.g., confidence/conflict prompting, selective injection) showing that misuse rate can be reduced. Otherwise the claim that transforming gaps into “explicit, factually reliable understanding” yields more accurate resolution overstates what the Resolver currently does with that knowledge.","section":"§V-B3, Finding 2, §VI-B"}],"minor_comments":[{"comment":"Abstract and §I claim “up to 4.4 percentage points with modest additional cost and time.” Table I and supplementary stage-wise breakdown support modest cost, but DeepSeek end-to-end time (1042 s) is close to LocAgent (1046 s) and higher than CoSIL (750 s). Qualify “modest” relative to which baseline class (lightweight localization vs MCTS/debate).","section":"Abstract, Table I"},{"comment":"Figure 2 percentages (e.g., 51.5% QA use in Locating) are useful; state inter-annotator agreement for the QA-related step labels, not only for the factual audit protocol.","section":"Figure 2, §V-B2"},{"comment":"§IV-E: Questioner temperature 0.7 vs Answerer/Resolver 0.0 is reasonable for diversity, but N-sensitivity (Figure 3) and FreeQ comparison may be sensitive to this choice; a one-sentence sensitivity note would help.","section":"§IV-E, Figure 3"},{"comment":"§VII External validity correctly flags Python-only prompts; also note SWE-bench Verified’s issue-style distribution when claiming generality of the four categories (Ecosystem & Standards is only ~2–3% of generated QA).","section":"§VII, Supplementary Table III"},{"comment":"Minor polish: “ACQUIRE(AgentCollaboration…” spacing in §I; consistent hyphenation of “pre-repair”; ensure arXiv-style citations that are clearly 2025–2026 preprints are marked as such where claims of priority (“first work”) depend on them.","section":"§I, Related Work"}],"recommendation":"minor_revision","confidential_remarks":"The empirical package is stronger than many concurrent SWE-agent papers (shared backbone, two models, human QA audit, honest regressions). The main risk for the journal is overstated certainty on a ~4 pp single-run delta, not a broken method. If the venue expects multi-seed agent results as standard, push for a subset re-run; otherwise McNemar plus claim tempering is enough for minor revision. Fit is appropriate for a software-engineering / agent systems venue; less so for a pure ML theory venue."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful takeaway is simple: they turn pre-repair exploration into an explicit Questioner–Answerer stage with a four-category template, then inject the QA into Mini-SWE-Agent. On SWE-bench Verified that beats the shared backbone and LocAgent/CoSIL/LingmaAgent/SWE-Debate on both DeepSeek-V3.2 and GPT-5-mini, with modest cost/time and public code.\n\nWhat is actually new is not “agents explore repos” (everyone does that) but treating repository QA as an internal, agent-generated pre-repair capability, decoupled from patching, with categories drawn from failure taxonomies plus a small oracle study. They do the work that makes that claim checkable: shared repair backbone, stage-wise cost/time, N-sensitivity peaking at 2, ablations (proposal vs QA drops 4.8pp; free questions vs categories drops 3.8pp), human audit of 232 pairs (99.1% Supported), trajectory stage shifts on Fail→Pass, and an honest Pass→Fail breakdown (only 5/22 labeled misleading framing). The sphinx case study is concrete about mechanism vs keyword chase. Citations cover the right agent and repo-QA lines without looking like a closed club.\n\nSoft spots, in proportion: the stress-test is right that 70.8 vs 66.4 (net +22 instances) is one deterministic trajectory per instance with no multi-seed, bootstrap, or paired test. In this area that is common, but it means “up to 4.4pp” is a point estimate, not a stability claim. Taxonomy is hand-derived from 116 failed Lite instances; external validity is Python/SWE-bench only; Resolver still misuses good QA sometimes. None of that sinks the central design argument.\n\nThis is for people building coding agents and pre-repair context pipelines. Worth a serious referee. I would engage, cite the pipeline and ablations if I work on agent knowledge stages, and send it to review rather than desk-reject—ask for uncertainty on the headline metric and clearer limits on when QA framing hurts.","headline":"Solid empirical SE methods paper: QA-before-fix is a clean idea with real ablations and a 99% QA audit; the 4.4pp claim is a single-run point estimate and should be read that way.","tokens_in":27703,"tokens_out":557,"would_cite":true,"duration_ms":7524,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Coding agents fix more GitHub issues when they first ask and answer targeted repository questions instead of diving straight into patches.","keywords":["software issue resolution","coding agents","repository knowledge acquisition","question answering","SWE-bench","pre-repair exploration","LLM agents","automated program repair"],"falsifier":"On the same SWE-bench Verified setup, replace category-guided QA with an equal-cost pre-repair baseline (or free questions / single-pass proposals as in the paper’s own ablations) and check whether Pass@1 gains disappear, or whether human audit shows a large rise in ungrounded answers or misleading-framing regressions that erase the net Fail-to-Pass recovery.","tokens_in":27567,"feed_emoji":"🔧","tokens_out":1025,"duration_ms":10185,"temperature":0.7,"pith_summary":"LLM coding agents often fail on real repository bugs not because they cannot reason, but because the issue description leaves out the internal knowledge they need—how a mechanism works, what an API contract allows, where modules live, or what external standards apply. Existing pre-repair exploration still tends to be fix-driven: it follows keywords or builds summaries without first naming what the agent does not know, so the context it gathers is often imprecise. This paper argues that the right order is the one experienced developers use: acquire the missing knowledge first, then repair. Its framework, ACQUIRE, has a Questioner generate a small set of category-guided questions, parallel Answerers explore the repository read-only to produce evidence-grounded answers, and a Resolver then patches with that QA set pre-injected. On SWE-bench Verified the approach raises single-attempt success by up to 4.4 percentage points over the shared base agent and over other pre-repair methods, while keeping cost and time modest, because the acquired knowledge is largely fact-supported and shortens the locating and fixing phases of repair trajectories.","feed_headline":"Ask the repo first: QA before patch lifts fix rates 4.4 points","feed_subtitle":"Targeted repository questions give coding agents the knowledge issue text leaves out, at modest extra cost.","key_machinery":"ACQUIRE: a two-stage pipeline in which a Questioner, guided by a four-category taxonomy (Mechanism & Behavior, Design & Usage, Locating & Structure, Ecosystem & Standards), emits N targeted questions (default N=2); independent read-only Answerers explore the repository in parallel and return evidence-grounded answers; and a Resolver receives the static QA set before any repair step so that localization and editing start from explicit repository understanding rather than keyword search alone.","core_discovery":"The paper claims that explicitly decoupling repository knowledge acquisition from patch generation—via category-guided questions answered by autonomous, evidence-grounded exploration—turns implicit knowledge gaps into reliable QA context that measurably improves automated issue resolution. On SWE-bench Verified this yields Pass@1 of 70.8% with DeepSeek-V3.2 and 62.2% with GPT-5-mini, gains of +4.4 and +3.8 points over the base Mini-SWE-Agent, outperforming LocAgent, CoSIL, LingmaAgent, and SWE-Debate at lower cost and time than the heavyweight alternatives.","pith_inferences":["The same Questioner–Answerer split could be reused mid-trajectory when a repair hypothesis changes, if cost of refresh is controlled.","Category schemas might be learned from large repair-failure corpora instead of fixed from a small oracle analysis of one agent’s failures.","Making the Resolver explicitly critique and selectively discard QA framing could cut the minority of misleading Pass-to-Fail cases without abandoning pre-repair knowledge."],"forward_implications":["Pre-repair agents should identify knowledge gaps as explicit questions rather than only ranking files or emitting summaries.","Narrow, decomposable repository questions reduce hallucination risk relative to one-shot root-cause proposals.","Injected QA knowledge mainly accelerates locating and fixing stages, freeing steps for reproduction and verification.","Two complementary QA pairs strike a better accuracy–cost trade-off than zero, one, or three pairs under the reported settings.","Regressions that remain tend to come from how the repairer uses otherwise reliable QA, not from answer factuality alone."],"fun_headline_variants":["Know before fix: QA repo knowledge lifts Pass@1 by 4.4 points","ACQUIRE: question the repo first then patch for +4.4 Pass@1","Targeted repo QA before repair raises SWE-bench Pass@1 4.4 pts","Decouple knowledge QA from patching: +4.4 Pass@1 over base agent","Ask repo via QA first: evidence-grounded answers lift fix rates 4.4"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That a small set of questions from a hand-derived four-category template, answered without gold files and then statically injected, will usually supply the right framing rather than a plausible but wrong one that steers the repairer off course.","fun_headline_variants_meta":{"raw":{"variants":["Know before fix: QA repo knowledge lifts Pass@1 by 4.4 points","ACQUIRE: question the repo first then patch for +4.4 Pass@1","Targeted repo QA before repair raises SWE-bench Pass@1 4.4 pts","Decouple knowledge QA from patching: +4.4 Pass@1 over base agent","Ask repo via QA first: evidence-grounded answers lift fix rates 4.4"]},"model":"grok-4.5","effort":"low","cost_usd":0.003942,"raw_usage":{"total_tokens":1277,"prompt_tokens":834,"num_sources_used":0,"completion_tokens":98,"cost_in_usd_ticks":39420000,"prompt_tokens_details":{"text_tokens":834,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":345,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":834,"tokens_out":98,"duration_ms":3543,"temperature":1.0,"reasoning_tokens":345,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T06:58:39.749984+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On the same SWE-bench Verified setup, replace category-guided QA with an equal-cost pre-repair baseline (or free questions / single-pass proposals as in the paper’s own ablations) and check whether Pass@1 gains disappear, or whether human audit shows a large rise in ungrounded answers or misleading-framing regressions that erase the net Fail-to-Pass recovery.","supporting_citations":[],"review_version":1}