Pith. sign in

REVIEW 3 major objections 5 minor 92 references

Know Before Fix: QA-Driven Repository Knowledge Acquisition for Software Issue Resolution

T0 review · 3 major / 5 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Coding agents fix more GitHub issues when they first ask and answer targeted repository questions instead of diving straight into patches.

desk verdict Solid empirical SE methods paper: QA-before-fix is a clean idea with real ablations and a 99% QA audit; the 4.4pp claim is a single-run point estimate and should be read that way. read the letter →

arxiv 2607.11111 v1 pith:L4MNE2XJ submitted 2026-07-13 cs.SE

Haotian Lin (1) , Silin Chen (1) , Xiaodong Gu (1) , Yuling Shi (1) , Chengxi Pan (2) , Jiaqi Ge (3) , Mengfan Li (1) , Jianghong Huang (1)
show 5 more authors
Mengchieh Chuang (1) Beijun Shen (1) Haibing Guan (1) ((1) Shanghai Jiao Tong University (2) University of Pittsburgh (3) Guangdong Technion-Israel Institute of Technology)
This is my paper · ORCID
classification cs.SE
keywords softwareissueresolutioncodingagentsrepositoryknowledgeacquisitionquestionansweringSWE-benchpre-repairexplorationLLMautomatedprogramrepair
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LLM coding agents often fail on real repository bugs not because they cannot reason, but because the issue description leaves out the internal knowledge they need—how a mechanism works, what an API contract allows, where modules live, or what external standards apply. Existing pre-repair exploration still tends to be fix-driven: it follows keywords or builds summaries without first naming what the agent does not know, so the context it gathers is often imprecise. This paper argues that the right order is the one experienced developers use: acquire the missing knowledge first, then repair. Its framework, ACQUIRE, has a Questioner generate a small set of category-guided questions, parallel Answerers explore the repository read-only to produce evidence-grounded answers, and a Resolver then patches with that QA set pre-injected. On SWE-bench Verified the approach raises single-attempt success by up to 4.4 percentage points over the shared base agent and over other pre-repair methods, while keeping cost and time modest, because the acquired knowledge is largely fact-supported and shortens the locating and fixing phases of repair trajectories.

What carries the argument

ACQUIRE: a two-stage pipeline in which a Questioner, guided by a four-category taxonomy (Mechanism & Behavior, Design & Usage, Locating & Structure, Ecosystem & Standards), emits N targeted questions (default N=2); independent read-only Answerers explore the repository in parallel and return evidence-grounded answers; and a Resolver receives the static QA set before any repair step so that localization and editing start from explicit repository understanding rather than keyword search alone.

What would settle it

On the same SWE-bench Verified setup, replace category-guided QA with an equal-cost pre-repair baseline (or free questions / single-pass proposals as in the paper’s own ablations) and check whether Pass@1 gains disappear, or whether human audit shows a large rise in ungrounded answers or misleading-framing regressions that erase the net Fail-to-Pass recovery.

Watch

Extended reading notes

Core claim

The paper claims that explicitly decoupling repository knowledge acquisition from patch generation—via category-guided questions answered by autonomous, evidence-grounded exploration—turns implicit knowledge gaps into reliable QA context that measurably improves automated issue resolution. On SWE-bench Verified this yields Pass@1 of 70.8% with DeepSeek-V3.2 and 62.2% with GPT-5-mini, gains of +4.4 and +3.8 points over the base Mini-SWE-Agent, outperforming LocAgent, CoSIL, LingmaAgent, and SWE-Debate at lower cost and time than the heavyweight alternatives.

Load-bearing premise

That a small set of questions from a hand-derived four-category template, answered without gold files and then statically injected, will usually supply the right framing rather than a plausible but wrong one that steers the repairer off course.

Editorial extensions

If this is right

  • Pre-repair agents should identify knowledge gaps as explicit questions rather than only ranking files or emitting summaries.
  • Narrow, decomposable repository questions reduce hallucination risk relative to one-shot root-cause proposals.
  • Injected QA knowledge mainly accelerates locating and fixing stages, freeing steps for reproduction and verification.
  • Two complementary QA pairs strike a better accuracy–cost trade-off than zero, one, or three pairs under the reported settings.
  • Regressions that remain tend to come from how the repairer uses otherwise reliable QA, not from answer factuality alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same Questioner–Answerer split could be reused mid-trajectory when a repair hypothesis changes, if cost of refresh is controlled.
  • Category schemas might be learned from large repair-failure corpora instead of fixed from a small oracle analysis of one agent’s failures.
  • Making the Resolver explicitly critique and selectively discard QA framing could cut the minority of misleading Pass-to-Fail cases without abandoning pre-repair knowledge.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ACQUIRE, a two-stage agent framework that decouples repository knowledge acquisition from patch generation for automated issue resolution. A Questioner, guided by a four-category taxonomy (Mechanism & Behavior, Design & Usage, Locating & Structure, Ecosystem & Standards), emits targeted questions; parallel read-only Answerers produce evidence-grounded QA pairs; a Resolver then repairs with static pre-injection of that knowledge. On SWE-bench Verified (500 instances), with Mini-SWE-Agent as the shared repair backbone, ACQUIRE reports Pass@1 gains of +3.8 pp (GPT-5-mini: 62.2% vs 58.4%) and +4.4 pp (DeepSeek-V3.2: 70.8% vs 66.4%) over the bare agent, and outperforms LocAgent, CoSIL, LingmaAgent, and SWE-Debate at modest extra cost/time (Table I). Supporting analyses include a human audit of 232 QA pairs (99.1% Supported), trajectory stage composition on Fail→Pass cases, ablations of proposal-style injection and free (non-category) questions, N-sensitivity peaking at N=2, and an honest Pass→Fail regression breakdown (5/22 misleading).

Significance. If the reported gains hold under modest re-evaluation, the work is a clear contribution to agentic software engineering: it reframes pre-repair exploration as explicit knowledge-gap identification via repository QA rather than keyword- or graph-driven localization, and is, to the authors’ claim, the first integration of repository-level QA into issue resolution. Strengths that should be credited include a controlled shared-backbone design across two model families, stage-wise cost/time, mechanism-focused ablations, a substantial human factual audit, trajectory-stage evidence that speedup concentrates in Locating/Fixing, public code and data, and unusually transparent regression analysis. The practical message—that a small number of targeted, evidence-grounded QA pairs can improve both Pass@1 and repair efficiency—is actionable for the community even if absolute deltas remain modest.

major comments (3)
  1. [Table I, §V-A, §V-B1] Table I / §V-A: The headline claim that ACQUIRE “consistently outperforms” baselines and raises Pass@1 by “up to 4.4 percentage points” rests on single deterministic trajectories (Answerer/Resolver temperature 0.0; §IV-E) with no multi-seed runs, bootstrap intervals, or paired significance tests. The net gain is only +22 instances (44 Fail→Pass − 22 Pass→Fail; §V-B1). For N=500 agent runs this is within the range of typical scaffold variance. At minimum, report McNemar (or equivalent paired) tests on the Mini-SWE-Agent vs ACQUIRE outcome pairs already available, and either multi-seed Pass@1 on a stratified subset or explicit softening of “consistently / up to 4.4 pp” language. Without this, the central quantitative claim is under-supported relative to how it is stated.
  2. [§II, §III.B.1, §V-C2] §II–III.B.1: The four-category taxonomy is hand-derived from 116 oracle questions on Mini-SWE-Agent failures under DeepSeek-V3.2, then the main evaluation reuses Mini-SWE-Agent on SWE-bench Verified (including DeepSeek). FreeQ ablation (§V-C2, Table II–III) shows categories help, but does not fully address whether the taxonomy is overfit to this agent’s failure modes or transfers to other repair scaffolds. Please (i) state the derivation sample’s overlap with the evaluation set more explicitly, (ii) report category distributions on held-out or non-Mini-SWE failures if available, and (iii) discuss transfer risk when the Resolver is not Mini-SWE-Agent.
  3. [§V-B3, Finding 2, §VI-B] §V-B3: The paper correctly finds that 5/22 Pass→Fail regressions are driven by plausible but wrong QA framing rather than ungrounded facts, and that the bottleneck is Resolver utilization. This is load-bearing for the “factually reliable understanding → more accurate resolution” narrative. The current discussion treats misuse as future work; the manuscript should either (a) quantify how often QA is consulted vs overridden in successful vs failed trajectories beyond the 44 Fail→Pass ring chart, or (b) add a controlled experiment (e.g., confidence/conflict prompting, selective injection) showing that misuse rate can be reduced. Otherwise the claim that transforming gaps into “explicit, factually reliable understanding” yields more accurate resolution overstates what the Resolver currently does with that knowledge.
minor comments (5)
  1. [Abstract, Table I] Abstract and §I claim “up to 4.4 percentage points with modest additional cost and time.” Table I and supplementary stage-wise breakdown support modest cost, but DeepSeek end-to-end time (1042 s) is close to LocAgent (1046 s) and higher than CoSIL (750 s). Qualify “modest” relative to which baseline class (lightweight localization vs MCTS/debate).
  2. [Figure 2, §V-B2] Figure 2 percentages (e.g., 51.5% QA use in Locating) are useful; state inter-annotator agreement for the QA-related step labels, not only for the factual audit protocol.
  3. [§IV-E, Figure 3] §IV-E: Questioner temperature 0.7 vs Answerer/Resolver 0.0 is reasonable for diversity, but N-sensitivity (Figure 3) and FreeQ comparison may be sensitive to this choice; a one-sentence sensitivity note would help.
  4. [§VII, Supplementary Table III] §VII External validity correctly flags Python-only prompts; also note SWE-bench Verified’s issue-style distribution when claiming generality of the four categories (Ecosystem & Standards is only ~2–3% of generated QA).
  5. [§I, Related Work] Minor polish: “ACQUIRE(AgentCollaboration…” spacing in §I; consistent hyphenation of “pre-repair”; ensure arXiv-style citations that are clearly 2025–2026 preprints are marked as such where claims of priority (“first work”) depend on them.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ACQUIRE’s claims are empirical Pass@1 gains on an external benchmark, not results forced by definition or self-citation.

full rationale

ACQUIRE is an empirical software-engineering methods paper. Its load-bearing claim is that a two-stage Questioner–Answerer–Resolver pipeline raises Pass@1 on SWE-bench Verified relative to Mini-SWE-Agent and other pre-repair baselines (Table I: +4.4 pp DeepSeek-V3.2, +3.8 pp GPT-5-mini). Pass@1 is scored by developer unit tests in a fixed harness; success is not defined in terms of the QA taxonomy, N, or the injected context. The four-category question template is motivated by a manual analysis of 116 oracle questions plus prior failure taxonomies (§II), but that analysis is used only to design prompts; the main evaluation is on the full 500-instance Verified set with independent Answerer exploration (no gold files) and ablations (ACQUIRE-Proposal, ACQUIRE-FreeQ, N-sweep). There is no fitted parameter re-labeled as a prediction, no uniqueness theorem imported from the authors, and no self-citation that forces the reported deltas. Related-work citations are normal positioning, not load-bearing premises of the result. The derivation chain is therefore self-contained against an external benchmark; circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The central empirical claim rests on standard SWE-bench evaluation practice, a hand-built four-category question schema, fixed agent hyperparameters (especially N and temperatures), and the assumption that Mini-SWE-Agent plus static QA injection isolates the effect of pre-repair knowledge. No new physical entities; invented pieces are architectural roles and the taxonomy.

free parameters (3)
  • N (number of QA pairs)
    Default N=2 chosen after a small sweep (N=0..3) where Pass@1 peaks at 2; performance and cost depend on this hand-selected count.
  • Questioner temperature
    Set to 0.7 to encourage diverse questions; Answerer/Resolver use 0.0—choices that affect coverage and reproducibility of trajectories.
  • Answerer/Resolver step and cost budgets
    Answerer: 150 steps, $2 cap, 120s timeout; Resolver inherits Mini-SWE-Agent 250 steps / $3—operational limits that can truncate exploration and thus QA quality.
assumptions (4)
  • domain assumption SWE-bench Verified Pass@1 under unit tests is a valid primary measure of issue-resolution quality.
    Used throughout §IV–V; paper notes broader benchmark reliability debates in related work but does not re-validate the metric.
  • ad hoc to paper Four knowledge-deficit categories (Mechanism & Behavior, Design & Usage, Locating & Structure, Ecosystem & Standards) adequately guide useful pre-repair questions.
    Taxonomy derived from manual analysis of 116 oracle questions plus prior failure taxonomies (§II); treated as fixed template in Questioner prompts.
  • domain assumption Holding the Mini-SWE-Agent repair scaffold fixed isolates gains to pre-repair knowledge rather than repair-loop changes.
    Stated design choice in §III.C and §IV.E; standard for agent ablations but assumes no interaction effects with other repair backbones.
  • domain assumption Read-only shell exploration can produce evidence-grounded answers without modifying repository state.
    Answerer protocol in §III.B.2; factual audit supports high support rate under this setup.
invented entities (2)
  • ACQUIRE three-agent pipeline (Questioner, Answerer, Resolver)
    purpose: Decouple repository knowledge acquisition from patch generation via structured QA.
    Architectural invention of the paper; evaluated only inside this work’s experiments.
  • Category-guided question taxonomy for repair knowledge gaps
    purpose: Steer Questioner toward complementary, diagnostically useful repository questions without oracle gold patches.
    Constructed from oracle-question analysis and failure literature; not an independently validated ontology outside this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Know Before Fix: QA-Driven Repository Knowledge Acquisition for Software Issue Resolution." pith.science (2026). https://pith.science/paper/L4MNE2XJ

@misc{pith2026260711111,
  author       = {Pith},
  title        = {Pith review of: Know Before Fix: QA-Driven Repository Knowledge Acquisition for Software Issue Resolution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L4MNE2XJ}},
  note         = {Machine review of arXiv:2607.11111}
}
read the original abstract

LLM-based coding agents have significantly advanced automated software issue resolution, yet they remain highly prone to factual errors caused by insufficient repository understanding. Recent methods attempt to mitigate this limitation through pre-repair repository exploration; however, their fix-driven strategies explore repositories without identifying the agent's knowledge gaps, often yielding imprecise context that fails to bridge the underlying understanding deficit. In this paper, we propose ACQUIRE, a QA-driven framework for software issue resolution. Mirroring how experienced developers first comprehend unfamiliar code before attempting a fix, ACQUIRE explicitly acquires repository knowledge prior to repair. The framework decouples knowledge acquisition from patch generation through two stages: in the first stage, a Questioner and an Answerer collaborate to acquire structured repository knowledge, where the Questioner poses targeted questions and the Answerer produces evidence-grounded answers through autonomous exploration; in the second stage, the Resolver leverages the resulting QA knowledge to generate informed patches. By transforming implicit knowledge gaps into explicit, factually reliable understanding, ACQUIRE accelerates knowledge-intensive repair stages and enables more accurate resolution. Experiments on SWE-bench Verified demonstrate that ACQUIRE consistently outperforms representative pre-repair methods, raising Pass@1 by up to 4.4 percentage points with modest additional cost and time.

Figures

Figures reproduced from arXiv: 2607.11111 by the authors.

Figure 1
Figure 1. Overview of ACQUIRE. tion through autonomous repository exploration, producing evidence-grounded QA pairs. In the repair stage, the Resolver leverages this structured QA knowledge to generate informed patches. This design transforms implicit knowledge gaps into explicit, structured understanding before editing begins. Experiments on SWE-bench Verified [13], a curated bench￾mark of 500 real-world GitHub issues, demon… view at source ↗
Figure 2
Figure 2. Trajectory stage composition on 44 Fail→Pass instances w/o vs. w/ QA Injection. Darker shading in the right ring marks QA-related steps. a per-transition API-call shift analysis are provided in the supplementary material. Since the acceleration is most pronounced on Fail→Pass instances, we focus the remaining trajectory analysis on these 44 cases to study where within each trajectory QA knowledge takes effect. We di… view at source ↗
Figure 3
Figure 3. Pass@1 and average cost per instance under varying the number of [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Case Study of ACQUIRE with instance sphinx-doc [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 1
Figure 1. Figure 1: Agent round distribution w/o vs. w/ QA injection under DeepSeek [PITH_FULL_IMAGE:figures/full_fig_p019_1.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

92 extracted references · 21 linked inside Pith

  1. [1]

    Swe-debate: Competitive multi-agent debate for software issue resolution,

    H. Li, Y . Shi, S. Lin, X. Gu, H. Lian, X. Wang, Y . Jia, T. Huang, and Q. Wang, “Swe-debate: Competitive multi-agent debate for software issue resolution,”arXiv preprint arXiv:2507.23348, 2025

    arXiv 2025

  2. [2]

    Swe-pruner: Self-adaptive context pruning for coding agents,

    Y . Wang, Y . Shi, M. Yang, R. Zhang, S. He, H. Lian, Y . Chen, S. Ye, K. Cai, and X. Gu, “Swe-pruner: Self-adaptive context pruning for coding agents,”arXiv preprint arXiv:2601.16746, 2026

    arXiv 2026

  3. [3]

    Locagent: Graph-guided llm agents for code localization,

    Z. Chen, X. Tang, G. Deng, F. Wu, J. Wu, Z. Jiang, V . Prasanna, A. Cohan, and X. Wang, “Locagent: Graph-guided llm agents for code localization,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Vienna, Austria: Association for Computational Linguistics, jul 2025, pp. 8697–8727. [Online]. ...

    2025

  4. [4]

    SWE-agent: Agent-Computer Interfaces Enable Auto- mated Software Engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. R. Narasimhan, and O. Press, “SWE-agent: Agent-Computer Interfaces Enable Auto- mated Software Engineering,” inThe Thirty-eighth Annual Conference on Neural Information Processing Systems, Nov. 2024

    2024

  5. [5]

    Agentless: Demystifying LLM-based software engineering agents,

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Agentless: Demystifying LLM-based software engineering agents,” 2024. [Online]. Available: https://arxiv.org/abs/2407.01489

    arXiv 2024

  6. [6]

    Autocoderover: Autonomous program improvement,

    Y . Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024

    2024

  7. [7]

    An empirical study on failures in automated issue solving,

    S. Liu, F. Liu, L. Li, X. Tan, Y . Zhu, X. Lian, and L. Zhang, “An empirical study on failures in automated issue solving,” September

  8. [8]

    Available: https://arxiv.org/abs/2509.13941

    [Online]. Available: https://arxiv.org/abs/2509.13941

    arXiv

Show all 92 references
  1. [9]

    Understanding software engineering agents: A study of thought-action-result trajectories,

    I. Bouzenia and M. Pradel, “Understanding software engineering agents: A study of thought-action-result trajectories,” inProceedings of the 40th IEEE/ACM International Conference on Automated Software Engineer- ing, 2025

    2025

  2. [10]

    SWE-Adept: An LLM-based agentic framework for deep codebase analysis and structured issue resolution,

    K. He and K. Roy, “SWE-Adept: An LLM-based agentic framework for deep codebase analysis and structured issue resolution,” March

  3. [11]

    Available: https://arxiv.org/abs/2603.01327

    [Online]. Available: https://arxiv.org/abs/2603.01327

    arXiv

  4. [12]

    SWE-Effi: Re-evaluating software AI agent system effectiveness under resource constraints,

    Z. Fan, K. Vasilevski, D. Lin, B. Chen, Y . Chen, Z. Zhong, J. M. Zhang, P. He, and A. E. Hassan, “SWE-Effi: Re-evaluating software AI agent system effectiveness under resource constraints,” September

  5. [13]

    Available: https://arxiv.org/abs/2509.09853

    [Online]. Available: https://arxiv.org/abs/2509.09853

  6. [14]

    Issue localiza- tion via llm-driven iterative code graph searching,

    Z. Jiang, X. Ren, M. Yan, W. Jiang, Y . Li, and Z. Liu, “Issue localiza- tion via llm-driven iterative code graph searching,” inProceedings of the 40th IEEE/ACM International Conference on Automated Software Engineering, 2025, pp. 3034–3045

    2025

  7. [15]

    Alibaba LingmaAgent: Improving automated issue resolution via comprehensive repository exploration,

    Y . Ma, Q. Yang, R. Cao, B. Li, F. Huang, and Y . Li, “Alibaba LingmaAgent: Improving automated issue resolution via comprehensive repository exploration,” inCompanion Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering (FSE Companio...

    2025

  8. [16]

    Swe-bench verified,

    OpenAI, “Swe-bench verified,” https://openai.com/index/ introducing-swe-bench-verified/, 2024

    2024

  9. [17]

    Expertise in debugging computer programs: A process analysis,

    I. Vessey, “Expertise in debugging computer programs: A process analysis,”International Journal of Man-Machine Studies, vol. 23, no. 5, pp. 459–494, 1985

    1985

  10. [18]

    Debugging by skilled and novice programmers,

    L. Gugerty and G. M. Olson, “Debugging by skilled and novice programmers,” inProceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1986, pp. 171–174

    1986

  11. [19]

    Swe-qa: Can language models answer repository-level code questions?

    W. Peng, Y . Shi, Y . Wang, X. Zhang, B. Shen, and X. Gu, “Swe-qa: Can language models answer repository-level code questions?”arXiv preprint arXiv:2509.14635, 2025

    2025 arXiv

  12. [20]

    Ambig- swe: Interactive agents to overcome underspecificity in software engi- neering,

    S. Vijayvargiya, X. Zhou, A. Yerukola, M. Sap, and G. Neubig, “Ambig- swe: Interactive agents to overcome underspecificity in software engi- neering,” inProceedings of the International Conference on Learning Representations (ICLR), 2026

    2026

  13. [21]

    mini-SWE-agent: The minimal AI software engineering agent,

    J. Yang, C. E. Jimenez, K. Lieret, A. Wettig, S. Yao, K. Narasimhan, and O. Press, “mini-SWE-agent: The minimal AI software engineering agent,” GitHub repository, 2024, accessed: 2026-03-26. [Online]. Available: https://github.com/SWE-agent/mini-swe-agent

    2024

  14. [22]

    Swe-bench: Can language models resolve real- world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “Swe-bench: Can language models resolve real- world github issues?” inProceedings of the International Conference on Learning Representations (ICLR), 2024

    2024

  15. [23]

    Deepseek-v3.2: Push- ing the frontier of open large language models,

    DeepSeek-AI, A. Liu, A. Mei, B. Linet al., “Deepseek-v3.2: Push- ing the frontier of open large language models,”arXiv preprint arXiv:2512.02556, 2025

    2025 arXiv

  16. [24]

    Eet: Experience-driven early termination for cost-efficient software engineering agents,

    Y . Guo, Y . Xiao, J. M. Zhang, M. Harman, Y . Lou, Y . Liu, and Z. Chen, “Eet: Experience-driven early termination for cost-efficient software engineering agents,”arXiv preprint arXiv:2601.05777, 2026

    2026 arXiv

  17. [25]

    Codescout: Contextual problem statement enhancement for software agents,

    M. Suri, X. Li, M. Shojaie, S. Han, C.-C. Hsu, S. Garg, A. A. Deshmukh, and V . Kumar, “Codescout: Contextual problem statement enhancement for software agents,”arXiv preprint arXiv:2603.05744, 2026

    2026 arXiv

  18. [26]

    Live-swe-agent: Can software engineering agents self-evolve on the fly?

    C. S. Xia, Z. Wang, Y . Yang, Y . Wei, and L. Zhang, “Live-swe-agent: Can software engineering agents self-evolve on the fly?”arXiv preprint arXiv:2511.13646, 2025

    2025

  19. [28]

    OpenAI GPT-5 system card,

    A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlinet al., “OpenAI GPT-5 system card,” 2025, technical Report by OpenAI. [Online]. Available: https://arxiv.org/abs/2601.03267

    2025 arXiv

  20. [29]

    A survey on LLM-as-a-judge,

    J. Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, W. Li, Y . Shen, S. Ma, H. Liu, S. Wang, K. Zhang, Y . Wang, W. Gao, L. Ni, and J. Guo, “A survey on LLM-as-a-judge,”The Innovation, vol. 7, p. 101253, 2026

    2026

  21. [30]

    Judging the judges: A systematic study of position bias in LLM-as-a-judge,

    L. Shi, C. Ma, W. Liang, X. Diao, W. Ma, and S. V osoughi, “Judging the judges: A systematic study of position bias in LLM-as-a-judge,” inProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of...

    2025

  22. [31]

    Pairwise or point- wise? evaluating feedback protocols for bias in llm-based evaluation,

    T. Tripathi, M. Wadhwa, G. Durrett, and S. Niekum, “Pairwise or point- wise? evaluating feedback protocols for bias in llm-based evaluation,” in Proceedings of the Conference on Language Modeling (COLM), 2025

    2025

  23. [32]

    Lost in the middle: How language models use long contexts,

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” Transactions of the Association for Computational Linguistics, vol. 12, pp. 157–173, 2024

    2024

  24. [33]

    Large language models can be easily distracted by irrelevant context,

    F. Shi, X. Chen, K. Misra, N. Scales, D. Dohan, E. H. Chi, N. Sch ¨arli, and D. Zhou, “Large language models can be easily distracted by irrelevant context,” inProceedings of the 40th International Conference on Machine Learning. PMLR, 2023, pp. 31 210–31 227

    2023

  25. [34]

    How is LLM reasoning distracted by irrelevant context? an analysis using a controlled benchmark,

    M. Yang, E. Huang, L. Zhang, M. Surdeanu, W. Y . Wang, and L. Pan, “How is LLM reasoning distracted by irrelevant context? an analysis using a controlled benchmark,” inProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Suzhou, China: Associ...

    2025

  26. [35]

    Llm agents can see code repositories,

    D. Ma, S. Chen, Y . Yang, Y . Shi, Y . Yan, and X. Gu, “Llm agents can see code repositories,” 2026. [Online]. Available: https://arxiv.org/abs/2606.14061

    2026 arXiv

  27. [36]

    Repo2Run: Automated building executable environment for code repository at scale,

    R. Hu, C. Peng, X. Wang, J. Xu, and C. Gao, “Repo2Run: Automated building executable environment for code repository at scale,”arXiv preprint arXiv:2502.13681, 2025. [Online]. Available: https://arxiv.org/abs/2502.13681

    2025

  28. [37]

    Are “solved issues

    Y . Wang, M. Pradel, and Z. Liu, “Are “solved issues” in SWE- bench really solved correctly? an empirical study,”arXiv preprint arXiv:2503.15223, 2025. [Online]. Available: https://arxiv.org/abs/2503. 15223

    2025 arXiv

  29. [38]

    Investigating test overfitting on SWE-bench,

    T. Ahmed, J. Ganhotra, A. Shinnar, and M. Hirzel, “Investigating test overfitting on SWE-bench,”arXiv preprint arXiv:2511.16858, 2025. [Online]. Available: https://arxiv.org/abs/2511.16858

    2025 arXiv

  30. [39]

    SWE-ABS: Adversarial benchmark strengthening exposes inflated success rates on test-based benchmark,

    B. Yu, Y . Cao, Y . Zhang, L. Lin, J. Xu, Z. Zhong, Q. Xu, G. Wang, J. Cao, S.-C. Cheung, P. He, and L. Briand, “SWE-ABS: Adversarial benchmark strengthening exposes inflated success rates on test-based benchmark,”arXiv preprint arXiv:2603.00520, 2026. [Online]. Available: htt...

    2026

  31. [40]

    SPICE: An automated SWE-Bench labeling pipeline for issue clarity, test coverage, and effort estimation,

    G. A. Oliva, G. K. Rajbahadur, A. Bhatia, H. Zhang, Y . Chen, Z. Chen, A. Leung, D. Lin, B. Chen, and A. E. Hassan, “SPICE: An automated SWE-Bench labeling pipeline for issue clarity, test coverage, and effort estimation,”arXiv preprint arXiv:2507.09108, 2025. [Online]. Availa...

    2025

  32. [42]

    Available: https://arxiv.org/abs/2510.18270

    [Online]. Available: https://arxiv.org/abs/2510.18270

    arXiv

  33. [44]

    Available: https://arxiv.org/abs/2506.09289

    [Online]. Available: https://arxiv.org/abs/2506.09289

    arXiv

  34. [45]

    SWE- Bench++: A framework for the scalable generation of software engineering benchmarks from open-source repositories,

    L. Wang, L. Ramalho, A. Celestino, P. A. Pham, Y . Liu, U. K. Sinha, A. Portillo, O. Osunwa, and G. Maduekwe, “SWE- Bench++: A framework for the scalable generation of software engineering benchmarks from open-source repositories,”arXiv preprint arXiv:2512.17419, 2025. [Online...

    2025

  35. [46]

    Saving SWE-Bench: A benchmark mutation approach for realistic agent evaluation,

    S. Garg, B. Steenhoek, and Y . Huang, “Saving SWE-Bench: A benchmark mutation approach for realistic agent evaluation,”arXiv preprint arXiv:2510.08996, 2025, accepted at CAIN 2026. [Online]. Available: https://arxiv.org/abs/2510.08996

    2025

  36. [47]

    Repograph: Enhancing ai software engineering with repository-level code graph,

    S. Ouyang, W. Yu, K. Ma, Z. Xiao, Z. Zhang, M. Jia, J. Han, H. Zhang, and D. Yu, “Repograph: Enhancing ai software engineering with repository-level code graph,” in13th International Conference on Learning Representations, ICLR 2025. International Conference on Learning Repres...

    2025

  37. [48]

    Compressing code context for LLM-based issue resolution,

    H. Jia, E. T. Barr, and S. Mechtaev, “Compressing code context for LLM-based issue resolution,”arXiv preprint arXiv:2603.28119, 2026. [Online]. Available: https://arxiv.org/abs/2603.28119

    2026

  38. [49]

    Llms as continuous learners: Improving the reproduction of defective code in software issues,

    Y . Lin, Y . Ma, R. Cao, B. Li, F. Huang, X. Gu, and Y . Li, “Llms as continuous learners: Improving the reproduction of defective code in software issues,”arXiv preprint arXiv:2411.13941, 2024

    2024 arXiv

  39. [50]

    Swe-exp: Experience-driven software issue resolution,

    S. Chen, S. Lin, X. Gu, Y . Shi, H. Lian, L. Yun, D. Chen, W. Sun, L. Cao, and Q. Wang, “Swe-exp: Experience-driven software issue resolution,” arXiv preprint arXiv:2507.23361, 2025

    2025

  40. [51]

    Experepair: Dual- memory enhanced llm-based repository-level program repair,

    F. Mu, J. Wang, L. Shi, S. Wang, S. Li, and Q. Wang, “Experepair: Dual- memory enhanced llm-based repository-level program repair,”arXiv preprint arXiv:2506.10484, 2025

    2025 arXiv

  41. [52]

    Agent kb: Leveraging cross-domain experience for agentic problem solving,

    X. Tang, T. Qin, T. Peng, Z. Zhou, D. Shao, T. Du, X. Wei, P. Xia, F. Wu, H. Zhu, G. Zhang, J. Liu, X. Wang, S. Hong, C. Wu, H. Cheng, C. Wang, and W. Zhou, “Agent kb: Leveraging cross-domain experience for agentic problem solving,”arXiv preprint arXiv:2507.06229, 2025

    2025

  42. [53]

    Self-abstraction from grounded experience for plan-guided policy refinement,

    H. Hayashi, B. Pang, W. Zhao, Y . Liu, A. Gokul, S. Bansal, C. Xiong, S. Yavuz, and Y . Zhou, “Self-abstraction from grounded experience for plan-guided policy refinement,”arXiv preprint arXiv:2511.05931, 2025

    2025

  43. [54]

    Se-agent: Self-evolution trajectory optimization in multi- step reasoning with llm-based agents,

    J. Lin, Y . Guo, Y . Han, S. Hu, Z. Ni, L. Wang, M. Chen, H. Liu, R. Chen, Y . Heet al., “Se-agent: Self-evolution trajectory optimization in multi- step reasoning with llm-based agents,”arXiv preprint arXiv:2508.02085, 2025

    2025

  44. [55]

    Same signal, different semantics: A cross-framework behavioral analysis of software engineering agents,

    W. Ma, Z. Chen, J. Gu, T. Li, S. Liu, and L. Jiang, “Same signal, different semantics: A cross-framework behavioral analysis of software engineering agents,”arXiv preprint arXiv:2605.18332, 2026. [Online]. Available: https://arxiv.org/abs/2605.18332

    2026 arXiv

  45. [56]

    Codesearchnet challenge: Evaluating the state of semantic code search,

    H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “Codesearchnet challenge: Evaluating the state of semantic code search,” arXiv preprint arXiv:1909.09436, 2019

    1909 arXiv

  46. [57]

    Cosqa+: Enhancing code search evaluation with a multi-choice benchmark and test-driven agents,

    J. Gong, Y . Wu, L. Liang, Y . Wang, J. Chen, and M. Liu, “Cosqa+: Enhancing code search evaluation with a multi-choice benchmark and test-driven agents,”IEEE Transactions on Software Engineering, vol. 52, no. 1, pp. 206–220, 2026

    2026

  47. [58]

    Repoqa: Evaluating long context code understanding,

    J. Liu, J. L. Tian, V . Daita, Y . Wei, Y . Ding, Y . K. Wang, J. Yang, and L. Zhang, “Repoqa: Evaluating long context code understanding,”arXiv preprint arXiv:2406.06025, 2024

    2024 arXiv

  48. [59]

    Coderepoqa: A large-scale benchmark for software engineering question answering,

    R. Hu, C. Peng, J. Ren, B. Jiang, X. Meng, Q. Wu, P. Gao, X. Wang, and C. Gao, “Coderepoqa: A large-scale benchmark for software engineering question answering,”arXiv preprint arXiv:2412.14764, 2024

    2024 arXiv

  49. [60]

    Coreqa: Uncovering potentials of language models in code repository question answering,

    J. Chen, K. Zhao, J. Liu, C. Peng, J. Liu, H. Zhu, P. Gao, P. Yang, and S. Deng, “Coreqa: Uncovering potentials of language models in code repository question answering,”arXiv preprint arXiv:2501.03447, 2025

    2025 arXiv

  50. [61]

    Repocoder: Repository-level code completion through iterative retrieval and generation,

    F. Zhang, B. Chen, Y . Zhang, J. Keung, J. Liu, D. Zan, Y . Mao, J.-G. Lou, and W. Chen, “Repocoder: Repository-level code completion through iterative retrieval and generation,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapo...

    2023

  51. [62]

    Empowering repoqa-agent based on reinforcement learn- ing driven by monte-carlo tree search,

    G. Li, Y . Liu, Z. Qin, Y . Wang, J. Zhong, C. Zhi, B. Li, F. Huang, Y . Li, and S. Deng, “Empowering repoqa-agent based on reinforcement learn- ing driven by monte-carlo tree search,”arXiv preprint arXiv:2510.26287, 2025. Supplementary Material I. PROMPTTEMPLATES A. Questione...

    2025

  52. [63]

    Include a THOUGHT section explaining your reasoning and what you’re trying to accomplish

  53. [64]

    Provide exactly ONE bash command to execute ## Important Rules - READ ONLY: You can read files, search code, list directories, run analysis commands - DO NOT MODIFY: Do not create, edit, or delete any files in /testbed - DO NOT RUN: Avoid running tests or executing application...

  54. [65]

    Understand the question thoroughly

  55. [66]

    Explore the repository structure to locate relevant code

  56. [67]

    Read and analyze relevant files

  57. [68]

    Search for patterns, function definitions, or specific implementations

  58. [69]

    Synthesize information to form your answer

  59. [70]

    Submit your final answer ## Command Execution Rules You are operating in an environment where:

  60. [71]

    You write a single command

  61. [72]

    The system executes that command in a subshell

  62. [73]

    You write your next command Each response should include:

  63. [74]

    A **THOUGHT** section where you explain your reasoning and plan

  64. [75]

    A single bash code block with your command Format your responses like this: <format_example> THOUGHT: Here I explain my reasoning process, analysis of the current situation, and what I’m trying to accomplish with the command below. ‘‘‘bash your_command_here ‘‘‘ </format_exampl...

  65. [76]

    Error occurred

    Combine them in one block using && or || ‘‘‘bash command1 && command2 || echo "Error occurred" ‘‘‘

  66. [77]

    *.py" | head -20 ‘‘‘ ### Search for specific code patterns: ‘‘‘bash grep -r

    Wait for the first command to complete, see its output, then issue the next command in your following response. ## Environment Details - You have a full Linux shell environment - Always use non-interactive flags (-y, -f) for commands - Avoid interactive tools like vi, nano, or...

  67. [78]

    Evaluate the question only using the provided problem statement and the question text itself

    Scoring – Individual Question: System Prompt You are a rigorous evaluator assessing the quality of a single repository-focused diagnostic question generated from a GitHub issue. Evaluate the question only using the provided problem statement and the question text itself . A st...

  68. [79]

    - 3-4: Weakly related but mostly about side details or symptoms

    Relevance - 1-2: Irrelevant to the issue’s core failure or likely fix path. - 3-4: Weakly related but mostly about side details or symptoms. - 5-6: Moderately relevant but still broad or indirect. - 7-8: Strongly relevant to understanding or resolving the issue. - 9-10: Precis...

  69. [80]

    - 3-4: Only loosely tied to the repository; answerable mostly with generic software reasoning

    Repository Answerability - 1-2: Requires mainly outside knowledge or general internet or documentation lookup. - 3-4: Only loosely tied to the repository; answerable mostly with generic software reasoning. - 5-6: Partly grounded in repository artifacts but still broad. - 7-8: ...

  70. [81]

    - 3-4: Some value, but leaves the search space very large

    Diagnostic Utility - 1-2: Adds almost no debugging value. - 3-4: Some value, but leaves the search space very large. - 5-6: Moderately useful for narrowing the investigation. - 7-8: Strongly helps isolate a subsystem, state transition, boundary condition, or execution path. - ...

  71. [82]

    - 3-4: Simple one-file or one-symbol lookup

    Reasoning Depth - 1-2: Trivial lookup or superficial restatement. - 3-4: Simple one-file or one-symbol lookup. - 5-6: Requires some local reasoning over behavior or flow. - 7-8: Requires tracing meaningful interactions across functions, files, or states. - 9-10: Requires subst...

  72. [83]

    Parser crashes when optional config is missing

    Clarity - 1-2: Ambiguous, confusing, or combines multiple unrelated asks. - 3-4: Understandable but poorly scoped or imprecise. - 5-6: Acceptable but somewhat verbose or fuzzy. - 7-8: Clear, well-scoped, and technically precise. - 9-10: Very concise, unambiguous, and sharply t...

  73. [84]

    Evaluate the set only using the provided problem statement and the question texts themselves

    Scoring – Question Set (Coverage): System Prompt You are an expert evaluator assessing the quality of a set of repository-focused diagnostic questions generated from a GitHub issue. Evaluate the set only using the provided problem statement and the question texts themselves. A...

  74. [85]

    scores": {

    Coverage Coverage measures how well this set uses its limited question slots to cover the key diagnostic angles needed to debug this specific issue. When scoring coverage, consider all of the following together: - Core relevance: Are the questions collectively centered on the ...

  75. [86]

    Evaluate the two sets only using the provided problem statement and the question texts

    Voting: System Prompt You are a rigorous evaluator comparing two candidate sets of repository-focused diagnostic questions generated for the same GitHub issue. Evaluate the two sets only using the provided problem statement and the question texts. Your goal is to decide which ...

  76. [87]

    Relevance - Which set better targets the core failure mechanism or likely repair path?

  77. [88]

    Repository Answerability - Which set is more clearly answerable from repository code, tests, config, or docs rather than generic software reasoning or outside knowledge?

  78. [89]

    Diagnostic Utility - Which set would do more to reduce debugging uncertainty and narrow the search space?

  79. [90]

    Reasoning Depth - Which set more often asks for meaningful behavioral or structural reasoning rather than superficial lookup?

  80. [91]

    winner":

    Coverage and Efficiency - Which set better covers complementary diagnostic angles while minimizing semantic overlap and wasted question slots? ## Guidance Repository Answerability is about where the answer comes from. Diagnostic Utility is about how much the answer would help ...

  81. [92]

    On the full set of 500 instances (left), QA injection shifts the distribution leftward, reducing the mean number of rounds by 7.1%

    Agent Round Distribution:Figure 1 plots the distribu- tion of agent rounds with and without QA injection under DeepSeek-V3.2. On the full set of 500 instances (left), QA injection shifts the distribution leftward, reducing the mean number of rounds by 7.1%. The shift becomes f...

  82. [93]

    TABLE II API-CALL SHIFT BETWEENMINI-SWE-AGENT ANDACQUIREON SWE-BENCHVERIFIED UNDERDEEPSEEK-V3.2

    API-call Shift by Transition Type:To further examine how QA injection affects exploration effort across different outcome groups, we compute the signed difference in the number of API calls between ACQUIRE and Mini-SWE- Agent (∆ =API ACQU IRE −API Mini-SWE-Agent) for each inst...

  83. [94]

    TABLE IV REPAIR-BEHAVIOR BREAKDOWN OF THE5MISLEADINGPASS→FAIL CASES UNDERDEEPSEEK-V3.2

    Misleading Cases:We further categorize the 5 mislead- ing Pass→Fail cases by the Resolver’s primary repair behavior, as summarized in Table IV. TABLE IV REPAIR-BEHAVIOR BREAKDOWN OF THE5MISLEADINGPASS→FAIL CASES UNDERDEEPSEEK-V3.2. Repair behavior # Relation to QA Wrong modifi...

  84. [95]

    Non-misleading Cases:For the remaining 17 non- misleading Pass→Fail cases, QA generally played a more constructive role: it often helped the Resolver identify rele- vant files, understand the failure mechanism, or make partial progress toward the gold fix. The eventual failure...

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.