Pith. sign in

REVIEW 3 major objections 5 minor 92 references

Coding agents fix more GitHub issues when they first ask and answer targeted repository questions instead of diving straight into patches.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 06:58 UTC pith:L4MNE2XJ

load-bearing objection Solid empirical SE methods paper: QA-before-fix is a clean idea with real ablations and a 99% QA audit; the 4.4pp claim is a single-run point estimate and should be read that way. the 3 major comments →

arxiv 2607.11111 v1 pith:L4MNE2XJ submitted 2026-07-13 cs.SE

Know Before Fix: QA-Driven Repository Knowledge Acquisition for Software Issue Resolution

Haotian Lin (1) , Silin Chen (1) , Xiaodong Gu (1) , Yuling Shi (1) , Chengxi Pan (2) , Jiaqi Ge (3) , Mengfan Li (1) , Jianghong Huang (1)
show 5 more authors
Mengchieh Chuang (1) Beijun Shen (1) Haibing Guan (1) ((1) Shanghai Jiao Tong University (2) University of Pittsburgh (3) Guangdong Technion-Israel Institute of Technology)
This is my paper
classification cs.SE
keywords software issue resolutioncoding agentsrepository knowledge acquisitionquestion answeringSWE-benchpre-repair explorationLLM agentsautomated program repair
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

LLM coding agents often fail on real repository bugs not because they cannot reason, but because the issue description leaves out the internal knowledge they need—how a mechanism works, what an API contract allows, where modules live, or what external standards apply. Existing pre-repair exploration still tends to be fix-driven: it follows keywords or builds summaries without first naming what the agent does not know, so the context it gathers is often imprecise. This paper argues that the right order is the one experienced developers use: acquire the missing knowledge first, then repair. Its framework, ACQUIRE, has a Questioner generate a small set of category-guided questions, parallel Answerers explore the repository read-only to produce evidence-grounded answers, and a Resolver then patches with that QA set pre-injected. On SWE-bench Verified the approach raises single-attempt success by up to 4.4 percentage points over the shared base agent and over other pre-repair methods, while keeping cost and time modest, because the acquired knowledge is largely fact-supported and shortens the locating and fixing phases of repair trajectories.

Core claim

The paper claims that explicitly decoupling repository knowledge acquisition from patch generation—via category-guided questions answered by autonomous, evidence-grounded exploration—turns implicit knowledge gaps into reliable QA context that measurably improves automated issue resolution. On SWE-bench Verified this yields Pass@1 of 70.8% with DeepSeek-V3.2 and 62.2% with GPT-5-mini, gains of +4.4 and +3.8 points over the base Mini-SWE-Agent, outperforming LocAgent, CoSIL, LingmaAgent, and SWE-Debate at lower cost and time than the heavyweight alternatives.

What carries the argument

ACQUIRE: a two-stage pipeline in which a Questioner, guided by a four-category taxonomy (Mechanism & Behavior, Design & Usage, Locating & Structure, Ecosystem & Standards), emits N targeted questions (default N=2); independent read-only Answerers explore the repository in parallel and return evidence-grounded answers; and a Resolver receives the static QA set before any repair step so that localization and editing start from explicit repository understanding rather than keyword search alone.

Load-bearing premise

That a small set of questions from a hand-derived four-category template, answered without gold files and then statically injected, will usually supply the right framing rather than a plausible but wrong one that steers the repairer off course.

What would settle it

On the same SWE-bench Verified setup, replace category-guided QA with an equal-cost pre-repair baseline (or free questions / single-pass proposals as in the paper’s own ablations) and check whether Pass@1 gains disappear, or whether human audit shows a large rise in ungrounded answers or misleading-framing regressions that erase the net Fail-to-Pass recovery.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Pre-repair agents should identify knowledge gaps as explicit questions rather than only ranking files or emitting summaries.
  • Narrow, decomposable repository questions reduce hallucination risk relative to one-shot root-cause proposals.
  • Injected QA knowledge mainly accelerates locating and fixing stages, freeing steps for reproduction and verification.
  • Two complementary QA pairs strike a better accuracy–cost trade-off than zero, one, or three pairs under the reported settings.
  • Regressions that remain tend to come from how the repairer uses otherwise reliable QA, not from answer factuality alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same Questioner–Answerer split could be reused mid-trajectory when a repair hypothesis changes, if cost of refresh is controlled.
  • Category schemas might be learned from large repair-failure corpora instead of fixed from a small oracle analysis of one agent’s failures.
  • Making the Resolver explicitly critique and selectively discard QA framing could cut the minority of misleading Pass-to-Fail cases without abandoning pre-repair knowledge.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ACQUIRE, a two-stage agent framework that decouples repository knowledge acquisition from patch generation for automated issue resolution. A Questioner, guided by a four-category taxonomy (Mechanism & Behavior, Design & Usage, Locating & Structure, Ecosystem & Standards), emits targeted questions; parallel read-only Answerers produce evidence-grounded QA pairs; a Resolver then repairs with static pre-injection of that knowledge. On SWE-bench Verified (500 instances), with Mini-SWE-Agent as the shared repair backbone, ACQUIRE reports Pass@1 gains of +3.8 pp (GPT-5-mini: 62.2% vs 58.4%) and +4.4 pp (DeepSeek-V3.2: 70.8% vs 66.4%) over the bare agent, and outperforms LocAgent, CoSIL, LingmaAgent, and SWE-Debate at modest extra cost/time (Table I). Supporting analyses include a human audit of 232 QA pairs (99.1% Supported), trajectory stage composition on Fail→Pass cases, ablations of proposal-style injection and free (non-category) questions, N-sensitivity peaking at N=2, and an honest Pass→Fail regression breakdown (5/22 misleading).

Significance. If the reported gains hold under modest re-evaluation, the work is a clear contribution to agentic software engineering: it reframes pre-repair exploration as explicit knowledge-gap identification via repository QA rather than keyword- or graph-driven localization, and is, to the authors’ claim, the first integration of repository-level QA into issue resolution. Strengths that should be credited include a controlled shared-backbone design across two model families, stage-wise cost/time, mechanism-focused ablations, a substantial human factual audit, trajectory-stage evidence that speedup concentrates in Locating/Fixing, public code and data, and unusually transparent regression analysis. The practical message—that a small number of targeted, evidence-grounded QA pairs can improve both Pass@1 and repair efficiency—is actionable for the community even if absolute deltas remain modest.

major comments (3)
  1. [Table I, §V-A, §V-B1] Table I / §V-A: The headline claim that ACQUIRE “consistently outperforms” baselines and raises Pass@1 by “up to 4.4 percentage points” rests on single deterministic trajectories (Answerer/Resolver temperature 0.0; §IV-E) with no multi-seed runs, bootstrap intervals, or paired significance tests. The net gain is only +22 instances (44 Fail→Pass − 22 Pass→Fail; §V-B1). For N=500 agent runs this is within the range of typical scaffold variance. At minimum, report McNemar (or equivalent paired) tests on the Mini-SWE-Agent vs ACQUIRE outcome pairs already available, and either multi-seed Pass@1 on a stratified subset or explicit softening of “consistently / up to 4.4 pp” language. Without this, the central quantitative claim is under-supported relative to how it is stated.
  2. [§II, §III.B.1, §V-C2] §II–III.B.1: The four-category taxonomy is hand-derived from 116 oracle questions on Mini-SWE-Agent failures under DeepSeek-V3.2, then the main evaluation reuses Mini-SWE-Agent on SWE-bench Verified (including DeepSeek). FreeQ ablation (§V-C2, Table II–III) shows categories help, but does not fully address whether the taxonomy is overfit to this agent’s failure modes or transfers to other repair scaffolds. Please (i) state the derivation sample’s overlap with the evaluation set more explicitly, (ii) report category distributions on held-out or non-Mini-SWE failures if available, and (iii) discuss transfer risk when the Resolver is not Mini-SWE-Agent.
  3. [§V-B3, Finding 2, §VI-B] §V-B3: The paper correctly finds that 5/22 Pass→Fail regressions are driven by plausible but wrong QA framing rather than ungrounded facts, and that the bottleneck is Resolver utilization. This is load-bearing for the “factually reliable understanding → more accurate resolution” narrative. The current discussion treats misuse as future work; the manuscript should either (a) quantify how often QA is consulted vs overridden in successful vs failed trajectories beyond the 44 Fail→Pass ring chart, or (b) add a controlled experiment (e.g., confidence/conflict prompting, selective injection) showing that misuse rate can be reduced. Otherwise the claim that transforming gaps into “explicit, factually reliable understanding” yields more accurate resolution overstates what the Resolver currently does with that knowledge.
minor comments (5)
  1. [Abstract, Table I] Abstract and §I claim “up to 4.4 percentage points with modest additional cost and time.” Table I and supplementary stage-wise breakdown support modest cost, but DeepSeek end-to-end time (1042 s) is close to LocAgent (1046 s) and higher than CoSIL (750 s). Qualify “modest” relative to which baseline class (lightweight localization vs MCTS/debate).
  2. [Figure 2, §V-B2] Figure 2 percentages (e.g., 51.5% QA use in Locating) are useful; state inter-annotator agreement for the QA-related step labels, not only for the factual audit protocol.
  3. [§IV-E, Figure 3] §IV-E: Questioner temperature 0.7 vs Answerer/Resolver 0.0 is reasonable for diversity, but N-sensitivity (Figure 3) and FreeQ comparison may be sensitive to this choice; a one-sentence sensitivity note would help.
  4. [§VII, Supplementary Table III] §VII External validity correctly flags Python-only prompts; also note SWE-bench Verified’s issue-style distribution when claiming generality of the four categories (Ecosystem & Standards is only ~2–3% of generated QA).
  5. [§I, Related Work] Minor polish: “ACQUIRE(AgentCollaboration…” spacing in §I; consistent hyphenation of “pre-repair”; ensure arXiv-style citations that are clearly 2025–2026 preprints are marked as such where claims of priority (“first work”) depend on them.

Circularity Check

0 steps flagged

No circularity: ACQUIRE’s claims are empirical Pass@1 gains on an external benchmark, not results forced by definition or self-citation.

full rationale

ACQUIRE is an empirical software-engineering methods paper. Its load-bearing claim is that a two-stage Questioner–Answerer–Resolver pipeline raises Pass@1 on SWE-bench Verified relative to Mini-SWE-Agent and other pre-repair baselines (Table I: +4.4 pp DeepSeek-V3.2, +3.8 pp GPT-5-mini). Pass@1 is scored by developer unit tests in a fixed harness; success is not defined in terms of the QA taxonomy, N, or the injected context. The four-category question template is motivated by a manual analysis of 116 oracle questions plus prior failure taxonomies (§II), but that analysis is used only to design prompts; the main evaluation is on the full 500-instance Verified set with independent Answerer exploration (no gold files) and ablations (ACQUIRE-Proposal, ACQUIRE-FreeQ, N-sweep). There is no fitted parameter re-labeled as a prediction, no uniqueness theorem imported from the authors, and no self-citation that forces the reported deltas. Related-work citations are normal positioning, not load-bearing premises of the result. The derivation chain is therefore self-contained against an external benchmark; circularity score is 0.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

The central empirical claim rests on standard SWE-bench evaluation practice, a hand-built four-category question schema, fixed agent hyperparameters (especially N and temperatures), and the assumption that Mini-SWE-Agent plus static QA injection isolates the effect of pre-repair knowledge. No new physical entities; invented pieces are architectural roles and the taxonomy.

free parameters (3)
  • N (number of QA pairs)
    Default N=2 chosen after a small sweep (N=0..3) where Pass@1 peaks at 2; performance and cost depend on this hand-selected count.
  • Questioner temperature
    Set to 0.7 to encourage diverse questions; Answerer/Resolver use 0.0—choices that affect coverage and reproducibility of trajectories.
  • Answerer/Resolver step and cost budgets
    Answerer: 150 steps, $2 cap, 120s timeout; Resolver inherits Mini-SWE-Agent 250 steps / $3—operational limits that can truncate exploration and thus QA quality.
axioms (4)
  • domain assumption SWE-bench Verified Pass@1 under unit tests is a valid primary measure of issue-resolution quality.
    Used throughout §IV–V; paper notes broader benchmark reliability debates in related work but does not re-validate the metric.
  • ad hoc to paper Four knowledge-deficit categories (Mechanism & Behavior, Design & Usage, Locating & Structure, Ecosystem & Standards) adequately guide useful pre-repair questions.
    Taxonomy derived from manual analysis of 116 oracle questions plus prior failure taxonomies (§II); treated as fixed template in Questioner prompts.
  • domain assumption Holding the Mini-SWE-Agent repair scaffold fixed isolates gains to pre-repair knowledge rather than repair-loop changes.
    Stated design choice in §III.C and §IV.E; standard for agent ablations but assumes no interaction effects with other repair backbones.
  • domain assumption Read-only shell exploration can produce evidence-grounded answers without modifying repository state.
    Answerer protocol in §III.B.2; factual audit supports high support rate under this setup.
invented entities (2)
  • ACQUIRE three-agent pipeline (Questioner, Answerer, Resolver) no independent evidence
    purpose: Decouple repository knowledge acquisition from patch generation via structured QA.
    Architectural invention of the paper; evaluated only inside this work’s experiments.
  • Category-guided question taxonomy for repair knowledge gaps no independent evidence
    purpose: Steer Questioner toward complementary, diagnostically useful repository questions without oracle gold patches.
    Constructed from oracle-question analysis and failure literature; not an independently validated ontology outside this paper.

pith-pipeline@v1.1.0-grok45 · 30765 in / 3279 out tokens · 35642 ms · 2026-07-14T06:58:39.749984+00:00 · methodology

0 comments
read the original abstract

LLM-based coding agents have significantly advanced automated software issue resolution, yet they remain highly prone to factual errors caused by insufficient repository understanding. Recent methods attempt to mitigate this limitation through pre-repair repository exploration; however, their fix-driven strategies explore repositories without identifying the agent's knowledge gaps, often yielding imprecise context that fails to bridge the underlying understanding deficit. In this paper, we propose ACQUIRE, a QA-driven framework for software issue resolution. Mirroring how experienced developers first comprehend unfamiliar code before attempting a fix, ACQUIRE explicitly acquires repository knowledge prior to repair. The framework decouples knowledge acquisition from patch generation through two stages: in the first stage, a Questioner and an Answerer collaborate to acquire structured repository knowledge, where the Questioner poses targeted questions and the Answerer produces evidence-grounded answers through autonomous exploration; in the second stage, the Resolver leverages the resulting QA knowledge to generate informed patches. By transforming implicit knowledge gaps into explicit, factually reliable understanding, ACQUIRE accelerates knowledge-intensive repair stages and enables more accurate resolution. Experiments on SWE-bench Verified demonstrate that ACQUIRE consistently outperforms representative pre-repair methods, raising Pass@1 by up to 4.4 percentage points with modest additional cost and time.

Figures

Figures reproduced from arXiv: 2607.11111 by (2) University of Pittsburgh, (3) Guangdong Technion-Israel Institute of Technology), Beijun Shen (1), Chengxi Pan (2), Haibing Guan (1) ((1) Shanghai Jiao Tong University, Haotian Lin (1), Jianghong Huang (1), Jiaqi Ge (3), Mengchieh Chuang (1), Mengfan Li (1), Silin Chen (1), Xiaodong Gu (1), Yuling Shi (1).

Figure 1
Figure 1. Figure 1: Overview of ACQUIRE. tion through autonomous repository exploration, producing evidence-grounded QA pairs. In the repair stage, the Resolver leverages this structured QA knowledge to generate informed patches. This design transforms implicit knowledge gaps into explicit, structured understanding before editing begins. Experiments on SWE-bench Verified [13], a curated bench￾mark of 500 real-world GitHub iss… view at source ↗
Figure 2
Figure 2. Figure 2: Trajectory stage composition on 44 Fail→Pass instances w/o vs. w/ QA Injection. Darker shading in the right ring marks QA-related steps. a per-transition API-call shift analysis are provided in the supplementary material. Since the acceleration is most pronounced on Fail→Pass instances, we focus the remaining trajectory analysis on these 44 cases to study where within each trajectory QA knowledge takes eff… view at source ↗
Figure 3
Figure 3. Figure 3: Pass@1 and average cost per instance under varying the number of [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Case Study of ACQUIRE with instance sphinx-doc [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 1
Figure 1. Figure 1: Agent round distribution w/o vs. w/ QA injection under DeepSeek [PITH_FULL_IMAGE:figures/full_fig_p019_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

92 extracted references · 21 linked inside Pith

  1. [1]

    Swe-debate: Competitive multi-agent debate for software issue resolution,

    H. Li, Y . Shi, S. Lin, X. Gu, H. Lian, X. Wang, Y . Jia, T. Huang, and Q. Wang, “Swe-debate: Competitive multi-agent debate for software issue resolution,”arXiv preprint arXiv:2507.23348, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    Swe-pruner: Self-adaptive context pruning for coding agents,

    Y . Wang, Y . Shi, M. Yang, R. Zhang, S. He, H. Lian, Y . Chen, S. Ye, K. Cai, and X. Gu, “Swe-pruner: Self-adaptive context pruning for coding agents,”arXiv preprint arXiv:2601.16746, 2026

    Pith/arXiv arXiv 2026

  3. [3]

    Locagent: Graph-guided llm agents for code localization,

    Z. Chen, X. Tang, G. Deng, F. Wu, J. Wu, Z. Jiang, V . Prasanna, A. Cohan, and X. Wang, “Locagent: Graph-guided llm agents for code localization,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Vienna, Austria: Association for Computational Linguistics, jul 2025, pp. 8697–8727. [Online]. ...

    2025

  4. [4]

    SWE-agent: Agent-Computer Interfaces Enable Auto- mated Software Engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. R. Narasimhan, and O. Press, “SWE-agent: Agent-Computer Interfaces Enable Auto- mated Software Engineering,” inThe Thirty-eighth Annual Conference on Neural Information Processing Systems, Nov. 2024

    2024

  5. [5]

    Agentless: Demystifying LLM-based software engineering agents,

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Agentless: Demystifying LLM-based software engineering agents,” 2024. [Online]. Available: https://arxiv.org/abs/2407.01489

    Pith/arXiv arXiv 2024

  6. [6]

    Autocoderover: Autonomous program improvement,

    Y . Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024

    2024

  7. [7]

    An empirical study on failures in automated issue solving,

    S. Liu, F. Liu, L. Li, X. Tan, Y . Zhu, X. Lian, and L. Zhang, “An empirical study on failures in automated issue solving,” September

  8. [8]

    Available: https://arxiv.org/abs/2509.13941

    [Online]. Available: https://arxiv.org/abs/2509.13941

    arXiv

  9. [9]

    Understanding software engineering agents: A study of thought-action-result trajectories,

    I. Bouzenia and M. Pradel, “Understanding software engineering agents: A study of thought-action-result trajectories,” inProceedings of the 40th IEEE/ACM International Conference on Automated Software Engineer- ing, 2025

    2025

  10. [10]

    SWE-Adept: An LLM-based agentic framework for deep codebase analysis and structured issue resolution,

    K. He and K. Roy, “SWE-Adept: An LLM-based agentic framework for deep codebase analysis and structured issue resolution,” March

  11. [11]

    Available: https://arxiv.org/abs/2603.01327

    [Online]. Available: https://arxiv.org/abs/2603.01327

    Pith/arXiv arXiv

  12. [12]

    SWE-Effi: Re-evaluating software AI agent system effectiveness under resource constraints,

    Z. Fan, K. Vasilevski, D. Lin, B. Chen, Y . Chen, Z. Zhong, J. M. Zhang, P. He, and A. E. Hassan, “SWE-Effi: Re-evaluating software AI agent system effectiveness under resource constraints,” September

  13. [13]

    Available: https://arxiv.org/abs/2509.09853

    [Online]. Available: https://arxiv.org/abs/2509.09853

    arXiv

  14. [14]

    Issue localiza- tion via llm-driven iterative code graph searching,

    Z. Jiang, X. Ren, M. Yan, W. Jiang, Y . Li, and Z. Liu, “Issue localiza- tion via llm-driven iterative code graph searching,” inProceedings of the 40th IEEE/ACM International Conference on Automated Software Engineering, 2025, pp. 3034–3045

    2025

  15. [15]

    Alibaba LingmaAgent: Improving automated issue resolution via comprehensive repository exploration,

    Y . Ma, Q. Yang, R. Cao, B. Li, F. Huang, and Y . Li, “Alibaba LingmaAgent: Improving automated issue resolution via comprehensive repository exploration,” inCompanion Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering (FSE Companion ’25). New York, NY , USA: Association for Comput- ing Machinery, 2025, pp. 238–249

    2025

  16. [16]

    Swe-bench verified,

    OpenAI, “Swe-bench verified,” https://openai.com/index/ introducing-swe-bench-verified/, 2024

    2024

  17. [17]

    Expertise in debugging computer programs: A process analysis,

    I. Vessey, “Expertise in debugging computer programs: A process analysis,”International Journal of Man-Machine Studies, vol. 23, no. 5, pp. 459–494, 1985

    1985

  18. [18]

    Debugging by skilled and novice programmers,

    L. Gugerty and G. M. Olson, “Debugging by skilled and novice programmers,” inProceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1986, pp. 171–174

    1986

  19. [19]

    Swe-qa: Can language models answer repository-level code questions?

    W. Peng, Y . Shi, Y . Wang, X. Zhang, B. Shen, and X. Gu, “Swe-qa: Can language models answer repository-level code questions?”arXiv preprint arXiv:2509.14635, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Ambig- swe: Interactive agents to overcome underspecificity in software engi- neering,

    S. Vijayvargiya, X. Zhou, A. Yerukola, M. Sap, and G. Neubig, “Ambig- swe: Interactive agents to overcome underspecificity in software engi- neering,” inProceedings of the International Conference on Learning Representations (ICLR), 2026

    2026

  21. [21]

    mini-SWE-agent: The minimal AI software engineering agent,

    J. Yang, C. E. Jimenez, K. Lieret, A. Wettig, S. Yao, K. Narasimhan, and O. Press, “mini-SWE-agent: The minimal AI software engineering agent,” GitHub repository, 2024, accessed: 2026-03-26. [Online]. Available: https://github.com/SWE-agent/mini-swe-agent

    2024

  22. [22]

    Swe-bench: Can language models resolve real- world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “Swe-bench: Can language models resolve real- world github issues?” inProceedings of the International Conference on Learning Representations (ICLR), 2024

    2024

  23. [23]

    Deepseek-v3.2: Push- ing the frontier of open large language models,

    DeepSeek-AI, A. Liu, A. Mei, B. Linet al., “Deepseek-v3.2: Push- ing the frontier of open large language models,”arXiv preprint arXiv:2512.02556, 2025

    Pith/arXiv arXiv 2025

  24. [24]

    Eet: Experience-driven early termination for cost-efficient software engineering agents,

    Y . Guo, Y . Xiao, J. M. Zhang, M. Harman, Y . Lou, Y . Liu, and Z. Chen, “Eet: Experience-driven early termination for cost-efficient software engineering agents,”arXiv preprint arXiv:2601.05777, 2026

    Pith/arXiv arXiv 2026

  25. [25]

    Codescout: Contextual problem statement enhancement for software agents,

    M. Suri, X. Li, M. Shojaie, S. Han, C.-C. Hsu, S. Garg, A. A. Deshmukh, and V . Kumar, “Codescout: Contextual problem statement enhancement for software agents,”arXiv preprint arXiv:2603.05744, 2026

    Pith/arXiv arXiv 2026

  26. [26]

    Live-swe-agent: Can software engineering agents self-evolve on the fly?

    C. S. Xia, Z. Wang, Y . Yang, Y . Wei, and L. Zhang, “Live-swe-agent: Can software engineering agents self-evolve on the fly?”arXiv preprint arXiv:2511.13646, 2025

    arXiv 2025

  27. [28]

    OpenAI GPT-5 system card,

    A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlinet al., “OpenAI GPT-5 system card,” 2025, technical Report by OpenAI. [Online]. Available: https://arxiv.org/abs/2601.03267

    Pith/arXiv arXiv 2025

  28. [29]

    A survey on LLM-as-a-judge,

    J. Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, W. Li, Y . Shen, S. Ma, H. Liu, S. Wang, K. Zhang, Y . Wang, W. Gao, L. Ni, and J. Guo, “A survey on LLM-as-a-judge,”The Innovation, vol. 7, p. 101253, 2026

    2026

  29. [30]

    Judging the judges: A systematic study of position bias in LLM-as-a-judge,

    L. Shi, C. Ma, W. Liang, X. Diao, W. Ma, and S. V osoughi, “Judging the judges: A systematic study of position bias in LLM-as-a-judge,” inProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics. Mumbai, India: The Asian Federat...

    2025

  30. [31]

    Pairwise or point- wise? evaluating feedback protocols for bias in llm-based evaluation,

    T. Tripathi, M. Wadhwa, G. Durrett, and S. Niekum, “Pairwise or point- wise? evaluating feedback protocols for bias in llm-based evaluation,” in Proceedings of the Conference on Language Modeling (COLM), 2025

    2025

  31. [32]

    Lost in the middle: How language models use long contexts,

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” Transactions of the Association for Computational Linguistics, vol. 12, pp. 157–173, 2024

    2024

  32. [33]

    Large language models can be easily distracted by irrelevant context,

    F. Shi, X. Chen, K. Misra, N. Scales, D. Dohan, E. H. Chi, N. Sch ¨arli, and D. Zhou, “Large language models can be easily distracted by irrelevant context,” inProceedings of the 40th International Conference on Machine Learning. PMLR, 2023, pp. 31 210–31 227

    2023

  33. [34]

    How is LLM reasoning distracted by irrelevant context? an analysis using a controlled benchmark,

    M. Yang, E. Huang, L. Zhang, M. Surdeanu, W. Y . Wang, and L. Pan, “How is LLM reasoning distracted by irrelevant context? an analysis using a controlled benchmark,” inProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Suzhou, China: Association for Computational Linguistics, nov 2025, pp. 13 329–13 347. [Online]. Avai...

    2025

  34. [35]

    Llm agents can see code repositories,

    D. Ma, S. Chen, Y . Yang, Y . Shi, Y . Yan, and X. Gu, “Llm agents can see code repositories,” 2026. [Online]. Available: https://arxiv.org/abs/2606.14061

    Pith/arXiv arXiv 2026

  35. [36]

    Repo2Run: Automated building executable environment for code repository at scale,

    R. Hu, C. Peng, X. Wang, J. Xu, and C. Gao, “Repo2Run: Automated building executable environment for code repository at scale,”arXiv preprint arXiv:2502.13681, 2025. [Online]. Available: https://arxiv.org/abs/2502.13681

    arXiv 2025

  36. [37]

    Are “solved issues

    Y . Wang, M. Pradel, and Z. Liu, “Are “solved issues” in SWE- bench really solved correctly? an empirical study,”arXiv preprint arXiv:2503.15223, 2025. [Online]. Available: https://arxiv.org/abs/2503. 15223

    Pith/arXiv arXiv 2025

  37. [38]

    Investigating test overfitting on SWE-bench,

    T. Ahmed, J. Ganhotra, A. Shinnar, and M. Hirzel, “Investigating test overfitting on SWE-bench,”arXiv preprint arXiv:2511.16858, 2025. [Online]. Available: https://arxiv.org/abs/2511.16858

    Pith/arXiv arXiv 2025

  38. [39]

    SWE-ABS: Adversarial benchmark strengthening exposes inflated success rates on test-based benchmark,

    B. Yu, Y . Cao, Y . Zhang, L. Lin, J. Xu, Z. Zhong, Q. Xu, G. Wang, J. Cao, S.-C. Cheung, P. He, and L. Briand, “SWE-ABS: Adversarial benchmark strengthening exposes inflated success rates on test-based benchmark,”arXiv preprint arXiv:2603.00520, 2026. [Online]. Available: https://arxiv.org/abs/2603.00520

    arXiv 2026

  39. [40]

    SPICE: An automated SWE-Bench labeling pipeline for issue clarity, test coverage, and effort estimation,

    G. A. Oliva, G. K. Rajbahadur, A. Bhatia, H. Zhang, Y . Chen, Z. Chen, A. Leung, D. Lin, B. Chen, and A. E. Hassan, “SPICE: An automated SWE-Bench labeling pipeline for issue clarity, test coverage, and effort estimation,”arXiv preprint arXiv:2507.09108, 2025. [Online]. Available: https://arxiv.org/abs/2507.09108

    arXiv 2025

  40. [42]

    Available: https://arxiv.org/abs/2510.18270

    [Online]. Available: https://arxiv.org/abs/2510.18270

    Pith/arXiv arXiv

  41. [44]

    Available: https://arxiv.org/abs/2506.09289

    [Online]. Available: https://arxiv.org/abs/2506.09289

    Pith/arXiv arXiv

  42. [45]

    SWE- Bench++: A framework for the scalable generation of software engineering benchmarks from open-source repositories,

    L. Wang, L. Ramalho, A. Celestino, P. A. Pham, Y . Liu, U. K. Sinha, A. Portillo, O. Osunwa, and G. Maduekwe, “SWE- Bench++: A framework for the scalable generation of software engineering benchmarks from open-source repositories,”arXiv preprint arXiv:2512.17419, 2025. [Online]. Available: https://arxiv.org/abs/2512. 17419

    arXiv 2025

  43. [46]

    Saving SWE-Bench: A benchmark mutation approach for realistic agent evaluation,

    S. Garg, B. Steenhoek, and Y . Huang, “Saving SWE-Bench: A benchmark mutation approach for realistic agent evaluation,”arXiv preprint arXiv:2510.08996, 2025, accepted at CAIN 2026. [Online]. Available: https://arxiv.org/abs/2510.08996

    arXiv 2025

  44. [47]

    Repograph: Enhancing ai software engineering with repository-level code graph,

    S. Ouyang, W. Yu, K. Ma, Z. Xiao, Z. Zhang, M. Jia, J. Han, H. Zhang, and D. Yu, “Repograph: Enhancing ai software engineering with repository-level code graph,” in13th International Conference on Learning Representations, ICLR 2025. International Conference on Learning Representations, ICLR, 2025, pp. 30 361–30 384

    2025

  45. [48]

    Compressing code context for LLM-based issue resolution,

    H. Jia, E. T. Barr, and S. Mechtaev, “Compressing code context for LLM-based issue resolution,”arXiv preprint arXiv:2603.28119, 2026. [Online]. Available: https://arxiv.org/abs/2603.28119

    arXiv 2026

  46. [49]

    Llms as continuous learners: Improving the reproduction of defective code in software issues,

    Y . Lin, Y . Ma, R. Cao, B. Li, F. Huang, X. Gu, and Y . Li, “Llms as continuous learners: Improving the reproduction of defective code in software issues,”arXiv preprint arXiv:2411.13941, 2024

    Pith/arXiv arXiv 2024

  47. [50]

    Swe-exp: Experience-driven software issue resolution,

    S. Chen, S. Lin, X. Gu, Y . Shi, H. Lian, L. Yun, D. Chen, W. Sun, L. Cao, and Q. Wang, “Swe-exp: Experience-driven software issue resolution,” arXiv preprint arXiv:2507.23361, 2025

    arXiv 2025

  48. [51]

    Experepair: Dual- memory enhanced llm-based repository-level program repair,

    F. Mu, J. Wang, L. Shi, S. Wang, S. Li, and Q. Wang, “Experepair: Dual- memory enhanced llm-based repository-level program repair,”arXiv preprint arXiv:2506.10484, 2025

    Pith/arXiv arXiv 2025

  49. [52]

    Agent kb: Leveraging cross-domain experience for agentic problem solving,

    X. Tang, T. Qin, T. Peng, Z. Zhou, D. Shao, T. Du, X. Wei, P. Xia, F. Wu, H. Zhu, G. Zhang, J. Liu, X. Wang, S. Hong, C. Wu, H. Cheng, C. Wang, and W. Zhou, “Agent kb: Leveraging cross-domain experience for agentic problem solving,”arXiv preprint arXiv:2507.06229, 2025

    arXiv 2025

  50. [53]

    Self-abstraction from grounded experience for plan-guided policy refinement,

    H. Hayashi, B. Pang, W. Zhao, Y . Liu, A. Gokul, S. Bansal, C. Xiong, S. Yavuz, and Y . Zhou, “Self-abstraction from grounded experience for plan-guided policy refinement,”arXiv preprint arXiv:2511.05931, 2025

    arXiv 2025

  51. [54]

    Se-agent: Self-evolution trajectory optimization in multi- step reasoning with llm-based agents,

    J. Lin, Y . Guo, Y . Han, S. Hu, Z. Ni, L. Wang, M. Chen, H. Liu, R. Chen, Y . Heet al., “Se-agent: Self-evolution trajectory optimization in multi- step reasoning with llm-based agents,”arXiv preprint arXiv:2508.02085, 2025

    arXiv 2025

  52. [55]

    Same signal, different semantics: A cross-framework behavioral analysis of software engineering agents,

    W. Ma, Z. Chen, J. Gu, T. Li, S. Liu, and L. Jiang, “Same signal, different semantics: A cross-framework behavioral analysis of software engineering agents,”arXiv preprint arXiv:2605.18332, 2026. [Online]. Available: https://arxiv.org/abs/2605.18332

    Pith/arXiv arXiv 2026

  53. [56]

    Codesearchnet challenge: Evaluating the state of semantic code search,

    H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “Codesearchnet challenge: Evaluating the state of semantic code search,” arXiv preprint arXiv:1909.09436, 2019

    Pith/arXiv arXiv 1909

  54. [57]

    Cosqa+: Enhancing code search evaluation with a multi-choice benchmark and test-driven agents,

    J. Gong, Y . Wu, L. Liang, Y . Wang, J. Chen, and M. Liu, “Cosqa+: Enhancing code search evaluation with a multi-choice benchmark and test-driven agents,”IEEE Transactions on Software Engineering, vol. 52, no. 1, pp. 206–220, 2026

    2026

  55. [58]

    Repoqa: Evaluating long context code understanding,

    J. Liu, J. L. Tian, V . Daita, Y . Wei, Y . Ding, Y . K. Wang, J. Yang, and L. Zhang, “Repoqa: Evaluating long context code understanding,”arXiv preprint arXiv:2406.06025, 2024

    Pith/arXiv arXiv 2024

  56. [59]

    Coderepoqa: A large-scale benchmark for software engineering question answering,

    R. Hu, C. Peng, J. Ren, B. Jiang, X. Meng, Q. Wu, P. Gao, X. Wang, and C. Gao, “Coderepoqa: A large-scale benchmark for software engineering question answering,”arXiv preprint arXiv:2412.14764, 2024

    Pith/arXiv arXiv 2024

  57. [60]

    Coreqa: Uncovering potentials of language models in code repository question answering,

    J. Chen, K. Zhao, J. Liu, C. Peng, J. Liu, H. Zhu, P. Gao, P. Yang, and S. Deng, “Coreqa: Uncovering potentials of language models in code repository question answering,”arXiv preprint arXiv:2501.03447, 2025

    Pith/arXiv arXiv 2025

  58. [61]

    Repocoder: Repository-level code completion through iterative retrieval and generation,

    F. Zhang, B. Chen, Y . Zhang, J. Keung, J. Liu, D. Zan, Y . Mao, J.-G. Lou, and W. Chen, “Repocoder: Repository-level code completion through iterative retrieval and generation,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore: Association for Computational Linguistics, dec 2023, pp. 2471–2484. [Online]...

    2023

  59. [62]

    Empowering repoqa-agent based on reinforcement learn- ing driven by monte-carlo tree search,

    G. Li, Y . Liu, Z. Qin, Y . Wang, J. Zhong, C. Zhi, B. Li, F. Huang, Y . Li, and S. Deng, “Empowering repoqa-agent based on reinforcement learn- ing driven by monte-carlo tree search,”arXiv preprint arXiv:2510.26287, 2025. Supplementary Material I. PROMPTTEMPLATES A. Questioner Prompt System Prompt You are an expert software engineer tasked with generatin...

    arXiv 2025

  60. [63]

    Include a THOUGHT section explaining your reasoning and what you’re trying to accomplish

  61. [64]

    Provide exactly ONE bash command to execute ## Important Rules - READ ONLY: You can read files, search code, list directories, run analysis commands - DO NOT MODIFY: Do not create, edit, or delete any files in /testbed - DO NOT RUN: Avoid running tests or executing application code unless necessary for understanding ## Recommended Workflow

  62. [65]

    Understand the question thoroughly

  63. [66]

    Explore the repository structure to locate relevant code

  64. [67]

    Read and analyze relevant files

  65. [68]

    Search for patterns, function definitions, or specific implementations

  66. [69]

    Synthesize information to form your answer

  67. [70]

    Submit your final answer ## Command Execution Rules You are operating in an environment where:

  68. [71]

    You write a single command

  69. [72]

    The system executes that command in a subshell

  70. [73]

    You write your next command Each response should include:

  71. [74]

    A **THOUGHT** section where you explain your reasoning and plan

  72. [75]

    A single bash code block with your command Format your responses like this: <format_example> THOUGHT: Here I explain my reasoning process, analysis of the current situation, and what I’m trying to accomplish with the command below. ‘‘‘bash your_command_here ‘‘‘ </format_example> Commands must be specified in a single bash code block: ‘‘‘bash your_command_...

  73. [76]

    Error occurred

    Combine them in one block using && or || ‘‘‘bash command1 && command2 || echo "Error occurred" ‘‘‘

  74. [77]

    *.py" | head -20 ‘‘‘ ### Search for specific code patterns: ‘‘‘bash grep -r

    Wait for the first command to complete, see its output, then issue the next command in your following response. ## Environment Details - You have a full Linux shell environment - Always use non-interactive flags (-y, -f) for commands - Avoid interactive tools like vi, nano, or any that require user input - All standard Unix tools are available (grep, find...

  75. [78]

    Evaluate the question only using the provided problem statement and the question text itself

    Scoring – Individual Question: System Prompt You are a rigorous evaluator assessing the quality of a single repository-focused diagnostic question generated from a GitHub issue. Evaluate the question only using the provided problem statement and the question text itself . A strong question should be tightly grounded in the repository, highly relevant to r...

  76. [79]

    - 3-4: Weakly related but mostly about side details or symptoms

    Relevance - 1-2: Irrelevant to the issue’s core failure or likely fix path. - 3-4: Weakly related but mostly about side details or symptoms. - 5-6: Moderately relevant but still broad or indirect. - 7-8: Strongly relevant to understanding or resolving the issue. - 9-10: Precisely targets the likely failure mechanism, root cause, or key repair decision

  77. [80]

    - 3-4: Only loosely tied to the repository; answerable mostly with generic software reasoning

    Repository Answerability - 1-2: Requires mainly outside knowledge or general internet or documentation lookup. - 3-4: Only loosely tied to the repository; answerable mostly with generic software reasoning. - 5-6: Partly grounded in repository artifacts but still broad. - 7-8: Clearly answerable through repository code, tests, config, or docs. - 9-10: Stro...

  78. [81]

    - 3-4: Some value, but leaves the search space very large

    Diagnostic Utility - 1-2: Adds almost no debugging value. - 3-4: Some value, but leaves the search space very large. - 5-6: Moderately useful for narrowing the investigation. - 7-8: Strongly helps isolate a subsystem, state transition, boundary condition, or execution path. - 9-10: Highly discriminative; answering it would sharply narrow the fault to a sp...

  79. [82]

    - 3-4: Simple one-file or one-symbol lookup

    Reasoning Depth - 1-2: Trivial lookup or superficial restatement. - 3-4: Simple one-file or one-symbol lookup. - 5-6: Requires some local reasoning over behavior or flow. - 7-8: Requires tracing meaningful interactions across functions, files, or states. - 9-10: Requires substantial multi-hop reasoning across decoupled components, state changes, or abstra...

  80. [83]

    Parser crashes when optional config is missing

    Clarity - 1-2: Ambiguous, confusing, or combines multiple unrelated asks. - 3-4: Understandable but poorly scoped or imprecise. - 5-6: Acceptable but somewhat verbose or fuzzy. - 7-8: Clear, well-scoped, and technically precise. - 9-10: Very concise, unambiguous, and sharply targeted. ## Hard Flags Set each to true or false. - requires_external_knowledge:...

Showing first 80 references.