Pith. sign in

REVIEW 3 major objections 5 minor 130 references

PhoenixRepair claims that expanding the repair search space—by sampling multiple candidate edit locations and iteratively refining patches—consistently lifts automated issue resolution over single-location agents, reaching 76.0% Pass@1 on S

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 14:06 UTC pith:EYF7A3TF

load-bearing objection A strong, well-executed SWE-bench scaffold with state-of-the-art numbers, but the evidence is single-run and the internal selection mechanism is less validated than the ablations imply. the 3 major comments →

arxiv 2607.18859 v1 pith:EYF7A3TF submitted 2026-07-21 cs.AI

PhoenixRepair: Rethinking Repair Strategy Exploration in Software Agents

Tianyue Jiang , Yanlin Wang , Xin He , Daya Guo , Jiachi Chen , Ming Wen , Ensheng Shi , Xilin Liu
show 2 more authors
Yuchi Ma Guanbin Li
This is my paper
classification cs.AI
keywords automated issue resolutionmulti-agent LLM frameworkfault localizationmulti-location samplingiterative patch refinementSWE-bench-Verifiedtest selectionsoftware repair agents
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that the main bottleneck in agentic software repair is not the model alone but the exploration strategy: existing agents commit too early to one edit location and one patch. PhoenixRepair widens the search at both levels, sampling several candidate edit locations and then repeatedly generating, testing, selecting, and refining patches at each location. If correct, the result is a scaffold-level gain that transfers across different backbone LLMs and even across other agent scaffolds. The strongest reported numbers are a 76.0% Pass@1 with MiniMax-M2.5 and a 7.8% relative improvement over SWE-agent under DeepSeek-V3.1 on SWE-bench-Verified.

Core claim

On its own terms, the paper's discovery is that a repair agent's search space can be expanded without changing the underlying LLM. The authors identify two failure modes in existing agents: repeated sampling runs tend to converge on the same confident-but-wrong edit location (P1), and even when the correct location is found, a single repair attempt often produces an incomplete patch (P2). PhoenixRepair addresses both: a localization sampling agent produces multiple candidate edit locations, optionally augmented by graph-based cross-file dependency information for hard tasks; a coder agent generates patches for each location; a selector agent ranks candidates using a weighted combination of L

What carries the argument

The load-bearing mechanism is a generate–analyze–select halving loop over edit locations. A localization sampling agent iteratively outputs up to N candidate edit locations, deduplicates them, and for difficult tasks merges in candidates from a graph-based localization agent that captures cross-file dependencies. A coder agent then generates a patch for each candidate location. A selector agent scores every patch using a weighted combination of (a) LLM-judged reproduction and edge-test quality and (b) regression-test pass rate, keeps the top ceil(w/2) locations, and repeats until one location remains. Finally, an analysis agent distills insights from the entire history at that location into

Load-bearing premise

The whole gain rests on at least one of the five initially sampled edit locations being the correct one and on the selection score keeping that location in the top half at every halving round; the paper does not validate either premise against the hidden test suite.

What would settle it

Run PhoenixRepair on a sample of SWE-bench-Verified tasks while logging the proxy rank of the gold edit location at each selection round; if the correct location falls out of the top ceil(w/2) in a large share of resolved tasks, or if replacing the proxy-based selection with random top-half selection preserves the Pass@1 gain, then the exploration mechanism, not the selector, is carrying the result.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Across all five backbone LLMs tested, PhoenixRepair reports the highest Pass@1, with the largest relative gain (7.8%) over SWE-agent under DeepSeek-V3.1.
  • The two core mechanisms—multi-location sampling and iterative reflection—transfer to other agent scaffolds; integrating them into Mini-SWE-agent and Live-SWE-agent yields consistent Pass@1 and localization-accuracy gains.
  • Fault localization accuracy improves at file, module, and function granularities, with the largest relative gains at the function level, indicating the framework narrows searches to precise code regions.
  • Ablation results attribute the largest performance drops to removing multi-location sampling or iterative reflection, while graph-based additional location information contributes a smaller but positive margin.
  • Raw API cost per task is about 1.85x SWE-agent, but the paper argues that prefix caching from LLM providers substantially lowers the effective overhead in practice.
  • The reported improvements are model-agnostic: they persist from moderate-capability models (DeepSeek-V3.1) to high-capability ones (MiniMax-M2.5), suggesting the exploration mechanism rather than raw model strength drives the gains.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The decisive internal assumption is that the selector's proxy (LLM-judged reproduction/edge test quality weighted with regression pass rate) ranks the correct location in the top half at every round. The paper reports no correlation between this proxy and the hidden FAIL_TO_PASS tests, so a direct calibration study would either confirm or refute the selection mechanism as the cause of the gains.
  • For tasks with exactly one deduplicated location (|L'|=1), multi-location sampling adds nothing, so any gain on those tasks must come entirely from iterative refinement at that single location. Isolating that subset would measure the two mechanisms' independent contributions more cleanly than the reported ablation.
  • If prefix caching reduces effective cost as claimed, then PhoenixRepair's real cost-performance trade-off could be competitive with much cheaper baselines; an end-to-end cost measurement with caching enabled, including cache hit rates, would settle whether the 1.85x raw overhead is the right number to worry about.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes PhoenixRepair, a multi-agent framework for automated issue resolution on SWE-bench-Verified. It targets two purported weaknesses of existing agentic repair pipelines: insufficient exploration of candidate edit locations (P1) and insufficient repair attempts per location (P2). The framework proceeds in three phases: (1) multi-location sampling, with five sequential localization iterations deduplicated and categorized by difficulty, and top-2 graph-localizer entities added for tasks with more than four distinct locations; (2) iterative reflection and refinement, in which a coder generates one patch per candidate location and a selector ranks patches by a weighted combination of LLM-judged reproduction/edge-test quality Q_test and regression-test pass rate R_pass, keeping the top ceil(w/2) locations each round until one remains; and (3) final-round generation, in which an analysis agent distills guidance from historical attempts and the coder produces the final patch. Experiments with five backbone LLMs report consistent relative Pass@1 gains over SWE-agent (5.3-7.8%), a best absolute rate of 76.0% with MiniMax-M2.5, improved file/module/function localization accuracy, transferability to Mini-SWE-agent and Live-SWE-agent, and a 1.85x raw API cost overhead. A case study on pylint-dev#6528 illustrates the refinement mechanism.

Significance. The framework is well aligned with current work on test-time scaling for software agents, and its decomposition of localization sampling, patch selection, and reflection is sensible. If the results hold, the key findings are noteworthy: systematic multi-location sampling combined with iterative refinement produces consistent, scaffold-transferable Pass@1 gains across five backbone models, and the ablations indicate that the two core mechanisms dominate over graph augmentation. Concrete strengths include internally consistent numbers across Tables I, III, and V; a transparent raw-cost analysis (Table V); a transferability study (RQ3); and a detailed case study. The headline evaluation, however, rests on a single run per configuration with no uncertainty quantification, and the paper's central mechanism--the selector-based halving loop--is not directly validated: selection weights are unreported, and the ablations conflate selector quality with additional compute. These gaps prevent full endorsement of the mechanism claims, although the empirical trend is plausible and worth publishing once the evidence is strengthened.

major comments (3)
  1. [III-B, Table I] The headline claims (5.3-7.8% relative gains in Table I) rest on a single run of a stochastic pipeline per configuration. For a binary metric on 500 tasks, the binomial standard error of a Pass@1 value is roughly 2 percentage points; the smallest margin (MiniMax-M2.5: 76.0 vs 72.2, i.e., 3.8 pp) is only about 1.8 standard errors. No confidence intervals, paired per-task tests, or repeat runs are reported. Because the 'consistent, model-agnostic gains' argument is built on five positive comparisons, I request uncertainty quantification: per-task bootstrap confidence intervals and/or a McNemar test per backbone, and ideally at least one full repeat for one or two configurations to quantify run-to-run variance.
  2. [II-B, Table IV, Fig. 5] The iterative phase is the paper's core mechanism, yet the selector that implements it is never validated. Section II-B ranks patches by a weighted combination of Q_test (LLM-judged reproduction/edge-test quality) and R_pass (regression pass rate), but the weights and exact scoring formula are not reported. The halving rule keeps only the top ceil(w/2) locations; if the correct location falls below the cutoff in any round it is permanently discarded. The paper provides no measurement of how often the gold location survives these rounds, and the only selection evidence is the anecdotal pylint-dev#6528 case. The ablation 'w/o Iterative Refl.' (Table IV) removes the entire loop, so its 3.4-4.0% relative drop conflates the selector's ranking quality with extra compute. Please report the scoring formula and weights, the survival rate of gold locations across rounds, and a compute-matched cont
  3. [II-A, II-B, Table IV] The paper classifies tasks into |L'|=1, 1<|L'|<=4, and |L'|>4 but never reports the counts or per-category Pass@1. This is load-bearing for mechanism interpretation: for |L'|=1 tasks the iterative halving loop is not exercised, so gains there must come solely from parallel final-round sampling (t=3) and distilled guidance; for |L'|=2 tasks a single binary selector decision can discard the correct location. Also, the ablation 'w/o Additional Location Info.' shows only a 0.3% relative drop (0.2 percentage points), within single-run noise, so the conclusion that the graph-based component 'contributes positively' is unsupported. Please report per-category task counts and Pass@1, and consider per-category ablations.
minor comments (5)
  1. [III-A, III-E, Tables I-IV] The model naming is inconsistent: Table I uses 'Qwen-Coder-Plus,' while Tables II and IV use 'Qwen3-Coder-Plus' (Qwen3-CP). If these are the same model, unify the name; if they differ, state which backbone was used in each table.
  2. [III-F, Table V] The prefix-caching discussion is qualitative. Table V reports only raw API cost; the claim that the effective cost is 'considerably lower' requires measuring actual cached/un-cached token billing. As written, the 1.85x overhead is the only cost figure supported by reported data.
  3. [II-C] The 'final round edit location' is used before being defined. Clarify that this location is the single one surviving the last selection round for medium/challenging tasks, and the sole location for simple tasks. Also consider a table listing hyperparameters N=5, k=4, l=2, t=3, currently given only in prose.
  4. [I, II-A, IV-A] The introduction contains a duplicated sentence ('Our experimental evaluation demonstrates ... Our evaluation demonstrates ...'). Other typos: 'the epository navigation' (Section IV-A), 'which effectivelyaddresses P1' (Section I), 'we treats each entity' (Section II-A).
  5. [Abstract, I] The phrase 'attains the highest resolved rate of 76.0% Pass@1' could be read as a global state-of-the-art claim; as reported in Table I it is the best among the compared methods under the tested backbones. Please clarify.

Circularity Check

0 steps flagged

No circularity: Pass@1 is an external benchmark evaluation; no claim reduces by construction to fitted inputs or self-citations.

full rationale

PhoenixRepair's central claims are empirical benchmark results on SWE-bench-Verified: Pass@1 is measured by applying the framework's final generated patch to hidden FAIL_TO_PASS/PASS_TO_PASS tests. The iterative selection mechanism (Section II-B) ranks candidate patches using reproduction/edge-test quality and regression-test pass rate, but these are explicitly separate from the final SWE-bench evaluation, which is applied post hoc to the final patch. No equation in the paper defines a prediction in terms of the benchmark outcome, and the selection weights are tuning details rather than fitted parameters whose values are then reported as predictions. The ablation study (Section III-E) compares framework variants under the same evaluation protocol, so the reported drops are empirical measurements rather than tautological consequences. The paper's own limitation statement (Section V, Construct Validity) acknowledges that the analysis agent's test-quality judgment may be subjective, but that is a stated threat to validity, not a circular reduction. Self-citations appear only in background/related work (e.g., code generation and code search references) and are not load-bearing for the framework's claimed contribution; the method's technical dependencies (SWE-agent, Agentless, LocAgent, Trae-agent) are external prior work. No self-definitional step, fitted-input-as-prediction, imported uniqueness theorem, or ansatz-by-self-citation was found. The central result is an externally falsifiable benchmark comparison and is not equivalent to its own inputs by construction.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

No new physical or ontological entities are introduced; PhoenixRepair is a software scaffold composed of existing agent and localization components. The empirical central claim rests on hand-set hyperparameters (N, k, graph-l, t, unreported selector weights) and on two unvalidated domain assumptions: the predictive power of the selection proxies and single-run representativeness.

free parameters (5)
  • N (localization sampling iterations) = 5
    Set in Section III-A4 ('we sample edit locations for five iterations'). No sensitivity analysis; controls breadth of location exploration and, with k, determines task difficulty categorization.
  • k (difficulty threshold) = 4
    Set in Section III-A4 ('high fault localization difficulty if ... exceeds four'). Tasks with more than 4 deduplicated locations receive graph-based augmentation; hand-picked and material to which tasks use LocAgent.
  • top-l graph entities = 2
    Section III-A4: 'we retain the top two entities identified by the graph-based localization agent.' Controls the size of graph augmentation.
  • t (parallel executions for simple tasks) = 3
    Section III-A4: 'we perform three parallel executions on the final round edit location.' Controls final-round sampling for tasks with |L'|=1.
  • Selection weights (Q_test vs R_pass) = not reported
    Section II-B: the selector ranks patches using 'a weighted combination' of reproduction/edge test quality and regression pass rate, but the weights are never specified, leaving the selection rule incompletely defined.
axioms (4)
  • domain assumption The selection proxies (LLM-judged reproduction/edge test quality and regression test pass rate) are predictive of whether a patch passes the hidden SWE-bench FAIL_TO_PASS tests.
    The halving procedure in Section II-B relies on this correlation; if the proxies are uninformative or overfit, the correct edit location can be permanently discarded. The paper provides no analysis of selection accuracy.
  • domain assumption The hidden FAIL_TO_PASS test patch is not available to the agents during repair, so regression/reproduction test signals are not the evaluation itself.
    Standard SWE-bench protocol, but the paper never states whether reproduction tests are drawn from the withheld gold tests. If the evaluation tests were visible to the selector, Pass@1 would be partially circular.
  • domain assumption Single-run evaluation is representative of each method's performance.
    No seeds or variance are reported (Section III-B); stochastic LLM sampling and reflection make run-to-run variation possible, and the headline relative gains are small enough to be affected by it.
  • domain assumption Baseline implementations were run faithfully under the authors' settings.
    The authors state in Section V that they 'strictly followed' baseline settings, but baseline numbers are self-reported re-runs; the undefined 'Moatless-tools' entry in Table I cannot be checked.

pith-pipeline@v1.3.0-alltime-deepseek · 23176 in / 13979 out tokens · 122460 ms · 2026-08-01T14:06:18.263188+00:00 · methodology

0 comments
read the original abstract

While Large Language Models have greatly advanced automated issue resolution, existing agent-based methods exhibit a fundamental limitation in their insufficient exploration of repair strategies. This insufficiency manifests in two key aspects. First, the exploration of multiple potential edit locations is limited. Second, the exploration of repair attempts at each location is also insufficient. To address these challenges, we present PhoenixRepair, a multi-agent framework that systematically explores multiple candidate edit locations and performs iterative reflection and refinement on patch generation, thereby expanding the search space of repair strategies. Our framework begins with multi-location sampling, optionally augmented with graph-based localization information for difficult tasks, followed by iterative reflection and refinement to generate better patches, culminating in final-round generation guided by distilled insights from all historical attempts. Experiments on SWE-bench-Verified demonstrate that PhoenixRepair achieves the largest relative improvement of 7.8\% over SWE-agent under DeepSeek-V3.1, and attains the highest resolved rate of 76.0\% Pass@1 under MiniMax-M2.5. Meanwhile, it achieves higher fault localization accuracy than existing approaches. Our code is available at https://github.com/DeepSoftwareAnalytics/PhoenixRepair.

Figures

Figures reproduced from arXiv: 2607.18859 by Daya Guo, Ensheng Shi, Guanbin Li, Jiachi Chen, Ming Wen, Tianyue Jiang, Xilin Liu, Xin He, Yanlin Wang, Yuchi Ma.

Figure 1
Figure 1. Figure 1: Comparison of MiniMax-M2.5 Performance Under [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: A motivating example of solving django#12193, the agent consistently attempts repairs at an incorrect location across multiple sampling, but generates the right patch when constrained to the correct location. ineffective in producing a correct repair. P2: The exploration of repair attempts on each edit location is insufficient. As shown in [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of our PhoenixRepair framework. The framework comprises three main phases: (1) multiple location sampling, [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ablation study results for Mini-SWE-agent (w/ Re [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: A case study of PhoenixRepair on pylint-dev#6528. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

130 extracted references · 44 linked inside Pith

  1. [1]

    Autocoderover: Autonomous program improvement,

    Y . Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024, pp. 1592–1604

    2024

  2. [2]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruanet al., “Deepseek-v3 technical report,”arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  4. [4]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lvet al., “Qwen3 technical report,”arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?”arXiv preprint arXiv:2310.06770, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Swe-agent: Agent-computer interfaces enable automated software engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated software engineering,”Advances in Neural Information Processing Systems, vol. 37, pp. 50 528–50 652, 2024

    2024

  7. [7]

    Trae agent: An llm-based agent for software engineering with test-time scaling,

    P. Gao, Z. Tian, X. Meng, X. Wang, R. Hu, Y . Xiao, Y . Liu, Z. Zhang, J. Chen, C. Gaoet al., “Trae agent: An llm-based agent for software engineering with test-time scaling,”arXiv preprint arXiv:2507.23370, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    Openhands: An open platform for ai software developers as generalist agents,

    X. Wang, B. Li, Y . Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y . Song, B. Li, J. Singhet al., “Openhands: An open platform for ai software developers as generalist agents,”arXiv preprint arXiv:2407.16741, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Large language model-based agents for software engineering: A survey,

    J. Liu, K. Wang, Y . Chen, X. Peng, Z. Chen, L. Zhang, and Y . Lou, “Large language model-based agents for software engineering: A survey,”arXiv preprint arXiv:2409.02977, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    A deep relevance matching model for ad-hoc retrieval,

    J. Guo, Y . Fan, Q. Ai, and W. B. Croft, “A deep relevance matching model for ad-hoc retrieval,” inProceedings of the 25th ACM inter- national on conference on information and knowledge management, 2016, pp. 55–64

    2016

  11. [11]

    Codemonkeys: Scaling test-time compute for software engineering,

    R. Ehrlich, B. Brown, J. Juravsky, R. Clark, C. R ´e, and A. Mirhoseini, “Codemonkeys: Scaling test-time compute for software engineering,” arXiv preprint arXiv:2501.14723, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    On the limits of language generation: Trade-offs between hallucination and mode-collapse,

    A. Kalavasis, A. Mehrotra, and G. Velegkas, “On the limits of language generation: Trade-offs between hallucination and mode-collapse,” in Proceedings of the 57th Annual ACM Symposium on Theory of Com- puting, 2025, pp. 1732–1743

    2025

  13. [13]

    The curious decline of linguistic diversity: Training language models on synthetic text,

    Y . Guo, G. Shang, M. Vazirgiannis, and C. Clavel, “The curious decline of linguistic diversity: Training language models on synthetic text,” inThe Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL), 2024

    2024

  14. [14]

    One fish, two fish, but not the whole sea: Alignment reduces language models’ conceptual diversity,

    S. Murthy, T. Ullman, and J. Hu, “One fish, two fish, but not the whole sea: Alignment reduces language models’ conceptual diversity,” inThe Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL), 2025

    2025

  15. [15]

    Se-agent: Self-evolution trajectory optimiza- tion in multi-step reasoning with llm-based agents,

    J. Lin, Y . Guo, Y . Han, S. Hu, Z. Ni, L. Wang, M. Chen, H. Liu, R. Chen, Y . Heet al., “Se-agent: Self-evolution trajectory optimiza- tion in multi-step reasoning with llm-based agents,”arXiv preprint arXiv:2508.02085, 2025

    arXiv 2025

  16. [16]

    Agentless: De- mystifying llm-based software engineering agents,

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Agentless: De- mystifying llm-based software engineering agents,”arXiv preprint arXiv:2407.01489, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Locagent: Graph-guided llm agents for code localization,

    Z. Chen, R. Tang, G. Deng, F. Wu, J. Wu, Z. Jiang, V . Prasanna, A. Cohan, and X. Wang, “Locagent: Graph-guided llm agents for code localization,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025, pp. 8697–8727

    2025

  18. [18]

    Introducing SWE-bench Verified,

    OpenAI, “Introducing SWE-bench Verified,” https://openai.com/index/introducing-swe-bench-verified/, 2024

    2024

  19. [19]

    Swe-search: Enhancing software agents with monte carlo tree search and iterative refinement,

    A. Antoniades, A. ¨Orwall, K. Zhang, Y . Xie, A. Goyal, and W. Wang, “Swe-search: Enhancing software agents with monte carlo tree search and iterative refinement,”arXiv preprint arXiv:2410.20285, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    Mini-swe-agent,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. R. Narasimhan, and O. Press, “Mini-swe-agent,” 2024. [Online]. Available: https://github.com/SWE-agent/mini-swe-agent

    2024

  21. [21]

    Live-swe-agent: Can software engineering agents self-evolve on the fly?

    C. S. Xia, Z. Wang, Y . Yang, Y . Wei, and L. Zhang, “Live-swe-agent: Can software engineering agents self-evolve on the fly?”arXiv preprint arXiv:2511.13646, 2025

    arXiv 2025

  22. [22]

    Context caching on disk,

    DeepSeek, “Context caching on disk,” 2024. [Online]. Available: https://api-docs.deepseek.com/news/news0802

    2024

  23. [23]

    Prompt caching,

    Anthropic, “Prompt caching,” 2024. [Online]. Available: https: //platform.claude.com/docs/en/build-with-claude/prompt-caching

    2024

  24. [24]

    Towards an understanding of large language models in software engineering tasks,

    Z. Zheng, K. Ning, Q. Zhong, J. Chen, W. Chen, L. Guo, W. Wang, and Y . Wang, “Towards an understanding of large language models in software engineering tasks,”Empirical Software Engineering, vol. 30, no. 2, p. 50, 2025

    2025

  25. [25]

    Large language models are qualified benchmark builders: Rebuilding pre-training datasets for advancing code intelligence tasks,

    K. Yang, X. Mao, S. Wang, Y . Wang, T. Zhang, B. Lin, Y . Qin, Z. Zhang, Y . Lu, and K. Al-Sabahi, “Large language models are qualified benchmark builders: Rebuilding pre-training datasets for advancing code intelligence tasks,”arXiv preprint arXiv:2504.19444, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    A survey of large language models for code: Evolution, benchmarking, and future trends,

    Z. Zheng, K. Ning, Y . Wang, J. Zhang, D. Zheng, M. Ye, and J. Chen, “A survey of large language models for code: Evolution, benchmarking, and future trends,”arXiv preprint arXiv:2311.10372, 2023

    Pith/arXiv arXiv 2023

  27. [27]

    Towards an under- standing of context utilization in code intelligence,

    Y . Wang, K. Duan, D. Zheng, E. Shi, F. Zhang, Y . Wang, J. Chen, X. Liu, Y . Ma, H. Zhang, Q. Wang, and Z. Zheng, “Towards an under- standing of context utilization in code intelligence,”ACM Computing Surveys, vol. 58, no. 11, pp. 1–43, 2026

    2026

  28. [28]

    Agents in software engineering: Survey, landscape, and vision,

    Y . Wang, W. Zhong, Y . Huang, E. Shi, M. Yang, J. Chen, H. Li, Y . Ma, Q. Wang, and Z. Zheng, “Agents in software engineering: Survey, landscape, and vision,”Automated Software Engineering, vol. 32, no. 2, p. 70, 2025

    2025

  29. [29]

    Adaptive-solver framework for dynamic strategy selection in large language model reasoning,

    J. Zhou, W. Zhong, Y . Wang, and J. Wang, “Adaptive-solver framework for dynamic strategy selection in large language model reasoning,” Information Processing & Management, vol. 62, no. 3, p. 104052, 2025. IEEE TRANSACTIONS ON SOFTW ARE ENGINEERING, VOL. XX, NO. X, XX 2026 12

    2025

  30. [30]

    Identifying smart contract security issues in code snippets from stack overflow,

    J. Chen, C. Chen, J. Hu, J. Grundy, Y . Wang, T. Chen, and Z. Zheng, “Identifying smart contract security issues in code snippets from stack overflow,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024, pp. 1198–1210

    2024

  31. [31]

    Hyperion: Unveiling dapp inconsistencies using llm and dataflow-guided symbolic execution,

    S. Yang, X. Lin, J. Chen, Q. Zhong, L. Xiao, R. Huang, Y . Wang, and Z. Zheng, “Hyperion: Unveiling dapp inconsistencies using llm and dataflow-guided symbolic execution,”arXiv preprint arXiv:2408.06037, 2024

    Pith/arXiv arXiv 2024

  32. [32]

    Sotana: The open-source software development assistant,

    E. Shi, F. Zhang, Y . Wang, B. Chen, L. Du, H. Zhang, S. Han, D. Zhang, and H. Sun, “Sotana: The open-source software development assistant,”arXiv preprint arXiv:2308.13416, 2023

    Pith/arXiv arXiv 2023

  33. [33]

    Re- pomincoder: Improving repository-level code generation based on information loss screening,

    Y . Li, E. Shi, D. Zheng, K. Duan, J. Chen, and Y . Wang, “Re- pomincoder: Improving repository-level code generation based on information loss screening,” inProceedings of the 15th Asia-Pacific Symposium on Internetware, 2024, pp. 229–238

    2024

  34. [34]

    Beyond functional correctness: Investigating coding style inconsistencies in large language models,

    Y . Wang, T. Jiang, M. Liu, J. Chen, M. Mao, X. Liu, Y . Ma, and Z. Zheng, “Beyond functional correctness: Investigating coding style inconsistencies in large language models,”Proceedings of the ACM on Software Engineering, vol. 2, no. FSE, pp. 690–712, 2025

    2025

  35. [35]

    Top general performance= top domain performance? do- maincodebench: A multi-domain code generation benchmark,

    D. Zheng, Y . Wang, E. Shi, X. Liu, Y . Ma, H. Zhang, and Z. Zheng, “Top general performance= top domain performance? do- maincodebench: A multi-domain code generation benchmark,”arXiv preprint arXiv:2412.18573, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Llm hallucinations in practical code gen- eration: Phenomena, mechanism, and mitigation,

    Z. Zhang, C. Wang, Y . Wang, E. Shi, Y . Ma, W. Zhong, J. Chen, M. Mao, and Z. Zheng, “Llm hallucinations in practical code gen- eration: Phenomena, mechanism, and mitigation,”Proceedings of the ACM on Software Engineering, vol. 2, no. ISSTA, pp. 481–503, 2025

    2025

  37. [37]

    S*: Test time scaling for code generation,

    D. Li, S. Cao, C. Cao, X. Li, S. Tan, K. Keutzer, J. Xing, J. E. Gonzalez, and I. Stoica, “S*: Test time scaling for code generation,” arXiv preprint arXiv:2502.14382, 2025

    Pith/arXiv arXiv 2025

  38. [38]

    Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings,

    S. Quan, J. Yang, B. Yu, B. Zheng, D. Liu, A. Yang, X. Ren, B. Gao, Y . Miao, Y . Fenget al., “Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings,”arXiv preprint arXiv:2501.01257, 2025

    Pith/arXiv arXiv 2025

  39. [39]

    Design2code: Benchmarking multimodal code generation for automated front-end engineering,

    C. Si, Y . Zhang, R. Li, Z. Yang, R. Liu, and D. Yang, “Design2code: Benchmarking multimodal code generation for automated front-end engineering,” inProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2025, pp. 3956–3974

    2025

  40. [40]

    What to retrieve for effective retrieval-augmented code generation? an empirical study and beyond,

    W. Gu, J. Chen, Y . Wang, T. Jiang, X. Li, M. Liu, X. Liu, Y . Ma, and Z. Zheng, “What to retrieve for effective retrieval-augmented code generation? an empirical study and beyond,”arXiv preprint arXiv:2503.20589, 2025

    Pith/arXiv arXiv 2025

  41. [41]

    Rmcbench: Benchmarking large language models’ resistance to malicious code,

    J. Chen, Q. Zhong, Y . Wang, K. Ning, Y . Liu, Z. Xu, Z. Zhao, T. Chen, and Z. Zheng, “Rmcbench: Benchmarking large language models’ resistance to malicious code,” inProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, 2024, pp. 995–1006

    2024

  42. [42]

    Humanevo: An evolution-aware benchmark for more re- alistic evaluation of repository-level code generation,

    D. Zheng, Y . Wang, E. Shi, R. Zhang, Y . Ma, H. Zhang, and Z. Zheng, “Humanevo: An evolution-aware benchmark for more re- alistic evaluation of repository-level code generation,”arXiv preprint arXiv:2406.06918, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    Code completion by modeling flattened abstract syntax trees as graphs,

    Y . Wang and H. Li, “Code completion by modeling flattened abstract syntax trees as graphs,” inProceedings of the AAAI conference on artificial intelligence, vol. 35, no. 16, 2021, pp. 14 015–14 023

    2021

  44. [44]

    Rlcoder: Reinforcement learning for repository-level code comple- tion,

    Y . Wang, Y . Wang, D. Guo, J. Chen, R. Zhang, Y . Ma, and Z. Zheng, “Rlcoder: Reinforcement learning for repository-level code comple- tion,”arXiv preprint arXiv:2407.19487, 2024

    Pith/arXiv arXiv 2024

  45. [45]

    Analogcoder: Analog circuit design via training-free code generation,

    Y . Lai, S. Lee, G. Chen, S. Poddar, M. Hu, D. Z. Pan, and P. Luo, “Analogcoder: Analog circuit design via training-free code generation,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 1, 2025, pp. 379–387

    2025

  46. [46]

    Domaineval: An auto-constructed benchmark for multi-domain code generation,

    Q. Zhu, J. Cao, Y . Lu, H. Lin, X. Han, L. Sun, and S.-C. Cheung, “Domaineval: An auto-constructed benchmark for multi-domain code generation,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 24, 2025, pp. 26 148–26 156

    2025

  47. [47]

    Unveiling the black box of plms with semantic anchors: towards interpretable neural semantic parsing,

    L. Nie, J. Sun, Y . Wang, L. Du, S. Han, D. Zhang, L. Hou, J. Li, and J. Zhai, “Unveiling the black box of plms with semantic anchors: towards interpretable neural semantic parsing,” inProceedings of the AAAI conference on artificial intelligence, vol. 37, no. 11, 2023, pp. 13 400–13 408

    2023

  48. [48]

    Arkrepobench: A repository-level code completion benchmark for harmonyos development,

    Y . Wang, B. Zhang, Y . Wang, D. Guo, T. Y . Zhuo, J. Chen, M. Liu, X. Zhang, and Z. Zheng, “Arkrepobench: A repository-level code completion benchmark for harmonyos development,” inFindings of the Association for Computational Linguistics: ACL 2026, 2026, pp. 19 409–19 429

    2026

  49. [49]

    Realsec-bench: A benchmark for evaluating secure code generation in real-world repositories,

    Y . Wang, Z. Zhang, C. Wang, X. Xu, M. Liu, Y . Wang, J. Chen, and Z. Zheng, “Realsec-bench: A benchmark for evaluating secure code generation in real-world repositories,” inFindings of the Association for Computational Linguistics: ACL 2026, 2026, pp. 35 866–35 883

    2026

  50. [50]

    Secret: Towards scalable and efficient code retrieval via segmented deep hashing,

    W. Gu, E. Shi, Y . Wang, L. Du, S. Han, H. Zhang, D. Zhang, and M. R. Lyu, “Secret: Towards scalable and efficient code retrieval via segmented deep hashing,”arXiv preprint arXiv:2412.11728, 2024

    Pith/arXiv arXiv 2024

  51. [51]

    Spencer: Self-adaptive model distillation for efficient code retrieval,

    W. Gu, Z. Lyu, Y . Wang, H. Zhang, C. Gao, and M. R. Lyu, “Spencer: Self-adaptive model distillation for efficient code retrieval,”ACM Transactions on Software Engineering and Methodology, 2025

    2025

  52. [52]

    Cosqa+: Enhancing code search evaluation with a multi-choice bench- mark and test-driven agents,

    J. Gong, Y . Wu, L. Liang, Y . Wang, J. Chen, M. Liu, and Z. Zheng, “Cosqa+: Enhancing code search evaluation with a multi-choice bench- mark and test-driven agents,”IEEE Transactions on Software Engineer- ing, vol. 52, no. 1, pp. 206–220, 2025

    2025

  53. [53]

    Accelerating code search with deep hashing and code classification,

    W. Gu, Y . Wang, L. Du, H. Zhang, S. Han, D. Zhang, and M. Lyu, “Accelerating code search with deep hashing and code classification,” inProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 2534– 2544

    2022

  54. [54]

    Revisiting code search in a two-stage paradigm,

    F. Hu, Y . Wang, L. Du, X. Li, H. Zhang, S. Han, and D. Zhang, “Revisiting code search in a two-stage paradigm,” inProceedings of the sixteenth ACM international conference on Web search and data mining, 2023, pp. 994–1002

    2023

  55. [55]

    Cocosoda: Effective contrastive learning for code search,

    E. Shi, Y . Wang, W. Gu, L. Du, H. Zhang, S. Han, D. Zhang, and H. Sun, “Cocosoda: Effective contrastive learning for code search,” in 2023 IEEE/ACM 45th International Conference on Software Engineer- ing (ICSE). IEEE, 2023, pp. 2198–2210

    2023

  56. [56]

    Rethinking negative pairs in code search,

    H. Li, X. Zhou, L. A. Tuan, and C. Miao, “Rethinking negative pairs in code search,”arXiv preprint arXiv:2310.08069, 2023

    Pith/arXiv arXiv 2023

  57. [57]

    Who needs the most research effort? investigating the importance of smart contract weaknesses,

    S. Chen, Y . Wang, and Z. Zheng, “Who needs the most research effort? investigating the importance of smart contract weaknesses,” inInternational Conference on Blockchain and Trustworthy Systems. Springer, 2023, pp. 197–210

    2023

  58. [58]

    You augment me: Exploring chatgpt- based data augmentation for semantic code search,

    Y . Wang, L. Guo, E. Shi, W. Chen, J. Chen, W. Zhong, M. Wang, H. Li, H. Zhang, Z. Lyuet al., “You augment me: Exploring chatgpt- based data augmentation for semantic code search,” in2023 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2023, pp. 14–25

    2023

  59. [59]

    Tackling long code search with splitting, encoding, and aggregating,

    F. Hu, Y . Wang, L. Du, H. Zhang, D. Zhang, and X. Li, “Tackling long code search with splitting, encoding, and aggregating,” inProceedings of the 2024 Joint International Conference on Computational Lin- guistics, Language Resources and Evaluation (LREC-COLING 2024), 2024, pp. 15 500–15 510

    2024

  60. [60]

    Improving code search with hard negative sampling based on fine-tuning,

    H. Dong, J. Lin, Y . Wang, Y . Leng, J. Chen, and Y . Xie, “Improving code search with hard negative sampling based on fine-tuning,” in2024 31st Asia-Pacific Software Engineering Conference (APSEC). IEEE, 2024, pp. 221–230

    2024

  61. [61]

    Costv: Accelerating code search with two-stage paradigm and vector retrieval,

    D. Zheng, Y . Wang, W. Chen, J. Chen, and Z. Zheng, “Costv: Accelerating code search with two-stage paradigm and vector retrieval,” in2024 31st Asia-Pacific Software Engineering Conference (APSEC). IEEE, 2024, pp. 383–392

    2024

  62. [62]

    Search-o1: Agentic search-enhanced large reasoning models,

    X. Li, G. Dong, J. Jin, Y . Zhang, Y . Zhou, Y . Zhu, P. Zhang, and Z. Dou, “Search-o1: Agentic search-enhanced large reasoning models,”arXiv preprint arXiv:2501.05366, 2025

    Pith/arXiv arXiv 2025

  63. [63]

    An empirical study on code search pre-trained models: Academic progresses vs. industry requirements,

    K. Chi, C. Li, J. Ge, and B. Luo, “An empirical study on code search pre-trained models: Academic progresses vs. industry requirements,” inProceedings of the 15th Asia-Pacific Symposium on Internetware, 2024, pp. 41–50

    2024

  64. [64]

    En- riching query semantics for code search with reinforcement learning,

    C. Wang, Z. Nong, C. Gao, Z. Li, J. Zeng, Z. Xing, and Y . Liu, “En- riching query semantics for code search with reinforcement learning,” Neural Networks, vol. 145, pp. 22–32, 2022

    2022

  65. [65]

    Are decoder-only large language models the silver bullet for code search?

    Y . Chen, M. Liu, G. Ou, A. Li, D. Dai, Y . Wang, and Z. Zheng, “Are decoder-only large language models the silver bullet for code search?” arXiv preprint arXiv:2410.22240, 2024

    Pith/arXiv arXiv 2024

  66. [66]

    Code search debiasing: improve search results beyond overall ranking performance,

    S. Zhang, H. Li, Y . Wang, Z. Wei, Y . Xiu, J. Wang, and R. Ji, “Code search debiasing: improve search results beyond overall ranking performance,”arXiv preprint arXiv:2311.14901, 2023

    Pith/arXiv arXiv 2023

  67. [67]

    Omnigirl: A multilingual and multimodal benchmark for github issue resolution,

    L. Guo, W. Tao, R. Jiang, Y . Wang, J. Chen, X. Liu, Y . Ma, M. Mao, H. Zhang, and Z. Zheng, “Omnigirl: A multilingual and multimodal benchmark for github issue resolution,”Proceedings of the ACM on Software Engineering, vol. 2, no. ISSTA, pp. 24–46, 2025

    2025

  68. [68]

    Magis: Llm-based multi-agent framework for github issue resolution,

    W. Tao, Y . Zhou, Y . Wang, W. Zhang, H. Zhang, and Y . Cheng, “Magis: Llm-based multi-agent framework for github issue resolution,” Advances in Neural Information Processing Systems, vol. 37, pp. 51 963–51 993, 2024. IEEE TRANSACTIONS ON SOFTW ARE ENGINEERING, VOL. XX, NO. X, XX 2026 13

    2024

  69. [69]

    Swe-exp: Experience-driven software issue resolution,

    S. Chen, S. Lin, X. Gu, Y . Shi, H. Lian, L. Yun, D. Chen, W. Sun, L. Cao, and Q. Wang, “Swe-exp: Experience-driven software issue resolution,”arXiv preprint arXiv:2507.23361, 2025

    arXiv 2025

  70. [70]

    Alibaba lingmaagent: Improving automated issue resolution via comprehensive repository exploration,

    Y . Ma, Q. Yang, R. Cao, B. Li, F. Huang, and Y . Li, “Alibaba lingmaagent: Improving automated issue resolution via comprehensive repository exploration,” inProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, 2025, pp. 238–249

    2025

  71. [71]

    Swe-debate: Competitive multi-agent debate for software issue resolution,

    H. Li, Y . Shi, S. Lin, X. Gu, H. Lian, X. Wang, Y . Jia, T. Huang, and Q. Wang, “Swe-debate: Competitive multi-agent debate for software issue resolution,”arXiv preprint arXiv:2507.23348, 2025

    Pith/arXiv arXiv 2025

  72. [72]

    Swe- fixer: Training open-source llms for effective and efficient github issue resolution,

    C. Xie, B. Li, C. Gao, H. Du, W. Lam, D. Zou, and K. Chen, “Swe- fixer: Training open-source llms for effective and efficient github issue resolution,”arXiv preprint arXiv:2501.05040, 2025

    Pith/arXiv arXiv 2025

  73. [73]

    Prometheus: Unified knowledge graphs for issue resolution in multilingual codebases,

    Z. Chen, Y . Pan, S. Lu, J. Xu, C. L. Goues, M. Monperrus, and H. Ye, “Prometheus: Unified knowledge graphs for issue resolution in multilingual codebases,”arXiv preprint arXiv:2507.19942, 2025

    arXiv 2025

  74. [74]

    Swe data construction, automatically!

    L. Guo, Y . Wang, C. Li, W. Tao, P. Yang, J. Chen, H. Song, D. Tang, and Z. Zheng, “Swe data construction, automatically!”Proceedings of the ACM on Software Engineering, vol. 3, no. FSE, pp. 525–546, 2026

    2026

  75. [75]

    On the evaluation of neural code summarization,

    E. Shi, Y . Wang, L. Du, J. Chen, S. Han, H. Zhang, D. Zhang, and H. Sun, “On the evaluation of neural code summarization,” inPro- ceedings of the 44th international conference on software engineering, 2022, pp. 1597–1608

    2022

  76. [76]

    Cast: Enhancing code summarization with hierarchical splitting and recon- struction of abstract syntax trees,

    E. Shi, Y . Wang, L. Du, H. Zhang, S. Han, D. Zhang, and H. Sun, “Cast: Enhancing code summarization with hierarchical splitting and recon- struction of abstract syntax trees,”arXiv preprint arXiv:2108.12987, 2021

    Pith/arXiv arXiv 2021

  77. [77]

    Repotransbench: A real-world benchmark for repository-level code translation,

    Y . Wang, Y . Wang, S. Wang, D. Guo, J. Chen, J. Grundy, X. Liu, Y . Ma, M. Mao, H. Zhanget al., “Repotransbench: A real-world benchmark for repository-level code translation,”arXiv preprint arXiv:2412.17744, 2024

    arXiv 2024

  78. [78]

    Repository- level code translation benchmark targeting rust,

    G. Ou, M. Liu, Y . Chen, X. Peng, and Z. Zheng, “Repository- level code translation benchmark targeting rust,”arXiv preprint arXiv:2411.13990, 2024

    arXiv 2024

  79. [79]

    Lost in translation: A study of bugs introduced by large language models while translating code,

    R. Pan, A. R. Ibrahimzada, R. Krishna, D. Sankar, L. P. Wassi, M. Merler, B. Sobolev, R. Pavuluri, S. Sinha, and R. Jabbarvand, “Lost in translation: A study of bugs introduced by large language models while translating code,” inProceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1–13

    2024

  80. [80]

    Unraveling the potential of large language models in code translation: How far are we?

    Q. Tao, T. Yu, X. Gu, and B. Shen, “Unraveling the potential of large language models in code translation: How far are we?” in2024 31st Asia-Pacific Software Engineering Conference (APSEC). IEEE, 2024, pp. 353–362

    2024

Showing first 80 references.