Pith. sign in

REVIEW 5 major objections 6 minor 89 references

Refine After Generation: Toward Correct and Concise Patches in LLM-based Program Repair

T0 review · 5 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A small post-generation refiner can reduce LLM repair patches to developer-size changes while preserving or improving resolution.

desk verdict The verbosity characterization is the real contribution; the headline size-correctness tradeoff is partly an artifact of averaging over different resolved-instance sets before and after refinement. read the letter →

arxiv 2608.13292 v1 pith:2U536KPQ submitted 2026-08-13 cs.SE

Wenqiang Luo , Jacky Keung , Xiaoyu Shi , Yicheng Sun , Boyang Yang , Zhou Yang , Haoye Tian This is my paper · ORCID
classification cs.SE
keywords automaticprogramrepairpatchverbosityrefinementpost-generationdirectpreferenceoptimizationLLM-basedminimalpatchescodereview
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models now resolve real repository-level bugs, but the patches they produce are far bulkier than what a developer would write. This paper claims that patch verbosity is systemic: across 28 leading repair approaches, every one exceeded the developer's patch in total line changes, with the median at +121.78%. It then proposes treating minimality as a separate post-generation step rather than a generator constraint. The paper's central result is that a small, trained refiner placed after generation can cut average total changes from +242.14% to +4.24% relative to developer patches across four host systems, while preserving or increasing the number of resolved issues by up to 42. If true, concise repairs do not require retraining large repair agents.

What carries the argument

The central mechanism is post-generation patch refinement, formalized as a task that takes problem context, code context, and a candidate patch and returns a patched program that satisfies the intended fix specification while making a smaller change. The instantiation is RECAP, a three-part adapter: a collector that standardizes context and patch format from a host framework, a filter that decides whether refinement is safe under an unconditional, judge-guided, or oracle-guided deployment mode, and a small open-source refiner trained on the refinement task. The training data is constructed from function-level repair pairs, tangled commits paired with atomic fixes, and synthetic repository-level examples made by inflating concise developer patches; SFT teaches the verbose-to-concise transformation and DPO supplies the negative signal that stops the model from over-compressing.

What would settle it

Take a set of real candidate patches from a new agent whose verbosity comes from exploration leftovers—failed alternatives, reverted edits, repeated tool calls—rather than from the paper's listed inflation patterns, run RECAP on them, and check whether they shrink without failing tests. If the refiner cannot reduce this structurally different verbosity, the central assumption about the synthetic training distribution is false.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that correctness and concision can be decoupled: a compact model that never sees the original bug can take a successful but verbose candidate patch, along with the issue and code context, and rewrite it into a smaller, less complex patch that still passes the tests. The evidence is a refinement pipeline trained on 5,540 verbosity-reducing pairs from three sources, using supervised fine-tuning followed by direct preference optimization with distilled reasoning traces. Across four host systems, this adapter reduced average total changes from +242.14% to +4.24% and net changes from +348.24% to -39.75% relative to developer patches, while strict test-gated deployment improved resolution by up to 42 instances and never dropped a resolved one. The paper also reports a mechanical origin for the problem: iterative refinement and higher-resolution capability are the design factors most associated with larger patches, while output format and minimality prompts have little effect.

Load-bearing premise

The whole approach depends on the synthetic training data—gold patches inflated according to a hand-written list of bad patterns—being a faithful stand-in for how real repair agents actually make patches verbose; if real verbosity has a different structure, the refiner's size gains may not transfer.

Editorial extensions

If this is right

  • Verbosity is a systemic property, not an outlier effect: all 28 evaluated approaches exceed developer patch size, so future repair systems should be measured on patch size as well as resolution.
  • Minimality prompts and structured output formats will not fix verbosity; only changing how the patch is produced or adding a post-generation step changes the outcome.
  • A compact refiner can be bolted onto an existing repair pipeline without retraining the generator, and the biggest size gains come on the most verbose agents.
  • Refinement can turn failing candidates into passing ones, recovering up to 43 instances on one host, so removing extraneous edits is itself a correctness improvement.
  • Optimizing a single size metric is unsafe: the SFT-only refiner wins on one metric but resolves the fewest instances, while DPO restores correctness without giving back concision.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference beyond the paper: a refinement adapter of this shape could sit in any diff-producing pipeline—code review bots, CI auto-fixers, or commit squashing tools—where the goal is to make machine-written edits reviewable, not just to repair bugs.
  • Inference beyond the paper: the reported capability–concision tradeoff implies that as repair agents get stronger through more iterative search, their patches will get larger; post-generation refinement may therefore become a necessary companion to scaling, not a temporary patch.
  • Inference beyond the paper: using a stronger oracle than test-passing, such as property-based tests or differential testing, would test whether the refiner's smaller patches preserve behavior beyond what a benchmark test suite can verify.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper identifies patch verbosity as a systemic problem in LLM-based APR. It characterizes 28 SWE-bench Verified approaches and reports that the median resolved patch is 121.78% larger in total changes, 80.91% larger in net changes, and 43.99% higher in cyclomatic complexity than the developer (gold) patch. It then proposes RECAP, a post-generation adapter with a collector, a filter, and a compact refiner trained via SFT and DPO on a multi-source dataset (function-level pairs, tangled commits, and synthetic inflated SWE-bench patches). The evaluation on four host systems across three deployment modes (UR, JGR, OGR) claims a substantially better size-correctness tradeoff than prompting, commit-untangling, and minimality-aware baselines, cutting average total changes from +242.14% to +4.24% while preserving or improving resolution. The paper also presents an ablation showing SFT alone over-compresses and DPO restores correctness, and a generalizability experiment with three open-source backbones.

Significance. If the central claims hold, this is a valuable contribution. The RQ-1 characterization of verbosity across 28 public leaderboard systems is a useful empirical resource, and the proposed adapter design is practical: it is plug-and-play, tested on multiple hosts and backbones, and the transition decomposition in Table IX is a thoughtful way to expose broken and recovered instances. The paper ships concrete examples of refinement beyond simple line deletion, and the SFT/DPO ablation cleanly illustrates a known failure mode of aggressive minimization. The significance is somewhat tempered by the fact that the size-reduction headline is measured as distance to the gold patch, which is also the training target, and by the absence of repeated runs; nevertheless, the empirical characterization and the adapter framework are likely to be useful to the APR community.

major comments (5)
  1. [§V-D, Table IX] The headline RQ-2 size-reduction numbers are not computed on a fixed set of instances, so the reported drop from +242.14% to +4.24% is partly compositional. The per-instance Δ metrics are averaged over each method's resolved instances (Section V-B3, applied in Section V-C), but Table IX shows the resolved sets differ before and after refinement: for SWE-agent UR, 314 instances are C→C, 19 are C→I, and 43 are I→C, so the post-refinement average includes 43 recovered instances and excludes 19 broken ones. The effect is strongest for OGR, where the oracle filter forces C→I=0 and adds I→C gains by construction (Eq. 4), so the post-refinement set is systematically different from the host's resolved set. Without size statistics per transition, the paper cannot attribute the size reduction to per-patch refinement on the original resolved instances. Please report ΔTotal/ΔNet on the C→C intersection, and separately for the I→C recovered set, or run a per-instance paired comparison on the common resolved subset.
  2. [Abstract and §V-D, Table VI] The abstract's claim that RECAP preserves or improves resolution is contradicted by the paper's own Table VI: on Moatless, RECAP-UR resolves 350/500 (-4) and RECAP-JGR resolves 348/500 (-6). Similarly, Table VIII shows resolution losses for Gemma-4-4B on SWE-agent (-5), Moatless (-24), and Openhands (-7), and for Mistral on Moatless (-8). The claim should be qualified to the OGR mode, or to 'preserving or improving resolution in most settings', and the exceptions should be discussed. This matters because the 'preserving or improving resolution' phrase is a central part of the advertised tradeoff.
  3. [§III-C, Threats to Validity] The transferability of the refiner rests on an unverified assumption that the synthetic training distribution resembles real APR verbosity. The paper constructs verbose inputs by asking GPT-5.2 to inflate gold patches according to the hand-written Table I patterns, or by rejection-sampling 10 LLM-written patches and keeping the largest. No evidence is provided that real host patches (e.g., SWE-agent's or Openhands's patches) exhibit the same structure of verbosity, rather than, say, agentic exploration noise or poor localization. The Threats to Validity section acknowledges LLM-judging and inflation as possible noise sources, but does not validate the synthetic distribution against real host patches. A concrete test would be to compare distributional properties (edit locality, file scope, line-level vs block-level changes) of synthetic and real host patches, or to train on a sample of real host patches and measure whether the gains persist.
  4. [§V-C and §III-C] The size-reduction target and the evaluation anchor are both the gold developer patch: the model is trained to map inflated gold patches back to gold patches, and results are reported as percentage distance from the same kind of reference. This alignment is close to construction for the concision metric, as the reader's circularity concern notes. The paper acknowledges that developer patches are not guaranteed to be minimal, but calling them 'relative reference points' does not resolve the circularity. Please include absolute patch sizes, report a reference-free measure (e.g., human reviewability judgments on a sample of refined patches), or at least discuss what happens when the gold patch is itself not minimal.
  5. [§V-B, Tables VI-VIII] No repeated runs or error bars are reported. The differences of a few resolved instances (e.g., Moatless UR -4 vs JGR -6, Agentless UR +4 vs JGR +11) are likely within stochastic variation of both the host agents and the refiner. The resolved-instance counts, and especially the small deltas, should be accompanied by at least three independent runs or a bootstrap over instances with confidence intervals. As written, the 'preserving or improving resolution' claim rests on single-run point estimates.
minor comments (6)
  1. [§IV-D, Eq. (4)] The text states that OGR 'guarantees it cannot drop an already-resolved instance', but Eq. (4) only guarantees this with respect to the test oracle T. A refined patch that passes the tests may still alter behavior in untested ways. The later discussion acknowledges this, but the earlier phrasing should be corrected.
  2. [§IV-B, Figure 1] The collector uses an LLM to generate a Python extraction script from a single localization artifact; it is unclear how failures or hallucinations in the generated script are detected and handled. Please describe the fallback mechanism or a sanity check on the extraction script.
  3. [Table VI caption] The 'Resolved' column reports absolute counts and, in parentheses, deltas relative to the host's resolved count; this should be stated explicitly in the caption to avoid confusion for the baseline rows.
  4. [Table VIII] Some entries appear to be missing leading signs (e.g., the SWE-agent row for Gemma-4-4B shows '5.94', '47.71', '47.15' without signs). Ensure all values are signed consistently with the metric definitions.
  5. [§V-B, metrics] Net Changes is defined as added minus deleted LOC, so it can be negative for a patch that is smaller than the gold patch. The interpretation of negative ΔNet values in the tables should be clarified, since a negative net change is not always desirable if it means removing necessary lines.
  6. [§IV-C, curriculum scheduling] The curriculum λ values (0, 0.3, 0.6, 0.9, 1.0) appear ad hoc. A sensitivity analysis or a citation to a prior work using the same schedule would strengthen the claim that curriculum learning is beneficial.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the size-reduction results are held-out generalization of a trained refiner, not forced by construction.

full rationale

The paper's derivation chain is self-contained for both central claims. RQ-1's verbosity characterization is an external empirical measurement against developer/gold patches on held-out SWE-bench Verified instances; no parameter is fitted to that data. RQ-2's size reduction is not forced by construction: the refiner is trained on synthetic pairs (inflated gold patches -> gold patches) built from the SWE-bench training split, while the evaluation is on the disjoint test split and on real host patches the refiner has never seen. The gold patch appearing both as training target and as the relative size reference means the size axis measures the trained objective, which is normal supervised evaluation, not circular prediction; the correctness axis is independently anchored by the SWE-bench test suite. The one measurement concern, that the before/after size averages are computed over different resolved-instance sets (Table IX shows C->I and I->C transitions), is a compositional validity issue rather than a circularity, since no fitted parameter or model output is algebraically determined by the reported average. Self-citations ([3], [55]) are not load-bearing: [3] is a survey cited for general background and [55] for curriculum learning, which the paper also specifies independently. No uniqueness theorem, renamed known result, or ansatz-smuggling via citation is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No physics-style free parameters are fitted to the headline numbers; the listed values are ML training hyperparameters chosen by hand. The ledger tracks them because the size reduction and correctness preservation results depend on these choices. The main domain assumptions are the test-suite oracle, the gold-patch reference, and the realism of synthetic training data.

free parameters (4)
  • Curriculum schedule lambda values = 0, 0.3, 0.6, 0.9, 1.0
    Hand-chosen mixing ratios over five SFT epochs control how quickly hard repository-level data is introduced. The refinement capability depends on this schedule, but no sensitivity analysis is reported.
  • As-is example ratio in SFT set = 20%
    The fraction of SFT instances whose input and target are identical gold patches. It determines how strongly the model is discouraged from over-refining already concise patches.
  • SFT to DPO split ratio = 4:1
    Disjoint split of 2,054 sampled synthetic instances into 1,643 SFT and 411 DPO seeds. These proportions affect the balance between concision and correctness preservation.
  • Rejection sampling keep-largest rule = 10 candidates, keep largest
    Sampling 10 patches per instance and keeping the largest defines the verbose input for DPO examples. It is an ad hoc choice for creating hard refinement targets.
assumptions (4)
  • domain assumption The SWE-bench Verified test suite is an executable oracle for patch correctness.
    Section V-B and the threats section state that passing tests shows plausibility, not semantic correctness. All resolution and size-correctness claims are measured against this oracle.
  • domain assumption Developer gold patches are a valid reference for patch minimality.
    Per-instance size ratios are computed relative to gold patches. The threats section concedes developer patches are not guaranteed to be globally minimal.
  • ad hoc to paper Inflated synthetic patches, per Table I patterns, approximate real APR verbosity.
    Section III-C constructs most repository-level training data by inflating gold patches with GPT-5.2. The realism of this distribution is assumed, not validated against real host patches.
  • standard math SWE-bench instances can be treated as independent observations for statistical tests.
    RQ-1.2 uses Mann-Whitney-Wilcoxon and Kruskal-Wallis tests over instances, but multiple instances come from the same repository, so instance-wise tests may understate correlation and inflate significance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Refine After Generation: Toward Correct and Concise Patches in LLM-based Program Repair." pith.science (2026). https://pith.science/paper/2U536KPQ

@misc{pith2026260813292,
  author       = {Pith},
  title        = {Pith review of: Refine After Generation: Toward Correct and Concise Patches in LLM-based Program Repair},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2U536KPQ}},
  note         = {Machine review of arXiv:2608.13292}
}
read the original abstract

Large language models (LLMs) have advanced automatic program repair (APR) to the point where agentic systems routinely resolve real-world, repository-level issues. Yet the generated patch has received little scrutiny beyond whether it passes tests. In this paper, we identify patch verbosity as a major yet overlooked concern in LLM-based APR. Characterizing 28 state-of-the-art approaches on SWE-bench Verified, we find that even successful patches are consistently larger and more complex than developer patches, with the median approach producing 121.78% more total changes, 80.91% more net changes, and 43.99% higher cyclomatic complexity. We further show that this verbosity is rooted in capability-oriented design choices such as iterative refinement and broad context, and can hardly be reduced by surface-level controls such as output format or minimality prompts. Motivated by these findings, we formulate post-generation patch refinement and propose RECAP, a lightweight, plug-and-play adapter that attaches to existing repair frameworks after generation. RECAP's refiner is trained via supervised fine-tuning and direct preference optimization with distilled reasoning traces, on a dataset of patch pairs we construct from multiple sources. Across four host systems, prompting, commit-untangling, and minimality-aware baselines reduce patch size only by sacrificing 49 to 217 resolved instances. In contrast, RECAP achieves a substantially better size-correctness tradeoff, cutting average total changes from +242.14% to +4.24% and net changes from +348.24% to -39.75% relative to developer patches while preserving or improving resolution by up to 42 instances. Our results indicate that minimality cannot be simply reduced to syntactic compression, and that decoupling minimization from generation offers a practical path to more reviewable repairs.

Figures

Figures reproduced from arXiv: 2608.13292 by the authors.

Figure 1
Figure 1. Overview of RECAP. 0/30/60/90/100%, so the model consolidates simpler refine￾ments before specializing on repository-level patches. DPO phase. We apply DPO [23] to contrast concise targets with less-preferred model-sampled patches. We use DPO over online RL such as PPO [57] and GRPO [58], since optimizing a static preference dataset needs no separate reward model or on-policy rollouts, keeping preference tuning chea… view at source ↗
Figure 3
Figure 3. An example of fine-grained patch refinement. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 2
Figure 2. An example of patch refinement by only removal. [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

89 extracted references · 46 canonical work pages

  1. [1]

    A systematic literature review on large language models applications in computer programming teaching evaluation process,

    A. F. Pereira and R. F. Mello, “A systematic literature review on large language models applications in computer programming teaching evaluation process,”IEEE Access, 2025

    2025

  2. [2]

    Patch generation in apr: A survey from the perspectives of utilizing llms and using apr-specific information,

    Y . Yang, C. Li, Z. Han, R. Li, K. Xu, Q. Li, W. Zhong, Z. Shen, Z. Fei, J. Geet al., “Patch generation in apr: A survey from the perspectives of utilizing llms and using apr-specific information,”ACM Transactions on Software Engineering and Methodology, 2025

    2025

  3. [3]

    A survey of llm-based automated program repair: Taxonomies, design paradigms, and applications,

    B. Yang, Z. Cai, F. Liu, B. Le, L. Zhang, T. F. Bissyand ´e, Y . Liu, and H. Tian, “A survey of llm-based automated program repair: Taxonomies, design paradigms, and applications,”arXiv preprint arXiv:2506.23749, 2025

    arXiv 2025

  4. [4]

    When large language models confront repository-level automatic program repair: How well they done?

    Y . Chen, J. Wu, X. Ling, C. Li, Z. Rui, T. Luo, and Y . Wu, “When large language models confront repository-level automatic program repair: How well they done?” inProceedings of the 2024 IEEE/ACM 46th inter- national conference on software engineering: companion proceedings, 2024, pp. 459–471

    2024

  5. [5]

    Agentic software issue resolution with large language models: A survey,

    Z. Jiang, D. Lo, and Z. Liu, “Agentic software issue resolution with large language models: A survey,”arXiv preprint arXiv:2512.22256, 2025

    arXiv 2025

  6. [6]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” inInternational Conference on Learning Representa- tions, vol. 2024, 2024, pp. 54 107–54 157

    2024

  7. [7]

    Automated program repair,

    C. Le Goues, M. Pradel, and A. Roychoudhury, “Automated program repair,”Communications of the ACM, vol. 62, no. 12, pp. 56–65, 2019

    2019

  8. [8]

    Small patches get in!

    P. Weißgerber, D. Neu, and S. Diehl, “Small patches get in!” in Proceedings of the 2008 international working conference on Mining software repositories, 2008, pp. 67–76

    2008

Show all 89 references
  1. [9]

    Evaluating agent-based program repair at google,

    P. Rondon, R. Wei, J. Cambronero, J. Cito, A. Sun, S. Sanyam, M. Tufano, and S. Chandra, “Evaluating agent-based program repair at google,” in2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 2025, pp. 365–376

    2025

  2. [10]

    Writing acceptable patches: An empirical study of open source project patches,

    Y . Tao, D. Han, and S. Kim, “Writing acceptable patches: An empirical study of open source project patches,” in2014 IEEE International Conference on Software Maintenance and Evolution. IEEE, 2014, pp. 271–280

    2014

  3. [11]

    An exploratory study on confusion in code reviews,

    F. Ebert, F. Castor, N. Novielli, and A. Serebrenik, “An exploratory study on confusion in code reviews,”Empirical Software Engineering, vol. 26, no. 1, p. 12, 2021

    2021

  4. [12]

    Toward understanding the rhetoric of small source code changes,

    R. Purushothaman and D. E. Perry, “Toward understanding the rhetoric of small source code changes,”IEEE Transactions on Software Engi- neering, vol. 31, no. 6, pp. 511–526, 2005

    2005

  5. [13]

    Directfix: Looking for simple program repairs,

    S. Mechtaev, J. Yi, and A. Roychoudhury, “Directfix: Looking for simple program repairs,” in2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, vol. 1. IEEE, 2015, pp. 448–458

    2015

  6. [14]

    Aligning llms to fully utilize the cross-file context in repository-level code completion,

    J. Li, H. Zhu, H. Liu, X. Shi, H. Zong, Y . Dong, K. Zhang, S. Jiang, Z. Jin, and G. Li, “Aligning llms to fully utilize the cross-file context in repository-level code completion,” in2025 40th IEEE/ACM Interna- tional Conference on Automated Software Engineering (ASE). IEEE, ...

    2025

  7. [15]

    Evaluating representation learning of code changes for predicting patch correctness in program repair,

    H. Tian, K. Liu, A. K. Kabor ´e, A. Koyuncu, L. Li, J. Klein, and T. F. Bissyand ´e, “Evaluating representation learning of code changes for predicting patch correctness in program repair,” inProceedings of the 35th IEEE/ACM international conference on automated software engin...

    2020

  8. [16]

    Utango: untangling commits with context-aware, graph-based, code change clustering learning model,

    Y . Li, S. Wang, and T. N. Nguyen, “Utango: untangling commits with context-aware, graph-based, code change clustering learning model,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022,...

    2022

  9. [17]

    Atomizer: An llm-based collaborative multi-agent framework for intent-driven commit untangling,

    K. Zhu, Z. Tian, S. Wang, M. Leng, and X. Mao, “Atomizer: An llm-based collaborative multi-agent framework for intent-driven commit untangling,”arXiv preprint arXiv:2601.01233, 2026

    2026

  10. [18]

    Toward a better understanding of probabilistic delta debugging,

    M. Zhang, Z. Xu, Y . Tian, X. Cheng, and C. Sun, “Toward a better understanding of probabilistic delta debugging,” in2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 2025, pp. 2024–2035

    2025

  11. [19]

    Wdd: Weighted delta debugging,

    X. Zhou, Z. Xu, M. Zhang, Y . Tian, and C. Sun, “Wdd: Weighted delta debugging,” in2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 2025, pp. 1592–1603

    2025

  12. [20]

    Less is more: Adaptive program repair with bug localization and preference learning,

    Z. Dai, B. Chen, Z. Zhao, X. Tang, S. Wu, C. Yao, Z. Gao, and J. Chen, “Less is more: Adaptive program repair with bug localization and preference learning,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 1, 2025, pp. 128–136

    2025

  13. [21]

    Qimeng-prepair: Precise code repair via edit- aware reward optimization,

    C. Ke, R. Zhang, J. Guo, Y . Wen, L. Ding, S. Wang, X. Zhu, X. Peng, D. Huang, Z. Duet al., “Qimeng-prepair: Precise code repair via edit- aware reward optimization,”arXiv preprint arXiv:2604.05963, 2026

    2026 arXiv

  14. [22]

    Swe-bench leaderboards,

    SWE-bench, “Swe-bench leaderboards,” https://www.swebench.com/, 2025

    2025

  15. [23]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,”Advances in neural information processing systems, vol. 36, pp. 53 728–53 741, 2023

    2023

  16. [24]

    An empirical study of sft-dpo interac- tion and parameterization in small language models,

    Y . Feng and C. Yang, “An empirical study of sft-dpo interac- tion and parameterization in small language models,”arXiv preprint arXiv:2603.20100, 2026

    2026

  17. [25]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhouet al., “Chain-of-thought prompting elicits reasoning in large language models,”Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022

    2022

  18. [26]

    Unicott: A unified framework for structural chain-of-thought distillation,

    X. Zhuang, Z. Zhu, Z. Wang, X. Cheng, and Y . Zou, “Unicott: A unified framework for structural chain-of-thought distillation,” inThe Thirteenth International Conference on Learning Representations, 2025

    2025

  19. [27]

    Cot-self-instruct: Building high- quality synthetic prompts for reasoning and non-reasoning tasks,

    P. Yu, J. Lanchantin, T. Wang, W. Yuan, O. Golovneva, I. Kulikov, S. Sukhbaatar, J. Weston, and J. Xu, “Cot-self-instruct: Building high- quality synthetic prompts for reasoning and non-reasoning tasks,”arXiv preprint arXiv:2507.23751, 2025

    2025 arXiv

  20. [28]

    Swe-agent: Agent-computer interfaces enable automated soft- ware engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated soft- ware engineering,”Advances in Neural Information Processing Systems, vol. 37, pp. 50 528–50 652, 2024

    2024

  21. [29]

    The openhands software agent sdk: A composable and extensible foundation for pro- duction agents,

    X. Wang, S. Rosenberg, J. Michelini, C. Smith, H. Tran, E. Nyst, R. Malhotra, X. Zhou, V . Chen, R. Brennanet al., “The openhands software agent sdk: A composable and extensible foundation for pro- duction agents,”arXiv preprint arXiv:2511.03690, 2025

    2025 arXiv

  22. [30]

    Agentless: De- mystifying llm-based software engineering agents,

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Agentless: De- mystifying llm-based software engineering agents,”arXiv preprint arXiv:2407.01489, 2024

    2024 arXiv

  23. [31]

    Autocoderover: Autonomous program improvement,

    Y . Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024, pp. 1592–1604

    2024

  24. [32]

    Swe-search: Enhancing software agents with monte carlo tree search and iterative refinement,

    A. Antoniades, A. ¨Orwall, K. Zhang, Y . Xie, A. Goyal, and W. Wang, “Swe-search: Enhancing software agents with monte carlo tree search and iterative refinement,” inInternational Conference on Learning Representations, vol. 2025, 2025, pp. 64 485–64 515

    2025

  25. [33]

    Alibaba lingmaagent: Improving automated issue resolution via comprehensive repository exploration,

    Y . Ma, Q. Yang, R. Cao, B. Li, F. Huang, and Y . Li, “Alibaba lingmaagent: Improving automated issue resolution via comprehensive repository exploration,” inProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, 2025, pp. 238– 249

    2025

  26. [34]

    Swe- fixer: Training open-source llms for effective and efficient github issue resolution, 2025,

    C. Xie, B. Li, C. Gao, H. Du, W. Lam, D. Zou, and K. Chen, “Swe- fixer: Training open-source llms for effective and efficient github issue resolution, 2025,”URL https://arxiv. org/abs/2501.05040, vol. 1, no. 2, p. 3

    2025 arXiv

  27. [35]

    Deepswe: Training a state-of-the-art coding agent from scratch by scaling rl,

    M. Luo, N. Jain, J. Singh, S. Tan, A. Patel, Q. Wu, A. Ariyak, C. Cai, T. Venkat, S. Zhu, B. Athiwaratkun, M. Roongta, C. Zhang, L. E. Li, R. A. Popa, K. Sen, and I. Stoica, “Deepswe: Training a state-of-the-art coding agent from scratch by scaling rl,” 2025, notion Blog

    2025

  28. [36]

    Swe-exp: Experience-driven software issue resolution,

    S. Chen, S. Lin, Y . Shi, H. Lian, X. Gu, L. Yun, D. Chen, L. Cao, J. Liu, N. Xiaet al., “Swe-exp: Experience-driven software issue resolution,” arXiv preprint arXiv:2507.23361, 2025

    2025

  29. [37]

    Lingxi: Repository-level issue resolution framework enhanced by procedural knowledge guided scaling,

    X. Yang, J. Zhou, M. Pacheco, W. Zhu, P. He, S. Wang, K. Liu, and R. Pan, “Lingxi: Repository-level issue resolution framework enhanced by procedural knowledge guided scaling,”arXiv preprint arXiv:2510.11838, 2025

    2025

  30. [38]

    Skywork-swe: Unveiling data scaling laws for software engineering in llms,

    L. Zeng, Y . Li, Y . Xiao, C. Li, C. Y . Liu, R. Yan, T. Wei, J. He, X. Song, Y . Liuet al., “Skywork-swe: Unveiling data scaling laws for software engineering in llms,”arXiv preprint arXiv:2506.19290, 2025

    2025 arXiv

  31. [39]

    Patchpilot: A cost-efficient software engineering agent with early attempts on formal verification,

    H. Li, Y . Tang, S. Wang, and W. Guo, “Patchpilot: A cost-efficient software engineering agent with early attempts on formal verification,” arXiv preprint arXiv:2502.02747, 2025

    2025 arXiv

  32. [40]

    Flexeme: Untangling commits using lexical flows,

    P.-P. P ˆart,achi, S. K. Dash, M. Allamanis, and E. T. Barr, “Flexeme: Untangling commits using lexical flows,” inProceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’20). ACM, 2020, pp. ...

    2020 doi

  33. [41]

    Perses: Syntax-guided program reduction,

    C. Sun, Y . Li, Q. Zhang, T. Gu, and Z. Su, “Perses: Syntax-guided program reduction,” inProceedings of the 40th International Conference on Software Engineering. Association for Computing Machinery, 2018, p. 361–371

    2018

  34. [42]

    C2d2: Extracting critical changes for real-world bugs with dependency-sensitive delta debugging,

    X. Song, Y . Wu, S. Liu, B. C. 0001, Y . L. 0001, and X. P. 0001, “C2d2: Extracting critical changes for real-world bugs with dependency-sensitive delta debugging,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, Vien...

    2024

  35. [43]

    Detecting multiple semantic concerns in tangled code commits,

    B. Koh, N. Walkinshaw, and D. Shin, “Detecting multiple semantic concerns in tangled code commits,”arXiv preprint arXiv:2601.21298, 2026

    2026

  36. [44]

    A first look at conventional commits classification,

    Q. Zeng, Y . Zhang, Z. Qiu, and H. Liu, “A first look at conventional commits classification,” in2025 IEEE/ACM 47th International Confer- ence on Software Engineering (ICSE). IEEE, 2025, pp. 2277–2289

    2025

  37. [45]

    Nemotron-cortexa: Enhancing llm agents for software engineering tasks via improved localization and solution diversity,

    A. Sohrabizadeh, J. Song, M. Liu, R. Roy, C. Lee, J. Raiman, and B. Catanzaro, “Nemotron-cortexa: Enhancing llm agents for software engineering tasks via improved localization and solution diversity,” in Forty-second International Conference on Machine Learning, 2025

    2025

  38. [46]

    Llms- as-judges: a comprehensive survey on llm-based evaluation methods,

    H. Li, Q. Dong, J. Chen, H. Su, Y . Zhou, Q. Ai, Z. Ye, and Y . Liu, “Llms- as-judges: a comprehensive survey on llm-based evaluation methods,” arXiv preprint arXiv:2412.05579, 2024

    2024 arXiv

  39. [47]

    Hybrid automated program repair by combining large language models and program analysis,

    F. Li, J. Jiang, J. Sun, and H. Zhang, “Hybrid automated program repair by combining large language models and program analysis,”ACM Transactions on Software Engineering and Methodology, vol. 34, no. 7, pp. 1–28, 2025

    2025

  40. [48]

    An empirical study on failures in automated issue solving,

    S. Liu, F. Liu, L. Li, X. Tan, Y . Zhu, X. Lian, and L. Zhang, “An empirical study on failures in automated issue solving,”arXiv preprint arXiv:2509.13941, 2025

    2025

  41. [49]

    Chatgpt for code refactoring: Analyzing topics, interaction, and effective prompts,

    E. A. AlOmar, L. Xu, S. Martinez, A. Peruma, M. W. Mkaouer, C. D. Newman, and A. Ouni, “Chatgpt for code refactoring: Analyzing topics, interaction, and effective prompts,” in2025 IEEE International Conference on Collaborative Advances in Software and COmputiNg (CASCON). IEEE,...

    2025

  42. [50]

    Aligning the objective of llm-based program repair,

    J. Xu, Y . Fu, S. H. Tan, and P. He, “Aligning the objective of llm-based program repair,” in2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 2025, pp. 2548–2560

    2025

  43. [51]

    Empirical eval- uation of generalizable automated program repair with large language models,

    V . Campos, R. Shariffdeen, A. Ulges, and Y . Noller, “Empirical eval- uation of generalizable automated program repair with large language models,”arXiv preprint arXiv:2506.03283, 2025

    2025

  44. [52]

    Robotic control via embodied chain-of-thought reasoning,

    M. Zawalski, W. Chen, K. Pertsch, O. Mees, C. Finn, and S. Levine, “Robotic control via embodied chain-of-thought reasoning,”arXiv preprint arXiv:2407.08693, 2024

    2024 arXiv

  45. [53]

    Llms can easily learn to reason from demonstrations structure, not content, is what matters!

    D. Li, S. Cao, T. Griggs, S. Liu, X. Mo, E. Tang, S. Hegde, K. Hakhamaneshi, S. G. Patil, M. Zahariaet al., “Llms can easily learn to reason from demonstrations structure, not content, is what matters!” arXiv preprint arXiv:2502.07374, 2025

    2025 arXiv

  46. [54]

    Mind the gap: Bridging thought leap for improved chain-of-thought tuning,

    H. Xu, Y . Yan, Y . Shen, W. Zhang, G. Hou, S. Jiang, K. Song, W. Lu, J. Xiao, and Y . Zhuang, “Mind the gap: Bridging thought leap for improved chain-of-thought tuning,”Advances in Neural Information Processing Systems, vol. 38, pp. 143 665–143 694, 2026

    2026

  47. [55]

    Semantics-aligned, curriculum-driven, and reasoning-enhanced vulnerability repair framework,

    C. Yang, T. Zhang, J. Jiang, X. Zhou, H. Tian, M. Du, J. Shi, J. Chen, Y . Li, E. L. Ouhet al., “Semantics-aligned, curriculum-driven, and reasoning-enhanced vulnerability repair framework,”arXiv preprint arXiv:2510.01002, 2025

  48. [56]

    Improving llm code generation via requirement-aware curriculum reinforcement learning,

    S. Yin, Z. Tian, J. Chen, and S. Guo, “Improving llm code generation via requirement-aware curriculum reinforcement learning,”arXiv preprint arXiv:2605.00433, 2026

    2026 arXiv

  49. [57]

    Prox- imal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Prox- imal policy optimization algorithms,”arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  50. [58]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models,

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . Li, Y . Wuet al., “Deepseekmath: Pushing the limits of mathematical reasoning in open language models,”arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  51. [59]

    Introducing gpt-5.2 — openai,

    OpenAI, “Introducing gpt-5.2 — openai,” https://openai.com/index/ introducing-gpt-5-2/, 2025

    2025

  52. [60]

    Introducing our most intelligent model yet. with state-of-the-art reasoning to help you learn, build, and plan anything

    Google, “Introducing our most intelligent model yet. with state-of-the-art reasoning to help you learn, build, and plan anything.” https://deepmind. google/models/gemini/, 2025

    2025

  53. [61]

    Qwen3.5: Towards native multimodal agents,

    Qwen Team, “Qwen3.5: Towards native multimodal agents,” February

  54. [62]

    Introducing claude sonnet 4.5,

    Anthropic, “Introducing claude sonnet 4.5,” https://www.anthropic.com/ news/claude-sonnet-4-5, 2025

    2025

  55. [63]

    Our most intelligent open models, built from gemini 3 research and technology to maximize intelligence-per-parameter,

    G. Deepmind, “Our most intelligent open models, built from gemini 3 research and technology to maximize intelligence-per-parameter,” https: //deepmind.google/models/gemma/gemma-4/, 2026

    2026

  56. [64]

    Introducing mistral 3 — mistral ai,

    M. AI, “Introducing mistral 3 — mistral ai,” https://mistral.ai/news/ mistral-3, 2025

    2025

  57. [65]

    Interactive code review for systematic changes,

    T. Zhang, M. Song, J. Pinedo, and M. Kim, “Interactive code review for systematic changes,” in2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, vol. 1. IEEE, 2015, pp. 111– 122

    2015

  58. [66]

    Automated patch transplantation,

    R. S. Shariffdeen, S. H. Tan, M. Gao, and A. Roychoudhury, “Automated patch transplantation,”ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 30, no. 1, pp. 1–36, 2020

    2020

  59. [67]

    Review participation in modern code review: An empirical study of the android, qt, and openstack projects,

    P. Thongtanunam, S. McIntosh, A. E. Hassan, and H. Iida, “Review participation in modern code review: An empirical study of the android, qt, and openstack projects,”Empirical Software Engineering, vol. 22, no. 2, pp. 768–817, 2017

    2017

  60. [68]

    Investigating technical and non-technical factors influencing modern code review,

    O. Baysal, O. Kononenko, R. Holmes, and M. W. Godfrey, “Investigating technical and non-technical factors influencing modern code review,” Empirical Software Engineering, vol. 21, no. 3, pp. 932–959, 2016

    2016

  61. [69]

    Git - git-diff documentation,

    Git, “Git - git-diff documentation,” https://git-scm.com/docs/git-diff, 2026

    2026

  62. [70]

    Trae agent: An llm-based agent for software engineering with test-time scaling,

    P. Gao, Z. Tian, X. Meng, X. Wang, R. Hu, Y . Xiao, Y . Liu, Z. Zhang, J. Chen, C. Gaoet al., “Trae agent: An llm-based agent for software engineering with test-time scaling,”arXiv preprint arXiv:2507.23370, 2025

    2025 arXiv

  63. [71]

    Swe-rizzo - software engineering on bittensor - a team rizzo subnet,

    SWE-Rizzo, “Swe-rizzo - software engineering on bittensor - a team rizzo subnet,” https://github.com/brokespace/code, 2025

    2025

  64. [72]

    Codesweep - autopilot for enterprise software mainte- nance,

    CodeSweep, “Codesweep - autopilot for enterprise software mainte- nance,” https://codesweep.ai/, 2025

    2025

  65. [73]

    Joycode swe-bench agent pipeline,

    JD, “Joycode swe-bench agent pipeline,” https://github.com/ jd-opensource/joycode-agent/, 2025

    2025

  66. [74]

    S. M. C. Ai, “Refact,” https://github.com/smallcloudai/refact, 2025

    2025

  67. [75]

    Composio sdk,

    Composio, “Composio sdk,” https://github.com/ComposioHQ/composio, 2024

    2024

  68. [76]

    Appmap navie,

    AppMap, “Appmap navie,” https://appmap.io/product/appmap-navie. html, 2024

    2024

  69. [77]

    Augment swe-bench verified agent,

    AugmentCode, “Augment swe-bench verified agent,” https://github.com/ augmentcode/augment-swebench-agent, 2025

    2025

  70. [78]

    Z.ai: Free ai chatbot,

    Z.ai, “Z.ai: Free ai chatbot,” https://chat.z.ai/, 2025

    2025

  71. [79]

    Gru ai - your agi assistant,

    Gru, “Gru ai - your agi assistant,” https://gru.ai/, 2024

    2024

  72. [80]

    Building coding agents via entropy-enhanced multi-turn preference optimization,

    J. Yu, Z. Cheng, X. Wu, and X. Xing, “Building coding agents via entropy-enhanced multi-turn preference optimization,”arXiv preprint arXiv:2509.12434, 2025

    2025

  73. [81]

    M. H. Halstead,Elements of Software Science (Operating and program- ming systems series). Elsevier Science Inc., 1977

    1977

  74. [82]

    Cyclomatic complexity,

    C. Ebert, J. Cain, G. Antoniol, S. Counsell, and P. Laplante, “Cyclomatic complexity,”IEEE software, vol. 33, no. 6, pp. 27–29, 2016

    2016

  75. [83]

    Automated clustering and pro- gram repair for introductory programming assignments,

    S. Gulwani, I. Radi ˇcek, and F. Zuleger, “Automated clustering and pro- gram repair for introductory programming assignments,”ACM SIGPLAN Notices, vol. 53, no. 4, pp. 465–480, 2018

    2018

  76. [84]

    On a test of whether one of two random variables is stochastically larger than the other,

    H. B. Mann and D. R. Whitney, “On a test of whether one of two random variables is stochastically larger than the other,”The annals of mathematical statistics, pp. 50–60, 1947

    1947

  77. [85]

    Individual comparisons by ranking methods,

    F. Wilcoxon, “Individual comparisons by ranking methods,” inBreak- throughs in statistics: Methodology and distribution. Springer, 1992, pp. 196–202

    1992

  78. [86]

    Methodology and application of the kruskal-wallis test,

    E. Ostertagova, O. Ostertag, and J. Kov ´aˇc, “Methodology and application of the kruskal-wallis test,”Applied mechanics and materials, vol. 611, pp. 115–120, 2014

    2014

  79. [87]

    A simple sequentially rejective multiple test procedure,

    S. Holm, “A simple sequentially rejective multiple test procedure,” Scandinavian journal of statistics, pp. 65–70, 1979

    1979

  80. [88]

    Dominance statistics: Ordinal analyses to answer ordinal questions

    N. Cliff, “Dominance statistics: Ordinal analyses to answer ordinal questions.”Psychological bulletin, vol. 114, no. 3, p. 494, 1993

    1993

  81. [2026]

    Available: https://qwen.ai/blog?id=qwen3.5

    [Online]. Available: https://qwen.ai/blog?id=qwen3.5

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.