Pith. sign in

REVIEW 4 major objections 6 minor 26 references

Three or four repair iterations capture nearly all LLM coding gains

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-07 23:50 UTC pith:GXJHYYFF

load-bearing objection Diminishing returns after 3-4 repair iterations is a real and useful empirical observation, but the specific threshold is softer than the paper's framing suggests. the 4 major comments →

arxiv 2607.05197 v1 pith:GXJHYYFF submitted 2026-07-06 cs.SE

Is Three the Magic Number? An Empirical Evaluation of LLM-Based Repair Loops

Tobias Kiecker , Eik Reichmann , Hosung Kang , Gabin An , Lars Grunske This is my paper
classification cs.SE
keywords repairengineeringllm-basedsoftwareacrossbudgetscodedesign
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks a deceptively simple question: when LLM-based software engineering tools iteratively generate, validate, and repair code artifacts, how many repair rounds actually matter? The authors evaluate five repair-loop tools across three tasks (code generation, test generation, code translation), three low-cost LLMs, and five datasets, measuring per-step completion rates up to ten repair iterations. They find a consistent concave-diminishing-returns curve: the first three to four repair steps account for the vast majority of achievable improvement, and by steps five through seven marginal gains approach zero. Crucially, the shape of this curve is driven more by how each tool orchestrates feedback and validation than by which LLM sits underneath — the same tool tends to produce similarly shaped completion curves across all three models, even when absolute starting performance differs substantially. The authors argue that because iteration limits directly affect cost, runtime, reproducibility, and reported performance, repair budgets should be treated as explicit experimental variables rather than arbitrary implementation details, and should be documented and justified in future work.

Core claim

Across five tools, three LLM backbones, and three software engineering tasks, the completion-rate curve as a function of repair-step count is consistently concave: the first three to four iterations capture most achievable gains, and marginal improvements drop to single-digit or sub-single-digit percentages by step three and to near zero by steps five through seven. The curve's shape is more strongly determined by workflow orchestration and feedback design than by the choice of backbone model — within each tool, different models produce similar curve shapes despite different absolute starting points. The two OCI variants illustrate this: including test cases in the initial prompt versus only

What carries the argument

The repair loop: a generate → validate → repair cycle where an LLM produces a candidate artifact, an automated validator (compiler, test suite, static analyzer) checks it, and validation failures are fed back to the LLM for another attempt, up to a fixed iteration limit. The central measured object is the per-step completion rate — the percentage of tasks successfully completed after each repair step — and its rate of change (mean relative improvement per step).

Load-bearing premise

The specific threshold of three to four steps rests on single-run experiments at temperature 0.2 with three low-cost models, without repeated executions or confidence intervals. Because LLMs are non-deterministic, the exact inflection point could shift across runs, and larger or newer models might show different diminishing-returns profiles.

What would settle it

If repeated runs showed that the inflection point varies substantially — say, sometimes landing at step two and sometimes at step six — then the specific 'three to four' claim would not hold, even if the general concave shape persisted. Alternatively, if a larger or reasoning-augmented model showed continued meaningful gains through steps five to eight on any of these tasks, the universality of the diminishing-returns threshold would be undermined.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Researchers evaluating LLM-based SE tools should report repair-loop limits as first-class experimental parameters, since different budgets yield different headline performance numbers and make cross-study comparisons misleading when limits are undocumented.
  • System designers building autonomous coding agents can cap repair iterations at three to four for most tasks, saving compute and latency without meaningfully sacrificing success rates.
  • Tool architecture — feedback format, validation strategy, prompt structure — may be a higher-leverage design target than swapping backbone models, since it shapes the repair-trajectory curve more than the model does.
  • The diminishing-returns pattern, if it holds for larger models, would further compress the practical repair budget, since stronger initial generation would leave even less room for iterative improvement.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The finding that curve shape is tool-driven rather than model-driven suggests a possible taxonomy of repair-loop architectures by their diminishing-returns profile — some tools (like OCI) flatten after one step while others (like the basic LLM call) continue rising through step three, which could inform tool selection by task.
  • If the concave pattern generalizes to larger or reasoning-augmented models, it would imply that repair loops face a structural ceiling tied to the feedback signal itself rather than model capacity — the validator can only surface so many distinct failure modes before the model re-encounters errors it already failed to fix.
  • The observation that models converge toward similar ceilings within a tool hints that the feedback channel may be the binding constraint: if two models with different starting points flatten at the same step, the bottleneck may be what information the validation step can provide, not what the model can do with it.
  • A natural extension would test whether richer feedback (e.g., natural-language explanations of failures rather than raw error messages) shifts the inflection point later, which would directly test whether feedback design is the curve-shaping variable the authors hypothesize.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper investigates how repair-loop iteration limits affect performance in LLM-based software engineering tools. The authors evaluate five tools (Basic LLM Call, OCI in two variants, INTERVENOR, CASCADE, LDB) across three tasks (code generation, test generation, code translation) and three low-cost LLMs (Gemma-4, Qwen3.5, GPT-4o-mini), measuring per-step completion rates up to 10 repair iterations. The central finding is that the first 3–4 repair iterations account for most achievable gains, with later iterations yielding only marginal improvements, and that workflow orchestration and feedback design influence repair behavior more than the backbone model. The paper concludes that repair budgets should be treated as explicit experimental variables.

Significance. The question of how many repair iterations are worthwhile is practically important for the LLM-based SE community, and the per-step evaluation across multiple tools, tasks, and models is a useful contribution. The observation that repair curves are concave and that tool architecture matters more than model choice is a finding that can inform experimental design and cost-benefit decisions. The study covers a reasonable diversity of tasks and tools, and the authors are commendably transparent about the modifications they made to tool internals (§3). The recommendation that repair budgets be explicitly reported is well-motivated by the data.

major comments (4)
  1. §3, Table 2: The central quantitative claim—that the first 3–4 repair steps account for most gains—is primarily supported by Table 2, which reports 'mean relative improvement' (percentage increase over the previous step). This metric is problematic for establishing a threshold because relative improvement mechanically diminishes as performance approaches a ceiling. For example, OCI starts near 90% completion at step 0, so even a meaningful absolute gain of 1–2 percentage points appears as a tiny relative improvement. Conversely, OCI (w/out tests) starts very low, making its step-1 relative improvement of 871.4% look dramatic but potentially misleading. The paper should supplement Table 2 with absolute marginal gains (percentage-point increases per step) to verify that the '3–4' threshold holds when measured in absolute terms. Without this, the specific threshold is not well-established.
  2. §3, §5 (Threats to Validity): No repeated runs or variance estimates are reported despite using temperature=0.2. The authors acknowledge this limitation but argue that the goal is to analyze trends rather than exact numbers. However, the paper makes a specific quantitative claim about a '3–4 step' threshold, and the reader cannot assess whether this threshold is stable across runs. At minimum, the authors should run a subset of configurations (e.g., 2–3 tools × 1 model) with multiple seeds and report whether the inflection point shifts. Without any variance information, the precision of the threshold claim is unsupported.
  3. §3: The modification to OCI—from rebuilding the prompt from scratch each iteration to using multi-turn conversation—is acknowledged by the authors as altering OCI's semantics. This is not a minor compatibility change: it fundamentally changes how feedback accumulates, since in multi-turn mode the model sees the full history of prior attempts and errors, whereas in prompt-rebuilding it sees only the original prompt plus the latest error. The paper itself argues that feedback design substantially influences repair effectiveness (§4). Given that OCI variants show the most dramatic patterns in Table 2, this modification could materially affect the central findings. The authors should either (a) run the original OCI prompt-rebuilding variant as well to isolate the effect of this change, or (b) provide a more detailed justification for why this modification does not threaten the generalizanceh
  4. §4: The '3–4' threshold is presented as a general finding, but Table 2 shows substantial variation across tools. INTERVENOR and OCI drop to single-digit relative improvements by step 2, while Basic LLM Call still shows 11.5% and 12.7% at steps 3 and 4. The paper does acknowledge per-tool differences in the text, but the abstract and conclusion present '3–4' as a uniform finding. The authors should clarify that this is an average across tools with heterogeneous behavior and discuss whether a single recommended threshold is appropriate given this variation.
minor comments (6)
  1. §3: The models are referred to as 'Gemma-4,' 'Qwen3.5,' and 'gpt-4o-mini.' The HuggingFace links suggest these are gemma-4-31B-it and Qwen3.5-27B. The paper should state the full model identifiers in the main text, not just footnotes, for reproducibility.
  2. Figure 2: The y-axis ranges differ across panels (e.g., OCI ranges from 88–98%, Basic LLM Call from 0–70%), which makes cross-tool comparison of curve shapes difficult. Consider adding a normalized version or at least noting this explicitly.
  3. Table 1: The dataset for CASCADE is listed as 'CASCADE [9],' which is self-referential. A brief description of this dataset would help readers unfamiliar with it.
  4. §4: The paper mentions 'Gemma model achieves the strongest overall performance' but does not provide a table or statistical test supporting this claim. A summary table of final completion rates per tool×model would strengthen this.
  5. §6 (Related Work): The categorization of prior work into three types is useful but brief. The paper could better position itself against Olausson et al. [12], who also study repair vs. sampling under compute budgets, by discussing what additional insights the per-step analysis provides beyond their findings.
  6. §3: The authors state 'Each datasets was also used in the tools original papers.' This should be 'Each dataset was also used in the tool's original paper.'

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee raises four major points: (1) Table 2's relative-improvement metric may distort the threshold claim; (2) no variance estimates despite stochastic sampling; (3) the OCI multi-turn modification may confound findings; (4) the '3–4 step' threshold is presented as uniform despite heterogeneous per-tool behavior. We agree with points 1, 2, and 4 and will revise accordingly. On point 3, we provide a detailed justification but also commit to running the original OCI variant as a supplementary check.

read point-by-point responses

  1. Referee: §3, Table 2: The central quantitative claim—that the first 3–4 repair steps account for most gains—is primarily supported by Table 2, which reports 'mean relative improvement' (percentage increase over the previous step). This metric is problematic for establishing a threshold because relative improvement mechanically diminishes as performance approaches a ceiling. The paper should supplement Table 2 with absolute marginal gains (percentage-point increases per step) to verify that the '3–4' threshold holds when measured in absolute terms.

    Authors: The referee is correct that relative improvement is a misleading metric when baseline completion rates differ substantially across tools, and that it can both exaggerate gains from low baselines (e.g., OCI w/out tests at 871.4%) and mask meaningful absolute gains near ceilings (e.g., OCI starting near 90%). We will add a supplementary table reporting absolute marginal gains (percentage-point increases per step) for all tool–model combinations. From the raw completion curves in Figure 2, the absolute gains do flatten within the same 3–4 step window, but we agree this must be shown explicitly rather than inferred. We will also revise the text to reference both metrics and note where they diverge. revision: yes

  2. Referee: No repeated runs or variance estimates are reported despite using temperature=0.2. The authors should run a subset of configurations with multiple seeds and report whether the inflection point shifts.

    Authors: We agree that the specific threshold claim requires some variance evidence. We will run 3 tools (Basic LLM Call, INTERVENOR, OCI w/out tests) × 1 model (GPT-4o-mini) with 5 seeds each and report the per-step completion rates with confidence intervals. This subset was chosen to span the range of starting points and curve shapes. If the inflection point shifts across seeds, we will report the range and qualify the threshold accordingly. If it is stable, we will state so explicitly. We acknowledge that without this evidence the precision of '3–4' is unsupported, and the revision will address that gap. revision: yes

  3. Referee: The modification to OCI—from rebuilding the prompt from scratch each iteration to using multi-turn conversation—fundamentally changes how feedback accumulates. The authors should either (a) run the original OCI prompt-rebuilding variant as well, or (b) provide a more detailed justification for why this modification does not threaten the generalizability of the findings.

    Authors: The referee correctly identifies that the multi-turn modification changes OCI's feedback semantics. We want to clarify the scope of our claim: the paper's central finding is about the general concavity of repair curves across diverse tools, not about OCI specifically. OCI is one of five tools studied, and the concave pattern holds across all of them, including tools (INTERVENOR, LDB, CASCADE) where no comparable modification was made. That said, we agree that isolating the effect would strengthen the OCI-specific analysis, particularly since the two OCI variants show the most dramatic patterns. We will therefore run the original prompt-rebuilding variant of OCI on MBPP with all three models and report the results alongside the multi-turn version. If the curve shape differs materially, we will discuss the implications; if not, we will note the robustness. We will also expand the justification in §3 for why the modification was necessary (the original OCI implementation relied on API-specific session handling incompatible with locally hosted models) and acknowledge the semantic difference more prominently. revision: yes

  4. Referee: The '3–4' threshold is presented as a general finding, but Table 2 shows substantial variation across tools. The abstract and conclusion present '3–4' as a uniform finding. The authors should clarify that this is an average across tools with heterogeneous behavior and discuss whether a single recommended threshold is appropriate.

    Authors: This is a fair point. The abstract and conclusion do present '3–4' more uniformly than the per-tool data supports. Basic LLM Call still shows double-digit relative improvements at steps 3 and 4, while INTERVENOR and OCI flatten by step 2. We will revise the abstract and conclusion to state that the threshold is an aggregate observation with tool-specific variation, and that the appropriate budget depends on the workflow architecture. We will also add a sentence in §4 explicitly discussing whether a single threshold recommendation is warranted, noting that tools with slower convergence (e.g., Basic LLM Call) may benefit from extended budgets while others saturate quickly. The core recommendation—that repair budgets should be explicitly reported and justified—does not require a single universal threshold and remains valid. revision: yes

Circularity Check

0 steps flagged

No circularity: the paper reports direct empirical observations of repair-loop completion rates without fitting models, tuning parameters, or making predictions that reduce to fitted inputs.

full rationale

This is an empirical study that measures completion rates at each repair step across tools, datasets, and models, then observes diminishing returns. There is no fitted model, no parameter tuning to a subset of data, and no 'prediction' that reduces to a fitted value. The claim that 'the first three to four repair iterations account for most achievable gains' is a direct empirical observation from Figure 2 and Table 2, not a derived quantity that equals its input by construction. The 'mean relative improvement' metric (Table 2) is computed as percentage increase to the previous step — a straightforward descriptive statistic, not a model fit. One author (Kiecker) is also an author of CASCADE [9], one of the evaluated tools, but this does not create circularity: CASCADE is used as a data source among five other tools, its repair-loop control logic was left unchanged, and the paper does not cite CASCADE to justify its central premise or to forbid alternatives. The paper's conclusions are observations from the data, not theoretical derivations that could be circular. The skeptic's concerns about tool modifications (OCI's prompt-rebuilding to multi-turn change), metric ceiling effects, and cross-tool averaging are correctness and generalizability concerns, not circularity — they question whether the observations generalize or are confounded, not whether the conclusion is tautologically equivalent to the input.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 0 invented entities

No new entities, particles, or frameworks are introduced. The paper uses existing tools, models, and datasets. The two free parameters (temperature, max iterations) are experimental settings, not fitted constants.

free parameters (2)
  • Temperature = 0.2
    Fixed across all experiments; not tuned but chosen as a constant.
  • Maximum repair iterations = 10
    Chosen as the upper bound for evaluation; not fitted but set as a measurement ceiling.
axioms (3)
  • domain assumption Automated validation (compilation, test execution) provides a sufficient correctness signal for measuring repair effectiveness.
    §3 and §5: all tools use automated validation; the paper acknowledges this may not capture semantic correctness.
  • domain assumption Results from low-cost LLMs (Gemma-4, Qwen3.5, GPT-4o-mini) generalize to LLM-based SE repair loops broadly.
    §5 External Validity: authors flag this as a threat; larger models may require fewer iterations.
  • ad hoc to paper Single-run results (no repeated executions) are sufficient to identify repair-loop trends.
    §5 Internal Validity: authors acknowledge non-determinism but argue trends hold; no variance is quantified.

pith-pipeline@v1.1.0-glm · 12531 in / 2132 out tokens · 88276 ms · 2026-07-07T23:50:07.725819+00:00 · methodology

0 comments
read the original abstract

Iterative repair loops have become a core design pattern in LLM-based software engineering systems. These workflows repeatedly generate, validate, and repair artifacts using feedback such as compiler errors or test failures. Despite their widespread use, the impact of repair-loop iteration limits remains poorly understood, as most prior work adopts fixed, often arbitrary, repair budgets. We study repair-loop effectiveness across multiple software engineering tasks, including code generation, test generation, and code translation. Across several representative workflows, datasets, and contemporary low-cost LLMs, we observe a consistent pattern of diminishing returns: the first three to four repair iterations account for most achievable gains, while later iterations contribute only marginal improvements. We further find that repair behavior is influenced more strongly by workflow orchestration and feedback design than by the underlying model itself. These results suggest that repair budgets should be treated as an explicit experimental variable, as they directly affect evaluation outcomes, computational cost, runtime, and reproducibility in LLM-based software engineering research.

Figures

Figures reproduced from arXiv: 2607.05197 by Eik Reichmann, Gabin An, Hosung Kang, Lars Grunske, Tobias Kiecker.

Figure 1
Figure 1. Figure 1: Generalized SE workflow with LLM-repair loop [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Completion rate as a function of repair steps across six tool–dataset combinations and three LLM backbones. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 26 canonical work pages · 3 internal anchors

  1. [1]

    Program Synthesis with Large Language Models

    Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models.CoRR abs/2108.07732 (2021). arXiv:2108.07732 https://arxiv.org/abs/2108.07732

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  2. [2]

    Vulnerability detection with code language models: How far are we?

    Islem Bouzenia, Premkumar T. Devanbu, and Michael Pradel. 2025. RepairAgent: An Autonomous, LLM-Based Agent for Program Repair. In47th IEEE/ACM In- ternational Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, 2025. IEEE, 2188–2200. doi:10.1109/ICSE55347.2025.00157

    work page doi:10.1109/icse55347.2025.00157 2025

  3. [3]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  4. [4]

    Kuen Sum Cheung, Mayuri Kaul, Gunel Jahangirova, Mohammad Reza Mousavi, and Eric Zie. 2025. Comparative Analysis of Carbon Footprint in Manual vs. LLM-Assisted Code Development. InProceedings of the 1st International Workshop on Responsible Software Engineering, ResponsibleSE 2025, Trondheim, Norway, June 23-28, 2025. ACM, 13–20. doi:10.1145/3711919.3728678

    work page doi:10.1145/3711919.3728678 2025

  5. [5]

    Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. 2023. Large language models for software engineering: Survey and open problems. In2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). IEEE, 31–53

    work page 2023

  6. [6]

    Cuiyun Gao, Xing Hu, Shan Gao, Xin Xia, and Zhi Jin. 2025. The Current Challenges of Software Engineering in the Era of Large Language Models.ACM Trans. Softw. Eng. Methodol.34, 5 (2025), 127:1–127:30. doi:10.1145/3712005

    work page doi:10.1145/3712005 2025

  7. [7]

    René Just, Darioush Jalali, and Michael D. Ernst. 2014. Defects4J: a database of existing faults to enable controlled testing studies for Java programs. InInterna- tional Symposium on Software Testing and Analysis, ISSTA ’14, San Jose, CA, USA - July 21 - 26, 2014, Corina S. Pasareanu and Darko Marinov (Eds.). ACM, 437–440. doi:10.1145/2610384.2628055

    work page doi:10.1145/2610384.2628055 2014

  8. [8]

    Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. 2024. When Can LLMsActuallyCorrect Their Own Mistakes? A Critical Survey of Self- Correction of LLMs.Trans. Assoc. Comput. Linguistics12 (2024), 1417–1440. doi:10.1162/TACL_A_00713

    work page doi:10.1162/tacl_a_00713 2024

  9. [9]

    Tobias Kiecker, Jan Arne Sparka, Martin Reuter, Albert Ziegler, and Lars Grunske

    work page

  10. [10]

    doi:10.1145/3808175

    Cascade: Detecting Inconsistencies between Code and Documentation with Automatic Test Generation.Proceedings of the ACM on Software Engineering 3, FSE, Article FSE168 (July 2026), 23 pages. doi:10.1145/3808175

    work page doi:10.1145/3808175 2026

  11. [11]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. SELF-REFINE: Iterative Refinement with Self-Feedback. InProceedings of the 37th International C...

    work page doi:10.5555/3666122.3668141 2023

  12. [12]

    Thu-Trang Nguyen, Thanh Trong Vu, Hieu Dinh Vo, and Son Nguyen. 2025. An empirical study on capability of Large Language Models in understanding code semantics.Information and Software Technology185 (2025), 107780. doi:10.1016/j. infsof.2025.107780

    work page doi:10.1016/j 2025

  13. [13]

    Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama

    Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. 2024. Is Self-Repair a Silver Bullet for Code Generation?. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum? id=y0GJXRungR

    work page 2024

  14. [14]

    Rangeet Pan, Myeongsoo Kim, Rahul Krishna, Raju Pavuluri, and Saurabh Sinha

    work page

  15. [15]

    In47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice, SEIP@ICSE 2025, Ottawa, ON, Canada, April 27 - May 3, 2025 (ICSE-SEIP 2025)

    ASTER: Natural and Multi-Language Unit Test Generation with LLMs. In47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice, SEIP@ICSE 2025, Ottawa, ON, Canada, April 27 - May 3, 2025 (ICSE-SEIP 2025). IEEE Computer Society, Los Alamitos, CA, USA, 413–424. doi:10.1109/ICSE-SEIP66354.2025.00042

    work page doi:10.1109/icse-seip66354.2025.00042 2025

  16. [16]

    Tal Ridnik, Dedy Kredo, and Itamar Friedman. 2024. Code Generation with Alpha- Codium: From Prompt Engineering to Flow Engineering.CoRRabs/2401.08500 (2024). arXiv:2401.08500 doi:10.48550/ARXIV.2401.08500

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2401.08500 2024

  17. [17]

    Baptiste Rozière, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lam- ple. 2020. Unsupervised Translation of Programming Languages. InAdvances in Neural Information Processing Systems 33: Annual Conference on Neural In- formation Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, M...

    work page 2020

  18. [18]

    Arkadii Sapozhnikov, Mitchell Olsthoorn, Annibale Panichella, Vladimir Ko- valenko, and Pouria Derakhshanfar. 2024. TestSpark: IntelliJ IDEA’s Ultimate Test Generation Companion. InProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, ICSE-Companion 2024, Lisbon, Portugal, April 14-20, 2024 (ICSE-Co...

    work page doi:10.1145/3639478.3640024 2024

  19. [19]

    Hao Tang, Keya Hu, Jin Zhou, Sicheng Zhong, Wei-Long Zheng, Xujie Si, and Kevin Ellis. 2024. Code Repair with LLMs gives an Exploration- Exploitation Tradeoff. (2024). http://papers.nips.cc/paper_files/paper/2024/ hash/d5c56ec4f69c9a473089b16000d3f8cd-Abstract-Conference.html

    work page 2024

  20. [20]

    Hanbin Wang, Zhenghao Liu, Shuo Wang, Ganqu Cui, Ning Ding, Zhiyuan Liu, and Ge Yu. 2024. INTERVENOR: Prompting the Coding Ability of Large Language Models with the Interactive Chain of Repair. InFindings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024 (Findings of ACL), Lun-Wei Ku, And...

    work page 2024

  21. [21]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. (2024). http://papers.nips. cc/paper_files/paper/2024/hash/5a7c947568c1b1328ccc5230172e1e7c-Abstract- Conference.html

    work page 2024

  22. [22]

    Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, Houda Bouamor, Ju...

    work page doi:10.18653/v1/2023.emnlp-main.151 2023

  23. [23]

    Huan Zhang, Wei Cheng, Yuhan Wu, and Wei Hu. 2024. A Pair Programming Framework for Code Generation via Multi-Plan Exploration and Feedback-Driven Refinement. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE 2024, Sacramento, CA, USA, October 27 - November 1, 2024 (ASE ’24). Association for Computing Machi...

    work page doi:10.1145/3691620.3695506 2024

  24. [24]

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Au- toCodeRover: Autonomous Program Improvement. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, Vienna, Austria, September 16-20, 2024, Maria Christakis and Michael Pradel (Eds.). ACM, 1592–1604. doi:10.1145/3650212.3680384

    work page doi:10.1145/3650212.3680384 2024

  25. [25]

    Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. 2024. OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement. InFindings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024 (Findings of ACL), Lun-Wei Ku, Andre Ma...

    work page 2024

  26. [26]

    Li Zhong, Zilong Wang, and Jingbo Shang. 2024. Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step by Step. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024 (Findings of ACL), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Associati...

    work page doi:10.18653/v1/2024.findings-acl.49 2024