Repeton: Structured Bug Repair with ReAct-Guided Patch-and-Test Cycles

Anh Chung Hoang; Chris Ngo; Nguyen Phu Vinh; Truong-Son Hy

REVIEW 3 major objections 5 minor 17 references

Repeton: Structured Bug Repair with ReAct-Guided Patch-and-Test Cycles

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A fully open, retrieval-free debug loop fixes 11.67% of SWE-bench Lite tasks.

desk verdict A reproducible open-source repair agent whose 11.67% is real but whose RAG comparison and validation-path analysis are too thin to support the abstract's claim. read the letter →

arxiv 2506.08173 v1 pith:7WLDPMBG submitted 2025-06-09 cs.SE cs.AI

Nguyen Phu Vinh , Anh Chung Hoang , Chris Ngo , Truong-Son Hy This is my paper

classification cs.SEcs.AI

keywords automatedprogramrepairLLMagentsSWE-benchLiteReActpatch-and-testretrieval-freeiterativerefinementGitrepositories

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Repeton is a fully open-source debugging agent built to show that real GitHub bugs can be repaired without embedding-based retrieval or task-specific fine-tuning. Instead of generating one holistic fix, it runs a structured patch-and-test cycle: the issue is summarized, keywords are used to locate candidate files and code regions, a single minimal patch is proposed, and a ReAct-driven Testing module tries to reproduce the bug before and after each patch. On SWE-bench Lite it resolves 35 of 300 tasks (11.67%), above the Moatless Tools baseline and below the retrieval-augmented Swe-Fixer, which the authors treat as evidence that the modular design is competitive while staying transparent and locally deployable.

What carries the argument

The load-bearing mechanism is the Iterative Repair and Validation (IRV) loop, wrapped around the Iterative Code Search and Repair (ICSR) process. ICSR moves through four stages — keyword extraction and file identification, file-structure summarization, precise code-region localization, and patch generation — with rollback at each stage and a rule that only one code region changes per iteration. IRV wraps this in a ReAct-style testing flow that first constructs a test reproducing the bug, then re-executes that test after each patch, feeding pass/fail information back into the next patch decision. The persistent problem summary, the truncation of conversation history, and the default of accepting the last patch when reproduction fails complete the system.

What would settle it

Run Repeton on the 300 SWE-bench Lite tasks twice: once with the Testing module active and once with a trivial script that always reports the bug as reproduced and accepts the last patch. If the success rates are nearly equal, the claimed test-driven validation is not carrying the result; if the rate drops sharply when reproduction fails, the bottleneck claim is confirmed.

Watch

Extended reading notes

Core claim

The central claim is that the separation of diagnosis, search, patching, and validation is what makes autonomous repair work, not the scale or retrieval apparatus of the model. Repeton's result is a workflow result: an open-weight LLM, guided by keyword heuristics and a persistent problem summary, can localize faults and produce patches that pass benchmark tests, and the pipeline can backtrack when a direction fails. The paper also reports where the loop breaks: reasoning sequences that exhaust the context window leave 152 issues with empty patches, and failures to reproduce the bug cause valid patches to be rejected in favor of the last generated patch. Those failure cases are treated as the useful output of the study, pointing to context management and test generation as the bottlenecks.

Load-bearing premise

The whole pipeline depends on the Testing module being able to write a test that reliably reproduces the reported bug; when that fails, valid patches are discarded and the last patch is accepted by default, so the resolution rate is bounded by test-generation ability rather than patch quality.

Editorial extensions

If this is right

If 11.67% on SWE-bench Lite is reproducible with any open-weight model, retrieval-free patch-and-test agents become a viable transparent baseline for autonomous repair, and benchmark comparisons should control for test-oracle quality rather than just model choice.
Because 152 of 300 failures end in empty patches from context exhaustion, changes that cap or compress the reasoning trace could translate directly into more resolved issues without any new retrieval machinery.
The default accept-last-patch rule implies that improving bug reproduction alone would recover earlier, more correct patches and could raise the success rate even if patch generation stays unchanged.
The single-region-per-iteration edit rule keeps patches minimal, so the pipeline is inspectable by a human reviewer in a way that holistic patch generators are not.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the paper's numbers suggest the binding constraint is not search or patch quality but the availability of a trustworthy test oracle; if test generation improves, most of the current failure modes should shrink.
Editorial inference: because the pipeline is embedding-free and open, it could be applied to codebases where embedding infrastructure is unavailable or where privacy rules forbid sending the full repository to a retrieval service.
A natural next experiment the paper does not run is to swap the test generator for a stronger one and measure the delta; a large delta would confirm that reproduction, not localization, is the current ceiling.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The headline is that Repeton is a fully open-source, embedding-free agent that scores 11.67% on SWE-bench Lite. That's a real data point: 35/300 resolved, with code and weights released, and a failure analysis that is more honest than most. The pipeline — ReAct-style reasoning, keyword search, conservative single-region patches, and a test-based validation loop — is a clean engineering configuration, not a new algorithm, but it's reproducible and the open release has value.

The good: the failure analysis in Section 5 is useful. It shows 152 of 300 runs produced an empty patch, and it openly admits that when the test generator can't reproduce the bug, the loop falls back to accepting the last patch without validation. That is exactly the kind of limitation a reader needs to know. The paper also positions itself against the right related work.

The soft spots are in the evaluation. The abstract claims "good performance compared to RAG-based methods," but the only RAG-adjacent baseline is Swe-Fixer, which they don't beat; Moatless Tools with DeepSeekV3 is not a RAG method and scores 0.00, so the comparison is weak. There are no error bars, no repeated runs, and no variance estimate. That would be fine for a short result if the claim were modest, but "good performance compared to RAG-based methods" overstates what the table shows.

The stress-test note about the fallback acceptance is on point, but it needs to be put in proportion. The final score is measured by the official SWE-bench tests, so a fallback-accepted patch still has to pass those tests to count. The concern is about the internal mechanism: if most of the 35 resolved were fallback acceptances, then the paper's story about test-driven validation being the reason for success is weakened. The paper should report, for the 35 resolved instances, how many were validated by a successful reproduction test versus accepted via the fallback path. Without that breakdown, the causal link between the patch-and-test loop and the score is not established. That's a missing analysis, not a fatal flaw.

Verdict: this deserves a serious referee. The contribution is a reproducible open-source baseline with an honest failure analysis. A referee should push for (1) a comparison that includes at least one RAG-based method the agent actually outperforms, (2) variance or at least a per-instance breakdown, and (3) the resolution-path breakdown. With those, the paper's claims would be properly supported. I would bring it to a reading group to discuss how to evaluate agentic repair systems, but I wouldn't cite it as evidence for a SOTA claim.

Referee Report

3 major / 5 minor

Summary. The paper presents Repeton, a fully open-source LLM-based agent for automated bug repair in real-world Git repositories. Instead of generating holistic fixes, Repeton runs a structured patch-and-test loop: it summarizes the issue, searches for relevant files using keyword heuristics, locates precise code regions, generates minimal patches, and validates each patch through a test-producing Testing module. The pipeline is guided by ReAct-style prompting and avoids embedding-based retrieval. The authors evaluate on SWE-bench Lite and report 11.67% resolved instances (35/300), which outperforms the Moatless Tools + DeepSeekV3 baseline (0.00%) but is below Swe-Fixer (24.67%). They also present a failure-case analysis and claim that the method shows good performance compared to RAG-based methods in patch validity and interpretability.

Significance. If the claims are substantiated, the contribution is a transparent, reproducible, embedding-free repair agent that decomposes debugging into modular, verifiable stages and uses only open-weight models without task-specific fine-tuning. The explicit strengths are the open-source release, the rollback mechanisms in the iterative search process, the use of runtime test feedback, and the candid failure analysis. The significance is currently limited by the thin evaluation: the comparative claim about RAG-based methods rests on only two baselines, one of which scores zero, and the validation mechanism is acknowledged to be bypassed when reproduction fails. The paper is a reasonable systems contribution, but the empirical evidence is not yet strong enough for the central comparative claim.

major comments (3)

[Section 4.1, Table 1] The comparative claim that Repeton shows "good performance compared to RAG-based methods" is unsupported by the reported evidence. Only two baselines are given: Moatless Tools + DeepSeekV3, which scores 0.00, and Swe-Fixer, which scores 24.67. Unless Swe-Fixer's text-retrieval module is the intended RAG baseline, no RAG-based method is directly evaluated; if it is the RAG baseline, then the RAG baseline outperforms Repeton by 13 percentage points. No error bars, repeated runs, or significance tests are reported, so the 11.67% score cannot be distinguished from noise relative to a single non-zero baseline. The claim should be scoped to "competitive with one open-weight baseline" or supported with direct RAG-based comparisons and variability estimates.
[Section 5] The fallback acceptance path undermines the central causal claim that the patch-and-test loop yields validated repairs. The paper states that when the agent "fails to reproduce the reported error," multiple valid patches can be rejected and "only the last patch is accepted by default." At this point the test-based validation guard is bypassed. The paper does not report, for the 35 resolved instances, how many were accepted via this fallback versus how many passed a genuine reproduction-and-validation cycle, nor whether fallback-accepted patches pass the official SWE-bench tests. Without this breakdown, the headline 11.67% could reflect test-generation failures rather than patch quality. Please report the number of fallback acceptances and the validation success rate among resolved instances.
[Section 5, Table 2] The failure taxonomy in Table 2 is ambiguous. The three categories "Resolved," "Unresolved," and "Empty Patch" sum to 300, but it is unclear whether "Unresolved" denotes instances where a non-empty patch was generated and tested but failed, and whether "Empty Patch" denotes instances where the agent returned no patch at all. Conflating these failure modes prevents the reader from assessing how often the Testing module actually produced a reproducing test and validated a patch. A clarified taxonomy with counts of generated patches, tested patches, and validated patches would directly address the load-bearing question raised by the fallback mechanism.

minor comments (5)

[Section 4.1] The text says "Swebench-lite includes 300 GitHub repositories," but SWE-bench Lite consists of 300 task instances drawn from a smaller set of repositories; this should be corrected to avoid overstating benchmark breadth.
[Figures 1 and 2] The acronyms IRV and IVR are used inconsistently: the text defines Iterative Repair and Validation (IRV), while Figure 1 and Figure 2's caption use "IVR." Please unify the terminology.
[Section 2] There is a typo in the sentence "Performance in the SWE benchmark reveals the limitations of mand LLMs" which should presumably read "many LLMs." Also, "Swebench" and "SWE-bench" are used inconsistently throughout the manuscript.
[Section 3.1] The testing flow is described in a way that is easy to misread: the Testing module's main goal is to produce a consistently failing test, yet the following sentence says that if the test passes the patch is finalized. Please clarify the control flow between test generation, test execution, and patch finalization.
[References] The Moatless Tools reference [1] lists only a GitHub repository without a version, commit, or retrieval date, which makes exact reproduction of the baseline difficult.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the evaluation is anchored to the external SWE-bench Lite benchmark, with no fitted parameters, no benchmark-specific tuning, and no load-bearing self-citation chain.

full rationale

Repeton's central claim is an empirical performance result on SWE-bench Lite (35/300 = 11.67%) and a qualitative claim about interpretability. The pipeline is compared against externally reported open-weight baselines (Moatless Tools + DeepSeekV3, Swe-Fixer); no parameter is fitted to the benchmark, no test-set information is used to tune the framework, and no prior result by the same authors is invoked as load-bearing support. The design choices (ReAct-style prompting, keyword search, patch-and-test loop) are grounded in external prior work [15, 5, 14] and are not defined in terms of the outcome they are meant to explain. The only noteworthy passage is the Section 5 admission that when the agent fails to reproduce the reported error, 'multiple valid patches can be rejected. Eventually, only the last patch is accepted by default.' This is a genuine limitation and a threat to the interpretation of the 11.67% score, but it is not circularity: it does not make the benchmark result equal to an assumption or fitted input; it describes a fallback path that weakens the causal link between test-driven validation and resolved issues. The paper self-reports this failure mode rather than hiding it, and the benchmark score remains externally anchored. I therefore find no derivation-equivalent step, no fitted-input-renamed-as-prediction, and no self-citation chain.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claims rest on empirical assumptions about LLM behavior and the benchmark, not on fitted parameters. No numeric parameters are fitted; the framework's success depends on the model's ability to generate reproducing tests, the keyword search to find relevant files, and test outcomes to indicate patch correctness.

assumptions (3)

domain assumption The LLM can generate a test that consistently fails on the buggy code and passes on the fixed code.
Section 3.1 says the Testing module's main goal is to produce a consistently failing test; Section 5 reports reproduction failures cause valid patches to be rejected, so this assumption is load-bearing but imperfect.
domain assumption Keyword-based file search over the project tree, without embeddings, identifies relevant files often enough for a successful fix.
Section 3.2 defines Stage 1 of ICSR; the 11.67% result depends on this search strategy, and Section 5 notes the agent sometimes fails to identify correct files.
domain assumption Passing the agent-generated test indicates the patch resolves the issue without breaking the project.
The IRV loop finalizes a patch only when the test passes; Section 5 explains that inability to reproduce the bug can lead to incorrect rejection or acceptance of patches.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Repeton: Structured Bug Repair with ReAct-Guided Patch-and-Test Cycles." pith.science (2026). https://pith.science/paper/7WLDPMBG

@misc{pith2026250608173,
  author       = {Pith},
  title        = {Pith review of: Repeton: Structured Bug Repair with ReAct-Guided Patch-and-Test Cycles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7WLDPMBG}},
  note         = {Machine review of arXiv:2506.08173}
}

read the original abstract

Large Language Models (LLMs) have shown strong capabilities in code generation and comprehension, yet their application to complex software engineering tasks often suffers from low precision and limited interpretability. We present Repeton, a fully open-source framework that leverages LLMs for precise and automated code manipulation in real-world Git repositories. Rather than generating holistic fixes, Repeton operates through a structured patch-and-test pipeline: it iteratively diagnoses issues, proposes code changes, and validates each patch through automated testing. This stepwise process is guided by lightweight heuristics and development tools, avoiding reliance on embedding-based retrieval systems. Evaluated on the SWE-bench Lite benchmark, our method shows good performance compared to RAG-based methods in both patch validity and interpretability. By decomposing software engineering tasks into modular, verifiable stages, Repeton provides a practical path toward scalable and transparent autonomous debugging.

Figures

Figures reproduced from arXiv: 2506.08173 by the authors.

**Figure 1.** Iterative Repair and Validation workflow. The flow combines code search and patch generation with error reproduction [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Iterative Code Search and Repair workflow, which includes many steps from locating to fixing errors, and example [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 6 canonical work pages

[1]

AorWall. 2024. Moatless Tools. GitHub repository. https://github.com/aorwall/ moatless-tools

work page 2024
[2]

Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2024. RepairA- gent: An Autonomous, LLM-Based Agent for Program Repair.arXiv preprint arXiv:2403.17134(2024)

arXiv 2024
[3]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique P. Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clem...

arXiv 2021
[4]

Claire Le Goues, ThanhVu Nguyen, Stephanie Forrest, and Westley Weimer. 2012. GenProg: A Generic Method for Automatic Software Repair.IEEE Transactions on Software Engineering38, 1 (2012), 54–72. doi:10.1109/TSE.2011.104

doi:10.1109/tse.2011.104 2012
[5]

Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?arXiv preprint arXiv:2310.06770(2024)

arXiv 2024
[6]

Xia, Longji Yang, Jen tse Huang, Zhouruixing Zhu, Ling- ming Zhang, and Michael R

Cheryl Lee, Chunqiu S. Xia, Longji Yang, Jen tse Huang, Zhouruixing Zhu, Ling- ming Zhang, and Michael R. Lyu. 2024. FixAgent: Hierarchical Multi-Agent Framework for Unified Software Debugging.arXiv preprint arXiv:2404.17153 (2024)

arXiv 2024
[7]

Zhengyao Liu, Yunlong Ma, Jingxuan Xu, Junchen Ai, Xiang Gao, Hailong Sun, and Abhik Roychoudhury. 2025. Agent That Debugs: Dynamic State-Guided Vulnerability Repair.arXiv preprint arXiv:2504.07634(2025)

arXiv 2025
[8]

Sergey Mechtaev, Jooyong Yi, and Abhik Roychoudhury. 2016. Angelix: Scalable Multiline Program Patch Synthesis via Symbolic Analysis. InProceedings of the 38th International Conference on Software Engineering (ICSE). ACM, 691–701

work page 2016

Show all 17 references

[9]

Anton Osika. 2023. GPT-Engineer: CLI Platform to Experiment with Code Gen- eration. GitHub repository. https://github.com/AntonOsika/gpt-engineer

2023
[10]

Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi Q. D. Bui, Junnan Li, and Steven C. H. Hoi. 2023. CodeT5+: Open Code Large Language Models for Code Understanding and Generation. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). ...

2023
[11]

Scott Wu. 2024. Introducing Devin, the First AI Software Engineer. Cognition.ai blog. https://cognition.ai/blog/introducing-devin

2024
[12]

Chengxing Xie, Bowen Li, Chang Gao, He Du, Wai Lam, Difan Zou, and Kai Chen. 2025. SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolution. InProceedings of the Deep Learning for Code (DL4C) Workshop at ICLR 2025

2025
[13]

Hui Yang, Sifu Yue, and Yunzhong He. 2023. Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions.arXiv preprint arXiv:2306.02224 (2023)

2023 arXiv
[14]

Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer Inter- faces Enable Automated Software Engineering.arXiv preprint arXiv:2405.15793 (2024)

2024 arXiv
[15]

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629 [cs.CL] https://arxiv.org/abs/2210.03629

2023 arXiv
[16]

Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury
[2024]

Received 8 June 2025

AutoCodeRover: Autonomous Program Improvement.arXiv preprint arXiv:2404.05427(2024). Received 8 June 2025

2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

AorWall. 2024. Moatless Tools. GitHub repository. https://github.com/aorwall/ moatless-tools

work page 2024

[2] [2]

Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2024. RepairA- gent: An Autonomous, LLM-Based Agent for Program Repair.arXiv preprint arXiv:2403.17134(2024)

arXiv 2024

[3] [3]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique P. Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clem...

arXiv 2021

[4] [4]

Claire Le Goues, ThanhVu Nguyen, Stephanie Forrest, and Westley Weimer. 2012. GenProg: A Generic Method for Automatic Software Repair.IEEE Transactions on Software Engineering38, 1 (2012), 54–72. doi:10.1109/TSE.2011.104

doi:10.1109/tse.2011.104 2012

[5] [5]

Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?arXiv preprint arXiv:2310.06770(2024)

arXiv 2024

[6] [6]

Xia, Longji Yang, Jen tse Huang, Zhouruixing Zhu, Ling- ming Zhang, and Michael R

Cheryl Lee, Chunqiu S. Xia, Longji Yang, Jen tse Huang, Zhouruixing Zhu, Ling- ming Zhang, and Michael R. Lyu. 2024. FixAgent: Hierarchical Multi-Agent Framework for Unified Software Debugging.arXiv preprint arXiv:2404.17153 (2024)

arXiv 2024

[7] [7]

Zhengyao Liu, Yunlong Ma, Jingxuan Xu, Junchen Ai, Xiang Gao, Hailong Sun, and Abhik Roychoudhury. 2025. Agent That Debugs: Dynamic State-Guided Vulnerability Repair.arXiv preprint arXiv:2504.07634(2025)

arXiv 2025

[8] [8]

Sergey Mechtaev, Jooyong Yi, and Abhik Roychoudhury. 2016. Angelix: Scalable Multiline Program Patch Synthesis via Symbolic Analysis. InProceedings of the 38th International Conference on Software Engineering (ICSE). ACM, 691–701

work page 2016