RepoLaunch: Automating Build and Management of Code Repositories across Languages and Platforms

Bowen Li; Chaoyun Zhang; Chengxing Xie; Dongmei Zhang; Elsie Nallipogu; Geng Zhang; Kenan Li; Liao Zhu; Linghao Zhang; Qingwei Lin

REVIEW 4 major objections 6 minor 2 cited by

RepoLaunch is an agent that automatically builds, tests, and maintains software repositories across arbitrary languages and operating systems.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-15 14:50 UTC pith:B7F5KVTY

load-bearing objection Real multi-language/Windows repo-build agent with open artifacts and useful failure analysis; the 18-point win and “any language” framing are softer than the abstract sells, but the systems contribution still holds. the 4 major comments →

arxiv 2603.05026 v2 pith:B7F5KVTY submitted 2026-03-05 cs.SE cs.LGcs.MA

RepoLaunch: Automating Build and Management of Code Repositories across Languages and Platforms

Kenan Li , Rongzhi Li , Linghao Zhang , Qirui Jin , Liao Zhu , Xiaosong Huang , Geng Zhang , Yikai Zhang

show 12 more authors

Shilin He Chengxing Xie Xin Zhang Zijian Jin Bowen Li Chaoyun Zhang Yu Kang Yufan Huang Elsie Nallipogu Saravan Rajmohan Qingwei Lin Dongmei Zhang

This is my paper

classification cs.SE cs.LGcs.MA

keywords repository build automationLLM agentssoftware engineering benchmarksmulti-languagecross-platformdependency resolutiontest status extractionSWE dataset creation

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Building real code repositories is still mostly manual: incomplete docs, shifting dependencies, and platform-specific toolchains force repeated trial and error. This paper claims that a multi-agent system can take only a base OS image and short language-specific build/test prompts and then resolve dependencies, compile the project, run regression tests, extract per-test outcomes, and emit minimal rebuild and test commands for later use. On a multi-language comparison set it reports roughly 78% build success and beats the prior Python/Linux-only system by 18 points. The same machinery powers a pipeline that turns GitHub issues into executable SWE benchmark tasks with human effort limited to task design. If the approach holds, scalable evaluation and training of coding agents no longer require hand-crafted sandboxes for every repository.

Core claim

RepoLaunch is presented as the first agent that, given only language-specific build/test guidance and a base OS image, can autonomously explore a repository, install dependencies, compile it, verify that most tests pass, and produce reusable rebuild commands, test commands, and a log parser that work across languages and both Linux and Windows. Empirically it reaches about 78% build success overall and 18 percentage points higher than the previous Python-only agent on the same Python set, while also enabling fully automated creation of multi-language and Windows SWE-bench-style tasks.

What carries the argument

A three-stage multi-agent workflow—Preparation (base-image and file selection), Build (Setup agent that installs and compiles plus Verify agent that checks test status), and Release (Organize agent that distills minimal rebuild/test commands and a parser)—that turns free exploration into durable sandbox images and reusable scripts.

Load-bearing premise

Success still depends on hand-written language-specific instruction prompts and curated base images, because current models lack enough framework knowledge on their own; if those prompts are missing or wrong, the agent loop cannot invent the missing expertise.

What would settle it

Remove or deliberately corrupt the language-specific instruction prompts for a held-out language or framework, re-run the same repositories, and check whether build success collapses to near the no-instruction baseline; a large drop would falsify the claim that agentic exploration alone generalizes to any language.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Executable sandboxes and per-test status extractors can be produced for large numbers of public repositories without manual Dockerfiles.
New multi-language and Windows SWE benchmarks can be generated with human effort restricted to choosing repositories, commits, and task design.
Coding agents and LLMs can be trained and evaluated on execution feedback from repositories outside the Python/Linux niche.
After code edits, minimal rebuild and re-test commands become available automatically, supporting long-running repository maintenance loops.
Failure modes of automated build/test (timeouts, missing test frameworks, dependency resolution) become measurable at scale and can guide future agent design.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Because the system already separates setup from issue resolution, future agent benchmarks can score environment construction and patch quality independently rather than conflating them.
The same pipeline could be pointed at private or enterprise monorepos if the base images and instruction prompts are extended to internal package managers and CI conventions.
Token cost of free multi-turn exploration is the practical bottleneck; hybrid systems that first try hard-coded parsers for common frameworks and only fall back to the agent would likely cut cost while preserving coverage.
Windows-specific success remaining competitive with Linux suggests platform gaps for coding agents are more about tooling support than inherent model inability.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Real multi-language/Windows repo-build agent with open artifacts and useful failure analysis; the 18-point win and “any language” framing are softer than the abstract sells, but the systems contribution still holds.

read the letter

The useful takeaway is simple: RepoLaunch is a working multi-agent pipeline that actually builds and tests non-Python repos on Linux and Windows, ships rebuild/test artifacts and parsers, and turns that into an automated SWE-task factory. That is the bottleneck people keep hitting, and the paper attacks it directly rather than waving at it.

What is new is not “agents can run bash.” Prior work (Repo2Run, the Python SWE-bench-Live path, SWE-rebench) already did Python/Linux. The advance is the Preparation/Build/Release split, Verify-agent checks against Setup hallucinations, distilled rebuild/test commands, generated log parsers, and explicit multi-language plus Windows support. Table 1’s staged retain rates, the failure taxonomy (Figure 3 / Table 5), and the MultiLang/Windows Live sets are the parts I would actually use. Open-source code, DockerHub images, and claimed external adoption are real credit, not decoration.

Soft spots, in proportion. The “any language / any platform” claim is bounded by hand-written language instructions and curated base images; the paper itself says those instructions move success rates because LLMs lack framework knowledge. That is a design choice, not a secret, but it should not be sold as pure agent discovery. The 18-point Python gain over repo2run sits on 33 selected SWE-bench commits and a majority-pass redefinition of the baseline; overall Table 4 is 75.63%, the intro says ~70%, the abstract says 78%. None of that collapses the result, but the headline numbers are noisier than they look. Step budgets, timeouts, disabled Release feedback, and proprietary LLMs also mean the rates are not a clean, re-runnable constant. Circular math is not the issue; sample size and criterion choices are.

Who this is for: people building coding-agent benchmarks, training loops, or sandbox farms. Not a theory paper. Math is light by design; citations cover the right prior systems. I would send it to peer review. Engage if you care about executable SWE data at scale; treat the abstract percentages as directional until the comparison set and success criteria are tightened.

Referee Report

4 major / 6 minor

Summary. The paper presents RepoLaunch, a multi-agent system (Preparation, Setup, Verify, Organize) that, given language-specific build/test instructions and a base OS image, installs dependencies, compiles repositories, extracts minimal rebuild/test commands, and generates test-log parsers across multiple languages and both Linux and Windows. The authors report roughly 70–78% build success depending on the evaluation set, claim an 18-point gain over the Python/Linux-only prior system repo2run on a curated comparison set (Table 4), and demonstrate an end-to-end pipeline that turns GitHub PRs into SWE-bench-style task instances with human effort limited to task design, releasing MultiLang and Windows subsets and open-sourcing the system.

Significance. If the empirical claims hold under clarified success criteria and broader sampling, RepoLaunch addresses a genuine bottleneck in agentic SWE: scalable construction of executable multi-language, multi-OS sandboxes and structured test status extraction. Strengths that should be credited include the open-source release, the multi-stage agent design with explicit rebuild/parser artifacts, the failure taxonomy (Figure 3, Table 5), head-to-head comparison against SWE-agent and repo2run, and demonstrated adoption for automated task generation and agent training. The Windows support and Win-agent evaluation fill a clear gap left by prior Linux-only SWE benchmarks. These contributions are practically useful even if the absolute success rates and “any language/any platform” framing need tighter qualification.

major comments (4)

[§5.3, Table 4, Appendix B.1] Abstract and §5.3 / Table 4: The load-bearing claim that RepoLaunch “outperforms the Python/Linux-only prior system by 18%” (78.79% vs repo2run 60.61% on 33 Python instances) rests on a small, non-random set of three temporally distant commits per repository from SWE-bench-Verified/Multilingual (Appendix B.1). Temporal extremes can systematically alter dependency drift relative to the Live crawl in Table 1. The paper should either enlarge and randomize the comparison set, report confidence intervals / per-repo variance, or clearly demote the 18-point figure from a headline result to a limited head-to-head on this fixed set.
[§5.3, Abstract, §1] §5.3: Success for repo2run is redefined as “more than half of the tests pass” because its original importability criterion was “too lax,” while RepoLaunch’s own success also hinges on a majority-pass threshold plus Verify-Agent checks, and Release-stage feedback to Setup was disabled for efficiency (§3.4). Without a side-by-side under a single pre-registered criterion (e.g., full FAIL_TO_PASS extraction as used for SWE-agent’s Release&Validation column), the 18-point gap and the abstract’s “78%” figure are not cleanly comparable. Please report both original and majority-pass criteria for repo2run, and align abstract/intro/Table 4 numbers (abstract 78%, intro ~70%, Table 4 overall Build 75.63%).
[§3.2, Title, Abstract, §1] §3.2 and the “any language and any platform” framing (title, abstract, contribution 1): The paper states that language-specific instructions “do affect the success rate, as current LLMs lack sufficient knowledge of the diverse frameworks,” while base images mainly reduce steps/size. Generalization is therefore bounded by hand-authored utilities rather than pure agent discovery. This is an important and honest limitation; the central claim should be restated as “given language-specific prompts and base images for the supported languages/OSes,” and the manuscript should specify what is required to add a new language (prompt content, image construction, validation protocol) so the scope is falsifiable rather than absolute.
[Table 1, Table 4, Figure 3, Table 5] Table 1 vs Table 4: Build success on the Live MultiLang/Windows crawl (60–77% by language; Windows 64.5%) is materially lower than the curated SWE-bench comparison set (overall 75.63%, Python 78.79%). The paper attributes low Validation retain rates largely to PRs lacking FAIL_TO_PASS tests, which is plausible, but does not quantify how much of the Build gap is due to repository noise, timeouts (dominant in Figure 3 / Table 5), or agent limits. A short analysis reconciling the two evaluation regimes is needed so readers do not over-generalize the higher Table 4 rates to arbitrary GitHub repositories.

minor comments (6)

[Figure 2, §4.1, Table 1] Throughout: “RepoLauch” appears repeatedly (e.g., Figure 2 caption, §4.1 headings, Table 1 caption) instead of “RepoLaunch.”
[§5.1] §5.1 Build failure list item 5: “TThe build or test” has a doubled capital T.
[§3.1] §3.1 formalization uses placeholder glyphs for sets/functions that render poorly in the provided text; ensure symbols for repository, state space, transition, and verification are typeset consistently in the camera-ready version.
[Table 2] Table 2: Python column is only filled for SWE-agent rows; clarify whether other agent–model pairs were not run on Python or were omitted for space, and how the Linux average is weighted when Python is excluded.
[Code Availability, Data Availability] Code/data availability URLs are redacted as boxes in the manuscript text; ensure final links and DockerHub namespace are complete and stable.
[§2] Related work could more explicitly position against SWE-bench-Live’s prior use of the authors’ Python-only RepoLaunch version so novelty of the multi-language/Windows extension is crisp.

Circularity Check

1 steps flagged

Empirical systems paper with no derivation-by-construction; only a minor non-load-bearing self-reference to prior SWE-bench-Live Python stats that used an earlier RepoLaunch version.

specific steps

self citation load bearing [Table 1 caption; Section 2 (Related Works); Section 5.1]
"Python* statistics were picked from previous experiments contributed to the SWE-bench-Live Python subset (Zhang et al., 2025) of July and August 2025, where test log parsing adopts the simple pytest format parser of SWE-bench ... so it does not have the Release step. ... SWE-bench-Live adopts our initial version of RepoLaunch Agent designed for Python only"

Table 1’s Python retain rates are imported from prior work by overlapping authors that already used an earlier RepoLaunch version, so that row is not an independent external measurement of the present multi-agent system. This is only a minor reporting self-reference: the load-bearing 18% Python gain and multi-lang/Windows results rest on Table 4 and new Live crawls against independent baselines, not on this Python* row.

full rationale

RepoLaunch’s central claims—multi-language/multi-OS build success (~70–78%), an 18-point gain over repo2run on Python, and automated SWE task creation—are empirical measurements on external GitHub repositories and independent baselines (repo2run, SWE-agent, SWE-bench-Verified/Multilingual), not quantities forced by definition, fitted parameters renamed as predictions, or uniqueness theorems. Success criteria (majority tests pass, Verify Agent checks, FAIL_TO_PASS extraction) are operational evaluation rules applied to observed command/test outcomes; they do not make the reported rates true by construction. Language-specific instruction prompts and curated base images are acknowledged human utilities that bound the “any language” claim (Section 3.2), but that is a scope/assumption limitation, not circular derivation. The only mild self-reference is Table 1’s Python* retain rates taken from prior SWE-bench-Live work that used an earlier Python-only RepoLaunch; that row is not the load-bearing comparison (Table 4 vs repo2run/SWE-agent on independently selected instances). No self-definitional loop, fitted-input-as-prediction, uniqueness import, ansatz smuggling, or renaming of a known result is present. Score 1 reflects only that minor self-citation, not circularity of the main result.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 3 invented entities

The central empirical claim depends less on free mathematical constants than on engineering choices: hand-written language instructions, curated base images, step/timeout budgets, and a majority-tests-pass success definition. The multi-agent roles are invented workflow entities without independent evidence outside this system. Domain assumptions include that public GitHub repos with stars and language dominance are representative, and that FAIL_TO_PASS extraction after PR patches is a valid task filter.

free parameters (4)

Setup agent step budget = 60 Linux / 90 Windows
Capped at 60 steps on Linux and 90 on Windows; directly affects whether hard builds succeed or time out.
Verify / Organize stage step budgets = 20 steps each
Verify capped at 20 steps; rebuild/test/parser stages each limited to 20 steps; Release failures not fed back to Setup in experiments.
Bash command timeout = 30 minutes
30-minute per-command timeout; timeouts are a major reported failure mode, so this threshold shapes measured success.
Majority-tests-pass success threshold = >50% tests pass
Build is accepted when a majority of test cases pass; this is an author-chosen criterion used both for handoff and for fair comparison with repo2run.

axioms (4)

ad hoc to paper Hand-authored language-specific build/test instructions plus curated base images are sufficient utilities for cross-language generalization.
Section 3.2 states instructions affect success rate because LLMs lack framework knowledge; these utilities are not derived, they are provided.
domain assumption A repository is successfully built if regression tests can run and a majority of cases pass, without requiring full production readiness or zero failures.
Used in Setup/Verify handoff and in rewriting repo2run’s success criterion (Section 5.3).
domain assumption GitHub issues/PRs with stars>1k and language proportion>60%, after LLM filtering, yield valid SWE task instances when FAIL_TO_PASS tests exist.
Appendix D pipeline assumptions for SWE-bench-Live/MultiLang & Windows creation.
domain assumption LLM agents with Bash and WebSearch tools can explore noisy public repositories case-by-case better than fixed rule-based build scripts.
Motivating claim in Related Works and Introduction; underpins why an agentic method is necessary.

invented entities (3)

RepoLaunch multi-agent workflow (Preparation, Setup, Verify, Organize agents) no independent evidence
purpose: Decompose repository preparation, build, verification, and release-artifact extraction into specialized agent stages.
The staged agent roles and their handoff criteria are system design choices introduced by the paper; independent evidence is only the reported empirical success of this implementation.
Generated test-log parser and minimal rebuild/test command artifacts no independent evidence
purpose: Make a built sandbox reusable for repeated evaluation after code patches without redoing full exploratory setup.
These artifacts are outputs of the Organize agent and are central to the automated SWE dataset pipeline; their correctness is validated inside the same system.
Win-agent no independent evidence
purpose: Provide a Windows-container coding agent with SWE-agent/OpenHands-like tools for Windows benchmark evaluation.
Introduced because existing agents lack Windows container support; results in Table 3 depend on this custom agent.

pith-pipeline@v1.1.0-grok45 · 21278 in / 3852 out tokens · 43021 ms · 2026-07-15T14:50:43.805458+00:00 · methodology

0 comments

read the original abstract

Language model (LM) agents have driven substantial progress in automated software engineering (SWE), yet building and testing software repositories at scale remains a largely manual and labor-intensive bottleneck. In this work, we introduce RepoLaunch, a novel agentic framework that automatically resolves dependencies, compiles source code, and extracts test results across diverse programming languages and operating systems. RepoLaunch achieves a 78% build success rate, outperforming the Python/Linux-only prior system by 18%. To demonstrate its application, we further present a fully automated pipeline for SWE dataset creation driven by RepoLaunch, which only requires human input at the task-design stage. RepoLaunch is open-sourced, and its automated task-generation pipeline has already been adopted by several recent works on agentic benchmarking and training.

discussion (0)

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Dockerless: Environment-Free Program Verifier for Coding Agents
cs.SE 2026-06 unverdicted novelty 7.0

Dockerless uses agentic repository exploration to verify patches without execution, enabling SFT and RL training of coding agents that reach 62.0/50.0/35.2% resolve rates on SWE-bench Verified/Multilingual/Pro while m...
EvoArena: Tracking Memory Evolution for Robust LLM Agents in Dynamic Environments
cs.CL 2026-06 unverdicted novelty 6.0

Introduces EvoArena benchmark for dynamic LLM agent environments and EvoMem memory paradigm that improves accuracy by 1.5% on EvoArena and gains on GAIA and LoCoMo.

Reference graph

Works this paper leans on

17 extracted references · 1 linked inside Pith · cited by 2 Pith papers

[1]

Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, An- drei Andriushchenko, Maria Trofimova, Daria Litv- intseva, and Boris Yangel

Swe-mera: A dynamic benchmark for agen- ticly evaluating large language models on software engineering tasks.�� . Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, An- drei Andriushchenko, Maria Trofimova, Daria Litv- intseva, and Boris Yangel. 2025. Swe-rebench: An automated pipeline f...

2025
[2]

Eric Horton and Chris Parnin

Training long-context, multi-turn software en- gineering agents with reinforcement learning.�� . Eric Horton and Chris Parnin. 2019. Dockerizeme: Automatic inference of environment dependencies for python code snippets. In�� , pages 328–338. IEEE. Ruida Hu, ...

2019
[3]

FAIL_TO_PASS

Swe-smith: Scaling data for software engineer- ing agents.��, arXiv:2504.21798. Hongjie Ye, Jiahong Zhou, Wei Chen, Jiaxin Zhu, Guo- quan Wu, and Jun Wei. 2021. Dockergen: A knowl- edge graph based approach for software containeriza- tion. In�� , pages 986–991. IEEE. Karina...

Pith/arXiv arXiv 2021
[4]

Then, filter these repos with the proportion of any of the six languages > 60%

Crawl.Filter GitHub public repos with stars > 1k. Then, filter these repos with the proportion of any of the six languages > 60%. Crawl merged pull requests (PRs) that are related to some issues during a specific time period
[5]

Windows-specific problems should be built sandbox images on Windows images instead of the Linux images adopted by previ- ous SWE benchmarks

Split.Use a Reasoning LLM (GPT-5- Thinking-Medium here) to categorize issue de- scriptions as either general/Linux or Windows- specific. Windows-specific problems should be built sandbox images on Windows images instead of the Linux images adopted by previ- ous SWE benchmarks
[6]

Launch.Use RepoLaunch to build the sand- box images, extract the rebuild and test com- mands, and run the test log parser for each task instance before the PR was merged
[7]

FAIL_TO_PASS

Validation.Apply the test patch of the PR and run rebuild commands, test commands and test log parser. Get the test statusbefore the solution patch. Apply the testandsolution patch of the PR and run again to get the test statusafterthe solution patch. Compare the test status before and after the solution patch. ExtractPASS_TO_PASStest cases that pass both...
[8]

In our experiment we filter out these 2 cases out with a reasoning LLM (GPT-5- Thinking-Medium)

Verification.An additional step is to ver- ify whether the test patch contains require- ments that are not specified in the issue de- scription, or the issue descritpion directly con- tains ground truths either in natural language or code. In our experiment we filter out these 2 cases out with a reasoning LLM (GPT-5- Thinking-Medium). E Failure Pattern of...
[9]

Reproduction Failure.The reproduction test file written by agent runs successfully but can- not reproduce the issue
[10]

Execution Failure.The agent even does not know how to run the code repository, so it fails to run the written reproduction test file
[11]

File Localization Failure.The agent did not view all the files that need to be edited in ground truth patch
[12]

5.Overly Specific Implementation

Line Localization Failure.The agent failed to edit correct locations that the ground truth patch edits. 5.Overly Specific Implementation
[13]

Normal Functionality Corrupted.The agent’s edits corrupted originally normal func- tionalities
[14]

Incorrect Solutionto the problem statement
[15]

9.Submit Prematurely

Failed to Recover from Edit.The agent fell into edit-error-edit-error loop. 9.Submit Prematurely
[16]

Key commands Timeout.Running impor- tant shell commands like testing or modifying dependencies went timeout so verification of edits is not possible
[17]

We then use GPT-5.2-Thinking-Medium to clas- sify all the failed trajectories on SWE-bench- 15 Live/MultiLang and Windows

Max Step Reached, before completing the task. We then use GPT-5.2-Thinking-Medium to clas- sify all the failed trajectories on SWE-bench- 15 Live/MultiLang and Windows. Result is shown in Figure 4. Failure pattern on Linux illustrates that: (1) The reproduction of the problem and the vali- dation of the fix mostly fails to work because the agents do not k...

[1] [1]

Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, An- drei Andriushchenko, Maria Trofimova, Daria Litv- intseva, and Boris Yangel

Swe-mera: A dynamic benchmark for agen- ticly evaluating large language models on software engineering tasks.�� . Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, An- drei Andriushchenko, Maria Trofimova, Daria Litv- intseva, and Boris Yangel. 2025. Swe-rebench: An automated pipeline f...

2025

[2] [2]

Eric Horton and Chris Parnin

Training long-context, multi-turn software en- gineering agents with reinforcement learning.�� . Eric Horton and Chris Parnin. 2019. Dockerizeme: Automatic inference of environment dependencies for python code snippets. In�� , pages 328–338. IEEE. Ruida Hu, ...

2019

[3] [3]

FAIL_TO_PASS

Swe-smith: Scaling data for software engineer- ing agents.��, arXiv:2504.21798. Hongjie Ye, Jiahong Zhou, Wei Chen, Jiaxin Zhu, Guo- quan Wu, and Jun Wei. 2021. Dockergen: A knowl- edge graph based approach for software containeriza- tion. In�� , pages 986–991. IEEE. Karina...

Pith/arXiv arXiv 2021

[4] [4]

Then, filter these repos with the proportion of any of the six languages > 60%

Crawl.Filter GitHub public repos with stars > 1k. Then, filter these repos with the proportion of any of the six languages > 60%. Crawl merged pull requests (PRs) that are related to some issues during a specific time period

[5] [5]

Windows-specific problems should be built sandbox images on Windows images instead of the Linux images adopted by previ- ous SWE benchmarks

Split.Use a Reasoning LLM (GPT-5- Thinking-Medium here) to categorize issue de- scriptions as either general/Linux or Windows- specific. Windows-specific problems should be built sandbox images on Windows images instead of the Linux images adopted by previ- ous SWE benchmarks

[6] [6]

Launch.Use RepoLaunch to build the sand- box images, extract the rebuild and test com- mands, and run the test log parser for each task instance before the PR was merged

[7] [7]

FAIL_TO_PASS

Validation.Apply the test patch of the PR and run rebuild commands, test commands and test log parser. Get the test statusbefore the solution patch. Apply the testandsolution patch of the PR and run again to get the test statusafterthe solution patch. Compare the test status before and after the solution patch. ExtractPASS_TO_PASStest cases that pass both...

[8] [8]

In our experiment we filter out these 2 cases out with a reasoning LLM (GPT-5- Thinking-Medium)

Verification.An additional step is to ver- ify whether the test patch contains require- ments that are not specified in the issue de- scription, or the issue descritpion directly con- tains ground truths either in natural language or code. In our experiment we filter out these 2 cases out with a reasoning LLM (GPT-5- Thinking-Medium). E Failure Pattern of...

[9] [9]

Reproduction Failure.The reproduction test file written by agent runs successfully but can- not reproduce the issue

[10] [10]

Execution Failure.The agent even does not know how to run the code repository, so it fails to run the written reproduction test file

[11] [11]

File Localization Failure.The agent did not view all the files that need to be edited in ground truth patch

[12] [12]

5.Overly Specific Implementation

Line Localization Failure.The agent failed to edit correct locations that the ground truth patch edits. 5.Overly Specific Implementation

[13] [13]

Normal Functionality Corrupted.The agent’s edits corrupted originally normal func- tionalities

[14] [14]

Incorrect Solutionto the problem statement

[15] [15]

9.Submit Prematurely

Failed to Recover from Edit.The agent fell into edit-error-edit-error loop. 9.Submit Prematurely

[16] [16]

Key commands Timeout.Running impor- tant shell commands like testing or modifying dependencies went timeout so verification of edits is not possible

[17] [17]

We then use GPT-5.2-Thinking-Medium to clas- sify all the failed trajectories on SWE-bench- 15 Live/MultiLang and Windows

Max Step Reached, before completing the task. We then use GPT-5.2-Thinking-Medium to clas- sify all the failed trajectories on SWE-bench- 15 Live/MultiLang and Windows. Result is shown in Figure 4. Failure pattern on Linux illustrates that: (1) The reproduction of the problem and the vali- dation of the fix mostly fails to work because the agents do not k...