Pith. sign in

REVIEW 4 major objections 5 minor 67 references

OmniGIRL: A Multilingual and Multimodal Benchmark for GitHub Issue Resolution

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The OmniGIRL benchmark shows that current LLMs resolve only 8.6% of real multilingual GitHub issues.

desk verdict A useful SWE-bench extension to multiple languages and a small image subset, but the current draft has internal numeric inconsistencies and an over-stated image-subset rate that need fixing before the numbers are citable. read the letter →

arxiv 2505.04606 v1 pith:AIMOYO2U submitted 2025-05-07 cs.SE

Lianghong Guo , Wei Tao , Runhan Jiang , Yanlin Wang , Jiachi Chen , Xilin Liu , Yuchi Ma , Mingzhi Mao
show 2 more authors
Hongyu Zhang Zibin Zheng
This is my paper · ORCID
classification cs.SE
keywords GitHubissueresolutionmultilingualbenchmarkmultimodallargelanguagemodelsexecution-basedverificationcross-filecodechangesimageunderstandingsoftwareengineeringagents
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces OmniGIRL, a benchmark for automatic GitHub issue resolution built from 959 real resolved issues across four programming languages, fifteen repositories, and eight domains. The authors want to show that current large language models are far weaker at this task than single-language, text-only benchmarks suggest. On the full benchmark the strongest configuration, GPT-4o with an adapted workflow called Agentless-X, resolves 8.6% of issues. When an issue description includes an image that is essential to the fix, the best reported resolve rate is 10.5%, achieved by Claude-3.5-Sonnet with oracle file retrieval and image-augmented text over three repeated runs. The paper also isolates two recurring failure modes: models that do not emit parseable localization output, and models that edit only one file when the gold fix requires changes across several files.

What carries the argument

The central object is the OmniGIRL task instance: a real merged pull request paired with its linked issue text, optional images and website links, a base commit, a gold patch, and a set of fail-to-pass tests that must turn from failing to passing after the fix is applied. The construction pipeline filters merged pull requests by issue linkage and test-file changes, then verifies each instance in a Docker environment built from the base commit, retaining only instances with at least one fail-to-pass test and manually checking that those tests match the pull request's test patch. The evaluation machinery compares three baselines (oracle retrieval, a hierarchical localization-then-repair workflow, and an agentic search-and-repair loop) on each instance. Its work is to convert 'issue resolved' into an executable, reproducible pass/fail check, and the paper's numbers inherit all their meaning from that conversion.

What would settle it

Run each designated fail-to-pass test suite multiple times on the untouched base-commit environment from the released data. If any of those tests ever passes before the gold patch is applied, or fails after it, that instance does not cleanly encode the issue and the affected resolve rates are over- or understated.

Watch

Extended reading notes

Core claim

OmniGIRL's central claim is that current LLM-based issue resolution degrades sharply as soon as evaluation leaves the single-language, text-only setting: on 959 validated instances from Python, JavaScript, TypeScript, and Java repositories, the best tested model resolves 8.6% of issues, and on the 19 instances whose images are essential, the best resolve rate is 10.5%. The paper argues this is not simply a retrieval problem, because giving models the exact files to edit (oracle retrieval) improves some results, and because rewriting image content into text raises Claude-3.5-Sonnet's image-subset rate from 3.5% to 10.5%. It attributes the failures to concrete causes: parse-format errors in the localization stage, which a one-line prompt constraint fixes for Claude-3.5-Sonnet (1.9% to 7.4%), and a systematic single-file editing bias on fixes that require multi-file changes. The implied conclusion is that multilingual and multimodal issue resolution remains largely unsolved and needs new mechanisms rather than prompt tweaks alone.

Load-bearing premise

The load-bearing premise is that the fail-to-pass tests extracted from each merged pull request actually fail before the fix, pass after it, and fully capture the reported issue; if any test is flaky, environment-dependent, or unrelated to the issue text, every resolve rate in the benchmark is unreliable.

Editorial extensions

If this is right

  • If the 8.6% full-benchmark rate holds, claims of software-engineering agent competence should be scoped to single-language, text-only benchmarks until multilingual evidence is available.
  • Multimodal issue resolution is a distinct bottleneck; the jump from 3.5% to 10.5% on the image subset with image-augmented text suggests converting visual content into text is a cheap and effective first step.
  • Prompt robustness can move a model's resolve rate by a factor of about four: adding an explicit format constraint raised Claude-3.5-Sonnet from 1.9% to 7.4%.
  • Cross-file issue resolution is a separate weakness; even with oracle file retrieval, models modify only a single file in roughly 73% to 86% of cross-file cases.
  • Syntax-tree-based localization favors languages whose structure is class- and method-centric (Python, Java) and underperforms on JavaScript and TypeScript, where interfaces and anonymous functions carry much of the structure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct validity check a reader could run on the released data is to execute each fail-to-pass suite several times on the untouched base commit; any test that sometimes passes before the fix would inflate every reported resolve rate.
  • The visual subset is only 19 instances (57 runs per setting), so the 10.5% figure is directionally suggestive but not a precise capability estimate.
  • Because the benchmark annotates website links but does not evaluate them, a natural next experiment is to give a repair agent browsing access to those links and measure the gain over text-only and image-augmented inputs.
  • The same image-augmented-text rewrite could be transferred to other multimodal code-repair benchmarks to test whether the gain is specific to this dataset or a general property of current vision-language models.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces OmniGIRL, a GitHub issue-resolution benchmark of 959 instances drawn from 15 repositories across Python, JavaScript, TypeScript, and Java, with a manually filtered 19-instance subset containing images that are essential for resolving the issue. The construction pipeline follows SWE-bench: merged pull requests are filtered by issue references and changed test files, task instances are assembled with base commits, test patches, gold patches, and FAIL2PASS/PASS2PASS annotations, and Docker-based execution is used to verify environments and filter instances. The paper evaluates GPT-4o, Claude-3.5-Sonnet, and DeepSeek-V2.5 using oracle retrieval, Agentless-X, and AutoCodeRover-X, reporting a best resolve rate of 8.6% (GPT-4o with Agentless-X) on the full benchmark and 10.5% (Claude-3.5-Sonnet with oracle retrieval and image-augmented text) on the 19-image subset. It also presents failure analyses of parsing errors in Agentless-X and of poor performance on multi-file issues.

Significance. If the benchmark's execution-based verification is sound, OmniGIRL is a useful contribution: it extends issue-resolution evaluation beyond Python-only SWE-bench, covers diverse domains and languages, and introduces a small but genuinely multimodal subset. The evaluation is carried out with reasonable, established baseline methods, and the code and data are open-sourced. The headline result that current LLMs resolve only 8.6% of issues is plausible and consistent with the broader SWE-bench literature. The failure analysis on prompt-format sensitivity and multi-file edits is informative. The main significance hinges, however, on demonstrating that the FAIL2PASS tests and Docker environments actually encode each issue, and on clarifying the statistical basis of the multimodal results; these points currently receive insufficient evidence.

major comments (4)
  1. [§3.4.1–3.4.3] The validity of every reported resolve rate depends on the assertion that FAIL2PASS tests extracted from merged PRs fail before the gold patch and pass after it for issue-related reasons, and that Docker environments faithfully reproduce the pre-fix state. The paper does not report the number of PRs collected, the number removed at each attribute filter, the environment build success rate, the number of instances discarded because the gold patch did not produce a FAIL2PASS test, or any measure of inter-annotator agreement for the manual FAIL2PASS check in §3.4.3. The paper's own §6 acknowledges that the path-keyword test-file detection 'could miss files' and that filtering 'may filter out some instances,' but it does not quantify how often the retained tests could be wrong. I ask the authors to provide a full filtering funnel and a human-validated subset (analogous to SWE-bench Verified) or an independent verification study on a random sample, since without this the 8.6% headline is not yet trustworthy.
  2. [§5.3, Table 9] The image-subset results report percentages over 57 runs for 19 unique instances (three repetitions each), so the 10.5% best result corresponds to only 6/57 runs and may represent as few as two distinct resolved issues (if all three runs of two instances resolved). With 19 instances, each unique resolved instance changes the rate by approximately 5.3 percentage points, making the multimodal claim extremely sensitive to a handful of successes. Please report the number of unique resolved instances and their instance IDs for each setting, and discuss the per-instance run-to-run agreement.
  3. [§5.4 and Table 10] There is an internal inconsistency in the central failure-analysis statistic: the text of §5.4 states Claude-3.5-Sonnet's parsing success rate is 11.8%, while Table 10 reports 10.0% (96/959). These cannot both be correct, and the discrepancy directly affects the claimed cause of the 1.9% resolve rate. Please reconcile the numbers and verify the corresponding apply rates (52.2% in Table 10 vs. 52.0% in Table 8 for DeepSeek-V2.5 Agentless-X).
  4. [Table 8] Table 8 contains internally inconsistent percentages that must be corrected: for Claude-3.5-Sonnet with Agentless-X on Python, the Apply Rate is shown as '1.6% (55/374)', but 55/374 is 14.7%, not 1.6%; the Java row also contains a typo ('4/105|'). Because apply rate is one of the three evaluation metrics, these errors need to be fixed and all percentages recomputed from the raw counts before publication.
minor comments (5)
  1. [§4.2.3] The modality distribution states that 4.1% (39/959) of instances contain image information, but Table 2 shows that after filtering only 19 instances remain with images judged necessary. Please clarify whether the 4.1% figure refers to the pre-filtering count and report the post-filtering distribution (19/959 ≈ 2.0%) consistently.
  2. [Throughout] There are several typos and misspellings: 'intergrated' in §5.1.3, 'AutoCoderOver' in the §5.2 text, 'Statitics modeling' in Table 6, 'preformance' in the RQ1 summary, and the duplicated '(d)' label in Figure 2. These should be cleaned up.
  3. [Table 9] The abbreviation 'IAG-Text' is defined in the table caption as 'Image-augmented Text', but the caption for Table 9 says 'Image-augmented Text' is shortened to 'IAG-Text' while the table header uses 'IAG-Text'; please use the abbreviation consistently and define it explicitly the first time it appears in the text.
  4. [§5.1.2] The oracle retrieval baseline is described as providing the exact files from the gold patch, but the paper does not specify whether the oracle baseline also receives the exact function or line locations; please clarify the input granularity, since it affects the interpretation of the 7.8% result.
  5. [§2.3 and Table 1] Table 1 lists SWE-bench-java as having 91 instances, while Section 2.3 also says 91; however, the abstract-level related-work text in Section 1 states 93 instances. Please ensure the counts are consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: OmniGIRL is an external evaluation benchmark whose resolve rates are measured against independent, merged-PR-derived tests, not derived from its own construction choices.

full rationale

OmniGIRL is an evaluation benchmark: the headline measurements (e.g., 8.6% resolve rate for GPT-4o with Agentless-X) are obtained by applying LLM-generated patches to Docker environments and running FAIL2PASS tests extracted from merged pull requests. These test cases and gold patches are external to the paper's own claims and are not fitted parameters, so the resolve rates are not defined in terms of the benchmark's construction choices. The construction pipeline follows SWE-bench: attribute filtering, environment verification, and execution-based filtering select instances with observable FAIL2PASS tests; no equation or fitted quantity is later renamed as a prediction. The oracle-retrieval baseline is explicitly described as an 'ideal-condition evaluation' that provides gold-patch file locations, so its use is transparent rather than a disguised prediction. The only self-citations (e.g., MAGIS in Section 2.2, cited as prior work by overlapping authors) are background mentions in related work and are not load-bearing for the benchmark's validity or for any measured result. Section 6 states limitations such as potential over-filtering and test-file detection misses; these are threats to benchmark validity and completeness, not circular steps, because they do not make the evaluation target equivalent to the construction inputs. The image-subset result (10.5% as 6/57 runs rather than unique resolved issues out of 19) is a reporting and interpretability concern, but it is not a self-referential derivation. Overall, the paper's central empirical claims are externally testable and do not reduce to their own inputs.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No free parameters are fitted; the benchmark introduces no new theoretical entities. The only hand-set choices are the repository selection rules and filtering thresholds, which are not numerical parameters in the claim. The assumptions above are the load-bearing premises.

assumptions (5)
  • domain assumption Merged pull requests that reference an issue and change a test file provide valid issue-solution ground truth.
    Used in Section 3.2.2 attribute-based filtering; inherited from SWE-bench; if PR/test linkage is wrong, the benchmark labels are wrong.
  • domain assumption Test files are identifiable by path keywords such as 'test' or by Java class naming conventions.
    Used in Sections 3.2.2 and 3.3 ('Test Patch'); the authors themselves note in Section 6 that unconventional test file names may be missed.
  • domain assumption Docker environments built from the base commit and documented setup commands reproduce the pre-fix failing state of each repository.
    Section 3.4.1 Environment Construction; if environments do not reproduce failures, FAIL2PASS verification is meaningless.
  • domain assumption The manual image filtering in Section 3.5 correctly identifies which images are crucial for resolving an issue.
    Only 19 of 39 image-bearing instances are kept; this human judgment determines the scope of the multimodal claim.
  • domain assumption FAIL2PASS test cases are a valid proxy for whether an issue is resolved.
    Section 3.4.3 Execution-Based Filtering; follows SWE-bench, but tests may be too weak or too strong relative to the reported issue.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OmniGIRL: A Multilingual and Multimodal Benchmark for GitHub Issue Resolution." pith.science (2026). https://pith.science/paper/AIMOYO2U

@misc{pith2026250504606,
  author       = {Pith},
  title        = {Pith review of: OmniGIRL: A Multilingual and Multimodal Benchmark for GitHub Issue Resolution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AIMOYO2U}},
  note         = {Machine review of arXiv:2505.04606}
}
read the original abstract

The GitHub issue resolution task aims to resolve issues reported in repositories automatically. With advances in large language models (LLMs), this task has gained increasing attention, and several benchmarks are proposed to evaluate the issue resolution ability of LLMs. However, existing benchmarks have three main limitations. First, current benchmarks focus on a single programming language, limiting the evaluation of issues from repositories across different languages. Second, they usually cover a narrow range of domains, which may fail to represent the diversity of real-world issues. Third, existing benchmarks rely solely on textual information in issue descriptions, overlooking multimodal information such as images in issues. In this paper, we propose OmniGIRL, a GitHub Issue ResoLution benchmark that is multilingual, multimodal, and multi-domain. OmniGIRL includes 959 task instances, which are collected from repositories across four programming languages (i.e., Python, JavaScript, TypeScript, and Java) and eight different domains. Our evaluation shows that current LLMs show limited performances on OmniGIRL. Notably, the best-performing model, GPT-4o, resolves only 8.6% of the issues. Besides, we find that current LLMs struggle to resolve issues requiring understanding images. The best performance is achieved by Claude-3.5-Sonnet, which resolves only 10.5% of the issues with image information. Finally, we analyze the reasons behind current LLMs' failure on OmniGIRL, providing insights for future improvements.

Figures

Figures reproduced from arXiv: 2505.04606 by the authors.

Figure 1
Figure 1. Overview of the evaluation pipeline of GitHub issue resolution. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Overview of benchmark construction. 3.1.2 Repository Selection. To ensure the popularity of the target repositories, we use the download counts from language-specific package management tools. For Python, we select repositories based on pip download counts. For Java, we use Maven, and for JavaScript and TypeScript, we rely on npm. For each language, we select the top 200 repositories with the highest download counts… view at source ↗
Figure 3
Figure 3. An example of task instance tailwindlabs__tailwindcss-10212. 3.3 Task Instance Construction After collecting raw data of pull requests, we build task instances using GitHub Developer API [6] to obtain key information from each pull request. An example of a task instance is shown in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Examples of images contained in issue descriptions. [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: An example of website links within an issue. [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Responses of different LLMs in the localization stage of Agentless-X. [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 30 canonical work pages

  1. [1]

    [n. d.]. JParser: JSON Stream Parser for Java. https://github.com/javadev/jparser. Accessed: 2023-10-26

    work page 2023

  2. [2]

    [n. d.]. npm - Node Package Manager. https://www.npmjs.com/. Accessed: 2023-10-25

    work page 2023

  3. [3]

    [n. d.]. pip - The Python Package Installer. https://pip.pypa.io/en/stable/. Accessed: 2023-10-25

    work page 2023

  4. [4]

    2024. Babel. https://babeljs.io/

    work page 2024

  5. [5]

    2024. docker. https://www.docker.com/

    work page 2024

  6. [6]

    github-rest-api

    2024. github-rest-api. https://docs.github.com/en/rest

    work page 2024

  7. [7]

    2024. maven. https://maven.apache.org/

    work page 2024

  8. [8]

    tree-sitter

    2024. tree-sitter. https://tree-sitter.github.io/

    work page 2024

Show all 67 references
  1. [9]

    Wasi Uddin Ahmad, Md Golam Rahman Tushar, Saikat Chakraborty, and Kai-Wei Chang. 2021. Avatar: A parallel corpus for java-python program translation. arXiv preprint arXiv:2108.11590 (2021)

    2021 arXiv

  2. [10]

    Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. 2024. SWE- Bench+: Enhanced Coding Benchmark for LLMs. arXiv preprint arXiv:2410.06992 (2024)

    2024 arXiv

  3. [11]

    Anthropic. 2024. Building Effective Agents. https://www.anthropic.com/research/building-effective-agents Accessed: 2024-12-31

    2024

  4. [12]

    Anthropic. 2024. claude-3-5-sonnet. https://www.anthropic.com/news/claude-3-5-sonnet

    2024

  5. [13]

    Daman Arora, Atharv Sonwane, Nalin Wadhwa, Abhav Mehrotra, Saiteja Utpala, Ramakrishna Bairi, Aditya Kanade, and Nagarajan Natarajan. 2024. MASAI: Modular Architecture for Software-engineering AI Agents. arXiv preprint arXiv:2406.11638 (2024)

    2024 arXiv

  6. [14]

    Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, et al. 2023. Multi-lingual Evaluation of Code Generation Models. OpenReview (2023). https://arxiv.org/abs/2210.14868 ICLR 2023

    2023 arXiv

  7. [15]

    SWE bench Team. 2024. SWE-bench Leaderboard. https://www.swebench.com/ Accessed: 2024-12-30

    2024

  8. [16]

    BigCode. 2023. MultiPL-E: A Multi-programming Language Benchmark for Evaluating Code Generation. https: //nuprl.github.io/MultiPL-E/

    2023

  9. [17]

    Bissyandé, David Lo, Lingxiao Jiang, Laurent Réveillère, Jacques Klein, and Yves Le Traon

    Tegawendé F. Bissyandé, David Lo, Lingxiao Jiang, Laurent Réveillère, Jacques Klein, and Yves Le Traon. 2013. Got issues? Who cares about it? A large scale investigation of issue trackers from GitHub. InISSRE. IEEE Computer Society, 188–197

    2013

  10. [18]

    Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2024. Repairagent: An autonomous, llm-based agent for program repair. arXiv preprint arXiv:2403.17134 (2024)

    2024 arXiv

  11. [19]

    Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian-Gang Wang, Anton Cheshkov, Jun Sun, Hao Yu, Guo- liang Dong, Artem Aliev, et al . 2024. CodeR: Issue Resolving with Multi-Agent and Task Graphs. arXiv preprint arXiv:2406.01304 (2024)

    2024 arXiv

  12. [20]

    Jiachi Chen, Qingyuan Zhong, Yanlin Wang, Kaiwen Ning, Yongkun Liu, Zenan Xu, Zhe Zhao, Ting Chen, and Zibin Zheng. 2024. RMCBench: Benchmarking Large Language Models’ Resistance to Malicious Code. In Proceedings of the 39th IEEE/ACM International Conference on Automated Softw...

    2024

  13. [21]

    Yang Chen. 2024. Flakiness Repair in the Era of Large Language Models. In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings . 441–443

    2024

  14. [22]

    Pygments contributors. 2023. Pygments: Python syntax highlighter. https://pygments.org/ Version 2.15.1

    2023

  15. [23]

    Robert Feldt, Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2023. Towards autonomous testing agents via conversational large language models. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 1688–1693

    2023

  16. [24]

    Python Software Foundation. 2024. Python Documentation. https://docs.python.org/3/contents.html Accessed: 2024-10-29

    2024

  17. [25]

    Dávid Hidvégi, Khashayar Etemadi, Sofia Bobadilla, and Martin Monperrus. 2024. Cigar: Cost-efficient program repair with llms. arXiv preprint arXiv:2402.06598 (2024). Proc. ACM Softw. Eng., Vol. 2, No. ISSTA, Article ISSTA002. Publication date: July 2025. ISSTA002:22 L. Guo, W...

    2024 arXiv

  18. [26]

    Yanxian Huang, Wanjun Zhong, Ensheng Shi, Min Yang, Jiachi Chen, Hui Li, Yuchi Ma, Qianxiang Wang, Zibin Zheng, and Yanlin Wang. 2024. Agents in Software Engineering: Survey, Landscape, and Vision.arXiv preprint arXiv:2409.09030 (2024)

    2024 arXiv

  19. [27]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 (2023)

    2023 arXiv

  20. [28]

    James A Jones and Mary Jean Harrold. 2005. Empirical evaluation of the tarantula automatic fault-localization technique. In Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering . 273–282

    2005

  21. [29]

    Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. 2024. Large language model-based agents for software engineering: A survey. arXiv preprint arXiv:2409.02977 (2024)

    2024 arXiv

  22. [30]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems 36 (2024)

    2024

  23. [31]

    Yizhou Liu, Pengfei Gao, Xinchen Wang, Chao Peng, and Zhao Zhang. 2024. MarsCode Agent: AI-native Automated Bug Fixing. arXiv preprint arXiv:2409.00899 (2024)

    2024 arXiv

  24. [32]

    Yiling Lou, Ali Ghanbari, Xia Li, Lingming Zhang, Haotian Zhang, Dan Hao, and Lu Zhang. 2020. Can automated program repair refine fault localization? a unified debugging approach. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis . 75–87

    2020

  25. [33]

    Yingwei Ma, Qingping Yang, Rongyu Cao, Binhua Li, Fei Huang, and Yongbin Li. 2024. How to Understand Whole Software Repository? arXiv preprint arXiv:2406.01422 (2024)

    2024 arXiv

  26. [34]

    Mozilla Developer Network (MDN). 2024. JavaScript Documentation. https://developer.mozilla.org/en-US/docs/Web/ JavaScript Accessed: 2024-10-29

    2024

  27. [35]

    Microsoft. 2024. TypeScript Handbook. https://www.typescriptlang.org/docs/handbook/intro.html Accessed: 2024-10-29

    2024

  28. [36]

    NUS APR. 2024. AutoCodeRover Configuration Example. https://github.com/nus-apr/auto-code-rover/blob/main/ conf/example.conf Accessed: 2025-01-02

    2024

  29. [37]

    Openai. 2024. gpt-4o. https://platform.openai.com/docs/models

    2024

  30. [38]

    OpenAI. 2024. SWE-bench Verified: A Human-Validated Subset for AI Model Evaluation. https://openai.com/index/ introducing-swe-bench-verified. Accessed: 2024-10-21

    2024

  31. [39]

    Oracle. 2024. The Java Tutorials - Inheritance and Interfaces. https://docs.oracle.com/javase/tutorial/java/IandI/ Accessed: 2024-10-29

    2024

  32. [40]

    Siru Ouyang, Wenhao Yu, Kaixin Ma, Zilin Xiao, Zhihan Zhang, Mengzhao Jia, Jiawei Han, Hongming Zhang, and Dong Yu. 2024. RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph. arXiv preprint arXiv:2410.14684 (2024)

    2024 arXiv

  33. [41]

    Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. 2024. Training Software Engineering Agents and Verifiers with SWE-Gym. arXiv preprint arXiv:2412.21139 (2024)

    2024 arXiv

  34. [42]

    Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, et al. 2021. Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks. arXiv preprint arXiv:2105.12655 (2021)

    2021 arXiv

  35. [43]

    Haifeng Ruan, Yuntong Zhang, and Abhik Roychoudhury. 2024. SpecRover: Code Intent Extraction via LLMs. arXiv preprint arXiv:2408.02232 (2024)

    2024 arXiv

  36. [44]

    Wei Tao, Yanlin Wang, Ensheng Shi, Lun Du, Shi Han, Hongyu Zhang, Dongmei Zhang, and Wenqiang Zhang. 2021. On the Evaluation of Commit Message Generation Models: An Experimental Study. In ICSME. IEEE, 126–136

    2021

  37. [45]

    Wei Tao, Yanlin Wang, Ensheng Shi, Lun Du, Shi Han, Hongyu Zhang, Dongmei Zhang, and Wenqiang Zhang. 2022. A large-scale empirical study of commit message generation: models, datasets and evaluation. Empir. Softw. Eng. 27, 7 (2022), 198

    2022

  38. [46]

    Wei Tao, Yucheng Zhou, Yanlin Wang, Hongyu Zhang, Haofen Wang, and Wenqiang Zhang. 2024. KADEL: Knowledge- Aware Denoising Learning for Commit Message Generation.ACM Trans. Softw. Eng. Methodol.33, 5 (2024), 133:1–133:32

    2024

  39. [47]

    Wei Tao, Yucheng Zhou, Yanlin Wang, Wenqiang Zhang, Hongyu Zhang, and Yu Cheng. 2024. MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution. arXiv preprint arXiv:2403.17927 (2024)

    2024 arXiv

  40. [48]

    DeepSeek Team. 2024. DeepSeek-V2.5: Advanced Open-Source Large Language Model. https://www.deepseek.com/. Accessed: 2024-11-01

    2024

  41. [49]

    Linus Torvalds et al. [n. d.]. Git. https://git-scm.com/

  42. [50]

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. 2024. Opendevin: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741 (2024)

    2024 arXiv

  43. [51]

    W Eric Wong, Ruizhi Gao, Yihao Li, Rui Abreu, and Franz Wotawa. 2016. A survey on software fault localization. IEEE Transactions on Software Engineering 42, 8 (2016), 707–740. Proc. ACM Softw. Eng., Vol. 2, No. ISSTA, Article ISSTA002. Publication date: July 2025. OmniGIRL: A ...

    2016

  44. [52]

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.01489 (2024)

    2024 arXiv

  45. [53]

    Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. Automated program repair in the era of large pre-trained language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 1482–1494

    2023

  46. [54]

    Chunqiu Steven Xia and Lingming Zhang. 2023. Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT. arXiv preprint arXiv:2304.00385 (2023)

    2023 arXiv

  47. [55]

    Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang, Tingyu Lin, Weishan Zhao, Li Zhu, Shuiguang Deng, et al. 2023. Codescope: An execution-based multilingual multitask multidimensional benchmark for evaluating llms on code understanding and generation. arXi...

    2023 arXiv

  48. [56]

    Weixiang Yan, Yuchen Tian, Yunzhe Li, Qian Chen, and Wen Wang. 2023. Codetransocean: A comprehensive multilingual benchmark for code translation. arXiv preprint arXiv:2310.04951 (2023)

    2023 arXiv

  49. [57]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. arXiv preprint arXiv:2405.15793 (2024)

    2024 arXiv

  50. [58]

    Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. 2024. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. In Proceedings of the 46th IEEE/ACM International Conference on Sof...

    2024

  51. [59]

    Daoguang Zan, Zhirong Huang, Ailun Yu, Shaoxin Lin, Yifan Shi, Wei Liu, Dong Chen, Zongshuai Qi, Hao Yu, Lei Yu, et al. 2024. SWE-bench-java: A GitHub Issue Resolving Benchmark for Java. arXiv preprint arXiv:2408.14354 (2024)

    2024 arXiv

  52. [60]

    Jiarui Zhang, Yicheng Luo, Yutao Xu, et al. 2023. HumanEval-X: Extending HumanEval to Evaluate Code Generation in Multilingual Contexts. arXiv preprint arXiv:2303.17568 (2023). https://arxiv.org/abs/2303.17568

    2023 arXiv

  53. [61]

    Kexun Zhang, Weiran Yao, Zuxin Liu, Yihao Feng, Zhiwei Liu, Rithesh Murthy, Tian Lan, Lei Li, Renze Lou, Jiacheng Xu, et al. 2024. Diversity empowers intelligence: Integrating expertise of software engineering agents. arXiv preprint arXiv:2408.07060 (2024)

    2024 arXiv

  54. [62]

    Lyuye Zhang, Kaixuan Li, Kairan Sun, Daoyuan Wu, Ye Liu, Haoye Tian, and Yang Liu. 2024. Acfix: Guiding llms with mined common rbac practices for context-aware repair of access control vulnerabilities in smart contracts. arXiv preprint arXiv:2403.06838 (2024)

    2024 arXiv

  55. [63]

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Autocoderover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis . 1592–1604

    2024

  56. [64]

    Dewu Zheng, Yanlin Wang, Ensheng Shi, Hongyu Zhang, and Zibin Zheng. 2024. How Well Do LLMs Generate Code for Different Application Domains? Benchmark and Evaluation. arXiv preprint arXiv:2412.18573 (2024)

    2024 arXiv

  57. [65]

    Dewu Zheng, Yanlin Wang, Ensheng Shi, Ruikai Zhang, Yuchi Ma, Hongyu Zhang, and Zibin Zheng. 2024. Towards more realistic evaluation of LLM-based code generation: an experimental study and beyond. arXiv preprint arXiv:2406.06918 (2024)

    2024 arXiv

  58. [66]

    Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, et al. 2023. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowle...

    2023

  59. [67]

    Zibin Zheng, Kaiwen Ning, Jiachi Chen, Yanlin Wang, Wenqing Chen, Lianghong Guo, and Weicheng Wang. 2023. To- wards an understanding of large language models in software engineering tasks (2023). arXiv preprint arXiv:2308.11396 (2023). Received 2024-10-31; accepted 2025-03-31 ...

    2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.