Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

A Real-World Benchmark for Evaluating Fine-Grained Issue Solving Capabilities of Large Language Models

T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that evaluating issue solving as three separate subtasks—QA, fault localization, and code editing—reveals task-specific LLM strengths that end-to-end benchmarks miss, and introduces FAUN-Eval, a 300-entry benchmark that…

desk verdict A genuinely useful fine-grained benchmark artifact whose fault-localization ground truth conflates 'file changed in the merged PR' with 'fault location'—the central claims need that proxy validated before the rankings are trusted. read the letter →

arxiv 2411.18019 v1 pith:LEK5HMXN submitted 2024-11-27 cs.SE

Ruida Hu , Chao Peng , Jingyi Ren , Bo Jiang , Xiangxin Meng , Qinyun Wu , Pengfei Gao , Xinchen Wang
show 1 more author
Cuiyun Gao
This is my paper · ORCID
classification cs.SE
keywords FAUN-EvalLLMevaluationfaultlocalizationcodeeditingquestionansweringGitHubissuessoftwareengineeringbenchmarkpromptsensitivity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that end-to-end benchmarks like SWE-Bench cannot show where LLMs fail inside the issue-solving pipeline, and offers FAUN-Eval, a benchmark that splits the pipeline into three scored subtasks: answering the issue, locating the faulty file, and editing the code. The dataset is built from 300 real GitHub issue–pull-request pairs across 30 repositories and five languages, with automated and human validation. Ten models are evaluated, and the results show that the best model differs by subtask: Gemini-1.5-Pro leads QA and fault localization while DeepSeek-Coder-V2 leads code editing. The paper also reports that issue titles can actively mislead models and that longer inputs generally hurt performance, especially for smaller models. If these findings hold, benchmark users get per-subtask scores for model selection rather than a single end-to-end number.

What carries the argument

The machinery is a four-stage dataset construction pipeline: it crawls GitHub metadata via REST and GraphQL, pairs issues with PRs using cross-reference events and GitHub's closing keywords, filters pairs down to merged, one-to-one, single-file, non-documentation code changes, and runs a semantic check with an LLM plus manual review. The three task formats are then derived: QA uses issue title and body as input and a commenter's reply as reference; fault localization uses issue text plus a pruned repository folder tree as input and the changed file path as ground truth; code editing uses issue and PR text plus the pre-change file as input and the PR patch (SEARCH/REPLACE blocks) as reference. The central object that carries the benchmark is the issue–PR fix pair, and the central assumption is that the single changed file in the merged PR is the fault location.

What would settle it

Take a random sample of FAUN-Eval fix pairs and have an independent panel mark the actual root-cause file(s) by reading the issue and the PR diff; if a substantial share of PRs touch a file that is not the root cause (for example, a symptom-level workaround), then the fault-localization scores rest on wrong labels. A second check: run the code-editing task with ground truth defined as all files touched by the PR rather than the single filtered file and see whether model rankings change.

Watch

Extended reading notes

Core claim

The central claim is that fine-grained evaluation of issue solving is feasible and informative: FAUN-Eval provides 300 validated issue–PR entries formatted as three tasks, and using it reveals that no single model dominates all subtasks. On fault localization, GPT-4o reaches 0.65 Exact Match while two open models score zero because they ignore output-format instructions. On code editing, Gemini-1.5-Flash and DeepSeek-Coder-V2 outperform GPT-4 and GPT-4o; on QA all models score low, with the best average around 0.18. The paper concludes that task-specific strengths vary, closed-source status and parameter count do not guarantee superiority, and issue titles sometimes mislead models, so the benchmark can guide which model to deploy for each stage of issue resolution.

Load-bearing premise

The benchmark assumes that the one file changed by the merged pull request is the true and complete location of the fault, and that the issue text plus a folder tree is enough context for a model to find it; if a fix touches a different file than the actual root cause, or needs multiple files, the ground-truth labels misstate localization ability.

Editorial extensions

If this is right

  • Model selection should be subtask-specific: a pipeline could pick Gemini-1.5-Pro for understanding and locating the issue but DeepSeek-Coder-V2 for producing the edit.
  • Prompt and benchmark design should weigh issue titles carefully, since removing titles improved QA performance across all tested models and helped some models in localization.
  • Input-length effects mean evaluations should report performance stratified by issue length; a model that tops a short-text leaderboard may not lead on long issue threads.
  • Instruction-following is a measurable failure mode: models that answered fault localization with explanations instead of a path scored zero, so output-format compliance is part of capability.
  • Open-source and smaller models can beat proprietary and larger models on specific subtasks, so budget-conscious deployments have viable choices.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the single-file restriction likely under-reports the hardest real-world cases: multi-file fixes are common, and excluding them may make FAUN-Eval's localization and editing scores optimistic for practice.
  • The choice of a single commenter's reply as QA ground truth is a weak anchor; future extensions could aggregate multiple maintainer replies or use the eventual fix as the answer target.
  • The title-misleads finding suggests a testable extension: systematically rewrite issue titles and measure how much QA and localization scores move, which would quantify how much models rely on framing versus content.
  • A routing system that sends each issue to the best model per subtask, based on estimated input length, could beat any single model; FAUN-Eval provides the per-subtask scores needed to test that.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. FAUN-Eval is a benchmark intended to measure fine-grained issue-solving capabilities of LLMs across three subtasks: code question answering (QA), fault localization, and code editing. The authors crawl 30 popular GitHub repositories, pair issues with merged pull requests, apply a multi-stage filtering and semantic-validation pipeline, and release 300 entries. Ten proprietary and open-source LLMs are evaluated with generic text/code similarity metrics, producing per-task rankings and ablation findings such as 'the issue body is crucial for QA and fault localization' and 'issue titles can mislead models.' The paper's contribution is mainly empirical: a reusable dataset and a decomposition of end-to-end issue resolution into three evaluable stages.

Significance. If the benchmark's operationalizations are accepted, FAUN-Eval would be a useful complement to end-to-end benchmarks like SWE-Bench, because it provides per-subtask scores that can guide model selection and identify where models fail. The data-construction pipeline is detailed and transparent, and the authors explicitly release the dataset and model outputs, which supports reproducibility. The three-task framing is sensible and maps onto a natural workflow. However, the validity of the benchmark's headline findings depends on several unvalidated choices, especially the use of the single file changed in a merged PR as the ground-truth fault location, and the reliance on exact-match and code-similarity metrics that conflate instruction following with capability. These issues affect a large share of the reported results, so the paper requires substantial revision before its claims can be accepted.

major comments (5)
  1. [§3.3.2, §3.4.2, §5.1 (Table 4)] The fault-localization ground truth is not validated as a fault location. Section 3.3.2 ('File-Changed Count') keeps only PRs that modify a single file, and Section 3.4.2 defines the task output as the 'changed path' of that PR. But Section 1 defines fault localization as pinpointing 'the specific file or section of code responsible for the reported issue.' A file changed by a merged PR is not necessarily the file responsible for the issue; the fix may reside in a different file than the one where the symptom appears, and some merged PRs are feature additions, refactors, or workarounds rather than causal fixes. The single-file filter also removes the multi-file cases that the paper's motivation emphasizes. Consequently, the EM scores in Table 4 (e.g., GPT-4o 0.65, GM-Pro 0.64) measure retrodiction of a developer's edited file on an easy subset, not fault localization as defined. Since Findings 1 and 3 rely substantially on this task, the authors should either validate the fault-location proxy (e.g., by manual annotation, bug-introducing-commit analysis, or test-based verification) or reframe the task as 'changed-file prediction' and temper the corresponding claims.
  2. [§4.4, §5.1 (Table 4)] The fault-localization metric conflates instruction following with localization ability. The paper uses Exact Match as the sole metric, and Table 4 reports EM = 0 for CodeQwen-1.5-Chat and DeepSeek-Coder 33B because these models output the correct path along with explanatory text. The paper itself notes this behavior in Section 5.1, but it still reports these as zero localization scores. This makes the FL rankings dependent on a strict output-format requirement rather than on the model's ability to identify the right file. The authors should either post-process and parse the model outputs (e.g., extract the first file path), use a containment or normalized-match metric, or report an instruction-following metric separately from localization accuracy. Without this, the zero scores and the averaged FL numbers in Table 4 and RQ2 are not comparable across models.
  3. [§3.4.1, §4.2] The QA ground truth is underspecified. The QA task treats the issue title and body as input and 'the body of the response from the reply' as the reference output. It is not clear how a single reply is selected from the issue thread, whether the selected commenter is a maintainer or otherwise authoritative, or whether the chosen response actually answers the issue. The semantic check in Section 3.3.3 validates issue-PR pairs, not QA reply pairs. As a result, the QA scores may measure similarity to an arbitrary comment rather than to a correct or representative answer. The authors need to specify and validate the reply-selection procedure, and ideally include inter-annotator agreement or a manual correctness check on the QA gold answers.
  4. [§4.4, §5.1 (Table 4)] CodeBLEU is not a measure of whether an edit resolves the issue. The code-editing task is evaluated exclusively with CodeBLEU, which compares lexical, AST, and data-flow similarity to the reference patch. A generated edit that is plausible but functionally incorrect can receive a high CodeBLEU score if it resembles the gold patch, while a correct but differently structured fix can be penalized. Since no tests or functional checks are used, the reported code-editing scores do not establish issue-solving capability. The paper should either add a functional verification component (e.g., running repository tests or applying the patch and checking behavior), or explicitly limit the claim to 'patch similarity' rather than 'code editing capability.'
  5. [§5.1, §5.3 (Table 4, Figure 3)] The ranking claims lack uncertainty quantification. With only 300 entries and no confidence intervals, significance tests, or per-repository breakdowns, small differences in Table 4 (e.g., QA BLEU 0.0943 vs. 0.0948; FL EM 0.6500 vs. 0.6400) and the ordering crossovers in Figure 3 may be noise. The paper states strong conclusions such as 'the top-performing models differ across the different tasks' and 'different models may excel at processing contexts of varying lengths' without statistical support. The authors should add confidence intervals, paired tests (e.g., bootstrap or McNemar for EM), and ideally report variance across repositories.
minor comments (6)
  1. [Author affiliations] 'Haribin Institute of Technology' should be 'Harbin Institute of Technology.'
  2. [§2.2.1] The word 'compability' appears to be a typo for 'compatibility.'
  3. [§3.3.2, §3.4.2] The paper states that test files are excluded from the dataset, but some issues may legitimately involve test code or test configuration; the rationale for excluding them (beyond simplicity) could be stated more explicitly.
  4. [§5.2 (Table 5)] The text says 'some models like DeepSeek-Coder-V2 show enhanced Exact Match scores after discarding the body,' but the corresponding Table 5 condition is 'IB' (issue body only), which removes the issue title, not the body. The wording appears to be inconsistent with the table.
  5. [§2 (Table 1)] Table 1 lists FAUN-Eval-QA and FAUN-Eval-fix separately, while the abstract and introduction describe FAUN-Eval as containing 300 entries. It would help to clarify how many issues are shared between the QA set and the fault-localization/code-editing sets, and how many QA entries have a corresponding fix pair.
  6. [§6.2] The threats-to-validity section does not discuss the risk that using GPT-4o during semantic filtering may bias the dataset toward issues that GPT-4o can understand; the manual review mitigates but does not eliminate this concern.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: task ground truths are externally sourced from GitHub history (human replies, real PR diffs, changed paths), and model scores are measured rather than derived from or fitted to the benchmark inputs.

full rationale

FAUN-Eval's central claims are empirical measurements, not derived quantities, so no enumerated circularity pattern applies. All three task ground truths come from GitHub artifacts that predate and are independent of the evaluated models: the QA reference is a human commenter's reply from the issue thread (Section 3.4.1, Section 4.2), the fault-localization reference is the path actually changed by the merged PR (Section 3.4.2, Section 4.2), and the code-editing reference is the PR's own patch (Section 3.4.3). No parameter is fitted to any model output, and no evaluation subset is defined in terms of a model score, so no prediction is forced by construction. Two candidate concerns were inspected and do not rise to circularity. First, GPT-4o participates in the semantic filter that selects issues presenting a 'distinct question-and-answer format' (Section 3.3.3), and GPT-4o is later scored on the QA task. This is a curation-bias/contamination risk, not a circular reduction: the filter judges whether a human question-answer pair is coherent, while the reference output is the human reply that GPT-4o never generates, and two researchers independently revalidate every retained entry (Section 3.3.3, Manual Review), so the dataset is not GPT-4o's own output. Second, the fault-localization task equates 'file changed in a single-file PR' with 'fault location' (Section 3.3.2 File-Changed Count; Section 3.4.2). That is a construct-validity threat (the fix can reside in a different file than the symptom site, and multi-file fixes are deliberately filtered out), but the EM results, e.g., GPT-4o at 0.6500 with several models at 0 (Table 4), show model predictions routinely deviate from the ground truth, so the metric is a possibly biased yardstick rather than a quantity equal to its input by definition. The code-editing task supplies the PR description as input alongside the target patch, which risks answer leakage, but the reference is the actual diff and the CodeBLEU scores are far from 1, indicating real measurable error. Self-citations (e.g., reference [25], which includes three co-authors) appear only as background related work and are not load-bearing for the benchmark's validity or its findings. Findings 1-4 (task-specific rankings, title misdirection, body importance, length degradation) are empirical aggregations of the measured scores and are not entailed by the dataset construction. Verdict: no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The benchmark's validity rests on domain assumptions about GitHub fix relationships and on several hand-chosen filtering thresholds that define the dataset's scope. These are not fitted to data but shape the central results.

free parameters (4)
  • Minimum issue length threshold = 200 characters
    Hand-chosen cutoff used in Section 3.3.1 to filter out short issues, affecting which issues enter the QA dataset.
  • Maximum event count = 10
    Issues with more than ten events are excluded in Section 3.3.1, removing issues with many automated messages.
  • Single-file change constraint = exactly one file
    Only PRs that modify a single file are kept in Section 3.3.2, simplifying fault localization and code editing but reducing realism.
  • Minimum participant count = 2 participants
    Issues with only one participant are removed in Section 3.3.1 to exclude unresolved discussions.
assumptions (4)
  • domain assumption GitHub closing keywords (fix, close, resolve) reliably identify issue-fixing PRs
    Used in Section 3.2.2 to build fix pairs; assumes these keywords cover the relevant fix relationships, though many PRs fix issues without using them.
  • domain assumption The repository folder tree at the PR base commit provides sufficient context for fault localization
    Assumed in Section 3.4.2; the tree omits file contents and may not reflect the actual fault.
  • domain assumption A single commenter's reply in the issue thread is the correct reference answer for QA
    Used in Sections 3.4.1 and 4.2; the issue thread may contain multiple or no definitive answers.
  • domain assumption Merged PRs represent valid fixes
    The Merged Check in Section 3.3.2 assumes maintainer acceptance corresponds to a correct resolution of the issue.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Real-World Benchmark for Evaluating Fine-Grained Issue Solving Capabilities of Large Language Models." pith.science (2026). https://pith.science/paper/LEK5HMXN

@misc{pith2026241118019,
  author       = {Pith},
  title        = {Pith review of: A Real-World Benchmark for Evaluating Fine-Grained Issue Solving Capabilities of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LEK5HMXN}},
  note         = {Machine review of arXiv:2411.18019}
}
read the original abstract

Automatically resolving software issues is crucial for software development in practice, impacting the software quality and user experience. The process of resolving real-world issues encompasses tasks such as question-answering (QA), fault localization, and code editing. Existing benchmarks such as HumanEval fall short in their ability to assess LLMs' proficiency in solving issues within a codebase. Although benchmarks like SWE-Bench are designed to evaluate the LLMs' capability to handle real-world GitHub issues, the end-to-end evaluation method cannot provide granular insights on the performance of subtasks involved in issue solving. To address existing deficiencies in benchmarking LLMs for practical software engineering tasks, we introduce FAUN-Eval, a benchmark specifically designed to evaluate the Fine-grAined issUe solviNg capabilities of LLMs. FAUN-Eval systematically assesses LLMs across three distinct tasks: QA, fault localization, and code editing. This benchmark is constructed using a dataset curated from 30 well-known GitHub repositories. For each entry, issue and pull request (PR) pairs are meticulously compiled and validated using cross-referencing and keyword verification methods. FAUN-Eval includes 300 entries and employs both LLM and manual checks to ensure data quality. We evaluate ten LLMs with FAUN-Eval, including four closed-source and six open-source models. Our experimental results reveal several key findings. We find that the top-performing LLMs differ across the different tasks. Additionally, features in issues may lead LLMs to generate incorrect information. Moreover, models may vary in their proficiency with texts of different lengths.

Figures

Figures reproduced from arXiv: 2411.18019 by the authors.

Figure 1
Figure 1. The architecture of our data collection framework. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Structures of QA, fault localization, and code editing data entries. [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Performance of different LLMs across distinct tasks for short, medium, and long input texts. [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: An Example of issue title guiding LLM into the correct response. [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Evaluating LLM Agents on Automated Software Analysis Tasks

    cs.SE 2026-04 unverdicted novelty 7.0 of 10

    A purpose-built, staged LLM agent correctly sets up and executes software analysis tools on 33 of 35 benchmark tasks, outperforming general-purpose agent baselines by at least 17 percentage points.

Reference graph

Works this paper leans on

44 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    Toufique Ahmed, Kunal Suresh Pai, Premkumar Devanbu, and Earl Barr. 2024. Automatic semantic augmentation of language model prompts (for code summarization). In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13

    2024

  2. [2]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021)

    arXiv 2021

  3. [3]

    Berkay Berabi, Jingxuan He, Veselin Raychev, and Martin Vechev. 2021. Tfix: Learning to fix coding errors with a text-to-text transformer. In International Conference on Machine Learning . PMLR, 780–791

    work page 2021

  4. [4]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

    arXiv 2021

  5. [5]

    Yinfang Chen, Huaibing Xie, Minghua Ma, Yu Kang, Xin Gao, Liu Shi, Yunjie Cao, Xuedong Gao, Hao Fan, Ming Wen, et al. 2024. Automatic root cause analysis via large language models for cloud incidents. In Proceedings of the Nineteenth European Conference on Computer Systems . 674–688

    2024

  6. [6]

    Viktor Csuvik and László Vidács. 2022. Fixjs: A dataset of bug-fixing javascript commits. In Proceedings of the 19th International Conference on Mining Software Repositories . 712–716

    work page 2022

  7. [7]

    Xiaohu Du, Ming Wen, Jiahao Zhu, Zifan Xie, Bin Ji, Huijun Liu, Xuanhua Shi, and Hai Jin. 2024. Generalization- Enhanced Code Vulnerability Detection via Multi-Task Instruction Fine-Tuning. arXiv preprint arXiv:2406.03718 (2024)

    arXiv 2024

  8. [8]

    facebook. 2024. react. https://github.com/facebook/react

    work page 2024

Show all 44 references
  1. [9]

    Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. 2023. Automated repair of programs from large language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 1469–1481

    2023

  2. [10]

    Lingyue Fu, Huacan Chai, Shuang Luo, Kounianhua Du, Weiming Zhang, Longteng Fan, Jiayi Lei, Renting Rui, Jianghao Lin, Yuchen Fang, et al. 2023. Codeapex: A bilingual programming evaluation benchmark for large language models. arXiv preprint arXiv:2309.01940 (2023)

    2023 arXiv

  3. [11]

    Mingyang Geng, Shangwen Wang, Dezun Dong, Haotian Wang, Ge Li, Zhi Jin, Xiaoguang Mao, and Xiangke Liao

  4. [12]

    google. 2024. guava. https://github.com/google/guava

    2024

  5. [13]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. 2021. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938 (2021)

    2021 arXiv

  6. [14]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. Proceedings of the International Conference on Learning Representations (ICLR) (2021). , Vol. 1, No. 1, Article . Pu...

    2021

  7. [15]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang

  8. [16]

    Sihao Hu, Tiansheng Huang, Fatih İlhan, Selim Furkan Tekin, and Ling Liu. 2023. Large language model-powered smart contract vulnerability detection: New perspectives. In 2023 5th IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applicatio...

    2023

  9. [17]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 (2023)

    2023 arXiv

  10. [18]

    René Just, Darioush Jalali, and Michael D Ernst. 2014. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the 2014 international symposium on software testing and analysis . 437–440

    2014

  11. [19]

    kubernetes. 2024. kubernetes. https://github.com/kubernetes/kubernetes

    2024

  12. [20]

    Changyoon Lee, Yeon Seonwoo, and Alice Oh. 2022. CS1QA: A dataset for assisting code-based question answering in an introductory programming course. arXiv preprint arXiv:2210.14494 (2022)

    2022 arXiv

  13. [21]

    Caroline Lemieux, Jeevana Priya Inala, Shuvendu K Lahiri, and Siddhartha Sen. 2023. Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 919–931

    2023

  14. [22]

    Bo Lin, Shangwen Wang, Ming Wen, Liqian Chen, and Xiaoguang Mao. 2024. One Size Does Not Fit All: Multi- granularity Patch Generation for Better Automated Program Repair. In Proceedings of the 33rd ACM SIGSOFT Interna- tional Symposium on Software Testing and Analysis

    2024

  15. [23]

    Chenxiao Liu and Xiaojun Wan. 2021. CodeQA: A question answering dataset for source code comprehension. arXiv preprint arXiv:2109.08365 (2021)

    2021 arXiv

  16. [24]

    Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. 2024. Large Language Model-Based Agents for Software Engineering: A Survey. arXiv preprint arXiv:2409.02977 (2024)

    2024 arXiv

  17. [25]

    Yizhou Liu, Pengfei Gao, Xinchen Wang, Chao Peng, and Zhao Zhang. 2024. MarsCode Agent: AI-native Automated Bug Fixing. arXiv preprint arXiv:2409.00899 (2024)

    2024 arXiv

  18. [26]

    Yingwei Ma, Qingping Yang, Rongyu Cao, Binhua Li, Fei Huang, and Yongbin Li. 2024. How to Understand Whole Software Repository? arXiv preprint arXiv:2406.01422 (2024)

    2024 arXiv

  19. [27]

    Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, and Juho Leinonen

  20. [28]

    Wonseok Oh and Hakjoo Oh. 2022. PyTER: effective program repair for Python type errors. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering . 922–934

    2022

  21. [29]

    Theo X Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. 2023. Is Self-Repair a Silver Bullet for Code Generation?. In The Twelfth International Conference on Learning Representations

    2023

  22. [30]

    In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V

    Experiences from using code explanations generated by large language models in a web software development e-book. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 . 931–937

  23. [31]

    Yangtian Zi Raymond Li, Loubna Ben Allal et al . 2023. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161 (2023)

    2023 arXiv

  24. [32]

    Steven I Ross, Fernando Martinez, Stephanie Houde, Michael Muller, and Justin D Weisz. 2023. The programmer’s assistant: Conversational interaction with a large language model for software development. In Proceedings of the 28th International Conference on Intelligent User Int...

    2023

  25. [33]

    Yihao Qin, Shangwen Wang, Yiling Lou, Jinhao Dong, Kaixin Wang, Xiaoling Li, and Xiaoguang Mao. 2024. AgentFL: Scaling LLM-based Fault Localization to Project-Level Context. arXiv preprint arXiv:2403.16362 (2024)

    2024 arXiv

  26. [34]

    Maolin Sun, Yibiao Yang, Yang Wang, Ming Wen, Haoxiang Jia, and Yuming Zhou. 2023. SMT solver validation empowered by large pre-trained language models. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1288–1300

    2023

  27. [35]

    Wei Tao, Yucheng Zhou, Wenqiang Zhang, and Yu Cheng. 2024. MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution. arXiv preprint arXiv:2403.17927 (2024)

    2024 arXiv

  28. [36]

    scipy. 2024. scipy. https://github.com/scipy/scipy

    2024

  29. [37]

    Yuxiang Wei, Chunqiu Steven Xia, and Lingming Zhang. 2023. Copiloting the copilots: Fusing large language models with completion engines for automated program repair. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations...

    2023

  30. [38]

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.01489 (2024)

    2024 arXiv

  31. [39]

    typeorm. 2024. typeorm. https://github.com/typeorm/typeorm

    2024

  32. [40]

    Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. 2024. Evaluating and improving chatgpt for unit test generation. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1703–1726

    2024

  33. [41]

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Autocoderover: Autonomous program improvement. arXiv preprint arXiv:2404.05427 (2024). , Vol. 1, No. 1, Article . Publication date: November 2024

    2024 arXiv

  34. [42]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. arXiv preprint arXiv:2405.15793 (2024). , Vol. 1, No. 1, Article . Publication date: ...

    2024 arXiv

  35. [2023]

    arXiv preprint arXiv:2308.10620 (2023)

    Large language models for software engineering: A systematic literature review. arXiv preprint arXiv:2308.10620 (2023)

    2023 arXiv

  36. [2024]

    In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering

    Large language models are few-shot summarizers: Multi-intent comment generation via in-context learning. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering . 1–13

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.