Pith. sign in

REVIEW 4 major objections 5 minor 53 references

Testing Refactoring Engine via Historical Bug Report driven LLM

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read RETESTER shows that LLM-generated, history-seeded refactoring-preserving variants can uncover 18 new bugs in the latest Eclipse and IntelliJ IDEA refactoring engines.

desk verdict Useful first step in LLM-based refactoring-engine testing, but the bug claims rest on a weak equivalence oracle and an internal count inconsistency. read the letter →

arxiv 2501.09879 v2 pith:SCW2MXPT submitted 2025-01-16 cs.SE

Haibo Wang , Zhuolin Xu , Shin Hwei Tan This is my paper · ORCID
classification cs.SE
keywords refactoringenginetestingLLM-basedtestgenerationdifferentialhistoricalbugreportsinputprogrammutationrefactoring-preservingtransformationEclipseJDTIntelliJIDEA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes RETESTER, a way to test refactoring engines by having a large language model generate new bug-triggering Java programs from historical bug reports. The key move is to preserve the applicability of the original refactoring while mutating the program with an error-prone construct such as lambda expressions, Java generics, or anonymous classes, so that any difference in the engine's output indicates a bug. On five seed reports covering five refactoring types, RETESTER generated 150 variants and found 18 previously unreported bugs in the latest Eclipse and IntelliJ IDEA refactoring engines; seven were confirmed by developers and three fixed by submission time. The paper also shows that abstracting the seed program into a structural template, rather than mutating the concrete code, is what lets the LLM explore diverse edge cases.

What carries the argument

The central mechanism is the refactoring-preserving transformation (RPT): a mutation Trans(P1)=P2 is valid only if the same refactoring operation O that applied to program element E1 in the seed still applies to some element E2 in the variant. This condition keeps generated programs inside the refactoring engine's precondition space so that engine misbehavior can be attributed to the refactoring itself. RETESTER operationalizes RPT with chain-of-thought prompts: the LLM first reads a refactoring definition, then an input-program structure template extracted from a historical bug report, then one error-prone characteristic, and must output the variant plus the elements and procedures for refactoring. Variants are filtered by javac compilation and by checking refactoring applicability, then the same refactoring is run in both IDEs under three oracles: uncompilable output, differing warning status, and differing refactored programs.

What would settle it

Run the same five-seed, three-characteristic pipeline on a held-out set of historical bug reports whose refactoring types are not among the five seeds, such as Rename, Move, or Change Signature, and count newly confirmed developer issues; if the bug yield does not exceed the no-mutation baseline, the claim that historical-bug-report-driven LLM mutation generalizes across refactoring types would be refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that a history-driven, LLM-based mutation pipeline can find real bugs in current refactoring engines without manually designing templates for each refactoring type. Starting from fixed historical bug reports, RETESTER extracts the input program, the refactoring type and procedure, and an abstract input-program structure template. It then prompts the LLM, in a chain-of-thought style, to produce a refactoring-preserving variant that introduces one of three error-prone Java features. Applying the same refactoring to each variant in Eclipse JDT (2024-09) and IntelliJ IDEA (2024.2.4) and using differential oracles, the authors found 18 new bugs, 15 unique in Eclipse and 3 in IntelliJ IDEA; seven were confirmed by developers and three were fixed. An ablation without the template found only six bugs, with three overlapping, and a no-mutation baseline found none, supporting the claim that the combination of historical structure and targeted mutation is what exposes the bugs.

Load-bearing premise

The bug counts depend on the premise that lambda expressions, Java generics, and anonymous classes, chosen from an earlier empirical study and applied to only five seed reports, are the dominant error-prone traits that trigger refactoring-engine bugs; if the real triggers lie elsewhere, the approach would find far fewer bugs on other inputs.

Editorial extensions

If this is right

  • If RETESTER's results hold, refactoring engines in current IDEs still emit uncompilable code on small programs that combine nested classes with lambda expressions, Java generics, or anonymous classes.
  • The extracted input-program structure templates can be reused by template-based testing tools as historically grounded templates instead of hand-crafted ones.
  • The pipeline extends to additional refactoring types simply by adding more historical bug reports, without redesigning the mutation prompts for each type.
  • Bug-triggering seeds tend to be small (median 9 lines of code), so test generators should favor small, structurally complex programs over larger, simpler ones.
  • Pull Up and Inline Method produced the most bugs in this sample, giving engine developers concrete areas to prioritize.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: combining the three error-prone characteristics in one variant, such as an anonymous class inside a generic type, could reveal interaction bugs that single-characteristic mutations miss; the paper leaves such combinations for future work.
  • Beyond the paper: the same template-extraction and refactoring-preserving prompt recipe could be applied to other languages or to compiler and JIT testing, where history-driven seeds are already known to help.
  • Beyond the paper: refactoring each variant by hand in the IDE limits throughput; automating the refactoring step would let the approach scale to thousands of variants and could change the measured bug yield.
  • Beyond the paper: the 15-to-3 split between Eclipse and IntelliJ IDEA may reflect differences in precondition checking rather than an intrinsic quality ranking, so the finding should not be read as a verdict on which IDE is better.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents RETESTER, an LLM-based approach to testing Java refactoring engines. It mines historical bug reports from Eclipse and IntelliJ IDEA, extracts seed programs and refactoring information using few-shot prompting, and then uses chain-of-thought prompts to generate variants that incorporate error-prone language characteristics (lambda, generics, anonymous classes) while preserving applicability of the original refactoring. The variants are applied to the latest engine versions and checked with uncompilable, warning-status, and differential oracles. The authors report discovering 18 new bugs (15 in Eclipse, 3 in IntelliJ IDEA), with 7 confirmed and 3 fixed by developers at submission time. An ablation shows that using an abstract program template yields more bugs than mutating the concrete input program.

Significance. If the reported bugs are genuine, the paper would be a valuable demonstration that LLMs can translate historical bug-triggering structures into new test programs for increasingly complex IDE refactoring engines, complementing earlier template-based and random-testing approaches. The dataset of 167 compilable Java programs extracted from bug reports, the 98/100 extraction accuracy, and the reproducible workflow are concrete strengths. However, the central claim rests on the manual validation of oracle violations, and the paper's current presentation does not adequately rule out false positives caused by semantic changes in LLM-generated variants, nor does it reconcile the reported bug statuses. These issues must be resolved before the results can be taken at face value.

major comments (4)
  1. [Section III-E, Definition 1; Section III-F] The refactoring-preserving transformation only requires that a program element exists in the variant on which the refactoring operation remains applicable; it does not require the variant to be semantically equivalent to the seed or the refactoring to be valid. The Uncompilable and Differential oracles then treat any syntax error or output difference as a potential engine bug. Because the LLM variants may change program semantics, an engine's refusal or different result could be correct for the modified program. The paper must either strengthen Definition 1 to require behavioral preservation (or at least a recorded proof that the refactoring precondition holds and the transformation is semantics-preserving) or provide an automated or manual semantic-equivalence check for each reported issue. In particular, for the 11 issues that developers have not confirmed or fixed, the evidence currently rests entirely on the authors' manual classification.
  2. [Abstract; Section V-A; Table VIII] The reported confirmation counts are inconsistent. The abstract and conclusion say 'seven of them were confirmed ... and three were fixed,' but Table VIII lists only four issues with status 'Confirmed' and three with 'Fixed'; the text in RQ1 says both 'seven have been confirmed' and 'four out of the eighteen bugs have been officially confirmed.' If 'confirmed' is meant to include 'fixed,' the table should use a single status field; if not, the abstract number is wrong. The authors should correct this and give the exact developer response for each issue.
  3. [RQ2, Table IX] The baseline comparison does not faithfully represent Gligoric et al.'s approach. The baseline is implemented by having an LLM propose refactoring operations on the seed input programs, rather than applying refactorings at randomly chosen program elements in real projects as in the original work. With only 50 variants per configuration, the finding that the baseline reports zero bugs is best interpreted as a no-mutation ablation, not as evidence that RETESTER outperforms the state of the art. The claim 'Without performing mutation on the input program, Gligoric's approach fails to find any bugs' is therefore overstated.
  4. [Data Availability; reference [25]] The Abstract and Contributions state that the data are open-sourced, but the Data Availability section says reference [25] is 'To be open-sourced.' This makes the dataset and the 18 bug artifacts unverifiable. Provide an accessible link or state a clear availability policy.
minor comments (5)
  1. [Table VII] The last column header '18 (15)' is unexplained in the footnote; add a note that the parenthetical is the number of unique Eclipse bugs after removing three overlaps between the template and no-template conditions.
  2. [Section IV-A; Table VII] Section IV-A states that 150 variants are generated, while Table VII totals 300 variants; clarify that the additional 150 come from the RQ4 ablation (Template=N rows).
  3. [Section V-D] The phrase 'lst two columns' should read 'last two columns.'
  4. [Section III-E] The term 'refactoring-preserving' is misleading because Definition 1 only preserves applicability, not behavior; consider renaming it to 'refactoring-applicability-preserving' or explicitly discussing the distinction.
  5. [Figure 3] The three code blocks in Figure 3 are difficult to read; increase the font size and separate the seed, template, and variant visually.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: RETESTER's bug findings are externally validated; the self-citation for characteristic selection is not load-bearing.

full rationale

The claimed derivation chain is: historical bug reports are mined and labeled; LLM extracts compilable seed programs, refactoring information, and structure templates; mutations are generated using three error-prone characteristics; variants are refactored with the same operation in Eclipse and IntelliJ IDEA; oracle violations are manually triaged and reported; developers confirm or fix a subset. The only self-citation is reference [10], the authors' own prior empirical study, used to justify choosing lambda, generics, and anonymous class as mutation characteristics. This is a design heuristic, not a mathematical derivation, and the central claim—18 new refactoring-engine bugs—does not reduce to it: the bugs are found on newly generated variant programs, triaged manually, and seven are independently confirmed or fixed by external developers. The Uncompilable Oracle is a validity threat because LLM-generated variants may change program semantics, but it is not circular: the oracle checks the engine's output on new inputs, not the inputs themselves, and the paper does not define a bug as equivalent to the mutation. No fitted parameter is renamed as a prediction, and no equation reduces to its own input. The reliance on the authors' prior study is self-citation but not load-bearing circularity, so the appropriate score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on design choices inherited from prior work, especially the selection of three error-prone characteristics, and on assumptions about what constitutes a valid test input and a bug. No new physical or ontological entities are introduced. The approach is empirical and does not derive quantitative predictions from fitted constants.

free parameters (5)
  • Set of error-prone input program characteristics = lambda, Java generics, anonymous class
    The paper selects three of 38 characteristics from prior work [10] to bound the search space; the choice is hand-made and not optimized or validated in this study. Section III-E.
  • Number of variants per seed = 30 (10 per characteristic)
    Hand-chosen; the paper generates 10 variants for each of 3 characteristics per seed, with no sensitivity analysis. Section IV-A.
  • Number of seed bug reports = 5
    Randomly selected from 167 compilable seeds; small sample limits generalizability. Section IV-A.
  • LLM model = gpt-4o-mini
    Chosen for the main experiments; results are model-dependent. Section IV-A.
  • JDK version = 22.0.1
    Used in mutation prompts and compilation checks. Section IV-A.
assumptions (4)
  • domain assumption The three selected input program characteristics are the most error-prone for refactoring engines.
    Section III-E: 'we choose three most error-prone input program characteristics as shown in Table IV in our study' based on prior work [10].
  • domain assumption A variant is a valid test input if it compiles and the same refactoring remains applicable, as defined by Refactoring-Preserving Transformation.
    Definition 1 and Section III-E; the paper relies on compilation and applicability as proxies for validity.
  • domain assumption Uncompilable refactored output after applying a refactoring is an engine bug after manual verification.
    Section III-F Uncompilable Oracle; manual verification is done but relies on annotators' judgment.
  • domain assumption Historical bug reports from the same engines contain structures that are still bug-prone in current versions.
    The history-driven approach assumes past bug patterns generalize to present versions; not proven in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Testing Refactoring Engine via Historical Bug Report driven LLM." pith.science (2026). https://pith.science/paper/SCW2MXPT

@misc{pith2026250109879,
  author       = {Pith},
  title        = {Pith review of: Testing Refactoring Engine via Historical Bug Report driven LLM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SCW2MXPT}},
  note         = {Machine review of arXiv:2501.09879}
}
read the original abstract

Refactoring is the process of restructuring existing code without changing its external behavior while improving its internal structure. Refactoring engines are integral components of modern Integrated Development Environments (IDEs) and can automate or semi-automate this process to enhance code readability, reduce complexity, and improve the maintainability of software products. Similar to traditional software systems such as compilers, refactoring engines may also contain bugs that can lead to unexpected behaviors. In this paper, we propose a novel approach called RETESTER, a LLM-based framework for automated refactoring engine testing. Specifically, by using input program structure templates extracted from historical bug reports and input program characteristics that are error-prone, we design chain-of-thought (CoT) prompts to perform refactoring-preserving transformations. The generated variants are then tested on the latest version of refactoring engines using differential testing. We evaluate RETESTER on two most popular modern refactoring engines (i.e., ECLIPSE, and INTELLIJ IDEA). It successfully revealed 18 new bugs in the latest version of those refactoring engines. By the time we submit our paper, seven of them were confirmed by their developers, and three were fixed.

Figures

Figures reproduced from arXiv: 2501.09879 by the authors.

Figure 1
Figure 1. The general workflow of a refactoring engine. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Screenshot of an ECLIPSE historical bug report [27]. public class A { public class BaseInner {} public class Outer { public int x = 0; public void foo(){}; public class Inner extends BaseInner { void innerMethod() { // Pull up to BaseInner System.out.println(Outer.this.x); Outer.this.foo(); }}}} public class OuterClass { public class BaseTargetClass {} public class OriginalClass { public DataType memberVariable; pub… view at source ↗
Figure 3
Figure 3. A bug-triggering input program extracted from histor [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Overall workflow of RETESTER. adopt an automatic way by using a combination of few-shot￾learning of LLM and JVM compiler. Different from traditional natural language corpus (e.g., news), a historical bug report (e.g., a bug report for Eclipse [27]) is a mixture of seve…
Figure 5
Figure 5. Figure 5: The venn diagram for the number of bugs detected by [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 47 canonical work pages

  1. [25]

    (2024) To be open-sourced

    work page 2024

  2. [1]

    Becker, M

    P. Becker, M. Fowler, K. Beck, J. Brant, W. Opdyke, and D. Roberts, Refactoring: improving the design of existing code . Addison-Wesley Professional, 1999

    work page 1999

  3. [2]

    Refactoring-improving coupling and cohesion of existing code,

    B. Du Bois, S. Demeyer, and J. Verelst, “Refactoring-improving coupling and cohesion of existing code,” in 11th working conference on reverse engineering. IEEE, 2004, pp. 144–151

    work page 2004

  4. [3]

    An empirical study of refactoringchallenges and benefits at microsoft,

    M. Kim, T. Zimmermann, and N. Nagappan, “An empirical study of refactoringchallenges and benefits at microsoft,” IEEE Transactions on Software Engineering, vol. 40, no. 7, pp. 633–649, 2014

    work page 2014

  5. [4]

    An empirical study on the impact of refactoring activities on evolving client-used apis,

    R. G. Kula, A. Ouni, D. M. German, and K. Inoue, “An empirical study on the impact of refactoring activities on evolving client-used apis,” Information and Software Technology , vol. 93, pp. 186–199, 2018

    2018

  6. [5]

    Improving code maintainabil- ity: A case study on the impact of refactoring,

    M. Wahler, U. Drofenik, and W. Snipes, “Improving code maintainabil- ity: A case study on the impact of refactoring,” in 2016 IEEE Inter- national Conference on Software Maintenance and Evolution (ICSME) . IEEE, 2016, pp. 493–501

    work page 2016

  7. [6]

    A survey of software refactoring,

    T. Mens and T. Tourwé, “A survey of software refactoring,” IEEE Transactions on software engineering, vol. 30, no. 2, pp. 126–139, 2004

    work page 2004

  8. [7]

    Ten years of jdeodor- ant: Lessons learned from the hunt for smells,

    N. Tsantalis, T. Chaikalis, and A. Chatzigeorgiou, “Ten years of jdeodor- ant: Lessons learned from the hunt for smells,” in 2018 IEEE 25th inter- national conference on software analysis, evolution and reengineering (SANER), 2018

    work page 2018

Show all 53 references
  1. [8]

    [Online]

    (2024) Eclipse. [Online]. Available: http://www.eclipse.org/

    2024

  2. [9]

    [Online]

    (2024) Intellij idea. [Online]. Available: http://www.jetbrains.com/idea/

    2024

  3. [10]

    An empirical study of refactoring engine bugs,

    H. Wang, Z. Xu, H. Zhang, N. Tsantalis, and S. H. Tan, “An empirical study of refactoring engine bugs,” arXiv preprint arXiv:2409.14610 , 2024

    2024 arXiv

  4. [11]

    Automated testing of refactoring engines,

    B. Daniel, D. Dig, K. Garcia, and D. Marinov, “Automated testing of refactoring engines,” in Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering , 2007, pp. 185– 194

    2007

  5. [12]

    Saferefactor-tool for checking refactoring safety,

    G. Soares, D. Cavalcanti, R. Gheyi, T. Massoni, D. Serey, and M. Cornélio, “Saferefactor-tool for checking refactoring safety,” Tools Session at SBES , pp. 49–54, 2009

    2009

  6. [13]

    Scaling testing of refactoring engines,

    M. Mongiovi, “Scaling testing of refactoring engines,” in Companion Proceedings of the 2016 ACM SIGPLAN International Conference on Systems, Programming, Languages and Applications: Software for Humanity, 2016, pp. 15–17

    2016

  7. [14]

    Making program refactoring safer,

    G. Soares, “Making program refactoring safer,” in Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering- Volume 2, 2010, pp. 521–522

    2010

  8. [15]

    Automated behavioral testing of refactoring engines,

    G. S. Soares, “Automated behavioral testing of refactoring engines,” in Proceedings of the 3rd annual conference on Systems, programming, and applications: software for humanity , 2012, pp. 49–52

    2012

  9. [16]

    Identifying overly strong con- ditions in refactoring implementations,

    G. Soares, M. Mongiovi, and R. Gheyi, “Identifying overly strong con- ditions in refactoring implementations,” in 2011 27th IEEE International Conference on Software Maintenance (ICSM) . IEEE, 2011, pp. 173– 182

    2011

  10. [17]

    Generating unit tests for checking refactoring safety,

    G. Soares, R. Gheyi, T. Massoni, M. Cornélio, and D. Cavalcanti, “Generating unit tests for checking refactoring safety,” in Brazilian Symposium on Programming Languages, vol. 1175, 2009, pp. 159–172

    2009

  11. [18]

    Systematic testing of refactoring engines on real software projects,

    M. Gligoric, F. Behrang, Y . Li, J. Overbey, M. Hafiz, and D. Marinov, “Systematic testing of refactoring engines on real software projects,” in ECOOP 2013–Object-Oriented Programming: 27th European Confer- ence, Montpellier, France, July 1-5, 2013. Proceedings 27 . Springer, ...

    2013

  12. [19]

    Refac- toring programs using large language models with few-shot examples,

    A. Shirafuji, Y . Oda, J. Suzuki, M. Morishita, and Y . Watanobe, “Refac- toring programs using large language models with few-shot examples,” in 2023 30th Asia-Pacific Software Engineering Conference (APSEC) . IEEE, 2023, pp. 151–160

    2023

  13. [20]

    Iterative refactoring of real-world open- source programs with large language models,

    J. Choi, G. An, and S. Yoo, “Iterative refactoring of real-world open- source programs with large language models,” in International Sym- posium on Search Based Software Engineering . Springer, 2024, pp. 49–55

    2024

  14. [21]

    How to refactor this code? an exploratory study on developer-chatgpt refactoring conversations,

    E. A. AlOmar, A. Venkatakrishnan, M. W. Mkaouer, C. Newman, and A. Ouni, “How to refactor this code? an exploratory study on developer-chatgpt refactoring conversations,” in Proceedings of the 21st International Conference on Mining Software Repositories , 2024, pp. 202–206

    2024

  15. [22]

    Em-assist: Safe automated ex- tractmethod refactoring with llms,

    D. Pomian, A. Bellur, M. Dilhara, Z. Kurbatova, E. Bogomolov, A. Sokolov, T. Bryksin, and D. Dig, “Em-assist: Safe automated ex- tractmethod refactoring with llms,” in Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, 20...

    2024

  16. [23]

    Compiler testing using template java programs,

    Z. Zang, N. Wiatrek, M. Gligoric, and A. Shi, “Compiler testing using template java programs,” in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022, pp. 1–13

    2022

  17. [24]

    Java jit testing with template extraction,

    Z. Zang, F.-Y . Yu, A. Thimmaiah, A. Shi, and M. Gligoric, “Java jit testing with template extraction,” Proceedings of the ACM on Software Engineering, vol. 1, no. FSE, pp. 1129–1151, 2024

    2024

  18. [26]

    Statistical analysis of refactoring bug reports in eclipse bugzilla,

    E. Lacker, J. Kim, A. Kumar, L. Chandrashekar, S. Paramaiahgari, and J. Howard, “Statistical analysis of refactoring bug reports in eclipse bugzilla,” in 2021 36th IEEE/ACM International Conference on Auto- mated Software Engineering Workshops (ASEW). IEEE, 2021, pp. 9–13

    2021

  19. [27]

    [Online]

    (2024) Pull up method refactoring for method in the inner class fails. [Online]. Available: https://github.com/eclipse-jdt/eclipse.jdt.ui/issues/ 1533

    2024

  20. [28]

    [Online]

    (2024) Add outer class checking to pull up refactoring. [Online]. Available: https://github.com/eclipse-jdt/eclipse.jdt.ui/pull/1590

    2024

  21. [29]

    [Online]

    (2024) [bug][pull up refactoring] pull up refactoring for the method in anonymous class produce uncompilable code. [Online]. Available: https://github.com/eclipse-jdt/eclipse.jdt.ui/issues/1766

    2024

  22. [30]

    Toward understanding compiler bugs in gcc and llvm,

    C. Sun, V . Le, Q. Zhang, and Z. Su, “Toward understanding compiler bugs in gcc and llvm,” in Proceedings of the 25th international sympo- sium on software testing and analysis , 2016, pp. 294–305

    2016

  23. [31]

    A comprehensive study of deep learning compiler bugs,

    Q. Shen, H. Ma, J. Chen, Y . Tian, S.-C. Cheung, and X. Chen, “A comprehensive study of deep learning compiler bugs,” in Proceedings of the 29th ACM Joint meeting on european software engineering conference and symposium on the foundations of software engineering , 2021, pp. 968–980

    2021

  24. [32]

    [Online]

    (2022) Github apis. [Online]. Available: https://docs.github.com/en/rest? apiVersion=2022-11-28

    2022

  25. [33]

    A comprehensive study of autonomous vehicle bugs,

    J. Garcia, Y . Feng, J. Shen, S. Almanee, Y . Xia, and Q. A. Chen, “A comprehensive study of autonomous vehicle bugs,” in Proceedings of the ACM/IEEE 42nd international conference on software engineering , 2020, pp. 385–396

    2020

  26. [34]

    A comprehensive study on deep learning bug characteristics,

    M. J. Islam, G. Nguyen, R. Pan, and H. Rajan, “A comprehensive study on deep learning bug characteristics,” in Proceedings of the 2019 27th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering , 2019, pp. 510– 520

    2019

  27. [35]

    Compat- ibility issues in deep learning systems: Problems and opportunities,

    J. Wang, G. Xiao, S. Zhang, H. Lei, Y . Liu, and Y . Sui, “Compat- ibility issues in deep learning systems: Problems and opportunities,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2023...

    2023

  28. [36]

    Towards automated detection of unethical behavior in open-source software projects,

    H. M. Win, H. Wang, and S. H. Tan, “Towards automated detection of unethical behavior in open-source software projects,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2023, pp. 644– 656

    2023

  29. [37]

    Learning to prioritize test programs for compiler testing,

    J. Chen, Y . Bai, D. Hao, Y . Xiong, H. Zhang, and B. Xie, “Learning to prioritize test programs for compiler testing,” in 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). IEEE, 2017, pp. 700–711

    2017

  30. [38]

    History-driven test program synthesis for jvm testing,

    Y . Zhao, Z. Wang, J. Chen, M. Liu, M. Wu, Y . Zhang, and L. Zhang, “History-driven test program synthesis for jvm testing,” in Proceedings of the 44th International Conference on Software Engineering , 2022, pp. 1133–1144

    2022

  31. [39]

    Selecting initial seeds for better jvm fuzzing,

    T. Gao, J. Chen, D. Wang, Y . Guo, Y . Zhao, and Z. Wang, “Selecting initial seeds for better jvm fuzzing,” arXiv preprint arXiv:2408.08515 , 2024

    2024 arXiv

  32. [40]

    Boosting compiler testing by injecting real-world code,

    S. Li, T. Theodoridis, and Z. Su, “Boosting compiler testing by injecting real-world code,” Proceedings of the ACM on Programming Languages, vol. 8, no. PLDI, pp. 223–245, 2024

    2024

  33. [41]

    [Online]

    (2024) Bug 92519 - [refactoring] inline method - result does not compile. [Online]. Available: https://bugs.eclipse.org/bugs/show_bug. cgi?id=92519

    2024

  34. [42]

    Generalizing from a few examples: A survey on few-shot learning,

    Y . Wang, Q. Yao, J. T. Kwok, and L. M. Ni, “Generalizing from a few examples: A survey on few-shot learning,” ACM computing surveys (csur), vol. 53, no. 3, pp. 1–34, 2020

    2020

  35. [43]

    Large language models are edge-case generators: Crafting unusual programs for fuzzing deep learning libraries,

    Y . Deng, C. S. Xia, C. Yang, S. D. Zhang, S. Yang, and L. Zhang, “Large language models are edge-case generators: Crafting unusual programs for fuzzing deep learning libraries,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1–13

    2024

  36. [44]

    [Online]

    (2024) Openai api. [Online]. Available: https://platform.openai.com/ docs/overview

    2024

  37. [45]

    [Online]

    (2024) Openai api default settings. [Online]. Available: https: //platform.openai.com/docs/api-reference/chat/create

    2024

  38. [46]

    Fowler, Refactoring: improving the design of existing code

    M. Fowler, Refactoring: improving the design of existing code . Addison-Wesley Professional, 2018

    2018

  39. [47]

    Can llm replace stack overflow? a study on robustness and reliability of large language model code generation,

    L. Zhong and Z. Wang, “Can llm replace stack overflow? a study on robustness and reliability of large language model code generation,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 19, 2024, pp. 21 841–21 849

    2024

  40. [48]

    A review on code generation with llms: Application and evaluation,

    J. Wang and Y . Chen, “A review on code generation with llms: Application and evaluation,” in 2023 IEEE International Conference on Medical Artificial Intelligence (MedAI) . IEEE, 2023, pp. 284–289

    2023

  41. [49]

    [Online]

    (2023) Make static refactoring preconditions. [Online]. Available: https://github.com/eclipse-jdt/eclipse.jdt.ui/issues/590

    2023

  42. [50]

    Finding and understanding bugs in c compilers,

    X. Yang, Y . Chen, E. Eide, and J. Regehr, “Finding and understanding bugs in c compilers,” in Proceedings of the 32nd ACM SIGPLAN conference on Programming language design and implementation, 2011, pp. 283–294

    2011

  43. [51]

    Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models,

    Y . Deng, C. S. Xia, H. Peng, C. Yang, and L. Zhang, “Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models,” in Proceedings of the 32nd ACM SIGSOFT interna- tional symposium on software testing and analysis , 2023, pp. 423–435

    2023

  44. [52]

    Smt solver validation empowered by large pre-trained language models,

    M. Sun, Y . Yang, Y . Wang, M. Wen, H. Jia, and Y . Zhou, “Smt solver validation empowered by large pre-trained language models,” in 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2023, pp. 1288–1300

    2023

  45. [53]

    Whitefox: White-box compiler fuzzing empowered by large language models,

    C. Yang, Y . Deng, R. Lu, J. Yao, J. Liu, R. Jabbarvand, and L. Zhang, “Whitefox: White-box compiler fuzzing empowered by large language models,” Proceedings of the ACM on Programming Languages , vol. 8, no. OOPSLA2, pp. 709–735, 2024

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.