Pith. sign in

REVIEW 4 major objections 5 minor 55 references

This paper claims that a structured refactoring prompt template lets developers reach a concise, accurate, and satisfactory ChatGPT response in about one turn, versus a median of three and a mean above thirteen turns for unguided prompts.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-04 21:17 UTC pith:LVWB577C

load-bearing objection Useful taxonomy and prompt template, but the headline claim about accuracy and satisfaction isn't supported by the metrics; the evaluation is in-sample and the stats table is garbled. the 4 major comments →

arxiv 2509.08090 v1 pith:LVWB577C submitted 2025-09-09 cs.SE

ChatGPT for Code Refactoring: Analyzing Topics, Interaction, and Effective Prompts

Eman Abdullah AlOmar , Luo Xu , Sofia Martinez , Anthony Peruma , Mohamed Wiem Mkaouer , Christian D. Newman , Ali Ouni This is my paper
classification cs.SE
keywords code refactoringChatGPTprompt engineeringLLMrefactoring taxonomydeveloper-AI interactionthematic analysisprompt template
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks what developers actually want when they ask ChatGPT to refactor code, and whether a deliberately structured prompt can get them there faster. Mining 715 refactoring conversations drawn from 29,778 shared ChatGPT prompts, it builds a 25-theme taxonomy of refactoring topics under Quality, Objective, Testing, and Design, and shows where ChatGPT signals trouble through apology phrases and success through affirmations. The central practical claim is that its refactoring prompt template—with explicit role, working set, context, task, steps, output format, example, and clarification fields—produces usable refactorings in roughly one interaction, compared with a median of three and a mean above thirteen turns for the original developer prompts. If true, prompt structure, not model capability alone, is a lever for making LLM refactoring practical.

Core claim

The paper's discovery is that refactoring conversations with ChatGPT are patterned enough to be classified and engineered. From 715 interactions it derives a two-tier taxonomy with four root categories—Quality, Objective, Testing, and Design—and 25 sub-themes. It then observes that ChatGPT's refactoring responses contain systematic apologetic phrases for introduced bugs, misunderstood code, missed updates, and bad practices, as well as affirmative phrases such as 'certainly' and 'absolutely' marking quality improvements or refactoring extensions. Finally, it constructs a refactoring prompt template whose role, working set, context, task, steps, output format, example, and clarification field

What carries the argument

The central object is the refactoring prompt template: a structured, fill-in-the-blank instruction block that assigns ChatGPT a role, supplies the working set, context, and constraints, specifies the refactoring task and intent, lists steps to analyze, apply strategies, give multiple solutions, validate behavior, and comment/document, and demands a specific output format with test cases. It carries the argument by turning unstructured developer requests into the exact information the paper hypothesizes ChatGPT needs, which is why it can reach the same refactoring result in one turn. The 25-theme taxonomy is the supporting classification machinery that identifies what topics those prompts mus

Load-bearing premise

The load-bearing premise is that fewer turns and shorter prompts or responses equal accurate and satisfactory refactoring; the paper assumes this equivalence without comparing template output to the developer's final code or collecting satisfaction ratings, and it designs and tests the template on the same 715 conversations.

What would settle it

Run the template on a held-out sample of refactoring requests, compile and test the generated code, diff it against the developers' merged refactorings, and collect developer satisfaction judgments; if the one-turn output is not at least as correct and satisfactory as the original multi-turn conversation, the central claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Using the template should let a developer obtain an acceptable refactoring in a single ChatGPT turn, cutting the average interaction cost from roughly 13.6 turns to 1.45.
  • The 25-theme taxonomy gives researchers and tool builders a standard vocabulary for labeling refactoring requests and for auditing whether an LLM response stays on the requested topic.
  • Apology phrases in ChatGPT output can serve as a practical red flag: they cluster around introduced bugs, misunderstood intent, missed updates, and bad practices, telling developers where to re-prompt.
  • Even with the template, ChatGPT frequently skips required test cases and multiple solution versions and mishandles markdown and commit-message formatting, so one-shot success is not uniform across refactoring tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the template was designed and evaluated on the same 715 conversations, the one-turn advantage is likely optimistic; a held-out set of refactoring conversations would be needed to rule out overfitting to those examples.
  • The evaluation never executes or diffs the generated code, so the words 'accurate and satisfactory' are an extrapolation; comparing one-shot template output against the developer's merged refactoring with test execution and developer ratings would directly test the claim.
  • If turn count is a proxy for quality, the large effect size justifies treating prompt design as a first-class software-engineering artifact; a cross-LLM replication would show whether the gain comes from the template or from ChatGPT's particular behavior.
  • The taxonomy's Objective and Design themes align with refactoring types that static-analysis-backed tools handle well, suggesting a hybrid workflow: use the template for exploration and use IDE-integrated refactoring tools for validation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper analyzes 715 developer-ChatGPT refactoring interactions drawn from the DevGPT dataset. It makes three contributions: (1) a thematic taxonomy of 25 refactoring-related conversation topics organized under four root themes (Quality, Objective, Testing, Design), produced through manual coding with a reported disagreement rate of 6%; (2) a qualitative analysis of ChatGPT's apology and affirmation phrases, categorized by scenarios such as bug introduction, misunderstanding, and best-practice violations; and (3) a structured refactoring prompt template, evaluated on the same DevGPT conversations by comparing the number of turns, prompt length, and response length against the original developer prompts. The abstract claims that the prompt template enables 'concise, accurate, and satisfactory responses with minimal interactions.' The paper includes a replication package and a discussion of LLM-generated code unreliability.

Significance. If the main claims were fully supported, the paper would be a useful addition to the growing literature on LLM-assisted refactoring: the taxonomy could inform future studies of developer-AI refactoring dialogue, and a validated prompt template would have practical value. The authors are also transparent about the limitations of ChatGPT's output, and they provide a replication artifact, which is a strength. However, the central quantitative claim about the prompt template's effectiveness is not supported by the evidence presented: the evaluation measures only interaction effort, not correctness or user satisfaction, and the evaluation is performed on the same conversations used to design the template. The taxonomy and RQ2 analysis are plausible qualitative contributions, but the lack of reported frequencies for apology/affirmation categories weakens their evidentiary value. Thus the paper's significance is contingent on either substantially revising the claims or adding missing validation.

major comments (4)
  1. [Abstract and Section III-C (RQ3)] The claim that the prompt template yields 'concise, accurate, and satisfactory responses' is not supported by the metrics in Table II. RQ3 measures only number of turns, prompt length, and response length; it does not compare the generated code against the developer's final code, check behavior preservation, or collect satisfaction ratings. The paper's own Discussion (Takeaway #3) states that ChatGPT 'often fails to follow the entire prompt,' skips test cases, produces hallucinations, and generates invalid Git commit messages. These admissions directly contradict 'accurate and satisfactory.' Please either temper the claim to 'fewer interactions' or add a correctness/satisfaction evaluation (e.g., diff-based comparison with the DevGPT ground truth, human ratings, or automated tests).
  2. [Section II-C (Prompt Construction and Evaluation)] The prompt template is constructed by manually reviewing the DevGPT conversations (Section II-C, Step #1) and then evaluated by 'reproducing the corresponding developer's prompt from DevGPT' on the same conversations (Step #2). This is an in-sample evaluation: the template is tested on the very data used to design it. The observed reduction in turns could partly reflect overfitting to the test conversations. Please validate the template on a held-out set of refactoring requests or an external benchmark, or explicitly frame the result as a proof-of-concept on the training distribution.
  3. [Section III-C / Table II] Table II is critically hard to interpret: the row and column alignments are corrupted (e.g., the values for 'Prompt length' and 'Response length' appear to merge with adjacent columns), making the p-values and Cliff's delta values unverifiable. Since the statistical comparison in RQ3 is one of the paper's main quantitative results, the table must be corrected and the reported effect sizes should be reconciled with the displayed distributions (e.g., the number-of-turns mean of 13.58 vs. median 3 suggests a heavily skewed distribution; please clarify how 'large (0.56)' is computed and whether it is consistent with the reported medians).
  4. [Section III-B (RQ2)] RQ2 identifies apology and affirmation categories but reports no counts, percentages, or inter-rater reliability for these classifications. The reader cannot tell how prevalent each scenario is (e.g., how often ChatGPT apologizes for bug introduction vs. lack of understanding) or whether the proposed subcategories are comprehensive. Please report the distribution of categories and, ideally, a measure of annotation agreement, as is standard for qualitative coding studies.
minor comments (5)
  1. [Abstract and Section I] The abstract says 'text mining 715 refactoring-related interactions,' but the methodology is primarily manual thematic analysis. Please clarify the role of automated text mining versus manual coding.
  2. [Section II-B] The conflict resolution rate of 6% is reported, but no Cohen's kappa or equivalent agreement coefficient is given. Adding one would strengthen confidence in the taxonomy's reliability.
  3. [Figure 2] The taxonomy figure is difficult to read because the sub-themes under each root category are not visually distinguished. Please format the figure so that the 25 sub-themes are clearly associated with their four root themes.
  4. [Section IV (Discussion)] Takeaway #1 calls the prompt template a 'successful prompt' and a 'guideline,' but Takeaway #3 documents several frequent failures. Please align these statements and indicate explicitly what 'successful' means in light of the observed limitations.
  5. [References] Some references are incomplete (e.g., [3] lacks a publication venue; [10] is a bare URL). Please ensure all citations follow the journal's style.

Circularity Check

2 steps flagged

RQ3's 'fewer turns' result is built into the one-shot evaluation protocol and the template was fitted to the same DevGPT conversations used for evaluation, so the abstract's 'accurate and satisfactory' claim is not independently supported.

specific steps
  1. self definitional [Section III-C (RQ 3), Table II; abstract]
    "We evaluate the efficiency of the template by assessing its ability to reproduce in one shot the exact refactored code that took developers several rounds of prompts to reach it. ... Looking at Table II, we found that our prompts differ (i.e., fewer turns (µ=1.45), shorter prompt (µ=4855.52), and shorter response (µ=6975.71)) from original developer’s prompts."

    The template is run as a single one-shot prompt, so its number of turns is 1 (or about 1.45 when clarification is requested) by experimental construction. Comparing this fixed one-turn condition to DevGPT's multi-turn developer prompts and then reporting 'fewer turns' with a large effect size is not an empirical finding about prompt effectiveness; the outcome is embedded in the definition of the test. The abstract's 'minimal interactions' claim restates the one-shot setup rather than measuring it.

  2. fitted input called prediction [Section II-C Step #1-#2 and Section III-C]
    "Then, we started by manually reviewing the unique set of prompts in which developers initiated a conversation with ChatGPT about refactoring. Based on our observations, we recognized that there is a certain format that leads to a shorter interaction between ChatGPT and developers. ... Consequently, we construct our first prompt. ... After constructing the prompt, we evaluated it by reproducing the corresponding developer's prompt from DevGPT, which serves as our ground truth."

    The template is manually fitted to the same DevGPT conversations that later serve as the ground truth for RQ3. The designers explicitly looked for a format that leads to shorter interactions, refined the prompt on these examples, and then 'evaluated' it on those same examples. No held-out or external set is used, so the reported reductions in turns and lengths are an in-sample fit, not an independent prediction. The paper's own limitation statement that effectiveness still needs to be assessed using other LLMs does not address this in-sample evaluation.

full rationale

The primary circularity is in RQ3. The template's 'fewer turns' is guaranteed by the one-shot evaluation protocol (the authors run the template once and count that as one turn), so the comparison to multi-turn DevGPT conversations reports a difference that is built into the experimental setup. Compounding this, the template was manually constructed from the same DevGPT conversations used for evaluation, so the measured reductions are in-sample rather than predictive. The abstract's 'accurate and satisfactory' wording goes beyond the three metrics actually tested (turns, prompt length, response length); no correctness or satisfaction measure is reported, and Discussion Takeaway #3 concedes the model often fails to follow the prompt, skips tests, and hallucinates. I treat that as a validity/overclaim issue rather than an additional circularity. The RQ1 taxonomy and RQ2 apology/affirmation analysis are descriptive and independent; citation [14] to the authors' prior study is used only as design motivation and is not the basis for the effectiveness claim. Overall, the central template-effectiveness result is partially circular due to the one-shot construction and in-sample fitting, so the score is 6.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No numeric parameters are fitted; this is a qualitative empirical study with statistical comparisons. The main assumptions are about dataset representativeness, keyword filtering, coding reliability, and the validity of the in-sample evaluation. No new physical or theoretical entities are introduced.

axioms (4)
  • domain assumption DevGPT snapshot provides a representative sample of developer-ChatGPT refactoring interactions.
    The entire analysis is built on conversations from DevGPT; the paper does not verify the representativeness of this corpus beyond noting it aggregates GitHub artifacts.
  • domain assumption Prompts containing the keyword 'refactor' are a valid and sufficient filter for refactoring-related interactions.
    This filter is used in data curation (Section II-A) and the paper acknowledges it may miss synonymous terms, a validity threat.
  • domain assumption Manual thematic coding by two authors is a reliable measure of topics and of ChatGPT's behavior.
    The taxonomy in RQ1 and the apology/affirmation categories in RQ2 rest entirely on subjective labeling; no inter-rater reliability statistic is reported.
  • domain assumption The developer's eventual resolved code in each DevGPT conversation is a valid ground truth for evaluating the prompt template.
    RQ3 treats the original conversation outcome as the reference, but does not check whether the one-shot output is behaviorally equivalent to that code, only that fewer turns and lengths are needed.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of ChatGPT for Code Refactoring: Analyzing Topics, Interaction, and Effective Prompts." pith.science (2026). https://pith.science/paper/LVWB577C

@misc{pith2026250908090,
  author       = {Pith},
  title        = {Pith review of: ChatGPT for Code Refactoring: Analyzing Topics, Interaction, and Effective Prompts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LVWB577C}},
  note         = {Machine review of arXiv:2509.08090}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large Language Models (LLMs), such as ChatGPT, have become widely popular and widely used in various software engineering tasks such as refactoring, testing, code review, and program comprehension. Although recent studies have examined the effectiveness of LLMs in recommending and suggesting refactoring, there is a limited understanding of how developers express their refactoring needs when interacting with ChatGPT. In this paper, our goal is to explore interactions related to refactoring between developers and ChatGPT to better understand how developers identify areas for improvement in code, and how ChatGPT addresses developers' needs. Our approach involves text mining 715 refactoring-related interactions from 29,778 ChatGPT prompts and responses, as well as the analysis of developers' explicit refactoring intentions.

Figures

Figures reproduced from arXiv: 2509.08090 by Ali Ouni, Anthony Peruma, Christian D. Newman, Eman Abdullah AlOmar, Luo Xu, Mohamed Wiem Mkaouer, Sofia Martinez.

Figure 1
Figure 1. Figure 1: An illustration of how a developer interacts with ChatGPT and how the generated outcomes have been included in a [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: A taxonomy of the ChatGPT refactoring topics. principle. On the other hand, ChatGPT’s responses about the GRASP principles offer guidelines for assigning responsi￾bilities to classes such as low-coupling and high-cohesion. Additionally, it discussed refactoring changes such as method composition, feature move, and class generalization. This indicates that ChatGPT asked developers to strive to design classe… view at source ↗
Figure 3
Figure 3. Figure 3: Chatbot interaction in the context of refactoring. • Bugs introduction. ChatGPT apologizes when its refac￾toring suggestions introduce bugs or errors into the code. For example, it may suggest a change that inadvertently breaks the functionality of the program. This type of apology is crucial as it reflects the model’s recognition of its mistake and its impact on the code’s functionality. • Lack of underst… view at source ↗
Figure 4
Figure 4. Figure 4: Refactoring prompt template. 7 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 43 canonical work pages · 1 internal anchor

  1. [1]

    Artificial intelligence vs. software engineers: An empirical study on performance and efficiency using chatgpt,

    N. Nathalia, A. Paulo, and C. Donald, “Artificial intelligence vs. software engineers: An empirical study on performance and efficiency using chatgpt,” inProceedings of the 33rd Annual International Conference on Computer Science and Software Engineering, pp. 24–33, 2023

    work page 2023

  2. [2]

    Towards human-bot collaborative software architecting with chatgpt,

    A. Ahmad, M. Waseem, P. Liang, M. Fahmideh, M. S. Aktar, and T. Mikkonen, “Towards human-bot collaborative software architecting with chatgpt,” inProceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering, pp. 279–285, 2023

    work page 2023

  3. [3]

    You augment me: Exploring chatgpt-based data augmentation for semantic code search,

    Y . Wanga, L. G. E. Shic, W. C. J. Chena, W. Z. M. W. H. Lie, and H. Z. Z. L. Z. Zhenga, “You augment me: Exploring chatgpt-based data augmentation for semantic code search,”

    work page

  4. [4]

    A prompt pattern catalog to enhance prompt engineering with chatgpt,

    J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. El- nashar, J. Spencer-Smith, and D. C. Schmidt, “A prompt pattern catalog to enhance prompt engineering with chatgpt,”arXiv preprint arXiv:2302.11382, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    The potential use of chatgpt for debugging and bug fixing,

    M. A. Haque and S. Li, “The potential use of chatgpt for debugging and bug fixing,”EAI Endorsed Transactions on AI and Robotics, vol. 2, no. 1, pp. e4–e4, 2023

    work page 2023

  6. [6]

    The scope of chatgpt in software engineering: A thorough investigation,

    W. Ma, S. Liu, W. Wang, Q. Hu, Y . Liu, C. Zhang, L. Nie, and Y . Liu, “The scope of chatgpt in software engineering: A thorough investigation,”arXiv preprint arXiv:2305.12138, 2023

    Pith/arXiv arXiv 2023

  7. [7]

    Investigating code generation performance of chat-gpt with crowd- sourcing social data,

    Y . Feng, S. Vanam, M. Cherukupally, W. Zheng, M. Qiu, and H. Chen, “Investigating code generation performance of chat-gpt with crowd- sourcing social data,” inProceedings of the 47th IEEE Computer Software and Applications Conference, pp. 1–10, 2023

    work page 2023

  8. [8]

    From copilot to pilot: Towards ai supported software development,

    R. Pudari and N. A. Ernst, “From copilot to pilot: Towards ai supported software development,”arXiv preprint arXiv:2303.04142, 2023

    Pith/arXiv arXiv 2023

  9. [9]

    Is chatgpt a good translator? a preliminary study,

    W. Jiao, W. Wang, J.-t. Huang, X. Wang, and Z. Tu, “Is chatgpt a good translator? a preliminary study,”arXiv preprint arXiv:2301.08745, 2023. 9

    Pith/arXiv arXiv 2023

  10. [10]

    https://futurism.com/the-byte/github-92-percent-programmers-using-ai

    work page

  11. [11]

    Automatic code summarization via chatgpt: How far are we?,

    W. Sun, C. Fang, Y . You, Y . Miao, Y . Liu, Y . Li, G. Deng, S. Huang, Y . Chen, Q. Zhang,et al., “Automatic code summarization via chatgpt: How far are we?,”arXiv preprint arXiv:2305.12865, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    https://github.com/UIUC-Chatbot/ai-ta-backend/issues/57

    work page

  13. [13]

    An empirical study on developers shared conversa- tions with chatgpt in github pull requests and issues,

    H. Hao, K. A. Hasan, H. Qin, M. Macedo, Y . Tian, S. H. Ding, and A. E. Hassan, “An empirical study on developers shared conversa- tions with chatgpt in github pull requests and issues,”arXiv preprint arXiv:2403.10468, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    How to refactor this code? an exploratory study on developer-chatgpt refactoring conversations,

    E. A. AlOmar, A. Venkatakrishnan, M. W. Mkaouer, C. Newman, and A. Ouni, “How to refactor this code? an exploratory study on developer-chatgpt refactoring conversations,” inProceedings of the 21st International Conference on Mining Software Repositories, pp. 202–206, 2024

    work page 2024

  15. [15]

    Fowler, K

    M. Fowler, K. Beck, J. Brant, W. Opdyke, and d. Roberts,Refactoring: Improving the Design of Existing Code. Boston, MA, USA: Addison- Wesley Longman Publishing Co., Inc., 1999

    work page 1999

  16. [16]

    A survey of software refactoring,

    T. Mens and T. Tourw ´e, “A survey of software refactoring,”IEEE Transactions on software engineering, vol. 30, no. 2, pp. 126–139, 2004

    2004

  17. [17]

    Next-generation refactoring: Combining llm insights and ide capabilities for extract method,

    D. Pomian, A. Bellur, M. Dilhara, Z. Kurbatova, E. Bogomolov, T. Bryksin, and D. Dig, “Next-generation refactoring: Combining llm insights and ide capabilities for extract method,” in2024 IEEE Inter- national Conference on Software Maintenance and Evolution (ICSME), pp. 275–287, IEEE, 2024

    work page 2024

  18. [18]

    Exploring the potential of general purpose llms in automated software refactoring: an empirical study,

    B. Liu, Y . Jiang, Y . Zhang, N. Niu, G. Li, and H. Liu, “Exploring the potential of general purpose llms in automated software refactoring: an empirical study,”Automated Software Engineering, vol. 32, no. 1, p. 26, 2025

    2025

  19. [19]

    https://smilevo.github.io/self-affirmed-refactoring/

    work page

  20. [20]

    Devgpt: Studying developer-chatgpt conversations,

    T. Xiao, C. Treude, H. Hata, and K. Matsumoto, “Devgpt: Studying developer-chatgpt conversations,” in2024 IEEE/ACM 21st International Conference on Mining Software Repositories (MSR), pp. 227–230, IEEE, 2024

    work page 2024

  21. [21]

    Gathering refactoring data: a comparison of four methods,

    E. Murphy-Hill, A. P. Black, D. Dig, and C. Parnin, “Gathering refactoring data: a comparison of four methods,” inProceedings of the 2nd Workshop on Refactoring Tools, pp. 1–5, 2008

    work page 2008

  22. [22]

    Can refactoring be self-affirmed? an exploratory study on how developers document their refactoring activities in commit messages,

    E. A. AlOmar, M. W. Mkaouer, and A. Ouni, “Can refactoring be self-affirmed? an exploratory study on how developers document their refactoring activities in commit messages,” inInternational Workshop on Refactoring-accepted. IEEE, 2019

    work page 2019

  23. [23]

    A preliminary investigation of self- admitted refactorings in open source software (s),

    Z. Di, B. Li, Z. Li, and P. Liang, “A preliminary investigation of self- admitted refactorings in open source software (s),” inInternational Conferences on Software Engineering and Knowledge Engineering, vol. 2018, pp. 165–168, KSI Research Inc. and Knowledge Systems Institute Graduate School, 2018

    work page 2018

  24. [24]

    On the relation of refactorings and software defect prediction,

    J. Ratzinger, T. Sigmund, and H. C. Gall, “On the relation of refactorings and software defect prediction,” inProceedings of the 2008 International Working Conference on Mining Software Repositories, MSR ’08, (New York, NY , USA), pp. 35–38, ACM, 2008

    work page 2008

  25. [25]

    Toward the automatic classification of self-affirmed refactoring,

    E. A. AlOmar, M. W. Mkaouer, and A. Ouni, “Toward the automatic classification of self-affirmed refactoring,”Journal of Systems and Software, vol. 171, p. 110821, 2021

    work page 2021

  26. [26]

    Refactoring practices in the context of modern code review: An industrial case study at xerox,

    E. A. AlOmar, H. AlRubaye, M. W. Mkaouer, A. Ouni, and M. Kessen- tini, “Refactoring practices in the context of modern code review: An industrial case study at xerox,” in2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 348–357, IEEE, 2021

    work page 2021

  27. [27]

    Recommended steps for thematic synthesis in software engineering,

    D. S. Cruzes and T. Dyba, “Recommended steps for thematic synthesis in software engineering,” in2011 international symposium on empirical software engineering and measurement, pp. 275–284, IEEE, 2011

    work page 2011

  28. [28]

    Why we refactor? confessions of github contributors,

    D. Silva, N. Tsantalis, and M. T. Valente, “Why we refactor? confessions of github contributors,” inProceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2016, (New York, NY , USA), pp. 858–870, ACM, 2016

    work page 2016

  29. [29]

    A Lot of Talk and a Badge: An Exploratory Analysis of Personal Achievements in GitHub

    F. Calefato, L. Quaranta, and F. Lanubile, “A lot of talk and a badge: An empirical analysis of personal achievements in github,”arXiv preprint arXiv:2303.14702, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  30. [30]

    Deterministic automatic refactoring at scale,

    J. Gehring, “Deterministic automatic refactoring at scale,” in2023 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 541–546, IEEE, 2023

    work page 2023

  31. [31]

    Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design,

    J. White, S. Hays, Q. Fu, J. Spencer-Smith, and D. C. Schmidt, “Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design,” inGenerative AI for Effective Software Development, pp. 71–108, Springer, 2024

    work page 2024

  32. [32]

    Refac- toring programs using large language models with few-shot examples,

    A. Shirafuji, Y . Oda, J. Suzuki, M. Morishita, and Y . Watanobe, “Refac- toring programs using large language models with few-shot examples,” arXiv preprint arXiv:2311.11690, 2023

    Pith/arXiv arXiv 2023

  33. [33]

    Exploring chatgpt’s code refactoring capabilities: An empirical study,

    K. DePalma, I. Miminoshvili, C. Henselder, K. Moss, and E. A. AlOmar, “Exploring chatgpt’s code refactoring capabilities: An empirical study,” Expert Systems with Applications, vol. 249, p. 123602, 2024

    work page 2024

  34. [34]

    Context-enhanced llm- based framework for automatic test refactoring,

    Y . Gao, X. Hu, X. Yang, and X. Xia, “Context-enhanced llm- based framework for automatic test refactoring,”arXiv preprint arXiv:2409.16739, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Iterative refactoring of real-world open- source programs with large language models,

    J. Choi, G. An, and S. Yoo, “Iterative refactoring of real-world open- source programs with large language models,” inInternational Sympo- sium on Search Based Software Engineering, pp. 49–55, Springer, 2024

    work page 2024

  36. [36]

    ismell: Assembling llms with expert toolsets for code smell detection and refactoring,

    D. Wu, F. Mu, L. Shi, Z. Guo, K. Liu, W. Zhuang, Y . Zhong, and L. Zhang, “ismell: Assembling llms with expert toolsets for code smell detection and refactoring,” inProceedings of the 39th IEEE/ACM In- ternational Conference on Automated Software Engineering, pp. 1345– 1357, 2024

    work page 2024

  37. [37]

    Improved program repair methods using refactoring with gpt models,

    R. Ishizue, K. Sakamoto, H. Washizaki, and Y . Fukazawa, “Improved program repair methods using refactoring with gpt models,” inPro- ceedings of the 55th ACM Technical Symposium on Computer Science Education V . 1, pp. 569–575, 2024

    work page 2024

  38. [38]

    Refactorbench: Evaluating stateful reasoning in language agents through code,

    D. Gautam, S. Garg, J. Jang, N. Sundaresan, and R. Z. Moghad- dam, “Refactorbench: Evaluating stateful reasoning in language agents through code,” inNeurIPS 2024 Workshop on Open-World Agents, 2024

    work page 2024

  39. [39]

    One-to-one or one-to-many? suggesting extract class refactoring opportunities with intra-class dependency hypergraph neural network,

    D. Cui, Q. Wang, Y . Zhao, J. Wang, M. Wei, J. Hu, L. Wang, and Q. Li, “One-to-one or one-to-many? suggesting extract class refactoring opportunities with intra-class dependency hypergraph neural network,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 1529–1540, 2024

    work page 2024

  40. [40]

    Three heads are better than one: Suggesting move method refactoring opportunities with inter-class code entity dependency en- hanced hybrid hypergraph neural network,

    D. Cui, J. Wang, Q. Wang, P. Ji, M. Qiao, Y . Zhao, J. Hu, L. Wang, and Q. Li, “Three heads are better than one: Suggesting move method refactoring opportunities with inter-class code entity dependency en- hanced hybrid hypergraph neural network,” inProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineer- ing, pp. 745–757, 2024

    work page 2024

  41. [41]

    Move method refactoring recommendation based on deep learning and llm-generated information,

    Y . Zhang, Y . Li, G. Meredith, K. Zheng, and X. Li, “Move method refactoring recommendation based on deep learning and llm-generated information,”Information Sciences, vol. 697, p. 121753, 2025

    work page 2025

  42. [42]

    Preference-guided refactored tuning for retrieval augmented code gen- eration,

    X. Gao, Y . Xiong, D. Wang, Z. Guan, Z. Shi, H. Wang, and S. Li, “Preference-guided refactored tuning for retrieval augmented code gen- eration,” inProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, pp. 65–77, 2024

    work page 2024

  43. [43]

    Copilot-in-the-loop: Fixing code smells in copilot-generated python code using copilot,

    B. Zhang, P. Liang, Q. Feng, Y . Fu, and Z. Li, “Copilot-in-the-loop: Fixing code smells in copilot-generated python code using copilot,” inProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, pp. 2230–2234, 2024

    work page 2024

  44. [44]

    Refactoring to pythonic idioms: A hybrid knowledge-driven approach leveraging large language models,

    Z. Zhang, Z. Xing, X. Ren, Q. Lu, and X. Xu, “Refactoring to pythonic idioms: A hybrid knowledge-driven approach leveraging large language models,”Proceedings of the ACM on Software Engineering, vol. 1, no. FSE, pp. 1107–1128, 2024

    work page 2024

  45. [45]

    W. J. Conover,Practical nonparametric statistics, vol. 350. John Wiley & Sons, 1998

    work page 1998

  46. [46]

    Dominance statistics: Ordinal analyses to answer ordinal questions,

    N. Cliff, “Dominance statistics: Ordinal analyses to answer ordinal questions,”Psychological Bulletin, vol. 114, no. 3, p. 494, 1993

    work page 1993

  47. [47]

    Appropriate statistics for ordinal level data,

    J. Romano, J. Kromrey, J. Coraggio, and J. Skowronek, “Appropriate statistics for ordinal level data,” inProceedings of the Annual Meeting of the Florida Association of Institutional Research, pp. 1–3, 2006

    work page 2006

  48. [48]

    https://chat.openai.com/share/e9f4664e-50a5-40c1-8604-befe89a2dd36

    work page

  49. [49]

    https://chat.openai.com/share/dbd163f7-aa90-4351-a9a7-a2deb906120e

    work page

  50. [50]

    Prompt programming for large language models: Beyond the few-shot paradigm,

    L. Reynolds and K. McDonell, “Prompt programming for large language models: Beyond the few-shot paradigm,” p. 10, 2021

    work page 2021

  51. [51]

    https://chatgpt.com/share/73e56b34-fb0d-4056-bfcc-daf800b5d213

    work page

  52. [52]

    How is chatgpt’s behavior changing over time?,

    L. Chen, M. Zaharia, and J. Zou, “How is chatgpt’s behavior changing over time?,”Harvard Data Science Review, vol. 6, no. 2, 2024

    work page 2024

  53. [53]

    Together we go further: Llms and ide static analysis for extract method refactoring,

    D. Pomian, A. Bellur, M. Dilhara, Z. Kurbatova, E. Bogomolov, T. Bryksin, and D. Dig, “Together we go further: Llms and ide static analysis for extract method refactoring,”arXiv preprint arXiv:2401.15298, 2024

    Pith/arXiv arXiv 2024

  54. [54]

    Unveiling chatgpt’s usage in open source projects: A mining-based study,

    R. Tufano, A. Mastropaolo, F. Pepe, O. Dabi ´c, M. Di Penta, and G. Bavota, “Unveiling chatgpt’s usage in open source projects: A mining-based study,”arXiv preprint arXiv:2402.16480, 2024

    Pith/arXiv arXiv 2024

  55. [55]

    How we refactor, and how we know it,

    E. Murphy-Hill, C. Parnin, and A. P. Black, “How we refactor, and how we know it,”IEEE Transactions on Software Engineering, vol. 38, pp. 5–18, Jan 2012. 10

    work page 2012

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.