REVIEW 3 major objections 4 minor 43 references
Teaching Code Refactoring Using LLMs
T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read The paper claims that a structured, reflection-required LLM refactoring workflow helped students identify code smells, develop prompt-refinement skills, and treat AI suggestions critically rather than accept them blindly.
desk verdict A genuinely useful teaching-practice report with a reusable scaffold, but the learning claim rests on thin self-report data; worth a referee's time as an Innovative Practice paper, not as evidence of effectiveness. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a four-step, documentation-forcing workflow: (1) identify problematic code through manual inspection and static analysis; (2) apply LLM-generated suggestions; (3) record the prompt, the output, and any manual revisions; (4) commit the change with a reflection log stating the rationale and result. The reflection log is the piece that converts an AI convenience into a learning event: it obliges the student to decide, with reasons, whether to accept, modify, or reject the suggestion. The supporting structure is a set of prepared prompt templates organised by refactoring goal—understanding code, detecting smells, suggesting improvements, step-by-step refactoring, an
What would settle it
Run a controlled comparison across two course sections with the same open-source refactoring project: one section uses the required LLM-plus-reflection workflow, the other refactors with only static-analysis tools and human review. Blindly rate pre- and post-project code on cyclomatic complexity, duplication, coupling, and cohesion, and give both groups the same standardised short test on identifying code smells and planning a refactoring. If the LLM section shows no significantly larger gains on those measures, the claim that LLMs bridge theory and practice is not supported.
Extended reading notes
Core claim
At the centre of the paper is a single pedagogical intervention: make LLM use mandatory but scaffolded. Students refactoring a module of a long-lived open-source project in a graduate software engineering course were required to use an LLM, were given a prompt guide, and had to record every refactoring act—prompt, output, manual revisions—and commit it with a reflection log stating the rationale. The paper's central claim is that this combination produced a specific learning pattern: LLMs were fast at finding local problems such as duplicated code, long methods, and poor names; iterative prompt refinement became a learned skill; and the requirement to justify acceptance or rejection turned t
Load-bearing premise
The load-bearing premise is that the voluntary self-reports of 24 of 46 students, plus the instructor's impressions, are an accurate proxy for actual learning—the paper itself notes (Section V-G) that it has no concrete measurements of how much code quality improved, so if those self-reports are unrepresentative or shaped by what students thought was expected, the central claim does not follow.
Editorial extensions
If this is right
- A corollary is that educators can offer personalised, context-aware feedback on large codebases at a scale that manual review cannot match.
- Prompt refinement becomes a learnable, assessable skill rather than an incidental mechanic of chatbot use.
- LLM tools function as first-pass scanners for localised code smells but not as architects: class-to-class responsibility moves need separate instruction.
- Assessment that focuses only on final code will miss the reported learning gains, which come from students' reasoned accept, modify, or reject decisions.
- Most students in the study said they would use LLMs for refactoring again, implying the practice will follow them into professional work.
Reading between the lines
- The instructor's observation that LLMs avoid moving responsibilities between classes points to a curriculum implication the paper leaves implicit: architectural refactoring should be taught and assessed as a separate skill, with LLMs reserved for tactical cleanup.
- Because the evidence is voluntary self-report from one cohort, the mechanism worth testing next is the reflection log itself: a controlled comparison with and without mandatory documentation would isolate whether the learning gain comes from the LLM or from the forced reasoning.
- A direct check of transferability would run the same workflow with open-weight or self-hosted models; the study's students mostly used commercial chat assistants, so the reported benefits may partly reflect those specific tools.
- A multi-cohort deployment with blind before/after ratings of code quality and a standardised refactoring-knowledge test would turn the exploratory finding into a causal claim.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This Innovative Practice paper reports on integrating LLM-assisted code refactoring into a graduate software-engineering course. Students worked on the Expertiza OSS project and were required to use LLMs with structured prompt templates, keeping reflection logs and committing refactoring changes. The evaluation draws on pre/post surveys (24 of 46 enrolled students, 52% response rate), open-ended survey responses, student reflection logs, and instructor reviews. Thematic findings indicate that students found LLMs helpful for local refactorings (e.g., renaming, extracting methods), that prompt refinement became a learned skill, and that students mostly engaged critically with LLM outputs. The paper claims these results suggest LLMs can bridge theoretical and practical learning of refactoring, while explicitly acknowledging the absence of concrete code-quality measurements.
Significance. If the central claim were strongly supported, the paper would offer a useful, transferable approach for teaching refactoring on real codebases, with practical prompt templates in Appendix A that other instructors can adopt. The authors are to be credited for being unusually transparent in Section V-G, explicitly noting the lack of code-quality measurements, the single-cohort design, and the small sample. They also appropriately frame the analysis as exploratory. However, the evidence is dominated by voluntary self-reports and informal qualitative coding; there is no comparison group, no human-developer baseline, and no objective learning or code-quality metric. The paper is therefore useful as a teaching-experience report, but the stronger claims about 'deeper understanding' and 'bridging theoretical and practical learning' go beyond what the data can currently support.
major comments (3)
- [Section IV-E, RQ1] RQ1 asks how effective LLMs are at identifying refactoring opportunities 'compared to human developers,' but no comparison data are presented. The answer in Section IV-E reports only student perceptions (e.g., 'fast and helpful starting point'), and the survey instrument described in Section III-D contains no human-developer baseline or expert judgement on the same code modules. Thus RQ1 is not actually answered, and contribution (2) in the Introduction is overstated. Please reframe RQ1 as 'perceived effectiveness' or add a comparison condition, or remove the comparison wording.
- [Section V-G] The abstract and conclusion claim that LLMs 'bridge theoretical and practical learning' and support 'deeper understanding' of maintainability and refactoring. The only supporting evidence is self-reported survey responses (n=24/46, Section IV-B) and reflection logs. Section V-G explicitly concedes: 'we do not have concrete measurements of how much code quality improved.' There is no pre-/post-test of refactoring skill, no comparison condition, and no check for nonresponse bias or demand characteristics (the instructors designed both the intervention and the evaluation). The central learning claim is therefore not distinguishable from prior ability, regression to the mean, or positive response bias. Please temper the abstract and conclusion to 'students perceived gains' unless objective outcome measures are added.
- [Section IV, Evaluation and Results] The qualitative analysis is described as informal: two members 'independently reviewed responses and discussed recurring patterns, but no formal coding rubric was used.' No coding scheme, codebook, or inter-rater reliability is reported. Consequently, the themes in Section IV-D (e.g., 'Improved refactoring with LLMs,' 'Critical thinking in code reviews') are not systematically derived. This is acceptable for an exploratory study, but Section V-A's sentence about 'greatest gains in learning' overreaches relative to the analytic method. Please describe the theme-extraction procedure in more detail and include supporting quotes or tallies, or reduce the strength of the interpretive claims.
minor comments (4)
- [Section IV-C3] Figure 3 caption ends with a double period: 'correctness..' Also, 'Errors or Unexpected behaviors' is not consistent with other caption capitalization. Please fix.
- [References] References [20] and [21] use inconsistent quotation marks and an en-dash style different from the rest of the bibliography; [25] is a non-archival blog/whitepaper and could be replaced with a peer-reviewed source or clearly marked as a practitioner report.
- [Section III-C] Step 4 of the workflow says students commit changes 'with a reflection log,' but the mechanics (e.g., a required text file, a commit-message convention, or a separate repository) are not described. A sentence on how the reflection logs were collected would improve reproducibility.
- [Appendix A] The phrase '23 Gang of Four patterns' should be '23 Gang of Four design patterns' for clarity, though this is a presentation issue only.
Circularity Check
No circularity: the paper draws no formal predictions from fitted inputs; the self-report evidence base is a validity limitation, not a circular derivation.
full rationale
The paper makes no formal derivation chain of the kind targeted by circularity analysis. The intervention is evaluated through voluntary post-surveys (Section IV-B, n=24 of 46), reflection logs, and instructor reviews; the paper explicitly frames the analysis as exploratory and interpretive: 'The open-ended responses and student reflections were analyzed using informal qualitative methods. Two members of the research team independently reviewed responses and discussed recurring patterns, but no formal coding rubric was used. As such, we describe our analysis as exploratory and interpretive' (Section IV). The central claim that LLMs can 'bridge theoretical and practical learning' is an inference from self-reports, not a quantity fitted to data and then renamed as a prediction. There are no fitted parameters, no equations whose output is an input by construction, and no load-bearing self-citations: none of the cited sources is the authors' own prior work. The acknowledged absence of objective code-quality measurement ('we do not have concrete measurements of how much code quality improved', Section V-G) weakens evidential strength but is not circularity; it is a stated threat to validity. Future-work plans for quantitative metrics (Section VII) further confirm that the current qualitative conclusions are not definitionally forced by the method. Accordingly, no circular step can be quoted and exhibited, and the score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption Self-reported survey responses and reflection logs are a valid proxy for learning and refactoring skill gains
- domain assumption Informal thematic analysis by two team members without a formal coding rubric reliably identifies recurring patterns
- domain assumption Instructor reviews of refactored projects are a sufficient check on correctness and student reasoning
Cite this review
Pith. "Pith review of Teaching Code Refactoring Using LLMs." pith.science (2026). https://pith.science/paper/DSKH26UL
@misc{pith2026250809332,
author = {Pith},
title = {Pith review of: Teaching Code Refactoring Using LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/DSKH26UL}},
note = {Machine review of arXiv:2508.09332}
}
read the original abstract
This Innovative Practice full paper explores how Large Language Models (LLMs) can enhance the teaching of code refactoring in software engineering courses through real-time, context-aware feedback. Refactoring improves code quality but is difficult to teach, especially with complex, real-world codebases. Traditional methods like code reviews and static analysis tools offer limited, inconsistent feedback. Our approach integrates LLM-assisted refactoring into a course project using structured prompts to help students identify and address code smells such as long methods and low cohesion. Implemented in Spring 2025 in a long-lived OSS project, the intervention is evaluated through student feedback and planned analysis of code quality improvements. Findings suggest that LLMs can bridge theoretical and practical learning, supporting a deeper understanding of maintainability and refactoring principles.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
A survey of software refactoring,
T. Mens and T. Tourw ´e, “A survey of software refactoring,” IEEE Trans. Softw. Eng., vol. 30, no. 2, pp. 126–139, 2004
work page 2004
-
[2]
A comparative study of manual and automated refactorings,
S. Negara, N. Chen, M. Vakilian, R. E. Johnson, and D. Dig, “A comparative study of manual and automated refactorings,” in ECOOP 2013 – Object-Oriented Programming , pp. 552–576, 2013
work page 2013
-
[3]
Peer review in programming courses: Benefits, implementations, and experiences,
A. Bosu, J. Carver, M. Guadagno, B. Wilson, D. Bassett, and C. Parnin, “Peer review in programming courses: Benefits, implementations, and experiences,” in IEEE/ACM 42nd Int. Conf. on Software Engineering Education and Training , pp. 21–32, 2020
work page 2020
-
[4]
How we refactor, and how we know it,
E. Murphy-Hill, C. Parnin, and A. P. Black, “How we refactor, and how we know it,” IEEE Trans. Softw. Eng. , vol. 38, no. 1, pp. 5–18, 2012
work page 2012
-
[5]
CodeRL: Mastering code generation through pretrained models and deep reinforcement learning,
S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blondel, et al., “CodeRL: Mastering code generation through pretrained models and deep reinforcement learning,” in Advances in Neural Information Processing Systems, pp. 11214–11227, 2022
work page 2022
- [6]
-
[7]
Why we refactor? Confes- sions of GitHub contributors,
D. Silva, N. Tsantalis, and M. T. Valente, “Why we refactor? Confes- sions of GitHub contributors,” in Proc. 24th ACM SIGSOFT Int. Symp. Foundations Softw. Eng., pp. 858–870, 2016
work page 2016
-
[8]
Exploring the impact of inter-smell relations on software maintainability: An empirical study,
A. Yamashita and L. Moonen, “Exploring the impact of inter-smell relations on software maintainability: An empirical study,” in Proc. 35th Int. Conf. Softw. Eng. , pp. 682–691, 2013
work page 2013
Show all 43 references
-
[9]
Do they really smell bad? A study on developers’ perception of bad code smells,
F. Palomba, G. Bavota, M. Di Penta, R. Oliveto, and A. De Lucia, “Do they really smell bad? A study on developers’ perception of bad code smells,” in IEEE Int. Conf. Softw. Maintenance Evolution , pp. 101–110, 2014
2014
-
[10]
Teaching object oriented modelling using concept maps,
J. B ¨orstler and M. T. G¨orzen, “Teaching object oriented modelling using concept maps,” in Proc. 6th Baltic Sea Conf. Computing Education Research, pp. 131–133, 2006
2006
-
[11]
An empirical investigation on the use of automated refactoring tools,
A. Peruma, K. Newman, M. W. Mkaouer, A. Ouni, and F. Palomba, “An empirical investigation on the use of automated refactoring tools,” in Proc. 3rd Int. Workshop Refactoring , pp. 30–37, 2019
2019
-
[12]
Preparing the next generation of software engineers to tackle cloud- based challenges,
C. Szabo, N. Falkner, A. Soria, D. Pagu, I. Bownds, and K. Szabo, “Preparing the next generation of software engineers to tackle cloud- based challenges,” in IEEE 43rd Annual Computer Software and Appli- cations Conference, pp. 238–243, 2019
2019
-
[13]
Work for us? An empirical study on student volunteers contributing to OSS projects,
G. Pinto, F. Castor, R. Bonif ´acio, and M. Rebouc ¸as, “Work for us? An empirical study on student volunteers contributing to OSS projects,” in IEEE/ACM 40th Int. Conf. Software Engineering: Software Engineering Education and Training , pp. 1–10, 2018
2018
-
[14]
JDeodorant: Iden- tification and removal of type-checking bad smells,
N. Tsantalis, T. Chaikalis, and A. Chatzigeorgiou, “JDeodorant: Iden- tification and removal of type-checking bad smells,” in 12th European Conf. Software Maintenance and Reengineering , pp. 329–331, 2008
2008
-
[15]
On the diffusion of test smells in automatically generated test code: An empirical study,
F. Palomba, D. Di Nucci, A. Panichella, R. Oliveto, and A. De Lucia, “On the diffusion of test smells in automatically generated test code: An empirical study,” in 9th Int. Workshop Search-Based Software Testing , pp. 5–14, 2016
2016
-
[16]
Evaluating large language models trained on code,
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. O. de Almeida, et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021
2021 arXiv
-
[17]
An empirical study on learning bug-fixing patches in the wild via neural machine translation,
M. Tufano, C. Watson, G. Bavota, M. Di Penta, M. White, and D. Poshyvanyk, “An empirical study on learning bug-fixing patches in the wild via neural machine translation,” ACM Trans. Softw. Eng. Methodol., vol. 28, no. 4, pp. 1–29, 2019
2019
-
[18]
Pythia: AI- assisted code completion system,
A. Svyatkovskiy, Y . Zhao, S. Fu, and N. Sundaresan, “Pythia: AI- assisted code completion system,” in Proc. 25th ACM SIGKDD Int. Conf. Knowledge Discovery & Data Mining , pp. 2727–2735, 2019
2019
-
[19]
On the use of large language models for computer science education,
D. Denny, E. Prather, B. A. Becker, Z. Papamichail, C. Liew, and J. Luxton-Reilly, “On the use of large language models for computer science education,” in Proc. 54th ACM Technical Symp. Computer Science Education, pp. 965–971, 2023
2023
-
[20]
Fakhoury, D
S. Fakhoury, D. Roy, A. Hassan, and V . Arnaoudova, ”Improving source code readability: Theory and practice,” in Proc. 27th Int. Conf. Program Comprehension, 2019, pp. 25-35
2019
-
[21]
S. Lau, I. Drosos, J. Markel, and P. J. Guo, ”The design space of AI coding assistants,” Proc. ACM Hum.-Comput. Interact., vol. 6, no. CSCW2, pp. 1-34, 2022
2022
-
[22]
Zhang, Y
Y . Zhang, Y . Li, G. Meredith, K. Zheng, and X. Li, ”Move method refactoring recommendation based on deep learning and LLM-generated information,” School Information Science and Engineering, Hebei Uni- versity of Science and Technology, Shijiazhuang, China, 2025
2025
-
[23]
Zhang, Z
Z. Zhang, Z. Xing, X. Ren, Q. Lu, and X. Xu, ”Refactoring to Pythonic Idioms: A Hybrid Knowledge-Driven Approach Leveraging Large Language Models,” Proc. ACM Softw. Eng., vol. 1, no. FSE, Article 50, pp. 1-22, 2024
2024
-
[24]
Thinking like a developer? Comparing the attention of humans with neural models of code,
A. Mastropaolo, L. Pascarella, E. Aghajani, D. Falessi, and G. Bavota, “Thinking like a developer? Comparing the attention of humans with neural models of code,” in Proc. 43rd Int. Conf. Software Engineering , 2021, pp. 1656–1668
2021
-
[25]
Refactoring vs Refuctoring Whitepaper,
J. Warden, “Refactoring vs Refuctoring Whitepaper,” [Online]. Available: https://jessewarden.com/2024/01/ refactoring-vs-refuctoring-whitepaper.html, 2024
2024
-
[26]
An empirical study on the potential of LLMs in automated software refactoring,
Anonymous, “An empirical study on the potential of LLMs in automated software refactoring,” arXiv preprint arXiv:2411.04444v1 , 2024
2024 arXiv
-
[27]
How effective are neural networks for fixing security vulnerabilities,
Y . Wu, N. Jiang, H. V . Pham, T. Lutellier, J. Davis, L. Tan, P. Babkin, and S. Shah, “How effective are neural networks for fixing security vulnerabilities,” in Proc. 32nd ACM SIGSOFT Int. Symp. Software Testing and Analysis, 2023, pp. 1282–1294
2023
-
[28]
An empirical study of pre-trained model-based code refactoring,
X. Ren, S. Ding, X. Chen, W. Zou, J. Guo, and J. Qin, “An empirical study of pre-trained model-based code refactoring,” IEEE Trans. Softw. Eng., vol. 49, no. 5, pp. 2587–2603, 2023. APPENDIX A TIPS FOR USING LLM S TO REFACTOR CODE A. General strategy When refactoring code, the...
2023
-
[29]
Can you explain what this class does step by step?
Understanding the existing code: • “Can you explain what this class does step by step?” • “Identify all dependencies in this class and how they interact.” • “Summarize the core logic of this class in simpler terms.”
-
[30]
What potential code smells do you see in this class?
Detecting code smells: • “What potential code smells do you see in this class?” • “Does this class follow the Single Responsibility Princi- ple?” • “Are there any redundant database queries or performance bottlenecks?”
-
[31]
Refactor this class to improve readability and maintain- ability
Suggesting improvements: • “Refactor this class to improve readability and maintain- ability.” • “Can you split this class into smaller sub-classes without changing behavior?” • “Rewrite this code using design patterns like the 23 Gang of Four patterns.”
-
[32]
Refactor this function gradually, explaining each step and ensuring correctness
Step-by-step refactoring: • “Refactor this function gradually, explaining each step and ensuring correctness.” • “Rewrite this method to be more modular while keeping the same input-output behavior.” • “Can you reduce code duplication in this file while keeping the logic intact?”
-
[33]
Provide test cases to verify the correctness of the refac- tored class
Validating and testing refactored code: • “Provide test cases to verify the correctness of the refac- tored class.” • “Check if the new implementation maintains the same functionality as the original.” • “Are there any edge cases that this class does not handle?” C. Specific r...
-
[34]
Here’s an example of a refactored method: [Exam- ple]. Now, refactor this method in a similar way: [Method to refactor]
One-Shot Prompting: Provide a single example of a refactored method and ask the LLM to refactor a similar method: “Here’s an example of a refactored method: [Exam- ple]. Now, refactor this method in a similar way: [Method to refactor]”
-
[35]
Here are three examples of refactored methods: [Examples]. Using these as guidelines, refactor the following method: [Method to refactor]
Few-Shot Prompting: Supply multiple examples of refactored code to guide the LLM: “Here are three examples of refactored methods: [Examples]. Using these as guidelines, refactor the following method: [Method to refactor]”
-
[36]
Chain-of-Thought Prompting: Guide the LLM through a step-by-step refactoring process: “Let’s refactor this method step by step:
-
[37]
Identify any code smells in the method
-
[38]
Suggest how to address each code smell
-
[39]
Rewrite the method, explaining each change
-
[40]
Now, start with step 1 for this method: [Method to refactor]” D
Verify that the new method maintains the orig- inal functionality. Now, start with step 1 for this method: [Method to refactor]” D. Additional examples of prompting
-
[41]
Refactor only one class/method at a time and explain why the change is needed
Incremental refactoring: • “Refactor only one class/method at a time and explain why the change is needed.” • “List the top three improvements for this file and imple- ment them step by step.”
-
[42]
How does this class fit into the larger project? Suggest improvements in that context
Context-aware refactoring: • “How does this class fit into the larger project? Suggest improvements in that context.” • “Given that this project uses [Rails/Node.js], how can this class follow best practices?”
-
[43]
Modify this code to follow Ruby on Rails best prac- tices
Style guide compliance: • “Modify this code to follow Ruby on Rails best prac- tices.” • “Ensure this class adheres to SOLID principles.”
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.