REVIEW 4 major objections 5 minor 37 references
ROSE: Transformer-Based Refactoring Recommendation for Architectural Smells
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A transformer fine-tuned on mined refactorings can recommend the correct fix for an architectural smell with 97% accuracy.
desk verdict A competent refactoring-type classifier with an overstated smell-recommendation claim: the mapping from refactoring labels to smell classes is assumed, not validated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing objects are the one-to-one smell-to-refactoring mapping and the corpus built from it: God Class maps to Extract Method, Cyclic Dependency to Move Class, Hub-like Dependency to Pull Up Method, and every historical refactoring mined by RefactoringMiner from 11,149 Java projects is labelled with one of these three classes. CodeT5 and CodeBERT are fine-tuned as classifiers on Java snippets paired with these labels, with CodeT5's encoder feeding a linear classification head. The mapping turns the recommendation problem into a three-class classification problem, and the dataset size—2,086,898 refactoring instances plus 1,006,653 negative samples, balanced by undersampling—is what makes the transformer fine-tuning feasible.
What would settle it
Run a smell detector such as Designite or Arcan on a held-out random sample of snippets labelled Extract Method and check whether each contains a God Class; if a substantial share contain no such smell, then the 97% accuracy reflects refactoring-type classification rather than smell-aware recommendation. A second check is to retrain on project-out splits and see whether the accuracy drop exceeds the paper's reported ten-fold within-corpus figures.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that an encoder–decoder transformer used as a classifier can reliably map a code fragment that contains an architectural smell to the refactoring operation that repairs it, with CodeT5 achieving 97% accuracy and 95% F1 on balanced data. CodeBERT, the encoder-only comparison, reaches 85% accuracy and 85% F1. The paper presents this as the first large-scale evidence that transformer models can close the loop between architectural-smell detection and actionable refactoring recommendation, and it attributes CodeT5's advantage to its encoder–decoder pre-training and its ability to capture structural cues that distinguish refactoring types.
Load-bearing premise
Everything rests on the hand-made assumption that each refactoring label in the training data is the correct repair for a real architectural smell in that snippet; if the labels are just refactoring types from commit history, the classifier may be learning to name refactorings without detecting architecture-level problems.
Editorial extensions
If this is right
- Tool builders can attach the classifier to existing detectors and replace static smell warnings with a suggested refactoring.
- A hierarchical recommender becomes plausible: first select the repair family for a smell, then rank specific refactorings within that family, which would address the residual Extract-versus-Move confusion.
- The released corpus of over two million refactorings enables longitudinal studies of how developers actually repair architectural smells across projects and time.
- The pattern that Pull Up Method is easiest and Extract Method hardest suggests that future work should move beyond the three-smell catalogue to smells such as Unstable Interface and Cyclically-Dependent Abstraction.
- Extending the same pipeline to interactive repair is feasible by coupling the classifier with a patch generator, such as the CodeT5 decoder.
Reading between the lines
- Going beyond the paper: because every training label comes from the hand-made one-to-one mapping rather than from a smell detector, the high accuracy may demonstrate refactoring-type classification without demonstrating that the model detects architectural smells; a project-out evaluation with independently detected smells would settle this.
- Going beyond the paper: the same architecture likely transfers to other smell-to-refactoring pairs; a cheap test is to mine additional canonical mappings, such as Feature Envy to Move Method, and check whether the accuracy gap reappears.
- Going beyond the paper: a direct falsification check would be to hold out entire projects, not just random snippets, and measure whether accuracy survives; the paper's ten-fold cross-validation may still mix code from the same repository across folds.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents ROSE, a pipeline that fine-tunes CodeBERT and CodeT5 as three-class classifiers to assign one of three refactoring labels (Extract Method, Move Class, Pull Up Method) to Java code snippets. The snippets are derived from over two million refactoring instances mined from 11,149 open-source projects in the Aniche et al. corpus, and the authors assert a one-to-one mapping from three architectural smells (God Class, Cyclic Dependency, Hub-like Dependency) to these three refactorings. Reported results show CodeT5 reaching 96.98% accuracy and 95.16% F1 versus CodeBERT's 85.28% accuracy, and the paper claims this demonstrates that transformers can reliably map architectural smells to refactoring operations. The authors release code, data splits, and model checkpoints.
Significance. If the central claim were valid, the work would be a useful large-scale demonstration that pretrained code models can recommend concrete refactorings for architectural smells, complementing existing smell detectors. The main strengths are the substantial dataset, the reproducible experimental setup (fixed seeds, pinned library versions, released artifacts), and the open acknowledgment of several validity threats. However, the current experiment does not actually evaluate architectural-smell detection or the smell-to-refactoring link: the model is trained solely on refactoring-type labels, and the mapping from smells to refactorings is asserted rather than validated. As a result, the results are better characterized as refactoring-type classification, and the paper's advertised contribution is not supported by the evidence. The resource and the framing may still be useful to the community, but the central claim needs major additional validation or a substantial reframing.
major comments (4)
- [III-B and IV-A, IV-C] The load-bearing construct-validity gap is the equivalence between refactoring labels and smell ground truth. Section III-B states the mapping "God Class → Extract Method, Cyclic Dependency → Move Class, Hub-like Dependency → Pull Up Method" as a design decision, but the underlying Aniche et al. corpus labels are refactoring operations detected by RefactoringMiner, not architectural-smell annotations. The paper provides no evidence that a God Class exists before each Extract Method instance, that a dependency cycle exists before each Move Class instance, or that a hub-like dependency exists before each Pull Up Method instance. Consequently, RQ1's answer in Section IV-A ("transformers can reliably map architectural smells to refactoring operations") and RQ3's per-class accuracy analysis ("which architectural smells are most accurately detected") are not supported: the model never receives or predicts a smell label, and the per-class metrics in Section IV-C are metrics for refactoring types, not detected smell classes. Section V-C concedes this as a construct-validity limitation, but the concession does not repair the inference made in the abstract and in the RQ answers.
- [Abstract and IV-A] The abstract claims that CodeT5 "outperforms CodeBERT and traditional baselines," and Section I promises "a comparison of CodeBERT, CodeT5, metric-based, and classical ML baselines," but no traditional baseline results appear anywhere in Section IV. Section III-B mentions that Aniche et al. trained six classical classifiers and reported accuracies above 90%, but the current paper reports no comparison table, no baseline numbers, and no statistical significance tests across the ten folds. The only evidence presented is the CodeBERT-versus-CodeT5 comparison in Table I, which is insufficient to support the claimed margin over state-of-the-art baselines.
- [III-B and III-D] The dataset description is internally inconsistent regarding the negative class. Section III-B states that 1,006,653 non-refactored commits were sampled as negative instances and that random undersampling balanced minority and majority classes, but the classification task, the confusion matrices in Figure 3, and the per-class tables contain only the three positive refactoring classes, with no negative class. The paper also does not explain how the original feature-based Aniche et al. corpus, which contains CK metrics, process metrics, and ownership metrics, is converted into the "Java snippet + refactoring label" format used for transformer fine-tuning, nor how refactoring instances are associated with the specific code fragment fed to the model. This ambiguity affects reproducibility and obscures what the reported accuracy actually measures.
- [III-C and V-C] The claim in Section III-C that "ten-fold cross-validation safeguards against project-specific bias" is not supported by the described methodology. Standard random ten-fold cross-validation can place snippets from the same project in both training and test folds, which likely inflates accuracy if the model memorizes project-specific formatting or refactoring patterns. The authors acknowledge in Section V-C that "independent project-out or cross-repository evaluations are needed," which directly contradicts the earlier safeguarding claim. Either project-aware splitting should be used and reported, or the generalization claim should be substantially weakened.
minor comments (5)
- [IV-C] The RQ3 answer says that Hub-like Dependency is predicted with "≈88%" precision and recall, but Table II reports CodeT5's Pull Up Method precision and recall as 97.2% and 97.4%, and CodeBERT's as 89.5% and 90.7%; the 88% figure matches neither model and should be corrected.
- [IV-B and Table II] There is a typo in Table II's caption: "Table II rovides a detailed per-class comparison" should read "provides."
- [Throughout] The model names are formatted inconsistently as "CodeBERT," "C ODE BERT," "CODE BERT," and "CodeT5"/"CODE T5"; please standardize all occurrences.
- [III-D] Figure 1 is referenced as "Figure 1 outlines the workflow" but the figure is not included in the submitted text; please ensure all referenced figures are present.
- [II] The related work section mentions "Extract Class" as an example refactoring, but the experimental classes are Extract Method, Move Class, and Pull Up Method; the terminology should be made consistent.
Circularity Check
The architectural-smell recommendation claim reduces to refactoring-type classification because the smell labels are defined as refactoring labels by an unvalidated one-to-one mapping.
-
self definitional
[Section III-B (Dataset) and Section IV-C (RQ3)]
"Architectural smells were then linked to canonical refactorings as follows: • God Class → Extract Method. • Cyclic Dependency → Move Class. • Hub-like Dependency → Pull Up Method."
The training and test labels are refactoring operations mined by RefactoringMiner from commit history; the dataset contains no independent architectural-smell ground truth. By defining each smell as a refactoring label via the one-to-one mapping, the model's classification of refactoring types is, by construction, reported as detection and mapping of architectural smells. RQ3's per-class precision/recall for Pull Up Method, Move Class, and Extract Method is therefore a statement about which refactoring label is easiest to classify, not about which smell is most accurately detected: the 'smell class' is the refactoring class. The central claim that transformers 'reliably map architectural smells to refactoring operations' is not tested independently of this definitional link.
full rationale
The transformer evaluation itself is self-contained and reproducible: CodeT5 and CodeBERT are fine-tuned on balanced refactoring labels and achieve high classification accuracy, which is a real empirical result for refactoring-type prediction. However, the paper's stated contribution is architectural-smell remediation, and that part is circular in the specific sense that the smell labels are not measured or detected; they are the refactoring labels renamed through the Section III-B mapping. Since the mapping (God Class to Extract Method, Cyclic Dependency to Move Class, Hub-like Dependency to Pull Up Method) is asserted without validation, the 97% accuracy and 95% F1 support a claim about classifying refactoring operations from code snippets, not about detecting architectural smells or recommending correct repairs. The authors themselves flag this construct-validity limitation in Section V-C and note that independent project-out or cross-repository evaluations are missing. This is a partial circularity of the central claim (score 6), not an intentional deception: the underlying classification result appears valid for what it directly measures. No self-citation or imported uniqueness theorem is involved.
Assumptions & free parameters
free parameters (2)
- Random undersampling ratio =
unspecified
- Sliding window size and overlap =
200 tokens, 50% overlap
assumptions (4)
- ad hoc to paper Every refactoring instance of a given type implies the corresponding architectural smell was present before the refactoring (God Class for Extract Method, Cyclic Dependency for Move Class, Hub-like Dependency for Pull Up Method).
- domain assumption RefactoringMiner's detections are an accurate ground truth for refactoring type labels.
- domain assumption Ten-fold cross-validation without project-level splitting prevents evaluation bias.
- ad hoc to paper Code snippets used as model input are representative of code fragments containing architectural smells.
Cite this review
Pith. "Pith review of ROSE: Transformer-Based Refactoring Recommendation for Architectural Smells." pith.science (2026). https://pith.science/paper/AIWR5QWR
@misc{pith2026250712561,
author = {Pith},
title = {Pith review of: ROSE: Transformer-Based Refactoring Recommendation for Architectural Smells},
year = {2026},
howpublished = {\url{https://pith.science/paper/AIWR5QWR}},
note = {Machine review of arXiv:2507.12561}
}
read the original abstract
Architectural smells such as God Class, Cyclic Dependency, and Hub-like Dependency degrade software quality and maintainability. Existing tools detect such smells but rarely suggest how to fix them. This paper explores the use of pre-trained transformer models--CodeBERT and CodeT5--for recommending suitable refactorings based on detected smells. We frame the task as a three-class classification problem and fine-tune both models on over 2 million refactoring instances mined from 11,149 open-source Java projects. CodeT5 achieves 96.9% accuracy and 95.2% F1, outperforming CodeBERT and traditional baselines. Our results show that transformer-based models can effectively bridge the gap between smell detection and actionable repair, laying the foundation for future refactoring recommendation systems. We release all code, models, and data under an open license to support reproducibility and further research.
Figures
Reference graph
Works this paper leans on
-
[1]
Identifying refactoring opportunities in object-oriented code: A systematic literature review,
J. Al Dallal, “Identifying refactoring opportunities in object-oriented code: A systematic literature review,” Information and Software Tech- nology, vol. 58, pp. 231–249, 2015
work page 2015
-
[2]
Deep learning ap- proaches for bad smell detection: a systematic literature review,
A. Alazba, H. Aljamaan, and M. R. Alshayeb, “Deep learning ap- proaches for bad smell detection: a systematic literature review,” Empiri- cal Software Engineering, vol. 28, 2023, corpusID:258591793. [Online]. Available: https://api.semanticscholar.org/CorpusID:258591793
work page 2023
-
[3]
Enhancing software quality with AI: A transformer-based approach for code smell detection,
I. Ali, S. S. H. Rizvi, and S. H. Adil, “Enhancing software quality with AI: A transformer-based approach for code smell detection,” Applied Sciences, vol. 15, no. 8, p. 4559, 2025
work page 2025
-
[4]
The Effectiveness of Supervised Machine Learning Algorithms in Predicting Software Refactoring,
M. Aniche, E. Maziero, R. Durelli, and V . H. S. Durelli, “The Effectiveness of Supervised Machine Learning Algorithms in Predicting Software Refactoring,” IEEE Transactions on Software Engineering , vol. 48, no. 04, pp. 1432–1450, Apr. 2022. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/TSE.2020.3021736
arXiv 2022
-
[5]
Impact of architectural smells on software performance: an exploratory study,
F. Arcelli Fontana, M. Camilli, D. Rendina, A. G. Taraboi, and C. Trubiani, “Impact of architectural smells on software performance: an exploratory study,” in Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering , ser. EASE ’23. New York, NY , USA: Association for Computing Machinery, 2023, p. 22–31. [Onlin...
arXiv 2023
-
[6]
Arcan: A tool for architectural smells detection,
F. Arcelli Fontana, I. Pigazzini, R. Roveda, D. Tamburri, M. Zanoni, and E. Di Nitto, “Arcan: A tool for architectural smells detection,” in International Workshops on Software Architecture , 2017, pp. 282–285, international Workshops on Software Architecture
work page 2017
-
[7]
Architectural smells detected by tools: a catalogue proposal,
U. Azadi, F. Arcelli Fontana, and D. Taibi, “Architectural smells detected by tools: a catalogue proposal,” in 2019 IEEE/ACM International Conference on Technical Debt (TechDebt) , 2019, pp. 88–97
work page 2019
-
[8]
M. I. Azeem, F. Palomba, L. Shi, and Q. Wang, “Machine learning techniques for code smell detection: A systematic literature review and meta-analysis,” Information and Software Technology, vol. 108, pp. 115– 138, 2019
work page 2019
Show all 37 references
-
[9]
An empirical study on the code refactoring capability of large language models,
J. Cordeiro, S. Noei, and Y . Zou, “An empirical study on the code refactoring capability of large language models,” arXiv preprint arXiv:2411.02320, 2024
2024 arXiv
-
[10]
An empirical study of untangling patterns of two-class dependency cycles,
Q. Feng, S. Liu, H. Ji, X. Ma, and P. Liang, “An empirical study of untangling patterns of two-class dependency cycles,” 2023. [Online]. Available: https://arxiv.org/abs/2306.10599
2023 arXiv
-
[11]
Codebert: A pre-trained model for programming and natural languages,
Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “Codebert: A pre-trained model for programming and natural languages,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Associ...
2020
-
[12]
A review-based comparative study of bad smell detection tools,
E. Fernandes, J. Oliveira, G. Vale, T. Paiva, and E. Figueiredo, “A review-based comparative study of bad smell detection tools,” in Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering . ACM, 2016, pp. 18:1–18:12
2016
-
[13]
Fowler, K
M. Fowler, K. Beck, J. Brant, W. Opdyke, and D. Roberts, Refactoring: Improving the Design of Existing Code . Addison-Wesley, 1999
1999
-
[14]
Graphcodebert: Pre-training code representations with data flow,
D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, J. Yin, D. Jiang, and M. Zhou, “Graphcodebert: Pre-training code representations with data flow,” https://arxiv.org/abs/2009.08366, 2020, arXiv preprint arXiv:2009.08366
2009 arXiv
-
[15]
Refactorscore: Evaluating refactor prone code,
K. Jesse, C. Kuhmuench, and A. Sawant, “Refactorscore: Evaluating refactor prone code,” IEEE Transactions on Software Engineering, 2023, early Access
2023
-
[16]
Coderl: Mastering code generation through pretrained models and deep rein- forcement learning,
H. Le, Y . Wang, A. D. Gotmare, S. Savarese, and S. C. Hoi, “Coderl: Mastering code generation through pretrained models and deep rein- forcement learning,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 35, 2022, pp. 21 314–21 328
2022
-
[17]
Codexglue: A machine learning benchmark dataset for code understanding and generation,
S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. B. Clement, D. Drain, D. Jiang, D. Tang, and G. Li, “Codexglue: A machine learning benchmark dataset for code understanding and generation,” https://arxiv.org/abs/2102.04664, 2021, arXiv preprint arXiv:2102.04664
2021 arXiv
-
[18]
Examining deep learning’s capability to spot code smells: a systematic literature review,
R. Malhotra, B. Jain, and M. Kessentini, “Examining deep learning’s capability to spot code smells: a systematic literature review,” Cluster Computing, vol. 26, pp. 3473–3501, 2023, corpusID:263654376. [On- line]. Available: https://api.semanticscholar.org/CorpusID:263654376
2023
-
[19]
A systematic review on search-based refactoring,
T. Mariani and S. R. Vergilio, “A systematic review on search-based refactoring,” Information and Software Technology , vol. 83, pp. 14–34, 2017
2017
-
[20]
A survey of search-based refactoring for software maintenance,
M. Mohan and D. Greer, “A survey of search-based refactoring for software maintenance,” Journal of Software Engineering Research and Development, vol. 6, no. 1, pp. 3–55, 2018
2018
-
[21]
A systematic mapping study on architectural smells detection,
H. Mumtaz, P. Singh, and K. Blincoe, “A systematic mapping study on architectural smells detection,” Journal of Systems and Software, vol. 173, p. 110885, 2021. [Online]. Available: https: //www.sciencedirect.com/science/article/pii/S0164121220302752
2021
-
[22]
Deep learning-based code refactoring: A review of current knowledge,
P. Naik, S. Nelaballi, V . S. Pusuluri, and D.-K. Kim, “Deep learning-based code refactoring: A review of current knowledge,” SSRN Electronic Journal , 2023, corpusID:254267544. [Online]. Available: https://api.semanticscholar.org/CorpusID:254267544
2023
-
[23]
Mining commit messages to enhance software refactorings recommendation: A machine learning approach,
A. S. Nyamawe, “Mining commit messages to enhance software refactorings recommendation: A machine learning approach,” Machine Learning with Applications , vol. 9, p. 100316, 2022
2022
-
[24]
Multi-objective software architecture refactoring driven by quality attributes,
D. D. Pompeo and M. Tucci, “Multi-objective software architecture refactoring driven by quality attributes,” in 2023 IEEE 20th International Conference on Software Architecture Companion (ICSA- C). IEEE, Mar. 2023, p. 175–178. [Online]. Available: http: //dx.doi.org/10.1109/IC...
2023
-
[25]
A review of code smell mining techniques,
G. Rasool and Z. Arshad, “A review of code smell mining techniques,” Journal of Software: Evolution and Process , vol. 27, no. 11, pp. 867– 895, 2015
2015
-
[26]
A systematic review on the code smell effect,
J. A. M. Santos, J. B. Rocha-Junior, L. C. L. Prates, R. S. do Nascimento, M. F. Freitas, and M. G. de Mendonc ¸a, “A systematic review on the code smell effect,” Journal of Systems and Software , vol. 144, pp. 450–477, 2018
2018
-
[27]
On the evolution and impact of architectural smells—an industrial case study,
D. Sas, P. Avgeriou, and U. Uyumaz, “On the evolution and impact of architectural smells—an industrial case study,” Empirical Software Engineering , vol. 27, no. 86, 2022. [Online]. Available: https://doi.org/10.1007/s10664-022-10132-7
2022 doi
-
[28]
Designite: A software design quality assessment tool,
T. Sharma, P. Mishra, and R. Tiwari, “Designite: A software design quality assessment tool,” in International Workshop on Bringing Ar- chitectural Design Thinking into Developers’ Daily Activities , 2016, pp. 1–4, international Workshop on Bringing Architectural Design Thinkin...
2016
-
[29]
A systematic literature review: Refactoring for disclosing code smells in object oriented software,
S. Singh and S. Kaur, “A systematic literature review: Refactoring for disclosing code smells in object oriented software,” Ain Shams Engineering Journal, vol. 9, no. 4, pp. 2129–2151, 2018
2018
-
[30]
Attention is all you need,
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems , vol. 30. Curran Associates, Inc., 2017. [Online]. Available: https://proceedings.neurips.cc...
2017
-
[31]
Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,
Y . Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” https://arxiv.org/abs/2109.00859, 2021, arXiv preprint arXiv:2109.00859
2021 arXiv
-
[32]
Promises and perils of using transformer-based models for se research,
Y . Xiao, X. Zuo, X. Lu, J. S. Dong, X. Cao, and I. Beschastnikh, “Promises and perils of using transformer-based models for se research,” Neural Networks , vol. 184, p. 107067, 2025. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0893608024009961
2025
-
[33]
A systematic literature review on the code smells datasets and validation mechanisms,
M. Zakeri-Nasrabadi, S. Parsa, E. Esmaili, and F. Palomba, “A systematic literature review on the code smells datasets and validation mechanisms,” ACM Computing Surveys , vol. 55, no. 13s, p. 1–48, Jul
-
[34]
Code smell detection re- search based on pre-training and stacking models,
D. Zhang, S. Song, Y . Zhang, and H. Liu, “Code smell detection re- search based on pre-training and stacking models,” IEEE Latin America Transactions, vol. 22, no. 1, pp. 22–30, 2024
2024
-
[35]
Code bad smells: A review of current knowledge,
M. Zhang, T. Hall, and N. Baddoo, “Code bad smells: A review of current knowledge,” Journal of Software Maintenance and Evolution: Research and Practice , vol. 23, no. 3, pp. 179–202, 2011
2011
-
[36]
Code smell detection based on supervised learning models: A survey,
Y . Zhang, C. Ge, H. Liu, and K. Zheng, “Code smell detection based on supervised learning models: A survey,” Neurocomputing, vol. 565, p. 127014, 2024, dOI:10.1016/j.neucom.2023.127014
2024
-
[2023]
Available: http://dx.doi.org/10.1145/3596908
[Online]. Available: http://dx.doi.org/10.1145/3596908
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.