Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

This paper claims that LLMs identify refactoring motivations that agree with human judgment in 80% of cases but align with literature-based motivations in only 47%, revealing a surface-level focus.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

LLM-generated refactoring motivations agree with expert raters about 80% of the time, align with literature motivations in roughly half of cases, and correlate only weakly with software metrics.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Useful empirical work on LLM-extracted refactoring motivations, but the headline percentages don't reconcile and the ground truth is experts, not developers. the 3 major comments →

arxiv 2509.07763 v1 pith:GFQRV3Y2 submitted 2025-09-09 cs.SE cs.AIcs.PL

What Were You Thinking? An LLM-Driven Large-Scale Study of Refactoring Motivations in Open-Source Projects

Mikel Robredo , Matteo Esposito , Fabio Palomba , Rafael Pe\~naloza , Valentina Lenarduzzi This is my paper
classification cs.SE cs.AIcs.PL
keywords code refactoringrefactoring motivationslarge language modelsempirical software engineeringsoftware metricsmining software repositoriesJava open-source projectsrefactoring recommendation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that large language models can read version-control history and recover why developers refactored code, but only at a surface level. On a sample of 385 refactorings from 114 Java projects, LLM-generated motivations agreed with human expert judgment in about 80% of cases, yet only 47% matched the motivations previously documented in the literature. The authors argue the disagreements are systematic: models reason from local code cues while humans invoke architectural and strategic goals. They also show that standard software metrics correlate only weakly with motivation categories, so metrics alone cannot stand in for intent. If right, refactoring recommendation tools should combine LLM explanations with metric signals rather than rely on either alone.

Core claim

The paper's central discovery is a measured asymmetry: given the refactoring type, its machine-generated description, commit message, and diff, LLMs and human experts agree on the motivation roughly four times out of five (156/198; Cohen's kappa = 0.567), yet only about 47% of LLM motivations match the catalogue of developer-reported motivations from the reference study. A Bowker symmetry test shows the disagreements are not random: humans invoke architectural and strategic considerations, while LLMs lean on local syntax and immediate readability. The LLMs also extended 22% of aligned motivations with new detail and produced entirely novel motivations in 25% of cases, and over 55% of open-co

What carries the argument

The analytical engine is a four-model LLM pipeline: Marco-o1 ('LRM') proposes the motivation; Mistral NeMo, DeepSeek R1, and Microsoft Phi-4 validate it in sequence, with Phi-4 as arbiter; three human experts validate a statistically significant subset (198 pairs) by majority vote. Motivations were extracted with zero-shot chain-of-thought prompting over refactoring type, RefactoringMiner description, commit message, and diff; then open-coded into 14 categories. Metric influence was assessed with Random Forest (MDA, MDG) and XGBoost (information gain), and correlations with Spearman's rho and Kendall's tau under Bonferroni and Benjamini-Hochberg corrections.

Load-bearing premise

The study's results stand on the assumption that three human experts, given only the refactoring type, tool description, commit message, and diff, can recover the original developer's true motivation, and that their majority vote is valid ground truth for judging the LLM.

What would settle it

Ask the developers who performed a random sample of the 385 refactorings to state their actual motivation, then compare those self-reports with both the LLM's motivation and the experts' labels; if self-reports align with LLM output more than with expert inference, or if expert labels do not match developer self-reports, the 80% agreement and the 'struggles with architectural reasoning' conclusion would measure expert inference rather than true developer motivation.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • LLM-derived motivations should be treated as partial, localized explanations, not ground truth; recommendation systems that use them alone will inherit the systematic disagreement with holistic human judgment.
  • Metric-only refactoring recommendation is not viable: after correction only 17 of 574 metric-motivation correlations are significant, all weak, so metrics need LLM or other semantic context.
  • Hybrid systems that feed metrics as context into LLM reasoning are the indicated direction the paper argues for.
  • Because most motivations (over 55%) are pragmatic clarity/simplification goals, refactoring tools should prioritize short-term readability and redundancy reduction over abstract design principles.
  • LLMs can enrich documented motivations (22% of cases) with explicit rationale—readability, testability, naming—useful for documentation, onboarding, and code review.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the 20% LLM-human gap may shrink if prompts include project-level context (issue tracker links, architectural diagrams, historical design decisions); this is testable by an A/B study with enriched prompts on the same 385 observations.
  • Editorial inference: the 47% alignment with the literature may be as much a verdict on the reference catalogue as on LLMs; the 25% novel motivations suggest the 2016 catalogue misses motives that now dominate open-source practice.
  • Editorial inference: because the expert labels are themselves inferences from limited data, the 80% agreement is an upper bound on true motivation recovery; a developer self-report study on the same commits would calibrate both LLM and expert accuracy.
  • Editorial inference: the weak metric correlations suggest current product/process metrics measure change activity rather than intent; new indicators capturing semantic diff intent or naming-quality deltas might correlate more strongly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents a large-scale empirical study of refactoring motivations in 114 open-source Java projects, using four LLMs (Marco-o1, Mistral NeMo, DeepSeek R1, and Phi-4) to infer motivations from refactoring commits identified with RefactoringMiner. The authors sample 385 refactorings, validate LLM outputs with three human experts, compare LLM-derived motivations against the developer-reported motivations of Silva et al. (2016), and open-code the LLM-derived motivations into 14 categories. They then train Random Forest and XGBoost models and compute Spearman/Kendall correlations to assess which product and process metrics are associated with the resulting motivation categories. The headline findings are that LLMs agree with human raters in ~80% of cases, that only 47% of LLM motivations align with literature-based motivations, that 22% of motivations are enriched, and that metric–motivation correlations are statistically weak. The paper concludes that LLMs capture surface-level motivations but struggle with architectural reasoning, and advocates hybrid LLM-plus-metric systems.

Significance. If the central results were internally consistent, the paper would make a useful contribution to empirical software engineering: it extends a well-known dataset (Silva et al.), introduces a reproducible LLM-based pipeline for refactoring-motivation mining, and offers a large catalogue of motivation categories and metric-importance rankings. The availability of a replication package, the use of multiple LLMs with explicit roles, and the human validation protocol are concrete strengths. However, the central quantitative claims are not consistently derivable from the paper's own tables, and the RQ3 analysis is based on LLM-generated motivation labels rather than externally verified developer intent. These issues affect the main contributions as stated, so the results should not be accepted in their current form.

major comments (3)
  1. [Abstract; §3.1–§3.2; Tables 8 and 9] The central alignment percentages are mutually inconsistent. The abstract and the RQ1 result box state that only 47% of LLM-generated RMs aligned with the reference study. However, §3.1 states that 'Based on human annotations, 97 out of 198 RMs (approximately 49%) aligned with those reported in the reference study,' and Table 8 gives the Yes–Yes cell as 97 (48.99%). Section 3.2 and Table 9 report 105 direct matches (53.03%), 44 extensions (22.22%), and 49 non-matches (24.74%). The abstract's 47% is the complement of the Table 9 'Yes' percentage (44+49=93, 93/198=47%), which is not an alignment rate. Since the paper's conclusion that LLMs are 'surface-level' is calibrated to these numbers, the counts must be reconciled and the abstract and result boxes corrected before the claims can be evaluated.
  2. [§2.4.3; §3.3.1–§3.3.3] RQ3 is conducted on motivation categories obtained by open-coding the LLM-generated RMs. The 14 RMCs are therefore derived from the same LLM outputs whose validity RQ1/RQ2 only partially establish. The feature-importance and correlation analyses in §3.3.2 and §3.3.3 consequently describe relationships between software metrics and LLM labels, not between metrics and independently verified developer motivations. Human validation of the open coding does not remove this circularity, since the category definitions themselves were produced by the LLMs. The paper should either re-run RQ3 with human-coded or developer-reported motivations as the dependent variable, or explicitly reframe RQ3 and the discussion as an analysis of LLM-perceived motivations rather than actual developer intentions.
  3. [§2.4.1; §3.1] The 80% agreement and the subsequent qualitative conclusions about LLMs 'struggling with architectural reasoning' rest on agreement between LLMs and three expert raters, not between LLMs and the original developers. The expert raters saw only the RT, RMT description, commit message, and diff, which are exactly the localized cues the paper later argues are insufficient for architectural intent. The dataset used as reference contains the original developers' own written motivations, so a direct LLM-to-developer comparison is possible (or, at minimum, an expert-rater reliability check against the developer statements) and should be reported. Without this, the claim that LLMs miss 'strategic rationales' is an interpretation of expert disagreement rather than a measured property of developer intent.
minor comments (6)
  1. [§3.1, Table 7] The text refers to 'MC (13.64%)' but Table 7 lists 'MovC'; please use consistent abbreviation and define it.
  2. [Table 4] The RFC metric description contains the placeholder '[citeme]'.
  3. [Appendix A.1] Typo: 'checksytle/checkstyle' should be 'checkstyle/checkstyle'.
  4. [§3.1] 156/198 = 78.8%, which is reasonably described as 'approximately 80%' in prose, but the abstract and result boxes present '80%' without the approximation; consider stating the exact fraction.
  5. [§2.4.3] The notation 'H01,02' appears malformed; it should likely read 'H0C' or a similarly defined hypothesis label.
  6. [§3.3.1, Figure 5] The legend for 'unique motivation instances' is unclear: it is not obvious whether grey bars count unique strings, unique subcategories, or something else. Please clarify in the caption.

Circularity Check

0 steps flagged

No significant circularity: central claims are anchored to the external Silva et al. dataset and independent human validation; the RQ3 reliance on LLM-generated categories is a disclosed construct-validity limitation rather than a circular derivation.

full rationale

The paper's central chain—LLM-generated refactoring motivations compared against the externally published Silva et al. motivations with three human experts validating a statistically sampled subset—is not circular. The '80% human agreement' and '47%/53%/49% literature alignment' numbers come from contingency tables and human ratings, even though several percentages are internally inconsistent (a correctness/reporting concern, not a circularity concern). RQ3 uses the LLM-generated and human-validated motivation categories as the dependent variable for feature importance and correlation analysis; this means the metric rankings describe those categories rather than independently verified developer intent. However, the paper does not define the metrics in terms of the motivations, nor does it fit a parameter to a subset and then predict a mathematically forced quantity. The open-coding taxonomy is derived from LLM outputs with human validation on a sample, and the paper explicitly acknowledges in Section 5 (Construct Validity) that assuming LLM feedback is correct is a threat. No load-bearing step reduces to a self-citation: the reference study [65] and the metric-selection works [29,54] are external to the authors, and the authors' own prior works cited are not used as evidence for the central empirical claims. No uniqueness theorem, no ansatz smuggled via citation, and no renaming of a known result as a new derivation are present. The main weaknesses are reporting inconsistencies in the percentage claims and the construct-validity caveat on RQ3, neither of which amounts to circular reasoning by the paper's own construction.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 1 invented entities

The study assumes that commit artifacts contain enough signal to infer developer intent, that the Silva et al. reference is valid ground truth, and that expert raters can stand in for the original developers. The sampling scheme is assumed to represent 13.7M refactorings despite greedy stratification. The LLM temperature is a hand-set parameter affecting all outputs. The RMC taxonomy is generated by the LLM pipeline and used as ground truth for RQ3, which is the main circularity burden.

free parameters (1)
  • LLM temperature = 0.8 (fixed)
    Set by hand for all models to balance creativity and coherence; it directly influences the distribution of LLM-generated motivations and therefore all agreement rates.
axioms (5)
  • domain assumption Commit messages and code diffs provide sufficient signal to infer genuine developer motivation.
    The LLM receives only RT, RMT description, commit message, and diff (Section 2.4.1); if motivation is not encoded in these local artifacts, the entire extraction is invalid.
  • domain assumption Silva et al. developer-reported motivations are a valid reference ground truth.
    RQ1/RQ2 compare LLM motivations against Silva et al.; the reference is treated as ground truth (Section 2.2).
  • domain assumption Human expert raters can reliably reconstruct developers' motivations from the same commit context.
    Human validation uses three experts, not the original developers (Section 2.4.1); the study equates expert judgment with developer intent.
  • domain assumption RefactoringMiner correctly identifies refactorings.
    RMT is used to mine refactorings; the authors acknowledge a small related risk in Section 5.
  • domain assumption The sampled 385 refactorings represent the 13,725,139 mined refactorings.
    Greedy stratified sampling with quotas (Section 2.3.2) is assumed representative for motivation and correlation analysis.
invented entities (1)
  • 14-category refactoring motivation taxonomy (RMC) no independent evidence
    purpose: Serves as the dependent variable for RQ3 feature importance and correlation analysis
    Created by LLM open coding, with human validation only on a subsample; no external benchmark confirms the categories.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of What Were You Thinking? An LLM-Driven Large-Scale Study of Refactoring Motivations in Open-Source Projects." pith.science (2026). https://pith.science/paper/GFQRV3Y2

@misc{pith2026250907763,
  author       = {Pith},
  title        = {Pith review of: What Were You Thinking? An LLM-Driven Large-Scale Study of Refactoring Motivations in Open-Source Projects},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GFQRV3Y2}},
  note         = {Machine review of arXiv:2509.07763}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Context. Code refactoring improves software quality without changing external behavior. Despite its advantages, its benefits are hindered by the considerable cost of time, resources, and continuous effort it demands. Aim. Understanding why developers refactor, and which metrics capture these motivations, may support wider and more effective use of refactoring in practice. Method. We performed a large-scale empirical study to analyze developers refactoring activity, leveraging Large Language Models (LLMs) to identify underlying motivations from version control data, comparing our findings with previous motivations reported in the literature. Results. LLMs matched human judgment in 80% of cases, but aligned with literature-based motivations in only 47%. They enriched 22% of motivations with more detailed rationale, often highlighting readability, clarity, and structural improvements. Most motivations were pragmatic, focused on simplification and maintainability. While metrics related to developer experience and code readability ranked highest, their correlation with motivation categories was weak. Conclusions. We conclude that LLMs effectively capture surface-level motivations but struggle with architectural reasoning. Their value lies in providing localized explanations, which, when combined with software metrics, can form hybrid approaches. Such integration offers a promising path toward prioritizing refactoring more systematically and balancing short-term improvements with long-term architectural goals.

Figures

Figures reproduced from arXiv: 2509.07763 by Fabio Palomba, Matteo Esposito, Mikel Robredo, Rafael Pe\~naloza, Valentina Lenarduzzi.

Figure 1
Figure 1. Figure 1: Workflow diagram of the study design. • H1C. There is a statistically significant correlation between the considered metrics and RMs. Consequently, considering H1C as the accepted hypothesis and therefore the evidence of correla￾tion, we expand our hypotheses as follows: • H1C.1. The identified correlation is negative. • H1C.2. The identified correlation is positive. In this sense, this question aims at id… view at source ↗
Figure 2
Figure 2. Figure 2: Data collection process diagram. reference study [65]. Since their publication, these projects experienced an average increase of over 10,600 commits; e.g., JetBrains/intellij-community increased by 273,044 commits. We provide a table presenting the differences between the number of commits in the studied projects when the reference study was published and for the present study in the replication package (… view at source ↗
Figure 3
Figure 3. Figure 3: Diagram of the adopted sampling strategy. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Data analysis process diagram. whether they agreed or disagreed with the LRM’s motivation, noted the majority decision among the validation models, and identified the models they considered correct. • A second expert repeated the same evaluation independently and documented their level of agreement with the first expert’s judgments. • In cases of disagreement between the first two reviewers, a third expert… view at source ↗
Figure 5
Figure 5. Figure 5: Frequencies for extracted total motivations ( [PITH_FULL_IMAGE:figures/full_fig_p020_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: MDA and MDG importance results for Random Forest model trained with process metrics. [PITH_FULL_IMAGE:figures/full_fig_p022_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: MDA and MDG importance results for Random Forest model trained with product metrics. [PITH_FULL_IMAGE:figures/full_fig_p023_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: MDA and MDG importance results for Random Forest model trained with process and product metrics. [PITH_FULL_IMAGE:figures/full_fig_p024_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Bar chart for process metrics importance from the trained XGB model. [PITH_FULL_IMAGE:figures/full_fig_p024_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Bar chart for product metrics importance from the trained XGB model. [PITH_FULL_IMAGE:figures/full_fig_p025_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Bar chart for product and process metrics importance according to XGBoost. [PITH_FULL_IMAGE:figures/full_fig_p025_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Heat-map representing the Spearman correlation matrix of RMCs with SMs for the uncorrected, [PITH_FULL_IMAGE:figures/full_fig_p026_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Heat-map representing the Kendall correlation matrix of RMCs with SMs for the uncorrected [PITH_FULL_IMAGE:figures/full_fig_p027_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Example of Zero-Shot CoT prompt based on Kojima et al. [ [PITH_FULL_IMAGE:figures/full_fig_p041_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Identifying unique developers in OSS projects: A family of models

    cs.SE 2026-06 unverdicted novelty 5.0

    Creates a ground-truth dataset via LLM-assisted matching and benchmarks classical ML models against an Indel baseline for identifying duplicate developer identities in OSS commit data.

Reference graph

Works this paper leans on

85 extracted references · 67 canonical work pages · cited by 1 Pith paper · 1 internal anchor

  1. [1]

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. 2024. Phi-4 technical report.arXiv preprint arXiv:2412.08905(2024)

    Pith/arXiv arXiv 2024

  2. [2]

    Pouria Alikhanifard and Nikolaos Tsantalis. 2025. A Novel Refactoring and Semantic Aware Abstract Syntax Tree Differencing Tool and a Benchmark for Evaluating the Accuracy of Diff Tools.ACM Transactions on Software Engineering and Methodology34, 2, Article 40 (Jan. 2025), 63 pages. https://doi.org/10.1145/3696002

    doi:10.1145/3696002 2025

  3. [3]

    Vahid Alizadeh, Marouane Kessentini, Mohamed Wiem Mkaouer, Mel Ó Cinnéide, Ali Ouni, and Yuanfang Cai. 2018. An interactive and dynamic search-based approach to software refactoring recommendations.IEEE Transactions on Software Engineering46, 9 (2018), 932–961

    2018

  4. [4]

    goodness of fit

    Theodore W Anderson and Donald A Darling. 1952. Asymptotic theory of certain" goodness of fit" criteria based on stochastic processes.The annals of mathematical statistics(1952), 193–212

    1952

  5. [5]

    2015.Java code metrics calculator (CK)

    Maurício Aniche. 2015.Java code metrics calculator (CK). Available in https://github.com/mauricioaniche/ck/

    2015

  6. [6]

    Paris C Avgeriou, Davide Taibi, Apostolos Ampatzoglou, Francesca Arcelli Fontana, Terese Besker, Alexander Chatzi- georgiou, Valentina Lenarduzzi, Antonio Martini, Athanasia Moschou, Ilaria Pigazzini, et al. 2020. An overview and comparison of technical debt measurement tools.Ieee software38, 3 (2020), 61–71

    2020

  7. [7]

    Yoav Benjamini and Yosef Hochberg. 1995. Controlling the false discovery rate: a practical and powerful approach to multiple testing.Journal of the Royal statistical society: series B (Methodological)57, 1 (1995), 289–300

    1995

  8. [8]

    2013.Convergence of probability measures

    Patrick Billingsley. 2013.Convergence of probability measures. John Wiley & Sons

    work page 2013

  9. [9]

    Albert H Bowker. 1948. A test for symmetry in contingency tables.Journal of the american statistical association43, 244 (1948), 572–574

    work page 1948

  10. [10]

    L. Breiman. 2001. Random forests.Machine learning45, 1 (2001), 5–32

    work page 2001

  11. [11]

    Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A Scalable Tree Boosting System.Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD ’16, 785–794. https://doi.org/10.114 5/2939672.2939785

    arXiv 2016

  12. [12]

    Jacob Cohen. 1960. A coefficient of agreement for nominal scales.Educational and psychological measurement20, 1 (1960), 37–46

    1960

  13. [13]

    Cyril Dalmasso, Philippe Broet, and Thierry Moreau. 2005. A simple procedure for estimating the false discovery rate. Bioinformatics21, 5 (2005), 660–668

    work page 2005

  14. [14]

    2007.Statistics without maths for psychology

    CP Dancey. 2007.Statistics without maths for psychology. Prentice Hall

    work page 2007

  15. [15]

    Joshua Davis, Liesbet Van Bulck, Brigitte N Durieux, and Charlotta Lindvall. 2024. The temperature feature of ChatGPT: modifying creativity for clinical research.JMIR human factors11, 1 (2024), e53559

    work page 2024

  16. [16]

    Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. 2002. A fast and elitist multiobjective genetic algorithm: NSGA-II.IEEE transactions on evolutionary computation6, 2 (2002), 182–197

    2002

  17. [17]

    Matteo Esposito and Davide Falessi. 2023. Uncovering the Hidden Risks: The Importance of Predicting Bugginess in Untouched Methods. In2023 IEEE 23rd International Working Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 277–282

    work page 2023

  18. [18]

    Matteo Esposito, Xiaozhou Li, Sergio Moreschini, Noman Ahmad, Tomas Cerny, Karthik Vaidhyanathan, Valentina Lenarduzzi, and Davide Taibi. 2025. Generative AI for Software Architecture. Applications, Trends, Challenges, and Future Directions.arXiv preprint arXiv:2503.13310(2025)

    Pith/arXiv arXiv 2025

  19. [19]

    Matteo Esposito, Francesco Palagiano, Valentina Lenarduzzi, and Davide Taibi. 2024. Beyond Words: On Large Language Models Actionability in Mission-Critical Risk Analysis. InInternational Symposium on Empirical Software Engineering and Measurement, ESEM 2024, Xavier Franch, Maya Daneva, Silverio Martínez-Fernández, and Luigi Quaranta (Eds.). 517–527

    work page 2024

  20. [20]

    Matteo Esposito, Francesco Palagiano, Valentina Lenarduzzi, and Davide Taibi. 2024. On Large Language Models in Mission-Critical IT Governance: Are We Ready Yet?ICSE-SEIP ’25(2024)

    work page 2024

  21. [21]

    Matteo Esposito, Mikel Robredo, Murali Sridharan, Guilherme Horta Travassos, Rafael Peñaloza, and Valentina Lenarduzzi. 2025. A Call for Critically Rethinking and Reforming Data Analysis in Empirical Software Engineering. arXiv preprint arXiv:2501.12728(2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  22. [22]

    Davide Falessi, Simone Mesiano Laureani, Jonida Çarka, Matteo Esposito, and Daniel Alencar da Costa. 2023. Enhancing the defectiveness prediction of methods and classes via JIT.Empirical Software Engineering28, 2 (2023), 37

    work page 2023

  23. [23]

    Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters.Psychological bulletin76, 5 (1971), 378

    work page 1971

  24. [24]

    1999.Refactoring: Improving the Design of Existing Code

    Martin Fowler. 1999.Refactoring: Improving the Design of Existing Code. Addison-Wesley Professional. ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: September 2025. 36 Mikel Robredo, Matteo Esposito, Fabio Palomba, Rafael Peñaloza, and Valentina Lenarduzzi

    work page 1999

  25. [25]

    Felix Grund, Shaiful Alam Chowdhury, Nick C Bradley, Braxton Hall, and Reid Holmes. 2021. CodeShovel: Constructing method-level source code histories. InInternational Conference on Software Engineering (ICSE). IEEE, 1510–1522

    work page 2021

  26. [26]

    Hong Han, Xiaoling Guo, and Hua Yu. 2016. Variable selection using mean decrease accuracy and mean decrease gini based on random forest. InInternational conference on software engineering and service science (icsess). IEEE, 219–224

    work page 2016

  27. [27]

    Sture Holm. 1979. A simple sequentially rejective multiple test procedure.Scandinavian journal of statistics(1979), 65–70

    work page 1979

  28. [28]

    Yanjie Jiang, Xiaye Chi, Yuxia Zhang, Weixing Ji, Guangjie Li, Weixiao Wang, Yunni Xia, Lu Zhang, and Hui Liu. 2025. Automated Recommendation of Extracting Local Variable Refactorings.ACM Transactions on Software Engineering and Methodology(2025)

    work page 2025

  29. [29]

    Yasutaka Kamei, Emad Shihab, Bram Adams, Ahmed E Hassan, Audris Mockus, Anand Sinha, and Naoyasu Ubayashi

    work page

  30. [30]

    Satnam Kaur, Lalit K Awasthi, and AL Sangal. 2021. A brief review on multi-objective software refactoring and a new method for its recommendation.Archives of Computational Methods in Engineering28 (2021), 3087–3111

    work page 2021

  31. [31]

    Marouane Kessentini and Ali Ouni. 2017. Detecting android smells using multi-objective genetic programming. In 2017 IEEE/ACM 4th International Conference on Mobile Software Engineering and Systems (MOBILESoft). IEEE, 122–132

    work page 2017

  32. [32]

    Dae-Kyoo Kim. 2025. Comparative analysis of design pattern implementation validity in LLM-based code refactoring. Journal of Systems and Software230 (2025), 112519

    work page 2025

  33. [33]

    Miryung Kim, Thomas Zimmermann, and Nachiappan Nagappan. 2012. A field study of refactoring challenges and benefits. InProceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering. 1–11

    work page 2012

  34. [34]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners.Advances in neural information processing systems35 (2022), 22199–22213

    2022

  35. [35]

    Zarina Kurbatova, Ivan Veselov, Yaroslav Golubev, and Timofey Bryksin. 2020. Recommendation of move method refactoring using path-based representation of code. InProceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops. 315–322

    work page 2020

  36. [36]

    Guilherme Lacerda, Fabio Petrillo, Marcelo Pimenta, and Yann Gaël Guéhéneuc. 2020. Code smells and refactoring: A tertiary systematic review of challenges and observations.Journal of Systems and Software167 (2020), 110610

    work page 2020

  37. [37]

    J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data.Biometrics (1977), 159–174

    work page 1977

  38. [38]

    Valentina Lenarduzzi, Terese Besker, Davide Taibi, Antonio Martini, and Francesca Arcelli Fontana. 2021. A systematic literature review on technical debt prioritization: Strategies, processes, factors, and tools.Journal of Systems and Software171 (2021), 110827

    2021

  39. [39]

    Valentina Lenarduzzi, Fabiano Pecorelli, Nyyti Saarimaki, Savanna Lujan, and Fabio Palomba. 2023. A critical comparison on six static analysis tools: Detection, agreement, and precision.Journal of Systems and Software198 (2023), 111575

    work page 2023

  40. [40]

    Bo Liu, Yanjie Jiang, Yuxia Zhang, Nan Niu, Guangjie Li, and Hui Liu. 2025. Exploring the potential of general purpose LLMs in automated software refactoring: an empirical study.Automated Software Engineering32, 1 (2025), 26

    work page 2025

  41. [41]

    Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika12, 2 (1947), 153–157

    work page 1947

  42. [42]

    Prabhaker Mishra, Chandra M Pandey, Uttam Singh, Anshul Gupta, Chinmoy Sahu, and Amit Keshri. 2019. Descriptive statistics and normality tests for statistical data.Annals of cardiac anaesthesia22, 1 (2019), 67–72

    work page 2019

  43. [43]

    Mistral AI team Mistral AI. 2023. Mistral 7B. https://mistral.ai/news/announcing-mistral-7b

    work page 2023

  44. [44]

    Mistral AI team Mistral AI. 2024. Mistral Nemo. https://mistral.ai/news/mistral-nemo

    work page 2024

  45. [45]

    Kristin K Nicodemus. 2011. On the stability and ranking of predictors from random forest variable importance measures. Briefings in bioinformatics12, 4 (2011), 369–373

    work page 2011

  46. [46]

    Ally S Nyamawe, Hui Liu, Zhendong Niu, Wentao Wang, and Nan Niu. 2018. Recommending refactoring solutions based on traceability and code metrics.IEEE Access6 (2018), 49460–49475

    work page 2018

  47. [47]

    OpenAI. 2025. openai — pypi.org. https://pypi.org/project/openai/. [Accessed 28-04-2025]

    work page 2025

  48. [48]

    Fabio Palomba, Andy Zaidman, Rocco Oliveto, and Andrea De Lucia. 2017. An exploratory study on the relationship between changes and refactoring. In2017 IEEE/ACM 25th International Conference on Program Comprehension (ICPC). IEEE, 176–185

    work page 2017

  49. [49]

    Jevgenija Pantiuchina, Fiorella Zampetti, Simone Scalabrino, Valentina Piantadosi, Rocco Oliveto, Gabriele Bavota, and Massimiliano Di Penta. 2020. Why developers refactor source code: A mining-based study.ACM Transactions on Software Engineering and Methodology (TOSEM)29, 4 (2020), 1–30

    work page 2020

  50. [50]

    Anthony Peruma, Steven Simmons, Eman Abdullah AlOmar, Christian D Newman, Mohamed Wiem Mkaouer, and Ali Ouni. 2022. How do i refactor this? An empirical study on refactoring trends and topics in Stack Overflow.Empirical ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: September 2025. What Were You Thinking? An LLM-Driven Large-...

    work page 2022

  51. [51]

    Dorin Pomian, Abhiram Bellur, Malinda Dilhara, Zarina Kurbatova, Egor Bogomolov, Timofey Bryksin, and Danny Dig. 2024. Next-generation refactoring: Combining llm insights and ide capabilities for extract method. InInternational Conference on Software Maintenance and Evolution (ICSME). IEEE, 275–287

    work page 2024

  52. [52]

    Dorin Pomian, Abhiram Bellur, Malinda Dilhara, Zarina Kurbatova, Egor Bogomolov, Andrey Sokolov, Timofey Bryksin, and Danny Dig. 2024. Em-assist: Safe automated extractmethod refactoring with llms. InCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. 582–586

    work page 2024

  53. [53]

    Farhad Pourpanah, Moloud Abdar, Yuxuan Luo, Xinlei Zhou, Ran Wang, Chee Peng Lim, Xi-Zhao Wang, and QM Jonathan Wu. 2022. A review of generalized zero-shot learning methods.IEEE transactions on pattern analysis and machine intelligence45, 4 (2022), 4051–4070

    2022

  54. [54]

    Foyzur Rahman and Premkumar Devanbu. 2013. How, and why, process metrics are better. InInternational Conference on Software Engineering. IEEE Press, 432–441

    work page 2013

  55. [55]

    Mohammad Masudur Rahman and Chanchal K Roy. 2014. An insight into the pull requests of github. InProceedings of the 11th working conference on mining software repositories. 364–367

    work page 2014

  56. [56]

    Gopi Krishnan Rajbahadur, Shaowei Wang, Gustavo A Oliva, Yasutaka Kamei, and Ahmed E Hassan. 2021. The impact of feature importance methods on the interpretation of defect classifiers.IEEE Transactions on Software Engineering48, 7 (2021), 2245–2261

    work page 2021

  57. [57]

    Mikel Robredo, Nyyti Saarimäki, Matteo Esposito, Davide Taibi, Rafael Peñaloza, and Valentina Lenarduzzi. 2025. Evaluating time-dependent methods and seasonal effects in code technical debt prediction.Journal of Systems and Software(2025), 112545

    work page 2025

  58. [58]

    2011.Encyclopedia of machine learning

    Claude Sammut and Geoffrey I Webb. 2011.Encyclopedia of machine learning. Springer Science & Business Media

    work page 2011

  59. [59]

    Margarete Sandelowski. 1995. Sample size in qualitative research.Research in nursing & health18, 2 (1995), 179–183

    work page 1995

  60. [60]

    Simone Scalabrino, Mario Linares-Vásquez, Rocco Oliveto, and Denys Poshyvanyk. 2018. A comprehensive model for code readability.Journal of Software: Evolution and Process30, 6 (2018), e1958

    2018

  61. [61]

    Agnia Sergeyuk, Yaroslav Golubev, Timofey Bryksin, and Iftekhar Ahmed. 2024. Using AI-Based Coding Assistants in Practice: State of Affairs.Perceptions, and Ways Forward10 (2024)

    work page 2024

  62. [62]

    Samuel Sanford Shapiro and Martin B Wilk. 1965. An analysis of variance test for normality (complete samples). Biometrika52, 3/4 (1965), 591–611

    work page 1965

  63. [63]

    Atsushi Shirafuji, Yusuke Oda, Jun Suzuki, Makoto Morishita, and Yutaka Watanobe. 2023. Refactoring programs using large language models with few-shot examples. In2023 30th Asia-Pacific Software Engineering Conference (APSEC). IEEE, 151–160

    work page 2023

  64. [64]

    Danilo Silva, João Silva, Gustavo Jansen De Souza Santos, Ricardo Terra, and Marco Tulio O Valente. 2020. RefDiff 2.0: A Multi-language Refactoring Detection Tool.IEEE Transactions on Software Engineering(2020)

    work page 2020

  65. [65]

    Danilo Silva, Nikolaos Tsantalis, and Marco Túlio Valente. 2016. Why we refactor? confessions of GitHub contributors. InInternational Symposium on Foundations of Software Engineering, FSE, Thomas Zimmermann, Jane Cleland-Huang, and Zhendong Su (Eds.). ACM, 858–870

    work page 2016

  66. [66]

    Minjun Son, Yun-Jae Won, and Sungjin Lee. 2025. Optimizing Large Language Models: A Deep Dive into Effective Prompt Engineering Techniques.Applied Sciences (2076-3417)15, 3 (2025)

    work page 2025

  67. [67]

    Charles Spearman. 1961. The proof and measurement of association between two things.The American Journal of Psychology(1961)

    work page 1961

  68. [68]

    M. A. Stephens. 1974. EDF Statistics for Goodness of Fit and Some Comparisons.J. Amer. Statist. Assoc.69, 347 (1974), 730–737

    work page 1974

  69. [69]

    O’Reilly Media, Inc

    Tom Stuart. 2013.Understanding Computation: From Simple Machines to Impossible Programs. " O’Reilly Media, Inc. "

    work page 2013

  70. [70]

    Gábor Szoke, Gábor Antal, Csaba Nagy, Rudolf Ferenc, and Tibor Gyimóthy. 2014. Bulk fixing coding issues and its effects on software quality: Is it worth refactoring?. In2014 IEEE 14th International Working Conference on Source Code Analysis and Manipulation. IEEE, 95–104

    work page 2014

  71. [71]

    Gábor Szőke, Csaba Nagy, Rudolf Ferenc, and Tibor Gyimóthy. 2014. A case study of refactoring large-scale industrial systems to efficiently improve source code quality. InComputational Science and Its Applications–ICCSA 2014: 14th International Conference, Guimarães, Portugal, June 30–July 3, 2014, Proceedings, Part V 14. Springer, 524–540

    work page 2014

  72. [72]

    Nikolaos Tsantalis, Victor Guana, Eleni Stroulia, and Abram Hindle. 2013. A multidimensional empirical study on refactoring activity.. InCASCON. 132–146

    work page 2013

  73. [73]

    Jeffrey S Vitter. 1985. Random sampling with a reservoir.ACM Transactions on Mathematical Software (TOMS)11, 1 (1985), 37–57

    work page 1985

  74. [74]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al . 2022. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems35 (2022), 24824–24837. ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: September 2025. 38 Mikel Ro...

    work page 2022

  75. [75]

    Ohlsson, and Björn Regnell

    Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, and Björn Regnell. 2012.Experimentation in Software Engineering. Springer

    work page 2012

  76. [76]

    S Paul Wright. 1992. Adjusted p-values for simultaneous inference.Biometrics(1992), 1005–1013

    work page 1992

  77. [77]

    Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou

    work page

  78. [78]

    Yue Yu, Huaimin Wang, Gang Yin, and Charles X Ling. 2014. Reviewer recommender of pull-requests in GitHub. In 2014 IEEE International Conference on Software Maintenance and Evolution. IEEE, 609–612

    work page 2014

  79. [79]

    Yue Yu, Huaimin Wang, Gang Yin, and Tao Wang. 2016. Reviewer recommendation for pull-requests in GitHub: What can we learn from code review and bug assignment?Information and Software Technology74 (2016), 204–218

    work page 2016

  80. [80]

    Yang Zhang, Yanlei Li, Grant Meredith, Kun Zheng, and Xiaobin Li. 2025. Move method refactoring recommendation based on deep learning and LLM-generated information.Information Sciences697 (2025), 121753

    work page 2025

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.