{"id":"6caca599-4e00-4e78-afa6-9ca5398515f6","arxiv_id":"2507.12561","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A CodeT5 classifier reaches 96.9% accuracy at labeling snippets with three refactoring types, but the paper's smell-to-refactoring mapping is assumed, not tested.","lead":"The authors fine-tune CodeBERT and CodeT5 to classify code snippets into three refactorings and report 96.9% accuracy for CodeT5. They present this as recommending fixes for architectural smells, but the smell labels are assigned by a fixed mapping rather than detected or validated.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Label validity is the load-bearing gap: refactoring-type labels are assumed to equal smell ground truth, so 97% accuracy may not support the architectural-smell recommendation claim.","rationale":"The reader's weakest_assumption precisely identifies the issue: the hand-made one-to-one mapping in Section III-B makes refactoring labels serve as smell labels without any smell ground truth. My own reading confirms this is the most load-bearing concern. The main empirical result (CodeT5 97% accuracy) is reported as evidence for 'reliably map architectural smells to refactoring operations,' but the dataset construction never verifies that the refactoring instances actually resolve the corresponding architectural smell. Section III-B describes repository selection, refactoring extraction, and feature engineering, with no smell detection step; the smell mapping is asserted as a definition. Section V-C concedes construct-validity limitations and the need for project-out evaluation. The paper claims open science and reproducibility, which is credit where due, but artifact availability does not repair the label-validity gap. The reader's REJECT verdict at MODERATE confidence is appropriate: the central contribution is not supported by the experimental design as written, while no fraud or dishonesty is suggested. There is no internally inconsistent argument; the issue is an unvalidated assumption that is load-bearing for the stated contribution. My proposed concrete test is feasible because the authors release code, models, and data, and it would settle whether the concern lands: if smell prevalence in the labeled snippets is low, the headline claim fails; if it is high, my concern is resolved and the verdict could be reconsidered. I agree with the reader's weakest_assumption, and I recommend the verdict remain UNCHANGED because the reader's analysis already captured this concern.","tokens_in":10609,"tokens_out":1543,"duration_ms":14307,"concrete_test":"Run the released corpus (GitHub: archsmell transformers-66F8) through a smell detector such as Designite or Arcan on the pre-refactoring snippets for each class, and compute the fraction of Extract Method snippets that contain a God Class, Move Class snippets that contain a Cyclic Dependency, and Pull Up Method snippets that contain a Hub-like Dependency. If the fractions are far below 100%, the accuracy numbers measure refactoring-type classification, not smell-to-refactoring recommendation; also report per-class accuracy after filtering out snippets that fail the smell check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that transformers reliably map architectural smells to refactoring operations. For that claim to hold, each training snippet must actually exhibit the smell that the paired refactoring is said to repair. The pipeline in Section III-B does not establish this: the smell-to-refactoring mapping (God Class to Extract Method, Cyclic Dependency to Move Class, Hub-like Dependency to Pull Up Method) is asserted, not validated. The original Aniche et al. dataset labels refactoring operations detected by RefactoringMiner, not architectural smells. The paper presents no evidence that a God Class exists before an Extract Method instance, that a dependency cycle exists before a Move Class instance, or that a hub exists before a Pull Up Method instance. The high accuracy and F1 therefore support a claim about classifying refactoring types from code snippets, not about detecting architectural smells or recommending correct repairs. The authors themselves concede in Section V-C that the one-to-one mapping is a construct-validity limitation and that independent project-out evaluation is missing. The strongest claim in Section IV-A overstates what the experiment can show, and the reader's REJECT verdict is consistent with this gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ROSE, a pipeline that fine-tunes CodeBERT and CodeT5 as three-class classifiers to assign one of three refactoring labels (Extract Method, Move Class, Pull Up Method) to Java code snippets. The snippets are derived from over two million refactoring instances mined from 11,149 open-source projects in the Aniche et al. corpus, and the authors assert a one-to-one mapping from three architectural smells (God Class, Cyclic Dependency, Hub-like Dependency) to these three refactorings. Reported results show CodeT5 reaching 96.98% accuracy and 95.16% F1 versus CodeBERT's 85.28% accuracy, and the paper claims this demonstrates that transformers can reliably map architectural smells to refactoring operations. The authors release code, data splits, and model checkpoints.","tokens_in":10938,"tokens_out":4493,"duration_ms":52185,"significance":"If the central claim were valid, the work would be a useful large-scale demonstration that pretrained code models can recommend concrete refactorings for architectural smells, complementing existing smell detectors. The main strengths are the substantial dataset, the reproducible experimental setup (fixed seeds, pinned library versions, released artifacts), and the open acknowledgment of several validity threats. However, the current experiment does not actually evaluate architectural-smell detection or the smell-to-refactoring link: the model is trained solely on refactoring-type labels, and the mapping from smells to refactorings is asserted rather than validated. As a result, the results are better characterized as refactoring-type classification, and the paper's advertised contribution is not supported by the evidence. The resource and the framing may still be useful to the community, but the central claim needs major additional validation or a substantial reframing.","major_comments":[{"comment":"The load-bearing construct-validity gap is the equivalence between refactoring labels and smell ground truth. Section III-B states the mapping \"God Class → Extract Method, Cyclic Dependency → Move Class, Hub-like Dependency → Pull Up Method\" as a design decision, but the underlying Aniche et al. corpus labels are refactoring operations detected by RefactoringMiner, not architectural-smell annotations. The paper provides no evidence that a God Class exists before each Extract Method instance, that a dependency cycle exists before each Move Class instance, or that a hub-like dependency exists before each Pull Up Method instance. Consequently, RQ1's answer in Section IV-A (\"transformers can reliably map architectural smells to refactoring operations\") and RQ3's per-class accuracy analysis (\"which architectural smells are most accurately detected\") are not supported: the model never receives or predicts a smell label, and the per-class metrics in Section IV-C are metrics for refactoring types, not detected smell classes. Section V-C concedes this as a construct-validity limitation, but the concession does not repair the inference made in the abstract and in the RQ answers.","section":"III-B and IV-A, IV-C"},{"comment":"The abstract claims that CodeT5 \"outperforms CodeBERT and traditional baselines,\" and Section I promises \"a comparison of CodeBERT, CodeT5, metric-based, and classical ML baselines,\" but no traditional baseline results appear anywhere in Section IV. Section III-B mentions that Aniche et al. trained six classical classifiers and reported accuracies above 90%, but the current paper reports no comparison table, no baseline numbers, and no statistical significance tests across the ten folds. The only evidence presented is the CodeBERT-versus-CodeT5 comparison in Table I, which is insufficient to support the claimed margin over state-of-the-art baselines.","section":"Abstract and IV-A"},{"comment":"The dataset description is internally inconsistent regarding the negative class. Section III-B states that 1,006,653 non-refactored commits were sampled as negative instances and that random undersampling balanced minority and majority classes, but the classification task, the confusion matrices in Figure 3, and the per-class tables contain only the three positive refactoring classes, with no negative class. The paper also does not explain how the original feature-based Aniche et al. corpus, which contains CK metrics, process metrics, and ownership metrics, is converted into the \"Java snippet + refactoring label\" format used for transformer fine-tuning, nor how refactoring instances are associated with the specific code fragment fed to the model. This ambiguity affects reproducibility and obscures what the reported accuracy actually measures.","section":"III-B and III-D"},{"comment":"The claim in Section III-C that \"ten-fold cross-validation safeguards against project-specific bias\" is not supported by the described methodology. Standard random ten-fold cross-validation can place snippets from the same project in both training and test folds, which likely inflates accuracy if the model memorizes project-specific formatting or refactoring patterns. The authors acknowledge in Section V-C that \"independent project-out or cross-repository evaluations are needed,\" which directly contradicts the earlier safeguarding claim. Either project-aware splitting should be used and reported, or the generalization claim should be substantially weakened.","section":"III-C and V-C"}],"minor_comments":[{"comment":"The RQ3 answer says that Hub-like Dependency is predicted with \"≈88%\" precision and recall, but Table II reports CodeT5's Pull Up Method precision and recall as 97.2% and 97.4%, and CodeBERT's as 89.5% and 90.7%; the 88% figure matches neither model and should be corrected.","section":"IV-C"},{"comment":"There is a typo in Table II's caption: \"Table II rovides a detailed per-class comparison\" should read \"provides.\"","section":"IV-B and Table II"},{"comment":"The model names are formatted inconsistently as \"CodeBERT,\" \"C ODE BERT,\" \"CODE BERT,\" and \"CodeT5\"/\"CODE T5\"; please standardize all occurrences.","section":"Throughout"},{"comment":"Figure 1 is referenced as \"Figure 1 outlines the workflow\" but the figure is not included in the submitted text; please ensure all referenced figures are present.","section":"III-D"},{"comment":"The related work section mentions \"Extract Class\" as an example refactoring, but the experimental classes are Extract Method, Move Class, and Pull Up Method; the terminology should be made consistent.","section":"II"}],"recommendation":"reject","confidential_remarks":"The label-validity problem is fundamental: the experiment measures refactoring-type classification, not architectural-smell detection or smell-to-refactoring mapping. Adding a manual validation or a smell-detector-based check would be a major new study rather than a routine revision. I would not recommend inviting a major revision unless the authors are willing to either substantially reframe the contribution or add independent smell-validated ground truth and project-out evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this one. First, the paper is not a sham: it fine-tunes CodeBERT and CodeT5 on the Aniche et al. corpus to predict one of three refactoring types, and the reported CodeT5 numbers (96.98% accuracy, 95% F1) are plausible for a large, balanced, three-class classification task. Second, the headline claim that these models \"reliably map architectural smells to refactoring operations\" is not supported by the experiment, and the gap is load-bearing, not cosmetic.\n\nWhat is actually new: applying pre-trained code transformers to refactoring-type prediction on a mined corpus of 2M+ instances is a reasonable new task framing, and the open release of code, data, and checkpoints is real credit. The writing is clear and the limitations section is unusually candid. The authors themselves admit the one-to-one smell-to-refactoring mapping is a construct-validity threat and that project-out evaluation is missing.\n\nWhere it falls down: the training labels are refactoring operations detected by RefactoringMiner, not architectural smells. The paper simply asserts that Extract Method implies God Class, Move Class implies Cyclic Dependency, and Pull Up Method implies Hub-like Dependency. The evaluation then reports per-class accuracy on those refactoring labels as if they were detected smell classes. RQ3 asks which smells are \"most accurately detected,\" but the model never sees a smell annotation. The 97% accuracy is a statement about classifying refactoring types from code snippets, not about detecting smells or recommending correct repairs. The abstract's \"outperforming traditional baselines\" is also unbacked: no baseline numbers appear anywhere in the text, despite the method section saying classical ML baselines were compared. The instance-count inconsistency (2M vs 3M in the contributions) and the missing RefactoringMiner citation are minor but sloppy.\n\nProportion: if reframed honestly as \"can transformers classify refactoring types from code,\" the result is a decent engineering contribution. As it stands, the title, abstract, and RQs promise more than the design can deliver. With a serious referee demanding a reframing or actual smell-validation, the paper could become useful; as-is, the central claim should not be trusted.\n\nThe reader's REJECT and the stress-test note are on the mark. I would not cite it for the smell-recommendation claim, but I would send it to peer review with a clear request for major revision: either add independent smell ground truth or drop the architectural-smell framing. It deserves referee time, not a desk reject.","headline":"A competent refactoring-type classifier with an overstated smell-recommendation claim: the mapping from refactoring labels to smell classes is assumed, not validated.","tokens_in":11357,"tokens_out":1580,"would_cite":false,"duration_ms":19331,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A transformer fine-tuned on mined refactorings can recommend the correct fix for an architectural smell with 97% accuracy.","keywords":["architectural smells","refactoring recommendation","CodeT5","CodeBERT","transformer models","software maintenance","refactoring mining","God Class"],"falsifier":"Run a smell detector such as Designite or Arcan on a held-out random sample of snippets labelled Extract Method and check whether each contains a God Class; if a substantial share contain no such smell, then the 97% accuracy reflects refactoring-type classification rather than smell-aware recommendation. A second check is to retrain on project-out splits and see whether the accuracy drop exceeds the paper's reported ten-fold within-corpus figures.","tokens_in":10357,"feed_emoji":"🛠️","tokens_out":4963,"duration_ms":48363,"temperature":0.7,"pith_summary":"The paper claims that a pre-trained transformer fine-tuned on historical refactorings can look at a Java code snippet exhibiting an architectural smell and recommend the canonical refactoring that resolves it. It frames the problem as a three-class classification task—Extract Method for God Class, Move Class for Cyclic Dependency, Pull Up Method for Hub-like Dependency—and trains CodeBERT and CodeT5 on more than two million mined refactoring instances from 11,149 open-source Java projects. CodeT5 reaches 96.98% accuracy and 95.16% F1, roughly ten points above CodeBERT and above the classical machine-learning baselines. If the claim holds, smell detectors could be paired with concrete repair suggestions instead of stopping at warnings, and researchers gain a reusable corpus for studying how developers fix architectural flaws.","feed_headline":"CodeT5 maps architectural smells to fixes with 97% accuracy","feed_subtitle":"A transformer fine-tuned on 2 million real refactorings beats classical baselines and CodeBERT on three architectural smell types.","key_machinery":"The load-bearing objects are the one-to-one smell-to-refactoring mapping and the corpus built from it: God Class maps to Extract Method, Cyclic Dependency to Move Class, Hub-like Dependency to Pull Up Method, and every historical refactoring mined by RefactoringMiner from 11,149 Java projects is labelled with one of these three classes. CodeT5 and CodeBERT are fine-tuned as classifiers on Java snippets paired with these labels, with CodeT5's encoder feeding a linear classification head. The mapping turns the recommendation problem into a three-class classification problem, and the dataset size—2,086,898 refactoring instances plus 1,006,653 negative samples, balanced by undersampling—is what makes the transformer fine-tuning feasible.","core_discovery":"On its own terms, the paper's discovery is that an encoder–decoder transformer used as a classifier can reliably map a code fragment that contains an architectural smell to the refactoring operation that repairs it, with CodeT5 achieving 97% accuracy and 95% F1 on balanced data. CodeBERT, the encoder-only comparison, reaches 85% accuracy and 85% F1. The paper presents this as the first large-scale evidence that transformer models can close the loop between architectural-smell detection and actionable refactoring recommendation, and it attributes CodeT5's advantage to its encoder–decoder pre-training and its ability to capture structural cues that distinguish refactoring types.","pith_inferences":["Going beyond the paper: because every training label comes from the hand-made one-to-one mapping rather than from a smell detector, the high accuracy may demonstrate refactoring-type classification without demonstrating that the model detects architectural smells; a project-out evaluation with independently detected smells would settle this.","Going beyond the paper: the same architecture likely transfers to other smell-to-refactoring pairs; a cheap test is to mine additional canonical mappings, such as Feature Envy to Move Method, and check whether the accuracy gap reappears.","Going beyond the paper: a direct falsification check would be to hold out entire projects, not just random snippets, and measure whether accuracy survives; the paper's ten-fold cross-validation may still mix code from the same repository across folds."],"forward_implications":["Tool builders can attach the classifier to existing detectors and replace static smell warnings with a suggested refactoring.","A hierarchical recommender becomes plausible: first select the repair family for a smell, then rank specific refactorings within that family, which would address the residual Extract-versus-Move confusion.","The released corpus of over two million refactorings enables longitudinal studies of how developers actually repair architectural smells across projects and time.","The pattern that Pull Up Method is easiest and Extract Method hardest suggests that future work should move beyond the three-smell catalogue to smells such as Unstable Interface and Cyclically-Dependent Abstraction.","Extending the same pipeline to interactive repair is feasible by coupling the classifier with a patch generator, such as the CodeT5 decoder."],"supporting_citations":[{"why":"Supplies the 11,149-project corpus of 2,086,898 refactoring instances, the feature set, and the classical baselines whose accuracies the transformers are compared against.","marker":"[4]"},{"why":"Provides the refactoring-mining method used to extract the training labels from commit history.","marker":"[10]"},{"why":"CodeBERT is one of the two pretrained models fine-tuned as the encoder-only baseline.","marker":"[11]"},{"why":"CodeT5 is the encoder–decoder model whose encoder, with a classification head, produces the 97% accuracy result.","marker":"[31]"},{"why":"Defines the architectural-smell catalogue (God Class, Cyclic Dependency, Hub-like Dependency) that the one-to-one mapping to refactorings is based on.","marker":"[7]"}],"fun_headline_variants":["CodeT5 recommends refactorings for architectural smells with 97% accuracy","CodeT5 beats CodeBERT on refactoring recommendation at 97% accuracy","Fine-tuned CodeT5 suggests refactorings with 97% accuracy","97% accurate refactoring advice from CodeT5 for architectural smells","CodeT5 transformer recommends refactorings at 97% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the hand-made assumption that each refactoring label in the training data is the correct repair for a real architectural smell in that snippet; if the labels are just refactoring types from commit history, the classifier may be learning to name refactorings without detecting architecture-level problems.","fun_headline_variants_meta":{"raw":{"variants":["CodeT5 recommends refactorings for architectural smells with 97% accuracy","CodeT5 beats CodeBERT on refactoring recommendation at 97% accuracy","Fine-tuned CodeT5 suggests refactorings with 97% accuracy","97% accurate refactoring advice from CodeT5 for architectural smells","CodeT5 transformer recommends refactorings at 97% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001276,"raw_usage":{"total_tokens":5162,"prompt_tokens":836,"completion_tokens":4326,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":452,"completion_tokens_details":{"reasoning_tokens":4227}},"tokens_in":452,"tokens_out":4326,"duration_ms":33465,"temperature":1.0,"reasoning_tokens":4227,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:44:27.890515+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a smell detector such as Designite or Arcan on a held-out random sample of snippets labelled Extract Method and check whether each contains a God Class; if a substantial share contain no such smell, then the 97% accuracy reflects refactoring-type classification rather than smell-aware recommendation. A second check is to retrain on project-out splits and see whether the accuracy drop exceeds the paper's reported ten-fold within-corpus figures.","supporting_citations":[{"cited_title":"An Empirical Study of Untangling Patterns of Two-Class Dependency Cycles","cited_arxiv_id":"2306.10599","evidence_quote":"Provides the refactoring-mining method used to extract the training labels from commit history."},{"cited_title":"Codebert: A pre-trained model for programming and natural languages,","cited_arxiv_id":null,"evidence_quote":"CodeBERT is one of the two pretrained models fine-tuned as the encoder-only baseline."},{"cited_title":"Architectural smells detected by tools: a catalogue proposal,","cited_arxiv_id":null,"evidence_quote":"Defines the architectural-smell catalogue (God Class, Cyclic Dependency, Hub-like Dependency) that the one-to-one mapping to refactorings is based on."}],"review_version":1}