{"id":"59a668a5-0b85-4a89-ac8b-8551a541b9d6","arxiv_id":"1908.01501","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Refactoring types applied to highly coupled and loosely coupled classes largely overlap, suggesting developers do not systematically treat high coupling as a refactoring trigger.","lead":"This paper analyzed more than 6,000 code refactorings from three open-source Java projects and compared which refactoring types developers apply to classes with high versus low coupling. It found that the mix of refactorings was nearly the same in both groups, challenging the common assumption that developers target highly coupled classes for cleanup.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unspecified timing of coupling measurement: if CBO/CCBC are measured after refactoring, the observed excess of move-method/move-field in low-coupling classes is a post-treatment artifact, not evidence of developer indifference.","rationale":"The reader's weakest assumption is that Ref-Finder's precision/recall is not differential across coupling quartiles. I agree that is a real measurement concern, but it is secondary. Even a perfect refactoring detector would yield a misleading conclusion if the coupling variable is temporally misaligned. The paper's specific observation—coupling-removal refactorings more numerous in the lower quartile—is exactly the signature of post-treatment measurement: applying MM/MF lowers CBO, pushing the class into the LQ. The paper's discussion in Section III-A even interprets the high counts of MM/MF as surprising ('It was therefore surprising to see so many of each refactoring type in both quartiles'), which suggests the authors did not consider that their measurement window could cause this. No section of the paper, including Threats to Validity, addresses the temporal ordering of metric computation versus refactoring extraction. The public dataset makes this test feasible, so the concern can be settled rather than left as an ambiguity. For an exploratory paper, the raw counts may still be of interest, but the strong interpretive conclusion ('developers seem to be largely indifferent') should be conditional on the timing analysis. This is why I keep the reader's CONDITIONAL verdict rather than escalating to REJECT: the flaw is in the unspecified assumption, not in the data collection, and a straightforward re-analysis could restore or refute the claim.","tokens_in":7967,"tokens_out":9342,"duration_ms":95033,"concrete_test":"Download the Bavota et al. dataset (referenced as [3]) and inspect its schema to determine whether each class's CBO/CCBC value is computed from the release before the refactoring, the release after, or a fixed snapshot. If pre-refactoring values are available, re-run the entire quartile decomposition and compare the distribution of refactoring types. Specifically, recompute the percentage of MM+MF in the LQ and UQ for each system (the numbers behind Section III-A and III-B). If, using pre-refactoring coupling, MM+MF is no longer higher in the LQ (or the overlap between quartiles drops materially), the reported pattern is a post-treatment measurement artifact. If the dataset does not expose timing, reconstruct the pre-refactoring state by checking out the parent revision of each refactoring commit and recomputing CBO/CCBC on that snapshot, then repeat the analysis.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central conclusion—that developers treat high- and low-coupling classes similarly when refactoring—requires that each refactoring be assigned to a class whose coupling value reflects the state of that class before the refactoring decision. The paper never states whether the CBO and CCBC values used to construct the quartiles are measured before or after the refactorings, or at some single release snapshot. This is not a peripheral detail: Move Method (MM) and Move Field (MF), which the paper labels as coupling-removal refactorings, reduce the coupling of the source class. If the dataset's coupling values are computed after these refactorings (or at the latest release), then classes that were originally highly coupled and then refactored will systematically appear in the lower quartile. That would mechanically produce the paper's most specific finding—'refactorings usually associated with coupling removal were actually more numerous in the lower quartile in some cases' (Section III-A)—without any implication that developers are indifferent to high coupling. It would also mix originally high- and low-coupling classes in the LQ, artificially increasing the overlap between quartile distributions and thus manufacturing the 'no meaningful difference' result. Section V lists several threats but does not mention temporal alignment. The paper also shows in Table 3 that UQ and LQ classes differ substantially in size (LOC, WMC), so size is confounded with coupling; without controlling for the timing of measurement, the comparison cannot separate 'developers ignore high coupling' from 'coupling was reduced by prior refactoring.' Because the dataset from Bavota et al. is public, this assumption is checkable, but as written the central interpretive claim is not supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper analyzes 5,200 refactoring operations from three open-source Java systems (Xerces, Apache Ant, ArgoUML), dividing classes into upper and lower quartiles of the CBO and CCBC coupling metrics and comparing the types of refactorings applied in each quartile. The authors report substantial overlap in refactoring types across quartiles, observe that coupling-removal refactorings such as Move Method and Move Field are sometimes more numerous in the lower quartile, and note a scarcity of inheritance-related refactorings. They conclude that developers appear largely indifferent to high coupling when selecting refactoring types, and question the relevance of Fowler's full catalog of refactorings.","tokens_in":8198,"tokens_out":2846,"duration_ms":28566,"significance":"If the central finding is robust, the paper challenges a widely held assumption that high coupling motivates coupling-reduction refactorings, and it contributes descriptive evidence about which refactorings are actually used in practice. The use of an existing external dataset (Bavota et al.) and the clear presentation of quartile-level counts are strengths, as is the explicit comparison of two coupling metrics. However, the paper's main conclusion is currently supported only by informal comparisons of proportions, with no statistical tests or confidence intervals, and several potentially confounding aspects (notably the timing of metric measurement) are not addressed. The work is an interesting exploratory study but does not yet meet the standards for a definitive empirical contribution.","major_comments":[{"comment":"The paper never states whether the CBO and CCBC values used to construct the quartiles are measured before or after the refactorings, or at a single release snapshot. This timing is load-bearing: Move Method and Move Field reduce the coupling of the source class, so if coupling values are computed after these refactorings, originally high-coupling classes that were refactored would systematically appear in the lower quartile. That mechanism would mechanically produce the paper's key observation that coupling-removal refactorings are numerous in the lower quartile, without implying developer indifference. The authors must specify the measurement timing, and ideally re-analyze the data using pre-refactoring coupling values or at least discuss how this could change the results.","section":"Section III, Table 2"},{"comment":"The conclusion 'no meaningful difference' is asserted without any statistical comparison. The reported proportions include substantial differences, for example ArgoUML CBO data show MM and MF together accounting for 35.17% of refactorings in the LQ versus 10.39% in the UQ (Section III-A). The authors should provide confidence intervals, effect sizes, or formal tests (e.g., chi-square or bootstrap-based comparisons) to support the claim of no difference. Without such analysis, the impression of overlap may be misleading.","section":"Section III, Figs. 1 and 2"},{"comment":"The size confounding is acknowledged in the paper's own Table 3, which shows that UQ classes are much larger than LQ classes (e.g., Xerces mean LOC 1781 vs. 249; ArgoUML mean WMC 57.72 vs. 15.00). The analysis compares raw counts and proportions of refactorings without normalizing by class size or method count. Larger classes naturally offer more refactoring opportunities, so the comparison of refactoring type mixes across quartiles is confounded; the authors should either normalize the data or control for size statistically.","section":"Section III-C, Table 3"},{"comment":"The analysis relies entirely on Ref-Finder's reconstructed refactorings, which the paper reports as having 79% precision and 95% recall. If Ref-Finder's detection accuracy varies with class coupling (e.g., if move operations are easier to detect in smaller, low-coupling classes), the observed overlap between quartiles could be a measurement artifact. The authors do not validate the tool's output on these three systems, and the threats-to-validity section does not address this. At minimum, a sensitivity analysis or an explicit argument about why detection bias is unlikely would strengthen the claim.","section":"Section II"}],"minor_comments":[{"comment":"The phrase 'Results showed no very little difference' is grammatically incorrect and appears to convey the opposite of the intended meaning; it should read 'no difference' or 'very little difference.'","section":"Abstract and Introduction"},{"comment":"There is an inconsistent abbreviation: 'UP' should be 'UQ' when referring to the upper quartile (in the sentence about RMNwSC percentages).","section":"Section III-A"},{"comment":"The text states that Figs. 2a, 2c, and 2e represent the UQ, but the figure captions label Fig. 2a and 2c as 'LQ' and Fig. 2b and 2d as 'UQ.' This mismatch between text and figures must be corrected.","section":"Section III-B, Figs. 2a-2f"},{"comment":"The abstract claims 'over six thousand refactoring operations,' but Table 1 lists 7,502 refactorings for Xerces alone, and the total across all three systems is 12,046. After quartile filtering only 5,200 are used; the wording should specify the number analyzed after quartile selection.","section":"Abstract and Table 1"},{"comment":"The threats-to-validity section does not mention the timing of metric measurement, which is a potential major threat as discussed in the major comments. It should be explicitly addressed there.","section":"Section V"}],"recommendation":"major_revision","confidential_remarks":"The paper's central observation is potentially interesting, but the analysis is currently too thin to support its strong conclusion. The missing temporal alignment of coupling measurements is the most serious issue because it could invalidate the main interpretive claim; this is not just a presentation problem. The paper may also be a better fit for a workshop or a more exploratory venue unless the authors can strengthen the statistical and confounding controls. I would encourage the editor to request a revision that addresses the timing issue and adds inferential statistics rather than rejecting outright."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Coupling/refactoring paper: new quartile analysis of the Bavota et al. data, but the headline interpretation is shaky until the timing question is answered. The descriptive overlap between high- and low-coupling quartiles is a genuine observation, and the lack of inheritance refactorings is worth noting. The authors are honest about the exploratory scope and list relevant threats.\n\nThe main soft spot is exactly the one the stress-test flags. The paper never states whether CBO and CCBC are measured before the refactorings or after. Since Move Method and Move Field reduce source-class coupling, post-treatment measurement would push those very refactorings into the low-coupling quartile, which is what they see in some systems. That would make the 'developers treat high and low coupling the same' conclusion an artifact. The dataset is public, so the authors can check this; as written, the central claim is not supported. I'd call this the one serious issue, not because the result is wrong but because the ordering of measurement and events is silently doing work.\n\nTwo smaller issues. First, 'no meaningful difference' is an interpretation, not a statistical result – no tests, intervals, or effect sizes, and some of their own percentages (ArgoUML MM+MF 35% LQ vs 10% UQ) look like differences worth talking about. Second, the refactoring counts come from Ref-Finder, 79% precision, and the authors don't validate its output on these three systems. That's a known limitation, but it reinforces the need for cautious wording.\n\nWhat's genuinely new: the quartile split, the side-by-side CBO/CCBC comparison, and the observation that the same few refactoring types dominate both quartiles, which was not in the original Bavota et al. paper. Credit is due for reusing a public dataset and for keeping the analysis simple enough to reproduce from the paper's tables.\n\nRecommendation: yes, send it to a serious referee. The timing issue is addressable and the paper's contribution is a useful empirical note, but it needs a revision that either verifies the temporal alignment or weakens the 'indifference' framing. I would not cite it in its current form, but I'd read a revised version.","headline":"New quartile analysis of a public refactoring dataset, but the 'developer indifference' claim depends on an unstated timing assumption that could flip the result.","tokens_in":8815,"tokens_out":2937,"would_cite":false,"duration_ms":29031,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Developers do not refactor highly coupled classes differently from loosely coupled classes, according to more than five thousand refactorings from three open-source systems.","keywords":["refactoring","coupling metrics","CBO metric","CCBC metric","quartile analysis","empirical software engineering","open-source software","inheritance refactoring"],"falsifier":"A reader could test this by taking a random sample of classes from the same three systems and manually reconstructing refactorings from version-control diffs, independent of the automated tool. If move-method and move-field refactorings concentrate in the high-coupling sample rather than appearing equally across quartiles, the paper's reported indifference would fail.","tokens_in":7753,"feed_emoji":"🔧","tokens_out":8621,"duration_ms":82201,"temperature":0.7,"pith_summary":"Refactoring is supposed to improve internal structure, and excessive coupling is usually treated as a reason to refactor. This paper asks whether developers actually behave that way: do classes with high coupling receive different refactoring types than classes with low coupling? The authors analyze more than 5,200 refactorings from releases of three open-source systems, split into lower and upper quartiles by two coupling measures (CBO, a count of coupled classes, and CCBC, a text-similarity measure). They find no meaningful difference in the types of refactoring applied in either quartile for either metric, and they find that move-method and move-field refactorings, typically associated with coupling removal, appear in the lower quartile too, sometimes more often. If this result holds, it undercuts the intuition that high coupling triggers coupling-reducing refactoring and raises the question of why so few of the standard seventy-two refactorings ever get used.","feed_headline":"5,200 refactorings show coupling does not drive refactoring choices","feed_subtitle":"Move-method and move-field refactorings, meant to cut coupling, appeared just as often in already-loosely-coupled classes.","key_machinery":"The argument is carried by a quartile comparison. Each class is ranked by CBO (the number of other classes to which it is coupled) and separately by CCBC (the textual similarity of a class's tokens to other classes); the classes in the bottom 25% and top 25% of each ranking define the lower and upper quartiles. Refactorings assigned to classes in each quartile are then tallied by type, and the top-ten lists are compared by overlap and frequency. The refactoring data themselves were produced by a reconstruction tool with reported 95% recall and 79% precision, operating on the release history of three open-source systems; the paper treats those reconstructed refactorings as the record of what developers did, then checks whether the type mix changes with coupling quartile.","core_discovery":"The central claim is reported as a negative result: refactoring choices do not appear to track class coupling. For each of three systems, the paper splits classes into lower and upper quartiles by CBO and by CCBC, then compares the ten most frequent refactoring types in each quartile. The lists overlap heavily, with seven, six, and eight of the top ten common across the systems for CBO and comparable overlap for CCBC, and the same types (rename method, move method, move field, add or remove parameter, replace magic number with symbolic constant) dominate both quartiles. Move method and move field, which the paper identifies as the refactorings most strongly associated with coupling reduction, are numerous in the low-coupling quartile as well, and in some systems more numerous there. The paper also reports that only 112 of 5,200 analyzed refactorings (2.15%) were inheritance-related, mostly in one system. Its conclusion is that developers are largely indifferent to coupling when choosing refactoring types.","pith_inferences":["Left implicit in the paper: refactoring-recommendation systems that rank classes by coupling are likely to mismatch real developer attention; adding size, change frequency, or defect history as co-predictors is a natural test the authors did not run.","The paper reports that upper-quartile classes are much larger on average than lower-quartile classes, yet refactoring types look the same. That makes size a plausible confound: a follow-up should compare coupling quartiles within narrow size bands to see whether size, not coupling, drives the observed choices.","The authors interpret the result as developer indifference to coupling. A competing interpretation is that high-coupling classes had already been refactored or are stable enough that no coupling-related refactoring was needed; distinguishing these would require developer interviews or commit-level rationale, which the dataset cannot provide.","If replication on the larger 33-project dataset the paper names as future work shows the same overlap, then the absence of coupling-driven refactoring may be a general property of open-source maintenance rather than a quirk of these three systems."],"forward_implications":["High coupling alone will not predict where coupling-removal refactorings such as move method and move field occur; in this dataset they are common in low-coupling classes too.","The overlap in refactoring types across quartiles implies that developers' refactoring decisions are driven by something other than coupling level, possibly code smells like long methods, duplication, or parameter-list awkwardness.","Because only a handful of the seventy-two catalogued refactorings appear in either quartile, claims about refactoring practice based on the full catalog may describe an ideal rather than actual developer behavior.","The similar behavior of CBO and CCBC across quartiles suggests the two metrics may be interchangeable for this kind of refactoring study, as the paper notes."],"supporting_citations":[{"why":"Supplies the pre-existing dataset of refactorings from three open-source systems that the quartile analysis reuses.","marker":"[3]"},{"why":"Provides the reconstruction tool whose reported 95% recall and 79% precision produced the refactoring data.","marker":"[8]"},{"why":"Defines the CBO metric, the first of the two coupling measures used to form quartiles.","marker":"[6]"},{"why":"Defines the conceptual coupling metric (CCBC), the second coupling measure used to form quartiles.","marker":"[11]"},{"why":"Defines the catalog of seventy-two refactorings whose usage rates are compared across quartiles.","marker":"[7]"}],"fun_headline_variants":["Coupling doesn't drive refactoring types in 5,200 ops","Refactoring choices don't follow class coupling","Coupling level has no effect on refactoring types","High vs low coupling: same refactorings, study says","Move-method refactorings ignore coupling levels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole result assumes that the automated refactoring reconstruction finds refactorings equally well in highly coupled and loosely coupled classes, since the paper does not revalidate the tool on these three systems.","fun_headline_variants_meta":{"raw":{"variants":["Coupling doesn't drive refactoring types in 5,200 ops","Refactoring choices don't follow class coupling","Coupling level has no effect on refactoring types","High vs low coupling: same refactorings, study says","Move-method refactorings ignore coupling levels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000757,"raw_usage":{"total_tokens":3378,"prompt_tokens":976,"completion_tokens":2402,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":2320}},"tokens_in":592,"tokens_out":2402,"duration_ms":18012,"temperature":1.0,"reasoning_tokens":2320,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:10:22.337497+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could test this by taking a random sample of classes from the same three systems and manually reconstructing refactorings from version-control diffs, independent of the automated tool. If move-method and move-field refactorings concentrate in the high-coupling sample rather than appearing equally across quartiles, the paper's reported indifference would fail.","supporting_citations":[{"cited_title":"An experimental investigation on the innate relationship between quality and refactoring","cited_arxiv_id":null,"evidence_quote":"Supplies the pre-existing dataset of refactorings from three open-source systems that the quartile analysis reuses."},{"cited_title":"Ref-Finder: A refactoring reconstruction tool based on logic query temp lates","cited_arxiv_id":null,"evidence_quote":"Provides the reconstruction tool whose reported 95% recall and 79% precision produced the refactoring data."},{"cited_title":"A Metrics Suite for Object Oriented Design","cited_arxiv_id":null,"evidence_quote":"Defines the CBO metric, the first of the two coupling measures used to form quartiles."},{"cited_title":"Using infor- mation retrieval based coupling measures for impact analysis","cited_arxiv_id":null,"evidence_quote":"Defines the conceptual coupling metric (CCBC), the second coupling measure used to form quartiles."},{"cited_title":"Refactoring: Improving the Design of Existing Code","cited_arxiv_id":null,"evidence_quote":"Defines the catalog of seventy-two refactorings whose usage rates are compared across quartiles."}],"review_version":1}