{"id":"2bcd0e27-bcef-4412-a8e5-386832cff989","arxiv_id":"2508.11993","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Only about a third of behavior-preserving code changes map to known refactoring operations, but adding 67 new operation types raises coverage by over 128%.","lead":"This paper measures how much of a code change that keeps behavior identical can be broken down into known refactoring steps. It finds that existing tools catch only about a third of such changes, but adding 67 new operation types more than doubles the coverage.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Coverage gain from 67 newly defined operations may reflect post-hoc fitting to the same dataset rather than a generalizable expansion of the refactoring catalog.","rationale":"The reader's verdict is UNVERDICTED because the abstract provides insufficient information to verify the central measurement. My stress-test does not change that verdict; it sharpens the reason why the verdict must remain unverdictable. The single most load-bearing concern is the potential circularity in defining the 67 new operations after observing the data. If the operation catalog is post-hoc, the 128% coverage increase is not evidence of a general phenomenon; it is a curve-fitting result. The reader's weakest assumption explicitly identified this possibility, so I agree with that assessment. In good faith, the authors may have designed the catalog independently or used rigorous dataset construction, but the abstract gives no support for either. The proposed concrete test is a straightforward holdout evaluation or independent annotation, which would distinguish a genuinely useful expansion from an overfitted set of patterns. I recommend no change to the reader's UNVERDICTED verdict because the concern cannot be resolved from the abstract alone, and the paper needs revision or supplementary material to address it before the claim can be assessed.","tokens_in":689,"tokens_out":3016,"duration_ms":31302,"concrete_test":"Run a holdout evaluation: split the functionally-equivalent method pairs into training and evaluation sets. Use only the training set to design and refine the 67 new operations (and to tune the existing detector). Then measure the coverage increase on the evaluation set. If the increase on unseen pairs is materially below the reported 'over 128%' (e.g., not significantly above the 33.9% baseline), the new catalog is likely fitted to the original dataset. A complementary check: release the catalog and dataset and have two independent annotators label a random sample of residual diffs using the catalog; substantial inter-rater agreement would support the catalog's independent validity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central measurement claims that an existing detector explains only 33.9% of behavior-preserving changes, and that adding 67 newly defined functionally-equivalent operations raises coverage by over 128%. The most load-bearing condition for this claim is that the 67 operations are an independently motivated catalog, not labels invented to cover residual diffs observed in the same dataset. The abstract does not state how the dataset was constructed, how functional equivalence was verified, or how the new operations were designed. If the operations were derived by inspecting the unexplained diffs and then evaluated on those same diffs, the reported coverage increase is partly an artifact of overfitting: a sufficiently flexible set of bespoke patterns can fit any finite dataset. Without a holdout evaluation, a pre-registered catalog, or independent annotation, the 128% figure does not establish that expanding refactoring catalogs in general would recover most behavior-preserving changes. This is a validity risk for the empirical claim, not a disagreement with any community consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical measurement on a dataset of functionally-equivalent method pairs, finding that an existing refactoring detector identifies only 33.9% of behavior-preserving changes as refactoring operations, and that adding 67 newly defined functionally-equivalent operations increases coverage by over 128%. The authors also report an investigation of the remaining unexplained differences, interpreted as opportunities for improving refactoring-detection catalogs. Because the full text is unavailable, this report is based solely on the abstract, which does not describe the dataset construction, the equivalence verification, or the design of the new operations.","tokens_in":1029,"tokens_out":1630,"duration_ms":17558,"significance":"If the measurement is valid, the paper would provide a useful quantitative lower bound on the coverage of current refactoring detectors and a concrete argument for extending refactoring catalogs with additional behavior-preserving operations. The abstract states a clear, falsifiable empirical claim, which is a strength. However, the two load-bearing conditions—representativeness of the functionally-equivalent method pairs and independent motivation of the 67 new operations—are not evidenced in the abstract. The reported 128% coverage increase, in particular, is vulnerable to post-hoc fitting if the operations were derived from the same dataset on which they are evaluated, and no holdout or cross-validation is mentioned. The significance of the result therefore depends entirely on details that are not available in the abstract.","major_comments":[{"comment":"The abstract does not state how the dataset of functionally-equivalent method pairs was constructed, how large it is, or how functional equivalence was verified (e.g., by test-suite equivalence, formal equivalence checking, or human annotation). Without this information, the 33.9% coverage figure cannot be interpreted as a meaningful baseline.","section":"Abstract, coverage claim"},{"comment":"The central risk is that the 67 newly defined operations were introduced after inspecting the same dataset, and then evaluated on that same dataset, which would make the 128% coverage increase an artifact of overfitting rather than evidence of a generalizable catalog expansion. The abstract does not describe how the operations were designed or whether their coverage was evaluated on a held-out dataset or by independent annotation.","section":"Abstract, 67 newly defined operations"},{"comment":"The relative percentage increase is ambiguous without absolute coverage values. If 33.9% is the baseline, a 128% increase implies roughly 77% coverage, but this is not stated; the abstract should report the final coverage percentage and confidence intervals for both estimates.","section":"Abstract, 'coverage increased by over 128%'"},{"comment":"The abstract mentions an investigation of the remaining unexplained differences but gives no indication of the method (e.g., qualitative analysis, taxonomy of residual diffs, or additional experiments). Since the paper claims to identify improvement opportunities, this part needs a description of the analysis procedure and its reliability.","section":"Abstract, 'further investigation into the remaining unexplained differences'"}],"minor_comments":[{"comment":"The term 'functionally-equivalent method pairs' should be defined at least briefly in the abstract, since the entire measurement rests on this notion of equivalence.","section":"Abstract, terminology"},{"comment":"The phrase 'coverage increased by over 128%' could be misinterpreted as a percentage-point increase; the authors should clarify whether they mean a relative or absolute increase.","section":"Abstract, notation"}],"recommendation":"uncertain","confidential_remarks":"This review is based only on the abstract because no full text was provided. The central empirical claim is plausible but unverifiable from the available information. I would recommend obtaining the full manuscript before making a final decision; in particular, the authors need to demonstrate that the 67 new operations were not post-hoc fitted to the dataset."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nThe thing to know: this is an empirical paper that finally puts a number on what refactoring-aware tooling can and cannot decompose. The 33.9% baseline is a useful, citable result. The 128% coverage increase from adding 67 new operations is the headline, and on the basis of the abstract it should not be believed as a generalizable expansion.\n\nWhat is actually new: the question itself. I don't know of a prior systematic measurement of how much of behavior-preserving changes can be decomposed into primitive refactoring operations. Building a dataset of functionally-equivalent method pairs and running an existing detector over it is a clean design for an initial validation. The abstract is honest about it being an initial validation and mentions a residual analysis, which is a good sign.\n\nThe soft spots, in proportion. The most serious is the circularity risk. The 67 operations look like they were defined after inspecting the same diffs they are then used to explain. If that is what happened, the 128% figure is partly a measure of how well a fitted label set can cover a finite dataset, not a measure of how much catalog expansion would help in general. The abstract gives no indication of a holdout, pre-registered definitions, or independent annotation. This is load-bearing for the headline. The second soft spot is the dataset: no description of how the method pairs were sampled or how functional equivalence was verified. The baseline could shift with a different sampling of behavior-preserving changes. Third, and minor by comparison, the coverage numbers have no confidence intervals, so we have no sense of variance.\n\nI want to be clear: none of this sinks the paper. The baseline result about the limits of an off-the-shelf detector is informative regardless. And if the authors can show the 67 operations were motivated independently of the residuals, then the 128% jump becomes a genuinely useful finding. The residual analysis the abstract mentions may be the most honest and interesting part. The authors appear to know they are at an early stage.\n\nThis paper is for people in refactoring detection, diff comprehension, and any tooling that separates behavior-preserving from behavior-altering edits. It deserves a serious referee. The referee should spend time on dataset construction and the derivation of the new operations, and push for a holdout evaluation or other evidence that the catalog is not fit to the test set. If that evidence is there, it is a solid incremental contribution. If not, the baseline still stands and the headline should be tempered.\n\nRecommendation: send it to peer review, with the circularity concern as the primary question for the authors.","headline":"The 33.9% baseline is a real addition; the 128% coverage gain depends on 67 operations that look post-hoc, so the full paper needs a holdout.","tokens_in":1354,"tokens_out":3447,"would_cite":true,"duration_ms":33883,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper measures how much of a behavior-preserving code change can be decomposed into refactoring operations, finding that an existing detector covers only 33.9 percent of such changes and that adding 67 newly defined operations more…","keywords":["refactoring detection","behavior-preserving changes","functionally-equivalent method pairs","changeset decomposition","refactoring operations","empirical software engineering"],"falsifier":"A concrete test would be to apply the same detector-plus-catalog protocol to a new set of method pairs drawn from different open-source projects or from commits that developers explicitly marked as pure refactoring. If the 67-operation coverage gain largely disappears on that fresh data, the original gain is an artifact of the dataset; if it remains comparable, the measurement generalizes.","tokens_in":528,"feed_emoji":"🔧","tokens_out":3868,"duration_ms":41260,"temperature":0.7,"pith_summary":"Behavior-preserving modifications are code changes that do not change what the program does, and developers often want to separate such refactorings from feature additions when reading a diff. This paper tests how much of these modifications can be broken down into primitive, reusable operations. Using a dataset of functionally-equivalent method pairs, an existing refactoring detector recognized 33.9% of the changes as refactoring operations. When the authors added 67 newly defined functionally-equivalent operations, coverage increased by over 128%, meaning the expanded catalog identified more than twice as much of the changes. The remaining unexplained differences are examined as opportunities for further improvement.","feed_headline":"Refactoring tools explain only 33.9% of behavior-preserving edits","feed_subtitle":"A catalog of 67 additional operations more than doubles the share, yet most edits still elude automated decomposition.","key_machinery":"The measurement device is a comparison between functionally-equivalent method pairs: take a method before and after a change that produce the same behavior, ask a refactoring detector to decompose the difference into primitive operations, and count the fraction of changes it can label. The experiment then enlarges the operation catalog with 67 newly defined functionally-equivalent operations and measures how much coverage rises. The dataset of method pairs and the expanded catalog together are the engine of the result, since every coverage number is a ratio of labeled differences to total behavior-preserving differences.","core_discovery":"The central result is an empirical quantification: on a dataset of method pairs that behave the same way but have different code, an existing refactoring detector can label only 33.9% of the behavior-preserving differences as refactoring operations. After the authors define 67 additional behavior-preserving operations, coverage grows by more than 128%. This shows that the vocabulary of recognized refactoring operations is a major bottleneck: expanding that vocabulary can more than double how much of a changeset becomes automatically decomposable. Even after the expansion, a substantial portion of the differences remains unexplained, and the paper treats that remainder as evidence that further operation definitions and analysis strategies are still needed.","pith_inferences":["If the 67 new operations were crafted after examining the gaps left by the existing detector, part of the coverage gain may be dataset-specific; a validation on a fresh, independently collected set of method pairs would show whether the catalog generalizes.","The result suggests a practical route for improving refactoring tools: mine real code histories for repeated behavior-preserving transformations, codify them as new operations, and re-measure coverage on held-out changes.","Functional equivalence is a broad condition that admits changes developers would not necessarily call refactorings, so the coverage numbers may reflect the expressiveness of the operation catalog rather than the conceptual quality of the decomposition."],"forward_implications":["Tools that rely on current refactoring detectors will miss most of the behavior-preserving portion of a changeset, so developers cannot yet trust them to separate refactorings from feature changes automatically.","Enlarging the catalog of behavior-preserving operations can more than double the fraction of a changeset that automated tools can decompose.","A sizable unexplained remainder persists even with the expanded catalog, so operation-catalog growth alone will not achieve near-complete coverage.","The 33.9% baseline and the expanded-coverage figure give future refactoring detectors a concrete benchmark to compare against."],"supporting_citations":[],"fun_headline_variants":["Behavior-preserving edits: only 34% decomposable, new ops help","67 new ops double refactoring coverage, still not enough","Refactoring tools explain 34% of edits; 67 new ops push to 77%","Coverage of behavior-preserving edits jumps 128% with 67 new ops"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The finding depends on the collected functionally-equivalent method pairs being a fair representation of the behavior-preserving changes developers actually make, and on the 67 added operations being a genuine catalog rather than labels chosen after the fact to close the observed gaps.","fun_headline_variants_meta":{"raw":{"variants":["Behavior-preserving edits: only 34% decomposable, new ops help","67 new ops double refactoring coverage, still not enough","Refactoring tools explain 34% of edits; 67 new ops push to 77%","Coverage of behavior-preserving edits jumps 128% with 67 new ops"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001701,"raw_usage":{"total_tokens":6688,"prompt_tokens":851,"completion_tokens":5837,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":5751}},"tokens_in":467,"tokens_out":5837,"duration_ms":40685,"temperature":1.0,"reasoning_tokens":5751,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:25:06.435714+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test would be to apply the same detector-plus-catalog protocol to a new set of method pairs drawn from different open-source projects or from commits that developers explicitly marked as pure refactoring. If the 67-operation coverage gain largely disappears on that fresh data, the original gain is an artifact of the dataset; if it remains comparable, the measurement generalizes.","supporting_citations":[],"review_version":1}