{"id":"57e1bc6c-8edc-4be2-bdf0-d1eb2c0a8028","arxiv_id":"2411.10842","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A toolkit of 11 code refactoring operators reduces n-gram overlap with training corpora by up to 65 percentage points, though this drop is partly by construction and is not tied to downstream task performance.","lead":"This paper presents CODECLEANER, a toolkit with 11 code-refactoring operators that rewrite Python and Java code to reduce how much of an evaluation snippet matches a model's training corpus. The authors report a 65 percentage-point drop in overlap when all operators are applied, but the study relies on proxy metrics rather than measuring whether model performance estimates actually become trustworthy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper provides no behavioral validation that its refactoring operators preserve program semantics; without such validation, the reported drops in overlap and model familiarity may reflect code alteration rather than contamination mitigation.","rationale":"The reader's weakest assumption is the absence of a post-cutoff control to calibrate the two metrics. I agree that such a control is missing, but the more load-bearing premise is that refactoring preserves the code's semantics, because without it the metrics cannot be interpreted at all. The refactoring operators include semantic additions (Param, Deco, Inhr) and perturbations (Renm), and the paper explicitly concedes possible semantic changes in Section VII. If, say, Renm replaces an identifier with a synonym that changes behavior, or Loop introduces a bug in edge cases, the refactored program is a different program. Lower overlap would then just be an artifact of altered code, not evidence of contamination mitigation. The decisive test is therefore differential execution of original vs. refactored code on representative inputs. This is cheap and directly tests the premise. If a substantial failure rate emerges, the paper's conclusion should be weakened to 'the operators reduce n-gram overlap and model familiarity' without claiming contamination mitigation; if all outputs match, the missing post-cutoff control becomes the next decisive check. I therefore keep the reader's CONDITIONAL verdict but with a sharper condition: add behavioral validation and, if the semantic operators cannot be made behavior-preserving, temper the central claim.","tokens_in":19373,"tokens_out":7135,"duration_ms":75146,"concrete_test":"Select a random subset of 50 method-level Python snippets from the RQ1 dataset that can be executed with available dependencies. For each, generate 100-200 inputs (random values matching the parameter types, plus edge cases). Run the original and the refactored version produced by each of the 11 operators (and the all-operators pipeline) on the same inputs; compare return values and raised exceptions. Any behavioral difference (other than permitted non-observable additions like timing decorators) indicates the operator does not preserve semantics. Also parse/compile each refactored snippet to confirm syntactic validity.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim in Section I and V(A) that the toolkit's 11 operators 'demonstrate their effectiveness in addressing data contamination' presupposes that a refactored code snippet remains a valid evaluation instance for the original program. The paper never executes, compiles, or differentially tests any of the refactored outputs; it only measures character-level 50-gram overlap against The Stack (Section IV-B1) and model perplexity/Min-20% Prob (Section IV-B2). This is not a mere calibration issue: model-wise familiarity metrics are highly sensitive to code unnaturalness and semantic oddity. A snippet that is syntactically valid but semantically altered or simply 'strange' will have elevated perplexity regardless of whether it appears in the training corpus. The authors acknowledge exactly this confound in Section VII (Threats to Validity): the semantic operators 'may inadvertently change the code semantics, making the model find the code semantics strange and thus exacerbating model-wise data contamination.' Therefore the observed 65% overlap reduction and positive model-wise changes (Tables II and III) cannot be attributed to contamination removal as opposed to code deterioration. If the refactored code changes observable behavior, evaluating a CLM on it does not measure performance on the original code, so the headline 'mitigates data contamination' is unsupported. The paper offers no control (e.g., post-cutoff code) nor any behavioral equivalence check, leaving the central claim hanging on an untested premise.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents CodeCleaner, an open-source toolkit of 11 code refactoring operators (nine method-level, one class-level, one cross-class-level) for Python, with four operators migrated to Java, intended to mitigate data contamination in code language model (CLM) evaluation. The authors evaluate the operators on method-level and class-level code samples drawn from The Stack, using two families of metrics: a data-wise 50-gram overlap ratio against The Stack and model-wise perplexity/Min-20% Prob computed for four CLMs. The headline results are a reduction in overlap from 87% to 22% on method-level Python code and positive model-wise familiarity changes for semantic operators such as Deco and Renm. The paper also reports cross-language contamination severity (RQ3) and operator generalizability to Java (RQ4). The claims are that the operators 'demonstrate their effectiveness in addressing data contamination' and that the toolkit is a practical, automated alternative to building post-cutoff benchmarks.","tokens_in":19600,"tokens_out":3314,"duration_ms":37901,"significance":"The paper addresses a timely and industrially relevant problem—data contamination in CLM evaluation—and contributes a concrete, open-sourced artifact (CodeCleaner) along with the first systematic comparison of refactoring operators at method, class, and cross-class levels. The distinction between data-wise and model-wise measurements is useful, and the acknowledgment of limitations in Section VII is candid. The migration of operators to Java and the per-operator analysis of effectiveness are valuable empirical contributions. However, the validity of the central claim depends on two load-bearing issues: the data-wise overlap metric is nearly circular given the sampling design, and the model-wise metrics are confounded by the absence of any semantic-preservation validation. The paper's value would be substantially increased by adding a post-cutoff control corpus and behavioral equivalence checks, which are feasible within the manuscript's scope.","major_comments":[{"comment":"The data-wise effectiveness evidence is close to by construction. The code snippets used in RQ1–RQ2 are sampled from The Stack (Section IV-A2), and the overlap ratio is computed against The Stack (Section IV-B1); the operators are explicitly designed to disrupt consecutive character sequences, which is exactly the quantity measured. The reported drop from 87% to 22% in Figure 12 therefore restates the operators' design objectives rather than demonstrating that the refactored code is uncontaminated in any meaningful sense. The paper would need a control set—for example, code written after the models' training cutoff and therefore not in the corpus—to calibrate what overlap levels are actually indicative of contamination. Without such a control, the '65% decrease' claim is not evidence of contamination mitigation.","section":"§IV-A2, §IV-B1, §V-A (Fig. 12)"},{"comment":"The paper repeatedly asserts that the operators preserve program semantics (e.g., 'without affecting the original semantic integrity' in Section I and 'keeping the code semantics untouched' in Section III-A), but it never executes, compiles, or differentially tests any refactored output. This matters because the model-wise metrics (perplexity and Min-20% Prob) are sensitive to code unnaturalness and semantic oddity; a syntactically valid but semantically altered snippet could show higher perplexity simply because it is strange, not because it is less contaminated. The authors themselves acknowledge this confound in Section VII: 'the semantic operators we implemented may inadvertently change the code semantics, making the model find the code semantics strange and thus exacerbating model-wise data contamination.' As it stands, the model-wise improvements in Tables II–IV cannot be attributed to contamination removal rather than code deterioration. The paper should add a behavioral equivalence check (e.g., running test suites, differential execution on sampled inputs, or at minimum compiling/parsing all refactored snippets) and report the fraction of snippets whose behavior is verified preserved.","section":"§III, §VII and model-wise results (Tables II–IV)"},{"comment":"The model-wise evaluation lacks any estimate of uncertainty. Tables II–IV report per-model and average Min-20% Prob differences, and Figure 11 shows rankings, but there are no error bars, confidence intervals, or significance tests. Many reported differences are small (e.g., 0.001 vs. -0.016 in Table II) and, without variance information, could easily be noise. In addition, the choice of K=20 in Min-20% Prob is taken from the authors' prior work [9] rather than justified on this dataset; a sensitivity analysis over K (e.g., K=10, 20, 30) is needed to establish that the operator rankings are not an artifact of this free parameter. The paper should report the distribution over the 384/324 snippets and test whether the operator effects are statistically distinguishable from zero and from each other.","section":"§V-A, §V-B, Tables II–IV and Figure 11"},{"comment":"RQ3's comparison of contamination severity across programming languages is circular for the same reason as RQ1–RQ2. The snippets for Java, C, Rust, and Python are all sampled from The Stack (Section IV-A2), and the reported severity is the 50-gram overlap against The Stack. High overlap rates (88–98%) are expected by construction when measuring a corpus against itself, so the finding that 'Java has the highest at 98%' is an artifact of the sampling design, not a characterization of contamination severity. The RQ3 claim should be reframed as 'overlap with The Stack' rather than 'data contamination severity,' or, preferably, the authors should sample an independent post-cutoff corpus to measure actual contamination levels.","section":"§V-C (Figure 15)"}],"minor_comments":[{"comment":"The text says 'The data-wise results are shown in Table 15' but the referenced object is labeled 'Figure 15'; please correct the cross-reference.","section":"§V-C"},{"comment":"The definition of Min-K% Prob is omitted ('The equation is omitted due to space limitation'). Since this is one of the two core model-wise metrics, the equation and the token-selection procedure should be included or, at minimum, given in an appendix.","section":"§IV-B2"},{"comment":"The captions state 'The bluer, the better,' yet many entries are negative (e.g., Iter, Comm, Norm, Inhr in Tables II and III). Please clarify how the color scale handles negative values and how a reader should interpret a negative effect.","section":"Tables II–IV captions"},{"comment":"The setup paragraph says 'eight method-level operators' but the operator list in Figure 1 contains nine method-level operators (IFF, Loop, Iter, Comm, Deco, Param, Renm, Norm, Styl); please reconcile the count.","section":"§V-A Setup"},{"comment":"The sentence 'the other operators are based on the operator Norm' is unclear. Does it mean that all operators are applied to code that has already been normalized? If so, please state this explicitly and explain how the reported reductions for individual operators are computed relative to the normalized baseline.","section":"§V-A Data-wise Effect"},{"comment":"Reference [13] in the text is attributed to 'Razeghi et al.' but the bibliography entry lists 'Schick and Schütze, It's not just size that matters.' Please verify and correct the citation mapping.","section":"References"},{"comment":"The paragraph states a total of 7680 snippets computed as 4 * 5 * 384, which matches the earlier statements, but the phrase 'we sampled data for the years 2018 ~ 2022' should clarify whether the 384 snippets per language are spread across years or per year; the current wording is ambiguous.","section":"§IV-A2 RQ3 data preparation"}],"recommendation":"major_revision","confidential_remarks":"The paper is on an important topic and the artifact is a useful contribution, but the empirical core is not yet convincing. The data-wise claims are largely tautological given the sampling design, and the model-wise claims are confounded by the lack of semantic-preservation validation. I believe these issues are addressable within the manuscript's scope: adding a post-cutoff control corpus, running behavioral equivalence checks on refactored snippets, and reporting uncertainty/sensitivity analyses would substantially strengthen the paper. I would encourage the editor to invite a revision rather than reject, as the toolkit itself is novel and the research direction is valuable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The toolkit is real and the per-operator results are informative, but the central claim that refactoring 'mitigates data contamination' is not backed by the measurements. The data-wise metric—50-gram overlap against The Stack—is exactly what the operators are designed to reduce, so the 87%-to-22% drop is partly a restatement of the operators' definitions. The model-wise metrics (perplexity, Min-20% Prob) add some independence, but the paper never shows that the refactored code is behaviorally equivalent to the original. There are no execution tests, no differential testing, and no held-out post-cutoff control. The threat-to-validity section even admits the semantic operators 'may inadvertently change the code semantics,' which means the model-wise familiarity drops could reflect code deterioration rather than contamination removal.\n\nWhat's genuinely new: this is the first systematic comparison of refactoring operators at method-, class-, and cross-class scales across Python, Java, C, and Rust. The released CodeCleaner toolkit is a useful engineering artifact, and RQ3's per-language contamination numbers (Java ~98% overlap, others ~88-94%) are a solid empirical addition. The Java port of four operators in RQ4 is a reasonable generalization check. The per-operator rankings (semantic operators like Deco and Renm consistently move model-wise scores, syntactic operators often do little or backfire) are believable and consistent with prior work [9].\n\nWhere it's soft: (1) No behavioral validation. The strongest requirement for a refactoring-based mitigation is that the refactored snippet still represents the same program/task. The paper doesn't compile, run, or compare outputs for any refactored snippet. (2) No post-cutoff control. We don't know what overlap or perplexity levels genuinely clean code would have, so we don't know if 22% overlap is 'clean enough.' (3) Model-wise tables report averages without variance or significance tests; with 384 samples, error bars should be trivial to compute. (4) The choice of K=20 for Min-K% Prob is inherited from prior work without re-justification.\n\nThe stress-test concern lands. This is not a fatal flaw in the engineering contribution, but it's fatal to the current interpretation. The paper should be reframed as 'refactoring reduces n-gram overlap and model familiarity proxies' rather than 'mitigates data contamination.' Reviewers should ask for a behavioral equivalence check (e.g., run both versions on a set of inputs and compare outputs), a post-cutoff control set, and significance testing. If the authors can show that at least the syntactic/naming operators preserve behavior on the sampled snippets, the toolkit becomes much more credible.\n\nWho this is for: anyone working on CLM evaluation or contamination; it's a useful cautionary study about proxy metrics. It deserves a serious referee but needs major revision. I'd send it to review with the expectation that the authors add the missing control and behavioral validation.","headline":"Real toolkit, useful per-operator data, but the headline claim that refactoring 'mitigates data contamination' is not supported because the primary metric is partly by construction and the paper never checks behavioral equivalence.","tokens_in":20205,"tokens_out":3362,"would_cite":true,"duration_ms":36160,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that automated code refactoring can mitigate data contamination in code-language-model evaluation, reporting a 65-percentage-point drop in training-data overlap when all operators are applied to method-level Python code.","keywords":["data contamination","code language models","code refactoring","semantic-preserving transformations","n-gram overlap","perplexity","membership inference","Python"],"falsifier":"Take a set of code snippets written after the studied models' training cutoffs, measure their 50-gram overlap and Min-20% Prob scores, then apply CODECLEANER to contaminated pre-cutoff snippets; if the refactored snippets still score far above the uncontaminated control on either metric, the claim that refactoring mitigates contamination is weakened. Running the refactored code through its original unit tests would also check whether semantics were preserved.","tokens_in":19122,"feed_emoji":"🧹","tokens_out":7729,"duration_ms":70785,"temperature":0.7,"pith_summary":"Data contamination—evaluation code appearing in a model's training corpus—inflates performance numbers and blocks companies from trusting code-language-model benchmarks. This paper argues that automated code refactoring is a practical, lightweight mitigation: it introduces CODECLEANER, a toolkit of 11 refactoring operators for Python (with four migrated to Java), and reports that applying all operators to method-level Python code cuts the average 50-gram overlap with the training corpus from 87% to 22%. The paper also finds that semantic operators—decorator insertion, special-parameter appending, and identifier renaming—are the most stably effective at making models less familiar with refactored code, while some syntactic and style operators have little or even negative model-wise effect. If the claim holds, teams can sanitize evaluation snippets before benchmarking instead of building contamination-free datasets from scratch.","feed_headline":"Code refactoring cuts training-data overlap from 87% to 22%","feed_subtitle":"An 11-operator toolkit makes evaluation code less recognizable to both n-gram matchers and code models.","key_machinery":"The carrying mechanism is a set of 11 refactoring operators that rewrite code at the method, class, and cross-class level while aiming to preserve semantics: if-condition flipping, loop and iteration transformations, commutative-law operand shuffling, method shuffling, decorator insertion, special-parameter appending, inherited-method appending, identifier renaming, code normalization, and naming-style switching. The operators are designed to break up long runs of consecutive characters that n-gram matching and models recognize, and their effect is measured by the 50-gram overlap ratio, the fraction of 50-character sequences shared with the training corpus, and by model-wise perplexity and Min-20% Prob scores.","core_discovery":"The central claim is that a set of 11 semantics-preserving refactoring operators can reduce how much evaluation code resembles training data, as measured by character overlap and by model familiarity. The paper reports that applying all operators to method-level Python code cuts the average overlap from 87% to 22%, a 65-percentage-point drop, and that semantic operators such as decorator insertion, parameter appending, and identifier renaming produce the largest and most stable reductions in model familiarity. On class-level code the best overlap reduction is 37 percentage points, and four operators migrated to Java show positive but smaller effects. The paper frames these results as evidence that automated refactoring is a viable lightweight alternative to building contamination-free evaluation datasets.","pith_inferences":["Because the snippets were sampled from the same corpus used for overlap measurement, the post-refactoring 22% overlap is not compared with code that was never in training; a control set of post-cutoff code would be needed to know whether refactored snippets actually resemble clean code.","The model-wise gains from semantic operators could partly reflect code that is semantically altered or stylistically odd rather than merely 'unmemorized'; a functional test of refactored snippets would separate those explanations.","The toolkit could be combined with membership-inference detectors as a validation loop: apply operators until a chosen detector no longer flags the snippet, then check that the refactored code still passes its unit tests.","Because the study uses four models and a single training corpus, the operator rankings may shift for other model families or other training corpora; users should re-run the measurement on their own models."],"forward_implications":["If the claim holds, teams evaluating code language models can preprocess evaluation code with CODECLEANER to reduce the risk that reported performance is inflated by memorized training data.","Semantic operators such as decorator insertion and identifier renaming are the ones to rely on when the goal is to lower model familiarity; syntactic rewrites alone may not be sufficient.","The method-level operators transfer to class-level code, though with smaller gains, so contamination mitigation on larger code units needs stronger or additional operators.","The positive but small Java results suggest the operator design is partially language-agnostic, but per-language tuning remains necessary.","A 65-percentage-point drop in overlap means refactored snippets share far fewer 50-gram sequences with the training corpus, making n-gram-based contamination detectors less likely to flag them."],"supporting_citations":[{"why":"Prior study of countermeasures for code-language-model data contamination; supplies the motivating finding that some refactoring operations can increase model familiarity and the baseline for operator choices.","marker":"[9]"},{"why":"Provides the character-overlap measurement and processed artifacts used to compute the 50-gram overlap ratio against the training corpus.","marker":"[33]"},{"why":"The public code corpus used as the reference for overlap and as the source of sampled snippets.","marker":"[45]"},{"why":"Proposes Min-K% Prob, the membership-inference metric used for the model-wise contamination measurement.","marker":"[16]"},{"why":"Classic definition of perplexity, the other model-wise familiarity metric.","marker":"[31]"},{"why":"The model whose training-subset artifacts are used in the overlap measurement pipeline.","marker":"[54]"}],"fun_headline_variants":["11 refactoring operators cut code overlap from 87% to 22%","CODECLEANER: 11 ways to shrink training-data overlap to 22%","Refactoring toolkit slashes code leakage by 65 points","Semantic refactoring cuts data contamination from 87% to 22%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the two chosen measurements—50-gram overlap with the training corpus and model-wise perplexity and Min-20% Prob scores—faithfully capture how contaminated a code snippet is, so that dropping those scores means contamination was actually mitigated; no control set of post-cutoff code is measured to support this.","fun_headline_variants_meta":{"raw":{"variants":["11 refactoring operators cut code overlap from 87% to 22%","CODECLEANER: 11 ways to shrink training-data overlap to 22%","Refactoring toolkit slashes code leakage by 65 points","Semantic refactoring cuts data contamination from 87% to 22%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000707,"raw_usage":{"total_tokens":3205,"prompt_tokens":984,"completion_tokens":2221,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":2136}},"tokens_in":600,"tokens_out":2221,"duration_ms":15511,"temperature":1.0,"reasoning_tokens":2136,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:14:23.707282+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of code snippets written after the studied models' training cutoffs, measure their 50-gram overlap and Min-20% Prob scores, then apply CODECLEANER to contaminated pre-cutoff snippets; if the refactored snippets still score far above the uncontaminated control on either metric, the claim that refactoring mitigates contamination is weakened. Running the refactored code through its original unit tests would also check whether semantics were preserved.","supporting_citations":[{"cited_title":"The stack: 3 tb of permissively licensed source code,","cited_arxiv_id":null,"evidence_quote":"The public code corpus used as the reference for overlap and as the source of sampled snippets."}],"review_version":1}