Pith. sign in

REVIEW 4 major objections 7 minor 1 cited by

CODECLEANER: Elevating Standards with A Robust Data Contamination Mitigation Toolkit

T0 review · 4 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that automated code refactoring can mitigate data contamination in code-language-model evaluation, reporting a 65-percentage-point drop in training-data overlap when all operators are applied to method-level Python code.

desk verdict Real toolkit, useful per-operator data, but the headline claim that refactoring 'mitigates data contamination' is not supported because the primary metric is partly by construction and the paper never checks behavioral equivalence. read the letter →

arxiv 2411.10842 v1 pith:NJI5NNT7 submitted 2024-11-16 cs.SE cs.AI

Jialun Cao , Songqiang Chen , Wuqi Zhang , Hau Ching Lo , Shing-Chi Cheung This is my paper · ORCID
classification cs.SEcs.AI
keywords datacontaminationcodelanguagemodelsrefactoringsemantic-preservingtransformationsn-gramoverlapperplexitymembershipinferencePython
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Data contamination—evaluation code appearing in a model's training corpus—inflates performance numbers and blocks companies from trusting code-language-model benchmarks. This paper argues that automated code refactoring is a practical, lightweight mitigation: it introduces CODECLEANER, a toolkit of 11 refactoring operators for Python (with four migrated to Java), and reports that applying all operators to method-level Python code cuts the average 50-gram overlap with the training corpus from 87% to 22%. The paper also finds that semantic operators—decorator insertion, special-parameter appending, and identifier renaming—are the most stably effective at making models less familiar with refactored code, while some syntactic and style operators have little or even negative model-wise effect. If the claim holds, teams can sanitize evaluation snippets before benchmarking instead of building contamination-free datasets from scratch.

What carries the argument

The carrying mechanism is a set of 11 refactoring operators that rewrite code at the method, class, and cross-class level while aiming to preserve semantics: if-condition flipping, loop and iteration transformations, commutative-law operand shuffling, method shuffling, decorator insertion, special-parameter appending, inherited-method appending, identifier renaming, code normalization, and naming-style switching. The operators are designed to break up long runs of consecutive characters that n-gram matching and models recognize, and their effect is measured by the 50-gram overlap ratio, the fraction of 50-character sequences shared with the training corpus, and by model-wise perplexity and Min-20% Prob scores.

What would settle it

Take a set of code snippets written after the studied models' training cutoffs, measure their 50-gram overlap and Min-20% Prob scores, then apply CODECLEANER to contaminated pre-cutoff snippets; if the refactored snippets still score far above the uncontaminated control on either metric, the claim that refactoring mitigates contamination is weakened. Running the refactored code through its original unit tests would also check whether semantics were preserved.

Watch

Extended reading notes

Core claim

The central claim is that a set of 11 semantics-preserving refactoring operators can reduce how much evaluation code resembles training data, as measured by character overlap and by model familiarity. The paper reports that applying all operators to method-level Python code cuts the average overlap from 87% to 22%, a 65-percentage-point drop, and that semantic operators such as decorator insertion, parameter appending, and identifier renaming produce the largest and most stable reductions in model familiarity. On class-level code the best overlap reduction is 37 percentage points, and four operators migrated to Java show positive but smaller effects. The paper frames these results as evidence that automated refactoring is a viable lightweight alternative to building contamination-free evaluation datasets.

Load-bearing premise

The paper assumes that the two chosen measurements—50-gram overlap with the training corpus and model-wise perplexity and Min-20% Prob scores—faithfully capture how contaminated a code snippet is, so that dropping those scores means contamination was actually mitigated; no control set of post-cutoff code is measured to support this.

Editorial extensions

If this is right

  • If the claim holds, teams evaluating code language models can preprocess evaluation code with CODECLEANER to reduce the risk that reported performance is inflated by memorized training data.
  • Semantic operators such as decorator insertion and identifier renaming are the ones to rely on when the goal is to lower model familiarity; syntactic rewrites alone may not be sufficient.
  • The method-level operators transfer to class-level code, though with smaller gains, so contamination mitigation on larger code units needs stronger or additional operators.
  • The positive but small Java results suggest the operator design is partially language-agnostic, but per-language tuning remains necessary.
  • A 65-percentage-point drop in overlap means refactored snippets share far fewer 50-gram sequences with the training corpus, making n-gram-based contamination detectors less likely to flag them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the snippets were sampled from the same corpus used for overlap measurement, the post-refactoring 22% overlap is not compared with code that was never in training; a control set of post-cutoff code would be needed to know whether refactored snippets actually resemble clean code.
  • The model-wise gains from semantic operators could partly reflect code that is semantically altered or stylistically odd rather than merely 'unmemorized'; a functional test of refactored snippets would separate those explanations.
  • The toolkit could be combined with membership-inference detectors as a validation loop: apply operators until a chosen detector no longer flags the snippet, then check that the refactored code still passes its unit tests.
  • Because the study uses four models and a single training corpus, the operator rankings may shift for other model families or other training corpora; users should re-run the measurement on their own models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper presents CodeCleaner, an open-source toolkit of 11 code refactoring operators (nine method-level, one class-level, one cross-class-level) for Python, with four operators migrated to Java, intended to mitigate data contamination in code language model (CLM) evaluation. The authors evaluate the operators on method-level and class-level code samples drawn from The Stack, using two families of metrics: a data-wise 50-gram overlap ratio against The Stack and model-wise perplexity/Min-20% Prob computed for four CLMs. The headline results are a reduction in overlap from 87% to 22% on method-level Python code and positive model-wise familiarity changes for semantic operators such as Deco and Renm. The paper also reports cross-language contamination severity (RQ3) and operator generalizability to Java (RQ4). The claims are that the operators 'demonstrate their effectiveness in addressing data contamination' and that the toolkit is a practical, automated alternative to building post-cutoff benchmarks.

Significance. The paper addresses a timely and industrially relevant problem—data contamination in CLM evaluation—and contributes a concrete, open-sourced artifact (CodeCleaner) along with the first systematic comparison of refactoring operators at method, class, and cross-class levels. The distinction between data-wise and model-wise measurements is useful, and the acknowledgment of limitations in Section VII is candid. The migration of operators to Java and the per-operator analysis of effectiveness are valuable empirical contributions. However, the validity of the central claim depends on two load-bearing issues: the data-wise overlap metric is nearly circular given the sampling design, and the model-wise metrics are confounded by the absence of any semantic-preservation validation. The paper's value would be substantially increased by adding a post-cutoff control corpus and behavioral equivalence checks, which are feasible within the manuscript's scope.

major comments (4)
  1. [§IV-A2, §IV-B1, §V-A (Fig. 12)] The data-wise effectiveness evidence is close to by construction. The code snippets used in RQ1–RQ2 are sampled from The Stack (Section IV-A2), and the overlap ratio is computed against The Stack (Section IV-B1); the operators are explicitly designed to disrupt consecutive character sequences, which is exactly the quantity measured. The reported drop from 87% to 22% in Figure 12 therefore restates the operators' design objectives rather than demonstrating that the refactored code is uncontaminated in any meaningful sense. The paper would need a control set—for example, code written after the models' training cutoff and therefore not in the corpus—to calibrate what overlap levels are actually indicative of contamination. Without such a control, the '65% decrease' claim is not evidence of contamination mitigation.
  2. [§III, §VII and model-wise results (Tables II–IV)] The paper repeatedly asserts that the operators preserve program semantics (e.g., 'without affecting the original semantic integrity' in Section I and 'keeping the code semantics untouched' in Section III-A), but it never executes, compiles, or differentially tests any refactored output. This matters because the model-wise metrics (perplexity and Min-20% Prob) are sensitive to code unnaturalness and semantic oddity; a syntactically valid but semantically altered snippet could show higher perplexity simply because it is strange, not because it is less contaminated. The authors themselves acknowledge this confound in Section VII: 'the semantic operators we implemented may inadvertently change the code semantics, making the model find the code semantics strange and thus exacerbating model-wise data contamination.' As it stands, the model-wise improvements in Tables II–IV cannot be attributed to contamination removal rather than code deterioration. The paper should add a behavioral equivalence check (e.g., running test suites, differential execution on sampled inputs, or at minimum compiling/parsing all refactored snippets) and report the fraction of snippets whose behavior is verified preserved.
  3. [§V-A, §V-B, Tables II–IV and Figure 11] The model-wise evaluation lacks any estimate of uncertainty. Tables II–IV report per-model and average Min-20% Prob differences, and Figure 11 shows rankings, but there are no error bars, confidence intervals, or significance tests. Many reported differences are small (e.g., 0.001 vs. -0.016 in Table II) and, without variance information, could easily be noise. In addition, the choice of K=20 in Min-20% Prob is taken from the authors' prior work [9] rather than justified on this dataset; a sensitivity analysis over K (e.g., K=10, 20, 30) is needed to establish that the operator rankings are not an artifact of this free parameter. The paper should report the distribution over the 384/324 snippets and test whether the operator effects are statistically distinguishable from zero and from each other.
  4. [§V-C (Figure 15)] RQ3's comparison of contamination severity across programming languages is circular for the same reason as RQ1–RQ2. The snippets for Java, C, Rust, and Python are all sampled from The Stack (Section IV-A2), and the reported severity is the 50-gram overlap against The Stack. High overlap rates (88–98%) are expected by construction when measuring a corpus against itself, so the finding that 'Java has the highest at 98%' is an artifact of the sampling design, not a characterization of contamination severity. The RQ3 claim should be reframed as 'overlap with The Stack' rather than 'data contamination severity,' or, preferably, the authors should sample an independent post-cutoff corpus to measure actual contamination levels.
minor comments (7)
  1. [§V-C] The text says 'The data-wise results are shown in Table 15' but the referenced object is labeled 'Figure 15'; please correct the cross-reference.
  2. [§IV-B2] The definition of Min-K% Prob is omitted ('The equation is omitted due to space limitation'). Since this is one of the two core model-wise metrics, the equation and the token-selection procedure should be included or, at minimum, given in an appendix.
  3. [Tables II–IV captions] The captions state 'The bluer, the better,' yet many entries are negative (e.g., Iter, Comm, Norm, Inhr in Tables II and III). Please clarify how the color scale handles negative values and how a reader should interpret a negative effect.
  4. [§V-A Setup] The setup paragraph says 'eight method-level operators' but the operator list in Figure 1 contains nine method-level operators (IFF, Loop, Iter, Comm, Deco, Param, Renm, Norm, Styl); please reconcile the count.
  5. [§V-A Data-wise Effect] The sentence 'the other operators are based on the operator Norm' is unclear. Does it mean that all operators are applied to code that has already been normalized? If so, please state this explicitly and explain how the reported reductions for individual operators are computed relative to the normalized baseline.
  6. [References] Reference [13] in the text is attributed to 'Razeghi et al.' but the bibliography entry lists 'Schick and Schütze, It's not just size that matters.' Please verify and correct the citation mapping.
  7. [§IV-A2 RQ3 data preparation] The paragraph states a total of 7680 snippets computed as 4 * 5 * 384, which matches the earlier statements, but the phrase 'we sampled data for the years 2018 ~ 2022' should clarify whether the 384 snippets per language are spread across years or per year; the current wording is ambiguous.

Circularity Check

2 steps flagged · score 6.0 of 10

The headline 65% overlap reduction is definitional: operators are designed to break exactly the 50-gram character overlaps used as the primary evidence; model-wise results are partly independent but depend on a self-cited K and an admitted semantic-strangeness confound.

  1. self definitional [Section II and Section IV-B1 (data-wise metric), headline result in Section V-A]
    "the key idea of code refactoring in resolving data contamination is to disrupt the consecutive characters/tokens as much as possible while maintaining the semantic unchanged. ... we quantify the degree of overlap using 50-gram overlap against the Stack."

    The primary effectiveness metric counts overlaps of consecutive non-empty 50-character sequences against The Stack. The operators were explicitly built to 'disrupt the consecutive characters/tokens as much as possible' via renaming, branch flipping, normalization, and style changes. Therefore the reported reduction from 87% to 22% overlap is the success criterion of the operator definitions, not an independent measurement that contamination was removed. Without a post-cutoff control or a behavioral equivalence check, the overlap drop cannot be distinguished from code alteration; the claimed mitigation result is the operator design goal restated as an outcome.

  2. self citation load bearing [Section IV-B2 (model-wise metric K selection)]
    "We set K as 20 because it produces the most distinguishable differences as shown in prior work [9]."

    The model-wise pillar of evidence uses Min-20% Prob as a membership-inference metric, and its K=20 setting is justified solely by [9], the authors' own prior arXiv preprint (Cao, Zhang, Cheung, arXiv:2403.16898). The operator rankings and averages in Tables II-IV depend on this metric, so a self-cited, non-independently-verified parameter choice is load-bearing for the model-wise conclusion. This is secondary to the definitional overlap issue but contributes to the partial circularity.

full rationale

CODECLEANER's central claim rests on two evidence pillars. The data-wise pillar (50-gram character overlap against The Stack) is circular by construction: Section II states the entire refactoring idea is 'to disrupt the consecutive characters/tokens as much as possible,' and Section IV-B1 measures exactly consecutive non-empty character overlaps. The headline 87% to 22% drop is therefore the objective function of the operators, not an external validation of decontamination. The model-wise pillar (perplexity, Min-20% Prob) is not definitional, but it relies on K=20 from the authors' own prior work, and Section VII concedes that 'the semantic operators we implemented may inadvertently change the code semantics, making the model find the code semantics strange and thus exacerbating model-wise data contamination.' That admitted confound means the model-wise improvements are plausibly caused by code unnaturalness rather than contamination removal. No post-cutoff control corpus is measured, so the paper never establishes that refactored snippets resemble genuinely uncontaminated code. These are partly validity threats, but the overlap-reduction result reduces by construction, so a mid-range circularity score is warranted. No uniqueness-theorem import or renaming of a known result is present.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper's central measurements depend on two proxies (n-gram overlap and MIA scores) that are assumed, not validated, to capture data contamination. The overlap metric is almost by construction sensitive to the refactoring operators, since renaming and restructuring necessarily change text. Semantic preservation is asserted but not machine-checked or differential-tested.

free parameters (4)
  • K in Min-K% Prob = 20
    Set to 20 because 'it produces the most distinguishable differences as shown in prior work [9]' (Section IV-B2); the threshold is chosen to maximize the metric's sensitivity rather than from an independent principle.
  • Number of renamed variables in class-level code (Renm) = 3
    RQ2 renames no more than three variables 'due to the time cost to request the online synonym library' (Section V-B), a practical cap that limits the operator's measured effect.
  • Number of inherited methods appended (Inhr) = 3
    RQ2 appends no more than three inherited methods 'to avoid lengthy classes' (Section V-B).
  • Sampling sizes = 384 methods, 324 classes, 7680 methods
    Chosen with a 95% confidence level and 5% margin of error (Section IV-A2); these are standard but still arbitrary sample sizes that influence all reported averages.
assumptions (5)
  • domain assumption N-gram overlap with The Stack (via DataPortraits) measures data contamination severity.
    Used throughout Section V and defined in Section IV-B1. The paper never validates this proxy against a held-out set of post-cutoff code, which is the natural ground truth.
  • domain assumption Perplexity and Min-20% Prob are valid membership-inference signals for code data contamination.
    Adopted from prior work [16], [31] in Section IV-B2; whether these signals correspond to actual contamination-driven performance inflation for code is not tested.
  • domain assumption The 11 refactoring operators preserve code semantics.
    Section III claims operators 'maintain the original code's semantics', but the paper acknowledges in Section VII that semantic operators 'may inadvertently change the code semantics'. No differential testing or execution checks are provided.
  • domain assumption The Stack v1.2 is representative of the training data of all four studied CLMs.
    Table I lists different training sources for the four models (e.g., CodeLlama is trained on GitHub+StackOverflow), yet the overlap measurements are computed only against the StarCoder training subset of The Stack (Section IV-B1).
  • standard math 95% confidence and 5% margin sampling from a corpus yields a representative sample of contamination.
    Stated in Section IV-A2; standard statistical sampling, but the representativeness depends on the population definition, which is the whole Stack corpus.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CODECLEANER: Elevating Standards with A Robust Data Contamination Mitigation Toolkit." pith.science (2026). https://pith.science/paper/NJI5NNT7

@misc{pith2026241110842,
  author       = {Pith},
  title        = {Pith review of: CODECLEANER: Elevating Standards with A Robust Data Contamination Mitigation Toolkit},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NJI5NNT7}},
  note         = {Machine review of arXiv:2411.10842}
}
read the original abstract

Data contamination presents a critical barrier preventing widespread industrial adoption of advanced software engineering techniques that leverage code language models (CLMs). This phenomenon occurs when evaluation data inadvertently overlaps with the public code repositories used to train CLMs, severely undermining the credibility of performance evaluations. For software companies considering the integration of CLM-based techniques into their development pipeline, this uncertainty about true performance metrics poses an unacceptable business risk. Code refactoring, which comprises code restructuring and variable renaming, has emerged as a promising measure to mitigate data contamination. It provides a practical alternative to the resource-intensive process of building contamination-free evaluation datasets, which would require companies to collect, clean, and label code created after the CLMs' training cutoff dates. However, the lack of automated code refactoring tools and scientifically validated refactoring techniques has hampered widespread industrial implementation. To bridge the gap, this paper presents the first systematic study to examine the efficacy of code refactoring operators at multiple scales (method-level, class-level, and cross-class level) and in different programming languages. In particular, we develop an open-sourced toolkit, CODECLEANER, which includes 11 operators for Python, with nine method-level, one class-level, and one cross-class-level operator. A drop of 65% overlap ratio is found when applying all operators in CODECLEANER, demonstrating their effectiveness in addressing data contamination. Additionally, we migrate four operators to Java, showing their generalizability to another language. We make CODECLEANER online available to facilitate further studies on mitigating CLM data contamination.

Figures

Figures reproduced from arXiv: 2411.10842 by the authors.

Figure 1
Figure 1. Code Refactoring Operators in CODECLEANER Stack [32] as a prior work [9] did to conduct our experiment. We apply different operators to it and investigate whether the perplexity or n-gram overlap ratios show significant changes. We study four research questions (RQs) accordingly. • RQ1. How do different code refactoring operators affect contamination severity for method-level Python code? We apply each code refactor… view at source ↗
Figure 2
Figure 2. Motivating Example of Code Refactoring Resolving Data Contamination. Green highlights the exactly matched characters with the Stack. The result is provided by prior work [33] and can be reproduced online [34]. the original code’s semantics while perturbing the consecutive tokens as much as possible. Also, the operators are better able to be applied automatically without human intervention and assistance, making them… view at source ↗
Figure 3
Figure 3. An Example of IF-condition Flipping (IFF) without Else-branch. Original code [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: An Example of Iteration Transformation ( [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: An Example of Commutative Law Shuffling ( [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: An Example of Special Parameter Appending ( [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 10
Figure 10. Figure 10: RQ1: Data-wise Effectiveness of 9 Method￾level Operators in Python Code. The lower, the better. Green / Blue / Yellow : Syntactical/Semantic/Code style oper￾ators. The green texts show the median. and only keep the methods with more than three lines of code (LoC) as p…
Figure 11
Figure 11. Figure 11: RQ1: Two Model-wise Metrics at Method-level Python (model: StarCoder-Instruct). Each bar shows the scores of the refactored code subtracted from the scores of the original code. The larger, the better. Green / Blue / Yellow : Syntactical/Semantic/Code style operators.…
Figure 12
Figure 12. Figure 12: RQ1: Best Trial on Method-level Python Code (Data-wise Measurement). Green shows the original overlap, while blue shows the overlap after applying all the operators in CODECLEANER. The lower the overlap with the training set, the less severe the data contamination. TA…
Figure 15
Figure 15. Figure 15: RQ3: Contamination Severity of Various Programming [PITH_FULL_IMAGE:figures/full_fig_p009_15.png]
Figure 14
Figure 14. Figure 14: RQ2: Best Trial on Class-level Python Code. Green shows the original overlap, while blue shows the overlap after applying all the operators in CODECLEANER once. The lower the overlap, the less severe the data contamination. Finding: In terms of the model-wise data con…
Figure 16
Figure 16. Figure 16: RQ4: Comparison of Operators on Java and Python [PITH_FULL_IMAGE:figures/full_fig_p010_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation

    cs.LG 2026-02 conditional novelty 4.0 of 10

    A taxonomy-driven survey arguing that reward design is the central mechanism shaping reliable LLM reasoning, with maps of reward paradigms, reward-hacking failure modes, and benchmark pitfalls.

Reference graph

Works this paper leans on

77 extracted references · 38 canonical work pages · cited by 1 Pith paper

  1. [9]

    Concerned with data contamina- tion? assessing countermeasures in code language model,

    J. Cao, W. Zhang, and S.-C. Cheung, “Concerned with data contamina- tion? assessing countermeasures in code language model,” arXiv preprint arXiv:2403.16898, 2024

    arXiv 2024

  2. [1]

    Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models,

    Y . Deng, C. S. Xia, H. Peng, C. Yang, and L. Zhang, “Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models,” in Proceedings of the 32nd ACM SIGSOFT interna- tional symposium on software testing and analysis , 2023, pp. 423–435

    2023

  3. [2]

    Large language models are edge-case generators: Crafting unusual programs for fuzzing deep learning libraries,

    Y . Deng, C. S. Xia, C. Yang, S. D. Zhang, S. Yang, and L. Zhang, “Large language models are edge-case generators: Crafting unusual programs for fuzzing deep learning libraries,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering , 2024, pp. 1–13

    work page 2024

  4. [3]

    Fuzz4all: Universal fuzzing with large language models,

    C. S. Xia, M. Paltenghi, J. Le Tian, M. Pradel, and L. Zhang, “Fuzz4all: Universal fuzzing with large language models,” Proc. IEEE/ACM ICSE , 2024

    work page 2024

  5. [4]

    Automated program repair in the era of large pre-trained language models,

    C. S. Xia, Y . Wei, and L. Zhang, “Automated program repair in the era of large pre-trained language models,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2023, pp. 1482–1494

    2023

  6. [5]

    Time travel in llms: Tracing data contamination in large language models,

    S. Golchin and M. Surdeanu, “Time travel in llms: Tracing data contamination in large language models,” CoRR, vol. abs/2308.08493,

    arXiv

  7. [6]

    NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark,

    O. Sainz, J. Campos, I. Garc ´ıa-Ferrero, J. Etxaniz, O. L. de Lacalle, and E. Agirre, “NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark,” in Findings of the Association for Computational Linguistics: EMNLP 2023 . Singapore: Association for Computational Linguistics, Dec. 2023, pp. 10 776–10 787. [Online]. Availa...

    work page 2023

  8. [7]

    Task contamination: Language models may not be few-shot anymore,

    C. Li and J. Flanigan, “Task contamination: Language models may not be few-shot anymore,” CoRR, vol. abs/2312.16337, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2312.16337

    arXiv doi:10.48550/arxiv.2312.16337 2023

Show all 77 references
  1. [8]

    An open source data contamination report for llama series models,

    Y . Li, “An open source data contamination report for llama series models,” arXiv preprint arXiv:2310.17589 , 2023

    2023 arXiv

  2. [10]

    Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLMs,

    S. Balloccu, P. Schmidtov ´a, M. Lango, and O. Dusek, “Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLMs,” in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Pap...

    2024

  3. [11]

    Memorization without overfitting: Analyzing the training dynamics of large language models,

    K. Tirumala, A. Markosyan, L. Zettlemoyer, and A. Aghajanyan, “Memorization without overfitting: Analyzing the training dynamics of large language models,” Advances in Neural Information Processing Systems, vol. 35, pp. 38 274–38 290, 2022

    2022

  4. [12]

    Deduplicating training data mitigates privacy risks in language models,

    N. Kandpal, E. Wallace, and C. Raffel, “Deduplicating training data mitigates privacy risks in language models,” in International Conference on Machine Learning . PMLR, 2022, pp. 10 697–10 707

    2022

  5. [13]

    It’s not just size that matters: Small language models are also few-shot learners,

    T. Schick and H. Sch ¨utze, “It’s not just size that matters: Small language models are also few-shot learners,” arXiv preprint arXiv:2009.07118 , 2020

    2009 arXiv

  6. [14]

    Data contamination: From memorization to exploitation,

    I. Magar and R. Schwartz, “Data contamination: From memorization to exploitation,” arXiv preprint arXiv:2203.08242 , 2022

    2022 arXiv

  7. [15]

    Datasets for large language models: A comprehensive survey,

    Y . Liu, J. Cao, C. Liu, K. Ding, and L. Jin, “Datasets for large language models: A comprehensive survey,” 2024. [Online]. Available: https://arxiv.org/abs/2402.18041

    2024 arXiv

  8. [16]

    Detecting pretraining data from large language models,

    W. Shi, A. Ajith, M. Xia, Y . Huang, D. Liu, T. Blevins, D. Chen, and L. Zettlemoyer, “Detecting pretraining data from large language models,” arXiv preprint arXiv:2310.16789 , 2023

    2023 arXiv

  9. [17]

    Extracting training data from large language models,

    N. Carlini, F. Tram `er, E. Wallace, M. Jagielski, A. Herbert-V oss, K. Lee, A. Roberts, T. B. Brown, D. Song, ´U. Erlingsson, A. Oprea, and C. Raffel, “Extracting training data from large language models,” in 30th USENIX Security Symposium, USENIX Security 2021, August 11-13,...

    2021

  10. [18]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  11. [19]

    An ethnographic study of copy and paste programming practices in oopl,

    M. Kim, L. Bergman, T. Lau, and D. Notkin, “An ethnographic study of copy and paste programming practices in oopl,” in Proceedings. 2004 International Symposium on Empirical Software Engineering, 2004. ISESE ’04., 2004, pp. 83–92

    2004

  12. [20]

    Github copilot,

    “Github copilot,” https://copilot.microsoft.com/, 2023

    2023

  13. [21]

    Github survey finds nearly all developers using ai coding tools,

    “Github survey finds nearly all developers using ai coding tools,” https://www.infoworld.com/article/3489925/ github-survey-finds-nearly-all-developers-using-ai-coding-tools.html, 2024

    2024

  14. [22]

    Generalization or memorization: Data contamination and trustworthy evaluation for large language models,

    Y . Dong, X. Jiang, H. Liu, Z. Jin, B. Gu, M. Yang, and G. Li, “Generalization or memorization: Data contamination and trustworthy evaluation for large language models,” in Findings of the Association for Computational Linguistics ACL 2024 . Bangkok, Thailand and virtual meeti...

    2024

  15. [23]

    Inferring resource-oriented intentions using llms for static resource leak detection,

    C. Wang, J. Liu, X. Peng, Y . Liu, and Y . Lou, “Inferring resource-oriented intentions using llms for static resource leak detection,” 2024. [Online]. Available: https://arxiv.org/abs/2311.04448

    2024 arXiv

  16. [24]

    Mr-adopt: Automatic deduction of input transformation function for metamorphic testing,

    C. Xu, S. Chen, J. Wu, S.-C. Cheung, V . Terragni, H. Zhu, and J. Cao, “Mr-adopt: Automatic deduction of input transformation function for metamorphic testing,” in 39th IEEE/ACM International Conference on Automated Software Engineering , 2024

    2024

  17. [25]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter,...

    2021

  18. [26]

    SWE-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “SWE-bench: Can language models resolve real-world github issues?” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https: //openreview.net/forum?...

    2024

  19. [27]

    On leakage of code generation evaluation datasets,

    A. Matton, T. Sherborne, D. Aumiller, E. Tommasone, M. Alizadeh, J. He, R. Ma, M. V oisin, E. Gilsenan-McMahon, and M. Gall ´e, “On leakage of code generation evaluation datasets,” arXiv preprint arXiv:2407.07565 , 2024

    2024 arXiv

  20. [28]

    Refactoring programs using large language models with few-shot examples,

    A. Shirafuji, Y . Oda, J. Suzuki, M. Morishita, and Y . Watanobe, “Refactoring programs using large language models with few-shot examples,” arXiv preprint arXiv:2311.11690 , 2023

    2023 arXiv

  21. [29]

    How effective are neural networks for fixing security vulnerabilities,

    Y . Wu, N. Jiang, H. V . Pham, T. Lutellier, J. Davis, L. Tan, P. Babkin, and S. Shah, “How effective are neural networks for fixing security vulnerabilities,” in Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis , ser. ISSTA 2023. Ne...

    2023

  22. [30]

    Exploring multi-lingual bias of large code models in code generation,

    C. Wang, Z. Li, C. Gao, W. Wang, T. Peng, H. Huang, Y . Deng, S. Wang, and M. R. Lyu, “Exploring multi-lingual bias of large code models in code generation,” arXiv preprint arXiv:2404.19368 , 2024

    2024 arXiv

  23. [31]

    Perplexity—a measure of the difficulty of speech recognition tasks,

    F. Jelinek, R. L. Mercer, L. R. Bahl, and J. K. Baker, “Perplexity—a measure of the difficulty of speech recognition tasks,” The Journal of the Acoustical Society of America , vol. 62, no. S1, pp. S63–S63, 1977

    1977

  24. [32]

    The Stack

    “The Stack.” [Online]. Available: https://huggingface.co/datasets/bigcode/ the-stack

  25. [33]

    Data portraits: Recording foundation model training data,

    M. Marone and B. Van Durme, “Data portraits: Recording foundation model training data,” in Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2023. [Online]. Available: https://arxiv.org/abs/2303.03919

    2023 arXiv

  26. [34]

    DataPortraits

    “DataPortraits.” [Online]. Available: https://dataportraits.org/

  27. [35]

    Code of sklearn.externals. arff.LODGeneratorData class

    “Code of sklearn.externals. arff.LODGeneratorData class.” [Online]. Available: https://github.com/scikit-learn/scikit-learn/blob/1.5.2/sklearn/ externals/ arff.py#L591

  28. [36]

    Kreuzer and L

    M. Kreuzer and L. Robbiano, Computational commutative algebra . Springer, 2000, vol. 1

    2000

  29. [37]

    Semmt: a semantic-based testing approach for machine translation systems,

    J. Cao, M. Li, Y . Li, M. Wen, S.-C. Cheung, and H. Chen, “Semmt: a semantic-based testing approach for machine translation systems,” ACM TOSEM, vol. 31, no. 2, pp. 1–36, 2022

    2022

  30. [38]

    Testing your question answering software via asking recursively,

    S. Chen, S. Jin, and X. Xie, “Testing your question answering software via asking recursively,” in 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2021, pp. 104–116

    2021

  31. [39]

    Structure-invariant testing for machine translation,

    P. He, C. Meister, and Z. Su, “Structure-invariant testing for machine translation,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering , ser. ICSE ’20. New York, NY , USA: Association for Computing Machinery, 2020, pp. 961–973

    2020

  32. [40]

    Validation on machine reading compre- hension software without annotated labels: A property-based method,

    S. Chen, S. Jin, and X. Xie, “Validation on machine reading compre- hension software without annotated labels: A property-based method,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engine...

    2021

  33. [41]

    Word closure-based metamorphic testing for machine translation,

    X. Xie, S. Jin, S. Chen, and S.-C. Cheung, “Word closure-based metamorphic testing for machine translation,” ACM Trans. Softw. Eng. Methodol., 2024

    2024

  34. [42]

    https://github.com/johnbumgarner/wordhoard, 2024

    2024

  35. [43]

    codellama/codellama-7b-instruct-hf,

    “codellama/codellama-7b-instruct-hf,” https://huggingface.co/codellama/ CodeLlama-7b-Instruct-hf, 2023

    2023

  36. [44]

    Starcoder-code-instruct,

    “Starcoder-code-instruct,” https://huggingface.co/ GeorgiaTechResearchInstitute/starcoder-gpteacher-code-instruct, 2023

    2023

  37. [45]

    The stack: 3 tb of permissively licensed source code,

    D. Kocetkov, R. Li, L. Ben Allal, J. Li, C. Mou, C. Mu ˜noz Ferrandis, Y . Jernite, M. Mitchell, S. Hughes, T. Wolf, D. Bahdanau, L. von Werra, and H. de Vries, “The stack: 3 tb of permissively licensed source code,” Preprint, 2022

    2022

  38. [46]

    Huggingfaceh4/starchat-beta,

    “Huggingfaceh4/starchat-beta,” 2023. [Online]. Avail- able: https://huggingface.co/HuggingFaceH4/starchat-beta/commit/ 4d8424c69643590f193cc97dc7eebff66500ebc6

    2023

  39. [47]

    Wizardlm/wizardcoder,

    “Wizardlm/wizardcoder,” https://huggingface.co/WizardLMTeam/ WizardCoder-15B-V1.0, 2023

    2023

  40. [48]

    Codereval: A benchmark of pragmatic code generation with generative pre-trained models,

    H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y . Ma, G. Liang, Y . Li, T. Xie, and Q. Wang, “Codereval: A benchmark of pragmatic code generation with generative pre-trained models,” arXiv preprint arXiv:2302.00288, 2023

    2023 arXiv

  41. [49]

    Scikit-Learn Project

    “Scikit-Learn Project.” [Online]. Available: https://github.com/ scikit-learn/scikit-learn

  42. [50]

    Pandas Project

    “Pandas Project.” [Online]. Available: https://github.com/pandas-dev/ pandas

  43. [51]

    NumPy Project

    “NumPy Project.” [Online]. Available: https://github.com/numpy/numpy

  44. [52]

    Investigating data contamination for pre-training language models,

    M. Jiang, K. Z. Liu, M. Zhong, R. Schaeffer, S. Ouyang, J. Han, and S. Koyejo, “Investigating data contamination for pre-training language models,” arXiv preprint arXiv:2401.06059 , 2024

    2024 arXiv

  45. [53]

    Documenting large webtext corpora: A case study on the colossal clean crawled corpus,

    J. Dodge, M. Sap, A. Marasovi ´c, W. Agnew, G. Ilharco, D. Groeneveld, M. Mitchell, and M. Gardner, “Documenting large webtext corpora: A case study on the colossal clean crawled corpus,” arXiv preprint arXiv:2104.08758, 2021

    2021 arXiv

  46. [54]

    Starcoder: may the source be with you!

    R. Li, L. B. Allal, Y . Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim et al., “Starcoder: may the source be with you!” arXiv preprint arXiv:2305.06161 , 2023

    2023 arXiv

  47. [55]

    The secret sharer: Evaluating and testing unintended memorization in neural networks,

    N. Carlini, C. Liu, ´U. Erlingsson, J. Kos, and D. Song, “The secret sharer: Evaluating and testing unintended memorization in neural networks,” in 28th USENIX Security Symposium , 2019, pp. 267–284

    2019

  48. [56]

    Language models are few-shot learners,

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B...

    2020

  49. [57]

    Palm: Scaling language modeling with pathways,

    A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann et al. , “Palm: Scaling language modeling with pathways,” Journal of Machine Learning Research, vol. 24, no. 240, pp. 1–113, 2023

    2023

  50. [58]

    The pile

    “The pile.” [Online]. Available: https://pile.eleuther.ai/

  51. [59]

    The Stack-V2

    “The Stack-V2.” [Online]. Available: https://huggingface.co/datasets/ bigcode/the-stack-v2

  52. [60]

    Investigating data contamination in modern benchmarks for large language models,

    C. Deng, Y . Zhao, X. Tang, M. Gerstein, and A. Cohan, “Investigating data contamination in modern benchmarks for large language models,” arXiv preprint arXiv:2311.09783 , 2023

    2023 arXiv

  53. [61]

    Truthfulqa: Measuring how models mimic human falsehoods,

    S. Lin, J. Hilton, and O. Evans, “Truthfulqa: Measuring how models mimic human falsehoods,” arXiv preprint arXiv:2109.07958 , 2021

    2021 arXiv

  54. [62]

    Measuring massive multitask language understanding,

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” arXiv preprint arXiv:2009.03300 , 2020

    2009 arXiv

  55. [63]

    Privacy risk in machine learning: Analyzing the connection to overfitting,

    S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha, “Privacy risk in machine learning: Analyzing the connection to overfitting,” in IEEE Computer Security Foundations Symposium , 2018, pp. 268–282

    2018

  56. [64]

    Zlib compression library,

    J.-l. Gailly and M. Adler, “Zlib compression library,” 2004

    2004

  57. [65]

    Membership inference attacks against language models via neighbourhood comparison,

    J. Mattern, F. Mireshghallah, Z. Jin, B. Sch ¨olkopf, M. Sachan, and T. Berg-Kirkpatrick, “Membership inference attacks against language models via neighbourhood comparison,” arXiv preprint arXiv:2305.18462, 2023

    2023 arXiv

  58. [66]

    Do membership inference attacks work on large language models?

    M. Duan, A. Suri, N. Mireshghallah, S. Min, W. Shi, L. Zettlemoyer, Y . Tsvetkov, Y . Choi, D. Evans, and H. Hajishirzi, “Do membership inference attacks work on large language models?” arXiv preprint arXiv:2402.07841, 2024

    2024 arXiv

  59. [67]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le et al., “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732 , 2021

    2021 arXiv

  60. [68]

    Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation,

    J. Liu, C. S. Xia, Y . Wang, and L. Zhang, “Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation,” in Thirty-seventh Conference on Neural Information Processing Systems , 2023. [Online]. Available: https://openrevie...

    2023

  61. [69]

    Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation,

    X. Du, M. Liu, K. Wang, H. Wang, J. Liu, Y . Chen, J. Feng, C. Sha, X. Peng, and Y . Lou, “Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation,” 2023

    2023

  62. [70]

    Livecodebench: Holistic and contamination free evaluation of large language models for code,

    N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica, “Livecodebench: Holistic and contamination free evaluation of large language models for code,” arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  63. [71]

    Large language models for software engineering: Survey and open problems,

    A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Survey and open problems,” arXiv preprint arXiv:2310.03533 , 2023

    2023 arXiv

  64. [72]

    Ai-assisted coding: Experiments with gpt-4,

    R. A. Poldrack, T. Lu, and G. Begu ˇs, “Ai-assisted coding: Experiments with gpt-4,” arXiv preprint arXiv:2304.13187 , 2023

    2023 arXiv

  65. [73]

    Chatbots as fluent polyglots: Revisiting breakthrough code snippets,

    D. Noever and K. Williams, “Chatbots as fluent polyglots: Revisiting breakthrough code snippets,” arXiv preprint arXiv:2301.03373 , 2023

    2023 arXiv

  66. [74]

    CodeSearchNet challenge: Evaluating the state of semantic code search,

    H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “CodeSearchNet challenge: Evaluating the state of semantic code search,” arXiv preprint arXiv:1909.09436 , 2019

    1909 arXiv

  67. [75]

    Measuring coding challenge competence with apps,

    D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt, “Measuring coding challenge competence with apps,” NeurIPS, 2021

    2021

  68. [76]

    Code Language Models trained on Stack

    “Code Language Models trained on Stack.” [Online]. Available: https://huggingface.co/models?dataset=dataset:bigcode/the-stack

  69. [2023]

    Available: https://doi.org/10.48550/arXiv.2308.08493

    [Online]. Available: https://doi.org/10.48550/arXiv.2308.08493

    doi arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.