Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Investigating Training Data Detection in AI Coders

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Existing training-data detection methods, built for natural language, mostly fail on source code: six of seven score around 0.6 AUC on a new 9,000-function benchmark, while the prefix-relative method ReCaLL reaches 0.796 average AUC in…

desk verdict Useful benchmark and the first systematic TDD-on-code study; the ReCaLL headline survives the harder checks, but the margin is partly a distribution artifact and the labels need verification. read the letter →

arxiv 2507.17389 v1 pith:XSCZRBKJ submitted 2025-07-23 cs.SE cs.AI

Tianlin Li , Yunxiang Wei , Zhiming Li , Aishan Liu , Qing Guo , Xianglong Liu , Dongning Sun , Yang Liu This is my paper
classification cs.SEcs.AI
keywords trainingdatadetectionmembershipinferencecodelargelanguagemodelsclonetaxonomySnitchbenchmarkmutationrobustnessReCaLLsourceprivacy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper investigates whether training-data detection (TDD) methods developed for natural language transfer to source code. It builds CodeSnitch, a 9,000-function benchmark in Python, Java, and C++ with member and non-member labels assigned by temporal heuristics, and evaluates seven TDD methods across eight code models. The central finding is that six methods hover around 0.6 AUC on the original benchmark and degrade toward random guessing when member code is renamed or rewritten, while the prefix-relative method ReCaLL reaches an average AUC of 0.796 on Python and above 0.8 on C++ and Java under the member-only mutation setting. The paper argues that code membership should be defined through the Type-1 to Type-4 code clone taxonomy rather than verbatim equality, because simple reformatting or identifier renaming defeats token-level detectors. It matters because code models that emit memorized proprietary snippets are a live compliance risk, and current detection tools are not dependable for that job.

What carries the argument

The benchmark CodeSnitch is the load-bearing object: 9,000 function-level samples across Python, Java, and C++, with member code drawn from pre-2024 training corpora and non-member code from post-2024 commits. Around it the paper builds a mutation module that rewrites member code according to the Type-1 to Type-4 clone taxonomy (formatting changes, identifier renaming, statement-level edits at similarity thresholds 0.9/0.7/0.5, and a hybrid of renaming plus statement edits). The method that carries the main result is ReCaLL's relative conditional log-likelihood score, which measures how much a non-member context prefix lowers a snippet's loss; the n-gram overlap analysis shows the prefix is statistically closer to non-member data, yet ReCaLL still works when the prefix is drawn out-of-distribution.

What would settle it

Run every post-2024 non-member function in CodeSnitch through a code-clone detector against the pre-2024 corpora used to build the member side; if a meaningful share are Type-1 or Type-2 clones of training-era code, the labels are contaminated and every reported AUC is biased. Repeating the whole evaluation on a split whose labels are verified by direct lookup against the actual training corpus would settle the central claim.

Watch

Extended reading notes

Core claim

The paper's central claim is that membership detection in code is a different problem from natural language, and current methods fail at it. On CodeSnitch, perplexity, zlib, Min-K%, Min-K%++, reference-model, and neighbor-comparison scoring all land near 0.6 AUC, with Min-K%++ at 0.595 on Python; their scores fall to roughly 0.5 when identifiers are replaced with random strings or statements are rewritten, i.e., random guessing. ReCaLL, which scores a snippet by the change in its conditional log-likelihood when a non-member prefix is prepended, $s(x;M)=\log p(x;M)-\log p(x\mid \text{prefix};M)$, reaches 0.796 average AUC in Python, above 0.8 in C++ and Java in Setting 1, and stays the best method under all clone-type-aware settings. The paper also shows that detection improves with model size and with code length up to roughly 250 tokens, and that the relative ranking of methods is stable across the three languages.

Load-bearing premise

Everything rests on the benchmark labels: snippets published before 2024 are treated as training members and snippets from after 2024 as non-members, so any post-2024 snippet that actually copies older training code carries a wrong label.

Editorial extensions

If this is right

  • Deploying any of the six natural-language methods (perplexity, zlib, Min-K%, Min-K%++, reference-model, neighbor-comparison) for code membership detection would give near-random accuracy on renamed or rewritten code, so they are not dependable compliance tools without further adaptation.
  • ReCaLL keeps working across Python, Java, and C++, including under formatting and identifier-renaming mutations, which makes relative scoring against a non-member context the strongest current signal for code membership.
  • Identifier renaming (Type-2) and statement-level rewrites (Type-3) defeat token-probability methods, so the paper's clone-aware membership definition is not a formality: it changes which detections are even possible.
  • Detection accuracy improves with model size and with snippet length up to roughly 250 tokens, then plateaus or declines, so short functions and smaller models are the hardest cases for any TDD method.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next stress test is semantic (Type-4) rewrites, which the paper leaves out of its mutation module; token-level and even prefix-relative scores are unlikely to survive a full functional rewrite.
  • If clone-aware membership becomes the standard, benchmark builders will need ground truth from direct clone matching against actual training corpora, since temporal proxies only approximate true membership.
  • Because ReCaLL's advantage rests on the difference between member and non-member prefix statistics, a training-time augmentation strategy that exposes the model to non-member-style prefixes could shrink that gap; testing this requires a controlled training run, not an audit benchmark.
  • The reported AUCs should be read as upper bounds under the temporal labeling assumption; a verified split could move every number.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces CodeSnitch, a function-level benchmark of 9,000 code samples in Python, Java, and C++ labeled as training members or non-members using temporal heuristics, and evaluates seven training-data-detection (TDD) methods (PPL, zlib, Min-K%, Min-K%++, Ref, Neighbor, ReCaLL) across eight CodeLLMs. It reports that ReCaLL substantially outperforms the other methods on the original benchmark and across three clone-type-aware mutation settings, while the other methods cluster around 0.6 AUC and degrade under Type-2 and Type-3 mutations. The paper also analyzes code-length effects and cross-language consistency, and releases the benchmark and evaluation pipeline.

Significance. If the benchmark labels are trustworthy, the paper makes a useful empirical contribution: it provides a public multi-language benchmark, a systematic comparison across seven methods and eight models, and a mutation framework grounded in the Type-1 to Type-4 clone taxonomy. The release of CodeSnitch and the pipeline, and the out-of-distribution prefix control for ReCaLL, are concrete strengths. However, the headline comparative claim (ReCaLL is consistently the best TDD method for code) is contingent on the validity of the temporal membership labels and on the non-member construction not being confounded with ReCaLL's prefix mechanism. Those two issues are load-bearing and are not fully resolved in the current manuscript.

major comments (4)
  1. [§4.2.1] The membership labels in CodeSnitch are inferred solely from temporal heuristics: members are pre-2024 snippets associated with corpora like The Stack and DeepSeek-Coder, while non-members are post-2024 GitHub commits or code using new APIs. No verification is reported (e.g., deduplication against the training corpora, manual audit, or contamination testing). Because post-2024 repositories can contain older code and pre-2024 corpora can contain code that resurfaces later, the label noise is potentially large and directly biases every AUC in Tables 1–4. I ask for label-quality statistics and/or a robustness analysis, such as a training-data-dependent verification or a synthetic member/non-member split.
  2. [§4.2.5 / §5.5] ReCaLL's advantage may be partly a distribution-match artifact. Section 4.2.5 samples ReCaLL's 12 prefixes from D_nm, and Section 5.5 shows these in-distribution prefixes have much higher n-gram overlap with non-member test data than with member data. The OOD-prefix experiment addresses token-level overlap, but the OOD prefixes are themselves post-2024 GitHub snippets, so the score can still exploit temporal or API-style cues rather than true membership. The Setting 2 results in Table 3 are consistent with partial confounding: on StarCoder2-7B, ReCaLL's margin over PPL shrinks from roughly 0.22 on the original benchmark (0.830 vs. 0.608) to roughly 0.115 in the Verbatim row (0.769 vs. 0.654). The Answer to RQ1 should be qualified, or supported by a non-member set that is independent of the prefix source.
  3. [§4.2.5] The choice of K for Min-K% and Min-K%++ is described as guided by prior work and 'preliminary tests,' but the validation protocol is not specified. If the preliminary tests used CodeSnitch test samples, the reported Min-K% results are tuned on the test set and thus optimistically biased; if a separate development set or fixed defaults were used, this should be stated. Please report the exact selection procedure and, if needed, include a sensitivity sweep over K rather than a single pair of values.
  4. [Table 1 / §4.2.5] No uncertainty estimates are provided for any AUC in Tables 1–4; Section 4.2.5 reports only averages over four seeds. Fine-grained comparative statements in Section 5.1 (e.g., 'PPL > Min% > zlib' and 'Min-K%++ underperforms PPL') involve differences of 0.01–0.03 AUC, which are within plausible sampling noise. Reporting standard deviations or confidence intervals, and where appropriate paired tests across the four seeds, would make the ranking claims evaluable.
minor comments (6)
  1. [§3.2] The word 'membeship' is a typo for 'membership.'
  2. [Fig. 5 / Fig. 6] The legend of Fig. 5 and the caption of Fig. 6 use 'ReCall' inconsistently with 'ReCaLL' used elsewhere in the paper.
  3. [§5.5] The text says 'ODD prefixes' where 'OOD prefixes' is intended.
  4. [§5.2] The Setting 3 paragraph contains a duplicated sentence: 'As shown in Fig 4, methods generally maintain an AUC around 0.6, with ReCaLL achieving nearly 0.8' appears twice.
  5. [Table 3] The 'Verbatim' clone type in Table 3 is defined only in the body text; adding a table note or caption definition would improve readability.
  6. [Notation] The paper alternates between 'Min-20%' and 'Min-K%' notation; a short notation table or consistent 'Min-K% (K=20)' format would help readers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the study is an external evaluation, and the only potential confound (ReCaLL's in-distribution prefixes) is explicitly controlled by an OOD-prefix experiment.

full rationale

The paper's central claims are empirical comparisons of seven existing TDD methods against a newly built benchmark, CodeSnitch. Membership labels are defined by temporal heuristics (pre-2024 training corpora for members, post-2024 GitHub commits for non-members), not by any score or method under test. No parameter is fitted to the target labels, and no method's output is used to define membership. The ReCaLL implementation samples its 12 prefixes from D_nm, which could in principle make non-member test samples easier to identify, but the paper explicitly acknowledges this in Section 5.5 and runs an out-of-distribution prefix control, obtaining nearly identical AUCs (0.830 vs. 0.829 on StarCoder2-7B). The clone-type-aware relabeling in Setting 2 is a definitional evaluation protocol, not a fitted prediction. The few self-citations (e.g., Refs. [11], [19], [20], [29]) are contextual and not load-bearing for the main result. Any concern about the temporal-heuristic labels is a benchmark-validity issue, not circularity, and the paper includes relevant threats-to-validity discussion. The derivation chain is therefore self-contained and does not reduce to its inputs.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new physical or architectural entities are introduced; CodeSnitch is a dataset artifact, not a postulated entity.

free parameters (2)
  • Min-K% / Min-K%++ K = 20 and 30
    Chosen based on prior work and the authors' own preliminary tests on the benchmark, i.e., tuned on the test data.
  • ReCaLL prefix count = 12
    Following the original ReCaLL paper; not fitted to data, but a manually chosen method parameter.
assumptions (4)
  • domain assumption Pre-2024 code from the-stack/deepseek-coder is assumed to be true training data for all eight models
    Section 4.2.1: member set is curated as highly likely to be in training, but actual membership is not verified.
  • domain assumption Post-2024 GitHub repos are assumed to be non-members
    Section 4.2.1: non-member set is built from repos and commits dated after all model release dates.
  • domain assumption Type-1 to Type-4 code clone taxonomy is the correct notion of membership under mutation
    Sections 2.3 and 3.2: the study redefines training-data membership for code using clone types.
  • standard math AUC computed from a random split with 4 seeds is a stable estimator of detector quality
    Section 4.2.5: only mean AUC across 4 seeds is reported, no variance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Investigating Training Data Detection in AI Coders." pith.science (2026). https://pith.science/paper/XSCZRBKJ

@misc{pith2026250717389,
  author       = {Pith},
  title        = {Pith review of: Investigating Training Data Detection in AI Coders},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XSCZRBKJ}},
  note         = {Machine review of arXiv:2507.17389}
}
read the original abstract

Recent advances in code large language models (CodeLLMs) have made them indispensable tools in modern software engineering. However, these models occasionally produce outputs that contain proprietary or sensitive code snippets, raising concerns about potential non-compliant use of training data, and posing risks to privacy and intellectual property. To ensure responsible and compliant deployment of CodeLLMs, training data detection (TDD) has become a critical task. While recent TDD methods have shown promise in natural language settings, their effectiveness on code data remains largely underexplored. This gap is particularly important given code's structured syntax and distinct similarity criteria compared to natural language. To address this, we conduct a comprehensive empirical study of seven state-of-the-art TDD methods on source code data, evaluating their performance across eight CodeLLMs. To support this evaluation, we introduce CodeSnitch, a function-level benchmark dataset comprising 9,000 code samples in three programming languages, each explicitly labeled as either included or excluded from CodeLLM training. Beyond evaluation on the original CodeSnitch, we design targeted mutation strategies to test the robustness of TDD methods under three distinct settings. These mutation strategies are grounded in the well-established Type-1 to Type-4 code clone detection taxonomy. Our study provides a systematic assessment of current TDD techniques for code and offers insights to guide the development of more effective and robust detection methods in the future.

Figures

Figures reproduced from arXiv: 2507.17389 by the authors.

Figure 1
Figure 1. The Four Types of Code Clones In this paper, we adopt the code clone definitions to systemati￾cally investigate how existing TDD methods perform when applied to codeLLMs. 3 Motivation Here, we first analyze the necessity of conducting this study—that is, why existing TDD methods designed for natural language (NL) data may not perform as effectively on code data. 3.1 Perspective 1: Distinct Code Structure The structu… view at source ↗
Figure 2
Figure 2. Overview of Our Study extract function snippets that either utilize new APIs released in 2024 (e.g., marimo) or originate from modifications in recent com￾mits made after 2024. The rationale is that code using new APIs is unlikely to appear in the training data of pre-released models. Moreover, code from recent commits—often addressing new re￾quirements—is less likely to consist of boilerplate templates that models … view at source ↗
Figure 3
Figure 3. The AUC drop of various methods when the Type [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: AUC Scores under Setting 3 (Isolated Mutation Im [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: AUC Scores as Code Length Increases test samples in this setup. The performance trends after mutation observed in C++ and Java are largely consistent in Python. Under Setting 1, performance for all methods degrades as the mutation level increases, whereas under Setting…
Figure 6
Figure 6. Figure 6: Ngram overlap (in log scale) between prefix in Re [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. CoDA: Towards Effective Cross-domain Knowledge Transfer via CoT-guided Domain Adaptation

    cs.AI 2026-04 unverdicted novelty 6.0 of 10

    CoDA aligns cross-domain latent reasoning representations in LLMs via CoT distillation and MMD to enable effective knowledge transfer without in-domain demonstrations.

Reference graph

Works this paper leans on

33 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    Our Website

    2025. Our Website. https://sites.google.com/view/codellms-tdd

    work page 2025

  2. [2]

    Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, Logesh Kumar Umapathi, Carolyn Jane Anderson, Yangtian Zi, Joel Lamy Poirier, Hailey Schoelkopf, Sergey Troshin, Dmitry Abulkhanov, Manuel Romero, Michael Lappert, Francesco De Toni, Bernardo García ...

    work page

  3. [3]

    Business of Apps. 2025. Microsoft Copilot Revenue and Usage Statistics (2025). https://www.businessofapps.com/data/microsoft-copilot-statistics/. Accessed 2025-06-18

    work page 2025

  4. [4]

    Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting Training Data from Large Lan- guage Models. In 30th USENIX Security Symposium (USENIX Security 21) . USENIX Association, 2633–2650. https://www.usenix.o...

    2021

  5. [5]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. arXiv:2002.08155 [cs.CL] https://arxiv.org/abs/2002.08155

    arXiv 2020

  6. [6]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wen- feng Liang. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming – The Rise of Code Intelligence. arXiv:2401.14196 [cs.SE] https: //arxiv.org/abs/2401.14196

    arXiv 2024

  7. [7]

    Hongsheng Hu, Zoran Salcic, Lichao Sun, Gillian Dobbie, Philip S Yu, and Xuyun Zhang. 2022. Membership inference attacks on machine learning: A survey.ACM Computing Surveys (CSUR) 54, 11s (2022), 1–37

    work page 2022

  8. [8]

    Minhao Jiang, Ken Ziyu Liu, Ming Zhong, Rylan Schaeffer, Siru Ouyang, Jiawei Han, and Sanmi Koyejo. 2024. Investigating Data Contamination for Pre-training Language Models. arXiv:2401.06059 [cs.CL] https://arxiv.org/abs/2401.06059

    arXiv 2024

Show all 33 references
  1. [9]

    Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Car- los Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. 2022. The Stack: 3 TB of permissively licensed source code. arXiv:...

    2022 arXiv

  2. [10]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, ...

    2023 arXiv

  3. [11]

    Tianlin Li, Qian Liu, Tianyu Pang, Chao Du, Qing Guo, Yang Liu, and Min Lin

  4. [12]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. arXiv:2305.01210 [cs.SE] https://arxiv. org/abs/2305.01210

    2023 arXiv

  5. [13]

    Zhenhua Liu, Tong Zhu, Chuanyuan Tan, Haonan Lu, Bing Liu, and Wen- liang Chen. 2024. Probing Language Models for Pre-training Data Detection. arXiv:2406.01333 [cs.CL] https://arxiv.org/abs/2406.01333

    2024 arXiv

  6. [14]

    Burton F. Maben. 2025. Privacy Leakage Attacks Against Large Language Models (Part1). LinkedIn. https://www.linkedin.com/pulse/privacy-leakage-attacks- against-large-language-models-burton-f-maben-xexec/ Published March 10, 2025; accessed 2025-06-18

    2025

  7. [15]

    Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schölkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick. 2023. Membership In- ference Attacks against Language Models via Neighbourhood Comparison. arXiv:2305.18462 [cs.CL] https://arxiv.org/abs/2305.18462

    2023 arXiv

  8. [16]

    Li et al

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiao- qing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Conference acronym ’XX, June 03–05, 2018, Woodstock, NY T. Li et al. Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton...

    2018 arXiv

  9. [17]

    Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2024. Detecting Pretraining Data from Large Language Models. arXiv:2310.16789 [cs.CL] https://arxiv.org/abs/ 2310.16789

    2024 arXiv

  10. [18]

    Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov

  11. [19]

    Chong Wang, Zhenpeng Chen, Tianlin Li, Yilun Zhao, and Yang Liu. 2024. To- wards Trustworthy LLMs for Code: A Data-Centric Synergistic Auditing Frame- work. arXiv:2410.09048 [cs.SE] https://arxiv.org/abs/2410.09048

    2024 arXiv

  12. [20]

    Kaixin Wang, Tianlin Li, Xiaoyu Zhang, Chong Wang, Weisong Sun, Yang Liu, and Bin Shi. 2025. Software Development Life Cycle Perspective: A Survey of Bench- marks for Code Large Language Models and Agents. arXiv:2505.05283 [cs.SE] https://arxiv.org/abs/2505.05283

    2025

  13. [21]

    Yueming Wu, Wenqi Suo, Siyue Feng, Cong Wu, Deqing Zou, and Hai Jin. 2025. Fine-Grained Code Clone Detection by Keywords-Based Connection of Program Dependency Graph. IEEE Transactions on Reliability (2025), 1–15. doi:10.1109/ TR.2025.3550747

    2025

  14. [22]

    Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. Automated Pro- gram Repair in the Era of Large Pre-trained Language Models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 1482–1494. doi:10.1109/icse48619.2023.00129

    2023

  15. [23]

    Roy Xie, Junlin Wang, Ruomin Huang, Minxing Zhang, Rong Ge, Jian Pei, Neil Zhenqiang Gong, and Bhuwan Dhingra. 2025. ReCaLL: Membership In- ference via Relative Conditional Log-Likelihoods. arXiv:2406.15968 [cs.CL] https://arxiv.org/abs/2406.15968

    2025 arXiv

  16. [24]

    Chengran Yang, Hong Jin Kang, Jieke Shi, and David Lo. 2024. ACECode: A Re- inforcement Learning Framework for Aligning Code Efficiency and Correctness in Code Language Models. arXiv:2412.17264 [cs.SE] https://arxiv.org/abs/2412. 17264

    2024 arXiv

  17. [25]

    Zhou Yang, Zhipeng Zhao, Chenyu Wang, Jieke Shi, Dongsum Kim, Dong- gyun Han, and David Lo. 2024. Gotcha! This Model Uses My Code! Evalu- ating Membership Leakage Risks in Code Models. arXiv:2310.01166 [cs.SE] https://arxiv.org/abs/2310.01166

    2024 arXiv

  18. [26]

    Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. 2018. Pri- vacy Risk in Machine Learning: Analyzing the Connection to Overfitting. arXiv:1709.01604 [cs.CR] https://arxiv.org/abs/1709.01604

    2018 arXiv

  19. [27]

    Zhiyuan Yu, Yuhao Wu, Ning Zhang, Chenguang Wang, Yevgeniy Vorobeychik, and Chaowei Xiao. 2023. Codeipprompt: Intellectual property infringement assessment of code language models. In International Conference on Machine Learning. PMLR, 40373–40389

    2023

  20. [28]

    Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang, and Hai Li. 2025. Improved Baseline for Detecting Pre-Training Data from Large Language Models. arXiv:2404.02936 [cs.CL] https://arxiv.org/ abs/2404.02936

    2025 arXiv

  21. [29]

    Jian Zhang, Chong Wang, Anran Li, Wenhan Wang, Tianlin Li, and Yang Liu. 2024. VulAdvisor: Natural Language Suggestion Generation for Software Vulnerability Repair. In 2024 39th IEEE/ACM International Conference on Automated Software Engineering (ASE). 1932–1944

    2024

  22. [30]

    Weichao Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. 2025. Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method. arXiv:2409.14781 [cs.CL] https://arxiv. org/abs/2409.14781

    2025 arXiv

  23. [2017]

    arXiv:1610.05820 [cs.CR] https://arxiv.org/abs/1610.05820

    Membership Inference Attacks against Machine Learning Models. arXiv:1610.05820 [cs.CR] https://arxiv.org/abs/1610.05820

    arXiv

  24. [2023]

    SantaCoder: don’t reach for the stars! arXiv:2301.03988 [cs.SE] https: //arxiv.org/abs/2301.03988

    arXiv

  25. [2024]

    arXiv preprint arXiv:2402.14845 (2024)

    Purifying large language models by ensembling a small language model. arXiv preprint arXiv:2402.14845 (2024)

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.