Pith. sign in

REVIEW 6 major objections 6 minor 106 references

Differentiation-Based Extraction of Proprietary Data from Fine-Tuned LLMs

T0 review · 6 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that supervised fine-tuning data—the valuable instruction-response pairs behind specialized LLM services—can be extracted from a deployed model through ordinary API queries, and that DDE, its differentiation-based method…

desk verdict First solid formalization of targeted SFT extraction with a plausible attack, but the method's key branch-generation step is under-specified enough that the headline numbers should be treated with caution until the authors clarify it. read the letter →

arxiv 2506.17353 v1 pith:JYQ2R2CG submitted 2025-06-20 cs.CR cs.AI

Zongjie Li , Daoyuan Wu , Shuai Wang , Zhendong Su This is my paper · ORCID
classification cs.CRcs.AI
keywords supervisedfine-tuningdataextractionattackLLMprivacytokenlogitsbranchdeviationreconstructionretraininginstruction-responsepairs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that the supervised fine-tuning (SFT) data used to specialize a large language model can be extracted from the deployed model by an attacker who only queries the API and reads per-token probabilities. Its proposed method, Differentiated Data Extraction (DDE), targets low-confidence token positions in the model's output, generates alternative continuations from those positions in both the fine-tuned model and an ordinary base model, and selects the continuation most likely to match the original training text. Across code and math domains, the paper reports that DDE beats straightforward querying by 9.96% and an earlier extraction method by 5.73% on reconstruction, and by 9.41% and 11.52% on retraining. If the claim holds, a fine-tuned model's most valuable asset—its proprietary instruction-response dataset—is recoverable through the model's own public interface.

What carries the argument

The load-bearing mechanism is branch-point identification via a confidence threshold on per-token logits, followed by two-model branch comparison. DDE queries the fine-tuned model with greedy decoding, records the probability of each generated token, and flags any token below a threshold $\tau$ as a potential branching point. It then continues generation from those prefixes in both the fine-tuned model and a base model, forming an SFT branch set and a base branch set; Algorithm 1 selects from the SFT set the branch closest on average to all base branches and the branch most distant from its own siblings. These two representatives, plus the masked-data completion step for retraining attacks, carry the entire argument that uncertain-generation behaviors expose the original training text.

What would settle it

Apply DDE to a fine-tuned model served through an API that masks all token probabilities (the provider returns only the decoded string); if reconstruction BLEU falls to the Vanilla baseline's roughly 0.146 with 0% exact matches, the logits-based branch-point mechanism is the load-bearing component. Alternatively, compute NTC on a model fine-tuned on a private held-out dataset: if NTC is not clearly above the base model's next-token accuracy, the premise that SFT text remains retrievable is wrong.

Watch

Extended reading notes

Core claim

The central discovery is that fine-tuned models retain their SFT data at token level but ordinary generation rarely reproduces it because of branch deviation: one wrong token diverts the autoregressive process permanently. DDE removes that obstacle by using the fine-tuned model's confidence signal to find branch points, then comparing branches from the fine-tuned model with branches from a base model that lacks the SFT knowledge. From the fine-tuned model's branch set it picks two representatives, one closest to the base-model space and one most unlike its sibling branches, capturing underfitted and well-learned data respectively; a masked-data completion step then fills gaps in partially known instructions or responses. The paper reports an average Next Token Correction score of 0.8297, meaning about 83% of SFT tokens are recoverable if branch deviations are corrected, and presents experiments where DDE outperforms both baselines in every attack setting while matching the computational cost of simple querying.

Load-bearing premise

The attack collapses if the victim API does not return per-token logits or returns sanitized ones, because DDE needs those probabilities to identify branch points; the paper's vendor survey found that only 8 of 14 providers currently return non-null logits and one provider does not support the feature at all.

Editorial extensions

If this is right

  • A fine-tuned model served through an API can be forced to reproduce substantial portions of its proprietary SFT dataset, not just generic memorized text.
  • The extracted pairs are usable for retraining: models fine-tuned on DDE output approach the victim's benchmark performance, making SFT data a recoverable business asset.
  • Partial knowledge suffices: with a fraction of the instruction or response preserved, the missing side can be recovered, and higher retention rates generally improve recovery.
  • Attackers do not need the exact base model; different open models of similar capability retain most of DDE's advantage.
  • Providers can blunt DDE by rewriting returned logits to raise the top token's apparent confidence, leaving greedy decoding unchanged and most sampling behavior intact.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If logits become unavailable, the natural escalation is empirical distribution estimation through many one-token queries; the paper names this adaptive route but does not quantify its cost, leaving an open measurement.
  • DDE's branch-deviation logic should transfer to parameter-efficient fine-tuning (for example, LoRA), since the method assumes only a fine-tuned model and a base model with different behavior, not full-parameter SFT.
  • The closest/outlier selection rule could double as a training-data auditing tool: low-confidence branch points mark tokens the model did not firmly memorize, while outlier branches mark tokens it did.
  • Because the attack needs only query access and logits, the value of SFT datasets themselves becomes an attack surface, shifting part of the privacy problem from model weights to API design.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. This paper studies extraction of supervised fine-tuning (SFT) instruction-response pairs from fine-tuned LLMs. The authors formalize attack goals (reconstruction vs. retraining), attack types (I-R and R-I), and attack variants with partial information preservation (PWP, PSP, SSP). They propose Differentiated/Differentiate Data Extraction (DDE), which greedily generates from the fine-tuned model, flags low-confidence tokens as branch points, generates alternative branches, and selects a 'closest' branch and an 'outlier' branch by comparing SFT-model branches against base-model branches. The paper reports that DDE outperforms Vanilla extraction and DSR across code (CodeLlama/OSS-Instruct) and math (LLaMA2/MathInstruct) domains, with average improvements of 9.96% and 5.73% for reconstruction and 9.41% and 11.52% for retraining. It also proposes a logit-modification defense and evaluates the defense's impact on generation performance. The appendix includes a pilot NTC analysis, a vendor survey of logits availability, and additional ablation results.

Significance. If the results are reliable, this is a meaningful step: it is a first systematic treatment of targeted SFT data extraction from LLMs, it introduces a concrete attack mechanism that exploits fine-tuning-induced confidence changes, and it distinguishes reconstruction from retraining goals. The pilot NTC analysis (average 0.8297 in Section 3.4) is a useful diagnostic, and the vendor logits survey in Appendix I grounds the threat model in current API practice. The extensive evaluation across domains, preservation methods, and retention rates is a strength, as is the inclusion of both reconstruction and retraining objectives. However, the main claims are currently supported by point estimates without uncertainty quantification, and several method details (branch generation, distance metric) are under-specified; the defense is not directly evaluated against the attack. With these gaps addressed, the paper could be a solid contribution to LLM security.

major comments (6)
  1. [§4.2, Figure 3] Step ② (New Branches Generation) is not specified precisely enough to reproduce the claimed branch diversity. Section 4.1 says DDE forces M_FT to continue generation with the second most probable token at potential deviation points, but §4.2 only says that prefixes obtained from step ① are used to query both models and that branches are generated under these prefixes; it never states that each prefix is formed by replacing the flagged low-confidence token with its next-best alternative, nor does it specify the decoding strategy (greedy vs. sampling) for continuing the branch. If the prefix ends immediately before the flagged token, greedy decoding will regenerate the same token and hence the same original branch, making DDE equivalent to Vanilla with extra queries. Please provide the exact branch-generation procedure, including how the alternative token is chosen and how many alternatives are generated per branch point.
  2. [Algorithm 1] The distance metric used in Representative Selection is undefined. Algorithm 1 computes 'avg. distance between s_i and all b in B' and 'avg. distance between s_i and all s_j in S', but no distance function is specified anywhere in the paper; BLEU, continuous token matching, and embedding similarity are defined in §5 only as evaluation metrics. Because the 'Closest' and 'Outlier' branches are the outputs of DDE, the selection rule is not reproducible without this definition.
  3. [§6.1, Table 6] The central claim that DDE 'consistently outperforms' Vanilla and DSR is not supported by any measure of statistical uncertainty. Table 6 reports a single value per setting, and the two ablation tables (Tables 9 and 10) are each based on 100 queries. Several improvements in Table 6 are small (e.g., I-R CodeLlama BLEU 'Full': Vanilla 0.651 vs. DDE 0.676), so confidence intervals or multiple random seeds are needed to establish that the average improvements of 9.96% and 5.73% are not noise.
  4. [§6.4, Table 10] The branch-point threshold τ is selected on the same evaluation setup used for the main results. Table 10 sweeps τ on the code domain and concludes that 0.8 is optimal, and the main results in Table 6 use τ = 0.8 on the same data. Because DDE's BLEU at τ = 0.2 (0.6127) is below the Vanilla baseline (0.651), the choice of τ is load-bearing; a held-out tuning procedure or a sensitivity analysis that does not select on the test set is needed.
  5. [§5, Compared Baselines] The procedure for matching Vanilla's query count to DDE's branch count is underspecified. DDE generates multiple distinct branches per instruction, whereas greedy decoding is deterministic; if Vanilla repeats the same greedy output when given the same input, matching query counts does not give it additional chances to recover the ground truth. The paper should state whether Vanilla uses sampling (and at what temperature/decoding parameters) or whether the multiple queries are over different inputs, so that the fairness comparison is meaningful.
  6. [§7, Defense evaluation] The defense is claimed to 'fail DDE on the extracted data', but Figure 7 evaluates only the defended model's generation performance (HumanEval pass rates across temperature/top_p settings); there is no experiment measuring whether DDE's branch-point identification or extraction success is actually reduced. Without such an evaluation, the defensive contribution is not supported by the reported evidence.
minor comments (6)
  1. [Title/Abstract] The method name is inconsistent: 'Differentiate Data Extraction' is used in the Abstract and Section 1, while 'Differentiated Data Extraction' appears in the title and Section 4; please standardize.
  2. [Throughout] There are several typographical issues: 'V anilla' appears with a space throughout (e.g., Abstract, §3.4), 'seperately' in Section 1 should be 'separately', and 'inlines' in Section 6.1 should be 'aligns' or 'is in line with'.
  3. [§5, Table 5] The number of training epochs is not reported; the paper only states that the checkpoint with the lowest validation loss is used. Please add the epoch count or the early-stopping criterion.
  4. [Appendix F] The theoretical formula P(success) = 1 - (1 - NTC^k)^{(L-k+1)} is presented without defining k and L in the caption or surrounding text; please define these variables explicitly.
  5. [§6.4, Tables 9 and 10] Each configuration in Tables 9 and 10 is based on 100 queries; please report the standard deviation or justify why this sample size is sufficient, especially because Table 10 is used to select τ.
  6. [Appendix I, Table 15] Since only 8 of 14 vendors currently return non-null logits and one vendor (Replicate) does not support the feature, the abstract's phrase 'all attack settings' should be qualified to settings where per-token logits are available; the current text in Section 3.1 is accurate but the abstract overstates the practical scope.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: DDE is an empirical attack evaluated against external baselines; no derived quantity reduces to its inputs.

full rationale

The paper's central claim is empirical: DDE outperforms Vanilla and DSR at reconstructing and retraining on SFT data. No equation in the paper derives a predicted performance number from a fitted input, and no load-bearing argument reduces to a self-citation. The NTC metric in Section 3.4 is a direct measurement of next-token accuracy given ground-truth prefixes; it is evidence of retention, not a circular derivation of it. The branch-point threshold tau is tuned on the evaluation setup in Table 10, but the paper presents this as a hyperparameter sensitivity study rather than as a prediction, and the main comparisons are reported as measured attack outcomes against external baselines. The closest/outlier selection in Algorithm 1 is a heuristic justified by measured branch-contribution statistics, not a quantity defined in terms of the target ground truth. Self-citations such as [51] are used for baseline methodology references and are not load-bearing for the paper's new claims. Appendix I's observation that only 8 of 14 vendors currently return non-null logits is a threat-model limitation, not circularity. Overall, the derivation chain is self-contained as an experimental study.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central claim rests primarily on empirical assumptions about attacker access (logits, base model, partial knowledge of I or R) and on the heuristic validity of branch-point and representative-selection rules. There are no invented physical entities, and the only fitted parameters are the branch threshold (tau=0.8), max branch count (10), and the post hoc token-match window (25).

free parameters (3)
  • tau (branch point threshold) = 0.8
    Threshold below which a token is flagged as a branch point; set to 0.8 based on a sweep on the code domain (Table 10). The paper shows performance is stable for 0.6-0.9, but the headline results use 0.8.
  • MBR (max branch count) = 10
    Maximum number of branch points tracked per query; fixed at 10 without sensitivity analysis (Table 5).
  • Token match window = 25
    Continuous token matching uses 25 tokens instead of 50 in prior work; changed post hoc because 'overly long windows can be suboptimal' (Appendix F), which is a metric tuning choice affecting the reported Token results.
assumptions (6)
  • domain assumption Attackers can obtain token logits from the victim model.
    Required for DDE's branch point identification; supported by the authors' survey (Appendix I) which shows 8/14 vendors return logits, but this is not guaranteed for all APIs.
  • domain assumption Attackers have access to a reasonably capable base model, possibly not the exact one.
    DDE compares M_FT branches with M_Base branches; experiments in Section 6.4 show Gemma-7B and ChatGLM3-6B also work, so this is partially validated.
  • domain assumption Attackers have partial knowledge of instruction or response (PWP/PSP/SSP).
    The three preservation variants in Section 3.2 model partial knowledge at 25/50/75% retention; without such partial knowledge the attack cannot initiate I-R or R-I queries.
  • ad hoc to paper Low-probability tokens mark re-routable branch points.
    DDE's core mechanism assumes that when the fine-tuned model assigns low probability to a token, generating from the second-best alternative creates useful candidate branches; this is validated only empirically.
  • ad hoc to paper The base model's branch set is a useful reference: closest-to-base and outlier-from-SFT branches are the best candidates.
    Algorithm 1's selection heuristic is motivated by an underfitting/overfitting story but is not derived; the paper gives no formal argument that these two branches capture the original data.
  • domain assumption The attacker's threshold tau is known to the defender.
    Stated in Section 7: 'our defense assumes that the attacker's threshold is known.' This is a strong assumption that constrains the defense's practical use.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Differentiation-Based Extraction of Proprietary Data from Fine-Tuned LLMs." pith.science (2026). https://pith.science/paper/JYQ2R2CG

@misc{pith2026250617353,
  author       = {Pith},
  title        = {Pith review of: Differentiation-Based Extraction of Proprietary Data from Fine-Tuned LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JYQ2R2CG}},
  note         = {Machine review of arXiv:2506.17353}
}
read the original abstract

The increasing demand for domain-specific and human-aligned Large Language Models (LLMs) has led to the widespread adoption of Supervised Fine-Tuning (SFT) techniques. SFT datasets often comprise valuable instruction-response pairs, making them highly valuable targets for potential extraction. This paper studies this critical research problem for the first time. We start by formally defining and formulating the problem, then explore various attack goals, types, and variants based on the unique properties of SFT data in real-world scenarios. Based on our analysis of extraction behaviors of direct extraction, we develop a novel extraction method specifically designed for SFT models, called Differentiated Data Extraction (DDE), which exploits the confidence levels of fine-tuned models and their behavioral differences from pre-trained base models. Through extensive experiments across multiple domains and scenarios, we demonstrate the feasibility of SFT data extraction using DDE. Our results show that DDE consistently outperforms existing extraction baselines in all attack settings. To counter this new attack, we propose a defense mechanism that mitigates DDE attacks with minimal impact on model performance. Overall, our research reveals hidden data leak risks in fine-tuned LLMs and provides insights for developing more secure models.

Figures

Figures reproduced from arXiv: 2506.17353 by the authors.

Figure 1
Figure 1. Attack scenario on SFT models. “Inst” and “Res” [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Simplified examples of possible attack variants [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. An overview of DDE’s four-step workflow: (1) Branch points identification, (2) New branches generation, (3) Represen￾tative selection, and (4) Masked data completion. The figure illustrates an I-R attack example using the instruction “Why are apples good?”, with token generation probabilities in parentheses. It depicts the SFT and base model branches in Step ②, and data extraction for reconstruction and retraining a… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Example of extracted code from OSS-Instruct [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 6
Figure 6. Figure 6: Performance comparison of DDE across an increas￾ing number of examples. The graph shows cumulative aver￾age Embed scores for reconstruction attacks (left y-axis) and SFT performance for retraining attacks (right y-axis). behavior aligns with our initial hypothesis rega…
Figure 7
Figure 7. Figure 7: Impact of defense across various Temperature and [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Performance comparison of retraining attacks un [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Comparison of actual results and theoretical proba [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

106 extracted references · 41 canonical work pages

  1. [1]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

    2020

  2. [2]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Se- bastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022)

    arXiv 2022

  3. [3]

    Zongjie Li, Daoyuan Wu, Shuai Wang, and Zhendong Su. 2025. Api-guided dataset synthesis to finetune large code models. Proceedings of the ACM on Programming Languages 9, OOPSLA1 (2025), 786–815

    2025

  4. [4]

    Santiago Zanella-Béguelin, Lukas Wutschitz, Shruti Tople, Victor Rühle, Andrew Paverd, Olga Ohrimenko, Boris Köpf, and Marc Brockschmidt. 2020. Analyzing information leakage of updates to natural language models. In Proceedings of the 2020 ACM SIGSAC conference on computer and communications security . 363–375

    2020

  5. [5]

    2020.{Updates-Leak}: Data set inference and reconstruction attacks in online learning

    Ahmed Salem, Apratim Bhattacharya, Michael Backes, Mario Fritz, and Yang Zhang. 2020.{Updates-Leak}: Data set inference and reconstruction attacks in online learning. In 29th USENIX security symposium (USENIX Security 20)

    2020

  6. [6]

    Matthew Jagielski, Stanley Wu, Alina Oprea, Jonathan Ullman, and Roxana Geambasu. 2023. How to combine membership-inference attacks on multiple updated machine learning models. Proceedings on Privacy Enhancing Technolo- gies (2023)

    2023

  7. [7]

    Tian Hui, Farhad Farokhi, and Olga Ohrimenko. 2023. Information Leakage from Data Updates in Machine Learning Models. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security . 35–41

    2023

  8. [8]

    Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21) . 2633–2650

    2021

Show all 106 references
  1. [9]

    Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. 2023. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:...

    2023 arXiv

  2. [10]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.arXiv e-prints (2019). arXiv:1910.10683

    2019 arXiv

  3. [11]

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al

  4. [12]

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al

  5. [13]

    Daixuan Cheng, Shaohan Huang, and Furu Wei. 2024. Adapting Large Language Models via Reading Comprehension. In The Twelfth International Conference on Learning Representations

    2024

  6. [14]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)

    2021 arXiv

  7. [15]

    Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2021. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602 (2021)

    2021 arXiv

  8. [16]

    OpenAI. 2025. OpenAI API. https://openai.com/product

    2025

  9. [17]

    Together AI. 2025. Together AI. https://www.together.ai/. Differentiation-Based Extraction of Proprietary Data from Fine-Tuned LLMs Conference’17, July 2017, Washington, DC, USA

    2025

  10. [18]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhu- patiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295 (2024)

    2024 arXiv

  11. [19]

    [n. d.]. LLaMa2-7B Base. https://huggingface.co/meta-llama/Llama-2-7b-hf

  12. [20]

    André V Duarte, Xuandong Zhao, Arlindo L Oliveira, and Lei Li. 2024. De-cop: Detecting copyrighted content in language models training data. arXiv preprint arXiv:2402.09910 (2024)

    2024 arXiv

  13. [21]

    Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. arXiv preprint a...

    2023 arXiv

  14. [22]

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al . 2024. Lima: Less is more for alignment. Advances in Neural Information Processing Systems 36 (2024)

    2024

  15. [23]

    [n. d.]. OpenAI reports conversation leakage. https://arstechnica.com/security/ 2024/01/ars-reader-reports-chatgpt-is-sending-him-conversations-from- unrelated-ai-users/

    2024

  16. [24]

    Haoran Yang, Hongyuan Lu, Wai Lam, and Deng Cai. 2024. Exploring Compo- sitional Generalization of Large Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 4...

    2024

  17. [25]

    Keyon Vafa, Ashesh Rambachan, and Sendhil Mullainathan. 2024. Do Large Language Models Perform the way People Expect? Measuring the Human Generalization Function. In International Conference on Machine Learning

    2024

  18. [26]

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao

  19. [27]

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2024. Magicoder: Empowering code generation with oss-instruct. In Forty-first Inter- national Conference on Machine Learning

    2024

  20. [28]

    Zongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu, Shuai Wang, Cuiyun Gao, and Yang Liu. 2024. Split and Merge: Aligning Position Biases in LLM- based Evaluators. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yaser Al-Onaizan, Mo...

    2024

  21. [29]

    arXiv preprint arXiv:2304.03277 (2023)

    Instruction Tuning with GPT-4. arXiv preprint arXiv:2304.03277 (2023)

    2023 arXiv

  22. [30]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics . 311–318

    2002

  23. [31]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

    2023 arXiv

  24. [32]

    Chaozheng Wang, Zongjie Li, Cuiyun Gao, Wenxuan Wang, Ting Peng, Hail- iang Huang, Yuetang Deng, Shuai Wang, and Michael R Lyu. 2024. Exploring Multi-Lingual Bias of Large Code Models in Code Generation. arXiv preprint arXiv:2404.19368 (2024)

    2024 arXiv

  25. [33]

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. 2024. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv preprint arXiv:2405.04434 (2024)

    2024 arXiv

  26. [34]

    [n. d.]. DeepSeek API document. https://platform.deepseek.com/api-docs/

  27. [35]

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiao- qing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023)

    2023 arXiv

  28. [36]

    Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2023. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653 (2023)

    2023 arXiv

  29. [37]

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493 (2022)

    2022 arXiv

  30. [38]

    [n. d.]. gpt35. https://platform.openai.com/docs/models/gpt-3-5

  31. [39]

    Zongjie Li, Chaozheng Wang, Zhibo Liu, Haoxuan Wang, Dong Chen, Shuai Wang, and Cuiyun Gao. 2023. CCTEST: Testing and Repairing Code Completion Systems. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023 . IEEE, ...

    2023

  32. [40]

    Diederik Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Opti- mization. In International Conference on Learning Representations (ICLR) . San Diega, CA, USA

    2015

  33. [41]

    Chris Cummins, Volker Seeker, Dejan Grubisic, Baptiste Roziere, Jonas Gehring, Gabriel Synnaeve, and Hugh Leather. 2024. Meta Large Language Model Compiler: Foundation Models of Compiler Optimization. arXiv preprint arXiv:2407.02524 (2024)

    2024 arXiv

  34. [42]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

    2021 arXiv

  35. [43]

    Ali Al-Kaswan, Maliheh Izadi, and Arie Van Deursen. 2024. Traces of memori- sation in large language models for code. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering . 1–12

    2024

  36. [44]

    Ilya Loshchilov and Frank Hutter. 2017. SGDR: Stochastic Gradient Descent with Warm Restarts. In International Conference on Learning Representations

    2017

  37. [45]

    Yangruibo Ding, Marcus J Min, Gail Kaiser, and Baishakhi Ray. 2024. Cycle: Learning to self-refine the code generation. Proceedings of the ACM on Program- ming Languages 8, OOPSLA1 (2024), 392–418

    2024

  38. [46]

    Sydney Nguyen, Hannah McLean Babe, Yangtian Zi, Arjun Guha, Carolyn Jane Anderson, and Molly Q Feldman. 2024. How Beginning Programmers and Code LLMs (Mis) read Each Other. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1–26

    2024

  39. [47]

    Zhensu Sun, Xiaoning Du, Fu Song, Shangwen Wang, and Li Li. 2024. When Neural Code Completion Models Size up the Situation: Attaining Cheaper and Faster Completion through Dynamic Model Inference. In Proceedings of the IEEE/ACM 46th International Conference on Software Enginee...

    2024

  40. [48]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 (2022)

    2022 arXiv

  41. [49]

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-Instruct: Aligning Language Models with Self-Generated Instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Lingui...

    2023

  42. [50]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168 (2021)

    2021 arXiv

  43. [51]

    Zongjie Li, Chaozheng Wang, Pingchuan Ma, Chaowei Liu, Shuai Wang, Daoyuan Wu, and Cuiyun Gao. 2023. On the feasibility of specialized abil- ity stealing for large language code models. (2023)

    2023

  44. [52]

    [n. d.]. common crawl. https://commoncrawl.org/

  45. [53]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. https://vicuna.lmsys.org

    2023

  46. [54]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1–16

    2020

  47. [55]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Princip...

    2023

  48. [56]

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deep- speed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining ...

    2020

  49. [57]

    Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. WildChat: 1M ChatGPT Interaction Logs in the Wild. InThe Twelfth International Conference on Learning Representations

    2024

  50. [58]

    OpenAI. 2023. OpenAI Temperature. https://platform.openai.com/docs/api- reference/completions/create#completions-create-temperature

    2023

  51. [59]

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. 2024. ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools. arXiv preprint arXiv:2406.12793 (2024)

    2024 arXiv

  52. [60]

    Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. 2024. Jailbreaking leading safety-aligned llms with simple adaptive attacks. arXiv preprint arXiv:2404.02151 (2024)

    2024 arXiv

  53. [61]

    Seong Joon Oh, Bernt Schiele, and Mario Fritz. 2019. Towards reverse- engineering black-box neural networks. In Explainable AI: Interpreting, Ex- plaining and Visualizing Deep Learning . Springer, 121–144

    2019

  54. [62]

    Anselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos, and Yuandong Tian. 2024. Advprompter: Fast adaptive adversarial prompting for llms. arXiv preprint arXiv:2404.16873 (2024)

    2024 arXiv

  55. [63]

    Zongjie Li, Chaozheng Wang, Shuai Wang, and Gao Cuiyun. 2023. Protecting Intellectual Property of Large Language Model-Based Code Generation APIs via Watermarks. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, CCS 2023, Copenhagen, Den...

    2023

  56. [64]

    Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. [n. d.]. Stealing machine learning models via prediction apis. In USENIX Sec’16

  57. [65]

    Binghui Wang and Neil Zhenqiang Gong. 2018. Stealing hyperparameters in machine learning. In 2018 IEEE symposium on security and privacy (SP) . IEEE, 36–52. Conference’17, July 2017, Washington, DC, USA Zongjie Li, Daoyuan Wu, Shuai Wang, and Zhendong Su

    2018

  58. [66]

    Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami

  59. [67]

    Xueluan Gong, Yanjiao Chen, Wenbin Yang, Guanghao Mei, and Qian Wang

  60. [68]

    Honggang Yu, Kaichen Yang, Teng Zhang, Yun-Yun Tsai, Tsung-Yi Ho, and Yier Jin. 2020. CloudLeak: Large-Scale Deep Learning Models Stealing Through Adversarial Examples.. In NDSS

    2020

  61. [69]

    Xinyu Tang, Saeed Mahloujifar, Liwei Song, Virat Shejwalkar, Milad Nasr, Amir Houmansadr, and Prateek Mittal. 2022. Mitigating membership inference attacks by{Self-Distillation} through a novel ensemble architecture. In 31st USENIX Security Symposium (USENIX Security 22) . 1433–1450

    2022

  62. [70]

    Xiaoyong Yuan and Lan Zhang. 2022. Membership inference attacks and de- fenses in neural network pruning. In 31st USENIX Security Symposium (USENIX Security 22). 4561–4578

    2022

  63. [71]

    Depeng Chen, Xiao Liu, Jie Cui, and Hong Zhong. 2023. Poster: Membership Inference Attacks via Contrastive Learning. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security . 3555–3557

    2023

  64. [72]

    Zhihao Zhu, Chenwang Wu, Rui Fan, Defu Lian, and Enhong Chen. 2023. Mem- bership inference attacks against sequential recommender systems. In Proceed- ings of the ACM Web Conference 2023 . 1208–1219

    2023

  65. [73]

    Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership Inference Attacks Against Machine Learning Models. In 2017 IEEE Symposium on Security and Privacy (SP) . 3–18. doi:10.1109/SP.2017.41

    2017 doi

  66. [74]

    Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2024. Detecting Pretraining Data from Large Language Models. In The Twelfth International Conference on Learn- ing Representations

    2024

  67. [75]

    Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Yang, and Hai Li. 2024. Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models.arXiv preprint arXiv:2404.02936 (2024)

    2024 arXiv

  68. [76]

    Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Deepali Aneja, Zeyu Jin, Ramani Duraiswami, Dinesh Manocha, et al. 2024. A Closer Look at the Limitations of Instruction Tuning. arXiv preprint arXiv:2402.05119 (2024)

    2024 arXiv

  69. [77]

    Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. 2024. When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method.arXiv preprint arXiv:2402.17193 (2024)

    2024 arXiv

  70. [78]

    Teodora Baluta, Shiqi Shen, S Hitarth, Shruti Tople, and Prateek Saxena. 2022. Membership inference attacks and generalization: A causal perspective. In Pro- ceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security. 249–262

    2022

  71. [79]

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551 (2017)

    2017 arXiv

  72. [80]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In EMNLP

    2018

  73. [81]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al . 2023. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161 (2023)

    2023 arXiv

  74. [82]

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568 (2023)

    2023 arXiv

  75. [83]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language un- derstanding. arXiv preprint arXiv:2009.03300 (2020)

    2020 arXiv

  76. [84]

    [n. d.]. lm-evaluation-harness. https://github.com/EleutherAI/lm-evaluation- harness

  77. [85]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In Findings of the Association for Computational Linguistics: EMNLP ...

    2020

  78. [86]

    N Reimers. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT- Networks. arXiv preprint arXiv:1908.10084 (2019)

    2019 arXiv

  79. [87]

    Wei-Lin Chiang. 2025. LLM Leaderboard. https://lmarena.ai/

    2025

  80. [88]

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harki- rat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219 (2024)

    2024 arXiv

  81. [89]

    Replicate. 2025. Replicate. https://replicate.com/home

    2025

  82. [90]

    Google. 2025. Gemini. https://gemini.google.com/

    2025

  83. [91]

    Hugging Face. 2025. Hugging Face - The AI community building the future. https://huggingface.co/

    2025

  84. [92]

    Microsoft. 2025. Azure. https://learn.microsoft.com/en-us/azure/ai-services/

    2025

  85. [93]

    helicone. 2025. Top 11 LLM API Providers in 2025. https://www.helicone.ai/ blog/llm-api-providers

    2025

  86. [94]

    X.AI. 2025. Grok API. https://x.ai/api

    2025

  87. [95]

    DeepInfra. 2025. DeepInfra - AI Inference Platform. https://deepinfra.com/docs

    2025

  88. [96]

    Anthropic. 2025. Anthropic - AI research and deployment company. https: //www.anthropic.com/

    2025

  89. [97]

    Anyscale. 2025. Anyscale AI Platform. https://docs.anyscale.com/

    2025

  90. [98]

    Groq. 2025. Groq is Fast AI Inference. https://groq.com/

    2025

  91. [99]

    Fireworks. 2025. Fireworks AI. https://docs.fireworks.ai/getting-started/ introduction

    2025

  92. [100]

    OpenRouter. 2025. OpenRouter. https://openrouter.ai/

    2025

  93. [101]

    all-MiniLM-L6-v2

    Future of Life Institute. 2024. Undefined Behavior Sanitizer. https:// artificialintelligenceact.eu/. Differentiation-Based Extraction of Proprietary Data from Fine-Tuned LLMs Conference’17, July 2017, Washington, DC, USA A Metric Details We employ the Pass@k metric to evaluat...

    2024

  94. [103]

    Novita AI. 2025. Novita AI. https://novita.ai/

    2025

  95. [2016]

    In 2016 IEEE symposium on security and privacy (SP)

    Distillation as a defense to adversarial perturbations against deep neural networks. In 2016 IEEE symposium on security and privacy (SP)

    2016

  96. [2021]

    In IJCAI

    InverseNet: Augmenting Model Extraction Attacks with Training Data Inversion.. In IJCAI

  97. [2023]

    Large language models encode clinical knowledge.Nature (2023), 172–180

    2023

  98. [2024]

    Journal of Machine Learning Research 25, 70 (2024), 1–53

    Scaling instruction-finetuned language models. Journal of Machine Learning Research 25, 70 (2024), 1–53

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.