Pith. sign in

REVIEW 3 major objections 44 references

Given an earlier checkpoint from the same lineage, you can recover which teacher model a student was distilled from.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 17:38 UTC pith:O7IGY3QH

load-bearing objection Reference-based MIA recovers the true teacher near-perfectly under single-teacher SFT when a same-lineage earlier checkpoint exists; the open-world claims are correctly caveated. the 3 major comments →

arxiv 2607.09692 v1 pith:O7IGY3QH submitted 2026-06-19 cs.LG cs.CL

Reference-Based Distillation Detection in LLMs

Rajat Rawat , Sizhe Chen , Akshay Anand , Michael Duan , Bob Rotsted , Sewon Min This is my paper
classification cs.LG cs.CL
keywords distillation detectionreference-based membership inferenceLLM auditingteacher attributiono1 Unicode signalmodel provenance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether one can tell that a language model was trained on outputs from a stronger teacher model. Absolute detection from the student alone is hard, but the problem becomes tractable once an earlier-generation checkpoint from the same family is available as a reference. The method scores each candidate teacher by how much better the student explains that teacher's outputs than the reference does, then ranks teachers and tests whether the margin is statistically real. Across controlled single-teacher fine-tunes and real-world distilled models, the true teacher is recovered with near-perfect accuracy even when the original prompts and pipeline are unknown. A complementary glyph-level test also flags o1/o3-style Unicode habits. Applied to contemporary models, the same procedure surfaces possible links among QwQ, DeepSeek-R1, and GPT-OSS.

Core claim

In a reference-based setting—student S plus an earlier same-lineage checkpoint R—reference-normalized likelihood scoring recovers the true teacher of single-teacher, last-stage SFT distillation with near-perfect accuracy, both in controlled experiments and on real-world distilled models whose training pipelines are unknown.

What carries the argument

Reference-normalized alignment score: for each candidate teacher output y, f = −(ℓ_S(x,y) − ℓ_R(x,y)), which measures how much better the student explains the teacher’s text relative to the reference; teachers are ranked by mean score or by rank-matched per-probe wins, and a calibrated margin test decides whether distillation occurred.

Load-bearing premise

A suitable earlier checkpoint from the same lineage must exist that was never (or only lightly) exposed to the candidate teachers, and the distillation must be single-teacher last-stage SFT; without that reference the signal collapses.

What would settle it

Take a student known to be distilled from teacher T, supply a plausible same-family earlier checkpoint as R, run the reference-normalized ranking over a held-out candidate pool that includes T; if T is not ranked first with statistically significant per-probe win rate on both probe sets, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. The paper proposes reference-based distillation detection for LLMs: given a student model S and an earlier same-lineage checkpoint R, it scores candidate teachers by how much better S explains each teacher’s outputs relative to R (reference-normalized negative log-likelihood). In a hybrid evaluation of controlled single-teacher SFT distillations (19 retained students, four-way classification) and real-world DeepSeek-R1 distills plus s1.1-32B (ten-way), the method recovers the true teacher at near-perfect accuracy, while non-reference baselines fail inconsistently. The authors add per-probe binomial tests for identification, Bonferroni-corrected margin tests for binary detection, a proxy-prompt / few-shot strategy for unknown templates, a glyph-level ASCII–Unicode diagnostic for o1/o3, and open-world case studies on QwQ, DeepSeek-R1, and GPT-OSS.

Significance. If the result holds under the stated regime, this is a useful first step toward model-behavior auditing of distillation without provider logs or watermarks. Strengths include a clean controlled ground-truth setup, real-world recovery on officially distilled R1 models, properly specified statistical tests (exact binomial; margin tests with selection correction and cross-prompt agreement), ablations on reference choice and proxy-prompt domain shift, and public code. The reference-based formulation is shown to be load-bearing: raw likelihood, Min-k, and ZLIB collapse on several teachers while the reference-normalized score does not (Table 1). The work is scoped honestly to single-teacher last-stage SFT and suitable R; that scope is still practically relevant for many published distills and for lineage audits.

major comments (3)
  1. §5 / Tables 6–7: binary detection is substantially weaker than teacher identification. Leave-one-student-out works well when the teacher is represented in calibration, but leave-one-teacher-out accuracy falls to 60.5% (46/76) with false positives on teacher-removed cells for reasoning teachers. The paper correctly notes that signal scale is teacher-dependent, yet the abstract and §3.2 present detection as a co-equal contribution. Either restrict the detection claim to the student-transfer setting with a teacher-matched calibration set, or provide a teacher-robust threshold / open-world null procedure that does not require the true teacher’s regime to be seen at calibration time.
  2. Table 4 (bottom rows) and §4.3: when R1 reasoning traces are hidden and only final answers are scored, identification accuracy collapses to 0.0 Agg. / 0.5 Per-sample. This is a central practical limitation for closed reasoning teachers (and for o1/o3, which only expose summarized traces). The main claims and open-world rankings should state more prominently that reliable attribution currently depends on access to full (or high-fidelity) teacher traces; otherwise the method’s applicability to the models of greatest policy interest is overstated.
  3. §7 and abstract: open-world rankings for QwQ, DeepSeek-R1, and GPT-OSS are presented as “new evidence” of potential distillation, but the manuscript’s own caveats (entangled candidates, multi-stage pipelines, unsuitable R for GPT-OSS with GPT-2 XL, o1 traces only summarized) mean these results are diagnostic rankings, not attribution. Soften the abstract’s “yields new evidence” language to match the disclaimer in §7, and report the same margin / significance tests used in §5 for these open-world cells so readers can judge effect size rather than rank order alone.

Circularity Check

0 steps flagged

No significant circularity; the reference-normalized scoring function and evaluation protocol are independently defined and do not reduce teacher identity to a fitted constant by construction.

full rationale

The paper's central claim is an empirical detection method, not a first-principles derivation. The scoring function is defined as f(Tk(xi), S) = −(ℓS(xi, y(k)i) − ℓR(xi, y(k)i)) in §3.1; this is a standard reference-based membership-inference difference that does not encode any particular teacher identity. Teacher identification is performed by ranking mean or sorted per-sample scores; the true teacher is recovered only when the student was actually trained on that teacher's outputs (controlled distillations constructed by the authors, or official DeepSeek-R1 distill checkpoints). The detection threshold τ is chosen by maximizing F1 exclusively on training folds and never on the held-out evaluation cells (§3.2, §5.1). Statistical tests (exact binomial per-probe win rate against 1/K, Bonferroni-corrected one-sided t/Wilcoxon on the margin) are applied after ranking and do not force the identity of the top teacher. Open-world case studies (§7) are presented with explicit disclaimers that the results are diagnostic rather than conclusive. No equation, uniqueness theorem, or load-bearing self-citation reduces the claimed teacher attribution to an input by construction. The method is therefore self-contained against its own controlled and real-world benchmarks.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central claim rests on standard membership-inference intuitions plus three domain assumptions about distillation pipelines and reference availability; free parameters are the detection threshold and a few experimental choices. No new physical or mathematical entities are invented.

free parameters (3)
  • detection threshold τ = 0.067 (controlled in-sample); fold-dependent otherwise
    Chosen by maximizing F1 on calibration margins (observed training margins tried as cutpoints); fold-specific and not universal. Value 0.067 on full controlled set; varies under leave-one-out.
  • output truncation length = 2048 tokens
    All teacher outputs truncated to 2048 tokens to remove length confounds; chosen by authors, not derived.
  • number of proxy probes N = ~100–200
    Typically ~200 for main tests, 100 for o1 diagnostic; sample size affects power of binomial and t/Wilcoxon tests.
axioms (5)
  • domain assumption A suitable earlier-generation same-lineage reference R exists that is less exposed to candidate teachers than S.
    Stated in §3.1 and Limitations; without it the reference-normalized score loses meaning (GPT-OSS example).
  • domain assumption Distillation is single-teacher, last-stage, SFT-style (at most one teacher from the candidate pool).
    Explicit disclaimer in §4 and Limitations; multi-teacher or multi-stage pipelines are deferred.
  • domain assumption Teacher outputs (especially full reasoning traces) remain informative under proxy prompts and default chat templates.
    §3 and Table 4: hiding R1 traces collapses accuracy to zero; few-shot style alignment is needed for customized instructions.
  • domain assumption Standard membership-inference premise: training examples receive lower loss than non-training examples, and reference calibration strengthens the signal.
    Imported from Carlini et al. / Watson et al.; used to justify f = −(ℓ_S − ℓ_R).
  • standard math Binomial and t/Wilcoxon tests with Bonferroni correction adequately control false positives for teacher attribution and detection.
    §3.3; exact binomial for pre-specified teacher, one-sided t or Wilcoxon on margins vs τ*.

pith-pipeline@v1.1.0-grok45 · 31205 in / 3286 out tokens · 32707 ms · 2026-07-14T17:38:44.211888+00:00 · methodology

0 comments
read the original abstract

Model distillation -- training on outputs from stronger third-party models -- is widely used to boost performance, but raises concerns about unfair advantages and policy violations. This motivates a fundamental question: can we detect whether a model was distilled from another? We show that, while identifying a teacher model from a student in isolation is highly challenging, it becomes tractable in a reference-based setting: given a model and an earlier-generation checkpoint from the same lineage, we can identify the teacher model used to train the later checkpoint. We introduce a distillation detection method based on reference-based membership inference. By comparing how strongly a student model preferentially aligns with outputs from different candidate teachers relative to a reference checkpoint, our method identifies the most likely teacher and detects evidence of distillation. To handle unknown distillation pipelines such as hidden prompts, we infer proxy prompt templates directly from model outputs. We additionally identify a distinctive glyph-level signal specific to o1/o3 models. Evaluating distillation detection is challenging because modern model lineages are already heavily entangled. To address this, we develop a hybrid evaluation spanning both controlled distillation experiments and real-world models. Across both settings, our approach recovers the true teacher with near-perfect accuracy in single-teacher distillation scenarios, even when the underlying distillation pipeline is largely unknown. We further introduce statistical tests for both teacher attribution and distillation detection, and extend our framework to open-world settings where no teacher is guaranteed to be present among the candidates. Applying our method to contemporary models yields new evidence regarding potential distillation relationships involving QwQ, DeepSeek-R1, and GPT-OSS.

Figures

Figures reproduced from arXiv: 2607.09692 by Akshay Anand, Bob Rotsted, Michael Duan, Rajat Rawat, Sewon Min, Sizhe Chen.

Figure 1
Figure 1. Figure 1: Reference-based distillation detection. (a) Rather than attributing distillation from the final [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Reference-based MIA CDFs. Qwen-2.5-3B distilled from GPT-OSS￾120B on s1, probed with OMI [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Reference-based MIA after only 32 training examples (checkpoint 2). A Qwen-2.5-1.5B [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Results on a model distilled from Llama 3.3 70B Instruct, using highly customized [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Results for s1.1-32B, a real-world model trained by [13]. The true teacher model, R1, is ranked at the top by a large margin. True teacher p range vs chance DeepSeek-R1 1.0 × 10−200 – 3.4 × 10−138 [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Per-probe CDFs. CDFs for a within-family control and an o1-distilled Gemma student. The non-distilled gemma-3-27b-pt control has nearly overlapping ASCII and Unicode curves, whereas the o1-distilled gemma-3-4b-pt student separates consistently. Target ∆ASCII 95% CI p Sig. gemma-3-4b-pt (SFT) +0.0732 [+0.064, +0.083] <10−15 ✓ Qwen2.5-1.5B (SFT) +0.0594 [+0.050, +0.069] <10−15 ✓ Qwen2.5-3B (SFT) +0.0573 [+0.… view at source ↗
Figure 7
Figure 7. Figure 7: Our reference-based method applied to QwQ-32B, using QwQ-32B-Preview as a reference. Results may suggest QwQ-32B was distilled from R1 after the release of QwQ-32B-Preview. QwQ-32B. We evaluate QwQ-32B using QwQ-32B-Preview as the reference model. As shown in [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Our reference-based method and ASCII-Unicode diagnostic applied to [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Our reference-based method and ASCII-Unicode diagonistic applied to [PITH_FULL_IMAGE:figures/full_fig_p013_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Results of prompting each candidate teacher model with few-shot examples drawn from the outputs [PITH_FULL_IMAGE:figures/full_fig_p021_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Reference-based MIA results for DeepSeek-R1-Distill-Qwen-1.5B, using Qwen2.5-Math-1.5B as the reference model. 0 20 40 60 80 100 Percentile 0.0 0.2 0.4 0.6 0.8 1.0 Score Prompts: s1 0 20 40 60 80 100 Percentile Prompts: OMI Ranked by mean score 1. DeepSeek R1 2. QwQ 32B Preview 3. GPT-OSS 120B 4. Claude Opus 4.5 * 5. Gemma 3 27B 6. Claude Opus 4.6 * 7. Claude 3.5 Sonnet 8. o1 * 9. Llama 3.3 70B Instruct 1… view at source ↗
Figure 12
Figure 12. Figure 12: Reference-based MIA results for DeepSeek-R1-Distill-Qwen-7B, using Qwen2.5-Math-7B as the reference model. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Reference-based MIA results for DeepSeek-R1-Distill-Llama-8B, using Llama-3.1-8B as the reference model. 0 20 40 60 80 100 Percentile 0.0 0.2 0.4 0.6 0.8 1.0 Score Prompts: s1 0 20 40 60 80 100 Percentile Prompts: OMI Ranked by mean score 1. DeepSeek R1 2. QwQ 32B Preview 3. GPT-OSS 120B 4. Gemma 3 27B 5. Claude Opus 4.5 * 6. Claude 3.5 Sonnet 7. Claude Opus 4.6 * 8. o1 * 9. Llama 3.3 70B Instruct 10. o3 … view at source ↗
Figure 14
Figure 14. Figure 14: Reference-based MIA results for DeepSeek-R1-Distill-Qwen-14B, using Qwen2.5-14B as the reference model. 0 20 40 60 80 100 Percentile 0.0 0.2 0.4 0.6 0.8 1.0 Score Prompts: s1 0 20 40 60 80 100 Percentile Prompts: OMI Ranked by mean score 1. DeepSeek R1 2. QwQ 32B Preview 3. GPT-OSS 120B 4. Gemma 3 27B 5. Claude Opus 4.5 * 6. Claude Opus 4.6 * 7. o1 * 8. Claude 3.5 Sonnet 9. Llama 3.3 70B Instruct 10. o3 *… view at source ↗
Figure 15
Figure 15. Figure 15: Reference-based MIA results for DeepSeek-R1-Distill-Qwen-32B, using Qwen2.5-32B as the reference model. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Reference-based MIA results for DeepSeek-R1-Distill-Llama-70B, using Llama-3.3-70B-Instruct as the reference model. 0 20 40 60 80 100 Percentile 0.0 0.2 0.4 0.6 0.8 1.0 Score Prompts: s1 0 20 40 60 80 100 Percentile Prompts: OMI Ranked by mean score 1. Qwen 3 235B 2. DeepSeek R1 3. Gemma 3 27B 4. QwQ 32B Preview 5. GPT-OSS 120B 6. Claude Opus 4.5 * 7. Claude Opus 4.6 * 8. Llama 3.3 70B Instruct 9. Claude … view at source ↗
Figure 17
Figure 17. Figure 17: Reference-based MIA results on XCoder. Although XCoder was trained on code data, we probe it using math datasets (s1 and OMI) to test whether teacher-specific likelihood signals remain detectable out of domain. E Additional o1 Detection Details Background: ASCII and Unicode. ASCII is a 7-bit encoding covering 128 characters; Unicode assigns codepoints to over 140,000 characters spanning nearly all writing… view at source ↗
Figure 18
Figure 18. Figure 18: ASCII vs. Unicode CDFs for non-o1-distilled Llama controls. For Llama models not expected to be distilled from o1, the ASCII and Unicode reference-normalized loss distributions nearly overlap, indicating little to no encoding-specific separation. The within-family controls—Llama and Gemma checkpoints not expected to be distilled from o1 or o3—produce ASCII gaps concentrated near zero, with most pairs fall… view at source ↗
Figure 19
Figure 19. Figure 19: We apply reference-based MIA to GPT-OSS-120B, using GPT-OSS-20B as the reference model. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: We apply reference-based MIA to GPT-OSS-120B and GPT-OSS-20B, using GPT-2 XL as the reference model and OMI as probing data. 0 20 40 60 80 100 Percentile 0.0 0.2 0.4 0.6 0.8 1.0 Score GPT-OSS-20B 0 20 40 60 80 100 Percentile Unicode ASCII GPT-OSS-120B Ranked by mean score 1. DeepSeek R1 2. Claude 3.5 Sonnet 3. QwQ 32B Preview 4. Gemma 3 27B 5. Claude Opus 4.6 * 6. Llama 3.3 70B Instruct 7. o3 * 8. o1 * 9.… view at source ↗
Figure 21
Figure 21. Figure 21: Results for the GPT-OSS family. We evaluate these models with both reference-based MIA (left) and our ASCII–Unicode diagnostic (right). Target Reference Probe δUni−ASCII [95% CI] Test p Sig. DeepSeek-R1 DeepSeek-MoE-16B-Base OMI +0.937 [+0.746, +1.141] Wilcoxon 8.5×10−16 ✓ GPT-OSS-120B GPT-OSS-20B OMI +0.32 [+0.21, +0.44] Wilcoxon 3.5×10−8 ✓ GPT-OSS-120B GPT-2 XL OMI +0.83 [+0.69, +0.98] Wilcoxon 2.7×10−1… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 18 linked inside Pith

  1. [1]

    Phi-4 technical report.arXiv preprint arXiv:2412.08905, 2024

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report.arXiv preprint arXiv:2412.08905, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    Which models are our models built on? auditing invisible dependencies in modern llms, 2026

    Sanjay Adhikesaven, Haoxiang Sun, and Sewon Min. Which models are our models built on? auditing invisible dependencies in modern llms, 2026. URL https://arxiv.org/abs/2606. 12385

    2026

  3. [3]

    Detecting and preventing distillation attacks, 2026

    Anthropic. Detecting and preventing distillation attacks, 2026. URL https://www. anthropic.com/news/detecting-and-preventing-distillation-attacks

    2026

  4. [4]

    Program synthesis with large language models, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URLhttps://arxiv.org/abs/2108.07732

    Pith/arXiv arXiv 2021

  5. [5]

    Membership inference attacks from first principles

    Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. InIEEE Symposium on Security and Privacy, 2022

    2022

  6. [6]

    Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025

    Gemma Team. Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025. URL https://arxiv.org/abs/2503.19786

    Pith/arXiv arXiv 2025

  7. [7]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    Pith/arXiv arXiv 2015

  8. [8]

    Tinybert: Distilling bert for natural language understanding

    Xiaoqi Jiao, Yichun Yin, Shang Lifeng, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. InFindings of the Association for Computational Linguistics: EMNLP 2020, 2020

    2020

  9. [9]

    Sequence-level knowledge distillation

    Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016

    2016

  10. [10]

    Llm dataset inference: Did you train on my dataset?, 2024

    Pratyush Maini, Hengrui Jia, Nicolas Papernot, and Adam Dziedzic. Llm dataset inference: Did you train on my dataset?, 2024. URLhttps://arxiv.org/abs/2406.06443

    Pith/arXiv arXiv 2024

  11. [11]

    meta-llama/llama-3.2-3b-instruct

    Meta. meta-llama/llama-3.2-3b-instruct. https://huggingface.co/meta-llama/ Llama-3.2-3B-Instruct, . Hugging Face model card, accessed 2026-04-05

    2026

  12. [12]

    Llama 3.3 | model cards and prompt formats

    Meta. Llama 3.3 | model cards and prompt formats. https://www.llama.com/docs/ model-cards-and-prompt-formats/llama3_3/ , . Model card for Llama 3.3 70B Instruct, accessed 2026-04-05

    2026

  13. [13]

    s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393, 2025

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393, 2025. 14

    Pith/arXiv arXiv 2025

  14. [14]

    Google says attackers used 100,000+ prompts to try to clone ai chatbot gemini, 2026

    NBC News. Google says attackers used 100,000+ prompts to try to clone ai chatbot gemini, 2026. URL https://www.nbcnews.com/tech/security/ google-gemini-hit-100000-prompts-cloning-attempt-rcna258657

    2026

  15. [15]

    gpt-oss-120b & gpt-oss-20b model card.arXiv preprint arXiv:2508.10925, 2025

    OpenAI. gpt-oss-120b & gpt-oss-20b model card.arXiv preprint arXiv:2508.10925, 2025. URLhttps://arxiv.org/abs/2508.10925

    Pith/arXiv arXiv 2025

  16. [16]

    Leyi Pan, Aiwei Liu, Shiyu Huang, Yijian Lu, Xuming Hu, Lijie Wen, Irwin King, and Philip S. Yu. Can llm watermarks robustly prevent unauthorized knowledge distillation?, 2025. URL https://arxiv.org/abs/2502.11598

    Pith/arXiv arXiv 2025

  17. [17]

    From prompt to clone: Copyright challenges in ai model distillation.UC Law Science and Technology Journal, 17(1):49, 2026

    Claudia Philipp. From prompt to clone: Copyright challenges in ai model distillation.UC Law Science and Technology Journal, 17(1):49, 2026

    2026

  18. [18]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Qwen Team. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025. URL https: //arxiv.org/abs/2505.09388

    Pith/arXiv arXiv 2025

  19. [19]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2025

    Qwen Team, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao L...

    Pith/arXiv arXiv 2025

  20. [20]

    Watermarking makes language models radioactive, 2024

    Tom Sander, Pierre Fernandez, Alain Durmus, Matthijs Douze, and Teddy Furon. Watermarking makes language models radioactive, 2024. URLhttps://arxiv.org/abs/2402.14904

    Pith/arXiv arXiv 2024

  21. [21]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108, 2019

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108, 2019

    Pith/arXiv arXiv 1910

  22. [22]

    Openai says china’s deepseek trained its ai by distilling us models, memo shows, 2026

    Deepa Seetharaman and Fabiola Arámburo. Openai says china’s deepseek trained its ai by distilling us models, memo shows, 2026. URL https://www.reuters.com/world/china/ openai-accuses-deepseek-distilling-us-models-gain-advantage-bloomberg-news-2026-02-12/

    2026

  23. [23]

    Knowledge distillation detection for open-weights models.arXiv preprint arXiv:2510.02302, 2025

    Qin Shi, Amber Yijia Zheng, Qifan Song, and Raymond A Yeh. Knowledge distillation detection for open-weights models.arXiv preprint arXiv:2510.02302, 2025

    arXiv 2025

  24. [24]

    Membership inference attacks against machine learning models

    Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. InIEEE Symposium on Security and Privacy, 2017

    2017

  25. [25]

    Alpaca: A strong, replicable instruction- following model.Stanford Center for Research on F oundation Models

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction- following model.Stanford Center for Research on F oundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html, 3(6):7, 2023

    2023

  26. [26]

    Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data.arXiv preprint arXiv:2410.01560, 2024

    Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data.arXiv preprint arXiv:2410.01560, 2024

    Pith/arXiv arXiv 2024

  27. [27]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instruc- tions.arXiv preprint arXiv:2212.10560, 2022

    Pith/arXiv arXiv 2022

  28. [28]

    On the importance of difficulty calibration in membership inference attacks, 2022

    Lauren Watson, Chuan Guo, Graham Cormode, and Alex Sablayrolles. On the importance of difficulty calibration in membership inference attacks, 2022. URL https://arxiv.org/abs/ 2111.08440

    Pith/arXiv arXiv 2022

  29. [29]

    Training data provenance verification: Did your model use synthetic data from my generative model for training?, 2025

    Yuechen Xie, Jie Song, Huiqiong Wang, and Mingli Song. Training data provenance verification: Did your model use synthetic data from my generative model for training?, 2025. URL https://arxiv.org/abs/2503.09122. 15

    Pith/arXiv arXiv 2025

  30. [30]

    Detecting distillation data from reasoning models, 2025

    Hengxiang Zhang, Hyeong Kyu Choi, Sharon Li, and Hongxin Wei. Detecting distillation data from reasoning models, 2025. URLhttps://arxiv.org/abs/2510.04850

    Pith/arXiv arXiv 2025

  31. [31]

    Training data attribution: Was your model secretly trained on data created by mine?, 2024

    Likun Zhang, Hao Wu, Lingcui Zhang, Fengyuan Xu, Jin Cao, Fenghua Li, and Ben Niu. Training data attribution: Was your model secretly trained on data created by mine?, 2024. URL https://arxiv.org/abs/2409.15781. 16 A Experimental Details Base student models ( S0)We consider Llama-3.2-3B-Instruct [ 11], Gemma-3-4B-PT [6], and Qwen2.5-1.5B and Qwen2.5-3B [1...

    Pith/arXiv arXiv 2024

  32. [32]

    This was the default setting in our main experiments

    Similar-domain: X and ˆX are not identical but are from similar domains, e.g., ( X= s1, ˆX=OMI), or (X=OMI, ˆX=s1). This was the default setting in our main experiments

  33. [33]

    Different-domain: X and ˆX are from substantially different domains, e.g., X is s1 or OMI (math), while ˆXconsists of prompts for code generation. 10 We follow the same setup as in the main experiments: models are distilled using four base students, four candidate teachers, and two prompt sets (s1 or OMI), and the detection task is to identify the true te...

  34. [34]

    0 20 40 60 80 100 Percentile 0.0 0.2 0.4 0.6 0.8 1.0Score Prompts: s1 0 20 40 60 80 100 Percentile Prompts: OMI Ranked by mean score

    Llama 3.3 70B Instruct * summarized thinking traces instead of original Reference: Qwen2.5-Math-1.5B Figure 11: Reference-based MIA results forDeepSeek-R1-Distill-Qwen-1.5B, usingQwen2.5-Math-1.5Bas the reference model. 0 20 40 60 80 100 Percentile 0.0 0.2 0.4 0.6 0.8 1.0Score Prompts: s1 0 20 40 60 80 100 Percentile Prompts: OMI Ranked by mean score

  35. [36]

    22 0 20 40 60 80 100 Percentile 0.0 0.2 0.4 0.6 0.8 1.0Score Prompts: s1 0 20 40 60 80 100 Percentile Prompts: OMI Ranked by mean score

    o3 * * summarized thinking traces instead of original Reference: Qwen2.5-Math-7B Figure 12: Reference-based MIA results forDeepSeek-R1-Distill-Qwen-7B, usingQwen2.5-Math-7Bas the reference model. 22 0 20 40 60 80 100 Percentile 0.0 0.2 0.4 0.6 0.8 1.0Score Prompts: s1 0 20 40 60 80 100 Percentile Prompts: OMI Ranked by mean score

  36. [38]

    0 20 40 60 80 100 Percentile 0.0 0.2 0.4 0.6 0.8 1.0Score Prompts: s1 0 20 40 60 80 100 Percentile Prompts: OMI Ranked by mean score

    o3 * * summarized thinking traces instead of original Reference: Llama-3.1-8B Figure 13: Reference-based MIA results forDeepSeek-R1-Distill-Llama-8B, usingLlama-3.1-8Bas the reference model. 0 20 40 60 80 100 Percentile 0.0 0.2 0.4 0.6 0.8 1.0Score Prompts: s1 0 20 40 60 80 100 Percentile Prompts: OMI Ranked by mean score

  37. [40]

    0 20 40 60 80 100 Percentile 0.0 0.2 0.4 0.6 0.8 1.0Score Prompts: s1 0 20 40 60 80 100 Percentile Prompts: OMI Ranked by mean score

    o3 * * summarized thinking traces instead of original Reference: Qwen2.5-14B Figure 14: Reference-based MIA results forDeepSeek-R1-Distill-Qwen-14B, usingQwen2.5-14Bas the reference model. 0 20 40 60 80 100 Percentile 0.0 0.2 0.4 0.6 0.8 1.0Score Prompts: s1 0 20 40 60 80 100 Percentile Prompts: OMI Ranked by mean score

  38. [42]

    23 0 20 40 60 80 100 Percentile 0.0 0.2 0.4 0.6 0.8 1.0Score Prompts: s1 0 20 40 60 80 100 Percentile Prompts: OMI Ranked by mean score

    o3 * * summarized thinking traces instead of original Reference: Qwen2.5-32B Figure 15: Reference-based MIA results forDeepSeek-R1-Distill-Qwen-32B, usingQwen2.5-32Bas the reference model. 23 0 20 40 60 80 100 Percentile 0.0 0.2 0.4 0.6 0.8 1.0Score Prompts: s1 0 20 40 60 80 100 Percentile Prompts: OMI Ranked by mean score

  39. [43]

    0 20 40 60 80 100 Percentile 0.0 0.2 0.4 0.6 0.8 1.0Score Prompts: s1 0 20 40 60 80 100 Percentile Prompts: OMI Ranked by mean score

    Llama 3.3 70B Instruct * summarized thinking traces instead of original Reference: Llama-3.3-70B-Instruct Figure 16: Reference-based MIA results forDeepSeek-R1-Distill-Llama-70B, usingLlama-3.3-70B-Instruct as the reference model. 0 20 40 60 80 100 Percentile 0.0 0.2 0.4 0.6 0.8 1.0Score Prompts: s1 0 20 40 60 80 100 Percentile Prompts: OMI Ranked by mean score

  40. [45]

    Llama CoT

    o3 * * summarized thinking traces instead of original Reference: Qwen3-8B-Base Figure 17: Reference-based MIA results onXCoder. Although XCoder was trained on code data, we probe it using math datasets (s1 and OMI) to test whether teacher-specific likelihood signals remain detectable out of domain. E Additional o1 Detection Details Background: ASCII and U...

  41. [47]

    26 0 20 40 60 80 100 Percentile 0.0 0.2 0.4 0.6 0.8 1.0Score GPT-OSS-20B 0 20 40 60 80 100 Percentile GPT-OSS-120B Ranked by mean score

    Gemma 3 27B * summarized thinking traces instead of original Reference: GPT-OSS-20B Figure 19: We apply reference-based MIA to GPT-OSS-120B, using GPT-OSS-20B as the reference model. 26 0 20 40 60 80 100 Percentile 0.0 0.2 0.4 0.6 0.8 1.0Score GPT-OSS-20B 0 20 40 60 80 100 Percentile GPT-OSS-120B Ranked by mean score

  42. [49]

    0 20 40 60 80 100 Percentile 0.0 0.2 0.4 0.6 0.8 1.0Score GPT-OSS-20B 0 20 40 60 80 100 Percentile Unicode ASCII GPT-OSS-120B Ranked by mean score

    o1 * * summarized thinking traces instead of original Reference: GPT-2 XL Figure 20: We apply reference-based MIA to GPT-OSS-120B and GPT-OSS-20B, using GPT-2 XL as the reference model and OMI as probing data. 0 20 40 60 80 100 Percentile 0.0 0.2 0.4 0.6 0.8 1.0Score GPT-OSS-20B 0 20 40 60 80 100 Percentile Unicode ASCII GPT-OSS-120B Ranked by mean score

  43. [50]

    Llama 3.3 70B Instruct

  44. [51]

    Target Reference Probeδ Uni−ASCII [95% CI] TestpSig

    Claude Opus 4.5 * * summarized thinking traces instead of original Reference: GPT-2 XL Figure 21:Results for the GPT-OSS family.We evaluate these models with both reference-based MIA (left) and our ASCII–Unicode diagnostic (right). Target Reference Probeδ Uni−ASCII [95% CI] TestpSig. DeepSeek-R1 DeepSeek-MoE-16B-BaseOMI+0.937[+0.746,+1.141]Wilcoxon8.5×10 ...