Pith. sign in

REVIEW 4 major objections 4 minor 23 references

Clone What You Can't Steal: Black-Box LLM Replication via Logit Leakage and Distillation

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a black-box LLM whose API returns top-k logits can be functionally cloned with fewer than 10,000 queries, by SVD-recovering the output projection matrix and distilling the rest into a smaller student model.

desk verdict Reasonable steal-and-distill pipeline, but the core top-k SVD step is unsubstantiated: the paper assumes full logits while claiming top-k access, so the headline 97.6% clone claim cannot be trusted as written. read the letter →

arxiv 2509.00973 v1 pith:A2JBS44P submitted 2025-08-31 cs.CR cs.AI

Kanchon Gharami , Hansaka Aluvihare , Shafika Showkat Moni , Berker Peköz This is my paper
classification cs.CRcs.AI
keywords adversarialmachinelearninglargelanguagemodelsmodelextractionlogitleakageknowledgedistillationsingularvaluedecompositionblack-boxattackcompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to show that a black-box LLM is cloneable whenever its API leaks top-k logits, even under tight query and compute budgets. The proposed two-stage pipeline first recovers the model's final projection layer by running SVD on fewer than 10,000 collected logit vectors, then freezes that layer and distills the remaining behavior into a smaller transformer trained on public text. The headline results are a 6-layer student that recreates 97.6% of the teacher's hidden-state geometry with a 7.31% perplexity increase, and a 4-layer variant that is 17.1% faster and 18.1% smaller with modest fidelity loss. If correct, the work turns a seemingly minor API output, top-k logits, into a complete cloning vector.

What carries the argument

The load-bearing object is the logit matrix Q formed by stacking the API's top-k logit responses. Because the teacher's output projection W is low-rank, with hidden dimension d much smaller than vocabulary size V, the top d left singular vectors of Q span the same column space as W, so the estimate W_hat = U_{:,1:d} Sigma_{1:d,1:d} approximates W up to an unknown invertible matrix G; the spectral drop after d also estimates the hidden dimension. The second mechanism is temperature-scaled distillation, L(p) = tau^2 KL(softmax(z_T/tau) || softmax(z_S/tau)) + lambda CE(z_S, y), applied while the recovered projection and embedding layers stay frozen, which transfers the teacher's token-level beh

What would settle it

Take the same 6-layer teacher, return exactly k = d top logits per query or round all returned logits to one decimal place, run the described SVD extraction, and compare the column space of the recovered projection with the true one; if the two are essentially uncorrelated, the pipeline's first stage fails and the clone claim collapses.

Watch

Extended reading notes

Core claim

The paper claims that a black-box large language model can be functionally replicated through its API using only the top-k logits it returns. In the first stage, fewer than 10,000 queries are stacked into a logit matrix; a singular value decomposition of that matrix yields an estimate of the output projection matrix, correct up to an unknown invertible linear map, and the sharp drop in the singular-value spectrum reveals the hidden dimension. In the second stage, the recovered projection and embedding are frozen and a compact transformer student is trained to match the teacher's softened output distribution on public text. Reported results on distilGPT-2, a 6-layer model with roughly 81M par

Load-bearing premise

Everything rests on the unstated assumption that the logits the API does not return can be ignored or filled in without destroying the underlying structure that the SVD is supposed to recover.

Editorial extensions

If this is right

  • A deployment that exposes unrounded top-k logits with k at least one more than the hidden dimension should be treated as having already lost its final layer.
  • A 4-layer student clone can cut inference cost by 17.1% and parameter count by 18.1% while remaining within about 10.5% perplexity of the teacher, so the same recipe doubles as a compression method.
  • Because clones evaluated on WikiText-103 stay within a 10.5% perplexity gap, the attack captures transferable behavior, not just memorized prompts.
  • The full attack uses fewer than 10k queries and under 24 GPU-hours, putting high-fidelity cloning within reach of a cost-limited adversary and inside typical API rate limits.
  • The recovered projection is only determined up to an unknown invertible transform, so any downstream student must be invariant to that transform; the cosine-similarity metric used here is exactly such an invariant.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The SVD stage could be isolated as a cheap reconnaissance tool: even without distillation, the singular-value spectrum of the collected logit matrix reveals the teacher's hidden dimension, which an attacker could use to plan a larger steal.
  • If the pipeline is robust to logit noise and quantization, it extends to real commercial APIs; a simple test would be to add small Gaussian noise to the returned logits and watch whether the recovered column space degrades gracefully or snaps.
  • The 4- to 8-layer sweep suggests the method doubles as architecture discovery: student depth can be chosen by fitting a curve of perplexity versus parameter count, effectively probing how much of the teacher's capacity is redundant.
  • A cheap defense is implied by the k >= d+1 assumption: truncating top-k logits to fewer than d+1 entries, or applying a randomized linear perturbation to logits before returning them, may break projection recovery while leaving sampling behavior intact.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a two-stage black-box LLM replication pipeline. Stage 1 (Sec. IV-A) claims to recover the output projection matrix W of a teacher transformer by collecting top-k logits from fewer than 10,000 API queries, stacking them into a full logit matrix Q, and applying SVD; the top-d left singular vectors scaled by singular values yield W_hat. Stage 2 (Sec. IV-B) then distills the teacher's behavior into student transformers of 4-8 layers, with the recovered projection layer frozen, using a KL+CE distillation loss on WikiText-2. The headline results are that a 6-layer student achieves 97.6% cosine similarity with the teacher's hidden-state geometry, a 7.31% perplexity increase, and 7.58 NLL, while a 4-layer variant gives 17.1% speedup and 18.1% parameter reduction. The authors evaluate on distilGPT-2 as teacher and report in-distribution (WikiText-2) and out-of-distribution (WikiText-103) metrics.

Significance. If the central claim were valid, the paper would demonstrate a practically important attack: a cost-limited adversary with only top-k logit access could produce a deployable functional clone of a black-box LLM under realistic query budgets. The paper has a clear threat model, uses public data, and compares multiple student depths, which are useful features. However, the load-bearing SVD step is not justified under the stated top-k threat model, the embedding-recovery assumption is unstated, and the empirical evaluation lacks the error bars and ablations needed to attribute the reported fidelity to the proposed method. As presented, the evidence does not support the headline claims, despite the topical significance.

major comments (4)
  1. [Sec. IV-A, Eqs. (2)-(3), Algorithm 1 line 4] The method defines Q ∈ R^{V×n} as a matrix of full logit vectors and applies SVD to recover W's column space. But the threat model (Sec. III-B.3) grants only top-k logits with k ≥ d+1, so each column has V-k unobserved entries. The paper never states how missing entries are filled, masked, or completed before SVD. Filling them with a constant contaminates the recovered column space with the missingness pattern; leaving them missing makes Eq. (2) undefined. The sentence 'Although partial, these responses contain enough information pattern to recover W' is an assertion, not a derivation. Since W_hat is frozen in Stage 2, this invalidates the 'under 10k queries' claim and the entire pipeline.
  2. [Sec. IV-B] The text states 'The clone keeps the stolen embedding and projection layers fixed,' but no embedding-recovery procedure is described anywhere; only W is reconstructed. If the student is initialized with the public distilGPT-2 embedding/vocabulary, this must be stated as an explicit assumption, as it is not part of the black-box capabilities in Sec. III-B.2. Without this clarification, the attack does not actually clone the full model and the 'deployable clone' claim is unsupported.
  3. [Sec. V, Tables I-III] No standard errors, confidence intervals, or multiple-seed results are reported, and no direct reconstruction error for W_hat (e.g., subspace distance to the true W) is given. There is also no ablation separating the contribution of the SVD stage from ordinary distillation with a randomly initialized projection. As a result, the reported 97.6% cosine similarity and perplexity gaps cannot be attributed to the proposed projection recovery, and the necessity of the recovered W is not established.
  4. [Sec. V-A, Table I] The metric 'Cos-sim' is not defined. Which internal representations are compared, at which layer(s), and how the student's hidden states are aligned with the teacher's under black-box access? Without a precise definition, the central 'hidden-state geometry' claim is not checkable or falsifiable.
minor comments (4)
  1. [Throughout] Grammar and wording issues: 'combining combine', 'enough information pattern to recover W', 'ground of our threat model'. A careful proofread is needed.
  2. [Algorithm 2] The temperature schedule 'Update temperature τ based on epoch' is unspecified; Eqs. (4)-(5) treat τ as a fixed hyperparameter. Please clarify how τ changes and report the values used.
  3. [Figures and Tables] Several figures have rendering artifacts such as '/uni000...' sequences and missing axis labels (Figs. 4-8). Please replace with clean, readable plots.
  4. [References [20], [21]] WikiText-2 and WikiText-103 are cited via Kaggle URLs. Please cite the original Merity et al. dataset papers and report the exact train/validation splits, tokenizer, and the number of queries n and top-k value used in the experiments.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity. The derivation is self-contained; the main weakness is an unproven partial-logit recovery assumption, not a circular argument.

full rationale

The paper's derivation chain is not circular. Stage 1 (Sec. IV-A, Eqs. 2-3) recovers W_hat by SVD of the logit matrix Q = W·g(p); this is a standard linear-algebra column-space recovery argument with stated assumptions (full-rank hidden states, unrounded logits), and W_hat is not defined in terms of the student model. Stage 2 (Sec. IV-B, Eqs. 4-5) trains the student to minimize KL(s_T || s_S) + λ·CE; the reported fidelity metrics (PPL gap, KL, cosine similarity) are measured after training, not fitted constants, so the results are empirical outcomes of distillation rather than tautologies. There are no load-bearing self-citations: reference [6] is Carlini et al., not the present authors, and is used only as inspiration for the SVD approach. The paper does contain an unsupported assertion in Sec. IV-A: 'Although partial, these responses contain enough information pattern to recover W.' Under the threat model (Sec. III-B.3), the API returns only top-k logits with k ≥ d+1, so the V×n matrix Q in Eq. (2) cannot be fully populated; the paper never specifies how missing logits are handled before SVD. This is a significant correctness/rigor gap, but it is not circularity—no equation reduces to its own input, no fitted parameter is relabeled as a prediction, and no self-citation chain forces the conclusion.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The pipeline rests on one recovered matrix and a standard distillation loss; no new particles, forces, or model components are introduced. The main unstated costs are the incomplete-logit assumption, the hidden-state comparison protocol, and the free hyperparameters of the distillation schedule.

free parameters (6)
  • hidden dimension d_est = nominally 768 (distilGPT-2)
    Estimated in Algorithm 1 as argmax of the log singular value gap; controls the rank of W_hat and is never validated against the true d.
  • top-k count k = k >= d+1, exact k not reported
    Threat model assumption chosen so the API exposes at least d+1 logits; central to filling Q but no sensitivity analysis is given.
  • query count n = under 10k, exact value not reported
    Budget for the SVD stage; the paper never states n or shows reconstruction error as a function of n.
  • distillation temperature tau = scheduled by epoch, formula and values omitted
    Algorithm 2 updates tau each epoch but does not specify the schedule; tau changes the target distribution and affects all fidelity numbers.
  • loss weight lambda = 0.1
    Eq. (5); chosen by hand so the KL term dominates, with no ablation.
  • student depth = 4 to 8 layers; headline results use 6 and 4
    Architecture hyperparameter scanned post hoc; the 6-layer and 4-layer results are selected for the headline without significance testing.
assumptions (5)
  • domain assumption API returns at least k >= d+1 top-k logits, unrounded and unperturbed
    Stated in Sec. III-B.3; justifies that W's column space is observable, but not demonstrated against a real API.
  • ad hoc to paper Each prompt's full logit vector can be treated as observed even though only top-k entries are returned
    Eq. (2) defines Q in R^{V x n} and applies dense SVD; the paper never explains how the V-k unreturned logits are handled.
  • standard math The n sampled prompts produce hidden states whose span is full rank d
    Required for the column space of Q = W [g(p_1), ..., g(p_n)] to equal the column space of W; plausible but unverified.
  • domain assumption The student can use the same token embedding as the teacher, recovered as the transpose of the output projection
    Sec. IV-B says the clone keeps the stolen embedding and projection layers fixed, but does not state how the embedding is obtained in a black-box setting; assumes tied embeddings and shared tokenizer.
  • domain assumption Cosine similarity of hidden states is a meaningful fidelity measure across different depths
    Table I reports 'Cos-sim' without specifying which layers are compared or how the depth mismatch (4-8 layers vs 6 layers) is aligned.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Clone What You Can't Steal: Black-Box LLM Replication via Logit Leakage and Distillation." pith.science (2026). https://pith.science/paper/A2JBS44P

@misc{pith2026250900973,
  author       = {Pith},
  title        = {Pith review of: Clone What You Can't Steal: Black-Box LLM Replication via Logit Leakage and Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A2JBS44P}},
  note         = {Machine review of arXiv:2509.00973}
}
read the original abstract

Large Language Models (LLMs) are increasingly deployed in mission-critical systems, facilitating tasks such as satellite operations, command-and-control, military decision support, and cyber defense. Many of these systems are accessed through application programming interfaces (APIs). When such APIs lack robust access controls, they can expose full or top-k logits, creating a significant and often overlooked attack surface. Prior art has mainly focused on reconstructing the output projection layer or distilling surface-level behaviors. However, regenerating a black-box model under tight query constraints remains underexplored. We address that gap by introducing a constrained replication pipeline that transforms partial logit leakage into a functional deployable substitute model clone. Our two-stage approach (i) reconstructs the output projection matrix by collecting top-k logits from under 10k black-box queries via singular value decomposition (SVD) over the logits, then (ii) distills the remaining architecture into compact student models with varying transformer depths, trained on an open source dataset. A 6-layer student recreates 97.6% of the 6-layer teacher model's hidden-state geometry, with only a 7.31% perplexity increase, and a 7.58 Negative Log-Likelihood (NLL). A 4-layer variant achieves 17.1% faster inference and 18.1% parameter reduction with comparable performance. The entire attack completes in under 24 graphics processing unit (GPU) hours and avoids triggering API rate-limit defenses. These results demonstrate how quickly a cost-limited adversary can clone an LLM, underscoring the urgent need for hardened inference APIs and secure on-premise defense deployments.

Figures

Figures reproduced from arXiv: 2509.00973 by the authors.

Figure 1
Figure 1. High–level data flow inside a transformer LLM. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Pipeline of the model stealing attack. This method assumes access to full or top-k logits, which has been shown to be feasible in commercial black-box APIs through prompt manipulation [6]. While quantized or truncated outputs may reduce reconstruction fidelity, the singular value spectrum typically exhibits a sharp drop after the true hidden dimension, enabling reliable estimation of model depth. This recovered proj… view at source ↗
Figure 4
Figure 4. Token-probability alignment between each clone and the target. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (3 more)
Figure 6
Figure 6. Figure 6: Relative AIC and AICc scores of cloned models. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 5
Figure 5. Figure 5: Unique 20-grams as a function of parameter count. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 8
Figure 8. Figure 8: Perplexity increase per million parameters saved compared to the [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 16 canonical work pages

  1. [1]

    Fine-tuning and evaluating open-source large lan- guage models for the army domain,

    D. C. Ruiz and J. Sell, “Fine-tuning and evaluating open-source large lan- guage models for the army domain,” arXiv preprint arXiv:2410.20297 , 2024

    arXiv 2024

  2. [2]

    Leveraging large language models for integrated satellite-aerial-terrestrial networks: recent advances and future directions,

    S. Javaid, R. A. Khalil, N. Saeed, B. He, and M.-S. Alouini, “Leveraging large language models for integrated satellite-aerial-terrestrial networks: recent advances and future directions,” IEEE Open Journal of the Communications Society, 2024

    work page 2024

  3. [3]

    Fine-tuned language models as space systems con- trollers,

    E. M. Zucchelli, D. Wu, J. Briden, C. Hofmann, V . Rodriguez-Fernandez, and R. Linares, “Fine-tuned language models as space systems con- trollers,” arXiv preprint arXiv:2501.16588 , 2025

    arXiv 2025

  4. [4]

    Milchat: Introducing chain of thought reasoning and grpo to a multimodal small language model for remote sensing,

    A. Koksal and A. A. Alatan, “Milchat: Introducing chain of thought reasoning and grpo to a multimodal small language model for remote sensing,” arXiv preprint arXiv:2505.07984 , 2025

    arXiv 2025

  5. [5]

    Pllm- cs: Pre-trained large language model (llm) for cyber threat detection in satellite networks,

    M. Hassanin, M. Keshk, S. Salim, M. Alsubaie, and D. Sharma, “Pllm- cs: Pre-trained large language model (llm) for cyber threat detection in satellite networks,” Ad Hoc Networks , vol. 166, p. 103645, 2025

    work page 2025

  6. [6]

    Stealing part of a production language model,

    N. Carlini, D. Paleka, K. Dvijotham, T. Steinke, J. Hayase, A. F. Cooper, K. Lee, M. Jagielski, M. Nasr, A. Conmy et al. , “Stealing part of a production language model,” in Proceedings of the 41st International Conference on Machine Learning , 2024, pp. 5680–5705

    work page 2024

  7. [7]

    Teach llms to phish: Stealing private information from language mod- els,

    A. Panda, C. A. Choquette-Choo, Z. Zhang, Y . Yang, and P. Mittal, “Teach llms to phish: Stealing private information from language mod- els,” in The Twelfth International Conference on Learning Representa- tions, 2024

    work page 2024

  8. [8]

    Model stealing for any low-rank language model,

    A. Liu and A. Moitra, “Model stealing for any low-rank language model,” arXiv preprint arXiv:2411.07536 , 2024

    arXiv 2024

Show all 23 references
  1. [9]

    I know what you trained last summer: A survey on stealing machine learning models and defences,

    D. Oliynyk, R. Mayer, and A. Rauber, “I know what you trained last summer: A survey on stealing machine learning models and defences,” ACM Computing Surveys , vol. 55, no. 14s, pp. 1–41, 2023

    2023

  2. [10]

    Privacy backdoors: stealing data with corrupted pretrained models,

    S. Feng and F. Tram `er, “Privacy backdoors: stealing data with corrupted pretrained models,” in Proceedings of the 41st International Conference on Machine Learning , 2024, pp. 13 326–13 364

    2024

  3. [11]

    Can’t steal? cont-steal! contrastive stealing attacks against image encoders,

    Z. Sha, X. He, N. Yu, M. Backes, and Y . Zhang, “Can’t steal? cont-steal! contrastive stealing attacks against image encoders,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 16 373–16 383

    2023

  4. [12]

    Large language models for link stealing attacks against graph neural networks,

    F. Guan, T. Zhu, H. Sun, W. Zhou, and P. S. Yu, “Large language models for link stealing attacks against graph neural networks,” IEEE Transactions on Big Data , 2024

    2024

  5. [13]

    Forensic analysis of indirect prompt injection attacks on llm agents,

    M. Chernyshev, Z. Baig, and R. Doss, “Forensic analysis of indirect prompt injection attacks on llm agents,” in 2024 IEEE 6th International Conference on Trust, Privacy and Security in Intelligent Systems, and Applications (TPS-ISA), 2024, pp. 409–411

    2024

  6. [14]

    Llm-sentry: A model-agnostic human-in-the-loop framework for securing large language models,

    S. Irtiza, K. A. Akbar, A. Yasmeen, L. Khan, O. Daescu, and B. Thurais- ingham, “Llm-sentry: A model-agnostic human-in-the-loop framework for securing large language models,” in 2024 IEEE 6th International Conference on Trust, Privacy and Security in Intelligent Systems, and A...

    2024

  7. [15]

    Data stealing attacks against large language models via backdooring,

    J. He, G. Hou, X. Jia, Y . Chen, W. Liao, Y . Zhou, and R. Zhou, “Data stealing attacks against large language models via backdooring,” Electronics, vol. 13, no. 14, p. 2858, 2024

    2024

  8. [16]

    Efficient llm jailbreak via adaptive dense-to-sparse constrained optimization,

    K. Hu, W. Yu, Y . Li, T. Yao, X. Li, W. Liu, L. Yu, Z. Shen, K. Chen, and M. Fredrikson, “Efficient llm jailbreak via adaptive dense-to-sparse constrained optimization,” Advances in Neural Information Processing Systems, vol. 37, pp. 23 224–23 245, 2024

    2024

  9. [17]

    Multi-turn context jailbreak attack on large language models from first principles,

    X. Sun, D. Zhang, D. Yang, Q. Zou, and H. Li, “Multi-turn context jailbreak attack on large language models from first principles,” arXiv preprint arXiv:2408.04686, 2024

    2024 arXiv

  10. [18]

    Harmbench: A standardized evaluation framework for automated red teaming and robust refusal,

    M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li et al. , “Harmbench: A standardized evaluation framework for automated red teaming and robust refusal,” in ICML, 2024

    2024

  11. [19]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531 , 2015

    2015 arXiv

  12. [20]

    Wikitext-2 data,

    V . Mettu, “Wikitext-2 data,” https://www.kaggle.com/datasets/ vivekmettu/wikitext2-data, 2022, accessed: 2025-05-31

    2022

  13. [21]

    Wikitext103,

    L. D, “Wikitext103,” https://www.kaggle.com/datasets/dekomposition/ wikitext103, 2022, accessed: 2025-05-31

    2022

  14. [22]

    Scalable extraction of training data from (production) language models,

    M. Nasr, N. Carlini, J. Hayase, M. Jagielski, A. F. Cooper, D. Ippolito, C. A. Choquette-Choo, E. Wallace, F. Tram `er, and K. Lee, “Scalable extraction of training data from (production) language models,” arXiv preprint arXiv:2311.17035, 2023

    2023 arXiv

  15. [23]

    Quantile-based cumulative kullback-leibler divergence in past lifetime: Some properties and applications,

    S. Sunoj and P. Saranya, “Quantile-based cumulative kullback-leibler divergence in past lifetime: Some properties and applications,” Commu- nications in Statistics-Theory and Methods , vol. 54, no. 6, pp. 1880– 1894, 2025

    2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.