Pith. sign in

REVIEW 4 major objections 6 minor 72 references

Depth Gives a False Sense of Privacy: LLM Internal States Inversion

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM internal states, long treated as irreversible, can be inverted to recover near-verbatim user prompts — including a 4,112-token medical consultation recovered with 86.88 F1 token matching.

desk verdict Solid evidence that LLM internal states leak prompt content, with genuinely new optimization tricks, but the 'nearly perfect' headline is a cherry-picked operating point and the attack's practical robustness to noise is not established. read the letter →

arxiv 2507.16372 v1 pith:LP5EZC6K submitted 2025-07-22 cs.CR cs.AI

Tian Dong , Yan Meng , Shaofeng Li , Guoxing Chen , Zhen Liu , Haojin Zhu This is my paper
classification cs.CRcs.AI
keywords internalstateinversionpromptleakagecollaborativeinferencelargelanguagemodelsoptimization-basedattackgeneration-baseddifferentialprivacydefenseTokenBasisSelection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models pass hidden activation vectors (internal states) between layers, and two increasingly common practices—collaborative inference, which runs early layers on the user's device, and model safety auditing, which inspects mid-layer activations—deliver those vectors to parties the user has not authorized to read the prompt. The paper tries to establish that these internal states are not the 'irreversible' representation they are usually taken to be. It proposes four inversion attacks, two white-box optimization attacks for shallow and deep layers and two black-box attacks for when model weights are unknown, and reports near-verbatim recovery of prompts up to 4,112 tokens, including a medical consultation recovered with 86.88 F1 token matching from the middle layer of a Llama-3 model. If these results hold, internal-state exposure should be treated as prompt exposure, and defenses that merely perturb the states are unlikely to restore privacy without breaking the model.

What carries the argument

The load-bearing mechanism is a two-phase inversion pipeline: first reconstruct the input embedding whose forward pass would produce the observed internal states, then decode tokens by choosing, for each position, the dictionary token whose embedding is most cosine-similar to the recovered embedding. For deep layers, Token Basis Selection (TBS) replaces direct embedding optimization with optimization over coefficients of an orthogonal basis (typically derived from an SVD of the input embedding matrix), which shrinks the search space and bounds gradient magnitude so the loss does not explode. A distribution-matching penalty keeps recovered embeddings in-distribution, and a generation-based variant treats inversion as translation by projecting internal states into an encoder-decoder model's embedding space. These pieces together are what let the attack scale from BERT-style models with around a dozen layers to modern LLMs with 32 or 80 layers.

What would settle it

Measure the TBS attack on hidden states that have been rounded to an 8-bit fixed-point representation before reaching the adversary; if token-level F1 falls from the reported 86.88 to near chance, the near-perfect inversion result depends on pristine, high-precision internal states rather than on an intrinsic property of the states themselves.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that an adversary who observes the internal states at a chosen layer can reconstruct the original prompt almost exactly, in order and with preserved semantics, rather than merely inferring attributes about it. The enabling insight is that strong inversion is possible if the attack first recovers a plausible input embedding and only then selects tokens: Embedding Recovery does this directly for shallow layers, while Token Basis Selection expresses the candidate embedding as a learned combination of an orthogonal basis of the input embedding matrix, which stabilizes gradients and lets deep layers be inverted too. The same strategy transfers to black-box settings when the victim model is derived from a known base model, and a separate generation-based inversion model, trained on surrogate data with a projection module aligning internal states to an encoder, handles closed-sourced models. The paper reports this across six models, short and long prompts, medical and coding domains, and shows that the broad privacy conclusion holds up to the middle layer of a 70B model.

Load-bearing premise

The near-perfect inversion numbers rest on the adversary observing the internal states exactly as computed by the deployed model, with no added noise, rounding, or defensive perturbation; the paper's own defense experiments show that even mild differential privacy noise severely degrades the optimization attack.

Editorial extensions

If this is right

  • Collaborative inference cannot promise data locality as privacy: any party holding the rest of the layers can, in principle, reconstruct the user's prompt rather than only consuming it for inference.
  • Safety auditors that legitimately inspect internal states become high-value targets; an honest auditor's access is indistinguishable from an inversion adversary's access.
  • Inversion quality persists across model scales—from 7B to 70B—so choosing a larger model does not by itself restore confidentiality of prompts.
  • Domain-specialized models, such as a coding-tuned model, leak more than general-purpose counterparts, so fine-tuning on sensitive domains raises rather than lowers inversion risk.
  • No tested defense—quantization, dropout, noisy embeddings, or differential privacy—blocks inversion while keeping model utility intact; the paper therefore frames the problem as needing architectural changes or confidential computing rather than output perturbation alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if real serving stacks quantize or round activations, the headline TBS numbers may overstate real-world risk; testing the attack on fixed-point hidden states is a direct next experiment.
  • Beyond the paper: the paper's architectural hypothesis—that uniform layer width prevents information loss and thereby aids inversion—implies a cheap falsifiable test: train a small transformer with decreasing hidden width and measure whether the same attack loses fidelity.
  • Beyond the paper: the generation-based attack's success hinges on surrogate data resembling the victim's domain; a defender could therefore focus on domain separation or on making internal states task-specific rather than on adding noise.
  • Beyond the paper: the two-phase embedding-recovery recipe is not obviously limited to internal states and may transfer to other shared artifacts such as KV caches or per-layer gradients, where hidden values are also exposed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies whether the internal states (ISs) of modern LLMs can be inverted to recover the original user prompt. It proposes two white-box optimization attacks (Embedding Recovery for shallow layers and Token Basis Selection for deep layers), a black-box extension that first identifies and replicates a derived model, and a generation-based attack that trains an encoder-decoder with a projection module to translate ISs back into text. The evaluation covers six LLMs on short and long medical and coding prompts, claims up to 86.88 F1 / 95.19 semantic similarity for a 4,112-token medical prompt from the middle layer of Llama-3-8B-Instruct, and tests quantization, dropout, noisy input embeddings, and DP as defenses. The authors conclude that input inversion from ISs is practical and that existing defenses do not fully mitigate it.

Significance. The privacy question addressed is timely and the attack surface (ISs exposed by collaborative inference and auditing) is well chosen. The paper is among the first to scale inversion to long prompts and large models (up to 70B), and it provides an extensive comparison across six models and multiple baselines. The commitment to release code, pretrained inversion models, and logs is valuable for reproducibility. If the evaluation were more conservative—notably with a validation protocol and without test-set hyperparameter selection—the results would still support a meaningful claim: middle-layer ISs leak substantial token-level and semantic content under white-box access, and generation-based inversion transfers across similar distributions. As written, the headline 'nearly perfect' claim overstates what the evidence shows.

major comments (4)
  1. [Section 5.2.2 / Figure 7 / Figure 9 / Table 4] The long-context evaluation uses only the 25 longest prompts from each dataset, and the reported operating points are selected by sweeping learning rate, distance metric, and basis on these same test prompts. Table 4 shows that the same TBS attack on Mentalhealth ranges from 78.53 F1 (unbiased basis, µ=5e-4, λ=0) to 94.99 F1 (SVD basis, µ=5e-4, λ=0), so the headline numbers are an optimized envelope rather than a representative or predictive operating point. A separate validation split or a pre-registered configuration is needed before the abstract's general claim can be accepted.
  2. [Abstract / Figures 21–23 / Section 5.2.2] The abstract's 'nearly perfect' claim for the 4,112-token prompt is not traceable to the reported averages: Section 5.2.2 gives Aimedical averages of 98.12 CS and 96.7 F1, while the abstract quotes 86.88 F1 / 95.19 CS. If the latter is a single-example result, the paper should say so explicitly. In addition, the displayed inversion in Figures 21–23 contains numerous inserted non-English and special tokens (e.g., 'тися', 'ıldığı', '神马收录', '<|begin_of_text|>') and repeated words, which is difficult to reconcile with 'nearly perfectly inverted'; reporting per-example token precision/recall would clarify what the F1 score actually means here.
  3. [Section 5.4 / Section 7 (Limitation)] The defense evaluation tests only the generation-based attack, while the abstract's strongest result is produced by the optimization-based TBS attack. The paper's own Limitation section states that TBS 'cannot recover meaningful inputs even under the highest ε we tested because of noise sensitivity,' so the conclusion that the four defenses 'cannot perfectly prevent ISs inversion' is not established for the attack that carries the headline claim. The authors should evaluate TBS under the same defenses or explicitly scope the defense conclusion to the generation-based attack.
  4. [Section 4.2 / Appendix A] TBS's optimization assumes that the true input embeddings are representable in the arctan-bounded basis; no proof or validation of this assumption is provided, and the strong dataset-dependent sensitivity in Table 4 suggests the feasible set may be misspecified for some inputs. The assumption should be stated explicitly and tested, e.g., by measuring whether the converged loss can reach the value expected when a representable target is available.
minor comments (6)
  1. [Section 5.4] The section says 'four practical defenses including DP, pruning, quantization, dropout and noisy input embedding,' which lists five defenses and never evaluates pruning; please correct the enumeration.
  2. [Section 4.3] The Bayesian perspective is simply Bayes' theorem applied to the inversion objective and does not explain the design choices; it can be shortened or removed without affecting the method.
  3. [Figure 8] The color-highlighted missed tokens are not legible in grayscale; consider adding underlining or textual markers.
  4. [Section 5.2.2] 'unbridle the maximum sequence to 4,096' should read 'set the maximum sequence length to 4,096'.
  5. [Table 1] The symbols '#' and the checkmark/cross convention are not defined in the caption; please add a legend.
  6. [Section 5.2.1] The attack settings paragraph describes Llama-3 layer indices, but Table 3 reports results on Llama-2; clarify which settings apply to which model.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the inversion attacks are empirical optimizations and supervised fits, with results measured against held-out inputs.

full rationale

I find no load-bearing circular step. The optimization-based attacks (ER/TBS) minimize a distance between inverted and observed internal states, but the reported F1 and semantic-similarity numbers are evaluated against held-out ground-truth prompts (Table 3, Figures 8-9, 21-23), so the target text is not used to construct the output. The generation-based attack trains a T5 inversion model on surrogate data (NoRobots/Instruction-2M) and is evaluated on separate test data (Tables 7-9), which is standard supervised learning rather than a circular prediction. The 'Bayesian perspective' (Eq. 6) is a straightforward restatement of Bayes' rule and is not used to derive any quantity that is then reported as a prediction. Self-citations such as [14], [60], and [68] appear only in contextual related-work or speculative mitigation discussion and are not load-bearing for the paper's central claims. The paper's own Limitation section acknowledges hyperparameter and noise sensitivity, which is a robustness/correctness concern rather than evidence of circularity.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim that ISs leak sensitive input relies on a set of assumptions about the model's differentiability, the adversary's knowledge, the boundedness of embeddings, and the availability of similar-distribution data for black-box attacks.

free parameters (6)
  • Learning rate µ = 1e-4 to 1e-3
    Tuned per dataset and layer to achieve best inversion results; the paper reports different values in Tables 3, 4, and Figure 9.
  • Penalty weight λ = 0 or 1e-3
    Used in the distribution matching loss; chose 0 by default, 1e-3 for some settings (Table 4).
  • Distance metric = MSE or COS
    Choice between mean squared error and cosine similarity in the inversion loss; COS works better for long prompts (Figure 9).
  • Basis type = Unbiased or SVD
    TBS can use V or V^T from SVD; unbiased is default, SVD can improve results (Table 4).
  • α in ϕz = 5/π
    Scaling for arctan activation to bound z·B; fixed, but a design choice.
  • Max sequence length (generation) = 128-1024
    For the generative inversion model, sequence length is tuned (Figure 12, Table 8).
assumptions (4)
  • standard math Transformer layers are differentiable and gradients can be backpropagated through them
    The optimization attacks require gradient computation through the first l layers (Section 4.2).
  • domain assumption The adversary knows the exact layer index l and tokenizer of the target model
    Stated in the threat model (Section 3).
  • ad hoc to paper The true input embeddings lie within the bounded range representable by the arctan parametrization in TBS
    The TBS attack represents embeddings as α arctan(z·B); if the true embedding exceeds this range, the representation is lossy. The paper does not prove this bound.
  • domain assumption The adversary has access to a surrogate dataset with distribution similar to the victim's queries for black-box attacks
    Required for training the generative inversion model and for replication; the paper analyzes the effect of distribution mismatch (Section 4.3, Appendix B).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Depth Gives a False Sense of Privacy: LLM Internal States Inversion." pith.science (2026). https://pith.science/paper/LP5EZC6K

@misc{pith2026250716372,
  author       = {Pith},
  title        = {Pith review of: Depth Gives a False Sense of Privacy: LLM Internal States Inversion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LP5EZC6K}},
  note         = {Machine review of arXiv:2507.16372}
}
read the original abstract

Large Language Models (LLMs) are increasingly integrated into daily routines, yet they raise significant privacy and safety concerns. Recent research proposes collaborative inference, which outsources the early-layer inference to ensure data locality, and introduces model safety auditing based on inner neuron patterns. Both techniques expose the LLM's Internal States (ISs), which are traditionally considered irreversible to inputs due to optimization challenges and the highly abstract representations in deep layers. In this work, we challenge this assumption by proposing four inversion attacks that significantly improve the semantic similarity and token matching rate of inverted inputs. Specifically, we first develop two white-box optimization-based attacks tailored for low-depth and high-depth ISs. These attacks avoid local minima convergence, a limitation observed in prior work, through a two-phase inversion process. Then, we extend our optimization attack under more practical black-box weight access by leveraging the transferability between the source and the derived LLMs. Additionally, we introduce a generation-based attack that treats inversion as a translation task, employing an inversion model to reconstruct inputs. Extensive evaluation of short and long prompts from medical consulting and coding assistance datasets and 6 LLMs validates the effectiveness of our inversion attacks. Notably, a 4,112-token long medical consulting prompt can be nearly perfectly inverted with 86.88 F1 token matching from the middle layer of Llama-3 model. Finally, we evaluate four practical defenses that we found cannot perfectly prevent ISs inversion and draw conclusions for future mitigation design.

Figures

Figures reproduced from arXiv: 2507.16372 by the authors.

Figure 1
Figure 1. A curious-but-honest LLM safety auditor or collab￾orative inference party can observe ISs and recover the nearly exact user inputs even in deep layers (false inverted tokens are in red). mally, the ISs inversion is xˆ = argmin x ′∈domT d(h fA l (x ′ ),h fV l (x)), s.t.S(x ′ , x) ≥ τS, [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 3
Figure 3. Evaluation of strawman attack TS on Llama-3-8B-Instruct. The strawman approach, TS, fails to converge on the deeper layers even under improved settings. Moreover, when attacking the first layer, the inverted texts contain no overlapping tokens with the input texts. Z = [z1,··· ,z |hl | ] ∈ R|hl |×NT of each embeddings to invert i-th input token of by argmaxbzi , where bzi are the rows of Zb = argmax Z [PITH_FULL_IM… view at source ↗
Figure 4
Figure 4. Intuition behind our TBS inversion attack. Instead [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (19 more)
Figure 5
Figure 5. Figure 5: Comparison between unbiased basis and singular [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Distribution of gradient norm ∥wb∥2 and ∥bz∥2 for ER and TBS attacks, respectively. attacks against LLMs, because the adversary (e.g., collab￾orative inference server) can also exploit the output and logits for inversion. For fair comparison, we use their test dataset …
Figure 7
Figure 7. Figure 7: Distribution of token length from our test data (top 100 longest inputs of long-context benchmarks). The rightmost [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: An inversion example from Mentalhealth consisting of 384 tokens. The missed tokens are highlighted in color. [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Evaluation of our TBS attack with different learning rates and distance functions (MSE and COS). [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: The distribution of autoencoder’s reconstruction [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 11
Figure 11. Figure 11: Evaluation of model type identification. [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 13
Figure 13. Figure 13: Evaluation of converged inversion models. [PITH_FULL_IMAGE:figures/full_fig_p012_13.png]
Figure 12
Figure 12. Figure 12: Generative inversion on long-context datasets. [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]
Figure 14
Figure 14. Figure 14: Evaluation of defenses including noisy input embedding, dropout and differential privacy with Laplace mechanism. [PITH_FULL_IMAGE:figures/full_fig_p014_14.png]
Figure 15
Figure 15. Figure 15: Visualization of internal states for mainstream [PITH_FULL_IMAGE:figures/full_fig_p019_15.png]
Figure 16
Figure 16. Figure 16: Distribution of token length of NoRobots. [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]
Figure 17
Figure 17. Figure 17: Visualization of sampled input embeddings from [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]
Figure 18
Figure 18. Figure 18: Failure inversion example of TBS attack in MentalHealth. [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]
Figure 19
Figure 19. Figure 19: Failure inversion example of generation-based inversion in EvolCode. [PITH_FULL_IMAGE:figures/full_fig_p021_19.png]
Figure 20
Figure 20. Figure 20: Loss curves of TBS attack in Mentalhealth. [PITH_FULL_IMAGE:figures/full_fig_p021_20.png]
Figure 21
Figure 21. Figure 21: Inversion example of a 4,112-token prompt (Part 1). [PITH_FULL_IMAGE:figures/full_fig_p022_21.png]
Figure 22
Figure 22. Figure 22: Inversion example of a 4,112-token prompt (Part 2). [PITH_FULL_IMAGE:figures/full_fig_p023_22.png]
Figure 23
Figure 23. Figure 23: Inversion example of a 4,112-token prompt (Part 3). [PITH_FULL_IMAGE:figures/full_fig_p024_23.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

72 extracted references · 42 canonical work pages

  1. [1]

    X fact checks elon musk after he blasts apple-openai partnership as ‘creepy’ privacy nightmare

    Robert Hart. X fact checks elon musk after he blasts apple-openai partnership as ‘creepy’ privacy nightmare. https://www.forbes.com/sites/roberthart/2024/06/11/x- fact-checks-elon-musk-after-he-blasts-apple-openai- partnership-as-creepy-privacy-nightmare/, 2024

    work page 2024

  2. [2]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  3. [3]

    Splitlora: A split parameter-efficient fine- tuning framework for large language models

    Zheng Lin, Xuanjie Hu, Yuxin Zhang, Zhe Chen, Zihan Fang, Xianhao Chen, Ang Li, Praneeth Vepakomma, and Yue Gao. Splitlora: A split parameter-efficient fine- tuning framework for large language models. arXiv preprint arXiv:2407.00952, 2024

    arXiv 2024

  4. [4]

    Helix: Dis- tributed serving of large language models via max-flow on heterogeneous gpus

    Yixuan Mei, Yonghao Zhuang, Xupeng Miao, Juncheng Yang, Zhihao Jia, and Rashmi Vinayak. Helix: Dis- tributed serving of large language models via max-flow on heterogeneous gpus. In Proceedings of Interna- tional Conference on Architectural Support for Program- ming Languages and Operating Systems, ASPLOS 2025 , 2025

    work page 2025

  5. [5]

    Edgeshard: Efficient llm inference via collaborative edge computing

    Mingjin Zhang, Jiannong Cao, Xiaoming Shen, and Zeyang Cui. Edgeshard: Efficient llm inference via collaborative edge computing. arXiv preprint arXiv:2405.14371, 2024

    arXiv 2024

  6. [6]

    Fact sheet: President biden issues executive order on safe, secure, and trustworthy artificial intelligence. https://www.whitehouse.gov/briefing-room/statements- releases/2023/10/30/fact-sheet-president-biden-issues- executive-order-on-safe-secure-and-trustworthy- artificial-intelligence/, 2023. 15

    work page 2023

  7. [7]

    The internal state of an LLM knows when it‘s lying

    Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it‘s lying. In Findings of the As- sociation for Computational Linguistics: EMNLP 2023 , pages 967–976, December 2023. doi: 10.18653/v1/2023. findings-emnlp.68

    doi:10.18653/v1/2023 2023

  8. [8]

    INSIDE: LLMs’ internal states retain the power of hallucination detection

    Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. INSIDE: LLMs’ internal states retain the power of hallucination detection. In The Twelfth International Conference on Learning Representations, 2024

    work page 2024

Show all 72 references
  1. [9]

    Unsupervised real-time hallucination detection based on the internal states of large language models

    Weihang Su, Changyue Wang, Qingyao Ai, Yiran Hu, Zhijing Wu, Yujia Zhou, and Yiqun Liu. Unsupervised real-time hallucination detection based on the internal states of large language models. In Findings of the Asso- ciation for Computational Linguistics ACL 2024 , pages 14379–1...

    2024

  2. [10]

    Malik, and Yarin Gal

    Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth A. Malik, and Yarin Gal. Semantic entropy probes: Robust and cheap hallucination de- tection in llms. CoRR, abs/2406.15927, 2024. doi: 10.48550/ARXIV .2406.15927

    2024 doi arXiv

  3. [11]

    Physics of language models: Part 2.1, grade-school math and the hidden reasoning process

    Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen- Zhu. Physics of language models: Part 2.1, grade-school math and the hidden reasoning process. arXiv preprint arXiv:2407.20311, 2024

    2024 arXiv

  4. [12]

    Safety layers of aligned large language models: The key to llm security

    Shen Li, Liuyi Yao, Lan Zhang, and Yaliang Li. Safety layers of aligned large language models: The key to llm security. arXiv preprint arXiv:2408.17003, 2024

    2024 arXiv

  5. [13]

    Analyzing and editing inner mechanisms of backdoored language 353 models

    M Lamparth and A Reuel. Analyzing and editing inner mechanisms of backdoored language 353 models. In The 2024 ACM Conference on Fairness, Accountability, and Transparency (F AccT), page 354, 2024

    2024

  6. [14]

    The philosopher’s stone: Trojaning plugins of large language models

    Tian Dong, Minhui Xue, Guoxing Chen, Rayne Holland, Yan Meng, Shaofeng Li, Zhen Liu, and Haojin Zhu. The philosopher’s stone: Trojaning plugins of large language models. In Network and Distributed System Security Symposium, NDSS 2025. The Internet Society, 2025

    2025

  7. [15]

    Rep- resentation engineering: A top-down approach to ai transparency

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Rep- resentation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023

    arXiv

  8. [16]

    Discovering latent knowledge in language models without supervision

    Collin Burns, Haotian Ye, Dan Klein, and Jacob Stein- hardt. Discovering latent knowledge in language models without supervision. In The Eleventh International Con- ference on Learning Representations, 2023

    2023

  9. [17]

    States hidden in hidden states: Llms emerge dis- crete state representations implicitly

    Junhao Chen, Shengding Hu, Zhiyuan Liu, and Maosong Sun. States hidden in hidden states: Llms emerge dis- crete state representations implicitly. arXiv preprint arXiv:2407.11421, 2024

    2024 arXiv

  10. [18]

    Informa- tion leakage in embedding models

    Congzheng Song and Ananth Raghunathan. Informa- tion leakage in embedding models. In Proceedings of the 2020 ACM SIGSAC conference on computer and communications security, pages 377–390, 2020

    2020

  11. [19]

    Text embeddings reveal (almost) as much as text

    John Morris, V olodymyr Kuleshov, Vitaly Shmatikov, and Alexander Rush. Text embeddings reveal (almost) as much as text. In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 12448–12460, Singapore, December

    2023

  12. [20]

    Morris, and Vitaly Shmatikov

    Collin Zhang, John X. Morris, and Vitaly Shmatikov. Extracting prompts by inverting LLM outputs. In Pro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024 , pages 14753–14777. Association for Computational Linguis- tics, 2024

    2024

  13. [21]

    Deep learning and the information bottleneck principle

    Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pages 1–5, 2015. doi: 10.1109/ITW.2015.7133169

    2015

  14. [22]

    Privacy risks of general-purpose language models

    Xudong Pan, Mi Zhang, Shouling Ji, and Min Yang. Privacy risks of general-purpose language models. In 2020 IEEE Symposium on Security and Privacy (SP) , pages 1314–1331, 2020. doi: 10.1109/SP40000.2020. 00095

    2020

  15. [23]

    Sentence embedding leaks more information than you expect: Generative embedding inversion attack to recover the whole sentence

    Haoran Li, Mingshi Xu, and Yangqiu Song. Sentence embedding leaks more information than you expect: Generative embedding inversion attack to recover the whole sentence. arXiv preprint arXiv:2305.03010, 2023

    2023 arXiv

  16. [24]

    Language model inversion

    John Xavier Morris, Wenting Zhao, Justin T Chiu, Vitaly Shmatikov, and Alexander M Rush. Language model inversion. In The Twelfth International Conference on Learning Representations, 2024

    2024

  17. [25]

    TAG: gradient attack on transformer-based lan- guage models

    Jieren Deng, Yijue Wang, Ji Li, Chenghong Wang, Chao Shang, Hang Liu, Sanguthevar Rajasekaran, and Caiwen Ding. TAG: gradient attack on transformer-based lan- guage models. In Findings of the Association for Com- putational Linguistics: EMNLP 2021, pages 3600–3610,

    2021

  18. [26]

    Uncovering gradient inversion risks in prac- tical language model training

    Xinguo Feng, Zhongkui Ma, Zihan Wang, Eu Joe Chegne, Mengyao Ma, Alsharif Abuadbba, and Guang- dong Bai. Uncovering gradient inversion risks in prac- tical language model training. In Proceedings of the 16 2024 on ACM SIGSAC Conference on Computer and Communications Security, ...

    2024

  19. [27]

    Unveiling the vulnerability of private fine-tuning in split-based frameworks for large language models: A bidirectionally enhanced attack

    Guanzhong Chen, Zhenghan Qin, Mingxin Yang, Yajie Zhou, Tao Fan, Tianyu Du, and Zenglin Xu. Unveiling the vulnerability of private fine-tuning in split-based frameworks for large language models: A bidirectionally enhanced attack. In Proceedings of the 2024 on ACM SIGSAC Confe...

    2024

  20. [28]

    How alignment and jail- break work: Explain LLM safety through intermediate hidden states

    Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, and Yongbin Li. How alignment and jail- break work: Explain LLM safety through intermediate hidden states. In Findings of the Association for Compu- tational Linguistics: EMNLP 2024 , pages 2461–2488, November 202...

    2024 doi

  21. [29]

    Enterprise privacy at openai - does ope- nai review my business data? https://openai.com/ enterprise-privacy/, 2024

    OpenAI. Enterprise privacy at openai - does ope- nai review my business data? https://openai.com/ enterprise-privacy/, 2024

    2024

  22. [30]

    Petals: Collaborative inference and fine-tuning of large mod- els

    Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Maksim Riabinin, Younes Belkada, Artem Chu- machenko, Pavel Samygin, and Colin Raffel. Petals: Collaborative inference and fine-tuning of large mod- els. In Proceedings of the 61st Annual Meeting of the Association for Comput...

    2023 doi

  23. [31]

    Shad- ownet: A secure and efficient on-device model inference system for convolutional neural networks

    Zhichuang Sun, Ruimin Sun, Changming Liu, Am- rita Roy Chowdhury, Long Lu, and Somesh Jha. Shad- ownet: A secure and efficient on-device model inference system for convolutional neural networks. In2023 IEEE Symposium on Security and Privacy (SP) , pages 1596– 1612, 2023. doi: ...

    2023

  24. [32]

    Zhang, C

    Z. Zhang, C. Gong, Y . Cai, Y . Yuan, B. Liu, D. Li, Y . Guo, and X. Chen. No privacy left outside: On the (in-)security of tee-shielded dnn partition for on-device ml. In 2024 IEEE Symposium on Security and Privacy (SP), pages 55–55, may 2024

    2024

  25. [33]

    Large dual encoders are generalizable retrievers

    Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gus- tavo Hernández Ábrego, Ji Ma, Vincent Y Zhao, Yi Luan, Keith B Hall, Ming-Wei Chang, et al. Large dual encoders are generalizable retrievers. arXiv preprint arXiv:2112.07899, 2021

    2021 arXiv

  26. [34]

    Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, Fan Yang, Mengnan Du, and Yongfeng Zhang. Exploring concept depth: How large language models acquire knowledge at different layers? In Proceedings of...

    2025

  27. [35]

    Authorship attribution in the era of llms: Problems, methodologies, and challenges

    Baixiang Huang, Canyu Chen, and Kai Shu. Authorship attribution in the era of llms: Problems, methodologies, and challenges. arXiv preprint arXiv:2408.08946, 2024

    2024 arXiv

  28. [36]

    LLM internal states reveal hallucination risk faced with a query

    Ziwei Ji, Delong Chen, Etsuko Ishii, Samuel Cahyawi- jaya, Yejin Bang, Bryan Wilie, and Pascale Fung. LLM internal states reveal hallucination risk faced with a query. In Proceedings of the 7th BlackboxNLP Work- shop: Analyzing and Interpreting Neural Networks for NLP, pages 8...

  29. [37]

    Get my drift? catching llm task drift with activation deltas

    Sahar Abdelnabi, Aideen Fay, Giovanni Cherubin, Ahmed Salem, Mario Fritz, and Andrew Paverd. Get my drift? catching llm task drift with activation deltas. In SaTML, 2025

    2025

  30. [38]

    Minxin Du, Xiang Yue, Sherman S. M. Chow, Tian- hao Wang, Chenyu Huang, and Huan Sun. Dp-forward: Fine-tuning and inference on language models with dif- ferential privacy in forward pass. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security,...

    2023

  31. [39]

    Dataset condensation with distribution matching

    Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2023, 2023

    2023

  32. [40]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  33. [41]

    Bio-medical: A high- performance biomedical language model

    ContactDoctor. Bio-medical: A high- performance biomedical language model. https://huggingface.co/ContactDoctor/Bio-Medical- Llama-3-8B, 2024

    2024

  34. [42]

    Introducing llama 3.1: Our most capable mod- els to date

    Meta AI. Introducing llama 3.1: Our most capable mod- els to date. https://ai.meta.com/blog/meta-llama-3-1/, 2024

    2024

  35. [43]

    C-pack: Packaged resources to advance general chinese embedding, 2023

    Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023

    2023

  36. [44]

    Mteb: Massive text embedding bench- mark

    Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. Mteb: Massive text embedding bench- mark. arXiv preprint arXiv:2210.07316 , 2022. doi: 10.48550/ARXIV .2210.07316. 17

    2022 doi arXiv

  37. [45]

    Massive text em- bedding benchmark (mteb) leaderboard

    MTEB Leaderboard. Massive text em- bedding benchmark (mteb) leaderboard. https://huggingface.co/spaces/mteb/leaderboard, 2024

    2024

  38. [46]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019

    2019

  39. [47]

    Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J

    Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, E...

    2020

  40. [48]

    Ai medical dataset,

    Ruslan Magana Vsevolodovna. Ai medical dataset,

  41. [49]

    mental_health_counseling_conversations (revision 9015341), 2024

    Amod. mental_health_counseling_conversations (revision 9015341), 2024. URL https: //huggingface.co/datasets/Amod/mental_ health_counseling_conversations

    2024

  42. [50]

    Wizardcoder: Empowering code large language models with evol-instruct, 2023

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct, 2023

    2023

  43. [51]

    Measuring coding challenge competence with apps

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Man- tas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Stein- hardt. Measuring coding challenge competence with apps. NeurIPS, 2021

    2021

  44. [52]

    URL https://github.com/ruslanmv/ ai-medical-chatbot

  45. [53]

    The magical effect of the bias term: Rope + bias = better length extrapolation

    Jianlin Su. The magical effect of the bias term: Rope + bias = better length extrapolation. https://spaces. ac.cn/archives/9577, 2023

    2023

  46. [54]

    Rush, and Thomas Wolf

    Nazneen Rajani, Lewis Tunstall, Edward Beeching, Nathan Lambert, Alexander M. Rush, and Thomas Wolf. No robots. https://huggingface.co/datasets/ HuggingFaceH4/no_robots, 2023

    2023

  47. [55]

    QLoRA: Efficient finetuning of quan- tized LLMs

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quan- tized LLMs. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023

    2023

  48. [56]

    Big code models leaderboard

    bigcode. Big code models leaderboard. https://huggingface.co/spaces/bigcode/bigcode- models-leaderboard, 2025

    2025

  49. [57]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021

    2021

  50. [58]

    GAN you see me? enhanced data reconstruction attacks against split infer- ence

    Ziang Li, Mengda Yang, Yaxin Liu, Juan Wang, Hongxin Hu, Wenzhe Yi, and Xiaoyang Xu. GAN you see me? enhanced data reconstruction attacks against split infer- ence. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Syst...

    2023

  51. [59]

    Unveiling the vulnerability of private fine-tuning in split-based frameworks for large language models: A bidirection- ally enhanced attack

    Guanzhong Chen, Zhenhan Qin, Mingxin Yang, Yajie Zhou, Tao Fan, Tianyu Du, and Zenglin Xu. Unveiling the vulnerability of private fine-tuning in split-based frameworks for large language models: A bidirection- ally enhanced attack. In Proceedings of the 2024 ACM SIGSAC Confere...

    2024

  52. [60]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

    2020

  53. [61]

    Input reconstruction attack against verti- cal federated large language models

    Fei Zheng. Input reconstruction attack against verti- cal federated large language models. arXiv preprint arXiv:2311.07585, 2023

    2023 arXiv

  54. [62]

    A first look at efficient and secure on-device llm inference against kv leakage, 2024

    Huan Yang, Deyu Zhang, Yudong Zhao, Yuanchun Li, and Yunxin Liu. A first look at efficient and secure on-device llm inference against kv leakage, 2024

    2024

  55. [63]

    I know what you asked: Prompt leakage via kv-cache sharing in multi- tenant llm serving

    Guanlong Wu, Zheng Zhang, Weili Wang, Jianyu Niu, Yao Zhang, Ye Wu, and Yinqian Zhang. I know what you asked: Prompt leakage via kv-cache sharing in multi- tenant llm serving. In Network and Distributed System Security Symposium, NDSS 2025 . The Internet Society, 2025

    2025

  56. [64]

    Learn what you want to unlearn: Unlearning inver- sion attacks against machine unlearning

    Hongsheng Hu, Shuo Wang, Tian Dong, and Minhui Xue. Learn what you want to unlearn: Unlearning inver- sion attacks against machine unlearning. In 2024 IEEE Symposium on Security and Privacy (SP) , pages 3257– 3275, may 2024. doi: 10.1109/SP54263.2024.00248

    2024

  57. [65]

    Prompt in- ference attack on distributed large language model in- ference frameworks

    Xinjian Luo, Ting Yu, and Xiaokui Xiao. Prompt in- ference attack on distributed large language model in- ference frameworks. arXiv preprint arXiv:2503.09291, 2025

    2025 arXiv

  58. [66]

    Strengthening vm isolation with in- tegrity protection and more

    AMD Sev-Snp. Strengthening vm isolation with in- tegrity protection and more. White Paper , January, 53: 1450–1465, 2020

    2020

  59. [67]

    Confidential Compute on NVIDIA Hopper H100

    Rob Nertney. Confidential Compute on NVIDIA Hopper H100. https://images.nvidia.cn/ aem-dam/en-zz/Solutions/data-center/ HCC-Whitepaper-v1.0.pdf, 2023

    2023

  60. [68]

    Prompt Inversion Attack against Collaborative Inference of Large Language Models

    Wenjie Qu, Yuguang Zhou, Yongji Wu, Tingsong Xiao, Binhang Yuan, Yiming Li, and Jiaheng Zhang. Prompt Inversion Attack against Collaborative Inference of Large Language Models . In 2025 IEEE Symposium on Security and Privacy (SP) , pages 1602–1619. IEEE 18 Table 10: Evaluation...

    2025

  61. [72]

    Seeing is not always believing: The space of harmless perturbations

    Lu Chen, Shaofeng Li, Benhao Huang, Fan Yang, Zheng Li, Jie Li, and Yuan Luo. Seeing is not always believing: The space of harmless perturbations. arXiv preprint arXiv:2402.02095, 2024. A Additional Results Visualization of Internal States. We applies the t-SNE onto the ISs of...

    2024 arXiv

  62. [2021]

    doi: 10.18653/V1/2021.FINDINGS-EMNLP.305

    2021 doi

  63. [2023]

    doi: 10.18653/v1/2023.emnlp-main.765

    Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.765

    2023 doi

  64. [2024]

    doi: 10.18653/v1/2024.blackboxnlp-1.6

    Association for Computational Linguistics. doi: 10.18653/v1/2024.blackboxnlp-1.6

    2024 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.