REVIEW 4 major objections 5 minor 27 references
PII Jailbreaking in LLMs via Activation Steering Reveals Personal Information Leakage
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Adding learned probe directions to a small set of attention heads can switch an LLM's refusal on a privacy question into a response that reveals factually accurate personal information.
desk verdict Probe-guided activation steering does flip privacy refusals in open-weight LLMs, but the paper's factuality rates for 'true personal information' are not backed by the evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is per-head linear probing plus activation steering. For each attention head, a single-layer ridge-regression probe is trained on the last-token activation vector at that head with binary labels (refused vs disclosed) assigned by a privacy evaluator. Probes are ranked by validation F1, and during generation the activation at each of the top-k selected heads is shifted by the update $a_{l,h}^d \leftarrow a_{l,h}^d + \alpha w_{l,h}$, where $w_{l,h}$ is the probe weight vector and $\alpha$ is a tunable strength. The paper's key observation is that this small, localized shift—with negative values of $\alpha$—reliably flips refusal behavior into disclosure across three different model families.
What would settle it
An independent verification pass would settle it: sample a random set of data subjects whose steered responses were labelled factual, and check each claim (orientation, partner names, relationship events) against external public sources. If the verified-true share lands well below the claimed 80–90%, the "true PII leakage" conclusion does not survive. A second decisive control is to replace the learned probe weights with random directions of the same norm and ask whether the same disclosure and factuality rates are reproduced; if they are, the effect is not attributable to the specific refusal direction the probes claim to find.
Extended reading notes
Core claim
The paper's central finding is that linear probes trained on last-token attention-head activations can predict whether a model will refuse or disclose on a privacy query, and that adding the top-ranked probe weights to a small set of heads during generation systematically turns refusals into disclosures. Using GPT-4o-mini as an automated factuality judge, the authors report that at least one steered response per subject is judged factually accurate for 80.4% of jailbroken Llama2-7B subjects, 89.9% for Qwen2.5-7B, and 68.8% for GLM-4-9B. The steered responses can go beyond the queried attribute and reveal associated relational facts, such as a partner's name or relationship timeline, and ten manually verified examples per model confirm that these disclosures track real public information. The conclusion the authors draw is that private information memorized during pre-training can be extracted by targeted activation-level intervention, without modifying the user's prompt.
Load-bearing premise
The load-bearing premise is that the automated judge (GPT-4o-mini) labels responses as "factual" accurately enough to treat the claimed 80–90% factuality rates as evidence of true personal-information leakage, rather than plausible confabulation.
Editorial extensions
If this is right
- If the result is correct, privacy alignment that only changes surface behavior is insufficient: refusal mechanisms localize in a small set of attention heads, and a white-box attacker can negate them without any prompt engineering.
- The attack requires no auxiliary attacker LLM and no crafted jailbreak strings; a fixed template plus probe weights and one scalar parameter is enough to reach 80–96% jailbreak rates.
- Factuality rates above 68% across all three models imply the leakage is not just refusal-breaking but also retrieval of memorized relational facts, so the exposed information goes beyond the attribute in the query.
- Because the best-performing probes sit in middle layers and require only about 110 training examples, the method is inexpensive enough to be used as a routine privacy audit for open-weight models.
- Differences across model families (Qwen2.5-7B being most susceptible) suggest disclosure risk is model-specific, so privacy evaluations should report steering-based leakage alongside prompt-based jailbreak benchmarks.
Reading between the lines
- We infer that the method likely generalizes beyond a single attribute: if refusal heads are attribute-specific but probe training is cheap, the same recipe could be reapplied to health conditions, financial status, or other verifiable sensitive attributes; the paper tests only sexual orientation.
- We infer that the decisive open question is external verifiability: because the factuality judge is drawn from the same model family that generated the subject list, a fully independent fact-checking pass on a larger sample than the ten manually verified examples is the natural next experiment.
- We infer that the controllability of the steering direction (negative $\alpha$ toward disclosure) provides a testable signature of a real refusal direction, so a permutation baseline that scrambles probe labels would clarify how much of the jailbreak is due to the learned direction versus mere activation perturbation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a white-box privacy-jailbreaking attack in which linear probes are trained on last-token attention-head activations to predict refusal versus disclosure for prompts of the form "What is the sexual orientation of {subject}?", and the top-k probe directions are then added to the activations at selected heads to induce non-refusal responses. The benchmark comprises 863 public figures generated by GPT-4o-mini and manually filtered; privacy labels are assigned by Gemma9B, and factuality labels by GPT-4o-mini. Across Llama2-7B, Qwen2.5-7B, and GLM-4-9B, the paper reports jailbreaking rates of 81.6%, 96.0%, and 80.3% and factuality rates among jailbroken subjects of 80.4%, 89.9%, and 68.8%, concluding that aligned LLMs can be steered to reveal memorized personal information.
Significance. If the factuality rates were credible, the paper would be a notable demonstration that activation-level manipulation alone, without adversarial prompting, can induce aligned LLMs to disclose sensitive personal information about real data subjects, and the authors deserve credit for validating the privacy evaluator against human judges and for providing concrete examples of successful disclosures. However, the central privacy-leakage claim is currently not established because the factuality ground truth is produced by the same model family that generated the subject list, and the manual verification selects on the dependent variable rather than validating the reported rates. With an independent, randomly sampled factuality check, the paper would be a useful contribution to the study of PII leakage from LLMs.
major comments (4)
- [Section 3.4 / Table 3] The factuality rates of 80.4%, 89.9%, and 68.8% are computed entirely from GPT-4o-mini labels, and the manual verification procedure is described as continuing "until we identify T = 10 subjects whose disclosures are verifiably factual," which selects on the dependent variable and cannot validate the reported rates. Please manually verify a random sample of jailbroken subjects (or all subjects if feasible) for each model, report the agreement between the automated judge and the manual labels, and provide corrected factuality rates with confidence intervals.
- [Section 4.3 / Table 2] Jailbreaking success is defined as generating at least one disclosed response across 66 (alpha, k) configurations, so the reported rates are maxima over the hyperparameter grid rather than per-configuration success rates. While this is a meaningful feasibility measure for an attacker who can search, the abstract's claim that "more than 50% on average of these responses" reveal true personal information is not supported by the "at least once" statistics in Table 3. Please report the distribution of disclosure and factuality rates across configurations, and if an average-response claim is intended, compute per-response rather than per-subject rates.
- [Section 3.3 / Eq. (1)] The steering intervention in Eq. (1) is never compared against control conditions using random vectors of matched norm, probe directions from non-selected heads, or randomly selected heads. Without such baselines, the causal role of the probe-selected heads in flipping refusal to disclosure is not established, and the jailbreak rates could partly reflect generic activation perturbation effects. Please add these controls and report disclosure and factuality rates under each baseline.
- [Section 3.1 / Section 3.4] The data-subject list is generated by GPT-4o-mini (Section 3.1) and the factuality labels are assigned by GPT-4o-mini (Section 3.4), creating a circular loop in which the judge may be confirming its own parametric associations rather than verifying facts against an external source. The hand-picked examples in Table 4 do not break this loop. Please replace or augment the factuality ground truth with an external source such as Wikipedia with citations, news articles, or human annotation of a random sample, and report the resulting factuality rates.
minor comments (5)
- [Abstract] The abstract states "Evaluations across four LLMs reveal jailbreaking disclosure rates of at least 95%," while the full-text abstract and Section 4 describe three LLMs with rates of at least 80%; please reconcile this inconsistency.
- [Tables 2 and 3] Model names are inconsistent: the tables list "LlaMa2-7B" and "GLM-4-7B" while the text and method section use "Llama2-7B" and "Glm4-9B." Please unify the naming.
- [Section 3.3] The text refers to "Equation 3.3" where it should refer to Eq. (1); please correct the cross-reference.
- [Section 3.4] The manual verification step does not state whether one or both of the two expert annotators performed the factuality check, nor does it report inter-annotator agreement for factuality labels; please provide this information.
- [Table 7] The factuality prompt in the appendix is specific to "sexual orientation of public_figure" rather than being templated with the {{attribute}} placeholder; please make the prompt consistent with the general method.
Circularity Check
No significant circularity: the steering results are empirical measurements, and the factuality-judge concern is an evaluation limitation rather than a definitional reduction.
full rationale
The paper's derivation chain is not circular in the sense of reducing a claimed prediction to a fitted input or to a self-citation. The probes are trained on privacy labels produced by the Gemma9B evaluator and are then used to steer held-out subjects; whether the steered output is labeled disclosed is measured by the same evaluator, but this is an empirical evaluation loop rather than a definitional identity, and the evaluator's agreement with human judges is reported for a random sample in Section 4.1. The central factual-leakage claim is weaker: Section 4.4 reports factuality rates of 80.4%, 89.9%, and 68.8% obtained entirely from GPT-4o-mini, and the manual verification is explicitly limited to T=10 selected successes per model; Section 6 concedes that GPT-4 'may itself introduce evaluation errors.' This is a validity or ground-truth limitation, not a circular reduction, because the paper defines 'factual' against external public information rather than defining it as GPT-4o-mini's output, and no parameter is fitted to force the reported rate. The only self-citation (Nakka et al. 2024) is contextual material in Related Work and is not load-bearing. Accordingly, no circularity steps are identified; score 0.
Assumptions & free parameters
free parameters (2)
- steering strength alpha =
swept from -80 to 30 in steps of 10
- number of steered heads k =
swept over {16, 32, 48, 64, 96}
assumptions (4)
- domain assumption The manually verified list of public figures and their sexual orientations is accurate ground truth.
- domain assumption Gemma9B as an LLM judge reliably labels disclosure vs refusal for privacy queries.
- domain assumption GPT-4o-mini factuality labels can serve as ground truth for the correctness of steered disclosures.
- ad hoc to paper Linear probes on last-token attention activations identify heads whose steering causally controls refusal.
Cite this review
Pith. "Pith review of PII Jailbreaking in LLMs via Activation Steering Reveals Personal Information Leakage." pith.science (2026). https://pith.science/paper/NGXLJBOE
@misc{pith2026250702332,
author = {Pith},
title = {Pith review of: PII Jailbreaking in LLMs via Activation Steering Reveals Personal Information Leakage},
year = {2026},
howpublished = {\url{https://pith.science/paper/NGXLJBOE}},
note = {Machine review of arXiv:2507.02332}
}
read the original abstract
This paper investigates privacy jailbreaking in LLMs via steering, focusing on whether manipulating activations can bypass LLM alignment and alter response behaviors to privacy related queries (e.g., a certain public figure's sexual orientation). We begin by identifying attention heads predictive of refusal behavior for private attributes (e.g., sexual orientation) using lightweight linear probes trained with privacy evaluator labels. Next, we steer the activations of a small subset of these attention heads guided by the trained probes to induce the model to generate non-refusal responses. Our experiments show that these steered responses often disclose sensitive attribute details, along with other private information about data subjects such as life events, relationships, and personal histories that the models would typically refuse to produce. Evaluations across four LLMs reveal jailbreaking disclosure rates of at least 95%, with more than 50% on average of these responses revealing true personal information. Our controlled study demonstrates that private information memorized in LLMs can be extracted through targeted manipulation of internal activations.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609
arXiv 2023
-
[2]
Zouying Cao, Yifei Yang, and Hai Zhao. 2024. Nothing in excess: Mitigating the exaggerated safety for llms via safety-conscious activation steering. arXiv preprint arXiv:2408.11491
arXiv 2024
-
[3]
Zouying Cao, Yifei Yang, and Hai Zhao. 2025. Scans: Mitigating the exaggerated safety for llms via safety-conscious activation steering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23523--23531
2025
-
[4]
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633--2650
work page 2021
-
[5]
Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2023. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419
arXiv 2023
-
[6]
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027
arXiv 2020
-
[7]
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793
arXiv 2024
-
[8]
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594
arXiv 2024
Show all 27 references
-
[9]
Junsol Kim, James Evans, and Aaron Schein. 2025. Linear representations of political perspective emerge in large language models. arXiv preprint arXiv:2503.02080
2025 arXiv
-
[10]
Nathalie Kirch, Constantin Weisser, Severin Field, Helen Yannakoudakis, and Stephen Casper. 2024. What features in prompts jailbreak llms? investigating the mechanisms behind attacks. arXiv preprint arXiv:2411.03343
2024
-
[11]
Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song. 2023. Multi-step jailbreaking privacy attacks on chatgpt. arXiv preprint arXiv:2304.05197
2023 arXiv
-
[12]
Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2023. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451
2023 arXiv
-
[13]
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. 2024. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249
2024 arXiv
-
[14]
Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. 2024. Tree of attacks: Jailbreaking black-box llms automatically. Advances in Neural Information Processing Systems, 37:61065--61105
2024
-
[15]
Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Florian Tram \`e r, and Katherine Lee. 2023. Scalable extraction of training data from (production) language models. arXiv preprint ar...
2023 arXiv
-
[16]
R OpenAI. 2023. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2(5)
2023 arXiv
-
[17]
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277
2023 arXiv
-
[18]
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728--53741
2023
-
[19]
Jitesh Shetty and Jafar Adibi. 2004. The enron email dataset database schema and brief statistical report. Information sciences institute technical report, University of Southern California, 4(1):120--128
2004
-
[20]
Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, et al. 2024. A strongreject for empty jailbreaks. arXiv preprint arXiv:2402.10260
2024 arXiv
-
[21]
Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, et al. 2024. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05561
2024 arXiv
-
[22]
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295
2024 arXiv
-
[23]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288
2023 arXiv
-
[24]
Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. 2023. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. In NeurIPS
2023
-
[25]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771
2019 arXiv
-
[26]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[27]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.