Pith. sign in

REVIEW 4 major objections 5 minor 27 references

PII Jailbreaking in LLMs via Activation Steering Reveals Personal Information Leakage

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read Adding learned probe directions to a small set of attention heads can switch an LLM's refusal on a privacy question into a response that reveals factually accurate personal information.

desk verdict Probe-guided activation steering does flip privacy refusals in open-weight LLMs, but the paper's factuality rates for 'true personal information' are not backed by the evidence. read the letter →

arxiv 2507.02332 v2 pith:NGXLJBOE submitted 2025-07-03 cs.CR

classification cs.CR
keywords activationsteeringprivacyleakagejailbreakingattentionheadprobespersonalinformationLLMalignmentsexualorientationrefusalbehavior
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the privacy alignment of instruction-tuned LLMs can be bypassed by a purely internal intervention: adding a few learned probe directions to the activations of a small subset of attention heads. This steering converts refusal responses into disclosures for prompts like "What is the sexual orientation of {public figure}?", and the authors report that many of these disclosures contain factually accurate personal information that the unsteered model would refuse to give. Across Llama2-7B, Qwen2.5-7B, and GLM-4-9B, the jailbreak success rates are 81.6%, 96.0%, and 80.3%, and the automated factuality rates among jailbroken subjects are 80.4%, 89.9%, and 68.8%. If the automated factuality judge can be trusted, this is a concrete demonstration that memorized private data can be extracted by manipulating internal activations alone, without adversarial prompts. The paper offers this as a stress test for privacy alignment and a reason to test LLMs for this kind of leakage.

What carries the argument

The central mechanism is per-head linear probing plus activation steering. For each attention head, a single-layer ridge-regression probe is trained on the last-token activation vector at that head with binary labels (refused vs disclosed) assigned by a privacy evaluator. Probes are ranked by validation F1, and during generation the activation at each of the top-k selected heads is shifted by the update $a_{l,h}^d \leftarrow a_{l,h}^d + \alpha w_{l,h}$, where $w_{l,h}$ is the probe weight vector and $\alpha$ is a tunable strength. The paper's key observation is that this small, localized shift—with negative values of $\alpha$—reliably flips refusal behavior into disclosure across three different model families.

What would settle it

An independent verification pass would settle it: sample a random set of data subjects whose steered responses were labelled factual, and check each claim (orientation, partner names, relationship events) against external public sources. If the verified-true share lands well below the claimed 80–90%, the "true PII leakage" conclusion does not survive. A second decisive control is to replace the learned probe weights with random directions of the same norm and ask whether the same disclosure and factuality rates are reproduced; if they are, the effect is not attributable to the specific refusal direction the probes claim to find.

Watch

Extended reading notes

Core claim

The paper's central finding is that linear probes trained on last-token attention-head activations can predict whether a model will refuse or disclose on a privacy query, and that adding the top-ranked probe weights to a small set of heads during generation systematically turns refusals into disclosures. Using GPT-4o-mini as an automated factuality judge, the authors report that at least one steered response per subject is judged factually accurate for 80.4% of jailbroken Llama2-7B subjects, 89.9% for Qwen2.5-7B, and 68.8% for GLM-4-9B. The steered responses can go beyond the queried attribute and reveal associated relational facts, such as a partner's name or relationship timeline, and ten manually verified examples per model confirm that these disclosures track real public information. The conclusion the authors draw is that private information memorized during pre-training can be extracted by targeted activation-level intervention, without modifying the user's prompt.

Load-bearing premise

The load-bearing premise is that the automated judge (GPT-4o-mini) labels responses as "factual" accurately enough to treat the claimed 80–90% factuality rates as evidence of true personal-information leakage, rather than plausible confabulation.

Editorial extensions

If this is right

  • If the result is correct, privacy alignment that only changes surface behavior is insufficient: refusal mechanisms localize in a small set of attention heads, and a white-box attacker can negate them without any prompt engineering.
  • The attack requires no auxiliary attacker LLM and no crafted jailbreak strings; a fixed template plus probe weights and one scalar parameter is enough to reach 80–96% jailbreak rates.
  • Factuality rates above 68% across all three models imply the leakage is not just refusal-breaking but also retrieval of memorized relational facts, so the exposed information goes beyond the attribute in the query.
  • Because the best-performing probes sit in middle layers and require only about 110 training examples, the method is inexpensive enough to be used as a routine privacy audit for open-weight models.
  • Differences across model families (Qwen2.5-7B being most susceptible) suggest disclosure risk is model-specific, so privacy evaluations should report steering-based leakage alongside prompt-based jailbreak benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that the method likely generalizes beyond a single attribute: if refusal heads are attribute-specific but probe training is cheap, the same recipe could be reapplied to health conditions, financial status, or other verifiable sensitive attributes; the paper tests only sexual orientation.
  • We infer that the decisive open question is external verifiability: because the factuality judge is drawn from the same model family that generated the subject list, a fully independent fact-checking pass on a larger sample than the ten manually verified examples is the natural next experiment.
  • We infer that the controllability of the steering direction (negative $\alpha$ toward disclosure) provides a testable signature of a real refusal direction, so a permutation baseline that scrambles probe labels would clarify how much of the jailbreak is due to the learned direction versus mere activation perturbation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a white-box privacy-jailbreaking attack in which linear probes are trained on last-token attention-head activations to predict refusal versus disclosure for prompts of the form "What is the sexual orientation of {subject}?", and the top-k probe directions are then added to the activations at selected heads to induce non-refusal responses. The benchmark comprises 863 public figures generated by GPT-4o-mini and manually filtered; privacy labels are assigned by Gemma9B, and factuality labels by GPT-4o-mini. Across Llama2-7B, Qwen2.5-7B, and GLM-4-9B, the paper reports jailbreaking rates of 81.6%, 96.0%, and 80.3% and factuality rates among jailbroken subjects of 80.4%, 89.9%, and 68.8%, concluding that aligned LLMs can be steered to reveal memorized personal information.

Significance. If the factuality rates were credible, the paper would be a notable demonstration that activation-level manipulation alone, without adversarial prompting, can induce aligned LLMs to disclose sensitive personal information about real data subjects, and the authors deserve credit for validating the privacy evaluator against human judges and for providing concrete examples of successful disclosures. However, the central privacy-leakage claim is currently not established because the factuality ground truth is produced by the same model family that generated the subject list, and the manual verification selects on the dependent variable rather than validating the reported rates. With an independent, randomly sampled factuality check, the paper would be a useful contribution to the study of PII leakage from LLMs.

major comments (4)
  1. [Section 3.4 / Table 3] The factuality rates of 80.4%, 89.9%, and 68.8% are computed entirely from GPT-4o-mini labels, and the manual verification procedure is described as continuing "until we identify T = 10 subjects whose disclosures are verifiably factual," which selects on the dependent variable and cannot validate the reported rates. Please manually verify a random sample of jailbroken subjects (or all subjects if feasible) for each model, report the agreement between the automated judge and the manual labels, and provide corrected factuality rates with confidence intervals.
  2. [Section 4.3 / Table 2] Jailbreaking success is defined as generating at least one disclosed response across 66 (alpha, k) configurations, so the reported rates are maxima over the hyperparameter grid rather than per-configuration success rates. While this is a meaningful feasibility measure for an attacker who can search, the abstract's claim that "more than 50% on average of these responses" reveal true personal information is not supported by the "at least once" statistics in Table 3. Please report the distribution of disclosure and factuality rates across configurations, and if an average-response claim is intended, compute per-response rather than per-subject rates.
  3. [Section 3.3 / Eq. (1)] The steering intervention in Eq. (1) is never compared against control conditions using random vectors of matched norm, probe directions from non-selected heads, or randomly selected heads. Without such baselines, the causal role of the probe-selected heads in flipping refusal to disclosure is not established, and the jailbreak rates could partly reflect generic activation perturbation effects. Please add these controls and report disclosure and factuality rates under each baseline.
  4. [Section 3.1 / Section 3.4] The data-subject list is generated by GPT-4o-mini (Section 3.1) and the factuality labels are assigned by GPT-4o-mini (Section 3.4), creating a circular loop in which the judge may be confirming its own parametric associations rather than verifying facts against an external source. The hand-picked examples in Table 4 do not break this loop. Please replace or augment the factuality ground truth with an external source such as Wikipedia with citations, news articles, or human annotation of a random sample, and report the resulting factuality rates.
minor comments (5)
  1. [Abstract] The abstract states "Evaluations across four LLMs reveal jailbreaking disclosure rates of at least 95%," while the full-text abstract and Section 4 describe three LLMs with rates of at least 80%; please reconcile this inconsistency.
  2. [Tables 2 and 3] Model names are inconsistent: the tables list "LlaMa2-7B" and "GLM-4-7B" while the text and method section use "Llama2-7B" and "Glm4-9B." Please unify the naming.
  3. [Section 3.3] The text refers to "Equation 3.3" where it should refer to Eq. (1); please correct the cross-reference.
  4. [Section 3.4] The manual verification step does not state whether one or both of the two expert annotators performed the factuality check, nor does it report inter-annotator agreement for factuality labels; please provide this information.
  5. [Table 7] The factuality prompt in the appendix is specific to "sexual orientation of public_figure" rather than being templated with the {{attribute}} placeholder; please make the prompt consistent with the general method.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the steering results are empirical measurements, and the factuality-judge concern is an evaluation limitation rather than a definitional reduction.

full rationale

The paper's derivation chain is not circular in the sense of reducing a claimed prediction to a fitted input or to a self-citation. The probes are trained on privacy labels produced by the Gemma9B evaluator and are then used to steer held-out subjects; whether the steered output is labeled disclosed is measured by the same evaluator, but this is an empirical evaluation loop rather than a definitional identity, and the evaluator's agreement with human judges is reported for a random sample in Section 4.1. The central factual-leakage claim is weaker: Section 4.4 reports factuality rates of 80.4%, 89.9%, and 68.8% obtained entirely from GPT-4o-mini, and the manual verification is explicitly limited to T=10 selected successes per model; Section 6 concedes that GPT-4 'may itself introduce evaluation errors.' This is a validity or ground-truth limitation, not a circular reduction, because the paper defines 'factual' against external public information rather than defining it as GPT-4o-mini's output, and no parameter is fitted to force the reported rate. The only self-citation (Nakka et al. 2024) is contextual material in Related Work and is not load-bearing. Accordingly, no circularity steps are identified; score 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the reliability of the GPT-4o-mini factuality judge, the Gemma9B privacy judge, and the assumption that probe-selected heads are causally responsible. The steering strength and head count are freely swept. No new theoretical entities are introduced.

free parameters (2)
  • steering strength alpha = swept from -80 to 30 in steps of 10
    The attack success is reported as positive if any configuration discloses; alpha is chosen post hoc to maximize disclosure rate across the sweep.
  • number of steered heads k = swept over {16, 32, 48, 64, 96}
    The number of top-F1 heads to steer is a free attack parameter, selected by search, not fixed by a principle.
assumptions (4)
  • domain assumption The manually verified list of public figures and their sexual orientations is accurate ground truth.
    Section 3.1: the list is generated by GPT-4o-mini and then manually verified by two annotators. The list itself is not provided, so the ground truth cannot be independently audited.
  • domain assumption Gemma9B as an LLM judge reliably labels disclosure vs refusal for privacy queries.
    Section 3.2: validated against human majority on 500 samples with 92.8-98% agreement, but human labels themselves showed disagreement (Section 6), and the validation set is a convenience sample.
  • domain assumption GPT-4o-mini factuality labels can serve as ground truth for the correctness of steered disclosures.
    Section 3.4: used to label responses as factual or hallucinated. This is load-bearing and acknowledged as problematic in Section 6.
  • ad hoc to paper Linear probes on last-token attention activations identify heads whose steering causally controls refusal.
    Section 3.3: the method assumes that adding the probe weight direction to the top-F1 heads will flip behavior; no control baseline with random directions or random heads is provided, so a causal link is not established.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PII Jailbreaking in LLMs via Activation Steering Reveals Personal Information Leakage." pith.science (2026). https://pith.science/paper/NGXLJBOE

@misc{pith2026250702332,
  author       = {Pith},
  title        = {Pith review of: PII Jailbreaking in LLMs via Activation Steering Reveals Personal Information Leakage},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NGXLJBOE}},
  note         = {Machine review of arXiv:2507.02332}
}
read the original abstract

This paper investigates privacy jailbreaking in LLMs via steering, focusing on whether manipulating activations can bypass LLM alignment and alter response behaviors to privacy related queries (e.g., a certain public figure's sexual orientation). We begin by identifying attention heads predictive of refusal behavior for private attributes (e.g., sexual orientation) using lightweight linear probes trained with privacy evaluator labels. Next, we steer the activations of a small subset of these attention heads guided by the trained probes to induce the model to generate non-refusal responses. Our experiments show that these steered responses often disclose sensitive attribute details, along with other private information about data subjects such as life events, relationships, and personal histories that the models would typically refuse to produce. Evaluations across four LLMs reveal jailbreaking disclosure rates of at least 95%, with more than 50% on average of these responses revealing true personal information. Our controlled study demonstrates that private information memorized in LLMs can be extracted through targeted manipulation of internal activations.

Figures

Figures reproduced from arXiv: 2507.02332 by the authors.

Figure 1
Figure 1. PII Jailbreaking by activation steering can not only generate non-refusal responses but also risk leaking true personal information about the data sub￾ject. human safety expectations. Despite these efforts, recent research (Liu et al., 2023; Chao et al., 2023; Mehrotra et al., 2024) demonstrated that LLMs can be prompted or manipulated to bypass align￾ment constraints and generate harmful outputs (i.e. subjected to … view at source ↗
Figure 2
Figure 2. Response distribution. We test 7 open-source LLMs with a simple prompt xd, asking for the sexual ori￾entation of N = 863 data subjects and plot the distribu￾tion of response content, evaluated by Gemma9B (Team et al., 2024) as a privacy evaluator. prior work has applied steering to elicit unsafe or policy-violating content (Cao et al., 2025, 2024; Li et al., 2023; Kirch et al., 2024), our work uniquely focuses on us… view at source ↗
Figure 3
Figure 3. Top: We label the generated model responses yd with privacy labels yd using privacy evaluator P. Bottom: We extract attention head activations a l,h from the probing prompt xd (ie., without response generation) and train probes wl,h using the corresponding privacy labels {yd} and attention-head features {a l,h} across all layers and heads. Probes here refer to a set of binary classifiers. consider sexual orientation… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Privacy refusal behaviour emerges from internal activations. We visualize the attention activations corresponding to the last token of input prompts xd at the top-4 highest-scoring attention heads (left to right) for three different LLMs. Each point represents a subjec…
Figure 5
Figure 5. Figure 5: Performance of Probes. We plot the perfor￾mance of probes on Llama-2-7B (Touvron et al., 2023) using two evaluation metrics. The x-axis represents at￾tention head indices (sorted by performance), and the y-axis indicates the corresponding layer positions. For complete …
Figure 6
Figure 6. Figure 6: Performance of privacy jailbreaking with different steering parameters. We vary the steering strength α along the x-axis and number of top-k attention heads along the y-axis. Model Subjects Jailbreaking rates LlaMa2-7B 769 628 (81.6%) Qwen2.5-7B 301 289 (96.0%) Glm-4-7…
Figure 7
Figure 7. Figure 7: Performance of Probes with different models. We plot the results of the probes with three models, Llama-7B, Qwen7B and GLM9B from top to bottom. We train a linear probe independently on each attention head of every layer in the LLM and plot the performance of the probe…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 2 canonical work pages

  1. [1]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

  2. [2]

    Zouying Cao, Yifei Yang, and Hai Zhao. 2024. Nothing in excess: Mitigating the exaggerated safety for llms via safety-conscious activation steering. arXiv preprint arXiv:2408.11491

  3. [3]

    Zouying Cao, Yifei Yang, and Hai Zhao. 2025. Scans: Mitigating the exaggerated safety for llms via safety-conscious activation steering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23523--23531

  4. [4]

    Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633--2650

  5. [5]

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2023. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419

  6. [6]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027

  7. [7]

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793

  8. [8]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594

Show all 27 references
  1. [9]

    Junsol Kim, James Evans, and Aaron Schein. 2025. Linear representations of political perspective emerge in large language models. arXiv preprint arXiv:2503.02080

  2. [10]

    Nathalie Kirch, Constantin Weisser, Severin Field, Helen Yannakoudakis, and Stephen Casper. 2024. What features in prompts jailbreak llms? investigating the mechanisms behind attacks. arXiv preprint arXiv:2411.03343

  3. [11]

    Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song. 2023. Multi-step jailbreaking privacy attacks on chatgpt. arXiv preprint arXiv:2304.05197

  4. [12]

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2023. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451

  5. [13]

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. 2024. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249

  6. [14]

    Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. 2024. Tree of attacks: Jailbreaking black-box llms automatically. Advances in Neural Information Processing Systems, 37:61065--61105

  7. [15]

    Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Florian Tram \`e r, and Katherine Lee. 2023. Scalable extraction of training data from (production) language models. arXiv preprint ar...

  8. [16]

    R OpenAI. 2023. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2(5)

  9. [17]

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277

  10. [18]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728--53741

  11. [19]

    Jitesh Shetty and Jafar Adibi. 2004. The enron email dataset database schema and brief statistical report. Information sciences institute technical report, University of Southern California, 4(1):120--128

  12. [20]

    Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, et al. 2024. A strongreject for empty jailbreaks. arXiv preprint arXiv:2402.10260

  13. [21]

    Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, et al. 2024. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05561

  14. [22]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295

  15. [23]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

  16. [24]

    Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. 2023. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. In NeurIPS

  17. [25]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771

  18. [26]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  19. [27]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.