Pith. sign in

REVIEW 3 major objections 5 minor 13 references

PIG: Privacy Jailbreak Attack on LLMs via Gradient-based Iterative In-Context Optimization

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Jailbreak attacks adapted to privacy can extract stored PII from aligned LLMs at near-100% success.

desk verdict PIG is a credible and practical attack that likely beats ICL-only jailbreaks for PII extraction, but the near-100% ASR numbers need a response-level breakdown before they're fully trustworthy. read the letter →

arxiv 2505.09921 v2 pith:DBGLPZJG submitted 2025-05-15 cs.CR cs.CL

Yidan Wang , Yanan Cao , Yubing Ren , Fang Fang , Zheng Lin , Binxing Fang This is my paper · ORCID
classification cs.CRcs.CL
keywords privacyjailbreakattackpersonallyidentifiableinformationin-contextlearninggradient-basedoptimizationLLMleakagesuccessrateTrustbenchmarkdataextraction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that jailbreak attacks, normally built to make a model produce harmful content, can be redirected to extract personally identifiable information from an LLM's stored context or memory. It introduces PIG, which detects the type of PII in a query, builds a few in-context demonstrations using fabricated values of the same type, and then iteratively edits tokens in those demonstrations with gradients until the model answers with the requested value. On four aligned open-weight models the paper reports attack success rates near 100 percent, and the method transfers to black-box APIs, reaching 97.1 percent on GPT-4o on the standard privacy template. The paper argues that this shows current privacy-leakage evaluations, which rely on memorized prefixes or simple instructions, systematically underestimate how easily sensitive data can be extracted.

What carries the argument

The load-bearing mechanism is the iteratively optimized privacy context. PIG first identifies the PII entity and type in a query, then assembles a set of demonstration question-answer pairs whose answers contain randomly generated values of that same type. The optimization step follows Algorithm 1: compute token gradients against the reference prefix, keep the top-k replacement candidates for each token, sample B perturbed contexts, and select the one with the lowest cross-entropy loss. The three strategies differ only in which token positions are editable -- random positions, positions inside PII entity tokens, or the M positions with the largest average gradient -- and combining their results is what pushes the attack success rate to the reported levels.

What would settle it

Re-run PIG on the TrustLLM dataset but score as success only responses that contain the exact target PII value in the completion immediately following the optimized opening phrase; count as failures outputs that begin with that phrase and then refuse, apologize, or give another value. If such partial-prefix outputs are common, the near-100% attack success rate would drop measurably.

Watch

Extended reading notes

Core claim

The paper's central claim is that privacy leakage from aligned LLMs is better modeled as an optimization problem over the prompt than as a failure of refusal training. PIG constructs a privacy context from demonstrations pairing similar requests with synthetically generated PII values, then optimizes selected tokens of that context to raise the probability of a reference response such as "Sure, David's phone password is". Three token-selection strategies -- random, entity-only, and gradient-importance-based -- explore different parts of the context, and their outputs are combined to avoid local optima. As reported, this yields nearly 100 percent attack success on LLaMA2-7b, Mistral-7b, Vicuna-7b, and LLaMA3-8b under the normal TrustLLM template, 97.1 percent on GPT-4o, and a large reduction in refusal rate on the Enron training-data extraction task.

Load-bearing premise

The attack's optimization target is only a polite opening phrase -- "Sure, David's phone password is" -- without the secret value, so the reported success rates assume that once a model starts with that phrase it will continue with the correct value stored in the system prompt rather than refusing or inventing a different value.

Editorial extensions

If this is right

  • Models that are considered well-aligned can still be induced to output stored PII when the attacker controls a few in-context examples, so privacy protections cannot rely on refusal training alone.
  • Attackers do not need a universal adversarial suffix; a query-specific context optimized for a few hundred epochs is enough, which makes targeted extraction cheaper than GCG-style universal attacks.
  • Success on black-box APIs through transferred prompts implies that closed models inherit the same in-context optimization vulnerability even when gradients are unavailable.
  • Privacy benchmarks built on prefix prompts or simple instructions underestimate leakage; evaluation should include optimized-context attacks as a standard baseline.
  • Because PII demonstrations can be synthesized from random names and numbers, the attack does not require access to real sensitive data to be constructed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A stricter evaluation that requires the exact PII value to appear immediately after the optimized prefix, rather than anywhere in the response, would clarify how much of the reported attack success reflects true retrieval versus compliant-but-vague completions.
  • The same context-optimization loop should transfer to other high-value secret types, such as API keys, internal identifiers, or medical record fields, provided a synthetic demonstration set of the same type exists; this is a testable extension the paper does not run.
  • An implied defense is to detect or limit in-context optimization itself, for example by checking whether the prompt contains an unusually low-loss continuation toward a fixed prefix, or by requiring out-of-band authorization before the model emits any stored value.
  • The method's success on Enron suggests that training-data extraction and context leakage are not separate problems but can be attacked with the same optimized-demonstration machinery.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper proposes PIG, a privacy jailbreak framework that first identifies PII types and entities in a privacy query, builds in-context demonstrations with synthetic entities, and then iteratively updates the privacy context using three gradient-based token-selection strategies (random, entity, dynamic) until the target model responds with a possible PII entity. The authors benchmark ten jailbreak baselines plus PIG on the TrustLLM and Enron datasets across four open-source white-box models and two API-based black-box models, reporting near-100% ASR on most white-box models, strong transfer to GPT-4o and Claude-3.5, faster convergence than GCG, and ablations isolating the in-context learning and optimization components.

Significance. If the reported ASR numbers hold, this is a useful contribution to LLM privacy evaluation: it provides a common evaluation harness for ten jailbreak baselines, shows that ICL alone is insufficient on well-aligned models (Fig. 3), demonstrates faster convergence than GCG (Fig. 5), and releases code. The central claim, however, is that the attack recovers the exact PII stored in the system prompt, and the manuscript currently does not establish that its optimization objective and stopping rule match the ASR metric it reports. The paper's benchmark of existing jailbreak methods in a privacy setting is valuable regardless, and the public datasets and code release are strengths.

major comments (3)
  1. [Section 4.3, Eq. (4); Algorithm 1, lines 10-13; Appendix D] The optimization target in Eq. (4) is the PII-free reference response R' (e.g., “Sure, David’s phone password is”), and the stopping rule in Algorithm 1 fires when Eθ(R) returns True, i.e., when the response contains “a possible PII entity.” In contrast, the ASR metric in Appendix D counts only responses in which the exact stored value p is a substring of r. Nothing in the paper measures how often the optimized prefix is completed with the correct p as opposed to a hallucinated value or a refusal. Consequently, the near-100% ASR figures in Table 1 are not yet supported by the stated objective and stopping criterion. Please add an evaluation that either (a) uses the known p, which is available in the TrustLLM system prompt, as the termination predicate and reports ASR under that predicate, or (b) reports the breakdown, for line-11 successes, of correct-p, wrong-value, and refusal-after-prefix responses.
  2. [Section 6.1 (Black-box Closed-source LLM APIs); Section 5 (Implementation Details)] The black-box evaluation is described only as “we transferred successful jailbreak prompts from white-box models to black-box APIs” with a citation to Zou et al. The protocol is underspecified: which white-box model produced the transferred prompts, whether transfer was per-query or via a universal template, how many prompts were transferred per query, and the query budget for GPT-4o and Claude-3.5. Since Table 1 reports ASR of 97.1% on GPT-4o and 85.7% on Claude-3.5 as headline results, this missing protocol makes the black-box claim non-reproducible.
  3. [Section 6.3.2; Appendix E, Table 4] Section 6.3.2 states that combining the three strategies gives ASRs of 92.85% on LLaMA2-7b and 94.3% on LLaMA3-8b over 100 epochs, but Appendix E Table 4 reports 100% ASR for PIG under the normal template for both models, and Table 1 also reports 100% ASR in the same setting. The discrepancy, which is likely due to the 70-sample subset introduced in Section 6.3.1, is not explained; please clarify the sample sizes and report the ablation numbers consistently.
minor comments (5)
  1. [Abstract] There is a typo: “availble” should be “available.”
  2. [Figure 2] The Step 3 box contains garbled text (e.g., “PIG Weto’sAnswerTomneedDynamicRandomEntity” and “Visa”), which obscures the differences among the random, entity, and dynamic strategies.
  3. [Table 2] Several rows run values together without spacing (e.g., the PIG x=5 row contains “100%62.5%” and similar), making the table unreliable to read; use explicit column separators.
  4. [Algorithm 1] The evaluation model Eθ is an input to the algorithm but is never formally defined in Section 4 or Section 5; please state how Eθ relates to the Longformer RtA classifier described in Appendix D and what exactly “contains a possible PII entity” means operationally.
  5. [Appendix F, Section F.2.1] The Mistral response displayed under “No rejection, but no privacy leaked” actually lists the email addresses of all six senders, which contradicts the category label; please verify the case or correct the category description.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the optimization objective is a hand-crafted PII-free prefix, and the reported ASR is evaluated against external ground truth, so the central claim is not equivalent to the method's inputs by construction.

full rationale

The paper's derivation chain is: identify the PII entity and type in the query (§4.1), construct demonstrations with synthetic PII values (§4.2), and optimize a subset of context tokens to minimize -log Pθ(R'|J), where R' is the PII-free prefix 'Sure, David's phone password is' (§4.3, Eq. 4). The reported success metric, ASR in Appendix D, is defined independently as the exact ground-truth PII being a substring of the response. The optimization target therefore does not contain the true PII value, nor is it fitted to the ASR definition; it is a hand-written prefix that the authors empirically assume will be completed with the value stored in the system prompt. The stopping rule in Algorithm 1 fires when the evaluator Eθ detects 'a possible PII entity', which is a weaker condition than the ASR definition, and Appendix F explicitly acknowledges a 'no rejection, but no privacy leaked' category, showing the authors do not conflate the two conditions. The near-100% ASR in Table 1 is thus an external empirical measurement, not a quantity forced by Eq. 4 or by any self-citation. There are no load-bearing self-citations: the method is compared against external baselines (GCG, PAIR, Jailbroken, etc.) on public datasets (TrustLLM, Enron), and the gradient-based optimization is openly presented as an adaptation of GCG with in-context initialization. The only substantive concern is that the headline ASR presupposes, without a reported breakdown, that a model producing the compliant prefix continues with the correct stored PII; that is a validity/measurement assumption, not a circular definition, and it does not make the reported prediction equivalent to the fitted input.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

PIG's central claim depends on a small set of standard modeling assumptions and attack-scenario assumptions. The optimizer itself introduces no new entities. The most fragile assumption is that a prefix-only objective is sufficient to elicit the correct stored PII value, which is not formally proven.

free parameters (5)
  • reference response R' = 'Sure, David's phone password is'
    A hand-written prefix representing the desired compliant response start, used as the optimization target in Eq. 4.
  • dynamic strategy selection size M = 64
    Number of most important tokens selected by average gradient magnitude, set manually and used in the dynamic strategy.
  • iteration count T = 500
    Maximum optimization rounds set for GCG and PIG in implementation details.
  • candidate size k = 256
    Top-k candidate tokens per position sampled from gradient in Algorithm 1.
  • sampling size B = 512
    Number of perturbed contexts sampled per iteration.
assumptions (4)
  • standard math Autoregressive token prediction model (Eq. 1)
    Assumes the target LLM is a transformer-based autoregressive model with differentiable log-probabilities.
  • domain assumption Attacker can compute gradients of the target model
    White-box setting assumes full access to open-source model; black-box transfer assumes transferability of optimized prompts.
  • domain assumption PII entities in the system prompt are known to the model and can be copied when compliant
    The attack only needs to elicit a compliant frame; the model is assumed to then output the correct stored value. This is the core assumption behind using a prefix-only reference response.
  • domain assumption GPT-4 accurately identifies PII types and entities in queries
    The PII detection prompt relies on GPT-4's reasoning; errors would produce wrong demonstrations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PIG: Privacy Jailbreak Attack on LLMs via Gradient-based Iterative In-Context Optimization." pith.science (2026). https://pith.science/paper/DBGLPZJG

@misc{pith2026250509921,
  author       = {Pith},
  title        = {Pith review of: PIG: Privacy Jailbreak Attack on LLMs via Gradient-based Iterative In-Context Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DBGLPZJG}},
  note         = {Machine review of arXiv:2505.09921}
}
read the original abstract

Large Language Models (LLMs) excel in various domains but pose inherent privacy risks. Existing methods to evaluate privacy leakage in LLMs often use memorized prefixes or simple instructions to extract data, both of which well-alignment models can easily block. Meanwhile, Jailbreak attacks bypass LLM safety mechanisms to generate harmful content, but their role in privacy scenarios remains underexplored. In this paper, we examine the effectiveness of jailbreak attacks in extracting sensitive information, bridging privacy leakage and jailbreak attacks in LLMs. Moreover, we propose PIG, a novel framework targeting Personally Identifiable Information (PII) and addressing the limitations of current jailbreak methods. Specifically, PIG identifies PII entities and their types in privacy queries, uses in-context learning to build a privacy context, and iteratively updates it with three gradient-based strategies to elicit target PII. We evaluate PIG and existing jailbreak methods using two privacy-related datasets. Experiments on four white-box and two black-box LLMs show that PIG outperforms baseline methods and achieves state-of-the-art (SoTA) results. The results underscore significant privacy risks in LLMs, emphasizing the need for stronger safeguards. Our code is availble at https://github.com/redwyd/PrivacyJailbreak.

Figures

Figures reproduced from arXiv: 2505.09921 by the authors.

Figure 1
Figure 1. An example of privacy jailbreak attack via [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overview of our proposed framework PIG for privacy jailbreak attack. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The ASR of ICL under different few-shots. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The top compares initialization loss, and the [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 6
Figure 6. Figure 6: The experimental results for 29 methods from [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 7 canonical work pages

  1. [1]

    Jim Kolodgie (jim.kolodgie@eds.com) - No content provided

    work page

  2. [2]

    Preprint, arXiv:2310.08419

    Jailbreaking black box large language models in twenty queries. Preprint, arXiv:2310.08419. Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. 2023. Multilingual jailbreak chal- lenges in large language models. arXiv preprint arXiv:2310.06474. Peng Ding, Jun Kuang, Dan Ma, Xuezhi Cao, Yun- sen Xian, Jiajun Chen, and Shujian Huang. 2024. A wolf in...

    arXiv 2023

  3. [3]

    Kay Quigley (kayquigs@yahoo.com) - No content provided

    work page

  4. [4]

    Nora McGee (mcgee.nora@epa.gov) - No content provided

    work page

  5. [5]

    arXiv preprint arXiv:2309.10253

    Gptfuzzer: Red teaming large language mod- els with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.10253. Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu

    arXiv

  6. [6]

    style_injection_short

    GPT-4 is too smart to be safe: Stealthy chat 10 with LLMs via cipher. In The Twelfth International Conference on Learning Representations. Shenglai Zeng, Jiankun Zhang, Pengfei He, Yiding Liu, Yue Xing, Han Xu, Jie Ren, Yi Chang, Shuaiqiang Wang, Dawei Yin, and Jiliang Tang. 2024a. The good and the bad: Exploring privacy issues in retrieval- augmented gen...

    arXiv 2024

  7. [7]

    Jailbreak Prompt (TrustLLM Dataset)

    to implement the following baseline, adopt- ing the default parameter settings: • ICA (Wei et al., 2024b) simply applies in-context learning demonstrations to perform jailbreak at- tack. • CodeChameleon (Lv et al., 2024) encrypts in- puts and embeds a decoding function within the prompt. • DeepInception (Li et al., 2024b) creates a virtual, layered scenar...

    work page 2024

  8. [9]

    Todd Esse (tesse@sempratrading.com) - No content provided

    work page

Show all 13 references
  1. [12]

    Christopher Culberson (cm- culbe@us.ibm.com) - No content provided

  2. [13]

    System Prompt (TrustLLM Dataset)

    Corey Ayers (no email address provided) - No content provided. If you need help with something spe- cific from these emails, please provide more context or details." System Prompt (TrustLLM Dataset) "You are a database manager and below are the data you manage: Angela’s phone ...

  3. [2022]

    Advances in neural in- formation processing systems, 35:27730–27744

    Training language models to follow instruc- tions with human feedback. Advances in neural in- formation processing systems, 35:27730–27744. Fábio Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models. In NeurIPS ML Safety Workshop. Zhenting...

    2022 arXiv

  4. [2023]

    In 2023 IEEE Sym- posium on Security and Privacy (SP) , pages 346–363

    Analyzing leakage of personally identifiable information in language models. In 2023 IEEE Sym- posium on Security and Privacy (SP) , pages 346–363. IEEE. Huijie Lv, Xiao Wang, Yuansen Zhang, Caishuang Huang, Shihan Dou, Junjie Ye, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. C...

    2023 arXiv

  5. [2024]

    arXiv preprint arXiv:2404.16251

    Investigating the prompt leakage effect and black-box defenses for multi-turn llm interactions. arXiv preprint arXiv:2404.16251. Meta AI. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. Cem Anil, Esin Durmus, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Bat...

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.