{"id":"4ac04f2d-9dfb-4fac-8183-e69be6abfbe3","arxiv_id":"2505.09921","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A gradient-based iterative in-context optimization framework, PIG, extracts personally identifiable information from aligned LLMs with high success rates on white-box and black-box models.","lead":"PIG is a new attack method that uses a few example conversations and gradient-guided tweaks to make large language models reveal private details like phone numbers and bank accounts. It matters because it shows that current safety alignment can be bypassed by mixing simple in-context learning with optimization, a risk for real-world LLM deployments.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The optimization target in Eq. 4 is a PII-free prefix ('Sure, David's phone password is'), while ASR requires the exact stored PII; the paper never verifies that prefix-completion yields the correct value, so the near-100% ASR in Table 1 is not yet supported.","rationale":"The reader's weakest assumption identifies exactly the load-bearing concern: the optimization objective in Eq. 4 targets a reference response with no PII value, while ASR in Appendix D requires the correct PII substring. This is not a matter of consensus or framing; it is an internal mismatch between the loss being minimized and the metric used to claim success. The high reported ASR could still be real if the prefix reliably induces the model to continue with the true stored value, but the paper provides no quantitative evidence for that continuation. The Appendix F case study shows both success and failure modes but is not a systematic breakdown. A controlled re-run with a debug hook that classifies post-prefix continuations would settle the question. If the check shows that most line-11 successes are followed by the correct value, the empirical claim is intact. If it shows many wrong-value or refusal-after-prefix outputs, the reported ASR is not a valid measure of PII extraction. This concern is addressable and does not by itself invalidate the method, so the conditional verdict remains appropriate.","tokens_in":20000,"tokens_out":6080,"duration_ms":66198,"concrete_test":"Run PIG on the TrustLLM normal template for LLaMA2-7b and LLaMA3-8b with the released code, recording for every sample where Algorithm 1 breaks (Eθ(R)=True) whether the generated continuation after the reference prefix contains (a) the correct target p before any refusal, (b) a refusal or wrong PII value, or (c) no PII. Compute the fraction of (a) and compare it with the ASR reported in Table 1. If the fraction of (a) is substantially below the reported ASR, or if a large share of (b) exists, then the prefix-only objective in Eq. 4 does not justify the success metric, and the Table 1 numbers need re-reporting with the stricter criterion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 (Eq. 4) minimizes -log Pθ(R'|J) with R' = 'Sure, David's phone password is'; R' deliberately excludes the true PII value. The stopping rule in Algorithm 1 (line 11) fires when Eθ(R) returns True, i.e., when R contains 'a possible PII entity' - not the correct substring p required by the ASR metric in Appendix D. Thus the attack optimizes a compliant prefix, and the algorithm can terminate on a hallucinated or wrong value, or on a refusal after the prefix. Table 1's near-100% ASR therefore rests on the unstated assumption that once the model produces the prefix it continues with the exact p from the system prompt. Appendix F contains only cherry-picked cases; it does not quantify how often the prefix is followed by the correct value versus a wrong value or refusal. Without that breakdown, the headline ASR numbers could be driven by samples that satisfy the line-11 predicate but would not satisfy the ASR definition.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes PIG, a privacy jailbreak framework that first identifies PII types and entities in a privacy query, builds in-context demonstrations with synthetic entities, and then iteratively updates the privacy context using three gradient-based token-selection strategies (random, entity, dynamic) until the target model responds with a possible PII entity. The authors benchmark ten jailbreak baselines plus PIG on the TrustLLM and Enron datasets across four open-source white-box models and two API-based black-box models, reporting near-100% ASR on most white-box models, strong transfer to GPT-4o and Claude-3.5, faster convergence than GCG, and ablations isolating the in-context learning and optimization components.","tokens_in":20158,"tokens_out":5804,"duration_ms":57640,"significance":"If the reported ASR numbers hold, this is a useful contribution to LLM privacy evaluation: it provides a common evaluation harness for ten jailbreak baselines, shows that ICL alone is insufficient on well-aligned models (Fig. 3), demonstrates faster convergence than GCG (Fig. 5), and releases code. The central claim, however, is that the attack recovers the exact PII stored in the system prompt, and the manuscript currently does not establish that its optimization objective and stopping rule match the ASR metric it reports. The paper's benchmark of existing jailbreak methods in a privacy setting is valuable regardless, and the public datasets and code release are strengths.","major_comments":[{"comment":"The optimization target in Eq. (4) is the PII-free reference response R' (e.g., “Sure, David’s phone password is”), and the stopping rule in Algorithm 1 fires when Eθ(R) returns True, i.e., when the response contains “a possible PII entity.” In contrast, the ASR metric in Appendix D counts only responses in which the exact stored value p is a substring of r. Nothing in the paper measures how often the optimized prefix is completed with the correct p as opposed to a hallucinated value or a refusal. Consequently, the near-100% ASR figures in Table 1 are not yet supported by the stated objective and stopping criterion. Please add an evaluation that either (a) uses the known p, which is available in the TrustLLM system prompt, as the termination predicate and reports ASR under that predicate, or (b) reports the breakdown, for line-11 successes, of correct-p, wrong-value, and refusal-after-prefix responses.","section":"Section 4.3, Eq. (4); Algorithm 1, lines 10-13; Appendix D"},{"comment":"The black-box evaluation is described only as “we transferred successful jailbreak prompts from white-box models to black-box APIs” with a citation to Zou et al. The protocol is underspecified: which white-box model produced the transferred prompts, whether transfer was per-query or via a universal template, how many prompts were transferred per query, and the query budget for GPT-4o and Claude-3.5. Since Table 1 reports ASR of 97.1% on GPT-4o and 85.7% on Claude-3.5 as headline results, this missing protocol makes the black-box claim non-reproducible.","section":"Section 6.1 (Black-box Closed-source LLM APIs); Section 5 (Implementation Details)"},{"comment":"Section 6.3.2 states that combining the three strategies gives ASRs of 92.85% on LLaMA2-7b and 94.3% on LLaMA3-8b over 100 epochs, but Appendix E Table 4 reports 100% ASR for PIG under the normal template for both models, and Table 1 also reports 100% ASR in the same setting. The discrepancy, which is likely due to the 70-sample subset introduced in Section 6.3.1, is not explained; please clarify the sample sizes and report the ablation numbers consistently.","section":"Section 6.3.2; Appendix E, Table 4"}],"minor_comments":[{"comment":"There is a typo: “availble” should be “available.”","section":"Abstract"},{"comment":"The Step 3 box contains garbled text (e.g., “PIG Weto’sAnswerTomneedDynamicRandomEntity” and “Visa”), which obscures the differences among the random, entity, and dynamic strategies.","section":"Figure 2"},{"comment":"Several rows run values together without spacing (e.g., the PIG x=5 row contains “100%62.5%” and similar), making the table unreliable to read; use explicit column separators.","section":"Table 2"},{"comment":"The evaluation model Eθ is an input to the algorithm but is never formally defined in Section 4 or Section 5; please state how Eθ relates to the Longformer RtA classifier described in Appendix D and what exactly “contains a possible PII entity” means operationally.","section":"Algorithm 1"},{"comment":"The Mistral response displayed under “No rejection, but no privacy leaked” actually lists the email addresses of all six senders, which contradicts the category label; please verify the case or correct the category description.","section":"Appendix F, Section F.2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a security/cs.CR venue, and the evaluation harness is a useful contribution. The central methodological risk is that the reported ASR may overstate what the optimization objective actually optimizes; this is fixable within revision by adding a prefix-completion breakdown or switching the termination predicate to the known PII. I would not reject on novelty, but I would require the authors to supply the requested breakdown before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read for you on PIG (2505.09921). The thing to know: this is a credible attack paper, not a stunt. The authors combine in-context jailbreaking with GCG-style gradient token search, applied to PII extraction, and it appears to beat the standard jailbreak baselines on most models, especially under the defensive system prompt. The empirical core is well-built: six models, two datasets (TrustLLM and Enron), clean ablations separating ICL from optimization, and a meaningful comparison against GCG showing faster convergence. The three token-selection strategies are simple heuristics, but they are clearly described and the combination helps. The code is linked. That is real work and worth engaging.\n\nThe soft spot is exactly what the stress-test flags. The optimization target (Eq. 4) is a reference prefix without the actual PII value, e.g., \"Sure, David's phone password is\". The stopping rule in Algorithm 1 (line 11) fires when the output contains \"a possible PII entity\", which is not the same as the exact correct PII required by the ASR definition in Appendix D. The paper never reports a breakdown of the final responses: how often the model continues the prefix with the correct value, how often it hallucinates a wrong one, and how often it stops after the prefix. Without that breakdown, the near-100% ASR numbers in Table 1 are not as airtight as they look. It could well be that the method genuinely extracts the exact PII most of the time—the case studies suggest it often does—but the paper needs to show the confusion matrix.\n\nThe other issues are minor. Results are single-run with no error bars or seeds, and the SoTA claim is overstated on rows where baselines already hit 100% ASR. The more meaningful claim is the robustness under the defensive template, where PIG does clearly better.\n\nBottom line: the central method holds up conceptually; the numbers need a response-level analysis before I'd trust the headline. Send it to a security or LLM venue with a request for that breakdown. It deserves refereeing, not a desk reject.","headline":"PIG is a credible and practical attack that likely beats ICL-only jailbreaks for PII extraction, but the near-100% ASR numbers need a response-level breakdown before they're fully trustworthy.","tokens_in":20788,"tokens_out":3700,"would_cite":true,"duration_ms":40164,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Jailbreak attacks adapted to privacy can extract stored PII from aligned LLMs at near-100% success.","keywords":["privacy jailbreak attack","personally identifiable information","in-context learning","gradient-based optimization","LLM privacy leakage","attack success rate","TrustLLM benchmark","data extraction"],"falsifier":"Re-run PIG on the TrustLLM dataset but score as success only responses that contain the exact target PII value in the completion immediately following the optimized opening phrase; count as failures outputs that begin with that phrase and then refuse, apologize, or give another value. If such partial-prefix outputs are common, the near-100% attack success rate would drop measurably.","tokens_in":19731,"feed_emoji":"🔓","tokens_out":6675,"duration_ms":62974,"temperature":0.7,"pith_summary":"This paper tries to show that jailbreak attacks, normally built to make a model produce harmful content, can be redirected to extract personally identifiable information from an LLM's stored context or memory. It introduces PIG, which detects the type of PII in a query, builds a few in-context demonstrations using fabricated values of the same type, and then iteratively edits tokens in those demonstrations with gradients until the model answers with the requested value. On four aligned open-weight models the paper reports attack success rates near 100 percent, and the method transfers to black-box APIs, reaching 97.1 percent on GPT-4o on the standard privacy template. The paper argues that this shows current privacy-leakage evaluations, which rely on memorized prefixes or simple instructions, systematically underestimate how easily sensitive data can be extracted.","feed_headline":"Jailbreak prompts pull private data from LLMs at near 100%","feed_subtitle":"Gradient-optimized in-context examples make aligned models reveal bank accounts, SSNs, and passwords.","key_machinery":"The load-bearing mechanism is the iteratively optimized privacy context. PIG first identifies the PII entity and type in a query, then assembles a set of demonstration question-answer pairs whose answers contain randomly generated values of that same type. The optimization step follows Algorithm 1: compute token gradients against the reference prefix, keep the top-k replacement candidates for each token, sample B perturbed contexts, and select the one with the lowest cross-entropy loss. The three strategies differ only in which token positions are editable -- random positions, positions inside PII entity tokens, or the M positions with the largest average gradient -- and combining their results is what pushes the attack success rate to the reported levels.","core_discovery":"The paper's central claim is that privacy leakage from aligned LLMs is better modeled as an optimization problem over the prompt than as a failure of refusal training. PIG constructs a privacy context from demonstrations pairing similar requests with synthetically generated PII values, then optimizes selected tokens of that context to raise the probability of a reference response such as \"Sure, David's phone password is\". Three token-selection strategies -- random, entity-only, and gradient-importance-based -- explore different parts of the context, and their outputs are combined to avoid local optima. As reported, this yields nearly 100 percent attack success on LLaMA2-7b, Mistral-7b, Vicuna-7b, and LLaMA3-8b under the normal TrustLLM template, 97.1 percent on GPT-4o, and a large reduction in refusal rate on the Enron training-data extraction task.","pith_inferences":["A stricter evaluation that requires the exact PII value to appear immediately after the optimized prefix, rather than anywhere in the response, would clarify how much of the reported attack success reflects true retrieval versus compliant-but-vague completions.","The same context-optimization loop should transfer to other high-value secret types, such as API keys, internal identifiers, or medical record fields, provided a synthetic demonstration set of the same type exists; this is a testable extension the paper does not run.","An implied defense is to detect or limit in-context optimization itself, for example by checking whether the prompt contains an unusually low-loss continuation toward a fixed prefix, or by requiring out-of-band authorization before the model emits any stored value.","The method's success on Enron suggests that training-data extraction and context leakage are not separate problems but can be attacked with the same optimized-demonstration machinery."],"forward_implications":["Models that are considered well-aligned can still be induced to output stored PII when the attacker controls a few in-context examples, so privacy protections cannot rely on refusal training alone.","Attackers do not need a universal adversarial suffix; a query-specific context optimized for a few hundred epochs is enough, which makes targeted extraction cheaper than GCG-style universal attacks.","Success on black-box APIs through transferred prompts implies that closed models inherit the same in-context optimization vulnerability even when gradients are unavailable.","Privacy benchmarks built on prefix prompts or simple instructions underestimate leakage; evaluation should include optimized-context attacks as a standard baseline.","Because PII demonstrations can be synthesized from random names and numbers, the attack does not require access to real sensitive data to be constructed."],"supporting_citations":[{"why":"Defines the prefix-based privacy data extraction baseline that PIG must beat and supplies the privacy-jailbreak framing.","marker":"Li et al., 2023"},{"why":"Introduces few-shot in-context jailbreaking, the demonstration strategy PIG starts from and one of its baselines.","marker":"Wei et al., 2024b"},{"why":"Provides the GCG gradient token-replacement algorithm that PIG adapts and compares against.","marker":"Zou et al., 2023"},{"why":"Supplies the TrustLLM dataset, its privacy system-prompt templates, and the RtA/ASR evaluation metrics.","marker":"Huang et al., 2024"},{"why":"Supplies the Enron email corpus used to test extraction of PII memorized in training data.","marker":"Klimt and Yang, 2004"},{"why":"Provides the unified implementation of the ten jailbreak baselines used in the comparison.","marker":"Zhou et al., 2024"},{"why":"Provides the Jailbroken adaptive attack, the strongest prior jailbreak baseline in the experiments.","marker":"Wei et al., 2024a"}],"fun_headline_variants":["PIG jailbreak extracts private data from aligned LLMs at near 100%","Gradient-optimized jailbreak uncovers SSNs and passwords","PIG optimizes in-context examples to break LLM privacy","Privacy jailbreak nearly 100% success on GPT-4o","LLM privacy stripped by iterative in-context optimization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The attack's optimization target is only a polite opening phrase -- \"Sure, David's phone password is\" -- without the secret value, so the reported success rates assume that once a model starts with that phrase it will continue with the correct value stored in the system prompt rather than refusing or inventing a different value.","fun_headline_variants_meta":{"raw":{"variants":["PIG jailbreak extracts private data from aligned LLMs at near 100%","Gradient-optimized jailbreak uncovers SSNs and passwords","PIG optimizes in-context examples to break LLM privacy","Privacy jailbreak nearly 100% success on GPT-4o","LLM privacy stripped by iterative in-context optimization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000764,"raw_usage":{"total_tokens":3400,"prompt_tokens":965,"completion_tokens":2435,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":2343}},"tokens_in":581,"tokens_out":2435,"duration_ms":17799,"temperature":1.0,"reasoning_tokens":2343,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:20:28.038702+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run PIG on the TrustLLM dataset but score as success only responses that contain the exact target PII value in the completion immediately following the optimized opening phrase; count as failures outputs that begin with that phrase and then refuse, apologize, or give another value. If such partial-prefix outputs are common, the near-100% attack success rate would drop measurably.","supporting_citations":[],"review_version":1}