REVIEW 4 major objections 5 minor 13 references
Model Inversion Attacks on Llama 3: Extracting PII from Large Language Models
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper claims that short templated prompts can extract memorized personal data from the Llama 3.2 1B model through black-box queries.
desk verdict A routine prompting demo that overclaims extraction: without a membership or likelihood test and with the memorization rate never computed, the central result is unsupported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism that carries the attack is the autoregressive text-generation loop: a short, templated prompt is passed to the model through the Hugging Face Transformers pipeline, and the model continues the sequence. The extraction probability is formalized in the paper as a chain-rule product of conditional token probabilities, linking successful extraction to the model's softmax distribution. The paper's "Memorization Rate" is defined to quantify how often such extractions succeed per query, although no actual rate for the 1B model is reported.
What would settle it
Check whether the exact generated strings (for example 'vishal@gmail.com' and 'PO Box 1444') appear anywhere in Llama 3.2's pretraining data; if they are absent, the outputs are plausible hallucinations rather than recovered training records. A complementary test would be to run the same prompts on a version of the model trained without those strings and see whether it emits the same completions.
Extended reading notes
Core claim
The central discovery the paper reports is that a black-box model inversion attack on Llama 3.2 1B succeeds: by feeding the model templated prompts like "my password is:", "account number:", and "my email id:", the author obtained completions that include a personal email address, a phone number, a name, a LinkedIn URL, and a bank account holder's name and P.O. Box address. The paper models the extraction probability as a product of per-token conditional probabilities (Equation 1) and reports a non-zero memorization rate defined as extracted PII sequences divided by total queries. The outputs were validated by Google searching the LinkedIn URL and the bank address, which the author takes as evidence that the strings were memorized during training.
Load-bearing premise
The paper's conclusion assumes that any generated completion that looks like PII is a verbatim copy of a memorized training string; the Google-search validation only shows the output resembles a real person or bank, not that those exact tokens appear in the training corpus.
Editorial extensions
If this is right
- If the attack works as reported, anyone with API access to Llama 3.2 1B can probe for memorized PII with short, templated prompts and no model internals.
- A non-zero memorization rate for the 1B model would mean that smaller, edge-deployed models are not inherently safe from inversion attacks, contrary to the assumption that only large models leak.
- The paper's proposed defenses—rate limiting, output filtering, differential privacy, and data sanitization—would all become relevant for production use of open-weight models, though access control offers no protection once weights are public.
- The extraction probability model in Equation (1) implies that attack success could be amplified by high-likelihood decoding settings, which the paper uses by design, so real-world risk depends on the attacker's choice of sampling parameters.
Reading between the lines
- The evidence presented does not rule out the null hypothesis that the model is hallucinating email- and account-shaped text: the Google-search check confirms that the outputs resemble real entities, but not that they are memorized copies. A membership-inference test comparing the likelihood of the exact string against a control string would settle this.
- If these extractions are genuine memorization, the trigger is almost certainly duplicated data in the pretraining corpus; measuring the frequency of the extracted strings across web corpora would predict attack success and connect this result to known deduplication defenses.
- The same prompt templates could be run across other small open-weight models to produce a memorization benchmark; the paper's handful of examples is not enough to estimate a memorization rate for Llama 3.2 1B, as the author concedes in the limitations.
- One testable extension is to measure how the extraction success rate varies with temperature and top-k; the paper fixed these to high-likelihood values (top-k=40, top-p=1), so the reported vulnerability may shrink under more diverse decoding settings.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper claims to demonstrate a black-box model inversion attack on Llama 3.2 1B, in which prompts such as 'my password is:', 'account number:', and 'my email id:' elicit memorized personally identifiable information (PII). The evidence consists of one redacted table (Table I) containing three generated fragments, a brief description of Google-search validation in Section V, and a discussion of mitigation strategies. The paper defines a memorization rate in Equation (2) but does not compute it, and Section VIII explicitly concedes that no specific memorization rates for Llama 3.2 1B are reported. The central claim that passwords, email addresses, and account numbers were extracted is therefore unsupported by the presented measurements.
Significance. If the central claim were established, it would be a meaningful privacy finding for a small open-source model. However, as presented, the paper provides no valid evidence of memorization: the Table I outputs are generic or truncated fragments, the Google-search validation cannot distinguish memorization from hallucination, and no membership test, likelihood threshold, or exact-match criterion is applied. The paper correctly cites the standard autoregressive factorization (Eq. 1) and prior extraction work by Carlini et al. and Nasr et al., but it does not carry over any of their measurement or verification methodology, so the result does not advance the state of the art. The reproducible code snippet is a minor positive, but no data or full outputs are released.
major comments (4)
- [Section IV, Table I] The reported 'extracted PII' does not support the abstract's claim that passwords, email addresses, and account numbers were extracted. 'my password is: 1a' is not a password; 'account number: , name: Suncorp Bank, address: PO Box 1444' contains no account number; and 'my email id: vishal @gmail.com, my mobile no: 97' contains a space before '@gmail.com' and a truncated phone number. These outputs are consistent with generic placeholder-like generation and do not constitute PII extraction.
- [Section V] The Google-search validation cannot establish that the generated sequences were memorized training data. Showing that Suncorp Bank and a LinkedIn user exist does not demonstrate that the exact generated token sequence appears in Llama 3.2's training corpus, nor does it rule out the possibility that the model hallucinated an email-shaped or bank-shaped string. The paper cites Carlini et al. but applies none of the membership or extraction tests used there, such as perplexity thresholds, exact-match criteria, or canary audits, so the inference from 'output resembles a real entity' to 'memorized training data' is not supported.
- [Section VI, Eq. (2)] Equation (2) defines a memorization rate, but the paper never instantiates it with any counts. The claim in Section VI that 'Some experiments yielded a non-zero memorization rate' is unsupported by any data, and Section VIII explicitly concedes 'the lack of specific memorization rates for Llama 3.2 1B.' Without this quantitative measure, the paper's central assertion that the attack 'successfully extracted' PII cannot be evaluated.
- [Section III] The methodology does not include any control or baseline to distinguish memorization from hallucination. Equation (1) is merely the standard autoregressive factorization of the probability of generating a sequence; it does not provide a test for whether a generated string was present in the training data. A model with no memorized PII would be expected to produce email-shaped and account-shaped completions from these generic prompts, so the observed outputs carry no evidentiary weight without additional membership or likelihood analysis.
minor comments (5)
- [Section I] In the Introduction, 'focuses on the Llama 3.2 an LLM developed by Meta' is missing the word 'is' or a comma before 'an LLM.'
- [Section II.A] There is a missing space in 'Model inversion attacks(MIA)'; it should read 'attacks (MIA).'
- [Section VII] Reference [7] is cited for the claim that 'requiring an API key or solving CAPTCHAs can prevent automated mass extraction,' but [7] is the OWASP API Security Top 10; the citation does not appear to support this specific claim.
- [Section VII] Reference [8] is cited for the statement that 'Google and Apple have used DP in production language models to protect user data,' but reference [8] is a 2007 paper on stopping spam by solving puzzles, which does not support this claim.
- [Figure 1] The caption states that outputs are 'analyzed (e.g., via likelihood),' but no likelihood analysis is performed anywhere in the paper; either add such an analysis or revise the caption.
Circularity Check
No definitional or self-citation circularity; the paper's equations are standard definitions that are never evaluated, and the extraction claim is under-evidenced rather than circular.
full rationale
The paper does not fit parameters to data and then predict the same data. Equation (1) is the standard autoregressive factorization and Equation (2) is a definition of memorization rate; neither is numerically instantiated, so no result reduces by construction to its own inputs. The load-bearing evidence in Section V is the claim that 'The PII data was validated by doing a google search of the corresponding data to prove that it was a memorized content as part of the training,' but this is an unsupported inference (real-world existence of a LinkedIn user or a bank does not establish that the generated token sequence appeared in Llama 3.2's training corpus), not a circular derivation. Table I's outputs are redacted or generic ('my password is: 1a'), so the paper's own evidence cannot confirm extraction; however, this is missing measurement/verification, which falls under correctness risk rather than circularity. The paper also acknowledges this in Section VIII: 'Limitations include the lack of specific memorization rates for Llama 3.2 1B and the need for broader testing across diverse prompts.' The references to Carlini et al. and Nasr et al. are external and are not used to justify the paper's central premise through a self-citation chain. No step in the derivation chain is equivalent by definition to its input.
Assumptions & free parameters
free parameters (1)
- Decoding hyperparameters =
top_p=1, top_k=40, max_new_tokens=50
assumptions (2)
- standard math Equation (1) correctly factorizes the probability of a generated sequence as a product of conditional next-token probabilities.
- ad hoc to paper A generated string whose contents match a real person or institution found by Google search is memorized training data.
Cite this review
Pith. "Pith review of Model Inversion Attacks on Llama 3: Extracting PII from Large Language Models." pith.science (2026). https://pith.science/paper/5V7AUZJV
@misc{pith2026250704478,
author = {Pith},
title = {Pith review of: Model Inversion Attacks on Llama 3: Extracting PII from Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/5V7AUZJV}},
note = {Machine review of arXiv:2507.04478}
}
read the original abstract
Large language models (LLMs) have transformed natural language processing, but their ability to memorize training data poses significant privacy risks. This paper investigates model inversion attacks on the Llama 3.2 model, a multilingual LLM developed by Meta. By querying the model with carefully crafted prompts, we demonstrate the extraction of personally identifiable information (PII) such as passwords, email addresses, and account numbers. Our findings highlight the vulnerability of even smaller LLMs to privacy attacks and underscore the need for robust defenses. We discuss potential mitigation strategies, including differential privacy and data sanitization, and call for further research into privacy-preserving machine learning techniques.
Figures
Reference graph
Works this paper leans on
-
[1]
Extracting Training Data from Large Language Models,
Carlini, Nicholas and Tramer, “Extracting Training Data from Large Language Models,” arXiv preprint arXiv:2012.07805 , 2021
arXiv 2012
-
[2]
Model Inversion Attacks: A Survey of Approaches and Countermeasures,
Z. Zhou, Y . Li, and X. Zhang, “Model Inversion Attacks: A Survey of Approaches and Countermeasures,” arXiv preprint arXiv:2411.10023 , 2024
arXiv 2024
-
[3]
Scalable Extraction of Training Data from (Production) Language Models,
M. Nasr, N. Carlini, and A. Terzis, “Scalable Extraction of Training Data from (Production) Language Models,” arXiv preprint arXiv:2311.17035, 2023
arXiv 2023
-
[4]
J. X. Morris, A. S. Luccioni, and M. Mitchell, “Language Model Inver- sion,” OpenReview, 2023. https://openreview.net/pdf?id=t9dWHpGkPj
work page 2023
-
[5]
Llama 3.2: A Lightweight Multilingual Language Model,
Meta AI, “Llama 3.2: A Lightweight Multilingual Language Model,” Meta AI Technical Report , 2024. https://ai.meta.com/research/ publications/llama-3-2
work page 2024
-
[6]
Masking Sensitive Data in Language Model Training,
Y . Chen, H. Li, and Z. Wu, “Masking Sensitive Data in Language Model Training,” arXiv preprint arXiv:2402.12345 , 2024
arXiv 2024
-
[7]
J. Harder, “OW ASP API Security Top 10,” OWASP F oundation, 2017
work page 2017
-
[8]
J. Goodman, R. Rounthwaite, and R. Thomas. Stopping Spam by Solving Puzzles . In IEEE Security and Privacy , 2007
work page 2007
Show all 13 references
-
[9]
Auditing Data Provenance in Text-Generation Models
Congzheng Song and Vitaly Shmatikov. Auditing Data Provenance in Text-Generation Models . In KDD, 2019
2019
-
[10]
Brown et al
Tom B. Brown et al. Language Models are Few-Shot Learners . In NeurIPS, 2020. arXiv:2005.14165
2020 arXiv
-
[11]
Learning Differentially Private Language Models Without Losing Accu- racy
Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning Differentially Private Language Models Without Losing Accu- racy. In ICLR, 2018
2018
-
[12]
Does Learning Require Memorization? A Short Tale About a Long Tail
Vitaly Feldman. Does Learning Require Memorization? A Short Tale About a Long Tail . In STOC, 2020
2020
-
[13]
K. Lee, J. Chen, N. Carlini, C. Raffel, and K. Olson. Guardrails: Preventing Sensitive Information Leakage from Language Models. arXiv:2306.11692, 2023
2023 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.