Pith. sign in

REVIEW 4 major objections 5 minor 23 references

Sampling from a large language model reliably reproduces exact personal identifiers from training documents for a subset of individuals, while standard aggregate membership scores detect nothing.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 01:08 UTC pith:5UXZVVEI

load-bearing objection Solid aggregate negative result; the per-document PII extraction headline rests on a control that is too weak to carry it. the 4 major comments →

arxiv 2608.00144 v1 pith:5UXZVVEI submitted 2026-07-31 cs.LG cs.CLcs.CR

Leak It: A Probabilistic Approach to Training-Data Extraction from Black-Box Language Models

Victor Maricato This is my paper
classification cs.LG cs.CLcs.CR
keywords training-data extractionmembership inferencesampling-based attackprivacy auditpersonally identifying informationlanguage model memorizationper-document leakageblack-box attack
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the standard way of measuring whether a language model leaks its training data—aggregate membership-inference accuracy—is the wrong lens. It shows that sampling-based statistics do not beat a model-free baseline at telling members from non-members on a balanced split, yet the same sampling reproduces exact email addresses and phone numbers from training documents for a subset of individual documents. On a 6.9-billion-parameter open model, 16.6% of 500 documents bearing a real identifier had that exact identifier reproduced and not reproduced under a mismatched-prefix control; for email addresses the rate is 21.3%. Leakage grows with model size, is about three times stronger in code than in prose, and is not detectably reduced by deduplicating the training corpus. The paper concludes that privacy audits should report per-document extraction rates, decomposed by domain, rather than a single AUC.

Core claim

The central discovery is that verbatim extraction of training content—including personally identifying information—is a per-document tail phenomenon that aggregate metrics cannot see. Treating N samples from the model's output distribution as an empirical estimate, the paper uses an extreme-value functional (the maximum overlap between any sample and the held-out continuation) plus a mismatched-prefix control to attribute each leak to a specific document. On the Pythia-6.9B model, 83 of 500 Pile documents bearing a real email or phone number (16.6%; 21.3% for emails) had that exact identifier reproduced and not reproduced under a mismatched prefix. The rate rises from 5.6% at 410M parameters

What carries the argument

The load-bearing object is the extreme-value functional of the empirical sampling distribution: draw N continuations from the model given a prefix and take the maximum text-overlap (specifically, max 5-gram recall) between any sample and the held-out continuation. This functional preserves the rare tail of near-verbatim continuations that a mean-based statistic averages away. The per-document attribution is carried by a mismatched-prefix control: a document counts as leaked only if its identifier is reproduced under the true prefix and not reproduced under an unrelated prefix, discounting globally common strings. Temperature and nucleus sampling barely affect the result, and a 16-token prefi

Load-bearing premise

The mismatched-prefix control is assumed to fully separate document-specific leakage from globally common strings; if a model emits a document-specific identifier in unrelated contexts for other reasons—such as a strong corpus co-occurrence or a general propensity—the conjunction criterion would either misattribute a leak to the document or miss a true leak.

What would settle it

A concrete falsifier: take the 83 documents the paper counts as leaked and query each with 50 random, unrelated prefixes. If a large fraction (say, more than half) reproduce the same exact identifier under many of those random prefixes, then the mismatched-prefix control has not isolated document-specific leakage, and the reported 16.6% overstates per-document extraction. Alternatively, run the identical 500-document probe on a model trained on a disjoint corpus: if the conjunction rate does not fall to near zero, the attribution criterion is not specific.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Privacy audits should report per-document extraction rates and their domain decomposition, not a single aggregate AUC.
  • Differential privacy is the only mitigation with a formal guarantee; training-time deduplication at the tested threshold does not measurably reduce identifier extraction.
  • Inference-time sampling controls (temperature, nucleus truncation) are not a reliable defense; verbatim extraction is detectable at the string level, so filtering against known-sensitive lists can work.
  • An adversary with partial-record access—just text before an identifier—can trigger extraction with a 16-token prefix and a modest sample budget.
  • Extraction risk is uneven across domains; code-heavy corpora present a roughly threefold higher per-document disclosure rate.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The per-document tail phenomenon likely extends beyond identifiers to any high-entropy or uniquely formatted string; the paper's method could be applied to measure it, but the paper does not do so.
  • Because the mismatched-prefix control is conservative (it discounts any reproduction under an unrelated prefix), the true per-document leak rate may be if a model emits a document-specific identifier in other contexts for non-memorisation reasons; that would undercount, not overcount.
  • The strong code/prose split suggests models trained predominantly on code repositories will have higher per-document extraction risk, which is testable by running the same probe on a code-focused model.
  • If deduplication is intended as a privacy mitigation, the null result here implies that document-level near-deduplication is insufficient; stronger, sub-document-level or semantically aware deduplication should be tested, but the paper leaves that open.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies black-box, sampling-based training-data extraction from language models. It frames sampling-based leakage statistics as functionals of the empirical output distribution, then applies the blind-baseline discipline of Das et al. (2025) to the sampling regime. On WikiMIA a blind bag-of-words classifier achieves AUC 0.97 and sampling adds nothing; on the IID MIMIR Pile split, no sampling functional significantly beats the residual blind baseline, with incremental AUC confidence intervals including zero. The paper's positive claim is that the same sampling procedure verbatim-extracts a tail of training data that blind attacks cannot reach: on Pythia-6.9B, 83 of 500 Pile documents bearing an identifier (16.6%; 21.3% of email-bearing documents) have that exact identifier reproduced and not reproduced under a mismatched-prefix control, which the authors interpret as document-specific leakage. They further report capacity scaling, stronger leakage in code than prose, recovery of arbitrary held-out continuations concentrated in GitHub code, regime sweeps over temperature/top-p/prefix length/sample budget, and no detectable reduction from deduplication. The paper releases the leakit audit tool.

Significance. If the central extraction claim holds, the paper makes a valuable contribution to the privacy-audit literature: it demonstrates that aggregate MIA AUC is the wrong lens, that per-document verbatim extraction is the concrete harm, and that this harm is concentrated in a tail and varies sharply by domain. The methodology is unusually careful in several respects: the blind-baseline discipline is applied consistently; all leakage claims use the IID Pile train/test split rather than temporally shifted benchmarks; the paper reports Wilson intervals, cluster-bootstrap confidence intervals, and paired McNemar tests; and the tool and analysis scripts are released. These strengths are real and make the paper a useful template for future audits. However, the per-document attribution claim rests on a mismatched-prefix control that is statistically too weak to establish that an identifier is 'not globally common' and therefore 'attributable to that document.' Since the headline 83-document count and the capacity/domain decompositions built on it depend on this control, the central positive claim needs additional support before the paper can be accepted.

major comments (4)
  1. [Sec. 5.1, 'context control' paragraph] The per-document attribution uses a single mismatched-prefix control draw of N=32 continuations. An identifier that is in fact globally common, with per-continuation emission probability p in unrelated contexts, will pass the control (no reproduction) with probability (1-p)^32. For p=0.01 this is 0.72; for p=0.03 it is 0.38. The conjunction therefore does not establish that the identifier is 'not a globally common string.' The Wilson intervals on 83/500 quantify sampling variation over documents given the criterion, but not the per-document false-attribution rate. Please estimate the control emission probability with multiple independent control prefixes or draws per identifier, or otherwise bound the false-positive rate; without this, the headline claim that each leak is attributable to that document is not supported.
  2. [Sec. 5.2, code/prose decomposition] The mismatched-prefix construction is not specified. If the control prefix is drawn from a different domain than the document, an identifier that is common within code or within the same organizational context but not in the control domain will be systematically misclassified as document-specific. This directly affects the claim that identifier leakage is ~3x stronger in code than prose (35.8% vs 12.1% at 6.9B) and the prose rates in Figure 3a. Please specify how control prefixes are constructed and, ideally, match control prefixes by domain or report domain-matched control rates.
  3. [Sec. 5.3 / Fig. 4b] The arbitrary-continuation recovery result is reported per domain as gaps of +0.44 on GitHub and at most +0.014 on prose domains, but no confidence intervals are given for these domain-level gaps, unlike the identifier-extraction results in Sec. 5.2. Since the 'recovery is a code phenomenon' conclusion is a central part of the paper's risk decomposition, please report cluster-bootstrap CIs or a full per-domain table for the recovery gaps.
  4. [Sec. 6, Benchmark construction / Sec. 4] The aggregate negative result in Sec. 4.2 is based on a Pythia-2.8B MIMIR cell that is only at 77% coverage (773/1000 documents). The paper states that the reached source slices are balanced and quantities fall within the partial-run CI, but no coverage-by-domain table or CI for the partial run is provided. Since the aggregate null result is a central contribution, please report the coverage per domain and the relevant CIs in the main text or appendix.
minor comments (5)
  1. [Abstract / Sec. 5.1] The abstract says '83 of 500 Pile documents bearing a real identifier (16.6%; 21.3% of those bearing an email address)'; the 500-document set includes both emails and telephone numbers, so 'bearing a real identifier' is slightly misleading. Please make the composition explicit in the abstract.
  2. [Fig. 3 caption] The caption states that the outline 'adds the reproductions the context control discounts as globally common.' This is not immediately clear; please define whether the outline is raw reproduction or raw-minus-solid, and relate it to the control rate reported in the text.
  3. [Sec. 3.2] The 'parameter-free sself-word' uses word-token self-concentration, but the tokenization is not specified. Clarify whether this is whitespace tokens, subword tokens, or something else.
  4. [Sec. 5.2] The McNemar test is reported on raw reproduction counts, not on the document-specific conjunction counts. The text says 'both extract 77, standard-only 9, deduplicated-only 13,' which are raw counts. This is fine, but please state explicitly that the test is on raw reproduction and note whether the document-specific criterion gives the same conclusion.
  5. [Sec. 4.1] The blind bag-of-words classifier is described as 5-fold cross-validation, but the paper does not say whether the same folds are used across models and attacks. Please state this to ensure comparability.

Circularity Check

0 steps flagged

No significant circularity: the paper's claims are empirical measurements with an explicit—if imperfect—control, not quantities derived from their own definitions.

full rationale

The claimed derivation chain is not circular. Section 3 introduces a functional-estimation vocabulary, but the paper does not derive its empirical results from that vocabulary alone; Section 4 measures sampling statistics against public benchmarks and a blind baseline it reproduces from Das et al. (2025), and Section 5 counts exact verbatim reproductions of identifiers under a stated conjunction rule. The 83/500 figure is an operational count, not a fitted parameter renamed as a prediction: no parameter is fitted to the extraction data and then 'predicted' back. The scale-ladder and code/prose results are decompositions of the same measured events, and the deduplication comparison is a paired McNemar test. All load-bearing citations (Das et al., Duan et al., Kaneko et al., Carlini et al.) are external; there are no self-citations carrying the argument. The paper's own limitations (Sec. 6) concern coverage, seed-level variability, and residual distribution differences—validity threats, not identity between inputs and conclusions. The mismatched-prefix control could be statistically underpowered for ruling out globally emitted strings, but that is an inference-strength concern, not a circularity: the conclusion is not the criterion by construction, because the criterion is an evidence rule the paper could in principle have failed.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

No physically new entities are introduced. The load-bearing axioms are benchmark soundness and the sampling i.i.d. model. The main free parameters are experimental configurations and the tail threshold, all swept or acknowledged. The positive extraction result does not depend on fitting any model parameter.

free parameters (3)
  • 5-gram overlap threshold for recovery tail (>=50%) = 0.50
    The 'near-extraction tail (a sample recovering >=50% of y)' threshold is chosen by the authors; it is descriptive, not fitted to optimize a claim, but the specific cutoff affects the 18% vs 4% tail statement.
  • Prefix length k=64, target l=64, N=32, T=1.0, top-p=1.0 = 64/64/32/1.0/1.0
    Default encoding settings chosen by the authors. Sweeps show minor dependence for sampling regime, but 16-token works, so no single default is load-bearing.
  • Email/phone regex filter = not specified
    The regular-expression filter over the Pile stream determines which documents are probes; exact regex is not included in main text, and a different filter would shift the 500-doc sample and rates.
axioms (4)
  • domain assumption MIMIR's Pile train/test split is IID (the official split is identical-distribution by construction)
    Used in Sec 4 to claim blind baselines are honest. The paper tests a temporal probe (AUC 0.556) but does not prove full IID; MIMIR construction is trusted from Duan et al. (2024).
  • domain assumption Pythia-2.8B-dedup trains on the deduplicated Pile with ~1.5 passes
    Cites Biderman et al. (2023) for the dedup suite and acknowledges the epoch confound. The dedup conclusion is explicitly qualified as under-powered.
  • domain assumption Sampled completions C1..CN ~ p_theta(·|x) are i.i.d. draws from the conditional distribution
    The functional framework treats samples as i.i.d.; in practice decoding is sequential and batched, and the authors note GPU nondeterminism only in bootstraps, not i.i.d.-ness.
  • domain assumption Bounding samples to 100-200 words in MIMIR controls length / temporal confounds (from MIMIR)
    The IID claim relies on MIMIR construction; if the official split still differs in surface text, the residual 0.646 blind bag-of-words AUC suggests some imbalance that makes the 'no sampling adds signal' conclusion harder to interpret.

pith-pipeline@v1.3.0-alltime-deepseek · 13288 in / 7286 out tokens · 63433 ms · 2026-08-04T01:08:23.503536+00:00 · methodology

0 comments
read the original abstract

Membership inference (MIA) on language models is usually summarised by an aggregate ROC-AUC, but such evaluations are confounded: model-free blind baselines separate members from non-members from surface text alone. We study black-box, sampling-based training-data leakage through a probabilistic lens, treating N samples from p(.|x) as an estimate of the output distribution and casting leakage signals as functionals of it. We extend the blind-baseline critique into the sampling regime: on WikiMIA a blind bag-of-words classifier reaches AUC 0.97 (TPR 0.90 at 5% FPR) and sampling adds nothing, while on an IID Pile split (MIMIR) neither self-concentration nor gold-continuation recovery significantly beats a blind baseline (incremental AUC 95% CI includes zero). Aggregate metrics hide the real harm. The same sampling verbatim-extracts training data for a tail of documents no blind attack can reach. On Pythia-6.9B, 83 of 500 Pile documents bearing a real identifier (16.6%; 21.3% of those bearing an email address) have that exact identifier reproduced AND not reproduced under a mismatched-prefix control, so each leak is attributable to that document, not to a globally common string. This per-document disclosure is invisible to aggregate AUC and grows with capacity (5.6% to 16.6% from 410M to 6.9B). The risk is uneven: identifier leakage is ~3x stronger in code than prose, though prose stays clearly positive and also grows with capacity (4.0% to 12.1%), while recovery of arbitrary held-out continuations is confined to code (+0.44 member gap on GitHub vs at most +0.014 on prose). Temperature and nucleus sampling matter little, a 16-token prefix suffices, and we detect no reduction from corpus deduplication. Privacy audits should report per-document extraction, decomposed by domain, not a single AUC. We release leakit, a black-box extraction-audit tool.

Figures

Figures reproduced from arXiv: 2608.00144 by Victor Maricato.

Figure 1
Figure 1. Figure 1: Probabilistic black-box leakage. From a prefix x we query a black-box LLM for N continuations C1, . . . , CN ∼ pθ(· | x), an empirical estimate of the model’s output distribution. Leakage signals are functionals of these samples: overlap with a candidate continuation, and the mutual overlap among the samples themselves (shown). For a memorised (member) prefix the samples concentrate on the training continu… view at source ↗
Figure 2
Figure 2. Figure 2: Does black-box sampling beat a model-free classifier? Aggregate ROC-AUC for a blind [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Verbatim extraction of real identifiers from Pile-train documents on a shared [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Why sampling reproduces some training documents and not others (Pythia-2.8B, MIMIR, [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Max 5-gram recall of the held-out continuation versus sample budget [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Verbatim email extraction versus the context control (Pythia-2.8B, [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Extraction regime (Pythia-2.8B, MIMIR). (a) recovery gap across temperature and top-p (minor effect). (b) recovery gap and the samples needed N∗ as a function of prefix length: a short prefix needs more samples, a longer prefix needs one. F COST ANALYSIS Sampling cost. A single extraction probe at N=32, ℓ=64 requires ≈2k generated tokens per document. At current frontier-API prices (e.g. $10/M output for a… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 8 linked inside Pith

  1. [1]

    Pythia : A suite for analyzing large language models across training and scaling

    Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H., O'Brien, K., Hallahan, E., et al. Pythia : A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning (ICML), 2023

    2023

  2. [2]

    Extracting training data from large language models

    Carlini, N., Tramèr, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., et al. Extracting training data from large language models. In USENIX Security, 2021

    2021

  3. [3]

    Quantifying memorization across neural language models

    Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., and Zhang, C. Quantifying memorization across neural language models. In ICLR, 2023

    2023

  4. [4]

    Blind baselines beat membership inference attacks for foundation models

    Das, D., Zhang, J., and Tramèr, F. Blind baselines beat membership inference attacks for foundation models. In DATA-FM Workshop at ICLR, 2025. arXiv:2406.16201

    Pith/arXiv arXiv 2025

  5. [5]

    Do membership inference attacks work on large language models? In COLM, 2024

    Duan, M., Suri, A., Mireshghallah, N., Min, S., Shi, W., Zettlemoyer, L., et al. Do membership inference attacks work on large language models? In COLM, 2024

    2024

  6. [6]

    A., Menten, M

    Knolle, M. A., Menten, M. J., Jungmann, F., Meissen, F., Glocker, B., Rueckert, D., and Kaissis, G. Disparate privacy risks from medical AI . Nature, 2026

    2026

  7. [7]

    OLMo : Accelerating the science of language models

    Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., et al. OLMo : Accelerating the science of language models. In ACL, 2024

    2024

  8. [8]

    Sampling-based pseudo-likelihood for membership inference attacks

    Kaneko, M., Ma, Y., Wata, Y., and Okazaki, N. Sampling-based pseudo-likelihood for membership inference attacks. In Findings of ACL, 2025. arXiv:2404.11262

    Pith/arXiv arXiv 2025

  9. [9]

    S., Sutawika, L., Schoelkopf, H., Anthony, Q., Purohit, S., and Raff, E

    Biderman, S., Prashanth, U. S., Sutawika, L., Schoelkopf, H., Anthony, Q., Purohit, S., and Raff, E. Emergent and predictable memorization in large language models. In NeurIPS, 2023. arXiv:2304.11158

    Pith/arXiv arXiv 2023

  10. [10]

    Deduplicating training data makes language models better

    Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., and Carlini, N. Deduplicating training data makes language models better. In ACL, 2022. arXiv:2107.06499

    Pith/arXiv arXiv 2022

  11. [11]

    Analyzing leakage of personally identifiable information in language models

    Lukas, N., Salem, A., Sim, R., Tople, S., Wutschitz, L., and Zanella-B\'eguelin, S. Analyzing leakage of personally identifiable information in language models. In IEEE Symposium on Security and Privacy (S&P), 2023. arXiv:2302.00539

    Pith/arXiv arXiv 2023

  12. [12]

    F., Ippolito, D., Choquette-Choo, C

    Nasr, M., Rando, J., Carlini, N., Hayase, J., Jagielski, M., Cooper, A. F., Ippolito, D., Choquette-Choo, C. A., et al. Scalable extraction of training data from aligned, production language models. In International Conference on Learning Representations (ICLR), 2025. arXiv:2311.17035

    Pith/arXiv arXiv 2025

  13. [13]

    LLM dataset inference: Did you train on my dataset? In NeurIPS, 2024

    Maini, P., Jia, H., Papernot, N., and Dziedzic, A. LLM dataset inference: Did you train on my dataset? In NeurIPS, 2024

    2024

  14. [14]

    Z., and Dhingra, B

    Xie, R., Wang, J., Huang, R., Zhang, M., Ge, R., Pei, J., Gong, N. Z., and Dhingra, B. ReCaLL : Membership inference via relative conditional log-likelihoods. In EMNLP, 2024

    2024

  15. [15]

    Zaree, P., Mamun, M. A. A., Dong, Y., Alouani, I., and Abu-Ghazaleh, N. AttenMIA : LLM membership inference attack through attention signals. arXiv preprint arXiv:2601.18110, 2026

    arXiv 2026

  16. [16]

    Towards label-only membership inference attack against pre-trained large language models

    He, Y., et al. Towards label-only membership inference attack against pre-trained large language models. In USENIX Security, 2025. arXiv:2502.18943

    Pith/arXiv arXiv 2025

  17. [17]

    Practical membership inference attacks against fine-tuned large language models via self-prompt calibration

    Fu, W., et al. Practical membership inference attacks against fine-tuned large language models via self-prompt calibration. In NeurIPS, 2024. arXiv:2311.06062

    Pith/arXiv arXiv 2024

  18. [18]

    MoPe : Model perturbation-based privacy attacks on language models

    Li, M., Wang, J., Wang, J., and Neel, S. MoPe : Model perturbation-based privacy attacks on language models. In EMNLP, 2023

    2023

  19. [19]

    Membership inference attacks against language models via neighbourhood comparison

    Mattern, J., Mireshghallah, F., Jin, Z., Schölkopf, B., Sachan, M., and Berg-Kirkpatrick, T. Membership inference attacks against language models via neighbourhood comparison. In ACL Findings, 2023

    2023

  20. [20]

    Detecting pretraining data from large language models

    Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., et al. Detecting pretraining data from large language models. In ICLR, 2024

    2024

  21. [21]

    H., Zettlemoyer, L., and Aghajanyan, A

    Tirumala, K., Markosyan, A. H., Zettlemoyer, L., and Aghajanyan, A. Memorization without overfitting: Analyzing the training dynamics of large language models. In NeurIPS, 2022

    2022

  22. [22]

    Privacy risk in machine learning: Analyzing the connection to overfitting

    Yeom, S., Giacomelli, I., Fredrikson, M., and Jha, S. Privacy risk in machine learning: Analyzing the connection to overfitting. In IEEE CSF, 2018

    2018

  23. [23]

    Min-K\ In ICLR, 2025

    Zhang, J., Sun, J., Yeats, E., Ouyang, Y., Kuo, M., Zhang, J., Yang, H., and Li, H. Min-K\ In ICLR, 2025

    2025