REVIEW 5 major objections 6 minor 2 cited by
A Statistical and Multi-Perspective Revisiting of the Membership Inference Attack in Large Language Models
T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Membership inference attacks on large language models are usually no better than baselines when tested across thousands of settings, but a small set of outlier settings shows strong separability that grows with model size.
desk verdict A large-scale descriptive map of MIA performance across settings that is genuinely useful, but the statistical language overclaims and the Pile ground truth needs a control. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The engine of the paper is a statistical evaluation protocol: for each MIA method, it constructs many settings by crossing three text-split schemes (Truncate, Complete, Relative) over Pile domains, text-length ranges, six Pythia model sizes, and random seeds, producing 4,860 experiments per method. Instead of reporting a single AUC, it plots probability densities of ROC-AUC while fixing one dimension, counts differentiable outliers (ROC-AUC above 0.55), and tests whether member and non-member feature distributions differ. Supporting analyses use Davies-Bouldin scores of layer-wise embeddings, a transformer classifier trained on those embeddings, Spearman correlations with text length and 7-gram overlap, and accumulated entropy differences over decoding steps. These tools let the paper treat MIA performance as a distribution and connect scaling behavior to internal representation changes.
What would settle it
Rerun the same MIA battery on a synthetic corpus where membership is controlled by insertion time alone, with all texts drawn from one domain and identical formatting and deduplication. If the ROC-AUC distribution still improves with model size and shows domain variation, the effect is genuine memorization; if the signal mostly disappears, the Pile train/valid/test split carries corpus artifacts that explain the findings.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that membership inference performance is not a property of an attack method alone but of the whole setting—model size, domain, text length, and how member and non-member texts are sampled. Across thousands of experiments, most MIA methods fail to statistically outperform simple baselines such as loss or reference-model loss, which supports the recent negative results. Yet the distribution has a long tail: a small but consistent fraction of splits yields ROC-AUC above 0.55, reaching up to 0.81 for one method, and these differentiable outliers differ across methods. Performance improves with model size, especially between 1b and 2.8b parameters, and is higher in high-token-diversity domains like Wikipedia and FreeLaw. The paper further claims that thresholds learned in one domain or model size transfer poorly, that long text and low 7-gram overlap between members and non-members help attacks, that member/non-member separability appears in intermediate-layer embeddings and emerges at larger scale while last-layer embeddings are poorly separable, and that high-performing domains show faster accumulated entropy divergence during decoding.
Load-bearing premise
The load-bearing premise is that the only systematic difference between member texts and non-member texts in this study is whether the Pythia model trained on them; if the Pile train, validation, and test splits differ in domain mix, formatting, collection time, or deduplication, the measured membership signal could be a corpus artifact and all six findings would be confounded.
Editorial extensions
If this is right
- A single benchmark or setting cannot certify an MIA method; evaluations should report distributions over many settings, or they will keep yielding contradictory conclusions.
- Larger models are more vulnerable to membership inference in these experiments, at least up to 12b parameters, so scaling studies must treat MIA risk as growing unless evidence shows a downturn.
- MIA thresholds do not transfer across domains or model sizes, meaning any practical deployment needs per-setting threshold calibration or it will silently lose accuracy.
- Texts that are long and lexically distinct from surrounding corpus texts are easier to flag as members, which suggests MIA will work best on unusual, highly specific documents.
- Because different methods succeed on different outlier splits, comparing only average performance can hide complementary signals.
Reading between the lines
- The finding that last-layer embeddings are less separable than intermediate layers suggests an immediately testable extension: membership probes built from mid-layer hidden states should outperform current likelihood-based methods, since the paper's own data indicate the signal is stronger there.
- The outlier overlap matrix implies that an ensemble that combines several MIA methods, especially a gray-box and a black-box method, could recover much of the long tail, since each method is differentiable on different splits.
- If the Pile train/valid/test split carries time or formatting artifacts, the same statistical protocol could be rerun on synthetic corpora with controlled insertion times to separate true memorization from corpus structure.
- The domain dependence predicts that contamination-detection tools will be far more reliable for Wikipedia-like and legal text than for code, which could inform where data-leakage screening is worth deploying.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents a large-scale empirical re-evaluation of membership inference attacks (MIA) on LLMs. Using Pythia models of six sizes and Pile-based member/non-member splits with three construction methods, the authors run thousands of MIA experiments per method across domains and text lengths, then analyze performance densities, outlier cases, threshold transfer, text-length and similarity correlations, embedding separability, and decoding entropy. The main claims are that MIA performance improves with model size and varies by domain, that most methods do not statistically outperform simple baselines, that a notable number of differentiable member/non-member outliers exist and vary across methods, that threshold selection is an overlooked problem, that text dissimilarity and length help MIA, and that differentiability is reflected in embeddings and decoding dynamics. The paper positions its contribution as a statistical, multi-perspective explanation for the previously inconsistent MIA results in the literature.
Significance. If the findings hold, the paper would make a useful contribution by reframing MIA effectiveness as a distribution over settings rather than a single number, and by connecting positive and negative results in the literature. The experimental scale is genuinely impressive: 4,860 experiments per MIA method, multiple split constructions, six model sizes, and several complementary analyses. The use of Pythia, whose pre-training data is known, is appropriate for a membership study. The paper also makes a number of falsifiable claims, such as the model-size trend, the emergence of embedding separability around the 2.8b scale, and the negative correlation between text similarity and MIA performance. That said, the load-bearing statistical claims are currently asserted rather than demonstrated, and the membership ground truth is not validated against corpus artifacts. The study's value depends on both of these issues being resolved.
major comments (5)
- [4.1, Fig. 2, A.5] The central claim (I), that 'most methods do not statistically outperform baselines,' is not supported by the analyses as reported. Figure 2 shows probability densities and shaded seed variance, but these are descriptive summaries; there are no confidence intervals around AUC values, no pairwise significance tests between methods, and no multiple-comparison correction. The Appendix A.5 hypothesis-test analysis is also underspecified: the test is never named, no p-value threshold is given, and no multiple-testing correction is described. Since the word 'statistical' appears in the paper's title and central findings, the authors need to report concrete inferential statistics, such as bootstrap confidence intervals, permutation tests, or paired significance tests with appropriate corrections.
- [3.2-3.4] The Pile train/valid/test split is used as the ground-truth membership boundary without validating that the only systematic difference between the groups is training membership. Sections 3.2-3.4 assume that Pile train texts are members and valid/test texts are non-members, but the valid and test splits could differ from train in document composition, formatting, collection time, or residual duplication artifacts. The paper cites prior work on flawed MIA benchmarks but never applies the same scrutiny to its own splits. A control with a randomly initialized or differently trained Pythia model, a label-permutation test, or a comparison to an out-of-domain reference corpus would show whether the splits are separable without any training signal. Without such a control, findings (I) through (VI) could reflect corpus artifacts rather than membership, so this issue is upstream of the missing significance tests.
- [4.2, Table 1] The outlier analysis in Section 4.2 selects splits with ROC-AUC > 0.55 and calls them 'differentiable outliers,' but this threshold appears arbitrary and the counts are not accompanied by uncertainty estimates or a null model. Because the threshold is applied after observing performance, the analysis selects on the dependent variable; the overlap matrix in Figure 4 is also not compared to the overlap expected by chance. To support finding (II), the authors should calibrate the outlier definition using a null distribution obtained by shuffling member/non-member labels or by comparing against baseline methods, and they should report the variance of outlier counts across seeds and splits.
- [4.3, Fig. 3] The threshold-generalization claim in finding (III) is not directly tested. Figure 3 shows that threshold estimates vary across domains and model sizes, but variation alone does not establish that thresholds do not generalize. The authors should perform a quantitative transfer evaluation: estimate a threshold on one domain or model size, apply it to a held-out domain or model size, and report the resulting AUC or accuracy compared to the within-setting threshold. Without this, the claim that threshold selection is an 'overlooked challenge' remains suggestive rather than demonstrated.
- [4.4, Table 2] The Spearman correlations in Table 2 are reported without significance levels, confidence intervals, or any correction for the large number of correlations computed. Several coefficients are near zero and vary in sign across splits and methods, so the statement that there is a 'universal negative relation' with text similarity and a general positive relation with text length is stronger than the table supports. The authors should report p-values or bootstrap intervals and should quantify how consistently the sign of the correlation replicates across splits, domains, and model sizes.
minor comments (6)
- [3.3] The phrase 'ten-percental text length range' is awkward and unclear; it should be reworded, for example, to 'text lengths falling within each decile of the domain-specific test-set length distribution.'
- [4.1, Fig. 2] The caption's statement that 'Less area on the left side means statistically better MIA performance' is misleading because the figure does not contain statistical inference; rephrase to 'means higher ROC-AUC values in the density.'
- [Throughout] The paper inconsistently spells the metric as ROC-AUC, RUC-AUC, and RUC-AOC; all instances should be corrected to ROC-AUC.
- [4.4, Table 2] The SaMIA row contains a malformed entry ' - .31' that appears to be a formatting mistake.
- [6] In the Limitations section, the sentence 'We do deny the existence of better threshold calculation method exits' should likely read 'We do not deny the existence of better threshold calculation methods.'
- [3.5] The text contains the typo 'Ture Positive Rate'; it should be 'True Positive Rate.'
Circularity Check
No circularity: all findings are empirical measurements against external ground truth (Pile train/valid/test and WikiMIA); no derivation chain reduces to its own inputs.
full rationale
This paper is a large-scale empirical measurement study rather than a formal derivation. Its central findings (I)-(VI) are statements about computed ROC-AUC distributions, outlier counts, threshold variations, Spearman correlations, embedding separability scores, and entropy dynamics across Pythia models trained on the Pile. None of these results is obtained by fitting a parameter and then predicting the same quantity; the MIA methods are existing published methods evaluated as black boxes on externally defined member/non-member splits. There is no equation whose output is equivalent by construction to its input, and no fitted parameter is renamed as a prediction. The paper contains no self-citations by the authors, so no load-bearing self-citation chain or imported uniqueness theorem is present. The closest structural pattern to circularity is the outlier analysis in Section 4.2 and the embedding analysis in Section 4.5: splits are first labeled 'differentiable' by high ROC-AUC and then shown to have separable embeddings. This is selection on the outcome variable and limits what can be concluded about causal or predictive value, but it is not circular because the embedding DB scores and Transformer classifier accuracies are measured independently of the ROC-AUC threshold that defined the groups. Similarly, the use of the Pile train/valid/test boundary as membership ground truth is an assumption about corpus cleanliness; if the valid/test splits differ from train by domain, formatting, or duplication artifacts, the findings would be confounded. That is a validity threat, not a circularity, since the paper does not define membership in terms of its own measurements. The paper is self-contained against external benchmarks (Pile ground truth and WikiMIA) and does not derive its conclusions from the conclusions themselves. Therefore the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (9)
- Outlier threshold ROC-AUC > 0.55 =
0.55
- Min-k% k =
20
- EDA-PAC swap percentage =
30%
- EDA-PAC perturbed sentences =
5
- RECALL prefix count =
12
- SaMIA/CDD generation count =
10
- SaMIA/CDD temperature =
0.8
- Black-box max input length =
512 tokens
- Entropy sequence length =
36 tokens
assumptions (5)
- domain assumption The Pile train/valid/test split provides unconfounded membership labels for Pythia; there is no systematic distribution shift beyond membership.
- domain assumption Pythia model family is representative enough to draw general conclusions about MIA and model scaling.
- domain assumption ROC-AUC computed on about 100 positive and negative examples per split is a reliable measure of MIA performance without confidence intervals.
- domain assumption The 7-gram overlap normalized by total length is an adequate measure of text similarity for studying MIA.
- domain assumption The Transformer classifier trained on embeddings measures separability without overfitting; no held-out test is described.
Cite this review
Pith. "Pith review of A Statistical and Multi-Perspective Revisiting of the Membership Inference Attack in Large Language Models." pith.science (2026). https://pith.science/paper/5ZQ4YRRH
@misc{pith2026241213475,
author = {Pith},
title = {Pith review of: A Statistical and Multi-Perspective Revisiting of the Membership Inference Attack in Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/5ZQ4YRRH}},
note = {Machine review of arXiv:2412.13475}
}
read the original abstract
The lack of data transparency in Large Language Models (LLMs) has highlighted the importance of Membership Inference Attack (MIA), which differentiates trained (member) and untrained (non-member) data. Though it shows success in previous studies, recent research reported a near-random performance in different settings, highlighting a significant performance inconsistency. We assume that a single setting doesn't represent the distribution of the vast corpora, causing members and non-members with different distributions to be sampled and causing inconsistency. In this study, instead of a single setting, we statistically revisit MIA methods from various settings with thousands of experiments for each MIA method, along with study in text feature, embedding, threshold decision, and decoding dynamics of members and non-members. We found that (1) MIA performance improves with model size and varies with domains, while most methods do not statistically outperform baselines, (2) Though MIA performance is generally low, a notable amount of differentiable member and non-member outliers exists and vary across MIA methods, (3) Deciding a threshold to separate members and non-members is an overlooked challenge, (4) Text dissimilarity and long text benefit MIA performance, (5) Differentiable or not is reflected in the LLM embedding, (6) Member and non-members show different decoding dynamics.
Figures
Figures from the paper (7 more)
Forward citations
Cited by 2 Pith papers
-
Implicit Reasoning Steering via Concept Chaining
Reinforcement-learning-optimized concept-chain paragraphs covertly steer language-model multiple-choice preferences after continued pretraining, with far lower detectability than direct paraphrases.
-
LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems
A systematic review that categorizes LLM threats, severity scores, and mitigations across development and operation life cycles and multiple deployment scenarios.
Reference graph
Works this paper leans on
-
[1]
Similar to MIA performance, we observed that the number of splits that pass the hypothesis test increases with the model size. This confirms the analysis of the results of the RUC-AOC score using the probability density functions
-
[2]
Further, we also see that in this evaluation metrics, the best-performed method Min-K% ++ does not also show the best performances in pass- ing the hypothesis test. On the contrary, the best- performed MIA method is the Refer, which actu- ally has the lowest performance in the ROC-AUC analysis. The reason is that the hypothesis test method does not evalua...
-
[3]
Even though the hypothesis test does not pro- vide a method to differentiate members and non- Method 160m 410m 1b 2.8b 6.9b 12b Loss 0.08 0.067 0.087 0.107 0.120 0.167 Min-K% 0.08 0.087 0.073 0.153 0.207 0.220 Zlib 0.013 0.020 0.040 0.080 0.113 0.173 SaMIA 0.093 0.073 0.053 0.080 0.060 0.080 Min-K% ++ 0.033 0.073 0.133 0.273 0.453 0.567 Refer 0.040 0.040 ...
-
[4]
In this metric, we are also able to observe the same performance boost when transferring from the 1b to 2.8b model. This aligns with the obser- vation in the probability density analysis of RUC- AOC scores across dimensions, which confirms the emergent embedding change that we have discov- ered. A.6 Detailed Results in Each Split Method In this section, w...
-
[7]
For example, in the Trun- cate split, the EuroParl performs very well com- pared to other domains
In the first row, which shows the probability density over domains, we saw some more high- performance domains. For example, in the Trun- cate split, the EuroParl performs very well com- pared to other domains. One of the reasons may be that the EuroParl contains some non-English texts, which serve as an important feature for the member and non-member cla...
-
[8]
In the second row, which shows the proba- bility density over model sizes, we saw a uniform performance across different splits where the MIA performance positively scales with the model size
-
[9]
We are also able to observe some split-based differences
In the third row, which shows the probability density over the different MIA methods. We are also able to observe some split-based differences. In the Relative and Complete split, we can see that the Min-k% ++ performs better than other methods. However, in the Truncate split, we see mixed results where most methods do not show obvious performance differe...
-
[10]
The threshold still changes across domains with the existence of outliers for all those methods. The Refer method shows an extreme trend where the threshold in each domain is totally different, in- dicating a threshold decided from another domain totally failed to generalize to other domains
Show all 12 references
-
[11]
However, the Refer model has a stable threshold that is generalized well in other model sizes
Regarding the model size, we still observe that their thresholds change across model sizes. However, the Refer model has a stable threshold that is generalized well in other model sizes. This is probably because the referee relies on a refer- ence model, which makes it less de...
-
[12]
We saw that the perplexity, zlib, Min-k%, and DC-PDD all showed either a gradual increase or decreasing threshold values
Additionally, while the trend is not general, we are able to see that the change of threshold is not random in some methods. We saw that the perplexity, zlib, Min-k%, and DC-PDD all showed either a gradual increase or decreasing threshold values. This can increase the predicta...
-
[2018]
Anesthesia & Analgesia, 126:1
Correlation coefficients: Appropriate use and interpretation. Anesthesia & Analgesia, 126:1. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2023. Detecting pretraining data from large language models. Weiji...
2024
-
[2023]
In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787, Sin- gapore
NLP evaluation in trouble: On the need to mea- sure LLM data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787, Sin- gapore. Association for Computational Linguistics. Patrick Schober, Christa Boer, an...
2023
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.