{"id":"16ceb532-3705-475d-bad2-45777beb66df","arxiv_id":"2412.13475","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"A large-scale statistical evaluation shows membership inference attacks on LLMs are mostly near-random on average, but a small set of outlier settings are reliably attackable, and performance improves with model size.","lead":"This paper runs thousands of membership inference experiments on language models of six sizes and finds that most attack methods perform only slightly better than guessing, while results vary strongly by domain and text length. It also shows that a small number of 'outlier' cases are genuinely detectable, that choosing a threshold is a major unsolved problem, and that member and non-member texts differ measurably in the model's internal states.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pile train/valid/test split lacks a validation control; a random-model test could reveal corpus artifacts confound all findings.","rationale":"The reader's weakest-assumption identification is exactly the same as mine: Pile train/valid/test is treated as clean membership ground truth without validation. This is the most load-bearing concern because it sits at the base of the entire statistical comparison. If the split is confounded, the model-size trend, domain variation, outlier analysis, similarity/length correlations, embedding separability, and entropy dynamics all become ambiguous. The paper's lack of formal significance tests is a secondary issue: even if those tests were added, they would test differences that might be driven by artifacts rather than membership. The proposed random-model control directly isolates the artifact channel: a model with no training knowledge of the members should not be able to separate the two sets if the only difference is membership. Since the paper already uses Pythia checkpoints, step-0 checkpoints are readily available and the experiment is cheap compared to the existing generation-heavy setup. If the control passes (AUC near 0.50), the central claim is substantially strengthened; if it fails, the headline findings need to be re-evaluated. The verdict remains CONDITIONAL, matching the reader, because the paper currently does not supply this validation.","tokens_in":15527,"tokens_out":4072,"duration_ms":41275,"concrete_test":"Run the Loss and Min-K% MIA methods on the same Truncate, Complete, and Relative splits using a randomly initialized Pythia model (e.g., the step-0 checkpoint) or a Pythia model trained on a different corpus. If the ROC-AUC for these methods is not centered at 0.50 and instead exceeds 0.55 for multiple domains, the train/valid/test split is separable without any training membership, which would indicate a systematic corpus artifact. This control should be applied before adding significance tests or reinterpreting any of the findings.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central comparison treats Pile train texts as members and Pile valid/test texts as non-members (Sections 3.2-3.4), implicitly assuming the only systematic difference is training membership. But the valid/test splits may differ from train in document composition, formatting, time of collection, or residual duplication even within the same domain and length range. The paper itself cites flawed MIA benchmarks (Meeus et al., Das et al.) but never validates its own splits against non-membership artifacts. There is no control using an untrained model, a label permutation, or an out-of-domain reference corpus. If the splits are separable by token statistics alone, then findings (I) through (VI) — especially the model-size trend and domain variation — would reflect corpus artifacts rather than membership. This concern is upstream of the missing significance tests; even a fully rigorous statistical procedure would not remove a systematic split confound. The load-bearing assumption is therefore the cleanliness of the Pile train/valid/test membership boundary, and it is currently unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a large-scale empirical re-evaluation of membership inference attacks (MIA) on LLMs. Using Pythia models of six sizes and Pile-based member/non-member splits with three construction methods, the authors run thousands of MIA experiments per method across domains and text lengths, then analyze performance densities, outlier cases, threshold transfer, text-length and similarity correlations, embedding separability, and decoding entropy. The main claims are that MIA performance improves with model size and varies by domain, that most methods do not statistically outperform simple baselines, that a notable number of differentiable member/non-member outliers exist and vary across methods, that threshold selection is an overlooked problem, that text dissimilarity and length help MIA, and that differentiability is reflected in embeddings and decoding dynamics. The paper positions its contribution as a statistical, multi-perspective explanation for the previously inconsistent MIA results in the literature.","tokens_in":15652,"tokens_out":3961,"duration_ms":39274,"significance":"If the findings hold, the paper would make a useful contribution by reframing MIA effectiveness as a distribution over settings rather than a single number, and by connecting positive and negative results in the literature. The experimental scale is genuinely impressive: 4,860 experiments per MIA method, multiple split constructions, six model sizes, and several complementary analyses. The use of Pythia, whose pre-training data is known, is appropriate for a membership study. The paper also makes a number of falsifiable claims, such as the model-size trend, the emergence of embedding separability around the 2.8b scale, and the negative correlation between text similarity and MIA performance. That said, the load-bearing statistical claims are currently asserted rather than demonstrated, and the membership ground truth is not validated against corpus artifacts. The study's value depends on both of these issues being resolved.","major_comments":[{"comment":"The central claim (I), that 'most methods do not statistically outperform baselines,' is not supported by the analyses as reported. Figure 2 shows probability densities and shaded seed variance, but these are descriptive summaries; there are no confidence intervals around AUC values, no pairwise significance tests between methods, and no multiple-comparison correction. The Appendix A.5 hypothesis-test analysis is also underspecified: the test is never named, no p-value threshold is given, and no multiple-testing correction is described. Since the word 'statistical' appears in the paper's title and central findings, the authors need to report concrete inferential statistics, such as bootstrap confidence intervals, permutation tests, or paired significance tests with appropriate corrections.","section":"4.1, Fig. 2, A.5"},{"comment":"The Pile train/valid/test split is used as the ground-truth membership boundary without validating that the only systematic difference between the groups is training membership. Sections 3.2-3.4 assume that Pile train texts are members and valid/test texts are non-members, but the valid and test splits could differ from train in document composition, formatting, collection time, or residual duplication artifacts. The paper cites prior work on flawed MIA benchmarks but never applies the same scrutiny to its own splits. A control with a randomly initialized or differently trained Pythia model, a label-permutation test, or a comparison to an out-of-domain reference corpus would show whether the splits are separable without any training signal. Without such a control, findings (I) through (VI) could reflect corpus artifacts rather than membership, so this issue is upstream of the missing significance tests.","section":"3.2-3.4"},{"comment":"The outlier analysis in Section 4.2 selects splits with ROC-AUC > 0.55 and calls them 'differentiable outliers,' but this threshold appears arbitrary and the counts are not accompanied by uncertainty estimates or a null model. Because the threshold is applied after observing performance, the analysis selects on the dependent variable; the overlap matrix in Figure 4 is also not compared to the overlap expected by chance. To support finding (II), the authors should calibrate the outlier definition using a null distribution obtained by shuffling member/non-member labels or by comparing against baseline methods, and they should report the variance of outlier counts across seeds and splits.","section":"4.2, Table 1"},{"comment":"The threshold-generalization claim in finding (III) is not directly tested. Figure 3 shows that threshold estimates vary across domains and model sizes, but variation alone does not establish that thresholds do not generalize. The authors should perform a quantitative transfer evaluation: estimate a threshold on one domain or model size, apply it to a held-out domain or model size, and report the resulting AUC or accuracy compared to the within-setting threshold. Without this, the claim that threshold selection is an 'overlooked challenge' remains suggestive rather than demonstrated.","section":"4.3, Fig. 3"},{"comment":"The Spearman correlations in Table 2 are reported without significance levels, confidence intervals, or any correction for the large number of correlations computed. Several coefficients are near zero and vary in sign across splits and methods, so the statement that there is a 'universal negative relation' with text similarity and a general positive relation with text length is stronger than the table supports. The authors should report p-values or bootstrap intervals and should quantify how consistently the sign of the correlation replicates across splits, domains, and model sizes.","section":"4.4, Table 2"}],"minor_comments":[{"comment":"The phrase 'ten-percental text length range' is awkward and unclear; it should be reworded, for example, to 'text lengths falling within each decile of the domain-specific test-set length distribution.'","section":"3.3"},{"comment":"The caption's statement that 'Less area on the left side means statistically better MIA performance' is misleading because the figure does not contain statistical inference; rephrase to 'means higher ROC-AUC values in the density.'","section":"4.1, Fig. 2"},{"comment":"The paper inconsistently spells the metric as ROC-AUC, RUC-AUC, and RUC-AOC; all instances should be corrected to ROC-AUC.","section":"Throughout"},{"comment":"The SaMIA row contains a malformed entry ' - .31' that appears to be a formatting mistake.","section":"4.4, Table 2"},{"comment":"In the Limitations section, the sentence 'We do deny the existence of better threshold calculation method exits' should likely read 'We do not deny the existence of better threshold calculation methods.'","section":"6"},{"comment":"The text contains the typo 'Ture Positive Rate'; it should be 'True Positive Rate.'","section":"3.5"}],"recommendation":"major_revision","confidential_remarks":"This is a large and potentially useful empirical study, but its central claims are currently ahead of the evidence. The missing statistical tests and the unvalidated Pile split are fixable in revision, and the paper would become substantially more valuable if the authors add a random-model or label-permutation control and proper inferential statistics. I do not see a reason to reject outright, but the current version should not be accepted as-is."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is worth reading if you work on membership inference for LLMs. The scale is real: thousands of experiments per method across three split methods, ten-plus domains, six model sizes, and multiple random seeds. The descriptive findings are new and mostly credible: performance varies with domain and split method, outliers exist per method with little overlap, thresholds do not transfer across settings, and embedding separability decreases in the last layer. The threshold-generalization analysis and the outlier-overlap matrix are the most novel pieces and could shape how MIA evaluations are reported. I also appreciate that the authors are explicit about reproducing DC-PDD imperfectly and about not covering all possible methods.\n\nThat said, the central statistical framing is softer than the language suggests. The abstract says most methods do not statistically outperform baselines, but I did not find a single significance test or confidence interval in the paper. The density-overlap plots in Figure 2 are descriptive, not inferential. Appendix A.5 presents a hypothesis-test view but never names the test, reports the test statistic, or explains its assumptions—so it is not usable as evidence. This is fixable, but the claims should be reworded or the tests actually run.\n\nThe Pile split concern raised by the stress-test is real and, in my view, the biggest worry. The paper treats Pile train as members and valid/test as non-members without ever validating that the boundary is free of corpus artifacts. No random-model control, no label permutation, no out-of-domain reference. If the valid/test splits differ from train in format, domain composition, or duplication, findings (I) through (VI) could be confounded. The authors cite flawed-benchmark work (Meeus et al., Das et al.) but do not run the corresponding validation on their own setup. The use of WikiMIA as a second benchmark mitigates but does not eliminate the concern. This should be the referee's first request.\n\nWho is this for? Researchers evaluating MIA methods and anyone trying to interpret the inconsistent results in the literature. It deserves a serious referee, but the paper needs a revision that adds significance testing, a split-validation control, and a code/data release before the claims can be taken at face value.","headline":"A large-scale descriptive map of MIA performance across settings that is genuinely useful, but the statistical language overclaims and the Pile ground truth needs a control.","tokens_in":16242,"tokens_out":2265,"would_cite":true,"duration_ms":23767,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Membership inference attacks on large language models are usually no better than baselines when tested across thousands of settings, but a small set of outlier settings shows strong separability that grows with model size.","keywords":["membership inference attack","large language models","training data detection","statistical evaluation","ROC-AUC","embedding separability","threshold generalization","Pythia model"],"falsifier":"Rerun the same MIA battery on a synthetic corpus where membership is controlled by insertion time alone, with all texts drawn from one domain and identical formatting and deduplication. If the ROC-AUC distribution still improves with model size and shows domain variation, the effect is genuine memorization; if the signal mostly disappears, the Pile train/valid/test split carries corpus artifacts that explain the findings.","tokens_in":15233,"feed_emoji":"📊","tokens_out":7701,"duration_ms":62665,"temperature":0.7,"pith_summary":"The paper is trying to resolve a contradiction in membership inference attacks (MIA) on large language models: some studies report strong ability to tell which texts a model was trained on, while others report near-random performance. Its explanation is that a single benchmark setting samples only a tiny slice of a huge corpus, so members and non-members can come from different distributions and produce inconsistent results. To test this, the authors run thousands of MIA experiments spanning split methods, domains, text lengths, and six model sizes, then analyze ROC-AUC scores statistically rather than treating one number as the answer. They conclude that MIA effectiveness is a distribution over settings: most methods do not beat simple baselines on average, but a notable set of outlier settings shows strong separability, and separability generally grows with model size, text length, and member/non-member text dissimilarity.","feed_headline":"Most LLM membership attacks don't beat baselines","feed_subtitle":"A 4,860-experiment retest shows success depends on model size and domain, not the method","key_machinery":"The engine of the paper is a statistical evaluation protocol: for each MIA method, it constructs many settings by crossing three text-split schemes (Truncate, Complete, Relative) over Pile domains, text-length ranges, six Pythia model sizes, and random seeds, producing 4,860 experiments per method. Instead of reporting a single AUC, it plots probability densities of ROC-AUC while fixing one dimension, counts differentiable outliers (ROC-AUC above 0.55), and tests whether member and non-member feature distributions differ. Supporting analyses use Davies-Bouldin scores of layer-wise embeddings, a transformer classifier trained on those embeddings, Spearman correlations with text length and 7-gram overlap, and accumulated entropy differences over decoding steps. These tools let the paper treat MIA performance as a distribution and connect scaling behavior to internal representation changes.","core_discovery":"On the paper's own terms, the discovery is that membership inference performance is not a property of an attack method alone but of the whole setting—model size, domain, text length, and how member and non-member texts are sampled. Across thousands of experiments, most MIA methods fail to statistically outperform simple baselines such as loss or reference-model loss, which supports the recent negative results. Yet the distribution has a long tail: a small but consistent fraction of splits yields ROC-AUC above 0.55, reaching up to 0.81 for one method, and these differentiable outliers differ across methods. Performance improves with model size, especially between 1b and 2.8b parameters, and is higher in high-token-diversity domains like Wikipedia and FreeLaw. The paper further claims that thresholds learned in one domain or model size transfer poorly, that long text and low 7-gram overlap between members and non-members help attacks, that member/non-member separability appears in intermediate-layer embeddings and emerges at larger scale while last-layer embeddings are poorly separable, and that high-performing domains show faster accumulated entropy divergence during decoding.","pith_inferences":["The finding that last-layer embeddings are less separable than intermediate layers suggests an immediately testable extension: membership probes built from mid-layer hidden states should outperform current likelihood-based methods, since the paper's own data indicate the signal is stronger there.","The outlier overlap matrix implies that an ensemble that combines several MIA methods, especially a gray-box and a black-box method, could recover much of the long tail, since each method is differentiable on different splits.","If the Pile train/valid/test split carries time or formatting artifacts, the same statistical protocol could be rerun on synthetic corpora with controlled insertion times to separate true memorization from corpus structure.","The domain dependence predicts that contamination-detection tools will be far more reliable for Wikipedia-like and legal text than for code, which could inform where data-leakage screening is worth deploying."],"forward_implications":["A single benchmark or setting cannot certify an MIA method; evaluations should report distributions over many settings, or they will keep yielding contradictory conclusions.","Larger models are more vulnerable to membership inference in these experiments, at least up to 12b parameters, so scaling studies must treat MIA risk as growing unless evidence shows a downturn.","MIA thresholds do not transfer across domains or model sizes, meaning any practical deployment needs per-setting threshold calibration or it will silently lose accuracy.","Texts that are long and lexically distinct from surrounding corpus texts are easier to flag as members, which suggests MIA will work best on unusual, highly specific documents.","Because different methods succeed on different outlier splits, comparing only average performance can hide complementary signals."],"supporting_citations":[{"why":"Documents the near-random MIA results and supplies the Truncate Split construction that this paper extends to length ranges and many domains.","marker":"Duan et al., 2024"},{"why":"Shows simple word-difference baselines can cheat MIA benchmarks, motivating the statistical, multi-setting evaluation.","marker":"Das et al., 2024"},{"why":"Identifies flawed MIA benchmarks, supporting the claim that single-setting results may be misleading.","marker":"Meeus et al., 2024"},{"why":"Provides the WikiMIA separable benchmark and the Davies-Bouldin index used to measure embedding separability.","marker":"Shi et al., 2023"},{"why":"Supplies the Pile corpus and its train/valid/test distinction used as the main member/non-member data.","marker":"Gao et al., 2020"},{"why":"Supplies the Pythia model suite and its deduplicated Pile training, enabling controlled comparisons across six model sizes.","marker":"Biderman et al., 2023"},{"why":"Defines the reference-model and zlib-calibrated baselines and the general loss-based extraction framing the attacks build on.","marker":"Carlini et al., 2021"},{"why":"Establishes the loss-based membership inference baseline that most evaluated methods are measured against.","marker":"Yeom et al., 2018"},{"why":"Provides the Min-k%++ method that this paper identifies as the strongest overall gray-box attack.","marker":"Zhang et al., 2024b"},{"why":"Provides the SaMIA black-box method whose semantic-similarity behavior is compared against the other attacks.","marker":"Kaneko et al., 2024"}],"fun_headline_variants":["MIA success depends on domain and model size, not the method","Most membership attacks fail to beat simple baselines","Threshold selection is the overlooked MIA challenge","Long text and low overlap boost MIA success","Embeddings reveal member vs non-member separability"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the only systematic difference between member texts and non-member texts in this study is whether the Pythia model trained on them; if the Pile train, validation, and test splits differ in domain mix, formatting, collection time, or deduplication, the measured membership signal could be a corpus artifact and all six findings would be confounded.","fun_headline_variants_meta":{"raw":{"variants":["MIA success depends on domain and model size, not the method","Most membership attacks fail to beat simple baselines","Threshold selection is the overlooked MIA challenge","Long text and low overlap boost MIA success","Embeddings reveal member vs non-member separability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000962,"raw_usage":{"total_tokens":4130,"prompt_tokens":1013,"completion_tokens":3117,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":3043}},"tokens_in":629,"tokens_out":3117,"duration_ms":20164,"temperature":1.0,"reasoning_tokens":3043,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:06:00.446889+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the same MIA battery on a synthetic corpus where membership is controlled by insertion time alone, with all texts drawn from one domain and identical formatting and deduplication. If the ROC-AUC distribution still improves with model size and shows domain variation, the effect is genuine memorization; if the signal mostly disappears, the Pile train/valid/test split carries corpus artifacts that explain the findings.","supporting_citations":[],"review_version":1}