Synthetic Data Can Mislead Evaluations: Membership Inference as Machine Text Detection

Ali Naseh; Niloofar Mireshghallah

REVIEW 2 major objections 4 minor 13 references

Synthetic Data Can Mislead Evaluations: Membership Inference as Machine Text Detection

T0 review · 2 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that replacing human-written non-members with machine-generated continuations makes most membership inference attacks on LLMs score synthetic text as more member-like than actual training data, so synthetic-data…

desk verdict Synthetic non-member effect is real, but the machine-text detector claim needs a matched human continuation control. read the letter →

arxiv 2501.11786 v1 pith:O2HF4R2P submitted 2025-01-20 cs.CL cs.CRcs.LG

Ali Naseh , Niloofar Mireshghallah This is my paper · ORCID

classification cs.CLcs.CRcs.LG

keywords membershipinferenceattacksmachine-generatedtextdetectionsyntheticdataevaluationlargelanguagemodelsprivacyleakagelikelihood-basedMIMIRbenchmark

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that membership inference attacks on large language models are, in practice, machine-generated-text detectors rather than membership detectors. When non-members in an evaluation are replaced by synthetic continuations from any of five language models, loss-based, Min-K, and Min-K++ attacks rank those synthetic samples as more likely to be training members than the real members, producing AUC far below 0.5 instead of above it. In the headline case, AUC falls from 0.66 to 0.20 for GPT-Neo 2.7B on the Wikipedia subset. The authors conclude that synthetic non-members cannot support valid conclusions about model memorization or data leakage.

What carries the argument

The mechanism is the target model's token-probability surface. Likelihood-based MIAs assign membership scores by how probable the model finds a candidate text, either through raw loss, low-percentile token likelihoods (Min-K), or normalized likelihoods (Min-K%++). Machine-generated continuations are produced by sampling from a language model, so they occupy high-probability regions of the model's distribution, which is exactly the signature these attacks interpret as memorization. The reference-based attack and the zlib attack add normalizers, and zlib's compression-length normalization is the one that keeps synthetic and human non-members distinguishable in the reported experiments.

What would settle it

A matched-length control would settle it: have humans write continuations of the same 30-token prompts to 200 tokens, matching topic and style, and run the same five attacks. If loss, Min-K, and Min-K%++ keep AUC near 0.66 instead of dropping to 0.20, the inversion is specific to machine-generated text; if human continuations produce the same drop, the trigger is something other than machine generation.

Watch

Extended reading notes

Core claim

The central discovery is the inversion of membership scores when non-members are machine-generated. Using GPT-Neo 2.7B as the target model, human-written members and non-members from the MIMIR Wikipedia and ArXiv subsets, and five attack methods (loss, reference-based, zlib, Min-K%, Min-K%++), the authors find that replacing human non-members with continuations generated by LLaMA 2, LLaMA 3.1, GPT-3.5, Pythia, or GPT-Neo itself pushes loss, Min-K, and Min-K%++ well below random guessing, sometimes to an AUC near 0. The attacks therefore treat synthetic text as more member-like than actual training members. The pattern transfers across generator architectures and scales, while the zlib attack remains above 0.5 in most synthetic settings. The paper concludes that certain MIAs consistently misclassify synthetic data as training members, suggesting that these attacks function more as detectors of machine-generated text than as membership detectors.

Load-bearing premise

The central assumption is that a 200-token machine continuation of the first 30 tokens of a human non-member is a fair stand-in for a full human-written non-member, so that the AUC collapse is caused by the text's machine origin rather than by differences in length, topic, or generation artifacts.

Editorial extensions

If this is right

MIA evaluations that replace human non-members with synthetic text can report AUC near zero not because the attack is broken but because it is detecting machine text, so memorization and leakage conclusions drawn from such setups are unsupported.
The inversion transfers across generator models, including larger and commercially deployed ones, so no synthetic non-member source can be assumed safe for likelihood-based membership evaluation.
The zlib attack is the one evaluated method that mostly stays above 0.5 with synthetic non-members, suggesting compression-based normalization is a more robust signal for separating genuine members from generated text.
Because synthetic text is increasingly common on the web, future non-member pools may unknowingly contain machine-generated material, making the confound a practical risk rather than only a laboratory artifact.
The same likelihood signal is used in other evaluation settings, so machine-translated or model-generated substitutes for real samples may bias loss-based measurements beyond membership inference.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not test human-written continuations at equal length; if humans complete the same 30-token prompts to 200 tokens and the AUC collapse persists, then the trigger would be length or prompt matching rather than machine generation, which would narrow the paper's conclusion.
The extreme inversion of Min-K%++ could be repurposed as a cheap screening test for synthetic contamination because a non-member pool that flips a well-calibrated attack below chance is likely to contain generated text.
If synthetic text increasingly enters real training corpora, the boundary between machine-generated and memorized blurs, because a model can genuinely memorize web text that was itself machine-written, making the observed confusion partly a labeling problem.
The zlib outlier suggests a testable design principle: membership scores that normalize by text compressibility or other length-insensitive statistics may separate high-likelihood text from actual training exposure, a direction the paper leaves open.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Hi — you should know this paper has a real, visible result and a load-bearing hole. The result: on MIMIR, swap human non-members for 200-token synthetic continuations (generated from 30-token prefixes) and likelihood-based MIAs against GPT-Neo 2.7B go from AUC ~0.66 to ~0.20. That is a concrete demonstration that the increasingly common practice of using synthetic non-members can reverse membership scores. That matters.

What it does well: the experiment is simple and the effect is large. Five generators, two datasets, five attacks; the pattern in loss/min-k/min-k++ is consistent. The zlib attack staying above 0.5 is a useful outlier. Prior work had only drawn the conceptual parallel between MIAs and text detection; showing the empirical inversion is new.

Where it falls short: the interpretation is not established. Members are full MIMIR documents; synthetic non-members are short continuations. No control where human non-members are turned into matched continuations (the next 200 tokens after the same prefix) or truncated to the same length. So the AUC drop could be from length, document position, or coherence rather than the machine origin of the text. The paper treats 'synthetic' and 'machine-generated' as the only difference, which they aren't here. That missing control is the main issue.

Also, the wording outruns the data. 'Consistent patterns in all but one attack' is false—reference-based stays above 0.5 in most synthetic rows, zlib always. 'Persists across model architectures and sizes' is about generator models, not the target; only GPT-Neo is attacked. No error bars, no code or data, no baseline against actual text detectors. These are not fatal to the empirical observation, but they make the 'MIAs are text detectors' claim a hypothesis rather than a conclusion.

Bottom line: if I were refereeing, I'd ask for the matched human-continuation control and a more modest title. With that, this could be a useful caution for the MIA evaluation community. Without it, the headline result could be an artifact of comparing full documents to short continuations. It deserves a serious referee—the question is important and the result is striking—but it should not be accepted as is.

Referee Report

2 major / 4 minor

Summary. The paper studies whether membership inference attacks (MIAs) on large language models are misled when human-written non-members are replaced by synthetic continuations. Using the MIMIR benchmark (Wikipedia and ArXiv) and GPT-Neo 2.7B as the target model, the authors compare a conventional setup (human members vs. human non-members) with a synthetic setup (human members vs. 200-token machine-generated continuations of human non-member prefixes). For loss, Min-K, and Min-K++ attacks, AUC drops from roughly 0.65–0.79 to values well below 0.5, indicating that these attacks score synthetic continuations as more member-like than actual training members. The reference-based attack shows mixed results, and the zlib attack remains above 0.5. The paper interprets this as evidence that such MIAs function as machine-generated-text detectors, and it warns against using synthetic data in membership evaluation protocols.

Significance. If the empirical pattern is robust and the interpretation is correct, the paper identifies a serious methodological pitfall: using synthetic data as non-members can invert the ranking produced by likelihood-based MIAs, invalidating conclusions about memorization and leakage. The strengths of the paper are its simple, direct experimental design, the use of a public benchmark (MIMIR), the range of generator models (GPT-Neo, Pythia, LLaMA 2, LLaMA 3.1, GPT-3.5), and the clear reporting of per-attack AUC values in Table 1. The zlib outlier is also a useful contrast that suggests normalization matters. However, the central comparison is potentially confounded by document length and position, and one of the paper's empirical summary statements is inconsistent with its own table. These issues need to be resolved before the conclusions can be considered established.

major comments (2)

[§3.1, Table 1] The central comparison is confounded: the human members are full documents from MIMIR, while the synthetic non-members are 200-token continuations generated from the first 30 tokens of human non-members. These pools differ not only in human-vs-machine origin but also in length, document position, and discourse structure. Likelihood-based scores are sensitive to such properties: a short, coherent continuation can have systematically lower per-token loss than a full Wikipedia or ArXiv article with sections, citations, and formula-heavy passages. The paper provides no control where human non-members are converted into matched continuations (e.g., the actual next 200 tokens after the same 30-token prefix) or where all documents are truncated to the same length. Without such a control, the observed drop from AUC 0.66 to 0.20 could reflect length or composition effects rather than the machine origin of the text. This is load-bearing because the paper's headline conclusion—that MIAs act as machine-text detectors—rests on this comparison.
[§1, §4, Table 1] The claim that 'consistent patterns' are observed in 'all but one attack method (Zlib)' is contradicted by Table 1. The reference-based attack achieves AUC above 0.5 for three of the five synthetic generators on Wikipedia (LLaMA 2-7B: 0.559; LLaMA 3.1-8B: 0.593; GPT-3.5: 0.613) and for four of five on ArXiv (Pythia: 0.761; LLaMA 2-7B: 0.908; LLaMA 3.1-8B: 0.924; GPT-3.5: 0.892). Thus the inversion is not as uniform as the text suggests. The paper should either restrict the claim to loss, Min-K, and Min-K++ and discuss why reference-based and zlib behave differently, or provide a more nuanced interpretation that does not treat all non-zlib attacks as a single class.

minor comments (4)

[Abstract] The sentence 'This behavior persists across different model architectures and sizes, from open-source models to commercial ones such as GPT-3.5' could be misread as varying the target model; in the experiments only GPT-Neo 2.7B is used as the target, while the generator models vary. Please clarify that the variation is in the synthetic text generators, and state explicitly that target-model generalization is not yet demonstrated.
[§3.2] The reference-based attack is not fully specified: the paper does not state which reference model is used. Since the MIMIR codebase provides a default reference model for each target, this should be described for reproducibility.
[Table 1] No measure of variability or statistical significance is reported for the AUC values. Given that the differences between setups are the main result, reporting confidence intervals or multiple subsamples would strengthen the conclusions.
[General] There are minor typographical issues: 'University of Massachusettes Amherst' should be 'Massachusetts', and the citation 'Min-k++[Zhang et al., 2024b]' should have a space and consistent formatting.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports an empirical benchmark comparison and does not derive its conclusion from its setup by construction.

full rationale

The paper's central claim—that many likelihood-based MIAs score synthetic continuations as more member-like than human-written members—is an observed AUC comparison using the external MIMIR benchmark, documented MIA implementations, and a synthetic non-member construction procedure taken from prior work (Mitchell et al., 2023). The synthetic pool is not fitted to the MIA outputs, and the AUC values are independent measurements of fixed attack scores on fixed data pools. The interpretation that MIAs function as machine-text detectors is a post-hoc explanation of these measurements rather than a premise built into the attack definitions, the data generation, or the evaluation protocol. The same-model rows are admittedly less surprising because text sampled from the target model tends to have high likelihood under that model, but the cross-model and cross-generator results (e.g., GPT-3.5 and LLaMA 3.1 continuations against GPT-Neo 2.7B) provide independent empirical content. The possible absence of a matched human-continuation control is a validity or confound concern about comparability of document length and position, not a circular-derivation concern, and the paper does not invoke any self-citation as the load-bearing justification for its main result. No equation or fitted parameter reduces to the paper's own inputs, so no circularity is present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted; the study is an empirical evaluation. The main non-standard premises are the validity of the synthetic non-member construction and the interpretation of AUC inversion as misclassification. No new entities or mechanisms are introduced.

assumptions (3)

domain assumption MIMIR's membership labels for the Wikipedia and ArXiv subsets are correct.
All AUC values are computed against these labels (Section 3.1); label noise would shift every reported number.
ad hoc to paper A 200-token continuation prompted by the first 30 tokens of a non-member is a valid synthetic non-member, comparable to full human-written non-members.
The synthetic setup (Section 3.1) relies on this comparability; the paper does not check length or genre matching with the member pool.
domain assumption AUC below 0.5 indicates that the attack is 'misled' rather than producing a usable inverted signal.
The paper interprets inversion as failure (Sections 1 and 4); this is a reasonable assumption for a well-calibrated attack, but it is an interpretive premise.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Synthetic Data Can Mislead Evaluations: Membership Inference as Machine Text Detection." pith.science (2026). https://pith.science/paper/O2HF4R2P

@misc{pith2026250111786,
  author       = {Pith},
  title        = {Pith review of: Synthetic Data Can Mislead Evaluations: Membership Inference as Machine Text Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O2HF4R2P}},
  note         = {Machine review of arXiv:2501.11786}
}

read the original abstract

Recent work shows membership inference attacks (MIAs) on large language models (LLMs) produce inconclusive results, partly due to difficulties in creating non-member datasets without temporal shifts. While researchers have turned to synthetic data as an alternative, we show this approach can be fundamentally misleading. Our experiments indicate that MIAs function as machine-generated text detectors, incorrectly identifying synthetic data as training samples regardless of the data source. This behavior persists across different model architectures and sizes, from open-source models to commercial ones such as GPT-3.5. Even synthetic text generated by different, potentially larger models is classified as training data by the target model. Our findings highlight a serious concern: using synthetic data in membership evaluations may lead to false conclusions about model memorization and data leakage. We caution that this issue could affect other evaluations using model signals such as loss where synthetic or machine-generated translated data substitutes for real-world samples.

Figures

Figures reproduced from arXiv: 2501.11786 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 5 canonical work pages

[1]

Evaluations of machine learning privacy defenses are misleading

Michael Aerni, Jie Zhang, and Florian Tramèr. Evaluations of machine learning privacy defenses are misleading. In Proceedings of the 2024 ACM SIGSAC Conference on Computer and Communica- tions Security, pages 1271–1284. ACM,

work page 2024
[4]

Do membership inference attacks work on large language models? arXiv preprint arXiv:2402.07841,

Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. Do membership inference attacks work on large language models? arXiv preprint arXiv:2402.07841,

arXiv
[5]

The Pile: An 800gb dataset of diverse text for language modeling

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027,

arXiv
[8]

Mishaal Kazmi, Hadrien Lautraite, Alireza Akbari, Qiaoyue Tang, Mauricio Soroco, Tao Wang, Sébastien Gambs, and Mathias Lécuyer

URL https://arxiv.org/abs/2406.18664. Mishaal Kazmi, Hadrien Lautraite, Alireza Akbari, Qiaoyue Tang, Mauricio Soroco, Tao Wang, Sébastien Gambs, and Mathias Lécuyer. Panoramia: Privacy auditing of machine learning models without retraining. arXiv preprint arXiv:2402.09477,

arXiv
[9]

Llm dataset inference: Did you train on my dataset? arXiv preprint arXiv:2406.06443,

Pratyush Maini, Hengrui Jia, Nicolas Papernot, and Adam Dziedzic. Llm dataset inference: Did you train on my dataset? arXiv preprint arXiv:2406.06443,

arXiv
[10]

Smaller language models are better black-box machine-generated text detectors

Fatemehsadat Mireshghallah, Justus Mattern, Sicun Gao, Reza Shokri, and Taylor Berg-Kirkpatrick. Smaller language models are better black-box machine-generated text detectors. arXiv preprint arXiv:2305.09859,

arXiv
[11]

Detecting pretraining data from large language models

Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789,

arXiv
[12]

On Evaluating Multilingual Compositional Generalization with Translated Datasets

Zi Wang and Daniel Hershcovich. On evaluating multilingual compositional generalization with translated datasets. arXiv preprint arXiv:2306.11420,

work page Pith review arXiv

Show all 13 references

[13]

Privacy risk in machine learning: Analyzing the connection to overfitting

Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pages 268–282. IEEE,

2018
[2018]

Membership inference attacks can’t prove that a model was trained on your data

Jie Zhang, Debeshee Das, Gautam Kamath, and Florian Tramèr. Membership inference attacks can’t prove that a model was trained on your data. arXiv preprint arXiv:2412.06157, 2024a. Jingyang Zhang et al. Min-k++: Improved baseline for detecting pre-training data from large langu...

arXiv
[2020]

Synthetic is all you need: Removing the auxiliary data assumption for membership inference attacks against synthetic data

Florent Guépin, Matthieu Meeus, Ana-Maria Cretu, and Yves-Alexandre de Montjoye. Synthetic is all you need: Removing the auxiliary data assumption for membership inference attacks against synthetic data. arXiv preprint arXiv:2307.01701,

arXiv
[2021]

Blind baselines beat membership inference attacks for foundation models

Debeshee Das, Jie Zhang, and Florian Tramèr. Blind baselines beat membership inference attacks for foundation models. arXiv preprint arXiv:2406.16201,

arXiv
[2024]

Wen Bao and Others

doi: 10.1145/3658644.3690194. Wen Bao and Others. Fast-detectgpt: Enhancing detection of machine-generated text. NeurIPS Workshop on Robust NLP,

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Evaluations of machine learning privacy defenses are misleading

Michael Aerni, Jie Zhang, and Florian Tramèr. Evaluations of machine learning privacy defenses are misleading. In Proceedings of the 2024 ACM SIGSAC Conference on Computer and Communica- tions Security, pages 1271–1284. ACM,

work page 2024

[2] [4]

Do membership inference attacks work on large language models? arXiv preprint arXiv:2402.07841,

Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. Do membership inference attacks work on large language models? arXiv preprint arXiv:2402.07841,

arXiv

[3] [5]

The Pile: An 800gb dataset of diverse text for language modeling

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027,

arXiv

[4] [8]

Mishaal Kazmi, Hadrien Lautraite, Alireza Akbari, Qiaoyue Tang, Mauricio Soroco, Tao Wang, Sébastien Gambs, and Mathias Lécuyer

URL https://arxiv.org/abs/2406.18664. Mishaal Kazmi, Hadrien Lautraite, Alireza Akbari, Qiaoyue Tang, Mauricio Soroco, Tao Wang, Sébastien Gambs, and Mathias Lécuyer. Panoramia: Privacy auditing of machine learning models without retraining. arXiv preprint arXiv:2402.09477,

arXiv

[5] [9]

Llm dataset inference: Did you train on my dataset? arXiv preprint arXiv:2406.06443,

Pratyush Maini, Hengrui Jia, Nicolas Papernot, and Adam Dziedzic. Llm dataset inference: Did you train on my dataset? arXiv preprint arXiv:2406.06443,

arXiv

[6] [10]

Smaller language models are better black-box machine-generated text detectors

Fatemehsadat Mireshghallah, Justus Mattern, Sicun Gao, Reza Shokri, and Taylor Berg-Kirkpatrick. Smaller language models are better black-box machine-generated text detectors. arXiv preprint arXiv:2305.09859,

arXiv

[7] [11]

Detecting pretraining data from large language models

Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789,

arXiv

[8] [12]

On Evaluating Multilingual Compositional Generalization with Translated Datasets

Zi Wang and Daniel Hershcovich. On evaluating multilingual compositional generalization with translated datasets. arXiv preprint arXiv:2306.11420,

work page Pith review arXiv