Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

A tokenizer's public vocabulary can reveal which dataset trained it

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 11:17 UTC pith:G3RRHNIE

load-bearing objection Tokenizer membership inference is real and carefully measured under a same-distribution shadow assumption, but the real-world LLM claims outrun the evidence. the 3 major comments →

arxiv 2510.05699 v4 pith:G3RRHNIE submitted 2025-10-07 cs.CR cs.AI

Membership Inference Attacks on Tokenizers of Large Language Models

Meng Tong , Yuntao Du , Kejiang Chen , Weiming Zhang , Ninghui Li This is my paper
classification cs.CR cs.AI
keywords tokenizermembership inferencebyte-pair encodingvocabulary leakagelarge language modelsdataset membershipshadow tokenizerspower law
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes a new attack vector for membership inference against large language models: the tokenizer. Because tokenizers are trained on web-scale text and their vocabularies are publicly released for billing transparency, the authors argue that the presence of dataset-distinctive tokens in a target vocabulary leaks whether that dataset was used in the tokenizer's training. They propose five attacks, two of which—Vocabulary Overlap and Frequency Estimation—reach AUC scores around 0.74–0.77 on a 200,000-token tokenizer. They also show that larger vocabularies and larger target datasets make the leakage stronger, and that a simple rare-token removal defense (min count) only partially mitigates the risk. The significance is that tokenizer artifacts, often overlooked, become a practical privacy and copyright-evidence concern for open-sourced components.

Core claim

The central claim is the distinctive-token hypothesis: the more distinctive tokens from dataset D that appear in the target tokenizer's vocabulary V, the more likely V was trained on D. The paper operationalizes this with two complementary attacks: a shadow-tokenizer approach that measures set overlap of distinctive tokens after removing non-distinctive tokens, and a shadow-free approach that computes a new metric, RTF-SI (Relative Token Frequency with Self-information), which scores each late-merge token by the product of its relative frequency in D and the estimated self-information from a power-law frequency model. Both attacks target set-level membership (which datasets, not which docume

What carries the argument

The load-bearing object is the tokenizer vocabulary with its merge indices, produced by byte-pair encoding (BPE). Distinctive tokens—strings merged only because they appear in the target dataset—are the signal. Vocabulary Overlap uses a set-overlap count between shadow vocabularies trained with and without D after subtracting tokens common to both. Frequency Estimation uses the RTF-SI score, where the self-information term is estimated by fitting a power-law tail to token counts versus merge index on a single shadow tokenizer, then bounding SI from below via the theorem RTF-SI ≥ RTF · log(Σ i^α / j^α). The max RTF-SI over late-merge tokens is the membership signal.

Load-bearing premise

For the shadow-free Frequency Estimation attack, the load-bearing premise is that the tail of token frequencies in the target tokenizer's training data follows the same power-law distribution with parameters fitted on a single shadow tokenizer; if that transfer fails (different vocabulary sizes, corpus mixtures, or BPE implementations), the estimated self-information is a heuristic with no stated guarantee.

What would settle it

Train a tokenizer on a dataset D, then postprocess its vocabulary to remove every token whose total count in the full training corpus is below a high threshold (e.g., 100), ensuring that no distinctive tokens from D survive. If the Vocabulary Overlap attack still distinguishes D from non-members above chance, the distinctive-token hypothesis is wrong or incomplete. Conversely, apply the attack to a tokenizer trained on a corpus with all rare tokens removed at train time and measure AUC; if it stays at chance, the signal's dependence on distinctive tokens is confirmed.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Anyone with access to an open-sourced tokenizer can test whether a candidate dataset—such as a copyrighted corpus or private user collection—was used in LLM training.
  • As tokenizer vocabularies grow for better compression and model quality, membership leakage will increase, making future tokenizers more vulnerable, not less.
  • Larger datasets are easier to confirm as members, so the highest-value legal claims (massive corpora) are the ones the attack handles best.
  • Removing infrequent tokens (min count defense) lowers attack accuracy but also lowers compression efficiency; the attack remains above chance for large datasets even under strict filtering.
  • The power-law frequency model gives an efficient shadow-free attack: one shadow tokenizer and minutes of inference instead of dozens of shadow tokenizers and hours.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because tokenizer vocabularies are published as static artifacts and do not change after release, membership inference on tokenizers may be immune to defenses that perturb model outputs—so organizations should treat vocabularies as sensitive even if the LLM weights are public.
  • The power-law assumption could be tested against alternative frequency estimators (e.g., neural or count-based) on out-of-distribution data, and the attack methodology may transfer to other learned subword vocabularies (SentencePiece, Unigram) if the same distinctive-token phenomenon appears.
  • The distinctive-token signal might be exploitable for the inverse task: given a candidate dataset, identify which tokenizer (and thus which model) was trained on it—a tokenizer-owner attribution tool.
  • Deduplication or decontamination of web corpora would likely reduce distinctive tokens; if so, the attack's success rate on a particular corpus could serve as a measure of contamination.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces tokenizers as a new attack surface for membership inference against LLMs. It proposes five set-level MIA methods: Merge Similarity, Vocabulary Overlap, Frequency Estimation, Naive Bayes, and Compression Rate. The two main attacks—Vocabulary Overlap and Frequency Estimation—use shadow tokenizers trained on auxiliary data drawn from the same distribution as the target tokenizer's training corpus. Experiments on C4 website datasets with tokenizer vocabulary sizes 80k–200k report AUC up to 0.771 and 0.740 respectively, with TPR around 35% and 28% at 1% FPR. The paper also analyzes distinctive tokens in commercial tokenizers and proposes a min-count defense that partially mitigates the attacks at some utility cost. The authors release code.

Significance. If the results hold, this is a useful first study of tokenizer-level membership leakage. The experimental protocol is careful in several ways: it includes a distribution-shift check (bag-of-words random forest, AUC 0.513, Figure 7), a utility comparison to commercial tokenizers (Figure 6), blind baselines, low-FPR ROC analysis, and open-sourced code. The core observation—that rare dataset-specific tokens are overfitted into BPE vocabularies—is plausible and consistent with existing evidence on distinctive tokens in commercial tokenizers. However, the real-world transfer of the attacks is not demonstrated: all end-to-end evaluations use target tokenizers trained by the authors on C4 with auxiliary data sampled from exactly the same distribution. The paper also does not position itself against the closely related prior work by Hayase et al. on BPE tokenizer leakage. These issues do not invalidate the in-distribution result, but they materially limit what can be claimed about state-of-the-art LLM tokenizers.

major comments (3)
  1. [Section 3, Section 5.1, Section 7] The threat model assumes the adversary can sample auxiliary datasets Daux from the same distribution D as the target tokenizer's training data. The distribution-shift check in §5.1 (BoW random forest, AUC 0.513, Figure 7) only validates that the random split into members/non-members is balanced; it does not test whether a realistic adversary can match the training distribution of a commercial tokenizer. §5.5 shows only that real-world tokenizers contain distinctive tokens, while §7 concedes that no end-to-end MIA evaluation on commercial tokenizers is possible due to missing ground truth. Consequently, the reported AUCs of 0.771 and 0.740 (Table 2) are upper bounds achieved under perfectly matched auxiliary data. Please add experiments with a mismatched auxiliary distribution (e.g., C4 target vs. auxiliary from a different corpus or a different mixture) or an end-to-end case study on an
  2. [Section 4.3, Table 1, Theorem 4.2] Frequency Estimation relies on the power-law assumption Pr(t_i|V) ∝ 1/i^α, with α and x_min fitted on a single shadow tokenizer and then applied to target tokenizers with different vocabulary sizes and potentially different training corpora. Table 1 reports α decreasing from 1.717 to 1.460 as vocabulary size grows, but no goodness-of-fit test or transfer analysis is given; the tiny standard errors only reflect MLE precision under the assumed model. Furthermore, Theorem 4.2 gives only a lower bound on the self-information term, and the attack in Eq. (12) substitutes that lower bound for the actual SI. If the power law does not transfer, the Frequency Estimation signal is a heuristic with no stated guarantee. Please add robustness tests (e.g., fit α on multiple shadow tokenizers and report ROC variability, or compare against direct frequency estimates when available) and clarify the role o
  3. [Section 1, Section 6, Refs [37,38]] The claim that 'the potential of tokenizers as an attack vector remains unexplored' (Section 1) is overstated given the prior work by Hayase et al. on data-mixture inference from BPE tokenizers, which the paper cites as [37,38] but does not compare against or explicitly differentiate from. Membership inference for a target dataset is closely related to mixture inference on training-data proportions, and the contribution needs to be positioned with respect to that work. Please add a direct comparison or an explicit technical distinction between the two problem settings.
minor comments (5)
  1. [Section 4.2, Eq. (3)] Equation (3) defines V_non using a set difference, but the preceding text and Algorithm 1 line 11 use an intersection. Please correct the equation to match the algorithm.
  2. [Section 4.3, Algorithm 2] Algorithm 2 line 7 trains V_shadow on D_aux after the sampling loop, but D_aux is only the last sampled auxiliary dataset. The main text says the shadow tokenizer is trained using 'a Daux ⊆ \tilde{D}', which is ambiguous. Clarify whether the shadow tokenizer is trained on one auxiliary dataset or on the union, and fix the pseudocode accordingly.
  3. [Section 5, Tables 2–3] Reported AUC/BA/TPR values have no confidence intervals or significance tests. Since the evaluation covers 4,133 datasets, standard errors or confidence intervals would help assess the stability of the rankings.
  4. [Figure 7] A t-SNE visualization is not a statistical confirmation of 'no distribution shifts'; the text should rely on the reported BoW RF AUC of 0.513 rather than the visual impression.
  5. [Section 1] Typo: 'toknizers' should be 'tokenizers'. Also, references [37] and [38] appear to be the same paper and should be merged.

Circularity Check

0 steps flagged

No significant circularity: the core signal is an empirical correlation evaluated on held-out target tokenizers; the only fitted parameter (the power-law exponent) is calibrated on an independent shadow tokenizer.

full rationale

I walked the derivation chain and found no step in which a claimed prediction is equivalent by construction to an input, or in which a load-bearing premise is imported from the authors' own prior work. The central hypothesis is stated as an empirical conjecture: "the more distinctive tokens from D that are found in Vtarget, the more likely it is that Vtarget was trained on D" (Section 4.2). The Vocabulary Overlap attack operationalizes this with shadow tokenizers trained with and without D; the target membership labels are not used to select distinctive tokens or set thresholds, and the AUC is measured on target tokenizers trained from half of the C4 datasets, so the result is a genuine out-of-sample correlation. The Frequency Estimation attack fits the power-law parameters on a single shadow tokenizer: Algorithm 2 line 9, "α,x_min ← pl.fit(V_shadow,D_aux)". Those parameters are then applied to the target vocabulary; target labels never influence the fit. Theorem 4.2 is a mathematical lower bound conditional on the stated power-law assumption (Appendix A), not a restatement of the empirical membership result. The paper explicitly acknowledges the main external-validity limitation in Section 7: "due to the absence of ground-truth training data for commercial tokenizers, we are unable to evaluate our attacks on them." This is a limitation about transfer to real-world commercial tokenizers, not circularity. The adversary's auxiliary-data assumption in Section 3 ("we assume that the adversary is able to sample auxiliary datasets Daux from the same distribution as the training data used by the target tokenizer") is likewise an explicit assumption, not a hidden equivalence. Self-citations appear only in related-work contexts (e.g., [23], [56], [108]) and are not used to justify the central derivation. No evidence of self-definition, fitted-input-renamed-as-prediction, or ansatz-smuggling was found.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 1 invented entities

The central attacks rest on three pillars: shadow tokenizers trained on auxiliary data from the same distribution, the distinctive-token overfitting hypothesis, and, for the cheap attack, a power-law model of token frequency fitted on shadow data. The power-law fit contributes genuine free parameters (α, x_min); the shadow sampling assumption is the most fragile structural premise.

free parameters (3)
  • alpha_power_law = 1.717 (80k vocab) to 1.460 (200k vocab), Table 1
    Exponent in the power-law estimate Pr(t_i|V_target) ∝ 1/i^α, fitted on a shadow tokenizer and used in the SI term of Equation 12. Attack accuracy depends on this fit transferring to the target tokenizer.
  • xmin_power_law_cutoff = 9,782 (Table 1)
    Lower bound for the power-law regime; only tokens with merge index i > x_min are used by MIA via Frequency Estimation. Chosen by the power-law fit on shadow data, not derived for the target.
  • N_shadow_tokenizers = 96 for Vocabulary Overlap; 10 sampling times for Frequency Estimation
    Hyperparameters chosen by the attacker; the ablation in Figure 10 shows performance plateaus as N grows, so the reported numbers depend on these hand-picked settings.
axioms (5)
  • domain assumption The tokenizer's training data is representative of the LLM pre-training corpus and is largely public web data.
    Used throughout Sections 1 and 3 to justify training shadow tokenizers on sampled web data; if real tokenizers are trained on proprietary mixtures, the shadow-model setup breaks.
  • domain assumption The adversary can access the target tokenizer's vocabulary and merge order.
    Section 3 assumes commercial LLMs such as OpenAI-o3, Gemini-1.5, and Claude-2 open-source tokenizers for billing transparency; this is true today but is a contingent business practice.
  • domain assumption The adversary can sample auxiliary datasets D_aux from the same distribution D as the target tokenizer's training data.
    Section 3, D_aux ← D. This is the load-bearing premise for shadow-tokenizer calibration; if the adversary cannot obtain representative auxiliary data, both shadow-based attacks lose their calibration signal.
  • domain assumption Token frequencies in training data follow a power law in merge index.
    Equation 9 and Figure 5/Table 1 in Section 4.3. The Frequency Estimation attack's self-information estimate is a lower bound that only holds under this empirical law; the fit is performed on shadow data and assumed to transfer.
  • ad hoc to paper Distinctive dataset-specific tokens overfit into the BPE vocabulary when that dataset is in training.
    The central hypothesis of Section 4.2: 'the more distinctive tokens from D that are found in V_target, the more likely it is that V_target was trained on D.' This is an empirical assumption about BPE behavior, supported by examples but not proven.
invented entities (1)
  • RTF-SI (Relative Token Frequency with Self-information) no independent evidence
    purpose: A new score, Definition 4.1, used as the membership signal for MIA via Frequency Estimation; it combines how much of a token's occurrences come from dataset D with how rare the token is in the target vocabulary.
    This is a new ledger entry introduced by the paper. It has no independent falsifiable handle outside the paper's own evaluation; its usefulness is measured only by the reported attack AUC.

pith-pipeline@v1.3.0-alltime-deepseek · 29353 in / 15512 out tokens · 141738 ms · 2026-08-04T11:17:08.649937+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Membership Inference Attacks on Tokenizers of Large Language Models." pith.science (2026). https://pith.science/paper/G3RRHNIE

@misc{pith2026251005699,
  author       = {Pith},
  title        = {Pith review of: Membership Inference Attacks on Tokenizers of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G3RRHNIE}},
  note         = {Machine review of arXiv:2510.05699}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Membership inference attacks (MIAs) are widely used to assess the privacy risks associated with machine learning models. However, when these attacks are applied to pre-trained large language models (LLMs), they encounter significant challenges, including mislabeled samples, distribution shifts, and discrepancies in model size between experimental and real-world settings. To address these limitations, we introduce tokenizers as a new attack vector for membership inference. Specifically, a tokenizer converts raw text into tokens for LLMs. Unlike full models, tokenizers can be efficiently trained from scratch, thereby avoiding the aforementioned challenges. In addition, the tokenizer's training data is typically representative of the data used to pre-train LLMs. Despite these advantages, the potential of tokenizers as an attack vector remains unexplored. To this end, we present the first study on membership leakage through tokenizers and explore five attack methods to infer dataset membership. Extensive experiments on millions of Internet samples reveal the vulnerabilities in the tokenizers of state-of-the-art LLMs. To mitigate this emerging risk, we further propose an adaptive defense. Our findings highlight tokenizers as an overlooked yet critical privacy threat, underscoring the urgent need for privacy-preserving mechanisms specifically designed for them.

Figures

Figures reproduced from arXiv: 2510.05699 by Kejiang Chen, Meng Tong, Ninghui Li, Weiming Zhang, Yuntao Du.

Figure 1
Figure 1. Figure 1: Evaluation challenges in MIAs against LLMs. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Performance of our MIAs on tokenizers of LLMs. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Average merge index for tokens in Vin and Vout. It is shown that overall merge orders in Vin and Vout resemble. solely examine those distinctive tokens whose merge index differs between the vocabularies Vin and Vout. Our analysis suggests that only when the tokenizer is trained on dataset D, some distinctive tokens in D are more likely to be over￾fit in its vocabulary. Typically, these distinctive tokens m… view at source ↗
Figure 4
Figure 4. Figure 4: Distinctive tokens in MIA via Vocabulary Overlap. where it ranges from 0 to 1. (v) If the membership signal is larger than a decision-making threshold τ, output 1 (member). Otherwise, output 0. The detailed process of this attack is outlined in Algo￾rithm 1. However, like other shadow-based MIAs [11,94,103], we find that MIA via Vocabulary Overlap requires multiple shadow tokenizers (e.g., 96) to effective… view at source ↗
Figure 5
Figure 5. Figure 5: Relationship between token merge index and fre [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of tokenizer utility based on the metric of bytes per token. Specifically, “Ours-80000” refers to our trained [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Visualization of the test set using t-SNE [ [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Success rate of our attacks on tokenizers with different vocabulary sizes. The experimental results demonstrate that, [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Distribution of members and non-members. have shown that increasing the amount of data used for mem￾bership inference can improve the attack performance. This finding is particularly relevant in the context of high-value litigation nowadays, where the datasets at stake are often mas￾sive [3, 73]. Motivated by this, we investigate whether MIAs can more effectively infer the membership of larger datasets fro… view at source ↗
Figure 10
Figure 10. Figure 10: Impact of N. Left: MIA via Vocabulary Overlap, training N shadow tokenizers. Right: MIA via Frequency Estimation, sampling auxiliary datasets N times. achieves a competitive AUC score of 0.843 [PITH_FULL_IMAGE:figures/full_fig_p011_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Comparison of tokenizers in real-world LLMs. [PITH_FULL_IMAGE:figures/full_fig_p012_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Distinctive tokens in MIA via Vocabulary Overlap. 10 −4 10 −3 10 −2 10 −1 10 0 False Positive Rate 10 −4 10 −3 10 −2 10 −1 10 0 True Positive Rate Vocabulary Overlap Frequency Estimation Merge Similarity Naive Bayes, k=20,000 Naive Bayes, k=40,000 Naive Bayes, k=60,000 Compression Rate (a) Vocabulary Size: 80, 000 10 −4 10 −3 10 −2 10 −1 10 0 False Positive Rate 10 −4 10 −3 10 −2 10 −1 10 0 True Positive … view at source ↗
Figure 13
Figure 13. Figure 13: ROC curves for MIAs using different methods. [PITH_FULL_IMAGE:figures/full_fig_p021_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Dataset distribution based on MIA via Vocabulary Overlap. (a) Vocabulary Size: 80, 000 (b) Vocabulary Size: 110, 000 (c) Vocabulary Size: 140, 000 (d) Vocabulary Size: 170, 000 (e) Vocabulary Size: 200, 000 [PITH_FULL_IMAGE:figures/full_fig_p022_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Dataset distribution based on MIA via Frequency Estimation. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Security Considerations for Multi-agent Systems

    cs.CR 2026-03 unverdicted novelty 6.0

    No existing AI security framework covers a majority of the 193 identified multi-agent system threats in any category, with OWASP Agentic Security Initiative achieving the highest overall coverage at 65.3%.

Reference graph

Works this paper leans on

110 extracted references · 19 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Deep learning with differential privacy

    Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. InCCS, pages 308–318, 2016

    2016

  2. [2]

    An information-theoretic perspective of tf–idf measures.Information Processing & Manage- ment, 39(1):45–65, 2003

    Akiko Aizawa. An information-theoretic perspective of tf–idf measures.Information Processing & Manage- ment, 39(1):45–65, 2003

    2003

  3. [3]

    Judge allows new york times copyright lawsuit to go forward

    Bobby Allyn. Judge allows new york times copyright lawsuit to go forward. www.npr.org/2025/03/26/nx- s1-5288157/new-york-times-openai-copyright-case- goes-forward, 2025

    2025

  4. [4]

    Tokenizer for anthropic large language models

    Anthropic. Tokenizer for anthropic large language models. Tokenizer for Use with Anthropic’s Models, 2024

    2024

  5. [5]

    Claude opus 4 & claude sonnet 4

    Anthropic. Claude opus 4 & claude sonnet 4. System Card, 2025. Anthropic System Card

    2025

  6. [6]

    An efficient recommendation generation us- ing relevant jaccard similarity.Information Sciences, 483:53–64, 2019

    Sujoy Bag, Sri Krishna Kumar, and Manoj Kumar Tiwari. An efficient recommendation generation us- ing relevant jaccard similarity.Information Sciences, 483:53–64, 2019

    2019

  7. [7]

    Deepseek llm: Scaling open- source language models with longtermism.arXiv preprint arXiv:2401.02954, 2024

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open- source language models with longtermism.arXiv preprint arXiv:2401.02954, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Pythia: A suite for analyzing large language models across train- ing and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gre- gory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across train- ing and scaling. InICML, pages 2397–2430. PMLR, 2023

    2023

  9. [9]

    Gpt- neox-20b: An open-source autoregressive language model.Challenges & Perspectives in Creating Large Language Models, page 95, 2022

    Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Con- nor Leahy, Kyle McDonell, Jason Phang, et al. Gpt- neox-20b: An open-source autoregressive language model.Challenges & Perspectives in Creating Large Language Models, page 95, 2022

    2022

  10. [10]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 33:1877–1901, 2020

    1901

  11. [11]

    Member- ship inference attacks from first principles

    Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Member- ship inference attacks from first principles. In2022 IEEE Symposium on Security and Privacy (SP), pages 1897–1914. IEEE, 2022

    1914

  12. [12]

    Extracting training data from large lan- guage models

    Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-V oss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlings- son, et al. Extracting training data from large lan- guage models. In30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650, 2021

    2021

  13. [13]

    André M Carrington, Douglas G Manuel, Paul W Fieguth, Tim Ramsay, Venet Osmani, Bernhard Wernly, Carol Bennett, Steven Hawken, Olivia Magwood, Yusuf Sheikh, et al. Deep roc analysis and auc as balanced average accuracy, for improved classifier se- lection, audit and explanation.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):329– 341, 2022

    2022

  14. [14]

    Evaluating the dynamics of membership privacy in deep learning

    Yuetian Chen, Zhiqi Wang, Nathalie Baracaldo, Swanand Ravindra Kadhe, and Lei Yu. Evaluating the dynamics of membership privacy in deep learning. arXiv preprint arXiv:2507.23291, 2025

    Pith/arXiv arXiv 2025

  15. [15]

    How contaminated is your benchmark? measuring dataset leakage in large language models with kernel divergence

    Hyeong Kyu Choi, Maxim Khanov, Hongxin Wei, and Yixuan Li. How contaminated is your benchmark? measuring dataset leakage in large language models with kernel divergence. InForty-second International Conference on Machine Learning, 2025

    2025

  16. [16]

    Label-only membership inference attacks

    Christopher A Choquette-Choo, Florian Tramer, Nicholas Carlini, and Nicolas Papernot. Label-only membership inference attacks. InICML, pages 1964–

    1964

  17. [17]

    Power-law distributions in empirical data

    Aaron Clauset, Cosma Rohilla Shalizi, and Mark EJ Newman. Power-law distributions in empirical data. SIAM review, 51(4):661–703, 2009

    2009

  18. [18]

    United States Code. U.s. copyright act, title 17, section

  19. [19]

    Getting the most out of your tokenizer for pre- training and domain adaptation

    Gautier Dagan, Gabriel Synnaeve, and Baptiste Roz- ière. Getting the most out of your tokenizer for pre- training and domain adaptation. InProceedings of the 41st International Conference on Machine Learning, pages 9784–9805, 2024

    2024

  20. [20]

    Blind baselines beat membership inference attacks for foun- dation models

    Debeshee Das, Jie Zhang, and Florian Trantèr. Blind baselines beat membership inference attacks for foun- dation models. In2025 IEEE Security and Privacy Workshops (SPW), pages 118–125. IEEE, 2025

    2025

  21. [21]

    Bert: Pre-training of deep bidi- rectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In 14 Proceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, 2019

    2019

  22. [22]

    Dp-forward: Fine-tuning and inference on language models with differential privacy in forward pass

    Minxin Du, Xiang Yue, Sherman SM Chow, Tianhao Wang, Chenyu Huang, and Huan Sun. Dp-forward: Fine-tuning and inference on language models with differential privacy in forward pass. InProceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pages 2665–2679, 2023

    2023

  23. [23]

    Cascading and Proxy Membership Infer- ence Attacks

    Yuntao Du, Jiacheng Li, Yuetian Chen, Kaiyuan Zhang, Zhizhen Yuan, Hanshen Xiao, Bruno Ribeiro, and Ninghui Li. Cascading and Proxy Membership Infer- ence Attacks. In33th Annual Network and Distributed System Security Symposium (NDSS), 2026

    2026

  24. [24]

    Systematic assessment of tabular data synthesis algorithms.arXiv preprint arXiv:2402.06806, 2024

    Yuntao Du and Ninghui Li. Systematic assessment of tabular data synthesis algorithms.arXiv preprint arXiv:2402.06806, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    Do membership inference attacks work on large language models? InFirst Conference on Lan- guage Modeling, 2024

    Michael Duan, Anshuman Suri, Niloofar Mireshghal- lah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Ha- jishirzi. Do membership inference attacks work on large language models? InFirst Conference on Lan- guage Modeling, 2024

    2024

  26. [26]

    De-cop: detecting copyrighted content in language models training data

    André V Duarte, Xuandong Zhao, Arlindo L Oliveira, and Lei Li. De-cop: detecting copyrighted content in language models training data. InProceedings of the 41st International Conference on Machine Learning, pages 11940–11956, 2024

    2024

  27. [27]

    Differential privacy

    Cynthia Dwork. Differential privacy. InInternational colloquium on automata, languages, and programming, pages 1–12. Springer, 2006

    2006

  28. [28]

    Analysis of sparse bayesian learning.Advances in neural information processing systems, 14, 2001

    Anita Faul and Michael Tipping. Analysis of sparse bayesian learning.Advances in neural information processing systems, 14, 2001

    2001

  29. [29]

    Privacy in pharmacogenetics: An {End-to-End} case study of personalized warfarin dosing

    Matthew Fredrikson, Eric Lantz, Somesh Jha, Simon Lin, David Page, and Thomas Ristenpart. Privacy in pharmacogenetics: An {End-to-End} case study of personalized warfarin dosing. In23rd USENIX security symposium (USENIX Security 14), pages 17–32, 2014

    2014

  30. [30]

    Label inference attacks against vertical federated learning

    Chong Fu, Xuhong Zhang, Shouling Ji, Jinyin Chen, Jingzheng Wu, Shanqing Guo, Jun Zhou, Alex X Liu, and Ting Wang. Label inference attacks against vertical federated learning. In31st USENIX security sympo- sium (USENIX Security 22), pages 1397–1414, 2022

    2022

  31. [31]

    Zipf’s law and the growth of cities

    Xavier Gabaix. Zipf’s law and the growth of cities. American Economic Review, 89(2):129–132, 1999

    1999

  32. [32]

    Investigating the effectiveness of bpe: The power of shorter sequences

    Matthias Gallé. Investigating the effectiveness of bpe: The power of shorter sequences. InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP- IJCNLP), pages 1375–1381, 2019

    2019

  33. [33]

    Counting gemini text tokens locally with the vertex ai sdk, July 2024

    Google. Counting gemini text tokens locally with the vertex ai sdk, July 2024. Tokenizer for Use with Google’s Models

    2024

  34. [34]

    Likelihood-based diffusion language models.Ad- vances in Neural Information Processing Systems, 36:16693–16715, 2023

    Ishaan Gulrajani and Tatsunori B Hashimoto. Likelihood-based diffusion language models.Ad- vances in Neural Information Processing Systems, 36:16693–16715, 2023

    2023

  35. [35]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  36. [36]

    Weird gpt-4 behavior for “davidjl”

    Hacker News. Weird gpt-4 behavior for “davidjl”. news.ycombinator.com/item?id=36242914, 2023

    2023

  37. [37]

    Data mixture inference attack: Bpe tokenizers reveal training data compositions.Advances in Neural Information Processing Systems, 37:8956– 8983, 2024

    Jonathan Hayase, Alisa Liu, Yejin Choi, Sewoong Oh, and Noah A Smith. Data mixture inference attack: Bpe tokenizers reveal training data compositions.Advances in Neural Information Processing Systems, 37:8956– 8983, 2024

    2024

  38. [38]

    Data mixture inference attack: Bpe tokenizers reveal training data compositions

    Jonathan Hayase, Alisa Liu, Yejin Choi, Sewoong Oh, and Noah A Smith. Data mixture inference attack: Bpe tokenizers reveal training data compositions. InThe Thirty-eighth Annual Conference on Neural Informa- tion Processing Systems, 2024

    2024

  39. [39]

    Strong membership inference attacks on massive datasets and (moderately) large language models.arXiv preprint arXiv:2505.18773, 2025

    Jamie Hayes, Ilia Shumailov, Christopher A Choquette- Choo, Matthew Jagielski, George Kaissis, Kather- ine Lee, Milad Nasr, Sahra Ghalebikesabi, Niloofar Mireshghallah, Meenatchi Sundaram Mutu Selva An- namalai, et al. Strong membership inference attacks on massive datasets and (moderately) large language models.arXiv preprint arXiv:2505.18773, 2025

    arXiv 2025

  40. [40]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 770–778, 2016

    2016

  41. [41]

    To- wards label-only membership inference attack against pre-trained large language models

    Yu He, Boheng Li, Liu Liu, Zhongjie Ba, Wei Dong, Yiming Li, Zhan Qin, Kui Ren, and Chun Chen. To- wards label-only membership inference attack against pre-trained large language models. In34th USENIX Security Symposium (USENIX Security 25), 2025. 15

    2025

  42. [42]

    Membership inference attacks against vision-language models

    Yuke He, Zheng Li, Yang Zhang, Zhan Qin, Kui Ren, and Chun Chen. Membership inference attacks against vision-language models. In34th USENIX Security Symposium (USENIX Security 25), 2025

    2025

  43. [43]

    Scaling laws for autoregressive generative modeling.arXiv preprint arXiv:2010.14701, 2020

    Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling.arXiv preprint arXiv:2010.14701, 2020

    Pith/arXiv arXiv 2010

  44. [44]

    Training compute- optimal large language models.arXiv preprint arXiv:2203.15556, 2022

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Men- sch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Jo- hannes Welbl, Aidan Clark, et al. Training compute- optimal large language models.arXiv preprint arXiv:2203.15556, 2022

    Pith/arXiv arXiv 2022

  45. [45]

    Damia: Leveraging domain adaptation as a defense against membership inference attacks.IEEE Transactions on Dependable and Secure Computing, 19(5):3183–3199, 2021

    Hongwei Huang, Weiqi Luo, Guoqiang Zeng, Jian Weng, Yue Zhang, and Anjia Yang. Damia: Leveraging domain adaptation as a defense against membership inference attacks.IEEE Transactions on Dependable and Secure Computing, 19(5):3183–3199, 2021

    2021

  46. [46]

    Over-tokenized trans- former: V ocabulary is generally worth scaling

    Hongzhi Huang, Defa Zhu, Banggu Wu, Yutao Zeng, Ya Wang, Qiyang Min, et al. Over-tokenized trans- former: V ocabulary is generally worth scaling. In Forty-second International Conference on Machine Learning, 2025

    2025

  47. [47]

    Efficient reasoning for large reasoning language models via certainty-guided reflec- tion suppression.arXiv preprint arXiv:2508.05337, 2025

    Jiameng Huang, Baijiong Lin, Guhao Feng, Jierun Chen, Di He, and Lu Hou. Efficient reasoning for large reasoning language models via certainty-guided reflec- tion suppression.arXiv preprint arXiv:2508.05337, 2025

    arXiv 2025

  48. [48]

    Tokenizer

    Hugging Face. Tokenizer. https://huggingface.co/docs/ transformers/main_classes/tokenizer, 2025

    2025

  49. [49]

    Codeparrot github code dataset

    Hugging Face Datasets. Codeparrot github code dataset. https://huggingface.co/datasets/codeparrot/ github-code, 2025

    2025

  50. [50]

    Practical blind membership inference attack via differential compar- isons

    Bo Hui, Yuchen Yang, Haolin Yuan, Philippe Burlina, Neil Zhenqiang Gong, and Yinzhi Cao. Practical blind membership inference attack via differential compar- isons. InISOC Network and Distributed System Secu- rity Symposium (NDSS), 2021

    2021

  51. [51]

    Memguard: Defend- ing against black-box membership inference attacks via adversarial examples

    Jinyuan Jia, Ahmed Salem, Michael Backes, Yang Zhang, and Neil Zhenqiang Gong. Memguard: Defend- ing against black-box membership inference attacks via adversarial examples. InCCS, pages 259–274, 2019

    2019

  52. [52]

    Scal- ing laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scal- ing laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Pith/arXiv arXiv 2001

  53. [53]

    Stolen memories: Leveraging model memorization for calibrated{White- Box} membership inference

    Klas Leino and Matt Fredrikson. Stolen memories: Leveraging model memorization for calibrated{White- Box} membership inference. In29th USENIX security symposium (USENIX Security 20), pages 1605–1622, 2020

    2020

  54. [54]

    Se- qmia: sequential-metric based membership inference attack

    Hao Li, Zheng Li, Siyuan Wu, Chengrui Hu, Yutong Ye, Min Zhang, Dengguo Feng, and Yang Zhang. Se- qmia: sequential-metric based membership inference attack. InProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Secu- rity, pages 3496–3510, 2024

    2024

  55. [55]

    Enhanced label-only membership inference attacks with fewer queries

    Hao Li, Zheng Li, Siyuan Wu, Yutong Ye, Min Zhang, Dengguo Feng, and Yang Zhang. Enhanced label-only membership inference attacks with fewer queries. In Proceedings of the 34th USENIX Security Symposium (USENIX Security ’25). USENIX Association, 2025

    2025

  56. [56]

    Mem- bership inference attacks and defenses in classification models

    Jiacheng Li, Ninghui Li, and Bruno Ribeiro. Mem- bership inference attacks and defenses in classification models. InProceedings of the Eleventh ACM Confer- ence on Data and Application Security and Privacy, pages 5–16, 2021

    2021

  57. [57]

    Large language models can be strong differentially private learners.arXiv preprint arXiv:2110.05679, 2021

    Xuechen Li, Florian Tramer, Percy Liang, and Tat- sunori Hashimoto. Large language models can be strong differentially private learners.arXiv preprint arXiv:2110.05679, 2021

    Pith/arXiv arXiv 2021

  58. [58]

    Membership leakage in label-only exposures

    Zheng Li and Yang Zhang. Membership leakage in label-only exposures. InCCS, pages 880–895, 2021

    2021

  59. [59]

    SuperBPE: Space travel for language models

    Alisa Liu, Jonathan Hayase, Valentin Hofmann, Se- woong Oh, Noah A Smith, and Yejin Choi. SuperBPE: Space travel for language models. InSecond Confer- ence on Language Modeling, 2025

    2025

  60. [60]

    Please tell me more: Privacy impact of explainability through the lens of membership inference attack

    Han Liu, Yuhao Wu, Zhiyuan Yu, and Ning Zhang. Please tell me more: Privacy impact of explainability through the lens of membership inference attack. In 2024 IEEE Symposium on Security and Privacy (SP), pages 4791–4809. IEEE, 2024

    2024

  61. [61]

    Roberta: A ro- bustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692, 2019

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A ro- bustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692, 2019

    Pith/arXiv arXiv 1907

  62. [62]

    Visu- alizing data using t-sne.Journal of machine learning research, 9(Nov):2579–2605, 2008

    Laurens van der Maaten and Geoffrey Hinton. Visu- alizing data using t-sne.Journal of machine learning research, 9(Nov):2579–2605, 2008. 16

    2008

  63. [63]

    Llm dataset inference: Did you train on my dataset?Advances in Neural Information Pro- cessing Systems, 37:124069–124092, 2024

    Pratyush Maini, Hengrui Jia, Nicolas Papernot, and Adam Dziedzic. Llm dataset inference: Did you train on my dataset?Advances in Neural Information Pro- cessing Systems, 37:124069–124092, 2024

    2024

  64. [64]

    Dataset inference: Ownership resolution in machine learning.arXiv preprint arXiv:2104.10706, 2021

    Pratyush Maini, Mohammad Yaghini, and Nicolas Pa- pernot. Dataset inference: Ownership resolution in machine learning.arXiv preprint arXiv:2104.10706, 2021

    Pith/arXiv arXiv 2021

  65. [65]

    Tokens used by gpt-4 probably come from the reddit users

    Matthew Watkins. Tokens used by gpt-4 probably come from the reddit users. https://x.com/SoC_trilogy /status/1666714127438434304, 2023

    arXiv 2023

  66. [66]

    LLMs on the line: Data determines loss-to-loss scaling laws

    Prasanna Mayilvahanan, Thaddus Wiedemer, Sayak Mallick, Matthias Bethge, and Wieland Brendel. LLMs on the line: Data determines loss-to-loss scaling laws. InForty-second International Conference on Machine Learning, 2025

    2025

  67. [67]

    Did the neurons read your book? document-level membership inference for large language models

    Matthieu Meeus, Shubham Jain, Marek Rei, and Yves- Alexandre de Montjoye. Did the neurons read your book? document-level membership inference for large language models. In33rd USENIX Security Sympo- sium (USENIX Security 24), pages 2369–2385, 2024

    2024

  68. [68]

    Sok: Membership inference attacks on llms are rushing nowhere (and how to fix it)

    Matthieu Meeus, Igor Shilov, Shubham Jain, Manuel Faysse, Marek Rei, and Yves-Alexandre de Montjoye. Sok: Membership inference attacks on llms are rushing nowhere (and how to fix it). InIEEE Conference on Secure and Trustworthy Machine Learning ((SaTML, 2025). IEEE, 2025

    2025

  69. [69]

    Pointer sentinel mixture models.arXiv preprint arXiv:1609.07843, 2016

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models.arXiv preprint arXiv:1609.07843, 2016

    Pith/arXiv arXiv 2016

  70. [70]

    Large language models: A survey.arXiv preprint arXiv:2402.06196, 2024

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatri- ain, and Jianfeng Gao. Large language models: A survey.arXiv preprint arXiv:2402.06196, 2024

    Pith/arXiv arXiv 2024

  71. [71]

    Scaling data-constrained language models.Advances in Neural Information Processing Systems, 36:50358– 50376, 2023

    Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models.Advances in Neural Information Processing Systems, 36:50358– 50376, 2023

    2023

  72. [72]

    Com- prehensive privacy analysis of deep learning: Passive and active white-box inference attacks against central- ized and federated learning

    Milad Nasr, Reza Shokri, and Amir Houmansadr. Com- prehensive privacy analysis of deep learning: Passive and active white-box inference attacks against central- ized and federated learning. In2019 IEEE symposium on security and privacy (SP), pages 739–753. IEEE, 2019

    2019

  73. [73]

    Reddit sues anthropic over its data scraping to train large language models

    New York Times. Reddit sues anthropic over its data scraping to train large language models. https:// www.nytimes.com/2025/06/04/technology/reddit- anthropic-lawsuit-data.html, 2025

    2025

  74. [74]

    System card of chatgpt-o1

    OpenAI. System card of chatgpt-o1. https://cdn.openai. com/o1-system-card-20241205.pdf, 2024

    2024

  75. [75]

    tiktoken: Tokenizer for openai models

    OpenAI. tiktoken: Tokenizer for openai models. https: //github.com/openai/tiktoken, 2025

    2025

  76. [76]

    Black-box membership inference attacks against fine-tuned diffusion models

    Yan Pang and Tianhao Wang. Black-box membership inference attacks against fine-tuned diffusion models. arXiv preprint arXiv:2312.08207, 2023

    Pith/arXiv arXiv 2023

  77. [77]

    White-box membership inference attacks against diffusion models.arXiv preprint arXiv:2308.06405, 2023

    Yan Pang, Tianhao Wang, Xuhui Kang, Mengdi Huai, and Yang Zhang. White-box membership inference attacks against diffusion models.arXiv preprint arXiv:2308.06405, 2023

    Pith/arXiv arXiv 2023

  78. [78]

    Zipf’s word frequency law in natural language: A critical review and future direc- tions.Psychonomic bulletin & review, 21(5):1112– 1130, 2014

    Steven T Piantadosi. Zipf’s word frequency law in natural language: A critical review and future direc- tions.Psychonomic bulletin & review, 21(5):1112– 1130, 2014

    2014

  79. [79]

    Scaling up membership inference: When and how attacks succeed on large language mod- els

    Haritz Puerto, Martin Gubri, Sangdoo Yun, and Seong Joon Oh. Scaling up membership inference: When and how attacks succeed on large language mod- els. In Luis Chiruzzo, Alan Ritter, and Lu Wang, ed- itors,Findings of the Association for Computational Linguistics: NAACL 2025, pages 4165–4182, Albu- querque, New Mexico, April 2025. Association for Computati...

    2025

  80. [80]

    Text mining: use of tf-idf to examine the relevance of words to docu- ments.International journal of computer applications, 181(1):25–29, 2018

    Shahzad Qaiser and Ramsha Ali. Text mining: use of tf-idf to examine the relevance of words to docu- ments.International journal of computer applications, 181(1):25–29, 2018

    2018

Showing first 80 references.