{"id":"e6e4cfe2-22ee-48dc-a245-0645c9ef697d","arxiv_id":"2510.05699","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Tokenizers leak dataset membership: using distinctive tokens that appear in a dataset's text, an attacker can detect with AUC up to 0.77 whether that dataset was part of tokenizer training.","lead":"This paper shows that an adversary can tell whether a specific website-scale dataset was used to train a large language model by inspecting the model's tokenizer—the component that splits text into tokens and is often published openly. It reports attack AUCs up to 0.77 and describes a defense that only partially reduces the leak.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Real-world transfer unverified: attacks assume auxiliary data drawn from the exact target training distribution","rationale":"The reader identified the power-law transfer assumption as the weakest point. While that is a legitimate concern for Frequency Estimation, it is not the most load-bearing for the central claim: even if the power law fails, Vocabulary Overlap (AUC 0.771) still supports the existence of membership leakage. The distribution-mismatch concern, by contrast, threatens both main attacks and the paper's real-world conclusion. In the evaluation, the adversary has oracle access to the true underlying distribution D; Daux is sampled from the same distribution as the tokenizer's training data. This is a strong idealization. In practice, an attacker targeting OpenAI or DeepSeek tokenizers must approximate their undisclosed training distributions with public data like C4. If the shadow tokenizers learn different distinctive tokens than the target has, the overlap signal is lost. The paper's Figure 11 shows real tokenizers differ, but does not connect them to shadow predictions. Section 7 acknowledges this limitation but does not test robustness to it. Therefore, the conditional acceptance should remain, with a requirement to add a distribution-shift experiment between Daux and the target training distribution. This is a concrete, addressable check, not a fatal flaw. I agree with the reader's overall conditional verdict, and disagree with the specific location of the weakest assumption.","tokens_in":29743,"tokens_out":14289,"duration_ms":109911,"concrete_test":"Train a target tokenizer on distribution A (e.g., 2,066 C4 English websites as in the paper). Then run both MIA via Vocabulary Overlap and MIA via Frequency Estimation against this tokenizer using auxiliary data Daux sampled from a different distribution B (e.g., Wikipedia, or C4 non-English/Code). Use the same 4,133-dataset evaluation protocol and metrics (AUC, TPR@1% FPR). If AUC drops to near 0.5 under a moderate shift, the attacks' practical applicability to real-world tokenizers with unknown training distributions is not established. A second step: repeat with Daux as a mixture of A and B to quantify sensitivity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that tokenizers' publicly available vocabularies leak dataset membership—is supported only under the assumption that the adversary can sample auxiliary datasets Daux from exactly the same distribution D as the target tokenizer's training corpus. Both main attacks (Vocabulary Overlap, §4.2; Frequency Estimation, §4.3) rely on shadow tokenizers trained on Daux to identify distinctive tokens and estimate their statistics. The paper's only distribution-shift check (Figure 7, AUC 0.513) tests whether member and non-member datasets are distinguishable by bag-of-words, not whether Daux matches the target's training distribution. Section 5.5 demonstrates that real-world tokenizers contain distinctive tokens, but does not show that shadow tokenizers trained on a public corpus such as C4 can reliably predict those tokens for an unknown target. Section 7 explicitly acknowledges that the attacks cannot be evaluated on commercial tokenizers due to missing ground truth. If the distinctive-token signal is sensitive to corpus mixture, language, or preprocessing differences—as is likely—the reported AUCs (0.771 and 0.740) may represent an upper bound that does not transfer to the claimed real-world targets. This gap is load-bearing because the abstract and title assert vulnerabilities in state-of-the-art LLMs, yet the only end-to-end attack demonstrations are on tokenizers trained by the authors on C4 subsets with perfectly matched auxiliary data.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces tokenizers as a new attack surface for membership inference against LLMs. It proposes five set-level MIA methods: Merge Similarity, Vocabulary Overlap, Frequency Estimation, Naive Bayes, and Compression Rate. The two main attacks—Vocabulary Overlap and Frequency Estimation—use shadow tokenizers trained on auxiliary data drawn from the same distribution as the target tokenizer's training corpus. Experiments on C4 website datasets with tokenizer vocabulary sizes 80k–200k report AUC up to 0.771 and 0.740 respectively, with TPR around 35% and 28% at 1% FPR. The paper also analyzes distinctive tokens in commercial tokenizers and proposes a min-count defense that partially mitigates the attacks at some utility cost. The authors release code.","tokens_in":30043,"tokens_out":9900,"duration_ms":85532,"significance":"If the results hold, this is a useful first study of tokenizer-level membership leakage. The experimental protocol is careful in several ways: it includes a distribution-shift check (bag-of-words random forest, AUC 0.513, Figure 7), a utility comparison to commercial tokenizers (Figure 6), blind baselines, low-FPR ROC analysis, and open-sourced code. The core observation—that rare dataset-specific tokens are overfitted into BPE vocabularies—is plausible and consistent with existing evidence on distinctive tokens in commercial tokenizers. However, the real-world transfer of the attacks is not demonstrated: all end-to-end evaluations use target tokenizers trained by the authors on C4 with auxiliary data sampled from exactly the same distribution. The paper also does not position itself against the closely related prior work by Hayase et al. on BPE tokenizer leakage. These issues do not invalidate the in-distribution result, but they materially limit what can be claimed about state-of-the-art LLM tokenizers.","major_comments":[{"comment":"The threat model assumes the adversary can sample auxiliary datasets Daux from the same distribution D as the target tokenizer's training data. The distribution-shift check in §5.1 (BoW random forest, AUC 0.513, Figure 7) only validates that the random split into members/non-members is balanced; it does not test whether a realistic adversary can match the training distribution of a commercial tokenizer. §5.5 shows only that real-world tokenizers contain distinctive tokens, while §7 concedes that no end-to-end MIA evaluation on commercial tokenizers is possible due to missing ground truth. Consequently, the reported AUCs of 0.771 and 0.740 (Table 2) are upper bounds achieved under perfectly matched auxiliary data. Please add experiments with a mismatched auxiliary distribution (e.g., C4 target vs. auxiliary from a different corpus or a different mixture) or an end-to-end case study on an","section":"Section 3, Section 5.1, Section 7"},{"comment":"Frequency Estimation relies on the power-law assumption Pr(t_i|V) ∝ 1/i^α, with α and x_min fitted on a single shadow tokenizer and then applied to target tokenizers with different vocabulary sizes and potentially different training corpora. Table 1 reports α decreasing from 1.717 to 1.460 as vocabulary size grows, but no goodness-of-fit test or transfer analysis is given; the tiny standard errors only reflect MLE precision under the assumed model. Furthermore, Theorem 4.2 gives only a lower bound on the self-information term, and the attack in Eq. (12) substitutes that lower bound for the actual SI. If the power law does not transfer, the Frequency Estimation signal is a heuristic with no stated guarantee. Please add robustness tests (e.g., fit α on multiple shadow tokenizers and report ROC variability, or compare against direct frequency estimates when available) and clarify the role o","section":"Section 4.3, Table 1, Theorem 4.2"},{"comment":"The claim that 'the potential of tokenizers as an attack vector remains unexplored' (Section 1) is overstated given the prior work by Hayase et al. on data-mixture inference from BPE tokenizers, which the paper cites as [37,38] but does not compare against or explicitly differentiate from. Membership inference for a target dataset is closely related to mixture inference on training-data proportions, and the contribution needs to be positioned with respect to that work. Please add a direct comparison or an explicit technical distinction between the two problem settings.","section":"Section 1, Section 6, Refs [37,38]"}],"minor_comments":[{"comment":"Equation (3) defines V_non using a set difference, but the preceding text and Algorithm 1 line 11 use an intersection. Please correct the equation to match the algorithm.","section":"Section 4.2, Eq. (3)"},{"comment":"Algorithm 2 line 7 trains V_shadow on D_aux after the sampling loop, but D_aux is only the last sampled auxiliary dataset. The main text says the shadow tokenizer is trained using 'a Daux ⊆ \\tilde{D}', which is ambiguous. Clarify whether the shadow tokenizer is trained on one auxiliary dataset or on the union, and fix the pseudocode accordingly.","section":"Section 4.3, Algorithm 2"},{"comment":"Reported AUC/BA/TPR values have no confidence intervals or significance tests. Since the evaluation covers 4,133 datasets, standard errors or confidence intervals would help assess the stability of the rankings.","section":"Section 5, Tables 2–3"},{"comment":"A t-SNE visualization is not a statistical confirmation of 'no distribution shifts'; the text should rely on the reported BoW RF AUC of 0.513 rather than the visual impression.","section":"Figure 7"},{"comment":"Typo: 'toknizers' should be 'tokenizers'. Also, references [37] and [38] appear to be the same paper and should be merged.","section":"Section 1"}],"recommendation":"major_revision","confidential_remarks":"The transfer gap is the main scientific concern. If the authors can add an end-to-end evaluation on a tokenizer with known training data or demonstrate robustness to auxiliary-distribution mismatch, the paper could become a solid contribution. The novelty overlap with Hayase et al. also needs to be addressed clearly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the core result is solid: a BPE tokenizer's vocabulary does leak set-level membership when the adversary has shadow data from the same distribution, and the two main attacks (Vocabulary Overlap and Frequency Estimation) beat reasonable baselines across vocabulary sizes. Second, the paper's real-world framing outruns the evidence—every headline AUC comes from tokenizers the authors trained on C4, not from any actual LLM tokenizer, because ground truth is unavailable. That gap is the whole story.\n\nWhat is new: framing tokenizer membership inference as a distinct attack surface; the vocabulary-overlap signal via distinctive tokens; the RTF-SI frequency-estimation attack with a power-law lower bound; and the observation that larger vocabularies are more leaky. The evaluation is more careful than most MIA papers: bag-of-words distribution-shift check (AUC 0.513), low-FPR ROC analysis, blind baselines (compression rate, Naive Bayes), and utility comparisons to commercial tokenizers. The Appendix A proof is a valid lower bound given the power-law premise. Credit where due: this is a genuinely new attack vector with honest negative results (merge similarity doesn't work).\n\nSoft spots. The main one is the Daux ~ D assumption. The threat model in Section 3 gives the adversary auxiliary data drawn from the exact training distribution. That assumption is doing real work: both attacks need the shadow tokenizers to surface the same distinctive tokens the target would, which depends on corpus mixture, language, and preprocessing. Section 5.5 only shows real tokenizers differ from each other; it does not show a C4-trained shadow predicts a production tokenizer. Section 7 admits they cannot evaluate on commercial tokenizers. So the abstract's 'vulnerabilities in state-of-the-art LLMs' is an extrapolation, not a measurement. The mechanism is demonstrated; the deployment is not.\n\nMinor issues: Equation (3) defines Vnon as set difference while Algorithm 1 uses intersection—one is a typo, but it matters for reproduction. No error bars anywhere. And 'first study' overclaims given Hayase et al.'s BPE mixture inference (NeurIPS 2024), which should have been a baseline or at least a careful positioning.\n\nBottom line: worth serious peer review. As a controlled measurement of a new leakage channel, it's a useful contribution; the real-world section should be rewritten as an open problem, not a finding.","headline":"Tokenizer membership inference is real and carefully measured under a same-distribution shadow assumption, but the real-world LLM claims outrun the evidence.","tokens_in":30578,"tokens_out":3065,"would_cite":true,"duration_ms":22507,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A tokenizer's public vocabulary can reveal which dataset trained it","keywords":["tokenizer","membership inference","byte-pair encoding","vocabulary leakage","large language models","dataset membership","shadow tokenizers","power law"],"falsifier":"Train a tokenizer on a dataset D, then postprocess its vocabulary to remove every token whose total count in the full training corpus is below a high threshold (e.g., 100), ensuring that no distinctive tokens from D survive. If the Vocabulary Overlap attack still distinguishes D from non-members above chance, the distinctive-token hypothesis is wrong or incomplete. Conversely, apply the attack to a tokenizer trained on a corpus with all rare tokens removed at train time and measure AUC; if it stays at chance, the signal's dependence on distinctive tokens is confirmed.","tokens_in":29588,"feed_emoji":"🔓","tokens_out":4516,"duration_ms":36156,"temperature":0.7,"pith_summary":"The paper establishes a new attack vector for membership inference against large language models: the tokenizer. Because tokenizers are trained on web-scale text and their vocabularies are publicly released for billing transparency, the authors argue that the presence of dataset-distinctive tokens in a target vocabulary leaks whether that dataset was used in the tokenizer's training. They propose five attacks, two of which—Vocabulary Overlap and Frequency Estimation—reach AUC scores around 0.74–0.77 on a 200,000-token tokenizer. They also show that larger vocabularies and larger target datasets make the leakage stronger, and that a simple rare-token removal defense (min count) only partially mitigates the risk. The significance is that tokenizer artifacts, often overlooked, become a practical privacy and copyright-evidence concern for open-sourced components.","feed_headline":"Tokenizers leak which datasets trained an LLM","feed_subtitle":"Public vocabularies reveal membership with AUC up to 0.77; larger tokenizers leak more.","key_machinery":"The load-bearing object is the tokenizer vocabulary with its merge indices, produced by byte-pair encoding (BPE). Distinctive tokens—strings merged only because they appear in the target dataset—are the signal. Vocabulary Overlap uses a set-overlap count between shadow vocabularies trained with and without D after subtracting tokens common to both. Frequency Estimation uses the RTF-SI score, where the self-information term is estimated by fitting a power-law tail to token counts versus merge index on a single shadow tokenizer, then bounding SI from below via the theorem RTF-SI ≥ RTF · log(Σ i^α / j^α). The max RTF-SI over late-merge tokens is the membership signal.","core_discovery":"The central claim is the distinctive-token hypothesis: the more distinctive tokens from dataset D that appear in the target tokenizer's vocabulary V, the more likely V was trained on D. The paper operationalizes this with two complementary attacks: a shadow-tokenizer approach that measures set overlap of distinctive tokens after removing non-distinctive tokens, and a shadow-free approach that computes a new metric, RTF-SI (Relative Token Frequency with Self-information), which scores each late-merge token by the product of its relative frequency in D and the estimated self-information from a power-law frequency model. Both attacks target set-level membership (which datasets, not which docume","pith_inferences":["Because tokenizer vocabularies are published as static artifacts and do not change after release, membership inference on tokenizers may be immune to defenses that perturb model outputs—so organizations should treat vocabularies as sensitive even if the LLM weights are public.","The power-law assumption could be tested against alternative frequency estimators (e.g., neural or count-based) on out-of-distribution data, and the attack methodology may transfer to other learned subword vocabularies (SentencePiece, Unigram) if the same distinctive-token phenomenon appears.","The distinctive-token signal might be exploitable for the inverse task: given a candidate dataset, identify which tokenizer (and thus which model) was trained on it—a tokenizer-owner attribution tool.","Deduplication or decontamination of web corpora would likely reduce distinctive tokens; if so, the attack's success rate on a particular corpus could serve as a measure of contamination."],"forward_implications":["Anyone with access to an open-sourced tokenizer can test whether a candidate dataset—such as a copyrighted corpus or private user collection—was used in LLM training.","As tokenizer vocabularies grow for better compression and model quality, membership leakage will increase, making future tokenizers more vulnerable, not less.","Larger datasets are easier to confirm as members, so the highest-value legal claims (massive corpora) are the ones the attack handles best.","Removing infrequent tokens (min count defense) lowers attack accuracy but also lowers compression efficiency; the attack remains above chance for large datasets even under strict filtering.","The power-law frequency model gives an efficient shadow-free attack: one shadow tokenizer and minutes of inference instead of dozens of shadow tokenizers and hours."],"fun_headline_variants":["Tokenizers leak LLM training sets","LLM tokenizers expose membership","New attack: tokenizer vocab reveals LLM data","Tokenizers are a privacy risk for LLMs","Distinctive tokens leak LLM training data"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"For the shadow-free Frequency Estimation attack, the load-bearing premise is that the tail of token frequencies in the target tokenizer's training data follows the same power-law distribution with parameters fitted on a single shadow tokenizer; if that transfer fails (different vocabulary sizes, corpus mixtures, or BPE implementations), the estimated self-information is a heuristic with no stated guarantee.","fun_headline_variants_meta":{"raw":{"variants":["Tokenizers leak LLM training sets","LLM tokenizers expose membership","New attack: tokenizer vocab reveals LLM data","Tokenizers are a privacy risk for LLMs","Distinctive tokens leak LLM training data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00088,"raw_usage":{"total_tokens":3634,"prompt_tokens":730,"completion_tokens":2904,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":474,"completion_tokens_details":{"reasoning_tokens":2837}},"tokens_in":474,"tokens_out":2904,"duration_ms":16440,"temperature":1.0,"reasoning_tokens":2837,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T11:17:08.649937+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a tokenizer on a dataset D, then postprocess its vocabulary to remove every token whose total count in the full training corpus is below a high threshold (e.g., 100), ensuring that no distinctive tokens from D survive. If the Vocabulary Overlap attack still distinguishes D from non-members above chance, the distinctive-token hypothesis is wrong or incomplete. Conversely, apply the attack to a tokenizer trained on a corpus with all rare tokens removed at train time and measure AUC; if it stays at chance, the signal's dependence on distinctive tokens is confirmed.","supporting_citations":[],"review_version":1}