{"id":"a48f5ee0-cf7d-4cc3-8c97-abda7279bcd9","arxiv_id":"2608.10690","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Released BPE vocabularies can be used to estimate per-token frequency ratios of a hidden training corpus, with mean relative errors as low as 3% in controlled settings and around 6% on SmolLM.","lead":"The paper shows that the mapping between token IDs and token frequencies in BPE vocabularies is similar across corpora, and uses that similarity to estimate how often individual tokens appear in an LLM's hidden training data. The proposed method, QGDE, achieves about 3% average error in controlled tests and about 6% on the released SmolLM tokenizer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Transferability evidence covers only same-algorithm BPE tokenizers on web-like corpora; production tokenizer pipelines may violate the stable ID-ratio shape that QGDE relies on.","rationale":"The reader's weakest assumption is transferability, and my stress pass agrees that this is the load-bearing point. I sharpen it in one way: because the estimator is a function of token ID only, transferability is not merely 'similar scatter plots' but equality of the full rank-frequency curve between source and target tokenizers. The current experiments support this only for a fairly narrow family of self-trained BPE tokenizers on web-like corpora, plus one SmolLM validation. The method's internal logic is sound, the K-ablation is informative, and the code release is a genuine strength. The concern is scope, not internal inconsistency: released tokenizers from GPT, Qwen, DeepSeek, or Llama use different algorithms and training distributions, and the paper does not yet demonstrate that their ID-ratio curves match the source curves. This does not warrant rejection; it warrants an explicit condition: either add an out-of-distribution transfer experiment (e.g., SentencePiece on a code-heavy mixture, or a low-resource language target) or restrict the central claim to the demonstrated setting. Since the reader already reached a conditional verdict, my analysis does not change that verdict.","tokens_in":16232,"tokens_out":12398,"duration_ms":166902,"concrete_test":"Keep the QGDE source tokenizers fixed as in the paper (FineWeb/Wiki/Code/Math and mC4 languages) and evaluate on two deliberately out-of-distribution targets: (a) a SentencePiece BPE tokenizer with byte_fallback=True trained on a 50/50 English-plus-code corpus, and (b) a HuggingFace BPE tokenizer trained on a low-resource language absent from the source set, such as Swahili. Report QGDE token-level MRE and the Section 3 directional transfer similarity for each. If either MRE exceeds roughly 15% or similarity drops below about 0.7, the stable-shape assumption is not universal across released tokenizer pipelines and corpus mixes, and the headline claim needs a scope restriction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is that the ID-ratio curve of the hidden target tokenizer coincides with the curve learned from known source tokenizers, because QGDE predicts a target token's ratio from its ID alone: Equations (3)-(7) depend on x=log t_i and the source density near that x, never on the token string. If the target's rank-frequency curve differs in slope, intercept, or vertical spread, the estimator inherits a systematic bias that local density weighting cannot correct. Section 3 demonstrates transferability only for self-trained HuggingFace BPE tokenizers on mC4 language slices and FineWeb/Wiki/Code/Math targets. These are web-like natural-language corpora, and the tokenizers share one implementation and presumably one vocabulary size. Production LLM tokenizers use different pre-tokenization, normalization, byte-fallback, sampling weights, deduplication, and sometimes SentencePiece or tiktoken-style ID orderings, any of which can alter the ID-ratio distribution. The SmolLM validation is a single released tokenizer with proxy corpora in the same broad domains; it is encouraging but does not establish universality across algorithms or across language/domain mixtures not represented in the source set. Unless the shape is shown to survive those variations, the central claim should be scoped to tokenizers and corpus types similar to those tested.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"Zhang et al. study whether released BPE tokenizer vocabularies allow token-level estimation of the corpus on which the tokenizer was trained. They establish an empirical regularity: across BPE tokenizers trained on different languages and domains, the joint distribution of log(token ID) and log(token ratio) has a stable global shape. They then introduce QGDE, which fits log-linear quantile trends to known ID-ratio points, selects a small anchor set via a coverage objective, and predicts each target token's ratio by Gaussian local-density weighting among candidate quantile values. Controlled experiments (mC4 to OSCAR for languages; FineWeb/Wikipedia/Code/Math to matched targets for domains) and a SmolLM validation with proxy corpora report token-level mean relative errors as low as 3.00% and category-level errors as low as 3.08%, with SmolLM results around 5.7-6.1%. The paper argues that released vocabularies provide a usable fine-grained corpus-composition signal.","tokens_in":16487,"tokens_out":8310,"duration_ms":88355,"significance":"If the transferability assumption holds, this is a useful advance over coarse mixture inference: per-token ratios can be aggregated to category mixtures, and the method is simple and released with code. The evaluation is careful in several ways: target ground truth is used only for scoring; the target-like source condition is explicitly diagnostic; and full K-sweeps are reported. The main risk is that the transferability evidence is drawn from a narrow family of tokenizers, whereas the prediction mechanism (Eq. 7) is ID-only; that gap is the main source of uncertainty. The SmolLM result is encouraging but does not fully close the gap because the proxy corpora are not the actual training components.","major_comments":[{"comment":"The transferability evidence is restricted to BPE tokenizers trained with the same implementation family, and the manuscript does not state the vocabulary size, pre-tokenization rules, or special-token handling used in the controlled experiments. Because QGDE's candidate estimates in Eq. (3) depend only on the target token ID, the method inherits the ID-ordering convention of the tokenizer implementation. Production tokenizers differ in pre-tokenization, byte fallback, ID assignment, and sometimes use SentencePiece or tiktoken-style orderings, any of which can alter the ID-ratio shape. Please test transferability with at least one tokenizer trained under a different BPE implementation or explicitly scope the claims to the tested tokenizer family.","section":"Section 3, Eqs. (3)-(7)"},{"comment":"The SmolLM validation uses proxies (RedPajama-C4, Wikipedia-arXiv, CodeParrot) for the actual training components (FineWeb-edu, Cosmopedia-v2, Python-edu). The category assignment weight in Eq. (8) is computed from counts of target tokens in these proxies, so any mismatch between proxies and real components directly biases both the fitted quantile trends and the aggregation weights. The text presents this as a realistic setting, but it is still a proxy setting. Please report sensitivity to alternative proxy corpora to establish that the 5.7-6.1% errors are not an artifact of the proxy choice.","section":"Section 8, Eq. (8)"},{"comment":"QGDE's behavior depends on the vertical bandwidth h_y, the horizontal window h_x, and the candidate quantile grid T, but the manuscript never states the values used in any experiment. These are free parameters of the method, and the coverage criterion in Eq. (4) and the Gaussian weights in Eq. (6) are directly controlled by them. Without reporting these values or a sensitivity analysis, the MREs are not reproducible. Please provide the exact hyperparameters used, including the grid resolution for Eq. (5).","section":"Section 4.2-4.3, Eqs. (4)-(7)"},{"comment":"The QAC saturation argument is used to justify K=14, but the token-level error curves do not align cleanly with that choice. In Table 1, the language Target-like column reaches its best MRE (3.00) at K=11 and worsens to 3.55 at K=14, while the domain single-source average still drops substantially between K=10 and K=14. The abstract's headline 3.00% and 3.08% therefore come from a K value that is not the recommended default and from a diagnostic target-like source mixture. Please either propose a principled source-only K-selection rule that tracks error, or report the default K=14 results separately from the best-case controlled results in the abstract and conclusion.","section":"Section 5.2 and Table 1"}],"minor_comments":[{"comment":"The framing emphasizes pretraining corpus composition, but the evidence concerns tokenizer-training corpora, which may differ from pretraining corpora; the footnote acknowledges this, but the abstract and conclusion should avoid implying that the estimator recovers the pretraining corpus itself.","section":"Introduction and Abstract"},{"comment":"The MRE is defined as an average over the vocabulary, but the paper does not state whether low-frequency tokens are included or whether any count threshold is applied; this matters because rare tokens dominate the vocabulary and can behave differently from high-frequency tokens.","section":"Section 6.1, Table 1"},{"comment":"The text says QGDE pies track the ground-truth mixtures closely, but the domain MRE is 5.3 and the displayed category proportions differ from the 25% reference by several percentage points; the wording should be qualified.","section":"Figure 6"},{"comment":"The smoothing constant epsilon in the density computation is left unspecified, and the transfer similarity score in Eq. (2) is used as primary motivation; the sensitivity of the score to epsilon should be reported.","section":"Appendix B, Eq. (12)"},{"comment":"The citation 'Regression, 2017' should be replaced with a standard quantile-regression reference, and the green color scaling in Tables 1 and 5 should be described so that the figures are interpretable without color.","section":"References and Figures"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the core idea is worth publishing after revision. The main risks are not integrity-related: target ground truth is used only for evaluation, and the target-like conditions are clearly labeled as diagnostic. The primary concerns are transferability scope and reproducibility of the free parameters, both of which can be addressed with additional experiments and reporting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real step beyond DMI and PoCTrace. QGDE gives per-token ratio estimates for arbitrary tokens, not just category mixtures or a single token group, and the evaluation is honest enough that the central claim—released BPE vocabularies carry usable ID-ratio signal—holds for the settings tested. The paper is worth a serious referee.\n\nThe new piece is the quantile-family plus local-density-weighting estimator, and the systematic demonstration that ID-ratio distributions transfer across the controlled corpora. Fitting multiple quantile trends instead of a single median does materially reduce error (Table 1), and the SmolLM validation is a nice touch; most LLM releases don't include training data, so getting a real released tokenizer with ground truth is valuable. The paper also ships code, which makes the method reproducible.\n\nSoft spots, in order. First, the transferability claim is broader than the evidence. Section 3 only covers HuggingFace-style BPE tokenizers trained on web-like corpora. Production tokenizers use different pre-tokenization, normalization, byte-fallback, and often SentencePiece or tiktoken. The SmolLM validation is one tokenizer in the same family. So 'BPE tokenizers share stable token ID-ratio distributions' should be scoped to tokenizers and corpus types similar to those tested. The stress-test note gets this right.\n\nSecond, the headline 'as low as 3.00%' is best-case controlled; realistic SmolLM error is 5.7-6.1%. The abstract is not false, but a reader might miss that the controlled settings are the favorable case. Worth rewording.\n\nThird, the key bandwidth hyperparameters h_y and h_x, the quantile grid T, and the smoothing epsilon are never specified in the paper. The code is available, so this is fixable, but as written the method is not fully reproducible from the text. That's a minor-to-moderate issue and should be addressed.\n\nThe circularity burden is low: anchors are selected on source coverage, and target labels are used only for evaluation. The 'target-like' source is a diagnostic, and the paper correctly shows other mixtures perform nearly as well. No fitting-to-target.\n\nBottom line: for readers working on training-data transparency or tokenizer analysis, this is a useful and citable contribution. It needs a revision that documents hyperparameters, scopes the transferability claim, and rewords the headline numbers, but it deserves peer review.","headline":"QGDE is a genuine step forward in tokenizer-based corpus estimation—token-level ratios, not just mixtures—but transferability is scoped and the hyperparameters need documenting.","tokens_in":17037,"tokens_out":2660,"would_cite":true,"duration_ms":27879,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Released tokenizer vocabularies reveal hidden corpus ratios, token by token.","keywords":["tokenizer vocabulary","byte-pair encoding","hidden corpus estimation","token-level ratio estimation","quantile regression","data mixture inference","LLM transparency","corpus composition"],"falsifier":"Train a BPE tokenizer on a corpus whose preprocessing deliberately differs from the reference corpora, such as aggressive deduplication or heavy quality filtering, fit QGDE using the standard reference corpora, and compare predicted token ratios to counts computed on that held-out corpus; if the mean relative error rises well above the paper's reported range, the transferability assumption is falsified.","tokens_in":16049,"feed_emoji":"🧩","tokens_out":6743,"duration_ms":62396,"temperature":0.7,"pith_summary":"This paper tries to establish that a released byte-pair-encoding (BPE) tokenizer vocabulary, even when model weights and training data are withheld, carries enough information to estimate how often arbitrary tokens appeared in the hidden training corpus. The key empirical claim is that token ID and token frequency ratio follow a stable global relationship across tokenizers trained on different languages and domains, so the relationship learned from known reference corpora transfers to a tokenizer trained on an unknown corpus. The paper packages this into Quantile-Guided Density Estimation (QGDE), which fits multiple quantile trends to the known ID-ratio cloud and weights them by local density to produce per-token estimates. In controlled tests QGDE reaches mean relative errors near 3.0%, and on the released SmolLM tokenizer roughly 5.7 to 6.1%, suggesting that released vocabularies are a useful signal for fine-grained corpus auditing.","feed_headline":"Tokenizer vocabulary alone estimates hidden corpus ratios to ~3% error","feed_subtitle":"A BPE vocabulary plus known corpora estimates per-token frequencies and category mixtures without model weights.","key_machinery":"The load-bearing object is the token ID-ratio distribution: for each token in a BPE vocabulary, the pair of the logarithm of its token ID and the logarithm of its corpus ratio, which forms a downward cloud in log-log space. QGDE approximates this shared cloud from known corpora by fitting a family of log-linear quantile trends, selecting a small set of quantile anchors that maximize coverage of known points within a vertical band, and then combining candidate estimates at each target token ID with Gaussian local-density weights based on nearby known points. The quantile family preserves the vertical spread of plausible ratios at a given ID, which is the part a single median trend discards, while local density weighting converts that spread into a token-specific point estimate.","core_discovery":"The central discovery is that the ID-ratio relationship has a stable global shape across BPE tokenizers: when you plot each token's logarithm of its corpus ratio against its logarithm of token ID, tokenizers trained on English, French, Japanese, Chinese, web text, Wikipedia, code, and math all show the same downward trend, with the main differences being vertical spread and tail behavior. The paper argues this shared shape is transferable, quantified by a directional similarity score based on KL divergence, and uses it to justify estimating a hidden tokenizer's per-token ratios from known corpora. With QGDE, token-level mean relative errors reach 3.00% in controlled settings and 5.72-5.78% on the released SmolLM tokenizer; aggregating the estimates into language or domain mixtures gives errors of 3.08% and about 5.9-6.1%, respectively, improving on direct ID-ratio transfer, single-median-trend estimation, and merge-rule-based Data Mixture Inference.","pith_inferences":["If the transferability holds beyond the tested corpora, tokenizer vocabularies could become a routine auditing signal: reviewers could check a released model's claimed data mix against the mix inferred from its vocabulary, without needing any training-time information.","The paper itself notes that ground truth for released tokenizers is scarce, so the SmolLM validation is the only realistic check; the controlled results should be read as proof-of-concept until more released training corpora allow broader checks.","A natural extension the authors do not run is to apply QGDE to multiple released tokenizers at once and check cross-tokenizer consistency, or to combine QGDE's token-level estimates with merge-rule methods as a joint constraint for more robust mixtures.","Another extension would be to use QGDE estimates to weight known corpora when building a proxy training set, since the per-token ratios provide a distributional target for corpus selection."],"forward_implications":["A released tokenizer alone, plus one or more known corpora, can yield per-token frequency estimates for arbitrary vocabulary tokens, not just for a pre-specified group of tokens.","Aggregating those token estimates recovers language and domain mixture proportions more accurately than merge-rule-based mixture inference, which only gives coarse category-level output.","The exact mixture ratios of the known corpora matter less than covering the relevant source components, so an analyst does not need to know the target mixture in advance.","The method transfers from controlled single-domain and single-language tokenizers to a real released tokenizer (SmolLM), suggesting it can be applied to other released tokenizers even when their training corpora are hidden.","Because the estimator needs no model weights or query access, it is a cheap, static transparency tool that can be run the moment a vocabulary is published."],"supporting_citations":[{"why":"Introduces BPE subword tokenization, whose merge order encodes corpus statistics and provides the basis for the ID-ratio signal.","marker":"Sennrich et al., 2016"},{"why":"Shows BPE merge rules reveal training data composition and supplies the coarse Data Mixture Inference baseline that QGDE is compared against.","marker":"Hayase et al., 2024"},{"why":"PoCTrace fits a single median ID-ratio trend for a specific token group; QGDE generalizes this from one trend to a full quantile family for arbitrary tokens.","marker":"Zhang et al., 2025"},{"why":"Zipf's law motivates modeling token ID and ratio on a log-log scale with approximately linear quantile trends.","marker":"Piantadosi, 2014; Saichev et al., 2009"},{"why":"Supplies the Gaussian kernel density estimation used for local density weighting around candidate quantile estimates.","marker":"Silverman, 2018"},{"why":"Provides the quantile regression objective used to fit each log-linear trend to known ID-ratio points.","marker":"Regression, 2017"},{"why":"The SmolLM release includes both a tokenizer and its training corpus, enabling the paper's realistic validation setting.","marker":"Allal et al., 2025"}],"fun_headline_variants":["Tokenizer alone estimates corpus ratios to 3% error","Released vocabularies predict hidden corpus mix within 3%","BPE tokenizer reveals corpus composition from IDs alone","Hidden corpus ratios estimated from tokenizer vocabulary","Vocabulary-only estimator yields 3% error on token ratios"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach assumes that the relationship between a token's ID and its frequency ratio in a hidden corpus can be inferred from the same relationship in known source corpora; if the hidden tokenizer was trained on a corpus with a different language mix, deduplication, or sampling weighting, the transferred trends are systematically biased.","fun_headline_variants_meta":{"raw":{"variants":["Tokenizer alone estimates corpus ratios to 3% error","Released vocabularies predict hidden corpus mix within 3%","BPE tokenizer reveals corpus composition from IDs alone","Hidden corpus ratios estimated from tokenizer vocabulary","Vocabulary-only estimator yields 3% error on token ratios"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000839,"raw_usage":{"total_tokens":3644,"prompt_tokens":921,"completion_tokens":2723,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":2646}},"tokens_in":537,"tokens_out":2723,"duration_ms":20198,"temperature":1.0,"reasoning_tokens":2646,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:21:09.654843+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a BPE tokenizer on a corpus whose preprocessing deliberately differs from the reference corpora, such as aggressive deduplication or heavy quality filtering, fit QGDE using the standard reference corpora, and compare predicted token ratios to counts computed on that held-out corpus; if the mean relative error rises well above the paper's reported range, the transferability assumption is falsified.","supporting_citations":[],"review_version":1}