Pith. sign in

REVIEW 4 major objections 5 minor 47 references

Can Released LLM Vocabularies Support Token-Level Estimation of Hidden Corpora?

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Released tokenizer vocabularies reveal hidden corpus ratios, token by token.

desk verdict QGDE is a genuine step forward in tokenizer-based corpus estimation—token-level ratios, not just mixtures—but transferability is scoped and the hyperparameters need documenting. read the letter →

arxiv 2608.10690 v1 pith:7ELODANB submitted 2026-08-11 cs.CL

Qingjie Zhang , Xingzhang Ren , Zixuan Chen , Jinfeng Li , YueFeng Chen , Yitong Yang , Hui Xue , Dayiheng Liu
show 1 more author
Han Qiu
This is my paper · ORCID
classification cs.CL
keywords tokenizervocabularybyte-pairencodinghiddencorpusestimationtoken-levelratioquantileregressiondatamixtureinferenceLLMtransparencycomposition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a released byte-pair-encoding (BPE) tokenizer vocabulary, even when model weights and training data are withheld, carries enough information to estimate how often arbitrary tokens appeared in the hidden training corpus. The key empirical claim is that token ID and token frequency ratio follow a stable global relationship across tokenizers trained on different languages and domains, so the relationship learned from known reference corpora transfers to a tokenizer trained on an unknown corpus. The paper packages this into Quantile-Guided Density Estimation (QGDE), which fits multiple quantile trends to the known ID-ratio cloud and weights them by local density to produce per-token estimates. In controlled tests QGDE reaches mean relative errors near 3.0%, and on the released SmolLM tokenizer roughly 5.7 to 6.1%, suggesting that released vocabularies are a useful signal for fine-grained corpus auditing.

What carries the argument

The load-bearing object is the token ID-ratio distribution: for each token in a BPE vocabulary, the pair of the logarithm of its token ID and the logarithm of its corpus ratio, which forms a downward cloud in log-log space. QGDE approximates this shared cloud from known corpora by fitting a family of log-linear quantile trends, selecting a small set of quantile anchors that maximize coverage of known points within a vertical band, and then combining candidate estimates at each target token ID with Gaussian local-density weights based on nearby known points. The quantile family preserves the vertical spread of plausible ratios at a given ID, which is the part a single median trend discards, while local density weighting converts that spread into a token-specific point estimate.

What would settle it

Train a BPE tokenizer on a corpus whose preprocessing deliberately differs from the reference corpora, such as aggressive deduplication or heavy quality filtering, fit QGDE using the standard reference corpora, and compare predicted token ratios to counts computed on that held-out corpus; if the mean relative error rises well above the paper's reported range, the transferability assumption is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that the ID-ratio relationship has a stable global shape across BPE tokenizers: when you plot each token's logarithm of its corpus ratio against its logarithm of token ID, tokenizers trained on English, French, Japanese, Chinese, web text, Wikipedia, code, and math all show the same downward trend, with the main differences being vertical spread and tail behavior. The paper argues this shared shape is transferable, quantified by a directional similarity score based on KL divergence, and uses it to justify estimating a hidden tokenizer's per-token ratios from known corpora. With QGDE, token-level mean relative errors reach 3.00% in controlled settings and 5.72-5.78% on the released SmolLM tokenizer; aggregating the estimates into language or domain mixtures gives errors of 3.08% and about 5.9-6.1%, respectively, improving on direct ID-ratio transfer, single-median-trend estimation, and merge-rule-based Data Mixture Inference.

Load-bearing premise

The approach assumes that the relationship between a token's ID and its frequency ratio in a hidden corpus can be inferred from the same relationship in known source corpora; if the hidden tokenizer was trained on a corpus with a different language mix, deduplication, or sampling weighting, the transferred trends are systematically biased.

Editorial extensions

If this is right

  • A released tokenizer alone, plus one or more known corpora, can yield per-token frequency estimates for arbitrary vocabulary tokens, not just for a pre-specified group of tokens.
  • Aggregating those token estimates recovers language and domain mixture proportions more accurately than merge-rule-based mixture inference, which only gives coarse category-level output.
  • The exact mixture ratios of the known corpora matter less than covering the relevant source components, so an analyst does not need to know the target mixture in advance.
  • The method transfers from controlled single-domain and single-language tokenizers to a real released tokenizer (SmolLM), suggesting it can be applied to other released tokenizers even when their training corpora are hidden.
  • Because the estimator needs no model weights or query access, it is a cheap, static transparency tool that can be run the moment a vocabulary is published.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the transferability holds beyond the tested corpora, tokenizer vocabularies could become a routine auditing signal: reviewers could check a released model's claimed data mix against the mix inferred from its vocabulary, without needing any training-time information.
  • The paper itself notes that ground truth for released tokenizers is scarce, so the SmolLM validation is the only realistic check; the controlled results should be read as proof-of-concept until more released training corpora allow broader checks.
  • A natural extension the authors do not run is to apply QGDE to multiple released tokenizers at once and check cross-tokenizer consistency, or to combine QGDE's token-level estimates with merge-rule methods as a joint constraint for more robust mixtures.
  • Another extension would be to use QGDE estimates to weight known corpora when building a proxy training set, since the per-token ratios provide a distributional target for corpus selection.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. Zhang et al. study whether released BPE tokenizer vocabularies allow token-level estimation of the corpus on which the tokenizer was trained. They establish an empirical regularity: across BPE tokenizers trained on different languages and domains, the joint distribution of log(token ID) and log(token ratio) has a stable global shape. They then introduce QGDE, which fits log-linear quantile trends to known ID-ratio points, selects a small anchor set via a coverage objective, and predicts each target token's ratio by Gaussian local-density weighting among candidate quantile values. Controlled experiments (mC4 to OSCAR for languages; FineWeb/Wikipedia/Code/Math to matched targets for domains) and a SmolLM validation with proxy corpora report token-level mean relative errors as low as 3.00% and category-level errors as low as 3.08%, with SmolLM results around 5.7-6.1%. The paper argues that released vocabularies provide a usable fine-grained corpus-composition signal.

Significance. If the transferability assumption holds, this is a useful advance over coarse mixture inference: per-token ratios can be aggregated to category mixtures, and the method is simple and released with code. The evaluation is careful in several ways: target ground truth is used only for scoring; the target-like source condition is explicitly diagnostic; and full K-sweeps are reported. The main risk is that the transferability evidence is drawn from a narrow family of tokenizers, whereas the prediction mechanism (Eq. 7) is ID-only; that gap is the main source of uncertainty. The SmolLM result is encouraging but does not fully close the gap because the proxy corpora are not the actual training components.

major comments (4)
  1. [Section 3, Eqs. (3)-(7)] The transferability evidence is restricted to BPE tokenizers trained with the same implementation family, and the manuscript does not state the vocabulary size, pre-tokenization rules, or special-token handling used in the controlled experiments. Because QGDE's candidate estimates in Eq. (3) depend only on the target token ID, the method inherits the ID-ordering convention of the tokenizer implementation. Production tokenizers differ in pre-tokenization, byte fallback, ID assignment, and sometimes use SentencePiece or tiktoken-style orderings, any of which can alter the ID-ratio shape. Please test transferability with at least one tokenizer trained under a different BPE implementation or explicitly scope the claims to the tested tokenizer family.
  2. [Section 8, Eq. (8)] The SmolLM validation uses proxies (RedPajama-C4, Wikipedia-arXiv, CodeParrot) for the actual training components (FineWeb-edu, Cosmopedia-v2, Python-edu). The category assignment weight in Eq. (8) is computed from counts of target tokens in these proxies, so any mismatch between proxies and real components directly biases both the fitted quantile trends and the aggregation weights. The text presents this as a realistic setting, but it is still a proxy setting. Please report sensitivity to alternative proxy corpora to establish that the 5.7-6.1% errors are not an artifact of the proxy choice.
  3. [Section 4.2-4.3, Eqs. (4)-(7)] QGDE's behavior depends on the vertical bandwidth h_y, the horizontal window h_x, and the candidate quantile grid T, but the manuscript never states the values used in any experiment. These are free parameters of the method, and the coverage criterion in Eq. (4) and the Gaussian weights in Eq. (6) are directly controlled by them. Without reporting these values or a sensitivity analysis, the MREs are not reproducible. Please provide the exact hyperparameters used, including the grid resolution for Eq. (5).
  4. [Section 5.2 and Table 1] The QAC saturation argument is used to justify K=14, but the token-level error curves do not align cleanly with that choice. In Table 1, the language Target-like column reaches its best MRE (3.00) at K=11 and worsens to 3.55 at K=14, while the domain single-source average still drops substantially between K=10 and K=14. The abstract's headline 3.00% and 3.08% therefore come from a K value that is not the recommended default and from a diagnostic target-like source mixture. Please either propose a principled source-only K-selection rule that tracks error, or report the default K=14 results separately from the best-case controlled results in the abstract and conclusion.
minor comments (5)
  1. [Introduction and Abstract] The framing emphasizes pretraining corpus composition, but the evidence concerns tokenizer-training corpora, which may differ from pretraining corpora; the footnote acknowledges this, but the abstract and conclusion should avoid implying that the estimator recovers the pretraining corpus itself.
  2. [Section 6.1, Table 1] The MRE is defined as an average over the vocabulary, but the paper does not state whether low-frequency tokens are included or whether any count threshold is applied; this matters because rare tokens dominate the vocabulary and can behave differently from high-frequency tokens.
  3. [Figure 6] The text says QGDE pies track the ground-truth mixtures closely, but the domain MRE is 5.3 and the displayed category proportions differ from the 25% reference by several percentage points; the wording should be qualified.
  4. [Appendix B, Eq. (12)] The smoothing constant epsilon in the density computation is left unspecified, and the transfer similarity score in Eq. (2) is used as primary motivation; the sensitivity of the score to epsilon should be reported.
  5. [References and Figures] The citation 'Regression, 2017' should be replaced with a standard quantile-regression reference, and the green color scaling in Tables 1 and 5 should be described so that the figures are interpretable without color.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: QGDE is fit entirely on known source corpora and target ground truth is used only for evaluation.

full rationale

The paper's derivation chain is self-contained rather than circular. The QGDE estimator is built exclusively from known-source ID–ratio pairs: quantile trends are fit on source points (Eq. 10), quantile anchors are selected by maximizing coverage over the known ID–ratio points (Eqs. 4–5), and local density weights use nearby known points (Eq. 6). The target tokenizer contributes only its visible token IDs, and target corpus ratios appear only in the evaluation tables as ground truth, not in any fitting objective. The controlled experiments deliberately hold target corpora fixed and vary only the source mixture; the 'Target-like' source is explicitly labeled a diagnostic setting, and the paper shows other source mixtures perform nearly as well (Tables 1 and 2). The SmolLM validation provides an external, released tokenizer with real training-corpus ground truth, further confirming that the approach is not tuned to the test data. The only overlapping-author citation, Zhang et al. (2025), is used as prior art and as a baseline (PoCTrace), not as the justification for transferability; the transferability claim is supported by the paper's own KL-divergence measurements over controlled corpora. No equation reduces a predicted quantity to a fitted target value, and no load-bearing premise is imported solely from a self-citation. Accordingly, there is no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method relies on Zipf's law, the assumption that BPE IDs encode frequency rank, and the empirical transferability of the ID to frequency shape. The main free parameters are the bandwidths h_x and h_y, the quantile anchor set, and the smoothing constant, none of which are given numeric values in the text.

free parameters (5)
  • h_y
    Vertical bandwidth used in Eq. (4) for anchor coverage and in Eq. (6) for the Gaussian kernel; no numeric value is given in the paper.
  • h_x
    Width of the local ID window in Eq. (6); no numeric value given.
  • K = 14 (default)
    Number of quantile anchors; swept from 3 to 14, default chosen after coverage saturation in Section 5.2. Not fit to target labels.
  • Quantile grid T
    Candidate quantile levels searched in Eq. (5); grid resolution unspecified.
  • epsilon
    Smoothing constant in the density estimate in Eq. (12); used only for the transfer similarity analysis.
assumptions (4)
  • domain assumption Log ID and log ratio are approximately linearly related (Zipf's law).
    Section 4.1 assumes this relationship to motivate log-linear quantile trends.
  • domain assumption BPE token IDs reflect merge order and thus corpus statistics.
    Sections 2.1 and 3 rely on BPE IDs encoding token frequency information.
  • domain assumption ID to ratio distributions transfer from known source corpora to hidden target corpora.
    Section 3 establishes this empirically on controlled corpora and then treats it as an assumption for hidden targets.
  • domain assumption The tokenizer training corpus approximates the pretraining corpus for real LLMs.
    Footnote 1 acknowledges differences but assumes tokenizer training corpora still reflect composition choices.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can Released LLM Vocabularies Support Token-Level Estimation of Hidden Corpora?." pith.science (2026). https://pith.science/paper/7ELODANB

@misc{pith2026260810690,
  author       = {Pith},
  title        = {Pith review of: Can Released LLM Vocabularies Support Token-Level Estimation of Hidden Corpora?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7ELODANB}},
  note         = {Machine review of arXiv:2608.10690}
}
read the original abstract

Pretraining corpus composition shapes LLM capabilities, but it often remains hidden even when model weights are released. Prior work has inferred corpus mixtures or traced specific token groups from released tokenizer vocabularies; in contrast, we estimate corpus ratios for arbitrary target tokens. We first show that BPE tokenizers trained on different corpora share stable token ID--ratio distributions, motivating distribution transfer from known corpora to a target tokenizer trained on hidden corpora. We then propose Quantile-Guided Density Estimation (QGDE), which approximates this distribution with multiple quantile trends and uses local density weighting to produce token-level estimates. In controlled settings and a realistic setting using the released SmolLM tokenizer, QGDE achieves mean relative errors as low as 3.00% for token-level estimation and 3.08% after aggregation into category-level mixtures. These results suggest that released tokenizer vocabularies provide a useful signal for fine-grained corpus estimation beyond coarse composition inference.

Figures

Figures reproduced from arXiv: 2608.10690 by the authors.

Figure 1
Figure 1. Overview of our work. QGDE transfers ID [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Token ID–ratio distributions are transferable across different corpora. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of quantile-guided density estimation. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Effect of the number of quantile anchors on [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Category-level mixture estimation, where QGDE matches the ground truth more closely in both language [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Token ID–ratio scatter plots for all eight controlled tokenizers. The top row shows language-specific mC4 [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 13 canonical work pages

  1. [1]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    arXiv

  2. [2]

    arXiv preprint arXiv:2412.19437 , year=

    Deepseek-v3 technical report , author=. arXiv preprint arXiv:2412.19437 , year=

    arXiv

  3. [3]

    arXiv preprint arXiv:2601.03267 , year=

    Openai gpt-5 system card , author=. arXiv preprint arXiv:2601.03267 , year=

    arXiv

  4. [4]

    arXiv preprint arXiv:2310.12941 , year=

    The foundation model transparency index , author=. arXiv preprint arXiv:2310.12941 , year=

    arXiv

  5. [5]

    Nature Machine Intelligence , volume=

    A large-scale audit of dataset licensing and attribution in AI , author=. Nature Machine Intelligence , volume=. 2024 , publisher=

    work page 2024

  6. [6]

    arXiv preprint arXiv:2406.04244 , year=

    Benchmark data contamination of large language models: A survey , author=. arXiv preprint arXiv:2406.04244 , year=

    arXiv

  7. [7]

    30th USENIX security symposium (USENIX Security 21) , pages=

    Extracting training data from large language models , author=. 30th USENIX security symposium (USENIX Security 21) , pages=

  8. [8]

    The Eleventh International Conference on Learning Representations , year=

    Quantifying memorization across neural language models , author=. The Eleventh International Conference on Learning Representations , year=

Show all 47 references
  1. [9]

    arXiv preprint arXiv:2311.04850 , year=

    Rethinking benchmark and contamination for language models with rephrased samples , author=. arXiv preprint arXiv:2311.04850 , year=

    arXiv

  2. [10]

    Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers) , pages=

    Neural machine translation of rare words with subword units , author=. Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers) , pages=

  3. [11]

    arXiv preprint arXiv:1609.08144 , year=

    Google's neural machine translation system: Bridging the gap between human and machine translation , author=. arXiv preprint arXiv:1609.08144 , year=

    arXiv

  4. [12]

    Proceedings of the 2021 conference on empirical methods in natural language processing , pages=

    Fast wordpiece tokenization , author=. Proceedings of the 2021 conference on empirical methods in natural language processing , pages=

    2021

  5. [13]

    Proceedings of the 2018 conference on empirical methods in natural language processing: System demonstrations , pages=

    SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing , author=. Proceedings of the 2018 conference on empirical methods in natural language processing: System demonstrations , pages=

    2018

  6. [14]

    Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Subword regularization: Improving neural network translation models with multiple subword candidates , author=. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  7. [15]

    arXiv preprint arXiv:2402.01035 , year=

    Getting the most out of your tokenizer for pre-training and domain adaptation , author=. arXiv preprint arXiv:2402.01035 , year=

    arXiv

  8. [16]

    arXiv preprint arXiv:2502.20273 , year=

    How much is enough? the diminishing returns of tokenization training data , author=. arXiv preprint arXiv:2502.20273 , year=

    arXiv

  9. [17]

    Advances in Neural Information Processing Systems , volume=

    Data mixture inference attack: BPE tokenizers reveal training data compositions , author=. Advances in Neural Information Processing Systems , volume=

  10. [18]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Fishing for magikarp: Automatically detecting under-trained tokens in large language models , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  11. [19]

    Proceedings of the ACM on Software Engineering , volume=

    Glitch tokens in large language models: Categorization taxonomy and effective detection , author=. Proceedings of the ACM on Software Engineering , volume=. 2024 , publisher=

    2024

  12. [20]

    2024 IEEE International Conference on Big Data (BigData) , pages=

    Problematic tokens: Tokenizer bias in large language models , author=. 2024 IEEE International Conference on Big Data (BigData) , pages=. 2024 , organization=

    2024

  13. [21]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    Speculating LLMs’ Chinese Training Data Pollution from Their Tokens , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  14. [22]

    Psychonomic bulletin & review , volume=

    Zipf’s word frequency law in natural language: A critical review and future directions , author=. Psychonomic bulletin & review , volume=. 2014 , publisher=

    2014

  15. [23]

    Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: Human language technologies , pages=

    mT5: A massively multilingual pre-trained text-to-text transformer , author=. Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: Human language technologies , pages=

    2021

  16. [24]

    Advances in Neural Information Processing Systems , volume=

    Doremi: Optimizing data mixtures speeds up language model pretraining , author=. Advances in Neural Information Processing Systems , volume=

  17. [25]

    arXiv preprint arXiv:2409.04556 , year=

    How does code pretraining affect language model task performance? , author=. arXiv preprint arXiv:2409.04556 , year=

    arXiv

  18. [26]

    Advances in neural information processing systems , volume=

    Language models are few-shot learners , author=. Advances in neural information processing systems , volume=

  19. [27]

    arXiv preprint arXiv:2309.16609 , year=

    Qwen technical report , author=. arXiv preprint arXiv:2309.16609 , year=

    arXiv

  20. [28]

    arXiv preprint arXiv:2401.02954 , year=

    Deepseek llm: Scaling open-source language models with longtermism , author=. arXiv preprint arXiv:2401.02954 , year=

    arXiv

  21. [29]

    2018 , publisher=

    Density estimation for statistics and data analysis , author=. 2018 , publisher=

    2018

  22. [30]

    2009 , publisher=

    Theory of Zipf's law and beyond , author=. 2009 , publisher=

    2009

  23. [31]

    Proceedings of the 58th annual meeting of the association for computational linguistics , pages=

    A monolingual approach to contextualized word embeddings for mid-resource languages , author=. Proceedings of the 58th annual meeting of the association for computational linguistics , pages=

  24. [32]

    Advances in Neural Information Processing Systems , volume=

    The fineweb datasets: Decanting the web for the finest text data at scale , author=. Advances in Neural Information Processing Systems , volume=

  25. [33]

    International Conference on Learning Representations , volume=

    Openwebmath: An open dataset of high-quality mathematical web text , author=. International Conference on Learning Representations , volume=

  26. [34]

    2021 , howpublished=

    2021

  27. [35]

    2023 , howpublished=

    2023

  28. [36]

    2020 , howpublished=

    2020

  29. [37]

    Advances in neural information processing systems , volume=

    Redpajama: an open dataset for training large language models , author=. Advances in neural information processing systems , volume=

  30. [38]

    arXiv preprint arXiv:2502.02737 , year=

    SmolLM2: When Smol Goes Big--Data-Centric Training of a Small Language Model , author=. arXiv preprint arXiv:2502.02737 , year=

    arXiv

  31. [39]

    Proceedings of the IEEE international conference on computer vision , pages=

    Aligning books and movies: Towards story-like visual explanations by watching movies and reading books , author=. Proceedings of the IEEE international conference on computer vision , pages=

  32. [40]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Dolma: An open corpus of three trillion tokens for language model pretraining research , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  33. [41]

    Boca Raton, FL, USA: CRC , year=

    Handbook of quantile regression , author=. Boca Raton, FL, USA: CRC , year=

  34. [42]

    Probabilistic Machine Learning: An Introduction , author=

  35. [43]

    Proceedings of the 58th annual meeting of the association for computational linguistics , pages=

    Don’t stop pretraining: Adapt language models to domains and tasks , author=. Proceedings of the 58th annual meeting of the association for computational linguistics , pages=

  36. [44]

    arXiv preprint arXiv:2203.15556 , year=

    Training Compute-Optimal Large Language Models , author=. arXiv preprint arXiv:2203.15556 , year=

    arXiv

  37. [45]

    Advances in Neural Information Processing Systems , volume=

    Datacomp-lm: In search of the next generation of training sets for language models , author=. Advances in Neural Information Processing Systems , volume=

  38. [46]

    arXiv preprint arXiv:2307.09288 , year=

    Llama 2: Open foundation and fine-tuned chat models , author=. arXiv preprint arXiv:2307.09288 , year=

    arXiv

  39. [47]

    Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages=

    Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus , author=. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages=

    2021

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.