Pith. sign in

REVIEW 4 major objections 5 minor 38 references

Pre-trained Models Perform the Best When Token Distributions Follow Zipf's Law

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a pre-trained model's downstream performance peaks when its tokenizer's rank-frequency distribution is closest to Zipf's law, across NLP, genomics, and chemistry, making the R^2 of a log-log fit a pre-training…

desk verdict The paper shows a real empirical trend—bigger BPE vocabularies make token distributions look more Zipfian—but the central claim that Zipfian alignment predicts the optimal vocabulary size is contradicted by the paper's own tables. read the letter →

arxiv 2507.22543 v1 pith:F537CTPT submitted 2025-07-30 cs.LG cs.CL

Yanjin He , Qingkai Zeng , Meng Jiang This is my paper
classification cs.LGcs.CL
keywords Zipf'slawtokenizationvocabularysizeBytePairEncodingtokenfrequencydistributionpower-lawfitpre-trainedlanguagemodelscross-modalgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to turn vocabulary-size selection for tokenizers from a heuristic into a data-driven choice: measure how closely the token rank-frequency distribution follows Zipf's law, then pick the vocabulary size where that fit stops improving. It argues that downstream performance and this Zipfian fit move together—performance rises while the fit improves, peaks at the vocabulary where the fit saturates, and then declines or plateaus. This correspondence is tested across English-language classification and translation, DNA-sequence classification, and chemistry with SMILES strings, using BPE tokenizers and transformer models trained at a range of vocabulary sizes. If the claim holds, a practitioner could choose a vocabulary size from token counts alone, before pre-training, without training candidate models.

What carries the argument

The carrying object is the token rank-frequency distribution plotted on log-log axes, with the coefficient of determination $R^{2}$ of a least-squares line fit used as a Zipf alignment score. An ideal power law f(r) proportional to $r^{{-k}}$ would plot as a straight line, so $R^{2}$ close to 1 means the distribution is nearly Zipfian. The selection procedure grows a BPE vocabulary in merge steps and tracks the score; when the score fails to improve beyond a small threshold epsilon over N steps, the current vocabulary is declared optimal. The paper does not specify the value of epsilon, so in practice the machinery identifies the vocabulary at the elbow where the log-log curve stops straightening.

What would settle it

Recompute the alignment score on the same corpora using a maximum-likelihood power-law goodness-of-fit test instead of the raw $R^{2}$ of a least-squares line; if the vocabularies that maximize downstream performance no longer coincide with the score's saturation points, then Zipfian alignment as measured here is not the operative criterion. A second check is to construct a vocabulary that is deliberately linear on log-log axes but semantically arbitrary and see whether a model trained with it matches the performance of the vocabulary selected by the $R^{2}$ rule.

Watch

Extended reading notes

Core claim

The central claim is that a tokenizer's vocabulary is near-optimal exactly when its token frequency distribution is as close to Zipf's law as that corpus allows. Concretely, as vocabulary size grows, the log-log rank-frequency curve straightens, the $R^{2}$ goodness-of-fit rises, and downstream task scores rise with it; the vocabulary size that corresponds to the elbow where $R^{2}$ stops growing is the same vocabulary that gives the best average performance. The evidence is a set of pre-training and fine-tuning sweeps—a BERT-style model on GLUE, an mBART-style model on three translation pairs, a DNA BERT on genomics tasks, and a SMILES BERT on chemistry tasks—where the reported optimal sizes (30K for English NLP, 4K for DNA, 3K for SMILES) coincide with the $R^{2}$ turning points. The paper calls this correspondence Zipfian alignment and proposes it as a generalizable criterion for vocabulary selection across modalities.

Load-bearing premise

The load-bearing premise is that the $R^{2}$ of a least-squares line fit on a log-log rank-frequency plot faithfully measures how Zipfian a token distribution is, and that the vocabulary where this score stops improving is the vocabulary where downstream performance peaks.

Editorial extensions

If this is right

  • Vocabulary size can be selected before pre-training by tokenizing the corpus at several sizes and choosing the smallest size at which the log-log rank-frequency fit stabilizes, avoiding the cost of training many candidate models.
  • The criterion transfers across modalities: the optimal vocabulary is dataset-dependent, around 30K for English text, 4K for DNA, and 3K for SMILES in the paper's setups, rather than a universal constant like 50K.
  • Common heuristic vocabularies are not optimal in the paper's experiments: on GLUE, a 30K vocabulary outperforms the larger 50K vocabulary, and further growth mostly plateaus or hurts performance.
  • Tokenization metrics that ignore the shape of the rank-frequency distribution, such as fertility and compression, are less reliable predictors of downstream performance than Zipfian alignment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the proposed stopping rule depends on an unspecified epsilon threshold, so a natural next step is to vary epsilon and check whether the selected vocabulary remains stable; if it does, the rule is genuinely automatic, and if not, the criterion is incomplete.
  • Editorial inference: R^2 measures linearity but not the slope of the log-log line, so two vocabularies with equal Zipfian fit can have different power-law exponents; an explicit comparison of slope versus fit would reveal whether the saturation point or the exponent carries the performance signal.
  • Editorial inference: if the correspondence holds at scale, it suggests a cheap tokenizer-only diagnostic for deciding whether a domain corpus is large enough to justify a larger vocabulary, before spending GPU hours on pre-training.
  • Editorial inference: nothing in the argument is specific to BPE or transformer models, so the same R^2-saturation rule should be testable on other merge-based tokenizers and on vision or audio tokenization, where vocabulary choice is also currently heuristic.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes to select tokenizer vocabulary size by measuring how closely the token frequency distribution adheres to Zipf's law, operationalized as the R2 of a least-squares line fit to the log-log rank-frequency plot. Using BPE tokenizers at multiple vocabulary sizes, the authors pre-train BERT on NLP, genomics, and chemistry data, mBART on translation, and report downstream performance (GLUE score, accuracy, ROC-AUC, BLEU). They claim that downstream task performance peaks when the token distribution is most Zipfian, and propose an iterative algorithm that stops expanding the vocabulary when the R2 improvement stagnates. The paper also includes case studies of tokenization granularity and a limitations section acknowledging small models and limited modalities.

Significance. Establishing a pre-training-only, corpus-statistics-based criterion for vocabulary size would be practically valuable, since vocabulary size is usually set by convention. The experiments are broad (three modalities, multiple tasks, three seeds per configuration), and the raw observation that R2 and downstream performance both rise with vocabulary size, at least up to the performance optimum, is interesting. However, the specific claim of a predictive criterion is not supported: the R2 metric keeps improving after the performance optimum in the paper's own tables, and the stagnation threshold is never specified. The manuscript also cites Clauset et al. (2009) for a procedure that reference explicitly warns against. With a properly specified criterion and a statistically sound goodness-of-fit measure, the underlying correlation could be a useful empirical finding, but as written the central claim is overclaimed.

major comments (4)
  1. [Section 3.3 and Section 5.1] The proposed stopping algorithm is not specified. The text says the score Zipft must fail to exceed Zipfmax by more than a small threshold epsilon after N steps, but no values for epsilon or N are given anywhere in the paper, and no pseudocode is provided. Because the R2 increments in the tables are small and positive past the performance peak (Table 1: +0.0053 from 30k to 40k; Table 2: +0.0017 from 4k to 7k; Table 3: +0.0010 from 3k to 5k), any epsilon smaller than those increments makes the algorithm continue past the optimum. The criterion is therefore not testable as stated. Please specify epsilon and N, justify them from token statistics alone, and report the vocabulary size the algorithm selects for each domain.
  2. [Section 5.1, Table 1; Section 5.2, Tables 2 and 3] The key claim is contradicted by the reported R2 values. In Table 1, average GLUE peaks at 30k (81.38) but R2 rises monotonically to 0.9425 at 40k. In Table 2, average gene accuracy peaks at 4k (85.92) but R2 rises to 0.9744 at 7k. In Table 3, ROC-AUC peaks at 3k (72.08) but R2 rises to 0.9751 at 5k. If R2 is the measure of Zipfian adherence, then 'models achieve peak performance when the token distribution closely adheres to Zipf's law' is false; the closest adherence occurs at a different, larger vocabulary. The authors need to either redefine their criterion (e.g., a principled threshold on the rate of R2 improvement) or substantially weaken the claim to 'performance improves as R2 increases, but the optimum occurs before the R2 maximum.'
  3. [Section 5.1, paragraph citing Clauset et al. (2009)] The manuscript justifies R2 as a goodness-of-fit measure by citing Clauset et al. (2009), but that paper explicitly argues that the R2 of a log-log least-squares fit is a poor and biased way to quantify power-law behavior (see their Sections 3.1-3.2 and Appendix). The citation therefore undercuts, rather than supports, the method. Because the entire criterion rests on R2, the authors should either (i) use a statistically valid power-law goodness-of-fit procedure (e.g., maximum likelihood estimation with a Kolmogorov-Smirnov test against synthetic data) and show the criterion still selects the observed performance-optimal vocabulary sizes, or (ii) provide an independent validation that R2 in this specific setting tracks a reliable measure of power-law adherence.
  4. [Appendix B, Table 4] The translation results also fail to support the coincidence claim. For De-En, BLEU reaches its maximum (30.01) at 80k, while R2 continues to rise and peaks at 0.9657 at 120k. For Fr-En, BLEU peaks at 37.59 at 70k, while R2 peaks at 0.9680 at 120k. For Zh-En, BLEU peaks at 21.21 at 130k while R2 reaches 0.9630 at 120k and 0.9596 at 130k. The sentence in Section 5.1, 'When the R2 metric reaches its optimal value, the BLEU score is also relatively high,' is much weaker than the abstract's 'consistently achieve peak performance when the token distribution closely adheres to Zipf's law.' Please quantify the discrepancy and either revise the central claim or demonstrate that the criterion is still predictive under a well-defined stopping rule.
minor comments (5)
  1. [Section 5.1 and Table 1] The phrase 'the R2 value stabilizes at a high value' after 30k is not supported by Table 1, where R2 continues to increase by about 0.002-0.005 per step (0.9372 at 30k, 0.9397 at 35k, 0.9408 at 37.5k, 0.9425 at 40k). Please use a precise criterion or plot with error bars.
  2. [Figure 2] The subplot labels and axis titles in Figure 2b are overlapping and difficult to read (e.g., 'V ocabulary Size' and the three side-by-side performance/R2 legends). Please reformat for clarity.
  3. [Section 4.2] The 'Insight for Bigger Models' paragraph cites Ruder et al. (2019) to argue that small-model conclusions extend to large models, but that reference does not address vocabulary size selection. Either cut this paragraph or add evidence directly relevant to scaling vocabulary size.
  4. [Section 5.1 and Tables 1-4] The paper does not report any variance or significance tests for the performance differences (e.g., Table 1 shows 81.38 at 30k vs 81.19 at 35k). Since each configuration is averaged over three seeds, please report standard deviations or confidence intervals so the reader can judge whether the identified 'peak' is robust.
  5. [Abstract and Section 6] The abstract claims a 'robust and generalizable criterion for vocabulary size selection,' but given the tables, the current evidence supports only a correlation up to the optimum. Please temper the wording to match the actual empirical support.

Circularity Check

1 steps flagged · score 6.0 of 10

The proposed Zipfian criterion is a post-hoc fit: epsilon and N in the stopping rule are never specified, and in the paper's own tables R2 keeps increasing after the performance peak, so the claimed 'R2 stabilizes at the optimum' is read off the performance curves rather than predicted from token statistics.

  1. fitted input called prediction [Section 3.3 (stopping rule) and Section 5.1 (R2 stabilization claim), Tables 1-3]
    "To automatically determine an appropriate vocabulary size, we design an iterative algorithm that gradually grows the vocabulary and monitors how well the resulting token distribution aligns with Zipf's law. ... if the score Zipft fails to exceed Zipfmax by more than a small threshold ϵ after N steps, we consider the Zipfian fit to have stabilized. At this point, the vocabulary is no longer expanded, and the current vocabulary is taken as the optimal set, denoted by Vopt. ..."

    epsilon and N are never specified, so 'stabilized' has no operational meaning. In the paper's own tables, R2 continues to rise after the claimed performance peak: Table 1 has R2=0.9372 at the 30,000 GLUE optimum but R2=0.9425 at 40,000; Table 2 has R2=0.9727 at the 4,000 genomics optimum but R2=0.9744 at 7,000; Table 3 has R2=0.9741 at the 3,000 chemistry optimum but R2=0.9751 at 5,000. Thus the assertion that R2 'stabilizes' at the performance-optimal vocabulary is not computed from token frequencies; it is made by choosing a plateau that matches the performance peak found in the same experiments.

full rationale

The paper's R2 metric is computed from tokenizer frequency distributions alone, so the raw correlation between R2 and downstream performance is not definitionally circular. There is no load-bearing self-citation chain. The circularity lies in the claimed selection criterion: Algorithm 1 (Section 3.3) defines Vopt as the vocabulary at which R2 growth falls below an unspecified threshold for an unspecified number of steps, and the paper then asserts that R2 stabilizes exactly at the empirically best vocabulary. Because epsilon and N are free, 'stabilized' can be made true at any tested vocabulary size; the published tables show R2 still increasing after the performance peak, so the equality between Zipfian saturation and performance optimum is imposed by choosing the plateau to match the outcome rather than read off the token statistics. This makes the central 'Zipfian alignment as a robust criterion' claim non-predictive and partially circular. The empirical observation that R2 and performance often move together remains interesting, but it is not validated as a selection rule without a specified epsilon and an out-of-sample check.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on treating R2 of a log-log line fit as a measure of Zipfian adherence, on a post-hoc plateau rule, and on transferring small-model results to large models. These are assumptions, not derived results, and no new physical or mathematical entity is introduced.

free parameters (3)
  • Stagnation threshold epsilon = not specified
    Algorithm in Section 3.3 stops when the Zipf score improvement falls below epsilon after N steps; the value is never given and the algorithm is not run in Section 5.
  • Stagnation window N = not specified
    Number of consecutive steps with no meaningful improvement; never specified or tested.
  • Slope and intercept of log-log linear fit = fit per vocabulary size
    R2, the proposed alignment score, comes from a least-squares slope and intercept. These are standard regression parameters, but the whole criterion depends on them.
assumptions (4)
  • ad hoc to paper R2 of a least-squares line on log-log rank-frequency data measures closeness to Zipf's law.
    Adopted in Section 5.1 despite the cited Clauset et al. (2009) warning that such fits are not a reliable power-law test.
  • domain assumption Token rank-frequency distributions across NLP, DNA, and SMILES are approximately single-regime power laws.
    Needed for a single straight-line fit to be meaningful; Section 3.2 itself notes real plots often have multiple slopes.
  • ad hoc to paper The vocabulary where R2 growth saturates is the vocabulary where downstream performance peaks.
    This is the bridge between R2 and performance; it is asserted from figures, not derived, and contradicted if the R2 maximum is used.
  • domain assumption Conclusions from small BERT and mBART models transfer to much larger models.
    Section 4.2 relies on Ruder et al. (2019); the paper's own Limitations section says findings may not generalize to billion-parameter models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Pre-trained Models Perform the Best When Token Distributions Follow Zipf's Law." pith.science (2026). https://pith.science/paper/F537CTPT

@misc{pith2026250722543,
  author       = {Pith},
  title        = {Pith review of: Pre-trained Models Perform the Best When Token Distributions Follow Zipf's Law},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F537CTPT}},
  note         = {Machine review of arXiv:2507.22543}
}
read the original abstract

Tokenization is a fundamental step in natural language processing (NLP) and other sequence modeling domains, where the choice of vocabulary size significantly impacts model performance. Despite its importance, selecting an optimal vocabulary size remains underexplored, typically relying on heuristics or dataset-specific choices. In this work, we propose a principled method for determining the vocabulary size by analyzing token frequency distributions through Zipf's law. We show that downstream task performance correlates with how closely token distributions follow power-law behavior, and that aligning with Zipfian scaling improves both model efficiency and effectiveness. Extensive experiments across NLP, genomics, and chemistry demonstrate that models consistently achieve peak performance when the token distribution closely adheres to Zipf's law, establishing Zipfian alignment as a robust and generalizable criterion for vocabulary size selection.

Figures

Figures reproduced from arXiv: 2507.22543 by the authors.

Figure 1
Figure 1. Log-log rank-frequency distribution of different vocabulary sizes on BookCorpus. As the size increases, [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Model performance with different vocabulary sizes across four distinct domains. Model performance [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Case study: With an appropriate vocabulary [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 20 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max L \"u bbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, and 1 others. 2024. Tokenizer choice for LLM training: Negligible or crucial? In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3907--3924

    work page 2024

  3. [3]

    Albert-L \'a szl \'o Barab \'a si and R \'e ka Albert. 1999. Emergence of scaling in random networks. science, 286(5439):509--512

    work page 1999

  4. [4]

    Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, and 1 others. 2016. Findings of the 2016 conference on machine translation ( WMT16 ). In First conference on machine translation, pages 131--198. Association for Computational Linguistics

    work page 2016

  5. [5]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  6. [6]

    Ramon Ferrer Cancho and Ricard V. Solé. 2001. https://doi.org/10.1076/jqul.8.3.165.4101 Two regimes in the frequency of words and the origins of complex lexicons: Zipf's law revisited . Journal of Quantitative Linguistics, 8(3):165--173

    doi:10.1076/jqul.8.3.165.4101 2001

  7. [7]

    Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Niehues, Sebastian St \"u ker, Katsuitho Sudoh, Koichiro Yoshino, and Christian Federmann. 2017. Overview of the iwslt 2017 evaluation campaign. In Proceedings of the 14th International Workshop on Spoken Language Translation, pages 2--14

    work page 2017

  8. [8]

    Mauro Cettolo, Jan Niehues, Sebastian St \"u ker, Luisa Bentivogli, Roldano Cattoni, and Marcello Federico. 2015. https://aclanthology.org/2015.iwslt-evaluation.1/ The IWSLT 2015 evaluation campaign . In Proceedings of the 12th International Workshop on Spoken Language Translation: Evaluation Campaign, pages 2--14, Da Nang, Vietnam

    work page 2015

Show all 38 references
  1. [9]

    Mauro Cettolo, Jan Niehues, Sebastian St \"u ker, Luisa Bentivogli, and Marcello Federico. 2014. Report on the 11th iwslt evaluation campaign. In Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign, pages 2--17

    2014

  2. [10]

    Aaron Clauset, Cosma Rohilla Shalizi, and M. E. J. Newman. 2009. https://doi.org/10.1137/070710111 Power-law distributions in empirical data . SIAM Review , 51(4):661--703

    2009 doi

  3. [11]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT : Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: huma...

    2019

  4. [12]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. https://openreview.net/forum?id=YicbFdNTTy An image is worth 1...

    2021

  5. [13]

    Philip Gage. 1994. A new algorithm for data compression. The C Users Journal, 12(2):23--38

    1994

  6. [14]

    Aaron Gokaslan and Vanya Cohen. 2019. Openwebtext corpus. https://skylion007.github.io/OpenWebTextCorpus/. Accessed: 2024-05-20

    2019

  7. [15]

    Omer Goldman, Avi Caciularu, Matan Eyal, Kris Cao, Idan Szpektor, and Reut Tsarfaty. 2024. https://doi.org/10.18653/v1/2024.findings-acl.134 Unpacking tokenization: Evaluating text compression and its correlation with model performance . In Findings of the Association for Comp...

    2024 doi

  8. [16]

    John J Irwin and Brian K Shoichet. 2005. ZINC --a free database of commercially available compounds for virtual screening. Journal of chemical information and modeling, 45(1):177--182

    2005

  9. [17]

    Hawoong Jeong, Sean P Mason, A-L Barab \'a si, and Zoltan N Oltvai. 2001. Lethality and centrality in protein networks. Nature, 411(6833):41--42

    2001

  10. [18]

    Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V Davuluri. 2021. DNABERT : pre-trained bidirectional encoder representations from transformers model for DNA -language in genome. Bioinformatics, 37(15):2112--2120

    2021

  11. [19]

    Taku Kudo and John Richardson. 2018. https://doi.org/10.18653/v1/D18-2012 S entence P iece: A simple and language independent subword tokenizer and detokenizer for neural text processing . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processin...

    2018 doi

  12. [20]

    Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. https://doi.org/10.1162/tacl_a_00343 Multilingual denoising pre-training for neural machine translation . Transactions of the Association for Computational...

    2020 doi

  13. [21]

    Marcelo A Montemurro. 2001. Beyond the Zipf -- Mandelbrot law in quantitative linguistics. Physica A: Statistical Mechanics and its Applications, 300(3-4):567--578

    2001

  14. [22]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 BLEU : a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...

    2002

  15. [23]

    Vilfredo Pareto. 1964. Cours d' \'e conomie politique , volume 1. Librairie Droz

    1964

  16. [24]

    David M. W. Powers. 1998. Applications and explanations of zipf's law. In Proceedings of the Joint Conferences on New Methods in Language Processing and Computational Natural Language Learning, NeMLaP3/CoNLL '98, page 151–160, USA. Association for Computational Linguistics

    1998

  17. [25]

    Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2020. https://doi.org/10.18653/v1/2020.acl-main.170 BPE -dropout: Simple and effective subword regularization . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1882--1892, O...

    2020 doi

  18. [26]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. PMLR

    2023

  19. [27]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog

    2019

  20. [28]

    Peters, Swabha Swayamdipta, and Thomas Wolf

    Sebastian Ruder, Matthew E. Peters, Swabha Swayamdipta, and Thomas Wolf. 2019. https://doi.org/10.18653/v1/N19-5004 Transfer learning in natural language processing . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Ling...

    2019 doi

  21. [29]

    Mike Schuster and Kaisuke Nakajima. 2012. https://doi.org/10.1109/ICASSP.2012.6289079 Japanese and korean voice search . In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149--5152

    2012

  22. [30]

    Hunter, Costas Bekas, and Alpha A

    Philippe Schwaller, Teodoro Laino, Th \'e ophile Gaudin, Peter Bolgar, Christopher A. Hunter, Costas Bekas, and Alpha A. Lee. 2019. https://doi.org/10.1021/acscentsci.9b00576 Molecular transformer: A model for uncertainty-calibrated chemical reaction prediction . ACS Central S...

    2019 doi

  23. [31]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. https://doi.org/10.18653/v1/P16-1162 Neural machine translation of rare words with subword units . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages ...

    2016 doi

  24. [32]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/W18-5446 GLUE : A multi-task benchmark and analysis platform for natural language understanding . In Proceedings of the 2018 EMNLP Workshop B lackbox NLP : A...

    2018 doi

  25. [33]

    Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, and 1 others. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXi...

    2016 arXiv

  26. [34]

    Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana V Davuluri, and Han Liu. 2024. https://openreview.net/forum?id=oMLQB4EZE1 DNABERT -2: Efficient foundation model and benchmark for multi-species genomes . In The Twelfth International Conference on Learning Representations

    2024

  27. [35]

    Yukun Zhu, Ryan Kiros, Richard S Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE International Conference o...

    2015

  28. [36]

    George Kingsley Zipf. 2013. The psycho-biology of language: An introduction to dynamic philology. Routledge

    2013

  29. [37]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  30. [38]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.