Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A tokenizer trained on more languages than the model gives LLMs much stronger language plasticity, with up to 20% higher win rates on new languages after continued pretraining.

desk verdict A well-executed, practice-relevant study of tokenizer coverage for multilingual plasticity, but the headline effect conflates tokenizer design with a 5% data allocation; the tokenizer-only effect is smaller and only measured for one cluster. read the letter →

arxiv 2506.10766 v1 pith:6EUDR2SB submitted 2025-06-12 cs.CL

Diana Abagyan , Alejandro R. Salamanca , Andres Felipe Cruz-Salinas , Kris Cao , Hangyu Lin , Acyr Locatelli , Marzieh Fadaee , Ahmet Üstün
show 1 more author
Sara Hooker
This is my paper
classification cs.CL
keywords multilingualtokenizationlanguageplasticitycontinuedpretrainingtokenizerdesignbytepairencodinglow-resourcelanguagesvocabularyexpansionLLMadaptation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a cheap up-front choice—training the tokenizer on more languages than the model itself will be pretrained on—makes a language model much easier to adapt to new languages later. Across three geographic language clusters and 69 languages, models pretrained with such a 'universal' BPE tokenizer beat models with cluster-specific tokenizers in continued pretraining, targeted fine-tuning, and adaptation to fully unseen languages. The universal tokenizer raises average win rates on expanded languages by 18.9% in continued pretraining, up to 5% on unseen languages, and reaches the baseline's final performance about 8x faster, while staying within roughly 2% of the baseline on primary languages. The upshot is that practitioners wanting to extend language coverage after training can do so more cheaply by broadening the tokenizer at the very start, instead of relying on post-hoc vocabulary-extension tricks.

What carries the argument

The load-bearing object is the universal tokenizer: a byte-pair-encoding vocabulary of 250k subword tokens trained on all 62 languages using a weighting scheme that balances the natural data distribution against language buckets formed by script and family. Compared with cluster-specific tokenizers, it produces far lower compression ratios on expanded languages, and the paper shows that better compression on those languages correlates with higher post-training win rates. Because the universal tokenizer is used from the start of pretraining, shared-token embeddings are reused naturally, and no post-hoc vocabulary replacement is needed. The paper's ablations show this up-front investment beats the standard alternative of cross-lingual vocabulary adaptation, where a new tokenizer is swapped in after pretraining with randomly or mean-initialized embeddings; the universal tokenizer leads by 7% average win rate over the best replacement variant on expanded languages.

What would settle it

Give the cluster-specific tokenizer exactly the same 5% expanded-language pretraining data as the universal tokenizer and rerun continued pretraining; if the win-rate gap on expanded languages collapses below the 12.8% seen at 0% data, then data allocation rather than tokenization drives the headline gain.

Watch

Extended reading notes

Core claim

The paper's central claim is that tokenizer coverage is a primary, cheap lever for 'language plasticity'—the ability of a model to adapt quickly to new linguistic distributions after pretraining. A universal tokenizer trained on all 62 experimental languages and used from the start of pretraining enables significantly better adaptation to an expanded language set than a cluster-specific tokenizer covering only the primary pretraining languages. In continued pretraining on primary plus expanded languages, universal-tokenizer models average 18.9% higher win rates on the expanded subset across three geo-clusters (up to 19.9% for the European cluster); in targeted supervised fine-tuning on expanded languages only, the average gain is 14.6%; and for seven fully unseen, severely under-resourced languages, the gain reaches up to 5%. Performance on the primary languages stays within about 2% of the cluster baseline, and in open-ended generation the universal tokenizer is often slightly better. The universal tokenizer also adapts much faster, reaching the cluster tokenizer's final expanded-language performance in about 300 training steps instead of 2,500.

Load-bearing premise

The load-bearing premise is that the tokenizer's broader coverage, rather than the 5% of extra pretraining data given only to the universal-tokenizer runs, is what produces the higher win rates; if tokenizer design must be isolated, the clean gain is the 12.8% from the 0%-data condition, not the headline 18.9%.

Editorial extensions

If this is right

  • A provider who plans to expand language coverage later can do so more cheaply by choosing a broad tokenizer at the start, without sacrificing accuracy on the primary languages.
  • The 8x faster adaptation means far less additional data and compute: the universal tokenizer reaches the baseline's final expanded-language win rate in roughly 150K samples versus 1.3M.
  • For severely under-resourced languages absent from all training data, the universal tokenizer reduces the burden of targeted adaptation by up to 5% in win rates.
  • Swapping in a new tokenizer after pretraining (cross-lingual vocabulary adaptation) is clearly inferior to having used a universal tokenizer from the start, even with mean-initialized embeddings for new tokens.
  • Because the universal tokenizer needs a 250k vocabulary to match or beat cluster tokenizers on primary languages, tokenizer budget and model-parameter reallocation matter when adopting this approach.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication not tested in the paper: if tokenizer design rather than data allocation carries the effect, the cleanest estimate of the tokenizer-only gain is the 12.8% win-rate improvement from the 0% expanded-language condition; the headline 18.9% bundles in a 5% data reallocation, so the higher number should be read as the combined intervention.
  • The compression-to-win-rate correlation the paper documents suggests a cheap screening test for tokenizer design: compression ratios on target languages might predict adaptation potential, replacing expensive continued-pretraining runs.
  • A natural scaling prediction follows from the paper's mechanism: at larger model sizes or longer pretraining budgets, token-level compression differences should compound, so the universal tokenizer's advantage may persist or grow; if instead the gap shrinks beyond 3.3B parameters, the result is size-specific.
  • Because only BPE is tested, the paper leaves open whether the plasticity benefit transfers to Unigram, byte-level, or character-level tokenizers; a test on one such algorithm would show whether the effect is about coverage or about BPE's merging behavior.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes using a 'universal tokenizer' trained on a superset of languages beyond the primary pretraining languages, introduced from the start of pretraining, as a cheap intervention to improve 'language plasticity' for later adaptation to new languages. The authors pretrain 3.3B-parameter models for 100B tokens on three geo-clusters (European, Asian, ME-Indic), with Cluster-specific and Universal tokenizers, and evaluate continued pretraining, targeted SFT on expanded languages, and targeted SFT on fully unseen languages. They report substantial win-rate gains for the Universal tokenizer on expanded and unseen languages with no more than about 2% drop on primary languages, plus an 8x faster adaptation claim based on checkpoint comparisons.

Significance. If the headline magnitudes were unconfounded, the result would be practically important: tokenizer design is a relatively cheap pretraining-stage choice, and an 8x adaptation speedup plus 12-20 percentage point win-rate gains would be a valuable recipe for multilingual LLM development. The paper's strengths are its unusually extensive experimental matrix — three clusters, multiple adaptation strategies, fully unseen languages, and a 0% data ablation — and its transparency about the 5% reallocation. The central concern is that the headline 18.9% continued-pretraining gain is not a clean tokenizer-only estimate, and the paper's own Euro-only 0% condition suggests a smaller effect. If the authors add cross-cluster 0% conditions or a Cluster+5% control and provide error bars for the win-rate estimates, the contribution would be solid and likely significant for the multilingual LLM community.

major comments (4)
  1. [§2.2, §4.1, §5.4] The headline continued-pretraining gain of 18.9% (Table 3) is a joint effect of tokenizer coverage and a 5% reallocation of pretraining data from English to expanded languages, because Section 2.2 states that Universal-tokenizer runs receive this 5% allocation while Cluster-tokenizer runs do not. Section 5.4's 0% condition, which removes this confound, is reported only for the European cluster and reduces the gain from 19.9% to 12.8%. Without a 0% condition (or a Cluster+5% control) for the Asian and ME-Indic clusters, the claim that the universal tokenizer 'enables' the 18.9% average gain is not established at the stated magnitude. Please report the 0%/1%/5% sweep for all three clusters or add a Cluster+5% condition, and adjust the abstract and Section 4.1 claims accordingly.
  2. [§5.1, Figure 4] The cross-lingual vocabulary adaptation (CVA) comparison in Section 5.1 is not matched on pretraining data exposure. The Universal-from-start model received 5% expanded-language pretraining data, while the CVA baseline starts from a Cluster-tokenizer model that received no such allocation. The 7% advantage attributed to 'using a Universal tokenizer from the start' may therefore reflect the data allocation rather than tokenizer design alone. Please run a CVA condition with the same 5% expanded-language pretraining data, or explicitly acknowledge that the comparison conflates tokenizer coverage with data exposure.
  3. [§2.4, Tables 3-4, Figure 3b] Win rates are reported as point estimates from LLM-as-a-judge evaluations on 200 held-out examples per language, with no confidence intervals, bootstrap estimates, or repeated judge samples. This is especially problematic for the small but load-bearing unseen-language claims: the 2-5 percentage point gains in Figure 3b are well within the plausible noise range for a 200-example evaluation. Please provide per-language confidence intervals or significance tests, and restrict strong claims to effects that exceed the measurement uncertainty.
  4. [§2.3, Appendix A] The tokenizer training description does not specify byte-fallback or unknown-token handling. This is central because the Cluster tokenizers are trained only on the primary language subsets, while the expanded subsets contain scripts (e.g., Arabic, Hebrew, Devanagari) that are likely absent from those primary subsets. The comparison is ill-defined unless byte-level or byte-fallback behavior is specified and held constant across the Universal and Cluster tokenizers. Please state the byte_fallback setting and confirm it is identical for both tokenizer families.
minor comments (6)
  1. [Abstract, §4.1, Table 3, §5.4] The abstract and conclusion cite an improvement 'up to 20.2%', but Table 3 reports a maximum of 19.9% and Section 5.4 states 19.8% for the same 5% condition; please harmonize these numbers.
  2. [§4.2, Figure 3b] The text attributes the 5.0-point gain to Nepali, but the plotted gain for Nepali (npi) is 12-9=3 points; the 5-point gain corresponds to Kazakh (kaz: 14-9=5). Please correct the language attribution.
  3. [Appendix C.2.2] The section heading and text say 'Table 5 presents the win rates', but the table shown is Table 6 (targeted adaptation). Please fix the cross-reference.
  4. [§4.1, Table 2, Table 5] Section 4.1 states that each primary language subset contains 5 languages and the expanded subset contains 10, but Table 2 evaluates the Euro model on 27 European languages and Section 3 calls these 'primary'. Please clarify the composition of the primary and expanded subsets for each cluster and make the tables consistent.
  5. [§2.1, Abstract, §7] The paper variously states 62, 69, and 70 languages (e.g., Section 2.1 says 62, the abstract and limitations say 69/70). Please make the language counts consistent throughout.
  6. [§5.2, Figure 1] The '8x faster adaptation' claim is based on a single Euro continued-pretraining run with intermediate checkpoint evaluations; please state whether the speed advantage is also observed in the Asian and ME-Indic clusters and whether the 300-step difference is statistically distinguishable, especially given the small evaluation sets.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the tokenizer is an input condition, outcomes are held-out benchmarks, and the disclosed 5% data reallocation is a confounding manipulation rather than a circular derivation.

full rationale

The paper's derivation chain is empirical: a universal BPE tokenizer is trained on 62 languages, models are pretrained with it, and adaptation is measured by win rates on held-out Dolly/Aya evaluation splits plus Belebele and M-MMLU. The tokenizer is an input condition, not a parameter fitted to the reported outcome, and no equation in the paper defines the outcome in terms of the intervention. The headline +18.9% continued-pretraining gain is partially attributable to the disclosed 5% reallocation of pretraining data toward expanded languages (Section 2.2), which is a confounding manipulation rather than a circular derivation; the Section 5.4 ablation at 0% expanded-language pretraining data reports a still-positive 12.8% gain for the Euro cluster, providing independent evidence for the tokenizer-level effect. Self-citations to Aya, Command-A, and Cohere works are contextual: they supply evaluation data, the judge model, and training recipes, but the central comparison (Universal vs. Cluster tokenizers) is implemented and measured in this paper against external datasets. The Limitations section notes scope constraints (BPE only, 3.3B models, 69 languages) that bound generality but do not make the result equivalent to its inputs. Under the rule that only explicit reductions or fit-renamed-as-prediction count, no circular step is present.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new theoretical entities or fitted laws are introduced. The central quantities are experimental design choices: the 5% expanded-language data allocation and the 250k vocabulary size. The main assumptions are about scale transfer, judge validity, data leakage, and tokenizer-algorithm generality.

free parameters (2)
  • Expanded-language pretraining proportion = 5% (ablated at 0%, 1%, 5%)
    In Section 2.2, Universal runs reallocate 5% of the training mix from English to expanded languages; this hand-set value inflates the headline plasticity gains and is not a tokenizer property. The 0% ablation in Section 5.4 gives the cleanest tokenizer-only comparison.
  • Vocabulary size = 250,000 tokens
    Section 5.3 ablates 100k, 175k, and 250k and chooses 250k because the Universal tokenizer needs a large vocabulary to match Cluster performance on primary languages. This is a per-run design choice, not a fitted parameter.
assumptions (4)
  • domain assumption The 3.3B-parameter, 100B-token pretraining setup provides sufficient signal to rank tokenizer strategies at scale.
    Stated in Section 2.2 and Limitations: the goal is relative signal, not full-scale performance. Section 7 assumes results transfer to larger models and budgets.
  • domain assumption LLM-as-a-judge win rates with Command-A are a valid measure of generation quality for low-resource languages.
    Section 2.4 relies on prior work that judges align with human preferences; no human validation is reported for the specific language set.
  • domain assumption Languages labeled 'fully unseen' are absent from both the universal tokenizer and pretraining corpora.
    Section 5.4 concedes contamination makes it hard to claim a language is truly new; no leakage audit is provided.
  • domain assumption BPE is a representative tokenization algorithm, and conclusions would transfer to Unigram or byte-level tokenizers.
    Stated in Limitations as an untested assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers." pith.science (2026). https://pith.science/paper/6EUDR2SB

@misc{pith2026250610766,
  author       = {Pith},
  title        = {Pith review of: One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6EUDR2SB}},
  note         = {Machine review of arXiv:2506.10766}
}
read the original abstract

Pretraining massively multilingual Large Language Models (LLMs) for many languages at once is challenging due to limited model capacity, scarce high-quality data, and compute constraints. Moreover, the lack of language coverage of the tokenizer makes it harder to address the gap for new languages purely at the post-training stage. In this work, we study what relatively cheap interventions early on in training improve "language plasticity", or adaptation capabilities of the model post-training to new languages. We focus on tokenizer design and propose using a universal tokenizer that is trained for more languages than the primary pretraining languages to enable efficient adaptation in expanding language coverage after pretraining. Our systematic experiments across diverse groups of languages and different training strategies show that a universal tokenizer enables significantly higher language adaptation, with up to 20.2% increase in win rates compared to tokenizers specific to pretraining languages. Furthermore, a universal tokenizer also leads to better plasticity towards languages that are completely unseen in the tokenizer and pretraining, by up to 5% win rate gain. We achieve this adaptation to an expanded set of languages with minimal compromise in performance on the majority of languages included in pretraining.

Figures

Figures reproduced from arXiv: 2506.10766 by the authors.

Figure 1
Figure 1. Universal tokenizer exhibits +2x higher plasticity with +8x faster adaptation compared to the cluster-specific baseline tokenizer. Average win rates on the expanded (new) language subset during continued pretraining that involves both primary and expanded language subsets (§ 2.1). This imbalance in language coverage has created a growing divide in the cost of use for particular language users as marginalized languag… view at source ↗
Figure 2
Figure 2. Win rates for models trained with the Universal and Cluster tokenizers against Dolly generations [Singh et al., 2024]. The shaded blue portion represents a subset of primary languages, and the white represents expanded languages. On average, Universal tokenizer achieves an increase of 18.9% win rate on expanded subset languages, and +0.3% on primary languages compared to the Cluster tokenizers across clusters. to th… view at source ↗
Figure 3
Figure 3. Language-specific results after targeted adaptation through SFT for the Euro cluster model to expanded languages (a) and fully unseen languages (b). Universal tokenizer outperforms the Cluster tokenizer in both language subsets, where the relative gains over the cluster-specific tokenizer go up to 22% on expanded languages (Hindi), and 5% on unseen languages (Kazakh). An experimental setting of great interest is the… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Win rates after continued pretraining, comparing Cross-lingual Vocabulary Adaptation (CVA) [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: (a) Universal tokenizer requires a larger vocabulary size to achieve the same (or better) pretraining performance with the Cluster tokenizers on primary languages, as evaluated in Belebele. (b) Universal tokenizer exhibits significantly higher adaptation gains over the…
Figure 6
Figure 6. Figure 6: Average performance on primary languages during pretraining (Euro), measured in Belebele. Shaded [PITH_FULL_IMAGE:figures/full_fig_p033_6.png]
Figure 7
Figure 7. Figure 7: Compression ratios for our tokenizer and the baseline uniform tokenizer. Our tokenizer uses a special [PITH_FULL_IMAGE:figures/full_fig_p035_7.png]
Figure 8
Figure 8. Figure 8: Adaptation results per language in Euro cluster together with tokenizers’ compression ratio. While [PITH_FULL_IMAGE:figures/full_fig_p036_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Weight Decay Improves Language Model Plasticity

    cs.LG 2026-02 conditional novelty 6.0 of 10

    Pretrained models trained with larger weight decay fine-tune better on downstream tasks, so the best pretraining checkpoint by loss is not always the best starting point for later training.

Reference graph

Works this paper leans on

13 extracted references · 7 canonical work pages · cited by 1 Pith paper

  1. [1]

    It should have correct reasoning,"

    work page

  2. [2]

    It should answer the request in the instruction,

    work page

  3. [3]

    It should be factually correct and semantically comprehensible,

    work page

  4. [4]

    Instruction: instruction Answer (A): completion_a Answer (B): completion_b FIRST provide a concise comparison of the two answers

    It should be grammatically correct and fluent. Instruction: instruction Answer (A): completion_a Answer (B): completion_b FIRST provide a concise comparison of the two answers. If one answer is better, explain which you prefer and why. If both answers are identical or equally good or bad, explain why. SECOND, on a new line, state exactly one of ’Answer (A...

    work page 2025

  5. [6]

    Egalitarian Language Representation in Language Models: It All Begins with Tokenizers

    Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.845. URL https://aclanthology.org/2024.acl-long.845/. AshishVaswani, NoamShazeer, NikiParmar, JakobUszkoreit, LlionJones, AidanNGomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and ...

    work page Pith review arXiv 2024

  6. [7]

    URLhttps://aclanthology.org/2025.coli ng-main.400/

    Association for Computational Linguistics. URLhttps://aclanthology.org/2025.coli ng-main.400/. Shumin Wang, Yuexiang Xie, Bolin Ding, Jinyang Gao, and Yanyong Zhang. Language adap- tation of large language models: An empirical study on LLaMA2. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schock- aert (ed...

    work page 2025

  7. [9]

    Scalable Training of Language Models using JAX pjit and TPUv4

    Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.653. URL https://aclanthology.org/2023.acl-long.653/. Joanna Yoo, Kuba Perlin, Siddhartha Rao Kamalakara, and João GM Araújo. Scalable training of language models using jax pjit and tpuv4.arXiv preprint arXiv:2204.06514, 2022. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, ...

    work page Pith review arXiv 2023

  8. [800]

    José Pombal, Dongkeun Yoon, Patrick Fernandes, Ian Wu, Seungone Kim, Ricardo Rei, Graham Neubig, and André FT Martins

    URL https://aclanthology.org/2021.emnlp-main.800/. José Pombal, Dongkeun Yoon, Patrick Fernandes, Ian Wu, Seungone Kim, Ricardo Rei, Graham Neubig, and André FT Martins. M-prometheus: A suite of open multilingual llm judges.arXiv preprint arXiv:2504.04953, 2025. Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre- train...

    arXiv 2021

Show all 13 references
  1. [2019]

    doi: 10.18653/v1/D19-1141

    Association for Computational Linguistics. doi: 10.18653/v1/D19-1141. URL https: //aclanthology.org/D19-1141/. Omer Goldman, Avi Caciularu, Matan Eyal, Kris Cao, Idan Szpektor, and Reut Tsarfaty. Un- packing tokenization: Evaluating text compression and its correlation with mo...

    arXiv

  2. [2020]

    doi: 10.18653/v1/2020.findings-emnlp.240

    Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp.240. URL https://aclanthology.org/2020.findings-emnlp.240/. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively mul...

    2020

  3. [2023]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt

    URL https://arxiv.org/abs/2311.08391. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. Valentin Hofmann, Hinrich Schuetze, and Janet Pierrehumbert. An embarrassingly...

    2021 arXiv

  4. [2024]

    doi: 10.18653/v1/2024.naacl-long.24

    Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.24. URL https://aclanthology.org/2024.naacl-long.24/. 17 Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip H. S. Torr, and Marc’Aurelio Ranzato. On...

    2024 arXiv

  5. [2025]

    Noam Shazeer

    URL https://arxiv.org/abs/2505.09738. Noam Shazeer. Glu variants improve transformer, 2020. URLhttps://arxiv.org/abs/2002.052 02. Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F. Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas...

    2020 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.