Pith. sign in

REVIEW 2 major objections 6 minor 48 references

Tokenization invariance fails outside English: multilingual LLMs drop sharply when the same text is segmented differently.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 19:47 UTC pith:TMUVX7WK

load-bearing objection Solid multilingual eval paper: English-style tokenization invariance does not hold, and the drops track fragmentation in a clean, multi-model way. the 2 major comments →

arxiv 2607.26831 v1 pith:TMUVX7WK submitted 2026-07-29 cs.CL

Language Models are not Equally Robust to Non-Canonical Tokenization across Languages

Poulami Ghosh , Preethi Jyothi This is my paper
classification cs.CL
keywords tokenization robustnessnon-canonical tokenizationmultilingual LLMstoken fragmentationLoRA fine-tuningcross-lingual transfersubword tokenization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Language models usually see only one official way of chopping text into tokens, even though many other valid chops decode to the same string. Earlier English results suggested modern instruction-tuned models barely notice those alternative chops. This paper asks whether that holds across languages. Across 27 languages, three model families, and six tasks, it does not: performance falls by roughly 10–24% on average under random non-canonical tokenizations, and more under character-level splits, with steeper drops where the tokenizer already fragments text more. The authors treat that sensitivity as a diagnostic of how tightly a model is locked to its tokenizer, and show that LoRA fine-tuning on multiple tokenizations of the same English examples transfers robustness gains to other languages.

Core claim

Tokenization invariance observed for English does not generalize. Instruction-tuned multilingual LLMs change behavior substantially under non-canonical but decode-identical tokenizations, with average relative drops of about 23.7% (Llama-3.1-8B), 11.4% (Qwen3-8B), and 9.9% (Gemma-3-12B) across six tasks and 27 languages; higher canonical token fragmentation predicts greater sensitivity, and multi-tokenization LoRA training—even on English alone—mitigates it.

What carries the argument

The set of valid tokenizations T(x): all token sequences that decode back to the same string x, contrasted with the single canonical tokenization. Sensitivity is measured as the performance gap between canonical and uniformly sampled non-canonical (or character-level) members of T(x), with fragmentation rate (tokens per word) as the main language-side predictor.

Load-bearing premise

That English-only multi-tokenization fine-tuning on one benchmark’s train split is enough to claim a general fix for tokenization sensitivity across languages and tasks.

What would settle it

Run the same multi-tokenization LoRA recipe on held-out multilingual tasks beyond Multilingual-ARC (and on non-English training data); if relative drops under random and character tokenizations do not shrink as claimed, the mitigation claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Robustness to alternative valid segmentations is language- and tokenizer-dependent, not a free property of stronger models.
  • Canonical fragmentation rate is a practical predictor of which languages will break under non-canonical inputs.
  • Tokenizer adaptation, vocabulary pruning, or transfer that changes segmentations can silently hurt multilingual accuracy even when surface text is unchanged.
  • Exposing models to diverse valid tokenizations at training time can raise both canonical accuracy and cross-lingual robustness.
  • Sentence-level embedding tasks (cross-lingual retrieval) are also sensitive, so the effect is not limited to generation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Safety and alignment filters that assume canonical segmentation may be unevenly bypassable across languages if adversarial non-canonical paths are cheaper in high-fragmentation scripts.
  • Tokenizer design that equalizes fragmentation across languages may buy more inference robustness than scale alone.
  • Evaluation suites that only score canonical tokenizations likely overstate multilingual reliability relative to real tokenizer-change scenarios.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper asks whether English-centric findings of LLM robustness to non-canonical but decode-identical tokenizations generalize multilingually. Across 27 languages, six downstream tasks, and three instruction-tuned models (Llama-3.1-8B-Instruct, Qwen3-8B, Gemma-3-12B-it), it reports large average relative drops under random non-canonical and character-level tokenizations (Table 1: ~23.8%/41.0% Llama, ~11.4%/16.1% Qwen, ~9.9%/20.9% Gemma). Sensitivity is systematic: higher canonical fragmentation predicts greater vulnerability (logistic analysis, Table 10; granularity buckets, Fig. 4), and length-matched controls (Table 6) indicate the effect is not solely sequence length. Cross-lingual retrieval (Figs. 5–7) and scaling (Table 2) reinforce the pattern. As mitigation, English-only LoRA multi-tokenization training on Multilingual-ARC (especially LR-Bucketed N×) improves canonical and non-canonical accuracy across languages (Table 3).

Significance. If the primary negative result holds—and the multi-model, multi-task, multi-language evidence with length controls and fragmentation analysis makes it persuasive—the paper supplies a clear, falsifiable diagnostic that tokenization invariance is not a universal LLM property but depends on language–tokenizer interaction. That is a useful corrective to English-only robustness claims and is practically relevant to tokenizer adaptation, vocabulary pruning, and cross-lingual transfer. Strengths include a clean definition of the tokenization space via MDDs, decode-identical perturbations, breadth of evaluation, and explicit length-ratio controls. The mitigation result is interesting as data augmentation via the tokenization lattice, including cross-lingual transfer from English-only fine-tuning, but is narrower in scope than the robustness diagnosis.

major comments (2)
  1. [§6–8, Table 3, Abstract, Limitations] Abstract, §6–8, and Table 3: the claim that “LoRA fine-tuning with multi-tokenization training data provides an effective mitigation for tokenization sensitivity” (and that English-only fine-tuning improves robustness “across languages”) is supported only on Multilingual-ARC, the sole benchmark with a train split (§6.2; Limitations). Model selection also uses an ARC-derived multi-tokenization validation set. This does not undermine the primary invariance-failure result, but it overreaches as stated. Either (i) add transfer evaluation of the ARC-tuned adapters on at least one held-out task family (e.g., Belebele or MLQA) under the same tokenization conditions, or (ii) explicitly scope the mitigation claim in abstract/conclusion to ARC-style multiple-choice reasoning and treat broader generality as a hypothesis.
  2. [§5.1, Table 10] §5.1 and Table 10: the fragmentation–sensitivity link is central (“Languages that exhibit higher token fragmentation show significantly greater sensitivity”). The logistic regression is reported in detail only for English (and briefly a few others in Table 10), with β_FR = −1.11 on ARC. For a claim framed as systematic across 27 languages, please report the FR coefficient (and significance) for the full language set used in the main average, or a pooled/mixed-effects analysis with language as a factor, so readers can judge how general the FR predictor is versus English-specific structure.
minor comments (6)
  1. [Figure 1] Figure 1 caption and body: Hungarian example text is Dutch (“Kleine stapjes…”); the language label should match the string or the string should be replaced with Hungarian.
  2. [Table 1, Abstract] Table 1 header uses “Llama-3.1-8B-Instruct” while the abstract cites “Llama-3.1-8B”; keep naming consistent with the instruct checkpoints actually evaluated.
  3. [§3] §3: Err(t1,t2)=d(f(t1),f(t2)) is defined abstractly; state explicitly that in the main tables d is relative task-metric drop vs canonical, not a representation distance, to avoid confusion with the retrieval experiments.
  4. [§4, Appendix C] §4 / language filter: the exclusion of near-character-canonical languages for Qwen is reasonable for cross-model fairness, but a one-sentence pointer in the main text (not only Appendix C) would help readers interpret the 27-language set.
  5. [Intro, §5.1] Minor prose: “through through embedding transfer” (Intro); “we observe that that short-answer” (§5.1); standardize “Rand Δ%” vs “Rand∆%” notation across tables.
  6. [§2] Related work could briefly situate subword regularization (Kudo 2018; Cognetta et al.) against the inference-time MDD sampling used here, since both exploit T(x).

Circularity Check

0 steps flagged

No circularity: empirical robustness gaps are measured on external benchmarks under decode-identical tokenizations, not derived from fitted targets or self-defining premises.

full rationale

This paper is a comparative empirical study, not a first-principles derivation. Non-canonical tokenizations are defined by decode(t)=x with t≠T_canon(x); Err(tc, tn) is then measured as task performance on external multilingual benchmarks (MGSM, MLQA, Multi-ARC, etc.). Fragmentation rate is an independent tokenizer statistic on FLORES-200, used as a correlate via logistic regression, not fitted to force the robustness claim. Length-matched controls (Table 6) and granularity buckets further separate boundary effects from sequence length. The LoRA multi-tokenization results (Table 3) are held-out test metrics after English-only fine-tuning; any scope limit (ARC-only training) is a generality caveat, not circularity. Citations to Zheng et al. (2025) and Geh et al. motivate the English baseline and MDD sampling but do not supply a uniqueness theorem or ansatz that forces the multilingual negative finding. No step reduces a claimed prediction to its own inputs by construction.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

Load-bearing content is empirical protocol, not a formal derivation. The claim rests on standard LM evaluation assumptions, the definition that decode-identical token sequences isolate segmentation effects, uniform/MDD sampling as a probe of the tokenization space, and the choice to treat relative performance drop under those probes as the robustness metric. Mitigation generality further assumes ARC English multi-tokenization is representative. No new physical entities; free choices are experimental knobs (N samples, length-ratio buckets, LoRA setup, language inclusion).

free parameters (4)
  • Number of uniform non-canonical samples N and seeds (N path samples, 5 seeds) = 5 seeds (main eval); multi-copy N in {10,20,30} for fine-tuning
    Random-tokenization metrics are Monte Carlo estimates over the MDD; reported RandΔ% depends on this sampling budget and seed set.
  • Length-ratio bucket boundaries for granularity and LR-Bucketed training = eight equal 0.25-width ratio bins up to 3.00
    Buckets [1.00,1.25)…[2.75,3.00] define both the granularity analysis and the strongest training scheme; boundaries are design choices, not derived.
  • LoRA and optimization hyperparameters = r=16, α=32, lr=2.5×10^-6, 5 epochs
    Mitigation results depend on r=16, α=32, dropout 0.1, lr=2.5e-6, 5 epochs, batch/accum settings (Table 7).
  • Language inclusion filter (27-language main set) = 27 languages in Table 9; Indic/Armenian set moved to appendix
    Languages with near character-level canonical tokenization under Qwen were excluded from the main comparison; this choice shapes average drops and cross-model fairness.
axioms (5)
  • domain assumption If decode(t1)=decode(t2)=x, differences in f(t1) vs f(t2) are attributable to segmentation, not to string identity.
    Stated in §3 Evaluation; underpins all Err(tc, tn) measurements. Ignores secondary effects (positional length, special-token borders) except where length is later controlled.
  • domain assumption Uniform sampling over MDD paths is a fair probe of 'non-canonical tokenization' robustness.
    §3 follows Geh et al.; adversarial or likelihood-weighted tokenizations could yield different drops.
  • ad hoc to paper Relative performance drop from canonical to non-canonical/character tokenization is the right summary of invariance failure.
    Table 1 headline metrics; absolute accuracy and calibration are secondary.
  • domain assumption Instruction-tuned open-weight models (Llama-3.1-8B-Instruct, Qwen3-8B, gemma-3-12b-it) and the chosen lm-eval tasks represent multilingual LLM behavior under retokenization.
    §4 Experimental Setup; no closed models, no base (non-instruct) full sweep in main tables.
  • ad hoc to paper Retokenizing the full user prompt including MC options, but not system prompt/special tokens, is the appropriate inference perturbation.
    §4 Task paragraph; different prompt-boundary choices could change measured sensitivity.
invented entities (2)
  • LR-Bucketed (N×) multi-tokenization training scheme no independent evidence
    purpose: Data-augmentation schedule that covers canonical, character, and eight length-ratio non-canonical buckets per example group.
    Introduced in §6.1 as the strongest mitigation variant; it is a procedural construct, not a latent model component, with evidence only from ARC LoRA runs.
  • Tokenization invariance / Err(t1,t2) robustness diagnostic tied to fragmentation independent evidence
    purpose: Frame non-canonical drop as a measure of how tightly a model is coupled to its tokenizer.
    Defined in §3 and used as the paper’s interpretive lens; operationalized via existing tasks rather than a new measurable physical quantity.

pith-pipeline@v1.2.0-daily-grok45 · 23912 in / 3709 out tokens · 77595 ms · 2026-07-30T19:47:20.886474+00:00 · methodology

0 comments
read the original abstract

Despite the existence of exponentially many valid tokenizations for a given string, language models operate on a single canonical sequence deterministically produced by the tokenizer, leaving the broader tokenization space largely uncharacterized. In this paper, we investigate this overlooked space by studying the behavior of language models under non-canonical tokenizations across diverse languages. For English, prior work shows that models are largely invariant to alternative tokenizations that represent the same underlying string. We ask whether this invariance generalizes to other languages beyond English. We conduct a multilingual study across 27 languages spanning diverse scripts and evaluate LLM behavior under alternative tokenizations across six downstream tasks. We find that tokenization invariance does not generalize: model behavior varies substantially across languages with instruction-tuned models exhibiting an average relative performance drop of 23.7% for Llama-3.1-8B, 11.4% for Qwen3-8B, and 9.9% for Gemma-3-12B. The variation of tokenization invariance is systematic across languages. Languages that exhibit higher token fragmentation show significantly greater sensitivity to non-canonical tokenizations. Our study of tokenization robustness serves as a diagnostic of how tightly a model is coupled to its tokenizer. These results demonstrate that tokenization robustness is not a universal property of language models, but depends strongly on the language and its interaction with the tokenizer. We also show that LoRA fine-tuning with multi-tokenization training data provides an effective mitigation for tokenization sensitivity. Fine-tuning on English alone improves tokenization robustness across languages, while systematically sampling diverse non-canonical tokenizations achieves the strongest overall performance.

Figures

Figures reproduced from arXiv: 2607.26831 by Poulami Ghosh, Preethi Jyothi.

Figure 1
Figure 1. Figure 1: LLM robustness to non-canonical tokenization is [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: MDD of the word “time” across different languages. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Impact of granularity of non-canonical tok [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 3
Figure 3. Figure 3: Fragmentation across languages under canon [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Retrieval experiments on FLORES-200 for Llama are comparatively modest, suggesting diminishing returns with further increases in model size. 5.4 Cross-lingual experiments In addition to benchmark evaluation, we further probe into how token segmentations influence the LLM’s internal representations. We perform a cross-lingual sentence retrieval experiment using the FLORES-200 multilingual benchmark dataset.… view at source ↗
Figure 6
Figure 6. Figure 6: Retrieval experiments on FLORES-200 for Qwen [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Retrieval experiments on FLORES-200 for Gemma [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Impact of granularity of non-canonical tokenization on MGSM performance. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Impact of LoRA fine-tuning across languages. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

48 extracted references · 1 canonical work pages

  1. [2]

    Proceedings of the 58th annual meeting of the association for computational linguistics , pages=

    MLQA: Evaluating cross-lingual extractive question answering , author=. Proceedings of the 58th annual meeting of the association for computational linguistics , pages=

  2. [3]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages=

    Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages=

    2023

  3. [4]

    arXiv preprint arXiv:2210.03057 , year=

    Language models are multilingual chain-of-thought reasoners , author=. arXiv preprint arXiv:2210.03057 , year=

    Pith/arXiv arXiv

  4. [5]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  5. [6]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    The belebele benchmark: a parallel reading comprehension dataset in 122 language variants , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  6. [7]

    arXiv preprint arXiv:2508.04796 , year=

    Parity-aware byte-pair encoding: Improving cross-lingual fairness in tokenization , author=. arXiv preprint arXiv:2508.04796 , year=

    Pith/arXiv arXiv

  7. [14]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Adversarial tokenization , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  8. [15]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Where is the signal in tokenization space? , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  9. [16]

    Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages=

    You should evaluate your language model on marginal likelihood over tokenisations , author=. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages=

    2021

  10. [17]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages=

    Should you marginalize over possible tokenizations? , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages=

  11. [18]

    Tokenization Workshop , year=

    Canonical autoregressive generation , author=. Tokenization Workshop , year=

  12. [19]

    arXiv preprint arXiv:2506.07956 , year=

    Language models over canonical byte-pair encodings , author=. arXiv preprint arXiv:2506.07956 , year=

    Pith/arXiv arXiv

  13. [20]

    Natural Language Engineering , volume=

    Emerging trends: Subwords, seriously? , author=. Natural Language Engineering , volume=. 2020 , publisher=

    2020

  14. [21]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Distributional properties of subword regularization , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  15. [25]

    2025 , eprint=

    Gemma 3 Technical Report , author=. 2025 , eprint=

    2025

  16. [30]

    Global MMLU : Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation

    Singh, Shivalika and Romanou, Angelika and Fourrier, Cl \'e mentine and Adelani, David Ifeoluwa and Ngui, Jian Gang and Vila-Suero, Daniel and Limkonchotiwat, Peerat and Marchisio, Kelly and Leong, Wei Qi and Susanto, Yosephine and Ng, Raymond and Longpre, Shayne and Ruder, Sebastian and Ko, Wei-Yin and Bosselut, Antoine and Oh, Alice and Martins, Andre a...

    2025

  17. [31]

    Advances in neural information processing systems , volume=

    Language model tokenizers introduce unfairness between languages , author=. Advances in neural information processing systems , volume=

  18. [32]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    Do all languages cost the same? tokenization in the era of commercial language models , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    2023

  19. [37]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    TokAlign: Efficient vocabulary adaptation via token alignment , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  20. [40]

    Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R Mortensen, Noah A Smith, and Yulia Tsvetkov. 2023. Do all languages cost the same? tokenization in the era of commercial language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9904--9923

    2023

  21. [41]

    Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. https://doi.org/10.18653/v1/2024.acl-long.44 The belebele benchmark: a parallel reading comprehension dataset in 122 language variants . In Proceedings of the 62nd Annual Meeting of t...

    doi:10.18653/v1/2024.acl-long.44 2024

  22. [42]

    Kris Cao and Laura Rimell. 2021. You should evaluate your language model on marginal likelihood over tokenisations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2104--2114

    2021

  23. [43]

    Nadezhda Chirkova, Germ \'a n Kruszewski, Jos Rozen, and Marc Dymetman. 2023. Should you marginalize over possible tokenizations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1--12

    2023

  24. [44]

    Marco Cognetta, Vil \'e m Zouhar, and Naoaki Okazaki. 2024. Distributional properties of subword regularization. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10753--10763

    2024

  25. [45]

    Marta R Costa-Juss \`a , James Cross, Onur C elebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, and 1 others. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672

    Pith/arXiv arXiv 2022

  26. [46]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others. 2023. https://doi.org/10.5281/zenodo.10256836 A framework...

    doi:10.5281/zenodo.10256836 2023

  27. [47]

    Renato Geh, Zilei Shao, and Guy Van den Broeck. 2025. Adversarial tokenization. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 20738--20765

    2025

  28. [48]

    Renato Geh, Honghua Zhang, Kareem Ahmed, Benjie Wang, and Guy Van den Broeck. 2024. Where is the signal in tokenization space? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3966--3979

    2024

  29. [49]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    Pith/arXiv arXiv 2024

  30. [50]

    Hoyeol Kim and Hyeonwoo Kim. 2026. Optimizing korean-centric llms via token pruning. arXiv preprint arXiv:2604.16235

    Pith/arXiv arXiv 2026

  31. [51]

    Taku Kudo. 2018. https://doi.org/10.18653/v1/P18-1007 Subword regularization: Improving neural network translation models with multiple subword candidates . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66--75, Melbourne, Australia. Association for Computational Linguistics

    doi:10.18653/v1/p18-1007 2018

  32. [52]

    Taku Kudo and John Richardson. 2018. https://doi.org/10.18653/v1/D18-2012 S entence P iece: A simple and language independent subword tokenizer and detokenizer for neural text processing . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66--71, Brussels, Belgium. Association for Compu...

    doi:10.18653/v1/d18-2012 2018

  33. [53]

    Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen. 2023. https://doi.org/10.18653/v1/2023.emnlp-demo.28 Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process...

    doi:10.18653/v1/2023.emnlp-demo.28 2023

  34. [54]

    Chong Li, Yingzhuo Deng, Wen Yang, Jiajun Zhang, and Chengqing Zong. 2026. Tokalign++: Advancing vocabulary adaptation via better token alignment. arXiv preprint arXiv:2605.13429

    Pith/arXiv arXiv 2026

  35. [55]

    Chong Li, Jiajun Zhang, and Chengqing Zong. 2025. Tokalign: Efficient vocabulary adaptation via token alignment. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4109--4126

    2025

  36. [56]

    Tomasz Limisiewicz, Ji r \'i Balhar, and David Mare c ek. 2023. https://doi.org/10.18653/v1/2023.findings-acl.350 Tokenization impacts multilingual language modeling: Assessing vocabulary allocation and overlap across languages . In Findings of the Association for Computational Linguistics: ACL 2023, pages 5661--5681, Toronto, Canada. Association for Comp...

    doi:10.18653/v1/2023.findings-acl.350 2023

  37. [57]

    Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. 2023. Language model tokenizers introduce unfairness between languages. Advances in neural information processing systems, 36:36963--36990

    2023

  38. [58]

    Yamshchikov, and Mark Fishel

    Taido Purason, Pavel Chizhov, Ivan P. Yamshchikov, and Mark Fishel. 2026. https://doi.org/10.18653/v1/2026.findings-eacl.341 Teaching old tokenizers new words: Efficient tokenizer adaptation for pretrained models . In Findings of the A ssociation for C omputational L inguistics: EACL 2026 , pages 6492--6516, Rabat, Morocco. Association for Computational L...

    doi:10.18653/v1/2026.findings-eacl.341 2026

  39. [59]

    Yuval Reif, Guy Kaplan, and Roy Schwartz. 2026. https://doi.org/10.18653/v1/2026.findings-acl.1618 Vocab diet: Reshaping the vocabulary of LLM s via vector arithmetic . In Findings of the A ssociation for C omputational L inguistics: ACL 2026 , pages 32334--32352, San Diego, California, United States. Association for Computational Linguistics

    work page doi:10.18653/v1/2026.findings-acl.1618 2026

  40. [60]

    Fran c ois Remy, Pieter Delobelle, Hayastan Avetisyan, Alfiya Khabibullina, Miryam de Lhoneux, and Thomas Demeester. 2024. Trans-tokenization and cross-lingual vocabulary transfers: Language adaptation of llms for low-resource nlp. arXiv preprint arXiv:2408.04303

    Pith/arXiv arXiv 2024

  41. [61]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. https://doi.org/10.18653/v1/P16-1162 Neural machine translation of rare words with subword units . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715--1725, Berlin, Germany. Association for Computational Linguistics

    doi:10.18653/v1/p16-1162 2016

  42. [62]

    Shivalika Singh, Angelika Romanou, Cl \'e mentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, Alice Oh, Andre Martins, Leshem Choshen, Daphne Ippolito, and 4 others. 2025. https://doi....

    doi:10.18653/v1/2025.acl-long.919 2025

  43. [63]

    Sophie Steger, Rui Li, Sofiane Ennadir, Anya Sims, Arno Solin, Franz Pernkopf, and Martin Trapp. 2026. Stochasticity in tokenisation improves robustness. arXiv preprint arXiv:2604.16037

    Pith/arXiv arXiv 2026

  44. [64]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025. https://arxiv.org/abs/2503.19786...

    Pith/arXiv arXiv 2025

  45. [65]

    Tim Vieira, Ben LeBrun, Mario Giulianelli, Juan Luis Gastaldi, Brian DuSell, John Terilla, Timothy J O'Donnell, and Ryan Cotterell. 2024. From language models over tokens to language models over characters. arXiv preprint arXiv:2412.03719

    Pith/arXiv arXiv 2024

  46. [66]

    Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, and 1 others. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144

    Pith/arXiv arXiv 2016

  47. [67]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388

    Pith/arXiv arXiv 2025

  48. [68]

    Brian Siyuan Zheng, Alisa Liu, Orevaoghene Ahia, Jonathan Hayase, Yejin Choi, and Noah A Smith. 2025. Broken tokens? your language model can secretly handle non-canonical tokenizations. arXiv preprint arXiv:2506.19004

    arXiv 2025