Pith. sign in

REVIEW 3 major objections 4 minor 43 references

Pre-trained LLMs can swap to a bigger tokenizer in place, preserving quality and cutting decode time for under-tokenized languages.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 23:46 UTC pith:AOLNCIKK

load-bearing objection Practical in-place tokenizer expansion recipe with real weights, but the headline speedup rests on a compression ratio measured on web text, not on the model's own output. the 3 major comments →

arxiv 2607.15232 v1 pith:AOLNCIKK submitted 2026-07-16 cs.CL cs.AIcs.LG

In-Place Tokenizer Expansion for Pre-trained LLMs

Jimmy T.H. Smith , Tarek Dakhran , Alberto Cabrera , Simon S. Lee , Paul Pak , Aditya Tadimeti , Tim Seyde , Maxime Labonne
show 2 more authors
Alexander Amini Mathias Lechner
This is my paper
classification cs.CL cs.AIcs.LG
keywords tokenizer expansioncontinued BPEvocabulary adaptationmultilingual LLMon-device inferenceembedding initializationcontinued pre-trainingdecode latency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that a pre-trained LLM's tokenizer can be upgraded in place — without retraining from scratch and without losing accumulated quality — by extending the existing BPE merges rather than replacing the tokenizer outright. The key move is to keep the old merge table fixed, append new merges from a multilingual corpus, and initialize new embedding rows as the mean of their source sub-token embeddings. A two-stage adaptation (embedding-only training, then full-model continued pre-training) recovers and even slightly exceeds the source checkpoint's quality on pre-training-style benchmarks. On an 8B-parameter on-device model, the expanded 128K tokenizer encodes Hindi, Vietnamese, Bengali, and Thai in 2.2–3.7× fewer tokens per character, which the paper synthesizes into a per-character decode speedup of the same magnitude after accounting for the larger vocabulary's per-token cost. A sympathetic reader would care because it suggests a practical recipe for extending language coverage of deployed models without discarding the compute already invested.

Core claim

The central claim is that tokenizer expansion — continuing the source tokenizer's BPE merges on a new corpus so that every new token decomposes exactly into source tokens, copying carried-over embedding rows and initializing new rows as the mean of their decomposition, then adapting with embedding-only training followed by full-model continued pre-training — allows an existing checkpoint to adopt a much larger vocabulary while matching or exceeding its original quality. The paper demonstrates this on an 8B Mixture-of-Experts model, LFM2-8B-A1B, upgrading its 65K tokenizer to 128K and reporting that the Stage 2 checkpoint beats the source on the aggregate eight-benchmark score (48.3 vs 44.7)

What carries the argument

The central object is the continued-BPE tokenizer construction: the new tokenizer is built by initializing the merge table from the source tokenizer's existing BPE merges and appending new merges learned from a multilingual corpus, so most source tokens map one-to-one into the expanded vocabulary and every new token has an exact decomposition into source tokens. This decomposition enables the embedding initialization rule — new rows are the mean of the corresponding source sub-token embeddings, copied rows for carried-over tokens — and ensures the adaptation has a clean starting point. The two-stage training recipe (Stage 1: embedding-only training on 600B tokens with new rows only; Stage 2:

Load-bearing premise

The headline decode-speedup figure rests on the assumption that the per-token decode cost of the 128K vocabulary, measured on a frozen model body with a resized embedding matrix, accurately reflects the per-token cost in the fully adapted LFM2.5-8B-A1B model across different output-token distributions and context lengths.

What would settle it

Decode the same Hindi, Thai, or Vietnamese prompt on an actual LFM2.5-8B-A1B model (with the 128K tokenizer) and an otherwise-comparable LFM2-8B-A1B model (with the 65K tokenizer) on the same device, measuring wall-clock time per character of generated output; if the ratio is below 2.2× for these languages, the synthesized speedup would be contradicted.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the recipe holds, model producers can extend an already-trained model's language coverage without retraining from scratch, making multilingual upgrades practical for on-device models.
  • The continued-BPE construction gives a deterministic source-token decomposition for every new token, which simplifies embedding initialization and makes the method reproducible across model families.
  • The measured per-token decode cost of a larger vocabulary (7–10% at 128K, up to 37% at 256K on the phone-class device) provides a concrete trade-off curve for choosing vocabulary size in on-device deployments.
  • The negative findings — that training all embedding rows in Stage 1 degrades generation, and that an English-heavy Stage 2 mixture causes a generative collapse not visible in multiple-choice benchmarks — are directly actionable for practitioners attempting similar recipes.
  • The released 128K tokenizer and model weights allow independent verification of the quality-preservation and decode-speedup claims.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The recipe's reliance on the model producer controlling the tokenizer design is a real boundary: it does not help users who must work with a fixed, third-party tokenizer, and the paper correctly points to zero-shot transfer methods for that case.
  • The measured per-token decode cost is drawn from a frozen-body benchmark with a synthetic matrix resize; the real adapted model's token distribution and MoE routing could shift the cost, so the 2.2–3.7× figure is an estimate rather than an end-to-end measurement, as the paper acknowledges.
  • The MCQ-versus-generation asymmetry documented in the ablations suggests a broader evaluation lesson: continued pre-training interventions can silently break generative fluency while leaving discriminative benchmarks intact, so practitioners should monitor repetition and free-form output, not just accuracy.
  • The compression ratios on under-tokenized languages are large enough that the 128K choice appears sensible; a natural extension would be to test whether a 192K or 256K vocabulary pays off on languages currently at parity, or to measure the decode cost on the actual LFM2.5 model after adaptation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes tokenizer expansion, an in-place recipe for upgrading a pre-trained LLM's tokenizer when the model producer controls the tokenizer. The method extends the existing BPE merge table on a multilingual corpus, so most source tokens carry over one-to-one and every new token has an exact decomposition into source tokens. Embedding rows for carried-over tokens are copied; new rows are initialized as the mean of their source sub-token embeddings. The model is then adapted in two stages: embedding-only training (600B tokens) and full-model continued pre-training (400B tokens), followed by the standard mid-training and post-training pipeline. The recipe is applied to an internal LFM2-8B-A1B checkpoint to produce LFM2.5-8B-A1B with a 128K tokenizer. The paper reports large token-count reductions for under-tokenized languages (up to 4.0x on Thai), quality preservation on an 8-benchmark aggregate and Global-MMLU, and synthesizes 2.2-3.7x per-character decode speedups on Thai, Bengali, Vietnamese, and Hindi by multiplying per-language compression ratios with a measured per-token decode-cost ratio.

Significance. If the results hold, this is a practically valuable contribution: it gives a production-scale recipe for upgrading a deployed model's tokenizer without retraining from scratch, with open weights and tokenizer released. The paper is transparent about its confounds and reports negative findings (Stage 1 full-embedding training, Stage 2 mixture imbalance) that are useful for practitioners. The factorization of decode speedup into compression and per-token cost is clear, and the per-token cost is measured with a controlled matrix-size experiment. The main risk is that the headline speedup numbers rely on web-text compression ratios applied to generated output, and on a single-run per-token cost measurement; these need validation or a sensitivity analysis before the quantitative claims can be accepted.

major comments (3)
  1. [§7.2, Table 5 (and §2, Fig. 2)] The synthesized per-character decode speedup is the product of a compression ratio and a per-token decode-cost ratio. The factorization is an identity, but the compression term is not a language constant: Figure 2 measures token-count ratios on FineWeb-2 web text plus English/code/JSON samples, while Table 5 is advertised for generated output. Tokenizer fertility is text-distribution dependent; assistant-style output is often more code-mixed, repetitive, and domain-specific. The paper never measures tokens per character on actual model-generated strings. If Hindi generation compresses at 1.6x instead of the web-text 2.38x, the Table 5 CPU speedup drops from about 2.18x to about 1.5x. This is a load-bearing empirical premise for the headline 2.2-3.7x. Please add a generation-based compression measurement (e.g., decode a fixed set of prompts with the final model and count source- vs expand
  2. [§6.1, Fig. 3, Table 3] The quality-recovery claim is confounded by the 400B tokens of Stage 2 full-model continued pre-training. The comparison is Stage 2 (35T source + 600B embedding + 400B CPT) against the 35T source checkpoint, so the +3.6 aggregate points could reflect additional training rather than successful preservation of the old checkpoint's capabilities under the new tokenizer. The paper acknowledges this for the surplus, but the abstract and §1 phrase the result as 'recovers source-checkpoint quality,' which is a causal claim. Without a control run that continued pre-trains the source model for 400B tokens with the original 65K tokenizer, the evidence establishes only that the full recipe produces a checkpoint that matches or exceeds source, not that tokenizer expansion alone avoids quality loss. Please either add such a control or explicitly reframe the claim as 'the full pipeline matches/exceeds
  3. [§7.1, Table 12] The per-token decode-cost ratio is measured on the source checkpoint with a resized embedding/LM-head matrix, not on the fully adapted LFM2.5 model, and is reported as single llama-bench runs without run-to-run variance. The memory-bandwidth argument makes the approximation plausible, since the ratio is dominated by the LM-head read, but the synthesis treats this ratio as a constant across languages, token distributions, and output lengths. Since Table 5 quotes two significant figures, please report at least the measurement spread. Ideally, validate on the final model for one language (e.g., Thai or Hindi) by timing actual generation with both tokenizers on held-out output text; this would also address the compression-distribution concern above.
minor comments (4)
  1. [§7, opening paragraph] The sentence 'the compression term is a property of the tokenizers alone' is misleading. Tokenizer compression is a property of the tokenizer and the text distribution; without specifying the distribution, the statement is false. Please rephrase to acknowledge the measured corpus.
  2. [§7.2, Table 5] The table reports synthesized speedups to two significant figures without any uncertainty or caveat in the table itself. Consider adding a footnote that these are estimates based on web-text compression and a single-run decode measurement.
  3. [§9.1, Limitations and Scope] The limitations list does not mention that the compression ratios are measured on held-out web text and may not transfer to generated assistant-style output. This is a central limitation of the speedup estimate and should be listed alongside the other scoping constraints.
  4. [Appendix B] The script-mediated explanation for the MGSM collapse pattern is appropriately hedged, but the appendix title 'Ablation Details' does not signal the speculative nature. A brief note that this is an observation rather than an established mechanism would help readers.

Circularity Check

0 steps flagged

No significant circularity: the speedup is an explicit product of two independently measured terms, and the self-citations are provenance references, not load-bearing derivation.

full rationale

The paper's central derivation is self-contained and transparent. The token-count compression ratios (Fig. 2) are measured on held-out text from FineWeb 2 plus English/code/JSON samples, independently of the model; the per-token decode-cost ratios (Table 12) are measured on a fixed LFM2-8B-A1B body with only the embedding/LM-head matrix resized, feeding identical token sequences. Section 7's synthesis is literally an identity: (chars/sec_new)/(chars/sec_old) = (chars/token_new)/(chars/token_old) × (tokens/sec_new)/(tokens/sec_old). Neither factor is defined in terms of the claimed output speedup, and neither parameter is fitted to the speedup target; the paper explicitly labels the result an estimate because the vocabulary-size cost is reused as a constant across languages and output lengths. That caveat, plus the skeptic's point that web-text compression may not transfer to generated assistant text, is an external-validity or robustness concern, not a definitional or fitted-input circularity. The quality-recovery evidence in Section 6 compares the expanded checkpoint to the source checkpoint on public benchmarks (MMLU-Pro, GSM8K, HumanEval+, Global-MMLU, etc.), so the preservation claim is externally checkable rather than derived from the recipe's assumptions. The citations to Liquid AI Team (2025, 2026) identify the source checkpoint and the surrounding release pipeline; they do not supply a uniqueness theorem, a fitted parameter, or an ansatz that does the argumentative work. The continued-BPE construction is credited to independent concurrent work (Purason et al., 2026), and the paper explicitly disclaims priority on that component. No circular step rises above routine self-citation for provenance, and none is load-bearing for the numerical claims.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

No physically new entities. The method introduces one engineered artifact (expanded 128K tokenizer) with exact mapping; its independence is testable via released weights and tokenizer. The main input assumptions are training budgets and bandwidth-bound decode.

free parameters (3)
  • Stage 1 token budget = 600B tokens
    Chosen by hand; not shown to be minimal or sufficient. The paper says they did not determine the minimum continued pre-training budget (§9.1). The quality-recovery claim is tied to this budget.
  • Stage 2 token budget = 400B tokens
    Chosen by hand; the added training confounds the quality-recovery result versus the source checkpoint, which the authors acknowledge in §6.1.
  • Stage 2 data mixture balance = unquantified 'balanced multilingual'
    Identified as critical by ablation (§8), but exact mixture weights are not specified, so a replication must re-derive them.
axioms (5)
  • domain assumption Continuing the source BPE merge table on a multilingual corpus, with existing merges held fixed and vocabulary capped at 128K, maps 63,151 source tokens to single tokens and decomposes every other new token exactly into source tokens.
    Section 2 / Table 1. This property is what makes row-copy and mean initialization possible; implementation caveats are deferred to Purason et al. (2026).
  • domain assumption Decode at batch size 1 is memory-bandwidth-bound, so per-token decode cost is governed by the bytes read per token, including the embedding/LM-head matrix size.
    Section 7, citing Xu et al. (2024). The entire latency synthesis depends on this assumption.
  • ad hoc to paper Mean-of-source-subtoken embedding initialization is a sufficient starting point for new rows.
    Section 3; justified empirically (Stage 1 recovers 4.8 of 5.8 aggregate points) but not derived, and no learned initializer is compared.
  • domain assumption Tied input/output embeddings in LFM2 make one matrix serve both roles; untied models initialize separately.
    Section 3; relevant to row copying and to the ablation where full-matrix training breaks generation.
  • domain assumption The internal 35T-token source checkpoint is the right control for 'source-checkpoint quality'.
    Sections 4 and 6.1; external readers cannot access this exact checkpoint, and the comparison includes extra Stage-2 tokens.

pith-pipeline@v1.3.0-alltime-deepseek · 20568 in / 13283 out tokens · 109755 ms · 2026-08-01T23:46:58.840338+00:00 · methodology

0 comments
read the original abstract

A tokenizer fixed at the start of pre-training allocates vocabulary in proportion to the pre-training corpus, reflecting the deployment priorities at that time. When those priorities shift, languages added later are split into many more tokens per word, which can raise latency, compute, and energy consumption for users of those languages. Cloud models can afford a broad vocabulary because the embedding and LM-head matrices are a small fraction of their parameters. On a compact model those matrices are a material share of per-token decode bandwidth, so on-device models ship small vocabularies and accept fragmentation outside a fixed language set. We present tokenizer expansion, an in-place recipe for upgrading a pre-trained model's tokenizer when the model producer controls its design. We continue the existing tokenizer's BPE merges on a multilingual corpus, so most source tokens carry over unchanged as single tokens and every new token has an exact decomposition into source tokens. We copy the carried-over embedding rows unchanged and initialize new rows as the mean of their source sub-token embeddings. A two-stage adaptation, embedding-only training then full-model continued pre-training, recovers source-checkpoint quality. We apply the recipe to a continued pre-trained checkpoint of LFM2-8B-A1B, an 8B-parameter Mixture-of-Experts model, to help produce LFM2.5-8B-A1B with a 128K tokenizer. The expanded tokenizer encodes Hindi and Vietnamese in roughly $2.4\times$ and $2.6\times$ fewer tokens than the source (up to $4.0\times$ on Thai). Combining these reductions with the measured per-token cost of the larger vocabulary, we estimate a $2.2$-$3.7\times$ per-character decode speedup for these languages across our reference devices. We release the model weights and the expanded tokenizer, and report the negative findings that shaped the recipe.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 28 linked inside Pith

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219,

    Marah Abdin et al. Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219,

    Pith/arXiv arXiv

  2. [4]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Pith/arXiv arXiv

  3. [6]

    Konstantin Dobler and Gerard de Melo

    arXiv:2402.01035. Konstantin Dobler and Gerard de Melo. FOCUS: Effective embedding initialization for monolingual spe- cialization of multilingual models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13440–13454,

    Pith/arXiv arXiv 2023

  4. [7]

    Token distillation: Attention-aware input em- beddings for new tokens.arXiv preprint arXiv:2505.20133,

    Konstantin Dobler, Desmond Elliott, and Gerard de Melo. Token distillation: Attention-aware input em- beddings for new tokens.arXiv preprint arXiv:2505.20133,

    arXiv

  5. [9]

    arXiv:2404.03608

    Association for Computational Linguistics. arXiv:2404.03608. Longxu Dou, Qian Liu, Fan Zhou, Changyu Chen, Zili Wang, Ziqi Jin, et al. Sailor2: Sailing in South-East Asia with inclusive multilingual LLMs.arXiv preprint arXiv:2502.12982,

    Pith/arXiv arXiv

  6. [10]

    Gemma Team

    arXiv:2404.17790. Gemma Team. Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

    Pith/arXiv arXiv

  7. [11]

    Charles Goddard and Fernando Fernandes Neto

    LLM inference in C/C++. Charles Goddard and Fernando Fernandes Neto. Training-free tokenizer transplantation via orthogonal matching pursuit.arXiv preprint arXiv:2506.06607,

    Pith/arXiv arXiv

  8. [12]

    The Llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Aaron Grattafiori et al. The Llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Pith/arXiv arXiv

  9. [13]

    ReTok: Replacing tokenizer to enhance representation efficiency in large language model.arXiv preprint arXiv:2410.04335,

    Shuhao Gu, Mengdi Zhao, Bowen Zhang, Liangdong Wang, Jijie Li, and Guang Liu. ReTok: Replacing tokenizer to enhance representation efficiency in large language model.arXiv preprint arXiv:2410.04335,

    Pith/arXiv arXiv

  10. [14]

    Multi-IF: Benchmarking LLMs on multi-turn and multilingual instruction following

    Yun He, Di Jin, Chaoqi Wang, Chloe Bi, Karishma Mandyam, Hejia Zhang, Chen Zhu, Ning Li, Tengyu Xu, Hongjiang Lv, et al. Multi-IF: Benchmarking LLMs on multi-turn and multilingual instruction following. arXiv preprint arXiv:2410.15553,

    Pith/arXiv arXiv

  11. [15]

    Measuring mathematical problem solving with the MATH dataset.arXiv preprint arXiv:2103.03874,

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset.arXiv preprint arXiv:2103.03874,

    Pith/arXiv arXiv

  12. [16]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica

    Accessed 2026-05. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974,

    Pith/arXiv arXiv 2026

  13. [17]

    Efficient and effective vocabulary expansion to- wards multilingual large language models.arXiv preprint arXiv:2402.14714,

    Seungduk Kim, Seungtaek Choi, and Myeongho Jeong. Efficient and effective vocabulary expansion to- wards multilingual large language models.arXiv preprint arXiv:2402.14714,

    Pith/arXiv arXiv

  14. [18]

    arXiv:2506.03523

    Association for Computational Linguistics. arXiv:2506.03523. Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. GSM-Plus: A comprehensive bench- mark for evaluating the robustness of LLMs as mathematical problem solvers. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages ...

    Pith/arXiv arXiv

  15. [19]

    doi: 10.18653/v1/2024.acl-long

    Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long

    doi:10.18653/v1/2024.acl-long 2024

  16. [20]

    Peiqin Lin, Shaoxiong Ji, J ¨org Tiedemann, Andr ´e F T Martins, and Hinrich Sch ¨utze

    arXiv:2305.20050. Peiqin Lin, Shaoxiong Ji, J ¨org Tiedemann, Andr ´e F T Martins, and Hinrich Sch ¨utze. MaLA-500: Massive language adaptation of large language models.arXiv preprint arXiv:2401.13303,

    Pith/arXiv arXiv

  17. [21]

    LFM2 technical report.arXiv preprint arXiv:2511.23404,

    Liquid AI Team. LFM2 technical report.arXiv preprint arXiv:2511.23404,

    arXiv

  18. [22]

    Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz

    arXiv:2305.01210. Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. WECHSEL: Effective initialization of sub- word embeddings for cross-lingual transfer of monolingual language models. InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language T echnologies, pages 3992–4006,

    Pith/arXiv arXiv 2022

  19. [23]

    Nandini Mundra, Aditya Nanda Kishore Khandavally, Raj Dabre, Ratish Puduppully, Anoop Kunchukut- tan, and Mitesh M

    arXiv:2405.07883. Nandini Mundra, Aditya Nanda Kishore Khandavally, Raj Dabre, Ratish Puduppully, Anoop Kunchukut- tan, and Mitesh M. Khapra. An empirical comparison of vocabulary expansion and initialization ap- proaches for language models. InProceedings of the 28th Conference on Computational Natural Language Learning (CoNLL), pages 84–104, Miami, FL, USA,

    arXiv

  20. [24]

    arXiv:2407.05841

    Association for Computational Linguistics. arXiv:2407.05841. Itay Nakash, Nitay Calderon, Eyal Ben David, Elad Hoffer, and Roi Reichart. AdaptiVocab: Enhancing LLM efficiency in focused domains through lightweight vocabulary adaptation. InConference on Language Modeling (COLM),

    Pith/arXiv arXiv

  21. [25]

    arXiv:2503.19693. OpenAI. MMMLU: Multilingual massive multitask language understanding.https://huggingface.co/ datasets/openai/MMMLU,

    Pith/arXiv arXiv

  22. [26]

    Malte Ostendorff and Georg Rehm

    Dataset accessed 2025-11-26. Malte Ostendorff and Georg Rehm. Efficient language model training through cross-lingual and progres- sive transfer learning.arXiv preprint arXiv:2301.09626,

    Pith/arXiv arXiv 2025

  23. [27]

    FineWeb2: One pipeline to scale them all – adapting pre-training data processing to every language.arXiv preprint arXiv:2506.20920,

    Guilherme Penedo, Hynek Kydl ´ıˇcek, Vinko Sabol ˇcec, Bettina Messmer, Negar Foroutan, Amir Hossein Kargaran, Colin Raffel, Martin Jaggi, Leandro Von Werra, and Thomas Wolf. FineWeb2: One pipeline to scale them all – adapting pre-training data processing to every language.arXiv preprint arXiv:2506.20920,

    Pith/arXiv arXiv

  24. [28]

    Yamshchikov, and Mark Fishel

    Taido Purason, Pavel Chizhov, Ivan P . Yamshchikov, and Mark Fishel. Teaching old tokenizers new words: Efficient tokenizer adaptation for pre-trained models. InFindings of the Association for Computational Lin- guistics: EACL 2026, pages 6492–6516, Rabat, Morocco,

    2026

  25. [30]

    Phillip Rust, Jonas Pfeiffer, Ivan Vuli´c, Sebastian Ruder, and Iryna Gurevych

    arXiv:2408.04303. Phillip Rust, Jonas Pfeiffer, Ivan Vuli´c, Sebastian Ruder, and Iryna Gurevych. How good is your tokenizer? On the monolingual performance of multilingual language models. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (V...

    Pith/arXiv arXiv

  26. [32]

    Shivalika Singh, Angelika Romanou, Cl ´ementine Fourrier, David I. Adelani, Jian Gang Ngui, Daniel Vila- Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Wei-Yin Ko, Sebastian Ruder, Madeline Smith, Antoine Bosselut, Alice Oh, Andre F. T. Mar- tins, Leshem Choshen, Daphne Ippolito, Enzo Ferrante, ...

    Pith/arXiv arXiv

  27. [33]

    Aya model: An instruction finetuned open- access multilingual language model.arXiv preprint arXiv:2402.07827,

    Ahmet ¨Ust ¨un, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et al. Aya model: An instruction finetuned open- access multilingual language model.arXiv preprint arXiv:2402.07827,

    Pith/arXiv arXiv

  28. [34]

    On-device language models: A comprehensive review.arXiv preprint arXiv:2409.00088,

    Jiajun Xu, Zhiyuan Li, Wei Chen, Qun Wang, Xin Gao, Qi Cai, and Ziyuan Ling. On-device language models: A comprehensive review.arXiv preprint arXiv:2409.00088,

    Pith/arXiv arXiv

  29. [35]

    An empirical study on cross-lingual vocab- ulary adaptation for efficient language model inference

    Atsuki Yamaguchi, Aline Villavicencio, and Nikolaos Aletras. An empirical study on cross-lingual vocab- ulary adaptation for efficient language model inference. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 6760–6785, Miami, Florida, USA,

    2024

  30. [36]

    arXiv:2402.10712

    Association for Computational Linguistics. arXiv:2402.10712. Atsuki Yamaguchi, Aline Villavicencio, and Nikolaos Aletras. How can we effectively expand the vocab- ulary of LLMs with 0.01GB of target language text?Computational Linguistics, 52(1):295–330,

    Pith/arXiv arXiv

  31. [37]

    arXiv:2406.11477

    doi: 10.1162/COLI.a.581. arXiv:2406.11477. An Yang et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    arXiv

  32. [38]

    Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911,

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911,

    Pith/arXiv arXiv

  33. [39]

    •Math (3 benchmarks).GSM8K (Cobbe et al., 2021), MATH500 (the 500-problem test subset of MATH (Hendrycks et al.,

    (we report the answer-ranking variant scored by log-probability over the option set, which is more stable across base and post-trained check- points than free-form answer extraction). •Math (3 benchmarks).GSM8K (Cobbe et al., 2021), MATH500 (the 500-problem test subset of MATH (Hendrycks et al.,

    2021

  34. [40]

    (2024)), and GSM-Plus (Li et al., 2024)

    selected by Lightman et al. (2024)), and GSM-Plus (Li et al., 2024). •Code (2 benchmarks).HumanEval+ (Liu et al.,

    2024

  35. [41]

    •Multilingual (2 benchmarks).MMMLU (OpenAI,

    and LiveCodeBench v5 (Jain et al., 2024). •Multilingual (2 benchmarks).MMMLU (OpenAI,

    2024

  36. [42]

    (multilingual GSM8K, generation-style). We exclude chat-template-dependent benchmarks (IFEval (Zhou et al., 2023), Multi-IF (He et al., 2024), and other tool-use and instruction-following benchmarks) from the Section 6.1 aggregate, for the reason given there: the source pre-training checkpoint has not been trained for chat-template adherence or tool- use ...

    2023

  37. [43]

    Zero-shot

    We read this as further evidence that the failure mode under the bad mixture is generation behavior rather than lost capability. The observed pattern is consistent with, though does not establish, a script-mediated 20 mechanism: when the model is steered toward English-style output by the predominantly English mix- ture, languages whose Latin script and s...

    2026

  38. [2016]

    Achieving tokenizer flexibility in language models through heuristic adaptation and supertoken learning.arXiv preprint arXiv:2505.09738,

    18 Shaurya Sharthak, Vinayak Pahalwan, Adithya Kamath, and Adarsh Shirawalmath. Achieving tokenizer flexibility in language models through heuristic adaptation and supertoken learning.arXiv preprint arXiv:2505.09738,

    Pith/arXiv arXiv

  39. [2021]

    Efficient and effective text encoding for Chinese LLaMA and Alpaca

    Yiming Cui, Ziqing Yang, and Xin Yao. Efficient and effective text encoding for Chinese LLaMA and Alpaca. arXiv preprint arXiv:2304.08177,

    Pith/arXiv arXiv

  40. [2023]

    doi: 10.18653/v1/2023.emnlp-main.614

    Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.614. Duarte M Alves, Jos´e Pombal, Nuno M Guerreiro, Pedro H Martins, Jo˜ao Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, et al. Tower: An open multilingual large language model for translation-related tasks.arXiv preprint arXiv:2402.17733,

    Pith/arXiv arXiv 2023

  41. [2024]

    Do all languages cost the same? Tokenization in the era of commercial language models

    Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. Do all languages cost the same? Tokenization in the era of commercial language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9904–9923, Singapore,

    2023

  42. [2025]

    Sailor: Open language models for South-East Asia

    Longxu Dou, Qian Liu, Guangtao Zeng, Jia Guo, Jiahui Zhou, Xin Mao, Ziqi Jin, Wei Lu, and Min Lin. Sailor: Open language models for South-East Asia. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 424–435, Miami, Florida, USA,

    2024

  43. [2026]

    doi: 10.18653/v1/2026.findings-eacl.341

    Association for Computational Linguistics. doi: 10.18653/v1/2026.findings-eacl.341. arXiv:2512.03989. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical report, OpenAI,

    arXiv 2026