BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization

Catherine Arnett; Sander Land

REVIEW 4 major objections 4 minor 19 references

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SCRIPT-BPE replaces UTF-8 bytes with script-based block and index token pairs, giving every character a uniform two-token cost, removing the byte premium and eliminating partial-character tokens.

desk verdict SCRIPT's two-token-per-character encoding plus constrained merges is a solid, useful tokenization result; just fix the overclaimed partial-UTF-8 elimination in §4.1 before it's citable. read the letter →

arxiv 2505.24689 v1 pith:RS4K2CIL submitted 2025-05-30 cs.CL

Sander Land , Catherine Arnett This is my paper

classification cs.CL

keywords BytePairEncodingtokenizationpretokenizationmultilingualUnicodescriptUTF-8compressionlanguageparity

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to fix two known weaknesses of Byte Pair Encoding for multilingual text: the byte premium that makes non-Latin scripts cost more tokens per character, and the fragile regex-based pretokenizers that split words at unnatural places. It proposes SCRIPT-BPE, which encodes each Unicode character as a pair of tokens derived from Unicode script and category properties, so all characters start at exactly two tokens. It also adds a constrained BPE merge rule that only merges complete characters or complete block-plus-index pairs, eliminating tokens that mix full and partial UTF-8 sequences. The paper reports that this scheme matches byte-level BPE compression on twelve languages and a large multilingual corpus while removing the encoding-based penalties and partial-character tokens. If correct, it offers a simpler and more robust pretokenization path for multilingual language models.

What carries the argument

The central object is the SCRIPT encoding: a bijective map from Unicode characters to ordered pairs of a block token and an index token, where the block is determined by Unicode Script plus one of five supercategories (Letters & Marks, Punctuation & Symbols, Numbers, Separators, Other) and large blocks are split into sub-blocks. This mapping gives every character a uniform two-token starting representation. The second piece is the constrained BPE merge: merges are allowed only between tokens that already represent one or more full characters, or between a block token and an index token that together form a complete character; the same rule applied to byte-level BPE only allows merges within a character's byte sequence or between complete characters. Together, the encoding and the merge constraint prevent the cascade of cross-character merges that creates partial UTF-8 tokens.

What would settle it

Train two otherwise identical multilingual language models, one with SCRIPT-BPE and one with byte-level BPE, and evaluate them on a battery of downstream tasks across scripts; if the SCRIPT model does not match or beat the byte-level model on at least some tasks despite equal or better compression, the central value claim fails, and a simpler check would be to count mixed full and partial UTF-8 tokens in the SCRIPT tokenizer's output, which the paper predicts to be zero.

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is that UTF-8 byte conversion is not a necessary substrate for BPE. By assigning each Unicode character a block token from 468 script-supercategory sub-block combinations and an index token from 1448 possible indices, SCRIPT-BPE makes the initial cost of every character exactly two tokens, independent of script. A rule-based pretokenizer that groups consecutive characters sharing script and supercategory then performs as well as regex-based pretokenizers, and a constrained merge rule that forbids cross-character merges removes all tokens containing partial character sequences. Across monolingual and multilingual tokenizers, constrained SCRIPT-BPE reaches tokens-per-character values competitive with byte-level BPE using the o200k regex, while improving compression for scripts like Thai, Hindi, and Punjabi that the cl100k regex penalized. The paper explicitly stops short of claiming downstream model gains; it claims the encoding and merge constraints remove a known source of bias and fragility without sacrificing compression.

Load-bearing premise

The paper's value claim rests on the assumption that compression and the absence of partial-character tokens are good proxies for tokenizer quality and downstream model fairness; no language model is trained, so better token counts are not shown to produce better or fairer models.

Editorial extensions

If this is right

Non-Latin scripts no longer carry an inherent token-count penalty: every character starts at two tokens, and final compression for Thai, Hindi, and Punjabi improves relative to the cl100k regex baseline.
Constrained merges eliminate tokens with mixed full and partial UTF-8 sequences, removing a class of semantically empty tokens that risk invalid encodings and under-training.
Rule-based SCRIPT pretokenization matches regex-based pretokenization on compression, so multilingual tokenizers do not have to depend on fragile regular expressions.
The merge constraint also helps byte-level BPE, meaning existing byte-based pipelines can adopt it without switching encodings.
Training time stays practical, about one hour with 16 CPUs for 256,000 merges, so the approach scales to large multilingual tokenizer training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper, uniform two-token character costs could change how multilingual models are priced and how context windows are consumed: if adopted, languages like Chinese and Thai would stop being charged a byte premium in token-based billing.
Beyond the paper, the constrained merge rule is directly testable on existing byte-level tokenizers: retraining merges with the boundary constraint on a fixed corpus should reproduce the near-complete elimination of partial-character tokens reported here.
Beyond the paper, because SCRIPT pretokenization is rule-based and script-aware, it may make tokenizer construction for low-resource languages easier and more predictable, though the paper does not evaluate low-resource downstream tasks.
Beyond the paper, the paper's acknowledged gap suggests a direct next experiment: train identical models with SCRIPT-BPE and byte-level BPE and compare downstream multilingual performance and fairness, since the compression results alone do not establish either.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague—quick read of Land & Arnett's BPE/SCRIPT paper. The new thing is clean: each Unicode character maps to two tokens (script/category block + index), which kills the byte premium by construction, and a merge constraint that only combines complete characters. That constraint is the more interesting part; it reduces partial-UTF-8 tokens to zero for SCRIPT and helps byte-level BPE too, with compression roughly on par with tiktoken's o200k for most languages and clear gains for Thai, Hindi, Punjabi. The rule-based pretokenizer without regexes is a genuinely simple engineering contribution.

Where it gets soft: Section 4.1 says constrained merges 'eliminate tokens with partial UTF-8 sequences,' but Table 2 shows 209 (cl100k) and 183 (o200k) mixed full/partial tokens for constrained byte-based BPE. The §3.2 constraint should never create those, so either the implementation doesn't fully enforce the constraint, the tiktoken vocabulary initializes with pre-existing mixed tokens that aren't pruned, or the metric counts something else. As written, the text and table are inconsistent, and that overstatement is load-bearing for the 'universal' character-integrity claim. It doesn't undercut the compression results, which are the empirical core, but it does mean the abstract's 'eliminating' language needs a fix, and the authors should clarify what exactly the constraint guarantees.

Also minor: no seeds or error bars on Tables 2/4, no downstream LM training (they acknowledge this in Section 5), and the byte-premium elimination is by construction, so it's not a separately tested prediction. The compression comparisons are benchmarked against tiktoken with no target re-tuning, which is fair.

Verdict: this deserves a serious referee. The core idea is simple, well-motivated, and mostly supported; the partial-token inconsistency is addressable. A careful reviewer should ask for the Table 2 reconciliation, error bars, and ideally a small-scale LM or downstream probe. I'd send it to a tokenization or multilingual modeling venue, maybe with the byte-level claim pulled back to 'substantially reduces.' For readers: tokenization engineers and anyone working on multilingual LLM efficiency would get value. I'd probably cite the constrained-merge result if I did tokenization work.

Referee Report

4 major / 4 minor

Summary. The paper proposes SCRIPT (Script Category Representation in PreTokenization), a Unicode-script-and-category-based encoding that maps every character to exactly two initial tokens (a block token and an index token). It combines this encoding with a rule-based pretokenizer that groups characters by script and supercategory, and with a constrained BPE merge strategy that only merges complete characters (or, for byte-level BPE, completes a character's byte sequence before merging it with other complete characters). The authors evaluate compression as tokens per Unicode character and count tokens containing partial UTF-8 sequences, comparing SCRIPT-BPE against byte-level BPE with tiktoken's cl100k and o200k regex pretokenizers, in monolingual and multilingual settings. They report that SCRIPT-BPE achieves compression competitive with byte-level BPE while eliminating tokens with partial UTF-8 sequences and avoiding the byte-premium penalty.

Significance. If the results hold, the paper offers a clean, regex-free pretokenization framework that addresses a real problem: byte-level BPE's uneven initial token costs across scripts and its tendency to create tokens that mix full and partial UTF-8 sequences. The manuscript's strengths are its simple and reproducible design, the public implementation, and the benchmarking against widely used tiktoken-based baselines. The compression gains for Thai, Hindi, and Punjabi are substantial and practically relevant. However, the paper evaluates only tokenizer-level metrics; Section 5 explicitly concedes that compression does not guarantee better downstream model performance, and no language model is trained. Consequently, the contribution is a tokenizer-design demonstration rather than an end-to-end validation of improved multilingual modeling. The uniform two-token initial cost is a property of the encoding by construction, so the central empirical content is that final compression remains competitive after BPE merging and that partial-character tokens can be avoided; that content is plausible but needs the clarifications below.

major comments (4)

[Section 4.1, Table 2] The text states that constraining BPE merges "eliminate[s] tokens with partial UTF-8 sequences" and the Discussion repeats that constrained merging "universally eliminated tokens representing a mix of full and partial characters." Table 2, however, reports 209 #Partial UTF-8 tokens for Bytes + cl100k regex with constrained merges and 183 for Bytes + o200k regex with constrained merges. If the #Partial UTF-8 column counts mixed full/partial-character tokens, these numbers directly contradict the claimed elimination; if it counts something else (e.g., tokens containing only partial byte sequences), the table header and text are misleading. Since the constrained-merge mechanism is a headline contribution, the authors should correct the claim, the metric definition, or the implementation so that the text and table are consistent.
[Section 4.3, Table 4] All numerical comparisons are reported for a single training run without seeds, standard deviations, or significance tests. Many differences are minute: Japanese final tokens/character is 0.5249 (Bytes cl100k) versus 0.5267 (SCRIPT), German is 0.2128 versus 0.2138, and English is 0.2152 versus 0.2150. Given the manual design choices in the method (the block-split threshold, the supercategory reassignments in Appendix A, and the whitespace-separating script list in Appendix B), the reader cannot tell whether these differences are real or noise. Please report results over several seeds or otherwise provide variance estimates, and ideally a sensitivity analysis for the main design choices.
[Section 4.1] The statement that constraining merges "is beneficial for all encodings and datasets" is not supported by the evidence shown. Table 2 reports only aggregated means over the monolingual corpora (mean tokens/char and mean #partial counts), not per-language or per-dataset values, and the compression improvements are described as "almost universal" without a per-language constrained-versus-unconstrained comparison. Please provide the per-language breakdown or soften the claim to match the presented evidence.
[Abstract, Section 4.3, Table 4] The abstract's claim that SCRIPT-BPE "eliminates encoding-based penalties for non-Latin-script languages" conflates a design property with a measured outcome. The uniform initial cost of two tokens per character is true by construction of the block+index encoding, not an empirical finding. The final compression results are more mixed: SCRIPT rule-based is worse than the best byte-based baseline for Chinese (0.6537 vs. 0.6259) and Japanese (0.5393 vs. 0.5267), and roughly tied elsewhere. The paper should distinguish clearly between the definitional two-token initial cost and the empirical final-compression comparison, and the abstract should be toned down accordingly.

minor comments (4)

[Section 5] The sentence "combining refining the handling of digits and leading spaces" is ungrammatical and should be revised.
[Table 4] The grouped header is difficult to parse: the two "Init. Tokens/Char" columns (Bytes, SCRIPT) and the four "Final Tokens/Character" columns should be explicitly labeled so that the reader can tell which baseline each o200k column refers to.
[Section 4] The text refers to "cl100k (GPT-4) and o200k (GPT-4o) variants" as if these were vocabularies; the paper elsewhere clarifies that only tiktoken's regex pretokenizers are used, but the shorthand is misleading and should be made explicit.
[Appendix B] The space-merging rule relies on a manually specified list of whitespace-separating scripts; please state whether this list is intended to be exhaustive and how it was selected, since it is a free design choice that could affect the reported compression.

Circularity Check

1 steps flagged · score 2.0 of 10

One definitional property, the fixed two-initial-token cost of SCRIPT, is presented as an experimentally eliminated byte premium; the central compression comparisons remain empirical and non-circular.

self definitional [Abstract and Section 1 (encoding overview); Section 3 (SCRIPT Encoding definition)]
"Our encoding uses two initial tokens to represent each character, eliminating crosslinguistic bias due to byte premium effects."

The elimination of the byte premium is not a derived prediction: SCRIPT is defined in Section 3 as mapping each Unicode character to exactly one block token and one index token, so every character costs exactly two initial tokens by construction. Table 4's Initial Tokens/Character column shows 2.00 for every SCRIPT row because the encoding is defined that way, not because an experiment established it. The abstract's stronger claim that experiments 'demonstrate ... eliminating encoding-based penalties for non-Latin-script languages' therefore reduces, for the initial encoding cost, to the encoding's own definition. The separate post-BPE compression results are genuinely empirical and do not reduce to this definition.

full rationale

The paper's central empirical content is the compression comparison in Tables 2 through 5, where SCRIPT-BPE and constrained byte-based BPE are measured against tiktoken-based baselines on monolingual and multilingual corpora. Those numbers are not fitted to any target result and do not reduce to the method's definitions, so the main compression claim is self-contained. The only step that reduces to its own input is the byte-premium elimination claim: because SCRIPT is defined as exactly two initial tokens per character, the absence of a variable per-character byte cost is a design guarantee rather than an experimentally discovered effect. The constrained-merge claim that partial UTF-8 tokens are eliminated is also a definitional consequence for SCRIPT, and it is partially contradicted for byte-based BPE by Table 2's residual counts of 209 and 183 mixed tokens; however, that is an internal consistency or correctness issue, not a circularity. The self-citations (Land & Bartolo 2024; Arnett et al. 2024; Land 2024) motivate the problem of partial tokens and byte premiums but do not carry the derivation of the compression results. Overall, the central contribution has independent empirical content, so the circularity score is low.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

SCRIPT introduces block and index token categories as encoding-level bookkeeping, but no new physical or mathematical entities. The free parameters are hand-chosen design choices about block splitting, supercategory grouping, and pretokenization rules, none of which are fitted to the target compression claims.

free parameters (3)

Block split threshold = 1,448 (size of Latin LM block)
Section 3 sets the threshold for splitting large script-supercategory blocks to the size of the Latin LM block; this is a hand-chosen design value, not derived from an independent criterion.
Supercategory grouping = five supercategories: LM, PS, N, Z, C
Section 3 and Appendix A define the grouping of Unicode general categories, including manual reassignments of newline, tab, U+30FC, U+FF70, and U+0640; these choices determine the block vocabulary and pretokenization behavior.
Whitespace-separating script list = 19 scripts listed in Appendix B
Appendix B hand-curates a list of scripts whose single-space groups may merge with following LM groups; this affects pretokenization for those scripts and is not derived from data.

assumptions (4)

domain assumption Unicode Script and Supercategory properties are meaningful units for pretokenization.
Section 3 and Appendix B assume that grouping by these properties yields useful pretokens and fair encoding across scripts.
domain assumption Compression in tokens per character is the relevant evaluation metric for tokenizer quality.
Section 4.3 measures compression, and Section 5 concedes that compression does not guarantee downstream performance, so the evidence rests on this proxy.
domain assumption Excluding Unassigned, Private Use, and Surrogate code points is fair across baselines.
Appendix A filters these code points and applies the same filter to all baselines, assuming this does not bias the comparison.
domain assumption A complete-character merge constraint does not prevent BPE from finding competitive vocabularies.
Section 3.2 imposes this constraint; the compression results show only small differences, but no downstream evaluation verifies the assumption.

how reviews work

0 comments

Cite this review

Pith. "Pith review of BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization." pith.science (2026). https://pith.science/paper/RS4K2CIL

@misc{pith2026250524689,
  author       = {Pith},
  title        = {Pith review of: BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RS4K2CIL}},
  note         = {Machine review of arXiv:2505.24689}
}

read the original abstract

Byte Pair Encoding (BPE) tokenizers, widely used in Large Language Models, face challenges in multilingual settings, including penalization of non-Western scripts and the creation of tokens with partial UTF-8 sequences. Pretokenization, often reliant on complex regular expressions, can also introduce fragility and unexpected edge cases. We propose SCRIPT (Script Category Representation in PreTokenization), a novel encoding scheme that bypasses UTF-8 byte conversion by using initial tokens based on Unicode script and category properties. This approach enables a simple, rule-based pretokenization strategy that respects script boundaries, offering a robust alternative to pretokenization strategies based on regular expressions. We also introduce and validate a constrained BPE merging strategy that enforces character integrity, applicable to both SCRIPT-BPE and byte-based BPE. Our experiments demonstrate that SCRIPT-BPE achieves competitive compression while eliminating encoding-based penalties for non-Latin-script languages.

Figures

Figures reproduced from arXiv: 2505.24689 by the authors.

**Figure 1.** A: Illustration of variable encoding length in UTF-8 encoding compared to consistent encoding length for SCRIPT encoding. B: In UTF-8, the same initial byte sequences may be used for characters in different scripts, however SCRIPT encoding uses consistent block tokens to represent characters from the same script block. C: Byte-based BPE can allow merges that create partial UTF-8 sequences, which then cascade through… view at source ↗

**Figure 2.** Distribution of block sizes in SCRIPT encoding, before splitting large blocks into sub-blocks. C. Monolingual tokenizer performance Compression performance for monolingual tokenizers on their training data ( [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 8 canonical work pages

[1]

wHy DoNt YoU jUsT uSe ThE lLaMa ToKeNiZeR?? Blog Post, 2024

Arnett, C. wHy DoNt YoU jUsT uSe ThE lLaMa ToKeNiZeR?? Blog Post, 2024. URL https://huggingface.co/blog/catherinearnett/dangers-of-tokenizer-recycling

work page 2024
[2]

A., and Bergen, B

Arnett, C., Chang, T. A., and Bergen, B. A Bit of a Problem: Measurement Disparities in Dataset Sizes across Languages . In Melero, M., Sakti, S., and Soria, C. (eds.), Proceedings of the 3rd Annual Meeting of the Special Interest Group on Under-resourced Languages @ LREC-COLING 2024, pp.\ 1--9, Torino, Italia, May 2024. ELRA and ICCL. URL https://aclanth...

work page 2024
[3]

A., Arnett, C., Tu, Z., and Bergen, B

Chang, T. A., Arnett, C., Tu, Z., and Bergen, B. K. Goldfish: Monolingual language models for 350 languages. arXiv preprint arXiv:2408.10441, 2024. URL https://arxiv.org/abs/2408.10441

arXiv 2024
[4]

Getting the most out of your tokenizer for pre-training and domain adaptation

Dagan, G., Synnaeve, G., and Roziere, B. Getting the most out of your tokenizer for pre-training and domain adaptation. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=ZFYBnLljtT

work page 2024
[5]

and Richardson, J

Kudo, T. and Richardson, J. SentencePiece : A simple and language independent subword tokenizer and detokenizer for neural text processing. In Blanco, E. and Lu, W. (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp.\ 66--71, Brussels, Belgium, November 2018. Association for Computatio...

doi:10.18653/v1/d18-2012 2018
[6]

Pre-tokenization on punctuation in GPT-4

Land, S. Pre-tokenization on punctuation in GPT-4 . Blog Post, 2024. URL https://tokencontributions.substack.com/p/pre-tokenization-on-punctuation-in

work page 2024
[7]

and Bartolo, M

Land, S. and Bartolo, M. Fishing for M agikarp: Automatically detecting under-trained tokens in large language models. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.\ 11631--11646, Miami, Florida, USA, November 2024. Association for Computational Linguistic...

doi:10.18653/v1/2024.emnlp-main.649 2024
[8]

Jamo-level subword tokenization in low-resource K orean machine translation

Lee, J., Cognetta, M., Moon, S., and Okazaki, N. Jamo-level subword tokenization in low-resource K orean machine translation. In Ojha, A. K., Liu, C.-h., Vylomova, E., Pirinen, F., Washington, J., Oco, N., and Zhao, X. (eds.), Proceedings of the Eighth Workshop on Technologies for Machine Translation of Low-Resource Languages (LoResMT 2025), pp.\ 66--80, ...

work page 2025

Show all 19 references

[9]

MYTE : Morphology-driven byte encoding for better and fairer multilingual language modeling

Limisiewicz, T., Blevins, T., Gonen, H., Ahia, O., and Zettlemoyer, L. MYTE : Morphology-driven byte encoding for better and fairer multilingual language modeling. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for...

2024 doi
[10]

A., and Choi, Y

Liu, A., Hayase, J., Hofmann, V., Oh, S., Smith, N. A., and Choi, Y. SuperBPE : Space travel for language models. arXiv preprint arXiv:2503.13423, 2025. URL https://arxiv.org/abs/2503.13423

2025 arXiv
[11]

and Low, B

Liu, T. and Low, B. K. H. Goat: Fine-tuned LLaMA outperforms GPT-4 on arithmetic tasks. arXiv preprint arXiv:2305.14201, 2023. URL https://arxiv.org/abs/2305.14201

2023 arXiv
[12]

V., Lai, V

Nguyen, T., Nguyen, C. V., Lai, V. D., Man, H., Ngo, N. T., Dernoncourt, F., Rossi, R. A., and Nguyen, T. H. CulturaX : A cleaned, enormous, and multilingual dataset for large language models in 167 languages, 2023. URL https://arxiv.org/abs/2309.09400

2023 arXiv
[13]

tiktoken: a fast BPE tokeniser for use with OpenAI 's models., 2024

OpenAI. tiktoken: a fast BPE tokeniser for use with OpenAI 's models., 2024. URL https://github.com/openai/tiktoken

2024
[14]

W., Reddy, V., Zhang, H., Alameddine, A., Uzan, O., Pinter, Y., and Tanner, C

Schmidt, C. W., Reddy, V., Zhang, H., Alameddine, A., Uzan, O., Pinter, Y., and Tanner, C. Tokenization is more than compression. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.\...

2024 doi
[15]

W., Reddy, V., Tanner, C., and Pinter, Y

Schmidt, C. W., Reddy, V., Tanner, C., and Pinter, Y. Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier . arXiv preprint arXiv:2504.00178, 2025. URL https://arxiv.org/abs/2504.00178

2025
[16]

Neural Machine Translation of Rare Words with Subword Units

Sennrich, R., Haddow, B., and Birch, A. Neural Machine Translation of Rare Words with Subword Units . In Erk, K. and Smith, N. A. (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 1715--1725, Berlin, G...

2016 doi
[17]

Singh, A. K. and Strouse, D. Tokenization counts: the impact of tokenization on arithmetic in frontier llms. arXiv preprint arXiv:2402.14903, 2024. URL https://arxiv.org/abs/2402.14903

2024 arXiv
[18]

and Sarveswaran, K

Velayuthan, M. and Sarveswaran, K. Egalitarian Language Representation in Language Models: It All Begins with Tokenizers . In Rambow, O., Wanner, L., Apidianaki, M., Al-Khalifa, H., Eugenio, B. D., and Schockaert, S. (eds.), Proceedings of the 31st International Conference on ...

2025
[19]

Tokenization is Sensitive to Language Variation

Wegmann, A., Nguyen, D., and Jurgens, D. Tokenization is Sensitive to Language Variation . arXiv preprint arXiv:2502.15343, 2025. URL https://arxiv.org/abs/2502.15343

2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

wHy DoNt YoU jUsT uSe ThE lLaMa ToKeNiZeR?? Blog Post, 2024

Arnett, C. wHy DoNt YoU jUsT uSe ThE lLaMa ToKeNiZeR?? Blog Post, 2024. URL https://huggingface.co/blog/catherinearnett/dangers-of-tokenizer-recycling

work page 2024

[2] [2]

A., and Bergen, B

Arnett, C., Chang, T. A., and Bergen, B. A Bit of a Problem: Measurement Disparities in Dataset Sizes across Languages . In Melero, M., Sakti, S., and Soria, C. (eds.), Proceedings of the 3rd Annual Meeting of the Special Interest Group on Under-resourced Languages @ LREC-COLING 2024, pp.\ 1--9, Torino, Italia, May 2024. ELRA and ICCL. URL https://aclanth...

work page 2024

[3] [3]

A., Arnett, C., Tu, Z., and Bergen, B

Chang, T. A., Arnett, C., Tu, Z., and Bergen, B. K. Goldfish: Monolingual language models for 350 languages. arXiv preprint arXiv:2408.10441, 2024. URL https://arxiv.org/abs/2408.10441

arXiv 2024

[4] [4]

Getting the most out of your tokenizer for pre-training and domain adaptation

Dagan, G., Synnaeve, G., and Roziere, B. Getting the most out of your tokenizer for pre-training and domain adaptation. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=ZFYBnLljtT

work page 2024

[5] [5]

and Richardson, J

Kudo, T. and Richardson, J. SentencePiece : A simple and language independent subword tokenizer and detokenizer for neural text processing. In Blanco, E. and Lu, W. (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp.\ 66--71, Brussels, Belgium, November 2018. Association for Computatio...

doi:10.18653/v1/d18-2012 2018

[6] [6]

Pre-tokenization on punctuation in GPT-4

Land, S. Pre-tokenization on punctuation in GPT-4 . Blog Post, 2024. URL https://tokencontributions.substack.com/p/pre-tokenization-on-punctuation-in

work page 2024

[7] [7]

and Bartolo, M

Land, S. and Bartolo, M. Fishing for M agikarp: Automatically detecting under-trained tokens in large language models. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.\ 11631--11646, Miami, Florida, USA, November 2024. Association for Computational Linguistic...

doi:10.18653/v1/2024.emnlp-main.649 2024

[8] [8]

Jamo-level subword tokenization in low-resource K orean machine translation

Lee, J., Cognetta, M., Moon, S., and Okazaki, N. Jamo-level subword tokenization in low-resource K orean machine translation. In Ojha, A. K., Liu, C.-h., Vylomova, E., Pirinen, F., Washington, J., Oco, N., and Zhao, X. (eds.), Proceedings of the Eighth Workshop on Technologies for Machine Translation of Low-Resource Languages (LoResMT 2025), pp.\ 66--80, ...

work page 2025