Tokenization Standards for Linguistic Integrity: Turkish as a Benchmark

Ahmet Semih G\"um\"u\c{s}; Ali Arda Fincan; Banu Diri; M. Ali Bayram; Sava\c{s} Y{\i}ld{\i}r{\i}m; Sercan Karaka\c{s}

REVIEW 4 major objections 5 minor 23 references

Tokenization Standards for Linguistic Integrity: Turkish as a Benchmark

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Valid-word tokens are the best predictor of Turkish LLM accuracy

desk verdict Useful new tokenizer diagnostics for Turkish, but the headline correlation is confounded by model identity and a sample of four; worth a careful revision, not a desk reject. read the letter →

arxiv 2502.07057 v2 pith:VVV4XNY6 submitted 2025-02-10 cs.CL

M. Ali Bayram , Ali Arda Fincan , Ahmet Semih Gümüş , Sercan Karakaş , Banu Diri , Savaş Y{i}ld{i}r{i}m This is my paper · ORCID

classification cs.CL

keywords Turkishtokenizationmorphologicallyrichlanguagestokenpuritylanguage-specificpercentagesTR-MMLUbenchmarksubwordagglutinativelow-resourceNLP

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a measurement framework for judging tokenizers by how well they preserve Turkish as a real language, rather than by compression or speed alone. On a Turkish MMLU benchmark of 6,200 multiple-choice questions, it computes five metrics for four widely used tokenizers and compares each metric with the models' accuracy on the same questions. The headline result is that the share of unique tokens that are valid Turkish words ($\%\mathrm{TR}$) tracks accuracy more closely ($r=0.90$) than the share of tokens that are morphologically pure roots or morphemes ($\%\mathrm{Pure}$, $r=0.68$). If the result holds, tokenizer quality for morphologically rich languages should be assessed primarily by language-specific valid-token coverage, and adding parameters alone will not compensate for poor linguistic alignment.

What carries the argument

The load-bearing machinery is the pair of new metrics defined on unique tokens: $\%\mathrm{TR}$, the percentage of unique tokens that are valid Turkish surface words, and $\%\mathrm{Pure}$, the percentage that cannot be decomposed into smaller meaningful morphemes. The metrics are computed with morphological analysis tools, then each tokenizer's values are correlated with its score on the TR-MMLU benchmark. The analysis that carries the argument is the correlation matrix among these metrics, model parameter count, vocabulary size, token count, processing time, and MMLU score, because the paper's conclusion follows directly from the relative sizes of those correlations.

What would settle it

Run the same TR-MMLU evaluation on a larger set of Turkish tokenizers, say fifteen or more, and compute rank correlations of $\%\mathrm{TR}$ and $\%\mathrm{Pure}$ with accuracy separately; if the $\%\mathrm{TR}$ correlation is not clearly above $\%\mathrm{Pure}$, or if a high-accuracy tokenizer with low $\%\mathrm{TR}$ appears, the paper's central ranking fails.

Watch

Extended reading notes

Core claim

The paper's central discovery is a ranking of tokenizer-side metrics by their correlation with downstream Turkish accuracy, with $\%\mathrm{TR}$ on top. It reports that Turkish token percentage correlates with MMLU at $r=0.90$, token purity at $r=0.68$, vocabulary size positively with both linguistic metrics ($r=0.77$ with $\%\mathrm{TR}$, $r=0.82$ with $\%\mathrm{Pure}$), and token count and processing time negatively with linguistic fidelity ($r=-0.93$ and $r=-0.60$, respectively). As a consequence, larger models are not automatically better: one 27.2-billion-parameter model outperformed a 70.6-billion-parameter model on Turkish MMLU because its tokenizer produced more valid Turkish words and purer tokens. The paper interprets this as evidence that language-specific tokenizer design, rather than raw scale, carries much of the burden for morphologically rich languages.

Load-bearing premise

The claim that $\%\mathrm{TR}$ predicts accuracy better than purity rests on correlations computed from just four tokenizer choices, and the paper treats those correlations as stable even though a four-point sample leaves the true ordering between 0.90 and 0.68 uncertain.

Editorial extensions

If this is right

Tokenizer evaluation for morphologically rich languages should add a valid-word coverage metric like $\%\mathrm{TR}$ alongside purity, since it was the metric most associated with accuracy.
Increasing model parameters alone is unlikely to fix poor tokenization: in the tested set, the largest model did not lead on Turkish accuracy, and a smaller model did.
Vocabulary size appears to support linguistic fidelity, while inflated token counts and long processing times are associated with worse linguistic metrics.
The same framework can be applied to other agglutinative languages such as Finnish, Hungarian, and Uyghur whenever equivalent linguistic validation tools exist.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the correlations come from just four tokenizer choices, the $r=0.90$ versus $r=0.68$ gap is best read as suggestive rather than settled; a larger sample could easily move the two metrics closer together or reverse them.
A tokenizer could raise $\%\mathrm{TR}$ by keeping long inflected surface words whole, which would also raise token count and lower morphological granularity, so $\%\mathrm{TR}$ should be paired with a fertility or length check before being used as a standalone optimization target.
The framework implies a testable design rule for multilingual models: reserving vocabulary capacity for language-specific surface forms of morphologically rich languages may buy more downstream accuracy than spending the same capacity on generic subword units, and this can be checked by ablating vocabulary allocation on TR-MMLU.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a decent empirical note with two new, simple metrics and a plausible but under-identified central finding. %TR (share of unique tokens that are valid Turkish words) is a sensible diagnostic, and the Table 1 measurements are the paper's real contribution. The claim that %TR correlates more strongly than %Pure with TR-MMLU accuracy (r=0.90 vs 0.68) is new and interesting, but the evidence doesn't support it as stated.

What it does well: the metric definitions are clean, the benchmark is a real native Turkish MMLU, and the correlations are easy to reproduce from Table 1. The paper acknowledges the role of morphological tools, even if it doesn't report their error rates. It also cites relevant prior work on Turkish tokenization and Arabic/Scandinavian leaderboards.

Soft spots, in rough order of severity. First, the confound the stress-test identified is the real issue: each row in Table 1 is a full model—Gemma-2, Llama-3.1, Qwen2.5, Aya-Expanse—with its own pretraining, architecture, and data. %TR may simply track how much Turkish text the model saw, which would raise MMLU directly. The r=0.90 for %TR could be driven by model-family effects, not tokenization. Section 5's point about parameter size not predicting performance doesn't fix this; you'd need tokenizer swaps under fixed model and data, or at least a larger sample with family controls. Second, n=4 with no confidence intervals means the %TR vs %Pure ranking is fragile; the reader's caution here is well placed. Third, the metrics depend on ITU NLP Web Service and Kalbur, and we're not told how errors in these tools propagate. Fourth, the repo is promised without a commit hash or verification instructions. Fifth, minor: several references look sloppy, and Section 5 claims 'remarkable initial results' for two unreleased tokenizers with no table or data.

Bottom line: the framework is worth refining, and the paper deserves peer review as a workshop-caliber contribution. Require the authors to address identification—before publishing the claim that %TR is the stronger driver—and to report uncertainty and tool accuracy. I'd accept it for review but not near acceptance as is.

Referee Report

4 major / 5 minor

Summary. The paper proposes a tokenizer evaluation framework for Turkish, centered on five metrics: vocabulary size, token count, processing time, language-specific token percentage (%TR), and pure token percentage (%Pure). Using the TR-MMLU dataset of 6,200 multiple-choice questions, the authors measure these metrics for the tokenizers of four models (gemma-2, llama-3.1, Qwen2.5, aya-expanse), compute correlations with MMLU scores, and report that %TR has the strongest positive correlation (r = 0.90), followed by %Pure (r = 0.68). The paper concludes that linguistic alignment, reflected by %TR, is more important than morphological purity for downstream performance in Turkish, and that parameter size is not a definitive predictor. The broader claim is that this framework sets a new standard for evaluating tokenization in morphologically rich and low-resource languages.

Significance. If the central claim were established, the paper would provide a practical, simple tokenizer-side diagnostic for morphologically rich languages: measuring the share of unique tokens that are valid Turkish words could guide tokenizer selection or design. The choice of a native Turkish benchmark (TR-MMLU) rather than translated data is a clear strength, as is the stated intention to release code and data. However, the headline quantitative result rests on correlations computed over only four complete model/tokenizer systems, with no uncertainty quantification and no separation of tokenizer effects from model-family effects. As presented, the finding is best viewed as a descriptive observation about four specific systems, not as evidence for a generalizable mechanism. The paper's contribution is therefore currently more modest than its conclusions claim, though the framework could become valuable with additional evidence.

major comments (4)

[Section 4, Table 1] The central claim that %TR is the tokenizer-side variable most associated with MMLU performance is derived from exactly four data points, each of which is an entire model system rather than a tokenizer in isolation. Gemma-2, Llama-3.1, Qwen2.5, and Aya-Expanse differ in architecture, pretraining corpus, training procedure, and parameter count, so the observed r = 0.90 between %TR and MMLU may be confounded by model-level properties such as the amount of Turkish pretraining text. A high %TR might simply indicate that the model's tokenizer sees a lot of Turkish as whole words, which in turn may reflect Turkish-heavy pretraining that boosts MMLU independently of tokenization mechanics. The same confound applies to the comparison of %TR with %Pure. Without an experiment that varies the tokenizer while holding the model fixed, or at least a substantially larger sample with family-level controls, the assertion that linguistic alignment 'plays a role' in downstream performance is not supported by the reported correlational evidence. Please either add such controls or explicitly reframe the conclusion as a descriptive observation about these four systems.
[Section 4, correlation matrix] With n = 4, a Pearson correlation of 0.90 has an extremely wide confidence interval, and the difference between r = 0.90 and r = 0.68 is not statistically meaningful. The paper reports no p-values, confidence intervals, bootstrap estimates, or sensitivity analyses, so the ranking of %TR over %Pure as 'the strongest' predictor is fragile. Additionally, the low outlier Qwen2.5, which has both the lowest MMLU score and the lowest %TR, may be driving the correlation. Please report uncertainty estimates and show whether the %TR–MMLU correlation survives removal of individual models; if it does not, the ranking claim should be withdrawn.
[Section 3, Equations (1) and (2)] The metrics %TR and %Pure depend on the ITU Turkish NLP Web Service and the Kalbur library for morphological analysis and token validation, but the manuscript does not report the accuracy or error rates of these tools, nor how ambiguous cases are resolved. If the morphological analyzer misclassifies valid words or morphemes, the values in Table 1 and all correlations derived from them change. Please provide a validation of the morphological tools on a gold-standard Turkish sample, or at least a sensitivity analysis showing that plausible error rates do not reverse the reported %TR/%Pure ordering.
[Section 5, Conclusion] The conclusion that 'tailored tokenization strategies can enable even smaller or less-optimized models to excel in morphologically complex settings' goes beyond the evidence, because no tokenizer was manipulated in the study. The data are observational correlations from four pre-existing models, so the conclusion should be softened to a hypothesis or design recommendation, not presented as a demonstrated effect.

minor comments (5)

[Section 3, Equations (1) and (2)] Please clarify whether %TR and %Pure are computed over all tokens or only unique tokens. The examples use unique tokens, and Table 1 lists 'Unique Token Count' as a separate row, but the metric definitions should state the denominator explicitly to avoid ambiguity in replication.
[Section 3, motivating example] The example states that 'de' is not a standalone Turkish word, but 'de' can function as a valid clitic/word written separately in Turkish (e.g., 'Ben de' meaning 'me too'). Since the metric depends on what counts as a valid word, please anchor the definition to a specific dictionary or morphological analyzer and make the example consistent with that resource.
[Section 5] The paragraph mentions 'AhmetSemih/tr_tokenizer' and 'aliarda/turkish_tokenizer' as achieving 'remarkable initial results', but no numerical results or table entries are provided for these tokenizers. Either include the data in the evaluation or remove the unsupported claim.
[References] Reference [6] lists the arXiv identifier 2301.12345, which appears to be a placeholder, and reference [8] lacks full bibliographic details; please verify and complete all references.
[Global presentation] There are several minor typographical issues, such as 'Language-Specific Token Percentagesevaluate' (missing space) and inconsistent spacing in 'V ocabulary'; a light copyedit is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: tokenizer metrics and MMLU scores are measured independently, and the correlation analysis fits no parameter to the target.

full rationale

The paper's derivation chain is not circular. The tokenizer-side metrics %TR and %Pure are defined by Equations 1 and 2 purely from tokenizer outputs and dictionary/morphological validation, with no reference to MMLU scores. The MMLU scores are separate model-evaluation results obtained on the TR-MMLU benchmark, which is cited from the authors' prior work but is a separately published dataset with its own construction and does not encode the tokenizer-metric correlations. The correlation analysis in Section 4 is descriptive: Pearson coefficients are computed from four model/tokenizer rows in Table 1, and no parameter is fitted to the MMLU target, so the headline claim is not equivalent to any input by construction. The self-citation to TR-MMLU is load-bearing only in the mundane sense that every benchmark-based study relies on its chosen dataset; it does not smuggle in the paper's conclusion. Concerns about n=4 and model/tokenizer confounding are threats to statistical identification and external validity, not circularity, and under the hard rules they do not warrant a circularity finding.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No numbers are fitted to the outcome in this paper, so the free-parameter ledger is empty. The central claims rest on three domain assumptions: the TR-MMLU accuracy scores are a valid proxy for downstream performance; the ITU and Kalbur tools correctly identify valid Turkish words and atomic morphemes; and four tokenizers suffice to estimate stable correlations. No new entities are introduced.

assumptions (3)

domain assumption TR-MMLU accuracy is a valid and noise-free proxy for downstream performance in Turkish.
Section 4 uses MMLU scores as the outcome variable in all correlations; if these scores have large evaluation variance, the reported r values become unreliable.
domain assumption ITU Turkish NLP Web Service and Kalbur correctly identify valid Turkish words and atomic morphemes.
Section 3 defines %TR and %Pure through these tools; misclassification of suffixes, clitics, or derived forms would change all metric values.
ad hoc to paper The four tested tokenizers are a representative sample for estimating correlations.
Section 4 computes Pearson correlations from Table 1 with n=4 and no uncertainty analysis; the representativeness of the sample is asserted by the study design rather than demonstrated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Tokenization Standards for Linguistic Integrity: Turkish as a Benchmark." pith.science (2026). https://pith.science/paper/VVV4XNY6

@misc{pith2026250207057,
  author       = {Pith},
  title        = {Pith review of: Tokenization Standards for Linguistic Integrity: Turkish as a Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VVV4XNY6}},
  note         = {Machine review of arXiv:2502.07057}
}

read the original abstract

Tokenization is a fundamental preprocessing step in NLP, directly impacting large language models' (LLMs) ability to capture syntactic, morphosyntactic, and semantic structures. This paper introduces a novel framework for systematically evaluating tokenization strategies, addressing challenges in morphologically rich and low-resource languages. Using a Turkish dataset of 6,200 multiple-choice questions from the Massive Multitask Language Understanding (MMLU) benchmark, the framework assesses tokenizers across five key metrics: vocabulary size, token count, processing time, language-specific token percentages (\%TR), and token purity. These metrics provide a structured approach to evaluating how well tokenizers preserve linguistic structures. While \%TR measures the proportion of valid words in the target language, \%Pure assesses the alignment of tokens with meaningful linguistic units, such as roots and valid morphemes, minimizing semantic fragmentation. The findings reveal that \%TR, introduced as a critical metric, exhibits a stronger correlation with downstream performance (e.g., MMLU scores) than token purity, emphasizing its role in improving model accuracy. Additionally, larger model parameters do not necessarily yield better tokenization quality or enhanced results, highlighting the importance of tailored tokenization strategies that prioritize linguistic alignment. This framework sets a new standard for developing robust tokenization methods optimized for morphologically complex and low-resource languages. Future work will refine morphological analysis, explore domain-specific customizations, and conduct cross-linguistic evaluations to further enhance tokenization practices.

Figures

Figures reproduced from arXiv: 2502.07057 by the authors.

**Figure 2.** Model Comparison: MMLU vs TR %, Parameter Size, and Pure %. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 17 canonical work pages

[1]

Tokenization Is More Than Compression

Craig W Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, and Chris Tanner. Tokenization Is More Than Compression. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 678–702, Miami, Florida, USA, November 2024. Associat...

work page 2024
[2]

Miguel Domingo, Mercedes García-Martínez, Alexandre Helle, Francisco Casacuberta, and Manuel Herranz. How much does tokenization affect neural machine translation? In Proceed- ings of the 22nd International Conference on Computational Linguistics and Intelligent Text Processing, pages 545–554. Springer, 2019

work page 2019
[3]

How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in japanese

Takuro Fujii, Koki Shibata, Atsuki Yamaguchi, Terufumi Morishita, and Yasuhiro Sogawa. How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in japanese. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 39–49. Association ...

work page 2023
[4]

Critical tokenization and its properties

Jun Guo. Critical tokenization and its properties. Computational Linguistics, 23(4):567–596, 1997

work page 1997
[5]

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing, August 2018

Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing, August 2018. arXiv:1808.06226 [cs]

arXiv 2018
[6]

Formal properties of tokenization in neural language models

Vilém Zouhar et al. Formal properties of tokenization in neural language models. arXiv preprint arXiv:2301.12345, 2023

arXiv 2023
[7]

Formalizing BPE Tokenization

Martin Berglund and Brink van der Merwe. Formalizing BPE Tokenization. Electronic Proceedings in Theoretical Computer Science, 388:16–27, September 2023. arXiv:2309.08715 [cs]

arXiv 2023
[8]

The Technical User’s Introduction to LLM Tokenization

Christopher Samiullah. The Technical User’s Introduction to LLM Tokenization

work page

Show all 23 references

[9]

A New Algorithm for Data Compression, 1994

Philip Gage. A New Algorithm for Data Compression, 1994

1994
[10]

github.com/riotu-lab/aranizer, December 2024

Anis Koubaa, Lahouari Ghouti, Omar Najar, and Serry Sebai. github.com/riotu-lab/aranizer, December 2024. original-date: 2023-12-19T07:57:47Z

2024
[11]

So many tokens, so little time: Introducing a faster, more flexible byte-pair tokenizer, December 2024

Hendrik van Antwerpen Neubeck, Alexander. So many tokens, so little time: Introducing a faster, more flexible byte-pair tokenizer, December 2024

2024
[12]

Arabic Tokenizers Leaderboard - a Hugging Face Space by MohamedRashad

Mohamed Rashad. Arabic Tokenizers Leaderboard - a Hugging Face Space by MohamedRashad
[13]

NbAiLab/tokenizer-benchmark, November 2024

Javier de la Rosa and Rolv Arild. NbAiLab/tokenizer-benchmark, November 2024. original-date: 2024-03-23T09:22:14Z

2024
[14]

Tokenizing on scale.Preprocessing large text corpora on the lexical and sentence level

Nils Diewald, Marc Kupietz, and Harald Lüngen. Tokenizing on scale.Preprocessing large text corpora on the lexical and sentence level. 2022. 9

2022
[15]

Analysis of Subword Tokenization Approaches for Turkish Language

Erencan Erkaya and Tunga Güngör. Analysis of Subword Tokenization Approaches for Turkish Language. In 2023 31st Signal Processing and Communications Applications Conference (SIU), pages 1–4, July 2023. ISSN: 2165-0608

2023
[16]

Guerreiro, Ricardo Rei, Duarte M

Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. EuroLLM: Multil...

2024 arXiv
[17]

How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models, June

Phillip Rust, Jonas Pfeiffer, Ivan Vuli´c, Sebastian Ruder, and Iryna Gurevych. How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models, June
[18]

Not All Tokens Are What You Need for Pretraining

Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, and Weizhu Chen. Not All Tokens Are What You Need for Pretraining
[19]

Ali Bayram, Ali Arda Fincan, Ahmet Semih Gümü¸ s, Banu Diri, Sava¸ s Yıldırım, and Öner Ayta¸ s

M. Ali Bayram, Ali Arda Fincan, Ahmet Semih Gümü¸ s, Banu Diri, Sava¸ s Yıldırım, and Öner Ayta¸ s. Setting Standards in Turkish NLP: TR-MMLU for Large Language Model Evaluation, January 2025. arXiv:2501.00593 [cs]

2025 arXiv
[20]

ITU Turkish NLP Web Service

Gül¸ sen Eryi˘git. ITU Turkish NLP Web Service. In Proceedings of the Demonstrations at the 14th Conference of the European Chapter of the Association for Computational Linguistics , pages 1–4, Gothenburg, Sweden, 2014. Association for Computational Linguistics

2014
[21]

ahmetax/kalbur, October 2024

Ahmet Aksoy. ahmetax/kalbur, October 2024. original-date: 2016-10-26T10:25:48Z

2024
[22]

Ali Bayram

M. Ali Bayram. malibayram/tokenizer_benchmark, December 2024. original-date: 2024-12- 06T14:03:02Z. 10

2024
[2021]

arXiv:2012.15613 [cs]

2012 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

Tokenization Is More Than Compression

Craig W Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, and Chris Tanner. Tokenization Is More Than Compression. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 678–702, Miami, Florida, USA, November 2024. Associat...

work page 2024

[2] [2]

Miguel Domingo, Mercedes García-Martínez, Alexandre Helle, Francisco Casacuberta, and Manuel Herranz. How much does tokenization affect neural machine translation? In Proceed- ings of the 22nd International Conference on Computational Linguistics and Intelligent Text Processing, pages 545–554. Springer, 2019

work page 2019

[3] [3]

How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in japanese

Takuro Fujii, Koki Shibata, Atsuki Yamaguchi, Terufumi Morishita, and Yasuhiro Sogawa. How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in japanese. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 39–49. Association ...

work page 2023

[4] [4]

Critical tokenization and its properties

Jun Guo. Critical tokenization and its properties. Computational Linguistics, 23(4):567–596, 1997

work page 1997

[5] [5]

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing, August 2018

Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing, August 2018. arXiv:1808.06226 [cs]

arXiv 2018

[6] [6]

Formal properties of tokenization in neural language models

Vilém Zouhar et al. Formal properties of tokenization in neural language models. arXiv preprint arXiv:2301.12345, 2023

arXiv 2023

[7] [7]

Formalizing BPE Tokenization

Martin Berglund and Brink van der Merwe. Formalizing BPE Tokenization. Electronic Proceedings in Theoretical Computer Science, 388:16–27, September 2023. arXiv:2309.08715 [cs]

arXiv 2023

[8] [8]

The Technical User’s Introduction to LLM Tokenization

Christopher Samiullah. The Technical User’s Introduction to LLM Tokenization

work page