In a four-tokenizer comparison on Turkish MMLU questions, the fraction of tokens that are valid Turkish words correlates with accuracy more strongly than the fraction of morphologically pure tokens.
Critical tokenization and its properties
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
citation-role summary
background 1
citation-polarity summary
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1roles
background 1polarities
unclear 1representative citing papers
citing papers explorer
-
Tokenization Standards for Linguistic Integrity: Turkish as a Benchmark
In a four-tokenizer comparison on Turkish MMLU questions, the fraction of tokens that are valid Turkish words correlates with accuracy more strongly than the fraction of morphologically pure tokens.