Morphological alignment of tokenizers across 70 languages explains only about 0.5% to 6% of variance in language model task performance, with a small negative trend.
Title resolution pending
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Evaluating Morphological Alignment of Tokenizers in 70 Languages
Morphological alignment of tokenizers across 70 languages explains only about 0.5% to 6% of variance in language model task performance, with a small negative trend.