Evaluating Morphological Alignment of Tokenizers in 70 Languages

Brendan O'Connor; Catherine Arnett; Marisa Hudspeth

REVIEW 4 major objections 5 minor 1 cited by

Evaluating Morphological Alignment of Tokenizers in 70 Languages

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that morphological alignment of tokenizers, measured by an expanded MorphScore across 70 languages, explains little to none of the variance in downstream language model performance, with a small negative correlation.

desk verdict A genuinely useful tokenizer alignment resource, but the null result rests on statistical reporting that has an impossible chi-square and internal language-list contradictions. read the letter →

arxiv 2507.06378 v1 pith:XWCSD4DV submitted 2025-07-08 cs.CL

Catherine Arnett , Marisa Hudspeth , Brendan O'Connor This is my paper

classification cs.CL

keywords morphologicalalignmenttokenizationmultilingualmodelsintrinsicevaluationMorphScoresubwordsegmentationUniversalDependencies

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper expands MorphScore, an intrinsic tokenizer evaluation that measures how often token boundaries align with morpheme boundaries, from 22 to 70 languages, and correlates these alignment scores with downstream task performance for five pretrained language models across seven tasks. It finds that morphological alignment explains only a small fraction of the variance in performance, with recall $R^2 = 0.024$ and precision $R^2 = 0.005$ in the full mixed-effects model, and that the correlation is negative. The authors interpret this as evidence that morphological alignment alone does not measure dimensions of tokenization quality relevant to model performance. A sympathetic reader would care because reliable intrinsic tokenizer evaluation would let practitioners pick a tokenizer before expensive model training; this result suggests morphological alignment is not that metric on its own.

What carries the argument

The central object is MorphScore, an evaluation that computes boundary-level precision and recall and subword-level precision, recall, and F1 by comparing tokenizer output against gold morphological segmentations. Gold segmentations are inferred automatically from Universal Dependencies wordform-lemma pairs by taking the longest shared substring as the stem and treating preceding and following characters as prefix and suffix. The scoring function, which penalizes oversegmentation through precision, is what carries the analysis; the dataset construction and scoring choices determine the alignment values that are then correlated with task performance.

What would settle it

Recompute morphological alignment on the same 70 languages using gold segmentations from a linguistically validated source (such as morphological analyzers or expert annotation) and re-run the correlation with model performance; if the correlation becomes substantially positive and explains a meaningful share of variance, the paper's null result would be shown to be an artifact of the approximate gold standard rather than a true absence of relationship.

Watch

Extended reading notes

Core claim

Across 70 languages, morphological alignment scores computed with the updated MorphScore show a small negative correlation with language model performance on downstream tasks, after controlling for model size and estimated training data proportion. The effect is so small that morphological alignment explains almost no variance in task scores, and the finding is consistent with the earlier 22-language result. The paper also shows that both the inclusion of one-token words and frequency scaling change alignment scores and relative tokenizer rankings, and that the frequency-scaled, one-token-excluded setting is slightly more predictive of performance, though still weak. The authors conclude that morphological alignment, taken in isolation, is not sufficient to classify tokenization as optimal.

Load-bearing premise

The gold morphological segmentation is derived by assuming the longest shared substring between a wordform and its lemma is the true stem, with all other characters treated as affixes, which is wrong for many words with spelling changes and makes the alignment score partly a measure of that heuristic rather than of real morphology.

Editorial extensions

If this is right

If morphological alignment does not predict performance, tokenizer selection for training should not be based on alignment scores alone, and existing claims that morpheme-aligned tokenization improves downstream performance need re-examination.
Combining morphological alignment with other intrinsic metrics such as compression or Rényi efficiency may yield a more informative evaluation of tokenizer quality than any single metric.
The released datasets and flexible evaluator, which include sentential context, part-of-speech, and morphological information, support fine-grained analyses of when alignment matters, such as by part of speech.
The finding that more frequent words are slightly more likely to be segmented morphemically, and that one-token inclusion inflates scores, means any tokenizer comparison should control for these scoring choices.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The null correlation could be partly an artifact of measurement error in the gold segmentation: the longest-shared-substring heuristic produces incorrect stems for words with spelling changes or non-concatenative morphology, which would attenuate any true relationship toward zero.
The result is specific to the evaluated tasks (topic classification, entailment, commonsense reasoning, and subject-verb agreement) and to autoregressive decoder-only models, so morphological alignment might still predict performance for other task families such as morphological generation or sequence labeling.
A testable extension would be to recompute alignment with linguistically validated segmentations (e.g., from morphological analyzers or human annotation) on the same 70-language sample and re-run the correlation; a strongly positive result would indicate the current null is driven by segmentation noise.
If alignment genuinely does not predict performance, then tokenization quality may be better captured by compression-related quantities or by how well tokens preserve meaning for frequent items, rather than by boundary fidelity to morphemes.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Hi, the short version: this paper expands MorphScore from 22 to 70 languages and reports that morphological alignment explains almost none of the variance in LM downstream performance. That null result is plausible, consistent with Arnett & Bergen (2025) and with several other negative findings, and it deserves attention. But the quantitative support is undercut by a few reporting and statistical problems that need to be fixed before the numbers can be taken at face value.

The real contribution is the resource: MorphScore for 70 languages with frequency scaling and one-token toggles, boundary vs subword precision/recall, plus released code and data. The parameter analysis is genuinely useful. They find that frequency scaling and excluding one-token words gives a slightly more predictive score, and they are honest that the differences are small. The limitations section is open about the coarse boundary operationalization.

Now the soft spots. First, Section 4 reports a chi-square of -6.99 for the precision ANOVA. Chi-square cannot be negative; the statistic is nonnegative by definition. That is a modeling or reporting error. It also reports recall with chi-square 391.42, which seems implausibly large for a model with such a small R2 (0.024). Something is off; the variance decomposition needs to be re-run from the raw data. Second, Appendix B lists a standard deviation of 0.00 for every language-tokenizer pair. Exactly zero standard deviation is not believable unless these are rounded or the value is a constant. The table needs a clear description of what the uncertainty measure is. Third, the text claims Semitic languages and isolating languages like Mandarin were excluded because they lacked sufficient items, but Appendix A includes Hebrew with 4641 items and Appendix B scores Mandarin Chinese. That is a direct internal contradiction and suggests the language list was not reconciled with the filtering criteria. Fourth, the gold segmentation uses the longest shared substring between wordform and lemma, which misidentifies stems when orthographic changes happen (studies -> stud+ies). The paper admits the coarseness, but the analysis proceeds as if the metric were valid. Measurement error here could attenuate a true positive correlation toward zero, so the null result should be framed as conditional on the validity of the segmentation.

The data quality problems are real but not load-bearing in the sense that the qualitative null is very likely correct. Still, a reader cannot currently reproduce the key numbers, and that weakens the paper.

This paper is for tokenizer evaluation and multilingual NLP researchers. It deserves a serious referee, and my recommendation is conditional acceptance: redo the statistics, reconcile the language lists, and either re-validate the gold segmentation or substantially soften the claim. With those changes it is a useful resource.

Referee Report

4 major / 5 minor

Summary. This paper extends MorphScore from 22 to 70 languages, introducing boundary-level and subword-level precision/recall metrics, and tests two parameter choices (frequency scaling and inclusion of single-token words). It then correlates alignment scores with reported downstream performance for BLOOM, XGLM, Llama2, Llama3, and Gemma3 across seven tasks. The main finding is a small negative correlation between morphological alignment and task performance (LMM R² ≈ 0.024 for recall, 0.005 for precision), which the authors interpret as evidence that morphological alignment alone does not measure task-relevant tokenization quality.

Significance. If correct, the result is a useful large-scale negative result: it challenges a common assumption that morphologically aligned tokenization benefits model performance, and it contributes a released dataset and evaluation framework for 70 languages. The expansion of language coverage and the sensitivity analysis over scoring parameters are valuable. However, the conclusion depends on the quality of the inferred gold segmentations and on the comparability of the alignment and performance data; the issues below must be resolved before the claim can be fully credited.

major comments (4)

[Section 2, Appendix A/B] The text states that Semitic and isolating languages (including Amharic, Arabic, Hebrew, Chinese, Vietnamese, and Thai) are not included because the dataset creation process did not yield sufficient items, yet Appendix A lists Hebrew with 4,641 items and Appendix B reports precision and recall scores for both Hebrew and Mandarin Chinese (e.g., Tables 3 and 5). This contradiction makes the actual language sample unclear. Since the central correlation is computed over the 70-language sample, please clarify whether Hebrew and Mandarin Chinese are included in the final 70, and if so, explain how they passed the filtering threshold and reconcile this with the exclusion claim in Section 2.
[Section 4, Results] The reported test statistic 'precision does not (χ2(1) = −6.99, p = 1)' is impossible because a chi-square statistic cannot be negative. This statistic is the basis for the claim that precision does not explain additional variance beyond the control variables, so as reported the result cannot be evaluated. Please re-run the ANOVA or likelihood-ratio comparison and report the correct statistic, or clarify if a different procedure was used.
[Section 2, Creating Evaluation Datasets] The gold segmentation is derived by taking the longest shared substring between wordform and lemma and treating remaining characters as affixes. This systematically misidentifies stems when inflection involves spelling changes, e.g., English 'studies' from 'study' yields stem 'stud' plus suffix 'ies' instead of stem 'study' plus suffix 's'; the recomposition filter passes such cases whenever concatenation reproduces the wordform. Because MorphScore precision and recall are computed against these boundaries, the predictor contains non-random measurement error, which can attenuate a true correlation with model performance. The Limitations section acknowledges that the operationalization is coarse, but §4's null conclusion requires evidence that this measurement error does not drive the near-zero R²; please quantify the error rate or validate the inferred segmentations against a morphologically annotated resource.
[Section 4, Method] The performance data include MultiBLiMP, which the authors note is itself derived from Universal Dependencies, and the MorphScore datasets are also built from UD treebanks. This shared data source creates a potential confound: language-specific properties of UD treebank size, annotation quality, or text domain could affect both the alignment scores and the MultiBLiMP performance scores. Please discuss this overlap explicitly and test the robustness of the correlation to excluding MultiBLiMP or restricting the analysis to tasks that are not derived from UD.

minor comments (5)

[Section 4, Method] There is a typo: 'using an ANOV A' should be 'using an ANOVA.'
[Section 4, Results] The phrase 'the proportion of training data in each languages' should be 'the proportion of training data in each language.'
[Figure 2] The caption reports R²=0.06 for the simple regressions, while the text reports LMM R²=0.024 and 0.005; please state in the caption which model each R² refers to.
[Appendix D] The condition names True_True, True_False, etc. are unintuitive because the ordering of the two booleans is not immediately clear; please define the naming convention explicitly in the caption, e.g., frequency_scaled.one_token_excluded.
[Table 3] Mandarin Chinese appears in the precision and recall tables despite the Section 2 exclusion claim; if it is intentionally included, add a note to the table explaining this.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the null correlation is an empirical result, not a consequence of how MorphScore is defined.

full rationale

The paper's central claim is an empirical negative correlation between MorphScore alignment and benchmark performance. The alignment scores are computed from tokenizer output against gold segmentations derived from UD wordform/lemma pairs, and the performance scores are taken from external published benchmarks (MultiBLiMP, XNLI, XCOPA, etc.) and from the authors' prior paper, but those prior scores are themselves external task results, not re-fitted values. No equation in Section 4 defines performance in terms of MorphScore, and no fitted parameter is renamed as a prediction; the 'optimal default settings' choice is a model-selection step over the same correlation data, but all conditions show the same small negative correlation, so the null finding does not reduce to the parameter choice. The self-citations to Arnett & Bergen (2025) introduce the original metric and performance table, but the present paper re-implements and expands the metric and adds new models and languages; the cited results are independent support rather than a forced premise. The coarse segmentation via longest shared substring is a measurement-validity limitation (acknowledged in the Limitations section), not a circular derivation: even if the gold boundaries are noisy, the regression result is not equivalent to the metric's construction.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central analysis relies on a coarse inferred gold segmentation and on performance scores from heterogeneous sources; the free parameters are design choices selected on the same data, which modestly inflates circularity.

free parameters (3)

language_filter_threshold = 100 items
Languages with fewer than 100 items after dataset creation are excluded; this threshold is chosen by hand and affects which languages enter the analysis.
frequency_scaling_setting = True (final default)
The scoring function can be weighted by wordform frequency; the paper selects the setting that maximizes predictive power on the same performance data, a data-dependent choice.
one_token_inclusion_setting = False (final default)
Whether single-token words are included in the score; the default is chosen by predictive power on the same data.

assumptions (5)

domain assumption The longest shared substring between a wordform and its UD lemma identifies the morphological stem for the selected items.
Used in Section 2 to construct gold segmentations; fails for non-concatenative morphology and spelling changes, which the authors partially exclude.
domain assumption Universal Dependencies annotations provide reliable wordform, lemma, and word-boundary information across 70 languages.
All evaluation items are built from UD treebanks; treebank quality varies by language.
domain assumption Reported downstream task scores from different papers and benchmarks are comparable for correlation analysis.
Used in Section 4; scores come from different evaluation protocols, model versions, and prompting setups.
domain assumption Estimated training data proportions from Hayase et al. (2024) are accurate enough as control variables.
Used as a fixed effect in the mixed models for models that do not disclose training data composition.
domain assumption MultiBLiMP subject-verb agreement is a valid representative downstream task for model performance across these languages.
MultiBLiMP provides the only performance signal for some languages in the sample.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Evaluating Morphological Alignment of Tokenizers in 70 Languages." pith.science (2026). https://pith.science/paper/XWCSD4DV

@misc{pith2026250706378,
  author       = {Pith},
  title        = {Pith review of: Evaluating Morphological Alignment of Tokenizers in 70 Languages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XWCSD4DV}},
  note         = {Machine review of arXiv:2507.06378}
}

read the original abstract

While tokenization is a key step in language modeling, with effects on model training and performance, it remains unclear how to effectively evaluate tokenizer quality. One proposed dimension of tokenizer quality is the extent to which tokenizers preserve linguistically meaningful subwords, aligning token boundaries with morphological boundaries within a word. We expand MorphScore (Arnett & Bergen, 2025), which previously covered 22 languages, to support a total of 70 languages. The updated MorphScore offers more flexibility in evaluation and addresses some of the limitations of the original version. We then correlate our alignment scores with downstream task performance for five pre-trained languages models on seven tasks, with at least one task in each of the languages in our sample. We find that morphological alignment does not explain very much variance in model performance, suggesting that morphological alignment alone does not measure dimensions of tokenization quality relevant to model performance.

Figures

Figures reproduced from arXiv: 2507.06378 by the authors.

**Figure 1.** Example of root template pattern in Arabic. rion, we could get gold segementations that would not be informative about the quality of tokenization. For example, the infinitival form of the verb ‘to be’ in Afrikaans is wees. The present form for all persons and numbers is is. Under our segmentation approach, the stem would be identified as -s and the proposed gold segmentation would be [i + s]. However, is is an irre… view at source ↗

**Figure 2.** Correlation between model performance on different tasks (color-coded) for recall (left) and precision (right). developers do not release that information about the pretraining data. We test the correlation using linear mixed effects models. As it is known that model size, in parameters, and proportion of the training data in each language impact performance (Kaplan et al., 2020 and Bagheri Nezhad & Agrawal, 2024; … view at source ↗

**Figure 3.** Correlation between morphological alignment measured with precision and task score. Model task is indicated by color. 0.00 0.25 0.50 0.75 1.00 Morph. Alignment (Recall) 0.2 0.4 0.6 0.8 1.0 Task Score True_True sib200 xstorycloze xnli wiki xwinograd xcopa multiblimp Best fit (R² = 0.02) 0.25 0.50 0.75 1.00 Morph. Alignment (Recall) 0.0 0.2 0.4 0.6 0.8 1.0 Task Score True_False sib200 xstorycloze xnli wiki xwinograd x… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Correlation between morphological alignment measured with recall and task score. Model task is indicated by color. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

MORFES: A Benchmark for Productive Inflectional Competence in Modern Greek
cs.CL 2026-07 conditional novelty 6.0 of 10

MORFES is the first expert-verified Modern Greek productive-inflection benchmark; Sophea-Genesis-1 leads it at 84% per-item production without losing general capability.

Reference graph

Works this paper leans on

21 extracted references · 11 canonical work pages · cited by 1 Pith paper

[8]

Dobrovoljc, K

Version 2.15. Dobrovoljc, K. and Ljubeši´c, N. Extending the SSJ Univer- sal Dependencies treebank for Slovenian: Was it worth it? In Pradhan, S. and Kuebler, S. (eds.), Proceedings of the 16th Linguistic Annotation Workshop (LAW-XVI) within LREC2022, pp. 15–22, Marseille, France, June 2022. Eu- ropean Language Resources Association. URL https: //aclantho...

work page 2022
[11]

Poster presentation

URL https://openreview.net/forum? id=0SRg6Cwx3h. Poster presentation. Heinecke, J. and Tyers, F. M. Development of a Uni- versal Dependencies treebank for Welsh. In Proceed- ings of the Celtic Language Technology Workshop, pp. 21–31, Dublin, 2019. European Association for Ma- chine Translation. URL https://www.aclweb. org/anthology/W19-6904. Hellwig, O., ...

arXiv 2019
[12]

10 Morphologically Aligned Tokenizers Kuzgun, A., Cesur, N., Yıldız, O

Version 2.15. 10 Morphologically Aligned Tokenizers Kuzgun, A., Cesur, N., Yıldız, O. T., Kuyrukçu, O., Yenice, A. B., Arıcan, B. N., and Sanıyar, E. Ud turkish-kenet. https: //github.com/UniversalDependencies/ UD_Turkish-Kenet, 2021. Version 2.8. Laan, K. Ud veps-vwt. https://github.com/ UniversalDependencies/UD_Veps-VWT, 2024. Version 2.14. Larasati, S....

work page 2021
[13]

Splintering Nonconcatenative Languages for Better Tokenization

URL https://aclanthology.org/2022. naacl-demo.13/. Faryad, J. and Zeman, D. Ud pashto-sikaram. https: //github.com/UniversalDependencies/ UD_Pashto-Sikaram, 2024. Version 2.14. Gage, P. A new algorithm for data compression. The C Users Journal, 12(2):23–38, 1994. Gallé, M. Investigating the effectiveness of bpe: The power of shorter sequences. In Proceedi...

work page Pith review arXiv 2022
[14]

org/CorpusID:270620835

URL https://api.semanticscholar. org/CorpusID:270620835. Lin, X. V ., Mihaylov, T., Artetxe, M., Wang, T., Chen, S., Simig, D., Ott, M., Goyal, N., Bhosale, S., Du, J., et al. Few-shot learning with multilingual language models. arXiv preprint arXiv:2112.10668, 2021. Liu, A., Hayase, J., Hofmann, V ., Oh, S., Smith, N. A., and Choi, Y . Superbpe: Space tr...

arXiv 2021
[16]

Ramasamy, L

Version 1.0. Ramasamy, L. and Žabokrtský, Z. Prague dependency style treebank for Tamil. In Chair), N. C. C., Choukri, K., Declerck, T., Do˘gan, M. U., Maegaard, B., Mariani, J., Moreno, A., Odijk, J., and Piperidis, S. (eds.), Proceed- ings of Eighth International Conference on Language Resources and Evaluation (LREC 2012), pp. 1888–1894, ˙Istanbul, Turk...

2012
[18]

doi: 10.18653/v1/2024.emnlp-main.40

Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.40. URL https:// aclanthology.org/2024.emnlp-main.40/. Schmidt, C. W., Reddy, V ., Tanner, C., and Pinter, Y . Bound- less byte pair encoding: Breaking the pre-tokenization barrier. arXiv preprint arXiv:2504.00178, 2025. Sennrich, R., Haddow, B., and Birch, A. Neural machine trans...

arXiv 2024
[19]

Talamo, L

Version 2.14. Talamo, L. Introducing staf: The saarbrücken treebank of albanian fiction. Journal of Open Humanities Data, 11 (1), 2025. Taulé, M., Martí, M. A., and Recasens, M. Ancora: Multi- level annotated corpora for catalan and spanish. In Lrec, volume 2008, pp. 96–101, 2008. Team, G., Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Pe...

arXiv 2025

Show all 21 references

[22]

eacl-srw.22

URL https://aclanthology.org/2021. eacl-srw.22. Samardži´c, T. and Ljubeši ´c, N. Ud serbian-set. https: //github.com/UniversalDependencies/ UD_Serbian-SET, 2024. Version 2.4. Sazdov, S. Sovremen makedonski jazik 4 . Tabernakul, Skopje, 2 edition, 2012. English title: Contempo...

2021
[114]

Borges Völker, E., Wendt, M., Hennig, F., and Köhn, A

IOS Press, 2016. Borges Völker, E., Wendt, M., Hennig, F., and Köhn, A. HDT-UD: A very large Universal Dependencies treebank for German. In Rademaker, A. and Tyers, F. (eds.), Proceedings of the Third Workshop on Univer- sal Dependencies (UDW, SyntaxFest 2019), pp. 46–57, Pari...

2016 doi
[185]

emnlp-main.185/

URL https://aclanthology.org/2020. emnlp-main.185/. Pretkalnin, a, L., Rituma, L., and Saul ¯ıte, B. Deriving en- hanced universal dependencies from a hybrid dependency- constituency treebank. In Text, Speech, and Dialogue: 21st International Conference, TSD 2018, Brno, Czech ...

2020
[247]

findings-naacl.247/

URL https://aclanthology.org/2024. findings-naacl.247/. Aranzabe, M. J., Atutxa, A., Bengoetxea, K., Diaz, A., de Ilarraza, I. G., Gojenola, K., and Uria, L. Automatic conversion of the basque dependency treebank to univer- sal dependencies. In International Workshop on Tree- ...

2024 arXiv
[324]

naacl-long.324/

URL https://aclanthology.org/2024. naacl-long.324/. Bejˇcek, E., Haji ˇc, J., Haji ˇcová, E., Kolá ˇrová, V ., and Vidová-Hladká, B. Ud_czech-cac: Czech cac treebank. https://github.com/ UniversalDependencies/UD_Czech-CAC,

2024
[414]

findings-emnlp.414

URL https://aclanthology.org/2020. findings-emnlp.414. Branco, A., Silva, J. R., Gomes, L., and António Rodrigues, J. Universal grammatical dependencies for Portuguese with CINTIL data, LX processing and CLARIN sup- port. In Calzolari, N., Béchet, F., Blache, P., Choukri, K., ...

2020
[614]

emnlp-main.614/

URL https://aclanthology.org/2023. emnlp-main.614/. Ahrenberg, L. LinES: An English-Swedish parallel tree- bank. In Nivre, J., Kaalep, H.-J., Muischnek, K., and Koit, M. (eds.), Proceedings of the 16th Nordic Confer- ence of Computational Linguistics (NODALIDA 2007), pp. 270–2...

2023
[2015]

Rahman, M

URL https://aclweb.org/anthology/ W15-1821.pdf. Rahman, M. U., Qureshi, S., Pirzada, S., Shah, S., Shaheer, M., Talpur, M. A. A., Sanjrani, Z., and Bauer, J. Ud sindhi-isra. https://github.com/ UniversalDependencies/UD_Sindhi-Isra,
[2018]

Tyers, F

to appear. Tyers, F. M. and Washington, J. N. Towards a free/open- source universal-dependency treebank for kazakh. In 3rd International Conference on Turkic Languages Process- ing, (TurkLang 2015), pp. 276–289, 2015. Uzan, O., Schmidt, C. W., Tanner, C., and Pinter, Y . Greed...

2015 arXiv
[2019]

tal-2.4/

URL https://aclanthology.org/2019. tal-2.4/. Guinovart, X. G. Recursos integrados da lingua galega para a investigación lingüística. In Gallæcia: Estudos de lingüística portuguesa e galega, pp. 1037–1048. Univer- sidad de Santiago de Compostela, 2017. Hakkani-Tür, D. Z., Oflaz...

2019
[2022]

Universal Dependencies 2.10 release. Bhat, R. A., Bhatt, R., Farudi, A., Klassen, P., Narasimhan, B., Palmer, M., Rambow, O., Sharma, D. M., Vaidya, A., Vishnu, S. R., et al. The hindi/urdu treebank project. In Handbook of Linguistic Annotation. Springer Press, 2017. Bielinski...

2017
[2023]

14 Morphologically Aligned Tokenizers A

Universal Dependencies v2 treebank. 14 Morphologically Aligned Tokenizers A. Language Sample Table 2 reports the number of items for each language after filtering and the UD treebank used to create the dataset for that language. The versions of each treebank that are used are ...

2016
[2024]

Dione, C

URL https://openreview.net/forum? id=jznbgiynus. Dione, C. B. Ud wolof-wtb. https://github.com/ UniversalDependencies/UD_Wolof-WTB,

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [8]

Dobrovoljc, K

Version 2.15. Dobrovoljc, K. and Ljubeši´c, N. Extending the SSJ Univer- sal Dependencies treebank for Slovenian: Was it worth it? In Pradhan, S. and Kuebler, S. (eds.), Proceedings of the 16th Linguistic Annotation Workshop (LAW-XVI) within LREC2022, pp. 15–22, Marseille, France, June 2022. Eu- ropean Language Resources Association. URL https: //aclantho...

work page 2022

[2] [11]

Poster presentation

URL https://openreview.net/forum? id=0SRg6Cwx3h. Poster presentation. Heinecke, J. and Tyers, F. M. Development of a Uni- versal Dependencies treebank for Welsh. In Proceed- ings of the Celtic Language Technology Workshop, pp. 21–31, Dublin, 2019. European Association for Ma- chine Translation. URL https://www.aclweb. org/anthology/W19-6904. Hellwig, O., ...

arXiv 2019

[3] [12]

10 Morphologically Aligned Tokenizers Kuzgun, A., Cesur, N., Yıldız, O

Version 2.15. 10 Morphologically Aligned Tokenizers Kuzgun, A., Cesur, N., Yıldız, O. T., Kuyrukçu, O., Yenice, A. B., Arıcan, B. N., and Sanıyar, E. Ud turkish-kenet. https: //github.com/UniversalDependencies/ UD_Turkish-Kenet, 2021. Version 2.8. Laan, K. Ud veps-vwt. https://github.com/ UniversalDependencies/UD_Veps-VWT, 2024. Version 2.14. Larasati, S....

work page 2021

[4] [13]

Splintering Nonconcatenative Languages for Better Tokenization

URL https://aclanthology.org/2022. naacl-demo.13/. Faryad, J. and Zeman, D. Ud pashto-sikaram. https: //github.com/UniversalDependencies/ UD_Pashto-Sikaram, 2024. Version 2.14. Gage, P. A new algorithm for data compression. The C Users Journal, 12(2):23–38, 1994. Gallé, M. Investigating the effectiveness of bpe: The power of shorter sequences. In Proceedi...

work page Pith review arXiv 2022

[5] [14]

org/CorpusID:270620835

URL https://api.semanticscholar. org/CorpusID:270620835. Lin, X. V ., Mihaylov, T., Artetxe, M., Wang, T., Chen, S., Simig, D., Ott, M., Goyal, N., Bhosale, S., Du, J., et al. Few-shot learning with multilingual language models. arXiv preprint arXiv:2112.10668, 2021. Liu, A., Hayase, J., Hofmann, V ., Oh, S., Smith, N. A., and Choi, Y . Superbpe: Space tr...

arXiv 2021

[6] [16]

Ramasamy, L

Version 1.0. Ramasamy, L. and Žabokrtský, Z. Prague dependency style treebank for Tamil. In Chair), N. C. C., Choukri, K., Declerck, T., Do˘gan, M. U., Maegaard, B., Mariani, J., Moreno, A., Odijk, J., and Piperidis, S. (eds.), Proceed- ings of Eighth International Conference on Language Resources and Evaluation (LREC 2012), pp. 1888–1894, ˙Istanbul, Turk...

2012

[7] [18]

doi: 10.18653/v1/2024.emnlp-main.40

Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.40. URL https:// aclanthology.org/2024.emnlp-main.40/. Schmidt, C. W., Reddy, V ., Tanner, C., and Pinter, Y . Bound- less byte pair encoding: Breaking the pre-tokenization barrier. arXiv preprint arXiv:2504.00178, 2025. Sennrich, R., Haddow, B., and Birch, A. Neural machine trans...

arXiv 2024

[8] [19]

Talamo, L

Version 2.14. Talamo, L. Introducing staf: The saarbrücken treebank of albanian fiction. Journal of Open Humanities Data, 11 (1), 2025. Taulé, M., Martí, M. A., and Recasens, M. Ancora: Multi- level annotated corpora for catalan and spanish. In Lrec, volume 2008, pp. 96–101, 2008. Team, G., Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Pe...

arXiv 2025