MoVoC: Morphology-Aware Subword Construction for Geez Script Languages

Dren Fazlija; Hailay Kidu Teklehaymanot; Wolfgang Nejdl

REVIEW 4 major objections 5 minor 9 references

MoVoC: Morphology-Aware Subword Construction for Geez Script Languages

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MoVoC-Tok aligns subword splits with morpheme boundaries in four Geez-script languages.

desk verdict Useful dataset release and a sensible constrained-BPE method for Geez-script languages, but the paper's central claim of consistent intrinsic improvements is undermined by circular evaluation and internal contradictions. read the letter →

arxiv 2509.08812 v1 pith:3SMDWHG2 submitted 2025-09-10 cs.CL cs.AI

Hailay Kidu Teklehaymanot , Dren Fazlija , Wolfgang Nejdl This is my paper · ORCID

classification cs.CLcs.AI

keywords GeezscriptAmharicTigrinyaTigreGe'ezmorpheme-awaretokenizationBytePairEncodinglow-resourceNLP

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes MoVoC, a method that builds a tokenizer vocabulary by mixing frequent morphemes from supervised morphological analysis with standard BPE subwords, and then restricts BPE merges so that no learned token crosses a morpheme boundary. The resulting tokenizer, MoVoC-Tok, is evaluated on Amharic, Tigrinya, Ge'ez, and Tigre. The authors claim consistent improvements in intrinsic measures of morphological fidelity—MorphoScore, morpheme boundary precision, and Rényi entropy—across all four languages, while acknowledging that machine translation quality does not significantly improve. They also release manually annotated morpheme datasets for the four languages and a morpheme-aware vocabulary for two of them.

What carries the argument

MoVoC (Morpheme-aware Subword Vocabulary Construction) is a three-step pipeline: (1) supervised morphological segmentation of the corpus into morphemes; (2) construction of a hybrid vocabulary that reserves a hyperparameter-controlled share of tokens for the most frequent morphemes and the rest for frequency-based BPE subwords; (3) constrained BPE training in which a merge operation is permitted only if the merging pair does not span a morpheme boundary. This final constraint is what makes MoVoC-Tok morphologically faithful rather than merely morpheme-initialized.

What would settle it

A concrete check: have human annotators who did not create the training boundaries segment a held-out set of words, then compare MoVoC-Tok and plain BPE against those independent boundaries. If MoVoC-Tok's boundary precision advantage disappears or reverses, the core claim collapses. A second check: train an unsupervised segmenter on the same corpora and see whether it matches or beats MoVoC-Tok's precision; if so, the gains are not due to morphology awareness.

Watch

Extended reading notes

Core claim

The central claim is that constraining subword vocabulary construction with morpheme boundaries yields tokenizations that better match expert-annotated morphological structure in low-resource, morphologically rich languages. The method allocates a fraction of the vocabulary to frequent morphemes extracted from a rule-based segmenter, trains BPE on the remaining budget, merges both sets, and then retrains BPE with merge candidates restricted to not cross the annotated boundaries. Across all four tested languages, MoVoC-Tok outperforms plain BPE on morpheme boundary precision and Rényi entropy, and it achieves MorphScores above the published fusional-language mean, indicating that the hybrid v

Load-bearing premise

The load-bearing premise is that the morpheme annotations used to build the vocabulary and to score the tokenizer are accurate, and that the test words do not overlap with the vocabulary-construction data; if those boundaries are wrong or the test set is not held out, the measured intrinsic gains could be an artifact of the annotation tool rather than a real property of the tokenizer.

Editorial extensions

If this is right

If the claim holds, tokenizers for low-resource morphologically rich languages can be made more linguistically faithful with only a small amount of supervised morpheme annotations.
The released morpheme-annotated datasets provide a benchmark for evaluating morphological segmentation in Amharic, Tigrinya, Ge'ez, and Tigre, which did not previously exist at this scale.
The hybrid vocabulary design can be applied to other fusional or agglutinative languages where a rule-based analyzer or a small annotated sample is available, potentially improving token-level interpretability.
The constrained-BPE mechanism can be reused in any existing BPE-based pipeline without changing the downstream model architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

One implication the authors do not explore is that morphology-aware tokenization may benefit tasks other than translation—such as language modeling, part-of-speech tagging, or morphological inflection—where token-level linguistic consistency matters more than n-gram overlap.
The method's value may be higher for languages with even less training data than Amharic and Tigrinya, since the paper shows the largest intrinsic gains for Tigre and Ge'ez, the two languages with the fewest resources.
A testable extension would be to apply the same constrained-BPE recipe to another script family (e.g., Arabic or Hebrew) and check whether the boundary-precision gains transfer; if they do not, the benefit may be specific to the Geez script's fusional patterns.
If the annotated test set overlaps with the vocabulary-construction data, the reported precision gains could be optimistic; an independent human-validated test set would make the comparison fairer.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The real contribution here is the dataset release and the application of a morphology-constrained BPE variant to four Geez-script languages. That is worth something, and the abstract is honest that MT gains are not significant. But the headline claim of consistent intrinsic improvements does not hold as stated, for reasons that are partly in the paper itself.

What the paper does well: it assembles and manually annotates morpheme data for Amharic, Tigrinya, Ge'ez, and Tigre, and it releases a vocabulary for two of them. The constraint on BPE merges to respect morpheme boundaries is simple and reasonable, and the qualitative examples show the kind of fragmentation BPE produces on these languages. Those resources will be useful to people working on low-resource Semitic NLP.

The soft spots are real, and the biggest is circularity. For Amharic and Tigrinya, HornMorpho is used both to build the morpheme vocabulary and as the gold standard for intrinsic evaluation. The tokenizer is explicitly forbidden from crossing those boundaries, then scored on how well it aligns with them. That measures how consistently the constraint is enforced, not whether the segmentation matches true morphology. The comparison with BPE is therefore largely by construction. To the paper's credit, this circularity does not apply to Ge'ez and Tigre, where the gold annotations are manual and the reported gains are larger. But the paper never explains how MoVoC-Tok, trained only on Amharic/Tigrinya, is applied to those languages, so that result is under-specified.

There are also internal contradictions that undermine trust. The abstract says no significant MT gains, but Section 6 and Table 3 claim MoVoC-Tok "consistently outperforms" across all four languages. Ge'ez is said to be evaluated only intrinsically, yet it appears in the MT table. Table 4 says "32k Vocabularies" while Table 5 reports a 152k MoVoC vocabulary, leaving unclear whether the intrinsic gains come from morphology or just from a larger vocabulary. And the test sets are only 100 sentences assembled from mixed sources, which is thin.

This paper is for people working on low-resource Geez-script tokenization or on the methodological pitfalls of using the same analyzer for training and evaluation. It deserves a serious referee rather than a desk reject, because the datasets are valuable and the idea is worth testing properly, but it needs substantial revision: either re-evaluate Amharic/Tigrinya against independent gold annotations, or explicitly limit the claims to the manually annotated languages, and clean up the contradictions.

Referee Report

4 major / 5 minor

Summary. The paper proposes MoVoC, a morpheme-aware vocabulary construction method for four Geez-script languages (Amharic, Tigrinya, Ge'ez, Tigre), and evaluates a tokenizer (MoVoC-Tok) that constrains BPE merges to respect morpheme boundaries derived from a supervised morphological analyzer. The authors release curated morpheme annotations and report intrinsic metrics (MorphoScore, Boundary Precision, Rényi entropy) plus extrinsic MT results. The abstract and limitations state that MT gains are not significant, while Section 6 and Table 3 claim consistent MT improvements across all languages.

Significance. If the central claim were supported, the curated datasets and a reproducible morphology-aware tokenizer would be useful resources for low-resource Geez-script NLP, and the finding that morphology-aware tokenization improves intrinsic segmentation quality without harming MT would align with and extend existing literature (e.g., Saleva and Lignos 2021; Arnett and Bergen 2025). The paper also contributes public code and data, which is a concrete strength. However, the current evidence has a circularity problem for the two languages with existing analyzers, and internal contradictions about MT performance undermine the main claim. The intrinsic results are therefore not established as evidence of genuine morphological fidelity, and the paper's contribution is reduced to a resource description plus a negative MT result.

major comments (4)

[§3.1, §3.2, §3.3, §5.2] The intrinsic evaluation for Amharic and Tigrinya is circular. HornMorpho is used to segment the training corpus into morphemes (§3.1), those segments are used to build the morpheme vocabulary (§3.2), the same boundaries are used to forbid BPE merges (§3.3), and 'the annotated morphemes serve as a gold-standard test set' (§3.1). MorphoScore and Boundary Precision then measure how faithfully MoVoC-Tok reproduces the HornMorpho training signal, not how well it matches true expert morphology. The reported gains over unconstrained BPE are largely by construction. The paper needs an independently annotated held-out test set for Amharic and Tigrinya, or at least a demonstration that HornMorpho's errors do not inflate the metrics.
[Abstract, §6, §9.1, Table 3] The paper contradicts itself about MT gains. The abstract states 'does not lead to significant gains in automatic translation quality,' and §9.1 repeats 'does not yield significant gains in automatic translation quality.' But §6 claims 'MoVoC-Tok consistently outperforms the other tokenizers across all three translation tasks,' and Table 3 shows MoVoC-Tok best on every language. Additionally, §4.2 says 'Ge’ez was evaluated only intrinsically,' yet Table 3 reports English→Ge’ez BLEU and chrF++ scores. This internal inconsistency must be resolved: either the MT results are significant and the limitations paragraph is wrong, or the MT results are not significant and Section 6/Table 3 overclaim.
[Table 4, Table 5, §5.1] The intrinsic comparison is confounded by vocabulary size. Table 4 is labeled '32k Vocabularies,' but Table 5 reports the Bilingual MoVoC vocabulary as 152k (32k BPE per language plus 80k morphemes per language), and §5.1 says the MT model vocabulary is 63,050 tokens. If MoVoC-Tok is evaluated with a much larger vocabulary than the BPE baseline, then better Rényi entropy and boundary precision may simply reflect more and longer tokens, not morphology awareness. The evaluation should match vocabulary sizes or otherwise control for token count and vocabulary capacity.
[§4.2, Table 3] The test sets are extremely small and their construction is unclear: only 100 sentence pairs per language, with Tigrinya and Tigre supplemented by 'human-validated' items and Ge'ez 'newly created,' while Flores-200 is mentioned but then not used consistently. Table 3 reports means and standard deviations but no significance tests, so 'consistent outperformance' is not statistically supported. The paper should report confidence intervals or pairwise significance tests, and should specify the exact origin, composition, and any overlap of the intrinsic test set with the vocabulary construction data.

minor comments (5)

[§6] 'across all three translation tasks' should be 'four translation tasks' (Amharic, Tigrinya, Tigre, Ge'ez) unless Ge'ez is intentionally excluded, which would contradict Table 3.
[§4.3] Typo: 'using the mplementations from HuggingFace' should be 'implementations.' Also, 'transforms version: "4.51.3"' should be 'transformers version.'
[Table 2] The column header 'No. Items' is vague; clarify whether these are annotated words, sentences, or morpheme tokens. Table 2 also says 'MoVoC-Toktokenizer' with a duplicated 'tok.'
[§4.2] The data description is hard to follow: the first paragraph says 'we finally consider 100 sentence pairs from OPUS' for all languages, but the next paragraph limits Ge'ez to intrinsic evaluation and then Table 3 includes Ge'ez MT. Please harmonize these statements.
[§5.2] MorphScore is described as recall-oriented and excluding unsegmented words; this should be stated before reporting scores in Table 2, and the effect of excluding unsegmented words should be discussed (it can inflate scores for vocabularies with many whole-word entries).

Circularity Check

1 steps flagged · score 6.0 of 10

Intrinsic gains for Amharic/Tigrinya are partly circular: HornMorpho supplies both the training constraints and the gold intrinsic labels.

fitted input called prediction [Sections 3.1, 3.2, 3.3, 5.2]
"§3.1: “The annotated morphemes serve as a gold-standard test set for evaluating token-to-morpheme boundary alignment.” §3.2: “extract_morphemes(P, s_morpheme) refers to a procedure that performs frequency-based morpheme extraction from a corpus that has already been segmented using a rule-based morphological analyzer, HornMorpho in our case.” §3.3: “limiting merge candidates to those that do not span morpheme boundaries defined by MoVoC.” §5.2: “we measure the alignment between BPE tokens and gold-standard morphemes using Morpheme boundary precision ... and MorphScore.”"

The same HornMorpho output is used twice in the pipeline. In §3.2 and Algorithm 1, HornMorpho segments the training corpora, producing Vmorpheme and the morpheme boundaries Mi. In §3.3, those boundaries are hard constraints: merges that cross Mi are forbidden during MoVoC-Tok training. Then §3.1 and §5.2 designate the annotated morphemes as the “gold-standard” test set and score MoVoC-Tok’s boundaries against them. For Amharic and Tigrinya, MoVoC-Tok is therefore trained to reproduce HornMorpho’s boundary decisions and then evaluated on how well it reproduces HornMorpho’s boundary decisions. The reported MorphScore and Boundary Precision for these two languages are a self-consistency check on the training signal, not an independent measure of morphological correctness. The comparison to BP

full rationale

The circularity is real but partial. The intrinsic evaluation for Amharic and Tigrinya reduces to measuring how faithfully MoVoC-Tok follows HornMorpho-derived training constraints against a HornMorpho-derived gold standard. However, for Ge’ez and Tigre, the annotations are manual, expert-created, used for testing only, and not part of vocabulary construction (§4.1); those intrinsic improvements are independent evidence and not circular. The extrinsic machine-translation evaluation uses FLORES-200 and OPUS references, which do not derive from HornMorpho, and the paper honestly reports a lack of significant MT gains, so that part is also not circular. The self-citation to Gidey et al. (2024) is background and not load-bearing. The inconsistent vocabulary-size reporting (Table 4 says 32k, Table 5 gives a 152k bilingual MoVoC, and §5.1 states a 63,050 token MT vocabulary) is a confound/correctness concern rather than a circularity step. Because the abstract’s central “across all four languages” intrinsic claim bundles two languages whose intrinsic results reduce by construction, the score is 6 rather than 0; the independent Ge’ez/Tigre intrinsic results and the independent MT evaluation prevent a higher score.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the accuracy of the morpheme annotations and on several unreported hyperparameters (r, vocabulary sizes). The intrinsic evaluation is not independent of the annotation source.

free parameters (3)

r (morpheme token ratio)
Hyperparameter controlling the fraction of morpheme tokens in the MoVoC vocabulary; value not reported in the paper.
Vocabulary sizes (s, smorpheme, sBPE) = BPE 32k, morpheme 80k, bilingual 152k; model 63k (inconsistent)
Vocabulary sizes are chosen by the authors and the paper gives inconsistent values for the final model.
Top-k morpheme frequency threshold
Morphemes selected by frequency; the exact k is derived from r and s, but not specified.

assumptions (3)

domain assumption HornMorpho provides correct morpheme segmentations for Amharic and Tigrinya
Used to build the vocabulary and gold test annotations; no independent verification provided.
domain assumption Manual morpheme annotations for Ge'ez and Tigre are consistent and correct
Created by the authors under linguistic supervision; no inter-annotator agreement reported.
domain assumption NLLB corpora are representative for training the tokenizers used in evaluation
Tokenizers trained on NLLB corpora but evaluated on OPUS sentences; distribution shift not discussed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of MoVoC: Morphology-Aware Subword Construction for Geez Script Languages." pith.science (2026). https://pith.science/paper/3SMDWHG2

@misc{pith2026250908812,
  author       = {Pith},
  title        = {Pith review of: MoVoC: Morphology-Aware Subword Construction for Geez Script Languages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3SMDWHG2}},
  note         = {Machine review of arXiv:2509.08812}
}

read the original abstract

Subword-based tokenization methods often fail to preserve morphological boundaries, a limitation especially pronounced in low-resource, morphologically complex languages such as those written in the Geez script. To address this, we present MoVoC (Morpheme-aware Subword Vocabulary Construction) and train MoVoC-Tok, a tokenizer that integrates supervised morphological analysis into the subword vocabulary. This hybrid segmentation approach combines morpheme-based and Byte Pair Encoding (BPE) tokens to preserve morphological integrity while maintaining lexical meaning. To tackle resource scarcity, we curate and release manually annotated morpheme data for four Geez script languages and a morpheme-aware vocabulary for two of them. While the proposed tokenization method does not lead to significant gains in automatic translation quality, we observe consistent improvements in intrinsic metrics, MorphoScore, and Boundary Precision, highlighting the value of morphology-aware segmentation in enhancing linguistic fidelity and token efficiency. Our morpheme-annotated datasets and tokenizer will be publicly available to support further research in low-resource, morphologically rich languages. Our code and data are available on GitHub: https://github.com/hailaykidu/MoVoC

Figures

Figures reproduced from arXiv: 2509.08812 by the authors.

**Figure 1.** MoVoC Pipeline. We first extract Amharic and Tigrinya words from our corresponding text corpora to perform token-based and morpheme-based separation resulting in four different vocabularies. We then merge all four vocabularies to generate a single MoVoC-based vocabulary (VMoVoC). by evidence from several languages, e.g., Korean (Lee et al., 2024), Arabic (Tawfik et al., 2019), Japanese (Bostrom and Durrett, 2020), … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

9 extracted references · 2 linked inside Pith

[5]

In Proceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing, pages 343–359, Singapore

CompoundPiece: Evaluating and improving decompounding performance of language models . In Proceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing, pages 343–359, Singapore. Association for Computational Linguistics. Javad Nouri and Roman Y angarber. 2016.A novel eval- uation method for morphological segmentation . In Proce...

2023
[8]

In Proceedings of the 29th International Conference on Computational Linguistics, pages 44–54, Gyeongju, Republic of Ko- rea

Measuring morphological fusion using par- tial information decomposition . In Proceedings of the 29th International Conference on Computational Linguistics, pages 44–54, Gyeongju, Republic of Ko- rea. International Committee on Computational Lin- guistics. Xinying Song, Alex Salcianu, Y ang Song, Dave Dop- son, and Denny Zhou. 2021. Fast WordPiece tok- en...

2021
[2016]

In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1715– 1725, Berlin, Germany

Neural machine translation of rare words with subword units . In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1715– 1725, Berlin, Germany. Association for Computa- tional Linguistics. Hyopil Shin and Hyunjo Y ou. 2009. Hybrid n-gram probability estimation in morphologically rich ...

2009
[2018]

likely”, “unlike

Marian: Cost-effective high-quality neural machine translation in C++ . In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 129–135, Melbourne, Australia. Association for Computational Linguistics. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tok- enizer and detokenizer fo...

2018
[2021]

Transactions of the Association for Computational Linguistics, 9:261–276

Morphology matters: A multilingual language modeling analysis. Transactions of the Association for Computational Linguistics, 9:261–276. Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation , pages 612–618, Copen- hagen, Denmark. Association for Computational Lin- guistics. Ricardo Rei...

2017
[2022]

Computational Linguistics, 48(3):673–732

Survey of low-resource machine translation . Computational Linguistics, 48(3):673–732. Valentin Hofmann, Hinrich Schuetze, and Janet Pierre- humbert. 2022. An embarrassingly simple method to mitigate undesirable properties of pretrained lan- guage model tokenizers . In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguisti...

arXiv 2022
[2023]

In Findings of the Association for Computational Linguistics: ACL 2023 , pages 5661–5681, Toronto, Canada

Tokenization impacts multilingual language modeling: Assessing vocabulary allocation and over- lap across languages. In Findings of the Association for Computational Linguistics: ACL 2023 , pages 5661–5681, Toronto, Canada. Association for Com- putational Linguistics. Beso Mikaberidze, Temo Saghinadze, Guram Mik- aberidze, Raphael Kalandadze, Konstantine ...

2023
[2024]

In Proceedings of the Fifth Workshop on Resources for African Indigenous Lan- guages @ LREC-COLING 2024 , pages 94–106, Torino, Italia

Morphological synthesizer for Ge‘ez lan- guage: Addressing morphological complexity and resource limitations . In Proceedings of the Fifth Workshop on Resources for African Indigenous Lan- guages @ LREC-COLING 2024 , pages 94–106, Torino, Italia. ELRA and ICCL. Dan Gillick, Cliff Brunk, Oriol Vinyals, and Amarnag Subramanya. 2016. Multilingual language pr...

arXiv 2024

Show all 9 references

[2218]

Thinh Truong, Yulia Otmakhova, Karin Verspoor, Trevor Cohn, and Timothy Baldwin

Citeseer. Thinh Truong, Yulia Otmakhova, Karin Verspoor, Trevor Cohn, and Timothy Baldwin. 2024. Revis- iting subword tokenization: A case study on aﬀixal negation in large language models . In Proceedings of the 2024 Conference of the North American Chap- ter of the Associati...

2024

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.

[1] [5]

In Proceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing, pages 343–359, Singapore

CompoundPiece: Evaluating and improving decompounding performance of language models . In Proceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing, pages 343–359, Singapore. Association for Computational Linguistics. Javad Nouri and Roman Y angarber. 2016.A novel eval- uation method for morphological segmentation . In Proce...

2023

[2] [8]

In Proceedings of the 29th International Conference on Computational Linguistics, pages 44–54, Gyeongju, Republic of Ko- rea

Measuring morphological fusion using par- tial information decomposition . In Proceedings of the 29th International Conference on Computational Linguistics, pages 44–54, Gyeongju, Republic of Ko- rea. International Committee on Computational Lin- guistics. Xinying Song, Alex Salcianu, Y ang Song, Dave Dop- son, and Denny Zhou. 2021. Fast WordPiece tok- en...

2021

[3] [2016]

In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1715– 1725, Berlin, Germany

Neural machine translation of rare words with subword units . In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1715– 1725, Berlin, Germany. Association for Computa- tional Linguistics. Hyopil Shin and Hyunjo Y ou. 2009. Hybrid n-gram probability estimation in morphologically rich ...

2009

[4] [2018]

likely”, “unlike

Marian: Cost-effective high-quality neural machine translation in C++ . In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 129–135, Melbourne, Australia. Association for Computational Linguistics. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tok- enizer and detokenizer fo...

2018

[5] [2021]

Transactions of the Association for Computational Linguistics, 9:261–276

Morphology matters: A multilingual language modeling analysis. Transactions of the Association for Computational Linguistics, 9:261–276. Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation , pages 612–618, Copen- hagen, Denmark. Association for Computational Lin- guistics. Ricardo Rei...

2017

[6] [2022]

Computational Linguistics, 48(3):673–732

Survey of low-resource machine translation . Computational Linguistics, 48(3):673–732. Valentin Hofmann, Hinrich Schuetze, and Janet Pierre- humbert. 2022. An embarrassingly simple method to mitigate undesirable properties of pretrained lan- guage model tokenizers . In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguisti...

arXiv 2022

[7] [2023]

In Findings of the Association for Computational Linguistics: ACL 2023 , pages 5661–5681, Toronto, Canada

Tokenization impacts multilingual language modeling: Assessing vocabulary allocation and over- lap across languages. In Findings of the Association for Computational Linguistics: ACL 2023 , pages 5661–5681, Toronto, Canada. Association for Com- putational Linguistics. Beso Mikaberidze, Temo Saghinadze, Guram Mik- aberidze, Raphael Kalandadze, Konstantine ...

2023

[8] [2024]

In Proceedings of the Fifth Workshop on Resources for African Indigenous Lan- guages @ LREC-COLING 2024 , pages 94–106, Torino, Italia

Morphological synthesizer for Ge‘ez lan- guage: Addressing morphological complexity and resource limitations . In Proceedings of the Fifth Workshop on Resources for African Indigenous Lan- guages @ LREC-COLING 2024 , pages 94–106, Torino, Italia. ELRA and ICCL. Dan Gillick, Cliff Brunk, Oriol Vinyals, and Amarnag Subramanya. 2016. Multilingual language pr...

arXiv 2024