Pith. sign in

REVIEW 4 major objections 4 minor 46 references

Beyond Literal Token Overlap: Token Alignability for Multilinguality

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that subword token alignability predicts cross-lingual transfer better than literal token overlap.

desk verdict Useful new metric for tokenizer evaluation, but the headline claim of superiority over JSD isn't yet backed by significance testing. read the letter →

arxiv 2502.06468 v1 pith:RMLUVRQQ submitted 2025-02-10 cs.CL

Katharina Hämmerl , Tomasz Limisiewicz , Jindřich Libovický , Alexander Fraser This is my paper · ORCID
classification cs.CL
keywords subwordtokenizationtokenalignabilitycross-lingualtransfermultilinguallanguagemodelswordalignmenteflomalJensen-Shannondivergencescriptdifference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multilingual models can transfer knowledge between languages even when the two languages share almost no literal token strings, and the paper argues that a new predictor explains this: subword token alignability. Instead of counting token overlap or comparing token distributions, it measures how well the subword tokens of one language can be statistically aligned to subword tokens of another as translation equivalents. The paper claims this eflomal-based score predicts downstream cross-lingual transfer better than the previous distributional-overlap metric, especially for language pairs with different scripts where literal overlap is near zero. If the claim holds, token alignability offers a principled way to choose language pairs for cross-lingual transfer and to guide the construction of fairer multilingual tokenisers.

What carries the argument

The central object is subword token alignability, computed as the eflomal score from the Bayesian word aligner eflomal. Given a parallel corpus tokenised by the multilingual tokeniser under study, eflomal estimates statistical links between subword tokens; the score is the maximum unnormalized log-probability of links in the last sampling iteration, averaged over both directions of the language pair. The mechanism is that this score captures regular statistical correspondences between token strings—including cross-script regularities—that literal token matching and distributional overlap miss. The paper contrasts it with the one-to-one alignment proportion, which by itself shows weak or no predictive correlation.

What would settle it

Recompute both metrics on the model's own pre-training corpus (CC-100) and compare Spearman correlations over the same language pairs: if JSD then matches or beats the eflomal score on POS, UD, or NER transfer, the claimed advantage for different-script pairs would be contradicted.

Watch

Extended reading notes

Core claim

The paper's central claim is that token alignability—how well subword tokens of one language can be aligned to subword tokens of another as translation equivalents—predicts cross-lingual transfer better than distributional token overlap. Operationally, alignability is the eflomal score, the tool's estimate of the maximum unnormalized log-probability of alignment links in its final sampling iteration, averaged over both directions of the language pair. Across three encoder tokeniser types (BPE, Unigram, TokMix), this score shows stronger Spearman correlations with POS, UD, and NER transfer than JSD does, and the advantage is clearest for language pairs with different scripts, where literal token overlap is near zero. The paper also reports that alignability correlates with cross-lingual embedding alignment, while the simpler proportion of one-to-one alignments shows weak or no predictive correlation. For decoder models the result is mixed: eflomal wins for one decoder while JSD wins for two, suggesting decoder cross-linguality may rely more on literal token matches.

Load-bearing premise

The result stands or falls on the assumption that eflomal alignment scores computed on FLORES-200 subword tokens, using priors trained on OPUS-100 and MultiCCAligned, faithfully capture the alignability of tokenisers in models trained on a different corpus, CC-100, for a small non-random set of language pairs.

Editorial extensions

If this is right

  • Eflomal score can serve as a tokenizer-level predictor for selecting language pairs in cross-lingual transfer experiments with encoder models, particularly when the pair uses different scripts.
  • Language pairs that share little literal token overlap can still be highly transferable if their subword tokens align well, which explains cases where related languages use different scripts.
  • The proportion of one-to-one subword alignments is not enough; the log-probability score carries the predictive signal.
  • Pre-training data size correlates with transfer from English for XNLI, but does not explain the full pattern, so alignability adds information beyond data size.
  • For decoder models, the paper's recommendation is provisional: alignability does not uniformly beat JSD across the three decoders tested, so tokenizer-level metrics alone do not fully determine decoder cross-linguality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A reader could test whether the eflomal score also works as a filtering signal during BPE merge selection or Unigram pruning, approximating the cost for batches of candidate tokens instead of checking every decision point.
  • The paper's logic implies that alignability should generalise beyond FLORES; a direct check is whether scores computed on a held-out parallel corpus still rank transfer performance for unseen language pairs.
  • For decoder-heavy applications, the mixed results suggest a combined predictor that uses literal overlap when scripts match and alignability when they do not, which the paper does not itself propose.
  • Because the metric is corpus-wide, reformulating it for word- or sentence-level tasks would be needed before it could guide dynamic tokenization during inference.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a new tokenizer-level metric, subword token alignability, computed from eflomal word alignments, and argues that it predicts cross-lingual transfer and representation alignment better than the existing distributional token-overlap metric (JSD), especially for language pairs with different scripts. The evidence is correlational: Spearman rank correlations between the metrics and downstream transfer scores for XNLI, POS, UD, and NER across three encoder tokenizers, plus retrieval-based alignment measures for one encoder and three decoder models. The authors also examine pre-training data size as a confounder and discuss implications for multilingual tokenizer construction. The manuscript is candid about scope limitations and releases code.

Significance. If the central claim is accepted, the paper contributes a useful predictor for choosing language pairs for cross-lingual transfer and for diagnosing multilingual tokenizers, particularly in the common setting where literal token overlap is low because scripts differ. The paper's strengths are that it reuses existing trained models and downstream transfer results, covers multiple tasks and tokenization algorithms, and provides code. The downstream transfer evidence is independent of the proposed metric, since the models and transfer results come from earlier work. The main results for POS and UD in encoder models are suggestive and align with the paper's narrative. However, the statistical evidence for a general advantage of eflomal over JSD is incomplete, and several results, especially for decoder models and NER, point in the opposite direction.

major comments (4)
  1. [Section 4.1, Tables 1 and 2] The central claim that eflomal is better than JSD rests on comparing Spearman's rho values without confidence intervals, p-values, or a test of the difference between dependent correlations. The language-pair observations are not independent because pairs share source or target languages (e.g., en-fr and en-de both contain English), which violates the independence assumption of ordinary Spearman inference and can yield overconfident readings. The set of pairs is also not a random sample of language pairs; it is drawn from 20 languages with heavy Indo-European representation. The manuscript should report cluster-bootstrap or permutation confidence intervals, or otherwise quantify whether the observed differences, such as POS rho -0.64 vs -0.45 for Unigram, are distinguishable from noise.
  2. [Section 3.4 and Section 4.1] The embedding-alignment evidence in Table 2 is partially circular. The tokenizer metrics are computed on FLORES-200, and the retrieval/margin evaluations in Section 3.4 are also computed on FLORES-200. The authors explicitly state that the strong FLORES correlations may be an upper bound because the same corpus was used, but this admission is not reflected in the strength of the conclusions drawn from Table 2. The paper should either provide out-of-corpus evidence for representation alignment, for example using Tatoeba alone or a held-out corpus for both metrics and alignment, or clearly restrict the embedding-alignment claim to an upper-bound analysis.
  3. [Section 4.2 and Table 6] The claim that data size cannot account for the whole effect is not supported by the analysis presented. Table 3 shows that target-language training size correlates with transfer, and Table 6 shows that both JSD and eflomal correlate with training size (e.g., eflomal rho -0.54 for BPE and JSD rho -0.40). The manuscript does not report partial correlations or a regression that controls for data size when comparing eflomal and JSD against transfer. Without such a control, the observed advantage of eflomal may be partly or wholly attributable to the confound that the authors themselves identify. A partial-correlation analysis or a matched comparison within narrow data-size bands is needed.
  4. [Section 4.3, Table 4] The decoder results substantially qualify the abstract's claim that eflomal predicts multilinguality 'much better.' In Table 4, JSD correlates more strongly than eflomal with FLORES F1 for Aya23 (-0.68 vs -0.49) and for Llama3 (-0.59 vs -0.32), and the same is true for average margin in Aya23. Only Mistral shows the opposite pattern. The manuscript should either limit the central claim to encoder models, provide a concrete explanation for the discrepancy, or weaken the generalization in the abstract and conclusion. As written, the paper recommends eflomal for identifying optimal language pairs without acknowledging that the metric does not appear to hold for two of the three decoder models tested.
minor comments (4)
  1. [Section 3.2] There is a typo: 'Seee Appendix A' should read 'See Appendix A'.
  2. [Tables 1, 2, and 4] The table headings use '=' and '̸=' without defining them in the captions; please add explicit definitions such as 'same script' and 'different script' to each table caption.
  3. [Caption of Figure 5] 'T atoeba' should be 'Tatoeba'.
  4. [Appendix C.1] The figures report p-values in the bottom-left corners, but the main tables do not; since the statistical significance of the correlations is central, consider reporting p-values or confidence intervals in the tables as well.

Circularity Check

1 steps flagged · score 4.0 of 10

Downstream transfer claims are independent, but the FLORES-based embedding-alignment evidence (Table 2) is circular by the paper's own admission.

  1. other [Section 4.1, Table 2; also Section 3.2 and Section 3.4]
    "All the correlations are much stronger on the FLORES dataset, likely because this dataset was used to calculate the tokeniser metrics in the first place. We can therefore see these as a kind of upper bound on how well the tokeniser metrics can predict cross-lingual alignment."

    The tokenizer metrics are computed on FLORES-200 (Section 3.2: 'As our test corpus, we use FLORES-200'), and the embedding-alignment retrieval/margin scores in Table 2 are also computed on FLORES-200 (Section 3.4: 'retrieval accuracy on the Tatoeba dataset as well as the FLORES-200 development set ... average margin distances on the latter'). The Table 2 correlations are therefore partly self-correlations from sharing the same source corpus, not independent confirmation of predictive power. The paper itself acknowledges this by labeling the correlations an 'upper bound.' This circularity is limited to the secondary embedding-alignment analysis; the main downstream-transfer comparison (Table 1) uses separately trained models and fixed transfer results from Limisiewicz et al.

full rationale

The primary claim that eflomal token alignability outperforms JSD for predicting downstream cross-lingual transfer is supported by correlations against an external, pre-existing set of models and transfer results (Limisiewicz et al., 2023), which are not fitted to the eflomal score in this paper. That part of the derivation chain is self-contained and not circular. The only genuinely circular element is the embedding-alignment analysis in Section 4.1/Table 2, where both the predictor metrics and the predicted retrieval/margin values come from FLORES-200; the paper openly states this makes the correlations an 'upper bound.' Because the authors disclose this and the main downstream result does not reduce to it, the paper does not rise to the level of forced circularity. Statistical concerns about missing confidence intervals and non-independent language pairs are robustness issues, not circularity, and are not scored here.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new free parameters or invented entities. The eflomal score depends on the aligner's learned priors, which are fitted to external parallel data, not to the downstream transfer results. The main untested assumptions are corpus generalization and the reliability of eflomal alignments on subword-tokenized text.

assumptions (3)
  • domain assumption eflomal produces reliable subword-level alignment links for tokenized multilingual text.
    The core metric is the eflomal score from a statistical word aligner run on FLORES-200, as described in Section 3.2. If these alignments are noisy or systematically biased for certain scripts, the metric inherits that bias.
  • domain assumption Alignability computed with priors from OPUS-100 and MultiCCAligned generalizes to tokenizers and models trained on CC-100.
    Section 3.2 explains that priors are trained on OPUS-100 for en-xx pairs and MultiCCAligned for others, while the encoder models from Limisiewicz et al. (2023) were trained on a CC-100 subset. The paper does not test whether this corpus mismatch affects the metric.
  • standard math Spearman rank correlation over available language pairs is an appropriate measure of predictive power.
    The paper consistently uses Spearman correlation, following Limisiewicz et al. (2023), but does not report confidence intervals or account for the non-independence of language pairs sharing a language.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Literal Token Overlap: Token Alignability for Multilinguality." pith.science (2026). https://pith.science/paper/RMLUVRQQ

@misc{pith2026250206468,
  author       = {Pith},
  title        = {Pith review of: Beyond Literal Token Overlap: Token Alignability for Multilinguality},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RMLUVRQQ}},
  note         = {Machine review of arXiv:2502.06468}
}
read the original abstract

Previous work has considered token overlap, or even similarity of token distributions, as predictors for multilinguality and cross-lingual knowledge transfer in language models. However, these very literal metrics assign large distances to language pairs with different scripts, which can nevertheless show good cross-linguality. This limits the explanatory strength of token overlap for knowledge transfer between language pairs that use distinct scripts or follow different orthographic conventions. In this paper, we propose subword token alignability as a new way to understand the impact and quality of multilingual tokenisation. In particular, this metric predicts multilinguality much better when scripts are disparate and the overlap of literal tokens is low. We analyse this metric in the context of both encoder and decoder models, look at data size as a potential distractor, and discuss how this insight may be applied to multilingual tokenisation in future work. We recommend our subword token alignability metric for identifying optimal language pairs for cross-lingual transfer, as well as to guide the construction of better multilingual tokenisers in the future. We publish our code and reproducibility details.

Figures

Figures reproduced from arXiv: 2502.06468 by the authors.

Figure 1
Figure 1. Eflomal score (bottom), a measure of token [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Unigram model: The eflomal score generally correlates better with downstream transfer than JSD. NER is [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. BPE model: The eflomal score correlates better with downstream transfer than JSD, with the exception [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: TokMix model: The eflomal score correlates better with downstream transfer than JSD, again with [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: BPE Model: Eflomal scores correlates well with cross-lingual embedding alignment. Nevertheless, both [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Aya23: Spearman’s rank correlation of cross [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 8
Figure 8. Figure 8: Mistral: Spearman’s rank correlation of cross [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 7
Figure 7. Figure 7: Llama3: Spearman’s rank correlation of cross [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 8 canonical work pages

  1. [1]

    Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Valentin Hoffman, Tomasz Limisiewicz, Yulia Tsvetkov, and Noah A. Smith. 2024. https://arxiv.org/abs/2407.08818 Magnet: Improving the multilingual fairness of language models with adaptive gradient-based tokenization . preprint, arXiv:2407.08818 [cs.CL]

    arXiv 2024

  2. [2]

    Mohamed Alkaoud and Mairaj Syed. 2020. https://aclanthology.org/2020.wanlp-1.11 On the importance of tokenization in A rabic embedding models . In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 119--129, Barcelona, Spain (Online). Association for Computational Linguistics

    work page 2020

  3. [3]

    Mikel Artetxe and Holger Schwenk. 2019. https://doi.org/10.1162/tacl_a_00288 Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond . Transactions of the Association for Computational Linguistics, 7:597--610

    doi:10.1162/tacl_a_00288 2019

  4. [4]

    Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015

    work page 2015

  5. [5]

    Khuyagbaatar Batsuren, Ekaterina Vylomova, Verna Dankers, Tsetsuukhei Delgerbaatar, Omri Uzan, Yuval Pinter, and Gábor Bella. 2024. https://arxiv.org/abs/2404.13292 Evaluating subword tokenization: Alien subword composition and oov generalization challenge . preprint, arXiv:2404.13292 [cs.CL]

    work page Pith review arXiv 2024

  6. [6]

    Peter Brown, Stephen Della Pietra , Vincent Della Pietra , and Robert Mercer. 1993. http://acl.ldc.upenn.edu/J/J93/J93-2003.pdf The mathematics of statistical machine translation : parameter estimation . Computational Linguistics, 19(2):263--311

    work page 1993

  7. [7]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://doi.org/10.18653/v1/2020.acl-main.747 Unsupervised cross-lingual representation learning at scale . In Proceedings of the 58th Annual Meeting of the Association for Comp...

    doi:10.18653/v1/2020.acl-main.747 2020

  8. [8]

    Alexis Conneau and Guillaume Lample. 2019. https://proceedings.neurips.cc/paper_files/paper/2019/file/c04c19c2c2474dbf5f7ac4372c5b9af1-Paper.pdf Cross-lingual language model pretraining . In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc

    work page 2019

Show all 46 references
  1. [9]

    Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. https://doi.org/10.18653/v1/D18-1269 XNLI : Evaluating cross-lingual sentence representations . In Proceedings of the 2018 Conference on Empirical Methods ...

    2018 doi

  2. [10]

    Ameet Deshpande, Partha Talukdar, and Karthik Narasimhan. 2022. https://doi.org/10.18653/v1/2022.naacl-main.264 When is BERT multilingual? isolating crucial ingredients for cross-lingual transfer . In Proceedings of the 2022 Conference of the North American Chapter of the Asso...

    2022 doi

  3. [11]

    Philipp Dufter and Hinrich Sch \"u tze. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.358 Identifying elements essential for BERT ' s multilinguality . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4423--4437, Onli...

    2020 doi

  4. [12]

    Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzm \'a n, and Philipp Koehn. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.480 CCA ligned: A massive collection of cross-lingual web-document pairs . In Proceedings of the 2020 Conference on Empirical Methods in Natural Langu...

    2020 doi

  5. [13]

    Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc ' Aurelio Ranzato, Francisco Guzm \'a n, and Angela Fan. 2022. https://doi.org/10.1162/tacl_a_00474 The F lores-101 evaluation benchmark for low-resource and multilingua...

    2022 doi

  6. [14]

    Francisco Guzm \'a n, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc ' Aurelio Ranzato. 2019. https://doi.org/10.18653/v1/D19-1632 The FLORES evaluation datasets for low-resource machine translation: N epali -- E nglish and S i...

    2019 doi

  7. [15]

    Katharina H \"a mmerl, Jind r ich Libovick \'y , and Alexander Fraser. 2024. https://doi.org/10.18653/v1/2024.findings-acl.649 Understanding cross-lingual A lignment --- A survey . In Findings of the Association for Computational Linguistics ACL 2024, pages 10922--10943, Bangk...

    2024 doi

  8. [16]

    Valentin Hofmann, Hinrich Schuetze, and Janet Pierrehumbert. 2022. https://doi.org/10.18653/v1/2022.acl-short.43 An embarrassingly simple method to mitigate undesirable properties of pretrained language model tokenizers . In Proceedings of the 60th Annual Meeting of the Associ...

    2022 doi

  9. [17]

    Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. https://arxiv.org/abs/2003.11080 Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization . preprint, arXiv:2003.11080 [cs.CL]

    2020 arXiv

  10. [18]

    Tianze Hua, Tian Yun, and Ellie Pavlick. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.103 m O thello: When do cross-lingual representation alignment and cross-lingual transfer emerge in multilingual models? In Findings of the Association for Computational Linguistics:...

    2024 doi

  11. [19]

    Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. 2019. https://doi.org/10.18653/v1/D19-1252 U nicoder: A universal language encoder by pre-training with multiple cross-lingual tasks . In Proceedings of the 2019 Conference on Empirical M...

    2019 doi

  12. [20]

    Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, and Fuzhen Zhuang. 2023. https://arxiv.org/abs/2307.16645 Scaling sentence embeddings with large language models . preprint, arXiv:2307.16645 [cs.CL]

    2023 arXiv

  13. [21]

    Alexander Jones, William Yang Wang, and Kyle Mahowald. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.471 A massively multilingual analysis of cross-linguality in shared embedding space . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process...

    2021 doi

  14. [22]

    Taku Kudo. 2018. https://doi.org/10.18653/v1/P18-1007 Subword regularization: Improving neural network translation models with multiple subword candidates . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page...

    2018 doi

  15. [23]

    Jind r ich Libovick \'y , Rudolf Rosa, and Alexander Fraser. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.150 On the language neutrality of pre-trained multilingual representations . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1663-...

    2020 doi

  16. [24]

    Tomasz Limisiewicz, Ji r \' Balhar, and David Mare c ek. 2023. https://doi.org/10.18653/v1/2023.findings-acl.350 Tokenization impacts multilingual language modeling: Assessing vocabulary allocation and overlap across languages . In Findings of the Association for Computational...

    2023 doi

  17. [25]

    Tomasz Limisiewicz, Terra Blevins, Hila Gonen, Orevaoghene Ahia, and Luke Zettlemoyer. 2024. https://doi.org/10.18653/v1/2024.acl-long.804 MYTE : Morphology-driven byte encoding for better and fairer multilingual language modeling . In Proceedings of the 62nd Annual Meeting of...

    2024 doi

  18. [26]

    J. Lin. 2006. https://doi.org/10.1109/18.61115 Divergence measures based on the shannon entropy . IEEE Trans. Inf. Theor., 37(1):145–151

    2006 doi

  19. [27]

    Md Mofijul Islam, Gustavo Aguilar, Pragaash Ponnusamy, Clint Solomon Mathialagan, Chengyuan Ma, and Chenlei Guo. 2022. https://doi.org/10.18653/v1/2022.repl4nlp-1.10 A vocabulary-free multilingual neural tokenizer for end-to-end task learning . In Proceedings of the 7th Worksh...

    2022 doi

  20. [28]

    Benjamin Muller, Yanai Elazar, Beno \^ t Sagot, and Djam \'e Seddah. 2021. https://doi.org/10.18653/v1/2021.eacl-main.189 First align, then predict: Understanding the cross-lingual ability of multilingual BERT . In Proceedings of the 16th Conference of the European Chapter of ...

    2021 doi

  21. [29]

    O stling and J \

    Robert \"O stling and J \"o rg Tiedemann. 2016. https://doi.org/10.1515/pralin-2016-0013 Efficient word alignment with markov chain monte carlo . The Prague Bulletin of Mathematical Linguistics, 106:125 -- 146

    2016 doi

  22. [30]

    Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. https://doi.org/10.18653/v1/P17-1178 Cross-lingual name tagging and linking for 282 languages . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (...

    2017 doi

  23. [31]

    Trinh Pham, Khoi Le, and Anh Tuan Luu. 2024. https://doi.org/10.18653/v1/2024.acl-long.174 U ni B ridge: A unified approach to cross-lingual transfer learning for low-resource languages . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistic...

    2024 doi

  24. [32]

    Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, and Chris Tanner

    Craig W. Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, and Chris Tanner. 2024. https://arxiv.org/abs/2402.18376 Tokenization is more than compression . preprint, arXiv:2402.18376 [cs.CL]

    2024 arXiv

  25. [33]

    Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2019. https://doi.org/10.18653/v1/N19-1380 Cross-lingual transfer learning for multilingual task oriented dialog . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computat...

    2019 doi

  26. [34]

    Anton Schäfer, Shauli Ravfogel, Thomas Hofmann, Tiago Pimentel, and Imanol Schlag. 2024. https://arxiv.org/abs/2404.07982 The role of language imbalance in cross-lingual generalisation: Insights from cloned language experiments . preprint, arXiv:2404.07982 [cs.CL]

    2024 arXiv

  27. [35]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. https://doi.org/10.18653/v1/P16-1162 Neural machine translation of rare words with subword units . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages ...

    2016 doi

  28. [36]

    NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prang...

    2022 arXiv

  29. [37]

    J \"o rg Tiedemann. 2012. http://www.lrec-conf.org/proceedings/lrec2012/pdf/463_Paper.pdf Parallel data, tools and interfaces in OPUS . In Proceedings of the Eighth International Conference on Language Resources and Evaluation ( LREC '12) , pages 2214--2218, Istanbul, Turkey. ...

    2012

  30. [38]

    Ra \'u l V \'a zquez, Umut Sulubacak, and J \"o rg Tiedemann. 2019. https://doi.org/10.18653/v1/W19-5441 The U niversity of H elsinki submission to the WMT 19 parallel corpus filtering task . In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task...

    2019 doi

  31. [39]

    Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. https://doi.org/10.18653/v1/2024.acl-long.820 Do llamas work in E nglish? on the latent language of multilingual transformers . In Proceedings of the 62nd Annual Meeting of the Association for Computati...

    2024 doi

  32. [40]

    Shijie Wu and Mark Dredze. 2019. https://doi.org/10.18653/v1/D19-1077 Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference o...

    2019 doi

  33. [41]

    Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. https://doi.org/10.1162/tacl_a_00461 B y T 5: Towards a token-free future with pre-trained byte-to-byte models . Transactions of the Association for Computa...

    2022 doi

  34. [42]

    Daniel Zeman, Joakim Nivre, et al. 2019. http://hdl.handle.net/11234/1-3105 Universal dependencies 2.5 . LINDAT / CLARIAH - CZ digital library at the Institute of Formal and Applied Linguistics ( \'U FAL ), Faculty of Mathematics and Physics, Charles University

    2019

  35. [43]

    Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020. https://doi.org/10.18653/v1/2020.acl-main.148 Improving massively multilingual neural machine translation and zero-shot translation . In Proceedings of the 58th Annual Meeting of the Association for Computationa...

    2020 doi

  36. [44]

    Vil \'e m Zouhar, Clara Meister, Juan Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell. 2023. https://doi.org/10.18653/v1/2023.acl-long.284 Tokenization and the noiseless channel . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (...

    2023 doi

  37. [45]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  38. [46]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.