Pith. sign in

REVIEW 5 major objections 5 minor 32 references

Conditional Unigram Tokenization with Parallel Data

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Conditioning a subword tokenizer on aligned source tokens lowers language-modeling perplexity on every tested language pair while leaving machine translation flat, a gap the paper traces to the quadratic co-occurrence table.

desk verdict New conditional unigram tokenization with honest negative MT results, but the small LM perplexity gains need error bars before we believe them. read the letter →

arxiv 2507.07824 v1 pith:UUEN4VCJ submitted 2025-07-10 cs.CL

Gianluca Vico , Jindřinch Libovický This is my paper
classification cs.CL MSC 68T50
keywords conditionaltokenizationunigramtokenizercross-lingualalignmentparalleldatasubwordsegmentationmachinetranslationlanguagemodelinglow-resourcelanguages
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a target-language tokenizer can be learned so that its tokens line up semantically with the tokens of a fixed source-language tokenizer, using parallel sentences as the only cross-lingual signal. The method replaces the unigram model's unconditional token probabilities with probabilities conditioned on a bag of aligned source tokens, then prunes the vocabulary by mutual information with those source tokens. On four language pairs spanning different families and resource levels, the conditional tokenizer keeps the statistical profile of a standard unigram tokenizer but does not improve machine translation quality; its consistent gain is a lower perplexity per byte in language modeling. The paper attributes the shortfall to a data-efficiency bottleneck: the co-occurrence table behind the conditional probabilities grows quadratically in vocabulary size while the training data stays fixed. If that diagnosis is right, the approach needs a more compact parameterization before cross-lingual tokenization of this kind can pay off for translation.

What carries the argument

The load-bearing object is the co-occurrence table $c(t, s)$ that counts alignments between target substrings $t$ and source tokens $s$ in word-aligned parallel data, together with the bag-of-tokens approximation of $p(t | S)$ in Equation (2), which sums co-occurrence counts over the source tokens of a sentence and normalizes over the target vocabulary. This table feeds the unigram model's dynamic-programming segmentation, and its marginals define the mutual-information criterion $I(t, V_{\mathrm{src}})$ that decides which subwords survive as the vocabulary is pruned. A second machinery element is the alignment pipeline: word alignments from Eflomal convert full parallel sentences into pairs of aligned words that serve as training examples, and the same aligner later produces the token alignments from which the one-to-one and unaligned metrics are computed.

What would settle it

Train the conditional tokenizer on a corpus large enough to saturate the co-occurrence table — the paper's own estimate puts the need at roughly 28M examples — and check whether parity and fertility then match or beat the unigram baseline; if they do not, quadratic data scaling is not the true bottleneck. A second decisive check is to recompute the one-to-one and unaligned metrics with a word aligner other than Eflomal and see whether the intrinsic advantages survive the change.

Watch

Extended reading notes

Core claim

The paper's central claim is that semantic alignment between tokenizers can be optimized directly during vocabulary learning by maximizing the unigram-style probability of target tokens conditioned on source tokens from a parallel sentence. Concretely, the conditional probability $p(t | S)$ is approximated from a co-occurrence count table $c(t, s)$ built from word-aligned parallel sentences, and the vocabulary is iteratively pruned using the mutual information $I(t, V_{\mathrm{src}})$ between target and source tokens. Tested on French-Italian, Czech-Ukrainian, Italian-Maltese, and German-Upper Sorbian, the method yields tokenizers with parity and fertility close to unigram baselines and fewer unaligned source tokens in some settings, yet it does not improve chrF++, BLEU, or COMET scores in machine translation. In contrast, the marginalization variant PairedSPM lowers perplexity per byte relative to the baseline on every language pair and every vocabulary size, in both monolingual and bilingual language-modeling settings, and the paper explains the flat translation results by quadratic scaling: the table for $p(t | S)$ has $|V_{\mathrm{tgt}}| \times |V_{\mathrm{src}}|$ entries, and roughly 28M parallel examples would be needed to match unigram fertility, with 4M needed for comparable one-to-one alignment.

Load-bearing premise

The whole method rests on the assumption that counting how often a target substring and a source token co-occur in word-aligned parallel sentences is a faithful proxy for translation equivalence, even though the approximation discards word order and the same alignment tool both produces the training signal and scores the resulting alignment.

Editorial extensions

If this is right

  • Because PairedSPM needs no source text at inference, its perplexity gains are available to purely monolingual pipelines that happen to have parallel data once, at tokenizer-training time.
  • The uniform machine-translation null result across four language pairs indicates that improved token correspondence, as measured here, does not translate into translation quality gains.
  • The paper's estimate of roughly 28M examples to match unigram fertility and 4M to match one-to-one alignment implies that available corpora are one to two orders of magnitude too small, making the quadratic-scaling hypothesis directly testable by scaling up data.
  • Bilingual language-model training amplifies the PairedSPM advantage on low-resource targets, so the conditioning signal and the training-data language mix interact.
  • The absence of correlation between intrinsic tokenization quality and perplexity (the worst intrinsic configuration gives the best perplexity) suggests the language-modeling gain is not a side effect of better fertility or alignment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the quadratic-scaling diagnosis is right, a low-rank or factored approximation of $c(t, s)$ — for example, embedding source and target tokens in a shared low-dimensional space and scoring pairs by an inner product — should recover most of the alignment signal at a fraction of the data; the 8k-vocabulary setting, where the conditioning signal is densest, is the natural place to look for the larg
  • The perplexity gain survives even in monolingual language-model training, which suggests the source-conditioned token distribution acts as a regularizer on the target token stream; a control experiment conditioning on a random, unrelated source language would show whether the gain comes from genuine translation equivalence or merely from smoothing.
  • Because the same aligner both generates the training pairs and scores the alignment metrics, the intrinsic advantages (fewer unaligned tokens) may partly reflect that aligner's own bias; recomputing the one-to-one and unaligned metrics with an independent word aligner would separate tokenizer-induced alignment from aligner artifacts.
  • The better stability of the expectation-maximization variant PairedSPEM, which never counts tokens absent from the first iteration, hints that count-update dynamics matter as much as the conditioning signal, so a reweighted target-only unigram tokenizer is a stronger baseline than the paper's SentencePiece comparison.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces conditional unigram tokenization, which extends unigram tokenization by conditioning target-token probabilities on source-language tokens from parallel data. Given a fixed source tokenizer, the method learns a target tokenizer that is intended to maximize cross-lingual semantic alignment, using a co-occurrence table between target subwords and source tokens. The authors evaluate on four language pairs with varying resources and scripts, comparing PairedSP and variants against SentencePiece unigram baselines on intrinsic tokenization metrics, machine translation, and language modeling. The main reported findings are that the conditional tokenizer does not improve machine translation quality, but achieves lower perplexity per byte in language modeling across most language pairs and vocabulary sizes. The authors hypothesize that the quadratic scaling of the conditional probability table with vocabulary size creates a data-efficiency bottleneck.

Significance. The paper addresses a relevant and underexplored problem: designing tokenizers that are semantically aligned across languages. It ships open-source code, covers four language pairs with different resource levels and scripts, and reports negative results honestly, which is a strength. If the language-modeling perplexity improvements were statistically robust, the finding would be a modest but useful step toward cross-lingually aware tokenization. However, the central positive claim currently rests on single runs with no uncertainty quantification, the alignment evaluation is partly circular, and a key training equation is underspecified. The contribution is therefore promising but not yet established.

major comments (5)
  1. [3, Eq. (3)] The sample update formula is independent of the specific source token s: csample(Ti:j, s) = p(Ti:j | S) p(T:i | S) p(Tj: | S) / length(S) contains no term that distinguishes one source token s from another, so every source token in S receives the same contribution. This contradicts the surrounding text, which says the amount is 'distributed across the source tokens.' Please clarify whether a factor involving p(s | Ti:j) or p(t, s) is missing, or whether the intended update is a uniform redistribution over source tokens; as written, the learning signal cannot favor particular target-source token pairs, which is the core mechanism of the method.
  2. [5.3, Table 6 and Figure 4] The central positive claim of the paper is that PairedSPM achieves consistent perplexity-per-byte reductions, but no variance, confidence intervals, number of seeds, or significance tests are reported. The reported gaps are extremely small (e.g., ces→ukr 8k: 1.009 vs 1.014; deu→hsb 32k monolingual: 1.085 vs 1.085, which is a tie). Without repeated runs or paired tests, 'consistent' improvements of this magnitude can easily arise from training noise, evaluation artifacts, or tokenizer-dependent text normalization. Please add multiple seeds with error bars, or at least bootstrap confidence intervals and a paired significance test across the evaluation sentences.
  3. [4.1] The intrinsic alignment evaluation is partly circular: Eflomal is used both to generate the word-level aligned training pairs in Section 3 and to compute the one-to-one and unaligned metrics on the test set in Section 4.1. Any systematic bias of Eflomal is shared between the training signal and the evaluation, so the observed alignment improvements may reflect properties of the aligner rather than of the tokenizer. Please evaluate with an independent alignment method (e.g., Awesome Align) or report agreement between multiple aligners as a robustness check.
  4. [5.3] The paper itself notes that the language-modeling improvement does not correlate with tokenization scores ('the PairedSPM model with the worst intrinsic evaluation scores achieves the lowest perplexity in language modeling'), referring to the degenerate single-character tokenizer at fra→ita 8k with fertility 5.61 and parity 3.95. This observation undermines the proposed mechanism that cross-lingual semantic alignment drives the perplexity gain. Please test the mechanism directly, for example by comparing PairedSPM against a control tokenizer matched for fertility or by ablating the conditioning signal (e.g., using the same vocabulary but unrelated parallel data).
  5. [6] The conclusions state that approximately 28M examples would be required to match unigram fertility and 4M examples for comparable one-to-one alignment performance, but no derivation, extrapolation method, or supporting table is provided. These estimates appear to be load-bearing for the paper's practical-applicability discussion. Please either show the data and procedure used to obtain them or remove the quantitative claims.
minor comments (5)
  1. [Appendix H, Tables 11 and 15] There are typos in the table headers: 'Fertiliy' in Table 11 and 'Lnegth ratio' in Table 15 should be 'Fertility' and 'Length ratio'.
  2. [5.3, Table 6] The claim that PairedSPM improves perplexity 'across all language pairs and vocabulary sizes' is not literally true: the (deu →) hsb 32k monolingual cell is a tie (1.085 vs 1.085). Please qualify the claim or note the tie.
  3. [4.1, text and Table 1] The notation 'PairedSP M' appears with a space in Table 1 and in a few places in the text, while 'PairedSPM' is used elsewhere; please unify the notation.
  4. [Appendix E, Algorithm 1] In Algorithm 1, the loop header 'for all (T ) ∈ (src, trg)' is incomplete or inconsistent with the main text's (S, T) notation, and the function COUNT uses 'src' in the denominator while the main text uses 'length(S)' in Eq. (3). Please align the pseudocode with the equations.
  5. [3, Eq. (6)] The marginalization p(t) = sum_s p(t, s) is written without first defining p(t, s) as a normalized quantity; the text immediately re-expresses it in terms of counts, but the notation should be made explicit to avoid confusion about the normalizer.

Circularity Check

1 steps flagged · score 4.0 of 10

Alignment evidence is partly circular because Eflomal both creates the training pairs and measures the alignment metrics; the LM and MT results remain independent.

  1. self definitional [Section 3 (training data) and Section 4.1 (intrinsic alignment metrics)]
    "To reduce the memory requirements and speed up the training, we pretokenize the input sentences and use Eflomal (Östling & Tiedemann, 2016) to align the words. Then, each pair of aligned words is used as a training example instead of the full sentences. ... For alignment quality assessment, we first get the token alignment on the test data using Eflomal and we compare PairedSP and SPtgt using: One-to-one ... Unaligned ..."

    The target tokenizer is trained on Eflomal word alignments, so its vocabulary and co-occurrence table are optimized to fit Eflomal's link preferences. The alignment evaluation then reuses the same aligner to compute the one-to-one and unaligned metrics. Consequently, the reported alignment improvements ("PairedSP shows improvement over the baseline on the unaligned metric") are not an independent confirmation of better semantic alignment; they partly reflect that the training signal and the evaluation instrument are the same system. The MT and LM results do not share this circularity, since they are measured with task metrics on independently tokenized data.

full rationale

The paper is mostly self-contained. The central empirical claim, consistent perplexity per byte reductions in language modeling (Table 6, Figure 4), is an independent downstream measurement: PairedSPM is trained with a cross-lingual objective on parallel data and evaluated by perplexity on target-only text, with no parameter fitted to the LM loss, so the reported gain is a genuine empirical result even if small and lacking significance tests, which is a statistical robustness concern rather than circularity. The MT finding of no improvement is likewise externally evaluated with chrF++, BLEU, TER, and COMET. The only load-bearing circularity is in the intrinsic alignment evaluation: Eflomal both generates the aligned-word training pairs (Section 3) and computes the one-to-one and unaligned metrics (Section 4.1), so the alignment improvements are not independent evidence. This is a partial, localized circularity and does not collapse the paper's LM claim.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper introduces algorithmic components, namely the conditional co-occurrence table c(t,s) and the mutual-information pruning objective, but it postulates no new physical or linguistic entities. The central claims rest on unstated training hyperparameters and on strong modeling assumptions: the bag-of-tokens approximation of the source sentence and the reliability of Eflomal alignments for both training and evaluation.

free parameters (3)
  • Maximum character span length
    The initial target vocabulary contains 'all character spans up to a fixed length' (Section 3), but the length is never specified, and it changes the candidate token set.
  • niterations and nsubiterations
    Algorithm 1 loops for niterations and prunes every nsubiterations, but the values are not reported for the main experiments, so replication requires guessing or reading the code.
  • Eflomal alignment settings
    Eflomal word alignments define the training pairs in Section 3, but alignment priors and hyperparameters are not reported, leaving the training signal under-specified.
assumptions (5)
  • ad hoc to paper The source sentence can be treated as a bag of tokens for estimating p(t|S)
    Equation (2) drops word order and assumes each source token contributes independently to the probability of a target token, which is a strong simplification not derived from data.
  • domain assumption Eflomal word alignments are a reliable training signal and a reliable evaluator of token alignment
    Training examples are built from Eflomal-aligned word pairs (Section 3), and the one-to-one and unaligned metrics are computed with Eflomal on test data (Section 4.1), so both supervision and evaluation assume Eflomal's alignment quality is correct.
  • standard math Unigram dynamic programming remains valid when probabilities are conditioned on source tokens
    Algorithm 2 inherits the unigram independence assumption: the score of a target token is independent of other target tokens given the source sentence S.
  • domain assumption Parallel data are available and the source tokenizer is fixed
    The problem setup in Section 3 assumes access to parallel sentences between source and target languages and a fixed source tokenizer, which limits applicability to monolingual or low-resource scenarios.
  • domain assumption Tokenization is invertible by concatenation
    Section 3 requires that the original text can be reconstructed by concatenating tokens, which is standard for subword tokenizers but excludes segmentations that delete or alter characters.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Conditional Unigram Tokenization with Parallel Data." pith.science (2026). https://pith.science/paper/UUEN4VCJ

@misc{pith2026250707824,
  author       = {Pith},
  title        = {Pith review of: Conditional Unigram Tokenization with Parallel Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UUEN4VCJ}},
  note         = {Machine review of arXiv:2507.07824}
}
read the original abstract

We introduce conditional unigram tokenization, a novel approach that extends unigram tokenization by conditioning target token probabilities on source-language tokens from parallel data. Given a fixed source tokenizer, our method learns a target tokenizer that maximizes cross-lingual semantic alignment. We evaluate our tokenizer on four language pairs across different families and resource levels, examining intrinsic properties and downstream performance on machine translation and language modeling. While our conditional tokenizer maintains comparable statistical properties to standard unigram tokenizers, results are mixed: we observe no improvements in machine translation quality, but find consistent perplexity reductions in language modeling. We hypothesize that quadratic scaling of conditional probability estimation with respect to the vocabulary size creates a data efficiency bottleneck. Our findings suggest that alternative parameterizations may be necessary for practical cross-lingual tokenization.

Figures

Figures reproduced from arXiv: 2507.07824 by the authors.

Figure 1
Figure 1. Fertility and parity scores of the tokenizers on the different language pairs, subdivided by vocabulary size (color) [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Alignment scores of the tokenizers on the different language pairs subdivided by vocabulary size. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. chrF++ (↑) scores on the different language pairs and vocabulary sizes. For most pairs, the baseline has higher scores than PairedSP and lower variance. depending on the vocabulary size) from scratch using the HuggingFace implementation on the target language of each language pair (hyperparameter details in Appendix G). We compare two settings, monolingual and bilingual, and each model is trained on a fixed number o… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Perplexity per byte of bilingual language models trained on the different languages, subdivided by vocabulary [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

32 extracted references · 15 canonical work pages

  1. [1]

    Do all languages cost the same? tokenization in the era of commercial language models

    Ahia, O., Kumar, S., Gonen, H., Kasai, J., Mortensen, D., Smith, N., and Tsvetkov, Y. Do all languages cost the same? tokenization in the era of commercial language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 9904--9923, Singapore, December 2023. Associ...

    doi:10.18653/v1/2023.emnlp-main.614 2023

  2. [2]

    Ba \ n \'o n, M., Chen, P., Haddow, B., Heafield, K., Hoang, H., Espl \`a -Gomis, M., Forcada, M. L., Kamran, A., Kirefu, F., Koehn, P., Ortiz Rojas, S., Pla Sempere, L., Ram \'i rez-S \'a nchez, G., Sarr \'i as, E., Strelec, M., Thompson, B., Waites, W., Wiggins, D., and Zaragoza, J. P ara C rawl: Web-scale acquisition of parallel corpora. In Jurafsky, D...

    2020

  3. [3]

    F., Della-Pietra, S

    Brown, P. F., Della-Pietra, S. A., Della-Pietra, V. J., and Mercer, R. L. The mathematics of statistical machine translation. Computational Linguistics, 19 0 (2): 0 263--313, 1993. URL http://acl.ldc.upenn.edu/J/J93/J93-2003.pdf

    work page 1993

  4. [4]

    and Gildea, D

    Chung, T. and Gildea, D. Unsupervised tokenization for machine translation. In Koehn, P. and Mihalcea, R. (eds.), Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pp.\ 718--726, Singapore, August 2009. Association for Computational Linguistics. URL https://aclanthology.org/D09-1075/

    work page 2009

  5. [5]

    Bilingual subword segmentation for neural machine translation

    Deguchi, H., Utiyama, M., Tamura, A., Ninomiya, T., and Sumita, E. Bilingual subword segmentation for neural machine translation. In Scott, D., Bel, N., and Zong, C. (eds.), Proceedings of the 28th International Conference on Computational Linguistics, pp.\ 4287--4297, Barcelona, Spain (Online), December 2020. International Committee on Computational Ling...

    work page doi:10.18653/v1/2020.coling-main.378 2020

  6. [6]

    and Neubig, G

    Dou, Z.-Y. and Neubig, G. Word alignment by fine-tuning embeddings on parallel corpora. In Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2021

    work page 2021

  7. [7]

    Dyer, C., Chahuneau, V., and Smith, N. A. A simple, fast, and effective reparameterization of IBM model 2. In Vanderwende, L., Daum \'e III, H., and Kirchhoff, K. (eds.), Proceedings of the 2013 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies , pp.\ 644--648, Atlanta, Georgia, June 20...

    work page 2013

  8. [8]

    Beyond literal token overlap: Token alignability for multilinguality

    H \"a mmerl, K., Limisiewicz, T., Libovick \'y , J., and Fraser, A. Beyond literal token overlap: Token alignability for multilinguality. In Chiruzzo, L., Ritter, A., and Wang, L. (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short ...

    work page 2025

Show all 32 references
  1. [9]

    The state and fate of linguistic diversity and inclusion in the NLP world

    Joshi, P., Santy, S., Budhiraja, A., Bali, K., and Choudhury, M. The state and fate of linguistic diversity and inclusion in the NLP world. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (eds.), Proceedings of the 58th Annual Meeting of the Association for Computat...

    2020 doi

  2. [10]

    Junczys-Dowmunt, M., Grundkiewicz, R., Dwojak, T., Hoang, H., Heafield, K., Neckermann, T., Seide, F., Germann, U., Fikri Aji, A., Bogoychev, N., Martins, A. F. T., and Birch, A. Marian: Fast neural machine translation in C++ . In Proceedings of ACL 2018, System Demonstrations...

    2018

  3. [11]

    Subword regularization: Improving neural network translation models with multiple subword candidates

    Kudo, T. Subword regularization: Improving neural network translation models with multiple subword candidates. In Gurevych, I. and Miyao, Y. (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 66--75, Me...

    2018 doi

  4. [12]

    and Fraser, A

    Libovick \'y , J. and Fraser, A. Findings of the WMT 2021 shared tasks in unsupervised MT and very low resource supervised MT . In Barrault, L., Bojar, O., Bougares, F., Chatterjee, R., Costa-jussa, M. R., Federmann, C., Fishel, M., Fraser, A., Freitag, M., Graham, Y., Grundki...

    2021

  5. [13]

    and Helcl, J

    Libovick \'y , J. and Helcl, J. Lexically grounded subword segmentation. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.\ 7403--7420, Miami, Florida, USA, November 2024. Associat...

    2024 doi

  6. [14]

    Tokenization impacts multilingual language modeling: Assessing vocabulary allocation and overlap across languages

    Limisiewicz, T., Balhar, J., and Mare c ek, D. Tokenization impacts multilingual language modeling: Assessing vocabulary allocation and overlap across languages. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Findings of the Association for Computational Linguistics: ...

    2023 doi

  7. [15]

    WECHSEL : Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models

    Minixhofer, B., Paischer, F., and Rekabsaz, N. WECHSEL : Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models. In Carpuat, M., de Marneffe, M.-C., and Meza Ruiz, I. V. (eds.), Proceedings of the 2022 Conference of the North A...

    2022 doi

  8. [16]

    NLLB Team , Costa-jussà, M. R., Cross, J., Çelebi, O., Elbayad, M., Heafield, K., Heffernan, K., Kalbassi, E., Lam, J., Licht, D., Maillard, J., Sun, A., Wang, S., Wenzek, G., Youngblood, A., Akula, B., Barrault, L., Gonzalez, G. M., Hansanti, P., Hoffman, J., Jarrett, S., Sad...

    2022 arXiv

  9. [17]

    and Tiedemann, J

    \"O stling, R. and Tiedemann, J. Efficient word alignment with M arkov C hain M onte C arlo. Prague Bulletin of Mathematical Linguistics, 106: 0 125--146, October 2016. URL http://ufal.mff.cuni.cz/pbml/106/art-ostling-tiedemann.pdf

    2016

  10. [18]

    Bleu: a Method for Automatic Evaluation of Machine Translation

    Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: a Method for Automatic Evaluation of Machine Translation . In Isabelle, P., Charniak, E., and Lin, D. (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , pp.\ 311--318, Phila...

    2002

  11. [19]

    Language Model Tokenizers Introduce Unfairness Between Languages

    Petrov, A., La Malfa, E., Torr, P., and Bibi, A. Language Model Tokenizers Introduce Unfairness Between Languages . Advances in Neural Information Processing Systems, 36: 0 36963--36990, December 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/74bb24dca833...

    2023

  12. [20]

    How multilingual is multilingual BERT ? In Korhonen, A., Traum, D., and M \`a rquez, L

    Pires, T., Schlinger, E., and Garrette, D. How multilingual is multilingual BERT ? In Korhonen, A., Traum, D., and M \`a rquez, L. (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 4996--5001, Florence, Italy, July 2019. Asso...

    2019 doi

  13. [21]

    A call for clarity in reporting BLEU scores

    Post, M. A call for clarity in reporting BLEU scores. In Bojar, O., Chatterjee, R., Federmann, C., Fishel, M., Graham, Y., Haddow, B., Huck, M., Yepes, A. J., Koehn, P., Monz, C., Negri, M., N \'e v \'e ol, A., Neves, M., Post, M., Specia, L., Turchi, M., and Verspoor, K. (eds...

    2018 doi

  14. [22]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. Technical report, OpenAI, 2019. URL https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

    2019

  15. [23]

    C., and Lavie, A

    Rei, R., Stewart, C., Farinha, A. C., and Lavie, A. COMET : A neural framework for MT evaluation. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.\ 2685--2702, Online, Novem...

    2020 doi

  16. [24]

    Tik-to- Tok : Translating Language Models One Token at a Time : An Embedding Initialization Strategy for Efficient Language Adaptation , October 2023

    Remy, F., Delobelle, P., Berendt, B., Demuynck, K., and Demeester, T. Tik-to- Tok : Translating Language Models One Token at a Time : An Embedding Initialization Strategy for Efficient Language Adaptation , October 2023. URL https://arxiv.org/abs/2310.03477v1

    2023 arXiv

  17. [25]

    Trans- Tokenization and Cross -lingual Vocabulary Transfers : Language Adaptation of LLMs for Low - Resource NLP , August 2024

    Remy, F., Delobelle, P., Avetisyan, H., Khabibullina, A., de Lhoneux, M., and Demeester, T. Trans- Tokenization and Cross -lingual Vocabulary Transfers : Language Adaptation of LLMs for Low - Resource NLP , August 2024. URL http://arxiv.org/abs/2408.04303. arXiv:2408.04303 [cs...

    2024 arXiv

  18. [26]

    How Good is Your Tokenizer ? On the Monolingual Performance of Multilingual Language Models

    Rust, P., Pfeiffer, J., Vulić, I., Ruder, S., and Gurevych, I. How Good is Your Tokenizer ? On the Monolingual Performance of Multilingual Language Models . In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Proceedings of the 59th Annual Meeting of the Association for Comp...

    2021 doi

  19. [27]

    W., Reddy, V., Zhang, H., Alameddine, A., Uzan, O., Pinter, Y., and Tanner, C

    Schmidt, C. W., Reddy, V., Zhang, H., Alameddine, A., Uzan, O., Pinter, Y., and Tanner, C. Tokenization Is More Than Compression . In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pp...

    2024

  20. [28]

    Neural machine translation of rare words with subword units

    Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. In Erk, K. and Smith, N. A. (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 1715--1725, Berlin, Ge...

    2016 doi

  21. [29]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is All you Need . In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://papers.nips.cc/paper_files/paper...

    2017

  22. [30]

    The U niversity of H elsinki submission to the WMT 19 parallel corpus filtering task

    V \'a zquez, R., Sulubacak, U., and Tiedemann, J. The U niversity of H elsinki submission to the WMT 19 parallel corpus filtering task. In Bojar, O., Chatterjee, R., Federmann, C., Fishel, M., Graham, Y., Haddow, B., Huck, M., Yepes, A. J., Koehn, P., Martins, A., Monz, C., Ne...

    2019

  23. [31]

    Vocabulary learning via optimal transport for neural machine translation

    Xu, J., Zhou, H., Gan, C., Zheng, Z., and Li, L. Vocabulary learning via optimal transport for neural machine translation. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th...

    2021 doi

  24. [32]

    Tokenization and the Noiseless Channel

    Zouhar, V., Meister, C., Gastaldi, J., Du, L., Sachan, M., and Cotterell, R. Tokenization and the Noiseless Channel . In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( Volume 1: Lon...

    2023 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.