Pith. sign in

REVIEW 4 major objections 5 minor 51 references

Retrofitting Large Language Models with Dynamic Tokenization

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A pretrained language model can be retrofitted with a dynamic, input-adaptive tokenizer that shortens token sequences by over 20% with less than 2% accuracy loss, using a hypernetwork to compute embeddings for newly merged tokens on the…

desk verdict A genuinely new idea—hypernetwork-generated embeddings for batch-adaptive BPE merges—with solid experiments, but the XNLI baseline is LoRA-mismatched and efficiency is FLOPs-based, so the headline numbers should be treated as provisional. read the letter →

arxiv 2411.18553 v3 pith:N6U5LTAV submitted 2024-11-27 cs.CL

Darius Feher , Ivan Vulić , Benjamin Minixhofer This is my paper · ORCID
classification cs.CL
keywords dynamictokenizationhypernetworkbyte-pairencodingtokenembeddingpredictioncross-lingualfairnessinferenceefficiencyvocabularyexpansionlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Language models normally lock in a static subword tokenizer at pretraining time, and that fixed vocabulary becomes a bottleneck for efficiency and for non-English languages. This paper claims the bottleneck is removable: it shows that a pretrained, frozen model can be retrofitted with dynamic tokenization, where a BPE-style merging step fuses frequent subword sequences within each batch and a pretrained hypernetwork generates embeddings for the resulting tokens on the fly, with no retraining of the base model. On encoder-style models like XLM-R the method shortens token sequences by 22.5% on XNLI and 26.4% on UNER across 14 languages while losing under 2% accuracy, and on decoder-style models like Mistral-7B it compresses prefilling and scoring inputs by up to 17% with minimal degradation. If correct, this would let existing models run faster and spread compute more fairly across languages without changing their weights.

What carries the argument

The load-bearing mechanism is the pairing of a batch-adaptive BPE-style merging algorithm with a pretrained embedding-prediction hypernetwork $H_\theta$. The update function $U(T_{\text{init}}(D_{\text{batch}}), m)$ repeatedly merges the most frequent adjacent subword tokens within a batch, never crossing word (pre-token) boundaries, and then each merged token $t$ receives an embedding via $E_{\phi_{\text{new}}}(t)=H_\theta(t)$. Because the hypernetwork's predictions do not depend on any fixed vocabulary, the effective vocabulary becomes unbounded for any input whose full token sequence is known in advance, which is what enables the efficiency gains for encoders, prefilling, and scoring.

What would settle it

Evaluate the method on a language or script that the hypernetwork did not see during training, or allow merges that cross word boundaries; if downstream accuracy drops substantially (for example, more than 5 points on XNLI) while the sequence-length gains hold, the central trade-off claim fails. A concrete check is to measure XLM-R's XNLI accuracy with word-level dynamic tokenization on a held-out language such as Swahili when the hypernetwork was trained without Swahili, and compare it against the original tokenizer baseline.

Watch

Extended reading notes

Core claim

The central discovery is that a hypernetwork trained for zero-shot tokenizer transfer—which makes embedding predictions independently for every token—can be repurposed to produce embeddings for tokens that appear in no fixed vocabulary, including multi-word merges formed from subword sequences. Combined with a greedy BPE-style merge that repeatedly fuses the most frequent adjacent subword tokens within a batch, constrained to stay inside word boundaries, this yields a batch-specific vocabulary and tokenization that compress sequences substantially: an average of over 20% sequence-length reduction for XLM-R across 14 languages with less than 2% accuracy loss, and up to 17% reduction for Mistral-7B in scoring and prefilling. For autoregressive generation, the paper replaces true dynamic tokenization with a large but bounded vocabulary of one million tokens, using longest-prefix tokenization and an approximate nearest-neighbor index to retrieve hypernetwork-computed embeddings, which preserves the model's parameter count while still shortening sequences.

Load-bearing premise

The load-bearing premise is that the pretrained hypernetwork produces accurate enough embeddings for arbitrary merged tokens, including multi-word sequences it never saw in training, so that the model's representations remain nearly as good as those built from its original embeddings.

Editorial extensions

If this is right

  • Encoder models like XLM-R can be retrofitted to operate with an effectively unbounded vocabulary, cutting token sequence lengths by over 20% across 14 languages while losing less than 2% accuracy.
  • For decoder models, the same method compresses the key-value cache during prefilling and scoring by up to 17% with minimal performance degradation, without changing the pretrained weights.
  • Expanding the vocabulary to one million tokens and retrieving embeddings with an approximate nearest-neighbor index transfers part of the benefit to autoregressive generation while keeping the model's parameter count fixed.
  • Because the hypernetwork's FLOPs stay below roughly 3% of the model's total FLOPs, the sequence-length reduction translates almost linearly into throughput gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the hypernetwork generalizes as the paper claims, dynamic tokenization could remove the need to fix a tokenizer at pretraining time: any frozen model with a pretrained hypernetwork could adapt to new languages, scripts, or domains on the fly, and the same recipe should transfer to other encoder and decoder families beyond XLM-R and Mistral-7B.
  • The adapter trained with a random number of merges per batch nearly closes the gap to the original tokenizer in English, hinting that stochastic tokenization acts as a regularizer; a testable extension is to use dynamic tokenization as a data-augmentation strategy during fine-tuning of tokenization-sensitive tasks.
  • A stronger stress test than the paper's own word-level constraint would allow merges across word boundaries, which would reveal whether the hypernetwork can compose embeddings at the phrase level and whether further compression is worth the accuracy loss.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a method to retrofit a pretrained language model with dynamic tokenization: at batch level, it applies a BPE-style merging algorithm to compress subword token sequences, and uses a pretrained hypernetwork (from Minixhofer et al., 2024) to compute embeddings for the resulting tokens on the fly. The approach is evaluated on encoder-style XLM-R with LoRA adapters on XNLI and UNER, reporting an average 22.5% and 26.4% sequence-length reduction with accuracy/F1 degradation around 1.9% and 1.7%, respectively. For decoder-style Mistral-7B, the same technique is applied to prefilling and scoring, with up to 17% sequence reduction at claimed minimal degradation, and a bounded 1M-token vocabulary with an ANN index is explored for autoregressive generation. The manuscript includes public code, detailed hyperparameter tables, and FLOPs-based throughput estimates. The central claim is an empirical benchmark result, not a theoretical derivation.

Significance. If the empirical claims hold, the work is practically valuable: it offers a way to shorten token sequences of existing multilingual LMs without retraining the base model, relying instead on embedding-prediction hypernetworks, and it provides a path toward more equitable compute allocation across languages. The paper ships code, uses established benchmarks across 14 languages, and includes ablations against FVT heuristics and original embeddings, which are strengths. The main significance is therefore conditional on two things: the reported accuracy degradation being measured under matched capacity, and the efficiency gain being demonstrated in wall-clock terms rather than only FLOPs. The paper's own limitations section honestly acknowledges the dependence on pretrained hypernetworks and the overhead of per-batch vocabulary generation, but the headline claims in the abstract go beyond what is directly supported by the experiments.

major comments (4)
  1. [§4.1, Table 6; Table 2] The headline XNLI result (Table 2, rows 1 vs 5, average gap 1.9 points) is not a matched comparison. Table 6 specifies that the XNLI task adapter with original subword tokenization uses LoRA rank 32 and scaling 64, while the joint task-and-dynamic-tokenization adapter uses rank 128 and scaling 256 and is trained for 10 or 15 epochs. The reported gap therefore conflates the effect of dynamic tokenization with a four-fold increase in adapter parameters and additional training. To support the abstract's 'less than 2% loss' claim, the authors need to report a rank-128/scaling-256 baseline trained on original subword tokenization, or otherwise match the adapter capacity across both conditions.
  2. [§5.2, Table 4, Appendix E] The efficiency claim is grounded in FLOPs and sequence-length reductions, not measured end-to-end latency. Table 4 reports model FLOPs and hypernetwork FLOPs but no wall-clock time for the BPE-style merging, on-the-fly HN embedding generation, or LRU cache operations. The text asserts that the dynamic-tokenization algorithm's overhead is 'minimal' and can be 'offloaded alongside other data loading logic', but no runtime measurement is provided. Because the abstract claims that dynamic tokenization 'substantially improves inference speed', the paper should include an end-to-end latency comparison or at least a measured overhead breakdown.
  3. [§5.2, Figure 4] For decoder prefilling and scoring, the manuscript states that dynamic tokenization yields 'minimal performance degradation' at up to 17% sequence-length reduction, but it never reports the numeric accuracy difference at that operating point. Figure 4 shows curves but no table gives the MMLU accuracy at 17% reduction relative to the 61.2 baseline. A quantitative statement is needed to substantiate the 'minimal degradation' claim for decoders.
  4. [§1, §5.1, Table 2] The abstract and contributions claim that dynamic tokenization promotes 'fairness across languages' and 'fairer compute allocation', but the paper does not define or measure a fairness metric. Table 2 shows substantial heterogeneity in per-language degradation (e.g., -4.9 points for Turkish, +1.1 points for Vietnamese), so the aggregate 1.9% average does not by itself establish improved fairness. The authors should either report a fairness/compute-parity metric or scope the claim to average accuracy and sequence-length reduction.
minor comments (5)
  1. [§4.1] There is a typo in 'merging frequenct adjacent tokens'; it should read 'merging frequent adjacent tokens'.
  2. [§3.1, Algorithm 1] The paper should clarify how the fixed merge count m is chosen to achieve a target reduction percentage, since the mapping from m to sequence-length reduction differs across languages and datasets, as acknowledged in Section 4.1.
  3. [Captions of Tables 2 and 3] The '50%' and '75%' reduction percentages are relative to the gap between subword-level and word-level tokenization, not absolute sequence-length reductions. The captions should make this explicit, as the term 'reduction percentage' can be misread.
  4. [Appendix B, Table 6] The XNLI joint adapter is trained for 10 or 15 epochs ({10,15}); the paper should specify which value produced the reported results and whether the numbers are averaged over multiple seeds.
  5. [Conclusion] The conclusion states 'up to 6% reduction in (absolute) sequence length on English' for decoder models, whereas the abstract reports 'up to 17% reduction'. The difference should be reconciled in the text, for example by distinguishing MMLU multilingual (17%) from MT-Bench English (6%).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is an empirical benchmark result enabled by a pretrained, code-released hypernetwork, and the reported length reductions and accuracy changes are measured rather than derived from the method's own assumptions.

full rationale

The paper's central claim is an empirical benchmark result, not a derivation. Dynamic tokenization is defined operationally as batch-level BPE-style merging (Eq. 2), and token embeddings are produced by a pretrained hypernetwork from prior work (Eq. 3). The accuracy at each compression level is measured directly, and no parameter in the reported trade-off curve is fitted to a target accuracy. The sequence-length reductions follow from the chosen merge counts and are reported as measured outcomes, not as predictions derived from the method's construction. The hypernetwork comes from Minixhofer et al. (2024), which overlaps with the present authorship, but that prior work is a code-released, pretrained component with its own external evaluation, and the paper additionally validates the HN embeddings against original embeddings (Setting 2 in Table 2) and against FVT baselines, so the self-citation is not an unverified load-bearing premise. The acknowledged dependence on a pretrained hypernetwork is stated in the Limitations section and Appendix F, which further reduces any appearance of circularity. The XNLI comparison does use unequal adapter capacities (rank 32 baseline vs. rank 128 joint adapter, Table 6), but that is a comparison-validity concern about whether the 'less than 2% degradation' is understated, not a circular reduction of the claim to its inputs. Similarly, the method's reliance on HN generalization to larger merges is an assumption that is tested empirically rather than assumed by definition. No load-bearing step reduces to a fit, to a self-citation chain, or to an equation that is equivalent to its input by construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the quality of the pretrained hypernetwork, the validity of batch-level BPE merging as a tokenization strategy, and the assumption that shorter sequences translate to faster inference without counting the cost of the merging stage itself. No new physical entities are introduced.

free parameters (2)
  • Sequence-length reduction targets for encoder adapters = 50% (XNLI), 75% (UNER)
    Chosen manually to define the fixed-merge adapter; the headline <2% average drop is reported at these operating points. The paper also shows full trade-off curves, so these are presentation choices rather than fitted parameters.
  • Hypernetwork embedding cache size
    The LRU cache size is not specified; its hit rate affects the real overhead of HN embedding generation and thus the efficiency claim.
assumptions (4)
  • domain assumption The pretrained hypernetwork H_theta from Minixhofer et al. (2024) generates accurate embeddings for arbitrary tokens created by dynamic BPE merging.
    Section 3.1, Eq. (3). The method's central premise; validated empirically but not proven.
  • domain assumption Merges are restricted to within pre-token word boundaries, which preserves task-relevant structure.
    Section 3.1: 'we constrain the merging process to never merge adjacent tokens which are part of different words.'
  • domain assumption Sequence-length reduction translates to proportional inference speedup, with negligible overhead from the BPE merging stage.
    Section 5.2 and Appendix E; the paper reports FLOPs, not wall-clock time, and does not measure the cost of per-batch pair counting.
  • domain assumption LoRA adapters can jointly adapt to a task and to coarser token granularities.
    Section 4.1; required for encoder experiments, where performance recovery depends on adapter training.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Retrofitting Large Language Models with Dynamic Tokenization." pith.science (2026). https://pith.science/paper/N6U5LTAV

@misc{pith2026241118553,
  author       = {Pith},
  title        = {Pith review of: Retrofitting Large Language Models with Dynamic Tokenization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N6U5LTAV}},
  note         = {Machine review of arXiv:2411.18553}
}
read the original abstract

Current language models (LMs) use a fixed, static subword tokenizer. This default choice typically results in degraded efficiency and language capabilities, especially in languages other than English. To address this issue, we challenge the static design and propose retrofitting LMs with dynamic tokenization: a way to dynamically decide on token boundaries based on the input text via a subword-merging algorithm inspired by byte-pair encoding. We merge frequent subword sequences in a batch, then apply a pre-trained embedding-prediction hypernetwork to compute the token embeddings on-the-fly. For encoder-style models (e.g., XLM-R), this on average reduces token sequence lengths by >20% across 14 languages while degrading performance by less than 2%. The same method applied to pre-filling and scoring in decoder-style models (e.g., Mistral-7B) results in minimal performance degradation at up to 17% reduction in sequence length. Overall, we find that dynamic tokenization can mitigate the limitations of static tokenization by substantially improving inference speed and promoting fairness across languages, enabling more equitable and adaptable LMs.

Figures

Figures reproduced from arXiv: 2411.18553 by the authors.

Figure 1
Figure 1. Dynamic tokenization applied to encoders and [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Accuracies on XNLI with different adapters as a function of sequence length reduction (%). Adapter names follow the format: Adapter type, Embeddings used. In this and subsequent figures, a 0% reduction refers to sequence length obtained with the original subword-level tokenization, while 100% indicates ‘upper bound’ word-level tokenization. Intermediate percentages show proportional reductions between these two extr… view at source ↗
Figure 3
Figure 3. F1-scores on UNER with different adapters as a function of sequence length reduction (%). domain results, the fixed-merges adapter consis￾tently shows stronger cross-lingual transferability than its counterpart trained with sampled merges. On XNLI, it obtains an average accuracy of 72.0% compared to 71.2% ( [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Performance of dynamic tokenization applied to decoders for scoring and prefilling, evaluated across [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: 5-shot prompt template used for MMLU evaluation [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Dynamic tokenization with expanded vocabu [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Tokens processed by the hypernetwork using an HN-specific LRU cache versus processing all unique [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 10 canonical work pages

  1. [1]

    Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.614 Do all languages cost the same? tokenization in the era of commercial language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9904--9923, ...

    doi:10.18653/v1/2023.emnlp-main.614 2023

  2. [2]

    Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max L \"u bbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, Charvi Jain, Alexander Weber, Lena Jurkschat, Hammam Abdelwahab, Chelsea John, Pedro Ortiz Suarez, Malte Ostendorff, Samuel Weinbach, Rafet Sifa, Stefan Kesselheim, and Nicolas Flores-Herr. 2024. https:...

    work page 2024

  3. [3]

    Kaj Bostrom and Greg Durrett. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.414 Byte pair encoding is suboptimal for language model pretraining . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4617--4624, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.findings-emnlp.414 2020

  4. [4]

    Cheng-Han Chiang and Hung-yi Lee. 2023. https://doi.org/10.18653/v1/2023.acl-long.870 Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607--15631, Toronto, Canada. Association for Computational Linguistics

    doi:10.18653/v1/2023.acl-long.870 2023

  5. [5]

    Clark, Dan Garrette, Iulia Turc, and John Wieting

    Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. 2022. https://doi.org/10.1162/tacl_a_00448 Canine: Pre-training an efficient tokenization-free encoder for language representation . Transactions of the Association for Computational Linguistics, 10:73--91

    doi:10.1162/tacl_a_00448 2022

  6. [6]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://doi.org/10.18653/v1/2020.acl-main.747 Unsupervised cross-lingual representation learning at scale . In Proceedings of the 58th Annual Meeting of the Association for Comp...

    doi:10.18653/v1/2020.acl-main.747 2020

  7. [7]

    Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. https://doi.org/10.18653/v1/D18-1269 XNLI : Evaluating cross-lingual sentence representations . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475--2485, Brussels, Belgium. Association...

    doi:10.18653/v1/d18-1269 2018

  8. [8]

    Gautier Dagan, Gabriel Synnaeve, and Baptiste Rozi \`e re. 2024. https://arxiv.org/abs/2402.01035 Getting the most out of your tokenizer for pre-training and domain adaptation . ArXiv preprint, abs/2402.01035

    arXiv 2024

Show all 51 references
  1. [9]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

    2019 doi

  2. [10]

    Konstantin Dobler and Gerard de Melo. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.829 FOCUS : Effective embedding initialization for monolingual specialization of multilingual models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process...

    2023 doi

  3. [11]

    Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, and Jun ' ichi Tsujii. 2020. https://doi.org/10.18653/v1/2020.coling-main.609 C haracter BERT : Reconciling ELM o and BERT for word-level open-vocabulary representations from characters . I...

    2020 doi

  4. [12]

    Takuro Fujii, Koki Shibata, Atsuki Yamaguchi, Terufumi Morishita, and Yasuhiro Sogawa. 2023. https://doi.org/10.18653/v1/2023.acl-srw.5 How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in J apanese . In Proceedings of the 61...

    2023 doi

  5. [13]

    Leonidas Gee, Andrea Zugarini, Leonardo Rigutini, and Paolo Torroni. 2022. https://doi.org/10.18653/v1/2022.emnlp-industry.41 Fast vocabulary transfer for language model compression . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: In...

    2022 doi

  6. [14]

    Siavash Golkar, Mariel Pettee, Michael Eickenberg, Alberto Bietti, Miles Cranmer, Geraud Krawezik, Francois Lanusse, Michael McCabe, Ruben Ohana, Liam Parker, Bruno Régaldo-Saint Blancard, Tiberiu Tesileanu, Kyunghyun Cho, and Shirley Ho. 2023. https://arxiv.org/abs/2310.02989...

    2023 arXiv

  7. [15]

    Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. 2020. http://proceedings.mlr.press/v119/guo20h.html Accelerating large-scale inference with anisotropic vector quantization . In Proceedings of the 37th International Conference on Ma...

    2020

  8. [16]

    Valentin Hofmann, Hinrich Schuetze, and Janet Pierrehumbert. 2022. https://doi.org/10.18653/v1/2022.acl-short.43 An embarrassingly simple method to mitigate undesirable properties of pretrained language model tokenizers . In Proceedings of the 60th Annual Meeting of the Associ...

    2022 doi

  9. [17]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lora: Low-rank adaptation of large language models . In The Tenth International Conference on Learning Representat...

    2022

  10. [18]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. https://arxiv.org/abs/2310.06825 Mistral 7b . ArXiv preprint, abs/2310.06825

    2023 arXiv

  11. [19]

    Taku Kudo. 2018. https://doi.org/10.18653/v1/P18-1007 Subword regularization: Improving neural network translation models with multiple subword candidates . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66--75

    2018 doi

  12. [20]

    Taku Kudo and John Richardson. 2018. https://doi.org/10.18653/v1/D18-2012 S entence P iece: A simple and language independent subword tokenizer and detokenizer for neural text processing . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processin...

    2018 doi

  13. [21]

    Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/d49042a5d49818711c401d34172f9900-Abstract-Datasets\_and\_Benchmarks.html MADLAD-400: A m...

    2023

  14. [22]

    Tian Lan, Deng Cai, Yan Wang, Heyan Huang, and Xian - Ling Mao. 2023. https://openreview.net/pdf?id=CROlOA9Nd8C Copy is all you need . In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net

    2023

  15. [23]

    Minghan Li, Xilun Chen, Ari Holtzman, Beidi Chen, Jimmy Lin, Wen-tau Yih, and Xi Victoria Lin. 2024. https://arxiv.org/abs/2405.19325 Nearest neighbor speculative decoding for llm generation and attribution . ArXiv preprint, abs/2405.19325

    2024 arXiv

  16. [24]

    Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, and Madian Khabsa. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.813 XLM - V : Overcoming the vocabulary bottleneck in multilingual masked language models . In Proceedings of...

    2023 doi

  17. [25]

    Tomasz Limisiewicz, Terra Blevins, Hila Gonen, Orevaoghene Ahia, and Luke Zettlemoyer. 2024. https://doi.org/10.18653/v1/2024.acl-long.804 MYTE : Morphology-driven byte encoding for better and fairer multilingual language modeling . In Proceedings of the 62nd Annual Meeting of...

    2024 doi

  18. [26]

    Yihong Liu, Peiqin Lin, Mingyang Wang, and Hinrich Schuetze. 2024. https://aclanthology.org/2024.findings-naacl.68 OFA : A framework of initializing unseen subword embeddings for efficient large-scale multilingual continued pretraining . In Findings of the Association for Comp...

    2024

  19. [27]

    Stephen Mayhew, Terra Blevins, Shuheng Liu, Marek Suppa, Hila Gonen, Joseph Marvin Imperial, B \"o rje Karlsson, Peiqin Lin, Nikola Ljube s i \'c , Lester James Miranda, Barbara Plank, Arij Riabi, and Yuval Pinter. 2024. https://aclanthology.org/2024.naacl-long.243 Universal N...

    2024

  20. [28]

    Sabrina J Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gall \'e , Arun Raja, Chenglei Si, Wilson Y Lee, Beno\^ i t Sagot, et al. 2021. https://arxiv.org/abs/2112.10508 Between words and characters: A brief history of open-vocabulary modeling and ...

    2021 arXiv

  21. [29]

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. https://arxiv.org/abs/2402.06196 Large Language Models: A Survey . ArXiv preprint, abs/2402.06196

    2024 arXiv

  22. [30]

    Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. 2022. https://doi.org/10.18653/v1/2022.naacl-main.293 WECHSEL : Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models . In Proceedings of the 2022 Conference of the Nor...

    2022 doi

  23. [31]

    Benjamin Minixhofer, Edoardo Maria Ponti, and Ivan Vuli \'c . 2024. https://arxiv.org/abs/2405.07883 Zero-shot tokenizer transfer . ArXiv preprint, abs/2405.07883

    2024

  24. [32]

    Piotr Nawrot, Jan Chorowski, Adrian Lancucki, and Edoardo Maria Ponti. 2023. https://doi.org/10.18653/v1/2023.acl-long.353 Efficient transformers with dynamic token pooling . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: ...

    2023 doi

  25. [33]

    Yuval Pinter, Robert Guthrie, and Jacob Eisenstein. 2017. https://doi.org/10.18653/v1/D17-1010 Mimicking word embeddings using subword RNN s . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 102--112, Copenhagen, Denmark. Associ...

    2017 doi

  26. [34]

    Phillip Rust, Jonas Pfeiffer, Ivan Vuli \'c , Sebastian Ruder, and Iryna Gurevych. 2021. https://doi.org/10.18653/v1/2021.acl-long.243 How good is your tokenizer? on the monolingual performance of multilingual language models . In Proceedings of the 59th Annual Meeting of the ...

    2021 doi

  27. [35]

    Timo Schick and Hinrich Sch \"u tze. 2019. https://doi.org/10.18653/v1/N19-1048 Attentive mimicking: Better word embeddings by attending to informative contexts . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguist...

    2019 doi

  28. [36]

    Timo Schick and Hinrich Sch \"u tze. 2020. https://doi.org/10.18653/v1/2020.acl-main.368 BERTRAM : Improved word embeddings have big impact on contextualized model performance . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3...

    2020 doi

  29. [37]

    Mike Schuster and Kaisuke Nakajima. 2012. https://doi.org/10.1109/ICASSP.2012.6289079 Japanese and korean voice search . In 2012 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5149--5152. IEEE

    2012

  30. [38]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. https://doi.org/10.18653/v1/P16-1162 Neural machine translation of rare words with subword units . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages ...

    2016 doi

  31. [39]

    Shivalika Singh, Angelika Romanou, Cl \'e mentine Fourrier, David I Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, et al. 2024. Global mmlu: Understanding and addressing cultural and linguistic biases in mul...

    2024 arXiv

  32. [40]

    Lichao Sun, Kazuma Hashimoto, Wenpeng Yin, Akari Asai, Jia Li, Philip Yu, and Caiming Xiong. 2020. https://arxiv.org/abs/2003.04985 Adv-bert: Bert is not robust on misspellings! generating nature adversarial samples on bert . ArXiv preprint, abs/2003.04985

    2020 arXiv

  33. [41]

    Tran, Sebastian Ruder, Jai Prakash Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler

    Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Prakash Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. 2022. https://openreview.net/forum?id=JtBRnrlOEFN Charformer: Fast character transformers via gradient-based subword tokenization . ...

    2022

  34. [42]

    Cagri Toraman, Eyup Halit Yilmaz, Furkan S ahinu c , and Oguzhan Ozcelik. 2023. Impact of tokenization on language models: An analysis for turkish. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(4):1--21

    2023

  35. [43]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. https://arxiv.org/abs/2307.09288 Llama 2: Open foundation and fine-tuned chat models . ArXiv preprint, abs/2...

    2023 arXiv

  36. [44]

    Omri Uzan, Craig W Schmidt, Chris Tanner, and Yuval Pinter. 2024. https://arxiv.org/abs/2403.01289 Greed is all you need: An evaluation of tokenizer inference methods . ArXiv preprint, abs/2403.01289

    2024 arXiv

  37. [45]

    Xinyi Wang, Sebastian Ruder, and Graham Neubig. 2021. https://doi.org/10.18653/v1/2021.naacl-main.40 Multi-view subword regularization . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol...

    2021 doi

  38. [46]

    Xu, Uri Alon, and Graham Neubig

    Frank F. Xu, Uri Alon, and Graham Neubig. 2023. https://proceedings.mlr.press/v202/xu23a.html Why do nearest neighbor language models work? In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine...

    2023

  39. [47]

    Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. https://doi.org/10.1162/tacl_a_00461 B y T 5: Towards a token-free future with pre-trained byte-to-byte models . Transactions of the Association for Computa...

    2022 doi

  40. [48]

    Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. 2018. https://openreview.net/forum?id=HkwZSG-CZ Breaking the softmax bottleneck: A high-rank RNN language model . In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada,...

    2018

  41. [49]

    Lili Yu, Daniel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/f8f78f8043f35890181a824e53a57134-Abstract-Conference.html MEGABYTE: predicting million-byte sequences with multiscale transformer...

    2023

  42. [50]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  43. [51]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.