Pith. sign in

REVIEW 3 major objections 3 minor 77 references

Byte-model knowledge could transfer while boundary placement stays put

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

The paper hypothesizes that next-byte and boundary distributions in byte-level LMs can be disentangled, proposes two experiments to test it, but provides no experimental results.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A plausible and well-framed hypothesis, but the missing preliminary data make it a roadmap rather than a result; worth refereeing as a position paper. the 3 major comments →

arxiv 2608.03599 v1 pith:S743BM3E submitted 2026-08-04 cs.CL

Disentangling Language Modeling and Boundaries

Mykola Haltiuk This is my paper
classification cs.CL
keywords byte-level language modelspatch boundariesboundary predictiontokenizer transferdistillationnext-byte predictiondisentanglementmodel interface
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that byte-level language models actually produce two separate distributions: one that predicts the next byte and one that decides where to place patch boundaries. The author hypothesizes these two can be disentangled and changed almost independently, so a model could absorb a teacher's skill without adopting its segmentation, or re-segment its input without losing knowledge. If true, the per-model tokenizer would no longer block capability transfer, making cross-tokenizer distillation exact and cheap. The paper proposes two concrete experiments to settle this, previewing the measurements needed to run them.

Core claim

The paper's central claim is that the language-modeling distribution and the boundary-prediction distribution of a byte-level model are separable enough that each can be changed almost independently of the other. Using Bolmo as the canonical architecture, the author defines a joint distribution over a byte and a boundary marker, whose marginals are exactly the two distributions of interest. Because every byte-level model emits over the same 256-byte output space, capability transfer via next-byte distillation should be exact, not approximate. The two experiments are: distill only the language-modeling distribution to move a capability while checking the student's boundary divergence stays un

What carries the argument

The joint distribution p(b, m) over a byte b and a boundary marker m, where every byte appears twice (plain and patch-ending), with marginals Σ_m p(b, m) and Σ_b p(b, m) as the language-modeling and boundary distributions. Bolmo emits this joint distribution, which makes the two marginals measurable and separately manipulable through distillation, carrying the entire disentangling argument.

Load-bearing premise

The claim rests on the assumption that a model's capabilities are stored in its deeper layers rather than its embeddings, so replacing embeddings with a byte interface does not erase them and the boundary predictor can be retrained without disturbing them.

What would settle it

Run the paper's first experiment: distill one byte-level model into another on next-byte predictions alone, choosing a teacher and student with strongly divergent boundary distributions. If the student's boundary divergence shifts substantially after distillation, or if the targeted capability fails to transfer, the disentangling hypothesis is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Capability transfer between any two byte-level models becomes exact and routine, since both emit distributions over the same 256-byte alphabet.
  • Boundary placement becomes a direct control knob: retrain the boundary predictor to change compression rate without retraining the backbone transformer.
  • The per-model tokenizer is no longer an obstacle, eliminating the need for tokenizer transfer or extensive synthetic data generation.
  • Approximate cross-tokenizer distillation methods become unnecessary bridges, subsumed by the shared byte interface.
  • If the hypothesis holds, the community gains a strong structural reason to standardize on a byte-level interface for all language models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same disentangling logic could extend to other jointly emitted structural distributions, such as word or sentence boundaries, not just patch boundaries.
  • Even a partial independence between the two distributions would meaningfully reduce the cost of adapting models to low-resource languages, where tokenizer mismatch currently forces expensive synthetic data pipelines.
  • The reported high divergence between SentencePiece- and BPE-derived boundary distributions suggests the proposed experiments already have ideal test pairs; a null result in one pair need not rule out disentangling in models with more separable boundary predictors.
  • If boundary retraining succeeds, compression rate becomes a practical efficiency lever for byte-level models, potentially lowering inference and memory costs without sacrificing capability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. This preprint argues that byte-level language models have a structural advantage beyond tokenizer robustness: because all byte-level models emit distributions over the same 256-byte alphabet, knowledge transfer between them is exact. The paper's central hypothesis is that the next-byte distribution and the patch-boundary distribution produced by such a model can be disentangled and changed almost independently, so a model could acquire a teacher's capabilities without changing its boundaries, or change its boundaries without losing capabilities. The paper motivates this idea via Bolmo-style byteification, defines boundary divergence as 1−F1, references (without showing) preliminary measurements on Ukrainian models and high-divergence settings, and describes two prospective experiments: (i) next-byte distillation between two Bolmo models while holding boundaries fixed, and (ii) boundary-predictor retraining while self-distilling the language-modeling distribution. It closes with a call for a shared byte-level interface.

Significance. If the hypothesis were established, the paper would identify a genuinely useful design point: a common byte interface would make cross-tokenizer distillation exact and cheap, and would allow tokenizer/boundary choice to be adjusted separately from language-modeling capabilities. The proposed experiments are concrete and falsifiable, and the focus on Bolmo's joint p(b,m) is a good testbed. These are real strengths. As it stands, however, the manuscript contains no experimental evidence; the preliminary measurements it invokes are not included, and the two decisive experiments are only planned. The contribution is therefore currently a plausible position piece rather than a demonstrated result. The missing data are the difference between a supported claim and a conjecture.

major comments (3)
  1. [Section 2 (Preliminaries), boundary-divergence definition] The manuscript claims two load-bearing empirical facts but reports none of the measurements: (i) 'our initial results confirm embedding resettability holds on Ukrainian models too' and (ii) 'we have already conducted some initial measurements across different domains and model families, and found promising, high-divergence settings', most clearly between SentencePiece- and BPE-derived models. The boundary-divergence metric is defined as 1−F1 (Section 2), yet no F1 values, model names, domains, dataset sizes, or experimental procedures are given. The second experiment also relies on the claim that a Bolmo model's boundaries 'barely differ' from the OLMo-family tokenizer it was byteified from, again without numbers. These are not optional extras: they are the empirical foundation for the disentangling hypothesis and for the feasibility of the proposed boundary-retraining experiment. The au
  2. [Section 3, first experiment] The first experiment 'will distill one Bolmo model into another over the language modeling distribution alone, leaving each model's boundaries untouched.' The premise that next-byte distillation leaves the boundary marginal p(b,m) unchanged is the core of the 'disentangling' claim, but the paper provides no argument or evidence for it. The shared byte alphabet makes the target distribution well-defined, but it does not imply independence between the two marginals of the joint output, nor that the internal boundary predictor is unaffected by distillation. The title's 'Disentangling' asserts more than the text establishes. A proof-of-concept result, or at least a formal statement of the independence claim, is needed before the hypothesis can be regarded as supported.
  3. [Section 3, second experiment] The second experiment aims to change boundaries while keeping capabilities, but the paper itself acknowledges a coupling: 'the boundaries decide how bytes are pooled into patches and so reshape the very input the backbone transformer sees' (Section 3). Since the backbone operates on patched inputs, changing the boundary predictor changes the model's input distribution, so it is not obvious how self-distilling the language-modeling distribution can 'keep it fixed' without altering the model's internal representations. The paper does not specify the self-distillation loss, the data, or the agreement procedure between the auxiliary boundary predictor and the decoding patching head. This is a load-bearing gap in the second direction of the hypothesis.
minor comments (3)
  1. [Section 3] The quantitative claim that Bolmo byteifies 'for under a percent of pretraining cost' is not attributed to a specific result in Minixhofer et al. (2025a); either cite the precise number or soften the wording.
  2. [Section 3] The phrase 'the right common alphabet' is left undefined. Since the paper argues against per-model tokenizers, it should state what alternatives to bytes are being considered (e.g., grapheme/CLP) or explicitly defer that question.
  3. [Abstract] The abstract promises 'preliminary measurements of the properties they rest on', but Section 2 contains only qualitative descriptions. The mismatch should be fixed even after adding the data.

Circularity Check

0 steps flagged

No circularity: the disentangling hypothesis is not derived from fitted data or self-citations, though its preliminary measurements are unreported.

full rationale

This is a position/hypothesis paper, not a derivation, and no step reduces a prediction to its fitted inputs or to a self-citation. The central claim is explicitly stated as a hypothesis to be tested in future experiments ('Our central hypothesis is that these two distributions can be disentangled'), and the experiments are described prospectively ('will distill', 'will try'). The only self-citations (Haltiuk & Smywinski-Pohl, 2026; Paniv et al., 2026) are in the background discussion of tokenizer-transfer costs and are not load-bearing. The supporting citation for embedding resettability (Minixhofer et al., 2025a) is an external group's work, not a self-citation. Per the reviewing rule, I flag the following as missing support, not circularity: Section 2, 'our initial results confirm embedding resettability holds on Ukrainian models too'; Section 2, 'We have already conducted some initial measurements across different domains and model families, and found promising, high-divergence settings'; and Section 2, 'We have also measured how far the Bolmo model's predicted boundaries diverge from the OLMo-family tokenizer it was byteified from, and found that they barely differ at all.' These unreported measurements are the only empirical basis offered for the hypothesis, so the central claim is currently unsupported. But unsupported is not circular: none of these statements is defined in terms of the hypothesis, no fitted quantity is renamed as a prediction, and the claim that the marginals 'can be manipulated separately' is the hypothesis itself, not a consequence derived from earlier definitions. If the preliminary numbers were supplied and the experiments run, the claim would be independently testable.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The paper introduces no new entities or fitted parameters. It depends on several unshown measurements and cited claims about the location of capabilities and the ease of learning boundary distributions.

axioms (4)
  • ad hoc to paper Embedding resettability holds on Ukrainian models
    Section 2 claims this based on the author's own 'initial results' which are not included in the paper.
  • domain assumption Capabilities live in deeper layers, not embeddings
    Taken from Minixhofer et al. 2025a, cited in Section 2.
  • domain assumption Boundary prediction can be measured as per-byte binary classification
    Defined in Section 2, a modeling choice without justification.
  • ad hoc to paper A Bolmo-style model can learn a given boundary distribution relatively easily
    Based on unshown measurements of divergence between Bolmo and the OLMo tokenizer, Section 2.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Disentangling Language Modeling and Boundaries." pith.science (2026). https://pith.science/paper/S743BM3E

@misc{pith2026260803599,
  author       = {Pith},
  title        = {Pith review of: Disentangling Language Modeling and Boundaries},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S743BM3E}},
  note         = {Machine review of arXiv:2608.03599}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Byte-level language models are usually argued for on the grounds of robustness, multilingual fairness, and character-level skills. We point to a different, structural advantage: because they read and write bytes, any two of them share an output space, so knowledge transfer between them is exact and independent of how either was originally tokenized. We hypothesize that the two distributions a byte-level model produces, one over the next byte, one over where its patch boundaries fall, can be disentangled and changed almost independently. A model could absorb a teacher's capability while keeping its own boundaries, or change how it places those boundaries while keeping its capabilities. We lay out the two experiments that would settle the hypothesis, alongside preliminary measurements of the properties they rest on. We argue that the community should move toward a byte-level interface as a shared standard: if the hypothesis holds, then once byte-level models are the norm, transferring capabilities and reshaping boundaries between them become cheap and routine, free of the per-model tokenizer that blocks them today.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

77 extracted references · 21 canonical work pages · 1 internal anchor

  1. [1]

    2021 , eprint=

    Transformer Feed-Forward Layers Are Key-Value Memories , author=. 2021 , eprint=

    2021

  2. [2]

    2024 , eprint=

    Understanding Factual Recall in Transformers via Associative Memories , author=. 2024 , eprint=

    work page 2024

  3. [3]

    2024 , eprint=

    Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages , author=. 2024 , eprint=

    work page 2024

  4. [4]

    Tokenizer Choice For LLM Training: Negligible or Crucial?

    Ali, Mehdi and Fromm, Michael and Thellmann, Klaudia and Rutmann, Richard and L. Tokenizer Choice For LLM Training: Negligible or Crucial?. Findings of the Association for Computational Linguistics: NAACL 2024. 2024. doi:10.18653/v1/2024.findings-naacl.247

    doi:10.18653/v1/2024.findings-naacl.247 2024

  5. [5]

    Unsupervised Cross-lingual Representation Learning at Scale

    Conneau, Alexis and Khandelwal, Kartikay and Goyal, Naman and Chaudhary, Vishrav and Wenzek, Guillaume and Guzm \'a n, Francisco and Grave, Edouard and Ott, Myle and Zettlemoyer, Luke and Stoyanov, Veselin. Unsupervised Cross-lingual Representation Learning at Scale. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. ...

    doi:10.18653/v1/2020.acl-main.747 2020

  6. [6]

    WECHSEL : Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models

    Minixhofer, Benjamin and Paischer, Fabian and Rekabsaz, Navid. WECHSEL : Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022. doi:10.18653/v1/2022.naacl-main.293

    doi:10.18653/v1/2022.naacl-main.293 2022

  7. [7]

    arXiv preprint arXiv:2302.13971 , year=

    Llama: Open and efficient foundation language models , author=. arXiv preprint arXiv:2302.13971 , year=

    Pith/arXiv arXiv

  8. [8]

    arXiv preprint arXiv:2307.09288 , year=

    Llama 2: Open foundation and fine-tuned chat models , author=. arXiv preprint arXiv:2307.09288 , year=

    Pith/arXiv arXiv

  9. [9]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    Pith/arXiv arXiv

  10. [10]

    arXiv preprint arXiv:2403.08295 , year=

    Gemma: Open models based on gemini research and technology , author=. arXiv preprint arXiv:2403.08295 , year=

    Pith/arXiv arXiv

  11. [11]

    arXiv preprint arXiv:2408.00118 , year=

    Gemma 2: Improving open language models at a practical size , author=. arXiv preprint arXiv:2408.00118 , year=

    Pith/arXiv arXiv

  12. [12]

    arXiv preprint arXiv:2503.19786 , year=

    Gemma 3 technical report , author=. arXiv preprint arXiv:2503.19786 , year=

    Pith/arXiv arXiv

  13. [13]

    Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

    OLMo: Accelerating the science of language models , author=. Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

  14. [14]

    arXiv preprint arXiv:2412.13663 , year=

    Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference , author=. arXiv preprint arXiv:2412.13663 , year=

    Pith/arXiv arXiv

  15. [15]

    Transactions of the association for computational linguistics , volume=

    Enriching word vectors with subword information , author=. Transactions of the association for computational linguistics , volume=. 2017 , publisher=

    2017

  16. [16]

    arXiv preprint arXiv:2503.10267 , year=

    An Expanded Massive Multilingual Dataset for High-Performance Language Technologies , author=. arXiv preprint arXiv:2503.10267 , year=

    Pith/arXiv arXiv

  17. [17]

    FOCUS : Effective Embedding Initialization for Monolingual Specialization of Multilingual Models

    Dobler, Konstantin and de Melo, Gerard. FOCUS : Effective Embedding Initialization for Monolingual Specialization of Multilingual Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. doi:10.18653/v1/2023.emnlp-main.829

    doi:10.18653/v1/2023.emnlp-main.829 2023

  18. [18]

    Proceedings of the 2013 conference of the North American chapter of the association for computational linguistics: human language technologies , pages=

    A simple, fast, and effective reparameterization of IBM model 2 , author=. Proceedings of the 2013 conference of the North American chapter of the association for computational linguistics: human language technologies , pages=

    work page 2013

  19. [19]

    arXiv preprint arXiv:2408.04303 , year=

    Trans-tokenization and cross-lingual vocabulary transfers: Language adaptation of LLMs for low-resource NLP , author=. arXiv preprint arXiv:2408.04303 , year=

    Pith/arXiv arXiv

  20. [20]

    Soboleva, Daria and Al-Khateeb, Faisal and Myers, Robert and Steeves, Jacob R and Hestness, Joel and Dey, Nolan , title =

  21. [21]

    arXiv preprint arXiv:2307.08691 , year=

    Flashattention-2: Faster attention with better parallelism and work partitioning , author=. arXiv preprint arXiv:2307.08691 , year=

    Pith/arXiv arXiv

  22. [22]

    O pen S ubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles

    Lison, Pierre and Tiedemann, J. O pen S ubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles. Proceedings of the Tenth International Conference on Language Resources and Evaluation ( LREC `16). 2016

    work page 2016

  23. [24]

    CoRR , volume =

    Ilya Loshchilov and Frank Hutter , title =. CoRR , volume =. 2017 , url =. 1711.05101 , timestamp =

    Pith/arXiv arXiv 2017

  24. [25]

    2023 , eprint=

    Mistral 7B , author=. 2023 , eprint=

    2023

  25. [26]

    arXiv preprint arXiv:2412.04261 , year=

    Aya expanse: Combining research breakthroughs for a new multilingual frontier , author=. arXiv preprint arXiv:2412.04261 , year=

    Pith/arXiv arXiv

  26. [27]

    arXiv preprint arXiv:2503.10497 , year=

    Mmlu-prox: A multilingual benchmark for advanced large language model evaluation , author=. arXiv preprint arXiv:2503.10497 , year=

    Pith/arXiv arXiv

  27. [28]

    MAUPQA: Massive Automatically-created Polish Question Answering Dataset

    MAUPQA: Massive automatically-created Polish question answering dataset , author=. arXiv preprint arXiv:2305.05486 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  28. [29]

    1: A Polish Language Model--Development, Insights, and Evaluation , author=

    Bielik 7B v0. 1: A Polish Language Model--Development, Insights, and Evaluation , author=. arXiv preprint arXiv:2410.18565 , year=

    arXiv

  29. [30]

    Byte Latent Transformer: Patches Scale Better Than Tokens

    Pagnoni, Artidoro and Pasunuru, Ramakanth and Rodriguez, Pedro and Nguyen, John and Muller, Benjamin and Li, Margaret and Zhou, Chunting and Yu, Lili and Weston, Jason E and Zettlemoyer, Luke and Ghosh, Gargi and Lewis, Mike and Holtzman, Ari and Iyer, Srini. Byte Latent Transformer: Patches Scale Better Than Tokens. Proceedings of the 63rd Annual Meeting...

    doi:10.18653/v1/2025.acl-long.453 2025

  30. [31]

    arXiv preprint arXiv:2507.07955 , year=

    Dynamic chunking for end-to-end hierarchical sequence modeling , author=. arXiv preprint arXiv:2507.07955 , year=

    Pith/arXiv arXiv

  31. [32]

    how to successfully recycle English GPT-2 to make models for other languages , author=

    As good as new. how to successfully recycle English GPT-2 to make models for other languages , author=. arXiv preprint arXiv:2012.05628 , year=

    Pith/arXiv arXiv 2012

  32. [33]

    Language Contamination Helps Explains the Cross-lingual Capabilities of E nglish Pretrained Models

    Blevins, Terra and Zettlemoyer, Luke. Language Contamination Helps Explains the Cross-lingual Capabilities of E nglish Pretrained Models. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022. doi:10.18653/v1/2022.emnlp-main.233

    doi:10.18653/v1/2022.emnlp-main.233 2022

  33. [34]

    Knowledge Neurons in Pretrained Transformers

    Dai, Damai and Dong, Li and Hao, Yaru and Sui, Zhifang and Chang, Baobao and Wei, Furu. Knowledge Neurons in Pretrained Transformers. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022. doi:10.18653/v1/2022.acl-long.581

    doi:10.18653/v1/2022.acl-long.581 2022

  34. [35]

    T ok A lign: Efficient Vocabulary Adaptation via Token Alignment

    Li, Chong and Zhang, Jiajun and Zong, Chengqing. T ok A lign: Efficient Vocabulary Adaptation via Token Alignment. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025. doi:10.18653/v1/2025.acl-long.207

    work page doi:10.18653/v1/2025.acl-long.207 2025

  35. [36]

    and Vuli\'

    Minixhofer, Benjamin and Ponti, Edoardo M. and Vuli\'. Zero-Shot Tokenizer Transfer , url =. Advances in Neural Information Processing Systems , editor =

    work page

  36. [37]

    arXiv preprint arXiv:2311.16867 , year=

    The falcon series of open language models , author=. arXiv preprint arXiv:2311.16867 , year=

    Pith/arXiv arXiv

  37. [38]

    Large Vocabulary Size Improves Large Language Models

    Takase, Sho and Ri, Ryokan and Kiyono, Shun and Kato, Takuya. Large Vocabulary Size Improves Large Language Models. Findings of the Association for Computational Linguistics: ACL 2025. 2025. doi:10.18653/v1/2025.findings-acl.57

    doi:10.18653/v1/2025.findings-acl.57 2025

  38. [39]

    XLM - V : Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models

    Liang, Davis and Gonen, Hila and Mao, Yuning and Hou, Rui and Goyal, Naman and Ghazvininejad, Marjan and Zettlemoyer, Luke and Khabsa, Madian. XLM - V : Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. doi:10.18653/v1/2023.emnlp-main.813

    doi:10.18653/v1/2023.emnlp-main.813 2023

  39. [40]

    On the Cross-lingual Transferability of Monolingual Representations

    Artetxe, Mikel and Ruder, Sebastian and Yogatama, Dani. On the Cross-lingual Transferability of Monolingual Representations. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020. doi:10.18653/v1/2020.acl-main.421

    doi:10.18653/v1/2020.acl-main.421 2020

  40. [41]

    Cross-lingual Transfer of Monolingual Models

    Gogoulou, Evangelia and Ekgren, Ariel and Isbister, Tim and Sahlgren, Magnus. Cross-lingual Transfer of Monolingual Models. Proceedings of the Thirteenth Language Resources and Evaluation Conference. 2022

    work page 2022

  41. [42]

    OFA : A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretraining

    Liu, Yihong and Lin, Peiqin and Wang, Mingyang and Schuetze, Hinrich. OFA : A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretraining. Findings of the Association for Computational Linguistics: NAACL 2024. 2024. doi:10.18653/v1/2024.findings-naacl.68

    doi:10.18653/v1/2024.findings-naacl.68 2024

  42. [43]

    arXiv preprint arXiv:2310.03477 , year=

    Tik-to-tok: Translating language models one token at a time: An embedding initialization strategy for efficient language adaptation , author=. arXiv preprint arXiv:2310.03477 , year=

    Pith/arXiv arXiv

  43. [44]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)

    HYPEROFA : Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initialization. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop). 2025. doi:10.18653/v1/2025.acl-srw.6

    doi:10.18653/v1/2025.acl-srw.6 2025

  44. [45]

    G lo V e: Global Vectors for Word Representation

    Pennington, Jeffrey and Socher, Richard and Manning, Christopher. G lo V e: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ( EMNLP ). 2014. doi:10.3115/v1/D14-1162

    doi:10.3115/v1/d14-1162 2014

  45. [46]

    arXiv preprint arXiv:2301.09626 , year=

    Efficient language model training through cross-lingual and progressive transfer learning , author=. arXiv preprint arXiv:2301.09626 , year=

    Pith/arXiv arXiv

  46. [47]

    Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Training

    Marchisio, Kelly and Lewis, Patrick and Chen, Yihong and Artetxe, Mikel. Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Training. Findings of the Association for Computational Linguistics: ACL 2023. 2023. doi:10.18653/v1/2023.findings-acl.338

    doi:10.18653/v1/2023.findings-acl.338 2023

  47. [48]

    Advances in Neural Information Processing Systems , volume=

    Improving language plasticity via pretraining with active forgetting , author=. Advances in Neural Information Processing Systems , volume=

    work page

  48. [49]

    Proceedings of the 40th International Conference on Machine Learning , pages =

    Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling , author =. Proceedings of the 40th International Conference on Machine Learning , pages =. 2023 , editor =

    2023

  49. [50]

    Post \#9194 on Telegram Channel Zaduha

    Bohdan Didenko. Post \#9194 on Telegram Channel Zaduha

    work page

  50. [51]

    doi:10.5281/zenodo.12608602 , url =

    Gao, Leo and Tow, Jonathan and Abbasi, Baber and Biderman, Stella and Black, Sid and DiPofi, Anthony and Foster, Charles and Golding, Laurence and Hsu, Jeffrey and Le Noac'h, Alain and Li, Haonan and McDonell, Kyle and Muennighoff, Niklas and Ociepa, Chris and Phang, Jason and Reynolds, Laria and Schoelkopf, Hailey and Skowron, Aviya and Sutawika, Lintang...

    doi:10.5281/zenodo.12608602

  51. [52]

    The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants

    Bandarkar, Lucas and Liang, Davis and Muller, Benjamin and Artetxe, Mikel and Shukla, Satya Narayan and Husa, Donald and Goyal, Naman and Krishnan, Abhinandan and Zettlemoyer, Luke and Khabsa, Madian. The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants. Proceedings of the 62nd Annual Meeting of the Association for Com...

    doi:10.18653/v1/2024.acl-long.44 2024

  52. [53]

    Global MMLU : Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation

    Singh, Shivalika and Romanou, Angelika and Fourrier, Cl \'e mentine and Adelani, David Ifeoluwa and Ngui, Jian Gang and Vila-Suero, Daniel and Limkonchotiwat, Peerat and Marchisio, Kelly and Leong, Wei Qi and Susanto, Yosephine and Ng, Raymond and Longpre, Shayne and Ruder, Sebastian and Ko, Wei-Yin and Bosselut, Antoine and Oh, Alice and Martins, Andre a...

    2025

  53. [54]

    arXiv preprint arXiv:2009.03300 , year=

    Measuring massive multitask language understanding , author=. arXiv preprint arXiv:2009.03300 , year=

    Pith/arXiv arXiv 2009

  54. [55]

    2207.04672 , archiveprefix =

    No Language Left Behind: Scaling Human-Centered Machine Translation , author =. 2207.04672 , archiveprefix =

    Pith/arXiv arXiv

  55. [56]

    The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation , author=

    work page

  56. [57]

    arXiv preprint arXiv:1902.01382 , year=

    Two New Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English , author=. arXiv preprint arXiv:1902.01382 , year=

    Pith/arXiv arXiv 1902

  57. [58]

    On the Path to Make U krainian a High-Resource Language

    Haltiuk, Mykola and Smywi \'n ski-Pohl, Aleksander. On the Path to Make U krainian a High-Resource Language. Proceedings of the Fourth Ukrainian Natural Language Processing Workshop (UNLP 2025). 2025. doi:10.18653/v1/2025.unlp-1.14

    doi:10.18653/v1/2025.unlp-1.14 2025

  58. [59]

    arXiv preprint arXiv:2309.16609 , year=

    Qwen technical report , author=. arXiv preprint arXiv:2309.16609 , year=

    Pith/arXiv arXiv

  59. [60]

    2025 , eprint=

    Qwen3 Technical Report , author=. 2025 , eprint=

    2025

  60. [61]

    Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing - Natural Language Processing in the Generative AI era , month =

    Paniv, Yurii , title =. Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing - Natural Language Processing in the Generative AI era , month =. 2025 , address =

    work page 2025

  61. [62]

    arXiv preprint arXiv:2410.05864 , year=

    From tokens to words: On the inner lexicon of LLMs , author=. arXiv preprint arXiv:2410.05864 , year=

    Pith/arXiv arXiv

  62. [63]

    The Fourteenth International Conference on Learning Representations , year=

    Token Distillation: Attention-Aware Input Embeddings for New Tokens , author=. The Fourteenth International Conference on Learning Representations , year=

    work page

  63. [64]

    Forty-first International Conference on Machine Learning , year=

    By Tying Embeddings You Are Assuming the Distributional Hypothesis , author=. Forty-first International Conference on Machine Learning , year=

    work page

  64. [65]

    The Smol Training Playbook: The Secrets to Building World-Class LLMs , author=

  65. [66]

    Saiful and Mubasshir, Kazi and Li, Yuan-Fang and Kang, Yong-Bin and Rahman, M

    Hasan, Tahmid and Bhattacharjee, Abhik and Islam, Md. Saiful and Mubasshir, Kazi and Li, Yuan-Fang and Kang, Yong-Bin and Rahman, M. Sohel and Shahriyar, Rifat. XL -Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. 2021. doi:10.18653/v1/2021.findings-acl.413

    doi:10.18653/v1/2021.findings-acl.413 2021

  66. [67]

    2502.12404 , archivePrefix=

    Daniel Deutsch and Eleftheria Briakou and Isaac Caswell and Mara Finkelstein and Rebecca Galor and Juraj Juraska and Geza Kovacs and Alison Lui and Ricardo Rei and Jason Riesa and Shruti Rijhwani and Parker Riley and Elizabeth Salesky and Firas Trabelsi and Stephanie Winkler and Biao Zhang and Markus Freitag , year=. 2502.12404 , archivePrefix=

    Pith/arXiv arXiv

  67. [68]

    2018 , editor =

    Chen, Zhao and Badrinarayanan, Vijay and Lee, Chen-Yu and Rabinovich, Andrew , booktitle =. 2018 , editor =

    work page 2018

  68. [69]

    Paniv, Yurii and Didenko, Bohdan and Haltiuk, Mykola and Humennyy, Vladyslav and Kravchenko, Andrian and Kyslyi, Roman and Makovska, Viktoriia and Orlovskyi, Artem and Ruban, Bohdan and Rudko, Maksym-Yurii and Senyk, Anastasiia and Drushchak, Nazarii and Chaplynskyi, Dmytro and Romanyshyn, Mariana , booktitle =. Data-. 2026 , note =

    work page 2026

  69. [70]

    arXiv preprint arXiv:1503.02531 , year=

    Distilling the knowledge in a neural network , author=. arXiv preprint arXiv:1503.02531 , year=

    Pith/arXiv arXiv

  70. [71]

    Yukhymenko, Hanna and Alexandrov, Anton and Vechev, Martin , year=. Mamay

    work page

  71. [72]

    Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets

    Yukhymenko, Hanna and Alexandrov, Anton and Vechev, Martin. Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets. Findings of the A ssociation for C omputational L inguistics: ACL 2026. 2026. doi:10.18653/v1/2026.findings-acl.2067

    work page doi:10.18653/v1/2026.findings-acl.2067 2026

  72. [73]

    arXiv preprint arXiv:2512.15586 , year=

    Bolmo: Byteifying the Next Generation of Language Models , author=. arXiv preprint arXiv:2512.15586 , year=

    arXiv

  73. [74]

    B y T 5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models

    Xue, Linting and Barua, Aditya and Constant, Noah and Al-Rfou, Rami and Narang, Sharan and Kale, Mihir and Roberts, Adam and Raffel, Colin. B y T 5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models. Transactions of the Association for Computational Linguistics. 2022. doi:10.1162/tacl_a_00461

    doi:10.1162/tacl_a_00461 2022

  74. [75]

    Mykola Haltiuk and Aleksander Smywinski-Pohl , year=. Model-. 2510.21954 , archivePrefix=

    Pith/arXiv arXiv

  75. [76]

    Universal

    Minixhofer, Benjamin and Vuli\'. Universal. Advances in Neural Information Processing Systems , editor =

    work page

  76. [77]

    Nicolas Boizard and Kevin El Haddad and Celine Hudelot and Pierre Colombo , journal=. Towards. 2025 , url=

    work page 2025

  77. [78]

    Cross-Tokenizer

    Avyav Kumar Singh and Yen-Chen Wu and Alexandru Cioba and Alberto Bernacchia and Davide Buffelli , booktitle=. Cross-Tokenizer. 2026 , url=

    work page 2026

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.