Search procedure (bottom-up vs. top-down) determines language model bits-per-byte more than optimisation objective (compression vs. log-likelihood) in tokeniser learning algorithms.
Causal Estimation of Tokenisation Bias
4 Pith papers cite this work. Polarity classification is still indexing.
4
Pith papers citing it
fields
cs.CL 4years
2026 4representative citing papers
LangMAP adapts UnigramLM for multilingual use to deliver language-specific tokenization from a shared vocabulary, boosting boundary alignment metrics across natural and programming languages with mixed downstream fine-tuning gains.
Tokenizer choice affects languages unevenly: languages with less language-model training data are most sensitive, and reallocating tokenizer capacity toward them does not reliably help.
Lexical frequency is a stronger predictor of metaphor novelty than LM surprisal, with the surprisal-novelty link peaking early in training before declining as surprisal becomes more aligned with frequency.
citing papers explorer
-
Objective vs. Search: Decomposing What Makes a Good Tokeniser
Search procedure (bottom-up vs. top-down) determines language model bits-per-byte more than optimisation objective (compression vs. log-likelihood) in tokeniser learning algorithms.
-
LangMAP: A Language-Adaptive Approach to Tokenization
LangMAP adapts UnigramLM for multilingual use to deliver language-specific tokenization from a shared vocabulary, boosting boundary alignment metrics across natural and programming languages with mixed downstream fine-tuning gains.
-
Language-Specific Effects of Tokenizer Choice in Multilingual Language Models
Tokenizer choice affects languages unevenly: languages with less language-model training data are most sensitive, and reallocating tokenizer capacity toward them does not reliably help.
-
The Frequency Confound in Language-Model Surprisal and Metaphor Novelty
Lexical frequency is a stronger predictor of metaphor novelty than LM surprisal, with the surprisal-novelty link peaking early in training before declining as surprisal becomes more aligned with frequency.