Pith. sign in

REVIEW 3 major objections 5 minor 50 references

Causal Estimation of Tokenisation Bias

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A subword's membership in a tokeniser's vocabulary causally changes the probability a trained model assigns to that string—by up to 17x in small models and about 2.7x even at 850M parameters—identified as a regression-discontinuity effect…

desk verdict Novel RD framing of tokenisation bias, but the headline 17x estimate likely rests on an unchecked greedy-tokenisation assumption that could inflate it substantially. read the letter →

arxiv 2506.03149 v1 pith:DPJBQGP7 submitted 2025-06-03 cs.CL cs.AIcs.LG

Pietro Lesci , Clara Meister , Thomas Hofmann , Andreas Vlachos , Tiago Pimentel This is my paper
classification cs.CLcs.AIcs.LG
keywords tokenisationbiasregressiondiscontinuitycausalinferencesubwordlanguagemodelsbyte-pairencodingWordPiecevocabularysize
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Language models are trained over subword sequences but ultimately define probabilities over character-strings; in an ideal world the tokeniser would be irrelevant, but this paper argues it is not. It defines tokenisation bias as the causal effect of including a subword in the tokeniser's vocabulary on the log-probability the model assigns to that subword's characters, and estimates it with a regression-discontinuity design: sequential tokenisers rank subwords by merge order, and the vocabulary size $K$ is an arbitrary cutoff, so subwords just below and above $K$ differ in treatment status but little else. In a 57M-parameter BPE model with $K=32\text{k}$ the estimated bias is 2.88 nats (about a 17-fold probability difference), and it remains near 1 nat (about 2.7x) even at 850M parameters. The bias grows over training, appears across BPE, WordPiece, and hybrid tokenisers, and also makes predictions less stable for out-of-vocabulary strings. If correct, these results turn tokeniser choice into a measurable, first-order design decision in language modelling.

What carries the argument

The central mechanism is the regression-discontinuity estimator with the merge index as running variable. Bottom-up tokenisers such as BPE and WordPiece build vocabularies by repeatedly adding the subword that maximises an objective; the iteration at which a subword would be added is its merge index $\gamma_v$, and the vocabulary size $K$ is a predetermined exogenous cutoff. Treatment is deterministic, $W_v = \mathbf{1}\{\gamma_v \le K\}$, so the causal estimand is the discontinuity in the conditional expectation of the observed log-probability at the cutoff, $\psi_{\text{RD}} = \lim_{k \to K^-} \mathbb{E}[Y_{\text{obs}} \mid \gamma_v = k] - \lim_{k \to K^+} \mathbb{E}[Y_{\text{obs}} \mid \gamma_v = k]$, estimated by fitting $Y_{\text{obs}}(v) = \alpha + \beta \gamma_v + \psi_{\text{RD}} W_v + \eta$ in a window around $K$. Two theorems bracket the effect: perfect models give $\psi_v = 0$, while a uniformly initialised model gives $\psi_v \approx \log|V|$.

What would settle it

A covariate-balance or placebo test around the cutoff would settle the matter. For instance, compute the same regression-discontinuity estimate at a fake cutoff offset from the true one (e.g., $K \pm 500$) where treatment status does not actually change; a significantly non-zero effect there would show the estimated discontinuity is not due to vocabulary membership. Alternatively, showing that subword length, corpus frequency, or context entropy changes discontinuously at $K$ would indicate the 2.88-nats estimate is confounded.

Watch

Extended reading notes

Core claim

Tokenisation bias is real, large, and persistent. Concretely, for a character-string whose subword falls just inside the vocabulary cutoff, removing that subword from the vocabulary reduces its average log-probability by 2.88 nats in a 57M BPE model (roughly a 17-fold reduction) and by about 1 nat ($\approx 2.7\times$) in 850M-parameter models. The paper obtains this estimate by comparing subwords immediately around the cutoff with a regression-discontinuity design, using the merge index as the running variable; the estimate is identified under the continuity assumption that potential outcomes are smooth in the merge index at the cutoff. It further shows the effect grows over training (the opposite of what the perfect-model ideal predicts), is weaker for smaller vocabularies but stabilises beyond 32k, is similar across BPE, WordPiece, and a hybrid, and is accompanied by an increase in the variability of predictions for out-of-vocabulary strings.

Load-bearing premise

The load-bearing assumption is continuity of potential outcomes in the merge index at the cutoff: subwords just below and just above the vocabulary boundary would have the same average character-string probability if their treatment status were swapped, and no other covariate, such as subword length, frequency, or morphological complexity, jumps discontinuously at the cutoff.

Editorial extensions

If this is right

  • A character-string tokenised as a single subword receives substantially more probability than the same string split into two subwords; in small models the gap is about 17x, so tokeniser choice alone can dominate lexical probability.
  • Tokenisation bias grows during training, implying that better-fitting models do not automatically approach the ideal of being tokenisation-invariant; instead the bias strengthens as cross-entropy decreases.
  • The bias persists at scale: 340M and 850M models still show roughly 1 nat of bias, so it is not an artifact of small models and is unlikely to vanish with further scaling alone.
  • Vocabulary size interacts with bias: smaller vocabularies (8k) show weaker bias, but beyond 32k the effect stabilises, so shrinking the vocabulary can buy efficiency without much added bias.
  • Because longer tokenisations are systematically penalised, tokenisation bias offers a mechanistic explanation for length bias in generation and for the lower likelihood of lower-resource languages whose tokenisations are longer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same regression-discontinuity design generalises to any tokeniser or preprocessing step with a deterministic ranking and an exogenous cutoff, such as pruning thresholds or quantisation levels, giving causal estimates of design choices without retraining each variant.
  • A direct test of the lexical-generalisation trade-off would measure $\psi_{\gamma_v}$ separately for subwords near the cutoff that are morphological variants or (near-)duplicates of larger subwords; the paper's released checkpoints make this measurement possible without new training.
  • Because the continuity assumption is testable, the 2.88-nats headline number should be paired with covariate-balance plots across the merge index; if subword length or frequency jumps at the cutoff, the estimate is an upper bound on the true causal effect.
  • A model-agnostic criterion for tokeniser selection follows from the sign and magnitude of $\psi_{\gamma_v}$: positive bias at the cutoff argues for vocabulary expansion, while a negligible or negative $\psi$ argues for shrinking the vocabulary, replacing heuristics such as Rényi efficiency.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper defines tokenisation bias as the causal effect of including a subword in a tokeniser's vocabulary on the model's log-probability of that subword's character string. It proposes a regression discontinuity design, using the merge index of BPE/WordPiece tokenizers as a running variable and the vocabulary size cutoff K as the threshold. The authors train Llama-style models of various sizes with different tokenizers, estimate the discontinuity at the cutoff, and report large effects (e.g., 2.88 nats for a 57M-parameter BPE model with K=32k, and about 1 nat for 850M-parameter models). They also provide theoretical consistency checks (zero bias for perfect models, roughly log|V| at initialisation) and robustness analyses varying window size and functional form.

Significance. The causal framing of tokenisation effects is novel and potentially useful: the RD design exploits the deterministic sequential construction of BPE and WordPiece vocabularies, avoiding the need to retrain models under different tokenizers. The paper is transparent about its assumptions, releases code and checkpoints, and reports multiple sensitivity checks. If the headline quantitative estimates are valid, the findings have clear implications for tokenizer design and for interpreting LM probabilities over character strings. The main significance rests on the validity of the outcome measurement, which is currently not established.

major comments (3)
  1. [3, Eq. (7)] The 'no tokenisation mismatch' assumption is load-bearing but unverified. You explicitly state that Eq. (7) holds 'assuming no tokenisation mismatch', yet the paper never measures how often τ(c_<t) is a prefix of τ(c_<t ∘ c_v) in the MiniPile evaluation contexts. When mismatch occurs, the product of subword conditional probabilities used to compute Y0 for out-of-vocabulary subwords is not the true character-level conditional probability; for such subwords it is systematically a lower bound of the marginal probability, so Y0 is underestimated and ψ_RD is inflated. The headline 17x estimate could therefore be partly an artifact of this approximation. I ask you to either compute exact character-level probabilities using the methods you cite (Phan et al., 2024, 2025; Vieira et al., 2024), or report the rate of mismatch and perform a sensitivity analysis that bounds or corrects for it. This is the central threat to the quantitative claim.
  2. [4.3, Assumption 1 (Eq. 18)] The continuity assumption is the identification condition for the RD design, but the paper provides no covariate-balance or density tests around the cutoff. If subword length, corpus frequency, or PMI jump at K, the estimated discontinuity would not be solely due to treatment. The smooth appearance of the outcome plots is not sufficient evidence; the authors should present a McCrary-style density test of the running variable and balance plots/tests for observable covariates on both sides of the cutoff, or argue explicitly why such jumps cannot occur given the construction algorithm. Without this, the causal interpretation of the discontinuity remains an assumption rather than an empirically supported conclusion.
  3. [5-6, Estimator and window choice] The 5k-subword window appears to be chosen after inspecting Fig. 5, which shows that estimates are unstable for smaller windows. This is a specification search, and the linear trend f is extrapolated over a relatively wide window. While the LOESS results in Fig. 9 are reassuring, the paper should compare its estimate with data-driven bandwidth selection methods (e.g., Calonico et al., 2014) and explicitly state that the window choice is post-hoc. This would strengthen the case that the reported ψ_RD is not an artifact of the chosen bandwidth.
minor comments (5)
  1. [Abstract] There is a typo in the abstract: 'T okenisation-Bias' should be 'Tokenisation-Bias'.
  2. [Fig. 2] The y-axis labels are partially truncated in the PDF; please ensure full labels are visible for all four panels.
  3. [Section 6, Results] The phrase 'up to 17 times' is misleading because 17x is the average effect estimated at the cutoff, not a maximum. Please rephrase to something like 'by a factor of about 17 on average'.
  4. [Section 3.1] The notation for individual treatment effects (Eq. 10) and local effects (Eq. 16) is confusing because ψγv is introduced later; consider defining ψγv alongside Eq. (10) to clarify that the paper focuses on the local (conditional-on-running-variable) effect.
  5. [Footnote 5] Your renaming of Phan et al.'s tokenisation bias as tokenisation mismatch is helpful, but please connect it explicitly to the assumption in Eq. (7) so readers understand that the assumption is the absence of the phenomenon they analyse.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the causal estimate is identified from observed log-probabilities under an RD design, not derived from the estimand itself.

full rationale

The paper's central claim is an empirical estimate of a causal effect, defined as a difference between potential outcomes (Definition 3 and eq. 10) and identified via a regression discontinuity design (eqs. 17-22). The estimator fits observed log-probabilities Yobs(v) as a function of the merge-index running variable and treatment indicator; it never uses the target causal quantity ψ as an input. Theorems 1 and 2 are consistency checks derived from the definition of pT and the uniform-initialisation assumption; they are not fitted to data and do not define the estimator. The no-tokenisation-mismatch identity in eq. (7) is a measurement approximation and the continuity assumption (Assumption 1) is an untested identification condition; both are validity threats, not circular reasoning. The self-citations (Pimentel and Meister 2024; Schafer et al. 2024; Whittington et al. 2025) are not load-bearing for the RD estimate: character-level marginalisation is elementary, and the cited works are used only as context or auxiliary discussion. Therefore no step in the derivation reduces to its own inputs by construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard causal inference assumptions rather than invented entities. The free parameters are the RD window and linear trend, chosen post-hoc; the axioms are domain assumptions about tokenization and model probabilities.

free parameters (2)
  • RD window size = 5,000 subwords
    Window width chosen for the main estimates; estimates are unstable below 500 and stabilize around 1k, after which the reported 5k is used. The choice affects the estimate but robustness is shown in Fig. 5.
  • Functional form of running-variable trend f = linear in gamma_v (scaled by 1/1000)
    The paper fits a linear f in eq. (30); LOESS robustness checks in Fig. 9 give similar estimates, reducing concern, but the linear form is an assumption that can bias the treatment effect if misspecified.
assumptions (4)
  • domain assumption Continuity of potential outcomes at the cutoff (Assumption 1).
    Stated in Sec. 4.3, eq. (18). The RD estimate is unbiased only if the conditional expectations of Y0 and Y1 are continuous at the cutoff; this is untestable and is the key identification condition.
  • domain assumption No tokenisation mismatch: for any context, tokenising a prefix yields a prefix of the tokenised string.
    Invoked in Sec. 3, eq. (7). Allows computing character-conditional log-probabilities as products of subword probabilities. In practice, for byte-level BPE this may sometimes fail, and the paper does not quantify the failure rate.
  • domain assumption SUTVA: treatment of one subword does not affect outcomes for others.
    Stated in Sec. C, footnote 17. The paper excludes subwords nested in larger vocabulary subwords, but non-nested interactions may remain and could bias the average effect.
  • domain assumption The running variable (merge index) is a valid, exogenous assignment variable with an arbitrary cutoff.
    Sec. 4.2. Vocabulary size K is preset before training, so the cutoff is exogenous, but subwords close to the cutoff are not randomly assigned; RD relies on smoothness rather than randomisation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Causal Estimation of Tokenisation Bias." pith.science (2026). https://pith.science/paper/DPJBQGP7

@misc{pith2026250603149,
  author       = {Pith},
  title        = {Pith review of: Causal Estimation of Tokenisation Bias},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DPJBQGP7}},
  note         = {Machine review of arXiv:2506.03149}
}
abstract

Modern language models are typically trained over subword sequences, but ultimately define probabilities over character-strings. Ideally, the choice of the tokeniser -- which maps character-strings to subwords -- should not affect the probability assigned to the underlying character-string; in practice, it does. We define this mismatch as tokenisation bias. In this work, we quantify one particular type of tokenisation bias: the effect of including or not a subword (e.g., $\langle hello \rangle$) in a tokeniser's vocabulary on the probability a trained model assigns to the corresponding characters (i.e., \textit{``hello''}). Estimating this effect is challenging because each model is trained with only one tokeniser. We address this by framing tokenisation bias as a causal effect and estimating it using the regression discontinuity design. Specifically, we exploit the fact that tokenisation algorithms rank subwords and add the first $K$ to a tokeniser's vocabulary, where $K$ is an arbitrary cutoff point. As such, we can estimate a causal effect by comparing similar subwords around this cutoff. Experimentally, we find that tokenisation consistently affects models' outputs across scales, vocabularies, and tokenisers. Notably, a subword's presence in a small model's vocabulary may increase its characters' probability by up to 17 times, highlighting tokenisation as a key design choice in language modelling.

Figures

Figures reproduced from arXiv: 2506.03149 by the authors.

Figure 1
Figure 1. Tokenisation bias. Consider subword ⟨hello⟩. If it is included in the vocabulary, then each occurrence of “hello” in the training data is represented as a single subword; otherwise, it is split into two subwords, i.e., ⟨he, llo⟩. If two models are trained under these two settings, the difference in the probability they assign to “hello” is the tokenisation bias we aim to estimate. these subword-strings are produced … view at source ↗
Figure 2
Figure 2. Tokenisation bias of v ∈ T for the last checkpoint of a model trained with BPE tokeniser and K = 32k. The x-axis shows the running variable γv, and the y-axis shows different outcome variables: mean, standard deviation, median, and interquartile range of a cv’s log-probability across contexts. Subwords on the left-hand side of the cutoff are in the tokeniser. The dashed orange line indicates the estimated counterfac… view at source ↗
Figure 3
Figure 3. Tokenisation bias ψγv of 57M across training for (left) BPE across vocabulary sizes 8k, 32k, and 128k; and for (right) vocabulary size 32k across tokenisers BPE, WP, and BPE2WP. Models and Tokenisers. Unless otherwise speci￾fied, we run our experiments with a BPE tokeniser with vocabulary size K = 32k, and a Llama (Tou￾vron et al., 2023) LM with 57M non-embedding parameters trained on the MiniPile (Kaddour, 2023) da… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: (Left to Center-right) Tokenisation bias [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Estimated effect ψbRD (y-axis) vs. window size used to estimate it (x-axis). Results for fully trained model with BPE and K ∈ {8k, 32k, 128k}. Shaded regions correspond to standard errors. effect of tokenisation bias on the stability of model outputs across contexts. T…
Figure 6
Figure 6. Figure 6: Average treatment effect for BPE with K ∈ {8k, 32k, 128k} at the last model checkpoints. Each row refers to a different outcome variable: mean, standard deviation, median, and interquartile range of a cv’s log-probability across contexts. Subwords on the left-hand side…
Figure 7
Figure 7. Figure 7: Estimated effect ψbRD (y-axis) vs. window size used to estimate it. Results for fully trained model with BPE and K ∈ {8k, 32k, 128k}. Shaded regions correspond to standard errors. The columns refer to different outcome variables: mean, standard deviation, median, and i…
Figure 8
Figure 8. Figure 8: Average treatment effect for fully trained models with [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Stability of the average treatment effect with respect to the functional form of [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 18 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max L \"u bbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, Charvi Jain, Alexander Weber, Lena Jurkschat, Hammam Abdelwahab, Chelsea John, Pedro Ortiz Suarez, Malte Ostendorff, Samuel Weinbach, Rafet Sifa, Stefan Kesselheim, and Nicolas Flores-Herr. 2024. https:...

    doi:10.18653/v1/2024.findings-naacl.247 2024

  4. [4]

    Joshua Angrist and J\" o rn-Steffen Pischke. 2015. https://www.masteringmetrics.com Mastering 'Metrics: The Path from Cause to Effect . Princeton University Press

    work page 2015

  5. [5]

    Angrist and J \"o rn-Steffen Pischke

    Joshua D. Angrist and J \"o rn-Steffen Pischke. 2009. https://www.dsecoaching.com/pdf/2008 Princeton university press

    work page 2009

  6. [6]

    Stella Biderman, Kieran Bicheno, and Leo Gao. 2022. https://doi.org/10.48550/arXiv.2201.07311 Datasheet for the Pile . arXiv preprint 2201.07311

    arXiv doi:10.48550/arxiv.2201.07311 2022

  7. [7]

    Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. 2023. https://proceedings.mlr.press/v202/biderman23a/biderman23a.pdf Pythia: A suite for analyzing large language models across t...

    work page 2023

  8. [8]

    Cattaneo, Nicolas Idrobo, and Roc\' o Titiunik

    Matias D. Cattaneo, Nicolas Idrobo, and Roc\' o Titiunik. 2024. https://doi.org/10.1017/9781009441896 A Practical Introduction to Regression Discontinuity Designs: E xtensions . Cambridge University Press

    work page doi:10.1017/9781009441896 2024

Show all 50 references
  1. [9]

    Yekun Chai, Yewei Fang, Qiwei Peng, and Xuhong Li. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.86 Tokenization falling short: On subword robustness in large language models . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1582--1599, ...

    2024 doi

  2. [10]

    Chang and Benjamin K

    Tyler A. Chang and Benjamin K. Bergen. 2022. https://doi.org/10.1162/tacl_a_00444 Word acquisition in neural language models . Transactions of the Association for Computational Linguistics, 10:1--16

    2022 doi

  3. [11]

    Cole and Constantine E

    Stephen R. Cole and Constantine E. Frangakis. 2009. https://journals.lww.com/epidem/fulltext/2009/01000/the\ Epidemiology, 20(1)

    2009

  4. [12]

    Li Du, Lucas Torroba Hennigen, Tiago Pimentel, Clara Meister, Jason Eisner, and Ryan Cotterell. 2023. https://doi.org/10.18653/v1/2023.acl-long.543 A measure-theoretic characterization of tight language models . In Proceedings of the 61st Annual Meeting of the Association for ...

    2023 doi

  5. [13]

    Philip Gage. 1994. https://dl.acm.org/doi/10.5555/177910.177914 A new algorithm for data compression . C Users Journal, 12(2):23–38

    1994

  6. [14]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. http://arxiv.org/abs/2101.00027 The Pile : An 800GB dataset of diverse text for language modeling . a...

    2020 arXiv

  7. [15]

    Thamme Gowda and Jonathan May. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.352 Finding the optimal vocabulary size for neural machine translation . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3955--3964, Online. Association for Com...

    2020 doi

  8. [16]

    Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tusha...

    2024

  9. [17]

    Jinyong Hahn, Petra Todd, and Wilbert Van der Klaauw. 2001. http://www.jstor.org/stable/2692190 Identification and estimation of treatment effects with a regression-discontinuity design . Econometrica, 69(1):201--209

    2001

  10. [18]

    Shengding Hu, Yuge Tu, Xu Han, Ganqu Cui, Chaoqun He, Weilin Zhao, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Xinrong Zhang, Zhen Leng Thai, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, dahai li, Zhiyuan Liu,...

    2024

  11. [19]

    Jean Kaddour. 2023. https://arxiv.org/abs/2304.08442 The M ini P ile challenge for data-efficient language models . arXiv preprint 2304.08442

    2023 arXiv

  12. [20]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2015. http://arxiv.org/abs/1412.6980 Adam: A method for stochastic optimization . In 3rd International Conference on Learning Representations

    2015 arXiv

  13. [21]

    L\' a szl\' o Kozma and Johannes Voderholzer. 2024. https://arxiv.org/abs/2411.08671 Theoretical analysis of byte-pair encoding . arXiv preprint 2411.08671

    2024 arXiv

  14. [22]

    Ilya Loshchilov and Frank Hutter. 2019. https://openreview.net/forum?id=Bkg6RiCqY7 Decoupled weight decay regularization . In International Conference on Learning Representations

    2019

  15. [23]

    Min Lu, Saad Sadiq, Daniel J Feaster, and Hemant Ishwaran. 2018. https://doi.org/10.1080/10618600.2017.1356325 Estimating individual treatment effect in observational data using random forest methods . Journal of Computational and Graphical Statistics, 27(1):209--219

    2018

  16. [24]

    Ian R. McKenzie, Alexander Lyzhov, Michael Martin Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Xudong Shen, Joe Cavanagh, Andrew George Gritsevskiy, Derik Kauffman, Aaron T. Kirtland, Zhengping Zhou, Yuhui Zhang, Sicong Huang, Daniel Wurgaft, Max Weiss, Al...

    2023

  17. [25]

    Kenton Murray and David Chiang. 2018. https://doi.org/10.18653/v1/W18-6322 Correcting length bias in neural machine translation . In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 212--223, Brussels, Belgium. Association for Computational Li...

    2018 doi

  18. [26]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu ...

    2019

  19. [27]

    Guilherme Penedo, Hynek Kydl \' c ek, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. https://openreview.net/forum?id=n6SCkn2QaG The FineWeb datasets: D ecanting the web for the finest text data at scale . In The Thir...

    2024

  20. [28]

    Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, and Adel Bibi. 2023. https://arxiv.org/abs/2305.15425 Language model tokenizers introduce unfairness between languages . In Advances in Neural Information Processing Systems

    2023 arXiv

  21. [29]

    Muckley, and Karen Ullrich

    Buu Phan, Brandon Amos, Itai Gat, Marton Havasi, Matthew J. Muckley, and Karen Ullrich. 2025. https://openreview.net/forum?id=zGej22CBnS Exact byte-level probabilities from tokenized language models for FIM -tasks and model ensembles . In The Thirteenth International Conferenc...

    2025

  22. [30]

    Muckley, and Karen Ullrich

    Buu Phan, Marton Havasi, Matthew J. Muckley, and Karen Ullrich. 2024. https://openreview.net/forum?id=OqfdrBj1y1 Understanding and mitigating tokenization bias in language models . In ICML 2024 Workshop on Theoretical Foundations of Foundation Models

    2024

  23. [31]

    Tiago Pimentel and Clara Meister. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.1020 How to compute the probability of a word . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18358--18375, Miami, Florida, USA. Association f...

    2024 doi

  24. [32]

    Nived Rajaraman, Jiantao Jiao, and Kannan Ramchandran. 2024. https://proceedings.neurips.cc/paper\ In Advances in Neural Information Processing Systems, volume 37, pages 62503--62556. Curran Associates, Inc

    2024

  25. [33]

    Donald B. Rubin. 1974. https://psycnet.apa.org/record/1975-06502-001?doi=1 Estimating causal effects of treatments in randomized and nonrandomized studies . Journal of Educational Psychology, 66(5):688--701

    1974

  26. [34]

    Donald B. Rubin. 2005. https://doi.org/10.1198/016214504000001880 Causal inference using potential outcomes . Journal of the American Statistical Association, 100(469):322--331

    2005 doi

  27. [35]

    Phillip Rust, Jonas Pfeiffer, Ivan Vuli \'c , Sebastian Ruder, and Iryna Gurevych. 2021. https://doi.org/10.18653/v1/2021.acl-long.243 How good is your tokenizer? O n the monolingual performance of multilingual language models . In Proceedings of the 59th Annual Meeting of the...

    2021 doi

  28. [36]

    Anton Sch \"a fer, Thomas Hofmann, Imanol Schlag, and Tiago Pimentel. 2024. https://doi.org/10.18653/v1/2024.findings-acl.571 On the effect of (near) duplicate subwords in language modelling . In Findings of the Association for Computational Linguistics: ACL 2024, pages 9580--...

    2024 doi

  29. [37]

    Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, and Chris Tanner

    Craig W. Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, and Chris Tanner. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.40 Tokenization is more than compression . In Proceedings of the 2024 Conference on Empirical Methods in Natural Langua...

    2024 doi

  30. [38]

    Mike Schuster and Kaisuke Nakajima. 2012. https://doi.org/10.1109/ICASSP.2012.6289079 J apanese and K orean voice search . In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149--5152

    2012

  31. [39]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. https://doi.org/10.18653/v1/P16-1162 Neural machine translation of rare words with subword units . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages ...

    2016 doi

  32. [40]

    Felix Stahlberg and Bill Byrne. 2019. https://doi.org/10.18653/v1/D19-1331 On NMT search errors and model errors: Cat got your tongue? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natura...

    2019 doi

  33. [41]

    Thistlewaite and Donald T

    Donald L. Thistlewaite and Donald T. Campbell. 1960. https://doi.org/10.1037/h0044319 Regression-discontinuity analysis: A n alternative to the ex-post facto experiment . Journal of Educational Psychology, 51(6):309--317

    1960 doi

  34. [42]

    Cagri Toraman, Eyup Halit Yilmaz, Furkan S ahi\.nu c , and Oguzhan Ozcelik. 2023. https://doi.org/10.1145/3578707 Impact of tokenization on language models: A n analysis for T urkish . ACM Trans. Asian Low-Resour. Lang. Inf. Process., 22(4)

    2023 doi

  35. [43]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  36. [44]

    O'Donnell, and Ryan Cotterell

    Tim Vieira, Ben LeBrun, Mario Giulianelli, Juan Luis Gastaldi, Brian DuSell, John Terilla, Timothy J. O'Donnell, and Ryan Cotterell. 2024. https://arxiv.org/abs/2412.03719 From language models over tokens to language models over characters . arXiv preprint 2412.03719

    2024 arXiv

  37. [45]

    Dixuan Wang, Yanda Li, Junyuan Jiang, Zepeng Ding, Ziqin Luo, Guochao Jiang, Jiaqing Liang, and Deqing Yang. 2025. https://arxiv.org/abs/2405.17067 Tokenization matters! D egrading large language models through challenging their tokenization . arXiv preprint 2405.17067

    2025 arXiv

  38. [46]

    Wang, David Leo Wright Hall, Percy Liang, and Tengyu Ma

    Kaiyue Wen, Zhiyuan Li, Jason S. Wang, David Leo Wright Hall, Percy Liang, and Tengyu Ma. 2025. https://openreview.net/forum?id=m51BgoqvbP Understanding warmup-stable-decay learning rates: A river valley loss landscape view . In The Thirteenth International Conference on Learn...

    2025

  39. [47]

    Philip Whittington, Gregor Bachmann, and Tiago Pimentel. 2025. https://arxiv.org/abs/2412.15210 Tokenisation is NP -complete . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics

    2025 arXiv

  40. [48]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen , Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mar...

    2020 doi

  41. [49]

    Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. 2022. https://openaccess.thecvf.com/content/CVPR2022/html/Zhai\ In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12104--12113

    2022

  42. [50]

    Vil \'e m Zouhar, Clara Meister, Juan Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell. 2023. https://doi.org/10.18653/v1/2023.acl-long.284 Tokenization and the noiseless channel . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (...

    2023 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.