Is There a Case for Conversation Optimized Tokenizers in Large Language Models?

Gonzalo Mart\'inez; Javier Conde; Pedro Reviriego; Raquel Ferrando

REVIEW 3 major objections 3 minor 14 references

Is There a Case for Conversation Optimized Tokenizers in Large Language Models?

T0 review · 3 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Retraining tokenizers on real chatbot conversations cuts token counts by 5-10% without hurting general-text efficiency.

desk verdict Useful benchmark but central domain-attribution claim needs a non-conversational control; still worth refereeing. read the letter →

arxiv 2506.18674 v1 pith:QVE2BYSZ submitted 2025-06-23 cs.CL cs.AI

Raquel Ferrando , Javier Conde , Gonzalo Martínez , Pedro Reviriego This is my paper

classification cs.CLcs.AI

keywords tokenizeroptimizationconversationalAILLMinferenceefficiencytokencountreductionenergy-awarefertilitymetricdomainadaptation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Tokenizers split text into the tokens that language models process, and every token carries a slice of compute and energy cost. This paper asks whether retraining a tokenizer's vocabulary on real chatbot dialogues, instead of on the books and web text used to build the original tokenizer, reduces the number of tokens needed for conversational input and output. Across a representative set of eight widely used tokenizers, retrained on a public corpus of one million real conversations and evaluated on held-out dialogues, the answer is yes: token counts drop by roughly 5% for some models and by more than 10% for others. The same retrained tokenizers lose almost nothing on the original general web-text corpus, less than 2% where they lose at all and a small gain for three of the eight, so the authors conclude that conversation-optimized tokenizers could yield 5-10% inference energy savings in chatbot deployments without penalizing other applications. The paper explicitly does not claim downstream model quality is unaffected, since checking that would require retraining the models themselves.

What carries the argument

The load-bearing mechanism is domain-retrained tokenization: a tokenizer's vocabulary is rebuilt from scratch on a sample of real chatbot conversations using the same tokenization algorithm and vocabulary size as the original, then the new tokenizer is evaluated on held-out conversations and on the original training-domain corpus. The performance comparison uses two quantities: the ratio of token counts between optimized and original tokenizers on the same text, which measures gain or loss directly, and fertility, defined as tokens per word, which allows fair comparison of different tokenizers on texts of different lengths. Retraining reassigns the vocabulary budget to the frequent words and subword units of conversational text, which is why a given dialogue becomes shorter; the same mechanism is what risks changing behavior on other domains, which the third experiment measures.

What would settle it

Retrain the same eight tokenizers on a similarly sized sample of non-conversational text, such as a random slice of the general web corpus, and measure token counts on the same held-out chat conversations; if that control achieves the same 5-10% reduction, the paper's specific claim that conversation data is the cause is refuted.

Watch

Extended reading notes

Core claim

The paper's central discovery is that tokenizers built for general LLM training text are measurably suboptimal for chatbot conversations, and that this gap is recoverable by retraining the tokenizer on conversational text with the same algorithm, vocabulary size, and configuration. On a held-out set of real user/assistant dialogues, every retrained tokenizer reduces token counts relative to the original; optimizing on the full conversation outperforms optimizing on user turns or assistant turns alone. Translating token counts into operations, the authors estimate energy savings of 5-10% at inference time. A second result is that the conversation-optimized tokenizers do not hurt general-text tokenization: three of the eight even improve on the training corpus, which the authors read as evidence that some original tokenizers were already inefficient. They treat the results as preliminary, noting that only one conversation corpus and one training corpus were used and that language distribution effects, such as a measured regression for Chinese on one model, need further study.

Load-bearing premise

The central claim depends on the assumption that retraining on conversations, rather than retraining on some other domain-matched corpus, is what produces the 5-10% token savings; the paper does not test a non-conversational retraining control, so that attribution is unexamined.

Editorial extensions

If this is right

For a model that uses the optimized tokenizer, the measured 5-10% token reduction on conversations translates directly into 5-10% fewer operations per chatbot request, since energy scales with token count.
Retraining on the full conversation yields the largest savings, so practical efforts should optimize tokenizer vocabularies on combined user-plus-assistant text rather than on user questions alone.
The general web-text token count changes by less than 2% where it worsens, and improves for three tokenizers, suggesting that adopting a conversation-optimized tokenizer does not force a large efficiency penalty on non-conversational tasks.
The language-distribution analysis shows that languages well represented in the retraining corpus gain tokens, whereas an underrepresented language can worsen, so language balance in the retraining sample is a design variable, not a detail.
Because the same algorithm and vocabulary size are used, the gains come from vocabulary reassignment rather than from any change in model architecture or training procedure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: the reported gains are not yet evidence that conversation is the special ingredient. The experiments retrain on chat data only, so a control retrained on a similarly sized sample of non-conversational text (news, code, or a random web slice) is needed to separate conversation-specific adaptation from generic domain re-estimation.
Beyond the paper: because three conversation-optimized tokenizers also improved on the general corpus, part of the observed savings may reflect that some production vocabularies were accidentally suboptimal; comparing against other domain-optimized vocabularies would quantify how much of the 5-10% is conversational versus merely better-estimated.
Beyond the paper: the paper measures token counts, not end-to-end quality. A practical next step would be to train or fine-tune a small model with both vocabularies on identical data and compare downstream task scores, which would turn the efficiency claim into a deployable trade-off statement.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague, quick take on arXiv:2506.18674.

What's actually new: the paper does a clean, reproducible measurement of tokenizer efficiency for eight production tokenizers on one million real chatbot conversations, with an 80/20 split and a cross-domain check on C4. Code and data are public. That is real work and the community can build on it. The main finding—retraining a tokenizer on conversations cuts token counts on held-out conversations by roughly 5% to over 10%—is plausible and consistent across the models tested. The fertility analysis (Experiment 1) and the language-wise breakdown add useful detail. The authors are also honest: they explicitly flag the single conversation dataset, the lack of downstream performance evaluation, and the unknown language distributions as limitations.

The soft spots. The biggest one is the missing control. Retraining on conversations is compared against the original production tokenizer, not against a tokenizer retrained on a similarly sized non-conversational sample. The paper's own Experiment 3 supplies the clue: conversation-retrained tokenizers for Mistral, Gemma, and BLOOM also cut tokens on C4 by up to 5%. That tells us part of the gain is just "retrained on a fresh, more targeted corpus" rather than "conversation-optimized" per se. Without a C4-retrained control, the 5–10% savings cannot be cleanly attributed to the conversational domain. This is a genuine methodological gap, not a nit. The second issue is the absence of any error bars or variance across random splits; tokenizer training is deterministic, but the train/test split is random, and we don't know how stable the numbers are. Both are fixable in a revision.

The paper's framing is slightly ahead of its evidence—"conversation-optimized tokenizers" is the natural reading, but the data support "retrained-on-LMSYS tokenizers." Still, the practical message survives: if you serve chatbots, retraining your tokenizer on a sample of chat logs is a cheap 5–10% inference-token saving with minimal downside on general text. That is worth knowing.

Who it's for: people doing efficient inference, tokenization research, and LLM serving. It deserves a serious referee; the review should demand the control experiment and variance estimates. I'd take it, with major revision.

Referee Report

3 major / 3 minor

Summary. The paper asks whether retraining LLM tokenizers on chatbot conversation data can reduce the number of tokens needed to tokenize conversational text, thereby reducing inference energy costs. Using the LMSYS Chat 1M dataset, the authors retrain eight tokenizers (GPT-4, GPT-4o, DeepSeek-R1, LLaMA-3.1-8B, Gemma-2-9B, Mistral-7B-v0.1, BLOOM, and Phi-4) on an 80% split of the conversation data, using either user inputs, assistant outputs, or both. They then measure token-count reductions on the held-out 20% split relative to the original tokenizers, and also evaluate the retrained tokenizers on the C4 corpus. They report consistent reductions of 5% to over 10% on conversational text, with small or even slightly negative impacts on C4 tokenization for several models.

Significance. If the results hold, the paper identifies a simple and practical lever for reducing inference cost in one of the most common LLM use cases, and the public repository and deterministic retraining pipeline make the core measurements reproducible. The study covers a broad and representative set of tokenizers, and the authors are careful to state several limitations, including the use of a single conversation dataset and the lack of downstream task evaluation. However, the central attribution of the observed gains to the conversational nature of the training data is not yet supported, because no control condition retrains a tokenizer on similarly sized non-conversational text. The significance of the paper therefore depends on whether the gains are specific to conversation-optimized training or are a general effect of domain-focused retraining.

major comments (3)

[§2.3.1, §3.2, Fig. 2] The experimental design lacks a control condition in which a tokenizer is retrained on a similarly sized non-conversational corpus (e.g., a C4 subset) using the same algorithm, configuration, and training-set size. Without this control, the 5–10% reductions on LMSYS cannot be attributed to the conversational nature of the training data; retraining on any domain-bearing corpus could yield similar gains because the original production tokenizers were trained on large mixed corpora. This concern is reinforced by §3.3, where three conversation-optimized tokenizers also reduce token counts on C4 by 1–5%, indicating general inefficiencies in the original tokenizers. Please add the control and report its LMSYS and C4 results.
[§2.3.1, Table 2] The paper states that tokenizers are retrained with "the same algorithm and configuration" as the original, but for GPT-4 and GPT-4o the original tokenizer's training algorithm and configuration are proprietary and not accessible, as noted in Table 2. The manuscript does not describe how these tokenizers were retrained or what approximations were used (e.g., a generic BPE implementation with a matching vocabulary size). Without this information, the GPT-4 and GPT-4o results are not reproducible and the comparison to "same configuration" is not supported. Please specify the exact retraining pipeline for each tokenizer, including any approximations.
[§3.2, Figs. 2–4] No variance estimates or repeated-split analyses are reported; conclusions such as "savings range from 5% to over 10%" rely on a single random 80/20 split. Although tokenization is deterministic for a fixed tokenizer and text, the random train/test split induces sampling variability. Please report results over multiple seeds or provide bootstrap confidence intervals for the token-count ratios.

minor comments (3)

[§2.3.1] Please report the random seed used for the train/test split and include the exact subset sizes and language filtering steps; this aids reproducibility.
[Table 2] "V ocabulary" contains an erroneous space, and the relation between Phi-4's and GPT-4's tokenizers is described informally as "almost identical"; this should be quantified or replaced with a precise statement.
[References] The reference "Ahmed" lacks a publication year and venue; please complete the citation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the central claim rests on held-out evaluation and an external corpus.

full rationale

The paper's central claim is that retraining tokenizers on a conversational corpus reduces token counts on held-out chatbot dialogues and has limited impact on general text. The derivation chain is empirical: tokenizers are retrained on an 80% split of LMSYS Chat 1M (Section 2.3.1), then evaluated on the disjoint 20% test split (Section 3.2) and on the external C4 corpus (Section 3.3). There is no equation equating the reported token reduction to the training objective itself; the held-out test and the C4 evaluation provide independent grounding. The tokenizer training objective (minimizing tokens on a training corpus) is not the same as the evaluation, because generalization to an unseen split is not guaranteed by construction. The paper includes no self-citations that are load-bearing; all references are external. The acknowledged limitations (Section 'Limitations') about using a single conversational dataset and not testing downstream performance are honest scope restrictions, not circular steps. The reviewer's concern about a missing non-conversational retraining control is a validity threat to the 'conversation-specific' attribution, but it is not a circularity: the observed improvements could still be genuine even if a control would show that any domain retraining produces similar gains. Thus the derivation is self-contained and non-circular.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper makes no derivational claims and introduces no new entities or fitted parameters. Its central result depends on the representativeness of the two datasets, the assumption that token count proxies energy, and the validity of the retraining procedure as a way to create domain-optimized tokenizers.

assumptions (4)

domain assumption LMSYS Chat 1M is representative of chatbot conversations in general.
The entire evaluation rests on this dataset; if it is not representative, the observed token reductions may not generalize. The authors acknowledge this in limitations.
domain assumption C4 is a good proxy for LLM training corpora.
The cross-domain check uses C4 as the sole representative of training text; other corpora such as The Pile might give different results.
domain assumption Token count is a valid proxy for computational and energy cost.
The paper assumes energy is proportional to token count, citing prior work, but does not measure energy directly.
ad hoc to paper Retraining a tokenizer with the original algorithm, configuration, and vocabulary size on a domain subset produces a valid domain-optimized tokenizer.
This is the core experimental intervention; the paper does not validate that the retrained tokenizer preserves the model's tokenization semantics or that this is how such a tokenizer would be deployed in practice.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Is There a Case for Conversation Optimized Tokenizers in Large Language Models?." pith.science (2026). https://pith.science/paper/QVE2BYSZ

@misc{pith2026250618674,
  author       = {Pith},
  title        = {Pith review of: Is There a Case for Conversation Optimized Tokenizers in Large Language Models?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QVE2BYSZ}},
  note         = {Machine review of arXiv:2506.18674}
}

read the original abstract

The computational and energy costs of Large Language Models (LLMs) have increased exponentially driven by the growing model sizes and the massive adoption of LLMs by hundreds of millions of users. The unit cost of an LLM is the computation of a token. Therefore, the tokenizer plays an important role in the efficiency of a model, and they are carefully optimized to minimize the number of tokens for the text in their training corpus. One of the most popular applications of LLMs are chatbots that interact with users. A key observation is that, for those chatbots, what is important is the performance of the tokenizer in the user text input and the chatbot responses. Those are most likely different from the text in the training corpus. So, a question that immediately arises is whether there is a potential benefit in optimizing tokenizers for chatbot conversations. In this paper, this idea is explored for different tokenizers by using a publicly available corpus of chatbot conversations to redesign their vocabularies and evaluate their performance in this domain. The results show that conversation-optimized tokenizers consistently reduce the number of tokens in chatbot dialogues, which can lead to meaningful energy savings, in the range of 5% to 10% while having minimal or even slightly positive impact on tokenization efficiency for the original training corpus.

Figures

Figures reproduced from arXiv: 2506.18674 by the authors.

**Figure 2.** Reduction on the number of tokens for the conversation-optimized tokenizers on the conversational test [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Token reduction achieved by conversation-optimized tokenizers on the top 10 languages in the LMSYS [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Increase on the number of tokens for the conversation-optimized tokenizers on the LLM training corpus [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

14 extracted references · 7 canonical work pages

[2]

https://allenai.org/

Allen institute for ai (ai2). https://allenai.org/. Accessed: 2025-04-11. Katherine Blunt and Jennifer Hiller

work page 2025
[3]

Accessed: 2024-06-08

Big tech’s latest obsession is finding enough energy. Accessed: 2024-06-08. Common Crawl. Common crawl: Open data for web research. DeepSeek-AI

work page 2024
[4]

arXiv preprint arXiv:2401.02954

Deepseek llm: Scaling open- source language models with longtermism. arXiv preprint arXiv:2401.02954. Hugging Face. Hugging face - the ai community build- ing the future. Hugging Face

arXiv
[8]

https://huggingface

C4: Colos- sal clean crawled corpus. https://huggingface. co/datasets/allenai/c4. Accessed: 2025-04-11. Phillip Rust, Jonas Pfeiffer, Ivan Vuli´c, Sebastian Ruder, and Iryna Gurevych

work page 2025
[9]

Preprint, arXiv:2012.15613

How good is your tok- enizer? on the monolingual performance of multilin- gual language models. Preprint, arXiv:2012.15613. Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Berg- eron, Jeremy Kepner, Devesh Tiwari, and Vijay Gade- pally

arXiv 2012
[11]

arXiv preprint arXiv:2403.20306

Towards greener llms: Bringing energy-efficiency to the forefront of llm inference. arXiv preprint arXiv:2403.20306. Changhan Wang, Kyunghyun Cho, and Jiatao Gu

arXiv
[13]

Preprint, arXiv:2211.05100

Bloom: A 176b-parameter open-access multilingual language model. Preprint, arXiv:2211.05100. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric. P Xing, Joseph E. Gonza- lez, Ion Stoica, and Hao Zhang

arXiv
[14]

Preprint, arXiv:2309.11998

Lmsys-chat- 1m: A large-scale real-world llm conversation dataset. Preprint, arXiv:2309.11998

arXiv

Show all 14 references

[2019]

Preprint, arXiv:1909.03341

Neural machine translation with byte-level subwords. Preprint, arXiv:1909.03341. Patrick Wilhelm, Thorsten Wittkopp, and Odej Kao

1909 arXiv
[2020]

https://pile.eleuther.ai/

The pile: An 800gb dataset of diverse text for language modeling. https://pile.eleuther.ai/. Accessed: 2025-04-

2025
[2021]

arXiv preprint arXiv:2112.10508

Between words and characters: A brief history of open-vocabulary modeling and tokenization in nlp. arXiv preprint arXiv:2112.10508. Mistral

arXiv
[2023]

In 2023 IEEE High Performance Extreme Computing Conference (HPEC), pages 1–9

From words to watts: Benchmarking the energy costs of large language model inference. In 2023 IEEE High Performance Extreme Computing Conference (HPEC), pages 1–9. Philipp Schmid, Omar Sanseviero, Pedro Cuenca, and Lewis Tunstall

2023
[2024]

Associa- tion for Computational Linguistics

Tokenizer choice for LLM training: Negligible or crucial? In Findings of the Association for Computational Lin- guistics: NAACL 2024, pages 3907–3924. Associa- tion for Computational Linguistics. Allen Institute for AI

2024
[2025]

https://www.demandsage.com/chatbot-statistics/

65 chatbot statis- tics for 2025 — new data released. https://www.demandsage.com/chatbot-statistics/. Llama

2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [2]

https://allenai.org/

Allen institute for ai (ai2). https://allenai.org/. Accessed: 2025-04-11. Katherine Blunt and Jennifer Hiller

work page 2025

[2] [3]

Accessed: 2024-06-08

Big tech’s latest obsession is finding enough energy. Accessed: 2024-06-08. Common Crawl. Common crawl: Open data for web research. DeepSeek-AI

work page 2024

[3] [4]

arXiv preprint arXiv:2401.02954

Deepseek llm: Scaling open- source language models with longtermism. arXiv preprint arXiv:2401.02954. Hugging Face. Hugging face - the ai community build- ing the future. Hugging Face

arXiv

[4] [8]

https://huggingface

C4: Colos- sal clean crawled corpus. https://huggingface. co/datasets/allenai/c4. Accessed: 2025-04-11. Phillip Rust, Jonas Pfeiffer, Ivan Vuli´c, Sebastian Ruder, and Iryna Gurevych

work page 2025

[5] [9]

Preprint, arXiv:2012.15613

How good is your tok- enizer? on the monolingual performance of multilin- gual language models. Preprint, arXiv:2012.15613. Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Berg- eron, Jeremy Kepner, Devesh Tiwari, and Vijay Gade- pally

arXiv 2012

[6] [11]

arXiv preprint arXiv:2403.20306

Towards greener llms: Bringing energy-efficiency to the forefront of llm inference. arXiv preprint arXiv:2403.20306. Changhan Wang, Kyunghyun Cho, and Jiatao Gu

arXiv

[7] [13]

Preprint, arXiv:2211.05100

Bloom: A 176b-parameter open-access multilingual language model. Preprint, arXiv:2211.05100. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric. P Xing, Joseph E. Gonza- lez, Ion Stoica, and Hao Zhang

arXiv

[8] [14]

Preprint, arXiv:2309.11998

Lmsys-chat- 1m: A large-scale real-world llm conversation dataset. Preprint, arXiv:2309.11998

arXiv