Pith. sign in

REVIEW 5 major objections 4 minor 31 references

Train It and Forget It: Merge Lists are Unnecessary for BPE Inference in Language Models

T0 review · 5 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that the BPE merge list can be dropped at inference: non-targeted, merge-list-free tokenization has little effect on downstream language model performance.

desk verdict A genuinely useful question about dropping BPE merge lists at inference, but the copy I have is unreadable beyond the abstract, so the empirical claim is unverifiable. read the letter →

arxiv 2508.06621 v1 pith:JF66CJ74 submitted 2025-08-08 cs.CL

Tomohiro Sawada , Kartik Goyal This is my paper
classification cs.CL
keywords BytePairEncodingtokenizationmergelistlanguagemodelinferencedownstreamevaluationprivacy-preservingtextcompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Byte-Pair Encoding (BPE) tokenizes text by pairing a learned token vocabulary with a detailed merge list that records the order in which character pairs are fused. This paper tests whether the merge list is actually needed when a trained language model reads new text. The authors compare targeted corruptions of the merge list (random orders, deletions, truncations) against non-targeted inference algorithms that ignore the list entirely and simply compress text greedily or exactly. Across question-answering benchmarks, machine translation, and open-ended generation, they find that ignoring the merge list altogether costs very little, while systematically corrupting it causes large drops. The paper concludes that a tokenizer can keep its vocabulary but drop its merge list at inference time, simplifying deployment and reducing the surface for training-data extraction attacks.

What carries the argument

The key object is the distinction between two ways of producing BPE tokens at inference: targeted deviation from the learned merge list versus non-targeted, merge-list-free compression. The load-bearing mechanism is the non-targeted encoder, which uses only the token vocabulary and compresses text either greedily or by finding an exactly shortest token sequence. Because these encoders never see the merge list, their near-baseline results are the evidence that the merge list is optional.

What would settle it

Evaluate a language model at several scales (e.g., about 100M to about 10B parameters) on tasks that require exact surface-form reproduction, such as code generation with string literals or spelling-sensitive QA, using greedy and exact merge-list-free tokenizers; if any scale or task shows a collapse of more than a few points compared with the standard BPE baseline, the claim of minimal impact would fail.

Watch

Extended reading notes

Core claim

The central claim is that the BPE merge list is not a load-bearing component during inference. The paper's evidence comes from two families of alternative encoders: 'targeted' schemes that deviate from the learned merge order in specific ways, and 'non-targeted' schemes that operate without any merge list, producing segmentations by greedy or exact text compression. On three task families (accuracy-based QA, machine translation, and open-ended generation), the non-targeted schemes land close to the standard BPE baseline, while targeted deviations degrade performance markedly. The authors interpret this as showing that what matters for downstream performance is a consistent, reasonably compre

Load-bearing premise

The results transfer across models, tasks, and vocabularies: the small drop from merge-list-free inference is not an artifact of the specific benchmarks, model sizes, or tokenizers tested.

Editorial extensions

If this is right

  • Tokenizers can ship with a vocabulary but no merge list, and inference-time encoding can use a simple greedy or exact compressor with minimal quality loss.
  • Deleting the merge list removes one known information-leak channel for training-data extraction without forcing a model retrain.
  • The sharp contrast between targeted corruption and merge-list-free inference implies that a corrupted or partially leaked merge list is best handled by ignoring it rather than repairing it.
  • Because non-targeted inference is deterministic and vocabulary-only, it makes tokenization easier to reproduce and verify across different software stacks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the finding transfers to other domains, tokenizer training could itself be simplified: learn the vocabulary and discard the merge-order objective, cutting memory and compute during tokenizer construction.
  • The attack-surface reduction could be quantified directly by measuring how much membership-inference or extraction success drops when the merge list is absent; the paper motivates but does not measure this.
  • The near-zero gap may reflect that transformer models read token IDs rather than characters, so any deterministic segmentation is acceptable as long as the model saw a consistent distribution at training time; a testable consequence is that fine-tuning with merge-list-free tokenization should close the residual gap entirely.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper claims that BPE tokenization can be used for language model inference without the learned merge list: instead of following the training-time merge order, one can segment text greedily or optimally over the learned vocabulary. The abstract reports experiments across QA benchmarks, machine translation, and open-ended generation, finding that targeted merge-list deviations (e.g., random orders, truncations) degrade performance, but non-targeted merge-list-free inference has minimal downstream impact, 'often much smaller than expected.' The paper concludes that merge lists are unnecessary for inference, enabling simpler and privacy-preserving tokenization. However, the provided full text is almost entirely corrupted (mojibake), so the experimental details, tables, and analysis are not readable, and the central empirical claim cannot be independently verified.

Significance. If the empirical claim holds, the paper would be a significant practical result: language model serving would not need to store or apply BPE merge lists, simplifying tokenization and removing a potential privacy channel exposed by merge-list membership. The contribution is entirely empirical, so the strength of the claim depends on the breadth of models/tasks, the magnitude of the reported degradations, and the statistical reliability of the measurements. The paper does not appear to provide formal proofs, code, or data in the visible fragments. Because the full text is unreadable, I cannot assess whether the experiments are adequately controlled (e.g., baseline comparison, number of seeds, variance) or whether the open-ended generation metric is tokenizer-independent. The potential significance is real, but the evidence as presented is inaccessible.

major comments (5)
  1. [Full text (all sections)] The manuscript text as supplied is corrupted throughout with encoding errors; none of the experimental setup, table numbers, or numerical results are readable. The central claim is empirical, so this missing evidence is load-bearing. I cannot verify whether the experiments were performed, the effect sizes, the baselines, or the statistical treatment. The authors must provide a readable, complete version with all tables and figures.
  2. [Abstract] The abstract states that non-targeted merge-list-free inference yields 'minimal impact ... often much smaller than expected,' but no quantitative threshold is given. The title asserts merge lists are 'unnecessary,' which is stronger than the abstract's hedged claim. The paper should define 'minimal' operationally (e.g., within a fixed margin of baseline accuracy, or a maximum relative drop) and report per-model, per-task effect sizes with confidence intervals.
  3. [Abstract (tasks and models)] The abstract does not report how many models, seeds, or tasks were used, nor the variance across them. The claim that merge-list-free inference is generally safe requires showing that no tested model-task cell has a large degradation. The authors should report a full per-cell table of results and explicitly discuss any outliers; otherwise the recommendation to discard merge lists is not justified for arbitrary deployments.
  4. [Abstract (open-ended generation)] The abstract lists open-ended generation as one of the evaluated tasks but does not state the evaluation metric. If token-level perplexity is used, then comparing models with different tokenizers is invalid because token counts and token boundaries differ; a lower token-level perplexity may simply reflect a different tokenization. The metric must be specified and must be tokenizer-independent (e.g., character-level perplexity or downstream human/LLM-judged quality).
  5. [Non-targeted BPE inference algorithms] The abstract describes greedy or exact compression over the learned vocabulary, but does not specify how ties are broken or how the exact optimization is defined. Tie-breaking rules can change segmentation substantially, and the success of the method may depend on the specific choice. The paper must give the exact algorithms, including tie-breaking, and ideally test sensitivity to those choices.
minor comments (4)
  1. [Title] The title 'Train It and Forget It: Merge Lists are Unnecessary for BPE Inference' overclaims relative to the abstract's 'often much smaller than expected.' Consider softening to 'can be omitted with minimal degradation in tested settings' or providing a quantitative bound that supports 'unnecessary.'
  2. [Abstract] The phrase 'These findings pave way for simpler' should be 'pave the way.'
  3. [Full text (available fragments)] The readable fragments contain tables with row/column labels such as 'Algorithm', 'Average', 'Quality', 'Factuality', but no numeric values could be extracted. Please ensure the final version is not corrupted and that all tables are legible with clear captions.
  4. [Reproducibility] No code or data repository is mentioned in the visible abstract. Given the empirical nature of the paper, releasing code and tokenizer implementations would strengthen the contribution and allow checking of the merge-list-free inference algorithms.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central claim is an empirical comparison, not a derivation from its own inputs.

full rationale

The paper's central claim—that non-targeted, merge-list-free BPE inference algorithms have minimal downstream impact—is an empirical claim evaluated by comparing inference algorithms on QA, MT, and open-ended generation benchmarks. The available text (abstract and structural fragments) shows no equation or definition that makes the outcome equal to the input by construction. The non-targeted greedy/exact algorithms are not fitted to the target metrics; they are fixed inference procedures over the learned vocabulary. The shared learned vocabulary is part of the experimental design (train tokenizer, then discard the merge list), not a circular construction, because the comparison is between merge-list-dependent and merge-list-free inference using the same vocabulary. No load-bearing self-citation, uniqueness theorem, or ansatz-smuggling citation is visible in the readable portions. The abstract's hedge ('often much smaller than expected') further indicates a qualitative empirical finding rather than a forced result. Even if the effect fails to generalize to other models or tasks, that would be a correctness or external-validity concern, not circularity. Therefore no circular step can be identified from the available manuscript text.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

No free parameters or invented entities are stated in the abstract. The main unstated assumptions are that the experimental setup (fixed model and vocabulary) is held constant across conditions, and that the chosen benchmarks are a representative sample of downstream use.

assumptions (2)
  • domain assumption Models were pretrained with standard BPE that uses a merge list, and the vocabulary is unchanged during inference.
    The comparison hinges on changing only the inference-time segmentation algorithm while keeping the model and vocabulary fixed, as stated in the abstract.
  • domain assumption The selected QA, MT, and generation benchmarks are representative of overall LM performance.
    The abstract's conclusion of 'minimal impact' generalizes from these tasks to other LLM uses.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Train It and Forget It: Merge Lists are Unnecessary for BPE Inference in Language Models." pith.science (2026). https://pith.science/paper/JF66CJ74

@misc{pith2026250806621,
  author       = {Pith},
  title        = {Pith review of: Train It and Forget It: Merge Lists are Unnecessary for BPE Inference in Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JF66CJ74}},
  note         = {Machine review of arXiv:2508.06621}
}
read the original abstract

Standard Byte-Pair Encoding (BPE) tokenization compresses text by pairing a learned token vocabulary with a detailed merge list. Recent work has shown that this merge list exposes a potential attack surface for extracting information about language model's training data. In this paper, we explore the downstream impact of BPE inference algorithms that do not rely on this merge list at all, and hence differ from the encoding process during BPE training. To address this question, we investigate two broad classes of BPE inference schemes that differ from BPE application during training: a) targeted deviation from merge-lists including random merge orders, and various corruptions of merge list involving deletion/truncation, and b) non-targeted BPE inference algorithms that do not depend on the merge list but focus on compressing the text either greedily or exactly. Extensive experiments across diverse language modeling tasks like accuracy-based QA benchmarks, machine translation, and open-ended generation reveal that while targeted deviation from the merge lists exhibits significant degradation in language model performance, the non-targeted merge-list-free inference algorithms result in minimal impact on downstream performance that is often much smaller than expected. These findings pave way for simpler and potentially more privacy-preserving tokenization schemes that do not catastrophically compromise model performance.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 7 canonical work pages

  1. [1]

    Satanjeev Banerjee and Alon Lavie. 2005. https://aclanthology.org/W05-0909/ METEOR : An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments . In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , pages 65--72, Ann Arbor, Michigan. Association for Compu...

    work page 2005

  2. [2]

    Lisa Beinborn and Yuval Pinter. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.272 Analyzing cognitive plausibility of subword tokenization . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4478--4486, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.emnlp-main.272 2023

  3. [3]

    Eric Bigelow, Ari Holtzman, Hidenori Tanaka, and Tomer Ullman. 2024. https://arxiv.org/abs/2412.07961 Forking paths in neural text generation . Preprint, arXiv:2412.07961

    arXiv 2024

  4. [4]

    Kaj Bostrom and Greg Durrett. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.414 Byte pair encoding is suboptimal for language model pretraining . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4617--4624, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.findings-emnlp.414 2020

  5. [5]

    Kris Cao and Laura Rimell. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.161 You should evaluate your language model on marginal likelihood over tokenisations . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2104--2114, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics

    doi:10.18653/v1/2021.emnlp-main.161 2021

  6. [6]

    Qi Cao, Takeshi Kojima, Yutaka Matsuo, and Yusuke Iwasawa. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.550 Unnatural error correction: GPT -4 can almost perfectly handle unnatural scrambled text . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8898--8913, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.emnlp-main.550 2023

  7. [7]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. https://doi.org/10.48550/arXiv.1803.05457 Think you have Solved Question Answering ? Try ARC , the AI2 Reasoning Challenge . arXiv preprint. ArXiv:1803.05457 [cs] version: 1

    arXiv doi:10.48550/arxiv.1803.05457 2018

  8. [8]

    Philip Gage. 1994. https://www.semanticscholar.org/paper/A-new-algorithm-for-data-compression-Gage/1aa9c0045f1fe8c79cce03c7c14ef4b4643a21f8 A new algorithm for data compression . The C Users Journal archive

    work page 1994

Show all 31 references
  1. [9]

    Juan Luis Gastaldi, John Terilla, Luca Malagutti, Brian DuSell, Tim Vieira, and Ryan Cotterell. 2025. https://openreview.net/forum?id=B5iOSxM2I0 The foundations of tokenization: Statistical and computational concerns . In The Thirteenth International Conference on Learning Rep...

    2025

  2. [10]

    Jonas Geiping, Alex Stein, Manli Shu, Khalid Saifullah, Yuxin Wen, and Tom Goldstein. 2024. https://arxiv.org/abs/2402.14020 Coercing llms to do and reveal (almost) anything . Preprint, arXiv:2402.14020

    2024 arXiv

  3. [11]

    Omer Goldman, Avi Caciularu, Matan Eyal, Kris Cao, Idan Szpektor, and Reut Tsarfaty. 2024 a . http://arxiv.org/abs/2403.06265 Unpacking Tokenization : Evaluating Text Compression and its Correlation with Model Performance . arXiv preprint. ArXiv:2403.06265 [cs]

    2024 arXiv

  4. [12]

    Omer Goldman, Avi Caciularu, Matan Eyal, Kris Cao, Idan Szpektor, and Reut Tsarfaty. 2024 b . https://doi.org/10.18653/v1/2024.findings-acl.134 Unpacking tokenization: Evaluating text compression and its correlation with model performance . In Findings of the Association for C...

    2024 doi

  5. [13]

    Jonathan Hayase, Alisa Liu, Yejin Choi, Sewoong Oh, and Noah A Smith. 2024. Data Mixture Inference : What do BPE Tokenizers Reveal about their Training Data ?

    2024

  6. [14]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://doi.org/10.48550/arXiv.2009.03300 Measuring Massive Multitask Language Understanding . arXiv preprint. ArXiv:2009.03300 [cs]

    2021 doi arXiv

  7. [15]

    Guy Kaplan, Matanel Oren, Yuval Reif, and Roy Schwartz. 2025. https://openreview.net/forum?id=328vch6tRs From tokens to words: On the inner lexicon of LLM s . In The Thirteenth International Conference on Learning Representations

    2025

  8. [16]

    Taku Kudo and John Richardson. 2018. https://doi.org/10.18653/v1/D18-2012 SentencePiece : A simple and language independent subword tokenizer and detokenizer for Neural Text Processing . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing ...

    2018 doi

  9. [17]

    Sander Land and Max Bartolo. 2024. https://arxiv.org/abs/2405.05417 Fishing for magikarp: Automatically detecting under-trained tokens in large language models . Preprint, arXiv:2405.05417

    2024 arXiv

  10. [18]

    Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020. https://doi.org/10.18653/v1/2020.acl-main.447 S2ORC : The Semantic Scholar Open Research Corpus . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 4969-...

    2020 doi

  11. [19]

    Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y

    Sabrina J. Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y. Lee, Benoît Sagot, and Samson Tan. 2021. https://doi.org/10.48550/arXiv.2112.10508 Between words and characters: A Brief History of Open - Vocabulary...

    2021 doi arXiv

  12. [20]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 Bleu: a Method for Automatic Evaluation of Machine Translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , pages 311--3...

    2002

  13. [21]

    Krishna Pillutla, Lang Liu, John Thickstun, Sean Welleck, Swabha Swayamdipta, Rowan Zellers, Sewoong Oh, Yejin Choi, and Zaid Harchaoui. 2023. https://doi.org/10.48550/arXiv.2212.14578 MAUVE Scores for Generative Models : Theory and Practice . arXiv preprint. ArXiv:2212.14578 [cs]

    2023 doi arXiv

  14. [22]

    Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2020. https://doi.org/10.18653/v1/2020.acl-main.170 BPE -dropout: Simple and effective subword regularization . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1882--1892, O...

    2020 doi

  15. [23]

    Language Models are Unsupervised Multitask Learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners

  16. [24]

    Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.213 COMET : A Neural Framework for MT Evaluation . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing ( EMNLP ) , pages 2685--270...

    2020 doi

  17. [25]

    Jonne Saleva and Constantine Lignos. 2023. https://doi.org/10.18653/v1/2023.insights-1.7 What changes when you randomly choose BPE merge operations? not much. In Proceedings of the Fourth Workshop on Insights from Negative Results in NLP, pages 59--66, Dubrovnik, Croatia. Asso...

    2023 doi

  18. [26]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. https://doi.org/10.18653/v1/P16-1162 Neural machine translation of rare words with subword units . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages ...

    2016 doi

  19. [27]

    Schmidt, Chris Tanner, and Yuval Pinter

    Omri Uzan, Craig W. Schmidt, Chris Tanner, and Yuval Pinter. 2024. https://doi.org/10.48550/arXiv.2403.01289 Greed is All You Need : An Evaluation of Tokenizer Inference Methods . arXiv preprint. ArXiv:2403.01289 [cs]

    2024 doi arXiv

  20. [28]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024 arXiv

  21. [29]

    Vil \'e m Zouhar, Clara Meister, Juan Gastaldi, Li Du, Tim Vieira, Mrinmaya Sachan, and Ryan Cotterell. 2023. https://doi.org/10.18653/v1/2023.findings-acl.38 A formal perspective on byte-pair encoding . In Findings of the Association for Computational Linguistics: ACL 2023, p...

    2023 doi

  22. [30]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  23. [31]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.