Pith. sign in

REVIEW 3 major objections 6 minor 50 references

Thunder-Tok: Minimizing Tokens per Word in Tokenizing Korean Texts for Generative Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A Korean tokenizer that cuts tokens per word by about 10 percent without lowering model accuracy.

desk verdict Genuinely new Korean tokenizer with a careful ablation, but the '10% token savings' headline is an artifact of averaging small benchmarks; on real Korean text the saving is ~4%. read the letter →

arxiv 2506.15138 v1 pith:5OJD4KDB submitted 2025-06-18 cs.CL cs.AI

Gyeongje Cho , Yeonkyoun So , Chanwoo Park , Sangmin Lee , Sungmok Jung , Jaejin Lee This is my paper · ORCID
classification cs.CLcs.AI MSC 68T50
keywords Koreantokenizationtokenfertilitypre-tokenizationbranchingentropysubwordBytePairEncodingUnigramtokenizergenerativelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Thunder-Tok is a Korean tokenizer built to make each word cost fewer tokens, and the paper's central claim is that it lowers tokens per word by about 10% relative to Byte Pair Encoding at a 128K vocabulary without reducing model quality. The savings come from pre-tokenization rules that keep whitespace-connected Korean words and sentence-final punctuation as single chunks, a seed vocabulary restricted to linguistically plausible boundaries, and a branching-entropy scoring step that favors tokens appearing in varied sentence contexts. If the claim holds, Korean generative models would need roughly 10% fewer inference steps and 2–4% fewer training tokens to reach the same or slightly better zero-shot accuracy on common Korean benchmarks. The paper reports the effect at 360M and 1.5B parameter scales, with the largest gains on Korean LAMBADA.

What carries the argument

The load-bearing mechanism is a three-stage vocabulary construction: a rule-based pre-tokenizer whose pattern `(?: ?\p{L}+)+(?:[.?!])?` forms multi-word and punctuation-attached chunks; a seed-vocabulary filter that admits only substrings which are concatenations of the next-lower linguistic unit (whole syllables within words, whole words within phrases); and a pruning loop that scores each token as BE(x)×P(x), where branching entropy BE(x) = -Σ_y P(y|x) log P(y|x) is computed over all sentences y using EM-derived segmentation probabilities from the Unigram tokenizer. The pruning loop removes the lowest-scoring m% tokens repeatedly until the target vocabulary size is reached. These three parts together determine which long, cross-space tokens survive, and that is what produces the fertility reduction without creating the semantically incoherent tokens that earlier work found harmful.

What would settle it

Train a 360M-parameter LLaMA3-style model on the same Korean corpus for one epoch at 128K vocabulary with Thunder-Tok and with BPE, then compare average tokens per word on the four benchmarks and mean zero-shot accuracy; the paper's claim fails if the fertility gap is meaningfully below 10% or Thunder-Tok's mean accuracy falls below BPE's by more than about one point.

Watch

Extended reading notes

Core claim

The paper's discovery is that token fertility in Korean can be cut without an accuracy penalty by changing three places in the tokenizer pipeline. First, pre-tokenization merges what BPE would keep separate: the regular expression `(?: ?\p{L}+)+(?:[.?!])?` treats whitespace-adjacent Korean letter sequences and a following sentence-final punctuation mark as one chunk, exploiting Korean's optional spacing and agglutinative endings. Second, the seed vocabulary only admits substrings that split cleanly at linguistic unit boundaries (e.g., a substring of a word must be a sequence of whole syllables), blocking incoherent fragments like "어사" from "영어사전". Third, tokens are scored by branching entropy computed over whole sentences rather than adjacent characters, using the Unigram tokenizer's EM estimates, multiplied by token probability; low-scoring tokens are pruned to reach the target vocabulary size. With these components, Thunder-Tok reaches average fertility 1.370 across HPLT v2.0 and four benchmarks versus 1.509 for BPE, and mean zero-shot accuracy 64.02 versus 63.06 at 128K vocabulary in the 360M model, with a similar pattern at 1.5B.

Load-bearing premise

The claim rests on the assumption that the regular-expression pre-tokenizer—merging whitespace-adjacent Korean letter runs and sentence-final punctuation into single chunks—always produces semantically coherent units, since the paper's own ablation shows accuracy drops when this pattern is removed or the seed filter is absent.

Editorial extensions

If this is right

  • Inference cost for Korean text falls roughly in proportion to token count: at about 10% lower fertility, a generation that previously took 100 token steps takes about 90, with accuracy unchanged or slightly better.
  • Training on the same Korean corpus requires 2–4% fewer tokens than BPE and 7–20% fewer than Unigram or BPE-Mecab, so a fixed compute budget could cover more data or a larger model.
  • The fertility advantage transfers to out-of-distribution text: on the four benchmark sets Thunder-Tok's fertility is 4–15% lower than BPE's, so the efficiency gain is not an artifact of the training corpus.
  • The seed-vocabulary filter is the component protecting accuracy: removing it (using GPT-2 style pre-tokenization or unfiltered N-gram seeds) drops average accuracy from 64.02 to as low as 62.46, while the branching-entropy scorer mainly buys fertility (from 1.418 to 1.370) rather than accuracy.
  • At larger scale the pattern holds: the 1.5B model with Thunder-Tok uses 33.0B training tokens versus 34.2B for BPE and posts the highest mean accuracy (67.26 vs 66.39).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Applied to autoregressive decoding, the 10% token reduction should translate into roughly 10% fewer forward passes and thus about 10% lower latency and cost, assuming the vocabulary does not change model width; the paper states this as an inference-speed improvement but does not measure wall-clock time directly.
  • The same three-stage recipe—rule-based chunking, linguistically constrained seed vocabulary, and entropy scoring—could be adapted to other languages where whitespace does not mark morpheme boundaries, such as Japanese or Turkish, but the pre-tokenization regex and the allowable-boundary filter would need re-deriving for each language.
  • A testable extension is to apply only the pre-tokenization pattern to a standard BPE tokenizer without retraining the seed vocabulary; the ablation suggests the pattern alone reduces fertility but hurts accuracy, so isolating the filter's protective effect would confirm the mechanism.
  • The sentence-level branching entropy definition is a general scoring criterion that could be evaluated on other morphologically rich languages or used to audit existing vocabularies for context-diverse tokens.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces Thunder-Tok, a Korean tokenizer designed to reduce token fertility (tokens per word) while maintaining downstream model performance. The method combines three components: (1) a rule-based pre-tokenization step that merges whitespace-connected Hangul sequences and attaches sentence-final punctuation to the preceding word; (2) a 'language-aware' seed vocabulary that only admits substring tokens that can be perfectly divided into lower-level linguistic units; and (3) a branching-entropy-based scoring function used to prune the vocabulary down to a target size. The authors evaluate Thunder-Tok against BPE, Unigram, and BPE-Mecab using LLaMA-style models of 360M and 1.5B parameters, measuring both fertility on the HPLT v2.0 corpus and four Korean benchmarks, and zero-shot accuracy on KoBEST HellaSwag, KoBEST COPA, Korean ARC-e, and Korean LAMBADA. The paper claims an approximately 10% fertility reduction over BPE with no loss in downstream performance.

Significance. If the headline claim held, the paper would offer a simple, linguistically motivated way to cut Korean token counts by about 10%, which is a practically meaningful inference-speed improvement with no accuracy cost. The core ideas—restricting the token vocabulary to morphologically plausible units and using branching entropy as a score—are interesting and worth testing in other morphologically rich languages. The paper is also careful to compare against strong baselines and includes an ablation study. However, the central quantitative claim is overstated: on the training corpus itself (HPLT v2.0) the fertility reduction is only 3.9%, and the training-token counts in Table 4 show a 3.5–3.7% reduction. The benchmark-specific reductions (5–14%) are inflated by short, templated prompts and answer choices. The downstream performance comparisons are based on a single run with no error bars, so the claim that performance is 'not compromised' is not statistically supported. The contribution is real but more modest than advertised.

major comments (3)
  1. [Abstract, Section 4.2, Table 3, Table 4] The headline claim of an approximately 10% reduction in token count and inference speed is not supported by the paper's own corpus-level numbers. In Table 3, the fertility reduction over BPE on HPLT v2.0 is 3.9% (1.638 vs 1.704), and the unweighted average of the five per-dataset reductions is about 8.6%, not 10%. More importantly, Table 4 shows that Thunder-Tok uses only 3.7% fewer training tokens than BPE at 128K/360M (13.0B vs 13.5B) and 3.5% fewer at 128K/1.5B (33.0B vs 34.2B). The larger benchmark-specific reductions likely arise from short, templated prompts and answer strings that the regex merges, not from general Korean text. The abstract and conclusion equate the benchmark average with a 10% reduction in tokens and inference speed, which overstates the practical benefit by approximately a factor of 2.5. The authors should report a corpus-weighted fertility reduction and revise the abstract and conclusion accordingly.
  2. [Section 4.3, Table 4] All downstream results are from a single run, with no error bars, confidence intervals, or significance tests. The differences between Thunder-Tok and BPE-Mecab are as small as 0.1 percentage points in the 1.5B setting (67.26 vs 67.17), and the test sets contain only 500–2,376 examples. Without variance estimates, the claim that Thunder-Tok 'does not compromise performance' is not rigorously established. The fertility measurements are deterministic and not affected by this issue, but the performance comparisons are load-bearing for the central claim. Please provide multiple seeds or a statistical test, or at least explicitly acknowledge that the accuracy differences are within the range of single-run noise.
  3. [Algorithm 1, Section 3.3] The removal threshold percentage m in Algorithm 1 is never specified in the manuscript. Since the final vocabulary and hence all experimental results depend on this hyperparameter, the experiments are not fully reproducible without knowing m. Additionally, the branching-entropy computation relies on the EM algorithm from the Unigram tokenizer, but the paper does not describe the initialization, number of iterations, or convergence criteria used. Please report these values.
minor comments (6)
  1. [Section 3.1] The example of preserving '2 cups' is not supported by the implemented regular expression `(?: ?\p{L}+)+(?:[.?!])?`, which matches only letters and does not include digits. Please either change the example or extend the regex to handle numeric-word combinations.
  2. [Algorithm 1] There is a typo: 'Caculate' should be 'Calculate'. Also, 'whitespaces' is used inconsistently (sometimes 'whitespace').
  3. [Table 3] The column header 'Average' is placed in a way that may be misread as belonging to 'Kobest HellaSwag'. Please reformat the table so the average column is clearly separate.
  4. [Section 4.2] The statement that 'replacing existing tokenizers ... could reduce inference costs by more than 10%' is not supported by the data, since the corpus-level reduction is about 4%. Please align this sentence with the actual measurements.
  5. [Appendix B, Table 10] There are minor typos in the appendix: 'Reasong' should be 'Reasoning', 'Hostpital' should be 'Hospital', and 'Becuase' should be 'Because'.
  6. [Figure 2] The Venn diagram percentages do not sum to 100 and are not clearly defined. Please clarify whether the numbers represent percentages of each individual vocabulary, of the union, or of some other base.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the fertility and downstream claims rest on direct measurements against external benchmarks.

full rationale

The paper's central claim, that Thunder-Tok reduces token fertility by roughly 10% versus BPE at a fixed 128K vocabulary without sacrificing downstream accuracy, is supported by direct measurement of trained tokenizers on the HPLT corpus and benchmark datasets, not by a quantity fitted from those benchmarks. The tokenizer construction in Sections 3.1-3.3 fixes the pre-tokenization rules, seed vocabulary, and branching-entropy scoring before the evaluations in Sections 4.2-4.3 are run; no parameter is optimized to reproduce the reported fertility numbers. The HPLT v2.0 fertility figure is in-sample in the sense that the tokenizer was built from 1M documents from that corpus, but the benchmark rows in Table 3 are measured on separately reformatted evaluation text and independently confirm the same direction and approximate magnitude (Thunder-Tok 1.254-1.431 vs BPE 1.402-1.510). The ablation study selects among configurations using the same evaluation protocol, which is a test-set selection concern rather than a circular derivation, because the reported numbers are still measurements and not implied by construction. The only citation that could appear in-group, Thunder Research Group (2025) for the Korean ARC-easy and LAMBADA datasets, is used as evaluation data rather than as a proof step; moreover, KoBEST HellaSwag and KoBEST COPA come from an independent source, so no load-bearing self-citation is present. The unweighted averaging that produces the 'approximately 10%' headline is a reporting and interpretation concern, not a circularity one: the underlying measurements are transparent and externally checkable. No step of the paper defines Thunder-Tok's outputs in terms of the claimed results or imports a uniqueness theorem to force its design choice.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the representativeness of the training and evaluation corpora, the standard Unigram EM estimation, and the design assumption that linguistically constrained tokens preserve semantics. No external entity or fitted physical constant is introduced; the tokenizer's hyperparameters are standard.

free parameters (1)
  • Removal threshold m in Algorithm 1
    The percentage of lowest-scoring tokens pruned in each iteration of vocabulary construction is a hyperparameter of the central method; its value is not reported, so the exact fertility result cannot be reproduced.
assumptions (4)
  • domain assumption HPLT v2.0 Korean corpus is a representative sample of Korean text for training the tokenizer and the language models.
    The tokenizer vocabulary and the LMs are trained on 1M sampled documents from HPLT v2.0; if this sample is unrepresentative, downstream conclusions may not generalize. Invoked in Section 4.1.
  • standard math The EM algorithm used by the Unigram tokenizer converges to reliable token probabilities P(x) and segmentation probabilities.
    Equation (4) and Algorithm 1 rely on P(x|y) and P(x) computed via EM; the paper gives no convergence details. This is a standard result from Kudo (2018).
  • domain assumption A single epoch of training on the same raw data with the same model architecture is a fair comparison across tokenizers.
    Because tokenizers have different fertility, one epoch yields different numbers of training steps; the paper treats this as the fair efficiency comparison in Section 4.3, but it confounds data replication with optimizer steps.
  • domain assumption The evaluation benchmarks (KoBEST, Korean ARC-easy, Korean LAMBADA) are reliable and independent measures of Korean LM performance.
    Two of the four benchmark sources are attributed to 'Thunder Research Group', which is not identified as external; independence is not established in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Thunder-Tok: Minimizing Tokens per Word in Tokenizing Korean Texts for Generative Language Models." pith.science (2026). https://pith.science/paper/5OJD4KDB

@misc{pith2026250615138,
  author       = {Pith},
  title        = {Pith review of: Thunder-Tok: Minimizing Tokens per Word in Tokenizing Korean Texts for Generative Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5OJD4KDB}},
  note         = {Machine review of arXiv:2506.15138}
}
read the original abstract

This paper introduces Thunder-Tok, a new Korean tokenizer designed to reduce token fertility without compromising model performance. Our approach uses a rule-based pre-tokenization method that aligns with the linguistic structure of the Korean language. We also create a seed vocabulary containing tokens that resemble linguistic units and employ a branching entropy-based selection algorithm. These techniques increase the average token length, thus lowering fertility while preserving linguistic information. Experimental results indicate that Thunder-Tok reduces fertility by approximately 10% (i.e., reduces the number of tokens by 10%, improving the inference speed by 10%) compared to BPE without compromising performance across various downstream tasks. These findings demonstrate that our linguistically informed approach is effective and practical for designing efficient tokenizers for language models.

Figures

Figures reproduced from arXiv: 2506.15138 by the authors.

Figure 1
Figure 1. Distribution of special-form tokens (multi [PITH_FULL_IMAGE:figures/full_fig_p013_1.png] view at source ↗
Figure 2
Figure 2. Venn diagrams comparing 128k token vocabu [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 12 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max L \"u bbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, et al. 2024. Tokenizer choice for llm training: Negligible or crucial? In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3907--3924

    work page 2024

  3. [3]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  4. [4]

    Gunjan Balde, Soumyadeep Roy, Mainack Mondal, and Niloy Ganguly. 2024. Adaptive bpe tokenization for enhanced vocabulary adaptation in finetuning pretrained language models. arXiv preprint arXiv:2410.03258

    work page Pith review arXiv 2024

  5. [5]

    Laurie Burchell, Ona de Gibert, Nikolay Arefyev, Mikko Aulamo, Marta Ba \ n \'o n, Pinzhen Chen, Mariia Fedorova, Liane Guillou, Barry Haddow, Jan Haji c , et al. 2025. An expanded massive multilingual dataset for high-performance language technologies. arXiv preprint arXiv:2503.10267

    arXiv 2025

  6. [6]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    arXiv 2018

  7. [7]

    Gautier Dagan, Gabriel Synnaeve, and Baptiste Roziere. 2024. Getting the most out of your tokenizer for pre-training and domain adaptation. arXiv preprint arXiv:2402.01035

    arXiv 2024

  8. [8]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems, 35:30318--30332

    2022

Show all 50 references
  1. [9]

    Takuro Fujii, Koki Shibata, Atsuki Yamaguchi, Terufumi Morishita, and Yasuhiro Sogawa. 2023. How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in japanese. arXiv preprint arXiv:2306.09572

    2023 arXiv

  2. [10]

    In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. 2024. Prompt cache: Modular attention reuse for low-latency inference. Proceedings of Machine Learning and Systems, 6:325--338

    2024

  3. [11]

    Omer Goldman, Avi Caciularu, Matan Eyal, Kris Cao, Idan Szpektor, and Reut Tsarfaty. 2024. Unpacking tokenization: Evaluating text compression and its correlation with model performance. arXiv preprint arXiv:2403.06265

    2024 arXiv

  4. [12]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  5. [13]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  6. [14]

    Valentin Hofmann, Hinrich Schuetze, and Janet B Pierrehumbert. 2022. An embarrassingly simple method to mitigate undesirable properties of pretrained language model tokenizers. Association for Computational Linguistics

    2022

  7. [15]

    Myeongjun Jang, Dohyung Kim, Deuk Sin Kwon, and Eric Davis. 2022. Kobest: Korean balanced evaluation of significant tasks. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3697--3708

    2022

  8. [16]

    Zhihui Jin and Kumiko Tanaka-Ishii. 2006. Unsupervised segmentation of chinese text by use of branching entropy. In Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, pages 428--435

    2006

  9. [17]

    Boseop Kim, HyoungSeok Kim, Sang-Woo Lee, Gichang Lee, Donghyun Kwak, Dong Hyeon Jeon, Sunghyun Park, Sungju Kim, Seonhoon Kim, Dongpil Seo, et al. 2021. What changes can large-scale language models bring? intensive study on hyperclova: Billions-scale korean generative pretrai...

    2021 arXiv

  10. [18]

    Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. arXiv preprint arXiv:1804.10959

    2018 arXiv

  11. [19]

    Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226

    2018 arXiv

  12. [20]

    Braulio Blanco Lambruschini, Patricia Becerra-Sanchez, Mats Brorsson, and Maciej Zurad. 2023. Reducing tokenizer’s tokens per word ratio in financial domain with t-mufin bert tokenizer. In Proceedings of the Fifth Workshop on Financial Technology and Natural Language Processin...

    2023

  13. [21]

    Percy Liang and Dan Klein. 2007. https://cs.stanford.edu/ pliang/papers/tutorial-acl2007-talk.pdf Bayesian nonparametric structured models . Tutorial presented at the 45th Annual Meeting of the Association for Computational Linguistics (ACL 2007)

    2007

  14. [22]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87--100

    2024

  15. [23]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  16. [24]

    Ilya Loshchilov and Frank Hutter. 2016. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983

    2016 arXiv

  17. [25]

    Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101

    2017 arXiv

  18. [26]

    Meta Platforms

    Inc. Meta Platforms. 2025. Meta llama 4: The future of multimodal ai. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5208228. Accessed: 2025-05-14

    2025

  19. [27]

    Sabrina J Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gall \'e , Arun Raja, Chenglei Si, Wilson Y Lee, Beno \^ t Sagot, et al. 2021. Between words and characters: A brief history of open-vocabulary modeling and tokenization in nlp. arXiv preprin...

    2021 arXiv

  20. [28]

    Kyubyong Park, Joohong Lee, Seongbo Jang, and Dawoon Jung. 2020. An empirical study of tokenization strategies for various korean nlp tasks. arXiv preprint arXiv:2010.02534

    2020 arXiv

  21. [29]

    Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. 2023. Language model tokenizers introduce unfairness between languages. Advances in neural information processing systems, 36:36963--36990

    2023

  22. [30]

    Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2019. Bpe-dropout: Simple and effective subword regularization. arXiv preprint arXiv:1910.13267

    2019 arXiv

  23. [31]

    Penghui Qi, Xinyi Wan, Guangxing Huang, and Min Lin. 2023. Zero bubble pipeline parallelism. arXiv preprint arXiv:2401.10241

    2023 arXiv

  24. [32]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  25. [33]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1--16. IEEE

    2020

  26. [34]

    Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical formalizations of commonsense reasoning, pages 90--95

    2011

  27. [35]

    Phillip Rust, Jonas Pfeiffer, Ivan Vuli \'c , Sebastian Ruder, and Iryna Gurevych. 2020. How good is your tokenizer? on the monolingual performance of multilingual language models. arXiv preprint arXiv:2012.15613

    2020 arXiv

  28. [36]

    Craig W Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, and Chris Tanner. 2024. Tokenization is more than compression. arXiv preprint arXiv:2402.18376

    2024 arXiv

  29. [37]

    Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5149--5152. IEEE

    2012

  30. [38]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909

    2015 arXiv

  31. [39]

    Claude E. Shannon. 1948. https://doi.org/10.1002/j.1538-7305.1948.tb00917.x A mathematical theory of communication . Bell System Technical Journal, 27(3):379--423

    1948

  32. [40]

    Yuchen Shen and Barnab \'a s P \'o czos. 2024. Graphbpe: Molecular graphs meet byte-pair encoding. arXiv preprint arXiv:2407.19039

    2024 arXiv

  33. [41]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118

    2024 arXiv

  34. [42]

    Thunder Research Group . 2025. Korean benchmarks. https://github.com/mcrl/korean_benchmarks. GitHub repository

    2025

  35. [43]

    Cagri Toraman, Eyup Halit Yilmaz, Furkan S ahinu c , and Oguzhan Ozcelik. 2023. Impact of tokenization on language models: An analysis for turkish. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(4):1--21

    2023

  36. [44]

    David Vilar and Marcello Federico. 2021. A statistical extension of byte-pair encoding. In Proceedings of the 18th International Conference on Spoken Language Translation (IWSLT 2021), pages 263--275

    2021

  37. [45]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical met...

    2020

  38. [46]

    Shaked Yehezkel and Yuval Pinter. 2022. Incorporating context into subword vocabularies. arXiv preprint arXiv:2210.07095

    2022 arXiv

  39. [47]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830

    2019 arXiv

  40. [48]

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody\_Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2023. Efficiently programming large language models using sglang

    2023

  41. [49]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  42. [50]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.