{"id":"adf0abb1-701e-494e-a297-d210ce0d6bdb","arxiv_id":"2506.15138","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Thunder-Tok, a Korean tokenizer with grammar-based pre-tokenization and branching-entropy vocabulary selection, cuts tokens per word by about 10% versus BPE while keeping downstream performance comparable.","lead":"This paper introduces Thunder-Tok, a Korean tokenizer that reduces the number of tokens per word by about 10% compared to standard byte-pair encoding. The tokenizer uses Korean grammar-aware pre-tokenization and a linguistically filtered vocabulary, aiming to cut inference cost without hurting model quality.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ~10% token reduction is an unweighted average over small benchmarks; on the HPLT training corpus the reduction is only ~3.9%, so the headline inference-speed claim overstates the practical token saving by about 2.5x.","rationale":"I read the paper in good faith. The tokenizer design is coherent: the pre-tokenization regex, the linguistically constrained seed vocabulary, the branching-entropy scoring, and the ablation study form a plausible argument. The ablation in Table 5 is genuinely informative: it shows that the seed-vocabulary filter is needed to recover the accuracy lost by the aggressive pre-tokenization, which addresses the reader's weakest-assumption concern. The deterministic fertility measurements are solid evidence for a token-count improvement, and the downstream numbers, though single-run, consistently show Thunder-Tok at or above BPE. My main concern is different from the reader's weakest assumption. The abstract's 'approximately 10%' fertility reduction is an unweighted average over five datasets whose sizes differ by orders of magnitude. The paper's own Table 4 training-token counts show only ~3.5-3.7% fewer tokens on the HPLT training corpus, and Table 3 shows 3.9% fertility reduction on HPLT. The 10% figure therefore derives from small evaluation sets with highly structured prompt formats rather than from general Korean text. This is a load-bearing problem for the central efficiency claim, because the abstract explicitly converts the 10% token reduction into a 10% inference-speed improvement. I recommend keeping the reader's conditional verdict, with an added condition: report corpus-weighted fertility and training-token reductions, and qualify the abstract accordingly. Since the reader already issued a conditional verdict, this does not move the verdict; it sharpens the condition.","tokens_in":16271,"tokens_out":7784,"duration_ms":82525,"concrete_test":"Compute fertility on a large held-out general Korean corpus (e.g., a fresh 100M-sentence split of HPLT v2.0 or a raw Korean web corpus without evaluation-prompt formatting), using Thunder-Tok and BPE at 128K vocabulary. Report the word-weighted percent reduction in tokens per word. Also report a corpus-weighted average of the Table 3 datasets, weighting each dataset by its word count. If the weighted reduction is ~4% rather than ~10%, the abstract and inference-speed claims should be revised accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim is that Thunder-Tok reduces token fertility by ~10% versus BPE. Table 3 shows this only as an unweighted average of five rows: HPLT v2.0 and four benchmark sets. On HPLT v2.0, the largest and most representative text source, fertility is 1.638 vs 1.704, a 3.9% reduction. The benchmark rows range from 5.2% to 14.3%, and their average is 9.2%, which rounds to the headline 10%. The training-token counts in Table 4 tell the same story: at 128K vocabulary, Thunder-Tok uses 13.0B vs BPE's 13.5B tokens (3.7% fewer) for the 360M model, and 33.0B vs 34.2B (3.5% fewer) for the 1.5B model. Thus the practical token-count reduction on natural Korean text is roughly 4%, not 10%. The larger benchmark-specific gains likely reflect repeated prompt templates and short answer strings that the pre-tokenization regex merges, not general linguistic fertility. Because the abstract explicitly equates the 10% figure with a 10% inference-speed improvement, the paper's headline overstates the practical benefit by a factor of about 2.5. This does not invalidate the tokenizer, but it does mean the claim 'reduces the number of tokens by 10%' is not supported by the paper's own corpus-level numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Thunder-Tok, a Korean tokenizer designed to reduce token fertility (tokens per word) while maintaining downstream model performance. The method combines three components: (1) a rule-based pre-tokenization step that merges whitespace-connected Hangul sequences and attaches sentence-final punctuation to the preceding word; (2) a 'language-aware' seed vocabulary that only admits substring tokens that can be perfectly divided into lower-level linguistic units; and (3) a branching-entropy-based scoring function used to prune the vocabulary down to a target size. The authors evaluate Thunder-Tok against BPE, Unigram, and BPE-Mecab using LLaMA-style models of 360M and 1.5B parameters, measuring both fertility on the HPLT v2.0 corpus and four Korean benchmarks, and zero-shot accuracy on KoBEST HellaSwag, KoBEST COPA, Korean ARC-e, and Korean LAMBADA. The paper claims an approximately 10% fertility reduction over BPE with no loss in downstream performance.","tokens_in":16605,"tokens_out":7467,"duration_ms":69602,"significance":"If the headline claim held, the paper would offer a simple, linguistically motivated way to cut Korean token counts by about 10%, which is a practically meaningful inference-speed improvement with no accuracy cost. The core ideas—restricting the token vocabulary to morphologically plausible units and using branching entropy as a score—are interesting and worth testing in other morphologically rich languages. The paper is also careful to compare against strong baselines and includes an ablation study. However, the central quantitative claim is overstated: on the training corpus itself (HPLT v2.0) the fertility reduction is only 3.9%, and the training-token counts in Table 4 show a 3.5–3.7% reduction. The benchmark-specific reductions (5–14%) are inflated by short, templated prompts and answer choices. The downstream performance comparisons are based on a single run with no error bars, so the claim that performance is 'not compromised' is not statistically supported. The contribution is real but more modest than advertised.","major_comments":[{"comment":"The headline claim of an approximately 10% reduction in token count and inference speed is not supported by the paper's own corpus-level numbers. In Table 3, the fertility reduction over BPE on HPLT v2.0 is 3.9% (1.638 vs 1.704), and the unweighted average of the five per-dataset reductions is about 8.6%, not 10%. More importantly, Table 4 shows that Thunder-Tok uses only 3.7% fewer training tokens than BPE at 128K/360M (13.0B vs 13.5B) and 3.5% fewer at 128K/1.5B (33.0B vs 34.2B). The larger benchmark-specific reductions likely arise from short, templated prompts and answer strings that the regex merges, not from general Korean text. The abstract and conclusion equate the benchmark average with a 10% reduction in tokens and inference speed, which overstates the practical benefit by approximately a factor of 2.5. The authors should report a corpus-weighted fertility reduction and revise the abstract and conclusion accordingly.","section":"Abstract, Section 4.2, Table 3, Table 4"},{"comment":"All downstream results are from a single run, with no error bars, confidence intervals, or significance tests. The differences between Thunder-Tok and BPE-Mecab are as small as 0.1 percentage points in the 1.5B setting (67.26 vs 67.17), and the test sets contain only 500–2,376 examples. Without variance estimates, the claim that Thunder-Tok 'does not compromise performance' is not rigorously established. The fertility measurements are deterministic and not affected by this issue, but the performance comparisons are load-bearing for the central claim. Please provide multiple seeds or a statistical test, or at least explicitly acknowledge that the accuracy differences are within the range of single-run noise.","section":"Section 4.3, Table 4"},{"comment":"The removal threshold percentage m in Algorithm 1 is never specified in the manuscript. Since the final vocabulary and hence all experimental results depend on this hyperparameter, the experiments are not fully reproducible without knowing m. Additionally, the branching-entropy computation relies on the EM algorithm from the Unigram tokenizer, but the paper does not describe the initialization, number of iterations, or convergence criteria used. Please report these values.","section":"Algorithm 1, Section 3.3"}],"minor_comments":[{"comment":"The example of preserving '2 cups' is not supported by the implemented regular expression `(?: ?\\p{L}+)+(?:[.?!])?`, which matches only letters and does not include digits. Please either change the example or extend the regex to handle numeric-word combinations.","section":"Section 3.1"},{"comment":"There is a typo: 'Caculate' should be 'Calculate'. Also, 'whitespaces' is used inconsistently (sometimes 'whitespace').","section":"Algorithm 1"},{"comment":"The column header 'Average' is placed in a way that may be misread as belonging to 'Kobest HellaSwag'. Please reformat the table so the average column is clearly separate.","section":"Table 3"},{"comment":"The statement that 'replacing existing tokenizers ... could reduce inference costs by more than 10%' is not supported by the data, since the corpus-level reduction is about 4%. Please align this sentence with the actual measurements.","section":"Section 4.2"},{"comment":"There are minor typos in the appendix: 'Reasong' should be 'Reasoning', 'Hostpital' should be 'Hospital', and 'Becuase' should be 'Because'.","section":"Appendix B, Table 10"},{"comment":"The Venn diagram percentages do not sum to 100 and are not clearly defined. Please clarify whether the numbers represent percentages of each individual vocabulary, of the union, or of some other base.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a reasonable, well-motivated tokenizer, but the headline claim is not supported by the paper's own corpus-level data. The benchmark-derived 10% figure should be replaced with a corpus-weighted measure. The lack of error bars for the downstream performance is also a concern for a top-tier venue, but it is fixable. The method itself is not fatally flawed; a careful revision that recalibrates the claims and adds basic statistical reporting could make this a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the bottom line: Thunder-Tok is a real contribution and worth taking seriously, but the 10% claim should be read as '10% on our benchmark prompts', not '10% on Korean text'. Table 3 shows the HPLT v2.0 corpus—the training data—gets 1.638 vs 1.704 fertility, a 3.9% reduction. The training token counts in Table 4 are the same story: 13.0B vs 13.5B at 128K/360M, 33.0B vs 34.2B at 1.5B, roughly 3.5–3.7% fewer tokens. The ~10% figure comes from the unweighted average of the four small benchmark sets, where short prompts and choices inflate the effect of the pre-tokenizer's chunk-merging regex. So the paper's own numbers support an efficiency win of around 4% on natural Korean, not the 10% promised in the abstract and the inference-speed claim.\n\nWhat is genuinely new: the pre-tokenization regex that merges whitespace-separated Hangul sequences and attaches sentence-final punctuation; the seed-vocabulary filter requiring substrings to decompose into complete lower-level units; and the sentence-context branching entropy scoring. The ablation is instructive—each component moves fertility and accuracy in the expected direction, and the authors report the full breakdown in the appendix. Downstream accuracy is on par with BPE and BPE-Mecab across model sizes, with no obvious degradation from the lower fertility. The paper is also transparent about single runs and the language-specific scope.\n\nSoft spots, in proportion. First, the benchmark 'Korean ARC-easy' and 'Korean LAMBADA' come from a 'Thunder Research Group'—the same name as the tokenizer—with no disclosure of relationship. Because these are translations of existing English benchmarks, the risk is limited, but the paper should clarify this to rule out a hidden dependency. Second, the removal threshold m in Algorithm 1 is left unspecified; without it and without released code, the seed-vocabulary construction is not fully reproducible. These are minor relative to the overstated headline, which is the main thing a referee should push on.\n\nWho this is for: anyone building Korean LLMs or studying tokenization for agglutinative languages. The method is a modest but real step, and the paper is honest enough to deserve a serious referee. I'd accept it with revisions, with the corrected summary statistic as a required change.","headline":"Genuinely new Korean tokenizer with a careful ablation, but the '10% token savings' headline is an artifact of averaging small benchmarks; on real Korean text the saving is ~4%.","tokens_in":17122,"tokens_out":5615,"would_cite":true,"duration_ms":54515,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T50"],"pacs":[],"model":"deepseek-v4-flash","headline":"A Korean tokenizer that cuts tokens per word by about 10 percent without lowering model accuracy.","keywords":["Korean tokenization","token fertility","pre-tokenization","branching entropy","subword tokenization","Byte Pair Encoding","Unigram tokenizer","generative language models"],"falsifier":"Train a 360M-parameter LLaMA3-style model on the same Korean corpus for one epoch at 128K vocabulary with Thunder-Tok and with BPE, then compare average tokens per word on the four benchmarks and mean zero-shot accuracy; the paper's claim fails if the fertility gap is meaningfully below 10% or Thunder-Tok's mean accuracy falls below BPE's by more than about one point.","tokens_in":16106,"feed_emoji":"✂️","tokens_out":6774,"duration_ms":55575,"temperature":0.7,"pith_summary":"Thunder-Tok is a Korean tokenizer built to make each word cost fewer tokens, and the paper's central claim is that it lowers tokens per word by about 10% relative to Byte Pair Encoding at a 128K vocabulary without reducing model quality. The savings come from pre-tokenization rules that keep whitespace-connected Korean words and sentence-final punctuation as single chunks, a seed vocabulary restricted to linguistically plausible boundaries, and a branching-entropy scoring step that favors tokens appearing in varied sentence contexts. If the claim holds, Korean generative models would need roughly 10% fewer inference steps and 2–4% fewer training tokens to reach the same or slightly better zero-shot accuracy on common Korean benchmarks. The paper reports the effect at 360M and 1.5B parameter scales, with the largest gains on Korean LAMBADA.","feed_headline":"Korean tokenizer cuts tokens per word by 10 percent","feed_subtitle":"Linguistically grounded pre-tokenization keeps downstream accuracy while trimming inference steps.","key_machinery":"The load-bearing mechanism is a three-stage vocabulary construction: a rule-based pre-tokenizer whose pattern `(?: ?\\p{L}+)+(?:[.?!])?` forms multi-word and punctuation-attached chunks; a seed-vocabulary filter that admits only substrings which are concatenations of the next-lower linguistic unit (whole syllables within words, whole words within phrases); and a pruning loop that scores each token as BE(x)×P(x), where branching entropy BE(x) = -Σ_y P(y|x) log P(y|x) is computed over all sentences y using EM-derived segmentation probabilities from the Unigram tokenizer. The pruning loop removes the lowest-scoring m% tokens repeatedly until the target vocabulary size is reached. These three parts together determine which long, cross-space tokens survive, and that is what produces the fertility reduction without creating the semantically incoherent tokens that earlier work found harmful.","core_discovery":"The paper's discovery is that token fertility in Korean can be cut without an accuracy penalty by changing three places in the tokenizer pipeline. First, pre-tokenization merges what BPE would keep separate: the regular expression `(?: ?\\p{L}+)+(?:[.?!])?` treats whitespace-adjacent Korean letter sequences and a following sentence-final punctuation mark as one chunk, exploiting Korean's optional spacing and agglutinative endings. Second, the seed vocabulary only admits substrings that split cleanly at linguistic unit boundaries (e.g., a substring of a word must be a sequence of whole syllables), blocking incoherent fragments like \"어사\" from \"영어사전\". Third, tokens are scored by branching entropy computed over whole sentences rather than adjacent characters, using the Unigram tokenizer's EM estimates, multiplied by token probability; low-scoring tokens are pruned to reach the target vocabulary size. With these components, Thunder-Tok reaches average fertility 1.370 across HPLT v2.0 and four benchmarks versus 1.509 for BPE, and mean zero-shot accuracy 64.02 versus 63.06 at 128K vocabulary in the 360M model, with a similar pattern at 1.5B.","pith_inferences":["Applied to autoregressive decoding, the 10% token reduction should translate into roughly 10% fewer forward passes and thus about 10% lower latency and cost, assuming the vocabulary does not change model width; the paper states this as an inference-speed improvement but does not measure wall-clock time directly.","The same three-stage recipe—rule-based chunking, linguistically constrained seed vocabulary, and entropy scoring—could be adapted to other languages where whitespace does not mark morpheme boundaries, such as Japanese or Turkish, but the pre-tokenization regex and the allowable-boundary filter would need re-deriving for each language.","A testable extension is to apply only the pre-tokenization pattern to a standard BPE tokenizer without retraining the seed vocabulary; the ablation suggests the pattern alone reduces fertility but hurts accuracy, so isolating the filter's protective effect would confirm the mechanism.","The sentence-level branching entropy definition is a general scoring criterion that could be evaluated on other morphologically rich languages or used to audit existing vocabularies for context-diverse tokens."],"forward_implications":["Inference cost for Korean text falls roughly in proportion to token count: at about 10% lower fertility, a generation that previously took 100 token steps takes about 90, with accuracy unchanged or slightly better.","Training on the same Korean corpus requires 2–4% fewer tokens than BPE and 7–20% fewer than Unigram or BPE-Mecab, so a fixed compute budget could cover more data or a larger model.","The fertility advantage transfers to out-of-distribution text: on the four benchmark sets Thunder-Tok's fertility is 4–15% lower than BPE's, so the efficiency gain is not an artifact of the training corpus.","The seed-vocabulary filter is the component protecting accuracy: removing it (using GPT-2 style pre-tokenization or unfiltered N-gram seeds) drops average accuracy from 64.02 to as low as 62.46, while the branching-entropy scorer mainly buys fertility (from 1.418 to 1.370) rather than accuracy.","At larger scale the pattern holds: the 1.5B model with Thunder-Tok uses 33.0B training tokens versus 34.2B for BPE and posts the highest mean accuracy (67.26 vs 66.39)."],"supporting_citations":[{"why":"Supplies Byte Pair Encoding, the principal baseline tokenizer whose fertility Thunder-Tok reduces by about 10%.","marker":"Sennrich et al., 2015"},{"why":"Supplies the Unigram tokenizer's EM-based token probabilities and the likelihood scoring baseline used in the ablation.","marker":"Kudo, 2018"},{"why":"Supplies BPE-Mecab, the morphological pre-tokenization baseline for Korean that Thunder-Tok is compared against.","marker":"Park et al., 2020"},{"why":"Defines tokenizer fertility as average tokens per word, the paper's central efficiency metric.","marker":"Rust et al., 2020"},{"why":"Introduces branching entropy, the statistical measure adapted for token scoring in vocabulary construction.","marker":"Jin and Tanaka-Ishii, 2006"},{"why":"Provides evidence that semantically incoherent tokens degrade model performance, motivating the seed-vocabulary filter.","marker":"Schmidt et al., 2024"},{"why":"Supplies the SentencePiece whitespace handling convention used in constructing the seed vocabulary.","marker":"Kudo and Richardson, 2018"},{"why":"Supplies the LLaMA3 architecture used for all trained models in the experiments.","marker":"Grattafiori et al., 2024"}],"fun_headline_variants":["Thunder-Tok tokenizer cuts Korean tokens 10%","Linguistic pre-tokenization shrinks Korean token count","Branching entropy tokenizer trims Korean token fertility","Korean tokenizer: 10% fewer tokens, accuracy intact","Tokenize Korean smarter: 10% fewer tokens per word"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on the assumption that the regular-expression pre-tokenizer—merging whitespace-adjacent Korean letter runs and sentence-final punctuation into single chunks—always produces semantically coherent units, since the paper's own ablation shows accuracy drops when this pattern is removed or the seed filter is absent.","fun_headline_variants_meta":{"raw":{"variants":["Thunder-Tok tokenizer cuts Korean tokens 10%","Linguistic pre-tokenization shrinks Korean token count","Branching entropy tokenizer trims Korean token fertility","Korean tokenizer: 10% fewer tokens, accuracy intact","Tokenize Korean smarter: 10% fewer tokens per word"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000684,"raw_usage":{"total_tokens":3092,"prompt_tokens":921,"completion_tokens":2171,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":2087}},"tokens_in":537,"tokens_out":2171,"duration_ms":16395,"temperature":1.0,"reasoning_tokens":2087,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:42:32.292860+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a 360M-parameter LLaMA3-style model on the same Korean corpus for one epoch at 128K vocabulary with Thunder-Tok and with BPE, then compare average tokens per word on the four benchmarks and mean zero-shot accuracy; the paper's claim fails if the fertility gap is meaningfully below 10% or Thunder-Tok's mean accuracy falls below BPE's by more than about one point.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces branching entropy, the statistical measure adapted for token scoring in vocabulary construction."}],"review_version":2}