{"id":"8be92acd-31c9-4050-9e94-11b4971c6afd","arxiv_id":"2608.08847","paper_version":2,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Explicit boundary markers with case shift codes eliminate space-duplicated vocabulary entries and improve language modeling bits per byte without changing compression.","lead":"This paper proposes replacing the usual leading-space convention in subword tokenizers with an explicit boundary marker and case codes, so words are not stored twice. The new schemes keep compression nearly unchanged but consistently lower bits per byte in small-scale language modeling, suggesting duplicate entries carry a real cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The downstream bits-per-byte claim in Table 3 may be confounded by an unspecified training budget: boundary[w] is ~9% worse in compression, so fixed-token and fixed-byte training give the schemes different amounts of compute.","rationale":"I focused on what the abstract's strongest claim actually requires. The compression and MorphScore results are controlled and interesting, and the invertibility issue raised by the reader is real but secondary: the fixed 20-script list plus the rule merging word runs across script changes appears to prevent adjacent marked spans in the tested languages, and in any case decoding is not exercised in the bits-per-byte evaluation. The critical gap is the missing training budget for Table 3. Because boundary[w] has roughly 9% worse compression, a fixed-step comparison gives it less data and a fixed-byte comparison gives it more updates; without knowing which, the headline 'every marker scheme reaches lower bits per byte' is not yet a controlled claim. The paper is otherwise carefully scoped and its code is available, so a conditional verdict asking for this one control is proportionate.","tokens_in":8993,"tokens_out":24982,"duration_ms":267780,"concrete_test":"Using the released training code, re-run the four English BPE and four MinGram tokenizers from Table 3 under two explicitly matched budgets: (a) equal optimizer steps with equal token batch size, and (b) equal bytes processed (scale the number of steps to compensate for compression differences). If the bits-per-byte ordering of Table 3 persists under both budgets, the concern is resolved; if it reverses or loses significance under either, the downstream claim must be restated as conditional on training budget.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim—every marker scheme tested downstream reaches lower bits-per-byte than plain—rests on Table 3, a depth-12 nanochat comparison on ClimbMix. The paper does not state whether the compared tokenizers were trained for a matched number of tokens/steps or a matched number of bytes. This matters because boundary[w] is about 9% worse in compression (Table 2). Under a fixed-token budget it is trained on roughly 9% fewer bytes than plain; under a fixed-byte budget it receives more optimizer updates per byte. The reported 0.5–1% bits-per-byte advantage could therefore reflect a training-compute difference rather than the boundary-marker representation. The Limitations section acknowledges English-only evaluation and one model scale, but not this budget ambiguity, so the headline result is not yet a controlled comparison.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an alternative to the standard leading-space convention in subword tokenization. Instead of attaching a space to the following word, it introduces an explicit boundary marker (and optional case-shift codes) emitted during pretokenization, so that words have a canonical form regardless of preceding whitespace and capitalization. The encoding is invertible. The authors evaluate five marking schemes on six languages with two vocabulary-learning algorithms (BPE and MinGram), measuring compression, morphological alignment (MorphScore), and downstream language-modeling bits per byte on a depth-12 nanochat model. They find that the best marker schemes are approximately compression-neutral, but that all schemes tested downstream achieve lower bits per byte than the leading-space baseline, which they interpret as evidence that vocabulary duplication carries a cost not captured by compression.","tokens_in":9151,"tokens_out":11903,"duration_ms":120456,"significance":"If the downstream result survives a matched-compute comparison, the paper offers a genuinely simple and practical pretokenization change: a drop-in modification applicable to any vocabulary-learning algorithm, with no change to the training objective or architecture, that reduces embedding duplication and improves language-modeling efficiency at a near-matched vocabulary size. The compression and duplication analyses over six languages and two tokenizers are careful, and the invertibility argument is well specified. The reported code availability and the use of standard deviations over three seeds are also strengths. However, the central headlining claim currently rests on a downstream comparison whose training budget is unspecified, so the significance is conditional.","major_comments":[{"comment":"The sentence 'Loss is normalized by the text's true UTF-8 length, so schemes that emit different numbers of tokens stay comparable' only makes the evaluation metric comparable, not the training budget. The number of optimizer steps, training tokens, or training bytes used for the nanochat runs is not reported. Since boundary[w] has about 9% worse compression than plain (Table 2), a fixed-token budget yields about 9% fewer bytes of training text for boundary[w], whereas a fixed-byte budget yields more optimizer updates per byte; in neither case is the comparison controlled for compute. The reported 0.5–1% bits-per-byte advantage of the marker schemes could therefore reflect a training-compute difference rather than the representation itself. Please report the exact training budget and include a matched-compute comparison (matched bytes or matched FLOPs) to support the claim that duplication carries a cost that compression does not capture.","section":"Section 5, Table 3"}],"minor_comments":[{"comment":"The phrase 'Incl100k' is missing a space and should read 'In cl100k'.","section":"Section 1"},{"comment":"The example sentence 'the in \"the is marked\"¦the¦' is difficult to read; rephrase as 'the word \"the\" in \"the is marked\" is encoded as ¦the¦, giving the same pretoken as the span in \"the cat\".'","section":"Section 3.1"},{"comment":"In the encoding of 'the, cat', the printed string '¦the¦ ,¦ ¦cat¦' shows a space between the comma's marker and the following marker; since that space is removed by the scheme, please adjust the formatting or add a note so that the marker adjacency is not misinterpreted.","section":"Appendix B.4, Table 6"},{"comment":"The sentence 'Every tokenizer learns 32,768 tokens beyond its atomic alphabet, which differs by at most three entries between schemes' means that the total vocabulary sizes are not exactly equal; please state the total vocabulary size for each scheme explicitly, or use the phrase 'matched learned-vocabulary size' to be precise.","section":"Section 4, Tokenizer training"},{"comment":"The reversal of the ordering when whole gold words are excluded deserves more analysis than the current one-sentence explanation; readers may otherwise interpret the high credit-mode scores as evidence of better morphological alignment, when the scores largely track the share of words left whole. Consider reporting the exclude-mode result as the primary metric or adding a brief analysis of why split words are aligned more poorly.","section":"Section 5, MorphScore discussion"},{"comment":"The claim that every scheme beats plain at p<0.01 is based on a two-sided paired t-test with only three shared seeds; please also report effect sizes and consider a permutation test, since the t-test's validity at this sample size is sensitive to distributional assumptions.","section":"Section 5, Downstream evaluation"}],"recommendation":"major_revision","confidential_remarks":"The paper is methodologically interesting and the intrinsic evaluation is solid. The main risk is the unspecified downstream training budget; the authors need to provide a matched-compute analysis. I would also encourage the editor to have the authors clarify the relationship between the credit and exclude MorphScore results before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a genuinely new pretokenization scheme: explicit boundary markers (¦) and case code tokens (↑, ⇑) that replace the leading-space convention, with a carefully specified invertible encoding. It's a clean idea, and the authors do real work showing why it doesn't help compression (it actually hurts: boundary[w] is ~9% worse characters per token) yet might help downstream LM quality. The evaluation is broad within its scope: six languages, two tokenizer algorithms, compression tables per language, and the marker schemes eliminate space-duplication by construction. The Limitations section is honest about English-only downstream and one scale.\n\nThe soft spot is real: the downstream claim in Table 3 is potentially confounded by an unstated training budget. The paper doesn't say whether the nanochat runs were matched on tokens/steps or on bytes. Since boundary[w] compresses ~9% worse than plain, that choice determines whether boundary[w] sees less text (fixed token budget) or more optimizer updates per byte (fixed byte budget). The first case would make the result stronger; the second would make it an artifact of compute. Right now we can't tell. This is a straightforward omission to fix, and the authors should address it.\n\nThe invertibility worry you passed along looks less serious on reading. The merging of word runs across script changes and the fixed list of space-using scripts are stated precisely. An unlisted script won't get markers, so there's no way to create an adjacent pair of markers that decodes to a spurious space. The encoding is safe, even if it silently skips scripts outside the list.\n\nI also think the MorphScore reversal under exclude is handled fairly: the marker schemes leave far more words whole, so excluding those flips the ordering, and the paper says so.\n\nBottom line: this deserves a serious referee. The central representation claim is novel and the intrinsic results are carefully documented. The LM result is plausible but needs a matched-compute statement or experiment before I'd trust it. Send it to review with a request to clarify training budget.","headline":"A well-argued tokenization proposal with a genuinely new marker scheme, but the headline LM gain may rest on an unspecified training budget.","tokens_in":9655,"tokens_out":3933,"would_cite":true,"duration_ms":40749,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Common words are stored twice by subword tokenizers, and explicit boundary markers remove the duplication and improve language modeling without improving compression.","keywords":["subword tokenization","word boundary markers","leading-space duplication","capitalization codes","pretokenization","language modeling","bits per byte","vocabulary compression"],"falsifier":"Run the proposed encoding on a corpus containing a script outside the fixed list of twenty space-separating scripts and round-trip decode; if a single spurious space appears, the invertibility guarantee fails. Alternatively, reproduce the English bits-per-byte comparison at a larger model scale: if marker schemes no longer beat the baseline, the claimed uncaptured duplication cost is not a general effect.","tokens_in":8788,"feed_emoji":"🔤","tokens_out":9183,"duration_ms":94120,"temperature":0.7,"pith_summary":"Subword tokenizers store many common words twice in space-using writing systems, once with a leading space and once without, so the same word occupies separate embedding rows that are trained independently. This paper proposes replacing the leading-space convention with an explicit word-boundary marker, encoding spaces as pairs of markers and title or upper case with shift codes, before any vocabulary is learned. The scheme eliminates space-driven duplicate entries, leaves compression within roughly one percent of the baseline, and every marker variant tested achieves lower bits per byte in downstream language modeling. The paper concludes that duplication carries a cost that compression alone does not measure, making explicit boundary markers a drop-in pretokenization change worth adopting when a canonical word form is wanted.","feed_headline":"Explicit word-boundary markers beat space-token duplication in LLMs","feed_subtitle":"Boundary markers cut duplicate word entries and lower bits per byte even when compression stays flat.","key_machinery":"The central mechanism is an atomic boundary marker token (written ¦) placed on both sides of word, punctuation, or digit spans during pretokenization; a single space between two marked spans is removed, so a pair of adjacent markers becomes the encoding of a space, and decoding reverses this by replacing each marker pair with a space and deleting the remaining markers. Two additional atomic shift codes, ↑ for title case and ⇑ for upper case, are attached before a span's opening marker while the span is lowercased, letting case variants share the same entry. Invertibility rests on the rule that word spans are always marked on both sides and never lie adjacent without a real space, which is guaranteed by merging word runs across script changes and by marking only a fixed list of twenty space-using scripts.","core_discovery":"On its own terms, the paper establishes that attaching a leading space to a word token creates systematic duplication, with a single word appearing in up to six forms once capitalization is counted, and that this duplication hurts language modeling even when tokenization compression is essentially unchanged. The proposed encoding marks word spans with an atomic boundary token, encodes inter-word spaces as pairs of those tokens, and uses title-case and upper-case codes so lower-case, title-case and upper-case forms can share one internal representation. Across six languages, two vocabulary-learning algorithms and three marker configurations, compression stays within one percent of the baseline for the punctuation- and digit-marking schemes, while all marker schemes tested improve bits per byte over the leading-space baseline at matched vocabulary size. The paper therefore claims that the benefit of removing duplication is real but invisible to compression metrics.","pith_inferences":["A direct test of the proposed mechanism would count effective embedding updates per word form before and after marking: the duplication-cost account predicts that the rare form of a common word receives many more updates under the marker scheme.","The digit-marking scheme is the least settled part of the design; splitting digit runs rather than marking whole runs could remove the four-form duplication of numbers and may reclaim the small compression losses seen in the current results.","If the bits-per-byte gain persists at larger scale and in non-English languages, the leading-space convention itself becomes the suspect design choice, and tokenizer evaluations would need to include downstream modeling quality, not just compression.","The case-code idea suggests a natural extension to mixed-case spans, though invertibility would likely require a more expressive code than a single prefix marker."],"forward_implications":["At matched vocabulary size, switching to explicit boundary markers lowers language-modeling bits per byte relative to the leading-space baseline, for both vocabulary-learning algorithms and for every marker configuration tested.","Because compression is essentially unchanged for the punctuation- and digit-marking schemes, the downstream gain is a separate effect of removing duplicated entries, not a side effect of better compression.","Space-driven duplicate vocabulary entries fall to zero by construction, and case-driven duplication drops as well when case codes are enabled.","The change is confined to pretokenization and leaves the vocabulary-learning algorithm untouched, so it can be applied to byte-pair-encoding-style, Unigram-style, and other tokenizers as a drop-in modification.","Words are emitted as single tokens far more often under the marker schemes than under the baseline, so morphological-alignment scores for whole words shift accordingly."],"supporting_citations":[{"why":"Supplies the structured base encoding and pretokenization that the paper extends with boundary markers.","marker":"Land and Arnett, 2025"},{"why":"Defines BPE, one of the two vocabulary-learning algorithms on which every scheme is measured.","marker":"Sennrich et al., 2016"},{"why":"Describes MinGram, the minimum-token-count Unigram trainer that provides the second algorithm.","marker":"Land, 2026a"},{"why":"Shares credit for the MinGram training procedure evaluated in the experiments.","marker":"Land and Pinter, 2026"},{"why":"Provides the pretraining corpus on which downstream bits-per-byte is measured.","marker":"Diao et al., 2025"},{"why":"Provides the monolingual held-out evaluation sets used for per-language compression.","marker":"Chang et al., 2026"},{"why":"Defines the morphological-alignment metric used to score whether tokenizers keep words whole.","marker":"Arnett et al., 2025"},{"why":"Provides the under-trained-token evidence the paper invokes to explain why duplication has a cost.","marker":"Land and Bartolo, 2024"}],"fun_headline_variants":["Explicit boundary markers cut token duplication, boost LM performance","Atomic word markers erase space-token duplication, aid modeling","Boundary markers unify word forms, lower LM bits per byte","Explicit markers stop duplicate word entries, improve LMs","One word, one token: explicit boundaries fix duplication"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The encoding is invertible only if two words never appear side by side without a real space between them; the method enforces this with a fixed list of twenty space-using scripts and by treating script changes within a word as one marking unit, so an unlisted or edge-case script could make decoding insert a spurious space.","fun_headline_variants_meta":{"raw":{"variants":["Explicit boundary markers cut token duplication, boost LM performance","Atomic word markers erase space-token duplication, aid modeling","Boundary markers unify word forms, lower LM bits per byte","Explicit markers stop duplicate word entries, improve LMs","One word, one token: explicit boundaries fix duplication"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000432,"raw_usage":{"total_tokens":2194,"prompt_tokens":926,"completion_tokens":1268,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":1186}},"tokens_in":542,"tokens_out":1268,"duration_ms":11101,"temperature":1.0,"reasoning_tokens":1186,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:21:51.278501+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the proposed encoding on a corpus containing a script outside the fixed list of twenty space-separating scripts and round-trip decode; if a single spurious space appears, the invertibility guarantee fails. Alternatively, reproduce the English bits-per-byte comparison at a larger model scale: if marker schemes no longer beat the baseline, the claimed uncaptured duplication cost is not a general effect.","supporting_citations":[{"cited_title":"ICML 2025 Tokenization Workshop , url=","cited_arxiv_id":null,"evidence_quote":"Supplies the structured base encoding and pretokenization that the paper extends with boundary markers."}],"review_version":1}