{"id":"a4ace1e8-9606-413b-bac2-43a09a1194d4","arxiv_id":"2608.02376","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Across six tokenizers and three corpora, storing text as token IDs compresses as well as or better than byte codecs, and a frequency-reordered vocabulary makes integer coding nearly as dense as entropy coding while decoding about 7x faster.","lead":"Instead of storing text as bytes, this paper proposes storing it as the token IDs that language models actually read, reporting up to 3.30x compression on English and a direct hand-off to model agents. The idea matters because AI agents are becoming the main readers of stored text, and a tokenized storage format could change how databases serve model workloads.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Byte-level tokenizer coverage is the load-bearing assumption: if any of the six tokenizers cannot losslessly round-trip arbitrary bytes, token-native storage is not a general text format and can expand instead of compress.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern: lossless byte-level tokenizer coverage. I agree. The central claim has two pillars: token IDs compress well and the representation is a lossless bijection for stored text. The compression pillar is an empirical measurement and could be verified from the methods and artifact, but the bijection pillar is more fundamental because if it fails, token-native storage cannot be a general text format regardless of compression ratios. The abstract explicitly mentions tokenizer sharing as the only requirement, which is insufficient: sharing a tokenizer does not guarantee that every byte sequence is tokenizable without information loss. GPT-2's r50k is byte-level and probably covers all bytes, but two of the three corpora (code and Hindi) are precisely regimes where non-byte-level tokenizers or byte-fallback pieces are likely, so the empirical ratios may be optimistic for general text. This is not an internal inconsistency in the abstract; it is an unstated scope restriction. A concrete round-trip and compression test on byte-fallback-heavy data would settle whether the concern lands. Since the full text is unavailable, the existing UNVERDICTED verdict remains appropriate; the concern reinforces the need for full verification rather than changing the verdict.","tokens_in":1113,"tokens_out":5056,"duration_ms":55266,"concrete_test":"For each of the six tokenizers, run tokenizer.encode followed by decode on: (1) all 256 single-byte inputs; (2) a byte-random 1 MB file; (3) Hindi text with combining marks and code with unusual Unicode. If any round-trip fails, record the first failing byte. Then compress the byte-fallback-heavy file with the paper's uint16 and entropy-coded token-ID formats and compare to UTF-8 and zstd; if the token-ID ratio is below 1.0 or below zstd, the universality claim fails. Re-run the compression table on an all-bytes corpus to see how much byte-fallback affects the reported 2.25x/3.30x numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Token-native storage is only a general replacement for UTF-8 if encode->decode is the identity on every byte sequence the store might hold. The abstract states the only requirement is that reader and writer share a tokenizer, but sharing a tokenizer does not imply the tokenizer covers every byte sequence. GPT-2's r50k is byte-level and likely covers all 256 bytes, but several common BPE/SentencePiece tokenizers without byte_fallback emit UNK for unseen bytes or Unicode combining marks; those UNK tokens do not reconstruct the original bytes. Even for byte-level tokenizers, arbitrary bytes that never appear in BPE merges are represented as one token per byte, so uint16 packing costs 2 bytes per byte of text, twice UTF-8, and an entropy coder can at best recover the original byte entropy plus token overhead. Thus the claimed 'matches or beats every byte codec' depends on the corpora being predominantly composed of mergeable token sequences. The Hindi and code corpora are exactly the places where byte-fallback pieces are likely, so the six-tokenizer/three-corpus result may not transfer to general text. The paper needs to state and test byte-level coverage explicitly; otherwise the central generality claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes storing text as BPE token IDs rather than UTF-8 bytes, arguing that token-centric readers (embedders, rerankers, agents) benefit from avoiding re-tokenization and that token-ID streams are more compressible than byte streams. The abstract reports that uint16 packing of r50k token IDs achieves 2.25x compression relative to UTF-8 on English, an entropy coder reaches 3.30x, and that across six tokenizers and three corpora (English, code, Hindi) compressing token IDs matches or beats byte codecs including a corpus-trained zstd dictionary. A second finding is that reordering vocabulary tokens by frequency lets the streamvbyte integer codec approach the entropy coder's ratio while decoding about 7x faster. The paper argues for standardization of a shared, frequency-published vocabulary.","tokens_in":1238,"tokens_out":4679,"duration_ms":54231,"significance":"If the claims hold, the paper identifies a practical systems optimization and a concrete, low-risk change to how tokenizer vocabularies are published (frequency ordering). The compression-ratio claims are falsifiable and the re-ranking result is testable independently. However, the significance is currently limited by missing experimental details and an under-specified generality claim; the reader cannot verify whether the results transfer to arbitrary stored text.","major_comments":[{"comment":"The abstract states that the only requirement for token-native storage is that reader and writer share a tokenizer, but this is not sufficient: the tokenizer must be a lossless bijection over every byte sequence the store may contain. Many common BPE/SentencePiece tokenizers without byte_fallback emit UNK tokens for unseen bytes or Unicode sequences (for example, combining marks in Hindi), and such UNK tokens do not reconstruct the original bytes. Even for byte-level tokenizers, arbitrary byte sequences that never appear in BPE merges are represented as one token per byte, which can double the storage cost relative to UTF-8. The paper needs to state and empirically verify byte-level round-trip coverage for all six tokenizers and report the fraction of tokens that are single-byte fallbacks; otherwise the claim that token-native storage 'matches or beats every byte codec' on general text is unsupported.","section":"Abstract (generality claim)"},{"comment":"The reported ratios (2.25x for uint16 packing, 3.30x with an entropy coder) and the comparative claim against byte codecs are presented without corpus sizes, error bars, or train/test separation. The entropy coder's probability model and the frequency table used for vocabulary renumbering appear to be fitted on the corpora being compressed; if so, the ratios may be optimistic. The corpus-trained zstd dictionary also needs to specify whether its size is included in the storage overhead and how it was trained. Without a precise description of the evaluation protocol, the central empirical claim is not reproducible.","section":"Experimental methodology (compression ratios)"},{"comment":"The statement that compressing token IDs 'matches or beats every byte codec' is ambiguous without a list of baseline codecs and their configurations. In particular, comparing uint16 packing of token IDs against raw UTF-8 is not a meaningful baseline because UTF-8 is also compressible; the meaningful comparison is against compressed byte storage (e.g., zstd, gzip, or a specialized byte codec). The abstract does not specify which byte codecs were included, at what compression levels, or how the 2.25x figure relates to a compressed-byte baseline. Please define the baseline set and report the full comparison table.","section":"Abstract (comparison baselines)"}],"minor_comments":[{"comment":"The abstract refers to 'byte-pair-encoding (BPE) token IDs' but the six tokenizers likely include SentencePiece/unigram models; please clarify the tokenization families and vocabulary sizes, especially whether all vocabularies fit in uint16.","section":"Abstract (tokenizer scope)"},{"comment":"The 2.25x figure is for r50k with uint16 packing; tokenizers with vocabularies larger than 65,536 tokens require wider integers or varint encoding, which changes the density calculations. Please report the packing width for each tokenizer.","section":"Abstract (packing width)"},{"comment":"If frequency re-ranking assigns new IDs, the stored IDs are no longer the tokenizer's original IDs; feeding them directly to a model requires permuting the model's embedding and output matrices. Please clarify whether the evaluated 'decode ~7x faster' scenario includes this mapping or assumes a reordered vocabulary.","section":"Abstract (renumbering and model compatibility)"},{"comment":"The 'only requirement is that reader and writer share a tokenizer' also implies that tokenizer versions must be pinned; a store written with one vocabulary version will be unreadable after a tokenizer update. Please discuss versioning and migration.","section":"Abstract (tokenizer versioning)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript was reviewed from the abstract and the reader's report only; the full text will be needed to verify the experimental claims. The main risk is that the generality claim overreaches without explicit byte-coverage guarantees. If the full text contains the missing methodology and coverage experiments, the paper can be a useful systems contribution; otherwise it needs substantial revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bob,\n\nYou asked for a quick read on this. The useful thing is the frequency-renumbering idea: BPE numbers tokens by merge order, which is arbitrary for storage, and re-ranking by empirical frequency lets streamvbyte get close to an entropy coder's density while decoding much faster. That's a concrete, testable systems trick, and the 7x speedup claim is the kind of thing a good referee can verify in an afternoon. Also, the framing of token-native storage as a way to hand IDs directly to model readers is sensible.\n\nThe 2.25x on English is just arithmetic: at roughly 4.5 characters per token, 2-byte IDs beat UTF-8 by that factor. Fine.\n\nThe soft spots are real but not fatal yet, because the full text is missing. The abstract says token-ID compression \"matches or beats every byte codec, even a corpus-trained zstd dictionary\" across six tokenizers and three corpora, but there are no error bars, no train/test separation for the frequency table or the entropy model, and no mention of how the zstd dictionary was trained. More importantly, the stress-test note lands: the only requirement stated is that reader and writer share a tokenizer, but sharing a tokenizer does not mean every byte sequence round-trips. Many BPE/SentencePiece tokenizers without byte_fallback emit UNK for unseen bytes, especially in code or with combining marks in Hindi. If that happens, token-native storage is not lossless, and in the worst case you get 2 bytes per byte of text, which is worse than UTF-8. Even for byte-level tokenizers, rare bytes become one token per byte and kill the compression. So the \"general replacement for UTF-8\" claim is overreached without an explicit byte-coverage analysis.\n\nThat said, this is the kind of paper that should go to peer review, not be desk-rejected. The novelty of the renumbering trick is enough to justify a careful look. A serious referee should demand the full methods, a round-trip test on a byte-diverse corpus, and an artifact. If the full paper delivers those, it could be a solid systems result. My verdict is the same as yours: unverified until we see the methods section, but worth engaging with.","headline":"The frequency-renumbering trick is genuinely new, but the byte-coverage gap in the abstract means the 'general replacement for UTF-8' claim needs a serious test before it holds.","tokens_in":1899,"tokens_out":2658,"would_cite":false,"duration_ms":106845,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Text should be stored in the model's own token IDs, not UTF-8, because the readers are agents.","keywords":["token-native storage","byte-pair encoding","token IDs","text compression","streamvbyte","zstd baseline","vocabulary standardization","language-model agents"],"falsifier":"Take a byte sequence the tokenizer never saw in training, such as UTF-8 for a script with many combining marks or arbitrary binary data, tokenize it, compress the token IDs, and attempt a byte-exact reconstruction; a single round-trip failure, or a compression ratio that falls below the byte-codec baseline on such input, would falsify the claim that token-native storage is generally smaller and lossless.","tokens_in":755,"feed_emoji":"🗄️","tokens_out":5131,"duration_ms":62179,"temperature":0.7,"pith_summary":"This paper argues that text storage should be token-native: keep text as the byte-pair-encoding (BPE) token IDs a language model actually reads, rather than as UTF-8 bytes. The claim is that packing token IDs into integer codecs already beats byte-oriented storage on size, and that serving token IDs directly eliminates the per-read tokenization step for AI agents. On English, packing r50k token IDs as uint16 gives 2.25x compression over UTF-8 with no entropy coding, and an entropy coder on top reaches 3.30x. Across six tokenizers and three corpora (English, code, Hindi), token-ID compression is reported to match or beat every byte codec tested, including a corpus-trained zstd dictionary. The only stated requirement is that reader and writer share the tokenizer, so the paper argues for standardizing a published vocabulary the way ASCII and UTF-8 standardized text.","feed_headline":"Store text as token IDs, not UTF-8: 2.25x smaller","feed_subtitle":"For AI agents that read and write text, token-native storage cuts size and skips re-tokenizing on every read.","key_machinery":"The central object is the token-ID stream produced by a BPE tokenizer, treated as a compressed representation of text. Two packing schemes carry the argument: plain uint16 packing of token IDs for speed and simplicity, and an entropy coder layered on top for maximum ratio. The third piece is a simple reordering of the vocabulary, from merge order to frequency order, which lets a fast integer codec (streamvbyte) approximate the entropy coder's density while decoding much faster. Together these make the token-ID stream the file format, with the tokenizer as the implicit codec shared by writer and reader.","core_discovery":"The central discovery is that a BPE token-ID stream, treated as a storage format in its own right, is denser than byte-oriented encodings for text that models read. Packing r50k token IDs as plain uint16 beats UTF-8 by 2.25x on English with no further compression, and adding an entropy coder raises the ratio to 3.30x. In comparisons across six tokenizers and three corpora, compressing token IDs matches or beats every byte codec, including a corpus-trained zstd dictionary. A second finding is that BPE assigns token numbers by merge order rather than by frequency, and that re-ranking the vocabulary by token frequency lets a plain integer codec such as streamvbyte recover most of the entropy coder's compression ratio while decoding about 7x faster. Because a model consumes token IDs directly, the paper concludes that a token-native store can hand over the IDs without re-tokenizing on every read.","pith_inferences":["A token-native store is only lossless for input that the tokenizer actually covers; text in scripts with many unseen byte combinations, or arbitrary binary data, may fall back to byte-level tokens and silently lose both the density gain and exact reversibility unless the store also keeps a byte-fallback path.","If vocabularies become standardized and versioned, migrating between tokenizer versions becomes analogous to charset migration, and old models will need to read multiple tokenizer versions, effectively turning the tokenizer into part of the schema.","The frequency-reordering result suggests a broader principle: learned vocabularies optimized for tokenization quality are not automatically optimized for compression or codec efficiency, so storage-aware vocabulary tuning could become a design axis for future tokenizers."],"forward_implications":["A database that stores text as token IDs can serve an agent its exact input IDs without running the tokenizer on every read, removing a per-query translation cost.","Token-native compression offers an immediate space win on English text: 2.25x from packing alone and 3.30x with entropy coding, which could reduce storage costs for text-heavy tables and indexes.","Since frequency-reordered vocabularies make a plain integer codec nearly as dense as an entropy coder, AI labs can capture most of the benefit simply by publishing tokenizer vocabularies ordered by frequency rather than by merge order.","If token-native storage becomes common, a shared, standardized vocabulary becomes a systems-level requirement comparable to ASCII or UTF-8, because both the writer and the reader must agree on the tokenizer.","Token-native storage naturally suits agent-oriented workloads where the same tokenizer is used across retrieval, ranking, and generation, making the stored representation directly consumable by the model."],"supporting_citations":[],"fun_headline_variants":["Token-native storage: BPE IDs beat UTF-8 by 2.25x","Store text as token IDs, skip UTF-8 and re-tokenizing","BPE tokens as storage: 2.25x denser, 7x faster decode","Agent-native storage: keep tokens, cut size and latency","Token IDs beat UTF-8 for storage, 2.25x smaller"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that every byte sequence the store must hold can be losslessly converted to token IDs by the shared tokenizer, with no unknown-token or byte-fallback pieces; if a corpus contains bytes the tokenizer never learned, the token-native file either cannot be decoded back to the original text or becomes much larger.","fun_headline_variants_meta":{"raw":{"variants":["Token-native storage: BPE IDs beat UTF-8 by 2.25x","Store text as token IDs, skip UTF-8 and re-tokenizing","BPE tokens as storage: 2.25x denser, 7x faster decode","Agent-native storage: keep tokens, cut size and latency","Token IDs beat UTF-8 for storage, 2.25x smaller"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000376,"raw_usage":{"total_tokens":2044,"prompt_tokens":1027,"completion_tokens":1017,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":643,"completion_tokens_details":{"reasoning_tokens":914}},"tokens_in":643,"tokens_out":1017,"duration_ms":12697,"temperature":1.0,"reasoning_tokens":914,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:59:57.306891+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a byte sequence the tokenizer never saw in training, such as UTF-8 for a script with many combining marks or arbitrary binary data, tokenize it, compress the token IDs, and attempt a byte-exact reconstruction; a single round-trip failure, or a compression ratio that falls below the byte-codec baseline on such input, would falsify the claim that token-native storage is generally smaller and lossless.","supporting_citations":[],"review_version":2}