Pith. sign in

REVIEW 3 major objections 4 minor 15 references

Token-Native Storage: Read and Write in your Agent's Language

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Text should be stored in the model's own token IDs, not UTF-8, because the readers are agents.

desk verdict The frequency-renumbering trick is genuinely new, but the byte-coverage gap in the abstract means the 'general replacement for UTF-8' claim needs a serious test before it holds. read the letter →

arxiv 2608.02376 v2 pith:TBPNRYTT submitted 2026-08-03 cs.DB cs.CLcs.IR

Kumar Shivendu This is my paper
classification cs.DBcs.CLcs.IR
keywords token-nativestoragebyte-pairencodingtokenIDstextcompressionstreamvbytezstdbaselinevocabularystandardizationlanguage-modelagents
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that text storage should be token-native: keep text as the byte-pair-encoding (BPE) token IDs a language model actually reads, rather than as UTF-8 bytes. The claim is that packing token IDs into integer codecs already beats byte-oriented storage on size, and that serving token IDs directly eliminates the per-read tokenization step for AI agents. On English, packing r50k token IDs as uint16 gives 2.25x compression over UTF-8 with no entropy coding, and an entropy coder on top reaches 3.30x. Across six tokenizers and three corpora (English, code, Hindi), token-ID compression is reported to match or beat every byte codec tested, including a corpus-trained zstd dictionary. The only stated requirement is that reader and writer share the tokenizer, so the paper argues for standardizing a published vocabulary the way ASCII and UTF-8 standardized text.

What carries the argument

The central object is the token-ID stream produced by a BPE tokenizer, treated as a compressed representation of text. Two packing schemes carry the argument: plain uint16 packing of token IDs for speed and simplicity, and an entropy coder layered on top for maximum ratio. The third piece is a simple reordering of the vocabulary, from merge order to frequency order, which lets a fast integer codec (streamvbyte) approximate the entropy coder's density while decoding much faster. Together these make the token-ID stream the file format, with the tokenizer as the implicit codec shared by writer and reader.

What would settle it

Take a byte sequence the tokenizer never saw in training, such as UTF-8 for a script with many combining marks or arbitrary binary data, tokenize it, compress the token IDs, and attempt a byte-exact reconstruction; a single round-trip failure, or a compression ratio that falls below the byte-codec baseline on such input, would falsify the claim that token-native storage is generally smaller and lossless.

Watch

Extended reading notes

Core claim

The central discovery is that a BPE token-ID stream, treated as a storage format in its own right, is denser than byte-oriented encodings for text that models read. Packing r50k token IDs as plain uint16 beats UTF-8 by 2.25x on English with no further compression, and adding an entropy coder raises the ratio to 3.30x. In comparisons across six tokenizers and three corpora, compressing token IDs matches or beats every byte codec, including a corpus-trained zstd dictionary. A second finding is that BPE assigns token numbers by merge order rather than by frequency, and that re-ranking the vocabulary by token frequency lets a plain integer codec such as streamvbyte recover most of the entropy coder's compression ratio while decoding about 7x faster. Because a model consumes token IDs directly, the paper concludes that a token-native store can hand over the IDs without re-tokenizing on every read.

Load-bearing premise

The argument assumes that every byte sequence the store must hold can be losslessly converted to token IDs by the shared tokenizer, with no unknown-token or byte-fallback pieces; if a corpus contains bytes the tokenizer never learned, the token-native file either cannot be decoded back to the original text or becomes much larger.

Editorial extensions

If this is right

  • A database that stores text as token IDs can serve an agent its exact input IDs without running the tokenizer on every read, removing a per-query translation cost.
  • Token-native compression offers an immediate space win on English text: 2.25x from packing alone and 3.30x with entropy coding, which could reduce storage costs for text-heavy tables and indexes.
  • Since frequency-reordered vocabularies make a plain integer codec nearly as dense as an entropy coder, AI labs can capture most of the benefit simply by publishing tokenizer vocabularies ordered by frequency rather than by merge order.
  • If token-native storage becomes common, a shared, standardized vocabulary becomes a systems-level requirement comparable to ASCII or UTF-8, because both the writer and the reader must agree on the tokenizer.
  • Token-native storage naturally suits agent-oriented workloads where the same tokenizer is used across retrieval, ranking, and generation, making the stored representation directly consumable by the model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A token-native store is only lossless for input that the tokenizer actually covers; text in scripts with many unseen byte combinations, or arbitrary binary data, may fall back to byte-level tokens and silently lose both the density gain and exact reversibility unless the store also keeps a byte-fallback path.
  • If vocabularies become standardized and versioned, migrating between tokenizer versions becomes analogous to charset migration, and old models will need to read multiple tokenizer versions, effectively turning the tokenizer into part of the schema.
  • The frequency-reordering result suggests a broader principle: learned vocabularies optimized for tokenization quality are not automatically optimized for compression or codec efficiency, so storage-aware vocabulary tuning could become a design axis for future tokenizers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes storing text as BPE token IDs rather than UTF-8 bytes, arguing that token-centric readers (embedders, rerankers, agents) benefit from avoiding re-tokenization and that token-ID streams are more compressible than byte streams. The abstract reports that uint16 packing of r50k token IDs achieves 2.25x compression relative to UTF-8 on English, an entropy coder reaches 3.30x, and that across six tokenizers and three corpora (English, code, Hindi) compressing token IDs matches or beats byte codecs including a corpus-trained zstd dictionary. A second finding is that reordering vocabulary tokens by frequency lets the streamvbyte integer codec approach the entropy coder's ratio while decoding about 7x faster. The paper argues for standardization of a shared, frequency-published vocabulary.

Significance. If the claims hold, the paper identifies a practical systems optimization and a concrete, low-risk change to how tokenizer vocabularies are published (frequency ordering). The compression-ratio claims are falsifiable and the re-ranking result is testable independently. However, the significance is currently limited by missing experimental details and an under-specified generality claim; the reader cannot verify whether the results transfer to arbitrary stored text.

major comments (3)
  1. [Abstract (generality claim)] The abstract states that the only requirement for token-native storage is that reader and writer share a tokenizer, but this is not sufficient: the tokenizer must be a lossless bijection over every byte sequence the store may contain. Many common BPE/SentencePiece tokenizers without byte_fallback emit UNK tokens for unseen bytes or Unicode sequences (for example, combining marks in Hindi), and such UNK tokens do not reconstruct the original bytes. Even for byte-level tokenizers, arbitrary byte sequences that never appear in BPE merges are represented as one token per byte, which can double the storage cost relative to UTF-8. The paper needs to state and empirically verify byte-level round-trip coverage for all six tokenizers and report the fraction of tokens that are single-byte fallbacks; otherwise the claim that token-native storage 'matches or beats every byte codec' on general text is unsupported.
  2. [Experimental methodology (compression ratios)] The reported ratios (2.25x for uint16 packing, 3.30x with an entropy coder) and the comparative claim against byte codecs are presented without corpus sizes, error bars, or train/test separation. The entropy coder's probability model and the frequency table used for vocabulary renumbering appear to be fitted on the corpora being compressed; if so, the ratios may be optimistic. The corpus-trained zstd dictionary also needs to specify whether its size is included in the storage overhead and how it was trained. Without a precise description of the evaluation protocol, the central empirical claim is not reproducible.
  3. [Abstract (comparison baselines)] The statement that compressing token IDs 'matches or beats every byte codec' is ambiguous without a list of baseline codecs and their configurations. In particular, comparing uint16 packing of token IDs against raw UTF-8 is not a meaningful baseline because UTF-8 is also compressible; the meaningful comparison is against compressed byte storage (e.g., zstd, gzip, or a specialized byte codec). The abstract does not specify which byte codecs were included, at what compression levels, or how the 2.25x figure relates to a compressed-byte baseline. Please define the baseline set and report the full comparison table.
minor comments (4)
  1. [Abstract (tokenizer scope)] The abstract refers to 'byte-pair-encoding (BPE) token IDs' but the six tokenizers likely include SentencePiece/unigram models; please clarify the tokenization families and vocabulary sizes, especially whether all vocabularies fit in uint16.
  2. [Abstract (packing width)] The 2.25x figure is for r50k with uint16 packing; tokenizers with vocabularies larger than 65,536 tokens require wider integers or varint encoding, which changes the density calculations. Please report the packing width for each tokenizer.
  3. [Abstract (renumbering and model compatibility)] If frequency re-ranking assigns new IDs, the stored IDs are no longer the tokenizer's original IDs; feeding them directly to a model requires permuting the model's embedding and output matrices. Please clarify whether the evaluated 'decode ~7x faster' scenario includes this mapping or assumes a reordered vocabulary.
  4. [Abstract (tokenizer versioning)] The 'only requirement is that reader and writer share a tokenizer' also implies that tokenizer versions must be pinned; a store written with one vocabulary version will be unreadable after a tokenizer update. Please discuss versioning and migration.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: all load-bearing claims are empirical benchmark comparisons against external byte codecs, not derivations from a target result.

full rationale

This abstract-only review contains no equations, no self-citations, and no fitted parameter that is later renamed as a prediction. The central claims are measured compression ratios (2.25x from uint16 packing, 3.30x with an entropy coder, and streamvbyte recovering most of the ratio while decoding faster) compared against external baselines such as UTF-8, byte codecs, and a corpus-trained zstd dictionary. These are empirical benchmark statements, not derivations whose conclusion is equivalent to an input. The frequency re-ranking of BPE token IDs is an encoding choice; if its frequency table were estimated from the same corpus used for evaluation, that would be a benchmark-design weakness or potential overfitting, but it would still not make the compression ratio a circular conclusion, because the ratio is measured against an external codec rather than derived from the fit. The byte-level tokenizer coverage requirement is a correctness and generality assumption, not a circular step: the paper states that reader and writer must share a tokenizer, but sharing a tokenizer does not by construction guarantee lossless round-tripping of arbitrary bytes, and the abstract does not define token-native storage in terms of the observed compression ratios. Because no load-bearing step reduces to its own inputs, the appropriate finding is no circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on two fitted quantities (the frequency table for renumbering and the entropy coder's probability model), one baseline fitted for comparison (a corpus-trained zstd dictionary), and four assumptions: lossless byte coverage, shared tokenizer, merge-order numbering, and corpus representativeness. The 2.25x baseline requires no free parameters; it is arithmetic given the average English token length. The lossless-coverage assumption is the most fragile because it is unstated in the abstract and would be violated by any tokenizer without a byte-level fallback.

free parameters (3)
  • Corpus frequency table for vocabulary renumbering
    The re-ranking that lets streamvbyte approach entropy-coder density needs empirical token frequencies from a corpus; the abstract does not state which corpus or how it relates to the evaluation corpora.
  • Entropy coder probability model
    The 3.30x ratio comes from an entropy coder whose symbol probabilities are estimated from token statistics; if the model is fit on the same corpora used for evaluation, part of the reported ratio is fitted rather than predictive, and this is not stated in the abstract.
  • Corpus-trained zstd dictionary
    The baseline is fit to a corpus it is then evaluated against, which is fair, but dictionary size and training corpus are unstated, and the comparison depends on those choices.
assumptions (4)
  • domain assumption The tokenizer losslessly represents every stored byte sequence, with reversible tokenization and no UNK fallback.
    The claims of beating byte codecs and handing IDs directly to model readers require that arbitrary text maps to valid token IDs and back; the abstract acknowledges only tokenizer sharing, not byte coverage. Many BPE vocabularies cannot encode every byte sequence.
  • domain assumption Reader and writer share the same tokenizer.
    Stated in the abstract as 'the only requirement is that reader and writer share a tokenizer'; this is the acknowledged boundary of the proposal and the basis of the standardization argument.
  • domain assumption BPE vocabularies number tokens by merge order, not by frequency.
    The frequency-renumbering finding depends on this property of standard BPE vocabularies; some published vocabularies are already renumbered, which would limit the generality of the 7x finding.
  • ad hoc to paper The evaluated corpora, English, code, and Hindi, represent general stored text.
    The universal claim, matches or beats every byte codec, extrapolates from three corpora and six tokenizers; other languages, mixed scripts, or binary-adjacent text may not behave the same.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Token-Native Storage: Read and Write in your Agent's Language." pith.science (2026). https://pith.science/paper/TBPNRYTT

@misc{pith2026260802376,
  author       = {Pith},
  title        = {Pith review of: Token-Native Storage: Read and Write in your Agent's Language},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TBPNRYTT}},
  note         = {Machine review of arXiv:2608.02376}
}
read the original abstract

Search and database engines still store text as UTF-8, a format built for humans. But the systems that increasingly read and write that text (embedders, rerankers, and language-model agents) work with token IDs, not characters, so every access pays to translate between the two. As agents become the primary readers and writers of stored text, we argue for token-native storage: keep the text as the model's own byte-pair-encoding (BPE) token IDs. Packing r50k IDs as uint16 already beats UTF-8 by 2.25x on English with no compression, and an entropy coder on top reaches 3.30x. Across six tokenizers and three corpora (English, code, Hindi), compressing token IDs matches or beats every byte codec, even a corpus-trained zstd dictionary. Two findings sharpen the case. BPE numbers tokens by merge order instead of frequency, and re-ranking by frequency lets a plain integer codec (streamvbyte) recover most of the entropy coder's ratio while decoding ~7x faster, a near-free change to how AI labs publish vocabularies. And because a model reads token IDs, not text, a token-native store hands over the IDs directly instead of re-tokenizing on every read. The only requirement is that reader and writer share a tokenizer, and different model families often use different ones today, so we argue for standardization: a published, shared vocabulary, the way ASCII and UTF-8 standardized text.

Figures

Figures reproduced from arXiv: 2608.02376 by the authors.

Figure 1
Figure 1. Every model read from a UTF-8 store pays a decompress-and-tokenize cost ( [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Token-native methods form the ratio-latency frontier (English, o200k, the tokenizer [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Median compression ratio over UTF-8 by method, one bar per corpus. Token-native [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Compression ratio vs. chunk size (o200k, English). Order-0 token methods ( [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The tokenization cost moves towards human reads and writes (log scale, [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The tokenizer step (red) is the cost that moves. For an agent it sits on the UTF-8 store, [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Review history (2 revisions) →

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 5 canonical work pages

  1. [1]

    R. Bamler. Understanding entropy coding with asymmetric numeral systems and the constriction library. https://github.com/bamler-lab/constriction, 2022

    work page 2022

  2. [2]

    J. Chen, X. Jiang, Z. Wang, et al. UniSearch: Rethinking Search System with a Unified Generative Architecture. arXiv:2509.06887, 2025

    arXiv 2025

  3. [3]

    Del\'etang, A

    G. Del\'etang, A. Ruoss, P.-A. Duquenne, et al. Language Modeling Is Compression. In ICLR, 2024. arXiv:2309.10668

    arXiv 2024

  4. [4]

    J. Duda. Asymmetric numeral systems. arXiv:1311.2540, 2013

    arXiv 2013

  5. [5]

    L. Gao, S. Biderman, S. Black, et al. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv:2101.00027, 2020

    arXiv 2020

  6. [6]

    J. He, R. He Bai, S. Williamson, J. Z. Pan, N. Jaitly, and Y. Zhang. CLaRa: Bridging Retrieval and Generation with Continuous Latent Reasoning. arXiv:2511.18659, 2025

    arXiv 2025

  7. [7]

    M. Kalcher. Frequency-Ordered Tokenization for Better Text Compression. arXiv:2602.22958, 2026

    work page arXiv 2026

  8. [8]

    Lester, J

    B. Lester, J. Lee, A. Alemi, et al. Training LLMs over Neurally Compressed Text. arXiv:2404.03626, 2024

    arXiv 2024

Show all 15 references
  1. [9]

    Megatron-Core Data Preparation: the IndexedDataset format

    NVIDIA. Megatron-Core Data Preparation: the IndexedDataset format. Developer documentation, 2024. https://docs.nvidia.com/megatron-core/

    2024

  2. [10]

    tiktoken : a fast BPE tokenizer

    OpenAI. tiktoken : a fast BPE tokenizer. https://github.com/openai/tiktoken, 2022

    2022

  3. [11]

    Raffel, N

    C. Raffel, N. Shazeer, A. Roberts, et al. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (C4). JMLR, 21(140), 2020. arXiv:1910.10683

    2020 arXiv

  4. [12]

    Sennrich, B

    R. Sennrich, B. Haddow, and A. Birch. Neural Machine Translation of Rare Words with Subword Units. In ACL, 2016. arXiv:1508.07909

    2016 arXiv

  5. [13]

    A. Ulla. LoPace: A Lossless Optimized Prompt Accurate Compression Engine for LLM Applications. arXiv:2602.13266, 2026

    2026

  6. [14]

    Wenzek, M.-A

    G. Wenzek, M.-A. Lachaux, A. Conneau, et al. CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data. In LREC, 2020. arXiv:1911.00359

    2020 arXiv

  7. [15]

    Zhang, C

    J. Zhang, C. Peng, M. Sun, et al. OneGen: Efficient One-Pass Unified Generation and Retrieval for LLMs. arXiv:2409.05152, 2024

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.