Pith. sign in

REVIEW 2 major objections 5 minor 15 references

Token-Native Storage: Read and Write in your Agent's Language

T0 review · 2 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Storing text as a model's token IDs beats UTF-8 for both size and speed, and a one-line vocabulary change makes it faster still.

desk verdict Useful, honest benchmark paper with a real gap: the default +freq codec stores frequency ranks, not token IDs, and the paper understates what it takes to serve those to a model. read the letter →

arxiv 2608.02376 v1 pith:TBPNRYTT submitted 2026-08-03 cs.DB cs.CLcs.IR

Kumar Shivendu This is my paper
classification cs.DBcs.CLcs.IR
keywords token-nativestorageBPEtokenIDsUTF-8vscompressiontokenizationcostfrequencyorderingintegercodecsagentworkloadsdatabase
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that databases and search engines should store text as the serving model's BPE token IDs rather than UTF-8 bytes, because the models and agents that read and write that text already work in token IDs. Doing so is smaller and faster: raw ID packing compresses English text 2.25x versus UTF-8 with no compression step, an entropy coder reaches 3.30x, and reads return IDs in microseconds instead of paying roughly 235 microseconds per chunk to re-tokenize. The paper also shows that BPE vocabularies are numbered by merge order, not by how often tokens appear, and that re-ranking IDs by frequency lets a lightweight integer codec recover most of the entropy coder's ratio while decoding about seven times faster. The main barrier is portability across tokenizers, so the paper argues for a standardized, published shared vocabulary, the way ASCII and UTF-8 standardized text.

What carries the argument

The central object is the BPE token-ID sequence produced by the serving model's tokenizer. It does double duty: it is the representation the model already consumes and emits, and it is itself a compressed encoding of the source text. The mechanism that sharpens the result is the frequency-ordering fix: BPE assigns IDs by merge-discovery order rather than usage, so re-ranking the vocabulary by corpus frequency before packing with a SIMD variable-length integer codec (streamvbyte) lets a lightweight decoder recover most of the entropy coder's compression ratio while decoding roughly seven times faster. The tokenizer is what supplies most of the gain; an entropy coder over raw UTF-8 bytes reach

What would settle it

Run an end-to-end agent workload (for example, 100 retrieved chunks per query) with a token-native store and with a UTF-8 store, using a model that accepts only text input through its API. If total query latency is not meaningfully lower for the token-native store, or if the measured re-tokenization cost per chunk is far below the roughly 235 microseconds reported, the central latency claim would be undermined.

Watch

Extended reading notes

Core claim

Token IDs are the natural read/write representation for model-facing storage. If a database keeps text losslessly as the serving model's BPE token IDs, it can hand those IDs directly to embedders, rerankers, and language models, eliminating the per-access tokenization pass that a UTF-8 store forces. The same tokenizer that models use also compresses for free: a token covers roughly three-quarters of an English word, so packing IDs into fixed-width integers already beats UTF-8 before any compression algorithm runs. Two refinements complete the case: re-ordering BPE IDs by frequency lets a simple integer codec (streamvbyte) achieve most of an entropy coder's compression at a fraction of the de

Load-bearing premise

The latency win assumes that models, embedders, and rerankers can consume and emit token IDs directly, so a token-native store can skip tokenization on every read and write; if the APIs require text, only the compression benefit remains and the 10-600x speedup disappears.

Editorial extensions

If this is right

  • Within a single-model stack that already shares a tokenizer, token-native storage is implementable today and delivers both compression and microsecond read/write latencies on the agent path.
  • If AI labs publish vocabularies with IDs in frequency order, every downstream integer compressor benefits for free, with no per-application re-ranking effort.
  • Standardizing a shared tokenizer vocabulary would make one stored copy serve embedders, rerankers, and LLMs without re-tokenization, removing a portability barrier that currently limits the approach to single-model deployments.
  • On agent-heavy workloads, the savings compound across the pipeline: the same text is tokenized once at the human boundary instead of repeatedly at embed, rerank, and read stages, and I/O-bound work speeds up because less data moves.
  • Storage-footprint reductions are material at scale: the paper estimates roughly 4.2 TB saved per billion documents on English using raw r50k packing with an entropy coder, which also lowers SSD and network costs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If token-ID-native inference APIs remain unavailable in closed model services, the latency advantage collapses and only the compression benefit survives, so the strongest version of the claim is contingent on that ecosystem change.
  • The frequency-ordering insight suggests a simple convention for future tokenizer releases: ship a frequency-ranked vocabulary or a frequency table alongside the tokenizer, which would let any downstream integer codec immediately improve without per-corpus training.
  • The same token-native logic could extend beyond text to other discrete neural representations, such as image or audio tokenizers, where a database could store and serve latent codes directly to generative models.
  • A testable extension is an end-to-end comparison of an agentic RAG system using a token-native store versus a UTF-8 store, measuring wall-clock latency per query across embed, rerank, and generation steps; the paper reports component-level numbers but not a full integrated benchmark.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes storing text as the serving model's BPE token IDs rather than as UTF-8 bytes, arguing that this is both smaller and faster for model-facing workloads. It reports that raw fixed-width packing of token IDs beats UTF-8 by 2.25x on English; an entropy coder (+ANS) reaches 3.30x; and a frequency-remap plus streamvbyte scheme (+freq) reaches 2.60x with reads in about 4 microseconds, versus about 235 microseconds to decompress and re-tokenize a UTF-8 chunk. The claims are supported by benchmarks over three corpora (English, code, Hindi), six tokenizers, and byte-codec baselines including a corpus-trained zstd dictionary. The paper also makes two ecosystem recommendations: publish tokenizer vocabularies in frequency order, and develop token-ID-native inference APIs so that models can consume and emit token IDs directly.

Significance. If the compression results hold, they are a useful empirical contribution: the evaluation uses held-out test data, multiple tokenizers and languages, standard codecs, and a plausible mechanism (BPE collapses frequent character sequences before any coder runs). The latency argument is a forward-looking systems claim that is valid under the explicit assumption of token-ID-native inference APIs; this assumption is stated clearly in Section 7. The frequency-ordering observation is concrete, actionable, and likely to be useful to the community. The paper is honest about its limitations and makes code and benchmarks available. However, the recommended default codec (+freq) has a load-bearing gap: it stores frequency ranks, not the serving model's original token IDs, and the paper does not establish that its reported read/write latencies include the required permutation mapping. That issue must be resolved before the central 'serve token IDs directly' claim is supported for the default codec.

major comments (2)
  1. [§4, §5.4, Table 2] The +freq method, which is recommended as the default, stores frequency ranks rather than the serving model's original BPE token IDs. Section 4 says 're-rank IDs by corpus frequency' and pack with streamvbyte; the stored values are therefore the ranks, not the IDs the model was trained on. Returning rank 5 to vLLM when the model expects original ID 40000 is semantically wrong. The ~4 µs read latency in Table 2 and the 'serve IDs directly' claim in §5.4 do not state whether an inverse rank-to-original-ID mapping is included. If it is not, the default codec cannot actually be consumed by a token-ID-native API without an extra per-token lookup, so the headline read latency is understated. The same issue affects writes if the model emits original IDs and the store converts them to ranks. This is fixable: either benchmark and report the mapping cost, or recommend raw/+ANS for the agent-facing
  2. [§3.1, §5.4] The paper claims that a consumer with a different tokenizer 'detokenizes and re-tokenizes, exactly as it does with UTF-8 today, so token-native storage is never worse.' This is not correct: from a UTF-8 store the consumer only tokenizes, while from a token-native store it must first detokenize the stored token IDs and then re-tokenize the text. That is an extra detokenization step. The claim does not affect the same-tokenizer compression and latency results, but it should be corrected to 'at most one extra detokenization' or qualified to the case where the consumer shares the stored tokenizer.
minor comments (5)
  1. [§7 (item 3)] The recommendation to publish IDs in frequency order is evaluated using corpus-specific frequency ranks trained on each evaluation domain. A vendor-published frequency order would be based on the vendor's training corpus and may not match a user's domain. The paper should either test the benefit of using a tokenizer's native frequency ordering when available, or explicitly state that users should re-rank on their own corpus.
  2. [§5.4] The 235 µs tokenization figure is presented without hardware/software context. Please state the CPU, Python/OS version, and tiktoken version so readers can gauge the number's generality.
  3. [Table 2] The raw row is garbled in the provided text: '0.56.0 5.8' and '0.45.3 5.0' appear to be missing spaces. The table should also annotate which latencies include detokenization/tokenization, since this is exactly the point a reader must verify.
  4. [Abstract/§1] Several words have missing spaces in the manuscript text ('work intoken IDs', 'fortoken-native', 'compressing token IDs and then applying token ID compression'). A copyedit pass is needed.
  5. [§5.1] The sentence 'The coders trace a ratio/decode-speed frontier ... that token-native storage can pick any point on' would benefit from a pointer to Figure 2, which illustrates this frontier.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical benchmarks against external baselines; no fitted constant or self-citation chain supports the central claims.

full rationale

The paper's central claims—token-ID packing compresses better than byte codecs and token-native reads avoid retokenization—are supported by held-out benchmarks (Tables 1, 2; Figures 3–5) against external baselines (LZ4, gzip, zstd, brotli, UTF-8). The compression ratios are measurements, not outputs of a fitted model: ANS tables, frequency ranks, and zstd dictionaries are trained on train splits and evaluated on held-out test chunks. The §3.1 heuristic (6 bytes/word × 3/4 word/token vs 2 bytes/ID) is explicitly illustrative ('a way to understand it') and is not used to define the measured ratios. The read/write latency comparison compares measured tokenization costs (~235 µs) with measured decode times (0.4–25 µs); no parameter fitted to those endpoint values is then called a prediction. The paper's one recommendation that could confuse the read path—+freq stores frequency ranks rather than original BPE IDs, so serving a model may require an inverse rank→ID mapping—is an implementation/clarity concern, not a circular reduction: the reported +freq ratio and latency are still measured on held-out data, and raw packing/+ANS do not share the issue. References are external (tiktoken, constriction, Megatron-Core, Kalcher); none is a self-citation carrying the argument. Section 8 discloses limitations (shared vocabulary, tokenizer coverage, corpus-specific tables, component-level results) rather than concealing them. No step in the derivation chain reduces to its own input by construction.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new physical entities are posited. 'Token-native storage', the 'token' field type, and a standardized shared vocabulary are proposed system designs, not standalone predicted entities; they have no falsifiable handle outside the paper beyond benchmarks and interface sketches.

free parameters (3)
  • ANS unigram frequency table = trained per-corpus on train split (not reported as learned values)
    Used by +ANS in §3.2/§5; the entropy coder's ratios depend on this corpus-fitted table.
  • Frequency ranks for +freq = per-corpus re-ranking of token IDs, not reported
    Central to the streamvbyte result in §4; ranks are fitted to the train split and applied to held-out test.
  • Corpus-trained zstd dictionary for +dict = not reported
    Used in Table 1 to beat zstd--train on code; dictionary is fitted to training substrings, as the paper itself notes.
assumptions (5)
  • standard math BPE tokenization is losslessly invertible: the tokenizer maps token IDs back to the exact source text.
    Assumed in §3.1 and required for the claim that a token-native store can serve humans by detokenizing.
  • domain assumption Inference/embedding APIs can accept and return token IDs directly, avoiding re-tokenization.
    Invoked in §5.4 and §7; without it the latency win is not realized.
  • domain assumption The measured tokenization cost (~235 µs per 512-token chunk) is representative of the read path.
    Used in §5.4 to derive the 10–600x speedup; measured on a single core with tiktoken and not independently verified here.
  • domain assumption English C4, codeparrot, and Hindi Wikipedia, with six modern tokenizers, are representative of agent text workloads.
    Generalization of the compression ratios in §5 rests on these corpora; the paper itself notes r50k's poor Hindi coverage as a limitation.
  • domain assumption BPE vocabularies number tokens by merge order, not frequency.
    Fact asserted in §4 and used to justify the +freq re-ranking; true for standard tiktoken-style BPE but not guaranteed for all tokenizer implementations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Token-Native Storage: Read and Write in your Agent's Language." pith.science (2026). https://pith.science/paper/TBPNRYTT

@misc{pith2026260802376,
  author       = {Pith},
  title        = {Pith review of: Token-Native Storage: Read and Write in your Agent's Language},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TBPNRYTT}},
  note         = {Machine review of arXiv:2608.02376}
}
read the original abstract

Search and database engines still store text as UTF-8, a format built for humans. But the systems that increasingly read and write that text (embedders, rerankers, and language-model agents) work in token IDs, not characters, so every access pays to translate between the two. As agents become the primary readers and writers of stored text, we argue for token-native storage: keep the text as the model's own byte-pair-encoding (BPE) token IDs. This is both smaller and faster. Packing r50k IDs as uint16 already beats UTF-8 by 2.25x on English with no compression, and an entropy coder reaches 3.30x. Across six tokenizers and three corpora (English, code, Hindi), compressing token IDs matches or beats every byte codec, even a corpus-trained zstd dictionary. Two findings sharpen the case. BPE numbers tokens by merge order, not frequency, and re-ranking by frequency lets a plain integer codec (streamvbyte) recover most of the entropy coder's ratio while decoding ~7x faster, a one-line change we ask AI labs to make when they publish vocabularies. And because a model reads token IDs, not text, a token-native store hands them over directly instead of re-tokenizing on every read, ~10-600x faster. The only barrier is that sharing token IDs requires a common tokenizer, which is not always true across model families yet, so we argue for standardization: a published, shared vocabulary, the way ASCII and UTF-8 standardized text.

Figures

Figures reproduced from arXiv: 2608.02376 by the authors.

Figure 1
Figure 1. Every model read from a UTF-8 store pays a decompress-and-tokenize cost ( [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Token-native methods form the ratio-latency frontier (English, o200k, the tokenizer [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Median compression ratio over UTF-8 by method, one bar per corpus. Token-native [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Compression ratio vs. chunk size (o200k, English). Order-0 token methods ( [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The tokenization cost moves towards human reads and writes (log scale, [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 9 linked inside Pith

  1. [1]

    R. Bamler. Understanding entropy coding with asymmetric numeral systems and the constriction library. https://github.com/bamler-lab/constriction, 2022

    2022

  2. [2]

    J. Chen, X. Jiang, Z. Wang, et al. UniSearch: Rethinking Search System with a Unified Generative Architecture. arXiv:2509.06887, 2025

    arXiv 2025

  3. [3]

    Del\'etang, A

    G. Del\'etang, A. Ruoss, P.-A. Duquenne, et al. Language Modeling Is Compression. In ICLR, 2024. arXiv:2309.10668

    arXiv 2024

  4. [4]

    J. Duda. Asymmetric numeral systems. arXiv:1311.2540, 2013

    arXiv 2013

  5. [5]

    L. Gao, S. Biderman, S. Black, et al. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv:2101.00027, 2020

    arXiv 2020

  6. [6]

    J. He, R. He Bai, S. Williamson, J. Z. Pan, N. Jaitly, and Y. Zhang. CLaRa: Bridging Retrieval and Generation with Continuous Latent Reasoning. arXiv:2511.18659, 2025

    arXiv 2025

  7. [7]

    M. Kalcher. Frequency-Ordered Tokenization for Better Text Compression. arXiv:2602.22958, 2026

    arXiv 2026

  8. [8]

    Lester, J

    B. Lester, J. Lee, A. Alemi, et al. Training LLMs over Neurally Compressed Text. arXiv:2404.03626, 2024

    arXiv 2024

Show all 15 references
  1. [9]

    Megatron-Core Data Preparation: the IndexedDataset format

    NVIDIA. Megatron-Core Data Preparation: the IndexedDataset format. Developer documentation, 2024. https://docs.nvidia.com/megatron-core/

    2024

  2. [10]

    tiktoken : a fast BPE tokenizer

    OpenAI. tiktoken : a fast BPE tokenizer. https://github.com/openai/tiktoken, 2022

    2022

  3. [11]

    Raffel, N

    C. Raffel, N. Shazeer, A. Roberts, et al. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (C4). JMLR, 21(140), 2020. arXiv:1910.10683

    2020 arXiv

  4. [12]

    Sennrich, B

    R. Sennrich, B. Haddow, and A. Birch. Neural Machine Translation of Rare Words with Subword Units. In ACL, 2016. arXiv:1508.07909

    2016 arXiv

  5. [13]

    A. Ulla. LoPace: A Lossless Optimized Prompt Accurate Compression Engine for LLM Applications. arXiv:2602.13266, 2026

    2026

  6. [14]

    Wenzek, M.-A

    G. Wenzek, M.-A. Lachaux, A. Conneau, et al. CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data. In LREC, 2020. arXiv:1911.00359

    2020 arXiv

  7. [15]

    Zhang, C

    J. Zhang, C. Peng, M. Sun, et al. OneGen: Efficient One-Pass Unified Generation and Retrieval for LLMs. arXiv:2409.05152, 2024

    2024 arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.