Pith. sign in

REVIEW 2 major objections 5 minor 35 references

TokTier makes LLM tokenization stateful: it re-tokenizes only the text a coding agent appends, not the whole transcript, and still returns exactly the same token IDs as full reference tokenization.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 01:59 UTC pith:BYOOE7TE

load-bearing objection Genuinely strong systems paper with a real internal contradiction: the headline O(Δ) complexity claim is falsified by its own Appendix C's measured O(N) prefix-verification scan; the core ideas and validation survive, but the paper needs to fix that before acceptance. the 2 major comments →

arxiv 2607.29678 v1 pith:BYOOE7TE submitted 2026-07-31 cs.CL cs.DCcs.PF

TokTier: Exact Stateful Tokenization for Agentic LLM Serving

Zhenyu Zhang , Zhichao Cao This is my paper
classification cs.CL cs.DCcs.PF
keywords stateful tokenizationagentic LLM servingincremental repairsplice certificateBPE tokenizationGPU tokenizationprefix cachingexactness contract
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large-language-model serving systems cache the model's hidden state but re-tokenize the entire request text on every call; for coding agents that resubmit a long transcript after appending a small tool result, this makes tokenization a growing share of time to first token. TokTier claims tokenization can be treated as stateful: keep the previous turn's token IDs, re-tokenize only the new text plus a small window, and splice the old and new sequences after proving a stable boundary exists, with a full re-tokenization fallback when the proof fails. For requests with no reusable prefix, it claims the regex pre-tokenizer can be rewritten into run-local parallel rules that produce exactly the same piece boundaries, enabling exact full tokenization on a GPU. If the contract holds, session tokenization cost follows the appended change rather than context length, and full tokenization of million-character contexts drops to under a millisecond.

Core claim

TokTier's central claim is a contract: the token IDs it emits are always identical to those of full reference tokenization of the request text. It meets the contract with two mechanisms: incremental repair, which re-tokenizes a window around the appended text and accepts a splice only when the window contains a character-class transition that provably resets the pre-tokenizer, and exact GPU tokenization, which decomposes the sequential regex pre-tokenization into per-character piece-start predicates computed from maximal character-class runs. The paper reports zero divergence over 1.5×10^10 split checks, a 12.4 TB real-text corpus, and 93,000+ replayed agent steps, with repair latency stayin

What carries the argument

The load-bearing identity is the splice certificate: a junction inside a matched run of token records that satisfies two conditions — no pre-tokenization unit straddles it, and both the cached and fresh windows produce identical units on each side. The certificate is discharged family-by-family using synchronizing boundaries, character-class transitions where the pre-tokenizer's output to the right provably does not depend on text to the left; on the GPU path, run decomposition turns the regex's leftmost-match dependency into independent per-character piece-start predicates over maximal character-class runs. These objects carry the exactness argument.

Load-bearing premise

The load-bearing premise is that the tokenizer's model stage is per-piece and stateless, and that the per-family synchronizing-boundary transitions capture every way the reference regex's lookahead can couple across a boundary; a deployed tokenizer that violates either can make a checked splice silently diverge.

What would settle it

Run an accepted splice on a tokenizer family whose pre-tokenization regex has a lookahead longer than the repair window or whose BPE merge stage carries state across pieces, then compare every emitted ID against full reference tokenization of the same text; a single mismatch on an input that passes the stable-boundary check would refute the contract.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Tokenization front ends can become stateful without changing model-visible outputs; every emitted sequence matches full reference tokenization, so prefix-cache keys remain valid.
  • For agent workloads where the median append is about 1.4K characters, tokenization time stops growing with context; repair stays flat from 100K to 3M characters.
  • Rare full-context calls — session starts and rebuilds, 1–3.6% of traffic — can be served by exact GPU tokenization, removing the CPU tail that dominates burst latency.
  • A service built this way sustains 1,821 requests/s under a 50 ms P99 objective with four repair cores and one GPU, versus 40 requests/s for a stateless CPU front end on the same host.
  • The same protocol handles mid-context edits and multiple frozen tokenizer versions; unsupported families fall back to full reference tokenization without violating the contract.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The exactness contract is established for the tested frozen tokenizer configurations; a new tokenizer version must re-run family admission, so the method's generality hinges on how many families admit a synchronizing-boundary discharge.
  • The measured workload comes from six users' coding-agent traces plus public corpora; if other agent applications produce much larger appends relative to context, the repair advantage shrinks and the GPU reroute described in the appendix becomes the relevant path.
  • The same stateful-repair idea could transfer to other sequential text transforms that factor into a front-end stage plus a per-piece stateless model stage, provided a synchronizing-boundary set can be derived.
  • Shadow verification as a runtime guard suggests a general pattern for deploying exact fast paths: per-request certificates plus sampled differential checking against a pinned reference, rather than full formal verification.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. TokTier is a stateful tokenization service whose contract is that emitted token IDs always equal full reference tokenization of the request text. For session continuations, it re-tokenizes only a window around the appended text, verifies a stable pre-tokenization boundary, and splices cached and fresh token records; for requests without reusable session state, it performs full tokenization on a GPU using a run-local reformulation of GPT-family regex pre-tokenization. The paper reports extensive differential validation: 1.5e10 split-level checks, a 12.4 TB corpus sweep, 92,484 replayed certified splices, adversarial oracles, and a runtime shadow verifier. Measured latency is 0.5–1.1 ms for incremental repair from 100K to 3M characters, 0.87 ms for GPU full tokenization of a 1M-character request, and sustained serving improvements with vLLM are reported. The central exactness claim is supported by unusually strong empirical evidence, but I find a load-bearing inconsistency between the claimed O(Δ+w) repair complexity and the measured O(N) term in Appendix C, and an associated gap in the mechanism for establishing that a stored session prefix matches the new request.

Significance. If the claims hold, TokTier addresses a real and growing bottleneck in agentic LLM serving: full re-tokenization of long contexts on every call despite KV-cache reuse. The validation methodology is a particular strength: version-pinned differential testing at both pre-tokenization-boundary and final-ID levels, full-corpus sweeps, certified splice replay, and a shadow verifier that caught a history-dependent bug in a widely deployed external tokenizer. This is substantially stronger evidence than is typical for systems papers. The workload characterization, with median appends of ~1.4K characters and 94.1% prompt-cache hit rates, is also valuable. However, the exactness and complexity story as written is internally inconsistent: the repair path appears to include an O(N) prefix-verification scan that is not accounted for in the O(Δ+w) claim, and the mechanism for guaranteeing prefix equality for a 'session state hit' is not described.

major comments (2)
  1. [§3.4, Appendix C, Table 5] The headline complexity claim is 'O(Δ+w) work end to end, independent of N' (§3.4), echoed in the abstract as 'follows the append.' Appendix C's own measurements of the shipped repair path show a clear O(N) term: at Δ=1K, P50 grows from 0.43 ms at 100K context to 6.17 ms at 8M context, a 14× increase for an 80× context growth, and the text labels this 'the prefix-verification scan.' This scan is not mentioned in §3.2 or §3.4. The contradiction is load-bearing: either the repair protocol performs a full-prefix verification, in which case the O(Δ+w) claim is false and all statements derived from it (§1, §5.1, §5.3, §8) need revision; or there is no full-prefix verification, in which case a mid-context edit outside the repair window (e.g., at position N−1024 with the default w=512) would retain stale cached IDs before the window and violate the 'always identical' contract. The authors must
  2. [§3.2, §3.4] The repair protocol's correctness depends on the assumption that a 'session state hit' actually has the stored text as a prefix of the new request, but no data structure for establishing this in O(1) is described. The session store keeps token IDs, byte spans, a position index, and the tokenizer hash (§3.4); it does not include a text hash, Merkle tree, or any other prefix-authentication mechanism. The request arrives as full text. Without a full-prefix check, an edit at position N−1024 with w=512 would produce a splice of stale cached IDs before the window and fresh IDs after it, with no signal. With a full-prefix check, the repair cost is at least O(N) unless a more sophisticated mechanism is introduced. The sentence in §3.2 that 'the same procedure applied on both sides of an edit handles mid-context mutations' assumes the edit position is known; the paper does not describe how the sy
minor comments (5)
  1. [§1 / Table 1] The abstract and Table 1 report 17 tokenizer families, but Table 4 lists six frozen tokenizers and the split-level campaigns cover four pattern families. The mapping from '17 families' to the tested configurations should be stated explicitly, since the reader cannot otherwise tell how many distinct pre-tokenizer rules were actually validated.
  2. [§5.4, Table 3] The P90 value of 133.1 ms at 4.4M characters in the array-delivery row is a striking outlier caused by the NFC quick-check fallback. The text explains this, but the row would be clearer if the fail-fast vs. fallback split were shown directly in the table, since the P90 alone invites misreading.
  3. [§3.1] The claim that 'our adversarial oracle produced cascading counterexamples for every bounded-radius rule we formulated' is central to motivating the certificate approach, but no definition of the oracle or the rule space is given in the main text. A brief pointer to the companion document and a one-sentence description of the input generator would help reproducibility.
  4. [§5.8] The history-dependent Rust tokenizer bug is reported as 'identification is pending maintainer confirmation.' If the reproduction is deterministic and has been shared upstream, this is fine; please state whether the artifact includes the reproduction script, since this is the strongest evidence for the shadow verifier's value.
  5. [Appendix C] The appendix labels the Δ=1K column's context growth as 'the prefix-verification scan,' but no such scan appears in the protocol description of §3.2. This is the same issue as Major Comment 1; please ensure the appendix protocol description is updated in lockstep with the main text.

Circularity Check

0 steps flagged

No significant circularity: the exactness argument is a theorem with explicit stated assumptions plus differential validation against an independent frozen reference, not a fitted or self-cited result.

full rationale

The paper's central derivation chain is not circular. The splice theorem (Appendix A.2) reduces exactness of repair to the existence of a splice certificate (Definition A.4), under the explicit Assumption 2 that the model stage is per-unit and stateless. The real work is the per-family discharge of conditions (C1)/(C2) for each tokenizer's pre-tokenizer, with the paper reporting split-level and end-to-end differential campaigns against a pinned HuggingFace reference. No free parameter is fitted to make a correctness claim true, and the reference tokenizer is an external frozen artifact rather than an output of the system. The GPU run-decomposition is derived constructively from regex alternatives and then checked at piece-boundary level against the same reference; again there is no fitted coefficient or renamed target. There are no load-bearing self-citations: the per-family discharge proofs are deferred to a companion artifact, which is a completeness gap in the paper as a standalone document, but not a circular reduction. Separately, the claimed O(Δ+w) end-to-end work in §3.4 is in tension with the O(N) 'prefix-verification scan' acknowledged in Appendix C; that is a consistency/correctness risk, not a circularity. Accordingly, the honest finding is no significant circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

No invented entities: 'splice certificate', 'synchronizing boundary', and 'run decomposition' are analytic techniques, not postulated system/physical entities needing external falsifiable handles. No free parameter is fitted to make a performance or correctness claim true: window size, thresholds, and retries are search parameters with provable fallback (miss ⇒ widen ⇒ full retokenization), and the per-tokenizer boundary sets are derived from or probed against the frozen reference. The load-bearing assumptions are the per-piece statelessness of the model stage (Appendix A.2 Assumption 2, explicitly named by the authors), the run-local reducibility of the production regex families (§4.1), and the choice of the pinned HuggingFace implementation as ground truth.

free parameters (4)
  • repair window size w = 512 characters (default)
    Window re-tokenized around the append (§3.2). A search parameter, not a correctness parameter: a miss widens the window (doubling, ≤5 retries) or falls back to full tokenization. Chosen for latency/coverage tradeoff; 99.995% of 56,052 real splices accept on the first window.
  • splice certificate thresholds = run ≥ 2 tokens; run covers > 128 normalized chars (probed per tokenizer); ≤ 5 widening retries
    Rules to exclude accidental single-token matches and bound fallback cost (§3.2). Correctness-neutral because failure always falls back to more work, never to different IDs.
  • per-tokenizer max added-token literal length and synchronizing-boundary sets = max literal 6–30 chars (Table 4); boundary sets per family (Appendix A.3)
    Constants for the leftmost-longest literal splitter and the stable-boundary certificate; derived from or probed against the frozen reference (class tables are probed directly out of the reference engine after the Unicode-skew episode, §5.2), not fitted to target outputs.
  • shadow verifier sampling rate = 5% in serving runs, 100% offline
    Operational parameter trading verification cost against detection latency (§5.8); does not affect correctness of emitted IDs.
axioms (6)
  • domain assumption Model stage (BPE/WordPiece) is per-piece and stateless across pieces, with no cross-piece state (Assumption 2).
    Load-bearing for the splice theorem and the GPU equivalence. The paper labels it 'the load-bearing assumption' (Appendix A.2), verifies it by differential campaigns on six pinned tokenizers, and §5.8 shows an external production engine violating its analogue with history-dependent IDs.
  • standard math Protocol: chunks are encoded with add_special_tokens=False, no padding, no truncation; sequence-level post-processing is applied once after the merge (Assumption 1).
    Defines the merge protocol under which the concatenation homomorphism (Lemma A.3) is proven. Checked configuration, not an argument about all tokenizers.
  • standard math BPE parallel-schedule lemmas: a round's minimum merge rank identifies one pair value; merging all leftmost non-overlapping occurrences of that pair equals iterated single merges; only pairs adjacent to a completed merge must be probed again.
    Licenses the parallel GPU BPE schedule (§4.2). Standard consequences of deterministic rank-order BPE, stated but not proven in the paper.
  • domain assumption Production GPT-family pre-tokenizers reduce to a four-class run-local predicate with ≤4-character lookback and three run-level aggregates; no backtracking beyond this locality is needed.
    Central to exact GPU pre-tokenization (§4.1). Constructively derived per regex alternative and validated by 1.5e10 split-level checks plus a 12.4 TB sweep, but not formally proven.
  • domain assumption The frozen HuggingFace reference implementation (tokenizers==0.22.2) is the ground truth for token IDs and source spans.
    Exactness is defined as equality to this pinned reference; all differential campaigns, the splice-certificate checks, and the shadow verifier compare against it (§5.2, §4.5).
  • domain assumption Workload-trace collection rules (count-only extraction, duplicate/replay detection, HMAC hashing) faithfully isolate true agent calls.
    The workload characterization (94.1% hit rate, median 1.4K-char appends, 1.0–3.6% initializations) rests on these parsing/dedup rules (Appendix B), which removed 26,578 phantom calls; cross-checked against provider aggregates, codex_swebenchpro, and TraceLab.

pith-pipeline@v1.3.0-daily-deepseek · 32668 in / 23927 out tokens · 259569 ms · 2026-08-03T01:59:24.869957+00:00 · methodology

0 comments
read the original abstract

LLM serving systems cache prompt KV state, yet most front ends still re-tokenize the full request text on every call. The cost lands on coding agents, which resubmit a long transcript after each small tool result, and reuse is hard because even a short append can change token boundaries near the end of the previous sequence. Across 153,951 calls from two agent ecosystems, the median call appends about 1.4K characters, and only 1.0-3.6% of calls start or rebuild a session with contexts of millions of characters. At a 94.1% fleet prompt-cache hit rate, tokenization reaches up to 64% of time to first token. TokTier is a stateful tokenization service with one contract: emitted token IDs are always identical to full reference tokenization of the request text. For a session continuation, it re-tokenizes a small window around the append and splices only after a per-request stable-boundary check, widening the window or falling back to full tokenization on failure. For a call without a reusable prefix, it decomposes GPT-family regex pre-tokenization into run-local rules and runs exact pre-tokenization and BPE on a GPU. A sampled shadow verifier re-checks live traffic. Across 17 tokenizer families, differential campaigns cover 1.5x10^10 split checks, a 12.4 TB real-text corpus, and 93,000+ replayed agent steps, with zero divergence. Incremental repair takes 0.5-1.1 ms from 100K to 3M characters, up to 437x faster than HF tokenization and 2.1x faster at 1M than the strongest cache-based baseline (Gigatoken) fully prewarmed. GPU full tokenization encodes a 1M-character request in 0.87 ms, up to 491x below HF and 23.4x below the fastest published CPU method. With vLLM, median time to first token drops 16-34% and P99 drops 23% under recorded bursts. Under a 50 ms P99 objective, four repair cores plus one GPU sustain 1,821 requests/s where a 16-core stateless front end saturates at 40.

Figures

Figures reproduced from arXiv: 2607.29678 by Zhenyu Zhang, Zhichao Cao.

Figure 1
Figure 1. Figure 1: One agent turn through today’s stack and through TokTier. The request appends a median ∼1.4K characters to a transcript that was tokenized on previous turns, and today’s front end re-scans the complete context before the prefix cache can act. TokTier repairs session continuations around the append, routes rare initializations and rebuilds to an exact GPU path or the reference CPU path, and samples all outp… view at source ↗
Figure 2
Figure 2. Figure 2: Joint distribution of context size 𝑁 and new tokens per call Δ over the 153,951 interactive calls, in the API’s token accounting. Session continuations concentrate one to three orders of magnitude below the diagonal, so most calls add little text to a large context. Session initializations and rebuilds sit on the Δ=𝑁 diagonal and carry complete contexts up to 106 tokens. Marginal distributions and per￾sour… view at source ↗
Figure 3
Figure 3. Figure 3: Request lifecycle through TokTier. The router checks for live session token state, then sends session contin￾uations to boundary repair, large state-miss segments to the GPU path, and small segments or any fast-path failure to the reference CPU path. The session store keeps token IDs and byte spans per live session. A shadow verifier re-tokenizes a sampled fraction of emitted IDs against the reference engi… view at source ↗
Figure 4
Figure 4. Figure 4: A boundary inside the word “pipeline” changes the token sequence (Llama-3.1-8B tokenizer, real token IDs below each box). Independent tokenization produces two tokens (␣pipe+line) where full tokenization produces one (␣pipeline). Repair re-tokenizes the affected region and reuses the cached prefix only after finding a stable boundary, reproducing the serial stream bit for bit. lifetimes. Large state-miss s… view at source ↗
Figure 6
Figure 6. Figure 6: Run decomposition. The reference regex exposes a serial dependency between matches. The equivalent formulation computes character classes, maximal runs, and a local piece-start predicate with parallel passes. The resulting pieces feed size-specialized BPE kernels. 4.2 GPU BPE After pre-tokenization, each piece is encoded independently. Merge ranks, vocabulary entries, and byte strings are packed into GPU h… view at source ↗
Figure 7
Figure 7. Figure 7: Performance against every measured baseline. (a) Single-request P50 latency versus complete context on identical real texts. Incremental-repair lines price one append on a live session under the protocol of §5.3, and full-tokenization lines price a full encode of a fresh request under the protocol of §5.4. Gigatoken runs in its most favorable, fully prewarmed mode and wins below the 100 K–500 K crossover. … view at source ↗
Figure 8
Figure 8. Figure 8: Incremental repair latency versus complete context length on the public-trace replay (log–log, solid P50, dashed P90). The Rust session store keeps repair at 0.5–1.1 ms from 100 K to 3 M characters, while full CPU re-tokenization grows with context size. The dotted line is the prior Python-store implementation on the same protocol [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Boundary-check behavior over 56,052 replayed real splices (SWE-smith streams under three families, plus the public Codex trace). (a) The default 512-character window accepts 56,049 splices on the first attempt at every append size, three splices widen the window once, and none falls back to full retokenization. (b) Repair wall time grows with the append and not with the context, as 𝑂(Δ + 𝑤) predicts. Laten… view at source ↗
Figure 10
Figure 10. Figure 10: Full tokenization single-request encode P50 (Qwen3 family, steady state). (a) With array token deliv￾ery, every dispatch variant stays near or below one millisec￾ond across three decades of request size, and the consumer RTX 5090 leads the server card. (b) The same fused+graph kernels behind two delivery channels. Materializing a Python list[int] adds interpreter interface cost that dominates be￾yond 100 … view at source ↗
Figure 11
Figure 11. Figure 11: Tail behavior in recorded burst scenarios. Adding CPU workers does not remove the session-initialization service￾time floor (a), and at low worker counts initialization work drags continuation P99 up (b). A four-core repair pool plus one GPU keeps continuation and initialization P99 low. The host-contention companion panel is [PITH_FULL_IMAGE:figures/full_fig_p013_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: P99 sojourn time versus offered load (Poisson, measured mixture, 60 s steady state per point, and open mark￾ers are backlogged points reported as-is). The tier reaches 1,821 requests/s under a 50 ms P99 objective, while stateless CPU configurations saturate at 33–40 requests/s and only the GPU-only front end holds the 10 ms objective. board power. These figures support deployment feasibility, while the la… view at source ↗
Figure 14
Figure 14. Figure 14: What session state costs and what its lifetime buys. (a) Store memory for one growing session on the replayed real trace. Accounted state is ∼16 bytes per token, the Rust store’s resident growth stays between 1.2 and 22 MB per session across strains and families, and the prior Python store paid 70–129 MB for the same sessions. (b) Share of the 153,951 trace calls that would find live token state, as a fun… view at source ↗
Figure 15
Figure 15. Figure 15: Anatomy of coding-agent tokenization workloads across nine machines, two ecosystems, and a public autonomous￾agent trace. (a) Per-call increment Δ. (b) Per-call cache hit ratio ℎ. (c) Context size 𝑁. (d) LLM calls per human turn. (e) Turn-gap distribution against the 5-minute and 1-hour cache TTLs. (f) Tokens-to-process per call, session continuations versus session initializations. Continuations process … view at source ↗
Figure 16
Figure 16. Figure 16: Per-core context throughput on the replay, two accounts kept separate (log–log). Under the served account, repair passes every 𝑂(𝑁) engine as context grows, crossing Gigatoken’s prewarmed ceiling (355 MB/s) at ∼200 K bytes, while its scanned account stays at 1.8–3.6 MB/s. it has not seen before. On this replay corpus even 100 K of new transcript is mostly repeated pretokens, so its P50 at 8 M moves by und… view at source ↗
Figure 17
Figure 17. Figure 17: Host-contention companion panel to [PITH_FULL_IMAGE:figures/full_fig_p021_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Measured routing phase diagram on the append￾size × context plane, under exploration-prototype account￾ing rather than shipped defaults. Small green points are incremental repairs from the append-size sweep, and the six large markers are cells where a GPU full-tokenization rebuild and the best CPU repair option were measured head to head. Real appends concentrate far left of the GPU region, and past the G… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 8 linked inside Pith

  1. [1]

    Anthropic. 2024. Prompt Caching with Claude. Explicit 5-minute and 1-hour cache TTLs

    2024

  2. [2]

    Anthropic. 2026. Models overview.https://platform.claude.com/docs/ en/about-claude/models/overview. Claude Fable 5 and Claude Opus 5: 1M token context window. Accessed July 30, 2026

    2026

  3. [3]

    Euijun Chung, Yuxiao Jia, Aaron Jezghani, and Hyesoon Kim. 2026. Characterizing CPU-Induced Slowdowns in Multi-GPU LLM Inference. arXiv:2603.22774

    Pith/arXiv arXiv 2026

  4. [4]

    DeepSeek-AI. 2024. DeepSeek-V3 Technical Report. arXiv:2412.19437

    Pith/arXiv arXiv 2024

  5. [5]

    HuggingFace. 2019. HuggingFace Tokenizers.https://github.com/ huggingface/tokenizers

    2019

  6. [6]

    Inferact. 2026. codex_swebenchpro_traces: Agentic Workload Traces of Codex on SWE-Bench Pro. HuggingFace dataset, MIT license

    2026

  7. [7]

    Vibhu Jawa. 2021. Run State of the Art NLP Workloads at Scale with RAPIDS, HuggingFace, and Dask. NVIDIA Devel- oper Blog.https://developer.nvidia.com/blog/run-state-of-the-art- nlp-workloads-at-scale-with-rapids-huggingface-and-dask/

    2021

  8. [8]

    Shenghu Jiang and Ruihao Gong. 2026. Incremental BPE Tokenization. InProceedings of ICML. arXiv:2605.30813

    Pith/arXiv arXiv 2026

  9. [9]

    Jimenez, John Yang, Alexander Wettig, et al

    Carlos E. Jimenez, John Yang, Alexander Wettig, et al . 2024. SWE- bench: Can Language Models Resolve Real-World GitHub Issues?. In Proceedings of ICLR

    2024

  10. [10]

    Venu Gopal Kadamba and Kanishkha Jaisankar. 2026. GPUTOK: GPU Accelerated Byte Level BPE Tokenization. arXiv:2603.02597

    arXiv 2026

  11. [11]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica

  12. [12]

    Yuhan Liu, Hanchen Li, Yihua Cheng, et al. 2024. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving. InProceedings of SIGCOMM

    2024

  13. [13]

    George C. Necula. 2000. Translation Validation for an Optimizing Com- piler. InProceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI)

    2000

  14. [14]

    NVIDIA. 2024. NVIDIA Blackwell Platform: GB200 NVL72. Vendor claim: up to 30× LLM-inference throughput vs. the same number of H100 GPUs.https://nvidianews.nvidia.com/news/nvidia-blackwell- platform-arrives-to-power-a-new-era-of-computing

    2024

  15. [15]

    NVIDIA. 2025. NVIDIA Dynamo: A Datacenter-Scale Distributed Inference Serving Framework. Router documentation: backend handlers receive pre-tokenized requests

    2025

  16. [16]

    OpenAI. 2023. tiktoken: A Fast BPE Tokeniser for Use with OpenAI’s Models.https://github.com/openai/tiktoken

    2023

  17. [17]

    OpenAI. 2026. GPT-5.6 Sol Model | OpenAI API.https://developers. openai.com/api/docs/models/gpt-5.6-sol. Context window: 1,050,000 tokens. Accessed July 30, 2026

    2026

  18. [18]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, et al. 2024. Splitwise: Ef- ficient Generative LLM Inference Using Phase Splitting. InProceedings of ISCA. Azure LLM inference traces

    2024

  19. [19]

    Amir Pnueli, Michael Siegel, and Eli Singerman. 1998. Translation Validation. InTools and Algorithms for the Construction and Analysis of Systems (TACAS)

    1998

  20. [20]

    Ruoyu Qin, Zheming Li, Weiran He, et al. 2025. Mooncake: Trading More Storage for Less Computation — A KVCache-Centric Architec- ture for Serving LLM Chatbot. InProceedings of FAST

    2025

  21. [21]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. GPT-2; byte-level BPE with regex pre-tokenization

    2019

  22. [22]

    Marcel Rød. 2026. Gigatoken: SIMD and Cache Hierarchies for 1000x Faster Byte-Pair Encoding Tokenization on Modern CPUs.https: //github.com/marcelroed/gigatoken

    2026

  23. [23]

    Wei Shao, Lingchao Zheng, Pengyu Wang, Peizhen Zheng, Jun Li, and Yuwei Fan. 2026. LoPT: Lossless Parallel Tokenization Acceleration for Long Context Inference of Large Language Model. InProceedings of ACL. ACL Anthology 2026.acl-long.1529

    2026

  24. [24]

    Shubham Tiwari, Tapan Chugh, Nash Rickert, Simon Peter, Ratul Ma- hajan, and Haiying Shen. 2026. CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agents. arXiv:2606.16824

    arXiv 2026

  25. [25]

    vLLM contributors. 2026. RFC: Rust-Based Serving Frontend.https: //github.com/vllm-project/vllm/issues/40846; parity tracking #44280

    2026

  26. [26]

    Yuxin Wang et al. 2024. BurstGPT: A Real-World Workload Dataset to Optimize LLM Serving Systems. arXiv:2401.17644; KDD’25

    Pith/arXiv arXiv 2024

  27. [27]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent- Computer Interfaces Enable Automated Software Engineering. In Proceedings of NeurIPS. arXiv:2405.15793

    Pith/arXiv arXiv 2024

  28. [28]

    Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Find- ing and Understanding Bugs in C Compilers. InProceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI)

    2011

  29. [29]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. InProceedings of ICLR. arXiv:2210.03629

    Pith/arXiv arXiv 2023

  30. [30]

    Amos You. 2025. BlockBPE: Parallel BPE Tokenization. arXiv:2507.11941; ICML 2025 Workshop

    Pith/arXiv arXiv 2025

  31. [31]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. InProceedings of OSDI

    2022

  32. [32]

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, et al. 2024. SGLang: Efficient Execution of Structured Language Model Programs. InPro- ceedings of NeurIPS. RadixAttention prefix caching

    2024

  33. [33]

    Yinmin Zhong, Shengyu Liu, Junda Chen, et al . 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized Large Language Model Serving. InProceedings of OSDI. 16 TokTier: Exact Stateful Tokenization for Agentic LLM Serving

    2024

  34. [34]

    HF tokenizers

    Kan Zhu, Mathew Jacob, Chenxi Ma, Yi Pan, Stephanie Wang, Arvind Krishnamurthy, and Baris Kasikci. 2026. TraceLab: Characterizing Coding Agent Workloads for LLM Serving. arXiv:2606.30560; dataset CC BY 4.0, github.com/uw-syfi/TraceLab v0.0.1. A The Splice Theorem This appendix states and proves the losslessness theorem behind the incremental repair’s spli...

    Pith/arXiv arXiv 2026

  35. [2023]

    InProceedings of SOSP

    Efficient Memory Management for Large Language Model Serving with PagedAttention. InProceedings of SOSP