REVIEW 6 cited by
The Foundations of Tokenization: Statistical and Computational Concerns
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Tokenization - the practice of converting strings of characters from an alphabet into sequences of tokens over a vocabulary - is a critical step in the NLP pipeline. The use of token representations is widely credited with increased model performance but is also the source of many undesirable behaviors, such as spurious ambiguity or inconsistency. Despite its recognized importance as a standard representation method in NLP, the theoretical underpinnings of tokenization are not yet fully understood. In particular, the impact of tokenization on language model estimation has been investigated primarily through empirical means. The present paper contributes to addressing this theoretical gap by proposing a unified formal framework for representing and analyzing tokenizer models. Based on the category of stochastic maps, this framework enables us to establish general conditions for a principled use of tokenizers and, most importantly, the necessary and sufficient conditions for a tokenizer model to preserve the consistency of statistical estimators. In addition, we discuss statistical and computational concerns crucial for designing and implementing tokenizer models, such as inconsistency, ambiguity, finiteness, and sequentiality. The framework and results advanced in this paper contribute to building robust theoretical foundations for representations in neural language modeling that can inform future theoretical and empirical research.
Forward citations
Cited by 6 Pith papers
-
From Language Models over Tokens to Language Models over Characters
A principled method computes and samples from the character-level distribution induced by any token-level language model, using a new covering enumeration with beam approximations.
-
Prompting Complexity: Shortest Prompts for Texts and Behaviors in LLMs
The paper defines prompting complexity as the length of the shortest plausible prompt that deterministically generates a target text with a fixed language model.
-
Zero-Shot Attribution for Large Language Models: A Distribution Testing Approach
Anubis re-frames LLM attribution as a distribution testing problem with EVAL access, and reports AUROC above 0.9 on code benchmarks with around 2000 samples, beating detectGPT.
-
Language Models over Canonical Byte-Pair Encodings
Enforcing canonical BPE tokenizations through conditioning or architectural constraints improves held-out likelihood for GPT-2 and Llama models.
-
Probabilistic Concept-Aware Steering for Trustworthy LLM Inference
PCS improves steering direction accuracy by adaptively sampling the intervention coefficient from a cosine-similarity-conditioned Gaussian, but its evaluation is partly circular because the optimal coefficient is chos...
-
DateLogicQA: Benchmarking Temporal Biases in Large Language Models
DateLogicQA evaluates 12 LLMs on 190 date-reasoning questions and claims separate representation-level and logical-level temporal biases, but the Semantic Integrity Metric is undefined.
Discussion (0). Continue with ORCID to comment.