REVIEW 6 cited by
Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Large Language Models (LLMs) have shown remarkable capabilities in language understanding and generation. Nonetheless, it was also witnessed that LLMs tend to produce inaccurate responses to specific queries. This deficiency can be traced to the tokenization step LLMs must undergo, which is an inevitable limitation inherent to all LLMs. In fact, incorrect tokenization is the critical point that hinders LLMs in understanding the input precisely, thus leading to unsatisfactory output. This defect is more obvious in Chinese scenarios. To demonstrate this flaw of LLMs, we construct an adversarial dataset, named as $\textbf{ADT (Adversarial Dataset for Tokenizer)}$, which draws upon the vocabularies of various open-source LLMs to challenge LLMs' tokenization. ADT consists of two subsets: the manually constructed ADT-Human and the automatically generated ADT-Auto. Our empirical results reveal that our ADT is highly effective on challenging the tokenization of leading LLMs, including GPT-4o, Llama-3, Deepseek-R1 and so on, thus degrading these LLMs' capabilities. Moreover, our method of automatic data generation has been proven efficient and robust, which can be applied to any open-source LLMs. In this paper, we substantially investigate LLMs' vulnerability in terms of challenging their token segmentation, which will shed light on the subsequent research of improving LLMs' capabilities through optimizing their tokenization process and algorithms.
Forward citations
Cited by 6 Pith papers
-
Causal Estimation of Tokenisation Bias
Using regression discontinuity, the paper shows that adding a subword to a tokenizer's vocabulary can raise the model's probability for that string by up to about 17 times in small models.
-
Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinese
A new benchmark shows LLMs are more accurate in Simplified Chinese for regional terms but favor Taiwanese names in simulated hiring, revealing task-dependent bias between Chinese script variants.
-
Incorporating Domain Knowledge into Materials Tokenization
A domain-knowledge-weighted tokenizer, MATTER, preserves material concepts and improves materials NLP performance by 4% on generation and 2% on classification tasks.
-
Addressing Tokenization Inconsistency in Steganography and Watermarking Based on Large Language Models
Removing tokens that would not survive re-tokenization at generation time fixes steganographic extraction errors and slightly improves watermark detectability.
-
TASE: Token Awareness and Structured Evaluation for Multilingual Language Models
TASE benchmark shows LLMs lag humans on token-level and structural language tasks across Chinese, English, and Korean despite strong high-level performance.
-
Concept-Level AI for Telecom: Moving Beyond Large Language Models
A position paper proposing Large Concept Models as the successor to LLMs for telecom network management, without experimental evidence.
Discussion (0). Sign in to comment.