TokenBreak: Bypassing Text Classification Models Through Token Manipulation

Kasimir Schulz; Kenneth Yeung; Kieran Evans

REVIEW 5 major objections 5 minor 26 references

TokenBreak: Bypassing Text Classification Models Through Token Manipulation

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TokenBreak shows that prepending a single letter to words that drive a text classifier's verdict can flip BPE- and WordPiece-based detectors into false negatives while leaving the text fully readable to downstream targets.

desk verdict A plausible tokenizer-level attack with a load-bearing semantic-preservation claim that outruns its evidence; worth refereeing after major revision. read the letter →

arxiv 2506.07948 v1 pith:O4KTTK5I submitted 2025-06-09 cs.LG cs.CR

Kasimir Schulz , Kenneth Yeung , Kieran Evans This is my paper

classification cs.LGcs.CR

keywords TokenBreakadversarialtextattacktokenizationclassificationpromptinjectionBPEWordPieceUnigram

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces TokenBreak, an attack that prepends a single character to the words that most influence a text classifier's verdict, so that models using BPE or WordPiece tokenization misclassify malicious input as benign. The manipulated text remains fully understandable to downstream targets, so an LLM or human reader can still be affected by the attack the classifier was meant to stop. Tests on nine models across prompt-injection, spam, and toxicity detection show that BPE and WordPiece models are susceptible, while three Unigram models were not evaded. The paper also proposes a defense that passes input through a Unigram tokenizer before feeding the model, which reduces attack success rates without retraining.

What carries the argument

The attack algorithm BreakPrompt scores each word by classification confidence, then tries prepending each alphabet letter until one flips the classifier to 'benign' with high confidence. The mechanism that makes it work is the difference between left-to-right subword segmentation (BPE, WordPiece) and probability-based segmentation (Unigram): a prepended letter shifts the start of the word and forces the left-to-right tokenizers to split the salient word into different, less meaningful subword pieces, while Unigram finds the most probable segmentation regardless of the added character and keeps the salient token intact. The defense, tokenizer translation, runs the input through a Unigram tokenizer first and maps the resulting tokens into the target model's vocabulary, restoring the salient subword boundaries.

What would settle it

Run TokenBreak on a held-out set of BPE, WordPiece, and Unigram models from families not tested in the paper; finding a single Unigram model that is evaded or a single BPE model that resists would contradict the claim that tokenizer type alone determines susceptibility.

Watch

Extended reading notes

Core claim

The central claim is that the tokenizer's word-splitting strategy, not the model's learned weights, is the attack surface: prepending a character to a high-impact word disrupts left-to-right BPE and WordPiece segmentation so that salient subwords no longer appear as single meaningful tokens, pushing the classifier to a false negative. Unigram tokenization, which segments by corpus-frequency probabilities rather than left-to-right merges, retains those salient subwords and therefore resists the attack. The authors argue this is a model-level vulnerability because tokenizer type is tied to model family, so an operator can predict susceptibility from the choice of model.

Load-bearing premise

The paper assumes that the nine selected checkpoints, one to three per model family, represent all models using each tokenizer algorithm, so that Unigram robustness and BPE/WordPiece vulnerability generalize to any model of those types.

Editorial extensions

If this is right

Protection models built on BPE or WordPiece tokenizers can be bypassed by single-character prefixing without retraining the attack per target.
Unigram-based classifiers tested (DeBERTa-v2, XLM-R) were not evaded, indicating that changing tokenizer type can harden a system.
Because tokenizer choice is tied to model family, an operator can assess vulnerability simply from the model architecture.
The tokenizer-translation defense reduces average TokenBreak success from 33.09% to 12.63% across BPE and WordPiece models, though WordPiece spam detection still saw 29.74% success.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If vulnerability is driven by left-to-right segmentation rather than model family, other left-to-right subword tokenizers beyond BPE and WordPiece may be similarly evadable; this is an inference, not tested in the paper.
The defense's residual success rates, especially 29.74% on WordPiece spam, suggest that tokenizer translation alone is not a complete fix; combining it with adversarial training or input filtering would be a natural next test.
The attack's single-character constraint is one point on a broader space of token-boundary perturbations; testing longer prefixes, infixes, or Unicode confusables would reveal how far the vulnerability extends.
A practical consequence the paper does not state is that model documentation should expose tokenizer type as a security-relevant property for deployed classifiers.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a real, modest empirical result that is overclaimed in one important place. The new bit is a systematic comparison: prepending a single character to high-impact words flips BPE and WordPiece classifiers across prompt-injection, spam, and toxicity tasks, while the three Unigram checkpoints they tried resisted; and a tokenizer-translation defense reduces success on most of those models. The mechanism—left-to-right BPE/WordPiece let the prepended letter disturb the segmentation, while Unigram can still extract the original word as a high-probability token—is plausible, and the worked tokenizer examples make it easy to follow. The writing is clear and the related work is honest about ORTH, Deep-WordBug, and Tokenization Confusion.

The stress-test concern is on target and is the biggest soft spot. The abstract's central claim is that manipulated text remains understandable and actionable to the downstream target, but the support is two LLM demos and three eyeballed spam/toxicity strings. Algorithm 1 only checks classifier confidence on isolated words; it never checks readability, meaning, or actionability. Some examples are still parseable, but 'dchat', 'isexy', and 'cfucking acunt' are not obviously fine, and the LLM demo itself notices the request is odd. The correct scope is 'some manipulated examples remain understandable,' not a general property of the technique. Second, no model identifiers, no code, and no confidence intervals are reported, so the numbers are not independently checkable. Third, the Unigram robustness claim comes from three checkpoints; 'models using this type of tokenizer are not susceptible' should be 'these three Unigram models were not evaded by this particular single-prefix attack.' The family-to-tokenizer mapping is less soft because public HuggingFace docs corroborate it. Finally, the defense is under-evaluated: no false-positive rate on benign inputs, no cost analysis, and in the BPE spam case success barely moved (4.28 to 4.08).

I would not desk-reject this. The mechanism is useful and the empirical pattern deserves referee time, but only after a major revision that names the checkpoints, releases artifacts, reports error bars, measures semantic preservation systematically, and tests the defense on benign inputs. Who should read it: NLP security researchers and practitioners choosing guardrail models; treat it as a warning that tokenizer choice matters, not as a definitive vulnerability map.

Referee Report

5 major / 5 minor

Summary. The paper introduces TokenBreak, an adversarial attack that prepends a single letter to selected words in an input text to make BPE- and WordPiece-based classifiers misclassify malicious prompts as benign, while aiming to preserve the text's meaning for the downstream target. The attack is evaluated on nine HuggingFace classifiers spanning prompt-injection, spam, and toxicity detection, with three tokenizer types. The paper also proposes a defense that passes input through a Unigram tokenizer before the model's own tokenizer. The central claims are that the attack preserves semantic intent, that Unigram-based models are immune, and that vulnerability can be predicted from model family.

Significance. If the results hold, TokenBreak is a practical, low-cost attack on common tokenizer-based guardrails, and the proposed defense is a simple mitigation that does not require retraining. The paper's strength is its attempt to demonstrate the attack across three distinct detection tasks and to validate a specific defense. The quantitative success rates are plausible, and the zero-success result for all three Unigram models is an interesting empirical observation. However, the paper lacks the reproducibility details and the systematic semantic-preservation evaluation needed to support the general claims in the abstract and conclusion, and the algorithmic description contains a serious flaw. With revisions, the core empirical findings could be a useful contribution.

major comments (5)

[§3.1, Algorithm 1] The deep-check branch in Algorithm 1 is an infinite recursion: the while loop increments the local variable threshold and then calls BreakPrompt(prompt) recursively with the original prompt. Because threshold is reset to 0.995 at the start of each call, the loop condition threshold<0.9999 is always true, so the function never returns. As written, this branch cannot produce any result, yet the results in Section 4 are attributed to an automated testing process that presumably included deep-check behavior. Please correct the pseudocode or specify how the deep-check mechanism was implemented in practice.
[§4.1, Tables 1, 4, and 7] No HuggingFace checkpoint identifiers are provided for the nine models, so the experiments are not reproducible. Additionally, each success rate is a single point estimate from 1000 samples with no confidence intervals or repeated runs; this is particularly problematic for the zero-success Unigram results in Tables 1, 4, and 7, where a single misclassified sample would change the conclusion. Please provide model names, tokenizer versions, and statistical uncertainty estimates.
[Abstract and §4.5] The claim that manipulated text 'can still be understood and respond' by the end target is load-bearing but is supported only by two LLM demonstrations (Tables 2 and 3) and a handful of visually inspected strings (Tables 5, 6, 8, and 9). Algorithm 1 does not optimize for or verify readability, and the LLM example in Table 2 shows the model noting the request is 'odd,' so actionability is not guaranteed in general. Please provide a systematic comprehension evaluation on a random sample of successful evasions, or explicitly narrow the claim to the demonstrated examples.
[§4.5 and Section 7, Table 14] The family-to-tokenizer mapping is derived from one to three checkpoints per family, yet Section 4.5 and Section 7 generalize to all models of those families and to the tokenizer classes, e.g., 'DistilBERT models use WordPiece, RoBERTa models use BPE and DeBERTa-v2 models use Unigram.' This overgeneralizes from a sample of nine models. Other checkpoints in the same families may use different tokenizers. Please either rephrase the claims as applying to the tested checkpoints or provide a broader survey to support the model-level vulnerability framing.
[§6.3, Tables 12 and 13] The defense reduces mean success from 33.09% to 12.63%, but the per-task results are uneven: for BPE spam the success rate is essentially unchanged (Table 12: 4.28% to 4.08%), and for WordPiece spam it remains at 29.74% (Table 13). The conclusion that the defense 'significantly reduced the susceptibility' should be qualified by per-task residual risk and by the assumption that a Unigram tokenizer is available for the translation step.

minor comments (5)

[§4.6.1] The marker character is rendered as '˙G', which is likely intended to be 'Ġ'; please use the correct Unicode character for clarity.
[References] Reference [5] (Tokenization Confusion) lacks author names, publication venue, and access date; please complete the bibliographic entry.
[Algorithm 1, line 9] The condition 'if cls = 1 or conf < threshold' is ambiguous: it should specify whether conf is the confidence for the positive class or the overall confidence.
[§4.1] Please state how the 1000 samples were distributed across the multiple datasets listed per task (e.g., how many from each dataset and whether the split was stratified).
[§4.2] The phrase 'actioned by the target LLM' is informal; consider replacing with 'acted upon' throughout.

Circularity Check

1 steps flagged · score 2.0 of 10

No significant circularity; the only same-author ingredient is a minor, non-load-bearing ShadowGenes family-identification step.

other [Section 4.5 and Section 7, Table 14]
"An apparent link between model family and tokenizer was also observed. It appears that DeBERTa typically leverages Unigram, RoBERTa typically leverages BPE, and DistilBERT and BERT typically leverage WordPiece. We go into this in more detail in section 7. We were able to validate the model families using the ShadowGenes[13] technique."

ShadowGenes is a same-author prior work, so the model-family identification that supports the 'model-level vulnerability' framing is self-cited. The self-citation is not load-bearing, however: Table 14 maps family to tokenizer using external HuggingFace documentation quoted in Section 7, and the TokenBreak success rates are measured on external checkpoints. This step does not reduce the paper's conclusions to its inputs.

full rationale

This is an empirical adversarial-attack paper, not a derivation from first principles. The central success rates are measured against nine external HuggingFace checkpoints, and Algorithm 1's internal test (c=0 and con>=threshold) is simply the definition of a false negative, so reporting 'TokenBreak Success' as the rate of induced false negatives is a benchmark outcome rather than a conclusion hidden in the input. The hand-set 0.995 threshold tunes the attack procedure, not the reported result. The only same-author ingredient is ShadowGenes, used in Sections 4.5 and 7 to identify model families; the family-to-tokenizer mapping is independently corroborated by HuggingFace documentation, so this self-citation is minor and non-load-bearing. The paper's genuine weaknesses are evidentiary, not circular: the 'still understood by the target' claim rests on two LLM demonstrations (Tables 2 and 3, including Qwen finding the request 'odd') and three visually inspected spam/toxicity examples (Tables 5-6 and 8-9), because Algorithm 1 never verifies readability or actionability; and Section 7 generalizes a model-level vulnerability from one to three checkpoints per family. These are correctness and robustness limitations, not circular reductions. Score 2 reflects the minor self-citation with no load-bearing circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on domain assumptions about tokenizer behavior and model-family mapping rather than on fitted constants. The only hand-fitted quantities are attack-algorithm thresholds. No new physical or theoretical entities are introduced.

free parameters (2)

high-confidence word selection threshold = 0.995
Algorithm 1 selects candidate words only if the classifier confidence is at least 0.995; this hand-set threshold controls which words receive a prefix and is not derived from any theory.
deep-check acceptance threshold = 0.9999
The deep check loop in Algorithm 1 raises the threshold in 0.0001 steps up to 0.9999 to seek a full-prompt misclassification; this is a hand-set stopping rule.

assumptions (4)

domain assumption BPE and WordPiece tokenizers split words left-to-right, while Unigram selects the highest-probability segmentation using corpus frequencies.
Used in Section 4.7.1 to explain why prefix insertion breaks BPE and WordPiece but not Unigram; the causal link between tokenizer algorithm and robustness is assumed rather than proven.
domain assumption Model family maps deterministically to tokenizer type for the relevant HuggingFace models.
Section 7 and Table 14 infer a family-to-tokenizer mapping from nine models plus documentation; the family-level vulnerability prediction depends on this mapping.
domain assumption Semantic intent is preserved for all successfully manipulated samples.
The paper supports this with two LLM demonstrations and a few text snippets, not a systematic readability or task-completion evaluation.
domain assumption The Unigram translation defense can be inserted without retraining and without harming classification of benign inputs.
Section 6 reports attack-success reduction but does not measure false-positive rates or accuracy on unmodified benign samples after translation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of TokenBreak: Bypassing Text Classification Models Through Token Manipulation." pith.science (2026). https://pith.science/paper/O4KTTK5I

@misc{pith2026250607948,
  author       = {Pith},
  title        = {Pith review of: TokenBreak: Bypassing Text Classification Models Through Token Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O4KTTK5I}},
  note         = {Machine review of arXiv:2506.07948}
}

read the original abstract

Natural Language Processing (NLP) models are used for text-related tasks such as classification and generation. To complete these tasks, input data is first tokenized from human-readable text into a format the model can understand, enabling it to make inferences and understand context. Text classification models can be implemented to guard against threats such as prompt injection attacks against Large Language Models (LLMs), toxic input and cybersecurity risks such as spam emails. In this paper, we introduce TokenBreak: a novel attack that can bypass these protection models by taking advantage of the tokenization strategy they use. This attack technique manipulates input text in such a way that certain models give an incorrect classification. Importantly, the end target (LLM or email recipient) can still understand and respond to the manipulated text and therefore be vulnerable to the very attack the protection model was put in place to prevent. The tokenizer is tied to model architecture, meaning it is possible to predict whether or not a model is vulnerable to attack based on family. We also present a defensive strategy as an added layer of protection that can be implemented without having to retrain the defensive model.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

26 extracted references · 24 canonical work pages

[1]

Tricking llms into disobedience: Formalizing, analyzing, and detecting jailbreaks, 2024

Abhinav Rao, Sachin Vashistha, Atharva Naik, Somak Aditya, and Monojit Choudhury. Tricking llms into disobedience: Formalizing, analyzing, and detecting jailbreaks, 2024

work page 2024
[2]

Black-box gener- ation of adversarial text sequences to evade deep learning classifiers, 2018

Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. Black-box gener- ation of adversarial text sequences to evade deep learning classifiers, 2018

work page 2018
[3]

Hotflip: White- box adversarial examples for text classification, 2018

Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. Hotflip: White- box adversarial examples for text classification, 2018

work page 2018
[4]

Synthetic and natural noise both break neural machine translation, 2018

Yonatan Belinkov and Yonatan Bisk. Synthetic and natural noise both break neural machine translation, 2018

work page 2018
[5]

Tokenization Confusion.https://specterops.io/blog/2025/06/03/ tokenization-confusion/

work page 2025
[6]

Gandalf the red: Adaptive security for llms.arXiv preprint arXiv:2501.07927, 2025

Niklas Pfister, V´ aclav Volhejn, Manuel Knott, Santiago Arias, Julia Bazi´ nska, Mykhailo Bichurin, Alan Commike, Janet Darling, Peter Dienes, Matthew Fiedler, et al. Gandalf the red: Adaptive security for llms.arXiv preprint arXiv:2501.07927, 2025

arXiv 2025
[7]

Qwen/Qwen3-0.6B.https://huggingface.co/Qwen/Qwen3-0.6B

work page
[8]

Twitter Spam Detection Dataset.https://www.kaggle.com/datasets/ greyhatboy/twitter-spam-dataset

work page

Show all 26 references

[9]

Email Spam Detection Dataset.https://www.kaggle.com/datasets/ zeeshanyounas001/email-spam-detection
[10]

Jigsaw Toxic Comment Dataset.https://www.kaggle.com/datasets/ julian3833/jigsaw-toxic-comment-classification-challenge? select=train.csv
[11]

WikiPedia Toxicity Dataset.https://www.kaggle.com/datasets/ manishguptads/wikipedia-toxicity
[12]

YouTube Toxic Comment Dataset.https://www.kaggle.com/datasets/ reihanenamdari/youtube-toxicity-data
[13]

Shadowgenes: Leveraging recurring patterns within computational graphs for model genealogy, 2025

Kasimir Schulz and Kieran Evans. Shadowgenes: Leveraging recurring patterns within computational graphs for model genealogy, 2025

2025
[14]

Tokenizer Components.https://huggingface.co/docs/tokenizers/ en/components
[15]

Neural machine trans- lation of rare words with subword units, 2016

Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine trans- lation of rare words with subword units, 2016

2016
[16]

BPE - HuggingFace documentation.https://huggingface.co/learn/ llm-course/en/chapter6/5. 19
[17]

WordPiece - HuggingFace documentation.https://huggingface.co/ learn/llm-course/en/chapter6/6
[18]

Subword regularization: Improving neural network translation models with multiple subword candidates, 2018

Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates, 2018

2018
[19]

Unigram - HuggingFace documentation.https://huggingface.co/docs/ transformers/en/tokenizer_summary#unigram
[20]

RoBERTa - HuggingFace documentation.https://huggingface.co/ docs/transformers/en/model_doc/roberta
[21]

Roberta: A robustly optimized bert pretraining approach, 2019

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019

2019
[22]

DistilBERT - HuggingFace documentation.https://huggingface.co/ docs/transformers/en/model_doc/distilbert
[23]

DeBERTa-v2 - HuggingFace documentation.https://huggingface.co/ docs/transformers/en/model_doc/deberta-v2
[24]

co/docs/transformers/en/model_doc/xlm-roberta

XLM-RoBERTa - HuggingFace documentation.https://huggingface. co/docs/transformers/en/model_doc/xlm-roberta
[25]

Unsupervised cross-lingual representa- tion learning at scale, 2020

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm´ an, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representa- tion learning at scale, 2020

2020
[26]

SentencePiece GitHub Repository.https://github.com/google/ sentencepiece/blob/273449044caa593c2fd7eb7550cb3ab2cff93f1a/ python/sentencepiece_python_module_example.ipynb#L590. 20

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Tricking llms into disobedience: Formalizing, analyzing, and detecting jailbreaks, 2024

Abhinav Rao, Sachin Vashistha, Atharva Naik, Somak Aditya, and Monojit Choudhury. Tricking llms into disobedience: Formalizing, analyzing, and detecting jailbreaks, 2024

work page 2024

[2] [2]

Black-box gener- ation of adversarial text sequences to evade deep learning classifiers, 2018

Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. Black-box gener- ation of adversarial text sequences to evade deep learning classifiers, 2018

work page 2018

[3] [3]

Hotflip: White- box adversarial examples for text classification, 2018

Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. Hotflip: White- box adversarial examples for text classification, 2018

work page 2018

[4] [4]

Synthetic and natural noise both break neural machine translation, 2018

Yonatan Belinkov and Yonatan Bisk. Synthetic and natural noise both break neural machine translation, 2018

work page 2018

[5] [5]

Tokenization Confusion.https://specterops.io/blog/2025/06/03/ tokenization-confusion/

work page 2025

[6] [6]

Gandalf the red: Adaptive security for llms.arXiv preprint arXiv:2501.07927, 2025

Niklas Pfister, V´ aclav Volhejn, Manuel Knott, Santiago Arias, Julia Bazi´ nska, Mykhailo Bichurin, Alan Commike, Janet Darling, Peter Dienes, Matthew Fiedler, et al. Gandalf the red: Adaptive security for llms.arXiv preprint arXiv:2501.07927, 2025

arXiv 2025

[7] [7]

Qwen/Qwen3-0.6B.https://huggingface.co/Qwen/Qwen3-0.6B

work page

[8] [8]

Twitter Spam Detection Dataset.https://www.kaggle.com/datasets/ greyhatboy/twitter-spam-dataset

work page