A semantic-aware tokenizer that merges similar and low-entropy text spans cuts long-context token counts by up to 59% and inference latency by roughly 2x, with no reported quality loss.
Title resolution pending
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2025 1verdicts
REJECT 1representative citing papers
citing papers explorer
-
SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modeling
A semantic-aware tokenizer that merges similar and low-entropy text spans cuts long-context token counts by up to 59% and inference latency by roughly 2x, with no reported quality loss.