Entropy-guided pre-tokenization, especially a PMI plus left/right entropy combination with lambda equal to 4, raises BPE word-boundary F1 from 49.30 to 58.73 on a 2,255-sentence PKU subset.
and Tanaka-Ishii, K
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Entropy-Driven Pre-Tokenization for Byte-Pair Encoding
Entropy-guided pre-tokenization, especially a PMI plus left/right entropy combination with lambda equal to 4, raises BPE word-boundary F1 from 49.30 to 58.73 on a 2,255-sentence PKU subset.