Subword token granularity substantially changes language model surprisal's fit to human reading times, with an intermediate vocabulary size near 8,000 being most predictive in this paper's setup.
Title resolution pending
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
The Impact of Token Granularity on the Predictive Power of Language Model Surprisal
Subword token granularity substantially changes language model surprisal's fit to human reading times, with an intermediate vocabulary size near 8,000 being most predictive in this paper's setup.