Pith. sign in

REVIEW 3 major objections 4 minor 48 references

The Impact of Token Granularity on the Predictive Power of Language Model Surprisal

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that subword token granularity is a first-order methodological variable in cognitive modeling: a vocabulary size around 8,000 makes surprisal the strongest predictor of naturalistic reading times, while coarser tokens…

desk verdict Systematic sweep of subword granularity shows tokenization matters for surprisal-based cognitive modeling, but the specific 8k optimum is not statistically supported. read the letter →

arxiv 2412.11940 v2 pith:YZ5ASFTY submitted 2024-12-16 cs.CL

Byung-Doh Oh , William Schuler This is my paper · ORCID
classification cs.CL
keywords surprisaltokengranularitysubwordtokenizationunigramlanguagemodelreadingtimesgarden-patheffectsstate-spacemodelscognitivemodeling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the choice of subword token granularity changes how well language-model surprisal, the negative log-probability a model assigns to a word, predicts human reading behavior. It trains unigram-tokenized Mamba-2 language models of three sizes with vocabularies from 256 to 128,000 tokens, then tests surprisal against ten measures from five reading-time corpora and against garden-path constructions. The paper concludes that token granularity matters substantially: before and after training, an intermediate vocabulary around 8,000 tokens produces surprisal that best fits naturalistic reading times, exceeding a widely used reference model. On garden-path sentences, coarser, more word-like tokens generally assign higher surprisal to critical regions, implying larger garden-path effects than previously reported. If correct, token granularity should be treated as a methodological variable rather than a neutral preprocessing choice.

What carries the argument

The central object is token granularity, operationalized by ULM, a unigram-language-model tokenizer that prunes a large character-based inventory down to a target vocabulary size. These tokenizers feed autoregressive Mamba-2 state-space language models in three sizes, and word probabilities are recovered through a leading-whitespace correction. Surprisal's predictive power is scored as the increase in regression model log-likelihood from adding surprisal to linear mixed-effects models of reading times, and garden-path effects are estimated by fitting a surprisal-to-reading-time linking function on filler items and comparing predicted reading times across ambiguous and control conditions.

What would settle it

A decisive check would be to retrain all eleven vocabulary conditions with several random seeds and more than one epoch, then test whether the roughly 8,000-token optimum in aggregate log-likelihood gain and the coarser-token garden-path trend still appear; if the optimum shifts or vanishes across seeds, the peak is an artifact of a single training run.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the subword vocabulary size is itself a cognitive-modeling parameter. Finer granularities make word surprisal track word length and character-level predictability, while coarser granularities make tokens increasingly word-like so that word surprisal approaches uniformity across common words and loses predictive power. The paper finds a sweet spot at roughly 8,000-token vocabularies on average across model sizes after training, with smaller models preserving an earlier peak near 4,000 tokens before training. In the garden-path experiment, coarser-grained tokens generally give larger differences in surprisal between ambiguous and unambiguous conditions at critical words, while still underestimating human garden-path effects by one or two orders of magnitude.

Load-bearing premise

The load-bearing premise is that Mamba-2 models trained for a single pass over Wiki-40B, with no repeated random initializations, stand in for language models generally; if longer training, a different architecture, or another seed changes how token granularity interacts with learning, the observed 8,000-token optimum and the garden-path granularity trend could be specific to this training regime.

Editorial extensions

If this is right

  • Vocabulary size alone can change a model's reading-time fit by hundreds of log-likelihood points, so studies using different tokenizers are not directly comparable.
  • The conventional vocabulary size around 50,000 is not optimal for naturalistic comprehension; models trained with roughly 8,000 tokens fit aggregate reading times better than the GPT-2 Small reference.
  • Coarser token granularity increases the raw surprisal difference between ambiguous and unambiguous garden-path sentences, which means tokenizer choice can inflate or deflate estimated garden-path effects.
  • The effect weakens as model size grows, so scaling may partially absorb the initial bias imposed by tokenization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An obvious extension is to repeat the vocabulary sweep on non-English corpora; because ULM uses characters as base units, writing systems with deeper orthography could shift the optimal vocabulary size away from 8,000.
  • The near-optimality of untrained 4,000-token surprisal suggests a cheap diagnostic: a researcher could select a tokenizer by reading-time fit on a small sample before committing to expensive language-model training.
  • If the size interaction generalizes, the practical recommendation is not a single best vocabulary but a size matched to model scale, with larger models tolerating coarser tokens.
  • A concrete test would be to hold the vocabulary at 8,000 and vary model size beyond the Large model used here; if the granularity peak flattens further, the cognitive relevance of token granularity may be a small-model phenomenon.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper investigates how subword token granularity (vocabulary size) affects the ability of language model surprisal to predict human reading times and to estimate garden-path effects. The authors train ULM tokenizers with 11 vocabulary sizes and Mamba-2 language models in three sizes, evaluate surprisal on 10 reading-time measures from five corpora, and on a garden-path benchmark following Huang et al. (2024). They report that an intermediate vocabulary size around 8,000 yields the best fit to naturalistic reading times (averaged over model sizes), and that coarser-grained tokens generally lead to larger surprisal differences at garden-path critical regions. They also compare with GPT-2 Small and provide untrained-model baselines to separate initial tokenization bias from learned representations.

Significance. If the findings are robust, the paper identifies a previously overlooked modeling decision—subword vocabulary size—as a first-order variable in cognitive modeling of reading. The experimental scope is substantial: 33 trained models, 11 vocabulary sizes, 10 reading-time datasets, and an external garden-path benchmark. The untrained-versus-trained comparison is a clean design for disentangling initial tokenization bias from learned representations, and the authors publicly release their tokenizers and models (a strength for reproducibility). However, in its current form the inferential support for the headline optimum is weak, and the garden-path claim is not consistently supported by the estimated effects.

major comments (3)
  1. [Section 3.3 and Section 3.4] The claim that 'a vocabulary size of around 8,000' yields the strongest predictors (Abstract; Section 3.4) is selected and reported on the same exploratory partition, and the held-out partition reserved for significance testing is never used. The evidence for 8,000 over neighboring sizes is small and inconsistent: summing the totals in Table 5 over the three model sizes gives 11,333.7 for |V|=8,000 versus 11,277.8 for |V|=1,000, and the Large models show the opposite ordering (3,606.9 vs. 3,643.2). Because eleven sizes are compared without confidence intervals, multiple-seed variability, or any significance test, the specific optimum is not established. The authors should either perform confirmatory tests on the held-out partition or substantially temper the headline claim to an 'intermediate granularity' trend.
  2. [Section 4.2, Figures 3 and 4] The abstract's conclusion that 'LMs trained on coarser-grained tokens generally assigned higher surprisal to critical regions ... suggesting a greater sensitivity to garden-path effects' is not supported by the estimated GPEs in Figure 3, which show no consistent monotonic trend: the Medium LM peaks at |V|=1,000, and the Large LMs are flat or noisy. The raw surprisal differences in Figure 4 do show a coarse-token trend, but the paper itself acknowledges that the estimated GPE is influenced by the linking function. The claim of greater GPE sensitivity should be restricted to raw surprisal, or the discrepancy with the estimated effects should be addressed.
  3. [Table 5 and Figure 2] The averaging across model sizes masks large heterogeneity in the post-training optimum: the Small LM peaks at |V|=8,000, but the Medium and Large LMs peak at |V|=1,000. Moreover, the untrained peak in Table 3 is at |V|=4,000 for all sizes, so the statement in Section 3.4 that Small LMs 'replicate the peak observed prior to LM training' is inaccurate—the Small LM's trained peak shifts from 4,000 to 8,000. The conclusion that 8,000 is the optimal granularity after training is therefore an artifact of averaging, and the paper should either present the model-size-specific patterns as the main result or justify the pooling.
minor comments (4)
  1. [Section 4.1] The phrase '24 items of the Main Verb/Reduced Relative (MV/RR), Direct Object/Sentential Complement (NP/S), and Transitive/Intransitive (NP/Z) garden-path constructions' is ambiguous—please clarify whether there are 24 items per construction or 24 items total.
  2. [Section 3.4] The comparison with GPT-2 Small is not matched on training data or model size; the paper should note that this is a reference point, not a controlled comparison, or provide a matched baseline.
  3. [Limitations] The limitations section does not mention the use of a single architecture (Mamba-2), a single training epoch, and no multiple random seeds; adding a sentence on these points would help calibrate the generalizability of the conclusions.
  4. [Abstract and Section 3.4] The abstract states 'a vocabulary size of 8,000' while Section 3.4 says 'around 8,000'; align these statements to avoid overstating the precision of the finding.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the vocabulary-size manipulation is evaluated against external reading-time corpora and a separate human garden-path benchmark, with no fitted parameter renamed as a prediction.

full rationale

The paper's core comparisons are self-contained against external behavioral data. Surprisal values are computed from LMs trained with vocabularies of 11 sizes and are then evaluated by their marginal improvement in log-likelihood over a baseline LME model fit to naturalistic reading-time corpora (Section 3.3), and by their ability to reproduce human garden-path effects using the benchmark and linking-function procedure of Huang et al. (2024) (Section 4.1). No equation defines the token-granularity result in terms of the outcome it is said to predict. The whitespace-correction procedure for word probabilities cites prior work including an independent source (Pimentel and Meister, 2024), and the citations to the authors' own earlier papers are for peripheral modeling choices, not for the central granularity claim. The only legitimate concern is statistical: the claimed "around 8,000" optimum is selected after inspecting the exploratory partition (Section 3.4), and the held-out partition, which Section 3.3 says is "reserved for any statistical significance testing," is not used; with eleven vocabulary sizes and no multiple seeds, the maximum could reflect noise. This is a robustness/selection issue, not circularity, because the reported quantity is an empirical fit to external data rather than a quantity forced by the construction. No step in the derivation reduces to its own input, so the appropriate score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on standard corpus and LME assumptions plus two paper-specific choices: the whitespace reallocation method for word probabilities and the one-epoch Mamba-2 training regime. The main free parameter is the vocabulary size optimum, which is selected from the same exploratory data used to report the result.

free parameters (3)
  • Optimal vocabulary size (|V|) = 8,000 (average over model sizes and datasets)
    Selected post hoc as the best-performing vocabulary size on the exploratory partition after sweeping 11 sizes; no held-out confirmation or significance test.
  • Slope and intercept of GPE trend lines versus log2(|V|) = Not reported in text; fitted in Figures 3 to 6
    Used to claim monotonic trends and p<0.05 stars in Experiment 2; these are fitted to the observed GPE and surprisal difference values, not independent predictions.
  • Random effects structures in LME models = Removed iteratively until convergence
    Maximal random effects pruned based on data, so the final model specification depends on the observed data and may vary across datasets.
assumptions (5)
  • domain assumption The whitespace reallocation procedure of Oh and Schuler (2024) and Pimentel and Meister (2024) correctly converts subword token probabilities to word probabilities.
    Section 3.3 Surprisal Calculation; word-level surprisal is the key predictor, so any error in this conversion affects all downstream fits.
  • domain assumption Mamba-2 multi-input SSMs are adequate stand-ins for the Transformer LMs typically used in cognitive modeling.
    Section 3.2; chosen because Transformers cannot efficiently handle long token sequences at fine granularity. If SSM surprisal behaves differently from Transformer surprisal, the granularity conclusions may not transfer.
  • domain assumption One epoch of training on Wiki-40B yields LMs whose 'end of training' surprisal is representative of trained LMs.
    Section 3.2 Training Setup: 'One iteration of the training data was provided'. No convergence checks or multiple epochs are reported, so interactions may be specific to undertrained models.
  • domain assumption The baseline LME specification (word length, position, unigram surprisal, previous-word fixation, spillover) is sufficient to isolate surprisal's contribution across tokenizers.
    Section 3.3; if baseline predictors interact with tokenizer in unmodeled ways, the DeltaLogLik comparisons may be confounded.
  • domain assumption A linking function estimated on Provo filler items transfers to garden-path stimuli, so model-derived predicted reading time differences can be interpreted as surprisal-based GPE.
    Section 4.1; GPE is not measured on humans but computed from LME-predicted reading times, and the validity depends on transfer across item types.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Impact of Token Granularity on the Predictive Power of Language Model Surprisal." pith.science (2026). https://pith.science/paper/YZ5ASFTY

@misc{pith2026241211940,
  author       = {Pith},
  title        = {Pith review of: The Impact of Token Granularity on the Predictive Power of Language Model Surprisal},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YZ5ASFTY}},
  note         = {Machine review of arXiv:2412.11940}
}
read the original abstract

Word-by-word language model surprisal is often used to model the incremental processing of human readers, which raises questions about how various choices in language modeling influence its predictive power. One factor that has been overlooked in cognitive modeling is the granularity of subword tokens, which explicitly encodes information about word length and frequency, and ultimately influences the quality of vector representations that are learned. This paper presents experiments that manipulate the token granularity and evaluate its impact on the ability of surprisal to account for processing difficulty of naturalistic text and garden-path constructions. Experiments with naturalistic reading times reveal a substantial influence of token granularity on surprisal, with tokens defined by a vocabulary size of 8,000 resulting in surprisal that is most predictive. In contrast, on garden-path constructions, language models trained on coarser-grained tokens generally assigned higher surprisal to critical regions, suggesting a greater sensitivity to garden-path effects than previously reported. Taken together, these results suggest a large role of token granularity on the quality of language model surprisal for cognitive modeling.

Figures

Figures reproduced from arXiv: 2412.11940 by the authors.

Figure 1
Figure 1. Smaller subword vocabulary sizes result in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. ∆LogLik from LM surprisal on the exploratory partitions and corpus-level perplexity after LM training, both aggregated over all reading time corpora. The ‘Av￾erage’ represents the arithmetic mean of ∆LogLik and the geometric mean of perplexity over the three model sizes. The red line denotes aggregate ∆LogLik from GPT-2 Small surprisal (Radford et al., 2019) for refer￾ence. See Appendix A for the results on each ind… view at source ↗
Figure 3
Figure 3. Estimated garden-path effects at the first spillover word (further in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Increase in LM surprisal of the critical word across conditions for the three garden-path constructions. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Estimated garden-path effects at the critical word (deserved in [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Estimated garden-path effects at the second spillover word (investigation in [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 18 canonical work pages

  1. [1]

    Suhas Arehalli, Brian Dillon, and Tal Linzen. 2022. https://aclanthology.org/2022.conll-1.20 Syntactic surprisal from neural models predicts, but underestimates, human processing difficulty from syntactic ambiguities . In Proceedings of the 26th Conference on Computational Natural Language Learning, pages 301--313

    work page 2022

  2. [2]

    Catherine Arnett and Benjamin Bergen. 2025. https://aclanthology.org/2025.coling-main.441/ Why do language models perform worse for morphologically complex languages? In Proceedings of the 31st International Conference on Computational Linguistics, pages 6607--6623

    work page 2025

  3. [3]

    Barr, Roger Levy, Christoph Scheepers, and Harry J

    Dale J. Barr, Roger Levy, Christoph Scheepers, and Harry J. Tily. 2013. https://doi.org/10.1016/j.jml.2012.11.001 Random effects structure for confirmatory hypothesis testing: Keep it maximal . Journal of Memory and Language, 68:255--278

    doi:10.1016/j.jml.2012.11.001 2013

  4. [4]

    Jason J. S. Barton, Hashim M. Hanif, Laura Eklinder Bj\"ornst\"om, and Charlotte Hills. 2014. https://doi.org/10.1080/02643294.2014.895314 The word-length effect in reading: A review . Cognitive Neuropsychology, 31(5--6):378--412

    work page arXiv 2014

  5. [5]

    Douglas Bates, Martin M \" a chler, Ben Bolker, and Steve Walker. 2015. https://doi.org/10.18637/jss.v067.i01 Fitting linear mixed-effects models using lme4 . Journal of Statistical Software, 67(1):1--48

    doi:10.18637/jss.v067.i01 2015

  6. [6]

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. https://proceedings.mlr.press/v202/biderman23a.html Pythia: A suite for analyzing large language models across trai...

    2023

  7. [7]

    Kaj Bostrom and Greg Durrett. 2020. https://aclanthology.org/2020.findings-emnlp.414 Byte pair encoding is suboptimal for language model pretraining . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4617--4624

    work page 2020

  8. [8]

    Agatha Christie. 1920. https://www.gutenberg.org The Mysterious Affair at Styles . John Lane. Retrieved from Project Gutenberg

    1920

Show all 48 references
  1. [9]

    Uschi Cop, Nicolas Dirix, Denis Drieghe, and Wouter Duyck. 2017. https://doi.org/10.3758/s13428-016-0734-0 Presenting GECO: An eyetracking corpus of monolingual and bilingual sentence reading . Behavior Research Methods, 49(2):602--615

    2017 doi

  2. [10]

    Tri Dao and Albert Gu. 2024. https://proceedings.mlr.press/v235/dao24a.html Transformers are SSM s: Generalized models and efficient algorithms through structured state space duality . In Proceedings of the 41st International Conference on Machine Learning, volume 235, pages 1...

    2024

  3. [11]

    Tily, Idan Blank, Anastasia Vishnevetsky, Steven Piantadosi, and Evelina Fedorenko

    Richard Futrell, Edward Gibson, Harry J. Tily, Idan Blank, Anastasia Vishnevetsky, Steven Piantadosi, and Evelina Fedorenko. 2021. https://doi.org/10.1007/s10579-020-09503-7 The Natural Stories corpus: A reading-time corpus of English texts containing rare syntactic constructi...

    2021 doi

  4. [12]

    Mario Giulianelli, Luca Malagutti, Juan Luis Gastaldi, Brian DuSell, Tim Vieira, and Ryan Cotterell. 2024. https://aclanthology.org/2024.emnlp-main.1032 On the proper treatment of tokenization in psycholinguistics . In Proceedings of the 2024 Conference on Empirical Methods in...

    2024

  5. [13]

    Aaron Gokaslan and Vanya Cohen. 2019. OpenWebText Corpus . http://Skylion007.github.io/OpenWebTextCorpus

    2019

  6. [14]

    Mandy Guo, Zihang Dai, Denny Vrande c i \'c , and Rami Al-Rfou. 2020. https://aclanthology.org/2020.lrec-1.297 W iki-40 B : Multilingual language model dataset . In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 2440--2452

    2020

  7. [15]

    John Hale. 2001. https://www.aclweb.org/anthology/N01-1021/ A probabilistic Earley parser as a psycholinguistic model . In Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics on Language Technologies , pages 1--8

    2001

  8. [16]

    Clark, and Philipp Koehn

    Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. 2013. https://www.aclweb.org/anthology/P13-2121/ Scalable modified Kneser-Ney language model estimation . In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages...

    2013

  9. [17]

    Kuan-Jung Huang, Suhas Arehalli, Mari Kugemoto, Christian Muxica, Grusha Prasad, Brian Dillon, and Tal Linzen. 2024. https://doi.org/10.1016/j.jml.2024.104510 Large-scale benchmark yields no evidence that language model surprisal explains syntactic disambiguation difficulty . ...

    2024

  10. [18]

    Carpenter

    Marcel Adam Just and Patricia A. Carpenter. 1980. https://doi.org/10.1037/0033-295X.87.4.329 A theory of reading: From eye fixations to comprehension . Psychological Review, 87(4):329--354

    1980 doi

  11. [19]

    Alan Kennedy, Robin Hill, and Joël Pynte. 2003. The Dundee Corpus . In Proceedings of the 12th European Conference on Eye Movement

    2003

  12. [20]

    Taku Kudo. 2018. https://aclanthology.org/P18-1007 Subword regularization: Improving neural network translation models with multiple subword candidates . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 66--75

    2018

  13. [21]

    Nelson Francis

    Henry Ku c era and W. Nelson Francis. 1967. Computational analysis of present-day American English. Brown University Press, Providence, RI

    1967

  14. [22]

    Sander Land and Max Bartolo. 2024. https://aclanthology.org/2024.emnlp-main.649 Fishing for M agikarp: Automatically detecting under-trained tokens in large language models . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11631--11646

    2024

  15. [23]

    Roger Levy. 2008. https://doi.org/10.1016/j.cognition.2007.05.006 Expectation-based syntactic comprehension . Cognition, 106(3):1126--1177

    2008 doi

  16. [24]

    Ilya Loshchilov and Frank Hutter. 2019. https://openreview.net/forum?id=Bkg6RiCqY7 Decoupled weight decay regularization . In International Conference on Learning Representations

    2019

  17. [25]

    Luke and Kiel Christianson

    Steven G. Luke and Kiel Christianson. 2018. https://doi.org/10.3758/s13428-017-0908-4 The Provo Corpus: A large eye-tracking corpus with predictability norms . Behavior Research Methods, 50(2):826--833

    2018 doi

  18. [26]

    Danny Merkx and Stefan L. Frank. 2021. https://doi.org/10.18653/v1/2021.cmcl-1.2 Human sentence processing: Recurrence or attention? In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 12--22

    2021 doi

  19. [27]

    Corrado, and Jeffrey Dean

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. https://papers.nips.cc/paper_files/paper/2013/hash/9aa42b31882ec039965f3c4923ce901b-Abstract.html Distributed representations of words and phrases and their compositionality . In Advances in N...

    2013

  20. [28]

    Sathvik Nair and Philip Resnik. 2023. https://aclanthology.org/2023.findings-emnlp.752 Words, subwords, and morphemes: What really matters in the surprisal-reading time relationship? In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11251--11260

    2023

  21. [29]

    Byung-Doh Oh, Christian Clark, and William Schuler. 2021. https://aclanthology.org/2021.acl-long.290 Surprisal estimators for human reading times need character models . In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Ling...

    2021

  22. [30]

    Byung-Doh Oh and William Schuler. 2023 a . https://aclanthology.org/2023.findings-emnlp.128/ Transformer-based language model surprisal predicts human reading times best with about two billion training tokens . In Findings of the Association for Computational Linguistics: EMNL...

    2023

  23. [31]

    Byung-Doh Oh and William Schuler. 2023 b . https://doi.org/10.1162/tacl_a_00548 Why does surprisal from larger T ransformer-based language models provide a poorer fit to human reading times? Transactions of the Association for Computational Linguistics, 11:336--350

    2023 doi

  24. [32]

    Byung-Doh Oh and William Schuler. 2024. https://aclanthology.org/2024.emnlp-main.202 Leading whitespaces of language models' subword vocabulary pose a confound for calculating word probabilities . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language P...

    2024

  25. [33]

    Tiago Pimentel and Clara Meister. 2024. https://aclanthology.org/2024.emnlp-main.1020 How to compute the probability of a word . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18358--18375

    2024

  26. [34]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf Language models are unsupervised multitask learners . OpenAI Technical Report

    2019

  27. [35]

    Keith Rayner, Marcia Carlson, and Lyn Frazier. 1983. https://doi.org/10.1016/S0022-5371(83)90236-0 The interaction of syntax and semantics during sentence processing: Eye movements in the analysis of semantically biased sentences . Journal of Verbal Learning and Verbal Behavio...

    1983 doi

  28. [36]

    Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, and Chris Tanner

    Craig W. Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, and Chris Tanner. 2024. https://aclanthology.org/2024.emnlp-main.40 Tokenization is more than compression . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language ...

    2024

  29. [37]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. https://www.aclweb.org/anthology/P16-1162 Neural machine translation of rare words with subword units . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1715--1725

    2016

  30. [38]

    Cory Shain. 2024. https://doi.org/10.1162/opmi_a_00119 Word frequency and predictability dissociate in naturalistic reading . Open Mind, 8:177--201

    2024 doi

  31. [39]

    Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cotterell, and Roger Levy. 2024. https://doi.org/10.1073/pnas.2307876121 Large-scale evidence for logarithmic effects of word predictability on reading time . Proceedings of the National Academy of Sciences, 121(10):e2307876121

    2024 doi

  32. [40]

    Smith and Roger Levy

    Nathaniel J. Smith and Roger Levy. 2013. https://doi.org/10.1016/j.cognition.2013.02.013 The effect of word predictability on reading time is logarithmic . Cognition, 128:302--319

    2013 doi

  33. [41]

    Marten van Schijndel and Tal Linzen. 2021. https://doi.org/10.1111/cogs.12988 Single-stage prediction models do not explain the magnitude of syntactic disambiguation difficulty . Cognitive Science, 45(6):e12988

    2021 doi

  34. [42]

    Gomez, ukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://papers.nips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html Attention is all you need . In Advances in Neura...

    2017

  35. [43]

    O'Donnell, and Ryan Cotterell

    Tim Vieira, Ben LeBrun, Mario Giulianelli, Juan Luis Gastaldi, Brian DuSell, John Terilla, Timothy J. O'Donnell, and Ryan Cotterell. 2024. https://arxiv.org/abs/2412.03719 From language models over tokens to language models over characters . arXiv preprint, arXiv:2412.03719

    2024 arXiv

  36. [44]

    Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger P. Levy. 2020. https://cognitivesciencesociety.org/cogsci20/papers/0375 On the predictive power of neural language models for human real-time comprehension behavior . In Proceedings of the 42nd Annual Meetin...

    2020

  37. [45]

    Ethan Gotlieb Wilcox, Clara Meister, Ryan Cotterell, and Tiago Pimentel. 2023 a . https://aclanthology.org/2023.emnlp-main.466 Language model quality correlates with psychometric predictive power in multiple languages . In Proceedings of the 2023 Conference on Empirical Method...

    2023

  38. [46]

    Ethan Gotlieb Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P. Levy. 2023 b . https://doi.org/10.1162/tacl_a_00612 Testing the predictions of surprisal theory in 11 languages . Transactions of the Association for Computational Linguistics, 11:1451--1470

    2023 doi

  39. [47]

    Weijie Xu, Jason Chon, Tianran Liu, and Richard Futrell. 2023. https://aclanthology.org/2023.findings-emnlp.1052 The linearity of the effect of surprisal on reading times across languages . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15711--15721

    2023

  40. [48]

    Vil \'e m Zouhar, Clara Meister, Juan Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell. 2023. https://aclanthology.org/2023.acl-long.284 Tokenization and the noiseless channel . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pag...

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.