Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

The Inverse Scaling Effect of Pre-Trained Language Model Surprisal Is Not Due to Data Leakage

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that the inverse scaling effect of pre-trained language model surprisal is a real phenomenon, not an artifact of data leakage.

desk verdict Solid, scale-appropriate test of the leakage hypothesis, but the abstract's 'data leakage' overstates what is measured: exact n-gram overlap only. read the letter →

arxiv 2506.01172 v1 pith:ENEWG7PB submitted 2025-06-01 cs.CL

Byung-Doh Oh , Hongao Zhu , William Schuler This is my paper
classification cs.CL
keywords inversescalingsurprisalreadingtimesdataleakagepretrainedlanguagemodelspsycholinguisticsCDAWGn-gramoverlap
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper defends the inverse scaling effect of pre-trained language model surprisal: the finding that larger models' next-word probabilities fit human reading times worse than smaller models' do. The worry had been that this trend is a data-leakage artifact, because the reading-time texts are publicly available and may appear inside pretraining corpora. The paper tests that worry directly: first by measuring exact token overlaps between five reading-time corpora and two large pretraining datasets, finding the overlaps short and rare; then by training Pythia-style models on a filtered, leakage-free subset and replicating the inverse scaling. It concludes that the trend is not an artifact of memorization, though severe leakage would make the inverse relationship look stronger than it is.

What carries the argument

The machinery is a pair of leakage-detection tools. First, a CDAWG (compact directed acyclic word graph) indexes each pretraining corpus, letting any reading-time passage be queried to find the longest contiguous subword-token sequence it shares and how often that sequence occurs in the pretraining data. Second, the same index is used as a filter to construct leakage-free training data: Pile chunks sharing 11 or fewer token overlaps with the reading-time corpora are selected, and three Pythia-like transformers are trained on that data. Fit is then quantified as the log-likelihood gain of linear mixed-effects reading-time regressions when surprisal is added to a baseline model.

What would settle it

Look for a reading-time passage, especially from Provo or Natural Stories, whose near-duplicate (paraphrase or lightly edited form) occurs in the Pile or OpenWebText, and show that a large model trained on that data gives that passage anomalously low surprisal. Alternatively, deduplicate the pretraining corpora against the reading-time corpora using fuzzy match measures and observe whether the inverse scaling slope between model size and surprisal fit collapses to zero; if the slope disappears entirely, the paper's central claim would be falsified.

Watch

Extended reading notes

Core claim

The central claim is that previously reported inverse scaling of surprisal's fit to reading times is not caused by data leakage. The authors build compact directed acyclic word graphs (CDAWGs) over the Pythia training subset of the Pile and over OpenWebText, and query them with the passages of the Dundee, Brown, GECO, Provo, and Natural Stories corpora. They report that almost no passage appears entirely in either pretraining corpus, and the longest overlapping token sequences are typically short and occur with low frequency. They then retrain Pythia-like transformers of three sizes on Pile chunks that contain no 12-token-or-longer overlap with the reading-time passages, and the negative relationship between model size and held-out log-likelihood gain from linear mixed-effects regression appears on all five datasets. Artificially fine-tuning these models on the reading-time corpora makes the negative relationship steeper, which the authors interpret as showing that genuine severe leakage would inflate, not create, the inverse scaling.

Load-bearing premise

The load-bearing premise is that data leakage can be fully detected as exact repeated sequences of subword tokens; if leakage arrives through paraphrases, reordered sentences, or light edits, the leakage-free training data may still contain the reading-time texts and the conclusion would not follow.

Editorial extensions

If this is right

  • Researchers can continue using Pythia- and GPT-2-style pretrained models for psycholinguistic surprisal studies without attributing the inverse scaling trend to exact-text memorization.
  • The released overlap annotations give a concrete resource for screening future reading-time corpora and pretraining data.
  • Because severe leakage is shown to steepen the inverse relationship, corpora that do contain long exact overlaps should be handled with extra caution in surprisal-fitting studies.
  • The finding supports the view that larger models' worse reading-time fit reflects learned distributional associations rather than verbatim memory.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors' operationalization of leakage as exact n-gram overlap leaves open the possibility of paraphrase-level leakage; a natural next experiment would be to deduplicate pretraining data with fuzzy near-duplicate detection (e.g., MinHash on embedded documents) and check whether the inverse scaling slope changes.
  • Since fine-tuning on reading-time data overestimates the negative relationship, the paper implies that original inverse-scaling slopes measured on leaky corpora may be inflated; one could predict that fully deduplicated web-scale models show the same trend but with a shallower slope.
  • The ≤11-token-overlap filtering threshold is a reusable, principled criterion for building leakage-free evaluation sets for psycholinguistics, and could generalize to other naturalistic stimulus materials such as speech transcripts or self-paced reading items.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper investigates whether the inverse scaling effect—larger pre-trained language models providing poorer surprisal fits to human reading times—is an artifact of data leakage. Study 1 uses CDAWGs to measure the longest exact token n-gram overlap between five reading-time corpora and two pre-training datasets (Pile and OpenWebText), finding relatively little overlap in most passages. Study 2 trains Pythia-style transformer LMs of three sizes on Pile chunks filtered to have minimal exact overlap with the reading-time corpora, and reports that the negative relationship between model size and surprisal fit is replicated on all five datasets; additional fine-tuning on the reading-time texts shows that severe exact leakage would inflate this negative relationship. The authors conclude that prior inverse-scaling results using Pythia and GPT-2 are not driven by data leakage.

Significance. If the conclusion is restricted to exact-sequence leakage, the paper makes a useful contribution: it provides a scalable method for quantifying a specific form of leakage, releases the overlapping sequences, and supplies a causal control experiment (leakage-free training plus artificial fine-tuning) that is a clear improvement over purely observational overlap checks. The convergence of five datasets and the direction of the fine-tuning manipulation are persuasive. The main limitation is that the paper measures and manipulates only exact token n-gram overlap, so the broad title-level claim about 'data leakage' is stronger than the evidence supports. The re-trained models are also small relative to the Pythia/GPT-2 models discussed, and the ΔLogLik comparisons lack uncertainty quantification.

major comments (3)
  1. [Abstract and Conclusion] The central claim is broader than the operationalization. The abstract and conclusion state that previous results are 'not driven by the effects of data leakage,' but Study 1 measures only exact token n-gram overlap and Study 2 defines leakage-free data by removing exact overlaps of more than 11 continuous tokens. The artificial-leakage manipulation in Section 3.1 also injects only exact copies of the reading-time texts. The Limitations section explicitly concedes that the diagnosis 'is insensitive to minor variations in form,' so the evidence supports the narrower conclusion 'not driven by exact token-sequence leakage.' The title and abstract should be revised to reflect this scope, otherwise the headline claim is unsupported.
  2. [Section 3.2 and Appendix B] The central result—the negative relationship between model size and ΔLogLik—is presented as point estimates with no confidence intervals or significance tests. Figure 2 shows trends across five corpora and three model sizes, but the paper does not quantify uncertainty, and Appendix B states that the remaining ~25% of the data was reserved for significance testing and 'not used in this work.' Given that the claim is about a monotonic decrease in fit, the authors should provide at least bootstrap confidence intervals or a permutation test across subjects or items, or explain why the consistent pattern is sufficient.
  3. [Section 3.1, Table 1] The re-trained models are 28M, 70M, and 162M parameters, which are at the small end of or smaller than the Pythia and GPT-2 models used in the prior inverse-scaling results cited. The paper claims in the Conclusion that the results 'generalize to trends observed from larger pre-trained LMs like Pythia and GPT-2,' but no model above 162M is studied. The authors should either temper this generalization or provide evidence that the inverse scaling trend is stable in this small-model range and that extrapolation to larger sizes is justified.
minor comments (4)
  1. [Section 3.1] The statement that chunks with 'no more than 11 continuous tokens of overlap' were selected 'which excludes all overlaps improbable enough to meet our threshold in Figure 1a' is unclear; the relationship between the 11-token cutoff and the chance-level threshold of Figure 1 should be explained explicitly.
  2. [Figure 1] The red pluses denoting the chance-level threshold in Figure 1b are described in the text but the caption would benefit from a self-contained definition of the threshold, including the log-probability values and the 0.05 criterion.
  3. [Section 2.2] The description of the Provo corpus overlaps as 'potentially concerning' before noting their low frequencies is slightly confusing; the two observations should be separated clearly so that the reader can assess the frequency evidence without re-reading.
  4. [General] There are a few typographical artifacts in the rendering of author names and references (e.g., 'Ku ˇcera'), likely from the PDF extraction, but these are not substantive.

Circularity Check

0 steps flagged · score 2.0 of 10

No circularity: the leakage-free replication is independent evidence; the n-gram-overlap operationalization is a stated limitation, not a constructed equivalence.

full rationale

The paper's central claim is tested with two independent studies. Study 1 directly measures token n-gram overlap between reading-time corpora and pre-training data using CDAWGs, an external indexing method. Study 2 trains new LMs on data filtered to have minimal overlap with the reading-time corpora and then evaluates surprisal fit on held-out partitions of those same corpora; this is a genuine out-of-sample test, not a re-statement of the prior inverse-scaling result. The prior findings by Oh and Schuler (2023a,b) are cited as the phenomenon to be explained, not as evidence for the new leakage-free replication. The main weakness is that data leakage is operationalized only as exact token n-gram overlap, which the authors explicitly acknowledge in the Limitations: "data leakage in this work is diagnosed mainly through token n-gram overlaps, which is insensitive to minor variations in form." This is a limitation on the breadth of the conclusion, not a circular reduction, because the paper never defines leakage as only exact token overlap; it uses overlap as a diagnostic and leaves open the possibility of paraphrastic leakage. No equation is fitted to the target result, and no parameter is renamed as a prediction. The artificial fine-tuning manipulation is an experimental control, not a circular inference. Thus the derivation chain is self-contained; the score reflects minor non-load-bearing self-citations and the acknowledged gap between the narrow measure and the broad conclusion, which falls under validity risk rather than circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities. It relies on measurement choices (overlap threshold, 11-token cutoff) and domain assumptions (exact-match proxy, OpenWebText approximation, representativeness of small trained LMs).

free parameters (2)
  • Chance-level overlap threshold = p < 0.05; 5-gram log-prob thresholds -28.87 (Pile) and -25.33 (OpenWebText)
    Hand-chosen threshold to separate chance overlaps from likely copied text in Figure 1b; affects interpretation of leakage severity, not the trained-model results.
  • Maximum allowed overlap in leakage-free data = 11 tokens
    Hand-chosen cutoff for filtering training chunks in Study 2; stricter cutoffs would reduce training data, looser cutoffs might admit more overlap.
assumptions (4)
  • domain assumption Exact token n-gram overlap is a valid proxy for data leakage
    Used throughout both studies; acknowledged in Limitations as insensitive to paraphrase. This is the weakest load-bearing premise.
  • domain assumption OpenWebText approximates GPT-2's undisclosed training data
    GPT-2 training data is not public; any leakage conclusions about GPT-2 rely on this approximation, as acknowledged in Limitations.
  • domain assumption The three trained model sizes (28M-162M parameters) are sufficient to represent the inverse scaling trend of larger pre-trained LMs
    Study 2 trains models up to 162M parameters, far smaller than the largest pre-trained LMs; generalizing the replication to those models assumes the trend is continuous across scale.
  • domain assumption Standard surprisal theory and linear mixed-effects baselines isolate the effect of surprisal on reading times
    The LME models assume linear surprisal effects and include standard baseline predictors (length, position, unigram surprisal, previous fixation); these are standard in the field.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Inverse Scaling Effect of Pre-Trained Language Model Surprisal Is Not Due to Data Leakage." pith.science (2026). https://pith.science/paper/ENEWG7PB

@misc{pith2026250601172,
  author       = {Pith},
  title        = {Pith review of: The Inverse Scaling Effect of Pre-Trained Language Model Surprisal Is Not Due to Data Leakage},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ENEWG7PB}},
  note         = {Machine review of arXiv:2506.01172}
}
abstract

In psycholinguistic modeling, surprisal from larger pre-trained language models has been shown to be a poorer predictor of naturalistic human reading times. However, it has been speculated that this may be due to data leakage that caused language models to see the text stimuli during training. This paper presents two studies to address this concern at scale. The first study reveals relatively little leakage of five naturalistic reading time corpora in two pre-training datasets in terms of length and frequency of token $n$-gram overlap. The second study replicates the negative relationship between language model size and the fit of surprisal to reading times using models trained on 'leakage-free' data that overlaps only minimally with the reading time corpora. Taken together, this suggests that previous results using language models trained on these corpora are not driven by the effects of data leakage.

Figures

Figures reproduced from arXiv: 2506.01172 by the authors.

Figure 1
Figure 1. Overlap between the five reading time corpora and the two pre-training corpora. Each ‘ [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. ∆LogLik due to surprisal on held-out data and corpus-level perplexity from LMs trained leakage-free (orange) and after 5 and 10 fine-tuning (FT) steps on reading time data (green and purple respectively). by-subject random effects (Barr et al., 2013) and assume a linear relationship between surprisal and reading times (Wilcox et al., 2023b; Xu et al., 2023; Shain et al., 2024) and a lingering influence of sur￾prisal… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Surprisal Theory is Tautological (without Rational Grounding)

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Unconstrained surprisal theory is a tautology: for any non-negative difficulty measure, a language model exists whose surprisal matches it affinely.

Reference graph

Works this paper leans on

38 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Shivanshu Purohit, Tri Songz, Phil Wang, and Samuel Weinbach. 2021. https://doi.org/10.5281/zenodo.5879544 GPT-NeoX : Large scale autoregressive language modeling in PyTorch

    doi:10.5281/zenodo.5879544 2021

  2. [2]

    Barr, Roger Levy, Christoph Scheepers, and Harry J

    Dale J. Barr, Roger Levy, Christoph Scheepers, and Harry J. Tily. 2013. https://doi.org/10.1016/j.jml.2012.11.001 Random effects structure for confirmatory hypothesis testing: Keep it maximal . Journal of Memory and Language, 68:255--278

    doi:10.1016/j.jml.2012.11.001 2013

  3. [3]

    Douglas Bates, Martin M \" a chler, Ben Bolker, and Steve Walker. 2015. https://doi.org/10.18637/jss.v067.i01 Fitting linear mixed-effects models using lme4 . Journal of Statistical Software, 67(1):1--48

    doi:10.18637/jss.v067.i01 2015

  4. [4]

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. https://proceedings.mlr.press/v202/biderman23a.html Pythia: A suite for analyzing large language models across trai...

    work page 2023

  5. [5]

    Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tram\`er, and Chiyuan Zhang. 2023. https://openreview.net/forum?id=TatRHT_1cK Quantifying memorization across neural language models . In Proceedings of the Eleventh International Conference on Learning Representations

    work page 2023

  6. [6]

    Agatha Christie. 1920. https://www.gutenberg.org The Mysterious Affair at Styles . John Lane. Retrieved from Project Gutenberg

    work page 1920

  7. [7]

    Uschi Cop, Nicolas Dirix, Denis Drieghe, and Wouter Duyck. 2017. https://doi.org/10.3758/s13428-016-0734-0 Presenting GECO: An eyetracking corpus of monolingual and bilingual sentence reading . Behavior Research Methods, 49(2):602--615

    doi:10.3758/s13428-016-0734-0 2017

  8. [8]

    Maxime Crochemore and Renaud V \'e rin. 1997. https://doi.org/10.1007/3-540-63246-8_12 On compact directed acyclic word graphs . In Jan Mycielski, Grzegorz Rozenberg, and Arto Salomaa, editors, Structures in Logic and Computer Science: A Selection of Essays in Honor of A. Ehrenfeucht, pages 192--211. Springer Berlin Heidelberg

    work page doi:10.1007/3-540-63246-8_12 1997

Show all 38 references
  1. [9]

    Tily, Idan Blank, Anastasia Vishnevetsky, Steven Piantadosi, and Evelina Fedorenko

    Richard Futrell, Edward Gibson, Harry J. Tily, Idan Blank, Anastasia Vishnevetsky, Steven Piantadosi, and Evelina Fedorenko. 2021. https://doi.org/10.1007/s10579-020-09503-7 The Natural Stories corpus: A reading-time corpus of English texts containing rare syntactic constructi...

    2021 doi

  2. [10]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. https://arxiv.org/abs/2101.00027 The Pile : An 800GB dataset of diverse text for language modeling . ...

    2020 arXiv

  3. [11]

    Aaron Gokaslan and Vanya Cohen. 2019. OpenWebText Corpus . http://Skylion007.github.io/OpenWebTextCorpus

    2019

  4. [12]

    John Hale. 2001. https://www.aclweb.org/anthology/N01-1021/ A probabilistic Earley parser as a psycholinguistic model . In Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics on Language Technologies , pages 1--8

    2001

  5. [13]

    Clark, and Philipp Koehn

    Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. 2013. https://www.aclweb.org/anthology/P13-2121/ Scalable modified Kneser-Ney language model estimation . In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages...

    2013

  6. [14]

    Shunsuke Inenaga, Hiromasa Hoshino, Ayumi Shinohara, Masayuki Takeda, Setsuo Arikawa, Giancarlo Mauri, and Giulio Pavesi. 2005. https://doi.org/10.1016/j.dam.2004.04.012 On-line construction of compact directed acyclic word graphs . Discrete Applied Mathematics, 146(2):156--179

    2005 doi

  7. [15]

    Alan Kennedy, Robin Hill, and Joël Pynte. 2003. The Dundee Corpus . In Proceedings of the 12th European Conference on Eye Movement

    2003

  8. [16]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2015. https://arxiv.org/abs/1412.6980 Adam: A method for stochastic optimization . In Proceedings of the 3rd International Conference on Learning Representations

    2015 arXiv

  9. [17]

    Nelson Francis

    Henry Ku c era and W. Nelson Francis. 1967. Computational analysis of present-day American English. Brown University Press, Providence, RI

    1967

  10. [18]

    Roger Levy. 2008. https://doi.org/10.1016/j.cognition.2007.05.006 Expectation-based syntactic comprehension . Cognition, 106(3):1126--1177

    2008 doi

  11. [19]

    Tal Linzen and Marco Baroni. 2021. https://doi.org/10.1146/annurev-linguistics-032020-051035 Syntactic structure from deep learning . Annual Review of Linguistics, 7:195--212

    2021 doi

  12. [20]

    Ilya Loshchilov and Frank Hutter. 2019. https://openreview.net/forum?id=Bkg6RiCqY7 Decoupled weight decay regularization . In Proceedings of the 7th International Conference on Learning Representations

    2019

  13. [21]

    Luke and Kiel Christianson

    Steven G. Luke and Kiel Christianson. 2018. https://doi.org/10.3758/s13428-017-0908-4 The Provo Corpus: A large eye-tracking corpus with predictability norms . Behavior Research Methods, 50(2):826--833

    2018 doi

  14. [22]

    Ivanova, Idan A

    Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2024. https://doi.org/10.1016/j.tics.2024.01.011 Dissociating language and thought in large language models . Trends in Cognitive Sciences, 28(6):517--540

    2024 doi

  15. [23]

    Smith, and Yanai Elazar

    William Merrill, Noah A. Smith, and Yanai Elazar. 2024. https://aclanthology.org/2024.emnlp-main.800/ Evaluating n -gram novelty of language models using R usty- DAWG . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14459--14473

    2024

  16. [24]

    Byung-Doh Oh and William Schuler. 2023 a . https://aclanthology.org/2023.findings-emnlp.128/ Transformer-based language model surprisal predicts human reading times best with about two billion training tokens . In Findings of the Association for Computational Linguistics: EMNL...

    2023

  17. [25]

    Byung-Doh Oh and William Schuler. 2023 b . https://doi.org/10.1162/tacl_a_00548 Why does surprisal from larger T ransformer-based language models provide a poorer fit to human reading times? Transactions of the Association for Computational Linguistics, 11:336--350

    2023 doi

  18. [26]

    Byung-Doh Oh and William Schuler. 2024. https://aclanthology.org/2024.emnlp-main.202 Leading whitespaces of language models' subword vocabulary pose a confound for calculating word probabilities . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language P...

    2024

  19. [27]

    Byung-Doh Oh, Shisen Yue, and William Schuler. 2024. https://aclanthology.org/2024.eacl-long.162/ Frequency explains the inverse correlation of large language models' size, training data amount, and surprisal's fit to reading times . In Proceedings of the 18th Conference of th...

    2024

  20. [28]

    Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2009. English Gigaword LDC2009T13

    2009

  21. [29]

    Tiago Pimentel and Clara Meister. 2024. https://aclanthology.org/2024.emnlp-main.1020 How to compute the probability of a word . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18358--18375

    2024

  22. [30]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf Language models are unsupervised multitask learners . OpenAI Technical Report

    2019

  23. [31]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. https://dl.acm.org/doi/10.5555/3433701.3433727 ZeRO : Memory optimizations toward training trillion parameter models . In Proceedings of the International Conference for High Performance Computing, Network...

    2020

  24. [32]

    Keith Rayner, Marcia Carlson, and Lyn Frazier. 1983. https://doi.org/10.1016/S0022-5371(83)90236-0 The interaction of syntax and semantics during sentence processing: Eye movements in the analysis of semantically biased sentences . Journal of Verbal Learning and Verbal Behavio...

    1983 doi

  25. [33]

    Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cotterell, and Roger Levy. 2024. https://doi.org/10.1073/pnas.2307876121 Large-scale evidence for logarithmic effects of word predictability on reading time . Proceedings of the National Academy of Sciences, 121(10):e2307876121

    2024 doi

  26. [34]

    Claude Elwood Shannon. 1948. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x A mathematical theory of communication . Bell System Technical Journal, 27:379--423

    1948

  27. [35]

    Smith and Roger Levy

    Nathaniel J. Smith and Roger Levy. 2013. https://doi.org/10.1016/j.cognition.2013.02.013 The effect of word predictability on reading time is logarithmic . Cognition, 128:302--319

    2013 doi

  28. [36]

    Ethan Gotlieb Wilcox, Clara Meister, Ryan Cotterell, and Tiago Pimentel. 2023 a . https://aclanthology.org/2023.emnlp-main.466 Language model quality correlates with psychometric predictive power in multiple languages . In Proceedings of the 2023 Conference on Empirical Method...

    2023

  29. [37]

    Ethan Gotlieb Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P. Levy. 2023 b . https://doi.org/10.1162/tacl_a_00612 Testing the predictions of surprisal theory in 11 languages . Transactions of the Association for Computational Linguistics, 11:1451--1470

    2023 doi

  30. [38]

    Weijie Xu, Jason Chon, Tianran Liu, and Richard Futrell. 2023. https://aclanthology.org/2023.findings-emnlp.1052 The linearity of the effect of surprisal on reading times across languages . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15711--15721

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.