Pith. sign in

REVIEW 3 major objections 6 minor 3 cited by

Large Language Models Are Human-Like Internally

T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that next-word surprisal drawn from internal layers of larger language models matches human sentence-processing data as well as or better than smaller models, overturning the earlier view that bigger models are less…

desk verdict A broadly useful layer-wise surprisal sweep that convincingly shows the final layer is not the best layer, but the headline scaling flip is confounded by best-of-N layer selection and needs a layer-count control. read the letter →

arxiv 2502.01615 v2 pith:E63HPCP6 submitted 2025-02-03 cs.CL

Tatsuki Kuribayashi , Yohei Oseki , Souhaib Ben Taieb , Kentaro Inui , Timothy Baldwin This is my paper · ORCID
classification cs.CL
keywords surprisallogitlensinternallayerscognitivemodelinglanguagemodelscalingreadingtimesN400psychometricpredictivepower
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to overturn the recent finding that larger language models make worse models of human reading. It argues that the earlier result came from looking only at the models' final output layer, and that when next-word probabilities are read off from internal layers instead, larger models fit reading times, gaze durations, maze-task times, and N400 brain potentials as well as or better than smaller models. If right, the apparent paradox of "bigger is less human-like" becomes a methodological correction, and cognitively plausible smaller models can be thought of as nested inside larger ones. The paper also proposes a layer-to-timescale mapping: earlier layers track fast measures such as first-pass gaze, while later layers track slower measures such as N400 and maze times.

What carries the argument

The engine is the logit lens: for a representation $h_{l,t}$ at layer $l$, the next-word probability is $p(w_t \mid w_{<t}; h_{l,t}) = \mathrm{softmax}(W_U\,\mathrm{LayerNorm}(h_{l,t}))[id(w_t)]$, which skips all later layers and converts any internal layer into a probability distribution over the vocabulary and hence into a surprisal value. Its tuned-lens variant adds a per-layer learned affine map to correct representation drift. The other load-bearing object is the psychometric predictive power $\Delta LL$, the log-likelihood difference between a regression model that includes surprisal plus baseline features and one with only baseline features; this score is plotted against layer depth and model size to compare layers and scaling trends.

What would settle it

Train a per-layer softmax head on held-out text for each internal layer of several model sizes and recompute best-layer $\Delta LL$; if the positive scaling trend of parameter count versus $\Delta LL$ disappears or reverses when properly calibrated heads replace the shared logit lens, the central claim fails. Alternatively, record first-pass gaze and N400 on the same words from the same participants; the paper predicts the best layer for gaze should sit earlier than the best layer for N400, and a reversal would falsify the timescale mapping.

Watch

Extended reading notes

Core claim

The central claim is that the cognitive plausibility of a language model is not a property of its final layer. By applying the logit lens, which projects each internal layer's representation through the unembedding matrix to obtain a next-word distribution, the authors compute layer-wise surprisal scores against 15 human datasets spanning self-paced reading, eye tracking, maze task, and EEG measures. Across 30 models from five families, the layer with the best psychometric predictive power is usually not the last layer, and larger models' best internal layers match or beat smaller models' best surprisal. The same data show a systematic ordering: FPGD and SPR align with earlier layers, while N400 and MAZE align with later layers, a pattern that survives clause-final-token controls and appears across ten MECO languages.

Load-bearing premise

The whole argument rests on treating logit-lens projections of intermediate layers as genuine next-word probabilities; if a model's hidden state at layer $l$ cannot be read as a probability distribution over the vocabulary, the layer-wise $\Delta LL$ scores do not measure internal cognitive plausibility.

Editorial extensions

If this is right

  • The default assumption that the final layer is the right place to extract surprisal should be dropped: across the paper's settings, the best layer is often internal, and for most tested models over 80% of internal layers beat the previous best final-layer score.
  • Model scaling no longer looks harmful: Pearson correlations between parameter count and best-layer $\Delta LL$ are significantly positive on average, so larger LMs appear more, not less, cognitively plausible once internal layers are counted.
  • The layer-measure interaction is significant: N400 and MAZE align with later layers than SPR and FPGD, and this ordering survives restricting the analysis to sentence-final tokens.
  • The apparent behavior-neurophysiology gap, where small models fit reading behavior and large models fit brain data, dissolves when all layers are considered.
  • In multilingual MECO data, the best internal layer outperforms the final layer in 43 of 50 settings, and negative scaling effects are mitigated in all ten languages tested.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If logit-lens surprisal is valid, layer choice stops being a nuisance parameter and becomes a substantive choice about which processing stage a behavioral measure indexes; future studies could report layer profiles rather than only best-layer peaks.
  • The fast-gaze-to-early-layer and N400-to-later-layer correspondence suggests a quantitative hypothesis the paper leaves implicit: across EEG components with different peak latencies, the best-fitting relative layer depth should increase with component latency, a prediction testable on the UCL corpus scores.
  • Because early-layer surprisal helps most on infrequent and long words, internal-layer surprisal might serve as a cheap drop-in replacement in psycholinguistic analyses that want human-like context use without training a new model.
  • The paper's layer-wise probability extraction could generalize beyond surprisal to entropy or entropy reduction; a direct test would be comparing layer-wise entropy reduction against the same human datasets to see whether the timescale ordering persists.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper revisits the recent finding that larger language models (LMs) produce surprisal estimates that fit human reading behavior worse than smaller LMs. Using logit-lens and tuned-lens readouts, the authors extract next-word probabilities from the internal layers of 30 LMs (up to 72B parameters) and evaluate their psychometric predictive power (ΔLL) against 15 human sentence-processing datasets spanning self-paced reading, first-pass gaze duration, MAZE, and N400 measures. They report that the best internal layer typically outperforms the final layer, that larger LMs' best internal layers yield higher ΔLL than smaller LMs' best layers (reversing the final-layer scaling trend), and that earlier layers align better with fast behavioral measures while later layers align with slower measures such as N400 and MAZE. The paper includes robustness checks for sentence-final tokens, multilingual MECO data, and tuned-lens readouts.

Significance. If the scaling-flip result were established, this would be an important corrective to the 'bigger is not always better' narrative in LM-based cognitive modeling and a useful bridge to mechanistic interpretability. The paper's empirical scope is a strength: systematic evaluation across 30 open models, 15 datasets, two readout methods, and a multilingual follow-up, with code released. The layer-measure alignment finding (early layers → fast gaze/SPR, late layers → N400/MAZE) is well supported by the per-layer regression in §6.2 and survives the sentence-final token control. However, the headline scaling result in §5.2/Figure 3 is not yet statistically established because of a best-of-N selection confound across models with different layer counts.

major comments (3)
  1. [§5.2, Figure 3] The central scaling claim is estimated from the per-model maximum ΔLL over all internal layers, but models differ by up to a factor of about 13 in layer count (e.g., GPT-2 small has 12 layers, Qwen2.5-72B has 80; see Table 4 and Appendix B). The maximum of L noisy ΔLL values grows with L, so even under a null of no true parameter-size effect, a positive parameter–ΔLL correlation can arise because larger models offer more candidate layers. The robustness check in §6.1 (win rate of internal layers against the final layer) is a within-model comparison and does not equalize the number of candidates across models; the one-sample t-test over the 34 dataset×lens correlations in §5.2 also does not include layer count as a covariate. Please add a control for log layer count, restrict the scaling analysis to models with matched layer counts, or report a permutation null that preserves per-model layer counts; without this, the 'flip' in Figure 3 is not established.
  2. [§3.1, Eq. (1)] The paper operationalizes 'internal plausibility' through logit-lens projections of intermediate representations into output vocabulary space using the final unembedding matrix and LayerNorm. This is an untrained linear readout; if it is a biased or lossy estimate of the model's internal predictive distribution, and if the bias varies with model depth or scale, then the cross-model comparisons in Figure 3 are not interpretable as differences in internal next-word probabilities. The tuned-lens readout (Eq. 2) is a stronger check because its parameters are trained on pretraining data, but the headline analyses report both lens types and the scaling conclusion is drawn from the logit-lens results as well. I recommend a validation of the logit-lens readout (e.g., comparison with a readout trained on the LM's own next-word objective) and/or reporting the scaling result primarily on the tuned-lens subset, where the probability interpretation is more defensible.
  3. [§5.2, Abstract] The statistical evidence for the 'flip' is a one-sample t-test over 34 Pearson correlations (Footnote 9), but those correlations are not independent: the same 30 models appear in every dataset×lens setting, and the logit-lens and tuned-lens scores for a given model are dependent. The test also does not model the layer-count confound described above. The narrative overstates the consistency of the result: Section 5.2 reports that NS MAZE shows a strictly negative scaling effect even for the best layer, yet the Abstract says the alignment holds 'consistently across behavioral and neurophysiological measures.' Please use a mixed-effects model with random intercepts for dataset and model family (or a cluster-robust test), and qualify the Abstract and Conclusion to reflect the NS MAZE exception.
minor comments (6)
  1. [§6.1, Table 2] The statement in §6.1 that the win rate is 'typically around 80%' is not supported by Table 2, which contains several values below 0.5 (e.g., 0.00 for DC FPGD with Pythia-1B and 0.10 for DC FPGD with Qwen2.5-3B); please report the distribution rather than a single typical value.
  2. [§3.1, §7.3] There are repeated typos: 'd-dimentaional' and 'd-dimentional' should be 'd-dimensional', 'internal representaion' should be 'internal representation', and 'suprisal' in §6.4 should be 'surprisal'.
  3. [§5.2, Figure 3 caption] The caption refers to 'maker type' (should be 'marker type'), and the relationship between red line, grey line, and the two scenarios should be stated more clearly; the marker shapes for model families should also be consistent with Figure 1.
  4. [§4.2] The phrase 'bilion-scale' should be 'billion-scale'.
  5. [§6.5] The heading 'Multilinguall generallity' contains typos; it should be 'Multilingual generality'.
  6. [§6.1] The definition of 'previous best' in §6.1 is ambiguous; please clarify whether the comparison is against the final layer of the same model or the best final-layer score across the model family.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the layer-wise ΔLL evaluation is computed against external human data, and the central scaling claim is not defined in terms of the conclusion.

full rationale

The paper's central quantities are ΔLL scores computed by regressing layer-wise surprisal (Eq. 1/2) against external human reading/EEG corpora (Table 1), with baseline features held fixed. No parameter of the LMs or lenses is fitted to human data; the tuned lens is trained on additional pretraining text, and the paper states 'we do not fine-tune any part of the LMs for human data.' The scaling claim in Section 5.2 is a regression of best-layer ΔLL on parameter count; the 'best' layer is selected post hoc from the same ΔLL values, but ΔLL is not defined in terms of model size or the conclusion, so the selection is not a term defined to equal the target quantity. The robustness check in Section 6.1 uses win rates against same-family last-layer scores, which is a within-model comparison; while the differing number of layers (6–80) creates a real best-of-N confound that could inflate the scaling slope, that is a statistical selection artifact rather than a circular reduction, and the paper's own Section 6.1 partially addresses the outlier concern. Self-citations (Kuribayashi et al., 2021, 2022, 2024) are used for preprocessing, corpus selection, and prior baseline results; they are not invoked as a uniqueness theorem or as the justification for the internal-layer readout. The logit-lens/tuned-lens ansatz is an assumption about representation geometry, not a premise that presupposes the human-alignment conclusion. No equation in the paper reduces to its own input by construction, so the derivation chain is self-contained against external benchmarks.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper's empirical claims rest on the validity of internal-layer surprisal readouts (logit lens), the standard linear surprisal theory, the sufficiency of the baseline features, and the fairness of best-layer selection across models with different depths.

free parameters (1)
  • best-layer index (per model, per dataset) = varies; the layer that maximizes delta-LL on the evaluation data
    The scaling analysis in Section 5.2 chooses the best layer from each LM's internal layers on the same human data that is later used to measure fit. Models with more layers have more opportunities to yield a high delta-LL, so this selection is a de facto free parameter that can bias the comparison.
assumptions (4)
  • ad hoc to paper LogitLens in Eq. 1, which projects intermediate representations into output vocabulary space via the final unembedding matrix and LayerNorm, produces valid next-word probability estimates for internal layers.
    All internal surprisal values are defined by this projection. The paper does not formally justify that intermediate representations encode next-word information in this way; it relies on prior interpretability work.
  • domain assumption Reading time and ERP amplitude are linear functions of surprisal, so the delta-LL from nested linear regressions isolates the contribution of surprisal.
    Standard in the field (Smith and Levy 2013, Shain et al. 2024); the paper adopts it without re-derivation.
  • domain assumption The baseline feature set in Eq. 3 (length, frequency, and previous two words' surprisal) adequately controls for low-level predictors.
    If the baseline leaves out relevant predictors, the delta-LL values may overstate surprisal's contribution. The paper uses the same baseline across datasets with a few documented exceptions.
  • ad hoc to paper Selecting the best-performing layer post hoc does not systematically favor models with more layers in cross-model comparisons.
    The paper never corrects for layer count when comparing best-layer delta-LL across models of different depths, but its central scaling claim depends on this comparability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Large Language Models Are Human-Like Internally." pith.science (2026). https://pith.science/paper/E63HPCP6

@misc{pith2026250201615,
  author       = {Pith},
  title        = {Pith review of: Large Language Models Are Human-Like Internally},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E63HPCP6}},
  note         = {Machine review of arXiv:2502.01615}
}
read the original abstract

Recent cognitive modeling studies have reported that larger language models (LMs) exhibit a poorer fit to human reading behavior (Oh and Schuler, 2023b; Shain et al., 2024; Kuribayashi et al., 2024), leading to claims of their cognitive implausibility. In this paper, we revisit this argument through the lens of mechanistic interpretability and argue that prior conclusions were skewed by an exclusive focus on the final layers of LMs. Our analysis reveals that next-word probabilities derived from internal layers of larger LMs align with human sentence processing data as well as, or better than, those from smaller LMs. This alignment holds consistently across behavioral (self-paced reading times, gaze durations, MAZE task processing times) and neurophysiological (N400 brain potentials) measures, challenging earlier mixed results and suggesting that the cognitive plausibility of larger LMs has been underestimated. Furthermore, we first identify an intriguing relationship between LM layers and human measures: earlier layers correspond more closely with fast gaze durations, while later layers better align with relatively slower signals such as N400 potentials and MAZE processing times. Our work opens new avenues for interdisciplinary research at the intersection of mechanistic interpretability and cognitive modeling.

Figures

Figures reproduced from arXiv: 2502.01615 by the authors.

Figure 1
Figure 1. Different measures of human sentence processing align with surprisal from different lay￾ers of language models (LMs), and the best layer is typically not the final layer. In the bottom plot, for example, gaze duration (blue dots) and EEG signal (red dots) correlate with earlier and later layers of LMs, respectively. Each dot corresponds to the fit of surprisal (y-axis) from a particular layer depth (x-axis) to human… view at source ↗
Figure 2
Figure 2. Relationships between layer depth (x-axis) and [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Scaling effect between model size (parameter counts in log scale) and [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Relationship between ∆LL and relative layer depth for each human measure. Different measures are associated with different layers; for example, good ∆LLs for FPGD are achieved in earlier layers, while those for MAZE are in the lat￾ter layers. layer_depth × FPGD is sign…
Figure 6
Figure 6. Figure 6: Scaling effects in MECO German part negative in some languages. We suspect that these mixed results might be biased by multilin￾gual LMs, which are reported to process every lan￾guage within the English subspace in their mid￾dle layers (Wendler et al., 2024), leading t…
Figure 5
Figure 5. Figure 5: Scaling effect between ∆LL and param￾eter counts in MECO. The grey lines are results relying on the last layer’s ∆LLs, and the red lines rely on the best internal layers’ ∆LLs. In all the languages, the negative correlation between pa￾rameter size and ∆LL is mitigated …
Figure 7
Figure 7. Figure 7: The markers correspond to all the inter [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Scaling effect between ∆LL and PPL (measured on respective datasets with final layer), instead of model parameter counts, as adopted in [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: Visulaition of layer–∆LL relationships (in addition to [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Surprisal Theory is Tautological (without Rational Grounding)

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Unconstrained surprisal theory is a tautology: for any non-negative difficulty measure, a language model exists whose surprisal matches it affinely.

  2. When the LM misunderstood the human chuckled: Analyzing garden path effects in humans and language models

    cs.CL 2025-02 conditional novelty 5.0 of 10

    Humans and large language models show similar comprehension failures on garden-path sentences, with stronger models correlating more closely with human performance across three tasks.

  3. A Spatio-Temporal Point Process for Fine-Grained Modeling of Reading Behavior

    cs.LG 2025-06

Reference graph

Works this paper leans on

102 extracted references · 43 canonical work pages · cited by 3 Pith papers

  1. [1]

    C Aurnhammer and S L Frank. 2019. https://repository.ubn.ru.nl/bitstream/handle/2066/213724/213724.pdf?sequence=1&isAllowed=y Comparing gated and simple recurrent neural network architectures as models of human sentence processing . In Proceedings of CogSci , pages 112--118

    2019

  2. [2]

    Khai Loong Aw, Syrielle Montariol, Badr AlKhamissi, Martin Schrimpf, and Antoine Bosselut. 2024. https://openreview.net/forum?id=nXNN0x4wbl Instruction-tuning aligns LLM s to the human brain . In COLM 2024

    2024

  3. [3]

    Andrea Banino, Jan Balaguer, and Charles Blundell. 2021. https://openreview.net/forum?id=1EuxRTe0WN PonderNet : Learning to ponder . In 8th ICML Workshop on Automated Machine Learning (AutoML)

    2021

  4. [4]

    Lisa Beinborn and Nora Hollenstein. 2024. Cognitive plausibility in natural language processing. Synthesis lectures on human language technologies. Springer International Publishing, Cham

    2024

  5. [5]

    Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Lev McKinney, Igor Ostrovsky, Stella Biderman, and Jacob Steinhardt. 2023. https://arxiv.org/abs/2303.08112 Eliciting latent predictions from transformers with the tuned lens . arXiv preprint

    arXiv 2023

  6. [6]

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. https://arxiv.org/abs/2304.01373 Pythia: A suite for analyzing large language models across training and scaling . In ICML 2023, pages 2397--2430. PMLR

    arXiv 2023

  7. [7]

    Veronica Boyce and Roger Philip Levy. 2023. https://api.semanticscholar.org/CorpusID:258086581 A-maze of natural stories: Comprehension and surprisal in the maze task . Glossa Psycholinguistics

    2023

  8. [8]

    Gino Brunner, Yang Liu, Damian Pascual, Oliver Richter, Massimiliano Ciaramita, and Roger Wattenhofer. 2019. https://openreview.net/forum?id=BJg1f6EFDB On identifiability in transformers . In ICLR 2019

    2019

Show all 102 references
  1. [9]

    Manuel G Calvo and Enrique Meseguer. 2002. https://www.cambridge.org/core/journals/spanish-journal-of-psychology/article/abs/eye-movements-and-processing-stages-in-reading-relative-contribution-of-visual-lexical-and-contextual-factors/A85DD970ED70A24BB97F28488DCD63E2 Eye movem...

    2002

  2. [10]

    Charlotte Caucheteux, Alexandre Gramfort, and Jean-R \'e mi King. 2023. https://www.nature.com/articles/s41562-022-01516-2 Evidence of a predictive coding hierarchy in the human brain listening to speech . Nature human behaviour, 7(3):430--441

    2023

  3. [11]

    Matthew W Crocker. 2007. https://www.coli.uni-saarland.de/ crocker/documents/crocker-nlp-handbook.pdf Computational psycholinguistics . The Handbook of Computational Linguistics and Natural Language Processing

    2007

  4. [12]

    Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2023. https://doi.org/10.18653/v1/2023.acl-long.893 Analyzing transformers in embedding space . In Proceedings of ACL 2023, pages 16124--16170

    2023 doi

  5. [13]

    Olaf Dimigen, Werner Sommer, Annette Hohlfeld, Arthur M Jacobs, and Reinhold Kliegl. 2011. https://psycnet.apa.org/record/2011-14094-001 Coregistration of eye movements and eeg in natural reading: analyses and review. Journal of experimental psychology: General, 140(4):552

    2011

  6. [14]

    Kawin Ethayarajh. 2019. https://aclanthology.org/D19-1006/ How contextual are contextualized word representations? comparing the geometry of BERT , ELMo , and GPT -2 embeddings . In Proceedings of EMNLP-IJCNLP 2019, pages 55--65

    2019

  7. [15]

    Kara D Federmeier, Edward W Wlotko, Esmeralda De Ochoa-Dewald, and Marta Kutas. 2007. https://www.sciencedirect.com/science/article/abs/pii/S0006899306019986 Multiple effects of sentential constraint on word processing . Brain Research, 1146:75--84

    2007

  8. [16]

    Kenneth I Forster, Christine Guerrera, and Lisa Elliot. 2009. https://link.springer.com/article/10.3758/brm.41.1.163 The maze task: measuring forced incremental sentence processing time . Behavior research methods, 41(1):163--171

    2009 doi

  9. [17]

    Stefan L Frank and Rens Bod. 2011. https://www.researchgate.net/publication/51140976_Insensitivity_of_the_Human_Sentence-Processing_System_to_Hierarchical_Structure Insensitivity of the human sentence-processing system to hierarchical structure . Psychological Science, 22(6):829--834

    2011

  10. [18]

    Stefan L Frank, Irene Fernandez Monsalve, Robin L Thompson, and Gabriella Vigliocco. 2013. https://link.springer.com/article/10.3758/s13428-012-0313-y Reading time data for evaluating broad-coverage models of english sentence processing . Behavior research methods, 45:1182--1190

    2013 doi

  11. [19]

    Frank, Leun J

    Stefan L. Frank, Leun J. Otten, Giulia Galli, and Gabriella Vigliocco. 2015. https://doi.org/https://doi.org/10.1016/j.bandl.2014.10.006 The erp response to the amount of information conveyed by words in sentences . Brain and Language, 140:1--11

    2015 doi

  12. [20]

    Richard Futrell, Edward Gibson, and Roger P. Levy. 2020. https://doi.org/10.1111/cogs.12814 Lossy-Context Surprisal: An Information-Theoretic Model of Memory Effects in Sentence Processing . Journal of Cognitive Science

    2020 doi

  13. [21]

    Richard Futrell, Edward Gibson, Harry J Tily, Idan Blank, Anastasia Vishnevetsky, Steven T Piantadosi, and Evelina Fedorenko. 2021. https://link.springer.com/article/10.1007/s10579-020-09503-7 The natural stories corpus: a reading-time corpus of english texts containing rare s...

    2021 doi

  14. [22]

    Mario Giulianelli, Luca Malagutti, Juan Luis Gastaldi, Brian DuSell, Tim Vieira, and Ryan Cotterell. 2024 a . https://aclanthology.org/2024.emnlp-main.1032/ On the proper treatment of tokenization in psycholinguistics . In Proceedings of EMNLP 2024, pages 18556--18572

    2024

  15. [23]

    Mario Giulianelli, Andreas Opedal, and Ryan Cotterell. 2024 b . https://aclanthology.org/2024.findings-emnlp.682/ Generalized measures of anticipation and responsivity in online language processing . In Findings of EMNLP 2024, pages 11648--11669

    2024

  16. [24]

    Aaron Gokaslan and Vanya Cohen. 2019. Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus

    2019

  17. [25]

    Adam Goodkind and Klinton Bicknell. 2018. https://doi.org/10.18653/v1/w18-0102 Predictive power of word surprisal for reading times is a linear function of language model quality . In Proceedings of CMCL, pages 10--18

    2018 doi

  18. [26]

    Alex Graves. 2016. https://arxiv.org/abs/1603.08983 Adaptive computation time for recurrent neural networks . arXiv preprint

    2016 arXiv

  19. [27]

    Michael Hahn, Richard Futrell, Roger Levy, and Edward Gibson. 2022. https://pubmed.ncbi.nlm.nih.gov/36260742/ A resource-rational model of human processing of recursive linguistic structure . Proceedings of the National Academy of Sciences of the United States of America, 119(...

    2022

  20. [28]

    John Hale. 2001. https://www.aclweb.org/anthology/N01-1021 A probabilistic Earley parser as a psycholinguistic model . In Proceedings of NAACL 2001, pages 159--166

    2001

  21. [29]

    John Hale, Chris Dyer, Adhiguna Kuncoro, and Jonathan R. Brennan. 2018. https://doi.org/10.18653/v1/p18-1254 Finding Syntax in Human Encephalography with Beam Search . In Proceedings of ACL 2018, pages 2727--2736

    2018 doi

  22. [30]

    Kenneth Heafield. 2011. https://aclanthology.org/W11-2123/ K en LM : Faster and smaller language model queries . In Proceedings of the sixth workshop on statistical machine translation, pages 187--197

    2011

  23. [31]

    Nora Hollenstein, Jonathan Rotsztejn, Marius Troendle, Andreas Pedroni, Ce Zhang, and Nicolas Langer. 2018. https://www.nature.com/articles/sdata2018291 ZuCo , a simultaneous EEG and eye-tracking resource for natural sentence reading . Scientific Data, 5:180291

    2018

  24. [32]

    Eghbal A Hosseini, Martin Schrimpf, Yian Zhang, Samuel Bowman, Noga Zaslavsky, and Evelina Fedorenko. 2024. https://direct.mit.edu/nol/article/5/1/43/119156/Artificial-Neural-Network-Language-Models-Predict Artificial neural network language models predict human brain response...

    2024

  25. [33]

    Jennifer Hu, Kyle Mahowald, Gary Lupyan, Anna Ivanova, and Roger Levy. 2024. https://www.pnas.org/doi/abs/10.1073/pnas.2400917121 Language models align with human judgments on key grammatical constructions . Proceedings of the National Academy of Sciences, 121(36):e2400917121

    2024 doi

  26. [34]

    Kuan-Jung Huang, Suhas Arehalli, Mari Kugemoto, Christian Muxica, Grusha Prasad, Brian Dillon, and Tal Linzen. 2024. https://www.sciencedirect.com/science/article/abs/pii/S0749596X24000135 Large-scale benchmark yields no evidence that language model surprisal explains syntacti...

    2024

  27. [35]

    Ryan J Hubbard, Joost Rommers, Cassandra L Jacobs, and Kara D Federmeier. 2019. https://www.frontiersin.org/journals/human-neuroscience/articles/10.3389/fnhum.2019.00291/full Downstream behavioral and electrophysiological consequences of word prediction on recognition memory ....

    2019

  28. [36]

    Just and Patricia A

    Marcel A. Just and Patricia A. Carpenter. 1980. https://doi.org/10.1037/0033-295X.87.4.329 A theory of reading: From eye fixations to comprehension . Journal of Psychological Review

    1980 doi

  29. [37]

    Yi g itcan Kaya, Sanghyun Hong, and Tudor Dumitras. 2019. https://proceedings.mlr.press/v97/kaya19a.html Shallow-Deep Networks : Understanding and mitigating network overthinking . In Proceedings of ICML 2019

    2019

  30. [38]

    Alan Kennedy, Robin Hill, and Joël Pynte. 2003. The dundee corpus . In Proceedings of the 12th European conference on eye movement

    2003

  31. [39]

    Nikita Kitaev, Steven Cao, and Dan Klein. 2019. https://doi.org/10.18653/v1/P19-1340 Multilingual constituency parsing with self-attention and pre-training . In Proceedings of ACL 2019, pages 3499--3505

    2019 doi

  32. [40]

    Nikita Kitaev and Dan Klein. 2018. https://doi.org/10.18653/v1/P18-1249 Constituency parsing with a self-attentive encoder . In Proceedings of ACL 2018, pages 2676--2686

    2018 doi

  33. [41]

    Tatsuki Kuribayashi, Yohei Oseki, and Timothy Baldwin. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.129 Psychometric predictive power of large language models . In Findings of NAACL 2024, pages 1983--2005

    2024 doi

  34. [42]

    Tatsuki Kuribayashi, Yohei Oseki, Ana Brassard, and Kentaro Inui. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.712 Context limitations make neural language models more human-like . In Proceedings of EMNLP 2022, pages 10421--10436

    2022 doi

  35. [43]

    Tatsuki Kuribayashi, Yohei Oseki, Takumi Ito, Ryo Yoshida, Masayuki Asahara, and Kentaro Inui. 2021. https://doi.org/10.18653/v1/2021.acl-long.405 Lower perplexity is not always human-like . In Proceedings of ACL-IJCNLP 2021, pages 5203--5217

    2021 doi

  36. [44]

    Marta Kutas and Kara D Federmeier. 2011. https://www.annualreviews.org/content/journals/10.1146/annurev.psych.093008.131123 Thirty years and counting: finding meaning in the N400 component of the event-related brain potential ( ERP ) . Annual review of psychology, 62(1):621--647

    2011

  37. [45]

    Ellen F Lau, Colin Phillips, and David Poeppel. 2008. https://www.nature.com/articles/nrn2532 A cortical network for semantics: (de)constructing the N400 . Nature Reviews Neuroscience, 9(12):920--933

    2008

  38. [46]

    Roger Levy. 2008. https://doi.org/10.1016/j.cognition.2007.05.006 Expectation-based syntactic comprehension . Journal of Cognition, 106(3):1126--1177

    2008 doi

  39. [47]

    Richard L Lewis and Shravan Vasishth. 2005. https://www.ling.uni-potsdam.de/ vasishth/pdfs/Lewis-VasishthCogSci2005.pdf An activation-based model of sentence processing as skilled memory retrieval . Cogn. Sci., 29(3):375--419

    2005

  40. [48]

    Falk Lieder and Thomas L Griffiths. 2019. https://www.cambridge.org/core/journals/behavioral-and-brain-sciences/article/abs/resourcerational-analysis-understanding-human-cognition-as-the-optimal-use-of-limited-computational-resources/586866D9AD1D1EA7A1EECE217D392F4A Resource-r...

    2019

  41. [49]

    Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O ' Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, M...

    2022 doi

  42. [50]

    David Marr. 1982. https://academic.oup.com/mit-press-scholarship-online/book/13528?sid=oup:oxfordacademic&genre=book&aulast=Marr&aufirst=David&title=Vision Henry Holt and Co., Inc., USA

    1982

  43. [51]

    Kate McCurdy and Michael Hahn. 2024. https://aclanthology.org/2024.conll-1.4/ Lossy context surprisal predicts task-dependent patterns in relative clause processing . In Proceedings of CoNLL 2024, pages 36--45

    2024

  44. [52]

    Clara Meister, Mario Giulianelli, and Tiago Pimentel. 2024. https://aclanthology.org/2024.emnlp-main.921/ Towards a similarity-adjusted surprisal theory . In Proceedings of EMNLP 2024, pages 16485--16498

    2024

  45. [53]

    Clara Meister, Tiago Pimentel, Thomas Clark, Ryan Cotterell, and Roger Levy. 2022. https://doi.org/10.18653/v1/2022.acl-short.3 Analyzing wrap-up effects through an information-theoretic lens . In Proceedings of ACL 2022, pages 20--28

    2022 doi

  46. [54]

    Danny Merkx and Stefan L. Frank. 2021. https://doi.org/10.18653/v1/2021.cmcl-1.2 Human sentence processing: Recurrence or attention? In Proceedings of CMCL, pages 12--22

    2021 doi

  47. [55]

    James Michaelov, Catherine Arnett, and Ben Bergen. 2024 a . https://openreview.net/forum?id=amhPBLFYWv Revenge of the fallen? recurrent models match transformers at predicting human language comprehension metrics . In COLM 2024

    2024

  48. [56]

    Michaelov, Megan D

    James A. Michaelov, Megan D. Bardolph, Cyma K. Van Petten, Benjamin K. Bergen, and Seana Coulson. 2024 b . https://doi.org/10.1162/nol_a_00105 Strong Prediction: Language Model Surprisal Explains Multiple N400 Effects . Neurobiology of Language, 5(1):107--135

    2024 doi

  49. [57]

    Sathvik Nair and Philip Resnik. 2023. https://arxiv.org/abs/2310.17774 Words, subwords, and morphemes: What really matters in the surprisal-reading time relationship? In Findings of EMNLP2023

    2023 arXiv

  50. [58]

    nostalgebraist. 2020. https://www.lesswrong. com/posts/AcKRB8wDpdaN6v6ru/ interpreting-gpt-the-logit-lens interpreting gpt: the logit lens

    2020

  51. [59]

    Samer Nour Eddine, Trevor Brothers, Lin Wang, Michael Spratling, and Gina R Kuperberg. 2024. https://www.sciencedirect.com/science/article/abs/pii/S0010027724000416 A predictive coding model of the N400 . Cognition, 246(105755):105755

    2024

  52. [60]

    Byung-Doh Oh, Christian Clark, and William Schuler. 2021. https://aclanthology.org/2021.acl-long.290/ Surprisal estimators for human reading times need character models . In Proceedings of ACL-IJCNLP 2021, pages 3746--3757

    2021

  53. [61]

    Byung-Doh Oh, Christian Clark, and William Schuler. 2022. https://www.frontiersin.org/journals/artificial-intelligence/articles/10.3389/frai.2022.777963/full Comparison of structural parsers and neural language models as surprisal estimators . Frontiers in Artificial Intellige...

    2022

  54. [62]

    Byung-Doh Oh and William Schuler. 2023 a . https://aclanthology.org/2023.findings-emnlp.128/ Transformer-based language model surprisal predicts human reading times best with about two billion training tokens . In Findings of EMNLP 2023, pages 1915--1921

    2023

  55. [63]

    Byung-Doh Oh and William Schuler. 2023 b . https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00548/115371/Why-Does-Surprisal-From-Larger-Transformer-Based Why does surprisal from larger transformer-based language models provide a poorer fit to human reading times? TACL, 1...

    2023 doi

  56. [64]

    Byung-Doh Oh and William Schuler. 2024. https://aclanthology.org/2024.emnlp-main.202/ Leading whitespaces of language models’ subword vocabulary pose a confound for calculating word probabilities . In Proceedings of EMNLP 2024, pages 3464--3472

    2024

  57. [65]

    Byung-Doh Oh, Shisen Yue, and William Schuler. 2024. https://aclanthology.org/2024.eacl-long.162/ Frequency explains the inverse correlation of large language models ' size, training data amount, and surprisal ' s fit to reading times . In Proceedings of EACL 2024, pages 2644--2663

    2024

  58. [66]

    Andreas Opedal, Eleanor Chodroff, Ryan Cotterell, and Ethan Wilcox. 2024. https://aclanthology.org/2024.emnlp-main.179/ On the role of context in reading time prediction . In Proceedings of EMNLP 2024, pages 3042--3058

    2024

  59. [67]

    Tiago Pimentel, Clara Meister, Ethan G Wilcox, Roger P Levy, and Ryan Cotterell. 2023. https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00603/118720 On the effect of anticipation on reading times . TACL, 11:1624--1642

    2023 doi

  60. [68]

    Grusha Prasad and Tal Linzen. 2024. https://aclanthology.org/2024.conll-1.14/ SPAWNing structural priming predictions from a cognitively motivated parser . In Proceedings of CoNLL 2024, pages 178--197

    2024

  61. [69]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf Language models are unsupervised multitask learners . OpenAI blog

    2019

  62. [70]

    Keith Rayner. 1998. https://psycnet.apa.org/record/1998-11174-004 Eye movements in reading and information processing: 20 years of research . Psychological bulletin, 124(3):372--422

    1998

  63. [71]

    Keith Rayner and Charles Clifton, Jr. 2009. https://www.sciencedirect.com/science/article/abs/pii/S0301051108001245 Language processing in reading and speech perception is fast and incremental: implications for event-related potential research . Biological Psychology, 80(1):4--9

    2009

  64. [72]

    Keith Rayner, Gretchen Kambe, and Susan A. Duffy. 2000. https://doi.org/10.1080/713755934 The Effect of Clause Wrap-Up on Eye Movements during Reading . Quarterly Journal of Experimental Psychology Section A: Human Experimental Psychology, 53(4):1061--1080

    2000 doi

  65. [73]

    Marten van Schijndel and Tal Linzen. 2021. https://pubmed.ncbi.nlm.nih.gov/34170031/ Single-Stage prediction models do not explain the magnitude of syntactic disambiguation difficulty . Cognitive Science, 45(6):e12988

    2021

  66. [74]

    Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. 2021. https://www.pnas.org/doi/abs/10.1073/pnas.2105646118 The neural architecture of language: Integrative modeling converges on predi...

    2021 doi

  67. [75]

    Skipper Seabold and Josef Perktold. 2010. https://www.statsmodels.org/stable/index.html statsmodels: Econometric and statistical modeling with Python . In 9th Python in Science Conference

    2010

  68. [76]

    Cory Shain, Idan A Blank, Evelina Fedorenko, Edward Gibson, and William Schuler. 2022. https://www.jneurosci.org/content/42/39/7412.abstract Robust effects of working memory demand during naturalistic language comprehension in language-selective cortex . Journal of Neuroscienc...

    2022

  69. [77]

    Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cotterell, and Roger Levy. 2024. https://www.pnas.org/doi/abs/10.1073/pnas.2307876121 Large-scale evidence for logarithmic effects of word predictability on reading time . Proceedings of the National Academy of Sciences, 121(10):...

    2024 doi

  70. [78]

    Noam Siegelman, Sascha Schroeder, Cengiz Acart \"u rk, Hee-Don Ahn, Svetlana Alexeeva, Simona Amenta, Raymond Bertram, Rolando Bonandrini, Marc Brysbaert, Daria Chernova, et al. 2022. https://link.springer.com/epdf/10.3758/s13428-021-01772-6?sharing_token=As4e3osuA15IaUCKtCvDT...

    2022 doi

  71. [79]

    Nathaniel J Smith and Roger Levy. 2013. https://www.sciencedirect.com/science/article/pii/S0010027713000413 The effect of word predictability on reading time is logarithmic . Cognition, 128(3):302--319

    2013

  72. [80]

    Robyn Speer. 2022. https://doi.org/10.5281/zenodo.7199437 rspeer/wordfreq: v3.0

    2022 doi

  73. [81]

    Jakub M Szewczyk and Kara D Federmeier. 2022. https://www.sciencedirect.com/science/article/pii/S0749596X21000942 Context-based facilitation of semantic access follows both logarithmic and linear functions of stimulus probability . Journal of Memory and Language, 123(104311):104311

    2022

  74. [82]

    Jakub M Szewczyk, Emily N Mech, and Kara D Federmeier. 2022. https://pmc.ncbi.nlm.nih.gov/articles/PMC9059672/ The power of ``good'': Can adjectives rapidly decrease as well as increase the availability of the upcoming noun? Journal of Experimental Psychology: Learning, Memory...

    2022

  75. [83]

    Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. https://aclanthology.org/P19-1452/ BERT rediscovers the classical NLP pipeline . In Proceedings of ACL 2019, pages 4593--4601

    2019

  76. [84]

    Mariya Toneva and Leila Wehbe. 2019. https://proceedings.neurips.cc/paper/2019/hash/749a8e6c231831ef7756db230b4359c8-Abstract.html Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain) . NeurIPS 2019, pages 14928--14938

    2019

  77. [85]

    Pranali Vani, Ethan Gotlieb Wilcox, and Roger Levy. 2021. https://escholarship.org/uc/item/3x34x7dz Using the interpolated maze task to assess incremental processing in english relative clauses . Proceedings of the Annual Meeting of the Cognitive Science Society, 43(43)

    2021

  78. [86]

    Andrea de Varda and Marco Marelli. 2023. https://doi.org/10.18653/v1/2023.acl-short.14 Scaling in cognitive modelling: a multilingual approach to human reading times . In Proceedings of ACL 2023, pages 139--149

    2023 doi

  79. [87]

    Andrea Gregor de Varda, Marco Marelli, and Simona Amenta. 2024. https://pubmed.ncbi.nlm.nih.gov/37880511/ Cloze probability, predictability ratings, and computational estimates for 205 english sentences, aligned with existing EEG and reading time data . Behavior Research Metho...

    2024

  80. [88]

    Shravan Vasishth, Katja Suckow, Richard L Lewis, and Sabine Kern. 2010. https://www.tandfonline.com/doi/abs/10.1080/01690960903310587 Short-term forgetting in sentence comprehension: Crosslinguistic evidence from verb-final structures . Language and Cognitive Processes, 25(4):533--567

    2010 doi

  81. [89]

    Andreas Waldis, Yotam Perlitz, Leshem Choshen, Yufang Hou, and Iryna Gurevych. 2024. https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00718/125534 Holmes a benchmark to assess the linguistic competence of language models . TACL, 12:1616--1647

    2024 doi

  82. [90]

    Daphne Wang, Mehrnoosh Sadrzadeh, Miloš Stanojević, Wing-Yee Chow, and Richard Breheny. 2024. https://aclanthology.org/2024.cmcl-1.14/ How can large language models become more human? In Proceedings of CMCL 2024, pages 166--176

    2024

  83. [91]

    Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. https://aclanthology.org/2024.acl-long.820/ Do llamas work in english? on the latent language of multilingual transformers . In Proceedings of ACL 2024, pages 15366--15394

    2024

  84. [92]

    Ethan Wilcox, Clara Meister, Ryan Cotterell, and Tiago Pimentel. 2023 a . https://aclanthology.org/2023.emnlp-main.466/ Language model quality correlates with psychometric predictive power in multiple languages . In Proceedings of EMNLP 2023, pages 7503--7511

    2023

  85. [93]

    Ethan Wilcox, Clara Meister, Ryan Cotterell, and Tiago Pimentel. 2023 b . https://doi.org/10.18653/v1/2023.emnlp-main.466 Language model quality correlates with psychometric predictive power in multiple languages . In Proceedings of EMNLP 2023, pages 7503--7511

    2023 doi

  86. [94]

    Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger Levy. 2020. http://arxiv.org/abs/2006.01912 On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior . In Proceedings of CogSci, pages 1707--1713

    2020 arXiv

  87. [95]

    Ethan Gotlieb Wilcox, Michael Hu, Aaron Mueller, Tal Linzen, Alex Warstadt, Leshem Choshen, Chengxu Zhuang, Ryan Cotterell, and Adina Williams. 2024. https://osf.io/preprints/psyarxiv/rfwgd Bigger is not always better: The importance of human-scale language modeling for psycho...

    2024

  88. [96]

    Naoko Witzel, Jeffrey Witzel, and Kenneth Forster. 2012. https://link.springer.com/article/10.1007/s10936-011-9179-x Comparisons of online reading paradigms: eye tracking, moving-window, and maze . Journal of Psycholinguistic Research, 41(2):105--128

    2012 doi

  89. [97]

    Edward W Wlotko and Kara D Federmeier. 2012. https://www.sciencedirect.com/science/article/abs/pii/S1053811912004508 So that's what you meant! event-related potentials reveal multiple aspects of context use during construction of message-level meaning . Neuroimage, 62(1):356--366

    2012

  90. [98]

    Ryo Yoshida, Hiroshi Noji, and Yohei Oseki. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.235 Modeling human sentence processing with left-corner recurrent neural network grammars . In Proceedings of EMNLP 2021, pages 2964--2973

    2021 doi

  91. [99]

    Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer...

    2022 arXiv

  92. [100]

    Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. 2020. https://proceedings.neurips.cc/paper/2020/hash/d4dd111a4fd973394238aca5c05bebe3-Abstract.html BERT loses patience: Fast and robust inference with early exit . NeurIPS 2020, abs/2006.04152

    2020 arXiv

  93. [101]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  94. [102]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.