pith. sign in

arxiv: 2606.07066 · v1 · pith:B7YUI4KSnew · submitted 2026-06-05 · 💻 cs.CL

Modeling semantic association in self-paced reading with language model embeddings

Sara M{\o}ller {\O}stergaard , Kenneth Enevoldsen , Afra Alishahi , Bruno Nicenboim This is my paper

Pith reviewed 2026-06-27 22:07 UTC · model grok-4.3

classification 💻 cs.CL
keywords semantic associationlanguage model embeddingsN400self-paced readingBayesian hierarchical modelsEEGDutch texts
0
0 comments X

The pith

Only sentence embeddings from language models detect semantic association effects on N400 and reading times beyond word predictability.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The study compares ten different ways of turning language model embeddings into a measure of how well a word fits its preceding context, using joint EEG and self-paced reading data from Dutch texts. It tests whether these measures still predict brain responses and reading times after word predictability has already been accounted for. The choice of embedding model and context length changes the size and reliability of the detected effects. Only the implementations that use full sentence embeddings produce consistent results on both the N400 component and reading times. This matters because it shows that the operationalization of semantic association is not neutral and directly affects conclusions about what drives comprehension.

Core claim

Semantic association between a word and its context influences the N400 and self-paced reading times even after word predictability is controlled, but this influence appears reliably on both neural and behavioral measures only when semantic association is quantified with sentence embeddings rather than other embedding implementations.

What carries the argument

Ten implementations of embedding-derived semantic association that differ by language model and context length, evaluated in Bayesian hierarchical models against N400 amplitude and reading times.

If this is right

  • Different choices of embedding model and context length produce different conclusions about whether semantic association affects reading beyond predictability.
  • Sentence embeddings are the only approach among the ten tested that yields reliable effects on both N400 and reading times.
  • Semantic association can be separated from predictability in natural text, but only with particular embedding choices.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Future reading studies may need to test sentence embeddings first when quantifying contextual fit.
  • The pattern could be checked in other languages to see whether sentence context remains the key scale.
  • Models of comprehension might be refined by treating sentence-level embeddings as a stronger proxy for integration processes.

Load-bearing premise

The ten embedding implementations are valid and comparable ways to measure semantic association, and the statistical models cleanly separate that association from word predictability without leftover confounding.

What would settle it

A new dataset of Dutch or similar reading times and EEG recordings in which sentence-embedding implementations no longer produce stronger or more reliable effects than the other nine implementations.

Figures

Figures reproduced from arXiv: 2606.07066 by Afra Alishahi, Bruno Nicenboim, Kenneth Enevoldsen, Sara M{\o}ller {\O}stergaard.

Figure 1
Figure 1. Figure 1: Pearson’s correlation coefficients be￾tween implementations of semantic association, log-probability of words, and Zipf word frequency for all content words in the corpus. and adverbs) were included in the analysis. As embeddings extracted from the word2vec model only exist for a finite number of words, the data loss slightly differed across different implementations of semantic association.5 The models we… view at source ↗
Figure 2
Figure 2. Figure 2: Regression coefficients and 95% credible [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Semantic association (as estimated by different implementations), log-probability, and Zipf [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Average semantic association between context and target words (expected, within, between, and [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
read the original abstract

Semantic association between a word and its context has been identified as an important component of reading comprehension, even when word predictability is accounted for. Recent research has highlighted the potential of language model ( LM) embeddings to quantify semantic association. Yet, embedding-based semantic association have been operationalized in a myriad of ways. In this study, we use embeddings from LMs to estimate semantic association on a corpus of joint electroencephalography (EEG) and self-paced reading of natural, Dutch texts. Semantic association is calculated in ten different implementations that vary the embedding model and context lengths. The effects of semantic association across the different implementations on the N400 and self-paced reading times are examined using Bayesian hierarchical models and Bayes factor. The results show that the choice of embedding model can alter the estimated effect of semantic association on both the N400 and self-paced reading times. Furthermore, the results demonstrate a promising potential of sentence embeddings for capturing semantic association, as only implementations relying on sentence embeddings indicate reliable results of semantic association beyond word predictability on both neural and behavioral measures. Together, these findings highlight the importance of methodological choices in quantifying semantic association.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The manuscript examines how different operationalizations of semantic association using language model embeddings affect N400 amplitudes and self-paced reading times in a corpus of Dutch texts, controlling for word predictability. Ten implementations are tested, varying the embedding model and context length. Bayesian hierarchical models and Bayes factors are used to evaluate the reliability of effects. The central result is that only sentence-embedding based implementations produce reliable semantic association effects beyond predictability on both measures.

Significance. If the findings are robust, the paper makes a useful methodological contribution by demonstrating that the choice of embedding model substantially influences the detection of semantic association effects in cognitive measures. The emphasis on sentence embeddings as a promising approach provides practical guidance for researchers using LMs to model semantic processing. The application of Bayes factors to assess reliability is a positive aspect of the analysis.

major comments (1)
  1. [Statistical analysis] Statistical analysis section: The description of the Bayesian hierarchical models does not include multicollinearity diagnostics or checks for residual confounding between semantic association predictors and word predictability. Since the central claim hinges on effects 'beyond word predictability' and only sentence embeddings crossing the reliability threshold, the lack of these checks is a load-bearing concern for interpreting the separation of variance.
minor comments (2)
  1. [Abstract] Grammatical error in abstract: 'embedding-based semantic association have been operationalized' should read 'has been operationalized'.
  2. [Methods] Methods: A summary table comparing the ten implementations (model, context length, embedding type) would improve readability and allow direct comparison of the operationalizations.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their constructive feedback and positive evaluation of the manuscript's contribution. We address the single major comment below.

read point-by-point responses

  1. Referee: [Statistical analysis] Statistical analysis section: The description of the Bayesian hierarchical models does not include multicollinearity diagnostics or checks for residual confounding between semantic association predictors and word predictability. Since the central claim hinges on effects 'beyond word predictability' and only sentence embeddings crossing the reliability threshold, the lack of these checks is a load-bearing concern for interpreting the separation of variance.

    Authors: We agree that multicollinearity diagnostics and checks for residual confounding would improve the transparency and interpretability of the results, especially since the models include both semantic association and word predictability as predictors. In the revised manuscript we will add (i) pairwise correlations between each semantic association measure and word predictability and (ii) variance inflation factor (VIF) values for the full set of predictors in every Bayesian hierarchical model. These diagnostics will be reported in the Statistical analysis section and briefly discussed in the Results. We expect the VIF values to remain well below conventional thresholds, but the explicit reporting will directly address the concern about variance separation. revision: yes

Circularity Check

0 steps flagged

Empirical comparison study; no derivation reduces to inputs

full rationale

The paper reports an empirical analysis comparing ten embedding implementations on observed Dutch EEG and self-paced reading data. Effects are estimated via standard Bayesian hierarchical models with Bayes factors; no first-principles derivation, uniqueness theorem, or ansatz is invoked. The central claim (sentence embeddings alone yield reliable effects beyond predictability) rests on data partitioning rather than any equation that equals its own fitted inputs by construction. No self-citation chain is load-bearing for the reported pattern.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review yields no explicit free parameters, axioms, or invented entities; standard Bayesian hierarchical modeling and Bayes factor analysis are assumed but not detailed.

pith-pipeline@v0.9.1-grok · 5743 in / 1012 out tokens · 25800 ms · 2026-06-27T22:07:29.431404+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 3 canonical work pages · 2 internal anchors

  1. [1]

    Modeling semantic association in self-paced reading with language model embeddings

    Introduction Humans process words in the context in which they are presented. How predictable a word is given its preceding context largely impacts the processing difficulty of the word (Kutas and Federmeier, 2011; Ehrlich and Rayner, 1981; Wong et al., 2024). For example, in the sentence pair, “By the end of the day, the hiker’s feet were extremely cold ...

    work page internal anchor Pith review Pith/arXiv arXiv 2011

  2. [2]

    Data The study used data from the Tilburg corpus of Nat- ural Dutch Texts (TiNT; Østergaard et al., 2025)

    Methods 2.1. Data The study used data from the Tilburg corpus of Nat- ural Dutch Texts (TiNT; Østergaard et al., 2025). The corpus consists of joined recordings ofEEG and self-paced reading (SPR) from 71 participants (whereof 56 participants were included in the analy- sis of the current study). All participants read eight medium-length(approx. 600words),...

    2025

  3. [3]

    Bayes fac- tor forβ2 across the regression models reported in Figure3

    Results Figure 2 displays the coefficients for semantic as- sociation (β2 in Equation 4) estimated by the 20 re- gression models using the values from the different implementations of semantic association to predict the N400 and self-paced reading times. Bayes fac- tor forβ2 across the regression models reported in Figure3. Theresults from theBayes factor...

    1983

  4. [4]

    dragon” appears in both sentences. Only theSE, Allimplementation captures the associa- tion between “draak

    Discussion In this study, we employed both uncontextualized wordembeddingsandcontextualizedsentenceem- beddings to estimate semantic association. These two embedding types appear to capture distinct patterns in the text. This is both apparent from the correlations between the different implementations of semantic association (Figure 1) but, more impor- ta...

    2018

  5. [5]

    The findings demonstrate that the conclusions critically depend on how semantic association is implemented, par- ticularly with respect to the embedding model

    Conclusion This study examined the effects ofLM embedding- basedsemanticassociationontheself-pacedread- ing of medium-length, Dutch texts. The findings demonstrate that the conclusions critically depend on how semantic association is implemented, par- ticularly with respect to the embedding model. While uncontextualized word embeddings (e.g., word2vec) ha...

    2024

  6. [6]

    Code availability Code for reproducing the analysis is publicly available from GitHub athttps://github.com/ saraoe/semantic_association

  7. [7]

    Ethics Statement The study utilized human data from the Tilburg corpus of Natural Dutch Texts (TiNT) collected by Østergaard et al. (2025). The data has received an ethics approval and is licensed under a CC-BY-NC- SA license

    2025

  8. [8]

    Bibliographical References Christoph Aurnhammer, Francesca Delogu, Harm Brouwer, and Matthew W. Crocker. 2023. The P600 as a continuous index of integration effort. Psychophysiology, 60(9):e14302. Nikolay Banar, Ehsan Lotfi, Jens Van Nooten, Cristina Arhiliuc, Marija Kliocaite, and Walter Daelemans. 2025. Mteb-nl and e5-nl: Embed- ding benchmark and model...

    2023

  9. [9]

    Getting real about Semantic Illusions: Rethinking the functional role of the P600 in language comprehension.Brain Research, 1446:127–143. Nyssa Z. Bulkes, Kiel Christianson, and Darren Tanner. 2020. Semantic constraint, reading con- trol, and the granularity of form-based expecta- tionsduringsemanticprocessing: Evidencefrom ERPs.Neuropsychologia, 137:1072...

    work page arXiv 2020

  10. [10]

    InProceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic

    SimCSE: Simple Contrastive Learning of Sentence Embeddings. InProceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. John Hale. 2001. A Probabilistic Earley Parser as a Psycholinguistic Model. InSecond Meeting of the No...

    2021

  11. [11]

    Multilingual E5 Text Embeddings: A Technical Report

    Testing the limits of the semantic illusion phenomenon: ERPs reveal temporary semantic change deafness in discourse comprehension. Cognitive Brain Research, 24(3):691–701. OpenAI. 2025. GPT-5.2 via OpenAI API. Mehdi Parviz, Mark Johnson, Blake Johnson, and Jon Brock. 2011. Using Language Models and Latent Semantic Analysis to Characterise the N400m Neural...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  12. [12]

    long” means the original and the generated longer context, “original

    Prediction in reading: A review of pre- dictability effects, their theoretical implications, and beyond.Psychonomic Bulletin & Review. Haoyin Xu, Masaki Nakanishi, and Seana Coul- son. 2024. Revisiting Joke Comprehension with Surprisal and Contextual Similarity: Implication from N400 and P600 Components.Proceedings of the Annual Meeting of the Cognitive S...

    2024