RECORD
Temperature-scaling surprisal estimates improve fit to human reading times -- but does it do so for the "right reasons"?
Computation and Language · Submitted 15 Nov 2023
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Secondly, we find that temperature-scaling probabilities lead to a systematically better fit to reading times (up to 89% improvement in delta log likelihood), across several reading time corpora.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
A wide body of evidence shows that human language processing difficulty is predicted by the information-theoretic measure surprisal, a word's negative log probability in context. However, it is still unclear how to best estimate these probabilities needed for predicting human processing difficulty -- while a long-standing belief held that models with lower perplexity would provide more accurate estimates of word predictability, and therefore lead to better reading time predictions, recent work has shown that for very large models, psycholinguistic predictive power decreases. One reason could be that language models might be more confident of their predictions than humans, because they have had exposure to several magnitudes more data. In this paper, we test what effect temperature-scaling of large language model (LLM) predictions has on surprisal estimates and their predictive power of reading times of English texts. Firstly, we show that calibration of large language models typically improves with model size, i.e. poorer calibration cannot account for poorer fit to reading times. Secondly, we find that temperature-scaling probabilities lead to a systematically better fit to reading times (up to 89% improvement in delta log likelihood), across several reading time corpora. Finally, we show that this improvement in fit is chiefly driven by words that are composed of multiple subword tokens.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | A wide body of evidence shows that human language processing difficulty is predicted by the information-theoretic measure surprisal, a word's negative log probability in context. | background 1.00 | 0.03 | 0.02 | asserted 1.00 |
| s1 | However, it is still unclear how to best estimate these probabilities needed for predicting human processing difficulty -- while a long-standing belief held that models with lower perplexity would provide more accurate estimates of word predictability, and therefore lead to better reading time predictions, recent work has shown that for very large models, psycholinguistic predictive power decreases. | gap 0.77 | 0.04 | 0.04 | asserted 0.89 |
| s2 | One reason could be that language models might be more confident of their predictions than humans, because they have had exposure to several magnitudes more data. | background 0.61 | 0.04 | 0.02 | hedged possibility 0.56 |
| s3 | In this paper, we test what effect temperature-scaling of large language model (LLM) predictions has on surprisal estimates and their predictive power of reading times of English texts. | contribution 0.98 | 0.57 | 0.05 | asserted 0.61 |
| s4 | Firstly, we show that calibration of large language models typically improves with model size, i.e. poorer calibration cannot account for poorer fit to reading times. | result 0.99 | 0.90 | 0.07 | asserted 1.00 |
| s5 | Secondly, we find that temperature-scaling probabilities lead to a systematically better fit to reading times (up to 89% improvement in delta log likelihood), across several reading time corpora. | result 1.00 | 0.97 | 0.61 | asserted 1.00 |
| s6 | Finally, we show that this improvement in fit is chiefly driven by words that are composed of multiple subword tokens. | result 0.97 | 0.95 | 0.25 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v2
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
A wide body of evidence shows that human language processing difficulty is predicted by the information-theoretic measure surprisal, a word's negative log probability in context. However, it is still unclear how to best estimate these probabilities needed for predicting human processing difficulty -- while a long-standing belief held that models with lower perplexity would provide more accurate estimates of word predictability, and therefore lead to better reading time predictions, recent work has shown that for very large models, psycholinguistic predictive power decreases. One reason could be that language models might be more confident of their predictions than humans, because they have had exposure to several magnitudes more data. In this paper, we test what effect temperature-scaling of large language model (LLM) predictions has on surprisal estimates and their predictive power of reading times of English texts. Firstly, we show that calibration of large language models typically improves with model size, i.e. poorer calibration cannot account for poorer fit to reading times. Secondly, we find that temperature-scaling probabilities lead to a systematically better fit to reading times (up to 89% improvement in delta log likelihood), across several reading time corpora. Finally, we show that this improvement in fit is chiefly driven by words that are composed of multiple subword tokens.
Human review
How the literature uses it
2 arXiv papers cite this in 3 sentences
Papers citing it · 2 sentences without a confident label
Builds on it 1 paper
-
Such an advantage has been reported with GPT-2 models [ref] ; we further show that this generalizes to other LLMs.
Psychometric Predictive Power of Large Language Models · Reproducing previous findings · 0.95
What it cites · 7 sentences without a confident label
Uses the method or tool of 1 paper
-
We measure the predictive power of surprisal estimates from different language models, which is denoted as the log-likelihood difference per data point between a linear mixed-effects (LME) regression model using lme4 package [ref] with a predictor of surprisal estimates (target model) and a model without surprisal (base model), following [ref] .
On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior · Metrics and evaluation · 1.00
Cites as background 1 paper
-
Temperature scaling has been successfully applied in several applications: In knowledge distillation [ref] , temperature scaling (with $T>1$) is used to “soften” the knowledge (i.e., probability distribution) provided by the teacher model; in text generation, temperature is used to shape the probability distribution to ease certain aspects of the problems of top-k sampling (e.g., choosing an appropriate $k$ value across varying contexts) [ref] .
Distilling the Knowledge in a Neural Network · Temperature-scaled surprisal · 0.88
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.