REVIEW 3 major objections 4 minor 4 references
Do Large Language Models know who did what to whom?
T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read The paper claims that LLMs can extract thematic roles but, relative to humans, this information influences their hidden-unit sentence representations more weakly: representational similarity tracks syntax, not agent-patient assignment…
desk verdict A well-designed controlled study showing that LLM sentence-geometry tracks syntax more than thematic roles, with a genuinely interesting attention-head dissociation — worth serious review despite a readout caveat. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is a critical sentence-pair contrast: same thematic role assignment with different syntax versus same syntax with opposite assignment. Sentence representations are extracted from a single token per sentence (the [CLS] token in BERT and the final period token in GPT2, Llama2, and Persimmon), and pairwise cosine similarity after per-unit normalization is used to test whether role information or syntax dominates. Experiment 2 adds linear support-vector-machine classifiers trained on hidden-unit activation differences and on attention-weight vectors between content words, with held-out syntactic structures; the key positive result is a BERT attention head (layer 11, head 5) whose weights distinguish agents from patients across structures.
What would settle it
Re-run Experiment 2's linear classifier on subject- and object-token activations rather than the final period token; if held-out structure classification accuracy then reaches or exceeds 0.79, the weak hidden-unit result is a readout artifact. Alternatively, ablate BERT layer-11 head 5 and measure next-word prediction on role-reversed passives; if behavior is unchanged, the head's role information is not functionally implicated.
Extended reading notes
Core claim
On the paper's own terms, LLM representations of sentences are dominated by grammatical form. In Experiment 1, every tested model (BERT, GPT2-Small, Llama2-7B, and Persimmon-8B) judged a pair with opposite agent and patient assignments but identical active syntax as more similar than a pair with the same thematic roles but different syntax. This is the reverse of human similarity ratings. Experiment 2, using ditransitive and cleft sentences, found little hidden-unit evidence for thematic-role information that generalizes across syntactic structures: linear classifiers trained on activation differences mostly stayed near chance or generalized poorly to held-out structures. However, a specific BERT attention head (layer 11, head 5) reached 0.79 classification accuracy and did so across sentence structures, attending to agents more than patients from the verb and direct object. The authors therefore claim that LLMs have the capacity to extract thematic roles, but this information has only weak influence in the hidden-unit sentence representation commonly used to compare models with humans.
Load-bearing premise
The claim that thematic roles influence LLM representations only weakly depends on the assumption that one sentence vector per sentence (the [CLS] token in BERT, the final period token in the other models), measured with cosine similarity and linear classifiers over normalized hidden units, would reveal role information if the model had it.
Editorial extensions
If this is right
- Comparisons between LLM hidden activations and human brain or behavior data should not be expected to carry thematic-role information, because this information is weak in exactly those activations.
- Scaling a word-prediction model from 117 million to 8 billion parameters did not make thematic roles more prominent in hidden-unit similarity, suggesting that more of the same training objective is not sufficient.
- Attention heads, not sentence-level vectors, are the right target for studying semantic-role knowledge in pretrained transformers.
- A model can contain extractable role information in attention heads while that information exerts little influence on its overall sentence representation, so the presence of information and its use are separate questions.
- The authors explicitly leave open the possibility that other architectures, training corpora, or non-linguistic training such as reinforcement learning from human feedback could change how prominently thematic roles influence hidden representations.
Reading between the lines
- An extension the paper does not make: if role information is stored at token positions other than the final token, the hidden-unit null result may understate LLM semantic knowledge; a natural follow-up is to train probes on subject, object, and verb token vectors separately.
- The attention-head finding suggests a causal test: ablating BERT layer-11 head 5 should reduce role sensitivity in downstream comprehension or next-word prediction if the head is functionally important.
- Because humans were explicitly asked to judge similarity while LLMs were probed without task framing, a fairer comparison might use LLM pairwise similarity judgments or human neural measures and could shift the apparent human-model gap.
- If attention heads encode roles independently of syntax, role extraction may not require large models; comparing head-level role accuracy across model sizes would test whether this capacity is scale-dependent.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper asks whether large language models trained on word prediction internally represent thematic roles ('who did what to whom'). In two experiments the authors compare LLM sentence representations (BERT, GPT2, Llama2, Persimmon) with human similarity ratings on controlled reversible sentences. Experiment 1 shows that, in the last hidden layer, cosine similarity of a single per-sentence token is more affected by shared syntax than by shared thematic roles, the opposite of the pattern in human ratings. Experiment 2 uses ditransitive stimuli with 12 structures and finds little evidence of thematic-role information in hidden-unit SVMs, while one BERT attention head (layer 11, head 5) classifies shared vs. reversed roles at 0.79 accuracy, and the authors characterize that head as attending to agents more than patients. The paper concludes that LLMs can extract thematic roles but, relative to humans, this information influences their representations more weakly.
Significance. The study is a careful, methodologically transparent contribution to the debate on whether LLMs acquire human-like semantic representations from a word-prediction objective. Its strengths include a clean within-subject design with reversible sentences that avoid plausibility shortcuts, a human benchmark collected under controlled conditions, results across four model families with multiple-comparison corrections, and public code and data on OSF. The attention-head finding, if it survives independent validation, provides evidence that role information can be encoded in a model component separate from the hidden units commonly used in representational-similarity analyses. The principal weakness is that the abstract's headline claim is broader than what the readout instruments (single-token cosine similarities and linear SVMs) can support; the authors themselves acknowledge this gap in the Discussion. With appropriate qualification or additional probes, the paper would be a useful contribution to the psycholinguistic and interpretability literatures.
major comments (3)
- [Discussion (final paragraph); Experiment 2 (SVM: hidden units)] The abstract's central claim that thematic-role information 'influences their representations more weakly' than in humans rests on cosine similarities of a single per-sentence token ([CLS] for BERT, '.' for the other models) and on linear SVMs trained on the difference of those same token vectors. The Discussion itself concedes that thematic-role information may be present non-linearly or in other tokens, and the positive attention-head result (BERT layer-11 head 5, 0.79) demonstrates that role information can be absent from this readout yet present in another model component. Because a negative result under a linear, single-token readout does not license the unqualified claim that thematic roles are represented more weakly in LLMs, the paper should either add probes that relax these readout assumptions (for example, nonlinear classifiers or token-level representations) or explicitly restrict the claim in the abstract and title to the specific readout examined.
- [Experiment 2, Results: SVM: attention heads; Figures 6 and 7] The positive attention-head result is obtained by selecting the single highest-accuracy head (BERT layer 11, head 5, accuracy 0.79) after scanning all 144 heads. Although the significance tests are Bonferroni-corrected, the subsequent functional characterization of this head (Figure 7 and the mixed-effects models) is carried out on the very head selected for maximal accuracy, so those descriptive statistics are not independent of the selection. The authors should present the functional characterization as exploratory or validate it on a second sample of heads (or a pre-registered head) before claiming that 'some attention heads robustly captured thematic roles, independently of syntax' as a general finding.
- [Experiment 1, Human Judgments; Discussion] The claim 'relative to humans' compares human explicit similarity ratings with LLM untasked representational cosine similarities, which are on different scales and involve different tasks. The authors explicitly decline to compare humans and LLMs directly, and in Experiment 2 the human 'implicit classification accuracy' of 0.703 is a derived proportion from rating directions, not a classification accuracy comparable to the SVM values. To support the relative-strength claim, the authors should either place both measures on a common scale (for example, convert human ratings into pairwise same/different judgments and compute an ROC or accuracy value) or reformulate the conclusion as a qualitative divergence in which factor (syntax vs. thematic roles) dominates each system's similarity, rather than a quantitative comparison of influence strength.
minor comments (4)
- [Introduction, fifth numbered paragraph] In the sentence 'give that we are asking whether meaning can be learned', the word 'give' should be 'given'.
- [Experiment 2, Methods, LLMs: attention heads] The list of words whose attention weights were extracted lists 'indirect object' twice; the second occurrence should presumably be 'direct object'.
- [Figure 7 caption] The caption says 'Each line corresponds to one for 24 sentence types'; this should read 'one of the 24 sentence types'.
- [Abstract] The phrase 'the central training objective of LLMs-word prediction' contains a stray hyphen; consider 'LLMs—word prediction—'.
Circularity Check
No significant circularity: the headline finding is a measured representational contrast against an external human benchmark, and the negative hidden-unit result is a conditional readout limitation, not an input-concealed derivation.
full rationale
The paper's central quantities are cosine similarities and cross-validated SVM accuracies, not fitted constants or derived laws. Experiment 1 compares pre-specified sentence conditions (same vs. reversed thematic roles crossed with same vs. different syntax) against an external human rating benchmark; the LLM similarities are computed directly from hidden-unit activations after normalization on an independent corpus (COCA), with no parameter fitted to the human data. The critical comparison (SEMd-SYNTs > SEMs-SYNTd) is a measured effect, not a predicted consequence of a fitted model. Experiment 2 trains linear SVMs on pairs labeled with shared vs. reversed roles, but tests them on held-out sentence structures in 66-fold cross-validation, so the classification accuracy is not trivially forced by the training labels; the same holds for the BERT attention-head SVM. The follow-up characterization of BERT layer-11 head-5 uses a mixed-effects model with thematic-role direction as a fixed effect, which is an independent statistical contrast on attention weights, not a relabeling of the SVM target. The authors explicitly acknowledge that thematic-role information might live outside the single-token linear readout ('It is possible that thematic role information is present non-linearly within the hidden representations' and 'It is possible that thematic role information may be present in other tokens'), and these are validity caveats about the negative result rather than circular reductions. Self-citations (e.g., Blank 2023; Mahowald et al. 2024; McGee and Blank 2024) are contextual and are not load-bearing for the main empirical claim. No fitted input is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via citation. The derivation chain is self-contained against external behavioral data, so the appropriate finding is no significant circularity.
Assumptions & free parameters
assumptions (5)
- domain assumption Thematic role assignment is a language-internal semantic capacity and can be isolated from world knowledge and reasoning.
- domain assumption Human similarity judgments reflect the prominence of thematic role information and are a valid benchmark for LLM representations.
- domain assumption The [CLS] or final-period token representation, with per-unit normalization, is an appropriate sentence-level readout for thematic role content.
- domain assumption Linear separability in SVM probing indicates that information is available in the representation in a usable form.
- domain assumption The four studied LLMs (BERT, GPT2-Small, Llama2, Persimmon) are representative of word-prediction-only pretrained models.
Cite this review
Pith. "Pith review of Do Large Language Models know who did what to whom?." pith.science (2026). https://pith.science/paper/6A4ZJMCZ
@misc{pith2026250416884,
author = {Pith},
title = {Pith review of: Do Large Language Models know who did what to whom?},
year = {2026},
howpublished = {\url{https://pith.science/paper/6A4ZJMCZ}},
note = {Machine review of arXiv:2504.16884}
}
read the original abstract
Large Language Models (LLMs) are commonly criticized for not understanding language. However, many critiques focus on cognitive abilities that, in humans, are distinct from language processing. Here, we instead study a kind of understanding tightly linked to language: inferring who did what to whom (thematic roles) in a sentence. Does the central training objective of LLMs-word prediction-result in sentence representations that capture thematic roles? In two experiments, we characterized sentence representations in four LLMs. In contrast to human similarity judgments, in LLMs the overall representational similarity of sentence pairs reflected syntactic similarity but not whether their agent and patient assignments were identical vs. reversed. Furthermore, we found little evidence that thematic role information was available in any subset of hidden units. However, some attention heads robustly captured thematic roles, independently of syntax. Therefore, LLMs can extract thematic roles but, relative to humans, this information influences their representations more weakly.
Figures
Reference graph
Works this paper leans on
-
[88]
N. Y. Holt Rinehart Winst. Syntax-Semant. Interface Psych-Verb Constr. 2341 201, 100363–100376. Frankland, S.M., Greene, J.D., 2020. Two ways to build a thought: distinct forms of compositional semantic representation across brain regions. Cereb. Cortex 30, 3838–3855. Frankland, S.M., Greene, J.D., 2015. An architecture for encoding sentence meaning in le...
work page 2020
-
[187]
Breaking NLI systems with sentences that require simple lexical inferences
https://doi.org/10.1016/j.jml.2007.07.004 Glockner, M., Shwartz, V., Goldberg, Y., 2018. Breaking NLI systems with sentences that require simple lexical inferences. ArXiv Prepr. ArXiv180502266. 29 Goldberg, A.E., 2019. Explain me this: Creativity, competition, and the partial productivity of constructions. Princeton University Press. Goldberg, A.E., 1995....
-
[240]
https://doi.org/10.1016/j.cognition.2018.06.019
-
[2021]
The neural architecture of language: Integrative modeling converges on predictive processing. Proc. Natl. Acad. Sci. 118, e2105646118. https://doi.org/10.1073/pnas.2105646118 Schuster, S., Linzen, T., 2022. When a sentence does not introduce a discourse entity, transformer-based models still sometimes refer to it. ArXiv Prepr. ArXiv220503472. Seyffarth, E...
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.