{"id":"d921a20d-c9ca-411a-90f0-f33928b10ce6","arxiv_id":"2504.16884","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In four word-prediction LLMs, hidden-unit sentence similarity reflects syntax more than thematic roles, while some BERT attention heads do capture thematic role assignments.","lead":"Testing four large language models, this paper finds that their sentence similarity judgments track syntax more than who did what to whom, opposite to human ratings. Some attention heads do encode thematic roles, so the models have the capacity even though hidden units de-emphasize it.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hidden-unit negative result depends on a single-token, linear readout; if nonlinear or multi-token probes recover thematic roles, the 'weak influence' claim collapses to a narrow geometry statement.","rationale":"The reader's weakest_assumption correctly identifies the readout of hidden units as the load-bearing point. I agree that the linear single-token readout could miss role information encoded nonlinearly or in other token positions. This concern is sharpened by the paper's own attention-head finding, which shows that role information is present in a component not captured by the hidden-unit readout. The Discussion acknowledges the possibility, but the acknowledgement does not reduce the risk to the central claim: the headline 'relative to humans, this information influences their representations more weakly' is about hidden-unit representations specifically, and the negative result is only as strong as the readout that produced it. A nonlinear or multi-token probe is the obvious decisive check. Because the authors already label the work as exploratory on some analyses and because the reader's condition is appropriate, I do not move the verdict; the conditional status should remain pending the probe-based verification.","tokens_in":23384,"tokens_out":10741,"duration_ms":112467,"concrete_test":"Re-run the Experiment 2 hidden-unit pipeline on the released OSF data with (i) an RBF-kernel SVM and (ii) a multi-token representation (concatenating hidden states at content-word tokens, or mean-pooling across tokens), using the same 66-fold held-out-structure split and per-unit z-scoring. If either probe exceeds roughly 0.70 held-out accuracy, comparable to the reported BERT attention head, the negative hidden-unit finding is a readout artifact and the 'weak influence' claim needs to be restricted to single-token linear geometry; if both probes remain near the reported 0.59-0.67 range, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that thematic-role information influences LLM representations 'more weakly' than in humans rests on a specific readout of hidden units: cosine similarity of one per-sentence token ([CLS] for BERT, final period for GPT2/Llama2/Persimmon) and, in Experiment 2, a linear SVM trained on difference vectors of those same single-token representations. The authors explicitly concede in the Discussion that 'thematic role information may be present in other tokens' and 'non-linearly within the hidden representations.' This is not a minor caveat. Their own positive attention-head result, BERT layer-11 head-5 at 0.79 held-out accuracy, demonstrates that role information can live in a model component that the hidden-unit readout does not expose. A linear SVM over 768/4096 dimensions with 9,000 training pairs and 66-fold cross-validation can easily miss a distributed nonlinear code, so the Experiment 2 statement that there is 'little evidence' in 'any subset of hidden units' overreaches. The human comparison is also indirect: humans gave explicit similarity ratings, while LLM 'influence' is an untasked representational cosine; the two are never placed on a common scale. But the readout issue alone is sufficient to make the headline claim conditional on an untested representational assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper asks whether large language models trained on word prediction internally represent thematic roles ('who did what to whom'). In two experiments the authors compare LLM sentence representations (BERT, GPT2, Llama2, Persimmon) with human similarity ratings on controlled reversible sentences. Experiment 1 shows that, in the last hidden layer, cosine similarity of a single per-sentence token is more affected by shared syntax than by shared thematic roles, the opposite of the pattern in human ratings. Experiment 2 uses ditransitive stimuli with 12 structures and finds little evidence of thematic-role information in hidden-unit SVMs, while one BERT attention head (layer 11, head 5) classifies shared vs. reversed roles at 0.79 accuracy, and the authors characterize that head as attending to agents more than patients. The paper concludes that LLMs can extract thematic roles but, relative to humans, this information influences their representations more weakly.","tokens_in":23560,"tokens_out":5620,"duration_ms":48951,"significance":"The study is a careful, methodologically transparent contribution to the debate on whether LLMs acquire human-like semantic representations from a word-prediction objective. Its strengths include a clean within-subject design with reversible sentences that avoid plausibility shortcuts, a human benchmark collected under controlled conditions, results across four model families with multiple-comparison corrections, and public code and data on OSF. The attention-head finding, if it survives independent validation, provides evidence that role information can be encoded in a model component separate from the hidden units commonly used in representational-similarity analyses. The principal weakness is that the abstract's headline claim is broader than what the readout instruments (single-token cosine similarities and linear SVMs) can support; the authors themselves acknowledge this gap in the Discussion. With appropriate qualification or additional probes, the paper would be a useful contribution to the psycholinguistic and interpretability literatures.","major_comments":[{"comment":"The abstract's central claim that thematic-role information 'influences their representations more weakly' than in humans rests on cosine similarities of a single per-sentence token ([CLS] for BERT, '.' for the other models) and on linear SVMs trained on the difference of those same token vectors. The Discussion itself concedes that thematic-role information may be present non-linearly or in other tokens, and the positive attention-head result (BERT layer-11 head 5, 0.79) demonstrates that role information can be absent from this readout yet present in another model component. Because a negative result under a linear, single-token readout does not license the unqualified claim that thematic roles are represented more weakly in LLMs, the paper should either add probes that relax these readout assumptions (for example, nonlinear classifiers or token-level representations) or explicitly restrict the claim in the abstract and title to the specific readout examined.","section":"Discussion (final paragraph); Experiment 2 (SVM: hidden units)"},{"comment":"The positive attention-head result is obtained by selecting the single highest-accuracy head (BERT layer 11, head 5, accuracy 0.79) after scanning all 144 heads. Although the significance tests are Bonferroni-corrected, the subsequent functional characterization of this head (Figure 7 and the mixed-effects models) is carried out on the very head selected for maximal accuracy, so those descriptive statistics are not independent of the selection. The authors should present the functional characterization as exploratory or validate it on a second sample of heads (or a pre-registered head) before claiming that 'some attention heads robustly captured thematic roles, independently of syntax' as a general finding.","section":"Experiment 2, Results: SVM: attention heads; Figures 6 and 7"},{"comment":"The claim 'relative to humans' compares human explicit similarity ratings with LLM untasked representational cosine similarities, which are on different scales and involve different tasks. The authors explicitly decline to compare humans and LLMs directly, and in Experiment 2 the human 'implicit classification accuracy' of 0.703 is a derived proportion from rating directions, not a classification accuracy comparable to the SVM values. To support the relative-strength claim, the authors should either place both measures on a common scale (for example, convert human ratings into pairwise same/different judgments and compute an ROC or accuracy value) or reformulate the conclusion as a qualitative divergence in which factor (syntax vs. thematic roles) dominates each system's similarity, rather than a quantitative comparison of influence strength.","section":"Experiment 1, Human Judgments; Discussion"}],"minor_comments":[{"comment":"In the sentence 'give that we are asking whether meaning can be learned', the word 'give' should be 'given'.","section":"Introduction, fifth numbered paragraph"},{"comment":"The list of words whose attention weights were extracted lists 'indirect object' twice; the second occurrence should presumably be 'direct object'.","section":"Experiment 2, Methods, LLMs: attention heads"},{"comment":"The caption says 'Each line corresponds to one for 24 sentence types'; this should read 'one of the 24 sentence types'.","section":"Figure 7 caption"},{"comment":"The phrase 'the central training objective of LLMs-word prediction' contains a stray hyphen; consider 'LLMs—word prediction—'.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a cognitive-science or psycholinguistics journal. The authors are honest about the limitations, which helps, but the abstract and conclusion oversell the readout. I am confident the central finding (syntax dominating similarity in hidden-unit readouts, opposite to humans) is solid; the fix is a matter of scope and additional analysis rather than a fundamental flaw. I would encourage the editor to request a revision rather than reject."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper does something real. It uses reversible sentences and a clean 2x2 design to show that four LLMs' cosine similarities track syntax over thematic roles, while human similarity judgments do the opposite. That's a new, controlled result. The attention-head finding — BERT layer 11 head 5 reliably classifies shared vs reversed roles across held-out structures, and its attention pattern is role-directed — is a nice dissociation, and it's characterized with proper mixed-effects models rather than just a single accuracy number.\n\nThe Experiment 1 critical comparison is well constructed: both critical conditions share lexical items with the base, so the syntax-vs-roles contrast isn't confounded by word overlap. Multiple-comparison correction is applied. Reproducibility is good (OSF data/code). The writing is clear about what was and wasn't tested.\n\nSoft spots, in order of real weight. First, the hidden-unit negative result rests on a single-token readout ([CLS] or final period) and a linear SVM. The authors say 'little evidence ... in any subset of hidden units,' but they only searched linear subsets of one token's representation. Their own positive attention-head result shows role information can live in a component their hidden-unit readout doesn't expose. They explicitly concede the nonlinear and other-token possibilities in the Discussion, so it's an acknowledged limitation, but the abstract's 'influences their representations more weakly' is a statement about their chosen geometry, not about all possible representations. Second, the human-LLM comparison is indirect — Likert ratings vs cosine similarities, never on a common scale. They do show the direction reverses on the same 24 stimulus sets, which is persuasive at the qualitative level, but 'weaker than humans' is a bit stronger than the data support. Third, the attention-head positive result is BERT-only and the headline head is selected post hoc. The characterization of that head mitigates this, but replication in another bidirectional model would help.\n\nNone of these are fatal. The central finding holds up for what it measures. This paper deserves a serious referee. I'd send it out.","headline":"A well-designed controlled study showing that LLM sentence-geometry tracks syntax more than thematic roles, with a genuinely interesting attention-head dissociation — worth serious review despite a readout caveat.","tokens_in":24136,"tokens_out":3725,"would_cite":true,"duration_ms":30481,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that LLMs can extract thematic roles but, relative to humans, this information influences their hidden-unit sentence representations more weakly: representational similarity tracks syntax, not agent-patient assignment…","keywords":["large language models","comprehension","thematic roles","representational similarity","agent-patient assignment","attention heads","sentence embeddings","linear probing"],"falsifier":"Re-run Experiment 2's linear classifier on subject- and object-token activations rather than the final period token; if held-out structure classification accuracy then reaches or exceeds 0.79, the weak hidden-unit result is a readout artifact. Alternatively, ablate BERT layer-11 head 5 and measure next-word prediction on role-reversed passives; if behavior is unchanged, the head's role information is not functionally implicated.","tokens_in":23141,"feed_emoji":"🤖","tokens_out":8661,"duration_ms":72538,"temperature":0.7,"pith_summary":"The paper asks whether word prediction—the objective on which large language models are pretrained—produces sentence representations that encode a core piece of sentence meaning: which participant did the action and which received it. Using reversible sentences that remove plausibility shortcuts, the authors find that when sentence similarity is measured over all hidden units, all four tested models are more similar for pairs that share syntax but reverse roles than for pairs that share roles but change syntax. Human similarity ratings show the opposite pattern. Linear probes over hidden units find little generalizable role information, yet a linear classifier on one BERT attention head reaches 79 percent accuracy at telling shared from reversed role assignments across held-out structures. The paper concludes that LLMs can extract thematic roles, but this information has a weaker influence on their hidden representations than it does in humans.","feed_headline":"Role knowledge sits in attention heads, not in LLM sentence vectors","feed_subtitle":"Four models show syntax, not roles, in their hidden layers; one BERT head classifies roles at 79 percent accuracy.","key_machinery":"The central machinery is a critical sentence-pair contrast: same thematic role assignment with different syntax versus same syntax with opposite assignment. Sentence representations are extracted from a single token per sentence (the [CLS] token in BERT and the final period token in GPT2, Llama2, and Persimmon), and pairwise cosine similarity after per-unit normalization is used to test whether role information or syntax dominates. Experiment 2 adds linear support-vector-machine classifiers trained on hidden-unit activation differences and on attention-weight vectors between content words, with held-out syntactic structures; the key positive result is a BERT attention head (layer 11, head 5) whose weights distinguish agents from patients across structures.","core_discovery":"On the paper's own terms, LLM representations of sentences are dominated by grammatical form. In Experiment 1, every tested model (BERT, GPT2-Small, Llama2-7B, and Persimmon-8B) judged a pair with opposite agent and patient assignments but identical active syntax as more similar than a pair with the same thematic roles but different syntax. This is the reverse of human similarity ratings. Experiment 2, using ditransitive and cleft sentences, found little hidden-unit evidence for thematic-role information that generalizes across syntactic structures: linear classifiers trained on activation differences mostly stayed near chance or generalized poorly to held-out structures. However, a specific BERT attention head (layer 11, head 5) reached 0.79 classification accuracy and did so across sentence structures, attending to agents more than patients from the verb and direct object. The authors therefore claim that LLMs have the capacity to extract thematic roles, but this information has only weak influence in the hidden-unit sentence representation commonly used to compare models with humans.","pith_inferences":["An extension the paper does not make: if role information is stored at token positions other than the final token, the hidden-unit null result may understate LLM semantic knowledge; a natural follow-up is to train probes on subject, object, and verb token vectors separately.","The attention-head finding suggests a causal test: ablating BERT layer-11 head 5 should reduce role sensitivity in downstream comprehension or next-word prediction if the head is functionally important.","Because humans were explicitly asked to judge similarity while LLMs were probed without task framing, a fairer comparison might use LLM pairwise similarity judgments or human neural measures and could shift the apparent human-model gap.","If attention heads encode roles independently of syntax, role extraction may not require large models; comparing head-level role accuracy across model sizes would test whether this capacity is scale-dependent."],"forward_implications":["Comparisons between LLM hidden activations and human brain or behavior data should not be expected to carry thematic-role information, because this information is weak in exactly those activations.","Scaling a word-prediction model from 117 million to 8 billion parameters did not make thematic roles more prominent in hidden-unit similarity, suggesting that more of the same training objective is not sufficient.","Attention heads, not sentence-level vectors, are the right target for studying semantic-role knowledge in pretrained transformers.","A model can contain extractable role information in attention heads while that information exerts little influence on its overall sentence representation, so the presence of information and its use are separate questions.","The authors explicitly leave open the possibility that other architectures, training corpora, or non-linguistic training such as reinforcement learning from human feedback could change how prominently thematic roles influence hidden representations."],"supporting_citations":[{"why":"Supplies BERT and the [CLS]-token sentence representation that anchors Experiment 1.","marker":"Devlin et al., 2018"},{"why":"Supplies GPT2-Small, one of the two small models whose hidden-unit similarities are measured.","marker":"Radford et al., 2019"},{"why":"Supplies Llama 2-7B, testing whether a larger word-prediction model shows stronger role influence.","marker":"Touvron et al., 2023"},{"why":"Supplies Persimmon-8B, the largest model and a second check of scale.","marker":"Elsen et al., 2023"},{"why":"Source of the base stimuli and the same-versus-different semantics-by-syntax design used in Experiment 1.","marker":"Fedorenko et al., 2020"},{"why":"Provides representational similarity analysis, the method behind the cosine-similarity comparisons.","marker":"Kriegeskorte et al., 2008"},{"why":"Establishes the practice of comparing LLM hidden activations to human brain data, the target the paper's conclusion speaks to.","marker":"Schrimpf et al., 2021"},{"why":"Documents how plausibility cues usually make thematic roles redundant in natural text, motivating the reversible stimuli.","marker":"Mahowald et al., 2023"},{"why":"Establishes the prior claim that LLM internal representations contain thematic-role information, which Experiment 2 refines.","marker":"Tenney, Das, et al., 2019"}],"fun_headline_variants":["Who did what? Attention heads know, LLM vectors don't","Syntax overrides roles in LLM space; one head fixes it","LLM hidden layers skip who-whom, but attention heads catch it","Thematic roles lie in attention heads, not sentence vectors","Grammar trumps meaning in LLMs; a single head recovers roles"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that thematic roles influence LLM representations only weakly depends on the assumption that one sentence vector per sentence (the [CLS] token in BERT, the final period token in the other models), measured with cosine similarity and linear classifiers over normalized hidden units, would reveal role information if the model had it.","fun_headline_variants_meta":{"raw":{"variants":["Who did what? Attention heads know, LLM vectors don't","Syntax overrides roles in LLM space; one head fixes it","LLM hidden layers skip who-whom, but attention heads catch it","Thematic roles lie in attention heads, not sentence vectors","Grammar trumps meaning in LLMs; a single head recovers roles"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000654,"raw_usage":{"total_tokens":2981,"prompt_tokens":913,"completion_tokens":2068,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":1976}},"tokens_in":529,"tokens_out":2068,"duration_ms":12879,"temperature":1.0,"reasoning_tokens":1976,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:53:31.431298+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run Experiment 2's linear classifier on subject- and object-token activations rather than the final period token; if held-out structure classification accuracy then reaches or exceeds 0.79, the weak hidden-unit result is a readout artifact. Alternatively, ablate BERT layer-11 head 5 and measure next-word prediction on role-reversed passives; if behavior is unchanged, the head's role information is not functionally implicated.","supporting_citations":[],"review_version":1}