REVIEW 5 major objections 5 minor 1 cited by
The Representational Alignment between Humans and Language Models is implicitly driven by a Concreteness Effect
T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper uses behavioral similarity judgments and embedding ablations to argue that the apparent alignment between humans and language models on single-word meaning is largely an alignment on the concreteness dimension.
desk verdict Useful conditional evidence that concreteness is a shared axis of human-model alignment for single nouns; the ablation-based causal claim needs stronger statistics and a linearity check before it will convince. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is Representational Similarity Analysis (RSA), which turns each space, whether human odd-one-out choices, model embeddings, or feature ratings, into a representational dissimilarity matrix (RDM) of pairwise distances and compares RDMs by Spearman correlation. The causal work is done by the ablation procedure: a Ridge regression predicts each model's word embeddings from a feature rating such as concreteness, and the residuals are treated as embeddings with that feature removed; re-running RSA on the residual RDMs quantifies how much of the human-model alignment depended on that feature. The concreteness ratings used for stimulus selection and ablation come from an automatic German norm set, validated against participants' own ratings (r = 0.86).
What would settle it
Train a nonlinear decoder, such as a small multilayer network, on the ridge-residual embeddings to predict participants' concreteness ratings; if it predicts held-out words above chance, the residuals still carry concreteness-related structure, and the linear-ablation drop cannot be read as the full contribution of concreteness to human-model alignment.
Extended reading notes
Core claim
The core claim is that concreteness is the principal shared dimension organizing single-word semantic representations in humans and in language models, even though neither system was explicitly trained or probed on it. According to the paper, the representational dissimilarity matrix derived from participants' odd-one-out choices correlates significantly with embedding-based RDMs for fastText, word2vec, BERT base, BERT large, and GPT2; both the human and model spaces correlate with a concreteness-rating RDM; and linear ablation of concreteness from the embeddings reduces the human-model RSA correlation by an average of 20.6% across models, with every model showing a significant drop, whereas ablation of frequency, length, or orthographic similarity produces effects below 7.6%. The authors take this as evidence that humans and language models converge on the concreteness dimension but not on other dimensions.
Load-bearing premise
The conclusion assumes concreteness is carried linearly in word embeddings, so regressing it out removes it; if concreteness is also encoded nonlinearly, the residual embeddings still contain concreteness-like structure and the measured drop does not isolate concreteness.
Editorial extensions
If this is right
- Word-similarity benchmarks that compare model embeddings with human judgments will need to control for concreteness before claiming broader semantic agreement.
- For model evaluation, matching human odd-one-out behavior on single nouns may be achievable mainly by reproducing the concrete/abstract split, so passing such benchmarks is weak evidence of deep semantic alignment.
- For psychology, the results show that an implicit concreteness axis can be recovered from similarity judgments without ever asking participants about concreteness.
- For engineering, improving human-model alignment beyond concreteness would require training objectives that capture other semantic structure rather than relying on current distributional statistics.
Reading between the lines
- Editorial inference: Because the ablation removes only the linearly decodable part of concreteness, the 20.6% drop should be read as the linear component of concreteness's contribution; a nonlinear probe could show the true contribution is larger.
- Editorial inference: The stimulus set was deliberately built to maximize concreteness variance, so the human RDM is partly structured around that axis; replication with words matched on concreteness but varying other dimensions could reveal alignment that this design underweights.
- Editorial inference: A natural extension is to test contextualized models with the same word in concrete and abstract senses; if the concreteness account is right, model-human alignment should move with the sense shift within a single word type.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates whether the representational alignment between human similarity judgments and language model embeddings for single words is driven by the concreteness of word meanings. The authors collected odd-one-out judgments on 40 carefully selected German nouns, built a behavioral representational dissimilarity matrix (RDM), and compared it via RSA to embeddings from fastText, word2vec, BERT base, BERT large, and GPT2. They report significant human-model alignment, and that both spaces are independently aligned to explicit concreteness ratings. Using a linear residualization ablation (Ridge regression), they find that removing concreteness from model embeddings reduces human-model alignment by an average of 20.6%, while removing frequency, length, or orthographic similarity (OLD20) has effects below 7.6%. They conclude that concreteness critically drives the human-model alignment.
Significance. If the result holds, it would provide evidence that a concreteness dimension is shared between human semantic representations and LLM embeddings, with implications for cognitive modeling, interpretability, and the evaluation of semantic representations. The paper's strengths include the use of an implicit behavioral task, multiple model architectures, and a comparison against several psycholinguistic control features. The authors also commit to releasing code and data, which supports reproducibility. However, the central claim rests on a linearity assumption that is not validated, a stimulus set that maximizes concreteness variance while restricting control-feature variance, and significance tests that treat non-independent RDM entries as independent. These issues make the headline conclusion stronger than the current evidence supports.
major comments (5)
- [3.6] The ablation conclusion assumes that a linear Ridge regression fully removes the targeted feature from the embeddings and that this removal is comparable across features. The manuscript states that the approach 'assumes a linear relationship' but provides no evidence that this assumption holds equally for concreteness and for word frequency, word length, and OLD20. If concreteness is encoded in a strong linear subspace while the control features are encoded more nonlinearly, linear residualization will preferentially delete concreteness structure and produce the observed asymmetry even if the true contributions to alignment are similar. Please report the R² of the Ridge regression for each feature and model, and validate the residualization with a nonlinear removal method or a decoding analysis on the residuals.
- [3.2 and 4.3] The stimulus set was selected from the cluster with the highest concreteness variance, with concrete and abstract words selected beyond one standard deviation, while control features (frequency, length, OLD20) were matched within one standard deviation. This design guarantees that concreteness dominates the RDM and restricts the variance of the control features. Consequently, the small ablation effects for frequency, length, and OLD20 may simply reflect their limited range in the stimulus set rather than their irrelevance to human-model alignment. The claim that alignment is 'not by other dimensions' is too strong. Please quantify the variances of all features in the chosen set and, where possible, provide positive controls demonstrating that the control features are detectable in the model RDMs under the same ablation procedure.
- [4.3 and Table 1] The results for GPT2 are internally inconsistent. Table 1 shows no significant alignment between GPT2 and either the explicit or automatic concreteness RDM (ρ = .06 and .01, both n.s.), yet Section 4.3 reports that removing concreteness significantly reduces GPT2's alignment with the behavioral RDM (p < .001). If GPT2 does not represent the concreteness ratings linearly, the ablation should have little effect on its RDM. The authors should explain this discrepancy or temper the claim that the concreteness effect is consistent across all models.
- [4.3] The significance of RSA correlations and ablation drops is assessed with the Williams test on correlations between RDMs that contain 780 off-diagonal entries derived from only 40 words. These entries are not independent, so the effective sample size is much smaller than 780, and the p-values are likely anti-conservative. Please re-evaluate the significance of the alignment and the ablation effects using item-level permutation tests (e.g., permuting word labels) or bootstrap over participants and words, and report the resulting confidence intervals.
- [Appendix A] The imageability control is not fully conclusive for the uniqueness claim. Imageability correlates .93 with concreteness in the stimulus set, and removing imageability also yields significant alignment drops for all models except word2vec. The conclusion that concreteness 'still plays a more critical role' is based on the relative size of the drop, but no statistical test is reported comparing the concreteness drop to the imageability drop. A direct comparison (e.g., bootstrap of the difference between drops) is needed to support the claim that concreteness is uniquely important.
minor comments (5)
- [3.5] The sentence 'we take the coded ratings (38 per combination)' is unclear: please specify how many participants contributed to each triplet and how the total of 9880 unique triplets was distributed across the 40 participants.
- [Figure 3] Please describe what the error bars in Figure 3 represent (e.g., standard error across participants, permutation intervals) and how the Williams test was applied to each model.
- [3.4] The reference to Müller (2015) is incomplete; please provide a full citation or a URL for the German word2vec embeddings.
- [Table 1] The note that 'behaviorally explicit concreteness ratings seem to perform better' than automatic ratings is not supported by a statistical comparison; either add a test or rephrase as a descriptive observation.
- [3.6] Please report the range of alpha values considered and the selected alpha for each Ridge regression, as well as the number of training embeddings used for each model (771 vs. 716).
Circularity Check
No significant circularity: the human-model alignment and concreteness-ablation results are empirical and not reducible to input definitions.
full rationale
The paper's derivation chain is self-contained at the level of definitions: the human implicit space is an odd-one-out RDM, the model spaces are cosine-distance RDMs from pretrained embeddings, and the explicit concreteness space is a rating-based RDM. The target quantity (the human-model RSA correlation and its change after ablation) is not used to define the concreteness feature or any control feature, and no parameter fitted to the outcome is relabeled as a prediction. The ridge residualization is explicitly an assumed linear operation ('this approach to removing features from embeddings assumes a linear relationship between the feature variables and the embedding'); this assumption limits the inference but does not make the observed 20.6% versus <7.6% drop a logical consequence of the input ratings, because the drop also depends on the independent behavioral odd-one-out RDM and on each model's embedding geometry. Stimulus selection (Section 3.2) intentionally maximized concreteness variance while matching controls, which raises legitimate questions about generalizability, but selecting items by a feature is not the same as defining the alignment result in terms of that feature, and the paper's Limitations section acknowledges the restricted word and frequency range. The only author self-citation (Fiebach and Friederici, 2004) supports a background neuroimaging claim and is externally established, so it is not load-bearing. No circular step can be exhibited with the required specificity.
Assumptions & free parameters
free parameters (3)
- Number of clusters k in stimulus selection =
19
- Concreteness and frequency extreme-selection threshold =
1 standard deviation beyond the cluster mean
- Control-feature matching window =
within 1 SD of the cluster mean
assumptions (4)
- domain assumption Spearman correlation between RDMs measures representational alignment
- domain assumption Odd-one-out choices reflect pairwise similarity, coded as 1 for the two unchosen words and 0 for pairs involving the chosen word
- ad hoc to paper Concreteness is linearly decodable from word embeddings, so Ridge residuals are concreteness-free
- domain assumption Token-averaged non-contextual BERT and GPT2 embeddings are adequate single-word representations
Cite this review
Pith. "Pith review of The Representational Alignment between Humans and Language Models is implicitly driven by a Concreteness Effect." pith.science (2026). https://pith.science/paper/QFF2SCJY
@misc{pith2026250515682,
author = {Pith},
title = {Pith review of: The Representational Alignment between Humans and Language Models is implicitly driven by a Concreteness Effect},
year = {2026},
howpublished = {\url{https://pith.science/paper/QFF2SCJY}},
note = {Machine review of arXiv:2505.15682}
}
read the original abstract
The nouns of our language refer to either concrete entities (like a table) or abstract concepts (like justice or love), and cognitive psychology has established that concreteness influences how words are processed. Accordingly, understanding how concreteness is represented in our mind and brain is a central question in psychology, neuroscience, and computational linguistics. While the advent of powerful language models has allowed for quantitative inquiries into the nature of semantic representations, it remains largely underexplored how they represent concreteness. Here, we used behavioral judgments to estimate semantic distances implicitly used by humans, for a set of carefully selected abstract and concrete nouns. Using Representational Similarity Analysis, we find that the implicit representational space of participants and the semantic representations of language models are significantly aligned. We also find that both representational spaces are implicitly aligned to an explicit representation of concreteness, which was obtained from our participants using an additional concreteness rating task. Importantly, using ablation experiments, we demonstrate that the human-to-model alignment is substantially driven by concreteness, but not by other important word characteristics established in psycholinguistics. These results indicate that humans and language models converge on the concreteness dimension, but not on other dimensions.
Figures
Figures from the paper (1 more)
Forward citations
Cited by 1 Pith paper
-
Large Language Models Show Signs of Alignment with Human Neurocognition During Abstract Reasoning
Only the largest tested LLMs (about 70 billion parameters) match human accuracy on an abstract reasoning task, and the internal geometry of their best layers correlates moderately with human frontal EEG activity.
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Jeffrey R Binder, Lisa L Conant, Colin J Humphries, Leonardo Fernandino, Stephen B Simons, Mario Aguilar, and Rutvik H Desai. 2016. Toward a brain-based componential semantic representation. Cognitive neuropsychology, 33(3-4):130--174
work page 2016
-
[4]
Jeffrey R Binder, Chris F Westbury, Kristen A McKiernan, Edward T Possing, and David A Medler. 2005. Distinct brain systems for processing concrete and abstract concepts. Journal of cognitive neuroscience, 17(6):905--917
work page 2005
-
[5]
Andrea Bruera, Yuan Tao, Andrew Anderson, Derya C okal, Janosch Haber, and Massimo Poesio. 2023. Modeling brain representations of words' concreteness in context using gpt-2 and human ratings. Cognitive Science, 47(12):e13388
work page 2023
-
[6]
Marc Brysbaert, Matthias Buchmeier, Markus Conrad, Arthur M Jacobs, Jens B \"o lte, and Andrea B \"o hl. 2011. The word frequency effect. Experimental psychology
work page 2011
-
[7]
Marc Brysbaert, Pawe Mandera, and Emmanuel Keuleers. 2018. The word frequency effect in word processing: An updated review. Current directions in psychological science, 27(1):45--50
work page 2018
-
[8]
Marc Brysbaert, Amy Beth Warriner, and Victor Kuperman. 2014. Concreteness ratings for 40 thousand generally known english word lemmas. Behavior research methods, 46:904--911
2014
Show all 42 references
-
[9]
Madalina Bucur and Costanza Papagno. 2021. An ale meta-analytical review of the neural correlates of abstract and concrete words. Scientific reports, 11(1):15727
2021
-
[10]
Branden Chan, Stefan Schweter, and Timo M \"o ller. 2020. https://doi.org/10.18653/v1/2020.coling-main.598 G erman`s next language model . In Proceedings of the 28th International Conference on Computational Linguistics, pages 6788--6796, Barcelona, Spain (Online). Internation...
2020 doi
-
[11]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...
2019 doi
-
[12]
Birk Diedenhofen and Jochen Musch. 2015. cocor: A comprehensive solution for the statistical comparison of correlations. PloS one, 10(4):e0121945
2015
-
[13]
Christian J Fiebach and Angela D Friederici. 2004. Processing concrete words: fmri evidence against a specific right-hemisphere involvement. Neuropsychologia, 42(1):62--70
2004
-
[14]
Klaus Fliessbach, Susanne Weis, Peter Klaver, Christian Erich Elger, and Bernd Weber. 2006. The effect of word concreteness on recognition memory. NeuroImage, 32(3):1413--1421
2006
-
[15]
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018)
2018
-
[16]
Martin N Hebart, Charles Y Zheng, Francisco Pereira, and Chris I Baker. 2020. Revealing the multidimensional mental representations of natural objects underlying human similarity judgements. Nature human behaviour, 4(11):1173--1185
2020
-
[17]
Hsu-Wen Huang and Kara D Federmeier. 2015. Imaginative language: What event-related potentials have revealed about the nature and source of concreteness effects. Language and linguistics, 16(4):503--515
2015
-
[18]
Carlton T James. 1975. The role of semantic information in lexical decisions. Journal of Experimental Psychology: Human Perception and Performance, 1(2):130
1975
-
[19]
Philipp Kanske and Sonja A Kotz. 2010. Leipzig affective norms for german: A reliability study. Behavior research methods, 42:987--991
2010
-
[20]
Maximilian K \"o per and Sabine Schulte Im Walde. 2016. Automatically generated affective norms of abstractness, arousal, imageability and valence for 350 000 german lemmas. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pa...
2016
-
[21]
Nikolaus Kriegeskorte and Rogier A Kievit. 2013. Representational geometry: integrating cognition, computation, and the brain. Trends in cognitive sciences, 17(8):401--412
2013
-
[22]
Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. 2008. Representational similarity analysis-connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2:249
2008
-
[23]
Alessandro Lenci, Magnus Sahlgren, Patrick Jeuniaux, Amaru Cuba Gyllensten, and Martina Miliani. 2022. A comparative evaluation and analysis of three generations of distributional semantic models. Language resources and evaluation, 56(4):1269--1313
2022
-
[24]
Guido L \"o hr. 2024. Does the mind care about whether a word is abstract or concrete? why concreteness is probably not a natural kind. Mind & Language, 39(5):627--646
2024
-
[25]
Gonzalo Mart \' nez, Juan Diego Molero, Sandra Gonz \'a lez, Javier Conde, Marc Brysbaert, and Pedro Reviriego. 2025. Using large language models to estimate features of multi-word expressions: Concreteness, valence, arousal. Behavior Research Methods, 57(1):1--11
2025
-
[26]
Maria Montefinese. 2019. Semantic representation of abstract and concrete words: A minireview of neural evidence. Journal of neurophysiology, 121(5):1585--1587
2019
-
[27]
Andreas Müller . 2015. https://devmount.github.io/GermanWordEmbeddings Analyse von Wort-Vektoren deutscher Textkorpora
2015
-
[28]
Subba Reddy Oota, Emin C elik, Fatma Deniz, and Mariya Toneva. 2024. https://doi.org/10.18653/v1/2024.acl-long.462 Speech language models lack important brain-relevant semantics . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volum...
2024 doi
-
[29]
Allan Paivio, John C Yuille, and Stephen A Madigan. 1968. Concreteness, imagery, and meaningfulness values for 925 nouns. Journal of experimental psychology, 76(1p2):1
1968
-
[30]
Pedregosa, G
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in P ython. Journal of Machine L...
2011
-
[31]
Francisco Pereira, Bin Lou, Brianna Pritchett, Samuel Ritter, Samuel J Gershman, Nancy Kanwisher, Matthew Botvinick, and Evelina Fedorenko. 2018. Toward a universal decoder of linguistic meaning from brain activation. Nature communications, 9(1):963
2018
-
[32]
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9
2019
-
[33]
Jamie Reilly, Cory Shain, Valentina Borghesani, Philipp Kuhnke, Gabriella Vigliocco, Jonathan E Peelle, Bradford Z Mahon, Laurel J Buxbaum, Asifa Majid, Marc Brysbaert, et al. 2024. What we mean when we say semantic: Toward a multidisciplinary semantic glossary. Psychonomic bu...
2024
-
[34]
Valery Solovyev. 2020. Concreteness/abstractness concept: State of the art. In International Conference on Cognitive Sciences, pages 275--283. Springer
2020
-
[35]
Ilia Sucholutsky, Lukas Muttenthaler, Adrian Weller, Andi Peng, Andreea Bobu, Been Kim, Bradley C Love, Erin Grant, Iris Groen, Jascha Achterberg, et al. 2023. Getting aligned on representational alignment. arXiv preprint arXiv:2310.13018
2023 arXiv
-
[36]
Bill Thompson and Gary Lupyan. 2018. Automatic estimation of lexical concreteness in 77 languages. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 40
2018
-
[37]
Jacopo Turini and Melissa Le-Hoa V \ o . 2022. Hierarchical organization of objects in scenes is reflected in mental representations of objects. Scientific Reports, 12(1):20068
2022
-
[38]
Raphael Vallat. 2018. Pingouin: statistics in python. J. Open Source Softw., 3(31):1026
2018
-
[39]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30
2017
-
[40]
Christian Wartena. 2024. Estimating word concreteness from contextualized embeddings. In Proceedings of the 20th Conference on Natural Language Processing (KONVENS 2024), pages 81--88
2024
-
[41]
Evan J Williams. 1959. The comparison of regression variables. Journal of the Royal Statistical Society: Series B (Methodological), 21(2):396--399
1959
-
[42]
Tal Yarkoni, David Balota, and Melvin Yap. 2008. Moving beyond coltheart’s n: A new measure of orthographic similarity. Psychonomic bulletin & review, 15(5):971--979
2008
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.