REVIEW 3 major objections 5 minor 36 references
Uncovering Visual-Semantic Psycholinguistic Properties from the Distributional Structure of Text Embedding Space
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper aims to establish that imageability and concreteness can be read off the geometry of text embeddings alone, using the peakedness of a word's nearest-neighbor neighborhood.
desk verdict A simple, cheap text-only imageability/concreteness estimator that clearly beats baselines, but the paper needs to fix a self-inclusion ambiguity and rule out frequency as the real driver. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Neighborhood Stability Measure (NSM), a distribution-free statistic over a collection of text embeddings. For a query word $q$ and radius $k$, it takes the $k$ approximate nearest neighbors $N_k(q)$ under cosine similarity and returns the fraction of those neighbors $u$ whose own nearest neighbor $N_1(u)$ is also in $N_k(q)$. That fraction is the $\alpha$-stability of the neighborhood (Definition 1), and the paper interprets a value near 1 as a sharp peak in the underlying distribution and a value near 0 as a flat, diffuse region. The machinery rests on approximate nearest neighbor search to make the computation scalable, on a precomputed nearest-neighbor table to reduce each query to $O(T)$, and on the natural-neighbor concept of mutual nearest neighbors, extended from pairs to sets.
What would settle it
Compute NSM on a caption collection after independently permuting the coordinates of each word's embedding so that cosine-similarity neighborhoods are destroyed while token frequency and vector norms are preserved; if the scrambled NSM still correlates with human imageability and concreteness ratings, the reported signal is not neighborhood geometry. A complementary check is to compare NSM within frequency-matched word bins: if the correlation disappears, frequency rather than visual semantics drives the effect.
Extended reading notes
Core claim
On the authors' own terms, the discovery is that the visual-semantic character of a word leaves a geometric trace in its contexts, and that a distribution-free statistic can read that trace. Hypothesis 1 states that the distribution of contexts around an imageable or concrete word forms a sharper peak in semantic space. The authors operationalize peakedness through $\alpha$-stability: a neighborhood is $\alpha$-stable when a proportion $\alpha$ of its points have their nearest neighbor inside that neighborhood, and the Neighborhood Stability Measure (NSM) of a query word is the $\alpha$ of its $k$-nearest-neighbor set. The empirical claim is that this single number correlates more strongly with ground-truth ratings than existing unsupervised methods, often by a wide margin, and that it acts as a strong predictor in imageable/non-imageable and concrete/abstract classification, with reliability increasing for extreme NSM values.
Load-bearing premise
The load-bearing premise is that imageability and concreteness appear as the geometric concentration of a word's contexts in embedding space, and that this concentration is not confounded by word frequency, register, or lexical fixedness.
Editorial extensions
If this is right
- NSM gives 100% coverage of the rating lexicons: any word that can be embedded receives a score, removing the vocabulary-mismatch bottleneck that leaves the HML baseline able to predict only about 13% of imageability words and 2.7% of concreteness words.
- The estimate is reusable: an embedding collection and its nearest-neighbor index are built once, and scoring a new word then costs one neighborhood query plus table lookups, in contrast to the roughly 120 GPU-hours reported for the image-generation baseline.
- The signal depends on the text domain: moving from caption collections to general text collections such as MS MARCO or Quora drops NSM's correlations substantially, indicating that the visual-semantic property is encoded most strongly in visual-descriptive regions of semantic space.
- Embedding dimensionality and collection size matter: smaller 384-dimensional embeddings and larger caption collections give stronger correlations, consistent with distance concentration in high dimensions.
- Extreme NSM values carry more meaning: when words with mid-range scores are excluded, classification AUC rises, so the measure can be used as a confidence-graded predictor.
Reading between the lines
- Because NSM is defined for any query vector, the same statistic could be applied to phrases and sentences; the authors state they have not tested this, and a human-rating study for sentence-level imageability would be the direct test.
- The paper does not control for word frequency or register when attributing neighborhood sharpness to visual semantics; a frequency-matched or residual analysis would show whether NSM adds signal beyond the caption-frequency baselines it reports.
- NSM could double as a diagnostic for embedding spaces: a model whose neighborhoods do not sharpen for concrete words may have anisotropic or poorly calibrated geometry, independent of the psycholinguistic prediction task.
- The caption-corpus requirement suggests a path to cross-lingual norms only where comparable visual-descriptive text exists, so languages without large caption collections would need translated or synthetically generated captions.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes the Neighborhood Stability Measure (NSM), an unsupervised, distribution-free statistic computed from text-only caption embeddings, intended to quantify the 'sharpness' of the neighborhood of a word in semantic space. The authors hypothesize that this sharpness correlates with psycholinguistic imageability and concreteness ratings. They evaluate NSM on COCO, CC3M, and CC12M captions embedded with three sentence transformer models, comparing against frequency counts, the visual-concreteness method of Hessel et al. (2018), and the text-to-image method of Wu and Smith (2023). The main results report higher Spearman correlations with MRC imageability and Brysbaert et al. concreteness than these baselines, as well as strong AUC in threshold-based classification. The paper also reports degraded performance on non-caption text collections and an analysis of the effect of radius k.
Significance. If the result holds, the paper offers a cheap, single-modal, unsupervised estimator for two psycholinguistic variables, with full vocabulary coverage and no need for generative image sampling. The proposed measure is simple, interpretable, and built on standard ANN machinery; the authors provide code and data. The empirical claim that text-embedding geometry alone captures imageability/concreteness is interesting and would be a useful contribution to computational psycholinguistics. However, the current manuscript contains a formal degeneracy in the definition of NSM and leaves the main mechanistic claim underdetermined by not controlling for frequency or contextual diversity.
major comments (3)
- [Section 4.2, Definition 1 and Algorithm 1] As written, NSM is identically 1 for every query, regardless of the data. In Definition 1, N1(u) is defined as the nearest neighbor of u in X (Equation 1). Since u itself belongs to X and δ(u,u) is maximal for any reasonable similarity (e.g., cosine similarity), the nearest neighbor of u is u itself. Therefore, for every u in Nk(q), the condition N1(u) ∈ Nk(q) holds trivially, making α = 1. The paper never states that the point itself is excluded from the nearest-neighbor search. This is a load-bearing issue: the reported correlations in Table 1 and the AUC results in Figures 3 and 4 cannot be reproduced from the stated algorithm. The fix is straightforward—define N1(u) as the nearest neighbor in X \ {u}, or explicitly use the second nearest neighbor—but it must be stated and ideally verified in the revised manuscript.
- [Section 5.1 and Table 1; Section 6] The manuscript never controls for word frequency or contextual diversity, so the central claim that embedding geometry specifically encodes visual-semantic properties is underdetermined. The FREQ baselines in Table 1 already show moderate correlations (up to 0.34 and 0.35), and NSM is a local geometric quantity computed from the same caption collections; it may largely be a proxy for frequency or contextual diversity, which are themselves correlated with imageability and concreteness. To support Hypothesis 1, the authors should report partial Spearman correlations controlling for log frequency, or match words across frequency bins and show NSM retains its discriminative power. The paper's own Discussion concedes that 'why NSM performs better on image-caption datasets' is left unanswered; this is exactly the point where the visual-semantic interpretation needs defense.
- [Table 1 and Table 2] The comparison with HML is not apples-to-apples. HML's correlation is computed only on the 12.9% of imageability words and 2.7% of concreteness words for which it can produce a score, whereas NSM and the other baselines are evaluated on the full rating vocabularies. Because Spearman's correlation is sensitive to the set of points, the reported superiority of NSM over HML could be partly an artifact of different evaluation subsets. The authors should either report HML on a common subset of words or explicitly discuss how the coverage disparity affects the comparison.
minor comments (5)
- [Abstract] There are missing spaces in the typeset text, e.g., 'NSMcorrelates' and 'NSMis', which should be corrected.
- [Section 4.2, Algorithm 1] The algorithm should specify that k must be smaller than the collection size |X|, since otherwise Nk(q) contains all points and the stability value is trivially defined.
- [Section 4.2, input notation] The Algorithm 1 input line says 'embedding collection X ∈ R^d' but X is a set of vectors in R^d; use X ⊂ R^d consistently.
- [Section 4.1, Figure 1] The tSNE visualization is illustrative only; its conclusions depend on tSNE hyperparameters and projection randomness. This is fine, but the text should not overstate what the figure demonstrates.
- [Section 5.1, Methods] The phrase 'this results in a near exact nearest neighbor search' is informal; specify the approximation quality or report the recall of the ANN index on a sample.
Circularity Check
No significant circularity: NSM is a purely geometric, unsupervised statistic validated against external human ratings; no load-bearing step reduces to its inputs.
full rationale
The paper's chain is: Hypothesis 1 links imageability/concreteness to the peakedness of a word's neighborhood in semantic embedding space; Algorithm 1 defines NSM as a distribution-free geometric statistic (the fraction of a query's k-nearest neighbors whose own nearest neighbor lies in the same set); and Section 5 measures Spearman correlation between NSM and external human ratings (MRC and Brysbaert et al.). The only tunable quantity is the radius k, and it is chosen on a validation split (Section 5.1: 'We randomly split each ratings dataset into non-overlapping validation (20%) and test (80%) subsets. If a method requires hyperparameter tuning, we do so on the validation split'), which is standard and not circular. NSM does not incorporate the ground-truth ratings in its definition, so the correlation is an empirical finding rather than a tautology. Self-citations are not load-bearing: Bruch (2024) is cited for ANN-search background and dimensionality-concentration effects, and Wu and Smith (2023) is cited as a baseline and for dataset comparisons; neither supplies the paper's central claim. No uniqueness theorem or ansatz is imported from the authors' prior work. The manuscript itself flag an admitted explanatory gap in Sections 6 and 7: 'We have also left unanswered the question of why NSM performs better on image-caption datasets' and whether its performance can be improved on general text collections. That acknowledgment is a limitation and a potential confound about visual-caption domain dependence, not a circular step. The skeptic's frequency-confound concern—that NSM may partly track word frequency or contextual diversity because Table 1 reports FREQ but does not partial it out—is a substantive correctness/validity threat, not a circularity, since the statistic is computed from geometry alone before any ratings are consulted. Overall, the central claim is empirically testable and tested against external benchmarks, so circularity is effectively absent.
Assumptions & free parameters
free parameters (2)
- radius k =
tuned on validation split; optimal values not reported, Figure 5 suggests 2^10 to 2^12
- ANN index parameters (nprobe, number of clusters) =
nprobe=128, num_clusters=8*sqrt(|X|)
assumptions (4)
- domain assumption The text portion of image-caption datasets contains sufficient distributional signals to estimate imageability and concreteness without the visual modality.
- domain assumption A finite collection of text embeddings is a faithful surrogate for the unknown semantic-space distribution around a word.
- ad hoc to paper The nearest neighbor of a point in a set is defined excluding the point itself, even though Definition 1 does not state this.
- domain assumption Cosine similarity between sentence embeddings reflects semantic-context similarity relevant to human concreteness judgments.
Cite this review
Pith. "Pith review of Uncovering Visual-Semantic Psycholinguistic Properties from the Distributional Structure of Text Embedding Space." pith.science (2026). https://pith.science/paper/RS4HUFEX
@misc{pith2026250523029,
author = {Pith},
title = {Pith review of: Uncovering Visual-Semantic Psycholinguistic Properties from the Distributional Structure of Text Embedding Space},
year = {2026},
howpublished = {\url{https://pith.science/paper/RS4HUFEX}},
note = {Machine review of arXiv:2505.23029}
}
read the original abstract
Imageability (potential of text to evoke a mental image) and concreteness (perceptibility of text) are two psycholinguistic properties that link visual and semantic spaces. It is little surprise that computational methods that estimate them do so using parallel visual and semantic spaces, such as collections of image-caption pairs or multi-modal models. In this paper, we work on the supposition that text itself in an image-caption dataset offers sufficient signals to accurately estimate these properties. We hypothesize, in particular, that the peakedness of the neighborhood of a word in the semantic embedding space reflects its degree of imageability and concreteness. We then propose an unsupervised, distribution-free measure, which we call Neighborhood Stability Measure (NSM), that quantifies the sharpness of peaks. Extensive experiments show that NSM correlates more strongly with ground-truth ratings than existing unsupervised methods, and is a strong predictor of these properties for classification. Our code and data are available on GitHub (https://github.com/Artificial-Memory-Lab/imageability).
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Sebastian Bruch. 2024. Foundations of Vector Retrieval. Springer Nature Switzerland
work page 2024
-
[4]
Marc Brysbaert, Amy Beth Warriner, and Victor Kuperman. 2014. Concreteness ratings for 40 thousand generally known english word lemmas. Behavior Research Methods, 46:904--911
work page 2014
-
[5]
Soravit Changpinyo, Piyush Kumar Sharma, Nan Ding, and Radu Soricut. 2021. https://api.semanticscholar.org/CorpusID:231951742 Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts . In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3557--3567
work page 2021
-
[6]
Jean Charbonnier and Christian Wartena. 2019. https://doi.org/10.18653/v1/W19-0415 Predicting word concreteness and imagery . In Proceedings of the 13th International Conference on Computational Semantics - Long Papers, pages 176--187, Gothenburg, Sweden
-
[7]
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. 2015. http://arxiv.org/abs/1504.00325 Microsoft COCO captions: Data collection and evaluation server
arXiv 2015
-
[8]
Max Coltheart. 1981. https://doi.org/10.1080/14640748108400805 The MRC psycholinguistic database . The Quarterly Journal of Experimental Psychology Section A, 33(4):497--505
Show all 36 references
-
[9]
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. http://arxiv.org/abs/2401.08281 The faiss library
2024 arXiv
-
[10]
Nick Ellis. 1991. https://doi.org/https://doi.org/10.1016/S0166-4115(08)60521-X Chapter 21 word meaning and the links between the verbal system and modalities of perception and imagery or in verbal memory the eyes see vividly, but ears only faintly hear, fingers barely feel an...
1991 doi
-
[11]
W. N. Francis and H. Kucera. 1979. http://icame.uib.no/brown/bcm.html Brown corpus manual . Technical report, Department of Linguistics, Brown University, Providence, Rhode Island, US
1979
-
[12]
Diego Frassinelli, Daniela Naumann, Jason Utt, and Sabine Schulte m Walde. 2017. Contextual characteristics of concrete and abstract words. In Proceedings of the 12th International Conference on Computational Semantics
2017
-
[13]
Ken J Gilhooly and Robert H Logie. 1980. Age-of-acquisition, imagery, concreteness, familiarity, and ambiguity measures for 1,944 words. Behavior research methods & instrumentation, 12(4):395--427
1980
-
[14]
Jack Hessel, David Mimno, and Lillian Lee. 2018. https://doi.org/10.18653/v1/N18-1199 Quantifying the visual concreteness of words and topics in multimodal datasets . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Ling...
2018 doi
-
[15]
Kastner, Ichiro Ide, Frank Nack, Yasutomo Kawanishi, Takatsugu Hirayama, Daisuke Deguchi, and Hiroshi Murase
Marc A. Kastner, Ichiro Ide, Frank Nack, Yasutomo Kawanishi, Takatsugu Hirayama, Daisuke Deguchi, and Hiroshi Murase. 2020. Estimating the imageability of words by mining visual characteristics from crawled image data. Multimedia Tools and Applications, 79:18167--18199
2020
-
[16]
Alessandro Lenci, Gianluca E Lebani, and Lucia C Passaro. 2018. The emotions of abstract words: A distributional semantic analysis. Topics in Cognitive Science, 10(3):550--572
2018
-
[17]
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. http://arxiv.org/abs/2308.03281 Towards general text embeddings with multi-stage contrastive learning
2023 arXiv
-
[18]
Lawrence Zitnick, and Piotr Dollár
Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. 2015. http://arxiv.org/abs/1405.0312 Microsoft coco: Common objects in context
2015 arXiv
-
[19]
Aaron Broadwell, Samira Shaikh, Tomek Strzalkowski, John Lien, Sarah Taylor, Laurie Feldman, Boris Yamrom, Nick Webb, Umit Boz, Ignacio Cases, and Ching-sheng Lin
Ting Liu, Kit Cho, G. Aaron Broadwell, Samira Shaikh, Tomek Strzalkowski, John Lien, Sarah Taylor, Laurie Feldman, Boris Yamrom, Nick Webb, Umit Boz, Ignacio Cases, and Ching-sheng Lin. 2014. http://www.lrec-conf.org/proceedings/lrec2014/pdf/178_Paper.pdf Automatic expansion o...
2014
-
[20]
Gonzalo Mart \'i nez, Juan Diego Molero, Sandra Gonz \'a lez, Javier Conde, Marc Brysbaert, and Pedro Reviriego. 2024. Using large language models to estimate features of multi-word expressions: Concreteness, valence, arousal. Behavior Research Methods, 57(1):5
2024
-
[21]
Daniela Naumann, Diego Frassinelli, and Sabine Schulte im Walde. 2018. https://aclanthology.org/S18-2008/ Quantitative semantic variation in the contexts of concrete and abstract words . In Proceedings of the 7th Joint Conference on Lexical and Computational Semantics, pages 7...
2018
-
[22]
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO : A human generated machine reading comprehension dataset
2016
-
[23]
Allan Paivio, John C Yuille, and Stephen A Madigan. 1968. Concreteness, imagery, and meaningfulness values for 925 nouns. Journal of experimental psychology, 76(1p2):1
1968
-
[24]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings...
2021
-
[25]
John T. E. Richardson. 1976. Imageability and concreteness. Bulletin of the psychonomic society, 7:429--431
1976
-
[26]
Jocelyn Schock, Michael J Cortese, and Maya M Khanna. 2012. Imageability estimates for 3,000 disyllabic words. Behavior Research Methods, 44:374--379
2012
-
[27]
Sabine Schulte im Walde and Diego Frassinelli. 2022. Distributional measures of semantic abstraction. Frontiers in Artificial Intelligence, 4
2022
-
[28]
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. https://doi.org/10.18653/v1/P18-1238 Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning . In Proceedings of the 56th Annual Meeting of the Association for Computa...
2018 doi
-
[29]
Tarun Tater, Sabine Schulte Im Walde, and Diego Frassinelli. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.1203 Unveiling the mystery of visual attributes of concrete and abstract concepts: Variability, nearest neighbors, and challenging categories . In Proceedings of the ...
2024 doi
-
[30]
Nandan Thakur, Nils Reimers, Andreas R \"u ckl \'e , Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR : A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Proceedings of the 35th Conference on Neural Information Processing Systems Dataset...
2021
-
[31]
Michael P Toglia and William F Battig. 1978. Handbook of semantic word norms. Lawrence Erlbaum
1978
-
[32]
Laurens van der Maaten and Geoffrey Hinton. 2008. http://jmlr.org/papers/v9/vandermaaten08a.html Visualizing data using t-sne . Journal of Machine Learning Research, 9(86):2579--2605
2008
-
[33]
Michael Wilson. 1988. MRC psycholinguistic database: Machine-usable dictionary, version 2.00. Behavior Research Methods, Instruments, & Computers, 20:6--10
1988
-
[34]
Si Wu and David Smith. 2023. https://doi.org/10.18653/v1/2023.wnu-1.16 Composition and deformance: Measuring imageability with a text-to-image model . In Proceedings of the 5th Workshop on Narrative Understanding, pages 106--117, Toronto, Canada. Association for Computational ...
2023 doi
-
[35]
Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li, and Min Zhang. 2024. http://arxiv.org/abs/2407.19669 mGTE : Generalized long-context text representation and reranking models for ...
2024 arXiv
-
[36]
Qingsheng Zhu, Ji Feng, and Jinlong Huang. 2016. Natural neighbor: A self-adaptive neighborhood method without parameter k. Pattern Recognition Letters, 80:30--36
2016
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.