{"id":"b6f352ea-a2ab-4c6b-a2f5-13f510f5464e","arxiv_id":"2505.23029","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Words whose caption-embedding neighborhoods are stable under nearest-neighbor agreement are rated by humans as more imageable and concrete, and this stability score outperforms prior unsupervised predictors.","lead":"A new text-only measure, the Neighborhood Stability Measure, estimates how imageable or concrete a word is by checking how tightly packed its neighbors are in an embedding space of image captions. It beats existing unsupervised baselines while avoiding expensive image generation or paired image data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"NSM's apparent signal may be largely a frequency/contextual-diversity confound: the paper never partials out or matches on frequency, so the claim that embedding geometry specifically encodes visual-semantic properties remains underdetermined.","rationale":"The reader's conditional verdict already centers on the same load-bearing concern: NSM measures geometric concentration, but the paper does not separate visual-semantic properties from frequency, lexical fixedness, or register. I agree with that identification. The empirical comparison is broad (three caption datasets, three embedding models, two rating datasets), and the reported advantage over FREQ makes it unlikely that NSM is simply raw frequency; nevertheless, a nonlinear local-density proxy for frequency or contextual diversity is plausible and untested. The self-inclusion ambiguity in Algorithm 1 is a secondary technical worry that should be settled by code inspection, but the main gap is the missing confound control. A frequency-matched or partial-correlation analysis would directly test the causal interpretation of Hypothesis 1. If it fails, the paper's contribution reduces to a predictive heuristic with an unsupported psycholinguistic explanation, so the manuscript should not be accepted as-is; the existing CONDITIONAL verdict is the appropriate outcome until the control is supplied.","tokens_in":13457,"tokens_out":8844,"duration_ms":102431,"concrete_test":"Run a frequency-controlled evaluation: compute log-frequency of each word in CC12M captions, then (i) stratify the MRC imageability and Brysbaert concreteness test words into frequency deciles and compute the Spearman correlation between NSM and ratings within each decile, and (ii) compute a partial Spearman correlation between NSM and ratings controlling for log-frequency (and, if feasible, contextual diversity and POS). Also inspect the released code to confirm whether q is excluded from X when computing Nk(q) and N1(u). If the within-decile or partial correlations collapse to near zero, NSM's signal is a frequency artifact; if a substantial positive correlation remains across strata, the confound concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise in Section 4.1 is that the peakedness of a word's neighborhood in embedding space is specifically governed by visual-semantic properties. NSM is computed from local geometry: it counts, among the k nearest neighbors of q, the fraction whose own nearest neighbor lies in the same set. Local geometric density in a token/caption embedding collection is strongly influenced by how many times a word occurs and by how varied its contexts are, and these lexical statistics are themselves correlated with imageability and concreteness ratings. Table 1 reports FREQ as a baseline, but the evaluation never controls for frequency: it only shows that raw NSM correlates more strongly than raw frequency. It does not test whether NSM retains its correlation after matching or partialling out frequency, POS, or contextual diversity. The self-inclusion ambiguity in Algorithm 1 (no exclusion of q from X) is a concrete pathway for this confound: if q is in the collection, q itself is in Nk(q), and any point whose nearest neighbor is q counts as stable, turning NSM into a near-duplicate/frequency detector. The paper's own Discussion concedes that 'why NSM performs better on image-caption datasets' is unanswered, which is exactly where the visual-semantic interpretation needs defense. Until a frequency-matched analysis is supplied, the central claim that the geometry encodes visual-semantic properties rather than generic lexical statistics is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the Neighborhood Stability Measure (NSM), an unsupervised, distribution-free statistic computed from text-only caption embeddings, intended to quantify the 'sharpness' of the neighborhood of a word in semantic space. The authors hypothesize that this sharpness correlates with psycholinguistic imageability and concreteness ratings. They evaluate NSM on COCO, CC3M, and CC12M captions embedded with three sentence transformer models, comparing against frequency counts, the visual-concreteness method of Hessel et al. (2018), and the text-to-image method of Wu and Smith (2023). The main results report higher Spearman correlations with MRC imageability and Brysbaert et al. concreteness than these baselines, as well as strong AUC in threshold-based classification. The paper also reports degraded performance on non-caption text collections and an analysis of the effect of radius k.","tokens_in":13778,"tokens_out":3641,"duration_ms":40833,"significance":"If the result holds, the paper offers a cheap, single-modal, unsupervised estimator for two psycholinguistic variables, with full vocabulary coverage and no need for generative image sampling. The proposed measure is simple, interpretable, and built on standard ANN machinery; the authors provide code and data. The empirical claim that text-embedding geometry alone captures imageability/concreteness is interesting and would be a useful contribution to computational psycholinguistics. However, the current manuscript contains a formal degeneracy in the definition of NSM and leaves the main mechanistic claim underdetermined by not controlling for frequency or contextual diversity.","major_comments":[{"comment":"As written, NSM is identically 1 for every query, regardless of the data. In Definition 1, N1(u) is defined as the nearest neighbor of u in X (Equation 1). Since u itself belongs to X and δ(u,u) is maximal for any reasonable similarity (e.g., cosine similarity), the nearest neighbor of u is u itself. Therefore, for every u in Nk(q), the condition N1(u) ∈ Nk(q) holds trivially, making α = 1. The paper never states that the point itself is excluded from the nearest-neighbor search. This is a load-bearing issue: the reported correlations in Table 1 and the AUC results in Figures 3 and 4 cannot be reproduced from the stated algorithm. The fix is straightforward—define N1(u) as the nearest neighbor in X \\ {u}, or explicitly use the second nearest neighbor—but it must be stated and ideally verified in the revised manuscript.","section":"Section 4.2, Definition 1 and Algorithm 1"},{"comment":"The manuscript never controls for word frequency or contextual diversity, so the central claim that embedding geometry specifically encodes visual-semantic properties is underdetermined. The FREQ baselines in Table 1 already show moderate correlations (up to 0.34 and 0.35), and NSM is a local geometric quantity computed from the same caption collections; it may largely be a proxy for frequency or contextual diversity, which are themselves correlated with imageability and concreteness. To support Hypothesis 1, the authors should report partial Spearman correlations controlling for log frequency, or match words across frequency bins and show NSM retains its discriminative power. The paper's own Discussion concedes that 'why NSM performs better on image-caption datasets' is left unanswered; this is exactly the point where the visual-semantic interpretation needs defense.","section":"Section 5.1 and Table 1; Section 6"},{"comment":"The comparison with HML is not apples-to-apples. HML's correlation is computed only on the 12.9% of imageability words and 2.7% of concreteness words for which it can produce a score, whereas NSM and the other baselines are evaluated on the full rating vocabularies. Because Spearman's correlation is sensitive to the set of points, the reported superiority of NSM over HML could be partly an artifact of different evaluation subsets. The authors should either report HML on a common subset of words or explicitly discuss how the coverage disparity affects the comparison.","section":"Table 1 and Table 2"}],"minor_comments":[{"comment":"There are missing spaces in the typeset text, e.g., 'NSMcorrelates' and 'NSMis', which should be corrected.","section":"Abstract"},{"comment":"The algorithm should specify that k must be smaller than the collection size |X|, since otherwise Nk(q) contains all points and the stability value is trivially defined.","section":"Section 4.2, Algorithm 1"},{"comment":"The Algorithm 1 input line says 'embedding collection X ∈ R^d' but X is a set of vectors in R^d; use X ⊂ R^d consistently.","section":"Section 4.2, input notation"},{"comment":"The tSNE visualization is illustrative only; its conclusions depend on tSNE hyperparameters and projection randomness. This is fine, but the text should not overstate what the figure demonstrates.","section":"Section 4.1, Figure 1"},{"comment":"The phrase 'this results in a near exact nearest neighbor search' is informal; specify the approximation quality or report the recall of the ANN index on a sample.","section":"Section 5.1, Methods"}],"recommendation":"major_revision","confidential_remarks":"The self-inclusion degeneracy is the most serious issue and must be fixed before publication. The frequency-confound concern is also substantive and requires additional experiments or a carefully argued partial-correlation analysis. The paper is otherwise clearly written and the empirical framework is solid, so I believe a major revision can address these points."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"NSM is a genuinely new and simple measure, and the empirical showing is better than the baselines by a clear margin. The paper deserves a serious referee, but not without two fixes: state the self-exclusion rule in Algorithm 1, and add a frequency- or diversity-matched analysis.\n\nThe core idea is crisp: for a word, embed its contexts, look at the k nearest neighbors, and count how many of those have their own nearest neighbor inside that set. That extends Zhu et al.'s natural neighbors from pointwise to set-level stability, and applying it to imageability/concreteness is new. The experiments are broad: three caption datasets, three embedding models, and sensible baselines including HML and AVGCLIP. The coverage table is a nice touch, showing HML only covers 2.7% of words.\n\nThe soft spots are real but fixable. Algorithm 1 as written never says whether the query q is excluded from the nearest-neighbor search. If q is in the collection, q is its own nearest neighbor, and many other points might have q as nearest neighbor, which could make NSM artificially high. The paper needs to state the exclusion rule.\n\nMore substantively, the paper reports FREQ as a baseline but never partials it out. Since FREQ correlates 0.26-0.35 with the ratings, and NSM correlates 0.58-0.66, it's possible that NSM is largely a frequency or contextual-diversity detector. The paper's hypothesis is about peakedness, but the evaluation doesn't discriminate between geometric sharpness and plain lexical statistics. The Discussion concedes that 'why NSM performs better on image-caption datasets' is unanswered; that is exactly where the mechanism needs defense. A matched-pairs analysis or partial correlation would settle it.\n\nMinor: Table 1 has no error bars. The 10-trial averaging is good, but reporting variance or significance would help.\n\nI recommend engaging with this paper. It's a solid empirical contribution with reproducible code. The conceptual claim is underdetermined, but the method works, and the authors are honest about the limits. A referee should ask for the self-exclusion clarification and a frequency-controlled experiment. With those, it's a clean paper.\n\nFor reading group, I'd say maybe—the confound discussion is worth a conversation. I'd cite it if I needed a cheap imageability estimator.","headline":"A simple, cheap text-only imageability/concreteness estimator that clearly beats baselines, but the paper needs to fix a self-inclusion ambiguity and rule out frequency as the real driver.","tokens_in":14243,"tokens_out":3907,"would_cite":true,"duration_ms":39594,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper aims to establish that imageability and concreteness can be read off the geometry of text embeddings alone, using the peakedness of a word's nearest-neighbor neighborhood.","keywords":["imageability","concreteness","text embeddings","neighborhood stability","unsupervised estimation","psycholinguistic norms","image captions","nearest neighbor geometry"],"falsifier":"Compute NSM on a caption collection after independently permuting the coordinates of each word's embedding so that cosine-similarity neighborhoods are destroyed while token frequency and vector norms are preserved; if the scrambled NSM still correlates with human imageability and concreteness ratings, the reported signal is not neighborhood geometry. A complementary check is to compare NSM within frequency-matched word bins: if the correlation disappears, frequency rather than visual semantics drives the effect.","tokens_in":13270,"feed_emoji":"🧠","tokens_out":10875,"duration_ms":95111,"temperature":0.7,"pith_summary":"The paper seeks to show that two psycholinguistic properties—imageability, how readily a word evokes a mental image, and concreteness, how perceptible its referent is—can be estimated from text alone, without images or multimodal models. Its proposed Neighborhood Stability Measure (NSM) computes, for each word, the fraction of its nearest embedding neighbors whose own nearest neighbor stays inside the same neighbor set; a self-contained, sharply peaked neighborhood is taken to indicate a more imageable or concrete word. Across three caption corpora and three embedding models, NSM shows higher rank correlation with human imageability and concreteness ratings than the unsupervised baselines tested, while covering every word in the rating lexicons. If the claim holds, psycholinguistic norms could be expanded cheaply and at scale from existing caption collections.","feed_headline":"Word neighborhoods predict imageability and concreteness","feed_subtitle":"Text-only measure beats image-based baselines and covers the full rating vocabulary.","key_machinery":"The central object is the Neighborhood Stability Measure (NSM), a distribution-free statistic over a collection of text embeddings. For a query word $q$ and radius $k$, it takes the $k$ approximate nearest neighbors $N_k(q)$ under cosine similarity and returns the fraction of those neighbors $u$ whose own nearest neighbor $N_1(u)$ is also in $N_k(q)$. That fraction is the $\\alpha$-stability of the neighborhood (Definition 1), and the paper interprets a value near 1 as a sharp peak in the underlying distribution and a value near 0 as a flat, diffuse region. The machinery rests on approximate nearest neighbor search to make the computation scalable, on a precomputed nearest-neighbor table to reduce each query to $O(T)$, and on the natural-neighbor concept of mutual nearest neighbors, extended from pairs to sets.","core_discovery":"On the authors' own terms, the discovery is that the visual-semantic character of a word leaves a geometric trace in its contexts, and that a distribution-free statistic can read that trace. Hypothesis 1 states that the distribution of contexts around an imageable or concrete word forms a sharper peak in semantic space. The authors operationalize peakedness through $\\alpha$-stability: a neighborhood is $\\alpha$-stable when a proportion $\\alpha$ of its points have their nearest neighbor inside that neighborhood, and the Neighborhood Stability Measure (NSM) of a query word is the $\\alpha$ of its $k$-nearest-neighbor set. The empirical claim is that this single number correlates more strongly with ground-truth ratings than existing unsupervised methods, often by a wide margin, and that it acts as a strong predictor in imageable/non-imageable and concrete/abstract classification, with reliability increasing for extreme NSM values.","pith_inferences":["Because NSM is defined for any query vector, the same statistic could be applied to phrases and sentences; the authors state they have not tested this, and a human-rating study for sentence-level imageability would be the direct test.","The paper does not control for word frequency or register when attributing neighborhood sharpness to visual semantics; a frequency-matched or residual analysis would show whether NSM adds signal beyond the caption-frequency baselines it reports.","NSM could double as a diagnostic for embedding spaces: a model whose neighborhoods do not sharpen for concrete words may have anisotropic or poorly calibrated geometry, independent of the psycholinguistic prediction task.","The caption-corpus requirement suggests a path to cross-lingual norms only where comparable visual-descriptive text exists, so languages without large caption collections would need translated or synthetically generated captions."],"forward_implications":["NSM gives 100% coverage of the rating lexicons: any word that can be embedded receives a score, removing the vocabulary-mismatch bottleneck that leaves the HML baseline able to predict only about 13% of imageability words and 2.7% of concreteness words.","The estimate is reusable: an embedding collection and its nearest-neighbor index are built once, and scoring a new word then costs one neighborhood query plus table lookups, in contrast to the roughly 120 GPU-hours reported for the image-generation baseline.","The signal depends on the text domain: moving from caption collections to general text collections such as MS MARCO or Quora drops NSM's correlations substantially, indicating that the visual-semantic property is encoded most strongly in visual-descriptive regions of semantic space.","Embedding dimensionality and collection size matter: smaller 384-dimensional embeddings and larger caption collections give stronger correlations, consistent with distance concentration in high dimensions.","Extreme NSM values carry more meaning: when words with mid-range scores are excluded, classification AUC rises, so the measure can be used as a confidence-graded predictor."],"supporting_citations":[{"why":"Provides the MRC Psycholinguistics Database imageability ratings for 4,848 words; this is the ground-truth axis for the imageability evaluation.","marker":"Coltheart (1981)"},{"why":"Provides concreteness ratings for 37,058 words; the ground-truth axis for the concreteness evaluation.","marker":"Brysbaert et al. (2014)"},{"why":"Defines the HML baseline measuring how clustered a word's associated images are; its vocabulary-mismatch limitation motivates NSM's embedding-space approach.","marker":"Hessel et al. (2018)"},{"why":"Supplies the AVGCLIP and COSINESIM text-to-image baselines for imageability that NSM is compared against and critiques for computational cost.","marker":"Wu and Smith (2023)"},{"why":"Supplies the approximate nearest neighbor foundations and the curse-of-dimensionality account used to explain why lower-dimensional embeddings give stronger NSM correlations.","marker":"Bruch (2024)"},{"why":"Introduces natural neighbors (mutual nearest neighbors), which Definition 1 extends from point pairs to neighborhood-level alpha-stability.","marker":"Zhu et al. (2016)"},{"why":"Provides the MS COCO images whose captions form one of the three caption collections used to build embedding collections.","marker":"Lin et al. (2015)"},{"why":"Provides the Microsoft COCO captions dataset, the MS COCO text portion used to construct a vector collection.","marker":"Chen et al. (2015)"},{"why":"Provides the Conceptual Captions 3M corpus of web image alt-text captions used as a second embedding collection.","marker":"Sharma et al. (2018)"},{"why":"Provides Conceptual Captions 12M, the larger caption collection that yields NSM's strongest correlations.","marker":"Changpinyo et al. (2021)"}],"fun_headline_variants":["Text geometry predicts imageability","Neighborhood stability gauges concreteness","Unsupervised metric rivals visual baselines","Semantic peaks reveal visual semantics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that imageability and concreteness appear as the geometric concentration of a word's contexts in embedding space, and that this concentration is not confounded by word frequency, register, or lexical fixedness.","fun_headline_variants_meta":{"raw":{"variants":["Text geometry predicts imageability","Neighborhood stability gauges concreteness","Unsupervised metric rivals visual baselines","Semantic peaks reveal visual semantics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000133,"raw_usage":{"total_tokens":1115,"prompt_tokens":901,"completion_tokens":214,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":164}},"tokens_in":517,"tokens_out":214,"duration_ms":3159,"temperature":1.0,"reasoning_tokens":164,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:55:26.270261+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute NSM on a caption collection after independently permuting the coordinates of each word's embedding so that cosine-similarity neighborhoods are destroyed while token frequency and vector norms are preserved; if the scrambled NSM still correlates with human imageability and concreteness ratings, the reported signal is not neighborhood geometry. A complementary check is to compare NSM within frequency-matched word bins: if the correlation disappears, frequency rather than visual semantics drives the effect.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MRC Psycholinguistics Database imageability ratings for 4,848 words; this is the ground-truth axis for the imageability evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides concreteness ratings for 37,058 words; the ground-truth axis for the concreteness evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the approximate nearest neighbor foundations and the curse-of-dimensionality account used to explain why lower-dimensional embeddings give stronger NSM correlations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces natural neighbors (mutual nearest neighbors), which Definition 1 extends from point pairs to neighborhood-level alpha-stability."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides Conceptual Captions 12M, the larger caption collection that yields NSM's strongest correlations."}],"review_version":1}