{"id":"a2a74a92-b5a3-4622-bbb5-b71fc111adb7","arxiv_id":"2412.10369","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new image-based measure of word meaning shows that across 30 languages, word classes form a consistent groundedness cline, with nouns most grounded and function words least, but still nonzero.","lead":"The paper introduces a new measure called groundedness, which estimates how much a word's occurrence in a sentence is predicted by the image the sentence describes, computed as a surprisal difference between an image captioning model and a text-only language model. Applied to parts of speech across 30 languages, it finds a consistent hierarchy (nouns most grounded, then adjectives, then verbs) and argues that even grammatical words carry measurable semantic content.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The two models in Eq. 1 are not matched in the required sense; the LM is fine-tuned from a vision-language pretrained model without image tokens, so functional-word groundedness may be a model artifact.","rationale":"The paper's central claim is that groundedness (Eq. 1) estimates PMI between a word and the image. This requires the captioning model and LM to be matched except for the image input. The reader's weakest assumption points to this, and my stress-test agrees: Section 4's matching procedure is not sufficient. The LM is initialized from PaliGemma pretrained, which has seen images; fine-tuning on captions only for three epochs may not erase image-conditioned priors. Moreover, the architectural difference (vision encoder plus 256 image tokens) means the input distributions differ. This can bias PMI for functional words, which are abundant and have low surprisal; a small systematic difference in log-probability could produce significant positive groundedness. The paper's significance test (Section 5.1) tests the mean of the computed PMIs, not the unbiasedness of the underlying estimate, so it cannot rule out this artifact. I propose a concrete control: compute the captioning model's image-marginal by averaging over many images and compare to the LM. If the log-ratio is nonzero for functional classes, the model-mismatch concern lands. This test is feasible with the released model and dataset. The paper is otherwise careful and the ranking result is plausible; the concern is empirical and addressable, so I maintain the CONDITIONAL verdict.","tokens_in":58055,"tokens_out":8528,"duration_ms":76710,"concrete_test":"Estimate the marginal distribution of the captioning model by averaging its next-word probabilities over a sample of images: p_phi(w_t | w_<t) = (1/N) * sum_i p_phi(w_t | image_i, w_<t), using held-out images from the same evaluation dataset. Then compute the mean log-ratio log p_phi(w_t | w_<t) - log p_theta(w_t | w_<t) for function-word classes (determiners, adpositions, auxiliaries, conjunctions, particles) across languages and datasets. If this mean is significantly above zero, the LM under-estimates function-word probabilities relative to the captioning model's image-marginal, and the positive groundedness of functional classes in Section 5.2 is at least partly a model-mismatch artifact rather than image-word PMI.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that groundedness (Eq. 1) estimates pointwise mutual information between a word and the image depends on the captioning model p_phi and the language model p_theta being matched except for the image input, so that the log-ratio is an unbiased estimate of PMI. Section 4 describes a careful attempt to match the models: the LM is initialized from the pretrained PaliGemma decoder and fine-tuned on COCO-35L captions only, while the captioning model is the COCO-35L fine-tuned PaliGemma. But the two models are not matched in the required sense. First, PaliGemma's pretraining (paligemma-3b-pt-224) includes large-scale vision-language training (WebLI, CC3M-35L), so the LM's initial weights are not those of a text-only model; fine-tuning on captions for three epochs (Appendix A.3) may not remove the inductive bias from seeing image tokens, and the LM never sees image tokens at all. Second, the captioning model includes a vision encoder and 256 image tokens prepended to the text, which changes the input distribution and attention patterns relative to the LM. These architectural and training differences can produce systematic log-probability differences for particular word classes even when the image contributes no information about them. In particular, if p_theta under-estimates the probability of function words in caption-style text, then the groundedness of function words will be positive even under a null hypothesis of zero image information. This directly threatens the paper's second headline claim that functional classes show significantly positive groundedness (Sections 5.1 and 5.2). The paper does not provide a control condition that would rule out such a model artifact. The permutation test in Section 5.1 only tests whether the computed PMIs have nonzero mean; it cannot detect a systematic bias in the model pair.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes 'groundedness,' an empirical measure of a word token's semantic contentfulness defined as the log-probability ratio between an image-conditioned captioning model and a text-only language model (Eq. 1), interpreted as pointwise mutual information between the word and the image. The measure is applied to Universal Dependencies part-of-speech classes across 30 languages using three multilingual image-caption datasets. The authors report that lexical classes are consistently more grounded than functional classes, that the ranking of classes is nearly total (proper nouns, nouns, adjectives, verbs above particles, auxiliaries, conjunctions, determiners, adpositions), and that functional classes show significantly positive groundedness, which they interpret as contradicting the view that functional classes do not convey content. They also report weak-to-moderate correlations with English psycholinguistic concreteness norms and release the groundedness scores.","tokens_in":58281,"tokens_out":4178,"duration_ms":42620,"significance":"If the measure is valid, the paper offers a novel, language-agnostic, corpus-based route to quantifying semantic contentfulness and to testing typological claims about word classes. The strength of the paper is its careful experimental scaffolding: evaluation on three datasets with complementary properties, an explicit attempt to match the training data of the two models, permutation tests with multiple-testing correction, ANOVA variance decomposition, and a public release of the scores. The measure itself is parameter-free: the ranking and class differences are outputs rather than fitted inputs, and the only post-hoc component (the uncertainty-coefficient normalization in Section 5.4) is clearly labeled. However, the central interpretation depends on the two neural models being matched except for the image input, and that premise is not validated with a control condition. The paper's headline claims about functional classes therefore require additional evidence.","major_comments":[{"comment":"The validity of groundedness as an estimate of PMI with the image depends on p_phi and p_theta being matched in every respect except the image input, but this is not the case. The language model is initialized from paligemma-3b-pt-224, a checkpoint that was pretrained on large-scale vision-language data including image tokens, then fine-tuned on COCO-35L captions only. The captioning model uses the same pretrained decoder plus a vision encoder and 256 prepended image tokens. These models therefore differ in architecture, input distribution, and pretraining exposure, not only in the presence of image information. If p_theta systematically underestimates function words in caption-style text, the log-ratio in Eq. (1) will be positive for functional classes even under a null hypothesis of zero image contribution. This directly threatens the Section 5.2 claim that functional classes 'do carry semantic content.' A control condition is needed: for example, computing groundedness with mismatched images (e.g., shuffled image-caption pairs) or comparing against a text-only pretrained LM fine-tuned on the same captions. Without such a check, the PMI interpretation of the reported positive values is not established.","section":"Section 4, Eq. (1)"},{"comment":"The one-sample permutation test randomly flips the signs of observed PMI values and compares the observed mean against the resulting null distribution. This procedure tests the null hypothesis that the PMI distribution is symmetric about zero, not the stated null hypothesis that the mean mutual information is zero. As Figure 3 shows, PMI distributions by part of speech are skewed and heavy-tailed, so a class with a skewed distribution but zero mean could be declared significantly grounded by this test. The authors should use a bootstrap confidence interval for the mean or a one-sample t-test on the average, or explicitly justify why sign-flipping is valid for their null of MI = 0. Since sample sizes are very large, the qualitative conclusions about the ranking may survive, but the reported p-values do not support the stated null as written.","section":"Section 5.1, Figure 2"},{"comment":"The claim of a 'near total ranking' is based on pairwise significance tests over very large token counts, where even tiny mean differences can become significant. The estimated marginal means and pairwise significance do not convey the magnitude or overlap of the class-level distributions. For example, the separation between adpositions and other functional classes, which underlies the claim that adpositions are not semi-lexical, should be reported with effect sizes or confidence intervals for the EMM differences. Without such information, the linguistic interpretation of the ranking (e.g., 'nouns > adjectives > verbs' as a substantive cline) is stronger than the displayed evidence supports.","section":"Section 5.2, Figure 3"}],"minor_comments":[{"comment":"There is a typo in 'pyscho- and neurolinguistics' (should be 'psycho-'), and 'word classs' appears in Section 5.1.","section":"Introduction"},{"comment":"The text uses 'ANOV A' instead of 'ANOVA' in the sentence describing the variance decomposition.","section":"Section 5.3"},{"comment":"The expectation notation 'Ep(Ci, m, w<t)' is not defined; clarifying that the average in Eq. (4) is the Monte Carlo estimator would help readers connect the formal definition to the implementation.","section":"Section 3, Eq. (3)"},{"comment":"The training details state 430,000 steps with a batch size of 4, which is about three epochs over COCO-35L; the relationship between steps, epochs, and dataset size should be made explicit to allow reproducibility.","section":"Appendix A.3"},{"comment":"The uncertainty coefficient is introduced as 'the average ratio between LM surprisal and captioning model surprisal,' but the standard definition is a proportional reduction in surprisal; the text should state the formula to avoid ambiguity.","section":"Section 5.4"}],"recommendation":"major_revision","confidential_remarks":"The paper presents an interesting and potentially useful measure, and the dataset release is a valuable contribution. The central concern is the model-matching assumption behind the PMI interpretation: the language model is initialized from a vision-language pretrained checkpoint, so the log-ratio in Eq. (1) may reflect architectural and pretraining differences rather than image information alone. This is addressable with a control condition (e.g., mismatched images) or by substantially weakening the PMI claims. I recommend asking the authors for such evidence before publication, rather than rejecting outright."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a new measure applied at a scale that makes it worth engaging, but the headline claim about functional classes is not yet supported because the two models in Eq. 1 are not shown to be matched in the way the PMI interpretation requires.\n\nWhat is actually new: groundedness, defined as the surprisal difference between an image-conditioned captioning model and a language model fine-tuned on the same captions. This is a sensible, language-agnostic operationalization of contentfulness that avoids subjective ratings. The experimental design is thoughtful: three datasets with different caption properties, matched training data, permutation tests, ANOVA variance decomposition, and a released dataset of groundedness scores for 30 languages. The ANOVA results are credible and the correlation with concreteness norms, while moderate, is a useful sanity check. The paper is generally well written and the limitations section is honest about images, language coverage, and POS tagging noise.\n\nThe soft spot is load-bearing. The claim that groundedness estimates pointwise mutual information with the image depends on the captioning model and the language model differing only in image input. They do not. The LM starts from paligemma-3b-pt-224, a vision-language pretrained decoder, and is fine-tuned on captions only; the captioning model also has a vision encoder and 256 image tokens prepended. That is not a clean matched pair. The paper has no control condition, such as pairing captions with random images to measure the baseline nonzero groundedness. The permutation test only asks whether the mean PMI is greater than zero; it cannot detect a systematic bias coming from model mismatch. So the second headline claim, that functional classes show significantly positive groundedness, is over-stated.\n\nThis does not sink the paper. The lexical-functional cline is consistent across three datasets, and the noun > adjective > verb ranking is plausible. But the \"functional classes carry content\" conclusion needs a proper control before being asserted. The limitations section should have flagged this.\n\nWho this is for: typologists and computational linguists who want a quantitative, reusable measure of contentfulness, and anyone building multilingual multimodal resources. I would send it to serious peer review and ask for a control condition and a softened conclusion on functional classes. With that, it would be a solid contribution.","headline":"A genuinely new measurement idea and a valuable 30-language dataset, but the central PMI interpretation rests on an unvalidated model-matching assumption, so the functional-class positive groundedness claim is provisional.","tokens_in":58915,"tokens_out":2414,"would_cite":true,"duration_ms":26313,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Groundedness measures how much of a word's predictability comes from the image it describes, and across 30 languages it ranks word classes from lexical to functional.","keywords":["groundedness","word classes","parts of speech","linguistic typology","mutual information","surprisal","multimodal language models","image captioning"],"falsifier":"Score the same models on captions paired with mismatched images or with no image; if functional words retain the same positive groundedness in that control, the signal reflects model differences rather than word–image association.","tokens_in":57808,"feed_emoji":"🖼️","tokens_out":8024,"duration_ms":65965,"temperature":0.7,"pith_summary":"The paper proposes 'groundedness', a measure of how much of a word's predictability comes from the image it describes, computed as the difference in surprisal between an image-conditioned captioning model and a matched text-only language model. Applied to word classes in 30 languages, the measure recovers the lexical/functional split: proper nouns, nouns, adjectives, and verbs rank above particles, auxiliaries, conjunctions, determiners, and adpositions, with a nearly total ordering across languages. The paper also finds that functional classes have significantly positive groundedness, which it takes as evidence that grammatical words carry semantic content. This gives typology a quantitative, image-grounded way to study meaning without relying on hand-built semantic categories.","feed_headline":"Image surprisal ranks word classes in 30 languages","feed_subtitle":"Lexical words like nouns top the ranking, but even function words carry image-related meaning.","key_machinery":"The carrying machinery is the paired-model surprisal comparison: a multilingual image captioning model $p_\\phi(w_t \\mid m, w_{<t})$ and a text-only language model $p_\\theta(w_t \\mid w_{<t})$ trained from the same pretrained weights on the same captions. Groundedness is the negative difference of their surprisals, i.e. the pointwise mutual information with the image, and images act as a language-neutral proxy for meaning. Word-class values come from a Monte Carlo average of token-level groundedness over all tokens carrying that part-of-speech tag, with one-sample permutation tests used to ask whether the class-level mutual information is significantly above zero.","core_discovery":"The central claim is that groundedness—defined as $\\log \\frac{p_\\phi(w_t \\mid m, w_{<t})}{p_\\theta(w_t \\mid w_{<t})}$ for a word token $w_t$ in context $w_{<t}$ with image $m$—estimates the pointwise mutual information between that word and the image it describes. Averaging this quantity over all tokens tagged with a given part of speech yields an estimate of the mutual information between that word class and images, and across 30 languages the paper finds a consistent, nearly total ranking of word classes: proper nouns, nouns, adjectives, and verbs above particles, auxiliaries, conjunctions, determiners, and adpositions. The paper also reports that functional classes show significantly positive groundedness, which it reads as evidence that grammatical words are not devoid of semantic content.","pith_inferences":["Using video instead of still images would likely raise verbs' groundedness relative to nouns, since verbs denote temporally extended events; this is a testable consequence of the paper's image-proxy assumption.","If groundedness tracks semantic contentfulness, a word undergoing grammaticalization should show falling groundedness as it moves from lexical to functional use, measurable on diachronic corpora.","The ranking could be validated behaviorally by testing whether groundedness predicts response times in lexical decision or picture-naming beyond standard concreteness norms."],"forward_implications":["Word classes can be treated as a graded lexical-to-functional cline rather than a dichotomy, since estimated marginal means place every part of speech on a continuous groundedness scale.","Grammatical words are not semantically inert: determiners, adpositions, auxiliaries, and particles show significant positive groundedness in most of the 30 languages.","Images provide a language-neutral meaning representation, so groundedness can be compared across typologically diverse languages directly.","The released dataset of per-token and per-class groundedness scores lets other researchers measure groundedness for constructions, morphemes, or finer subclasses without retraining the models."],"supporting_citations":[{"why":"It supplies the multilingual image captioning model whose probability estimates define the image-conditioned surprisal.","marker":"(Beyer et al., 2024)"},{"why":"It supplies the pretrained weights from which both the captioning model and the matched language model are initialized.","marker":"(Gemma, 2024)"},{"why":"It supplies the multilingual image–caption training data and the Crossmodal-3600 evaluation captions used in the experiments.","marker":"(Thapliyal et al., 2022)"},{"why":"It defines the part-of-speech inventory used to aggregate token-level groundedness into class-level mutual information.","marker":"(de Marneffe et al., 2021)"},{"why":"It supplies the automatic part-of-speech tagger that assigns tags to the caption data.","marker":"(Qi et al., 2020)"},{"why":"It provides the English concreteness norms against which the paper validates the semantic content of groundedness.","marker":"(Brysbaert et al., 2014)"},{"why":"It states the semi-lexical view of adpositions that the paper's groundedness ranking contradicts.","marker":"(Corver and Riemsdijk, 2001)"}],"fun_headline_variants":["Images rank nouns top, but function words still carry meaning","Image surprisal reveals universal word class hierarchy","New metric: how much each word type connects to images","30 languages: nouns most image-grounded, function words least"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire measure relies on the captioning model and the text-only language model being identical in everything except the image input, so that a larger word probability in the captioning model can only be attributed to information from the image.","fun_headline_variants_meta":{"raw":{"variants":["Images rank nouns top, but function words still carry meaning","Image surprisal reveals universal word class hierarchy","New metric: how much each word type connects to images","30 languages: nouns most image-grounded, function words least"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000138,"raw_usage":{"total_tokens":1131,"prompt_tokens":897,"completion_tokens":234,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":168}},"tokens_in":513,"tokens_out":234,"duration_ms":3062,"temperature":1.0,"reasoning_tokens":168,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:54:28.228739+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Score the same models on captions paired with mismatched images or with no image; if functional words retain the same positive groundedness in that control, the signal reflects model differences rather than word–image association.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It states the semi-lexical view of adpositions that the paper's groundedness ranking contradicts."}],"review_version":1}