Pith. sign in

REVIEW 1 major objections 5 minor 43 references

VLMs ignore distance when choosing 'this' versus 'that'

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 16:05 UTC pith:FRLI64OA

load-bearing objection New benchmark for multilingual spatial deixis in VLMs; results are directionally right but underpowered the 1 major comments →

arxiv 2607.07251 v1 pith:FRLI64OA submitted 2026-07-08 cs.CL

Evaluation of Multilingual Ability to Use Spatial Deictic Expressions in Vision-Language Models

Kaito Watanabe , Taisei Yamamoto , Tomoki Doi , Hitomi Yanaka This is my paper
classification cs.CL
keywords spatialexpressionsdeicticvlmsabilitymodelsabilitiesappropriate
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper constructs a benchmark to test whether vision-language models (VLMs) can use spatial deictic expressions — words like 'this' and 'that' whose meaning depends on physical distance to the referenced object — in a human-like way across four languages: English, Chinese, Japanese, and Korean. The benchmark adapts the 'memory game' paradigm from linguistics: objects are placed at controlled distances (0.25m, 1.50m, 2.75m) on a desk, and the model must describe each using a demonstrative plus color plus shape. The authors compare the resulting probability distributions of demonstrative choice against human data from a large cross-linguistic study. The central finding is that all tested models fail to reproduce the human pattern: humans shift from proximal ('this') to medial to distal ('that') demonstratives as objects move farther away, but VLMs largely do not. In Japanese and Korean, which have three demonstratives, models almost never use the distal form regardless of distance. The authors also find a modest negative correlation (r = −0.40) between object recognition accuracy and deviation from human demonstrative distributions, suggesting that models with better visual grounding tend to use demonstratives somewhat more like humans.

Core claim

Vision-language models do not adjust their choice of spatial demonstratives ('this' vs. 'that' and their cross-linguistic equivalents) based on the physical distance to the referenced object, unlike humans across all four languages tested. The human-like shift from proximal to medial to distal demonstratives as distance increases is absent or severely muted in every model evaluated. In three-demonstrative languages (Japanese, Korean), models almost entirely avoid the distal form. Better object recognition correlates weakly with more human-like demonstrative use, but the correlation is modest, indicating that visual grounding alone does not close the gap.

What carries the argument

The 'memory game' paradigm adapted for VQA: synthetic images generated in Blender showing a colored shape on a disk placed at one of three controlled distances, with the model prompted to output '[Demonstrative] [Color] [Shape].' Demonstrative probability distributions are computed from model logits and compared to human distributions using Jensen-Shannon distance.

Load-bearing premise

The benchmark assumes that a single fixed prompt, translated from English by an automated tool and manually patched with medial demonstratives, can elicit demonstrative usage from VLMs that is fairly comparable to the human memory game paradigm — but the paper itself acknowledges that it cannot determine whether the observed trends reflect intrinsic model capabilities or artifacts of the prompt's phrasing.

What would settle it

If a VLM, given a more naturalistic or embodied prompt setting (e.g., a first-person scene with conversational context rather than a static image and a rigid three-word template), were shown to shift demonstrative choice with distance in a human-like way, the central claim that VLMs lack distance-sensitive deictic competence would be substantially weakened.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If VLMs cannot properly use deictic expressions, they may fail in real-world assistive or robotic settings where 'this' vs. 'that' carries critical spatial information about which object to act upon.
  • The absence of distance-sensitive demonstrative shifting could serve as a diagnostic signal for whether a VLM possesses genuine spatial understanding versus surface-level pattern matching.
  • The near-total avoidance of distal demonstratives in three-demonstrative languages suggests training data imbalances may distort cross-linguistic competence in measurable, language-specific ways.
  • The modest correlation between recognition accuracy and human-likeness implies that improving visual grounding may partially improve spatial language competence, but a substantial gap would remain.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the deficit stems from a lack of genuine spatial representation rather than linguistic knowledge, VLMs might also fail on other distance-dependent linguistic phenomena — for instance, spatial adverbs like 'near'/'far,' or tense-aspect systems in languages where spatial distance modulates verbal morphology.
  • The fact that some models with strong object recognition still fail at demonstrative selection points to a gap in the language-vision alignment mechanism specifically, not just in visual perception — suggesting that the binding between spatial features in the visual encoder and distance-sensitive lexical choices in the language decoder is where the breakdown occurs.
  • Testing in embodied or first-person settings — where the model occupies a viewpoint within a scene rather than observing a static image — could reveal whether the deficit is specific to the static-image paradigm or reflects a deeper inability to model deictic reference from a speaker's perspective.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. This paper introduces a multilingual benchmark to evaluate whether vision-language models (VLMs) can use spatial deictic expressions (demonstratives like 'this' and 'that') in a human-like manner. The benchmark is based on the 'memory game' paradigm from psycholinguistics (Gudde et al., 2018; Coventry et al., 2023), where a disk with a colored shape is placed at varying distances (0.25 m, 1.50 m, 2.75 m) on a desk. Four VLMs (Gemma 3 4B/12B, Qwen3-VL 8B/32B) are prompted in four languages (English, Japanese, Korean, Chinese) to describe the disk using a fixed template: [Demonstrative] [Color] [Shape]. The authors compute the probability distribution of demonstrative usage from model logits and compare it to human data from Coventry et al. (2023) using Jensen-Shannon distance. The main finding is that VLMs fail to reproduce human-like shifts in demonstrative selection as a function of distance, particularly in Japanese and Korean, where distal demonstratives are almost never used.

Significance. The paper addresses a genuine gap: while spatial reasoning in VLMs has been benchmarked extensively, the use of spatial deixis—expressions whose meaning depends on the speaker's situational context—has not been explicitly studied. The cross-linguistic dimension (languages with two vs. three demonstratives) adds value. The benchmark is constructed independently of the tested models, the human reference data comes from an external source, and the metric (Jensen-Shannon distance) is standard, so there is no circularity concern. The code is publicly available. These are real strengths. However, the significance of the findings is tempered by a structural issue in the experimental design (see Major Comment 1) and very small effective sample sizes, which together make it difficult to attribute the observed failures specifically to a deficit in spatial deixis reasoning.

major comments (1)
  1. §4.2 (Prompt): The prompt never establishes the VLM as the deictic center (speaker/viewer). The instruction reads: 'Analyze the image and identify the shape on the disk. Describe it by filling in the following three-word template exactly: [Demonstrative] [Color] [Shape].' At no point does the prompt tell the model that it should judge proximity from the camera's viewpoint, that the camera position corresponds to the speaker's position, or that 'this' refers to objects near the camera. In the human memory game (§3.1), participants are physically present and inherently know they are the deictic center. The VLM has no such grounding. Without an established deictic center, the model has no reference point from which to judge whether an object is 'near' (proximal) or 'far' (distal). The observed failure to shift demonstratives with distance could therefore reflect a missing perspective-ground
minor comments (5)
  1. §4.2: The prompt was translated by Google Gemini and then manually patched with medial demonstratives for Japanese and Korean. The paper should report the exact final prompts used in each language, ideally in an appendix, to allow reproduction and to rule out translation artifacts.
  2. Table 1: The caption should clarify that these counts are out of 60 and that the probability distributions in Figure 2 are computed only on these valid samples.
  3. Figure 2: The figure is difficult to read in monochrome; the legend (green/orange/light blue) is hard to distinguish. Consider using patterns or distinct labels.
  4. §5.1: The phrase 'a general trend is to avoid distal demonstratives in languages with three demonstratives' could be quantified with a summary statistic across models/languages for clarity.
  5. §7 (Limitations): The authors acknowledge the single-prompt limitation and small dataset size. These are appropriate acknowledgments, but they also underscore the need for the controls suggested in Major Comment 1 before the central claim can be fully supported.

Circularity Check

0 steps flagged

No circularity found: benchmark, human reference, and metrics are all independent of the tested models

full rationale

The paper's central claim—that VLMs fail to reproduce human-like probability distributions of demonstratives as a function of distance—is derived from a chain that is free of circularity. (1) The benchmark images are constructed using Blender with fixed distance settings (0.25m, 1.50m, 2.75m) independent of any model output. (2) The human reference distribution is taken from an external source (Coventry et al., 2023), not produced or fitted by the present authors. (3) The metric (Jensen-Shannon distance, Eq. 1–2) is a standard, parameter-free measure applied between the model's output distribution and the external human distribution; no parameter is fitted to the evaluation data and then 'predicted' back. (4) The probability distribution of demonstratives for each VLM is computed directly from the model's logits via softmax normalization (§4.3), which is a measurement, not a fit. There are no self-citations that serve as load-bearing premises for the central result. The skeptic's concern about the absence of an explicit deictic-center instruction in the prompt is a correctness/validity concern (does the prompt measure what it claims to measure?), not a circularity concern (is the result equivalent to its inputs by construction?). The derivation chain is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

The paper introduces no new mathematical entities, particles, or forces. It is an empirical benchmark study. The axioms are domain assumptions about the validity of the experimental design, not mathematical postulates.

axioms (3)
  • domain assumption Human demonstrative usage data from Coventry et al. (2023), collected in physical rooms with human participants, is a valid ground-truth reference for evaluating VLM behavior on synthetic images.
    The entire evaluation framework depends on comparing VLM outputs to human distributions from Coventry et al. (2023). The paper does not discuss whether the shift from physical to synthetic visual input might change the distance perception or demonstrative selection behavior.
  • domain assumption A single fixed prompt with a three-word template can elicit naturalistic demonstrative usage from VLMs comparable to the human memory game.
    The prompt constrains output to '[Demonstrative] [Color] [Shape]' and the paper acknowledges in §7 that 'the variety of usage of demonstratives in natural situations is not sufficiently reflected.'
  • domain assumption The softmax-normalized logits of demonstrative tokens represent the model's probability distribution of demonstrative use.
    The metric in §4.3 computes P(this)/(P(this)+P(that)) from logits. This assumes that token-level logits faithfully represent the model's pragmatic choice, which may not hold if the model's internal spatial reasoning is not reflected in first-token logits.

pith-pipeline@v1.1.0-glm · 14223 in / 2433 out tokens · 394981 ms · 2026-07-09T16:05:32.287249+00:00 · methodology

0 comments
read the original abstract

One of the expected abilities of vision-language models (VLMs) is spatial reasoning ability based on a given text and image. To evaluate the spatial reasoning abilities of VLMs, we focus on the use of spatial deictic expressions, which are defined as spatial expressions whose referent is determined by their situational context, such as ``this'' and ``that''. To handle spatial deictic expressions, VLMs must jointly reason over language and visual space, grounding context-dependent references in the image's spatial structure. In addition, selecting appropriate spatial deictic expressions across languages requires VLMs to understand the language-specific spatial distinctions encoded by these expressions. In this paper, we develop a benchmark to evaluate the multilingual ability of VLMs to use spatial deictic expressions in four languages. Our experiments using this benchmark reveal that the tested models use demonstratives in a manner different from that of humans, particularly in selecting the appropriate demonstratives based on the distance to the object.

Figures

Figures reproduced from arXiv: 2607.07251 by Hitomi Yanaka, Kaito Watanabe, Taisei Yamamoto, Tomoki Doi.

Figure 1
Figure 1. Figure 1: Examples of images included in the bench [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Probability distributions across distances for each experimental setting. The red horizontal line separates [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 43 canonical work pages · 2 internal anchors

  1. [1]

    Nature human behaviour , volume=

    Spatial communication systems across languages reflect universal action constraints , author=. Nature human behaviour , volume=. 2023 , publisher=

    work page 2023

  2. [2]

    Journal of Visualized Experiments: JoVE , number=

    The (spatial) memory game: testing the relationship between spatial language, object knowledge, and spatial cognition , author=. Journal of Visualized Experiments: JoVE , number=

    work page

  3. [3]

    Blender Foundation, Blender Institute , year=

    Blender - a 3D modelling and rendering package , author=. Blender Foundation, Blender Institute , year=

    work page

  4. [4]

    ArXiv , year=

    Qwen3-VL Technical Report , author=. ArXiv , year=

    work page

  5. [5]

    The Thirteenth International Conference on Learning Representations , year=

    Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference under Ambiguities , author=. The Thirteenth International Conference on Learning Representations , year=

    work page

  6. [6]

    and Manning, Christopher D

    Hudson, Drew A. and Manning, Christopher D. , booktitle=. GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering , year=

    work page

  7. [7]

    2025 , eprint=

    Vision language models are unreliable at trivial spatial cognition , author=. 2025 , eprint=

    work page 2025

  8. [8]

    [The Sanseido Dictionary of Linguistics] Meikai gengogaku jiten

    Yoshio Saito and Yoshihisa Taguchi and Yoshiki Nishimura. [The Sanseido Dictionary of Linguistics] Meikai gengogaku jiten. 2015

    work page 2015

  9. [9]

    Advances in neural information processing systems , volume=

    Attention is all you need , author=. Advances in neural information processing systems , volume=

    work page

  10. [10]

    Proceedings of the IEEE/CVF international conference on computer vision , pages=

    Sigmoid loss for language image pre-training , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=

    work page

  11. [11]

    2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , year=

    A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges , author=. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , year=

    work page 2025

  12. [12]

    arXiv preprint arXiv:2402.12451 , year=

    The revolution of multimodal large language models: a survey , author=. arXiv preprint arXiv:2402.12451 , year=

    work page arXiv

  13. [13]

    ArXiv , year=

    DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs , author=. ArXiv , year=

    work page

  14. [14]

    2025 , eprint=

    Qwen3 Technical Report , author=. 2025 , eprint=

    work page 2025

  15. [15]

    ArXiv , year=

    Finetuned Language Models Are Zero-Shot Learners , author=. ArXiv , year=

    work page

  16. [16]

    Demonstratives : form, function, and grammaticalization

    Diessel, Holger. Demonstratives : form, function, and grammaticalization. 1999

    work page 1999

  17. [17]

    2025 , eprint=

    Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methods , author=. 2025 , eprint=

    work page 2025

  18. [18]

    and Han, Rilyn and Fei-Fei, Li and Xie, Saining , booktitle=

    Yang, Jihan and Yang, Shusheng and Gupta, Anjali W. and Han, Rilyn and Fei-Fei, Li and Xie, Saining , booktitle=. Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces , year=

    work page

  19. [19]

    Transactions of the Association for Computational Linguistics , volume=

    Visual spatial reasoning , author=. Transactions of the Association for Computational Linguistics , volume=. 2023 , publisher=

    work page 2023

  20. [20]

    European conference on computer vision , pages=

    Mmbench: Is your multi-modal model an all-around player? , author=. European conference on computer vision , pages=. 2024 , organization=

    work page 2024

  21. [21]

    2025 , eprint=

    PISA-Bench: The PISA Index as a Multilingual and Multimodal Metric for the Evaluation of Vision-Language Models , author=. 2025 , eprint=

    work page 2025

  22. [22]

    Natural Multimodal Fusion-Based Human–Robot Interaction: Application With Voice and Deictic Posture via Large Language Model , year=

    Lai, Yuzhi and Yuan, Shenghai and Nassar, Youssef and Fan, Mingyu and Gopal, Atmaraaj and Yorita, Arihiro and Kubota, Naoyuki and Rätsch, Matthias , journal=. Natural Multimodal Fusion-Based Human–Robot Interaction: Application With Voice and Deictic Posture via Large Language Model , year=

    work page

  23. [23]

    , journal=

    Han, Chang and Isaacs, Katherine E. , journal=. A Deixis-Centered Approach for Documenting Remote Synchronous Communication Around Data Visualizations , year=

    work page

  24. [24]

    1997 , publisher=

    Lectures on Deixis , author=. 1997 , publisher=

    work page 1997

  25. [25]

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xiong-Hui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Rongyao Fang, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, and 46 others. 2025. https://arxiv.org/abs/2511.21631 Qwen3-vl technical report . ArXiv, abs/2511.21631

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  26. [26]

    Blender Online Community . 2016. Blender - a 3d modelling and rendering package. Blender Foundation, Blender Institute

    work page 2016

  27. [27]

    Kenny R Coventry, Harmen B Gudde, Holger Diessel, Jacqueline Collier, Pedro Guijarro-Fuentes, Mila Vulchanova, Valentin Vulchanov, Emanuela Todisco, Maria Reile, Merlijn Breunesse, and 1 others. 2023. https://www.nature.com/articles/s41562-023-01697-4 Spatial communication systems across languages reflect universal action constraints . Nature human behavi...

    work page 2023

  28. [28]

    Holger Diessel. 1999. Demonstratives : form, function, and grammaticalization. Number v. 42 in Typological studies in language. J. Benjamins

    work page 1999

  29. [29]

    Fillmore

    C.J. Fillmore. 1997. Lectures on Deixis. Center for the Study of Language and Information Publication Lecture Notes. Cambridge University Press

    work page 1997

  30. [30]

    Gemma Team . 2025. https://goo.gle/Gemma3Report Gemma 3

    work page 2025

  31. [31]

    Harmen B Gudde, Debra Griffiths, and Kenny R Coventry. 2018. The (spatial) memory game: testing the relationship between spatial language, object knowledge, and spatial cognition. Journal of Visualized Experiments: JoVE, (132):56495

    work page 2018

  32. [32]

    Patrick Haller, Fabio Barth, Jonas Golde, Georg Rehm, and Alan Akbik. 2025. https://arxiv.org/abs/2510.24792 Pisa-bench: The pisa index as a multilingual and multimodal metric for the evaluation of vision-language models . Preprint, arXiv:2510.24792

    work page arXiv 2025

  33. [33]

    Chang Han and Katherine E. Isaacs. 2025. https://doi.org/10.1109/TVCG.2024.3456351 A deixis-centered approach for documenting remote synchronous communication around data visualizations . IEEE Transactions on Visualization and Computer Graphics, 31(1):930--940

    work page doi:10.1109/tvcg.2024.3456351 2025

  34. [34]

    Vision language models are unreliable at trivial spatial cognition

    Sangeet Khemlani, Tyler Tran, Nathaniel Gyory, Anthony M. Harrison, Wallace E. Lawson, Ravenna Thielstrom, Hunter Thompson, Taaren Singh, and J. Gregory Trafton. 2025. https://arxiv.org/abs/2504.16061 Vision language models are unreliable at trivial spatial cognition . Preprint, arXiv:2504.16061

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  35. [35]

    Yuzhi Lai, Shenghai Yuan, Youssef Nassar, Mingyu Fan, Atmaraaj Gopal, Arihiro Yorita, Naoyuki Kubota, and Matthias Rätsch. 2025. https://doi.org/10.1109/MRA.2025.3543957 Natural multimodal fusion-based human–robot interaction: Application with voice and deictic posture via large language model . IEEE Robotics & Automation Magazine, pages 2--11

    work page doi:10.1109/mra.2025.3543957 2025

  36. [36]

    Fangyu Liu, Guy Emerson, and Nigel Collier. 2023. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11:635--651

    work page 2023

  37. [37]

    Weichen Liu, Qiyao Xue, Haoming Wang, Xiangyu Yin, Boyuan Yang, and Wei Gao. 2025. https://arxiv.org/abs/2511.15722 Spatial reasoning in multimodal large language models: A survey of tasks, benchmarks and methods . Preprint, arXiv:2511.15722

    work page arXiv 2025

  38. [38]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, and 1 others. 2024. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216--233. Springer

    work page 2024

  39. [39]

    Yoshio Saito, Yoshihisa Taguchi, and Yoshiki Nishimura. 2015. https://ci.nii.ac.jp/ncid/BB19119305 [The Sanseido Dictionary of Linguistics] Meikai gengogaku jiten . Sanseido

    work page 2015

  40. [40]

    Settaluri Sravanthi, Meet Doshi, Pavan Tankala, Rudra Murthy, Raj Dabre, and Pushpak Bhattacharyya. 2024. https://doi.org/10.18653/v1/2024.findings-acl.719 PUB : A pragmatics understanding benchmark for assessing LLM s' pragmatics capabilities . In Findings of the Association for Computational Linguistics: ACL 2024, pages 12075--12097, Bangkok, Thailand. ...

    work page doi:10.18653/v1/2024.findings-acl.719 2024

  41. [41]

    In: IEEE Conf

    Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. 2025. https://doi.org/10.1109/CVPR52734.2025.00994 Thinking in space: How multimodal large language models see, remember, and recall spaces . In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10632--10643

    work page doi:10.1109/cvpr52734.2025.00994 2025

  42. [42]

    Wenyu Zhang, Wei En Ng, Lixin Ma, Yuwen Wang, Junqi Zhao, Allison Koenecke, Boyang Li, and Lu Wang. 2025 a . https://doi.org/10.18653/v1/2025.acl-long.568 SPHERE : Unveiling spatial blind spots in vision-language models through hierarchical evaluation . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: L...

    work page doi:10.18653/v1/2025.acl-long.568 2025

  43. [43]

    Zheyuan Zhang, Fengyuan Hu, Jayjun Lee, Freda Shi, Parisa Kordjamshidi, Joyce Chai, and Ziqiao Ma. 2025 b . https://openreview.net/forum?id=84pDoCD4lH Do vision-language models represent space and how? evaluating spatial frame of reference under ambiguities . In The Thirteenth International Conference on Learning Representations

    work page 2025