REVIEW 3 major objections 4 minor 44 references
Vision-language models describe the same image with less than ten percent lexical overlap when the observer's goal changes, and the paper argues this context-dependence reflects genuine computation rather than sampling noise.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 23:28 UTC pith:U23T2B6C
load-bearing objection The 90% context-dependence headline is confounded by prompt surface differences and should not be cited as-is, but the paper is transparent, ships code/data, and has a real qualitative finding; it deserves a serious referee. the 3 major comments →
Context-Dependent Affordance Computation in Vision-Language Models
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central discovery is 'affordance drift': the set of objects and uses a vision-language model reports for one image changes massively when the prompt changes the agent's identity or goal, from neutral observer to chef, security professional, child, wheelchair user, urgent survivor, or leisured explorer. The mean word-level Jaccard similarity between context conditions is 0.095 (95% CI [0.093, 0.096], p<0.0001), object-level 0.119, and a replication on a second, architecturally distinct model gives 0.181. Because sentence-embedding cosine similarity is much higher (0.415), the authors conclude that surface vocabulary shifts more than underlying meaning, leaving a semantic invariant core. T
What carries the argument
The carrying mechanism is a controlled comparison: the same image is prompted with seven agentic persona primes; outputs are parsed into object/affordance lists; and pairwise Jaccard similarity over word tokens and object names measures how much of the functional scene ontology changes with context. Stochastic baselines across four temperatures and five seeds separate genuine context effects from sampling noise, and Tucker decomposition — a three-mode factor analysis separating image, context, and embedding dimensions — reveals the latent structure of the shift. The Jaccard coefficient is the load-bearing statistic: the claim that more than 90 percent of lexical scene description is context-
Load-bearing premise
The load-bearing premise is that the seven differently worded prompts really create seven different agent contexts; because the paper has not yet run the same-prompt/different-persona and paraphrase controls it lists as future work, the 90 percent drift could in principle be instruction-following surface variation.
What would settle it
A decisive check is the same-prompt/different-persona control: keep the wording of the prime fixed and vary only the named persona, or vary the wording while keeping the persona fixed. If pairwise Jaccard similarity follows the wording rather than the persona, the 90 percent drift is prompt sensitivity, not agent context; if it follows the persona, the paper's interpretation survives.
If this is right
- If the drift is real, vision pipelines that first build a context-free geometric representation are spending most of their computation on the roughly 10 percent of functional signal that is invariant across tasks.
- The lexical-versus-semantic gap (90.5% vs 58.5% context-dependent) implies that scene meaning retains a substantial invariant core even as vocabulary changes, so context re-weights salience rather than creating or destroying affordances.
- The stable orthogonal factors (a culinary manifold and an access axis) mean context-dependent affordance structure is organized around functional categories, not diffuse variation.
- The effect replicates across two architecturally distinct vision-language models, suggesting it is a general property of models trained on naturalistic image-text data.
- For embodied AI, the result motivates just-in-time ontology: construct task-specific functional projections at query time rather than maintaining a single static world model, though the paper labels this a suggested direction rather than a proven design.
Where Pith is reading between the lines
- Extension: the paper's own list of remaining controls — same prompt with different personas, and prompt paraphrase — is the decisive test; if those controls show low drift, the 90% figure is surface prompt sensitivity, not agentic context.
- Extension: the roughly 40% semantic overlap that survives context shifts suggests a hybrid architecture: a small stable geometric core plus a large dynamic task projection, rather than the pure just-in-time alternative the paper sketches.
- Extension: the access axis separating child from wheelchair-user contexts offers a concrete testbed for the paper's untested action-distance prediction: compare spatial reasoning or navigation judgments under the two personas and see whether behavior tracks Euclidean or action-based distance.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript characterizes context-dependent affordance computation in vision-language models (VLMs). Using 479 COCO images and seven persona prompts (neutral, chef, security, child, wheelchair-user, urgent, leisure), it asks Qwen3-VL-30B-A3B to output JSON affordance descriptions and computes pairwise Jaccard similarity across primes. The mean word-level Jaccard of 0.095 is interpreted as '>90% of lexical scene description is context-dependent'; sentence-level cosine similarity of 0.415 is interpreted as 58.5% context-dependence. A LLaVA-1.5-13B replication, stochastic sampling controls, and Tucker decomposition of sentence embeddings are reported, and the paper proposes a 'semantic-first' processing architecture and Just-In-Time Ontology for robotics. The paper explicitly limits its claims to behavioral output and lists several untested controls.
Significance. The qualitative phenomenon—that a VLM's reported objects and affordances shift when the user prompt changes—is plausible and consistent with prior work on instruction-following and task-conditioned affordance ranking. The paper has notable strengths: public code/data, a large pairwise sample, cross-model replication, stochastic baselines, bootstrap stability analysis, and unusually explicit statements about what is not claimed. If the 90% magnitude were valid, it would be an important quantitative anchor for debates about static versus dynamic scene representations in robotics and cognitive AI. However, the central magnitude is not currently established: the primes differ in surface task demands, the key same-prompt/paraphrase controls are acknowledged as future work, and 1−Jaccard is not a variance-explained quantity. The headline contribution therefore requires substantial additional support.
major comments (3)
- [§4.4, §6.6, Appendix A.1 (Table 12)] The headline inference from J̄=0.095 to '>90% of functional scene ontology is context-dependent' requires that the seven primes isolate agent context. They do not. P0 asks for the 3 most prominent objects, geometric properties, and standard functions; P1–P4 ask for 3 role-specific critical/interesting objects; P5 asks for a single survival tool; P6 asks what 'catches your eye.' Word-level Jaccard (Eq. 8) is computed over token sets of concatenated name+affordance+reasoning, so changes in requested object count, output length, and instruction vocabulary directly deflate J; P5's single-object output alone caps overlap with three-object outputs near 1/3. The paper itself lists same-prompt/different-image and prompt-paraphrase sensitivity as 'Remaining Controls for Future Work' (§4.4, §6.6), i.e., the control that would separate agent context from prompt wording is absent. As it stands, the
- [§5.6, Table 10] The paper treats 1−J and 1−cosine as 'proportion of variance explained' and as 'Ctx-Dep.' percentages. Jaccard is a set-overlap coefficient; its complement is not a variance decomposition and cannot be interpreted as 'context explains approximately 90% of the variance.' The same holds for cosine similarity: 0.415 similarity does not imply 58.5% of variance is context-dependent. The only variance-explained quantity in the paper is the Tucker decomposition's 46.6%, and that is tensor-approximation variance, not a decomposition of the Jaccard contrast. This is a category error in the central statistic.
- [§5.4, §6.4, Table 7] Table 7 explicitly defines 'Var. %' as each factor's share of captured variance (summing to 100% of explained variance). Yet §5.4 says Dim1 'accounts for only 0.9% of variance,' and §6.4 cites 'Tucker Dim1's low variance contribution (0.9%)' to argue geometry-invariant features are a small residual. Because only 46.6% of total tensor variance is explained, Dim1's 0.9% is a share of that captured variance, not of total variance. The architectural inference drawn from this number is therefore not supported by the reported decomposition.
minor comments (4)
- [Abstract vs. Table 3 / §5.7] The abstract reports LLaVA mean J=0.160, whereas Table 3 reports 0.1807; the abstract reports 2,384 stochastic inference runs, while §5.7 describes 7,000 runs. Please harmonize these numbers.
- [§6.6 vs. §5.8] §6.6 refers to 'Lemmatized Jaccard' as an alternative metric, but §5.8/Table 10 implements stopword-filtered Jaccard, not lemmatization; Section 4.3.4 explicitly states that no lemmatization/stemming is used.
- [§5.3] The Visual Genome human baseline is not collected under goal priming, so it cannot validate context-dependent extraction; the comparison is suggestive only and should be labeled as such.
- [§4.4, Table 2] The null threshold J<0.5 is arbitrary; since the observed J=0.095 is within 2–9× of the paper's own random-baseline estimate (J≈0.01–0.05, §4.4), a more informative benchmark would be random-pair or same-prompt overlap.
Circularity Check
No significant circularity: the reported drift is an output-contrast measurement with acknowledged construct-validity limits, not a derivation from its own inputs.
full rationale
The paper's headline '>90% context-dependent' is the complement of the observed pairwise Jaccard similarity between model outputs under seven primes (Eq. 8). This is an operational measurement, not a fitted parameter renamed as a prediction. No equation reduces a target result to an input by construction. H2 is tested by comparing outputs, and the paper explicitly limits the claim to behavioral output ('We do not claim to establish processing order or architectural primacy', Section 7). The only self-citation (Farzulla, 2025) appears as a non-load-bearing aside linking theta_history to developmental training, not as support for the drift result. The acknowledged missing controls in Section 4.4 ('Same-prompt, different-image' and 'Prompt paraphrase sensitivity') and the metric-dependence caveat in Section 5.8 (cosine similarity yields 58.5% context-dependence) are construct-validity and confound concerns, not circular reductions. The Tucker 'Culinary Manifold' and 'Access Axis' labels are post-hoc interpretations of stable factor loadings, not predictions derived from those loadings. Thus no specific circular step can be exhibited, and the score is 0.
Axiom & Free-Parameter Ledger
free parameters (2)
- Primary inference temperature =
0.7 (primary; 0.0-1.0 in stochastic baseline)
- Tucker rank =
[10,3,10]
axioms (4)
- ad hoc to paper The threshold J<0.5 defines 'more difference than overlap' and thus context-dependence.
- ad hoc to paper 1-J measures the proportion of scene description that is context-dependent, and 'context explains ~90% of variance'.
- domain assumption all-MiniLM-L6-v2 sentence embeddings capture semantic similarity between affordance descriptions.
- domain assumption COCO-2017 validation images selected for 'multi-object scenes with high interaction potential' are representative of naturalistic scenes for affordance generalization.
read the original abstract
We characterize the phenomenon of context-dependent affordance computation in vision-language models (VLMs). Our primary study uses Qwen3-VL-30B-A3B ($n = 3{,}213$ scene-context pairs from COCO-2017: 479 images under 7 agentic personas), with a cross-model replication on LLaVA-1.5-13B. We demonstrate substantial affordance drift: mean Jaccard similarity between context conditions is $0.095$ (95% CI $[0.092, 0.097]$ across $N = 479$ images; $9{,}244$ prime pairs; $p < 0.0001$), indicating that more than 90% of lexical scene description is context-dependent; the LLaVA replication reproduces the effect (mean $J = 0.160$, 84% context-dependent). Sentence-level cosine similarity confirms drift at the semantic level (mean $= 0.415$, 58.5% context-dependent). Stochastic baseline experiments ($2{,}384$ inference runs across 4 temperatures and 5 seeds) confirm this reflects genuine context effects rather than generation noise: within-prime variance is substantially lower than cross-prime variance across all conditions. Tucker decomposition with bootstrap stability analysis ($n = 1{,}000$ resamples) reveals stable orthogonal latent factors: a "Culinary Manifold" isolated to chef contexts and an "Access Axis" spanning child-mobility contrasts. The gap between lexical (90%) and semantic (58.5%) measures indicates that surface vocabulary changes more than underlying meaning under context shifts. These findings suggest a direction for robotics: dynamic, query-dependent ontological projection (JIT Ontology) rather than static world modeling. We do not claim to establish processing order or architectural primacy; such claims require internal representational analysis beyond output behavior.
Figures
Reference graph
Works this paper leans on
-
[1]
Do as I can, not as I say: Grounding language in robotic affordances
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, et al. Do as I can, not as I say: Grounding language in robotic affordances. In Conference on Robot Learning (CoRL). PMLR, 2022
2022
-
[2]
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL : A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023
Pith/arXiv arXiv 2023
-
[3]
Emily M. Bender and Alexander Koller. Climbing towards NLU : On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 5185--5198. ACL, 2020. doi:10.18653/v1/2020.acl-main.463
-
[4]
Andrei Z. Broder. On the resemblance and containment of documents. In Compression and Complexity of Sequences, pages 21--29. IEEE, 1997. doi:10.1109/SEQUEN.1997.666900
arXiv 1997
-
[5]
An outline of a theory of affordances
Anthony Chemero. An outline of a theory of affordances. Ecological Psychology, 15 0 (2): 0 181--195, 2003. doi:10.1207/s15326969eco1502_5
-
[6]
Cortical mechanisms of action selection: The affordance competition hypothesis
Paul Cisek. Cortical mechanisms of action selection: The affordance competition hypothesis. Philosophical Transactions of the Royal Society B, 362 0 (1485): 0 1585--1599, 2007. doi:10.1098/rstb.2007.2054
arXiv 2007
-
[7]
Being There: Putting Brain, Body, and World Together Again
Andy Clark. Being There: Putting Brain, Body, and World Together Again. MIT Press, 1997
1997
-
[8]
Whatever next? P redictive brains, situated agents, and the future of cognitive science
Andy Clark. Whatever next? P redictive brains, situated agents, and the future of cognitive science. Behavioral and Brain Sciences, 36 0 (3): 0 181--204, 2013. doi:10.1017/S0140525X12000477
-
[9]
Hubert L. Dreyfus. What Computers Still Can't Do: A Critique of Artificial Reason. MIT Press, 1992
1992
-
[10]
Hubert L. Dreyfus. Why H eideggerian AI failed and how fixing it would require making it more H eideggerian. Philosophical Psychology, 20 0 (2): 0 247--268, 2007. doi:10.1080/09515080701239510
-
[11]
Training data and the maladaptive mind: A computational framework for developmental trauma
Murad Farzulla. Training data and the maladaptive mind: A computational framework for developmental trauma. Research Square, 2025. doi:10.21203/rs.3.rs-8634152/v1. Under review at Humanities & Social Sciences Communications (Nature). Zenodo: 10.5281/zenodo.17681336
-
[12]
James J. Gibson. The Senses Considered as Perceptual Systems. Houghton Mifflin, 1966
1966
-
[13]
James J. Gibson. The Ecological Approach to Visual Perception. Houghton Mifflin, 1979
1979
-
[14]
Tejas Gokhale. Towards robust visual understanding: A paradigm shift in computer vision from recognition to reasoning. AI Magazine, 45 0 (3): 0 396--403, 2024. doi:10.1002/aaai.12194. AAAI New Faculty Highlights invited talk
-
[15]
Melvyn A. Goodale and A. David Milner. Separate visual pathways for perception and action. Trends in Neurosciences, 15 0 (1): 0 20--25, 1992. doi:10.1016/0166-2236(92)90344-8
-
[16]
Visual affordance and function understanding: A survey
Mohammed Hassanin, Salman Khan, and Murat Tahtali. Visual affordance and function understanding: A survey. ACM Computing Surveys, 54 0 (3): 0 1--35, 2022. doi:10.1145/3446370
-
[17]
Sein und Zeit
Martin Heidegger. Sein und Zeit. Max Niemeyer Verlag, 1927
1927
-
[18]
Leverage task context for object affordance ranking
Haojie Huang, Hongchen Luo, Wei Zhai, Yang Cao, and Zheng-Jun Zha. Leverage task context for object affordance ranking. arXiv preprint arXiv:2411.16082, 2024
Pith/arXiv arXiv 2024
-
[19]
VoxPoser : Composable 3D value maps for robotic manipulation with language models
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. VoxPoser : Composable 3D value maps for robotic manipulation with language models. In Conference on Robot Learning (CoRL). PMLR, 2023
2023
-
[20]
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123 0 (1): 0 32--73, 2017. doi:10.1007/s11263-016-0981-7
-
[21]
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C. Lawrence Zitnick. Microsoft COCO : Common objects in context. In European Conference on Computer Vision (ECCV), pages 740--755. Springer, 2014. doi:10.1007/978-3-319-10602-1_48
-
[22]
Visual instruction tuning
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2024
2024
-
[23]
Urbano Lorenzo-Seva and Jos M. F. ten Berge. Tucker's congruence coefficient as a meaningful index of factor similarity. Methodology, 2 0 (2): 0 57--64, 2006. doi:10.1027/1614-2241.2.2.57
-
[24]
Ph \'e nom \'e nologie de la Perception
Maurice Merleau-Ponty. Ph \'e nom \'e nologie de la Perception . Gallimard, 1945
1945
-
[25]
Object representation in the ventral premotor cortex (area F5 ) of the monkey
Akira Murata, Luciano Fadiga, Leonardo Fogassi, Vittorio Gallese, Vassilis Raos, and Giacomo Rizzolatti. Object representation in the ventral premotor cortex (area F5 ) of the monkey. Journal of Neurophysiology, 78 0 (4): 0 2226--2230, 1997. doi:10.1152/jn.1997.78.4.2226
-
[26]
RT-Affordance : Affordances are versatile intermediate representations for robot manipulation
Soroush Nasiriany, Sean Kirmani, Tianhe Ding, Laura Smith, Yuke Zhu, Danny Driess, Dorsa Sadigh, and Ted Xiao. RT-Affordance : Affordances are versatile intermediate representations for robot manipulation. arXiv preprint arXiv:2411.02704, 2024. doi:10.1109/icra55743.2025.11127525
Pith/arXiv arXiv 2024
-
[27]
Anh Nguyen, Dimitrios Kanoulas, Darwin G. Caldwell, and Nikos G. Tsagarakis. Object-based affordances detection with convolutional neural networks and dense conditional random fields. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5908--5915, 2017. doi:10.1109/IROS.2017.8206484
arXiv 2017
-
[28]
Action in Perception
Alva No \"e . Action in Perception. MIT Press, 2004
2004
-
[29]
J. Kevin O'Regan and Alva No \"e . A sensorimotor account of vision and visual consciousness. Behavioral and Brain Sciences, 24 0 (5): 0 939--973, 2001. doi:10.1017/S0140525X01000115
-
[30]
AffordanceLLM : Grounding affordance from vision language models
Shengyi Qian, Weifeng Chen, Min Bai, Xiong Zhou, Zhuowen Tu, and Li Erran Li. AffordanceLLM : Grounding affordance from vision language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 7587--7597, 2024. doi:10.1109/CVPRW63382.2024.00754
arXiv 2024
-
[31]
Sentence- BERT : Sentence embeddings using S iamese BERT -networks
Nils Reimers and Iryna Gurevych. Sentence- BERT : Sentence embeddings using S iamese BERT -networks. In Conference on Empirical Methods in Natural Language Processing (EMNLP-IJCNLP), pages 3982--3992. ACL, 2019. doi:10.18653/v1/D19-1410
-
[32]
Giacomo Rizzolatti and Laila Craighero. The mirror-neuron system. Annual Review of Neuroscience, 27: 0 169--192, 2004. doi:10.1146/annurev.neuro.27.070203.144230
arXiv 2004
-
[33]
A survey on efficient vision-language models
Gaurav Shinde, Anuradha Ravi, Emon Dey, Shadman Sakib, Milind Rampure, and Nirmalya Roy. A survey on efficient vision-language models. WIREs Data Mining and Knowledge Discovery, 15 0 (3): 0 e70036, 2025. doi:10.1002/widm.70036
-
[34]
Daniel J. Simons and Christopher F. Chabris. Gorillas in our midst: Sustained inattentional blindness for dynamic events. Perception, 28 0 (9): 0 1059--1074, 1999. doi:10.1068/p281059
doi:10.1068/p281059 1999
-
[35]
Thomas A. Stoffregen. Affordances as properties of the animal-environment system. Ecological Psychology, 15 0 (2): 0 115--134, 2003. doi:10.1207/s15326969eco1502_2
-
[36]
Differential effects of contextual congruency on recognition and retrieval of perceptual details
Rebeca Su \'a rez, Bruno Lara, and Alejandra Ciria. Differential effects of contextual congruency on recognition and retrieval of perceptual details. Memory & Cognition, 2026. doi:10.3758/s13421-025-01848-0. Published online 3 February 2026
-
[37]
Mind in Life: Biology, Phenomenology, and the Sciences of Mind
Evan Thompson. Mind in Life: Biology, Phenomenology, and the Sciences of Mind. Harvard University Press, 2007. ISBN 978-0-674-02511-0
2007
-
[38]
Ledyard R. Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 31 0 (3): 0 279--311, 1966. doi:10.1007/BF02289464
-
[39]
Michael T. Turvey. Affordances and prospective control: An outline of the ontology. Ecological Psychology, 4 0 (3): 0 173--187, 1992. doi:10.1207/s15326969eco0403_3
-
[40]
Varela, Evan Thompson, and Eleanor Rosch
Francisco J. Varela, Evan Thompson, and Eleanor Rosch. The Embodied Mind: Cognitive Science and Human Experience. MIT Press, 1991
1991
-
[41]
Danfei Xu, Yuke Zhu, Christopher B. Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5410--5419, 2017. doi:10.1109/CVPR.2017.330
-
[42]
RoboPoint : A vision-language model for spatial affordance prediction for robotics
Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. RoboPoint : A vision-language model for spatial affordance prediction for robotics. In Conference on Robot Learning (CoRL), pages 4005--4020. PMLR, 2024
2024
-
[43]
Self-explainable affordance learning with embodied caption
Zhipeng Zhang, Zhimin Wei, Guolei Sun, Peng Wang, and Luc Van Gool. Self-explainable affordance learning with embodied caption. arXiv preprint arXiv:2404.05603, 2024
Pith/arXiv arXiv 2024
-
[44]
Places: A 10 million image database for scene recognition
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40 0 (6): 0 1452--1464, 2017. doi:10.1109/TPAMI.2017.2723009
arXiv 2017
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.