Pith. sign in

REVIEW 2 major objections 4 minor 36 references

Egocentric Bias in Vision-Language Models

T0 review · 2 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FlipSet, a 28-item benchmark that asks vision-language models what a monkey sees on the back of a rotated card, finds that 91.3% of 103 models score below the 25% chance level and that 75.9% of errors merely copy the camera's viewpoint.

desk verdict Promising diagnostic design and a likely-real egocentric-bias finding, but the stated physical setup would keep the monkey from seeing any characters—invalidating the labels as described. read the letter →

arxiv 2602.15892 v2 pith:H7TB5IEF submitted 2026-02-10 cs.CV cs.AI

Maijunxian Wang , Yijiang Li , Bingyang Wang , Tianwei Zhao , Ran Ji , Qingying Gao , Emmy Liu , Hokin Deng
show 1 more author
Dezhi Luo
This is my paper · ORCID
classification cs.CVcs.AI
keywords visualperspectivetakingegocentricbiasvision-languagemodelsmentalrotationtheoryofmindbenchmarkLevel-2VPTcompositionaldeficit
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that current vision-language models systematically fail at Level-2 visual perspective taking—inferring how a symbol appears from another agent's viewpoint—and that the failure is not simply missing component skills. It introduces FlipSet, a 28-item benchmark that asks a model what a plush monkey sees on the back of a card bearing a 2D character string, requiring a 180-degree mental rotation. Across 103 models, 91.3% score below the 25% chance level, and 75.88% of all errors reproduce the camera's own view, a pattern the paper calls egocentric bias. Control experiments on 24 models find high theory-of-mind accuracy (90.4%), near-chance isolated mental rotation (26.1%), and catastrophic full-task performance (10.3%), with L2 performance below the product of the component scores for 22 of 24 models. A sympathetic reader would care because the result suggests that today's multimodal systems can recognize that others see differently and can rotate shapes in isolation, yet cannot bind the two operations together—a specific architectural gap rather than a generic weakness.

What carries the argument

FlipSet's central device is a controlled multiple-choice item: an upright white card showing a 2D string such as '81', with a plush monkey on the opposite side facing the card's back. The question 'What does the monkey see on the card?' requires mentally rotating the string 180° (to '18'). Every item's four options are designed to diagnose the failure mode: correct perspective-transformed answer, egocentric camera-view answer, a contour-confusable distractor, and an unrelated random distractor, with 12 counterbalanced layouts per item to remove position bias. A companion control set reuses the same images under three prompts—theory-of-mind visibility judgment, pure mental rotation, and full

What would settle it

Run the same FlipSet battery with the character string printed on both sides of the card, so the monkey genuinely sees the rotated characters. If model accuracy jumps from near 10% to at or above chance, the original opaque-card geometry—not a compositional deficit—explains the failure; if accuracy stays near chance, the integration deficit is genuine. A complementary check is to ask human raters what the monkey sees in the original images; if they say the back is blank, the ground-truth labels are invalid.

Watch

Extended reading notes

Core claim

On the paper's own terms, FlipSet shows that 91.3% of 103 vision-language models perform below the 25% random baseline on a task that only requires reading a short character string and rotating it 180 degrees from a monkey's viewpoint. Mean accuracy is 8.96%, median 5.36%, and egocentric responses account for 75.88% of all answers. In the control battery, the same models average 90.4% on theory-of-mind recognition (does the monkey see a different string?), 26.1% on isolated mental rotation (what does the string become under 180 degrees?), and 10.3% on the full L2 task. The authors interpret the gap between the product of the component scores and the observed L2 accuracy—a deficit present in

Load-bearing premise

The claims rest on two assumptions: that the monkey actually sees the rotated character string, and that expected L2 performance equals theory-of-mind accuracy multiplied by mental-rotation accuracy; if either is wrong, the egocentric-bias and compositional-deficit conclusions lose their footing.

Editorial extensions

If this is right

  • Chain-of-thought prompting does not fix the egocentric bias and often amplifies it, implying the limitation is not a lack of verbal reasoning steps.
  • Models with near-perfect theory-of-mind scores and above-chance mental rotation still fail the integrated task, so the two component skills do not automatically compose in current architectures.
  • Mental rotation accuracy correlates strongly with L2 accuracy (r = 0.746) while theory of mind does not (r = 0.010), pointing to spatial transformation as the bottleneck skill.
  • FlipSet's 28-item, zero-shot protocol with counterbalanced answer positions offers a reusable diagnostic for tracking perspective-taking progress in future vision-language models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the ground-truth labels assume the monkey sees the rotated string, but the described setup—an upright, opaque card with characters printed on the camera-facing side—would show the monkey a blank back. Re-rendering the stimuli with the string visible on both sides (or a transparent card) would test whether the reported egocentric bias is partly a visual-geometry artifact.
  • Beyond the paper: the product-rule baseline treats theory of mind and mental rotation as independent and free to combine. Because the full L2 task adds prompt comprehension and coordination demands, any real task cost will automatically read as a 'compositional deficit'; adding a two-step control task would calibrate this baseline.
  • Beyond the paper: the benchmark only uses 180-degree rotations. Extending FlipSet to 90- and 270-degree rotations would test whether model error scales with rotation angle the way human response time does, linking the result to classical mental-rotation findings.
  • Beyond the paper: the egocentric bias may partly reflect training statistics, since front-view text is far more common than rotated text. Fine-tuning on multi-view or egocentric-to-allocentric data is a testable intervention that, if it reduces egocentric errors, would support a data-driven rather than architectural explanation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces FlipSet, a benchmark for Level-2 visual perspective taking (L2 VPT) in vision-language models. Each item shows a card with a character string and a plush monkey on the opposite side; the model must choose what the monkey sees. The authors report that 91.3% of 103 VLMs perform below 25% chance, with roughly 75.88% of errors being egocentric (outputting the camera viewpoint). Control experiments on 24 models separate theory of mind (ToM), mental rotation (MR), and L2 VPT, reporting high ToM accuracy, near-chance MR, and catastrophically low L2 VPT. They further claim a compositional deficit: L2 VPT is below the product ToM × MR. The paper frames these results as evidence that VLMs lack mechanisms for integrating social awareness with spatial transformation.

Significance. If the benchmark and its labels are valid, this is a valuable large-scale diagnostic result with clear implications for model architecture: it would demonstrate a systematic and specific failure in situated social reasoning. The scale (103 models), the error-type taxonomy, and the separation of ToM and MR are strong points. However, the entire quantitative edifice rests on the stimulus geometry producing the ground-truth 180° rotations, and the compositional-deficit conclusion relies on an unvalidated product-rule baseline. Both are load-bearing and currently unsupported.

major comments (2)
  1. [Methods (Main Experiments)] The described stimulus geometry is inconsistent with the ground-truth labels. The text says the card is 'placed upright on a wooden floor' and the monkey sits 'on the opposite side, facing the card's back.' An upright opaque card shows the monkey a blank back; a transparent card would show a mirror reflection (e.g., 'd' would appear 'b', not 'p'). The 180° rotations in Table 1 (d→p, nod→pou) correspond to the six/nine paradigm where the card lies flat between the two agents, not to the described vertical-card layout. Since all accuracy, egocentric-error, and ToM/MR/L2 statistics are computed against these labels, the benchmark's validity is at stake. The paper releases neither stimuli images nor a precise diagram. Please correct the geometry, provide the actual layout, or re-run the evaluation under the intended setup.
  2. [Control Results / Figure 3] The compositional-deficit claim rests on the asserted product rule L2 = ToM × MR. This rule is not derived and is not a neutral baseline: the L2 task is four-way multiple choice while the ToM task is binary, and L2 adds the demand of recognizing which option corresponds to the transformed string. Any such added task demand will automatically produce L2 < ToM × MR even without an integration deficit. For example, with ToM = 1.0 and MR = 0.505, the product is 0.505, but a 0.339 L2 score is not evidence of a binding failure unless the product rule is justified or replaced with a matched-task baseline. Please provide a formal task model or empirical calibration; otherwise the 'deficit' is an artifact of task design.
minor comments (4)
  1. [Control Experiment] The ToM task is described as 'recognizing that another agent's view differs', but the question is actually a Level-1 visibility judgment ('Is the monkey seeing a different string...?'). This conflation should be acknowledged more explicitly, as it may overstate what the high ToM accuracy measures.
  2. [Results / Figure 3] The claim that MR performance is 'above chance' (mean 26.1% vs 25%) is not statistically supported. With only 28 items per model per task, the standard error for a single model is about 8 percentage points; a t-test or confidence interval across the 24 models is needed before concluding MR is above chance.
  3. [Model Evaluations] The selection criterion for the 24 models used in the control experiments is not described. Since these models drive the compositional-deficit analysis, a clear sampling procedure or a justification for the subset is needed to rule out selection bias.
  4. [Appendix A1] Figure A1 shows fluctuations in confusable errors of up to 12 percentage points across answer layouts. The claim that 'answer position has limited influence' is supported only by visual inspection; a statistical test (e.g., ANOVA or chi-square) would strengthen the claim. Also, the main text contains typos such as 'In this respectm' in the Introduction.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central results are direct empirical measurements; the compositional-deficit baseline is an asserted comparison, not a fitted or self-cited input.

full rationale

The paper reports an empirical benchmark evaluation. The main claims (91.3% of models below chance; 75.88% egocentric errors; ToM 90.4%, MR 26.1%, L2 VPT 10.3%) are direct measurements of model outputs against fixed labels; no parameter is fitted and no equation is derived whose output equals an input. The egocentric option is deliberately the camera view, but that does not force models to select it, so the high egocentric rate is an observed result, not a construction artifact. The 'compositional deficit' compares observed L2 VPT to the asserted product-rule baseline ToM×MR. That baseline is a normative assumption, not fitted from the L2 data and not imported from a self-citation; the finding that L2 falls below it is a comparison, not a tautology. The baseline's appropriateness is debatable (e.g., it ignores differing chance levels and task demands), but that is a modeling/validity concern, not circularity. Several references in Related Work and Discussion are self-citations (Gao et al. 2024; Y. Li et al. 2025; B. Wang et al. 2025; Luo, Gao, & Deng 2025), but they are used for motivation or interpretation and are not load-bearing for the empirical claims. A serious construct-validity issue exists outside circularity: the Methods describe an upright opaque card with the monkey facing the blank back, which is inconsistent with ground-truth labels that require the monkey to see the 180°-rotated string; this threatens all reported numbers but is a correctness/geometry problem rather than a circular derivation.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on three unproven premises: the physical visibility of the rotated string to the monkey, the product-rule baseline for 'integrated' performance, and the font-dependent rotation mapping. No free parameters are fit; the empirical measurements are direct, but the interpretive layer (compositional deficit) is carried by the axioms above.

assumptions (3)
  • domain assumption The monkey, seated on the opposite side of an upright card, sees the 180°-rotated version of the printed string.
    Methods: 'a plush monkey sits on the opposite side, facing the card’s back.' The ground-truth labels (e.g., '81'→'18') require the rotated string to be visible to the monkey; if the card is opaque, the monkey sees a blank back. The paper never states transparency or double-sided printing.
  • ad hoc to paper If a model can integrate theory-of-mind and mental rotation, L2 VPT accuracy equals the product ToM × MR.
    Results, Control Results: 'If models could successfully integrate ToM and MR, L2 VPT accuracy should approximate their product (ToM× MR).' This product is asserted without derivation; it implicitly assumes independence of components and no additional processing cost, so any extra task demand produces an apparent 'deficit.'
  • domain assumption The 180° rotation of Latin digits/letters has a unique, well-defined ground truth (e.g., 'q'→'b', 'nod'→'pou') for all models.
    Table 1 lists ground truths; these depend on Arial glyph topology and assume models and designers share the same visual mapping. A model that has learned a different font geometry could produce what the paper labels 'confusable' while being correct under its own internal rendering.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Egocentric Bias in Vision-Language Models." pith.science (2026). https://pith.science/paper/H7TB5IEF

@misc{pith2026260215892,
  author       = {Pith},
  title        = {Pith review of: Egocentric Bias in Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H7TB5IEF}},
  note         = {Machine review of arXiv:2602.15892}
}
read the original abstract

Visual perspective taking--inferring how the world appears from another's viewpoint--is foundational to social cognition. We introduce FlipSet, a diagnostic benchmark for Level-2 visual perspective taking (L2 VPT) in vision-language models. The task requires simulating 180-degree rotations of 2D character strings from another agent's perspective, isolating spatial transformation from 3D scene complexity. Evaluating 103 VLMs reveals systematic egocentric bias: the vast majority perform below chance, with roughly three-quarters of errors reproducing the camera viewpoint. Control experiments expose a compositional deficit--models achieve high theory-of-mind accuracy and above-chance mental rotation in isolation, yet fail catastrophically when integration is required. This dissociation indicates that current VLMs lack the mechanisms needed to bind social awareness to spatial operations, suggesting fundamental limitations in model-based spatial reasoning. FlipSet provides a cognitively grounded testbed for diagnosing perspective-taking capabilities in multimodal systems.

Figures

Figures reproduced from arXiv: 2602.15892 by the authors.

Figure 1
Figure 1. FlipSet benchmark design and evaluation approach. (a) Prompt type and Error types in model responses across cognitive tasks. Each FlipSet item asks the model what a monkey sees on the back of a card—requiring a 180° mental rotation from the monkey’s viewpoint. Answer options correspond to distinct reasoning outcomes: Correct (successful perspective taking), Egocentric (front-view repetition), Confusable (visually si… view at source ↗
Figure 2
Figure 2. Error type distribution across 103 models on L2 [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Control Experiment Results: Model Performance and Task Correlations. (a) Performance comparison of 24 models on three cognitive tasks: ToM (theory of mind), L2 (Level 2 perspective taking), and MR (mental rotation). Models are grouped by family and sorted by L2 accuracy within each family. (b) Correlation analysis between the three cognitive tasks, showing the relationships between ToM, L2 VPT, and MR performance ac… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 18 linked inside Pith

  1. [1]

    Alayrac,J.-B.,Donahue,J.,Luc,P.,Miech,A.,Barr,I.,Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022). Flamingo: A visual language model for few- shot learning.Advances in neural information processing systems,35, 23716–23736

    2022

  2. [2]

    Ye, S., Tian, H., Liu, Z., Gu, L., Wang, X., Li, Q., Ren, Y., Chen,Z.,Luo,J.,Wang,J.,Jiang,T.,Wang,B.,... Wang,W. (2025). Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. https://arxiv.org/abs/2412.05271

    arXiv 2025

  3. [3]

    Cheng, K., Li, Y., Xu, F., Zhang, J., Zhou, H., & Liu, Y. (2024). Vision-language models can self-improve reasoning via reflection.arXiv preprint arXiv:2411.00855

    arXiv 2024

  4. [4]

    Zhu, Y., Zhang, W., Luo, Z., Zhao, D., & Bing, L. (2024). Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. https://arxiv.org/abs/ 2406.07476

    arXiv 2024

  5. [5]

    Edwards, K., & Low, J. (2019). Level 2 perspective-taking distinguishes automatic and non-automatic belief-tracking. Cognition,193, 104017

    2019

  6. [6]

    Flavell, J. H. (2013). Perspectives on perspective taking. In Piaget’s theory(pp. 107–139). Psychology Press

    2013

  7. [7]

    Fu, S., Bonnen, T., Guillory, D., & Darrell, T. (2025). Hidden in plain sight: Vlms overlook their visual representations. arXiv preprint arXiv:2506.08008

    arXiv 2025

  8. [8]

    Gallese, V. (2007). Before and below ‘theory of mind’: Em- bodied simulation and the neural correlates of social cog- nition.Philosophical Transactions of the Royal Society B, 362(1480), 659–669

    2007

Show all 36 references
  1. [9]

    Gallese, V., & Goldman, A. (1998). Mirror neurons and the simulation theory of mind-reading.Trends in Cognitive Sciences,2(12), 493–501

    1998

  2. [10]

    Gao, Q., Li, Y., Lyu, H., Sun, H., Luo, D., & Deng, H. (2024). Visionlanguagemodelsseewhatyouwantbutnotwhatyou see.arXiv preprint arXiv:2410.00324

    2024 arXiv

  3. [11]

    Gao, Q., Pi, X., Liu, K., Chen, J., Yang, R., Huang, X., Fang, X., Sun, L., Kishore, G., Ai, B., et al. (2025). Do vision- language models have internal world models? towards an atomic evaluation.arXiv preprint arXiv:2506.21876

    2025 arXiv

  4. [12]

    Gunia, A., Moraresku, S., & Vlček, K. (2021). Brain mecha- nismsofvisuospatialperspective-takinginrelationtoobject mental rotation and the theory of mind.Behavioural Brain Research,407, 113247

    2021

  5. [13]

    Jiang, Y., Wang, Y., Zhao, R., Parag, T., Chen, Z., Liao, Z., & Unnikrishnan, J. (2025). Videop2r: Video under- standing from perception to reasoning.arXiv preprint arXiv:2511.11113

    2025 arXiv

  6. [14]

    Li, J., Li, D., Savarese, S., & Hoi, S. (2023). Blip-2: Boot- strapping language-image pre-training with frozen image encoders and large language models.CONFERENCE. Li,X.,Wang,Y.,&Feng,J.(2024).Spatial-vqa:Benchmarking spatial reasoning in vision–language models.Proceedings of...

    2023

  7. [15]

    Luo, D., Li, Y., & Deng, H. (2025). The philosophical foundations of growing ai like a child.arXiv preprint arXiv:2502.10742

    2025

  8. [16]

    Moll, H., & Meltzoff, A. N. (2011). How does it look? level 2 perspective-taking at 36 months of age.Child Development, 82(2), 661–673

    2011

  9. [17]

    S., Bhagavatula, C., Mottaghi, R., Farhadi, A., & Choi,Y.(2020).Visualcomet:Reasoningaboutthedynamic context of a still image

    Park, J. S., Bhagavatula, C., Mottaghi, R., Farhadi, A., & Choi,Y.(2020).Visualcomet:Reasoningaboutthedynamic context of a still image. https://arxiv.org/abs/2004.10796

    2020 arXiv

  10. [18]

    (1954).The construction of reality in the child

    Piaget, J. (1954).The construction of reality in the child. Routledge

    1954

  11. [19]

    (1977).The development of thought: Equilibration of cognitive structures

    Piaget, J. (1977).The development of thought: Equilibration of cognitive structures. Viking Press

    1977

  12. [20]

    Krueger, G., & Sutskever, I. (2021). Learning transferable visual models from natural language supervision.arXiv preprint arXiv: 2103.00020

    2021 arXiv

  13. [21]

    N., & Metzler, J

    Shepard, R. N., & Metzler, J. (1971). Mental rotation of three-dimensional objects.Science,171(3972), 701–703

    1971

  14. [22]

    K., Kaur, G., & Batra, D

    Singh, Y. K., Kaur, G., & Batra, D. (2023). Rotated CLEVR: Assessing spatial reasoning of vision–language models. Proceedingsofthe17thPacificRimInternationalConference on Artificial Intelligence

    2023

  15. [23]

    Sun, H., Gao, Q., Lyu, H., Luo, D., Li, Y., & Deng, H. (2024). Probingmechanicalreasoninginlargevisionlanguagemod- els.arXiv preprint arXiv:2410.00318

    2024 arXiv

  16. [24]

    Sun, H., Yu, S., Li, Y., Gao, Q., Lyu, H., Deng, H., & Luo, D. (2025). Probing perceptual constancy in large vision language models.arXiv preprint arXiv:2502.10273

    2025

  17. [25]

    J., & Pinker, S

    Tarr, M. J., & Pinker, S. (1989). Mental rotation and orientation-dependenceinshaperecognition.CognitivePsy- chology,21(2), 233–282

    1989

  18. [26]

    one-image-probe-all

    Ramos, S., Yvinec, E., Casbon, M., Pot, E., Penchev, I., ... Hussenot, L. (2025). Gemma 3 technical report. https: //arxiv.org/abs/2503.19786 Wang,B.,Li,Y.,Zhou,Q.,Leong,H.Y.,Zhao,T.,Ye,L.,Deng, H., Luo, D., & Vasconcelos, N. (2025). Do vision language models infer human inten...

    2025 arXiv

  19. [27]

    Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., & Lin, J. (2024). Qwen2-vl:Enhancingvision-languagemodel’sperceptionof theworldatanyresolution.https://arxiv.org/abs/2409.12191 Wang,R.,Todd,G.,Xiao,Z.,Yuan,X.,Côté,M. -A.,Clar...

    2024 arXiv

  20. [28]

    Xu, G., Jin, P., Wu, Z., Li, H., Song, Y., Sun, L., & Yuan, L. (2024). Llava-cot: Let vision language models reason step-by-step.arXiv preprint arXiv:2411.10440

    2024 arXiv

  21. [29]

    Xu, Z., Jain, S., & Kankanhalli, M. (2024). Hallucination is inevitable: An innate limitation of large language models. arXiv preprint arXiv:2401.11817

    2024 arXiv

  22. [30]

    Yin, B., Wang, Q., Zhang, P., Zhang, J., Wang, K., Wang, Z., Zhang, J., Chandrasegaran, K., Liu, H., Krishna, R., et al. (2025). Spatial mental modeling from limited views. Structural Priors for Vision Workshop at ICCV’25

    2025

  23. [31]

    Zacks, J. M. (2008). Neuroimaging studies of mental rotation: A meta-analysis and review.Journal of Cognitive Neuro- science,20(1), 1–19

    2008

  24. [32]

    Zellers, R., Bisk, Y., Farhadi, A., & Choi, Y. (2019). From recognition to cognition: Visual commonsense reasoning. https://arxiv.org/abs/1811.10830

    2019 arXiv

  25. [33]

    Zhang, J., Hu, J., Khayatkhoei, M., Ilievski, F., & Sun, M. (2024). Exploring perceptual limitation of multimodal large language models.arXiv preprint arXiv:2402.07384

    2024 arXiv

  26. [34]

    Yang, Y., Pang, R., & Yang, Y. (2024). Improve vision language model chain-of-thought reasoning.arXiv preprint arXiv:2410.16198

    2024 arXiv

  27. [35]

    Zhang, Y., Wu, J., Li, W., Li, B., Ma, Z., Liu, Z., & Li, C. (2024). Video instruction tuning with synthetic data. https://arxiv.org/abs/2410.02713

    2024 arXiv

  28. [36]

    Zhang, Z., Hu, F., Lee, J., Shi, F., Kordjamshidi, P., Chai, J., & Ma, Z. (2024). Do vision–language models represent space and how? evaluating spatial frame of reference under ambiguities. Zhao,X.,Malle,B.,&Gweon,H.(2016).Isitanine,orasix? prosocial and selective perspective ...

    2024

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.