Pith. sign in

REVIEW 3 major objections 5 minor 56 references

The paper argues that today's multimodal models do not perform active visual observation: given tasks that force repeated, hypothesis-driven returns to an image, the strongest model solves 10.6% of items while humans average 96.1%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 21:08 UTC pith:6FOTDPAL

load-bearing objection A well-built benchmark of hard visual tasks, but the "active observation" label oversells what the design can actually isolate — the gap is real, the cause is inferred. the 3 major comments →

arxiv 2607.16165 v1 pith:6FOTDPAL submitted 2026-07-17 cs.CV cs.AIcs.CLcs.LG

An Exam for Active Observers

Jiarui Zhang , Muzi Tao , Shangshang Wang , Ollie Liu , Xuezhe Ma , Willie Neiswanger This is my paper
classification cs.CV cs.AIcs.CLcs.LG
keywords active visionmultimodal large language modelsbenchmarkiterative perceptionvisual reasoningdistributed scanningsequential traversalvisual attribute transfer
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

ActiveVision is a 17-task benchmark built to measure whether multimodal large language models can actively re-inspect an image as they reason, rather than answer from a single static description. The paper's central claim is that current frontier models lack this active-observation capability: on 85 items, the strongest model solves 10.6% and scores zero on 11 of 17 tasks, while three human participants average 96.1%. Scaling reasoning effort barely moves the needle, and even coding agents that write and run their own vision code reach at most 50.6%, with failures concentrated in exactly the tasks that require exhaustive scanning or step-by-step tracing. The paper argues that active visual observation is a distinct, measurable capability that current vision-language benchmarks hide.

Core claim

The paper shows a wide, systematic human–model gap on tasks that force repeated visual perception. Each item is designed so no concise language description carries the answer; the observer must keep returning to the pixels to scan distributed signals, follow connected structures, and compare fine-grained attributes across distant regions. Models fail these demands in characteristic ways: they undercount as scenes get crowded, lose the walk's frame at the very first steps of sequential traversal, and default to 'same' in visual comparison. Reasoning-effort scaling does not close the gap, and tool use narrows it only where perception reduces to clean algorithmic computation; catching tool fail

What carries the argument

The load-bearing object is the ActiveVision benchmark, organized into three task families—Distributed Scanning, Sequential Traversal, and Visual Attribute Transfer—that instantiate exhaustive enumeration, curve tracing, and fine-grained comparison. Each task begins as a deterministic procedural scaffold with exact ground truth, then is re-rendered into a photorealistic image by GPT-image-2 with the constraint that positions, counts, labels, and topology are preserved; models see only the photorealistic image, so any correct answer requires re-deriving the scaffold's structure from the pixels. Exact-match accuracy over 85 items is the primary metric, with a three-participant human baseline fo

Load-bearing premise

The central claim rests on the assumption that GPT-image-2's photorealistic re-rendering preserves the procedural scaffold's exact positions, counts, labels, and topology; the paper asserts this but does not quantify it, and if the renderer merges, splits, or drops elements, ground-truth answers may be wrong for the image models actually see.

What would settle it

Take a random sample of ActiveVision items and compare the procedural scaffold to the photorealistic image—either with independent human annotators or an automated geometric matcher—looking for count or topology mismatches; if even a few items show merged or dropped elements, model failures are partly explained by faulty ground truth. Alternatively, run the same models on the clean scaffold images: a large accuracy jump would indicate the bottleneck is rendering robustness, not active observation.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If correct, state-of-the-art multimodal models are not yet reliable for high-stakes visual work—radiology, satellite imagery, manufacturing inspection—where exhaustive scanning and sequential tracing are core operations.
  • Saturated static benchmarks do not mean visual understanding is solved; this gap is invisible on leaderboards that allow one-glance answers.
  • Increasing reasoning effort is not a substitute for a perception–action loop; closing the gap likely requires architectures or training objectives that let models revisit visual evidence.
  • Agentic tool use can substitute for active vision only where the task reduces to reliable computation; traversal and tangle-counting tasks resist all tested agents, making verification the new bottleneck.
  • The benchmark gives future systems a concrete target: to approach the human ~96% on the same 85 items.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The observed failure patterns (undercounting, early walk loss, 'same' default) are consistent with a one-pass visual encoding bottleneck; a direct test would be whether letting models re-encode image patches or attend with multiple glimpses closes part of the gap.
  • The paper's two-stage pipeline suggests a natural ablation: run models on the clean procedural scaffolds as well as the photorealistic renderings. If accuracy jumps on the clean versions, the bottleneck is partly rendering robustness rather than active observation alone.
  • A follow-up with real-world imagery (medical scans, aerial photos, wiring photos) and human-verified ground truth would test whether the gap transfers beyond controlled synthetic-but-realistic renderings.
  • The paper's own caveat that improved image description could erode task difficulty implies ActiveVision may need periodic regeneration to remain a valid measure as models improve.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces ActiveVision, a benchmark of 17 visual task generators (85 items) designed to require iterative, hypothesis-driven revisiting of an image. Tasks are grouped into Distributed Scanning, Sequential Traversal, and Visual Attribute Transfer. Each instance is procedurally generated with exact ground truth, then re-rendered photorealistically with GPT-image-2 under the assertion that geometry, counts, labels, and topology are preserved. The authors evaluate six frontier MLLMs at multiple reasoning-effort tiers, finding that the best model (GPT-5.5 xhigh) solves 10.6% of items, with zero accuracy on 11 of 17 tasks, while three human participants average 96.1%. They also test three autonomous coding agents that can write and run vision code, finding large but uneven gains (24.7–50.6%), with traversal tasks remaining difficult. The paper concludes that current MLLMs lack robust active visual observation, and that additional reasoning or tool use does not close the gap.

Significance. If the benchmark is valid, it addresses a real and under-measured capability: whether MLLMs can keep returning to visual evidence during reasoning, rather than relying on a single static encoding. The paper has several strong methodological features: deterministic procedural scaffolds with exact ground truth, a question-only control (2.4%), prefix-survival analysis of ordered walks, per-task error-mode tables, and a genuine agentic tool-use ablation with verbatim rollouts and cost accounting. These controls go beyond most existing vision benchmarks and make the reported human–model gap concrete. The weakness is that the central construct—'active observation'—is not directly measured, and the rendering-fidelity assumption is unquantified. Should these be addressed, the benchmark could be a useful diagnostic for the vision-language community.

major comments (3)
  1. [§3.2, §4.3] Construct-validity gap: The paper's title and abstract claim that MLLMs 'lack robust active visual observation,' but no condition or measurement isolates iterative re-engagement with the image from a one-pass perceptual deficit. The design principle in §3.2 rules out solutions from a single language description, yet it does not rule out a powerful single-pass visual encoder that extracts all relevant information at once. The process analyses in §4.3 (undercounting slopes, prefix survival, 'same' bias) describe error patterns, not evidence about whether models re-attend to image tokens as reasoning unfolds. A model with perfect one-shot perception could, in principle, answer all items without any 'return to the image.' I recommend either (a) adding a control that directly manipulates the opportunity for active observation (e.g., gating image access across reasoning steps, or comparing acc
  2. [§3.3, Figure 3] Rendering-fidelity assumption is load-bearing but unverified. The accuracy results are computed against ground truth derived from the procedural scaffold, yet the model is shown only the GPT-image-2 photorealistic image. The paper asserts that re-rendering preserves 'positions, counts, labels, and topology' without altering them, but no quantitative check is reported. If the renderer merges, splits, or drops elements (e.g., a rope loop or a contour), then model failures could partly reflect mismatched ground truth rather than a lack of active observation. I request a verification study—for example, human annotation of rendered images against scaffold ground truth, or automated segmentation-based comparison—for a sample of items across all 17 tasks. This is essential to validate every accuracy number in Tables 2 and 4.
  3. [§4.1, Table 2] Per-task accuracy rests on only N=5 items per task, and the human baseline is N=3. With five binary-scored items, the difference between 0/5 and 1/5 (or 2/5) is not statistically meaningful, yet the paper emphasizes 'zero on 11 of 17 tasks' as a headline result. The overall 85-item gap is robust, but the per-task and per-model comparisons are over-interpreted. The authors should report confidence intervals or exact binomial intervals, and avoid drawing strong conclusions from individual task zeros. Similarly, the effort-tier differences in §4.2 are acknowledged as within sampling noise; this should be stated more prominently rather than implied as a trend.
minor comments (5)
  1. [Figure 3] Typo in the figure: 'TANGLED LOOP COUNTIING' should read 'COUNTING.' Also, the figure would benefit from a side-by-side overlay of scaffold and rendered image with labeled correspondences to support the preservation claim.
  2. [§4.2, Figure 4] The x-axis is API cost per item, which is a proxy for reasoning effort but conflates token usage with other factors. Reporting output token counts (as in Table 4) is helpful; consider adding input token counts or a secondary x-axis for effort tier to aid interpretation.
  3. [§4.3, Table 3] The 'same' bias analysis is interesting, but the table would be clearer if it reported the base rate of 'same' answers in the dataset, so the reader can distinguish bias from accuracy. Also, the highlighted runs are not described in the caption; please explain what the highlighting indicates.
  4. [§5.2] The limitation paragraph correctly notes that tasks may erode as models improve at describing images. This is an honest caveat, but the paper does not propose a quantitative measure of 'information content' to detect such erosion. A simple operationalization (e.g., measuring question-only accuracy over time, or human agreement on whether a static description suffices) would strengthen future reproducibility.
  5. [§3.3, Table 1] For the 'Field Difference Spotting' and 'Signal Difference Spotting' tasks, the description says 'List the indexed cells/segments that differ,' but Table 2 scores exact match on the entire list. Please specify how ordering is handled in scoring (e.g., set equality vs. sequence equality), since this affects the difficulty and the reported accuracies.

Circularity Check

0 steps flagged

No substantive circularity: ActiveVision is externally grounded by human performance and ablations; the only mild concern is a design tautology, not a derivation that reduces to its inputs.

full rationale

The paper's central inference—frontier MLLMs lack robust active visual observation—is an empirical generalization from exact-match accuracy on 85 rendered instances, benchmarked against a three-participant human baseline (§4.1, Table 2). No parameter is fitted to the evaluation set and then reported as a prediction; the question-only control (2/85), reasoning-effort sweep (§4.2), and coding-agent ablation (§4.4) are genuine manipulations rather than circular probes. The design principle in §3.2 ('every task instance carries discriminative visual state whose information content exceeds what a single language description can losslessly carry') is a construction criterion, not an equation that equates the output with an input: it guarantees that language-summary shortcuts fail, but the actual failure of MLLMs on the resulting tasks is measured, not assumed. The self-citations [53,54] are contextual and non-load-bearing; [54] is cited only to note that attention shifts could in principle implement active observation, and the benchmark's conclusions do not depend on that claim. The stated limitation in §5.2 ('our images are synthetic even with photorealistic re-rendering') and the unquantified renderer-preservation assertion in §3.3 are validity/robustness concerns rather than circularity: if GPT-image-2 altered counts or topology, the ground truth could mismatch the rendered image, but that would be an annotation error, not a prediction constructed from the answer. Likewise, the construct-validity concern that low accuracy may reflect hard visual tasks rather than the absence of an iterative loop is a measurement-interpretation question, not a demonstration that the result is equivalent to its inputs. Therefore no circular step can be exhibited; the appropriate finding is no significant circularity, with a low score reflecting only the mild design tautology that tasks are deliberately engineered to defeat one-pass perception.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No fitted free parameters or invented physical entities appear; the paper's contribution is a measurement instrument. The main unverified assumptions are the rendering-fidelity transfer, the mapping of tasks to cognitive operations, and the representativeness of the human baseline.

axioms (4)
  • domain assumption Active observation is essential for the tested tasks; repeated gaze shifts are required for exhaustive enumeration, curve tracing, and fine-grained comparison.
    Borrowed from psychophysics and cognitive science (§2.1, §3.1); not independently established for MLLM evaluation, but grounded in cited human literature.
  • ad hoc to paper GPT-image-2 re-rendering preserves the scaffold's positions, counts, labels, and topology.
    Load-bearing for ground-truth transfer; asserted in §3.3 but not verified quantitatively.
  • domain assumption No concise language description of an image can losslessly carry the discriminative visual state required by the tasks.
    Design principle stated in §3.2; if false, tasks could be solved by summarizing rather than active looking, which would undermine the interpretation.
  • domain assumption Three self-paced human participants provide a valid reference for human active-observation performance.
    Small convenience sample in §4.1; no inter-rater reliability or demographic control is reported.

pith-pipeline@v1.3.0-alltime-deepseek · 14649 in / 10712 out tokens · 88647 ms · 2026-08-01T21:08:01.838837+00:00 · methodology

0 comments
read the original abstract

Human vision is a closed loop: gaze is continuously redirected by intermediate hypotheses rather than a single snapshot. Decades of psychophysics and cognitive science have argued that this active observation is essential for a wide range of tasks. Whether today's multimodal large language models (MLLMs) exercise active observation is an empirical question that current vision-language benchmarks do not answer. We introduce ActiveVision, a benchmark that makes active observation measurable for MLLMs, comprising 17 tasks across 3 categories. Tasks are designed to force repeated visual perception rather than a single static description. Frontier MLLMs collapse on ActiveVision: the highest-scoring model we evaluate, GPT-5.5 at the highest exposed reasoning-effort tier, solves only 10.6% of items and scores zero on 11 of the 17 tasks, and even Claude Fable 5, despite topping most reasoning and coding leaderboards, solves just 3.5%, far behind three human participants who average 96.1%. Furthermore, much of the gap persists even when models write and run their own vision code: such code is unreliable on realistic imagery, and catching its failures itself requires the active perception the models lack. Together, these results indicate that current MLLMs lack robust active visual observation, motivating architectures and training objectives that close the perception-reasoning loop.

Figures

Figures reproduced from arXiv: 2607.16165 by Jiarui Zhang, Muzi Tao, Ollie Liu, Shangshang Wang, Willie Neiswanger, Xuezhe Ma.

Figure 1
Figure 1. Figure 1: Seeing is not always a single-glance task. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of all 17 tasks in ActiveVision, grouped into three task families that probe distinct dimensions of active observation. For each task, we show one photorealistic instance produced by the rendering pipeline described in §3.3, together with a brief description of the required operation. benchmarks do not explicitly name. Its tasks resist shortcuts from any single language description and are rendere… view at source ↗
Figure 3
Figure 3. Figure 3: The four-stage generation pipeline, illustrated on Tangled Loop Counting. A deterministic Python generator (1) emits the geometric scaffold (2): random smooth closed curves with full ground truth. A task-specific GPT-image-2 prompt (3) re-renders the scaffold into a photorealistic image (4) without altering positions, counts, or topology. The benchmark question and ground-truth answer apply unchanged to bo… view at source ↗
Figure 4
Figure 4. Figure 4: Accuracy versus API cost per item (log scale, public list prices) on the 85-item ActiveVision split. Points are complete pure-CoT runs, with marker size encoding the reasoning-effort tier; lines connect runs of the same model. The dashed line and band are the human mean and range (N=3). 0 5 10 15 20 ground-truth count 0 5 10 15 20 predicted count slope = 0.69 y = x GPT-5.5 xhigh 0 5 10 15 20 ground-truth c… view at source ↗
Figure 5
Figure 5. Figure 5: Predicted versus ground-truth counts across eight counting tasks at each model’s highest reasoning effort. Solid lines are least-squares fits; gray dashed lines show the identity y = x. Slopes below one reveal increasing undercounting as the true count grows. 0 1 2 3 4 5 6 7 8 9 10 steps into the walk (k) 0 20% 40% 60% 80% 100% share of walks with no error yet Color Zone Sequencing Line Intersection Sequen… view at source ↗
Figure 7
Figure 7. Figure 7: Agentic tool use on the 85-item ActiveVision split. (a) Per-agent accuracy against the human reference (N=3); (b) per-task breakdown across the three agents. “Same” becomes the safe default in visual comparison [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Four examples illustrating two failure modes of agentic tool use, shown through the agents’ own intermediate artifacts and verbatim rollout excerpts. For each failure mode, the upper example is from Codex and the lower example is from Claude Code. 11 [PITH_FULL_IMAGE:figures/full_fig_p011_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 20 linked inside Pith

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Active vision.International journal of computer vision, 1(4):333–356, 1988

    John Aloimonos, Isaac Weiss, and Amit Bandyopadhyay. Active vision.International journal of computer vision, 1(4):333–356, 1988

    1988

  3. [3]

    Introducingclaudeopus4.8

    Anthropic. Introducingclaudeopus4.8. https://www.anthropic.com/news/claude-opus-4-8,

  4. [4]

    Introducing claude fable 5 and claude mythos 5.https://www.anthropic.com/news/ claude-fable-5-mythos-5, 2026

    Anthropic. Introducing claude fable 5 and claude mythos 5.https://www.anthropic.com/news/ claude-fable-5-mythos-5, 2026. Accessed: 2026-07-01

    2026

  5. [5]

    Active perception.Proceedings of the IEEE, 76(8):966–1005, 1988

    Ruzena Bajcsy. Active perception.Proceedings of the IEEE, 76(8):966–1005, 1988

    1988

  6. [6]

    Dana H. Ballard. Animate vision.Artificial Intelligence, 48(1):57–86, 1991

    1991

  7. [7]

    Ballard, Mary M

    Dana H. Ballard, Mary M. Hayhoe, and Jeff B. Pelz. Memory representations in natural tasks.Journal of Cognitive Neuroscience, 7(1):66–80, 1995

    1995

  8. [8]

    Consis- tency and variability among latent print examiners as revealed by eye tracking methodologies.Journal of Forensic Identification, 61(1):60–91, 2011

    Thomas Busey, Chen Yu, Dean Wyatte, John Vanderkolk, Francisco Parada, and Ruj Akavipat. Consis- tency and variability among latent print examiners as revealed by eye tracking methodologies.Journal of Forensic Identification, 61(1):60–91, 2011

    2011

  9. [9]

    Babyvision: Visual reasoning beyond language.arXiv preprint arXiv:2601.06521, 2026

    Liang Chen, Weichu Xie, Yiyan Liang, Hongfeng He, Hans Zhao, et al. Babyvision: Visual reasoning beyond language.arXiv preprint arXiv:2601.06521, 2026

    Pith/arXiv arXiv 2026

  10. [10]

    Are we on the right way for evaluating large vision-language models?arXiv preprint arXiv:2403.20330, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models?arXiv preprint arXiv:2403.20330, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Easy for children, hard for ai: The limits of mllms in early learning tasks recognition

    ChildBench Team. Easy for children, hard for ai: The limits of mllms in early learning tasks recognition. https://github.com/Jderder/ChildBench, 2025

    2025

  12. [12]

    Colin G. Drury. Inspection performance. In Gavriel Salvendy, editor,Handbook of Industrial Engineering, pages 2282–2314. Wiley, 2nd edition, 1992

    1992

  13. [13]

    Findlay and Iain D

    John M. Findlay and Iain D. Gilchrist.Active Vision: The Psychology of Looking and Seeing. Oxford University Press, 2003

    2003

  14. [14]

    Blink: Multimodal large language models can see but not perceive

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404.12390, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Expertise differences in the comprehension of visualizations: A meta-analysis of eye-tracking research in professional domains.Educational Psychology Review, 23(4):523–552, 2011

    Andreas Gegenfurtner, Erno Lehtinen, and Roger Säljö. Expertise differences in the comprehension of visualizations: A meta-analysis of eye-tracking research in professional domains.Educational Psychology Review, 23(4):523–552, 2011

    2011

  16. [16]

    Gemini 3.1 pro: A smarter model for your most complex tasks.https://blog

    Google DeepMind. Gemini 3.1 pro: A smarter model for your most complex tasks.https://blog. google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-pro/ ,

  17. [17]

    Gemini 3.5: Frontier intelligence with action

    Google DeepMind. Gemini 3.5: Frontier intelligence with action. https://blog.google/ innovation-and-ai/models-and-research/gemini-models/gemini-3-5/ , 2026. Ac- cessed: 2026-07-16

    2026

  18. [18]

    14 An Exam for Active Observers

    Accessed: 2026-05-05. 14 An Exam for Active Observers

    2026

  19. [19]

    FORGE: Fine-grained multimodal evaluation for manufacturing scenarios.arXiv preprint arXiv:2604.07413, 2026

    Xiangru Jian, Hao Xu, Wei Pang, Xinjian Zhao, Chengyu Tao, Qixin Zhang, Xikun Zhang, Chao Zhang, Guanzhi Deng, Alex Xue, Juan Du, Tianshu Yu, Garth Tarr, Linqi Song, Qiuzhuang Sun, and Dacheng Tao. FORGE: Fine-grained multimodal evaluation for manufacturing scenarios.arXiv preprint arXiv:2604.07413, 2026

    Pith/arXiv arXiv 2026

  20. [20]

    Eye movements in natural behavior.Trends in Cognitive Sciences, 9(4): 188–194, 2005

    Mary Hayhoe and Dana Ballard. Eye movements in natural behavior.Trends in Cognitive Sciences, 9(4): 188–194, 2005

    2005

  21. [21]

    Curve tracing: A possible basic operation in the perception of spatial relations.Memory & Cognition, 14(2):129–140, 1986

    Pierre Jolicoeur, Shimon Ullman, and Marilynne Mackay. Curve tracing: A possible basic operation in the perception of spatial relations.Memory & Cognition, 14(2):129–140, 1986

    1986

  22. [22]

    MAC: A live benchmark for multimodal large language models in scientific understanding.arXiv preprint arXiv:2508.15802, 2025

    Mohan Jiang, Jin Gao, Jiahao Zhan, and Dequan Wang. MAC: A live benchmark for multimodal large language models in scientific understanding.arXiv preprint arXiv:2508.15802, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    Prismatic VLMs: Investigating the design space of visually-conditioned language models.arXiv preprint arXiv:2402.07865, 2024

    Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic VLMs: Investigating the design space of visually-conditioned language models.arXiv preprint arXiv:2402.07865, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    BRAVE: Broadening the visual encoding of vision-language models.arXiv preprint arXiv:2404.07204, 2024

    Oğuzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, and Federico Tombari. BRAVE: Broadening the visual encoding of vision-language models.arXiv preprint arXiv:2404.07204, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    cadrille: Multi-modal CAD reconstruction with reinforcement learning

    Maksim Kolodiazhnyi, Denis Tarasov, Dmitrii Zhemchuzhnikov, Alexander Nikulin, Ilya Zisman, Anna Vorontsova, Anton Konushin, Vladislav Kurenkov, and Danila Rukhovich. cadrille: Multi-modal CAD reconstruction with reinforcement learning. InInternational Conference on Learning Representations (ICLR), 2026. arXiv:2505.22914

    arXiv 2026

  26. [26]

    E. L. Kaufman, M. W. Lord, T. W. Reese, and J. Volkmann. The discrimination of visual number.The American Journal of Psychology, 62(4):498–525, 1949

    1949

  27. [27]

    Kundel, Calvin F

    Harold L. Kundel, Calvin F. Nodine, and Dennis Carmody. Visual scanning, pattern recognition and decision-making in pulmonary nodule detection.Investigative Radiology, 13(3):175–181, 1978

    1978

  28. [28]

    Krupinski, Allison A

    Elizabeth A. Krupinski, Allison A. Tillack, Lynne Richter, Jeffrey T. Henderson, Achyut K. Bhattacharyya, Katherine M. Scott, Anna R. Graham, Michael R. Descour, John R. Davis, and Ronald S. Weinstein. Eye-movement study and human performance using telepathology virtual slides: Implications for medical education and differences with experience.Human Patho...

    2006

  29. [29]

    Luck and Edward K

    Steven J. Luck and Edward K. Vogel. The capacity of visual working memory for features and conjunc- tions.Nature, 390(6657):279–281, 1997

    1997

  30. [30]

    White, and Samuel G

    JonM.Laurent, JosephD.Janizek, MichaelRuzo, MichaelaM.Hinks, MichaelJ.Hammerling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D. White, and Samuel G. Rodriques. LAB-Bench: Measuring capabilities of language models for biology research.arXiv preprint arXiv:2407.10362, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    Mm1: Methods, analysis & insights from multimodal llm pre-training.arXiv preprint arXiv:2403.09611, 2024

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mm1: Methods, analysis & insights from multimodal llm pre-training.arXiv preprint arXiv:2403.09611, 2024

    Pith/arXiv arXiv 2024

  32. [32]

    RoboBench: Acomprehensiveevaluationbenchmarkformultimodallargelanguagemodelsasembodied brain.arXiv preprint arXiv:2510.17801, 2025

    Yulin Luo, Chun-Kai Fan, Menghang Dong, Jiayu Shi, Mengdi Zhao, Bo-Wen Zhang, Cheng Chi, Jiaming Liu, Gaole Dai, Rongyu Zhang, Ruichuan An, Kun Wu, Zhengping Che, Shaoxuan Xie, Guocai Yao, 15 An Exam for Active Observers Zhongxia Zhao, Pengwei Wang, Guang Liu, Zhongyuan Wang, Tiejun Huang, and Shanghang Zhang. RoboBench: Acomprehensiveevaluationbenchmarkf...

    Pith/arXiv arXiv 2025

  33. [33]

    ChatGPT images 2.0 (GPT-image-2)

    OpenAI. ChatGPT images 2.0 (GPT-image-2). https://openai.com/index/ introducing-chatgpt-images-2-0/, 2026

    2026

  34. [34]

    Introducing gpt-5.5.https://openai.com/index/introducing-gpt-5-5/, 2026

    OpenAI. Introducing gpt-5.5.https://openai.com/index/introducing-gpt-5-5/, 2026. Ac- cessed: 2026-05-05

    2026

  35. [35]

    Blueprint-Bench: Comparing spatial intelligence of LLMs, agents and image models.arXiv preprint arXiv:2509.25229, 2025

    Lukas Petersson, Axel Backlund, Axel Wennstöm, Hanna Petersson, Callum Sharrock, and Arash Dabiri. Blueprint-Bench: Comparing spatial intelligence of LLMs, agents and image models.arXiv preprint arXiv:2509.25229, 2025

    arXiv 2025

  36. [36]

    Kevin O’Regan and Alva Noë

    J. Kevin O’Regan and Alva Noë. A sensorimotor account of vision and visual consciousness.Behavioral and Brain Sciences, 24(5):939–973, 2001

    2001

  37. [37]

    Roelfsema

    Pieter R. Roelfsema. Elemental operations in vision.Trends in Cognitive Sciences, 9(4):226–233, 2005

    2005

  38. [38]

    Vision language models are blind.arXiv preprint arXiv:2407.06581, 2024

    Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen. Vision language models are blind.arXiv preprint arXiv:2407.06581, 2024

    Pith/arXiv arXiv 2024

  39. [39]

    Scheffer, C

    Louis K. Scheffer, C. Shan Xu, Michal Januszewski, et al. A connectome and analysis of the adult Drosophila central brain.eLife, 9:e57443, 2020

    2020

  40. [40]

    Roelfsema, Victor A

    Pieter R. Roelfsema, Victor A. F. Lamme, and Henk Spekreijse. Object-based attention in the primary visual cortex of the macaque monkey.Nature, 395(6700):376–381, 1998

    1998

  41. [41]

    What does a visual formal analysis of the world’s 500 most famous paintings tell us about multimodal llms? InThe Second Tiny Papers Track at ICLR 2024, 2024

    Muzi Tao and Saining Xie. What does a visual formal analysis of the world’s 500 most famous paintings tell us about multimodal llms? InThe Second Tiny Papers Track at ICLR 2024, 2024

    2024

  42. [42]

    Eagle: Exploring the design space for multimodal llms with mixture of encoders.arXiv preprint arXiv:2408.15998, 2024

    Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, et al. Eagle: Exploring the design space for multimodal llms with mixture of encoders.arXiv preprint arXiv:2408.15998, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024

    2024

  44. [44]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms.arXiv preprint arXiv:2406.16860, 2024

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms.arXiv preprint arXiv:2406.16860, 2024

    Pith/arXiv arXiv 2024

  45. [45]

    Visual routines.Cognition, 18(1–3):97–159, 1984

    Shimon Ullman. Visual routines.Cognition, 18(1–3):97–159, 1984. 16 An Exam for Active Observers

    1984

  46. [46]

    Trick and Zenon W

    Lana M. Trick and Zenon W. Pylyshyn. Why are small and large numbers enumerated differently? A limited-capacity preattentive stage in vision.Psychological Review, 101(1):80–102, 1994

    1994

  47. [47]

    CharXiv: Charting gaps in realistic chart understanding in multimodal LLMs

    Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora, and Danqi Chen. CharXiv: Charting gaps in realistic chart understanding in multimodal LLMs. InAdvances in Neural Information Processing Systems (NeurIPS) – Datasets and Benchmarks Track, volume 37, p...

    Pith/arXiv arXiv 2024

  48. [48]

    Vision language models are biased.arXiv preprint arXiv:2505.23941, 2025

    An Vo, Khai-Nguyen Nguyen, Mohammad Reza Taesiri, Vy Tuong Dang, Anh Totti Nguyen, and Daeyoung Kim. Vision language models are biased.arXiv preprint arXiv:2505.23941, 2025

    Pith/arXiv arXiv 2025

  49. [49]

    Children’s intelligence tests pose challenges for mllms? kidgym: A 2d grid-based reasoning benchmark for mllms

    Hengwei Ye, Yuanting Guan, Yuxuan Ge, Tianying Zhu, Zhenhan Guan, Yijia Zhong, et al. Children’s intelligence tests pose challenges for mllms? kidgym: A 2d grid-based reasoning benchmark for mllms. arXiv preprint arXiv:2603.20209, 2026

    arXiv 2026

  50. [50]

    Yarbus.Eye Movements and Vision

    Alfred L. Yarbus.Eye Movements and Vision. Plenum Press, New York, 1967

    1967

  51. [51]

    MMMU-Pro: A more robust multi-discipline multimodal understanding benchmark

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. MMMU-Pro: A more robust multi-discipline multimodal understanding benchmark. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15...

    2025

  52. [52]

    OSWorld2.0: Benchmarking computer use agents on long-horizon real-world tasks.arXiv preprint arXiv:2606.29537, 2026

    Mengqi Yuan, Zilong Zhou, Xinzhuang Xiong, Weiming Wu, Jiayang Sun, Jiamin Song, Tianbao Xie, Tao Yu, et al. OSWorld2.0: Benchmarking computer use agents on long-horizon real-world tasks.arXiv preprint arXiv:2606.29537, 2026

    Pith/arXiv arXiv 2026

  53. [53]

    Euclid: Supercharging multimodal llms with synthetic high-fidelity visual descriptions.arXiv preprint arXiv:2412.08737, 2024

    Jiarui Zhang, Ollie Liu, Tianyu Yu, Jinyi Hu, and Willie Neiswanger. Euclid: Supercharging multimodal llms with synthetic high-fidelity visual descriptions.arXiv preprint arXiv:2412.08737, 2024

    Pith/arXiv arXiv 2024

  54. [54]

    BenchCAD: A comprehensive, industry-standard benchmark for programmatic CAD.arXiv preprint arXiv:2605.10865, 2026

    Haozhe Zhang, Kaichen Liu, Miaomiao Chen, Lei Li, Shaojie Yang, Cheng Peng, and Hanjie Chen. BenchCAD: A comprehensive, industry-standard benchmark for programmatic CAD.arXiv preprint arXiv:2605.10865, 2026

    Pith/arXiv arXiv 2026

  55. [56]

    MLLMs know where to look: Training-free perception of small visual details with multimodal llms.arXiv preprint arXiv:2502.17422, 2025

    Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip Ilievski. MLLMs know where to look: Training-free perception of small visual details with multimodal llms.arXiv preprint arXiv:2502.17422, 2025. 17

    Pith/arXiv arXiv 2025

  56. [2026]

    Accessed: 2026-07-16

    2026