pith. sign in

arxiv: 2606.17410 · v1 · pith:EX7554VVnew · submitted 2026-06-16 · 💻 cs.CV

Attention Alignment Between Humans and Vision-Language Models

Isaac R. Christian , Udith Haputhanthrige , Hanna Hornfeld , Declan Campbell , Samuel Nastase , Taylor Webb , Michael Graziano This is my paper

Pith reviewed 2026-06-27 02:10 UTC · model grok-4.3

classification 💻 cs.CV
keywords vision-language modelsattention alignmenthuman fixationsLSTM decoderTransformer decoderspatial attention mapsvisual perceptioneye tracking
0
0 comments X

The pith

LSTM decoders in vision-language models produce attention maps that align with human fixations at 80-87 percent of the noise ceiling, far above the 40-59 percent for Transformer decoders.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper compares spatial attention maps from six vision-language models to human eye fixation heatmaps collected on the same 200 images under two captioning tasks. Models vary in a 2x2 design of CNN or ViT encoders crossed with LSTM or Transformer decoders, plus two larger models. Decoder architecture turns out to drive most of the alignment difference, with LSTM versions reaching 80-87 percent of the human noise ceiling and Transformer versions only 40-59 percent. Encoder choice adds a smaller 5-20 point effect favoring CNNs. The work also reports that LSTM maps remain diffuse and task-insensitive while Transformer maps concentrate sharply, and that the two architectures trade off in how well they predict synthetic neural responses.

Core claim

Decoder choice dominates alignment between vision-language model attention maps and human fixation heatmaps, with LSTM decoders achieving 80-87 percent of the human noise ceiling compared to 40-59 percent for Transformer decoders; CNN-LSTM models perform best overall, LSTM maps are spatially diffuse with minimal task differentiation, ViT-Transformer maps are the sharpest yet least aligned, ablating attention impacts LSTM decoders more in a hemispatial-neglect test, and CNN-Transformer maps better predict synthetic brain activity despite lower fixation alignment.

What carries the argument

Spatial attention maps extracted from the internal states of vision-language models, compared directly to human fixation heatmaps on identical images and tasks.

If this is right

  • LSTM-decoder attention maps remain spatially diffuse and show little differentiation between general description and social captioning tasks.
  • A hemispatial-neglect simulation that removes half the visual field impairs performance of LSTM-decoder models more than Transformer-decoder models.
  • CNN-Transformer attention maps predict synthetic neural responses better than LSTM maps despite lower alignment with human fixations.
  • Attention maps from all models align most strongly with early visual cortex responses in the synthetic neural data.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The dissociation between fixation alignment and synthetic neural prediction implies that different model components may capture distinct aspects of human visual processing.
  • If decoder architecture controls alignment so strongly, hybrid designs that retain LSTM-style recurrence in the decoder while using modern encoders could improve human-like attention behavior.
  • Task differentiation in attention may matter more for real-world applications than raw alignment scores alone.

Load-bearing premise

The assumption that attention maps extracted from vision-language model internals are directly comparable to human fixation heatmaps as measures of attention.

What would settle it

Extracting attention maps from the same models using an alternative method such as gradient-based saliency and obtaining reversed alignment rankings between LSTM and Transformer decoders would falsify the decoder-dominance claim.

Figures

Figures reproduced from arXiv: 2606.17410 by Declan Campbell, Hanna Hornfeld, Isaac R. Christian, Michael Graziano, Samuel Nastase, Taylor Webb, Udith Haputhanthrige.

Figure 1
Figure 1. Figure 1: (A) Describe task alignment. Model attention maps were correlated to human fixation patterns and correlations were averaged across images and participants. (B) The same as (A) but for the Social task. (C) Between-task spatial alignment. Correlations were calculated between each agent’s attention maps on Social vs. Describe images. All bars: % of human-human noise ceiling for panels A–B; raw Spearman r for … view at source ↗
Figure 2
Figure 2. Figure 2: Attention map examples for all models on three images and two tasks. Top rows: Social [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Caption similarity (vs. unablated baseline) under progressive left-side spatial ablation [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: (A) Whole-brain mean R2 for a banded ridge encoding model predicting TRIBE-simulated neural responses from each agent’s describe- and social-task attention maps. (B) Variance partition for CNN-Transformer on the social task, thresholded at q < 0.05 for display. Rows: unique attention R2 (“Where”), unique CLIP R2 (“What”), and shared R2 . Four cortical views per row (lateral left, medial left, lateral right… view at source ↗
read the original abstract

Visual perception depends on top-down goals and bottom-up sensory mechanisms. Vision-language models implement both, allowing us to treat each component as a separable hypothesis about what drives where we look. We compared spatial attention maps from six vision-language models against human fixation heatmaps recorded on 200 images during two tasks (general description and social captioning). The six models spanned a 2$\times$2 factorial of CNN vs.\ ViT encoders crossed with LSTM vs.\ Transformer decoders, plus Molmo 7B-D and Qwen3.5 9B. We found that both decoder and encoder architecture shaped alignment, but decoder choice dominated. LSTM vs.\ Transformer decoders increased alignment by 40--50 percentage points (80--87\% vs.\ 40--59\% of the human noise ceiling). In contrast, CNN vs.\ ViT encoders contributed a secondary 5--20 point advantage depending on decoder family, with CNN-LSTM the most aligned model overall (85--87\%). Despite their alignment advantage, LSTM-decoder attention maps were spatially diffuse and minimally task-differentiated; ViT-Transformer, the weakest in alignment, showed the sharpest spatial concentration and strongest task differentiation. A hemispatial-neglect simulation confirmed that ablating attention impacted LSTM decoders more than Transformer decoders. In an exploratory extension using TRIBE-simulated synthetic neural responses, fixation alignment and neural relevance dissociate: CNN-Transformer attention maps better predicted synthetic brain activity despite lower fixation alignment, with attention maps best predicting early visual cortex. Together, top-down and bottom-up components trade off what they predict in behavioral and synthetic neural data.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript compares spatial attention maps extracted from six vision-language models (a 2×2 design of CNN vs ViT encoders with LSTM vs Transformer decoders, plus Molmo 7B-D and Qwen3.5 9B) against human fixation heatmaps recorded on 200 images during two tasks (general description and social captioning). It claims decoder architecture dominates alignment, with LSTM decoders reaching 80–87% of the human noise ceiling versus 40–59% for Transformers (CNN-LSTM highest overall at 85–87%), while noting that LSTM maps are diffuse and minimally task-differentiated; additional results include a hemispatial-neglect ablation and a TRIBE-based dissociation between fixation alignment and synthetic neural predictivity (CNN-Transformer better for early visual cortex).

Significance. If the attention-map extraction and comparability assumptions hold, the work identifies a clear architectural trade-off: decoder choice strongly modulates behavioral alignment with human fixations, yet LSTM advantages come with reduced spatial specificity and task sensitivity, while also dissociating from neural relevance in synthetic data. This could guide VLM design choices balancing human-like attention against other objectives.

major comments (2)
  1. [Methods (attention map extraction)] Methods (attention map extraction): The headline result—that LSTM decoders outperform Transformers by 40–50 percentage points—requires that spatial attention maps extracted from internal states are commensurable across the 2×2 architecture grid. LSTM attention is typically a single set of weights over image regions while Transformer attention involves multi-head, multi-layer aggregation; without identical extraction, projection to image space, and normalization procedures, the gap could arise from representational differences rather than genuine alignment with human fixations. The abstract notes diffuse LSTM maps still score higher, which is consistent with this extraction mismatch.
  2. [Abstract/Methods] Abstract/Methods: No details are provided on attention map extraction procedures, noise ceiling computation, statistical tests, or image selection criteria. This leaves the central quantitative claims (e.g., 80–87% vs 40–59% of noise ceiling) vulnerable to unstated methodological choices that could affect the decoder-dominance conclusion.
minor comments (1)
  1. [Abstract] Abstract: The two tasks are mentioned but it is unclear whether alignment metrics are reported separately or aggregated; adding this clarification would improve interpretability of the decoder and encoder effects.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. The two major comments both concern methodological transparency and the commensurability of attention maps across architectures. We address each point below and will revise the manuscript accordingly to strengthen the presentation of our methods and results.

read point-by-point responses

  1. Referee: [Methods (attention map extraction)] The headline result—that LSTM decoders outperform Transformers by 40–50 percentage points—requires that spatial attention maps extracted from internal states are commensurable across the 2×2 architecture grid. LSTM attention is typically a single set of weights over image regions while Transformer attention involves multi-head, multi-layer aggregation; without identical extraction, projection to image space, and normalization procedures, the gap could arise from representational differences rather than genuine alignment with human fixations. The abstract notes diffuse LSTM maps still score higher, which is consistent with this extraction mismatch.

    Authors: We agree that explicit documentation of the extraction pipeline is essential for interpreting the decoder-dominance result. In the current manuscript the Methods section outlines that LSTM attention weights were taken directly from the decoder's region-level attention and that Transformer maps were obtained by averaging the final-layer cross-attention heads before upsampling to image resolution; all maps were then L1-normalized and bilinearly resized to the same 224×224 grid. Nevertheless, the referee is correct that these steps are described at a high level. In revision we will add a dedicated subsection that specifies the exact layer(s) and head-aggregation rule for each Transformer model, the precise normalization (sum-to-one versus max-normalization), and the interpolation method, together with a supplementary figure illustrating one example map from each architecture family. This will allow readers to evaluate whether the observed 40–50-point gap reflects genuine alignment differences or extraction artifacts. revision: yes

  2. Referee: [Abstract/Methods] No details are provided on attention map extraction procedures, noise ceiling computation, statistical tests, or image selection criteria. This leaves the central quantitative claims (e.g., 80–87% vs 40–59% of noise ceiling) vulnerable to unstated methodological choices that could affect the decoder-dominance conclusion.

    Authors: We acknowledge that the current Methods section is insufficiently detailed on these points. The noise ceiling was computed via split-half Spearman-Brown corrected reliability on the human fixation data (200 images, two tasks), statistical comparisons used paired t-tests with FDR correction across the six models, and images were a random subset of MS-COCO validation images balanced for object and scene diversity. In the revised manuscript we will expand the Methods with: (i) a step-by-step description of attention-map extraction and normalization for every model, (ii) the exact formula and split used for the noise-ceiling estimate, (iii) the full statistical procedure including effect-size reporting, and (iv) the image-selection protocol with a supplementary table of image IDs. These additions will make the quantitative claims fully reproducible and directly address the referee's concern. revision: yes

Circularity Check

0 steps flagged

No circularity: direct empirical comparison of model outputs to external human data

full rationale

The paper performs an empirical comparison of spatial attention maps extracted from six VLMs (spanning CNN/ViT encoders and LSTM/Transformer decoders, plus two large models) against human fixation heatmaps on 200 images across two tasks. Alignment is quantified as percentage of human noise ceiling, with results showing decoder architecture dominating (LSTM 80-87% vs Transformer 40-59%). No mathematical derivation, parameter fitting, or self-referential definitions are present; the central claims rest on direct measurement against independent human behavioral data. The extraction of attention maps from internal states is an operational choice but does not reduce any result to a fitted input or self-definition by construction. No self-citation chains or uniqueness theorems are invoked as load-bearing. This is a standard self-contained empirical study.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The paper introduces no new free parameters or invented entities. It rests on standard domain assumptions about attention measurement.

axioms (1)
  • domain assumption Human fixation heatmaps serve as a valid proxy for spatial attention that can be directly compared to model attention maps
    The entire alignment analysis depends on this equivalence between behavioral data and model internals.

pith-pipeline@v0.9.1-grok · 5843 in / 1308 out tokens · 39140 ms · 2026-06-27T02:10:26.858736+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 5 canonical work pages · 3 internal anchors

  1. [1]

    1967 , publisher=

    Eye Movements and Vision , author=. 1967 , publisher=

    1967

  2. [2]

    Vision Research , volume=

    A saliency-based search mechanism for overt and covert shifts of visual attention , author=. Vision Research , volume=. 2000 , publisher=

    2000

  3. [3]

    IEEE Transactions on Pattern Analysis and Machine Intelligence , volume=

    A model of saliency-based visual attention for rapid scene analysis , author=. IEEE Transactions on Pattern Analysis and Machine Intelligence , volume=. 1998 , publisher=

    1998

  4. [4]

    Advances in Neural Information Processing Systems , volume=

    Graph-based visual saliency , author=. Advances in Neural Information Processing Systems , volume=

  5. [5]

    Huang, Xun and Shen, Chengyao and Boix, Xavier and Zhao, Qi , booktitle=

  6. [6]

    DeepGaze II: Reading fixations from deep features trained on object recognition

    K. arXiv preprint arXiv:1610.01563 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    IEEE Transactions on Pattern Analysis and Machine Intelligence , volume=

    What do different evaluation metrics tell us about saliency models? , author=. IEEE Transactions on Pattern Analysis and Machine Intelligence , volume=. 2019 , publisher=

    2019

  8. [8]

    International Conference on Machine Learning , pages=

    Show, attend and tell: Neural image caption generation with visual attention , author=. International Conference on Machine Learning , pages=

  9. [9]

    Predicting human eye fixations via an

    Cornia, Marcella and Baraldi, Lorenzo and Serra, Giuseppe and Cucchiara, Rita , journal=. Predicting human eye fixations via an. 2018 , publisher=

    2018

  10. [10]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

    Image captioning with semantic attention , author=. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

  11. [11]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

    Visual grounding via accumulated attention , author=. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

  12. [12]

    Do wide and deep networks learn the same things? Uncovering how neural network representations vary with width and depth

    Do wide and deep networks learn the same things? Uncovering how neural network representations vary with width and depth , author=. arXiv preprint arXiv:2010.15327 , year=

    work page arXiv 2010

  13. [13]

    Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages=

    Quantifying attention flow in transformers , author=. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages=

  14. [14]

    Advances in Neural Information Processing Systems , volume=

    Visual instruction tuning , author=. Advances in Neural Information Processing Systems , volume=

  15. [15]

    Bai, Jinze and Bai, Shuai and Yang, Shengding and Wang, Shijie and Tan, Sinan and Wang, Peng and Lin, Junyang and Zhou, Chang and Zhou, Jingren , journal=

  16. [16]

    Wang, Peng and Bai, Shuai and Tan, Sinan and Wang, Shijie and Fan, Zhihao and Bai, Jinze and Chen, Keqin and Liu, Xuejing and Wang, Jialin and Ge, Wenbin and others , journal=

  17. [17]

    Molmo and

    Deitke, Matt and Clark, Christopher and Lee, Sangho and Tripathi, Rohun and Yang, Yue and Park, Jae Sung and Salehi, Mohammadreza and Muennighoff, Niklas and Lo, Kyle and Soldaini, Luca and others , journal=. Molmo and

  18. [18]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

    Deep residual learning for image recognition , author=. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

  19. [19]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    An image is worth 16x16 words: Transformers for image recognition at scale , author=. arXiv preprint arXiv:2010.11929 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  20. [20]

    International Conference on Learning Representations , year=

    Neural machine translation by jointly learning to align and translate , author=. International Conference on Learning Representations , year=

  21. [21]

    Sentence-

    Reimers, Nils and Gurevych, Iryna , booktitle=. Sentence-

  22. [22]

    Nature Communications , volume=

    A large-scale examination of inductive biases shaping high-level visual representation in brains and machines , author=. Nature Communications , volume=. 2024 , publisher=

    2024

  23. [23]

    arXiv preprint arXiv:2105.07197 , year=

    Are convolutional neural networks or transformers more like human vision? , author=. arXiv preprint arXiv:2105.07197 , year=

    work page arXiv

  24. [24]

    Nature Machine Intelligence , volume=

    Learning high-level visual representations from a child's perspective without strong inductive biases , author=. Nature Machine Intelligence , volume=. 2024 , publisher=

    2024

  25. [25]

    bioRxiv , year=

    Universality of representation in biological and artificial neural networks , author=. bioRxiv , year=

  26. [26]

    Nature Machine Intelligence , pages=

    Convolutional architectures are cortex-aligned de novo , author=. Nature Machine Intelligence , pages=. 2025 , publisher=

    2025

  27. [27]

    2026 , howpublished=

    A foundation model of vision, audition, and language for in-silico neuroscience , author=. 2026 , howpublished=

    2026

  28. [28]

    The American Statistician , volume=

    Hierarchical partitioning , author=. The American Statistician , volume=. 1991 , publisher=

    1991

  29. [29]

    Advances in Neural Information Processing Systems , volume=

    Passive attention in artificial neural networks predicts human visual selectivity , author=. Advances in Neural Information Processing Systems , volume=

  30. [30]

    Trends in Cognitive Sciences , year=

    Better artificial intelligence does not mean better models of biology , author=. Trends in Cognitive Sciences , year=

  31. [31]

    Proceedings of the IEEE International Conference on Computer Vision , pages=

    Learning where humans look , author=. Proceedings of the IEEE International Conference on Computer Vision , pages=

  32. [32]

    Journal of Vision , volume=

    K. Journal of Vision , volume=. 2022 , publisher=

    2022

  33. [33]

    Cognitive Psychology , volume=

    A feature-integration theory of attention , author=. Cognitive Psychology , volume=. 1980 , publisher=

    1980

  34. [34]

    Nature Human Behaviour , volume=

    Five factors that guide attention in visual search , author=. Nature Human Behaviour , volume=. 2017 , publisher=

    2017

  35. [35]

    Frontiers in Computer Science , volume=

    Self-attention in vision transformers performs perceptual grouping, not attention , author=. Frontiers in Computer Science , volume=. 2023 , publisher=

    2023

  36. [36]

    Neural Networks , year=

    Emergence of human-like attention and distinct head clusters in self-supervised vision transformers: A comparative eye-tracking study , author=. Neural Networks , year=

  37. [37]

    Frontiers in Computational Neuroscience , volume=

    Attention in psychology, neuroscience, and machine learning , author=. Frontiers in Computational Neuroscience , volume=. 2020 , publisher=

    2020

  38. [38]

    2023 , eprint=

    Multimodality and Attention Increase Alignment in Natural Language Prediction Between Humans and Computational Models , author=. 2023 , eprint=

    2023

  39. [39]

    Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics , pages=

    Do Transformer Models Show Similar Attention Patterns to Task-Specific Human Gaze? , author=. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics , pages=

  40. [40]

    Chen, Shi and Jiang, Ming and Yang, Jinhui and Zhao, Qi , booktitle=

  41. [41]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Human Attention in Image Captioning: Dataset and Analysis , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  42. [42]

    Proceedings of the Conference on Empirical Methods in Natural Language Processing , pages=

    Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions? , author=. Proceedings of the Conference on Empirical Methods in Natural Language Processing , pages=

  43. [43]

    Proceedings of the 25th Conference on Computational Natural Language Learning , pages=

    Sood, Ekta and K. Proceedings of the 25th Conference on Computational Natural Language Learning , pages=

  44. [44]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

    Bottom-up and top-down attention for image captioning and visual question answering , author=. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

  45. [45]

    Nature Reviews Neuroscience , volume=

    Control of goal-directed and stimulus-driven attention in the brain , author=. Nature Reviews Neuroscience , volume=. 2002 , publisher=

    2002

  46. [46]

    Analysis of Visual Behavior , editor=

    Two cortical visual systems , author=. Analysis of Visual Behavior , editor=. 1982 , publisher=

    1982

  47. [47]

    Trends in Neurosciences , volume=

    Separate visual pathways for perception and action , author=. Trends in Neurosciences , volume=. 1992 , publisher=

    1992

  48. [48]

    Trends in Cognitive Sciences , volume=

    Beyond binding: from modular to natural vision , author=. Trends in Cognitive Sciences , volume=. 2025 , doi=

    2025

  49. [49]

    Microsoft

    Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll. Microsoft. European Conference on Computer Vision , pages=. 2014 , publisher=

    2014

  50. [50]

    Learning what and where to attend

    Learning what and where to attend , author=. arXiv preprint arXiv:1805.08819 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  51. [51]

    Journal of cognitive neuroscience , volume=

    Diverse deep neural networks all predict human inferior temporal cortex well, after training and fitting , author=. Journal of cognitive neuroscience , volume=. 2021 , publisher=

    2021

  52. [52]

    NeuroImage , volume=

    Feature-space selection with banded ridge regression , author=. NeuroImage , volume=. 2022 , publisher=

    2022

  53. [53]

    Nature Machine Intelligence , pages=

    Adopting a human developmental visual diet yields robust and shape-based AI vision , author=. Nature Machine Intelligence , pages=. 2026 , publisher=

    2026