pith. sign in

arxiv: 2605.28818 · v1 · pith:FLBJHV4Qnew · submitted 2026-05-27 · 💻 cs.CL · q-bio.NC

VLMs May Not Globally Enhance Human Alignment over LLMs During Natural Reading

Jinzhou Wu , Zhengwu Ma , Jixing Li , Baoping Tang , Zitong Lu This is my paper

Pith reviewed 2026-06-29 12:50 UTC · model grok-4.3

classification 💻 cs.CL q-bio.NC
keywords vision language modelslarge language modelshuman alignmentnatural readingfMRIeye trackingmultimodal pretraining
0
0 comments X

The pith

Vision-language models do not provide a uniform global advantage over language models in aligning with human brain and eye responses during natural reading.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper compares matched pairs of language models and vision-language models, both run on text only, against human data from natural reading. It finds that multimodal pretraining history does not improve alignment uniformly across the brain or eye movements. Alignment differences appear only selectively in sentences with more visual content. This suggests that internal language representations matter more than added visual training for modeling how people process text. A sympathetic reader would care because it refines how we build computational models of human language use.

Core claim

By evaluating tightly matched LLM and VLM pairs under a strictly text-only setting on a human natural-reading dataset with whole-cortex fMRI and eye-tracking, the paper shows that multimodal pretraining may not confer a uniform global advantage in human alignment during natural reading. Language-internal representations remain the key factor, though VLM advantages can emerge selectively for sentences with stronger visual semantic content, with evidence from both fMRI and eye-movement measures.

What carries the argument

Tightly matched LLM-VLM pairs evaluated text-only against fMRI and eye-tracking data from natural reading, isolating multimodal training history effects.

Load-bearing premise

The chosen human natural-reading fMRI and eye-tracking dataset, along with the matched model pairs, can reliably detect global versus selective differences in alignment.

What would settle it

Finding a consistent global VLM advantage across all sentences or brain regions in the same dataset would contradict the claim of no uniform advantage.

Figures

Figures reproduced from arXiv: 2605.28818 by Baoping Tang, Jinzhou Wu, Jixing Li, Zhengwu Ma, Zitong Lu.

Figure 1
Figure 1. Figure 1: Overview of the matched LLM/VLM alignment pipeline. (a) Data acquisition. Participants read sentences [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Global fMRI alignment of different LLM–VLM pairs. Significant brain clusters from the single-model [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Eye-movement alignment for regressive sac [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Visual strength modulation of VLM–LLM alignment across matched model pairs. (a) Mean advantage in [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Subject-level LLM–VLM consistency for eye-movement alignment. Each point represents one participant; [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Subject-level LLM–VLM consistency for fMRI alignment. Format follows Figure [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Distribution of sentence-level visual strength [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: System prompt used for LLM-based visual strength scoring. The same prompt was used for all three [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: User prompt used for LLM-based visual strength scoring. The placeholder {sentence} was re￾placed with the target sentence at inference time. following fields: scene_content, visual_detail, im￾ageability and confidence. The raw judge score for model m on sentence i was computed as q (m) i = q (m) i,scene + q (m) i,detail + q (m) i,imagery 3 . (10) D.2.4 API Settings All judges were queried in text-only mode… view at source ↗
Figure 10
Figure 10. Figure 10: Example structured JSON output from one LLM judge for a single sentence. The three scoring dimensions are accompanied by a confidence rating [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗
read the original abstract

Large language models (LLMs) have become increasingly useful computational models of human language processing, but it remains unclear whether vision-language learning makes text representations more human-like during natural reading. Here, we address this question by comparing tightly matched LLM and vision-language model (VLM) pairs under a strictly text-only setting, allowing us to isolate the effect of multimodal training history from online visual input or cross-modal fusion. We evaluate model alignment with a human natural-reading dataset that includes whole-cortex fMRI responses and synchronized eye-tracking saccades. Our findings demonstrate that multimodal pretraining may not confer a uniform, global advantage in human alignment during natural reading, indicating that language-internal representations remain the key factor for modeling human text processing. However, the VLM advantage could emerge more selectively when sentences contain stronger visual semantic content, with converging evidence from both fMRI and eye-movement alignments. Together, our findings provide a controlled in silico framework for testing how visual learning history shapes model-human alignment of language processing, suggesting that multimodal pretraining contributes selectively rather than globally to human-like language representations during natural reading.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 1 minor

Summary. The manuscript claims that comparing tightly matched LLM-VLM pairs under strictly text-only input on a natural-reading dataset (whole-cortex fMRI and synchronized eye-tracking) shows multimodal pretraining confers no uniform global advantage in human alignment. Any VLM benefit appears selective to sentences with stronger visual semantic content, implying language-internal representations are the primary driver of human-like text processing. Converging evidence is reported from both modalities.

Significance. If the isolation of multimodal history holds, the result would indicate that vision-language pretraining does not globally improve alignment with human brain responses or eye movements beyond language modeling alone. This has implications for cognitive modeling of language and for when multimodal data might be expected to help. The use of two independent human modalities (fMRI and eye-tracking) is a methodological strength that could support selective rather than global effects if the model controls are adequate.

major comments (1)
  1. [Methods (model pairs)] Methods section (model-pair construction): The central claim requires that observed alignment patterns can be attributed specifically to the presence/absence of multimodal pretraining. The manuscript describes the pairs as 'tightly matched' under text-only input, but does not supply explicit verification that text-only pretraining corpora, tokenizers, parameter counts, and optimization schedules are identical. Without this, residual differences could explain the lack of global VLM advantage, directly weakening the inference that 'language-internal representations remain the key factor.'
minor comments (1)
  1. [Abstract] Abstract and §1: The criteria used to declare pairs 'tightly matched' should be stated more explicitly (e.g., exact architecture, data overlap, training steps) so readers can evaluate the isolation of the multimodal variable.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback. The single major comment concerns the need for explicit verification of model-pair matching. We address this below and will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: Methods section (model-pair construction): The central claim requires that observed alignment patterns can be attributed specifically to the presence/absence of multimodal pretraining. The manuscript describes the pairs as 'tightly matched' under text-only input, but does not supply explicit verification that text-only pretraining corpora, tokenizers, parameter counts, and optimization schedules are identical. Without this, residual differences could explain the lack of global VLM advantage, directly weakening the inference that 'language-internal representations remain the key factor.'

    Authors: We agree that explicit verification strengthens the attribution to multimodal pretraining history. The pairs were chosen from publicly released checkpoints selected for closest architectural and scale similarity (e.g., same base transformer family and comparable parameter counts). Complete pretraining corpora and optimization schedules are not always fully public, which limits perfect matching. In revision we will add a dedicated Methods subsection with a comparison table of known attributes (parameter counts, tokenizers, cited original papers) and explicitly discuss any residual differences as a limitation. This addresses the concern without changing the core results, which rest on converging fMRI and eye-tracking evidence. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical model comparison with independent measurements

full rationale

The paper performs an empirical comparison of tightly matched LLM-VLM pairs on alignment with human fMRI and eye-tracking data during natural reading. No derivations, first-principles predictions, fitted parameters renamed as predictions, or self-citation chains are present in the abstract or described methodology. The central claim rests on observable differences in alignment metrics, which are externally falsifiable against the human dataset and do not reduce to the inputs by construction. This matches the default expectation for non-circular empirical studies.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No mathematical axioms, free parameters, or invented entities are introduced; the work relies on existing models, public human datasets, and standard evaluation practices.

pith-pipeline@v0.9.1-grok · 5734 in / 1139 out tokens · 41285 ms · 2026-06-29T12:50:41.379549+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

9 extracted references · 3 canonical work pages · 2 internal anchors

  1. [1]

    Kimi K2.5: Visual Agentic Intelligence

    Scaling laws for language encoding models in fMRI. InThirty-Seventh Conference on Neural Information Processing Systems. Khai Loong Aw, Syrielle Montariol, Badr AlKhamissi, Martin Schrimpf, and Antoine Bosselut. 2024. Instruction-tuning Aligns LLMs to the Human Brain. InFirst Conference on Language Modeling. Tongtong Bai, Yifan Bai, Yiping Bao, S. H. Cai,...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  2. [2]

    arXiv preprint arXiv:2405.02246 , year=

    Do Multimodal Large Language Models and Humans Ground Language Similarly?Computa- tional Linguistics, 50(4):1415–1440. Tatsuki Kuribayashi, Yohei Oseki, and Timothy Bald- win. 2024. Psychometric Predictive Power of Large Language Models. InFindings of the Association for Computational Linguistics: NAACL 2024, pages 1983–2005, Mexico City, Mexico. Associat...

    work page arXiv 2024

  3. [3]

    Instruction-Tuned Video-Audio Models Eluci- date Functional Specialization in the Brain.Preprint, arXiv:2506.08277. Sara F. Popham, Alexander G. Huth, Natalia Y . Bilenko, Fatma Deniz, James S. Gao, Anwar O. Nunez- Elizalde, and Jack L. Gallant. 2021. Visual and lin- guistic semantic representations are aligned at the border of human visual cortex.Nature ...

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  4. [4]

    Use only the sentence itself

  5. [5]

    Ignore article context, discourse context, and task context

  6. [6]

    Do NOT rate general concreteness, emotional intensity, familiarity, importance, or linguistic complexity

  7. [7]

    Use ordinary lexical semantics only; do not enrich the sentence with elaborate world knowledge

  8. [8]

    Return JSON only, matching the required schema exactly

  9. [9]

    {sentence}

    Use a 0.0 to 5.0 scale, with at most one decimal place. Definitions: •scene_content: how much the sentence explicitly evokes visible entities, objects, scenes, or spatial layout. •visual_detail: how much the sentence contains visually discriminable attributes such as shape, color, appearance, motion, or spatial relations. •imageability: how easy it is to ...

    2026