REVIEW 4 major objections 4 minor 26 references
Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain)
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Instruction-tuned multimodal LLMs, prompted with natural-language instructions, predict human visual brain activity significantly better than vision-only models and on par with CLIP, with the prompting instruction selecting which visual…
desk verdict A solid, useful brain-encoding benchmark for instruction-tuned MLLMs, with a real but addressable confound in the instruction-specific steering claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the instruction-conditioned image representation: for each image and each of ten natural-language instructions, the model generates text tokens zero-shot, and the hidden states of those generated tokens are averaged and PCA-reduced to 1024 dimensions. That embedding is fed to voxel-wise bootstrap ridge regression encoding models trained to predict fMRI responses, with performance scored as Pearson correlation normalized by an estimated cross-subject prediction ceiling. Variance partitioning, which fits separate and joint encoding models for pairs of instructions and uses set arithmetic to split unique from shared explained variance, is what lets the paper claim that captioning subsumes most of the other instructions' signal. The instruction set itself, ten prompts borrowed from the MultiInstruct task taxonomy covering captioning, question answering, relationships, reasoning, understanding, and scene recognition, is the manipulator that makes a single model act as many task-specific encoders.
What would settle it
A control that would settle the claim: take the same images and models and replace the ten task instructions with ten length-matched, semantically empty prompts (for example, matched nonsense strings or unrelated sentences). If the instruction-specific differences in normalized brain alignment persist under nonsense prompts, then the task-specific alignment is driven by generation statistics or prompt wording rather than by the semantics of the instruction. A complementary check would record fMRI while participants actually perform each of the ten tasks; if the region-wise instruction alignment found under passive viewing does not sharpen under active task performance, the claim that these instructions capture task-specific neural processing would be weakened.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that prompting an instruction-tuned MLLM with different natural-language instructions changes the brain alignment of its representations in a structured way. Averaged over ten instructions, all three MLLMs significantly outperform randomly initialized versions and the unimodal ViT-H model, and perform comparably to CLIP-Text across the whole visual cortex and in five visual ROIs. The instruction that wins varies by region: image captioning aligns best in high-level visual areas (EBA, PPA, FFA), while image-understanding instructions such as 'describe the most dominant color' align best in early visual regions. Grouping instructions into visual concepts, the MLLMs capture count-related and recognition-related concepts with clear brain alignment, but color, positional understanding, and general scene understanding produce similar alignment patterns regardless of the specific concept. Variance partitioning shows that most of the explained variance is shared between image captioning and the other instructions, with captioning acting as an umbrella category; the authors interpret the large shared variance as identifying where instruction-tuned MLLMs could improve at differentiating tasks.
Load-bearing premise
The analysis assumes that averaging the hidden states of the text tokens an MLLM generates under a given instruction yields a faithful, instruction-specific representation of the image, so that differences in brain alignment across instructions reflect genuine task-specific neural processing rather than artifacts of generation length, language-model priors, or the exact wording of the ten chosen prompts.
Editorial extensions
If this is right
- Instruction-tuned MLLMs can be used off the shelf as brain encoders: one model, prompted with natural language, predicts visual fMRI as well as a model trained on ground-truth image-caption pairs, and no task-specific fine-tuning is needed.
- The instruction itself acts as a selection knob for brain regions: captioning-style prompts best predict high-level visual areas (EBA, PPA, FFA), while color-understanding prompts best predict early visual cortex.
- The large shared variance between image captioning and other instructions quantifies how much instruction-tuned MLLM task-following still collapses into generic captioning, giving a concrete metric for future instruction-tuning improvements.
- Because instruction-tuned models beat a non-instruction-tuned multimodal model (BLIP-2) at the same task, the brain-alignment gain is attributed to instruction tuning itself rather than to the underlying vision-language backbone.
Reading between the lines
- If instruction prompts genuinely select distinct neural computations, prompt variation becomes a cheap experimental probe: rotating instructions on a single MLLM could generate hypotheses about which cortical region computes which visual property, testable against targeted fMRI experiments.
- The shared-variance portrait suggests a testable design for the next generation of instruction-tuned models: training to minimize representational overlap across instructions should reduce the shared explained variance in brain responses, and if the mapping from instructions to neural processes is real, it should increase unique variance in the corresponding ROIs.
- The study cannot separate task semantics from surface phrasing with only ten hand-picked prompts; a follow-up using multiple paraphrases per task would show whether the instruction-specific alignment tracks the task or the wording.
- Because the fMRI data come from passive viewing, the alignment measured here is instruction-to-observation, not instruction-to-behavior; recording brain activity while subjects actually perform the instructed tasks could reveal stronger or different task-specific alignments.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates whether instruction-tuned multimodal large language models (MLLMs) can serve as brain-encoding models for visual stimuli. Using the Natural Scenes Dataset, the authors extract embeddings by mean-pooling hidden states of output tokens generated by InstructBLIP, mPLUG-Owl, and IDEFICS under ten natural-language instructions, then train voxel-wise ridge regression encoding models. They report that MLLMs show significantly better normalized brain alignment than a vision-only ViT-H baseline and perform comparably to CLIP-Text, that some instructions (e.g., image captioning, visual question answering) align better with specific visual ROIs, that MLLMs capture count- and recognition-related concepts, and that most explained variance is shared across instructions. The paper includes extensive appendices with layer-wise maps, category analyses, variance partitioning, and a limitations statement.
Significance. If the central claims hold, the paper would extend brain-encoding methodology by showing that a single instruction-tuned MLLM can be steered by natural-language prompts to produce representations with different degrees of alignment to visual cortex, potentially offering a flexible alternative to training separate task-specific models. The work uses a standard encoding pipeline on four NSD subjects with held-out test images, and it makes code publicly available, which are strengths. However, the load-bearing comparisons and the instruction-specific conclusions are currently undermined by the feature-extraction procedure, the unmatched CLIP baseline, and the absence of a stated significance test, so the significance of the claimed findings depends on whether these issues can be resolved.
major comments (4)
- [Section 4 (feature extraction)] The instruction-specific alignment results in Figures 3-7 rest on features obtained by mean-pooling hidden states over all generated output tokens for each instruction. This procedure conflates genuine image-plus-instruction representation with several generation artifacts: generation length varies widely across instructions, the lexical content of generated text carries language-model priors that may align with brain responses independently of visual processing, and autoregressive generation makes the pooled embedding depend on the whole text-generation trajectory rather than on a fixed, instruction-controlled visual representation. Because the fMRI data were collected during passive viewing with no task instruction given to participants (Appendix N), the only instruction-specific signal in the pipeline comes from the model's own outputs. The control in Appendix M (image only, instruction only) shows that visual input is necessary, but it does not exclude the text-generation confound because both compared conditions still use generated tokens. The authors should provide a fixed-length or generation-independent feature extraction (e.g., prompting only, or using the representation at the final prompt token before generation) and should control for generation length and lexical content, or explicitly show that the instruction-specific alignment patterns survive such controls.
- [Section 6.1, Figure 2 and Appendix K] The claim that MLLMs perform 'comparably to' CLIP is based on an unmatched comparison: the CLIP-Text model receives ground-truth oracle captions, while the MLLM embeddings come from model-generated tokens. The appendix also compares LLaMA-2-7B using captions, again with oracle input. This asymmetry can either inflate or deflate the relative performance in a way that is not controlled. Additionally, the asterisk and caret markers in Figure 2 are said to indicate 'statistically significantly better' with p <= 0.05, but no statistical test, multiple-comparison correction, or error-bar definition is given in the main text or figure caption; the only test described in the appendices (a one-way ANOVA in Appendix F) compares early versus higher visual ROIs, not model pairs. The authors should state the exact test used for each marker, report effect sizes and variance across subjects, and either match the CLIP baseline by using captions generated from the same model or clearly frame the comparison as 'MLLM-predicted text versus oracle captions.'
- [Section 6.2 (visual concept grouping)] The conclusion that MLLMs 'effectively capture count-related and recognition-related concepts' is based on grouping the ten instructions into five concept categories after the fact, with instruction-concept memberships chosen by the authors (e.g., VQ2 appears in both Count and Recognition, so the groups are not disjoint). The voxel-level winner-take-all maps in Figures 3-4 then select, for each voxel, the instruction with the highest alignment on the same data used to motivate the groupings. This post-hoc selection can create apparent concept specificity even if the underlying instruction representations are not specifically aligned with count or recognition processing. The authors should pre-specify the concept groupings, use disjoint instruction sets, or validate the grouping with a held-out set of instructions; without this, the 'effective capture' claim is not yet established.
- [Appendix N (Limitations)] The limitations section states that NSD was collected while subjects passively viewed images, 'so the dataset may not fully capture how brain activity aligns with task-specific instructions.' This is a direct acknowledgment that the passive-viewing data cannot independently ground the instruction-specific alignment maps that form the paper's central novelty. The authors should either soften the main-text claims (e.g., 'instruction-conditioned model representations' rather than 'instruction-specific neural processing') or add an analysis that ties the instruction-specific model differences to a neural measure that is not solely derived from the model's own outputs. The current framing overstates what can be concluded from passive-viewing fMRI.
minor comments (4)
- [Section 6.3, Figure 6 caption] The caption says the diagonal cells are empty, but the displayed matrix appears to contain values near 1 on the diagonal; please clarify whether the diagonal is excluded or set to 1 by construction.
- [Section 6.2, Figure 3 and Appendix G] The brain-map figures use a color code for ten instructions and then reuse the same color bar for concept groups in Figure 4; the captions should state whether the color assignment per instruction is identical across all subjects and whether the maps are thresholded, because the current presentation makes it hard to compare across subjects.
- [Section 4] The sentence 'The hidden states are dynamic and continuously evolve until the model generates the final token, which involves multiple forward passes in the process' is grammatically awkward; it should also specify whether the hidden states are taken from the language model backbone only or also from the vision encoder and Q-Former, since PCA is applied after pooling.
- [Section 5 (Evaluation metrics)] The normalized-brain-alignment procedure divides Pearson correlation by cross-subject prediction accuracy and then averages across voxels, regions, and participants; the paper should report the distribution of this normalized measure across voxels or subjects rather than only the mean, especially because Figure 2 reports values up to 0.9 that strongly depend on the choice of the voxel-inclusion threshold (cross-subject accuracy >= 0.05).
Circularity Check
No significant circularity: predictions are benchmarked against external NSD fMRI data, and no fitted parameter or self-citation is renamed as a derived result.
full rationale
The central claims—MLLM representations align better with brain activity than vision-only models and comparably to CLIP, and different instructions yield distinct alignment patterns—are evaluated against held-out fMRI responses from the external Natural Scenes Dataset. Features are extracted from pretrained MLLMs via zero-shot generation and mean-pooled hidden states; these features are fed to voxelwise ridge-regression encoding models whose test-set correlation is measured against neural recordings. This is an empirical external benchmark, not a derivation from the model's own outputs. The comparison baselines (ViT-H, CLIP, randomly initialized MLLMs) are independent, and the CLIP-text baseline even uses oracle captions while MLLMs use predicted generations, which strengthens rather than weakens the comparison. Variance partitioning uses standard shared/unique variance set arithmetic on the fitted normalized brain alignment values; the conclusion that much variance is shared is read off from the fitted models, not imposed by the definitions. The instruction-concept groupings and per-voxel winning-instruction maps are post-hoc selections made on the same test data, so they carry selection-sensitivity risk, but selecting the best of ten independently extracted feature sets is not a circular reduction: nothing in the feature construction is defined in terms of the brain responses being predicted. The mean-pooling of variable-length generated tokens may conflate text-generation artifacts with instruction-specific visual processing, but that is a validity or confound concern, not a circularity concern, because the predictions remain external to the model's own success measure. Self-citations appear mostly in related-work context and are background motivations rather than load-bearing premises; the present experiments are self-contained against public benchmarks. No equation, fitted parameter, or uniqueness claim reduces to its own input by construction.
Assumptions & free parameters
free parameters (3)
- PCA dimensionality for MLLM embeddings =
1024
- Voxel inclusion threshold =
cross-subject prediction accuracy >= 0.05
- Ridge regularization lambda =
10^-1 to 10^3, selected on 10% validation split
assumptions (4)
- domain assumption fMRI BOLD responses in visual cortex are a valid measure of visual neural activity for brain-encoding comparisons.
- domain assumption Linear ridge-regression voxel encoding models capture the relationship between model embeddings and fMRI sufficiently to rank models.
- domain assumption Hidden states of generated output tokens from MLLMs encode instruction-specific image understanding.
- ad hoc to paper The 10 hand-picked instructions and 5 concept groupings are a representative decomposition of visual semantics.
Cite this review
Pith. "Pith review of Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain)." pith.science (2026). https://pith.science/paper/NWUAE267
@misc{pith2026250520029,
author = {Pith},
title = {Pith review of: Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain)},
year = {2026},
howpublished = {\url{https://pith.science/paper/NWUAE267}},
note = {Machine review of arXiv:2505.20029}
}
read the original abstract
Transformer-based language models, though not explicitly trained to mimic brain recordings, have demonstrated surprising alignment with brain activity. Progress in these models-through increased size, instruction-tuning, and multimodality-has led to better representational alignment with neural data. Recently, a new class of instruction-tuned multimodal LLMs (MLLMs) have emerged, showing remarkable zero-shot capabilities in open-ended multimodal vision tasks. However, it is unknown whether MLLMs, when prompted with natural instructions, lead to better brain alignment and effectively capture instruction-specific representations. To address this, we first investigate brain alignment, i.e., measuring the degree of predictivity of neural visual activity using text output response embeddings from MLLMs as participants engage in watching natural scenes. Experiments with 10 different instructions show that MLLMs exhibit significantly better brain alignment than vision-only models and perform comparably to non-instruction-tuned multimodal models like CLIP. We also find that while these MLLMs are effective at generating high-quality responses suitable to the task-specific instructions, not all instructions are relevant for brain alignment. Further, by varying instructions, we make the MLLMs encode instruction-specific visual concepts related to the input image. This analysis shows that MLLMs effectively capture count-related and recognition-related concepts, demonstrating strong alignment with brain activity. Notably, the majority of the explained variance of the brain encoding models is shared between MLLM embeddings of image captioning and other instructions. These results suggest that enhancing MLLMs' ability to capture task-specific information could lead to better differentiation between various types of instructions, and thereby improving their precision in predicting brain responses.
Figures
Figures from the paper (19 more)
Reference graph
Works this paper leans on
-
[1]
Phi-3 technical re- port: A highly capable language model locally on your phone
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical re- port: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, August
-
[2]
15 Published as a conference paper at ICLR 2025 0.1 0.2 0.3 S1 S2 S3 S4 Cross-subject brain predictivity Figure 8: Cross-subject prediction accuracy for each subject of NSD dataset. InstructBLIP (Dai et al.,
work page 2025
-
[3]
From the examples in Table 6, we observe that the generated output tokens adhere more closely to captioning instructions, regardless of the specific task instructions provided. The outputs often consist of simple responses, such as “Yes,” “No,” or color names, and lack detailed descriptions. In contrast, instruction-tuned MLLMs excel at providing semantic...
work page 2025
-
[5]
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,
-
[8]
Microsoft coco: Common objects in context
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740–755. Springer,
2014
-
[10]
Subba Reddy Oota, Jashn Arora, Veeral Agarwal, Mounika Marreddy, Manish Gupta, and Bapi Surampudi. Neural language taskonomy: Which nlp tasks are the most predictive of fmri brain activity? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3220–3237, 2022a...
work page 2022
-
[11]
Speech language models lack important brain-relevant semantics
Subba Reddy Oota, Emin C ¸ elik, Fatma Deniz, and Mariya Toneva. Speech language models lack important brain-relevant semantics. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8503–8528, 2024a. Subba Reddy Oota, Zijiao Chen, Manish Gupta, Bapi Raju Surampudi, Ga ¨el Jobard, Fr ´ed´eri...
work page 2025
-
[13]
Tuning in to neural encoding: Linking human brain and artificial supervised representations of language
Jingyuan Sun, Xiaohan Zhang, and Marie-Francine Moens. Tuning in to neural encoding: Linking human brain and artificial supervised representations of language. In ECAI 2023, pp. 2258–2265. IOS Press,
2023
Show all 26 references
-
[14]
Llama: Open and efficient foundation language models
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971,
-
[16]
Natural language supervision with a large and diverse dataset builds better models of human high-level visual cor- tex
Aria Y Wang, Kendrick Kay, Thomas Naselaris, Michael J Tarr, and Leila Wehbe. Natural language supervision with a large and diverse dataset builds better models of human high-level visual cor- tex. BioRxiv, pp. 2022–09,
2022
-
[17]
Transformers: State-of-the-art natural language processing
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R ´emi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in ...
2020
-
[18]
mplug-owl: Modularization empowers large language models with multimodality
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178,
-
[19]
mid temporal lobe bodies
14 Published as a conference paper at ICLR 2025 A O VERVIEW OF APPENDIX SECTIONS • Section B: Visual functional localizers • Section C: Cross-subject brain predictivity • Section D Details of MLLMs with training details and their parameters • Section E: Model generated outputs...
2025
-
[21]
is a vision-language instruction-tuned model built upon the pre- trained BLIP-2 model (Li et al., 2023). It leverages a diverse set of instruction data (26 different datasets) to train a MLLM, which comprises an image encoder, a large language model (LLM), and a Query Transfor...
2023
-
[22]
is an MLLM based on Flamingo (Zhu et al., 2024), which accepts arbitrary sequences of image and text inputs and generates text tokens. We utilize the idefics-9b- instruct version (the model obtained by further training IDEFICS on supervised fine-tuning and instruction fine-tun...
2024
-
[23]
The color bar highlights color codes for each instruction
that led to the highest nor- malized brain alignment. The color bar highlights color codes for each instruction. The voxels are projected onto the flattened cortical surface of a representative subject (subject S1, S2, S5 and S7) for three MLLMs. 21 Published as a conference p...
2025
-
[25]
Overall, the analysis shows that MLLMs are most effective at aligning with brain activity when both task-specific and shared category features are present, suggesting that these features play a crucial role in brain alignment. However, in categories with fewer shared visual el...
2025
-
[33]
The color bar highlights color codes for each layer
that led to the highest normalized brain alignment. The color bar highlights color codes for each layer. The voxels are projected onto the flattened cortical surface of a representative subject (S2, S5 and S7) for three MLLMs. information could lead to greater precision in pre...
2025
-
[2003]
Shared computational principles for language processing in humans and deep language models.Nature Neuroscience, 25(3):369–380,
2https://cvnlab.slite.page/p/IB6BSeW_7o/Terms-and-Conditions 11 Published as a conference paper at ICLR 2025 Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, et al. Shared computational...
2025
-
[2018]
Mistral 7b
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825,
-
[2019]
Semantic scene descriptions as an objective of human vision
Adrien Doerig, Tim C Kietzmann, Emily Allen, Yihan Wu, Thomas Naselaris, Kendrick Kay, and Ian Charest. Semantic scene descriptions as an objective of human vision. arXiv preprint arXiv:2209.11737,
-
[2020]
What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines? bioRxiv, pp
Colin Conwell, Jacob S Prince, Kendrick N Kay, George A Alvarez, and Talia Konkle. What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines? bioRxiv, pp. 2022–03,
2022
-
[2021]
Learning transferable visual models from natural language supervision, 2021a
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021a. URL https:/...
-
[2022]
Neural taskonomy: Inferring the similarity of task- derived representations from brain activity
13 Published as a conference paper at ICLR 2025 Aria Wang, Michael Tarr, and Leila Wehbe. Neural taskonomy: Inferring the similarity of task- derived representations from brain activity. Advances in Neural Information Processing Systems, 32:15501–15511,
2025
-
[2023]
Instruction-tuning aligns llms to the human brain
Khai Loong Aw, Syrielle Montariol, Badr AlKhamissi, Martin Schrimpf, and Antoine Bosselut. Instruction-tuning aligns llms to the human brain. arXiv preprint arXiv:2312.00575,
-
[2024]
The brain tells a story: Unveiling distinct representations of semantic content in speech, objects, and stories in the human brain with large language models
Yuko Nakagi, Takuya Matsuyama, Naoko Koide-Majima, Hiroto Yamaguchi, Rieko Kubo, Shinji Nishimoto, and Yu Takagi. The brain tells a story: Unveiling distinct representations of semantic content in speech, objects, and stories in the human brain with large language models. bioR...
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.