The time course of visuo-semantic representations in the human brain is captured by combining vision and language models

Alessandro Thomas Gifford; Boyan Rong; Emrah D\"uzel; Radoslaw Martin Cichy

REVIEW 2 major objections 5 minor 8 references

The time course of visuo-semantic representations in the human brain is captured by combining vision and language models

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fusing vision and language models predicts human EEG better than either alone

desk verdict Solid fusion encoding model with real predictive gains; the 'language' interpretation is weaker than claimed because the text comes from a vision-language model that sees the image. read the letter →

arxiv 2506.19497 v1 pith:QUZVKUFA submitted 2025-06-24 q-bio.NC

Boyan Rong , Alessandro Thomas Gifford , Emrah Düzel , Radoslaw Martin Cichy This is my paper

classification q-bio.NC

keywords vision-languagemodelsencodinghumanEEGvisualobjectrecognitionsemanticrepresentations

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the human brain's visuo-semantic representations, measured with millisecond-level EEG, are best captured by one model or by a combination. The authors combine a vision deep neural network (CORnet-S) with a text-embedding large language model into a single encoding model and show that this fusion predicts EEG responses to over 16,000 natural images better than either component alone and better than prior multimodal models. The key insight is a clean division of labour: the vision component uniquely explains early, broadband signals over occipital cortex, while the language component uniquely explains later, low-frequency signals over occipito-temporal cortex. The paper argues that the language component's contribution comes from detailed visuo-semantic content — nouns and adjectives describing object parts, scene context, and properties — beyond mere object category. If correct, this gives a practical recipe for building more accurate neural encoding models and for parsing the temporal dynamics of semantic processing in the human visual system.

What carries the argument

The central object is the fusion encoding model: a ridge regression that maps a convex combination of vision and language representations, [(1-α)X, αY], onto time-resolved EEG responses, where α balances the two modalities. The vision pathway uses CORnet-S activations; the language pathway averages five GPT-4V-generated text descriptions per image into a text-embedding-3-large embedding. The authors isolate each modality's unique contribution via partial correlation, controlling for the other modality's predictions. This machinery lets them attribute early broadband occipital signals to vision and later low-frequency occipito-temporal signals to language.

What would settle it

A control experiment that feeds the LLM purely textual descriptions of images generated without any visual input (e.g., from category labels and attributes alone) and shows the late low-frequency prediction advantage disappears would directly test whether the claimed visuo-semantic contribution requires visual access in the language pathway.

Watch

Extended reading notes

Core claim

The central claim is that a fusion encoding model, built by concatenating weighted representations from a vision DNN and a large language model, outperforms encoding models based on either component alone in predicting human EEG responses to visual stimuli. The paper shows that the vision DNN (CORnet-S) and the LLM (text-embedding-3-large, fed with GPT-4V image descriptions) capture complementary information: the vision component uniquely predicts early (peak around 90-110 ms), broadband (2-70 Hz) activity at occipital electrodes, while the LLM component uniquely predicts later (peak around 365 ms), low-frequency (below 12 Hz) activity at occipito-temporal electrodes. It further demonstrates that the LLM benefit arises from detailed visuo-semantic text — specifically nouns and adjectives — rather than object category labels alone, and that the advantage holds across different vision DNNs and LLMs.

Load-bearing premise

The language pathway's 'semantic' representations come from text descriptions that GPT-4V generated while looking at the actual image, so the unique late contribution attributed to language may include additional visual information rather than pure meaning.

Editorial extensions

If this is right

Encoding models of visual processing should fuse visual and language representations rather than rely on one modality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the GPT-4V descriptions carry additional visual details that CORnet-S lacks, the 'language' component may partly serve as a richer visual encoder, and its late low-frequency signature could reflect scene-level visual analysis, not pure semantics.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Bottom line: the fusion model works, and the paper shows it carefully. The design is a model for EEG encoding-model comparisons: held-out test images, cross-validated alpha and lambda, FDR-corrected statistics, noise ceilings, multiple instantiations of vision and language backbones, plus code and data online. I believe the core predictive result, that adding LLM embeddings of image descriptions to CORnet-S features improves held-out EEG prediction, is real and will replicate.

What's new: the distinct temporal and spectral signatures of the two components (vision early/broadband occipital, LLM late/low-frequency occipito-temporal) are a useful addition to the LLM-brain alignment literature. The controls with category labels, parts of speech, and permuted descriptions are well thought out and address the obvious alternative explanations in a stepwise way.

The load-bearing weakness is the language pathway. GPT-4V literally sees the image before producing the text. So the 'LLM' representations are not language-only; they are text renderings of visual features. The unique late contribution could be additional visual detail (scene context, object parts) that CORnet-S lacks, rather than abstract semantic content. The category/noun/adjective controls show semantic structure, but the category-label control is pure text only in the sense that it lacks detail; it doesn't control for GPT-4V's visual access. A two-vision-DNN fusion control, or a pure-text description generator that receives no image information, would help. The paper doesn't acknowledge this confound explicitly, which is an omission, but it doesn't destroy the predictive claim.

Who this is for: researchers building encoding models of visual cortex, particularly with EEG/MEG, and anyone benchmarking LLM-brain alignment. It deserves a serious referee. With the GPT-4V confound addressed or at least discussed in the limitations, I'd be comfortable with publication.

Referee Report

2 major / 5 minor

Summary. This manuscript proposes and evaluates fusion encoding models that predict time-resolved human EEG responses to naturalistic object images by combining vision-DNN features (CORnet-S) with LLM text embeddings (text-embedding-3-large) of image descriptions generated by GPT-4V. On the THINGS EEG2 dataset (16,740 training images, 200 held-out test images, 10 participants), the authors train ridge-regression encoding models with cross-validated regularization lambda and fusion weight alpha, and report that the fusion model outperforms each unimodal model as well as two multimodal baselines (CLIP, VisualBERT). The vision-DNN component uniquely predicts early (about 90-110 ms), broadband, occipital signals, while the LLM component uniquely predicts later (about 365 ms), low-frequency (below 12 Hz), occipito-temporal signals. Text-ablation analyses indicate that nouns and adjectives, but not verbs or permuted descriptions, drive the fusion benefit over the vision model, which the authors interpret as evidence that the LLM contribution reflects detailed visuo-semantic content beyond object categories.

Significance. The paper has substantial strengths. The quantitative core is carefully designed: models are evaluated on a held-out test partition, the hyperparameters (alpha, lambda) are selected by 5-fold cross-validation on the training partition only, statistical significance is FDR-corrected across time points, channels, and frequencies, and performance is benchmarked against noise ceilings. The authors provide extensive robustness evidence: replication in every participant, pairwise decoding, partial correlation and variance partitioning converging on the same spectro-temporal pattern, and generalization across four vision DNNs and four LLMs. Public code is provided. If the fusion advantage is accepted as stated, the paper offers a practical and interpretable approach for modeling the time course of visual and semantic processing, and the spectro-temporal dissociation between the two components is a valuable datapoint for the encoding-model literature.

major comments (2)

[Methods 4.4.2, Fig. 1A, Results 2.3, Discussion 3.2] The 'language' pathway is a vision-to-language pipeline, not a purely textual one. The text descriptions are generated by GPT-4V, which receives the stimulus image (Methods 4.4.2; Fig. 1A), and the example captions in Fig. 2G show that the descriptions encode rich visual scene content (context, object parts, material, scene layout). The paper's central interpretive claims, that the LLM 'uniquely captured later and low frequency signals, as well as detailed visuo-semantic stimulus information' (Abstract) and that the late occipito-temporal signal reflects semantic processing linked to the N400 and the anterior temporal lobe hub (Discussion 3.2), presuppose that the unique contribution of the text-embedding model is carried by language-derived content. The same unique contribution could, however, be carried by visual detail extracted by GPT-4V that CORnet-S does not encode. The controls in Section 2.3 do not close this gap: the permuted-description control only rules out meaningless text, and the noun and adjective analyses operate on tokens extracted from GPT-4V's own output, so they inherit the same visual access. The human-annotated category-label control is pure text but lacks exactly the detailed content whose contribution is at issue. I recommend adding a control condition in which detailed descriptions come from a source without image access (e.g., human-written captions), or a vision-only control of comparable descriptive detail (e.g., features from a larger self-supervised vision transformer), or, failing that, systematically re-scoping all 'language' and 'LLM' claims to 'the vision-to-language captioning pipeline'. Without one of these, the title and abstract overstate what the experiments establish.
[Results 2.4, Fig. 3A-B, Discussion 3.1] The benchmark comparison against CLIP and VisualBERT is informative but asymmetric. The fusion model consumes, in addition to the image, text descriptions produced by GPT-4V, a large multimodal system with direct visual access, whereas the CLIP and VisualBERT baselines are evaluated without such an upstream captioning stage. The finding that the fusion model predicts EEG better than these baselines is therefore a statement about the full pipeline, not about the fusion architecture per se. The Discussion (Section 3.1) frames the outcome as demonstrating 'the advantage of the fusion model approach'; to be fully fair, the text should acknowledge that part of the advantage may reflect the additional visual-captioning resource (GPT-4V) rather than the benefit of combining two unimodal feature spaces. This does not invalidate the reported comparison, but the framing should be adjusted to match what the design can support.

minor comments (5)

[Methods 4.4.3] Typo: 'predictosrs' should be 'predictors'.
[Discussion 3.2] There is a stray '31' in the phrase 'revealing 31 two processing stages', apparently a broken citation or formatting artifact; please fix.
[Results 2.2, Fig. 2B] The statement that 'we did not observe qualitative differences between the images that most and least benefited from the fusion model' is based on informal visual inspection; a quantitative image-property comparison (e.g., low-level statistics, object size, clutter) would strengthen the claim.
[Supplementary Fig. 2] The pairwise-decoding measure peaks at 190 ms for the language encoding model, whereas the correlation measure peaks at 365 ms, and the fusion-vs-language decoding difference is not significant (p = 0.06). A sentence reconciling this discrepancy between the two evaluation measures would help readers interpret the decoding results.
[Methods 4.4.4] Please state explicitly whether the participant-averaged Pearson r time courses are obtained by averaging raw correlations or Fisher z-transformed correlations, since this affects the reported confidence intervals and the interpretation of the noise-ceiling comparisons.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: held-out test images and cross-validated hyperparameters make the fusion prediction a genuine out-of-sample result; the GPT-4V visual-access issue is an interpretive confound, not a circular derivation.

full rationale

The central quantitative claim is that a fusion of CORnet-S features and text-embedding-3-large embeddings predicts EEG responses on 200 held-out test images. Models are trained on the training partition only (16,495 images after exclusions), and the fusion weight alpha and ridge regularization lambda are selected via 5-fold cross-validation on training data (Methods 4.4.3). Test images are used only for evaluation, and prediction accuracy is computed by correlating predicted EEG with recorded EEG averaged over held-out repetitions (Methods 4.4.4). The unique-contribution analyses (partial correlation, subtraction, variance partitioning) are all computed on these out-of-sample predictions, so none of the reported improvements reduce to a fitted value by construction. The permuted-description control confirms that the language benefit depends on meaningful image-text correspondence. The mild concern that GPT-4V has visual access to the stimulus (Methods 4.4.2) is a construct-validity issue: the 'language' pathway is actually a multimodal pipeline (image -> GPT-4V description -> text embedding), so the late low-frequency contribution attributed to the LLM could reflect additional visual detail rather than purely abstract semantics. This is an interpretive limitation, not a circular step, because the text embeddings are fixed inputs and the held-out test preserves predictive validity. Self-citations are present (e.g., refs 8, 16, 32, 60 from the authors), but they concern dataset usage and background findings, not a load-bearing uniqueness theorem or an unverified ansatz. No equation-level reduction or fitted-input-renamed-as-prediction occurs. Therefore the derivation chain is self-contained with respect to circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard encoding-model assumptions and on the specific choice of GPT-4V as an image description generator. The main fitted parameters are the fusion weight alpha and the ridge regularization lambda, both cross-validated on training data. No new physical or conceptual entities are introduced.

free parameters (3)

alpha (fusion weighting coefficient) = grid searched [0, 1] with step 0.05, per subject, channel, and time point
Fusion representation defined as [(1-alpha)X, alpha Y]; alpha is optimized via cross-validation on the training partition, so it is a fitted parameter that controls the balance between vision and language features.
lambda (ridge regularization strength) = grid searched [10^-3, 10^4] with 100 log-spaced steps, per subject, channel, and time point
Standard ridge regression hyperparameter, selected by cross-validation on the training partition; the central results depend on this choice, though robustly so across the grid.
PCA dimensionality (1000 components per feature set) = 1000
Chosen by the authors for feature reduction of both vision and language representations; not fitted to neural data but affects all encoding models.

assumptions (5)

domain assumption Linear ridge regression can map model features to EEG responses
The encoding models assume a linear mapping from features to EEG amplitude at each channel and time point, a standard but nontrivial assumption in encoding-model research.
domain assumption THINGS EEG2 dataset provides reliable ground-truth neural responses
The paper relies on the dataset's quality and the averaging of 40 or 80 repetitions per test image to obtain stable EEG responses.
domain assumption GPT-4V descriptions faithfully represent image content for semantic embedding
The language pathway is built on text descriptions generated by GPT-4V; the validity of the study depends on these descriptions being accurate and information-preserving, and on the text embedding model capturing that information.
domain assumption Cosine similarity of model representations reflects representational overlap
Used in Figure 1B and Figure 2A to quantify the difference between vision and language representations; assumes cosine similarity is a valid measure of representational content.
domain assumption Noise ceiling bounds are valid estimates of explainable variance
The noise ceiling is computed from split-half reliability of EEG responses; the interpretation of model performance relies on the assumption that this measures the maximum explainable signal.

how reviews work

0 comments

Cite this review

Pith. "Pith review of The time course of visuo-semantic representations in the human brain is captured by combining vision and language models." pith.science (2026). https://pith.science/paper/QUZVKUFA

@misc{pith2026250619497,
  author       = {Pith},
  title        = {Pith review of: The time course of visuo-semantic representations in the human brain is captured by combining vision and language models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QUZVKUFA}},
  note         = {Machine review of arXiv:2506.19497}
}

read the original abstract

The human visual system provides us with a rich and meaningful percept of the world, transforming retinal signals into visuo-semantic representations. For a model of these representations, here we leveraged a combination of two currently dominating approaches: vision deep neural networks (DNNs) and large language models (LLMs). Using large-scale human electroencephalography (EEG) data recorded during object image viewing, we built encoding models to predict EEG responses using representations from a vision DNN, an LLM, and their fusion. We show that the fusion encoding model outperforms encoding models based on either the vision DNN or the LLM alone, as well as previous modelling approaches, in predicting neural responses to visual stimulation. The vision DNN and the LLM complemented each other in explaining stimulus-related signal in the EEG responses. The vision DNN uniquely captured earlier and broadband EEG signals, whereas the LLM uniquely captured later and low frequency signals, as well as detailed visuo-semantic stimulus information. Together, this provides a more accurate model of the time course of visuo-semantic processing in the human brain.

Figures

Figures reproduced from arXiv: 2506.19497 by the authors.

**Figure 2.** Factors determining the prediction performance of the fusion encoding model. A, Scatterplot of pairwise cosine similarity between pairs of stimulus representations of either the vision DNN or the LLM. The stimuli were the 25 stimulus images that most benefited from the fusion compared to the vision encoding model in terms of EEG prediction accuracy (color coded in red), and the 25 stimulus images that least benefitt… view at source ↗

**Figure 3.** Model comparisons and generalizability analysis. A, Prediction accuracy timecourse for the fusion encoding model, and the encoding models trained on representations from multimodal DNNs (CLIP and VisualBERT). B, Difference in prediction accuracy between the fusion encoding model, and the encoding models trained on representations from multimodal DNNs (CLIP and VisualBERT). C, Prediction accuracy timecourse for fusio… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Time-frequency resolved analysis and results. A, We decomposed the EEG responses into the timefrequency domain using Morlet wavelets. B-D, Prediction accuracy (Pearson’s r) of the EEG time-frequency data [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

8 extracted references · 4 canonical work pages

[3]

Discussion Our goal was to provide a modelling approach accounting for the time course of visuo-semantic processing in the human brain. Towards this goal, we trained encoding models on a large datasets of EEG responses to naturalistic images to test the hypothesis that human representations across the visual processing hierarchy are better modelled by a f...

work page
[4]

Describe the image in five different ways

Methods 4.1. Data For all our analyses we used the THINGS EEG2 dataset 32, a large-scale dataset of EEG responses to naturalistic images optimized to build encoding models of neural responses to visual stimulation. The THINGS EEG2 dataset contains EEG recordings of 10 participants viewing images of objects on natural backgrounds from the THINGS database 4...

1980
[12]

Doerig, A. et al. The neuroconnectionist research programme. Nat. Rev. Neurosci. 24, 431–450 (2023). 13. Khaligh-Razavi, S.-M. & Kriegeskorte, N. Deep Supervised, but Not Unsupervised, Models May Explain IT Cortical Representation. PLOS Comput. Biol. 10, e1003915 (2014). 14. Yamins, D. L. K. et al. Performance-optimized hierarchical models predict neural ...

work page Pith review arXiv doi:10.48550/arxiv.2202.04728 2023
[30]

A., Lal, V

Tang, J., Du, M., Vo, V. A., Lal, V. & Huth, A. G. Brain encoding models based on multimodal transformers can transfer across language and vision. Adv. Neural Inf. Process. Syst. 36, 29654–29666 (2023). 31. Grosbard, I. D. & Yovel, G. Self-supervision deep learning models are better models of human high-level visual cortex: The roles of multi-modality and...

arXiv doi:10.48550/arxiv.2208.02957 2023
[39]

The dynamics of meaning through time: Assessment of Large Language Models

Alrefaie, M. T., Salem, F., Morsy, N. E., Samir, N. & Gaber, M. M. The dynamics of meaning through time: Assessment of Large Language Models. Preprint at https://doi.org/10.48550/arXiv.2501.05552 (2025). 40. Kubilius, J. et al. Brain-like object recognition with high-performing shallow recurrent ANNs. in Proceedings of the 33rd International Conference on...

work page Pith review arXiv doi:10.48550/arxiv.2501.05552 2025
[48]

& Xiao, B

Liao, C., Sawayama, M. & Xiao, B. Probing the link between vision and language in material perception using psychophysics and unsupervised learning. PLOS Comput. Biol. 20, e1012481 (2024). 49. Nakagi, Y. et al. Unveiling Multi-level and Multi-modal Semantic Representations in the Human Brain using Large Language Models. in Proceedings of the 2024 Conferen...

work page 2024
[56]

language-aligned

Winawer, J. et al. Asynchronous broadband signals are the principal source of the BOLD response in human visual cortex. Curr. Biol. CB 23, 1145–1153 (2013). 57. van Kerkoerle, T. et al. Alpha and gamma oscillations characterize feedback and feedforward processing in monkey visual cortex. Proc. Natl. Acad. Sci. U. S. A. 111, 14332–14341 (2014). 58. Bastos,...

arXiv doi:10.48550/arxiv.2412.02104 2013
[85]

Noise ceiling calculation

Kutas, M. & Federmeier, K. D. Thirty years and counting: Finding meaning in the N400 component of the event related brain potential (ERP). Annu. Rev. Psychol. 62, 621–647 (2011). 86. Van Petten, C. & Luka, B. J. Neural localization of semantic context effects in electromagnetic and hemodynamic studies. Brain Lang. 97, 279–293 (2006). 87. Lau, E. F., Phill...

arXiv 2011

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [3]

Discussion Our goal was to provide a modelling approach accounting for the time course of visuo-semantic processing in the human brain. Towards this goal, we trained encoding models on a large datasets of EEG responses to naturalistic images to test the hypothesis that human representations across the visual processing hierarchy are better modelled by a f...

work page

[2] [4]

Describe the image in five different ways

Methods 4.1. Data For all our analyses we used the THINGS EEG2 dataset 32, a large-scale dataset of EEG responses to naturalistic images optimized to build encoding models of neural responses to visual stimulation. The THINGS EEG2 dataset contains EEG recordings of 10 participants viewing images of objects on natural backgrounds from the THINGS database 4...

1980

[3] [12]

Doerig, A. et al. The neuroconnectionist research programme. Nat. Rev. Neurosci. 24, 431–450 (2023). 13. Khaligh-Razavi, S.-M. & Kriegeskorte, N. Deep Supervised, but Not Unsupervised, Models May Explain IT Cortical Representation. PLOS Comput. Biol. 10, e1003915 (2014). 14. Yamins, D. L. K. et al. Performance-optimized hierarchical models predict neural ...

work page Pith review arXiv doi:10.48550/arxiv.2202.04728 2023

[4] [30]

A., Lal, V

Tang, J., Du, M., Vo, V. A., Lal, V. & Huth, A. G. Brain encoding models based on multimodal transformers can transfer across language and vision. Adv. Neural Inf. Process. Syst. 36, 29654–29666 (2023). 31. Grosbard, I. D. & Yovel, G. Self-supervision deep learning models are better models of human high-level visual cortex: The roles of multi-modality and...

arXiv doi:10.48550/arxiv.2208.02957 2023

[5] [39]

The dynamics of meaning through time: Assessment of Large Language Models

Alrefaie, M. T., Salem, F., Morsy, N. E., Samir, N. & Gaber, M. M. The dynamics of meaning through time: Assessment of Large Language Models. Preprint at https://doi.org/10.48550/arXiv.2501.05552 (2025). 40. Kubilius, J. et al. Brain-like object recognition with high-performing shallow recurrent ANNs. in Proceedings of the 33rd International Conference on...

work page Pith review arXiv doi:10.48550/arxiv.2501.05552 2025

[6] [48]

& Xiao, B

Liao, C., Sawayama, M. & Xiao, B. Probing the link between vision and language in material perception using psychophysics and unsupervised learning. PLOS Comput. Biol. 20, e1012481 (2024). 49. Nakagi, Y. et al. Unveiling Multi-level and Multi-modal Semantic Representations in the Human Brain using Large Language Models. in Proceedings of the 2024 Conferen...

work page 2024

[7] [56]

language-aligned

Winawer, J. et al. Asynchronous broadband signals are the principal source of the BOLD response in human visual cortex. Curr. Biol. CB 23, 1145–1153 (2013). 57. van Kerkoerle, T. et al. Alpha and gamma oscillations characterize feedback and feedforward processing in monkey visual cortex. Proc. Natl. Acad. Sci. U. S. A. 111, 14332–14341 (2014). 58. Bastos,...

arXiv doi:10.48550/arxiv.2412.02104 2013

[8] [85]

Noise ceiling calculation

Kutas, M. & Federmeier, K. D. Thirty years and counting: Finding meaning in the N400 component of the event related brain potential (ERP). Annu. Rev. Psychol. 62, 621–647 (2011). 86. Van Petten, C. & Luka, B. J. Neural localization of semantic context effects in electromagnetic and hemodynamic studies. Brain Lang. 97, 279–293 (2006). 87. Lau, E. F., Phill...

arXiv 2011