Pith. sign in

REVIEW 3 major objections 6 minor 103 references

What Makes Linguistic Representations Good Models of High-Level Visual Perception in the Human Brain?

T0 review · 3 major / 6 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Caption embeddings from text-only language models significantly predict brain activity in high-level visual regions, with detailed machine-generated captions and sentence-level embedders performing best.

desk verdict A solid, useful benchmark with an over-interpreted LM comparison; the empirical recipe is worth taking seriously, but the causal story about why text embedders win needs controls. read the letter →

arxiv 2607.16214 v1 pith:56P4SMGE submitted 2026-05-22 cs.CV cs.AI

Anna Bavaresco , Ina Klarić , Raquel Fernández , Marie-Francine Moens This is my paper
classification cs.CVcs.AI
keywords brainencodingcaptionembeddingslanguagemodelsrepresentationalsimilarityanalysishigh-levelvisualcortexfMRINaturalScenesDatasetintrinsicdimensionality
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks what makes a linguistic representation a good model of high-level visual perception in the human brain. It systematically varies how images are described—six caption types, human- and machine-generated—and which text-only language model converts those captions into vectors. The central finding is that machine-generated captions often predict fMRI responses in face-, body-, and place-selective areas better than the standard MS COCO captions used previously, and that sentence-level text embedders consistently beat next-word-prediction models. The authors interpret this as evidence that the meaning of a caption, not its surface form, drives brain predictivity, with predictivity peaking in intermediate layers shortly after a phase of high intrinsic dimensionality. If correct, text descriptions alone capture much of the structure the brain uses for high-level visual organization.

What carries the argument

The central machinery is the pairing of caption type and embedding model: six caption sources—MS COCO human captions and five vision-language-model-generated caption sets—each embedded with five text-only LMs (BERT, GPT-2, Llama3.1, Qwen3 Embedding, KaLM Embedding). Brain predictivity is measured with voxel-wise ridge encoding models, brain and behavioural alignment with RSA (Spearman correlation between representational dissimilarity matrices), and layer-wise information processing with the GRIDE intrinsic-dimensionality estimator. The comparison between different LMs and caption types, plus the layer analysis linking predictivity peaks to a high-dimensionality phase, is what carries the ar

What would settle it

Take the best-performing text embedder, and on the same brain-encoding pipeline compare a same-architecture autoregressive base model with and without contrastive semantic fine-tuning, matched for size and tokenizer. If the fine-tuned variant does not outperform its base on brain predictivity, the conclusion that semantic fine-tuning drives the gains is falsified; if a small matched model outperforms a larger autoregressive one, scale is not the confound.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that image captions embedded with text-only LMs are a powerful tool for modelling high-level vision: the best configurations nearly match vision-model features in brain encoding and outperform them in behavioural alignment with human similarity judgments. The effect is systematic: machine-generated captions from Pixtral, Qwen2.5-VL, and Molmo yield higher brain predictivity than the human MS COCO captions, and the choice of embedding LM is the stronger factor, with Qwen3 and KaLM Embedding leading across ROIs and metrics. Layer-wise analyses show brain predictivity and behavioural alignment peak in mid-late layers immediately after a high-dimensiona

Load-bearing premise

The load-bearing premise is that the five language models can be compared as if they differed mainly in training objective; in reality they also differ in scale, tokenizer, and how sentence embeddings are extracted, so an uncontrolled confound could produce the observed ordering.

Editorial extensions

If this is right

  • If caption embeddings rival vision features in high-level visual cortices, text-only supervision becomes a viable route for building brain-aligned vision models, and linguistic descriptions can serve as a cheap annotation for neural decoding studies.
  • Since text embedders outperform next-word models, fine-tuning on semantic tasks improves brain-relevant representations; future embedding models could be evaluated for neural predictivity as a benchmark.
  • Machine-generated, more fluent and detailed captions are better neural models than the short human captions used previously; averaging several human captions recovers comparable performance, suggesting that caption noise rather than inherent content limits the human annotations.
  • The double dissociation between face/body and place alignment across layers suggests that different high-level visual regions rely on different linguistic information, inviting region-specific caption representations.
  • Layer-wise peaks after high intrinsic dimensionality suggest the brain-like information is abstract semantic and syntactic structure, not word prediction or surface statistics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The conclusion that text embedders outperform autoregressive LMs may generalize to other brain regions and stimulus sets, but only if the same ordering persists when embedders are matched for size, tokenizer, and pooling; a matched-size control would directly test the training-objective hypothesis.
  • If caption embeddings capture brain-aligned semantics so well, they could be used as a low-cost surrogate to preselect stimuli or caption sets that maximize expected neural predictivity in future fMRI experiments.
  • The behavioural alignment result suggests language-derived similarity is a good proxy for human similarity judgments; this could make linguistic stimuli a practical tool for aligning vision-language models to human perception, not just to task labels.
  • The layer-wise dissociation predicts that probing a single layer will overfit to one region; using layer ensembles or region-specific layers may improve neural encoding models of high-level vision.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper asks which caption properties and which text-only language models make image-caption embeddings good predictors of human high-level visual brain responses and similarity judgments. Using NSD data (906 images, 8 participants, face/body/place ROIs), the authors compare six caption types (one random MS COCO caption and five VLM-generated sets) embedded with five LMs (BERT, GPT-2, Llama3.1, Qwen3 Embedding, KaLM Embedding). They evaluate with voxel-wise ridge encoding models, RSA with noise ceilings, and linear mixed-effects models, and additionally analyze layer-wise intrinsic dimensionality. They report that machine-generated captions are often more predictive than the single COCO captions used in previous work, that text embedders outperform autoregressive LMs, that layer-wise predictivity/alignment peaks after a high-dimensional phase, and that caption embeddings can align with human similarity judgments better than image features.

Significance. The study addresses an important question in NeuroAI with carefully described standard methods: cross-validated ridge encoding models, permutation tests with FDR correction, RSA with noise ceilings, and linear mixed-effects models. The behavioral similarity analysis appears to be the first on this NSD component, and the authors are transparent about the single-vs-averaged COCO issue and provide code. If the main empirical pattern holds—machine-generated captions, especially embedded with Qwen3/KaLM, are strongly predictive and behaviorally aligned—it is a meaningful contribution. However, the mechanistic interpretation of the LM-type comparison is not yet supported, and the statistical reporting has gaps. These issues are fixable but currently block full acceptance.

major comments (3)
  1. [§2.1, §2.5, Discussion; Methods 'Caption embeddings'] The claim that text embedders outperform autoregressive LMs because they are fine-tuned on semantic tasks is underdetermined: the five LMs differ simultaneously in scale (BERT/GPT-2 ≈0.1B; Llama3.1 8B; Qwen3/KaLM larger), base pretraining (BERT/GPT-2/Llama3/Gemma3), architecture, and pooling (CLS vs last token vs model-card instructions). The LME analyses include LM as a fixed effect but do not control for these confounds, so the ranking could be driven by scale or pooling rather than training objective. Add matched-size/same-base controls (e.g., an embedding model and its base LM with identical pooling) or restrict the conclusion to a descriptive claim and state the confound as a limitation. This is load-bearing because the Discussion attributes the gain to 'extensive training on supervised semantic tasks.'
  2. [§2.1 and Methods 'Linear mixed-effects models'] The unit of observation for the mixed-effects models is not specified. In brain encoding, the dependent variable is a per-voxel correlation, but the random-effect structure is only (1 | participant). If observations are voxels, this ignores within-participant voxel correlations and can make p-values anticonservative (note the SE=0.00 values in §2.1). If observations are participant-level, the effective N is only 8 per condition and the reported t-values would be implausible. Please report the number of observations, clarify the unit, and, if voxel-level, add a random intercept for voxels (or use participant-level summaries) and show that the LM ranking survives.
  3. [§2.1, Fig. 3, Methods 'Voxel-wise encoding models'] Encoding accuracies are reported after selecting the best layer per voxel and averaging only over voxels that passed FDR-corrected permutation tests. Since the permutation procedure is described per voxel but not per layer, it is unclear whether multiple-layer selection is accounted for. This selection can inflate absolute correlations and may bias comparisons if models differ in layer count or noise. Please provide supplementary analyses with a fixed layer or whole-ROI averages without per-voxel best-layer selection, to confirm that the across-caption and across-LM rankings are not artifacts of selection.
minor comments (6)
  1. [Abstract; §2.1; Discussion] The abstract says machine-generated captions 'often surpass human-annotated captions used in previous work,' but the comparison uses a single randomly sampled COCO caption, whereas previous work averaged the five COCO captions. The Discussion discloses that averaged COCO embeddings achieve comparable predictivity (Supp. Fig. 7); the wording should be qualified throughout.
  2. [§2.1] There is a typo in the sentence comparing Phi-4 perplexity: '49.64.14' should read '49.64'.
  3. [Methods 'Caption embeddings'] BERT is described as a masked-LM model, not autoregressive, so the binary 'autoregressive LMs vs text embedders' framing in the Abstract and Results excludes BERT awkwardly; consider describing the model set as 'early masked-LM and autoregressive models vs. recent text embedders.'
  4. [Fig. 3 and Fig. 5] Error bars are standard deviations across participants; in RSA, correlations are normalized by noise ceilings, but it is unclear whether layer selection is performed on raw or normalized values. Please clarify.
  5. [Methods 'Image features'] ImageNet classification is described as 'minimal language supervision'; this is better termed supervised image classification, since labels are not captions.
  6. [§2.3 and §2.4] The claim that predictivity/alignment 'peaks immediately after' the high-dimensional phase is based on visual inspection of Figs. 6–8; consider a quantitative test (e.g., comparing layer-wise correlations before/after the dimensionality peak) to support this temporal claim.

Circularity Check

1 steps flagged · score 1.0 of 10

No circular derivation in the central brain/behaviour benchmarks; one peripheral metric-interpretation step is partly definitional (visualness is a summed score, so its correlation with caption length is built into the metric).

  1. self definitional [Section 2, paragraph after Table 1]
    "Caption length positively correlates with visualness for all caption types (0.77< r <0.88,p≪0.001), confirming that longer captions provide additional visual information"

    The 'visualness' metric is defined in Methods as summing word-level Lancaster visualness ratings over the caption's words. Word count is exactly the number of terms in that sum, so a positive length-visualness correlation is largely guaranteed by the definition (more summands -> higher sum), independent of whether the added words are visually informative. The sentence treats this correlation as empirical confirmation that longer captions add visual information, but the confirmation is partially contained in the metric's construction. This step is peripheral: it is used only to describe caption-type differences, not to derive the brain-predictivity or alignment results.

full rationale

Score 1 rather than 0 only because of the peripheral visualness/length tautology; the paper's main derivation chain is self-contained. The central results are obtained by feeding fixed, publicly available caption embeddings (from text-only LMs) into standard ridge encoding models and RSA, evaluated against held-out NSD fMRI beta weights and behavioural dissimilarity judgments. No quantity that is defined in terms of the target brain/behaviour data is used as a predictor, and no parameter is fit to one subset and then reported as a prediction of a closely related subset: the best-layer selection is a standard (if optimistic) model-selection step, and the claims about intermediate-depth peaks come from separate layer-wise curves, not from the selected maxima. The only self-referential element is the definitional correlation between caption length and summed visualness, which is not load-bearing. The interpretation that text embedders win because of semantic fine-tuning is underdetermined by confounds such as model scale, base model, and pooling scheme, but confounding is a validity risk, not circularity. No self-citation chain or imported uniqueness theorem carries the argument.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new entities are introduced. Two preprocessing constants and a layer-selection rule are hand-set; the main interpretive load is carried by the abstraction-phase mapping and by the implicit assumption that the LM comparison isolates training objective. The COCO caption sampling choice is disclosed and its effect on the headline comparison is acknowledged.

free parameters (3)
  • PCA variance retained before ridge regression = 0.99 (99%)
    Chosen by hand; affects the dimensionality of features entering the encoding model (§4, Voxel-wise encoding models).
  • GRIDE maximum neighbour rank L = 5436 (906 images × 6 caption types)
    Sets the neighbourhood sizes for intrinsic dimensionality estimates; the layer-phase interpretation depends on these estimates (§4, Intrinsic dimensionality).
  • Layer selection for reported correlations = argmax over layers, per participant
    Brain and behavioural results report the best-performing layer for each participant/model; a selection over many layers can inflate the reported maximum, though applied consistently across conditions (Figs. 3, 5, 8).
assumptions (4)
  • domain assumption The fMRI responses in face-, body-, and place-selective ROIs contain semantic/categorical structure that a text caption can carry.
    Motivates why caption embeddings should predict these voxels at all (§1, §2).
  • ad hoc to paper The five LMs differ in the way that matters for the comparison; training objective (semantic fine-tuning vs autoregressive pretraining) is the operative factor, not model scale, data, or pooling.
    Used to conclude text embedders outperform autoregressive LMs; no matched-size or ablation control is provided (Discussion, 'Concerning the language models...').
  • domain assumption Intrinsic dimensionality peaks signal the emergence of abstract syntactic/semantic structure.
    Borrowed from [64,66]; the paper uses this mapping to interpret layer-wise curves (§2.3).
  • ad hoc to paper Comparing machine-generated captions to a single randomly sampled MS COCO caption is a valid way to assess caption quality against 'previous work'.
    The abstract's 'surpassing human-annotated captions' depends on this; prior work averaged the five captions, which the authors acknowledge changes results (Discussion, Supp. Fig. 7).

how reviews work

0 comments
Cite this review

Pith. "Pith review of What Makes Linguistic Representations Good Models of High-Level Visual Perception in the Human Brain?." pith.science (2026). https://pith.science/paper/56P4SMGE

@misc{pith2026260716214,
  author       = {Pith},
  title        = {Pith review of: What Makes Linguistic Representations Good Models of High-Level Visual Perception in the Human Brain?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/56P4SMGE}},
  note         = {Machine review of arXiv:2607.16214}
}
read the original abstract

Image descriptions represented with language models (LMs) predict human brain responses to naturalistic images in high-level visual regions, but the factors driving this predictivity remain unclear. To investigate this, we systematically studied how images are described and which language models are used to embed those descriptions. For a common set of images, we considered six caption types -- including human-annotated and multiple machine-generated captions -- differing along several dimensions. Each caption was represented with five LMs, spanning autoregressive LMs trained to predict upcoming words and text embedders, i.e., LMs fine-tuned on semantic tasks requiring sentence/document-level representations. Machine-generated captions yielded significant brain predictivity and alignment, often surpassing human-annotated captions used in previous work. Across caption types, text embedders consistently outperformed autoregressive LMs, a pattern replicated when measuring behavioural alignment with image-similarity judgments. Analyses of caption representations from different model layers further revealed that both brain predictivity and behavioural alignment peak at intermediate network depth, shortly after a point thought to mark the emergence of syntactic and semantic structure. Altogether, our results demonstrate that both the content of image captions and the LM used to represent them influence brain- and behaviour-modelling performance, establishing caption embeddings as a useful tool for studying high-level visual perception.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

103 extracted references · 2 canonical work pages

  1. [1]

    Deep learning: the good, the bad, and the ugly.Annual review of vision science5, 399–426 (2019)

    Serre, T. Deep learning: the good, the bad, and the ugly.Annual review of vision science5, 399–426 (2019)

    2019

  2. [2]

    A.et al.A deep learning framework for neuroscience.Nature neuroscience22, 1761–1770 (2019)

    Richards, B. A.et al.A deep learning framework for neuroscience.Nature neuroscience22, 1761–1770 (2019)

    2019

  3. [3]

    & Dobs, K

    Kanwisher, N., Khosla, M. & Dobs, K. Using artificial neural networks to ask ‘why’questions of minds and brains.Trends in Neurosciences46, 240–254 (2023)

    2023

  4. [4]

    Doerig, A.et al.The neuroconnectionist research programme.Nature Reviews Neuroscience24, 431–450 (2023)

    2023

  5. [5]

    URL https://openreview.net/ forum?id=Hiq7lUh4Yn

    Sucholutsky, I.et al.Getting aligned on representational alignment.Trans- actions on Machine Learning Research(2025). URL https://openreview.net/ forum?id=Hiq7lUh4Yn

    2025

  6. [6]

    Deep neural networks: a new framework for modeling biological vision and brain information processing.Annual review of vision science1, 417–446 (2015)

    Kriegeskorte, N. Deep neural networks: a new framework for modeling biological vision and brain information processing.Annual review of vision science1, 417–446 (2015)

    2015

  7. [7]

    L.et al.Performance-optimized hierarchical models predict neural responses in higher visual cortex.Proceedings of the national academy of sciences 111, 8619–8624 (2014)

    Yamins, D. L.et al.Performance-optimized hierarchical models predict neural responses in higher visual cortex.Proceedings of the national academy of sciences 111, 8619–8624 (2014)

    2014

  8. [8]

    & Thirion, B

    Eickenberg, M., Gramfort, A., Varoquaux, G. & Thirion, B. Seeing it all: Convolutional network layers map the function of the human visual system. NeuroImage152, 184–194 (2017)

    2017

Show all 103 references
  1. [9]

    & Kriegeskorte, N

    Khaligh-Razavi, S.-M. & Kriegeskorte, N. Deep supervised, but not unsuper- vised, models may explain it cortical representation.PLoS computational biology 10, e1003915 (2014)

    2014

  2. [10]

    & Van Gerven, M

    G¨ u¸ cl¨ u, U. & Van Gerven, M. A. Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream.Journal of Neuroscience35, 10005–10014 (2015)

    2015

  3. [11]

    I.et al.Distinct contributions of functional and deep neural network features to representational similarity of scenes in human brain and behavior

    Groen, I. I.et al.Distinct contributions of functional and deep neural network features to representational similarity of scenes in human brain and behavior. eLife7, e32962 (2018). URL https://doi.org/10.7554/eLife.32962. 24

    2018 doi

  4. [12]

    C., Abbott, J

    Peterson, J. C., Abbott, J. T. & Griffiths, T. L. Evaluating (and improving) the correspondence between deep neural networks and human representations. Cognitive science42, 2648–2669 (2018)

    2018

  5. [13]

    M., Zaremba, W., Fergus, R

    Lake, B. M., Zaremba, W., Fergus, R. & Gureckis, T. M. Deep neural networks predict category typicality ratings for images.Proceedings of the annual meeting of the cognitive science society, Vol. 37 (2015) (2015). URL https://escholarship. org/uc/item/4mv9c98f

    2015

  6. [14]

    & Ghanem, B

    Dubey, R., Peterson, J., Khosla, A., Yang, M.-H. & Ghanem, B. What makes an object memorable?Proceedings of the IEEE international conference on computer vision, 1089–1097 (2015) (2015)

    2015

  7. [15]

    & Op de Beeck, H

    Kubilius, J., Bracci, S. & Op de Beeck, H. P. Deep neural networks as a com- putational model for human shape sensitivity.PLoS computational biology12, e1004896 (2016)

    2016

  8. [16]

    & Kellman, P

    Baker, N., Lu, H., Erlikhman, G. & Kellman, P. J. Deep convolutional networks do not classify based on global object shape.PLoS computational biology14, e1006613 (2018)

    2018

  9. [17]

    Geirhos, R.et al.ImageNet-trained CNNs are biased towards texture; increas- ing shape bias improves accuracy and robustness.International conference on learning representations(2018) (2018)

    2018

  10. [18]

    Vaswani, A.et al.Attention is all you need.Advances in neural information processing systems30(2017)

    2017

  11. [19]

    & Dosovitskiy, A

    Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C. & Dosovitskiy, A. Ran- zato, M., Beygelzimer, A., Dauphin, Y., Liang, P. & Vaughan, J. W. (eds) Do vision transformers see like convolutional neural networks? (eds Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P. & Vaugh...

    2021

  12. [20]

    S., Kay, K

    Conwell, C., Prince, J. S., Kay, K. N., Alvarez, G. A. & Konkle, T. A large- scale examination of inductive biases shaping high-level visual representation in brains and machines.Nature communications15, 9383 (2024)

    2024

  13. [21]

    Raugel, J.et al.Disentangling the factors of convergence between brains and computer vision models.arXiv preprint arXiv:2508.18226(2025)

    2025 arXiv

  14. [22]

    A., Ritchie, J

    Zeman, A. A., Ritchie, J. B., Bracci, S. & Op de Beeck, H. Orthogonal repre- sentations of object shape and category in deep convolutional neural networks and human visual cortex.Scientific reports10, 2453 (2020). 25

    2020

  15. [23]

    G.et al.Representation of locomotive action affordances in human behavior, brains, and deep neural networks.Proceedings of the National Academy of Sciences122, e2414005122 (2025)

    Bartnik, C. G.et al.Representation of locomotive action affordances in human behavior, brains, and deep neural networks.Proceedings of the National Academy of Sciences122, e2414005122 (2025)

    2025

  16. [24]

    Y., Kay, K., Naselaris, T., Tarr, M

    Wang, A. Y., Kay, K., Naselaris, T., Tarr, M. J. & Wehbe, L. Better models of human high-level visual cortex emerge from natural language supervision with a large and diverse dataset.Nature Machine Intelligence5, 1415–1426 (2023)

    2023

  17. [25]

    S., Konkle, T., Isik, L

    Conwell, C., Prince, J. S., Konkle, T., Isik, L. & Alvarez, G. A. Rethinking language-alignment in human visual cortex with syntax manipulation and word models.OpenReview(2024)

    2024

  18. [26]

    Doerig, A.et al.High-level visual representations in the human brain are aligned with large language models.Nature Machine Intelligence1–15 (2025)

    2025

  19. [27]

    McCarthy, G., Puce, A., Gore, J. C. & Allison, T. Face-specific processing in the human fusiform gyrus.Journal of cognitive neuroscience9, 605–610 (1997)

    1997

  20. [28]

    & Yovel, G

    Kanwisher, N. & Yovel, G. The fusiform face area: A cortical region specialized for the perception of faces.Philosophical Transactions of the Royal Society B: Biological Sciences361, 2109–2128 (2006)

    2006

  21. [29]

    Y., Freiwald, W

    Tsao, D. Y., Freiwald, W. A., Tootell, R. B. & Livingstone, M. S. A cortical region consisting entirely of face-selective cells.Science311, 670–674 (2006)

    2006

  22. [30]

    Bracci, S., Ietswaart, M., Peelen, M. V. & Cavina-Pratesi, C. Dissociable neural responses to hands and non-hand body parts in human left extrastriate visual cortex.Journal of neurophysiology103, 3389–3397 (2010)

    2010

  23. [31]

    E., Jiang, Y., Shuman, M

    Downing, P. E., Jiang, Y., Shuman, M. & Kanwisher, N. A cortical area selective for visual processing of the human body.Science293, 2470–2473 (2001)

    2001

  24. [32]

    Peelen, M. V. & Downing, P. E. Selectivity for the human body in the fusiform gyrus.Journal of neurophysiology93, 603–608 (2005)

    2005

  25. [33]

    & Kanwisher, N

    Epstein, R. & Kanwisher, N. A cortical representation of the local visual environment.Nature392, 598–601 (1998)

    1998

  26. [34]

    building

    Aguirre, G. K., Zarahn, E. & D’Esposito, M. An area within human ventral cortex sensitive to “building” stimuli: evidence and implications.Neuron21, 373–383 (1998)

    1998

  27. [35]

    Brain123, 291–307 (2000)

    Cohen, L.et al.The visual word form area: spatial and temporal characterization of an initial stage of reading in normal subjects and posterior split-brain patients. Brain123, 291–307 (2000)

    2000

  28. [36]

    Baker, C. I.et al.Visual word processing and experiential origins of functional selectivity in human extrastriate cortex.Proceedings of the National Academy 26 of Sciences104, 9087–9092 (2007)

    2007

  29. [37]

    Lin, T.-Y.et al.Microsoft COCO: Common objects in context.Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, 740–755 (2014) (Springer, 2014)

    2014

  30. [38]

    J.et al.A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence.Nature neuroscience25, 116–126 (2022)

    Allen, E. J.et al.A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence.Nature neuroscience25, 116–126 (2022)

    2022

  31. [39]

    URL https://openreview.net/forum?id= zKv8qUL V6n

    Li, B.et al.LLaV A-OneVision: Easy visual task transfer.Transactions on Machine Learning Research(2025). URL https://openreview.net/forum?id= zKv8qUL V6n

    2025

  32. [40]

    Abouelenin, A.et al.Phi-4-Mini technical report: Compact yet powerful multi- modal language models via mixture-of-loras.arXiv preprint arXiv:2503.01743 (2025)

    2025 arXiv

  33. [41]

    Pixtral large model card (2024)

    MistralAI. Pixtral large model card (2024). URL https://mistral.ai/news/ pixtral-large

    2024

  34. [42]

    Bai, S.et al.Qwen2.5-VL technical report.arXiv preprint arXiv:2502.13923 (2025)

    2025 arXiv

  35. [43]

    Deitke, M.et al.Molmo and pixmo: Open weights and open data for state-of- the-art vision-language models.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 91–104 (2025) (2025)

    2025

  36. [44]

    H.et al.Ministral 3.arXiv preprint arXiv:2601.08584(2026)

    Liu, A. H.et al.Ministral 3.arXiv preprint arXiv:2601.08584(2026)

    2026 arXiv

  37. [45]

    & Carney, J

    Lynott, D., Connell, L., Brysbaert, M., Brand, J. & Carney, J. The Lan- caster Sensorimotor Norms: multidimensional measures of perceptual and action strength for 40,000 English words.Behavior research methods52, 1271–1291 (2020)

    2020

  38. [46]

    & Toutanova, K

    Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Burstein, J., Doran, C. & Solorio, T. (eds) BERT: Pre-training of deep bidirectional transformers for language understanding. (eds Burstein, J., Doran, C. & Solorio, T.)Proceedings of the 2019 Conference of the North American C...

    2019

  39. [47]

    Radford, A.et al.Language models are unsupervised multitask learners.OpenAI blog1, 9 (2019)

    2019

  40. [48]

    Grattafiori, A.et al.The Llama 3 herd of models.arXiv preprint arXiv:2407.21783(2024). 27

    2024 arXiv

  41. [49]

    Zhang, Y.et al.Qwen3 Embedding: Advancing text embedding and reranking through foundation models.arXiv preprint arXiv:2506.05176(2025)

    2025 arXiv

  42. [50]

    Zhao, X.et al.KaLM-Embedding-V2: Superior training techniques and data inspire a versatile embedding model.arXiv preprint arXiv:2506.20923(2025)

    2025

  43. [51]

    Schrimpf, M.et al.The neural architecture of language: Integrative modeling converges on predictive processing.Proceedings of the National Academy of Sciences118, e2105646118 (2021)

    2021

  44. [52]

    & King, J.-R

    Caucheteux, C. & King, J.-R. Brains and algorithms partially converge in natural language processing.Communications biology5, 134 (2022)

    2022

  45. [53]

    & Sun, J

    He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, 770–778 (2016) (2016)

    2016

  46. [54]

    URL https://openreview.net/forum?id=YicbFdNTTy

    Dosovitskiy, A.et al.An image is worth 16x16 words: Transformers for image recognition at scale.International Conference on Learning Representations (2021) (2021). URL https://openreview.net/forum?id=YicbFdNTTy

    2021

  47. [55]

    Proceedings of the IEEE/CVF international conference on computer vision, 9650–9660 (2021) (2021)

    Caron, M.et al.Emerging properties in self-supervised vision transformers. Proceedings of the IEEE/CVF international conference on computer vision, 9650–9660 (2021) (2021)

    2021

  48. [56]

    Transactions on Machine Learning Research(2024)

    Oquab, M.et al.DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research(2024). URL https://openreview. net/forum?id=a68SUt6zFt. Featured Certification

    2024

  49. [57]

    Deng, J.et al.ImageNet: A large-scale hierarchical image database.2009 IEEE Conference on Computer Vision and Pattern Recognition, 248–255 (2009) (2009)

    2009

  50. [58]

    Radford, A.et al.Learning transferable visual models from natural language supervision.International conference on machine learning, 8748–8763 (2021) (PmLR, 2021)

    2021

  51. [59]

    N., Nishimoto, S

    Naselaris, T., Kay, K. N., Nishimoto, S. & Gallant, J. L. Encoding and decoding in fMRI.Neuroimage56, 400–410 (2011)

    2011

  52. [60]

    & Gallant, J

    Dupr´ e la Tour, T., Visconti di Oleggio Castello, M. & Gallant, J. L. The vox- elwise encoding model framework: a tutorial introduction to fitting encoding models to fmri data.Imaging Neuroscience3, imag a 00575 (2025)

    2025

  53. [61]

    & De Martino, F

    Lage-Castellanos, A., Valente, G., Formisano, E. & De Martino, F. Meth- ods for computing the maximum performance of computational models of fmri responses.PLoS computational biology15, e1006397 (2019). 28

    2019

  54. [62]

    & Bandettini, P

    Kriegeskorte, N., Mur, M. & Bandettini, P. A. Representational similarity analysis—connecting the branches of systems neuroscience.Frontiers in systems neuroscience2, 249 (2008)

    2008

  55. [63]

    & Baroni, M

    Cheng, E., Kervadec, C. & Baroni, M. Bridging information-theoretic and geo- metric compression in language models.Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 12397–12420 (2023) (2023)

    2023

  56. [64]

    (eds Oh, A.et al.)Advances in Neural Infor- mation Processing Systems, Vol

    Valeriani, L.et al.Oh, A.et al.(eds) The geometry of hidden representations of large transformer models. (eds Oh, A.et al.)Advances in Neural Infor- mation Processing Systems, Vol. 36, 51234–51252 (2023) (Curran Associates, Inc., 2023). URL https://proceedings.neurips.cc/paper...

    2023

  57. [65]

    & Purver, M

    Razzhigaev, A.et al.Graham, Y. & Purver, M. (eds) The shape of learning: Anisotropy and intrinsic dimensions in transformer-based models. (eds Graham, Y. & Purver, M.)Findings of the Association for Computational Linguistics: EACL 2024, 868–874 (2024) (Association for Computat...

    2024

  58. [66]

    URL https://openreview.net/forum?id=0fD3iIBhlV

    Cheng, E.et al.Emergence of a high-dimensional abstraction phase in language transformers.The Thirteenth International Conference on Learning Represen- tations(2025) (2025). URL https://openreview.net/forum?id=0fD3iIBhlV

    2025

  59. [67]

    & Mira, A

    Denti, F., Doimo, D., Laio, A. & Mira, A. The generalized ratios intrinsic dimension estimator.Scientific Reports12, 20005 (2022)

    2022

  60. [68]

    L., Groen, I

    King, M. L., Groen, I. I., Steel, A., Kravitz, D. J. & Baker, C. I. Similarity judgments and cortical visual responses reflect different properties of object and scene categories in naturalistic images.NeuroImage197, 368–382 (2019)

    2019

  61. [69]

    S., Behrmann, M., Humphreys, K

    Scherf, K. S., Behrmann, M., Humphreys, K. & Luna, B. Visual category- selectivity for faces, places and objects emerges along different developmental trajectories.Developmental science10, F15–F30 (2007)

    2007

  62. [70]

    P., Pillet, I

    Op de Beeck, H. P., Pillet, I. & Ritchie, J. B. Factors determining where category-selective areas emerge in visual cortex.Trends in cognitive sciences 23, 784–797 (2019)

    2019

  63. [71]

    Peelen, M. V. & Downing, P. E. Category selectivity in human visual cortex: Beyond visual object recognition.Neuropsychologia105, 177–183 (2017)

    2017

  64. [72]

    Liao, W.et al.Differentiating ChatGPT-generated and human-written medical texts: quantitative study.JMIR medical education9, e48904 (2023)

    2023

  65. [73]

    F., Van Der Linden, D., Bakker, A

    Hopstaken, J. F., Van Der Linden, D., Bakker, A. B. & Kompier, M. A. A multi- faceted investigation of the link between mental fatigue and task disengagement. 29 Psychophysiology52, 305–315 (2015)

    2015

  66. [74]

    5, 321–328 (2011) (2011)

    Rogstadius, J.et al.An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets.Proceedings of the international AAAI conference on web and social media, Vol. 5, 321–328 (2011) (2011)

    2011

  67. [75]

    A., Burns, K., Darrell, T

    Rohrbach, A., Hendricks, L. A., Burns, K., Darrell, T. & Saenko, K. Riloff, E., Chiang, D., Hockenmaier, J. & Tsujii, J. (eds) Object hallucination in image captioning. (eds Riloff, E., Chiang, D., Hockenmaier, J. & Tsujii, J.)Proceedings of the 2018 Conference on Empirical Me...

    2018

  68. [76]

    F., G´ omez, L

    Biten, A. F., G´ omez, L. & Karatzas, D. Let there be a clock on the beach: Reduc- ing object hallucination in image captioning.Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 1381–1390 (2022) (2022)

    2022

  69. [77]

    & Reimers, N

    Muennighoff, N., Tazi, N., Magne, L. & Reimers, N. Vlachos, A. & Augen- stein, I. (eds) MTEB: Massive Text Embedding Benchmark. (eds Vlachos, A. & Augenstein, I.)Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, 2014–2...

    2014

  70. [78]

    URL https://openreview.net/forum?id=zl3pfz4VCV

    Enevoldsen, K.et al.MMTEB: Massive Multilingual Text Embedding Bench- mark.The Thirteenth International Conference on Learning Representations (2025) (2025). URL https://openreview.net/forum?id=zl3pfz4VCV

    2025

  71. [79]

    & Wehbe, L

    Schwartz, D., Toneva, M. & Wehbe, L. Wallach, H.et al.(eds) Inducing brain- relevant bias in natural language processing models. (eds Wallach, H.et al.) Advances in Neural Information Processing Systems, Vol. 32 (2019) (Curran Associates, Inc., 2019). URL https://proceedings.n...

    2019

  72. [80]

    Goldstein, A.et al.Temporal structure of natural language processing in the human brain corresponds to layered hierarchy of large language models.Nature communications16, 10529 (2025)

    2025

  73. [81]

    Cheng, E., Vaidya, A. R. & Antonello, R. Abstraction induces the brain alignment of language and speech models.arXiv preprint arXiv:2602.04081 (2026)

    2026 arXiv

  74. [82]

    Goldstein, A.et al.Shared computational principles for language processing in humans and deep language models.Nature neuroscience25, 369–380 (2022). 30

    2022

  75. [83]

    & King, J.-R

    Caucheteux, C., Gramfort, A. & King, J.-R. Evidence of a predictive coding hierarchy in the human brain listening to speech.Nature human behaviour7, 430–441 (2023)

    2023

  76. [84]

    van Dyck, L. E. & Dobs, K. Face and body representations converge along the visual hierarchy in models and cortex.bioRxiv2026–02 (2026)

    2026

  77. [85]

    URL https://openreview.net/forum?id= O-G91-4cMdv

    Marjieh, R.et al.Words are all you need? Language as an approximation for human similarity judgments.The Eleventh International Conference on Learn- ing Representations(2023) (2023). URL https://openreview.net/forum?id= O-G91-4cMdv

    2023

  78. [86]

    Linguistically modulated perception and cognition: The label- feedback hypothesis.Frontiers in psychology3, 54 (2012)

    Lupyan, G. Linguistically modulated perception and cognition: The label- feedback hypothesis.Frontiers in psychology3, 54 (2012)

    2012

  79. [87]

    & Clark, A

    Lupyan, G. & Clark, A. Words and the world: Predictive coding and the language-perception-cognition interface.Current Directions in Psychological Science24, 279–284 (2015)

    2015

  80. [88]

    C., de Lange, F

    Simanova, I., Francken, J. C., de Lange, F. P. & Bekkering, H. Linguistic priors shape categorical perception.Language, Cognition and Neuroscience31, 159– 165 (2016)

    2016

  81. [89]

    & Mur, M

    Kriegeskorte, N. & Mur, M. Inverse MDS: Inferring dissimilarity structure from multiple item arrangements.Frontiers in psychology3, 245 (2012)

    2012

  82. [90]

    Wolf, T.et al.HuggingFace’s Transformers: State-of-the-art natural language processing.arXiv preprint arXiv:1910.03771(2019)

    1910 arXiv

  83. [91]

    & Boyd, A

    Honnibal, M., Montani, I., Van Landeghem, S. & Boyd, A. spaCy: Industrial- strength natural language processing in Python (2020)

    2020

  84. [92]

    Yang, A.et al.Qwen3 technical report.arXiv preprint arXiv:2505.09388(2025)

    2025 arXiv

  85. [93]

    Team, G.et al.Gemma 3 technical report.arXiv preprint arXiv:2503.19786 (2025)

    2025 arXiv

  86. [94]

    Tuckute, G.et al.Driving and suppressing the human language network using large language models.Nature Human Behaviour8, 544–561 (2024)

    2024

  87. [95]

    Hosseini, E. A.et al.Artificial neural network language models predict human brain responses to language even after a developmentally realistic amount of training.Neurobiology of Language5, 43–63 (2024)

    2024

  88. [96]

    G.et al.Representation of locomotive action affordances in human behavior, brains, and deep neural networks.Proceedings of the National Academy of Sciences122, e2414005122 (2025)

    Bartnik, C. G.et al.Representation of locomotive action affordances in human behavior, brains, and deep neural networks.Proceedings of the National Academy of Sciences122, e2414005122 (2025). URL https://www.pnas.org/ doi/abs/10.1073/pnas.2414005122. 31

    2025 doi

  89. [97]

    & Hochberg, Y

    Benjamini, Y. & Hochberg, Y. Controlling the false discovery rate: a practi- cal and powerful approach to multiple testing.Journal of the Royal statistical society: series B (Methodological)57, 289–300 (1995)

    1995

  90. [98]

    Virtanen, P.et al.SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.Nature Methods17, 261–272 (2020)

    2020

  91. [99]

    & Walker, S

    Bates, D., M¨ achler, M., Bolker, B. & Walker, S. Fitting linear mixed-effects models using lme4.Journal of Statistical Software67, 1–48 (2015)

    2015

  92. [100]

    Kuznetsova, A., Brockhoff, P. B. & Christensen, R. H. B. lmerTest package: Tests in linear mixed effects models.Journal of Statistical Software82, 1–26 (2017)

    2017

  93. [101]

    Pedregosa, F.et al.Scikit-learn: Machine learning in Python.Journal of Machine Learning Research12, 2825–2830 (2011)

    2011

  94. [102]

    Nili, H.et al.A toolbox for representational similarity analysis.PLoS computational biology10, e1003553 (2014)

    2014

  95. [103]

    Patterns100589 (2022)

    Glielmo, A.et al.Dadapy: Distance-based analysis of data-manifolds in python. Patterns100589 (2022). URL https://www.sciencedirect.com/science/article/ pii/S2666389922002070. Acknowledgements This project originated during A. B.’s research visit to M.-F. M. and her LIIR lab at...

    2022

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.