Pith. sign in

REVIEW 4 major objections 5 minor 76 references

Exploring The Visual Feature Space for Multimodal Neural Decoding

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VINDEX, aligning fMRI to nine coarse-to-fine visual tokens with a denoising objective, produces the most accurate detailed brain captions and salient question answering in the paper's MG-BrainDub benchmark.

desk verdict Sensible extension of UMBRAE with a feature-space study and a new benchmark, but the benchmark's LLaVA-generated references weaken the headline NF9 claim. read the letter →

arxiv 2505.15755 v1 pith:PEJ4TOGA submitted 2025-05-21 cs.CV

Weihao Xia , Cengiz Oztireli This is my paper · ORCID
classification cs.CV
keywords fMRIdecodingmultimodallargelanguagemodelsvisualfeaturespacesnestedfeaturesdenoisingdiffusiondetailedcaptioningsalientquestionansweringzero-shotbrain
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the choice of visual feature space, not the size or sophistication of the language model, governs how much fine-grained visual detail can be decoded from brain activity. It introduces VINDEX, a zero-shot pipeline that trains a brain encoder on fMRI signals to predict the visual tokens used by a pre-trained multimodal large language model, then feeds those predicted tokens to the model for captioning and question answering. Across four feature-space designs — a single encoder, a mixture of encoders, aggregated multi-layer features, and coarse-to-fine nested token sets — the paper finds that a nested set of nine visual tokens gives the most accurate detailed captions and salient-object question answering, outperforming denser feature representations. To measure this, it builds the MG-BrainDub benchmark, whose metrics score objects, attributes, and relations separately rather than relying on n-gram matching. A reader should care because the result suggests that brain signals contain less visual information than images do, so matching the token budget to brain data can be more effective than throwing more visual computation at the problem.

What carries the argument

The central object is nested features: the target image is encoded by a vision transformer and its patch tokens are repeatedly downsampled by $2\times2$ pooling to form a hierarchy of token sets (144, 36, 9, 1 tokens), each coarser set derived from the finer one, so the brain encoder can be aligned with a token budget matching the information carried by brain signals. The other load-bearing piece is the masked denoising objective: during training a lightweight MLP denoiser learns to predict noise added to the clean visual tokens, conditioned on the brain encoder's prediction, and its loss is added to the regression loss, acting as implicit augmentation and regularization that stabilizes training and improves caption quality at no extra inference cost.

What would settle it

Take a random subset of the shared test images, ask independent human raters to write free-form descriptions and answer the benchmark's salient question-answering items, then score VINDEX's decoded captions against those human references using MG-BrainDub's object/attribute/relation metrics; if the pattern of results across feature spaces (nine-token nested best, aggregated worst) does not reproduce with human references, the paper's ranking is an artifact of using machine-generated ground truth.

Watch

Extended reading notes

Core claim

VINDEX treats the visual component of a multimodal large language model as the target space for brain alignment. A shared brain encoder maps fMRI responses to image features from a chosen vision encoder, using a weighted sum of a regression loss and a masked denoising loss in which a small denoiser predicts noise added to the target visual tokens conditioned on the brain prediction. During inference, the predicted tokens replace image tokens and flow through the connector and language model, enabling zero-shot instruction-following tasks with no textual or spatial annotations at training time. The paper's central empirical discovery is that among the feature spaces tested, the nested-features setting with nine visual tokens (coarse-to-fine downsampling of the single encoder's features) yields the best performance on detailed description and salient question answering, while dense aggregated features collapse to garbled outputs, and single-encoder features lag behind despite larger and more recent language models.

Load-bearing premise

The benchmark's ground-truth detailed captions are written by a multimodal language model (with human correction), not obtained from the human subjects themselves, so the measured 'detail' is fidelity to that model's descriptions of the stimuli rather than to what the subjects actually perceived.

Editorial extensions

If this is right

  • A brain decoder can be built zero-shot for any multimodal language model whose vision encoder matches the trained brain encoder, without collecting captions or bounding-box annotations during training.
  • Token pruning and merging ideas from vision can transfer to brain decoding: the leading result uses only nine visual tokens, implying much lower compute for fMRI-to-language than image-based multimodal pipelines.
  • Dense multi-layer feature representations are not just wasteful but harmful for brain decoding, producing gibberish or hallucinated output, so future brain-decoding architectures should not blindly inherit full-resolution visual features.
  • Detailed caption evaluation should use object/attribute/relation precision, recall, and F1 rather than n-gram or embedding-based metrics, since rule-based and CLIP-style metrics mis-rank long brain-decoded captions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If nine tokens encode what fMRI preserves, then the brain's visual bottleneck — not the model — is the limiting factor, and decoders for lower-spatial-resolution recording modalities such as EEG or MEG would likely need even smaller token budgets.
  • The nested-feature result suggests a design principle: feature spaces should be chosen so their information capacity matches the fidelity of the neural signal, a criterion that could be tested prospectively by ablating per-subject token counts.
  • Because the denoiser is removed at inference, the same brain encoder could be retrained with different denoising strengths or noise schedules to trade robustness against overfitting, a dimension the paper explores only coarsely.
  • The benchmark's reliance on machine-generated reference captions implies that rankings could shift if a stronger caption generator is used as ground truth, so adding a human-verified subset would make MG-BrainDub more stable across future model generations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies which visual feature spaces of multimodal large language models (MLLMs) best support zero-shot fMRI-to-text decoding. It proposes VINDEX, which trains a brain encoder to predict image features from CLIP, DINO, and SigLIP variants using a regression loss combined with a masked denoising objective, and then feeds the predicted tokens into LLaVA-family models for tasks such as concept localization, concise captioning, detailed captioning, and salient question answering. The paper also introduces MG-BrainDub, a benchmark for detailed descriptions and salient QA, and reports that a nested 9-token feature space (NF9) achieves the best detailed captioning and QA performance. Experiments are conducted on the Natural Scenes Dataset with standard train/test splits for four subjects.

Significance. If the evaluation-validity concerns are resolved, the paper is valuable to the brain-decoding community: it gives a systematic comparison of feature-space choices, a lightweight denoiser that stabilizes training, a publicly announced code release, and a new benchmark aimed at fine-grained evaluation. The concept-localization and concise-captioning results are compared against several existing methods and show competitive or superior numbers, which lends some independent credibility to the approach. However, the headline detailed-captioning and salient-QA claims rest almost entirely on MG-BrainDub, whose reference construction is not independent of the decoding MLLM; the paper is therefore not yet ready for acceptance without a strengthened evaluation.

major comments (4)
  1. [Sec. 4.1 / App. C.2 / Tab. 3] The reference standard for MG-BrainDub is internally inconsistent and is not an independent yardstick. Section 4.1 states that ground-truth descriptions were generated by MLLMs 'followed by manual error correction, missing element addition, and hallucination removal through expert human intervention,' but Appendix C.2 says only that 'Results from LLaVA-v1.5 7B (LLaVA) serve as pseudo ground truth for detailed captioning evaluation,' with no annotation counts, correction statistics, or inter-annotator agreement. Since the decoding MLLM in the NF9 row of Table 3 is also LLaVA-1.5 7B, the CAPTURE and object/attribute/relation F1 scores may reflect agreement with LLaVA's stylistic and hallucination priors rather than genuine brain-based decoding. Please provide a human-verified subset of the references, or references from a different model family, and report scores on that subset; without this, the claim that NF9 outperforms SE/ME/AF on detailed decoding is not externally validated.
  2. [Tab. 3] Table 3 reports a single run for each configuration with no error bars, confidence intervals, or significance tests. The NF9-versus-NF1 SQA gap is 1.24 accuracy points (83.83 vs. 82.59), and the CAPTURE gap is 0.0144 (0.5021 vs. 0.4877). Given the noise in fMRI data and the fact that the token count n was selected after inspecting these results, the claim that NF9 is the best feature space is not established. The authors should report repeated-seed or per-subject variability and should either pre-register the token count or validate it on a held-out split.
  3. [Tab. 2 (lower block) / Tab. 3] The detailed-captioning comparison includes only NeuroVLA as an external method; all other compared rows are VINDEX's own feature-space variants. No independent brain-decoding method, such as UMBRAE or MindEye2, is evaluated on MG-BrainDub under the same protocol. As a result, the paper does not support a claim of state-of-the-art detailed brain captioning; it only supports an internal comparison among feature spaces. Please add at least UMBRAE and one additional recent method to the MG-BrainDub evaluation.
  4. [Sec. 4.2 / Tab. 3] The SQA reference construction introduces additional annotation dependence that is not discussed in the validity analysis: salient objects are identified by SAM and input to 'the MLLM,' and the in-depth reasoning questions are based on GPT annotations. The paper does not report how many questions were manually verified or how often the MLLM's object and attribute labels were corrected. Because SQA accuracy is a headline result in Table 3, the SQA reference construction should be documented with the same rigor as the caption references, including human-verification statistics.
minor comments (5)
  1. [Sec. 5.5] Section 5.5 says 'using Salient Question Answering (SQA) (Sec. 5.5)', but SQA is described in Sec. 4.2; the cross-reference is wrong and should be corrected.
  2. [Tab. 2 / Tab. 3 captions] The relationship between the † symbol in Table 2 and the * symbol in Table 3 should be clarified: Table 3 says the starred models use the same settings as in Table 2, but Table 2's VINDEX is built on Shikra while Table 3's SE* row uses LLaVA-1.5 7B.
  3. [Abstract] The phrase 'The intrication of brain signals' in the abstract and introduction is unusual; 'complexity' or 'intricacy' appears to be intended.
  4. [App. C.1] The metric-calculation example appears to use mismatched candidate and reference captions: the candidate describes a city street with a truck, while the reference describes a beach, yet the matching example yields 100% precision. Please correct the example or add a note explaining why the reference objects listed are consistent with the candidate.
  5. [Sec. 5.1 / Tab. 4] The text in Section 6 correctly identifies the best denoiser setting as a one-depth MLP with width 1024 and beta 1.0, but Section 5.1 should state explicitly that this corresponds to configuration S4 in Table 4, since the table columns are not otherwise named in the text.

Circularity Check

1 steps flagged · score 4.0 of 10

MG-BrainDub's detailed-caption references are LLaVA-1.5 7B outputs while the decoding MLLM is also LLaVA-1.5 7B, making the NF9-best result self-referential; the alignment method itself is not circular.

  1. self definitional [Appendix C.2; see also Sec. 4.1 and Table 3 (Sec. 5.4)]
    "Results from LLaV A-v1.5 7B (LLaV A) [39] serve as pseudo ground truth for detailed captioning evaluation, while others, including LLaV A-MoF (MoF) [62], DenseConnector-v1.5 7B (DC) [71], and Matryoshka-MM-v1.5 7B (M3) [8] are for reference."

    The detailed-captioning benchmark that supports the headline ranking (NF9 best, Tab. 3) defines its reference captions as LLaVA-1.5 7B outputs ('pseudo ground truth'). The decoding MLLM for every VINDEX row in Tab. 3, including the winning NF9 row, is also LLaVA-1.5 7B. Thus the evaluation score is effectively similarity(LLaVA-1.5 7B(brain-predicted features), LLaVA-1.5 7B(image)): the reference is generated by the same model that writes the candidate, so the metric rewards reproducing LLaVA's language and content priors rather than independently verifying what the subject perceived. Sec.

full rationale

The core method is not circular: the brain encoder is trained to regress and denoise CLIP/DINO/SigLIP image features, and the denoiser is used only as a training regularizer, not as a prediction target. Concept localization, concise captioning on BrainHub, and SQA are evaluated against external or differently constructed references. The one significant circularity is the detailed-captioning benchmark: its references are LLaVA-1.5 7B outputs, while the decoding MLLM is also LLaVA-1.5 7B, and the paper itself labels these references as 'pseudo ground truth'. This makes the central NF9-best claim for detailed captioning self-referential, though the effect is partial because the benchmark also uses a T5-based element parser and the alignment objective itself is independent. I therefore assign a score of 4: one load-bearing self-referential evaluation contaminates the headline result, but the rest of the derivation chain is self-contained.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The ledger shows four post hoc tuning choices (token count, denoiser depth, width, loss weight) that the central NF9 result depends on. The benchmark validity rests on the unvalidated assumption that LLaVA-generated captions are an accurate ground truth. No new physical or computational entities are introduced.

free parameters (4)
  • Nested token count for NF (n) = 9 (best of {1, 9, 36, 144})
    Selected after evaluating all four token counts on the test set in Tab. 3; no validation split is reported, so this is a post hoc choice.
  • Denoiser depth d = 1 (best of {1, 2, 3})
    Ablation in Tab. 4 shows depth 1 yields the best CAPTURE score.
  • Denoiser width w = 1024 (best of {512, 1024})
    Ablation in Tab. 4 shows width 1024 is best.
  • Denoising loss weight beta = 1.0 (best among {0.5, 1.0, 1.5, 2.0})
    Ablation in Tab. 4 shows the default weight 1.0 gives the best CAPTURE score.
assumptions (4)
  • domain assumption NSD fMRI responses, preprocessed in nsdgeneral, contain enough signal to predict CLIP/DINO/SigLIP image features at the level needed for MLLM decoding.
    Sec. A.1 and Sec. 5.1 assume standard NSD preprocessing and feature alignment without demonstrating that the fMRI signal supports the claimed granularity.
  • domain assumption Pre-trained vision encoder features are a valid intermediate representation for brain decoding.
    Sec. 3 assumes aligning to V(v) and passing through the MLLM preserves brain-relevant visual information.
  • ad hoc to paper LLaVA-generated detailed captions are a reliable gold standard for evaluating brain decoding.
    Sec. 4.1 constructs ground truth with LLaVA plus manual correction; this assumption is load-bearing for the benchmark but not independently validated.
  • domain assumption Diffusion-style denoising loss acts as useful regularization for brain feature alignment.
    Sec. 3.2 relies on standard diffusion denoising theory to justify the auxiliary loss.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring The Visual Feature Space for Multimodal Neural Decoding." pith.science (2026). https://pith.science/paper/PEJ4TOGA

@misc{pith2026250515755,
  author       = {Pith},
  title        = {Pith review of: Exploring The Visual Feature Space for Multimodal Neural Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PEJ4TOGA}},
  note         = {Machine review of arXiv:2505.15755}
}
read the original abstract

The intrication of brain signals drives research that leverages multimodal AI to align brain modalities with visual and textual data for explainable descriptions. However, most existing studies are limited to coarse interpretations, lacking essential details on object descriptions, locations, attributes, and their relationships. This leads to imprecise and ambiguous reconstructions when using such cues for visual decoding. To address this, we analyze different choices of vision feature spaces from pre-trained visual components within Multimodal Large Language Models (MLLMs) and introduce a zero-shot multimodal brain decoding method that interacts with these models to decode across multiple levels of granularities. % To assess a model's ability to decode fine details from brain signals, we propose the Multi-Granularity Brain Detail Understanding Benchmark (MG-BrainDub). This benchmark includes two key tasks: detailed descriptions and salient question-answering, with metrics highlighting key visual elements like objects, attributes, and relationships. Our approach enhances neural decoding precision and supports more accurate neuro-decoding applications. Code will be available at https://github.com/weihaox/VINDEX.

Figures

Figures reproduced from arXiv: 2505.15755 by the authors.

Figure 1
Figure 1. Feature Spaces. We aim to capture multi-granular representations for zero-shot multimodal brain decoding by aligning brain signals across different feature spaces in MLLM’s visual component. The explored representative feature spaces include: (a) Single Encoder, which utilizes a single encoder to extract selected features; (b) Mixture of Encoders, using hybrid featured from different vision experts specialized in ta… view at source ↗
Figure 2
Figure 2. Training Procedure with Denoiser. Therefore, we introduce a masked denoising optimization objective as shown in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Example NSD Images. Below, we present detailed captions generated by MLLMs using images as input, alongside captions from our method using different feature spaces with brain signals as input. (d) (12 words) A red and yellow double decker bus on street next to trees. (…
Figure 5
Figure 5. Figure 5: Denoiser as a Training Stabilizer. (a) The vanilla regression loss decreases but exhibits significant oscillation; (b) The training process becomes less oscillatory with the incorporation of diffusion loss, which stabilizes the training and accelerates convergence [PI…
Figure 6
Figure 6. Figure 6: shows the UMAP visualization for predicted features from brain encoders B-CLIP224, B-CLIP336, B-DINO224, and B-SIGLIP384 (fMRI as input), along with the corresponding ground truth features from target vision encoders (associated visual stimuli as input). Refer to Tab. …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 65 canonical work pages

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023. 5

    arXiv 2023

  2. [2]

    A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence.Nature neuroscience, 25(1):116–126, 2022

    Emily J Allen, Ghislain St-Yves, Yihan Wu, Jesse L Breedlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence.Nature neuroscience, 25(1):116–126, 2022. 4, 7, 8, 9, 11, 19

    work page 2022

  3. [3]

    Spice: Semantic propositional image caption evaluation

    Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In ECCV, pages 382–398. Springer, 2016. 1, 4

    work page 2016

  4. [4]

    Leo: Boosting mixture of vision encoders for multimodal large language models.arXiv preprint arXiv:2501.06986, 2025

    Mozhgan Nasr Azadani, James Riddell, Sean Sedwards, and Krzysztof Czarnecki. Leo: Boosting mixture of vision encoders for multimodal large language models.arXiv preprint arXiv:2501.06986, 2025. 20

    arXiv 2025

  5. [5]

    Layer normalization.arXiv preprint arXiv:1607.06450, 2016

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization.arXiv preprint arXiv:1607.06450, 2016. 11 20

    arXiv 2016

  6. [6]

    Meteor: An automatic metric for mt evaluation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. InACL Workshop, pages 65–72, 2005. 1, 4, 8

    work page 2005

  7. [7]

    Token merging: Your vit but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. InICLR, 2023. 3, 19

    work page 2023

  8. [8]

    Matryoshka multimodal models

    Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. Matryoshka multimodal models. InICLR, 2025. 2, 3, 5, 7, 8, 9, 10, 11, 13

    work page 2025

Show all 76 references
  1. [9]

    Shikra: Unleashing multimodal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023. 2, 5, 6, 7, 8, 10, 11, 16, 17

    2023 arXiv

  2. [10]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InCVPR, pages 24185–24198, 2024. 20

    2024

  3. [11]

    Unifying specialized visual encoders for video language models.arXiv preprint arXiv:2501.01426, 2025

    Jihoon Chung, Tyler Zhu, Max Gonzalez Saez-Diez, Juan Carlos Niebles, Honglu Zhou, and Olga Russakovsky. Unifying specialized visual encoders for video language models.arXiv preprint arXiv:2501.01426, 2025. 9, 20

    2025 arXiv

  4. [12]

    Stimulus-selective properties of inferior temporal neurons in the macaque.Journal of Neuroscience, 4(8):2051–2062, 1984

    Robert Desimone, Thomas D Albright, Charles G Gross, and Charles Bruce. Stimulus-selective properties of inferior temporal neurons in the macaque.Journal of Neuroscience, 4(8):2051–2062, 1984. 4

    1984

  5. [13]

    Benchmarking and improving detail image caption.arXiv preprint arXiv:2405.19092, 2024

    Hongyuan Dong, Jiawen Li, Bohong Wu, Jiacong Wang, Yuan Zhang, and Haoyuan Guo. Benchmarking and improving detail image caption.arXiv preprint arXiv:2405.19092, 2024. 4, 8, 12, 13

    2024 arXiv

  6. [14]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. InICLR, 2021. 2, 9

    2021

  7. [15]

    EV A: Exploring the limits of masked visual representation learning at scale

    Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. EV A: Exploring the limits of masked visual representation learning at scale. InCVPR, 2023. 3

    2023

  8. [16]

    EV A-02: A visual representation for neon genesis.Image and Vision Computing, 2024

    Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. EV A-02: A visual representation for neon genesis.Image and Vision Computing, 2024. 20

    2024

  9. [17]

    Brain captioning: Decoding human brain activity into images and text.arXiv preprint arXiv:2305.11560, 2023

    Matteo Ferrante, Furkan Ozcelik, Tommaso Boccato, Rufin VanRullen, and Nicola Toschi. Brain captioning: Decoding human brain activity into images and text.arXiv preprint arXiv:2305.11560, 2023. 4, 6, 16, 17

    2023 arXiv

  10. [18]

    Onellm: One framework to align all modalities with language

    Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. InCVPR, 2024. 1, 2, 4, 6, 16, 17

    2024

  11. [19]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InCVPR, pages 770–778,

  12. [20]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. InCVPR, pages 16000–16009, 2022. 3, 4

    2022

  13. [21]

    Clipscore: A reference-free evaluation metric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. InEMNLP, 2021. 1, 4, 8

    2021

  14. [22]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. InNeurIPS Workshop on Deep Learning,

  15. [23]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. InNeurIPS, pages 6840–6851, 2020. 3, 4, 9, 11

    2020

  16. [24]

    GQA: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. InCVPR, 2019. 20

    2019

  17. [25]

    Perceiver: General perception with iterative attention

    Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. InICML, pages 4651–4664. PMLR, 2021. 2

    2021

  18. [26]

    The fusiform face area: a module in human extrastriate cortex specialized for face perception.Journal of neuroscience, 17(11):4302–4311, 1997

    Nancy Kanwisher, Josh McDermott, and Marvin M Chun. The fusiform face area: a module in human extrastriate cortex specialized for face perception.Journal of neuroscience, 17(11):4302–4311, 1997. 4, 7

    1997

  19. [27]

    Brave: Broadening the visual encoding of vision-language models

    O˘guzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, and Federico Tombari. Brave: Broadening the visual encoding of vision-language models. InECCV, pages 113–132. Springer, 2024. 2, 3

    2024

  20. [28]

    Analyzing and improving the image quality of StyleGAN

    Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. InCVPR, pages 8107–8116, 2020. 2

    2020

  21. [29]

    Token fusion: Bridging the gap between token pruning and token merging

    Minchul Kim, Shangqian Gao, Yen-Chang Hsu, Yilin Shen, and Hongxia Jin. Token fusion: Bridging the gap between token pruning and token merging. InWACV, pages 1383–1392, 2024. 3, 19

    2024

  22. [30]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything. InICCV, 2023. 3, 4, 20

    2023

  23. [31]

    Matryoshka representation learning

    Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representation learning. InNeurIPS, pages 30233–30249, 2022. 3

    2022

  24. [32]

    Pix2Struct: Screenshot parsing as pretraining for visual language understanding

    Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. Pix2Struct: Screenshot parsing as pretraining for visual language understanding. InICML,

  25. [33]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InICML, pages 12888–12900. PMLR, 2022. 1 21

    2022

  26. [34]

    Autoregressive image generation without vector quantization

    Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. NeurIPS, 37:56424–56445, 2024. 3, 4, 11

    2024

  27. [35]

    Evaluating object hallucination in large vision-language models.arXiv:2305.10355, 2023

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models.arXiv:2305.10355, 2023. 20

    2023 arXiv

  28. [36]

    Factual: A benchmark for faithful and consistent textual scene graph parsing

    Zhuang Li, Yuyang Chai, Terry Yue Zhuo, Lizhen Qu, Gholamreza Haffari, Fei Li, Donghong Ji, and Quan Hung Tran. Factual: A benchmark for faithful and consistent textual scene graph parsing. InACL, 2023. 4

    2023

  29. [37]

    Mind Reader: Reconstructing complex images from brain activities.NeurIPS, 35: 29624–29636, 2022

    Sikun Lin, Thomas Sprague, and Ambuj K Singh. Mind Reader: Reconstructing complex images from brain activities.NeurIPS, 35: 29624–29636, 2022. 18

    2022

  30. [38]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. InECCV, pages 740–755, 2014. 4, 5, 7, 9, 13

    2014

  31. [39]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InNeurIPS, 2023. 1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13

    2023

  32. [40]

    Nltk: The natural language toolkit

    Edward Loper and Steven Bird. Nltk: The natural language toolkit. InProceedings of the ACL Interactive Poster and Demonstration Sessions, 2004. 4

    2004

  33. [41]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InICLR, 2019. 5

    2019

  34. [42]

    Benchmarking large vision-language models via directed scene graph for comprehensive image captioning

    Fan Lu, Wei Wu, Kecheng Zheng, Shuailei Ma, Biao Gong, Jiawei Liu, Wei Zhai, Yang Cao, Yujun Shen, and Zheng-Jun Zha. Benchmarking large vision-language models via directed scene graph for comprehensive image captioning. InCVPR, 2025. 4, 5, 12, 13

    2025

  35. [43]

    Umap: Uniform manifold approximation and projection for dimension reduction

    Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018. 18

    2018 arXiv

  36. [44]

    Improved denoising diffusion probabilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. InICML, pages 8162–8171, 2021. 4, 9, 11

    2021

  37. [45]

    DINOv2: Learning robust visual features without supervision.TMLR, 2023

    Maxime Oquab, Timoth´ee Darcet, Th´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning robust visual features without supervision.TMLR, 2023. 3, 5, 9, 10, 19, 20

    2023

  38. [46]

    Brain-Diffuser: Natural scene reconstruction from fMRI signals using generative latent diffusion

    Furkan Ozcelik and Rufin VanRullen. Brain-Diffuser: Natural scene reconstruction from fMRI signals using generative latent diffusion. Scientific Reports, 13(1):15666, 2023. 1, 2, 5, 6, 18

    2023

  39. [47]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311–318, 2002. 1, 4, 8

    2002

  40. [48]

    Differential sensitivity of human visual cortex to faces, letterstrings, and textures: a functional magnetic resonance imaging study.Journal of neuroscience, 16(16):5205–5215, 1996

    Aina Puce, Truett Allison, Maryam Asgari, John C Gore, and Gregory McCarthy. Differential sensitivity of human visual cortex to faces, letterstrings, and textures: a functional magnetic resonance imaging study.Journal of neuroscience, 16(16):5205–5215, 1996. 4, 7

    1996

  41. [49]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InICML, pages 8748–8763. PMLR,

  42. [50]

    Exploring the limits of transfer learning with a unified text-to-text transformer.JMLR, 21(140):1–67, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.JMLR, 21(140):1–67, 2020. 4

    2020

  43. [51]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image synthesis with latent diffusion models. InCVPR, pages 10684–10695, 2022. 2, 3, 9

    2022

  44. [52]

    LAION-5B: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. L...

    2022

  45. [53]

    Reconstructing the mind’s eye: fmri-to-image with contrastive learning and diffusion priors

    Paul S Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Ethan Cohen, Aidan J Dempster, Nathalie Verlinde, Elad Yundler, David Weisberg, et al. Reconstructing the mind’s eye: fmri-to-image with contrastive learning and diffusion priors. In NeurIPS, 2023. 1...

    2023

  46. [54]

    Mindeye2: Shared-subject models enable fmri-to-image with 1 hour of data

    Paul S Scotti, Mihir Tripathy, Cesar Kadir Torrico Villanueva, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu, Thomas Naselaris, Kenneth A Norman, and Tanishq Mathew Abraham. Mindeye2: Shared-subject models enable fmri-to-image with 1 hour of d...

    2024

  47. [55]

    Neuro-vision to language: Enhancing brain recording-based visual reconstruction and language interaction

    Guobin Shen, Dongcheng Zhao, Xiang He, Linghao Feng, Yiting Dong, Jihang Wang, Qian Zhang, and Yi Zeng. Neuro-vision to language: Enhancing brain recording-based visual reconstruction and language interaction. InNeurIPS, pages 98083–98110, 2024. 1, 2, 5, 6, 7, 18

    2024

  48. [56]

    Mome: Mixture of multimodal experts for generalist multimodal large language models

    Leyang Shen, Gongwei Chen, Rui Shao, Weili Guan, and Liqiang Nie. Mome: Mixture of multimodal experts for generalist multimodal large language models. InNeurIPS, pages 42048–42070, 2025. 2, 9, 20

    2025

  49. [57]

    Eagle: Exploring the design space for multimodal llms with mixture of encoders

    Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, Bryan Catanzaro, Andrew Tao, Jan Kautz, Zhiding Yu, and Guilin Liu. Eagle: Exploring the design space for multimodal llms with mixture of ...

    2025

  50. [58]

    Super-convergence: Very fast training of neural networks using large learning rates

    Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. InArtificial intelligence and machine learning for multi-domain operations applications, pages 369–386. SPIE, 2019. 5

    2019

  51. [59]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. InICLR, 2021. 3, 9

    2021

  52. [60]

    Generative modeling by estimating gradients of the data distribution

    Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. InNeurIPS, 2019. 3, 9, 11

    2019

  53. [61]

    Improving visual image reconstruction from human brain activity using latent diffusion models via multiple decoded inputs.arXiv preprint arXiv:2306.11536, 2023

    Yu Takagi and Shinji Nishimoto. Improving visual image reconstruction from human brain activity using latent diffusion models via multiple decoded inputs.arXiv preprint arXiv:2306.11536, 2023. 1, 2, 4, 5, 6, 9, 16, 17, 18

    2023 arXiv

  54. [62]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. InCVPR, pages 9568–9578, 2024. 2, 5, 7, 8, 9, 10, 13

    2024

  55. [63]

    Cider: Consensus-based image description evaluation

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. InCVPR, pages 4566–4575, 2015. 1, 4, 8

    2015

  56. [64]

    Reconstructive visual instruction tuning

    Haochen Wang, Anlin Zheng, Yucheng Zhao, Tiancai Wang, Zheng Ge, Xiangyu Zhang, and Zhaoxiang Zhang. Reconstructive visual instruction tuning. InICLR, 2025. 3

    2025

  57. [65]

    Mindbridge: A cross-subject brain decoding framework

    Shizun Wang, Songhua Liu, Zhenxiong Tan, and Xinchao Wang. Mindbridge: A cross-subject brain decoding framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11333–11342, 2024. 18

    2024

  58. [66]

    ConvNeXt V2: Co-designing and scaling convnets with masked autoencoders

    Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. ConvNeXt V2: Co-designing and scaling convnets with masked autoencoders. InCVPR, 2023. 19, 20

    2023

  59. [67]

    Umbrae: Unified multimodal brain decoding

    Weihao Xia, Raoul de Charette, Cengiz Oztireli, and Jing-Hao Xue. Umbrae: Unified multimodal brain decoding. InECCV, pages 242–259, 2024. 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 16, 17, 18, 20

    2024

  60. [68]

    Dream: Visual decoding from reversing human visual system

    Weihao Xia, Raoul de Charette, Cengiz ¨Oztireli, and Jing-Hao Xue. Dream: Visual decoding from reversing human visual system. In WACV, pages 8226–8235, 2024. 1, 2, 5, 6, 9, 18

    2024

  61. [69]

    Mevox: Multi-task vision experts for brain captioning

    Weihao Xia and Cengiz ¨Oztireli. Mevox: Multi-task vision experts for brain captioning. InCVPR Workshop, 2025. 2, 6, 7, 17

    2025

  62. [70]

    Versatile diffusion: Text, images and variations all in one diffusion model

    Xingqian Xu, Zhangyang Wang, Eric Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all in one diffusion model. InICCV, 2023. 2

    2023

  63. [71]

    Dense connector for mllms

    Huanjin Yao, Wenhao Wu, Taojiannan Yang, YuXin Song, Mengxi Zhang, Haocheng Feng, Yifan Sun, Zhiheng Li, Wanli Ouyang, and Jingdong Wang. Dense connector for mllms. InNeurIPS, pages 33108–33140, 2025. 2, 3, 5, 7, 8, 9, 10, 11, 13

    2025

  64. [72]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InICCV, pages 11975–11986, 2023. 5, 7, 9, 10

    2023

  65. [73]

    Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment. InICLR, 2024. 20

    2024

  66. [74]

    Detrs with collaborative hybrid assignments training

    Zhuofan Zong, Guanglu Song, and Yu Liu. Detrs with collaborative hybrid assignments training. InICCV, pages 6748–6758, 2023. 20

    2023

  67. [75]

    Mova: Adapting mixture of vision experts to multimodal context

    Zhuofan Zong, Bingqi Ma, Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hongsheng Li, and Yu Liu. Mova: Adapting mixture of vision experts to multimodal context. InNeurIPS, 2024. 2, 9, 20 23

    2024

  68. [2021]

    1, 2, 3, 4, 5, 7, 9, 10, 19, 20

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.