{"id":"e1829331-9267-4080-8c85-efb1e5c1d665","arxiv_id":"2505.15755","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"VINDEX decodes fMRI into nested 9-token CLIP features that feed a frozen MLLM, improving detailed brain captioning and QA over a regression baseline, with a new benchmark.","lead":"This paper trains brain decoders to translate fMRI signals into visual features from pretrained image models, then feeds those features into multimodal language models to produce detailed captions, answers, and locations. It systematically compares four visual feature spaces and introduces a denoising loss and a new benchmark for fine-grained brain decoding.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MG-BrainDub's references are LLaVA-1.5 outputs labeled 'pseudo ground truth' in App. C.2 despite Sec. 4.1's claim of human correction; since the decoding MLLM is also LLaVA, the benchmark may reward LLaVA priors rather than neural decoding, so the NF9-best claim lacks a valid yardstick.","rationale":"The paper's strongest claim is a ranking of feature spaces on detailed brain captioning and salient QA. That ranking is only as meaningful as the benchmark on which it is computed. The benchmark's detailed-caption references are generated by LLaVA-1.5 7B, and the same model family (LLaVA-1.5/1.6) is used for the decoded captions. Appendix C.2 calls these references 'pseudo ground truth'; no human verification protocol, correction statistics, or inter-annotator agreement is reported anywhere, despite Section 4.1 asserting that expert human intervention was used. Without independent verification, the metric compares one LLaVA output to another LLaVA output and can reward shared priors, hallucination patterns, and stylistic tendencies rather than fidelity to the actual stimulus. The paper does report positive results on BrainHub concise captioning and concept localization that are independent of MG-BrainDub, which supports the general VINDEX approach; however, the detailed-captioning and SQA superiority of NF9 is the headline contribution and is benchmark-specific. The reader's conditional verdict already captures this risk, so I do not propose a different verdict: acceptance should remain conditional on validating the benchmark references, for example by the human-annotation check described above.","tokens_in":28582,"tokens_out":9466,"duration_ms":82560,"concrete_test":"Select 100 random MG-BrainDub test images. Have at least two independent human annotators write detailed captions and verify each object/attribute/relation in the LLaVA reference against the image, resolving disagreements with a third annotator. Recompute object/attribute/relation F1 and CAPTURE for VINDEX-SE, -ME, -AF, -NF1, -NF9, -NF36, and -NF144 using the human-verified reference set. If NF9 does not remain best, or if image-grounded LLaVA does not clearly outperform VINDEX on the same subset, the LLaVA-generated references are the load-bearing flaw and the central ranking is unverified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that NF9 is the best feature space for detailed brain captioning and salient QA rests entirely on scores computed against MG-BrainDub references. Those references are generated by LLaVA-1.5 7B. Section 4.1 states that ground truth descriptions were constructed using state-of-the-art MLLMs 'followed by manual error correction, missing element addition, and hallucination removal through expert human intervention,' but Appendix C.2 says simply that 'Results from LLaVA-v1.5 7B (LLaVA) serve as pseudo ground truth for detailed captioning evaluation.' No annotation protocol, correction statistics, or inter-annotator agreement is reported anywhere. Because the decoding backbone is also LLaVA (Tab. 3), candidate and reference captions share the same hallucination and stylistic priors; a decoder that echoes LLaVA priors could score well even when the brain signal carries little of the asserted content. The benchmark therefore may not measure neural decoding fidelity, and the reported ranking of feature spaces would not establish that VINDEX/NF9 outperforms SE/ME/AF in any externally valid sense. This is load-bearing: the headline contribution on detailed decoding is evaluated only on this benchmark, and no image-grounded oracle or human-verified subset is reported to calibrate the metric.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies which visual feature spaces of multimodal large language models (MLLMs) best support zero-shot fMRI-to-text decoding. It proposes VINDEX, which trains a brain encoder to predict image features from CLIP, DINO, and SigLIP variants using a regression loss combined with a masked denoising objective, and then feeds the predicted tokens into LLaVA-family models for tasks such as concept localization, concise captioning, detailed captioning, and salient question answering. The paper also introduces MG-BrainDub, a benchmark for detailed descriptions and salient QA, and reports that a nested 9-token feature space (NF9) achieves the best detailed captioning and QA performance. Experiments are conducted on the Natural Scenes Dataset with standard train/test splits for four subjects.","tokens_in":28907,"tokens_out":7632,"duration_ms":58501,"significance":"If the evaluation-validity concerns are resolved, the paper is valuable to the brain-decoding community: it gives a systematic comparison of feature-space choices, a lightweight denoiser that stabilizes training, a publicly announced code release, and a new benchmark aimed at fine-grained evaluation. The concept-localization and concise-captioning results are compared against several existing methods and show competitive or superior numbers, which lends some independent credibility to the approach. However, the headline detailed-captioning and salient-QA claims rest almost entirely on MG-BrainDub, whose reference construction is not independent of the decoding MLLM; the paper is therefore not yet ready for acceptance without a strengthened evaluation.","major_comments":[{"comment":"The reference standard for MG-BrainDub is internally inconsistent and is not an independent yardstick. Section 4.1 states that ground-truth descriptions were generated by MLLMs 'followed by manual error correction, missing element addition, and hallucination removal through expert human intervention,' but Appendix C.2 says only that 'Results from LLaVA-v1.5 7B (LLaVA) serve as pseudo ground truth for detailed captioning evaluation,' with no annotation counts, correction statistics, or inter-annotator agreement. Since the decoding MLLM in the NF9 row of Table 3 is also LLaVA-1.5 7B, the CAPTURE and object/attribute/relation F1 scores may reflect agreement with LLaVA's stylistic and hallucination priors rather than genuine brain-based decoding. Please provide a human-verified subset of the references, or references from a different model family, and report scores on that subset; without this, the claim that NF9 outperforms SE/ME/AF on detailed decoding is not externally validated.","section":"Sec. 4.1 / App. C.2 / Tab. 3"},{"comment":"Table 3 reports a single run for each configuration with no error bars, confidence intervals, or significance tests. The NF9-versus-NF1 SQA gap is 1.24 accuracy points (83.83 vs. 82.59), and the CAPTURE gap is 0.0144 (0.5021 vs. 0.4877). Given the noise in fMRI data and the fact that the token count n was selected after inspecting these results, the claim that NF9 is the best feature space is not established. The authors should report repeated-seed or per-subject variability and should either pre-register the token count or validate it on a held-out split.","section":"Tab. 3"},{"comment":"The detailed-captioning comparison includes only NeuroVLA as an external method; all other compared rows are VINDEX's own feature-space variants. No independent brain-decoding method, such as UMBRAE or MindEye2, is evaluated on MG-BrainDub under the same protocol. As a result, the paper does not support a claim of state-of-the-art detailed brain captioning; it only supports an internal comparison among feature spaces. Please add at least UMBRAE and one additional recent method to the MG-BrainDub evaluation.","section":"Tab. 2 (lower block) / Tab. 3"},{"comment":"The SQA reference construction introduces additional annotation dependence that is not discussed in the validity analysis: salient objects are identified by SAM and input to 'the MLLM,' and the in-depth reasoning questions are based on GPT annotations. The paper does not report how many questions were manually verified or how often the MLLM's object and attribute labels were corrected. Because SQA accuracy is a headline result in Table 3, the SQA reference construction should be documented with the same rigor as the caption references, including human-verification statistics.","section":"Sec. 4.2 / Tab. 3"}],"minor_comments":[{"comment":"Section 5.5 says 'using Salient Question Answering (SQA) (Sec. 5.5)', but SQA is described in Sec. 4.2; the cross-reference is wrong and should be corrected.","section":"Sec. 5.5"},{"comment":"The relationship between the † symbol in Table 2 and the * symbol in Table 3 should be clarified: Table 3 says the starred models use the same settings as in Table 2, but Table 2's VINDEX is built on Shikra while Table 3's SE* row uses LLaVA-1.5 7B.","section":"Tab. 2 / Tab. 3 captions"},{"comment":"The phrase 'The intrication of brain signals' in the abstract and introduction is unusual; 'complexity' or 'intricacy' appears to be intended.","section":"Abstract"},{"comment":"The metric-calculation example appears to use mismatched candidate and reference captions: the candidate describes a city street with a truck, while the reference describes a beach, yet the matching example yields 100% precision. Please correct the example or add a note explaining why the reference objects listed are consistent with the candidate.","section":"App. C.1"},{"comment":"The text in Section 6 correctly identifies the best denoiser setting as a one-depth MLP with width 1024 and beta 1.0, but Section 5.1 should state explicitly that this corresponds to configuration S4 in Table 4, since the table columns are not otherwise named in the text.","section":"Sec. 5.1 / Tab. 4"}],"recommendation":"major_revision","confidential_remarks":"The main obstacle is the validity of the detailed-captioning benchmark. If the authors add a human-verified reference subset, compare against independent brain-decoding methods on MG-BrainDub, and report variability, the paper could be acceptable. The concise-captioning and grounding results provide a partial independent check on the method's value."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nHere's my read on arXiv:2505.15755. It's a genuine extension of UMBRAE, and for that alone it deserves a look. The paper systematically compares four vision feature spaces for brain decoding—single encoder, mixture of encoders, aggregated multi-layer features, and nested coarse-to-fine tokens—and adds a denoising objective to the regression alignment. The central empirical finding, that NF9 (nine nested visual tokens) works best for detailed captioning and salient QA, is clearly presented, and the paper is honest enough to report that dense aggregated features (AF) fail badly. That negative result, plus the inclusion of a new benchmark, MG-BrainDub, makes this a useful contribution to the brain-decoding and MLLM-feature-space literature.\n\nWhere I worry: the benchmark's reference captions. Section 4.1 says they were constructed by MLLMs followed by manual correction and hallucination removal. Appendix C.2, though, calls LLaVA-1.5 outputs 'pseudo ground truth' and gives no details of any correction protocol, no counts, no inter-annotator agreement. Since the decoding MLLM is also LLaVA, the detailed-caption scores may be rewarding LLaVA's priors and style rather than neural decoding fidelity. This is load-bearing, because the headline NF9 claim rests entirely on that benchmark. The concise captioning and concept localization results are on independent metrics and are more reassuring, but the gains over UMBRAE there are within fractions of a point. There are also no error bars or significance tests anywhere, and NF9 appears to have been selected after looking at the numbers. Code isn't released yet.\n\nIf this lands in your inbox, I'd send it to peer review. The method and the systematic comparison deserve scrutiny, and the benchmark concerns are fixable—a human-validated subset of references, an image-grounded oracle, or at least a clear description of the correction process would go a long way. The paper is a solid, honest piece of work despite the evaluation issues. A careful referee can push it into a stronger state.\n\nBest","headline":"Sensible extension of UMBRAE with a feature-space study and a new benchmark, but the benchmark's LLaVA-generated references weaken the headline NF9 claim.","tokens_in":29408,"tokens_out":3259,"would_cite":false,"duration_ms":26863,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VINDEX, aligning fMRI to nine coarse-to-fine visual tokens with a denoising objective, produces the most accurate detailed brain captions and salient question answering in the paper's MG-BrainDub benchmark.","keywords":["fMRI decoding","multimodal large language models","visual feature spaces","nested features","denoising diffusion","detailed captioning","salient question answering","zero-shot brain decoding"],"falsifier":"Take a random subset of the shared test images, ask independent human raters to write free-form descriptions and answer the benchmark's salient question-answering items, then score VINDEX's decoded captions against those human references using MG-BrainDub's object/attribute/relation metrics; if the pattern of results across feature spaces (nine-token nested best, aggregated worst) does not reproduce with human references, the paper's ranking is an artifact of using machine-generated ground truth.","tokens_in":28379,"feed_emoji":"🧠","tokens_out":8144,"duration_ms":66123,"temperature":0.7,"pith_summary":"The paper tries to establish that the choice of visual feature space, not the size or sophistication of the language model, governs how much fine-grained visual detail can be decoded from brain activity. It introduces VINDEX, a zero-shot pipeline that trains a brain encoder on fMRI signals to predict the visual tokens used by a pre-trained multimodal large language model, then feeds those predicted tokens to the model for captioning and question answering. Across four feature-space designs — a single encoder, a mixture of encoders, aggregated multi-layer features, and coarse-to-fine nested token sets — the paper finds that a nested set of nine visual tokens gives the most accurate detailed captions and salient-object question answering, outperforming denser feature representations. To measure this, it builds the MG-BrainDub benchmark, whose metrics score objects, attributes, and relations separately rather than relying on n-gram matching. A reader should care because the result suggests that brain signals contain less visual information than images do, so matching the token budget to brain data can be more effective than throwing more visual computation at the problem.","feed_headline":"Nine visual tokens beat dense features for fMRI decoding","feed_subtitle":"Coarse-to-fine tokens beat dense features for brain captions and question answering.","key_machinery":"The central object is nested features: the target image is encoded by a vision transformer and its patch tokens are repeatedly downsampled by $2\\times2$ pooling to form a hierarchy of token sets (144, 36, 9, 1 tokens), each coarser set derived from the finer one, so the brain encoder can be aligned with a token budget matching the information carried by brain signals. The other load-bearing piece is the masked denoising objective: during training a lightweight MLP denoiser learns to predict noise added to the clean visual tokens, conditioned on the brain encoder's prediction, and its loss is added to the regression loss, acting as implicit augmentation and regularization that stabilizes training and improves caption quality at no extra inference cost.","core_discovery":"VINDEX treats the visual component of a multimodal large language model as the target space for brain alignment. A shared brain encoder maps fMRI responses to image features from a chosen vision encoder, using a weighted sum of a regression loss and a masked denoising loss in which a small denoiser predicts noise added to the target visual tokens conditioned on the brain prediction. During inference, the predicted tokens replace image tokens and flow through the connector and language model, enabling zero-shot instruction-following tasks with no textual or spatial annotations at training time. The paper's central empirical discovery is that among the feature spaces tested, the nested-features setting with nine visual tokens (coarse-to-fine downsampling of the single encoder's features) yields the best performance on detailed description and salient question answering, while dense aggregated features collapse to garbled outputs, and single-encoder features lag behind despite larger and more recent language models.","pith_inferences":["If nine tokens encode what fMRI preserves, then the brain's visual bottleneck — not the model — is the limiting factor, and decoders for lower-spatial-resolution recording modalities such as EEG or MEG would likely need even smaller token budgets.","The nested-feature result suggests a design principle: feature spaces should be chosen so their information capacity matches the fidelity of the neural signal, a criterion that could be tested prospectively by ablating per-subject token counts.","Because the denoiser is removed at inference, the same brain encoder could be retrained with different denoising strengths or noise schedules to trade robustness against overfitting, a dimension the paper explores only coarsely.","The benchmark's reliance on machine-generated reference captions implies that rankings could shift if a stronger caption generator is used as ground truth, so adding a human-verified subset would make MG-BrainDub more stable across future model generations."],"forward_implications":["A brain decoder can be built zero-shot for any multimodal language model whose vision encoder matches the trained brain encoder, without collecting captions or bounding-box annotations during training.","Token pruning and merging ideas from vision can transfer to brain decoding: the leading result uses only nine visual tokens, implying much lower compute for fMRI-to-language than image-based multimodal pipelines.","Dense multi-layer feature representations are not just wasteful but harmful for brain decoding, producing gibberish or hallucinated output, so future brain-decoding architectures should not blindly inherit full-resolution visual features.","Detailed caption evaluation should use object/attribute/relation precision, recall, and F1 rather than n-gram or embedding-based metrics, since rule-based and CLIP-style metrics mis-rank long brain-decoded captions."],"supporting_citations":[{"why":"Supplies the fMRI dataset of responses to natural images that all training and evaluation builds on.","marker":"[2]"},{"why":"Provides the contrastively pre-trained vision encoder whose features serve as the single-encoder and nested-feature targets.","marker":"[49]"},{"why":"Provides the brain-encoder architecture, the zero-shot brain-to-MLLM interface, and the baseline that VINDEX is compared against.","marker":"[67]"},{"why":"Provides the multimodal language model, prompt templates, and the machine-generated reference captions used in the benchmark.","marker":"[39]"},{"why":"Supplies the nested, coarse-to-fine representation idea behind the nested-feature settings.","marker":"[8]"},{"why":"Provides the long-caption metric and visual-element evaluation that MG-BrainDub's object/attribute/relation metrics extend.","marker":"[13]"},{"why":"Supplies the denoising diffusion objective and noise schedule that the masked denoiser is based on.","marker":"[23]"},{"why":"Provides the self-supervised vision encoder used in the mixture-of-encoders setting.","marker":"[45]"}],"fun_headline_variants":["Nine visual tokens top dense features for fMRI decoding","Coarse-to-fine tokens win in brain decoding and Q&A","VINDEX: nine tokens beat dense features in zero-shot fMRI tasks","Nested features outperform dense for brain captions and QA","Small token count, big decode: nine tokens beat dense fMRI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's ground-truth detailed captions are written by a multimodal language model (with human correction), not obtained from the human subjects themselves, so the measured 'detail' is fidelity to that model's descriptions of the stimuli rather than to what the subjects actually perceived.","fun_headline_variants_meta":{"raw":{"variants":["Nine visual tokens top dense features for fMRI decoding","Coarse-to-fine tokens win in brain decoding and Q&A","VINDEX: nine tokens beat dense features in zero-shot fMRI tasks","Nested features outperform dense for brain captions and QA","Small token count, big decode: nine tokens beat dense fMRI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000238,"raw_usage":{"total_tokens":1488,"prompt_tokens":902,"completion_tokens":586,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":502}},"tokens_in":518,"tokens_out":586,"duration_ms":5388,"temperature":1.0,"reasoning_tokens":502,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:11:15.887408+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random subset of the shared test images, ask independent human raters to write free-form descriptions and answer the benchmark's salient question-answering items, then score VINDEX's decoded captions against those human references using MG-BrainDub's object/attribute/relation metrics; if the pattern of results across feature spaces (nine-token nested best, aggregated worst) does not reproduce with human references, the paper's ranking is an artifact of using machine-generated ground truth.","supporting_citations":[{"cited_title":"A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence.Nature neuroscience, 25(1):116–126, 2022","cited_arxiv_id":null,"evidence_quote":"Supplies the fMRI dataset of responses to natural images that all training and evaluation builds on."},{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Provides the contrastively pre-trained vision encoder whose features serve as the single-encoder and nested-feature targets."},{"cited_title":"Umbrae: Unified multimodal brain decoding","cited_arxiv_id":null,"evidence_quote":"Provides the brain-encoder architecture, the zero-shot brain-to-MLLM interface, and the baseline that VINDEX is compared against."},{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Provides the multimodal language model, prompt templates, and the machine-generated reference captions used in the benchmark."},{"cited_title":"Matryoshka multimodal models","cited_arxiv_id":null,"evidence_quote":"Supplies the nested, coarse-to-fine representation idea behind the nested-feature settings."},{"cited_title":"Denoising diffusion probabilistic models","cited_arxiv_id":null,"evidence_quote":"Supplies the denoising diffusion objective and noise schedule that the masked denoiser is based on."},{"cited_title":"DINOv2: Learning robust visual features without supervision.TMLR, 2023","cited_arxiv_id":null,"evidence_quote":"Provides the self-supervised vision encoder used in the mixture-of-encoders setting."}],"review_version":1}