{"id":"6ad49adc-a6ce-440a-a1c1-d8936bded82d","arxiv_id":"2505.10356","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A multimodal brain-to-text decoder with learned routing reports state-of-the-art decoding on fMRI, EEG, and MEG benchmarks by aligning neural activity with text, image, and audio embeddings.","lead":"This paper presents a brain decoder that maps fMRI, EEG, and MEG signals into text by aligning them with text, image, and audio representations inside a multimodal language model. A routing module picks which representation matters for each brain sample, and the authors report state-of-the-art captioning scores on several benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FLUX auxiliary images for Pereira/Huth are likely generated from the target text, making the visual projector a label-derived pathway and undermining the multimodal-alignment claim on those datasets.","rationale":"The reader's weakest_assumption identified precisely the FLUX target-text leakage, and my reading of the manuscript confirms it as the most load-bearing threat to the central claim. The architecture itself is coherent and the NSD experiments, where images are genuine stimuli, provide independent support for a somewhat weaker version of the claim. However, the Pereira and Huth results, which are needed for the 'diverse brain signals and stimulus types' generalization, depend on an unspecified auxiliary-image protocol. If the FLUX prompts are the target captions, the visual projector becomes a second, label-derived pathway, so the ablation in Table 4 does not demonstrate that the brain signals contain independent visual information; it may simply show that multiple target-derived encoders help the LLM. This concern is concrete, falsifiable, and directly addresses the central empirical assertion. I agree with the reader that the paper should be conditionally accepted pending release of code, prompts, and datasets, or a controlled re-run; hence I recommend no change to the reader's CONDITIONAL verdict. I did not identify a more fundamental internal inconsistency: the method is mathematically well-specified, and the reported numbers are internally plausible once the leakage ambiguity is resolved.","tokens_in":12266,"tokens_out":5884,"duration_ms":63664,"concrete_test":"Release the training code and all text prompts used for FLUX generation, then re-run the Pereira and Huth experiments with the auxiliary images replaced by (a) a fixed neutral image (e.g., a gray frame) and (b) images generated from a caption belonging to a different sample in the same dataset, keeping all other hyperparameters fixed. If the omni-versus-unimodal gap in Table 4 collapses or shrinks to non-significance under either condition, the reported multimodal gain is an artifact of target-derived visual supervision rather than independent multimodal brain information.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the multimodal advantage comes from genuinely complementary visual, textual, and auditory brain representations. On NSD this is plausible because the images are the actual stimuli. On Pereira (text) and Huth (audio), however, there is no natural image stimulus; Section 4.4 states only 'We used FLUX.1-dev with a guidance scale of 3.5 for auxiliary image generation' without specifying the text prompt used to synthesize those images. If the prompt is the target sentence or transcript, as is the default for text-to-image generation, then the Brain-Image projector is trained to map brain signals into embeddings derived from the target text through FLUX. At inference, the fused representation H = sum w_i P_i(b) can then carry target-derived visual features, and the omni-versus-unimodal advantage in Table 4 may reflect ensembling multiple label-derived encoders rather than evidence of independent multimodal brain decoding. This is a load-bearing circularity: the paper's strongest evidence for its central contribution, the value of multimodal alignment for non-visual stimuli, is confounded by target supervision injected through the auxiliary image pathway. The paper does not report the FLUX prompt protocol, so the concern cannot be resolved from the manuscript alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a unified brain-to-text decoding framework that maps fMRI, EEG, and MEG signals into a shared multimodal embedding space derived from a Multimodal LLM (Qwen2.5-Omni). The architecture consists of separate brain projectors aligned to text, image, and audio auxiliary embeddings, plus a learned router that fuses the projected embeddings per sample before feeding the LLM. Training proceeds in two phases: multimodal instruction tuning (auxiliary encoders plus brain projectors aligned with captioning/alignment losses) and projectors fusion with load-balancing and routing losses. Experiments on three fMRI datasets (NSD, Pereira, Huth), one MEG dataset, and one EEG dataset report the best or near-best performance against several baselines, including an 8.48% claimed improvement on NSD and ablations showing the omni (multimodal) variant outperforms unimodal variants.","tokens_in":12553,"tokens_out":3898,"duration_ms":40676,"significance":"If the central claim holds, the paper would make a useful contribution: a single architecture that handles multiple neuroimaging modalities and stimulus types through adaptive multimodal fusion is interesting, and the router-weight/abstractness analysis (Figure 4) is a nice interpretability step. The paper also includes careful ablations over routing strategies and framework components. However, the main significance claim of state-of-the-art multimodal brain decoding currently rests on comparisons that omit important baselines, lack statistical grounding, and, most importantly, may be confounded by auxiliary images generated from the target text on non-visual datasets. The manuscript is not yet at the standard required for publication.","major_comments":[{"comment":"The auxiliary image generation for the Pereira (text) and Huth (audio) datasets is potentially label-derived. Section 4.4 states only \"We used FLUX.1-dev with a guidance scale of 3.5 for auxiliary image generation,\" without specifying the text prompts used. If, as is the default for text-to-image models, the FLUX prompts are the target sentences or transcripts, then the Brain-Image projector is trained on embeddings derived from the very text that the model must output. In that case, the omni-versus-unimodal advantage in Table 4 may reflect ensembling multiple label-derived encoders rather than independent visual information in the brain signal. This is load-bearing for the paper's central claim of multimodal alignment benefits on non-visual stimuli, so the prompt protocol must be specified, and the authors should either use stimulus-independent auxiliary images or re-run the Pereira/Huth ablations without the target-derived visual pathway.","section":"Section 4.4 and Table 4"},{"comment":"The claimed \"8.48% improvement on the most commonly used benchmark\" is undefined. Inspecting Table 1, the BLEU-1 gains of Ours (similarity merge) over MindLLM and UMBRAE are 3.48 and 3.90 absolute points (roughly 6.0% and 6.8% relative), while the CIDEr gain is about 23%. No entry in Table 1 corresponds to 8.48%. Please state the exact metric, baseline, and computation; also report error bars or significance tests over subjects/splits, since several differences in Table 1 are small relative to likely noise.","section":"Abstract and Table 1"},{"comment":"The \"with text prompt\" condition in Table 2 is underspecified. The table caption says textual context is provided as input to the LLM, but the text of that prompt is never defined. If the prompt includes the target sentence or any token derived from it, then those results are not clean brain-decoding results. Please specify exactly what the text prompt contains, for both the \"with text prompt\" and \"without text prompt\" settings, and clarify whether the prompt is derived from the target or from a fixed/instructional prefix.","section":"Table 2"},{"comment":"The MEG and EEG evaluations omit several strong baselines that are cited in the Related Work section, including MAD (Yang et al. 2024b), EEG2Text (Liu et al. 2024), and DeWave (Duan et al. 2023). Table 3 compares only random, linear, and BrainLLM. Without these baselines, the claim of \"highest performance across all metrics\" on MEG/EEG is unsupported. Please add these methods or justify their exclusion.","section":"Table 3"},{"comment":"The claim of generalizability is not established because the fMRI results in Table 1 are reported for a single subject (NSD subject 1), and the MEG/EEG experiments use separate datasets with separate subjects. No cross-subject, cross-dataset, or single-model-trained-on-all-data experiment is presented, so the architecture-level \"unified\" claim is not the same as the empirical generalizability claimed in the title and abstract. Please either add such experiments or temper the wording to \"a unified architecture applicable to different datasets and signal types.\"","section":"Title, Abstract, and Section 5.1"}],"minor_comments":[{"comment":"The Pearson correlation r = 0.247 (p = 0.043) is reported without the number of samples, confidence interval, or any multiple-comparison correction; please report n and the CI, and note whether this is a pre-registered or post-hoc analysis.","section":"Figure 4 and Section 5.2"},{"comment":"The input image resolution is stated as 112×112×3, but the CLIP ViT-L/14 backbone normally operates at 224×224; please clarify whether the resolution change is intentional and whether it affects the auxiliary and brain encoder performance.","section":"Section 4.4"},{"comment":"The description of the hard-select router is unclear: Eq. (1) produces continuous Gumbel-Softmax outputs, Eq. (2) applies Top-1 selection, and the text says gradients use a straight-through estimator, but the precise forward/backward pass is not specified. Please provide the exact implementation of the straight-through estimator.","section":"Section 3.1 and Figure 3"},{"comment":"There are several typos and formatting issues, including \"a adaptive routing\" in Section 1, \"auxilliary\" in Figure 2, and the unspaced \"Text EncoderImage EncoderAudio Encoder\" in Figure 2. A careful proofread is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a potentially interesting architecture and a substantial experimental scope, but the missing FLUX prompt protocol is a serious concern that goes to the validity of the central multimodal-alignment claim. The authors should be asked to disclose the exact prompt generation procedure and to re-analyze the Pereira/Huth results if the prompts are target-derived. The omission of several well-known MEG/EEG baselines also needs to be addressed before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a legitimate attempt at a unified brain decoder, and the architecture is the real contribution: three brain projectors aligned to text, image, and audio embeddings from Qwen2.5-Omni, plus a learned router that fuses them per sample. That combination hasn't been done before, and the authors show it works on NSD with clear gains over UMBRAE and MindLLM on text metrics. The router-vs-abstractness analysis is a nice interpretability check, even if the correlation is modest.\n\nThe paper's central empirical claim—that multimodal alignment inherently helps on text and audio datasets—rests on an unspecified detail. For Pereira and Huth, there are no natural images, so the authors generated auxiliary images with FLUX.1-dev. They never say what prompt they used. If the prompt is the target sentence or transcript, which is the default for text-to-image, then the 'vision' projector is trained on embeddings derived from the very text the model has to output. In that case, the omni-vs-unimodal advantage in Table 4 is not evidence of independent visual information in brain signals; it's ensembling multiple label-derived encoders. This is load-bearing, because the paper's headline is about multimodal alignment, not ensembling. The stress-test note has this right, and the manuscript does not resolve it.\n\nOther soft spots in proportion: no error bars or significance tests anywhere; the '8.48% improvement' in the abstract is undefined (I can't reproduce it from Table 1); MEG/EEG comparisons omit cited strong baselines like MAD; and the 'with text prompt' condition in Table 2 muddies the comparison, though the 'without' condition still shows the method winning on BLEU-1. Minor: it's not stated explicitly whether the LLM is frozen during training.\n\nOn the positive side, the ablations are systematic and the two-phase training is sensible. The architecture is coherent and the writing is clear. But the empirical evidence as reported is not reproducible without code and error bars.\n\nWho this is for: anyone working on fMRI-to-text or multi-signal BCI decoding will want to read it and benchmark against it. It deserves a serious referee. My recommendation: major revision, and the referee should demand the FLUX prompt protocol, error bars, stronger baselines on MEG/EEG, and a precise definition of the 8.48% claim. If the FLUX images turn out to be target-derived, the Pereira/Huth multimodal claim should be withdrawn or reframed.","headline":"A serious architectural contribution undermined by an unspecified auxiliary-image protocol and missing error bars—worth refereeing but not trusting as-is.","tokens_in":13076,"tokens_out":2701,"would_cite":false,"duration_ms":26074,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that multimodal alignment and adaptive routing yield state-of-the-art brain-to-text decoding across fMRI, EEG, and MEG, with an 8.48% gain on the leading benchmark.","keywords":["brain-to-text decoding","multimodal alignment","modality routing","fMRI decoding","EEG decoding","MEG decoding","multimodal large language models","brain-computer interface"],"falsifier":"Retrain the framework on Pereira and Huth with auxiliary images generated from independent captions that do not contain the target sentences, keeping all other settings fixed; if the omni model no longer beats the text-only projector by a similar margin, the central claim of modality-independent multimodal gains is falsified. A second check is to shuffle the pairing between brain samples and generated images and see whether the multimodal advantage persists.","tokens_in":12073,"feed_emoji":"🧠","tokens_out":8826,"duration_ms":80569,"temperature":0.7,"pith_summary":"The paper tries to establish that the brain's response to a seemingly unimodal stimulus still contains multimodal traces, and that a decoder can exploit them by aligning brain signals with a shared text-image-audio space. It describes a single framework with three brain projectors and a per-sample router, and reports that this omni-modal decoder beats all unimodal baselines on three fMRI datasets and on EEG and MEG, with an 8.48% improvement on the most commonly used benchmark. If true, the practical consequence is that one architecture can serve multiple recording technologies and stimulus types, and that cross-modal alignment is a better inductive bias for brain-to-text decoding than text-only alignment. The paper also claims the router's weights track sentence abstractness, giving an interpretable view of which modality the decoder leans on.","feed_headline":"Multimodal router lifts brain-to-text scores by 8.48%","feed_subtitle":"One decoder aligns fMRI, EEG, and MEG with text, images, and audio, beating single-modality baselines on every benchmark.","key_machinery":"The central object is the fusion of three brain projectors by a Modality Router. Each projector is a CLIP-ViT-based encoder that maps raw brain activity into the embedding space of a modality; an auxiliary module (a cross-attention text encoder, a CLIP image encoder, and a Whisper-based audio encoder) provides target embeddings during training only. The router computes per-sample weights by one of three strategies (soft merge via an MLP, hard select via Gumbel-Softmax, or similarity merge via dot-product attention), and the weighted sum of projector outputs is fed to the Qwen2.5-Omni language decoder. A progressive alignment loss and a load-balancing loss keep the three projectors jointly trained and prevent any one from vanishing. The two-phase schedule first aligns projectors to auxiliary embeddings, then fuses them for captioning.","core_discovery":"The central claim is that brain signals encode language in a distributed, multimodal manner, so mapping them into a single text space throws away useful signal. The paper proposes that aligning each brain sample to text, image, and audio embeddings in a shared space, then fusing the three aligned representations with learned per-sample weights, yields better text generation than any one modality. On the NSD fMRI benchmark the fused model reaches BLEU-1 61.53 versus 58.05 for the strongest unimodal baseline, the reported source of the 8.48% improvement, and it also leads on the Pereira, Huth, MEG, and EEG evaluations. The paper further reports that the router assigns more weight to the text projector for more abstract sentences ($r=0.247$, $p=0.043$), and that both training phases and the load-balancing loss are necessary for the result.","pith_inferences":["Because the auxiliary images for the text and audio datasets were generated with FLUX.1-dev, and if their prompts were the target sentences, the visual projector may be trained on target-derived information; the omni-versus-unimodal gap then needs a control experiment before it can be read as evidence of independent visual brain activity.","A direct extension would replace generated images with images retrieved without access to the target text, or with unrelated images, and check whether the improvement persists.","The router and two-phase alignment could be applied to other paired neural recordings, such as electrocorticography or implanted electrode arrays, wherever multiple auxiliary modalities are available.","The abstractness-weight correlation ($r=0.247$) is modest; comparing router weights against a random-weights null model would show whether the interpretability claim survives beyond a single dataset."],"forward_implications":["If the framework is correct, a single trained architecture can decode text from fMRI, EEG, and MEG without needing a separate decoder design for each recording technology.","Cross-modal alignment helps even when the stimulus is text or audio, which implies that the brain response to those stimuli contains visual and auditory components worth modeling.","The router's abstractness correlation offers a behavioral handle: increasing text weight for abstract sentences means the decoder's reliance on visual information is at least partly predictable from semantic content.","The load-balancing and two-phase training results imply that naive joint training of multiple brain projectors collapses to a single modality; without explicit balancing the multimodal advantage shrinks."],"supporting_citations":[{"why":"Provides the NSD fMRI-image dataset and standard train/test splits used for the main benchmark and the 8.48% improvement claim.","marker":"Allen et al. 2022"},{"why":"Supplies the text-stimulus fMRI dataset used to demonstrate decoding from language stimuli.","marker":"Pereira et al. 2018"},{"why":"Provides the Huth natural-story fMRI dataset used for auditory-stimulus decoding.","marker":"LeBel et al. 2023"},{"why":"Supplies the MEG recordings used in the cross-modality extension.","marker":"Wang et al. 2022"},{"why":"Provides the ZuCo2 EEG dataset used in the extension to electroencephalography.","marker":"Zou et al. 2022"},{"why":"Defines the Qwen2.5-Omni multimodal language model whose shared embedding space and decoder the framework builds on.","marker":"Xu et al. 2025"},{"why":"CLIP-ViT is used as the initialization for the brain projectors and as the image encoder in the auxiliary module.","marker":"Radford et al. 2021"},{"why":"Whisper-large-v3 serves as the audio encoder that produces auxiliary audio embeddings for alignment.","marker":"Radford et al. 2023"},{"why":"UMBRAE is the strongest unimodal image-alignment baseline on NSD, the direct comparison behind the claimed improvement.","marker":"Xia et al. 2024"},{"why":"Gumbel-Softmax provides the differentiable approximation that makes the hard-select routing strategy trainable.","marker":"Jang, Gu, and Poole 2016"}],"fun_headline_variants":["Multimodal router lifts brain-to-text by 8.48%","Align brain to text, images, audio: decoder gains 8.48%","One BCI decoder for fMRI, EEG, MEG beats unimodal","Adaptive fusion of brain signals improves text decoding","Brain-to-text translation: multimodal alignment wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the auxiliary images for the text and audio datasets were produced independently of the sentences the model must decode; the paper reports generating them with FLUX.1-dev, so if those images were synthesized from the target text itself, the visual pathway could leak the answer and the multimodal gain would not prove the brain signal carried independent visual information.","fun_headline_variants_meta":{"raw":{"variants":["Multimodal router lifts brain-to-text by 8.48%","Align brain to text, images, audio: decoder gains 8.48%","One BCI decoder for fMRI, EEG, MEG beats unimodal","Adaptive fusion of brain signals improves text decoding","Brain-to-text translation: multimodal alignment wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1337,"prompt_tokens":920,"completion_tokens":417,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":330}},"tokens_in":536,"tokens_out":417,"duration_ms":4204,"temperature":1.0,"reasoning_tokens":330,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:10:16.220378+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the framework on Pereira and Huth with auxiliary images generated from independent captions that do not contain the target sentences, keeping all other settings fixed; if the omni model no longer beats the text-only projector by a similar margin, the central claim of modality-independent multimodal gains is falsified. A second check is to shuffle the pairing between brain samples and generated images and see whether the multimodal advantage persists.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Huth natural-story fMRI dataset used for auditory-stimulus decoding."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MEG recordings used in the cross-modality extension."},{"cited_title":"J.; Kanwisher, N.; Botvinick, M.; and Fedorenko, E","cited_arxiv_id":null,"evidence_quote":"Supplies the text-stimulus fMRI dataset used to demonstrate decoding from language stimuli."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ZuCo2 EEG dataset used in the extension to electroencephalography."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"UMBRAE is the strongest unimodal image-alignment baseline on NSD, the direct comparison behind the claimed improvement."}],"review_version":1}