Pith. sign in

REVIEW 5 major objections 4 minor 43 references

Decoding the Multimodal Mind: Generalizable Brain-to-Text Translation via Multimodal Alignment and Adaptive Routing

T0 review · 5 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that multimodal alignment and adaptive routing yield state-of-the-art brain-to-text decoding across fMRI, EEG, and MEG, with an 8.48% gain on the leading benchmark.

desk verdict A serious architectural contribution undermined by an unspecified auxiliary-image protocol and missing error bars—worth refereeing but not trusting as-is. read the letter →

arxiv 2505.10356 v3 pith:GCWZOOBF submitted 2025-05-15 cs.CL

Chunyu Ye , Yunhao Zhang , Jingyuan Sun , Chong Li , Yang Zhao , Shaonan Wang This is my paper · ORCID
classification cs.CL
keywords brain-to-textdecodingmultimodalalignmentmodalityroutingfMRIEEGMEGlargelanguagemodelsbrain-computerinterface
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the brain's response to a seemingly unimodal stimulus still contains multimodal traces, and that a decoder can exploit them by aligning brain signals with a shared text-image-audio space. It describes a single framework with three brain projectors and a per-sample router, and reports that this omni-modal decoder beats all unimodal baselines on three fMRI datasets and on EEG and MEG, with an 8.48% improvement on the most commonly used benchmark. If true, the practical consequence is that one architecture can serve multiple recording technologies and stimulus types, and that cross-modal alignment is a better inductive bias for brain-to-text decoding than text-only alignment. The paper also claims the router's weights track sentence abstractness, giving an interpretable view of which modality the decoder leans on.

What carries the argument

The central object is the fusion of three brain projectors by a Modality Router. Each projector is a CLIP-ViT-based encoder that maps raw brain activity into the embedding space of a modality; an auxiliary module (a cross-attention text encoder, a CLIP image encoder, and a Whisper-based audio encoder) provides target embeddings during training only. The router computes per-sample weights by one of three strategies (soft merge via an MLP, hard select via Gumbel-Softmax, or similarity merge via dot-product attention), and the weighted sum of projector outputs is fed to the Qwen2.5-Omni language decoder. A progressive alignment loss and a load-balancing loss keep the three projectors jointly trained and prevent any one from vanishing. The two-phase schedule first aligns projectors to auxiliary embeddings, then fuses them for captioning.

What would settle it

Retrain the framework on Pereira and Huth with auxiliary images generated from independent captions that do not contain the target sentences, keeping all other settings fixed; if the omni model no longer beats the text-only projector by a similar margin, the central claim of modality-independent multimodal gains is falsified. A second check is to shuffle the pairing between brain samples and generated images and see whether the multimodal advantage persists.

Watch

Extended reading notes

Core claim

The central claim is that brain signals encode language in a distributed, multimodal manner, so mapping them into a single text space throws away useful signal. The paper proposes that aligning each brain sample to text, image, and audio embeddings in a shared space, then fusing the three aligned representations with learned per-sample weights, yields better text generation than any one modality. On the NSD fMRI benchmark the fused model reaches BLEU-1 61.53 versus 58.05 for the strongest unimodal baseline, the reported source of the 8.48% improvement, and it also leads on the Pereira, Huth, MEG, and EEG evaluations. The paper further reports that the router assigns more weight to the text projector for more abstract sentences ($r=0.247$, $p=0.043$), and that both training phases and the load-balancing loss are necessary for the result.

Load-bearing premise

The load-bearing premise is that the auxiliary images for the text and audio datasets were produced independently of the sentences the model must decode; the paper reports generating them with FLUX.1-dev, so if those images were synthesized from the target text itself, the visual pathway could leak the answer and the multimodal gain would not prove the brain signal carried independent visual information.

Editorial extensions

If this is right

  • If the framework is correct, a single trained architecture can decode text from fMRI, EEG, and MEG without needing a separate decoder design for each recording technology.
  • Cross-modal alignment helps even when the stimulus is text or audio, which implies that the brain response to those stimuli contains visual and auditory components worth modeling.
  • The router's abstractness correlation offers a behavioral handle: increasing text weight for abstract sentences means the decoder's reliance on visual information is at least partly predictable from semantic content.
  • The load-balancing and two-phase training results imply that naive joint training of multiple brain projectors collapses to a single modality; without explicit balancing the multimodal advantage shrinks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the auxiliary images for the text and audio datasets were generated with FLUX.1-dev, and if their prompts were the target sentences, the visual projector may be trained on target-derived information; the omni-versus-unimodal gap then needs a control experiment before it can be read as evidence of independent visual brain activity.
  • A direct extension would replace generated images with images retrieved without access to the target text, or with unrelated images, and check whether the improvement persists.
  • The router and two-phase alignment could be applied to other paired neural recordings, such as electrocorticography or implanted electrode arrays, wherever multiple auxiliary modalities are available.
  • The abstractness-weight correlation ($r=0.247$) is modest; comparing router weights against a random-weights null model would show whether the interpretability claim survives beyond a single dataset.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes a unified brain-to-text decoding framework that maps fMRI, EEG, and MEG signals into a shared multimodal embedding space derived from a Multimodal LLM (Qwen2.5-Omni). The architecture consists of separate brain projectors aligned to text, image, and audio auxiliary embeddings, plus a learned router that fuses the projected embeddings per sample before feeding the LLM. Training proceeds in two phases: multimodal instruction tuning (auxiliary encoders plus brain projectors aligned with captioning/alignment losses) and projectors fusion with load-balancing and routing losses. Experiments on three fMRI datasets (NSD, Pereira, Huth), one MEG dataset, and one EEG dataset report the best or near-best performance against several baselines, including an 8.48% claimed improvement on NSD and ablations showing the omni (multimodal) variant outperforms unimodal variants.

Significance. If the central claim holds, the paper would make a useful contribution: a single architecture that handles multiple neuroimaging modalities and stimulus types through adaptive multimodal fusion is interesting, and the router-weight/abstractness analysis (Figure 4) is a nice interpretability step. The paper also includes careful ablations over routing strategies and framework components. However, the main significance claim of state-of-the-art multimodal brain decoding currently rests on comparisons that omit important baselines, lack statistical grounding, and, most importantly, may be confounded by auxiliary images generated from the target text on non-visual datasets. The manuscript is not yet at the standard required for publication.

major comments (5)
  1. [Section 4.4 and Table 4] The auxiliary image generation for the Pereira (text) and Huth (audio) datasets is potentially label-derived. Section 4.4 states only "We used FLUX.1-dev with a guidance scale of 3.5 for auxiliary image generation," without specifying the text prompts used. If, as is the default for text-to-image models, the FLUX prompts are the target sentences or transcripts, then the Brain-Image projector is trained on embeddings derived from the very text that the model must output. In that case, the omni-versus-unimodal advantage in Table 4 may reflect ensembling multiple label-derived encoders rather than independent visual information in the brain signal. This is load-bearing for the paper's central claim of multimodal alignment benefits on non-visual stimuli, so the prompt protocol must be specified, and the authors should either use stimulus-independent auxiliary images or re-run the Pereira/Huth ablations without the target-derived visual pathway.
  2. [Abstract and Table 1] The claimed "8.48% improvement on the most commonly used benchmark" is undefined. Inspecting Table 1, the BLEU-1 gains of Ours (similarity merge) over MindLLM and UMBRAE are 3.48 and 3.90 absolute points (roughly 6.0% and 6.8% relative), while the CIDEr gain is about 23%. No entry in Table 1 corresponds to 8.48%. Please state the exact metric, baseline, and computation; also report error bars or significance tests over subjects/splits, since several differences in Table 1 are small relative to likely noise.
  3. [Table 2] The "with text prompt" condition in Table 2 is underspecified. The table caption says textual context is provided as input to the LLM, but the text of that prompt is never defined. If the prompt includes the target sentence or any token derived from it, then those results are not clean brain-decoding results. Please specify exactly what the text prompt contains, for both the "with text prompt" and "without text prompt" settings, and clarify whether the prompt is derived from the target or from a fixed/instructional prefix.
  4. [Table 3] The MEG and EEG evaluations omit several strong baselines that are cited in the Related Work section, including MAD (Yang et al. 2024b), EEG2Text (Liu et al. 2024), and DeWave (Duan et al. 2023). Table 3 compares only random, linear, and BrainLLM. Without these baselines, the claim of "highest performance across all metrics" on MEG/EEG is unsupported. Please add these methods or justify their exclusion.
  5. [Title, Abstract, and Section 5.1] The claim of generalizability is not established because the fMRI results in Table 1 are reported for a single subject (NSD subject 1), and the MEG/EEG experiments use separate datasets with separate subjects. No cross-subject, cross-dataset, or single-model-trained-on-all-data experiment is presented, so the architecture-level "unified" claim is not the same as the empirical generalizability claimed in the title and abstract. Please either add such experiments or temper the wording to "a unified architecture applicable to different datasets and signal types."
minor comments (4)
  1. [Figure 4 and Section 5.2] The Pearson correlation r = 0.247 (p = 0.043) is reported without the number of samples, confidence interval, or any multiple-comparison correction; please report n and the CI, and note whether this is a pre-registered or post-hoc analysis.
  2. [Section 4.4] The input image resolution is stated as 112×112×3, but the CLIP ViT-L/14 backbone normally operates at 224×224; please clarify whether the resolution change is intentional and whether it affects the auxiliary and brain encoder performance.
  3. [Section 3.1 and Figure 3] The description of the hard-select router is unclear: Eq. (1) produces continuous Gumbel-Softmax outputs, Eq. (2) applies Top-1 selection, and the text says gradients use a straight-through estimator, but the precise forward/backward pass is not specified. Please provide the exact implementation of the straight-through estimator.
  4. [Throughout] There are several typos and formatting issues, including "a adaptive routing" in Section 1, "auxilliary" in Figure 2, and the unspaced "Text EncoderImage EncoderAudio Encoder" in Figure 2. A careful proofread is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No equation-level circularity found; the multimodal alignment and routing derivation is self-contained, with only an under-specified auxiliary-image protocol as a reporting caveat.

full rationale

The paper's derivation chain is a standard supervised alignment: Brain Projectors are trained with MSE (Eq. 5) to match auxiliary embeddings produced by external encoders (CLIP-ViT, Whisper, Qwen2.5-Omni), and the Modality Router fuses the projector outputs via Eq. 3. The captioning loss (Eq. 4) and progressive alignment (Eq. 7) are ordinary training objectives; the target text is used as supervision, not as an inference-time input. The reported improvements and ablations are empirical results, not parameters fitted to a subset and then renamed as predictions. There is no load-bearing self-citation chain and no uniqueness theorem imported from the authors' prior work. The only notable caveat is in Sec. 4.4: 'We used FLUX.1-dev with a guidance scale of 3.5 for auxiliary image generation.' For the Pereira (text) and Huth (audio) datasets, the paper does not specify the text prompt used to synthesize those auxiliary images. If the prompt were the target sentence or transcript, the visual projector would be aligned to target-derived image embeddings, which could inflate the apparent multimodal advantage in Table 4. However, this is an under-specified experimental detail and a potential confound, not a circular derivation: inference still maps brain signals to text through learned projectors, and the target text is not an input at inference. Therefore, the derivation itself is self-contained, and no circular step meets the evidentiary bar.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The model is a supervised deep learning system; its central claim depends on a small number of hand-set loss weights and scheduling choices, plus the domain assumption that pretrained multimodal embeddings are a suitable alignment target. The most consequential burden is the FLUX-generated-image protocol, which can couple the auxiliary image pathway to the target text.

free parameters (6)
  • Progressive alignment schedule parameters (lambda, t0) = not reported
    Eq. (6) controls how fast the alignment loss is weighted; values are not given, and they affect convergence.
  • Loss weights lambda1, lambda2 = lambda1=1, lambda2=0.01
    Eq. (10); chosen by hand on NSD, may not transfer to EEG/MEG.
  • Gumbel temperature tau = not reported
    Eq. (1) controls hard-select sharpness; no value or schedule is given.
  • Number of projectors M per dataset = not reported
    M must match the selected auxiliary modalities; per-dataset modality subsets are not enumerated.
  • Soft prompt count and query length = 10 soft prompt tokens, 16 queries
    Given in Sec. 4.4; used across all datasets without tuning details.
  • FLUX.1-dev guidance scale = 3.5
    Used for auxiliary image generation; the generation setting is not justified or ablated.
assumptions (5)
  • domain assumption Pretrained MLLM (Qwen2.5-Omni) and CLIP/Whisper encoders define a shared semantic space that brain embeddings can be aligned into.
    Invoked throughout Sec 3; if this space is not actually shared, the alignment and captioning losses cannot produce the claimed transfer.
  • domain assumption fMRI, EEG, and MEG signals from unimodal stimuli carry enough multimodal content for a router to assign meaningful per-sample modality weights.
    Central motivation; used in Eq. (3) and the abstractness analysis in Sec 5.2.
  • domain assumption MSE alignment between brain embeddings and auxiliary embeddings preserves the semantic information needed for the LLM to generate text.
    Eq. (5); if MSE collapses details the LLM needs, the quality of decoding would not follow.
  • standard math Gumbel-Softmax with straight-through estimator provides a valid differentiable approximation to hard routing.
    Eqs. (1)-(2); standard technique, but convergence behavior is assumed.
  • ad hoc to paper Auxiliary images generated by FLUX.1-dev from target text are valid auxiliary visual targets that do not introduce label leakage.
    Sec 4.4; FLUX images are synthesized from the same sentences to be decoded, so the 'visual' pathway is not independent.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Decoding the Multimodal Mind: Generalizable Brain-to-Text Translation via Multimodal Alignment and Adaptive Routing." pith.science (2026). https://pith.science/paper/GCWZOOBF

@misc{pith2026250510356,
  author       = {Pith},
  title        = {Pith review of: Decoding the Multimodal Mind: Generalizable Brain-to-Text Translation via Multimodal Alignment and Adaptive Routing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GCWZOOBF}},
  note         = {Machine review of arXiv:2505.10356}
}
read the original abstract

Decoding language from the human brain remains a grand challenge for Brain-Computer Interfaces (BCIs). Current approaches typically rely on unimodal brain representations, neglecting the brain's inherently multimodal processing. Inspired by the brain's associative mechanisms, where viewing an image can evoke related sounds and linguistic representations, we propose a unified framework that leverages Multimodal Large Language Models (MLLMs) to align brain signals with a shared semantic space encompassing text, images, and audio. A router module dynamically selects and fuses modality-specific brain features according to the characteristics of each stimulus. Experiments on various fMRI datasets with textual, visual, and auditory stimuli demonstrate state-of-the-art performance, achieving an 8.48% improvement on the most commonly used benchmark. We further extend our framework to EEG and MEG data, demonstrating flexibility and robustness across varying temporal and spatial resolutions. To our knowledge, this is the first unified BCI architecture capable of robustly decoding multimodal brain activity across diverse brain signals and stimulus types, offering a flexible solution for real-world applications.

Figures

Figures reproduced from arXiv: 2505.10356 by the authors.

Figure 1
Figure 1. Top: Conventional BCI frameworks decode lan [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Architecture of the proposed BCI framework, consisting of four key components: an Auxiliary Module (providing [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Comparison of three routing strategies for calcu [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Line-and-scatter plot of Brain-Text Projector [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 20 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Alayrac, J.-B.; Donahue, J.; Luc, P.; Miech, A.; Barr, I.; Hasson, Y.; Lenc, K.; Mensch, A.; Millican, K.; Reynolds, M.; et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35: 23716--23736

    2022

  4. [4]

    J.; St-Yves, G.; Wu, Y.; Breedlove, J

    Allen, E. J.; St-Yves, G.; Wu, Y.; Breedlove, J. L.; Prince, J. S.; Dowdle, L. T.; Nau, M.; Caron, B.; Pestilli, F.; Charest, I.; et al. 2022. A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 25(1): 116--126

    2022

  5. [5]

    J.; Kiela, D.; Clark, S.; and Poesio, M

    Anderson, A. J.; Kiela, D.; Clark, S.; and Poesio, M. 2017. Visually grounded and textual semantic models differentially decode brain activity associated with concrete and abstract nouns. Transactions of the Association for Computational Linguistics, 5: 17--30

    work page 2017

  6. [6]

    Banerjee, S.; and Lavie, A. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, 65--72

    2005

  7. [7]

    Barsalou, L. W. 2008. Grounded cognition. Annu. Rev. Psychol., 59(1): 617--645

    work page 2008

  8. [8]

    C.; Ince, R

    Bolam, J.; Boyle, S. C.; Ince, R. A.; and Delis, I. 2022. Neurocomputational mechanisms underlying cross-modal associations and their influence on perceptual decisions. NeuroImage, 247: 118841

    work page 2022

Show all 43 references
  1. [9]

    B.; and Kuperman, V

    Brysbaert, M.; Warriner, A. B.; and Kuperman, V. 2014. Concreteness ratings for 40 thousand generally known English word lemmas. Behavior research methods, 46(3): 904--911

    2014

  2. [10]

    Duan, Y.; Zhou, J.; Wang, Z.; Wang, Y.-K.; and Lin, C.-T. 2023. Dewave: Discrete eeg waves encoding for brain dynamics to text translation. arXiv preprint arXiv:2309.14030

    2023 arXiv

  3. [11]

    Ferrante, M.; Ozcelik, F.; Boccato, T.; VanRullen, R.; and Toschi, N. 2023. Brain captioning: Decoding human brain activity into images and text. arXiv preprint arXiv:2305.11560

    2023 arXiv

  4. [12]

    Han, J.; Gong, K.; Zhang, Y.; Wang, J.; Zhang, K.; Lin, D.; Qiao, Y.; Gao, P.; and Yue, X. 2024. Onellm: One framework to align all modalities with language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 26584--26595

    2024

  5. [13]

    L.; and Choi, Y

    Hessel, J.; Holtzman, A.; Forbes, M.; Bras, R. L.; and Choi, Y. 2021. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718

    2021 arXiv

  6. [14]

    Huang, W.; Ma, K.; Xie, T.; and Wang, H. 2025. BrainChat: Interactive Semantic Information Decoding from fMRI Using Large-Scale Vision-Language Pretrained Models. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1--5. IEEE

    2025

  7. [15]

    Jang, E.; Gu, S.; and Poole, B. 2016. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144

    2016 arXiv

  8. [16]

    LeBel, A.; Wagner, L.; Jain, S.; Adhikari-Desai, A.; Gupta, B.; Morgenthal, A.; Tang, J.; Xu, L.; and Huth, A. G. 2023. A natural language fMRI dataset for voxelwise encoding models. Scientific Data, 10(1): 555

    2023

  9. [17]

    Lester, B.; Al-Rfou, R.; and Constant, N. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691

    2021 arXiv

  10. [18]

    Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, 19730--19742. PMLR

    2023

  11. [19]

    Lin, C.-Y. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, 74--81

    2004

  12. [20]

    Liu, H.; Hajialigol, D.; Antony, B.; Han, A.; and Wang, X. 2024. Eeg2text: Open vocabulary eeg-to-text decoding with eeg pre-training and multi-view transformer. arXiv preprint arXiv:2405.02165

    2024 arXiv

  13. [21]

    Mai, W.; and Zhang, Z. 2023. Unibrain: Unify image reconstruction and captioning all in one diffusion model from human brain activity. arXiv preprint arXiv:2308.07428

    2023 arXiv

  14. [22]

    Meyer, M.; Baumann, S.; Marchina, S.; and Jancke, L. 2007. Hemodynamic responses in human multisensory and auditory association cortex to purely visual stimulation. BMC neuroscience, 8(1): 14

    2007

  15. [23]

    M.; Shinkareva, S

    Mitchell, T. M.; Shinkareva, S. V.; Carlson, A.; Chang, K.-M.; Malave, V. L.; Mason, R. A.; and Just, M. A. 2008. Predicting human brain activity associated with the meanings of nouns. science, 320(5880): 1191--1195

    2008

  16. [24]

    E.; and Mitchell, T

    Palatucci, M.; Pomerleau, D.; Hinton, G. E.; and Mitchell, T. M. 2009. Zero-shot learning with semantic output codes. Advances in neural information processing systems, 22

    2009

  17. [25]

    Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 311--318

    2002

  18. [26]

    J.; Kanwisher, N.; Botvinick, M.; and Fedorenko, E

    Pereira, F.; Lou, B.; Pritchett, B.; Ritter, S.; Gershman, S. J.; Kanwisher, N.; Botvinick, M.; and Fedorenko, E. 2018. Toward a universal decoder of linguistic meaning from brain activation. Nature communications, 9(1): 963

    2018

  19. [27]

    Qiu, W.; Huang, Z.; Hu, H.; Feng, A.; Yan, Y.; and Ying, R. 2025. MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-Text Decoding. arXiv preprint arXiv:2502.15786

    2025 arXiv

  20. [28]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.00020

    2021 arXiv

  21. [29]

    W.; Xu, T.; Brockman, G.; McLeavey, C.; and Sutskever, I

    Radford, A.; Kim, J. W.; Xu, T.; Brockman, G.; McLeavey, C.; and Sutskever, I. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, 28492--28518. PMLR

    2023

  22. [30]

    Spence, C. 2011. Crossmodal correspondences: A tutorial review. Attention, Perception, & Psychophysics, 73(4): 971--995

    2011

  23. [31]

    Takagi, Y.; and Nishimoto, S. 2023. High-resolution image reconstruction with latent diffusion models from human brain activity. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 14453--14463

    2023

  24. [32]

    Tang, J.; LeBel, A.; Jain, S.; and Huth, A. G. 2023. Semantic reconstruction of continuous language from non-invasive brain recordings. Nature Neuroscience, 26(5): 858--866

    2023

  25. [33]

    Vedantam, R.; Lawrence Zitnick, C.; and Parikh, D. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, 4566--4575

    2015

  26. [34]

    Wang, P.; Bai, S.; Tan, S.; Wang, S.; Fan, Z.; Bai, J.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; et al. 2024. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191

    2024 arXiv

  27. [35]

    Wang, S.; Zhang, X.; Zhang, J.; and Zong, C. 2022. A synchronized multimodal neuroimaging dataset for studying brain language processing. Scientific Data, 9(1): 590

    2022

  28. [36]

    Xia, W.; de Charette, R.; Oztireli, C.; and Xue, J.-H. 2024. Umbrae: Unified multimodal brain decoding. In European Conference on Computer Vision, 242--259. Springer

    2024

  29. [37]

    Xu, J.; Guo, Z.; He, J.; Hu, H.; He, T.; Bai, S.; Chen, K.; Wang, J.; Fan, Y.; Dang, K.; et al. 2025. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215

    2025 arXiv

  30. [38]

    Yang, A.; Yang, B.; Hui, B.; Zheng, B.; Yu, B.; Zhou, C.; Li, C.; Li, C.; Liu, D.; Huang, F.; Dong, G.; Wei, H.; Lin, H.; Tang, J.; Wang, J.; Yang, J.; Tu, J.; Zhang, J.; Ma, J.; Yang, J.; Xu, J.; Zhou, J.; Bai, J.; He, J.; Lin, J.; Dang, K.; Lu, K.; Chen, K.; Yang, K.; Li, M....

    2024 arXiv

  31. [39]

    H.; Xu, R.; and Xiong, H

    Yang, Y.; Jo, H.; Duan, Y.; Zhang, Q.; Zhou, J.; Lee, W. H.; Xu, R.; and Xiong, H. 2024 b . Mad: Multi-alignment meg-to-text decoding. arXiv preprint arXiv:2406.01512

    2024

  32. [40]

    Ye, Z.; Ai, Q.; Liu, Y.; de Rijke, M.; Zhang, M.; Lioma, C.; and Ruotsalo, T. 2025. Generative language reconstruction from brain recordings. Communications Biology, 8(1): 346

    2025

  33. [41]

    Zhao, X.; Sun, J.; Wang, S.; Ye, J.; Zhang, X.; and Zong, C. 2024. Mapguide: A simple yet effective method to reconstruct continuous language from brain activities. arXiv preprint arXiv:2403.17516

    2024 arXiv

  34. [42]

    Zhao, Y.; Dong, G.; Zhu, L.; and Ying, X. 2025. Memory recall: Retrieval-Augmented mind reconstruction for brain decoding. Information Fusion, 103280

    2025

  35. [43]

    Zou, S.; Wang, S.; Zhang, J.; and Zong, C. 2022. Cross-modal cloze task: A new task to brain-to-word decoding. In Findings of the Association for Computational Linguistics: ACL 2022, 648--657

    2022

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.