On movie-watching fMRI data, multi-modal vision-audio transformers predict brain activity better than unimodal video or speech models, with video dominating cross-modal alignment and video plus audio jointly contributing in jointly-pretrained models.
Lxmert: Learning cross-modality encoder representations from transform- ers
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
q-bio.NC 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Multi-modal brain encoding models for multi-modal stimuli
On movie-watching fMRI data, multi-modal vision-audio transformers predict brain activity better than unimodal video or speech models, with video dominating cross-modal alignment and video plus audio jointly contributing in jointly-pretrained models.