REVIEW 11 cited by
Scaling laws for decoding images from brain activity
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Generative AI has recently propelled the decoding of images from brain activity. How do these approaches scale with the amount and type of neural recordings? Here, we systematically compare image decoding from four types of non-invasive devices: electroencephalography (EEG), magnetoencephalography (MEG), high-field functional Magnetic Resonance Imaging (3T fMRI) and ultra-high field (7T) fMRI. For this, we evaluate decoding models on the largest benchmark to date, encompassing 8 public datasets, 84 volunteers, 498 hours of brain recording and 2.3 million brain responses to natural images. Unlike previous work, we focus on single-trial decoding performance to simulate real-time settings. This systematic comparison reveals three main findings. First, the most precise neuroimaging devices tend to yield the best decoding performances, when the size of the training sets are similar. However, the gain enabled by deep learning - in comparison to linear models - is obtained with the noisiest devices. Second, we do not observe any plateau of decoding performance as the amount of training data increases. Rather, decoding performance scales log-linearly with the amount of brain recording. Third, this scaling law primarily depends on the amount of data per subject. However, little decoding gain is observed by increasing the number of subjects. Overall, these findings delineate the path most suitable to scale the decoding of images from non-invasive brain recordings.
Forward citations
Cited by 11 Pith papers
-
How Much is Brain Data Worth for Machine Learning?
Brain data is worth a variable number of task samples depending on task-brain alignment, noise levels, and latent dimension, with conditions under which it also improves robustness to test distribution shift.
-
NeuralBench: A Unifying Framework to Benchmark NeuroAI Models
NeuralBench is a new benchmarking framework for neuroAI models on EEG data that finds foundation models only marginally outperform task-specific ones while many tasks like cognitive decoding stay highly challenging.
-
A foundation model of vision, audition, and language for in-silico neuroscience
TRIBE v2 is a multimodal AI model that predicts human brain activity more accurately than linear encoding models and recovers established neuroscientific findings through in-silico testing.
-
Disentangling the Factors of Convergence between Brains and Computer Vision Models
By systematically varying model size, training amount, and image type in DINOv3 vision transformers, this paper shows that brain similarity increases with scale and human-centric data and emerges in a characteristic t...
-
Real-time Reconstruction of Human Visual Perception from fMRI
First demonstration that single-trial visual images can be decoded from fMRI in near-real-time (about 10-15 seconds) with roughly one hour of fine-tuning data.
-
Visualizing the Invisible: Generative Visual Grounding Empowers Universal EEG Understanding in MLLMs
Generative Visual Grounding creates visual proxy images from EEG to enhance MLLM understanding of brain signals beyond text-only alignment.
-
Visualizing the Invisible: Generative Visual Grounding Empowers Universal EEG Understanding in MLLMs
Generative Visual Grounding creates instance-specific visual proxy images from EEG signals to enhance MLLM understanding of brain activity beyond text-only alignment.
-
DANCE: Detect and Classify Events in EEG
DANCE frames EEG event identification as a set-prediction problem to jointly detect and classify events directly from raw, unaligned signals, outperforming existing methods on seizure monitoring and matching onset-inf...
-
Boosting Brain-to-Image Decoding with TRIBE v2 Data Augmentation
Augmenting limited fMRI datasets with synthetic responses from TRIBE v2 improves brain-to-image decoding accuracy and can yield above-chance performance using only synthetic data.
-
WorldWeaver: Generating Long-Horizon Video Worlds via Rich Perception
WorldWeaver reduces temporal drift in long-horizon video generation by jointly modeling RGB and depth perceptual conditions with segmented noise scheduling.
-
Foundation Models for Cross-Domain EEG Analysis Application: A Survey
A survey that organizes EEG foundation-model research into five output-modality categories: native EEG, text, vision, audio, and multimodal fusion, with a claim to be the first such comprehensive taxonomy.
Discussion (0). Sign in to comment.