REVIEW 4 major objections 5 minor 6 cited by
The Algonauts Project 2025 Challenge: How the Human Brain Makes Sense of Multimodal Movies
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read This paper introduces a brain-encoding challenge whose central claim is that the best models of the brain are those that generalize beyond their training distribution, and it operationalizes that claim by selecting the winner solely on…
desk verdict A well-specified challenge announcement, not a results paper; the OOD leaderboard may be too noisy to crown a reliable winner, but that is a fixable design gap, not a fatal flaw. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a two-phase, two-leaderboard evaluation split. In the six-month building phase, models train on 55 hours of the sitcom and 10 hours of four films, and are tested on the held-out subsequent season with unlimited submissions; this gives in-distribution performance. In the one-week selection phase, models predict brain responses to two hours of withheld movies from outside that distribution, with a maximum of ten submissions, and winners are ranked solely on this out-of-distribution score. Scoring is done by averaging Pearson correlation over the 1,000 functionally defined brain parcels, then over out-of-distribution movies or held-out episodes, then over the four subjects. The load-bearing baseline is a linearizing encoding model that maps extracted visual, audio, and language features to fMRI responses, achieving r=0.20 in-distribution and r=0.09 out-of-distribution.
What would settle it
Compute a feature-space distance, using a pretrained audio-visual or language model, between the withheld out-of-distribution movies and the sitcom and film training material; if that distance is no larger than the distance between two seasons of the same sitcom, the out-of-distribution set is not actually out-of-distribution and the challenge's main ranking would not measure generalization.
Extended reading notes
Core claim
The paper's claim is that the next generation of brain encoding models should be multimodal and should be selected by their out-of-distribution generalization, and it offers a concrete way to measure that: the largest single-subject fMRI movie-watching dataset assembled so far, split so that training, in-distribution testing, and out-of-distribution testing draw on different content. An encoding model is any algorithm that maps movie stimuli, including visual frames, audio, and time-stamped language transcripts, to predicted fMRI activity in 1,000 cortical parcels for each of four subjects. The quality of a model is a single number: Pearson correlation between predicted and recorded responses, averaged across parcels, then across stimuli, then across subjects. The challenge is organized so that the winner cannot be chosen by repeated probing of the out-of-distribution test set; the out-of-distribution movies are revealed only in a one-week selection phase, and only ten submissions are allowed. The intended payoff is a transparent leaderboard that ranks models both on a familiar test from the same TV series and on a genuinely novel test, making out-of-distribution robustness part of what a good brain model means.
Load-bearing premise
The whole design rests on two hours of withheld movies being different enough from the training films, and reliable enough, that an out-of-distribution correlation score is a fair and stable ranking of model quality; if those movies are too similar, too noisy, or too short, the central test of generalization fails.
Editorial extensions
If this is right
- Multimodal fusion of visual frames, audio, and transcripts becomes a prerequisite for top scores, since the stimuli and scoring combine all three modalities.
- Unlimited in-distribution submissions let teams tune their models on a familiar test, while the ten-submission out-of-distribution cap makes the final ranking a test of generalization rather than leaderboard overfitting.
- The single averaged correlation score makes models of any architecture directly comparable on identical data and identical preprocessing.
- The indefinite post-challenge phase turns the challenge into a permanent public benchmark with separate in-distribution and out-of-distribution leaderboards.
- If participants close the gap between the r=0.20 and r=0.09 baselines, that would demonstrate that data-hungry, end-to-end trained encoding models can generalize to new movies.
Reading between the lines
- A natural extension the paper gestures at but does not implement is disaggregating out-of-distribution scores by modality or by movie genre; such breakdowns could reveal exactly which stimulus properties models fail to capture.
- The design implicitly predicts that in-distribution leaderboard ranks will not match out-of-distribution ranks; if they do match closely, selecting winners by out-of-distribution performance would add little information.
- After the challenge, a decisive validity check would be to compare the out-of-distribution-selected winners against a fresh, never-revealed movie set; if their advantage evaporates, the two-hour out-of-distribution set was too small or too similar to the training data.
- If the benchmark works, the same train-withhold-out-of-distribution structure could be adopted for other brain-encoding problems, for example predicting responses during tasks with active cognition rather than passive movie watching.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript introduces the Algonauts Project 2025 challenge, which asks participants to build encoding models that predict fMRI responses to multimodal movie stimuli. The challenge uses the CNeuroMod dataset: four subjects, almost 80 hours of fMRI per subject, with training data from six seasons of Friends and a set of four movies (Movie10), an in-distribution test on season 7 of Friends, and a two-hour out-of-distribution test on withheld movies. Winners are selected solely on OOD performance during a one-week selection phase with at most ten submissions, after a six-month building phase with unlimited ID submissions. The paper describes the data, phases, rules, baseline model (a linearizing encoding model with visual, audio, and language features, reporting r=0.20 ID and r=0.09 OOD), development kit, and the scientific rationale for multimodal naturalistic stimulation and OOD generalization.
Significance. If the challenge runs as designed, it will provide the field with a large, open, multimodal fMRI encoding benchmark with a public leaderboard and an explicit OOD generalization test, which are valuable and complementary to existing initiatives such as Brain-Score and Sensorium. The choice of CNeuroMod data and the release of a development kit and automated scoring infrastructure are concrete strengths. The paper's central scientific promise, however, is that winners selected on the OOD score will be models that genuinely generalize beyond the training distribution; that promise depends on the statistical reliability of the OOD leaderboard, which the manuscript does not yet establish.
major comments (4)
- [Model selection phase and Baseline model] The central claim that winners are selected 'solely based on their OOD performance' depends on the OOD score being reliable enough to rank models. The paper reports only r=0.09 for the baseline OOD score, with no error bars, split-half reliability, bootstrap confidence intervals, or subject-wise and parcel-wise variance. With 2 hours of OOD stimuli per subject and averaging over 1,000 parcels, the sampling variance of this mean correlation is non-negligible; the manuscript should quantify the reliability of the OOD leaderboard (e.g., split-half correlation across OOD movies, bootstrap CI on the baseline, or a noise ceiling estimate) and state a criterion for when differences between models are meaningful.
- [Rules and Model selection phase] Because the OOD leaderboard is updated after each of the up to ten submissions, participants can choose their best of ten scores on the same small OOD set. If the OOD metric is as noisy as the r=0.09 baseline suggests, this protocol selects on noise rather than on generalization. The paper should describe a safeguard, such as a final hold-out split of OOD data used only after the ten submissions are frozen, or a statistical test comparing the top submission against the baseline and against other top submissions.
- [Baseline model] The baseline description is too underspecified to be reproduced: 'extracts visual, audio, and language features' does not state which features are used, how they are temporally aligned to the fMRI time series, what regression or regularization is applied, or whether the model is fit per subject and parcel. Since the baseline is the reference score against which all entries are judged, this omission weakens the scientific value of the benchmark and should be fixed by providing a precise specification or a link to the baseline code.
- [Model selection phase] The term 'out-of-distribution' is used without defining the distribution shift. Because the OOD movies are unrevealed until the selection phase, participants cannot know the shift, but the organizers should specify what dimensions of shift are intended (e.g., new narrative content, different genres, new audiovisual statistics) and ideally provide a planned post-hoc measure of distribution shift to verify that the OOD set is actually outside the training distribution.
minor comments (5)
- [Model building phase] The phrase 'all episodes of seasons 7 of the Friends dataset' should be 'season 7'.
- [Discussion] The word 'premiating' should be replaced with 'rewarding' or 'prizing'.
- [References] The two Richards et al. 2019 entries are identical; one duplicate should be removed.
- [Data] The sentence 'Further information on the challenge stimuli and fMRI data is provided on the challenge data repository and development kit' lacks a URL or specific citation; please add the repository link.
- [Figure 2c] The caption and text mention the maximum of ten submissions in the model selection phase, but the figure should visually indicate this cap to avoid ambiguity.
Circularity Check
No circularity: this challenge specification paper contains no derivation whose output reduces to its inputs, and the baseline scores are reference values, not predictions.
full rationale
The paper is a challenge specification, not a derivation-based study. It describes challenge data, phases, rules, and a baseline model, but it makes no scientific claim that is derived from fitted parameters. The baseline model is reported as a reference score (r = 0.20 in-distribution, r = 0.09 out-of-distribution) with the explicit status of a baseline in the 'Baseline model' section, not as evidence for an underlying theory. The OOD evaluation protocol is designed to avoid fitting to the withheld test set: the OOD stimuli are unrevealed until the model selection phase, and the winners are selected solely on OOD performance with a capped number of submissions. This is the opposite of a fitted-input-called-prediction pattern, since the withheld responses are not used to tune model parameters. The paper cites prior Algonauts challenge papers by overlapping authors, but these citations are contextual references to prior challenge editions and are not load-bearing for any central claim that requires independent validation; the current challenge is defined by its own new data, protocol, and benchmark structure. The skeptical concern that the two-hour OOD leaderboard may be too noisy to rank models reliably is a testable design risk about benchmark validity, not a circularity in the reasoning. Because there is no derivation chain whose outputs are equivalent to its inputs by construction, the circularity score is 0.
Assumptions & free parameters
assumptions (4)
- domain assumption fMRI responses in 1,000 Schaefer parcels, normalized to MNI, are a sufficient and meaningful target for encoding model evaluation.
- domain assumption Averaging Pearson correlations across parcels, then movies, then subjects produces a meaningful model ranking.
- domain assumption The withheld OOD movie set is sufficiently out-of-distribution and sufficiently long to rank models on generalization.
- domain assumption The provided training set, 55 hours of Friends seasons 1 to 6 plus 10 hours of Movie10, is large and representative enough to train generalizable multimodal encoding models.
Cite this review
Pith. "Pith review of The Algonauts Project 2025 Challenge: How the Human Brain Makes Sense of Multimodal Movies." pith.science (2026). https://pith.science/paper/E4EN5GUO
@misc{pith2026250100504,
author = {Pith},
title = {Pith review of: The Algonauts Project 2025 Challenge: How the Human Brain Makes Sense of Multimodal Movies},
year = {2026},
howpublished = {\url{https://pith.science/paper/E4EN5GUO}},
note = {Machine review of arXiv:2501.00504}
}
read the original abstract
There is growing symbiosis between artificial and biological intelligence sciences: neural principles inspire new intelligent machines, which are in turn used to advance our theoretical understanding of the brain. To promote further collaboration between biological and artificial intelligence researchers, we introduce the 2025 edition of the Algonauts Project challenge: How the Human Brain Makes Sense of Multimodal Movies (https://algonautsproject.com/). In collaboration with the Courtois Project on Neuronal Modelling (CNeuroMod), this edition aims to bring forth a new generation of brain encoding models that are multimodal and that generalize well beyond their training distribution, by training them on the largest dataset of fMRI responses to movie watching available to date. Open to all, the 2025 challenge provides transparent, directly comparable results through a public leaderboard that is updated automatically after each submission to facilitate rapid model assessment and guide development. The challenge will end with a session at the 2025 Cognitive Computational Neuroscience (CCN) conference that will feature winning models. We welcome researchers interested in collaborating with the Algonauts Project by contributing ideas and datasets for future challenges.
Forward citations
Cited by 6 Pith papers
-
NeuroWorld: A Latent Brain World Model for Stimulus-Conditioned Human Brain Dynamics
A latent world model of the brain, trained to predict the next latent fMRI state from past brain states and current movie stimuli, outperforms regression-based encoders in causal multi-step rollout on three naturalist...
-
RABBiT: Rapidly adaptive BOLD foundation model via brain-tuning for accurate zero-shot and few-shot prediction of speech-elicited responses in the brain
A 12.9M-parameter audio-to-fMRI encoder achieves zero-shot prediction of speech-evoked brain responses on 324 unseen participants and few-shot adaptation with ~10 minutes of data, outperforming larger baselines and pe...
-
A global predicted-fMRI drive signal from TRIBE does not predict YouTube replay heatmaps
TRIBE’s predicted cortical drive does not predict YouTube most-replayed heatmaps beyond position and low-level baselines, with the null bounded near r≈0.14.
-
TRIBE: TRImodal Brain Encoder for whole-brain fMRI response prediction
A transformer-based encoder that combines text, audio, and video embeddings predicts whole-brain fMRI responses to movies across subjects and won the Algonauts 2025 competition.
-
Probing Multimodal Fusion in the Brain: The Dominance of Audiovisual Streams in Naturalistic Encoding
On the Algonauts 2025 benchmark, a linear brain-encoding model with X-CLIP and Whisper features outperformed an attention-based model on out-of-distribution movies, and linguistic features were excluded after prelimin...
-
VIBE: Video-Input Brain Encoder for fMRI Response Modeling
A two-stage multimodal Transformer with fixed pretrained feature extractors predicts fMRI activity from movies with Pearson r = 0.32 (in-domain) and 0.21 (out-of-domain), beating the challenge baseline by about 0.12.
Reference graph
Works this paper leans on
-
[106]
Identifying natural images from human brain activity
https://doi.org/10.1162/nol_a_00101 Kay, K.N., Naselaris, T., Prenger, R.J., Gallant, J.L., 2008. Identifying natural images from human brain activity. Nature 452, 352–355. https://doi.org/10.1038/nature06713 Kell, A.J.E., Yamins, D.L.K., Shook, E.N., Norman- Haignere, S.V., McDermott, J.H., 2018. A Task- Optimized Neural Network Replicates Human Auditory...
arXiv 2008
-
[183]
The Sensorium competition on predicting large-scale mouse primary visual cortex activity
https://doi.org/10.1016/j.jmp.2016.06.009 Willeke, K.F., Fahey, P.G., Bashiri, M., Pede, L., Burg, M.F., Blessing, C., Cadena, S.A., Ding, Z., Lurz, K.- K., Ponder, K., Muhammad, T., Patel, S.S., Ecker, A.S., Tolias, A.S., Sinz, F.H., 2022a. The Sensorium competition on predicting large-scale mouse primary visual cortex activity. https://doi.org/10.48550/...
-
[423]
Leveraging the Human Ventral Visual Stream to Improve Neural Network Robustness
https://doi.org/10.1016/j.neuron.2020.07.040 Shao, Z., Ma, L., Li, B., Beck, D.M., 2024. Leveraging the Human Ventral Visual Stream to Improve Neural Network Robustness. https://doi.org/10.48550/ARXIV.2405.02564 Shmueli, G., 2010. To Explain or to Predict? Stat. Sci. 25. https://doi.org/10.1214/10-STS330 Sinz, F.H., Pitkow, X., Reimer, J., Bethge, M., Tol...
-
[2011]
Towards robust vision by multi-task learning on monkey visual cortex
Encoding and decoding in fMRI. NeuroImage 56, 400–410. https://doi.org/10.1016/j.neuroimage.2010.07.073 Ren, Y., Bashivan, P., 2024. How well do models of visual cortex generalize to out of distribution samples? PLOS Comput. Biol. 20, e1011145. https://doi.org/10.1371/journal.pcbi.1011145 Richards, B., Tsao, D., Zador, A., 2022. The application of artific...
work page Pith review arXiv doi:10.48550/arxiv.2107.14344 2010
-
[2018]
Local-Global Parcellation of the Human Cerebral Cortex from Intrinsic Functional Connectivity MRI. Cereb. Cortex 28, 3095–3114. https://doi.org/10.1093/cercor/bhx179 Schrimpf, M., Kubilius, J., Hong, H., Majaj, N.J., Rajalingham, R., Issa, E.B., Kar, K., Bashivan, P., Prescott-Roy, J., Geiger, F., Schmidt, K., Yamins, D.L.K., DiCarlo, J.J., 2018. Brain-Sc...
-
[2021]
Towards Out-Of-Distribution Generalization: A Survey. https://doi.org/10.48550/ARXIV.2108.13624 Madan, S., Xiao, W., Cao, M., Pfister, H., Livingstone, M., Kreiman, G., 2024. Benchmarking Out-of-Distribution Generalization Capabilities of DNN-based Encoding Models for the Ventral Visual Cortex. https://doi.org/10.48550/ARXIV.2406.16935 Mathis, M.W., Perez...
-
[2022]
A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nat. Neurosci. 25, 116–126. https://doi.org/10.1038/s41593-021-00962-x Boyle, J., Pinsard, B., Borghesani, V., Paugam, F., DuPre, E., Bellec, P., 2023. The Courtois NeuroMod project: quality assessment of the initial data release (2020), in: 2023 Conference on Cognitiv...
-
[2023]
Many but not all deep neural network audio models capture brain responses and exhibit correspondence between model stages and brain regions. PLOS Biol. 21, e3002366. https://doi.org/10.1371/journal.pbio.3002366 Turishcheva, P., Fahey, P.G., Hansel, L., Froebe, R., Ponder, K., Vystrčilová, M., Willeke, K.F., Bashiri, M., Wang, E., Ding, Z., Tolias, A.S., S...
Show all 9 references
- [2024]
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.