{"id":"57fd78ab-e719-48aa-bb49-cef47b9182c4","arxiv_id":"2507.22229","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A transformer-based encoder that combines text, audio, and video embeddings predicts whole-brain fMRI responses to movies across subjects and won the Algonauts 2025 competition.","lead":"Meta AI researchers trained TRIBE, a neural network that predicts which brain regions activate while people watch videos, by fusing text, audio, and video features from pretrained AI models. It took first place in the 263-team Algonauts 2025 brain-encoding competition, and the authors show that combining modalities helps most in association cortices.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validation-set ensembling of 1000 models may inflate the 0.0050 leaderboard margin; no nested check or error bars establish that the ensemble selection transfers to the test set.","rationale":"The paper is a strong engineering result: an external leaderboard confirms the rank, the ablation structure is coherent, and the code is released. My stress-test focuses on the single claim most connected to the headline: that the first-place margin is meaningful. The reader's weakest assumption already identified validation-driven ensemble selection as a risk, and I agree that is the most load-bearing concern. I do not elevate it to a rejection because the leaderboard is an externally administered test set and the ensemble could well generalize; the issue is that the paper supplies no evidence that it does. The noise-ceiling ambiguity is a genuine secondary issue but less central to the competition claim. A nested validation split and a bootstrap confidence interval on the margin would settle the question. Since the reader's verdict is already CONDITIONAL with exactly these requested checks, my recommendation is unchanged.","tokens_in":10787,"tokens_out":4232,"duration_ms":56017,"concrete_test":"Using the public code and Courtois NeuroMod data, create a second, disjoint validation split (e.g., an additional 10% of sessions per subject) that is never used for early stopping, hyperparameter sampling, or ensemble weight fitting. Compute three Table-1-style mean scores on this new split: (i) the per-parcel softmax ensemble selected on the original validation set, (ii) a uniform ensemble of the same 1000 models, and (iii) the best single model. If (i) does not beat (ii) by more than the 0.0050 margin, or if bootstrap resampling of sessions gives a 95% confidence interval on the gap with NCG that includes zero, the 'significant margin' claim is not robust to validation-set selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claims are the first-place Algonauts 2025 result and the 'significant margin' over the runner-up in Table 1. The most fragile step producing those numbers is the validation-driven ensembling in Section 2.4. For each of M=1000 models, with hyperparameters independently sampled per Appendix A, a softmax distribution over models is fit per parcel to the validation Pearson score using temperature 0.3; early stopping and stochastic weight averaging also use the same validation sessions. This is a high-capacity selection procedure: effectively 1000 x 1000 weights (one distribution over 1000 models for each of 1000 parcels) are optimized on roughly 10% of the recording sessions. If the validation set is not large enough to stabilize the per-parcel winner, the ensemble can memorize idiosyncrasies of those sessions, and the reported test margin (mean 0.0050 over NCG, with subject 5 separated by 0.0002) could be optimistically biased. The paper provides no error bars, no statistical comparison, and no check that the ensemble weights selected on validation actually transfer to the leaderboard. The noise-ceiling ambiguity in Section 3.2 is also real, but secondary: the 0.54 normalized Pearson depends on whether model rho and rho_self are computed on disjoint sessions, and even if inflated it would not overturn the competition rank.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces TRIBE, a transformer-based encoding model that predicts whole-brain fMRI responses from text, audio, and video features extracted from frozen pretrained models (Llama-3.2, Wav2Vec-Bert-2.0, V-JEPA 2). The model is trained on four subjects of the Courtois NeuroMod dataset and evaluated on the Algonauts 2025 competition, where it reports first place. The paper includes modality ablation analyses showing gains from multimodality, a noise-ceiling normalization reporting 54% of explainable variance, and scaling-law experiments. Central claims are that the pipeline is the first simultaneously nonlinear, multisubject, and multimodal encoder, and that it outperforms competitors by a significant margin.","tokens_in":11119,"tokens_out":6263,"duration_ms":67355,"significance":"The external competition result is strong evidence that the pipeline works in practice, and the code is released. If the statistical concerns below are addressed, this would be a valuable contribution to the brain-encoding literature. The combination of three modalities with a nonlinear multisubject model is practically useful, and the per-parcel maps of multimodal gains are potentially interesting to neuroscientists. However, the headline margin and the 54% explainable-variance fraction need stronger statistical support before the claims can be taken at face value.","major_comments":[{"comment":"The per-parcel ensemble selection over M=1000 models using validation Pearson scores and softmax temperature 0.3, combined with early stopping and hyperparameter sampling on the same validation set, is a high-capacity selection procedure; the manuscript reports no error bars or significance tests for the leaderboard gap, so the word 'significant' in the Abstract and Section 3.1 is not statistically supported, especially because the subject-5 margin is only 0.0002 (0.1720 vs 0.1718). Please provide a bootstrap over subjects or sessions, or at least report per-subject standard errors, and discuss whether the validation-based ensemble selection transfers to the test set.","section":"Section 2.4 and Table 1"},{"comment":"The manuscript does not state whether the model correlation ρ in Eq. (1) is computed on held-out data disjoint from the repeated-viewing movies (Hidden Figures, Life) used to estimate ρ_self; if the same sessions contribute to both quantities, the reported 54% of explainable variance is inflated because ρ and ρ_self are not independent. Please specify the exact data split used for ρ and ρ_self and, if necessary, recompute the normalized scores on disjoint sessions.","section":"Section 3.2, Eq. (1)"},{"comment":"The claim that TRIBE is 'the first encoding pipeline which is simultaneously nonlinear, multisubject and multimodal' is not established by the cited related work; in particular, the Deep Recurrent Encoder [19] is cited as relaxing linearity and may already be multisubject, so the novelty should be scoped to the specific combination of three input modalities and a transformer encoder, rather than to the individual properties. Please temper or justify this claim with a more systematic comparison.","section":"Section 4 and Abstract"}],"minor_comments":[{"comment":"The text reports 'out of 262 teams' while the Table 1 caption says 'out of 263 teams'; please reconcile the two numbers.","section":"Section 3.1 and Table 1"},{"comment":"The phrase 'start-of-the art' should be 'state-of-the-art'.","section":"Section 2.1"},{"comment":"The word 'substracting' should be 'subtracting'.","section":"Figure 5b caption"},{"comment":"The sentence 'text+audio (yellow) in can be observed in the superior temporal lobe' contains a grammatical error ('in' is repeated); please revise.","section":"Section 3.4"},{"comment":"The audio embeddings are noted to carry bidirectional information while text and video embeddings are causal; this asymmetry is not discussed in the modality ablation results and could affect their interpretation.","section":"Section 2.3"},{"comment":"The 'Layer groups' hyperparameter values such as [0.5, 0.75, 1] are not defined; please specify whether these are fractions of the total number of layers or absolute layer indices.","section":"Table 3"},{"comment":"The Pearson correlation values printed next to the brain parcels are not defined in the caption; please state that they are the correlation between predicted and actual BOLD for the first participant and the first held-out movie.","section":"Figure 1"},{"comment":"The code repository is mentioned only in the Abstract; please add a Code Availability section in the main text or at the end of the paper.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is essentially a competition-report-style manuscript with a strong external validation result. The main technical weakness is the absence of any statistical quantification around the headline numbers; the ensembling protocol and noise-ceiling computation are both plausible but under-specified. If the authors can add the requested analyses or clearly state the existing data splits, the paper would be suitable for publication. I would also suggest the editor check whether the claim of being 'first' in the combined setting is supported by prior art beyond the few references cited."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"TRIBE is the real deal: it does what it says, wins the Algonauts 2025 benchmark with a whole-brain encoder that is simultaneously nonlinear, multisubject, and multimodal. The external leaderboard result and the coherent ablations (unimodal vs multimodal, with/without transformer, with/without multisubject) make the central claims credible. I agree with the reader's overall conditional verdict, and I think the stress-test note identifies the right soft spot, though it's a soft spot rather than a fatal one.\n\nWhat's genuinely new: previous multimodal encoders used linear mappings; previous nonlinear ones were single-modality; TRIBE's combination of frozen text/audio/video features with a transformer and subject-conditional readout is a clean, useful step. The results on OOD movies (cartoons, silent films) are impressive generalization evidence. The code is released, which is a real plus.\n\nThe main concern is the ensembling. Fitting a per-parcel softmax over 1000 models on the validation set, with early stopping and SWA also tuned on that same set, means the ensemble weights are themselves fitted. The reported 0.0050 mean gap over NCG could be optimistic, and the paper gives no error bars or nested validation to show the selection transfers. That's a legitimate flaw in the reporting. However, this is a competition ranking, not a statistical claim about a scientific hypothesis; the rank is what it is, and the gap is small enough that the paper should not call it 'significant' without a test. Also the noise-ceiling normalization (eq. 1) needs to state whether model rho and rho_self are computed on disjoint data; as written, the 54% could be inflated. Both are addressable with minor revisions.\n\nThe 'first' phrasing in the abstract and Section 4 is a bit strong given the fast-moving literature, but the related work does cite the nearest competitors, and the claim is qualified enough in context.\n\nBottom line: this is a solid technical report that deserves serious review. I'd accept it with requests for error bars or at least a nested ensemble check, a clarification of the noise-ceiling data split, and softer 'first/margin' language. The paper is worth a reading group slot and I'd cite it as the current Algonauts winner.","headline":"Solid competition-winning encoding model with a real but addressable validation-selection concern.","tokens_in":11593,"tokens_out":2134,"would_cite":true,"duration_ms":25295,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single multimodal model predicts whole-brain fMRI responses to movies across subjects.","keywords":["brain encoding","fMRI","multimodal","transformer","naturalistic movies","whole-brain prediction","multi-subject learning","modality dropout"],"falsifier":"Re-run the pipeline with the ensemble weights and early-stopping decisions fixed on a split that is never used during development, and evaluate on a fresh held-out set; if the mean test correlation falls below the runner-up's score, the reported margin is an artifact of validation-based selection. Separately, restrict the noise-ceiling estimate to recording sessions that are disjoint from the sessions used to compute the model's correlations; a normalized Pearson well below 0.54 would show that the explainable-variance claim is inflated by session overlap.","tokens_in":10600,"feed_emoji":"🧠","tokens_out":6843,"duration_ms":71357,"temperature":0.7,"pith_summary":"The paper introduces TRIBE, a deep encoding pipeline that predicts the brain's BOLD responses to naturalistic videos from three input streams at once: the transcript, the audio, and the video frames. It combines frozen representations from a language model, a speech model, and a video model, then lets a transformer integrate them over time. The central claim is that this is the first encoding pipeline that is simultaneously nonlinear, multisubject, and multimodal, and that this combination is what lets a single model reach top accuracy across 1,000 cortical parcels and four participants. If the claim holds, whole-brain encoding no longer needs one linear model per modality per subject, and the parts of the brain that integrate senses can be studied with a predictive model rather than by comparing separate unimodal fits.","feed_headline":"Multimodal transformer predicts whole-brain fMRI from movies","feed_subtitle":"Combining frozen text, audio, and video features lifts encoding scores, with the biggest gains in association cortices.","key_machinery":"The load-bearing design is a transformer encoder fed with time-aligned embeddings from three frozen foundation models: Llama-3.2-3B for transcript words, Wav2Vec-Bert-2.0 for audio, and V-JEPA-2 for video, all resampled to a common 2 Hz grid. Each modality's intermediate layers are grouped and averaged, projected to a shared dimension, concatenated, and given learnable positional and subject embeddings. The transformer exchanges information across time, an adaptive pooling layer compresses the sequence to one embedding per fMRI repetition time, and a subject-conditional linear layer maps the result onto 1,000 parcels. During training, modality dropout randomly masks whole modalities so the model learns to predict from any subset; the final predictions come from a per-parcel softmax ensemble over 1,000 models.","core_discovery":"On its own terms, the paper establishes that combining text, audio, and video embeddings in one transformer-based encoder improves fMRI prediction over any unimodal encoder, and that the gains are largest in associative cortices such as prefrontal and parieto-occipito-temporal areas, while the best unimodal model still wins in primary visual cortex. The model reaches a mean Pearson correlation of 0.2146 on the competition's out-of-distribution held-out movies and 0.3195 on an in-distribution Friends episode, and it would capture about 54% of the noise-ceiling-normalized explainable variance. The paper also reports that removing the transformer drops the score from 0.31 to 0.23 and removing multi-subject training drops it to 0.29, and that adding either more recording sessions or longer language context keeps improving performance without plateauing.","pith_inferences":["The paper's spatial averaging of video patch tokens is identified by the authors themselves as a likely cost in retinotopic visual areas; a version that keeps patch positions should specifically improve low-level visual parcels, a test the paper's data makes possible.","The per-parcel softmax ensemble over 1,000 models suggests that different parcels may benefit from different hyperparameters; a single model with per-parcel readout heads might capture part of that benefit at a fraction of the training cost.","Because the frozen feature extractors are never fine-tuned, the approach predicts that further gains are available by adapting those representations to the brain-encoding objective, at the risk of overfitting the four available subjects.","The same architecture should transfer to other naturalistic recording settings, such as audiobooks or virtual-reality navigation, where text, audio, and video streams can be time-aligned in the same way."],"forward_implications":["Whole-brain encoding becomes a single-model problem: the same nonlinear, multimodal network can serve multiple subjects and all cortical parcels, removing the need to fit per-subject, per-modality ridge regressions.","Multisensory integration is where multimodality pays off: the largest gains are in association cortices, so encoding models aimed at those areas should not be built from a single modality.","Primary sensory areas still prefer their own modality, so unimodal and multimodal models are complementary rather than interchangeable.","Recording more data and giving the language model a longer context both improve predictions on a trend that has not plateaued, which predicts further gains from larger datasets and longer contexts.","Because modality dropout lets the model run with missing inputs, the same pipeline can be applied to silent film, audio-only, or text-only stimuli without retraining."],"supporting_citations":[{"why":"Defines the challenge's data split, evaluation metric, and held-out movies that the reported scores are measured on.","marker":"[37]"},{"why":"Provides the naturalistic movie-watching fMRI recordings that the model is trained on.","marker":"[36]"},{"why":"Supplies the frozen language model whose contextualized word embeddings are one of the three input modalities.","marker":"[6]"},{"why":"Supplies the frozen speech model whose hidden representations provide the audio input stream.","marker":"[9]"},{"why":"Supplies the frozen video model whose temporally averaged activations provide the video input stream.","marker":"[11]"},{"why":"Introduces the subject-conditional linear layer used to map transformer outputs onto each participant's parcels.","marker":"[42]"},{"why":"Provides the noise-ceiling formula used to compute the normalized Pearson correlation reported as 54% of explainable variance.","marker":"[47]"},{"why":"Defines the 1,000 cortical parcels that serve as the prediction targets.","marker":"[40]"},{"why":"Supports the interpretation that encoding performance keeps improving with more training recordings.","marker":"[48]"}],"fun_headline_variants":["TRIBE fuses text, audio, video to map brain responses","Multimodal transformer beats unimodal in associative cortices","Whole-brain fMRI prediction with triple-modality transformer","First deep brain encoder across modalities wins Algonauts 2025","Associative cortex encoding boosted by multimodal fusion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quantitative results, including the winning margin and the 54% of explainable variance, depend on the competition's held-out test scores being a faithful measure of generalization, while the model's early stopping, hyperparameter sampling, and per-parcel ensemble weights are all chosen on the validation set.","fun_headline_variants_meta":{"raw":{"variants":["TRIBE fuses text, audio, video to map brain responses","Multimodal transformer beats unimodal in associative cortices","Whole-brain fMRI prediction with triple-modality transformer","First deep brain encoder across modalities wins Algonauts 2025","Associative cortex encoding boosted by multimodal fusion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000822,"raw_usage":{"total_tokens":3576,"prompt_tokens":903,"completion_tokens":2673,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":2592}},"tokens_in":519,"tokens_out":2673,"duration_ms":24757,"temperature":1.0,"reasoning_tokens":2592,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:56:15.362857+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the pipeline with the ensemble weights and early-stopping decisions fixed on a split that is never used during development, and evaluate on a fresh held-out set; if the mean test correlation falls below the runner-up's score, the reported margin is an artifact of validation-based selection. Separately, restrict the noise-ceiling estimate to recording sessions that are disjoint from the sessions used to compute the model's correlations; a normalized Pearson well below 0.54 would show that the explainable-variance claim is inflated by session overlap.","supporting_citations":[{"cited_title":"cneuromod-things: a large-scale fmri dataset for task-and data-driven assessment of object representation and visual memory recognition in the human brain","cited_arxiv_id":null,"evidence_quote":"Provides the naturalistic movie-watching fMRI recordings that the model is trained on."},{"cited_title":"W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training","cited_arxiv_id":null,"evidence_quote":"Supplies the frozen speech model whose hidden representations provide the audio input stream."},{"cited_title":"Decoding speech perception from non-invasive brain recordings","cited_arxiv_id":null,"evidence_quote":"Introduces the subject-conditional linear layer used to map transformer outputs onto each participant's parcels."},{"cited_title":"Harper, Ben D","cited_arxiv_id":null,"evidence_quote":"Provides the noise-ceiling formula used to compute the normalized Pearson correlation reported as 54% of explainable variance."},{"cited_title":"Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity mri","cited_arxiv_id":null,"evidence_quote":"Defines the 1,000 cortical parcels that serve as the prediction targets."},{"cited_title":"Scaling laws for language encoding models in fmri","cited_arxiv_id":null,"evidence_quote":"Supports the interpretation that encoding performance keeps improving with more training recordings."}],"review_version":1}