REVIEW 1 major objections 4 minor
This paper claims that whether a brain-encoding model's predicted responses beat their own visual backbone for video memorability has no dataset-independent answer: the backbone wins on Memento10k, the brain projection wins on VideoMem, and
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 07:04 UTC pith:ZUDGWWMS
load-bearing objection Useful, careful paper with a real result, but the matched-control claim is under-specified: TRIBE v2 is trimodal and the paper never says what input it got. the 1 major comments →
Predicted Cortex Is Not a Domain-General Prior: A Matched-Control Audit of Brain-Encoding Features for Video Memorability
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Within Memento10k (499 clips), the V-JEPA2 backbone predicts short-term memorability better than TRIBE's predicted cortical response (rank correlation 0.594 vs 0.544, bootstrap delta -0.050). Within VideoMem (820 clips), the brain projection wins (0.415 vs 0.368, delta +0.047, 95% CI [+0.009, +0.088]). Both within-dataset gaps have bootstrap intervals excluding zero, in opposite directions. Cross-dataset transfer mirrors this: trained on Memento10k and tested on VideoMem, the brain projection beats the backbone (+0.076, CI [+0.019, +0.135]); trained on VideoMem and tested on Memento10k, it loses heavily (-0.311). The VideoMem advantage survives matched training size, a PCA-then-ridge pipelin
What carries the argument
The matched-control protocol is the central mechanism: each video clip passes through frozen TRIBE v2, which exposes two representations of the same stimulus—the V-JEPA2 visual backbone embedding (1024-dimensional, taken before the brain projection) and the predicted average-subject cortical response (20484-dimensional fsaverage5 vertices, reduced over time to per-vertex mean). Both feed the same standardized ridge-regression probe, and their memorability predictions are compared with paired bootstrap over clips. A vision-orthogonality analysis residualizes both predictions on the vision prediction and correlates the residual with memorability under a permutation null, which isolates what th
Load-bearing premise
The load-bearing premise is that the V-JEPA2 visual backbone is a matched control for the brain projection, meaning the backbone and the projection are fed exactly the same input; if TRIBE's audio or text encoders contributed to the predicted cortical responses without a corresponding backbone input, the VideoMem advantage could come from non-visual modalities rather than from brain-like projection.
What would settle it
Re-run the same protocol with TRIBE restricted to video input only (disabling audio and text encoders) and check whether the brain projection still beats the backbone on VideoMem and still shows a vision-orthogonal component; if the advantage disappears or shrinks to zero, the paper's claim that the brain projection itself carries the signal is falsified. A second decisive test: apply the same matched-control comparison to a third memorability dataset with intermediate clip length and content distribution—if the winner does not flip monotonically with clip length or content similarity, the dat
If this is right
- If the paper is correct, predicted-brain features are not a general upgrade for behavior tasks; their utility depends on the target dataset, and a single cross-dataset win cannot justify calling them a neural contribution.
- Cross-dataset transfer results should always be reported in both directions and compared against the model's own pre-projection backbone, because a representation that merely fits one dataset better will produce a one-directional transfer win that looks like generalization.
- The vision-orthogonal memorability signal, though small, is locally consistent with the known ventral-temporal memory network, suggesting that a model trained only to predict fMRI can recover a behaviorally relevant neural substrate without a scanner.
- Predicted BOLD dynamics at native clip lengths cannot resolve sub-second memorability effects, so time-averaged predicted responses are sufficient for this task and finer temporal statistics add variance rather than signal.
- Fusing vision and brain features improves memorability forecasting beyond vision alone (0.594 to 0.609 on Memento10k), so the brain projection contributes complementary signal even where it loses as a standalone feature set.
Where Pith is reading between the lines
- If the dissociation is driven by clip length or content distribution, then testing on additional benchmarks with intermediate properties should reveal a graded shift in which representation wins—a testable extension of the paper's two-point pattern.
- The paper does not report whether TRIBE's audio and text encoders were fed the clips; if they were, the brain projection's VideoMem advantage and its 'vision-orthogonal' component could be partly driven by non-visual modalities rather than by brain-like projection. An ablation feeding only video would settle this.
- A practical consequence the author leaves implicit: practitioners should evaluate brain-encoding features separately for each target domain and distribution, rather than assuming they transfer, and should always include the backbone as a baseline to avoid over-attributing gains to the 'brain' framing.
- The ventral-temporal localization is weak (0.179 vs 0.171 rank correlation) and the signal is significant across all ROIs, so the paper's characterization of a 'recovered network' is suggestive; per-subject brain-encoding models or a larger ROI set could sharpen or overturn this bias.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper asks whether predicted cortical responses from the brain-encoding foundation model TRIBE v2, used as features without a scanner, beat the model's own V-JEPA2 visual backbone for video memorability prediction. On two benchmarks the answer flips: within Memento10k the backbone wins (SRCC 0.594 vs 0.544), within VideoMem the brain projection wins (0.415 vs 0.368), with bootstrap intervals excluding zero in opposite directions. Cross-dataset transfer inherits this within-dataset affinity rather than transcending it. The paper reports controls for sample size, PCA dimensionality, backbone compression, and transfer-tuned regularization, and adds analyses of a vision-orthogonal component, its ventral-temporal localization, and a temporal-resolution limit.
Significance. The contribution is a clean matched-control protocol and a falsifiable double-dissociation result. Using the model's own pre-projection backbone as the control is a methodological step other work in this area should copy, and the paper is careful with bootstrap confidence intervals, seed stability, and honest limitations. The released code and predicted-response arrays improve reproducibility. However, the central matched-control claim is compromised by an underspecified input modality: TRIBE v2 is trimodal, and the paper does not state whether audio or text was fed to the model. If they were, the VideoMem advantage and the vision-orthogonal component could be non-visual artifacts rather than properties of the predicted cortical representation. Because the separation of the brain projection from its backbone carries the whole weight of the paper, this gap must be closed before the central claim can be accepted as stated.
major comments (1)
- [§1, §3.1, Figure 1] The matched-control assumption is load-bearing and under-specified. TRIBE v2 fuses V-JEPA2 video, Wav2Vec-BERT audio, and LLaMA-3.2 text (§1), but §3.1 says only that 'each clip' is fed through TRIBE and that the V-JEPA2 backbone is 'of the same clip'. The paper never states whether the clip's audio track or any text input was passed to TRIBE. If audio or text were used, the brain projection has strictly more input information than the 1024-d visual backbone, and the VideoMem delta (+0.047, CI [+0.009, +0.088]) and the partial Spearman 0.19 in §3.4 could be driven by non-visual memorability cues. Please specify exactly which modalities were fed to TRIBE in §3.1 and Figure 1. If audio/text were fed, rerun the core comparisons with a vision-only TRIBE variant or otherwise ablate them; if they were not, state that explicitly and add the caveat to §4.2. Without this, the phrase 'matched cont
minor comments (4)
- [§3.2, Table 1] The phrase 'probability that brain is not better = 0.98' is non-standard; please report one-sided p-values or bootstrap percentile intervals. Also, the Memento10k CI upper bound is printed as '-0.000'; show more decimal places so the reader can see whether the interval excludes zero.
- [§3.2–§3.3] Internal section references are off: 'Section 2' in §3.2 should be 'Section 3.3', and 'Section 1' in §3.3 should be 'Section 3.2'.
- [§3.5] The per-ROI analysis runs 148 tests and the ventral-bias Mann-Whitney p=0.016 is uncorrected for the two-group comparison and spatial autocorrelation. Given the paper's own caveat that the bias is 'suggestive', I recommend a split-half or permutation-based summary rather than a single uncorrected p-value.
- [§4.2] The limitations list should explicitly include the modality-matching question raised in the major comment; 'One task, one backbone' does not cover it.
Circularity Check
No circularity: the double dissociation is an empirical benchmark against a held-out backbone, not a derivation from fitted inputs.
full rationale
The paper's central claim is an empirical comparison, not a derivation. TRIBE v2's predicted cortical responses and the V-JEPA2 backbone embeddings are both fixed, public feature extractors; memorability scores come from external benchmarks (Memento10k, VideoMem); ridge regression is trained and evaluated on held-out folds or cross-dataset splits. No parameter is fitted to a subset and then reported as a prediction of a closely related quantity. The vision-orthogonal component is a partial Spearman correlation residualized on the vision prediction with a permutation null, which is a standard control rather than a circular reuse of the fitted values. The temporal analysis is a negative null result, explicitly measured rather than assumed. The paper contains no self-citations by the author and invokes no uniqueness theorem; citations to TRIBE, V-JEPA2, and the neural-basis literature are external support. The only notable weakness is an underspecified matched-control detail: TRIBE v2 is multimodal, and the paper never states whether audio/text encoders were fed the clips, so the brain projection may have access to non-visual input the backbone lacks. That is a validity concern about the comparison, not a circularity: the asserted brain-vs-backbone difference is still an empirical outcome, not an identity by construction. Hence no circular step meets the evidentiary bar, and a score of 0 is appropriate.
Axiom & Free-Parameter Ledger
free parameters (2)
- Ridge regularization alpha =
LOO-selected per probe (not reported numerically)
- PCA dimensionality in robustness check =
100
axioms (5)
- domain assumption TRIBE v2's predicted cortical responses are a usable proxy for brain responses to video stimuli.
- ad hoc to paper The V-JEPA2 embedding taken before the brain projection is a matched control.
- domain assumption Memento10k and VideoMem memorability scores are valid ground truth.
- domain assumption A linear ridge readout with LOO-selected alpha is a fair probe for both feature sets.
- standard math Bootstrap resampling over clips yields valid confidence intervals.
read the original abstract
Brain-encoding foundation models predict fMRI responses to video, audio and text well enough to win the Algonauts 2025 challenge. We ask whether their predicted responses, obtained with no scanner, are a useful feature lens for a human-behavior task: forecasting short-video memorability. Each clip is projected into TRIBE v2's predicted cortical space and scored by ridge regression against a matched control, the model's own V-JEPA2 visual backbone taken before the brain projection. The answer is dataset-dependent. Within Memento10k (499 clips) the backbone wins (Spearman 0.594 vs 0.544); within VideoMem (820 clips) the brain projection wins (0.415 vs 0.368). Because the claim is that the ordering reverses, we test the reversal itself: the dataset-by-representation interaction is +0.097, 95% CI [+0.032, +0.160], two-sided bootstrap p=0.001, and over 10 cross-validation seeds the datasets separate completely (0/10 seeds favor the brain projection on Memento10k, 10/10 on VideoMem). Cross-dataset transfer inherits the split: Memento10k->VideoMem the brain projection wins (+0.076); the reverse loses heavily (-0.311). The VideoMem advantage is not a sample-size artifact (it survives matched training size and PCA-then-ridge) and not mere compression (a compressed, heavily regularized or transfer-tuned backbone stays below it). Predicted-brain features thus carry a small but real memorability signal the backbone misses on one dataset and not the other: a dataset-specific representation, not a domain-general prior. A vision-orthogonal component (partial Spearman 0.19, permutation p=2.5e-4) localizes to ventral occipito-temporal cortex, and predicted BOLD dynamics add nothing beyond the time-average because 3-4 samples per clip cannot resolve the sub-second late memorability response. Our pre-specified within-dataset hypothesis returned NO-GO; the reversal is what survived.
Figures
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.