Pith. sign in

REVIEW 2 major objections 4 minor 28 references

Predicted brain responses are dataset-specific features for video memorability, not a domain-general prior: on one benchmark they beat their own visual backbone, on another they lose.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-04 04:19 UTC pith:ZUDGWWMS

load-bearing objection A genuinely careful and honest matched-control audit whose central claim is undermined by a missing modality control: TRIBE's predicted cortex sees audio, the V-JEPA2 backbone does not, and the paper never says audio was stripped. the 2 major comments →

arxiv 2607.16292 v4 pith:ZUDGWWMS submitted 2026-07-12 cs.CV cs.AIcs.LG

Predicted Cortex Is Not a Domain-General Prior: A Matched-Control Audit of Brain-Encoding Features for Video Memorability

classification cs.CV cs.AIcs.LG
keywords video memorabilitybrain-encoding foundation modelspredicted fMRImatched controldataset-by-representation interactionTRIBEV-JEPA2cross-dataset transfer
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether a brain-encoding foundation model's predicted cortical responses, produced with no scanner, carry useful signal for a human-behavior task—forecasting short-video memorability. Using a matched control, the model's own visual backbone before the brain projection, it finds the answer depends on the dataset: the backbone wins on Memento10k (Spearman 0.594 vs 0.544) and the brain projection wins on VideoMem (0.415 vs 0.368). The reversal is tested directly as a dataset-by-representation interaction (+0.097, 95% CI [+0.032, +0.160], p=0.001) and holds across all 10 cross-validation seeds. Cross-dataset transfer inherits the split, so a one-directional transfer win would have looked like generalization. The paper concludes predicted-brain features are a dataset-specific representation, not a domain-general prior.

Core claim

On the paper's own terms, the finding is that the better memorability forecaster between a brain-encoding model's predicted cortical response and the same model's pre-projection visual backbone flips depending on the dataset. Within Memento10k the backbone wins; within VideoMem the predicted brain projection wins. Because the claim is that the ordering reverses, the paper tests the reversal as an interaction rather than as two separate comparisons: +0.097 with 95% bootstrap CI [+0.032, +0.160] and p=0.001, with opposite signs in 96.6% of replicates. Ten cross-validation seeds separate the datasets completely. The VideoMem advantage survives matched training size, a PCA-then-ridge pipeline, a

What carries the argument

The central object is the matched-control protocol: for each video clip, the paper extracts two feature sets from the same frozen brain-encoding model—the predicted average-subject cortical response (a ~20k-vertex surface array reduced to per-vertex mean) and the model's own visual backbone embedding (1024-d) taken before the brain projection. Both feed an identical StandardScaler + RidgeCV probe, so any difference in memorability forecasting must come from the brain projection itself. The decisive test is the dataset-by-representation interaction: bootstrapping each dataset's clips independently and comparing the two within-dataset deltas, which directly asks whether the ordering reverses r

Load-bearing premise

The reversal claim assumes that the two benchmarks' Spearman deltas are commensurable despite different clip lengths, score distributions, annotation protocols, and sample sizes; if those metadata differences drive the interaction, the reversal reflects benchmark properties rather than representation quality.

What would settle it

A concrete test: re-run the interaction after matching the two datasets' score distributions (e.g., quantile-normalizing each benchmark's labels) and, separately, after downsampling VideoMem clips to 3 seconds. If the interaction's CI then includes zero, the reversal is an artifact of benchmark metadata rather than a property of the representations. Alternatively, on a third dataset with intermediate clip length, the ordering should fall consistently on one side if the effect is representation-driven.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Cross-dataset gains from predicted-brain features should not be read as evidence of a general neural prior; a matched-control comparison against the model's own backbone is needed to distinguish generalization from dataset-specific fit.
  • On datasets where the brain projection wins (VideoMem), the projection carries memorability signal the backbone misses, and simple compression or regularization of the backbone does not recover it—so predicted cortex can add value beyond vision features in some regimes.
  • The vision-orthogonal component's concentration in ventral occipito-temporal cortex means a model trained only to predict fMRI, run without a scanner, reconstructs a known piece of the brain's memorability geometry—opening the door to in-silico localization of behavior-relevant circuits.
  • The temporal result implies that using predicted BOLD dynamics from short clips is pointless for sub-second behavioral effects; studies with longer clips or higher temporal resolution would be needed to test for dynamic memorability signals.
  • The interaction test itself is a template: any claim that a representation beats a control should be tested as a dataset-by-representation interaction rather than as separate wins.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the dataset-dependence is driven by clip length or content distribution, then on a third benchmark with, say, 10-second clips the brain projection should win even more decisively on VideoMem-like distributions; this is testable with the released code.
  • The matched-control logic could be extended to other brain-encoding models and other behavior tasks (e.g., aesthetic judgment or perceived interestingness) to map where predicted cortex adds signal beyond generic vision.
  • The paper's pilot subset of Memento10k (499 clips) leaves open the possibility that on the full 10k set the within-dataset margin changes; scaling the audit is a direct next step.
  • Because pooling the two datasets hides the dissociation, future benchmark evaluations of brain-derived features should report per-dataset results rather than a pooled correlation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper asks whether predicted cortical responses from TRIBE v2, a brain-encoding foundation model, are useful features for video memorability prediction beyond the model's own visual backbone. Using ridge regression on two benchmarks, Memento10k and VideoMem, the authors report a dataset-dependent reversal: the V-JEPA2 backbone wins on Memento10k (SRCC 0.594 vs 0.544), while the brain projection wins on VideoMem (0.415 vs 0.368). They test the reversal directly as a dataset-by-representation interaction (+0.097, 95% CI [+0.032, +0.160], bootstrap p=0.001) and find complete separation over 10 cross-validation seeds. Cross-dataset transfer inherits this affinity: brain wins on Memento10k→VideoMem (+0.076) and loses on VideoMem→Memento10k (-0.311). The paper also reports a vision-orthogonal component (partial Spearman 0.19) with a weak ventral-temporal localization, and a null temporal-dynamics result. The authors are transparent that the pre-specified within-dataset hypothesis failed (NO-GO) and the reversal emerged post hoc.

Significance. If the central claim holds, the paper provides an important methodological caution: predicted brain features are not a domain-general prior for behavior tasks, and matched-control comparisons against the model's own backbone are essential. The study's strengths are its direct interaction test rather than separate comparisons, the use of complete CV-seed separation, the robustness checks (matched size, PCA, transfer-tuned backbone), and the release of code and derived arrays. The paper honestly reports a failed pre-registered-style hypothesis and subsequent exploratory reversal. However, the central attribution of the VideoMem advantage to the cortical projection is currently undermined by a missing modality-matched control: TRIBE v2 is trimodal, and the V-JEPA2 backbone is visual-only. If audio was not stripped, the brain projection's advantage could be an audio effect, not a brain projection effect. This must be resolved before the paper's conclusion can be accepted.

major comments (2)
  1. [§2.4, §3.1, Fig. 1] The decisive control is the V-JEPA2 visual backbone, but TRIBE v2 is trimodal (§2.4): it maps video, audio, and text through V-JEPA2, Wav2Vec-BERT, and LLaMA-3.2. The paper never states whether the audio track was stripped before feeding clips to TRIBE, and no non-brain audio or audiovisual control is included. If audio is present, the predicted cortical response can contain audio-derived information (via Wav2Vec-BERT) that the visual backbone cannot access. The VideoMem advantage (+0.047 within, +0.076 transfer, and the vision-orthogonal component) could then be an audio effect rather than a cortical-projection effect. This is load-bearing because the central claim—that predicted-brain features carry memorability signal the backbone misses—collapses if the signal is already present in another input modality. The authors must either document that audio was removed/not used, or add a Wav2
  2. [§3.2, §3.3] The interaction test assumes the two benchmarks' Spearman deltas are commensurable, but Memento10k (499 clips, 3 s, one scoring protocol) and VideoMem (820 clips, longer clips, different annotation protocol) differ in clip length, score distributions, and label noise. The paper acknowledges the score-range confound for pooling (§3.3) but not for the interaction itself. A dataset-by-representation interaction could be driven by these metadata differences rather than by a property of the representations. To strengthen the claim, the authors should provide evidence that the reversal tracks representation quality rather than benchmark properties—for example, by subgrouping clips by length or score range, or by adding a third dataset. As written, the reversal is robust statistically, but its interpretation as a representational dissociation is under-supported.
minor comments (4)
  1. [§3.2] The text says 'the transfer results in Section 2 inherit' but the transfer results are in Section 3.3, not Section 2. Similarly, §3.3 begins 'The transfer results follow Section 1'—likely meant Section 3.2. Please correct the cross-references.
  2. [Fig. 1] Figure 1 shows a 'video clip' entering TRIBE v2 but does not indicate whether audio is part of the input. Please add an explicit statement in the figure or caption about which modalities were used, and if audio was present, mark it. This is important given the modality-matched control issue.
  3. [§3.1, Regularization grid] The grid range is written as '10−1–104' and '10 −1–108'; use consistent superscript notation (e.g., 10^{-1}–10^4 and 10^{-1}–10^8) for clarity.
  4. [§3.4] The 'vision-orthogonal' component is computed only on Memento10k. Since the VideoMem advantage is the main positive finding, consider also reporting the analogous partial Spearman on VideoMem to show whether the orthogonal signal exists there as well.

Circularity Check

0 steps flagged

No circularity: the paper is an empirical matched-control audit; load-bearing claims are tested against the backbone, bootstrap interaction, permutation nulls, and external scanner results.

full rationale

The paper is a self-contained empirical comparison, not a derivation. The central claim—that predicted-brain features beat their own V-JEPA2 backbone on VideoMem but not Memento10k—is established by out-of-fold ridge forecasts, a paired bootstrap, a dataset-by-representation interaction test, and cross-validation seed separation; none of these reduce to the inputs by construction. The 'vision-orthogonal component' is indeed defined as a residual after regressing on the vision prediction, so its orthogonality is definitional, but the paper's substantive claim is that this residual correlates with memorability, which is tested by permutation against label-shuffled nulls. The ROI localization is compared against external scanner-derived atlases and prior findings, not against the paper's own assumptions. The temporal analysis reports a null result with a mechanistic resolution argument, not a circular rescue. There are no self-citations to prior uniqueness theorems; TRIBE v2 and V-JEPA2 are external, publicly released models. The paper's disclosure that the original directional hypothesis returned NO-GO and that the reversal was not pre-registered is a limitation about post-hoc analysis, not circularity. The strongest caveat—that TRIBE is trimodal and the V-JEPA2 control is visual-only, so the VideoMem advantage could be audio-driven—is a confound or external-validity threat, not a circular reduction: no equation or fitted parameter is being renamed as a prediction. The derivation chain therefore does not collapse into its own inputs.

Axiom & Free-Parameter Ledger

1 free parameters · 7 axioms · 0 invented entities

The central claim rests mainly on domain assumptions about the validity of predicted brain features, the matched control, and the comparability of the two benchmarks. No new entities are postulated. The only fitted hyperparameter is a standard ridge penalty.

free parameters (1)
  • Ridge penalty alpha = Selected per fold by RidgeCV (LOO) over grid 10^-1 to 10^4; individual values not reported
    The central comparisons use ridge regression, and alpha is fit to the training data by cross-validation. This is a standard hyperparameter, not a parameter chosen to manufacture the reversal, but it is technically fitted.
axioms (7)
  • domain assumption TRIBE v2 predicted cortical responses are a meaningful proxy for brain stimulus responses.
    The entire protocol uses TRIBE's predicted responses as features (§1, §3.1). If the predictions were noise, the comparison would be void; external support comes from the Algonauts 2025 win, but that is not checked here.
  • domain assumption V-JEPA2 backbone embeddings are an appropriate matched control representing generic visual features.
    The paper argues the backbone is the model's own pre-projection visual features (§1). If the backbone were not a fair control (e.g., if dimensionality or optimization differed in a way that disadvantages it), the interpretation would change.
  • domain assumption Memorability scores from Memento10k and VideoMem are ground truth and measure the same construct within each dataset.
    Labels are taken from the two benchmark papers (§3.1). Within-dataset comparisons assume the scores are reliable; the interaction assumes the per-dataset deltas are commensurable across different annotation protocols.
  • domain assumption A linear ridge probe is a fair readout for comparing representations.
    All forecasts use StandardScaler + RidgeCV (§3.1). The paper acknowledges in §4.2 that a nonlinear head might change the gaps, so the claim is scoped to linear readouts.
  • domain assumption TRIBE v2 and V-JEPA2 weights were not trained on Memento10k or VideoMem memorability labels.
    The paper uses frozen public weights (§5) and does not mention any training on these benchmarks. If leakage existed, the 'prediction' would be circular. This is an implicit but load-bearing assumption.
  • domain assumption The Destrieux fsaverage5 atlas correctly maps cortical ROIs for the localization analysis.
    ROI naming and ventral/MTL grouping in §3.5 rely on the standard atlas; if the atlas/parcellation were inappropriate, the localization conclusion would be compromised.
  • standard math Bootstrap and permutation null distributions are valid for the reported confidence intervals and p-values.
    The interaction test, paired-bootstrap CIs, and permutation p-values in §3.2–§3.5 assume standard resampling validity; this is routine but an unproved statistical assumption.

pith-pipeline@v1.3.0-alltime-deepseek · 10015 in / 11263 out tokens · 110289 ms · 2026-08-04T04:19:58.200277+00:00 · methodology

0 comments
read the original abstract

Brain-encoding foundation models predict fMRI responses to video, audio and text well enough to win the Algonauts 2025 challenge. We ask whether their predicted responses, obtained with no scanner, are a useful feature lens for a human-behavior task: forecasting short-video memorability. Each clip is projected into TRIBE v2's predicted cortical space and scored by ridge regression against a matched control, the model's own V-JEPA2 visual backbone taken before the brain projection. The answer is dataset-dependent. Within Memento10k (499 clips) the backbone wins (Spearman 0.594 vs 0.544); within VideoMem (820 clips) the brain projection wins (0.415 vs 0.368). Because the claim is that the ordering reverses, we test the reversal itself: the dataset-by-representation interaction is +0.097, 95% CI [+0.032, +0.160], two-sided bootstrap p=0.001, and over 10 cross-validation seeds the datasets separate completely (0/10 seeds favor the brain projection on Memento10k, 10/10 on VideoMem). Cross-dataset transfer inherits the split: Memento10k->VideoMem the brain projection wins (+0.076); the reverse loses heavily (-0.311). The VideoMem advantage is not a sample-size artifact (it survives matched training size and PCA-then-ridge) and not mere compression (a compressed, heavily regularized or transfer-tuned backbone stays below it). Predicted-brain features thus carry a small but real memorability signal the backbone misses on one dataset and not the other: a dataset-specific representation, not a domain-general prior. A vision-orthogonal component (partial Spearman 0.19, permutation p=2.5e-4) localizes to ventral occipito-temporal cortex, and predicted BOLD dynamics add nothing beyond the time-average because 3-4 samples per clip cannot resolve the sub-second late memorability response. Our pre-specified within-dataset hypothesis returned NO-GO; the reversal is what survived.

Figures

Figures reproduced from arXiv: 2607.16292 by Carson Rodrigues.

Figure 1
Figure 1. Figure 1: Predictions-only protocol. Each clip passes through frozen TRIBE v2, which exposes two [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 1
Figure 1. Figure 1: Predictions-only protocol. Each clip passes through frozen TRIBE v2, which exposes two [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Which representation wins is dataset-dependent, and transfer inherits it. Left: within each dataset, [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 2
Figure 2. Figure 2: Which representation wins is dataset-dependent, and transfer inherits it. Left: within each dataset, [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Where the vision-orthogonal signal lives. Left: per-ROI vision-controlled unique SRCC is weakly [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 3
Figure 3. Figure 3: Where the vision-orthogonal signal lives. Left: per-ROI vision-controlled unique SRCC is weakly [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

28 extracted references · 8 linked inside Pith

  1. [2]

    Bainbridge, Daniel D

    Wilma A. Bainbridge, Daniel D. Dilks, and Aude Oliva. Memorability: A stimulus-driven perceptual neural signature distinctive from memory. NeuroImage, 149: 0 141--152, 2017. doi:10.1016/j.neuroimage.2017.01.063

  2. [3]

    Romain Cohendet, Karthik Yadati, Ngoc Q. K. Duong, and Claire-H \'e l \`e ne Demarty. Annotating, understanding, and predicting long-term video memorability. In Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval (ICMR), pp.\ 178--186. ACM, 2018. doi:10.1145/3206025.3206056

  3. [4]

    Romain Cohendet, Claire-H \'e l \`e ne Demarty, Ngoc Q. K. Duong, and Martin Engilberge. VideoMem : Constructing, analyzing, predicting short-term and long-term video memorability. In IEEE/CVF International Conference on Computer Vision (ICCV), 2019. VideoMem dataset

  4. [5]

    TRIBE : TRImodal brain encoder for whole-brain fMRI response prediction

    St \'e phane d'Ascoli et al. TRIBE : TRImodal brain encoder for whole-brain fMRI response prediction. arXiv preprint arXiv:2507.22229, 2025. Algonauts 2025 winner

  5. [6]

    Andrew Jaegle, Vahid Mehrpour, Yalda Mohsenzadeh, Travis Meyer, Aude Oliva, and Nicole C. Rust. Population response magnitude variation in inferotemporal cortex predicts image memorability. eLife, 8: 0 e47596, 2019. doi:10.7554/eLife.47596

  6. [7]

    Visual perception of highly memorable images is mediated by a distributed network of ventral visual regions that enable a late memorability response

    Benjamin Lahner, Yalda Mohsenzadeh, Caitlin Mullin, and Aude Oliva. Visual perception of highly memorable images is mediated by a distributed network of ventral visual regions that enable a late memorability response. PLOS Biology, 22 0 (4): 0 e3002564, 2024. doi:10.1371/journal.pbio.3002564

  7. [8]

    TRIBE v2 : A predictive foundation model of how the human brain processes complex stimuli, 2026

    Meta FAIR . TRIBE v2 : A predictive foundation model of how the human brain processes complex stimuli, 2026. Model weights and card: https://huggingface.co/facebook/tribev2. Code: https://github.com/facebookresearch/tribev2. Released under CC BY-NC-4.0

  8. [9]

    Multimodal memorability: Modeling effects of semantics and decay on video memorability

    Anelise Newman, Camilo Fosco, Vincent Casser, Allen Lee, Barry McNamara, and Aude Oliva. Multimodal memorability: Modeling effects of semantics and decay on video memorability. In European Conference on Computer Vision (ECCV), 2020. doi:10.1007/978-3-030-58517-4_14. Memento10k dataset; arXiv:2009.02568

  9. [11]

    2025 , eprint =

    d'Ascoli, St. 2025 , eprint =

  10. [12]

    and others , year =

    Scotti, Paul S. and others , year =. Insights from the. 2508.10784 , archivePrefix =

  11. [13]

    2025 , eprint =

    Improving Multimodal Brain Encoding Model with Dynamic Subject-awareness Routing , author =. 2025 , eprint =

  12. [14]

    Feature Visualization Recovers Known Cortical Selectivity from

    Bladon, Stuart and others , year =. Feature Visualization Recovers Known Cortical Selectivity from. arXiv preprint arXiv:2605.13904 , note =. 2605.13904 , archivePrefix =

  13. [15]

    2026 , eprint =

    Mechanistically Interpretable Neural Encoding Reveals Fine-Grained Functional Selectivity in Human Visual Cortex , author =. 2026 , eprint =

  14. [16]

    Nature Neuroscience , volume =

    Functional Connectome Fingerprinting: Identifying Individuals Using Patterns of Brain Connectivity , author =. Nature Neuroscience , volume =. 2015 , doi =

  15. [17]

    Individual Identifiability Following

    Sasse, Leonard and others , journal =. Individual Identifiability Following. 2026 , doi =

  16. [18]

    A Low-Variance Subspace Underlies Individual Differences in Resting-State

    Borovykh, Anastasia and Weissenbacher, Max and Noble, Stephanie and Shinn, Maxwell , year =. A Low-Variance Subspace Underlies Individual Differences in Resting-State. doi:10.64898/2026.01.25.701594 , note =

  17. [19]

    A Shared Neural Encoding Model for the Prediction of Subject-Specific

    Khosla, Meenakshi and others , year =. A Shared Neural Encoding Model for the Prediction of Subject-Specific. 2006.15802 , archivePrefix =

  18. [20]

    2021 , eprint =

    Deep Recurrent Encoder: A Scalable End-to-End Network to Model Brain Signals , author =. 2021 , eprint =

  19. [21]

    and others , year =

    Scotti, Paul S. and others , year =. 2403.11207 , archivePrefix =

  20. [22]

    2510.27128 , archivePrefix =

    Wang, Haonan and others , year =. 2510.27128 , archivePrefix =

  21. [23]

    2505.15946 , archivePrefix =

    Wei, Yuxiang and others , year =. 2505.15946 , archivePrefix =

  22. [24]

    European Conference on Computer Vision (ECCV) , year =

    Multimodal Memorability: Modeling Effects of Semantics and Decay on Video Memorability , author =. European Conference on Computer Vision (ECCV) , year =. doi:10.1007/978-3-030-58517-4_14 , note =

  23. [25]

    IEEE/CVF International Conference on Computer Vision (ICCV) , year =

    Cohendet, Romain and Demarty, Claire-H. IEEE/CVF International Conference on Computer Vision (ICCV) , year =. 1812.01973 , archivePrefix =

  24. [26]

    arXiv preprint arXiv:2506.09985 , note =

    Assran, Mahmoud and others , year =. arXiv preprint arXiv:2506.09985 , note =. 2506.09985 , archivePrefix =

  25. [27]

    NeuroImage , volume =

    Memorability: A Stimulus-Driven Perceptual Neural Signature Distinctive from Memory , author =. NeuroImage , volume =. 2017 , doi =

  26. [28]

    eLife , volume =

    Population Response Magnitude Variation in Inferotemporal Cortex Predicts Image Memorability , author =. eLife , volume =. 2019 , doi =

  27. [29]

    PLOS Biology , volume =

    Visual Perception of Highly Memorable Images is Mediated by a Distributed Network of Ventral Visual Regions that Enable a Late Memorability Response , author =. PLOS Biology , volume =. 2024 , doi =

  28. [30]

    Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval (ICMR) , pages =

    Annotating, Understanding, and Predicting Long-term Video Memorability , author =. Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval (ICMR) , pages =. 2018 , publisher =