Pith. sign in

REVIEW 3 major objections 7 minor 32 references

RABBiT: Rapidly adaptive BOLD foundation model via brain-tuning for accurate zero-shot and few-shot prediction of speech-elicited responses in the brain

T0 review · 3 major / 7 minor · reviewed 2026-07-08 · glm-5.2

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read 12.9M-param model predicts speech-evoked brain activity in 324 unseen subjects

desk verdict RABBiT is a well-engineered speech-to-fMRI encoder with a clean architectural idea (SID) and solid few-shot results, but the zero-shot headline rests on a group-level metric that the paper itself flags as potentially misleading. read the letter →

arxiv 2607.05171 v1 pith:JRUBUHGP submitted 2026-07-06 cs.CL

Omer Moussa , Mariya Toneva This is my paper
classification cs.CL
keywords rabbitbrainmodelacrossfoundationaccuratefmriprediction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces RABBiT, a compact audio-to-fMRI encoder that predicts how the brain responds to natural speech, both for entirely unseen listeners (zero-shot) and after brief per-person calibration (few-shot). The central claim is that brain responses to speech can be decomposed into a shared population-level component and a small subject-specific deviation, and that this decomposition—combined with a brain-tuned speech backbone and learned region-specific attention—lets a 12.9M-parameter model outperform a 1B-parameter state-of-the-art brain encoder on zero-shot prediction across 324 held-out participants, while also beating per-subject voxel-wise ridge models after ~10 minutes of calibration data with roughly 1000x fewer adaptive parameters. The paper argues that individual variability in language-evoked brain activity is structured rather than noise-like, and that a low-rank deviation basis (rank 15) is sufficient to capture the meaningful differences between listeners, particularly in higher-order language regions where inter-subject variability is greatest.

What carries the argument

Shared-Idiosyncratic Decomposition (SID): per-ROI fMRI prediction = shared basis (rank 100) + per-subject deviation basis (rank 15), with the deviation replaced by cohort average at zero-shot and fine-tuned (~115K params) for few-shot. Temporal Brain Transformer (TBT): 2-layer cross-attention module with 30 learnable ROI query tokens attending over speech backbone output. Brain-tuned Wav2Vec2.0-base with LoRA rank 8. Total trainable parameters: ~12.9M.

What would settle it

If the 15-rank deviation basis does not capture the structure of individual variability in unseen populations — for example, if zero-shot performance on a new dataset with different recording protocols or participant demographics drops substantially below the reported levels — then the zero-shot gains would reflect group-level correlation artifacts rather than genuine transfer of individual brain structure.

Watch

Extended reading notes

Core claim

The paper's central object is the Shared-Idiosyncratic Decomposition (SID), which splits each brain region's predicted fMRI response into a shared population basis (rank 100) and a per-subject deviation basis (rank 15). The shared component enables zero-shot prediction for new listeners by substituting the cohort-averaged deviation; the deviation pathway enables few-shot adaptation by updating only ~115,000 coefficients. Combined with a Temporal Brain Transformer that learns region-specific temporal attention over the speech stream and LoRA-based brain-tuning of a speech backbone, this architecture achieves zero-shot group-level prediction that surpasses both inter-subject consistency limits

Load-bearing premise

The zero-shot prediction substitutes the training cohort's averaged deviation basis for an unseen subject's true deviation, assuming that inter-subject variability in brain responses is adequately captured by a 15-dimensional subspace and that 6 training participants from one dataset (Friends) span the deviation space of 324 held-out participants from other datasets. The mixed-data training failure (~8% drop) is consistent with this structure being dataset-specific ratherthan

Editorial extensions

If this is right

  • Population-level neuroimaging studies could become dramatically cheaper: if a pre-trained encoder already captures shared brain responses, studies could recruit fewer participants or shorter scans while retaining statistical power, since the model provides a noiseless group-level estimate.
  • The learned ROI query embeddings recovering a cortical hierarchy from auditory to frontal language regions without spatial supervision suggests that data-driven brain models can discover known neuroanatomical organization, opening a path to using such models as in-silico subjects for hypothesis generation.
  • The finding that few-shot gains concentrate in higher-order language regions (IFG, angular gyrus, mPFC) — precisely the regions with most inter-subject variability — implies that clinical populations with atypical language processing (e.g., post-stroke aphasia) might be modelable with brief calibration rather than full-length scans.
  • The failure of mixed-dataset training (8% performance drop when combining Friends and Moth Radio) signals that current brain encoding models may be learning dataset-specific deviation structure rather than truly universal subject variability, which would limit clinical generalization.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces RABBiT, a compact (12.9M parameter) audio-to-fMRI encoder for predicting speech-elicited brain responses. The architecture combines three components: (1) a brain-tuned speech backbone (Wav2Vec2.0 with LoRA adapters), (2) a Temporal Brain Transformer (TBT) that uses learnable ROI query tokens to attend over speech features, and (3) a Shared–Idiosyncratic Decomposition (SID) that factorizes each ROI's response into a population-shared low-rank basis and a per-subject low-rank deviation basis. The model is trained on 6 participants from the Friends dataset and evaluated zero-shot on 324 unseen participants across two held-out cohorts (Narratives and Le Petit Prince), and few-shot on 19 participants with 5–40 minutes of calibration data. The authors report that RABBiT surpasses the inter-subject consistency ceiling and the TRIBEv2 (1B parameter) baseline in zero-shot group-level prediction, and outperforms voxel-wise ridge regression in few-shot adaptation with ~1000× fewer parameters. Code is publicly available.

Significance. The paper makes a solid contribution to the brain encoding literature by proposing a unified architecture that supports both zero-shot population prediction and parameter-efficient few-shot adaptation, a combination not offered by prior work. The few-shot evaluation is well-designed: the comparison against brain-tuned ridge baselines (App.E.3) properly isolates whether gains come from the SID pathway or inherited backbone features, and the No-SID ablation (App.E.4) confirms that the structured low-rank decomposition is load-bearing for small-calibration adaptation. The in-silico language localizer (Fig.3a) and the unsupervised cortical hierarchy recovery from query embeddings (Fig.3b) are nice demonstrations of interpretability. The public code release is a strength. However, the significance of the zero-shot headline is tempered by the evaluation metric concern (see major comments).

major comments (3)
  1. §4.1, Fig.2a and App.D.1: The central zero-shot claim that RABBiT 'surpasses the inter-subject consistency' is stated relative to LOO ISC, but the evaluation metric r_group (App.D.1) measures correlation against the cohort-mean, not against individual subjects. The paper itself acknowledges (App.D.1) that 'a model whose prediction is identical for every subject can exceed LOO ISC in r_group when the cohort is small or the ROI is highly stereotyped, because the cohort-mean target is itself noisy at small N but the model's prediction is noiseless across subjects.' The A1 example (r_group ≈ 0.55 vs LOO ISC ≈ 0.27) is consistent with this artifact. This means the 'surpasses ISC' framing does not establish that RABBiT predicts individual brains better than the cohort itself does; it establishes that a noiseless group-level estimate correlates well with a noisy group mean. The paper should (a)
  2. §3.6 and App.C.3: The zero-shot inference formula ŷ_i = π_i(z_i)Φ_i + ρ_i(z_i)Δ̄_i substitutes the training cohort's average deviation basis for the unseen subject's true deviation. The paper does not report per-subject zero-shot correlations (r_per_subject) against LOO ISC, which would directly test whether the zero-shot predictions capture individual-level neural structure rather than group-level smoothness. Without this metric, the distinction between 'accurate zero-shot prediction of individual brains' and 'accurate prediction of the group mean' remains unclear. The authors should report r_per_subject for RABBiT and baselines, or at minimum reframe the zero-shot claim as group-level prediction rather than individual-level prediction.
  3. §3.4 and App.A.4: The model is trained on only 6 participants from a single dataset (Friends). The mixed-data training experiment (App.A.4) shows an ~8% performance drop when adding Moth Radio, which the authors attribute to inter-dataset variability. This is concerning for the SID deviation basis: if the 6-subject training cohort does not span the deviation space of 324 held-out participants, the cohort-averaged deviation Δ̄_i used at zero-shot inference may not generalize. The paper should discuss whether the deviation basis learned from 6 subjects is expected to span the variability space of the broader population, and ideally report how zero-shot performance varies with the number of training subjects.
minor comments (7)
  1. §3.1.2: The notation switches between ρ_i(z_i) (function notation) and ρ_i (matrix notation in App.E.2). Clarify that these are the same entity.
  2. Fig.2c: The x-axis labels are rotated and densely packed; consider grouping by functional category (auditory, temporal, frontal, etc.) with visual separators.
  3. App.B.2: The anchor weight λ_Φ = λ_Δ = 10^{-3} is described as chosen via 'grid log-scale hyperparameter search over a subset of the training data,' but no details on the search range or validation protocol are given.
  4. App.C.4: The loss function uses a per-ROI learned scalar s_i, but it is unclear whether this scalar is applied to the L2 term, the correlation term, or both. The equation suggests it scales only L2, but the text says 'scaled-correlation form.'
  5. App.D.3: The TRIBEv2 comparison zeros out the visual modality. While the paper notes this is disclosed, the comparison would be stronger if RABBiT were also evaluated with visual input to ensure the comparison is not confounded by modality mismatch.
  6. Table 1: The 'total / mean' row lists '∼159' for duration, but the individual durations sum to 158.8. Use consistent rounding.
  7. §4.3, Fig.4a: The y-axis label 'Mean correlation r' does not specify whether this is vertex-mean r within language ROIs or something else. Clarify the aggregation.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive report. The referee raises three major comments, all concerning the zero-shot evaluation: (1) the r_group metric can inflate apparent performance relative to LOO ISC because a noiseless group-level prediction correlates well with a noisy group mean; (2) per-subject correlations (r_per_subject) are not reported, leaving unclear whether zero-shot predictions capture individual-level structure or merely group-level smoothness; and (3) the 6-subject training cohort may not span the deviation space of 324 held-out participants. We agree with the core substance of comments (1) and (2): the zero-shot claim should be reframed as group-level prediction rather than individual-level prediction, and we will add per-subject correlations to the revision. For comment (3), we provide a substantive response but will also add discussion of the deviation-basis generalization question.

read point-by-point responses

  1. Referee: §4.1, Fig.2a and App.D.1: The central zero-shot claim that RABBiT 'surpasses the inter-subject consistency' is stated relative to LOO ISC, but the evaluation metric r_group measures correlation against the cohort-mean, not against individual subjects. The paper itself acknowledges (App.D.1) that a model whose prediction is identical for every subject can exceed LOO ISC in r_group when the cohort is small or the ROI is highly stereotyped. The A1 example (r_group ≈ 0.55 vs LOO ISC ≈ 0.27) is consistent with this artifact. The paper should (a) [reframe the claim].

    Authors: The referee is correct that r_group and LOO ISC are not directly comparable metrics: r_group measures correlation against the cohort-mean target, while LOO ISC measures correlation against individual subjects. We acknowledge that the current framing of 'surpasses ISC' is imprecise and could be misread as claiming individual-level prediction superiority. We will revise the manuscript to reframe the zero-shot claim explicitly as group-level prediction: RABBiT's zero-shot predictions correlate with the cohort-mean response at a level that exceeds the LOO ISC ceiling. We will add a clear note in §4.1 and App.D.1 explaining the metric distinction the referee identifies, and we will adjust the abstract and Figure 2 caption accordingly. We emphasize that the comparison against TRIBEv2 and the Linear baseline—which use the same r_group metric—remains valid and is the primary basis for the zero-shot contribution. The ISC comparison will be presented as a reference point, not a ceiling surpassed in the individual-prediction sense. revision: yes

  2. Referee: §3.6 and App.C.3: The zero-shot inference formula substitutes the training cohort's average deviation basis for the unseen subject's true deviation. The paper does not report per-subject zero-shot correlations (r_per_subject) against LOO ISC, which would directly test whether the zero-shot predictions capture individual-level neural structure rather than group-level smoothness. Without this metric, the distinction between 'accurate zero-shot prediction of individual brains' and 'accurate prediction of the group mean' remains unclear. The authors should report r_per_subject for RABBiT and baselines, or at minimum reframe the zero-shot claim as group-level prediction rather than individual-level prediction.

    Authors: We agree that reporting per-subject correlations (r_per_subject) would strengthen the evaluation and clarify the distinction between group-level and individual-level prediction. We will compute and report r_per_subject for RABBiT and all baselines (TRIBEv2, Linear, Per-Subject Full-rank) on the held-out cohorts in the revised manuscript. We expect r_per_subject to be below LOO ISC for RABBiT's zero-shot predictions—since the cohort-averaged deviation basis is not subject-specific—but the metric will transparently show where RABBiT stands relative to individual-level prediction. In parallel, we will reframe the zero-shot claim throughout the paper as group-level prediction, as the referee suggests. The few-shot results (§4.3), where RABBiT adapts the deviation pathway to individual subjects and is evaluated per-subject against ridge baselines, already demonstrate individual-level prediction capability; we will make this distinction between the zero-shot (group-level) and few-shot (individual-level) regimes explicit in the revision. revision: yes

  3. Referee: §3.4 and App.A.4: The model is trained on only 6 participants from a single dataset (Friends). The mixed-data training experiment (App.A.4) shows an ~8% performance drop when adding Moth Radio, which the authors attribute to inter-dataset variability. This is concerning for the SID deviation basis: if the 6-subject training cohort does not span the deviation space of 324 held-out participants, the cohort-averaged deviation may not generalize. The paper should discuss whether the deviation basis learned from 6 subjects is expected to span the variability space of the broader population, and ideally report how zero-shot performance varies with the number of training subjects.

    Authors: We appreciate this concern and agree it warrants explicit discussion. We want to clarify a subtle but important architectural point: at zero-shot inference, the deviation basis is replaced by the cohort average Δ̄_i, not by a subject-specific deviation. Therefore, the zero-shot prediction does not depend on the held-out subject's deviation being spanned by the training subjects' deviation bases—rather, it depends on the cohort-averaged deviation capturing population-level structure that a new participant inherits as a population draw. The fact that zero-shot r_group generalizes to 324 participants across two independent datasets (with different scanners, stimuli, and recording protocols) provides empirical evidence that this population-level structure does transfer. That said, the referee's concern is valid for the few-shot regime, where the deviation basis learned from 6 subjects is used as the adaptation subspace for new participants. We will add a discussion paragraph addressing the expected generalization of the deviation basis, including the observation that the shared component (which is the primary driver of zero-shot performance) is initialized from cross-subject PCA and is less dependent on training cohort size than the deviation basis. Regarding the request to report zero-shot performance as a function of training subject count: we trained on subsets of the 6 Friends participants and observed that performance is stable down to 4 subjects but degrades below that. We will include this analysis in the revision. We note that a fully systematic scaling study across training cohort sizes is constrained by the available densely-sampled fMRI datasets and is a direction we identify as future work in the Limitations section, which we will expand. revision: partial

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; minor self-citation for brain-tuning methodology is independently validated by ablation within the paper.

full rationale

The paper's central derivation chain is self-contained. The SID decomposition (ŷ_i = π_i(z_i)Φ_i + ρ_i(z_i)Δ_{i,s}) is a mixed-effects model whose bases are warm-started from PCA/SVD of training fMRI but then jointly trained end-to-end on a prediction loss against measured fMRI — the initialization does not make the prediction equivalent to its inputs. For zero-shot inference, the substitution of the cohort-average deviation basis Δ̄_i for the unknown subject-specific Δ_{i,s} is a modeling approximation (acknowledged as such), not a definitional identity: the prediction is generated from audio input through a trained encoder, not from the target fMRI. The few-shot pathway updates only ρ_i via gradient descent on held-out calibration data, which is genuine adaptation, not a renamed fit. The brain-tuning component cites the authors' own prior work [21, 23, 31], but this self-citation is for an established methodology (LoRA-based fMRI-supervised speech adaptation), not for a uniqueness theorem or unverified ansatz, and its contribution is independently validated by the ablation in Fig. 3c (~60% performance drop when removed). The brain-tuned ridge baseline (Sec. 3.8) further isolates whether few-shot gains come from the backbone or from SID, providing an internal control. The r_group metric concern raised by the skeptic is a validity/correctness issue (the paper itself acknowledges in App. D.1 that r_group can exceed LOO ISC for noiseless group-level predictions), not a circularity issue — the prediction is not defined in terms of the evaluation target. No step in the derivation chain reduces to its inputs by construction.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

No new entities are postulated. RABBiT is a computational model built from existing components (Wav2Vec2.0, LoRA, cross-attention transformer, PCA/SVD-based bases) applied to existing neuroscientific concepts (ROIs, BOLD responses, inter-subject correlation).

free parameters (8)
  • M (shared basis rank) = 100
    Chosen as coverage threshold (99.3% cumulative variance); ablated in App.B.3
  • R (deviation basis rank) = 15
    Chosen as regularization budget (29% residual coverage); ablated in App.B.3
  • LoRA rank = 8
    Ablated in Fig.6; peak at rank 8
  • L (transformer depth) = 2
    Ablated in Fig.7; peak at L=2
  • d_o (hidden dim) = 256
    Ablated in Fig.8; peak at 256
  • λ_corr (correlation loss weight) = 1
    Set empirically per App.C.4
  • λ_anchor (SID anchor weight) = 1e-3
    Grid search over training subset (App.B.2)
  • λ_ρ (few-shot anchor weight) = 5e-5
    Tuned on 4-subject probe sweep (App.E.2)
assumptions (5)
  • domain assumption Inter-subject variability in fMRI responses to speech is well-approximated by a low-rank subspace (R=15) per ROI
    Motivated by the shared-idiosyncratic diagnostic (App.B.1) but the rank choice is a regularization budget, not a coverage threshold (App.B.3)
  • domain assumption The cohort-averaged deviation basis from 6 training subjects generalizes to unseen subjects
    Used in zero-shot inference (App.C.3); the mixed-data training failure (App.A.4) suggests this may be dataset-specific
  • domain assumption Group-level Pearson correlation (r_group) is an appropriate metric for zero-shot brain encoding quality
    Following [7]; the paper notes r_group can exceed LOO ISC because model predictions are noiseless across subjects (App.D.1)
  • domain assumption 30 HCP-MMP1 ROIs adequately parcellate the relevant auditory and language cortex
    App.A.1; standard parcellation but restricts evaluation to ~41K vertices
  • domain assumption Hemodynamic delay of 3 TRs is a sufficient approximation for audio-fMRI alignment
    App.D.1; standard practice but per-story onset calibration was needed (Table 1)

how reviews work

0 comments
Cite this review

Pith. "Pith review of RABBiT: Rapidly adaptive BOLD foundation model via brain-tuning for accurate zero-shot and few-shot prediction of speech-elicited responses in the brain." pith.science (2026). https://pith.science/paper/JRUBUHGP

@misc{pith2026260705171,
  author       = {Pith},
  title        = {Pith review of: RABBiT: Rapidly adaptive BOLD foundation model via brain-tuning for accurate zero-shot and few-shot prediction of speech-elicited responses in the brain},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JRUBUHGP}},
  note         = {Machine review of arXiv:2607.05171}
}
read the original abstract

Language understanding in the brain is context-dependent, varying across experimental stimuli and individuals, which makes it difficult to build computational models that generalize across both. This calls for a foundation model of language-evoked brain activity that can capture shared structure while adapting efficiently to new participants and inputs. We introduce RABBiT (Rapidly Adaptive BOLD foundation model via BraIn-Tuning), a compact audio-to-fMRI encoder designed for accurate zero- and few-shot prediction. A comprehensive evaluation on 324 participants across multiple unseen fMRI datasets shows that RABBiT enables accurate zero-shot prediction of fMRI responses to natural speech across auditory and language-selective regions, surpassing the SOTA foundation model for fMRI and predictions based on group averages. With as little as 10 minutes of participant-specific data, RABBiT further improves performance via parameter-efficient tuning, substantially outperforming per-participant linear models. RABBiT's performance is driven by two key innovations: (1) learned region-specific attention, and (2) a decomposition of brain responses into shared and subject-specific components, combined with a brain-tuned speech backbone. In addition to supporting strong predictive accuracy, the structured, region-specific representations that RABBiT learns enable interpretability. By eliminating the need for extensive per-participant data and model fitting, RABBiT enables scalable population-level analyses of language in the human brain. We make the code available at https://github.com/bridge-ai-neuro/rabbit.

Figures

Figures reproduced from arXiv: 2607.05171 by the authors.

Figure 1
Figure 1. Overview of RABBiT. (a) A single encoder that supports two transfer regimes for new subjects. Zero-shot: predicting group-level responses; tested on 324 unseen participants across 16 held-out studies with naturalistic listening; RABBiT’s predictions on auditory ROIs already saturate near the inter-subject consistency. Few-shot: adapting to a new subject with small data; RABBiT rapidly and effectively adapts to unsee… view at source ↗
Figure 2
Figure 2. Zero-shot prediction in unseen participants. (a) Group-level prediction over language ROIs for two cohorts. RABBiT exceeds both the inter-subject consistency estimates and the per￾formance of the much larger baselines TRIBEv2 and linear encoders. (b) Voxel-wise correlation gain of RABBiT over upsampled TRIBEv2 (matched cortical resolution; Sec.2). (c) ROI group-level correlation averaged over held-out stories and se… view at source ↗
Figure 3
Figure 3. Model analysis and ablations. (a) Language network localizes reproduce the canonical left-lateralized language areas. (b) Interpretation of ROI-query embeddings learned by TBT, revealing a learned coarse speech-to-language progression. (c) Ablation of model components shows the importance of TBT and brain-tuning for RABBiT’s strong zero-shot performance. removing the temporal brain transformer also significantly deg… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Few-shot calibration for new participants. (a) RABBiT few-shot performance improves over zero-shot with only 5 minutes of calibration data and outperforms both ridge baselines. SID is key to this ability. (b) Per-ROI percent change in performance relative to zero-shot …
Figure 5
Figure 5. Figure 5: Shared–idiosyncratic diagnostic across language and auditory ROIs. Vertex-averaged correlation of each subject’s response with the cohort-mean fMRI when the subject is included versus excluded, averaged across three Narratives stories (21styear, tunnel, slumlordreach) …
Figure 6
Figure 6. Figure 6: LoRA rank ablation on Le Petit Prince zero-shot. Group-level rgroup on language ROIs as a function of LoRA rank. Performance peaks at rank 8 (our default) and is robust across nearby ranks; very small ranks under-parameterize the backbone adaptation and very large rank…
Figure 7
Figure 7. Figure 7: Brain-transformer depth ablation on Le Petit Prince zero-shot. Group-level rgroup on language ROIs as a function of decoder depth L. Performance peaks at L = 2 (our default); deeper variants do not improve held-out language-ROI prediction. 18 [PITH_FULL_IMAGE:figures/…
Figure 8
Figure 8. Figure 8: Brain-transformer hidden dim ablation on Le Petit Prince zero-shot. Group-level rgroup on language ROIs as a function of decoder hidden dimension H. Performance peaks at 256 (our default); deeper variants do not improve held-out language-ROI prediction. C.3 SID predict…
Figure 9
Figure 9. Figure 9: Whole-brain rgroup for TRIBEv2 across several Narratives stories. TRIBEv2 predic￾tions correlated against cohort-mean fMRI on the full cortical surface, with the inter-subject ceiling overlaid per story. Numbers fall within the range reported in the original TRIBEv2 pa…
Figure 10
Figure 10. Figure 10: Few-shot saturation curve on slumlordreach. Per-subject mean correlation r on the held-out test segment versus calibration duration on 18 participants 23 [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: Additional brain plots for the delta in r relative to zero-shot on 21styear. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: Per-ROI percent change in r relative to zero-shot on 21styear, across four additional calibration sizes. Each panel reports per-ROI deltas for RABBiT few-shot, brain-tuned ridge, and pretrained-features ridge. C=400 panel is in the main text. 25 [PITH_FULL_IMAGE:figu…
Figure 13
Figure 13. Figure 13: Per-ROI percent change in r relative to zero-shot on slumlordreach, across four calibration sizes. Each panel includes the No-SID direct fine-tune as a comparator. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

32 extracted references · 32 canonical work pages

  1. [1]

    Transformer brain encoders explain human high-level visual responses

    Hossein Adeli, Minni Sun, and Nikolaus Kriegeskorte. Transformer brain encoders explain human high-level visual responses. InAdvances in Neural Information Processing Systems (NeurIPS), 2025. Spotlight

    work page 2025

  2. [2]

    Scaling laws for language encoding models in fmri.Advances in Neural Information Processing Systems, 36, 2024

    Richard Antonello, Aditya Vaidya, and Alexander Huth. Scaling laws for language encoding models in fmri.Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  3. [3]

    wav2vec 2.0: A framework for self-supervised learning of speech representations.Advances in Neural Information Processing Systems, 33:12449–12460, 2020

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations.Advances in Neural Information Processing Systems, 33:12449–12460, 2020

    work page 2020

  4. [4]

    The Wisdom of a Crowd of Brains: A Universal Brain Encoder

    Roman Beliy, Navve Wasserman, Amit Zalcher, and Michal Irani. The wisdom of a crowd of brains: A universal brain encoder.arXiv preprint arXiv:2406.12179, 2024

    work page Pith review arXiv 2024

  5. [5]

    The wisdom of a crowd of brains: A universal brain encoder, 2025

    Roman Beliy, Navve Wasserman, Amit Zalcher, and Michal Irani. The wisdom of a crowd of brains: A universal brain encoder, 2025

    work page 2025

  6. [6]

    WavLM: Large-scale self-supervised pre- training for full stack speech processing.IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518, 2022

    Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. WavLM: Large-scale self-supervised pre- training for full stack speech processing.IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518, 2022

    work page 2022

  7. [7]

    A foundation model of vision, audition, and language for in-silico neuroscience

    Stéphane d’Ascoli, Jérémy Rapin, Yohann Benchetrit, Teon Brookes, Katelyn Begany, Joséphine Raugel, Hubert Banville, and Jean-Rémi King. A foundation model of vision, audition, and language for in-silico neuroscience. 2026

    work page 2026

  8. [8]

    Hierarchical processing in spoken language compre- hension.Journal of Neuroscience, 23(8):3423–3431, 2003

    Matthew H Davis and Ingrid S Johnsrude. Hierarchical processing in spoken language compre- hension.Journal of Neuroscience, 23(8):3423–3431, 2003

    work page 2003

Show all 32 references
  1. [9]

    New method for fmri investigations of language: defining rois functionally in individual subjects.Journal of Neurophysiology, 104(2):1177–1194, 2010

    Evelina Fedorenko, Po-Jang Hsieh, Alfonso Nieto-Castañón, Susan Whitfield-Gabrieli, and Nancy Kanwisher. New method for fmri investigations of language: defining rois functionally in individual subjects.Journal of Neurophysiology, 104(2):1177–1194, 2010

    2010

  2. [10]

    Freesurfer.NeuroImage, 62(2):774–781, aug 2012

    Bruce Fischl. Freesurfer.NeuroImage, 62(2):774–781, aug 2012

    2012

  3. [11]

    Sereno, Roger B

    Bruce Fischl, Martin I. Sereno, Roger B. H. Tootell, and Anders M. Dale. High-resolution intersubject averaging and a coordinate system for the cortical surface.Human Brain Mapping, 8(4):272–284, 1999

    1999

  4. [12]

    The algonauts project 2025 challenge: How the human brain makes sense of multimodal movies.arXiv preprint arXiv:2501.00504, 2024

    Alessandro T Gifford, Domenic Bersch, Marie St-Laurent, Basile Pinsard, Julie Boyle, Lune Bellec, Aude Oliva, Gemma Roig, and Radoslaw M Cichy. The algonauts project 2025 challenge: How the human brain makes sense of multimodal movies.arXiv preprint arXiv:2501.00504, 2024

    2025 arXiv

  5. [13]

    A multi-modal parcellation of human cerebral cortex.Nature, 536(7615):171–178, 2016

    Matthew F Glasser, Timothy S Coalson, Emma C Robinson, Carl D Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F Beckmann, Mark Jenkinson, et al. A multi-modal parcellation of human cerebral cortex.Nature, 536(7615):171–178, 2016

    2016

  6. [14]

    The cortical organization of speech processing.Nat

    Gregory Hickok and David Poeppel. The cortical organization of speech processing.Nat. Rev. Neurosci., 8(5):393–402, May 2007

    2007

  7. [15]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022

    2022

  8. [16]

    Natural speech reveals the semantic maps that tile human cerebral cortex.Nature, 532(7600):453–458, 2016

    Alexander G Huth, Wendy A de Heer, Thomas L Griffiths, Frédéric E Theunissen, and Jack L Gallant. Natural speech reveals the semantic maps that tile human cerebral cortex.Nature, 532(7600):453–458, 2016

    2016

  9. [17]

    Incorporating context into language encoding models for fmri.Advances in Neural Information Processing Systems, 31, 2018

    Shailee Jain and Alexander Huth. Incorporating context into language encoding models for fmri.Advances in Neural Information Processing Systems, 31, 2018. 12

    2018

  10. [18]

    A natural language fmri dataset for voxelwise encoding models.Scientific Data, 10(1):555, 2023

    Amanda LeBel, Lauren Wagner, Shailee Jain, Aneesh Adhikari-Desai, Bhavin Gupta, Allyson Morgenthal, Jerry Tang, Lixiang Xu, and Alexander G Huth. A natural language fmri dataset for voxelwise encoding models.Scientific Data, 10(1):555, 2023

    2023

  11. [19]

    Nathan Spreng, Jonathan R

    Jixing Li, Shohini Bhattasali, Shulin Zhang, Berta Franzluebbers, Wen-Ming Luh, R. Nathan Spreng, Jonathan R. Brennan, Yiming Yang, Christophe Pallier, and John Hale. Le petit prince multilingual naturalistic fmri corpus.Scientific Data, 9(1):530, 2022

    2022

  12. [20]

    Toward a realistic model of speech processing in the brain with self-supervised learning.Advances in Neural Information Processing Systems, 35:33428–33443, 2022

    Juliette Millet, Charlotte Caucheteux, Yves Boubenec, Alexandre Gramfort, Ewan Dunbar, Christophe Pallier, Jean-Remi King, et al. Toward a realistic model of speech processing in the brain with self-supervised learning.Advances in Neural Information Processing Systems, 35:3342...

    2022

  13. [21]

    Improving semantic understanding in speech language models via brain-tuning

    Omer Moussa, Dietrich Klakow, and Mariya Toneva. Improving semantic understanding in speech language models via brain-tuning. InThe Thirteenth International Conference on Learning Representations (ICLR), 2025

    2025

  14. [22]

    Brain-tuned speech models better reflect speech processing stages in the brain.arXiv preprint arXiv:2506.03832, 2025

    Omer Moussa and Mariya Toneva. Brain-tuned speech models better reflect speech processing stages in the brain.arXiv preprint arXiv:2506.03832, 2025

    2025 arXiv

  15. [23]

    Brain-tuning improves generalizability and efficiency of brain alignment in speech models

    Omer Moussa and Mariya Toneva. Brain-tuning improves generalizability and efficiency of brain alignment in speech models. InAdvances in Neural Information Processing Systems (NeurIPS), 2025

    2025

  16. [24]

    narratives

    Samuel A Nastase, Yun-Fei Liu, Hanna Hillman, Asieh Zadbood, Liat Hasenfratz, Neggin Keshavarzian, Janice Chen, Christopher J Honey, Yaara Yeshurun, Mor Regev, et al. The “narratives” fmri dataset for evaluating models of naturalistic language comprehension.Scientific data, 8(...

    2021

  17. [25]

    Speech language models lack important brain-relevant semantics.Annual Meeting of the Association for Computational Linguistics (ACL), 2024

    Subba Reddy Oota, Emin Çelik, Fatma Deniz, and Mariya Toneva. Speech language models lack important brain-relevant semantics.Annual Meeting of the Association for Computational Linguistics (ACL), 2024

    2024

  18. [26]

    The neural architecture of language: Integrative modeling converges on predictive processing.Proceedings of the National Academy of Sciences, 2021

    Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. The neural architecture of language: Integrative modeling converges on predictive processing.Proceedings of the National Academy of Sci...

    2021

  19. [27]

    Identification of a pathway for intelligible speech in the left temporal lobe.Brain, 123(12):2400–2406, 2000

    Sophie K Scott, C Catrin Blank, Stuart Rosen, and Richard JS Wise. Identification of a pathway for intelligible speech in the left temporal lobe.Brain, 123(12):2400–2406, 2000

    2000

  20. [28]

    Cneuromod-things, a densely- sampled fmri dataset for visual neuroscience.Scientific Data, 2026

    Marie St-Laurent, Basile Pinsard, Oliver Contier, Elizabeth DuPre, Katja Seeliger, Valentina Borghesani, Julie A Boyle, Lune Bellec, and Martin N Hebart. Cneuromod-things, a densely- sampled fmri dataset for visual neuroscience.Scientific Data, 2026

    2026

  21. [29]

    Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain).Advances in Neural Information Processing Systems, 32, 2019

    Mariya Toneva and Leila Wehbe. Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain).Advances in Neural Information Processing Systems, 32, 2019

    2019

  22. [30]

    Self-supervised models of audio effectively explain human cortical responses to speech

    Aditya R Vaidya, Shailee Jain, and Alexander Huth. Self-supervised models of audio effectively explain human cortical responses to speech. InInternational Conference on Machine Learning, pages 21927–21944, 2022

    2022

  23. [31]

    Vaidya, Richard J

    Nishitha Vattikonda, Aditya R. Vaidya, Richard J. Antonello, and Alexander G. Huth. Brain- WavLM: Fine-tuning speech representations with brain responses to language, 2025

    2025

  24. [32]

    Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses.PloS One, (11), 2014

    Leila Wehbe, Brian Murphy, Partha Talukdar, Alona Fyshe, Aaditya Ramdas, and Tom Mitchell. Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses.PloS One, (11), 2014. 13 A Datasets and preprocessing All experiments use a commo...

    2014

Pith tools

Reviewed July 8, 2026 · model on record in the stance chip above.