Pith. sign in

REVIEW 4 major objections 4 minor 18 references

VoxelFormer claims a single shared transformer can decode perceived images from fMRI across subjects without subject-specific layers, reaching 74.3% top-1 retrieval with 39M parameters.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A 39M-parameter transformer with token merging and query compression gives around 74% top-1 fMRI-to-image retrieval across six training subjects, far below the ~98% of larger state-of-the-art decoders.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Real engineering contribution with an unsupported headline: the 39M-parameter architecture is a genuine new combination, but the 'competitive' claim relies on a non-matching subject-set comparison and an internal contradiction with the Discussion. the 4 major comments →

arxiv 2509.09015 v1 pith:WRXYSN53 submitted 2025-09-10 cs.CV

VoxelFormer: Parameter-Efficient Multi-Subject Visual Decoding from fMRI

Chenqian Le , Yilin Zhao , Nikasadat Emami , Kushagra Yadav , Xujin "Chris" Liu , Xupeng Chen , Yao Wang This is my paper
classification cs.CV
keywords fMRI decodingvisual reconstructionmulti-subject learningtoken merging transformerQ-FormerCLIP alignmentimage retrievalNatural Scenes Dataset
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VoxelFormer claims that a single transformer shared across subjects can decode perceived images from fMRI voxels without subject-specific layers. It compresses each subject's visual-cortex voxels into a compact token set by attention-guided merging, then uses learnable queries to produce fixed-size embeddings aligned with CLIP image features. On the 7T Natural Scenes Dataset, the model reports 74.3% mean top-1 forward retrieval accuracy across training subjects S2–S7 with 39 million parameters, versus hundreds of millions for prior single-subject or aligned baselines. The paper argues that token merging and query-based distillation can substitute for per-subject capacity, making multi-subject training practical with far less data and compute.

Core claim

The central discovery claim is that raw fMRI voxels from different individuals can be projected into a shared CLIP-aligned embedding space by a single network, provided that voxel tokens are progressively merged by attention similarity and then distilled through a small set of learnable queries. The paper shows that this pipeline reaches competitive image-retrieval performance (74.3% mean top-1 forward retrieval on subjects S2–S7; above 66% for each included subject) while using 39M parameters, about 12x fewer than MindEye2 and 24x fewer than MindEye1. Because the Q-Former outputs fixed-size embeddings regardless of voxel count, the same shared modules handle all subjects, with no subject-sp

What carries the argument

The Token Merging Transformer (ToMer) encoder and the Q-Former. ToMer tokenizes voxels with a 1x1 convolution, adds sinusoidal positional embeddings derived from voxel coordinates, computes self-attention, and merges the most similar token pairs during training, progressively compressing the voxel set. The Q-Former then uses a fixed set of learnable query tokens that cross-attend to the compressed features, mapping variable-size neural latents to a fixed-size representation aligned with CLIP embeddings through MSE and contrastive losses.

Load-bearing premise

The load-bearing premise is that a single shared network, guided only by coordinate-based positional embeddings and attention-driven token merging, can absorb the anatomical and functional variability across different people's brains well enough to map every subject's voxels into one shared CLIP-aligned space, without subject-specific layers or explicit anatomical registration.

What would settle it

Train the same architecture on two subjects whose visual cortices differ substantially in shape, size, or coverage (or artificially misalign one subject's voxel grid), then check whether per-subject retrieval accuracy stays well above chance. If the shared network fails on one subject while succeeding on the other, the coordinate-based positional embeddings are not absorbing inter-subject variability, and the multi-subject claim collapses. A simpler calculation: plot per-subject accuracy against voxel-count or spatial-overlap differences; a sharp drop with anatomical distance would indicate th

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Multi-subject training with shared weights becomes feasible without per-subject adaptation, cutting model size to 39M and enabling deployment on resource-constrained hardware.
  • The fixed-size Q-Former output means a new subject's voxels can, in principle, be fed directly to the shared modules, supporting the intended zero-shot retrieval evaluation on held-out subjects.
  • Token merging during training reduces memory and compute, allowing longer or larger-scale training than dense transformer encoders.
  • The dual-branch design separates retrieval (contrastive MLP) from reconstruction (MSE prior), and the prior branch could serve as a conditioning signal for diffusion-based image generation.
  • Parameter-efficient multi-subject decoding could lower the data barrier for practical brain-computer interfaces that currently require tens of hours of per-subject scans.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The coordinate-based positional embeddings may be implicitly learning a functional correspondence across subjects; if so, explicit anatomical alignment might become unnecessary — this is testable by ablating or shuffling the positional encoding.
  • The reported 74.3% mean leaves a gap to MindEye2's 98.3%, so the paper's real claim is about accuracy per parameter, not peak accuracy; a fair trade-off analysis would fix a parameter budget and compare retrieval per million parameters.
  • The same token-merging plus query-distillation design could transfer to other high-dimensional neural or physiological signals (e.g., EEG or MEG decoding), where voxel-count variability and memory constraints are similar.
  • A direct extension would be to train on more subjects and test whether the shared network continues to absorb inter-subject variability, or whether accuracy plateaus as anatomical diversity grows.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes VoxelFormer, a two-stage transformer for multi-subject fMRI-based visual decoding. The first stage is a Token Merging Transformer (ToMer) that tokenizes raw visual-cortex voxels, adds coordinate-based positional embeddings, and progressively merges tokens by attention similarity. The second stage is a Q-Former with learnable queries that produce fixed-size neural embeddings, which are trained with MSE and contrastive losses to align with frozen CLIP image embeddings. The authors train on NSD subjects S2–S7 with a single shared network containing about 39M parameters, and report top-1 image retrieval accuracies per subject and on average. The paper claims competitive retrieval performance on training subjects with significantly fewer parameters than MindEye1 and MindEye2.

Significance. Parameter-efficient cross-subject fMRI decoding is a meaningful goal: subject-specific models such as MindEye1 require roughly 940M parameters per subject, and cross-subject models such as MindEye2 still use over 469M parameters. A shared 39M-parameter architecture that avoids subject-specific layers would be a useful contribution if its retrieval accuracy were truly competitive. The paper uses the standard NSD retrieval protocol with a 300-image candidate pool, and the code is publicly linked. I also see no circularity problem: retrieval is measured on held-out test images and the target CLIP embeddings are external. The main weakness is that the reported numbers do not support the word 'competitive,' as detailed in the major comments.

major comments (4)
  1. [Abstract; Table I; Section V] The central claim of 'competitive retrieval performance' is contradicted by the manuscript's own data. On the three subjects where both VoxelFormer and MindEye2 report forward top-1 accuracy, VoxelFormer is 86.54 vs 99.88 (S2), 73.03 vs 98.39 (S5), and 68.65 vs 96.89 (S7), a 13–28 point gap. Against MindEye1 on S2–S4 the gaps are also 10–15 points. The Discussion explicitly states that 'retrieval accuracy remains below state-of-the-art subject-specific approaches,' which undercuts the abstract's 'competitive.' Because this is the paper's headline claim, the claim needs to be either supported by matched-subject evidence or removed/reframed.
  2. [Table II; Table I] The mean comparison in Table II is not like-for-like. 'Ours' is averaged over S2–S7, 'MindEye1' over S1–S4, and 'MindEye2' over S1,2,5,7; these are different subject subsets. The per-subject table is incomplete: MindEye2 has no entries for S3/S4/S6 and MindEye1 has no entries for S5/S6/S7. No common-subject mean is reported. Additionally, no error bars, seeds, or confidence intervals are given; with a 300-image candidate pool, the 74.3% point estimate is not assessable. A matched-subject comparison is required before any claim of competitiveness can be evaluated.
  3. [Section IV, subject-wise performance] The paper states that subject 1 was held out for zero-shot retrieval, but no held-out S1 results are reported anywhere. Since the abstract emphasizes multi-subject training and the method's scalability, the held-out result is directly relevant to the central claim. Its absence leaves a stated evaluation incomplete and weakens the generalizability argument.
  4. [Section II-B; Section III-C] The paper notes that anatomical alignment 'may be necessary for meaningful multi-subject training,' yet VoxelFormer uses no subject-specific layers and no explicit anatomical alignment, relying on coordinate-based positional embeddings to absorb inter-subject differences. The reported training-subject results show this can work to some degree, but the paper provides no analysis or ablation of how the shared network handles cross-subject variability. A concrete test, such as comparing with an anatomical-alignment baseline or reporting per-subject token-merging behavior, is needed to support the claim that the architecture itself, rather than the particular NSD subjects chosen, provides the multi-subject capability.
minor comments (4)
  1. [Table II, paragraph] The sentence 'MindEye2 achieves slightly worse performance, even though each subject specific model is twice the size the MindEye2' is garbled. It appears to mean that MindEye1 performs slightly worse than MindEye2 despite having roughly twice the parameter count. Please correct the wording and clarify which model is being discussed.
  2. [Section III-D, Eq. (2)] Notation is incompletely defined. The variables p_i, lambda_i, k_i, and the relationship between batch size B in Eq. (1) and N in Eq. (2) should be stated explicitly. This makes the loss formula hard to check.
  3. [Section IV] Key hyperparameters are missing: the number of Q-Former queries, ToMer block count L, tokens merged per block M, latent dimension C, contrastive temperature tau, optimizer, learning rate, number of epochs, and the exact train/test split. Reporting only lambda_MSE=30 and lambda_contrastive=1 is insufficient for reproducibility.
  4. [Throughout] Minor typos and inconsistent naming: 'voxel partterns' should be 'voxel patterns'; 'V oxelFormer' and 'V oxFormer' appear alongside 'VoxelFormer.' Casual abbreviations such as 'SOTA' should be spelled out.

Circularity Check

0 steps flagged

No circularity: derivation is self-contained and target CLIP embeddings are external; the 'competitive' wording is a validity concern, not a circular one.

full rationale

The paper's derivation chain is: raw fMRI voxels → ToMer encoder (1×1 conv tokenizer, sinusoidal coordinate embeddings, attention, token merging) → Q-Former (learnable queries attend to compressed tokens) → fixed-size embeddings → two heads (prior transformer with MSE loss to frozen CLIP embeddings; MLP projector with contrastive loss). The CLIP image embeddings used as regression/contrastive targets are externally defined and frozen, so the model is trained to match an independent space, not its own output. The loss weights (λ_MSE=30, λ_contrastive=1) are hyperparameters, not fitted parameters disguised as predictions. Retrieval accuracy is measured by nearest-neighbor search against a candidate pool of 300 images, an external benchmark. Token Merging (ref 11), Q-Former/BLIP (refs 14/17), and the MindEye2 dual-pathway training strategy (ref 9) are all prior external works; the VoxelFormer authors do not overlap with these cited authors, so there is no load-bearing self-citation. The paper's claims that are questionable—such as 'competitive retrieval performance' despite the Discussion conceding 'retrieval accuracy remains below state-of-the-art subject-specific approaches,' and Table II comparing different subject sets—are issues of evidence quality and internal consistency, not circularity. No equation reduces to its own input, and no fitted parameter is relabeled as a prediction. Per the review rules, lack of standard consensus or unsupported competitive claims are correctness risks, not circularity. Therefore the circularity score is 0.

Axiom & Free-Parameter Ledger

7 free parameters · 4 axioms · 0 invented entities

The model has several undisclosed hyperparameters, and the cross-subject comparability of raw voxels is assumed without an explicit alignment or validation. No new entities are introduced.

free parameters (7)
  • lambda_MSE = 30
    Weight for MSE loss in Eq. 4; chosen by hand, not derived.
  • lambda_contrastive = 1
    Weight for contrastive loss in Eq. 4.
  • number_of_query_tokens
    Q-Former output size; not reported, needed for replication.
  • ToMer_block_count_L
    Number of stacked ToMer blocks; not reported.
  • tokens_merged_per_block_M
    Compression rate per block; not reported.
  • latent_dimension_C
    Hidden dimension of tokens; not reported.
  • contrastive_temperature_tau
    Temperature in BiMixCo/SoftCLIP; not reported.
axioms (4)
  • domain assumption Raw fMRI voxels from different subjects are directly comparable via coordinate-based positional embeddings without subject-specific alignment layers.
    Sections III-B and III-C rely on SiREN coordinate PE to handle inter-subject variability; the paper itself says anatomical alignment may be necessary in Section II-B.
  • domain assumption CLIP image embeddings are a sufficient target space such that cosine-similarity retrieval is a valid measure of decoding accuracy.
    Used throughout as training targets and retrieval space; inherited from prior work without new justification.
  • ad hoc to paper Token merging based on attention similarity preserves the task-relevant information needed for CLIP alignment.
    ToMer merges tokens with highest attention similarity during training; no analysis shows this is lossless for fMRI decoding.
  • domain assumption The 300-image candidate pool retrieval protocol yields a meaningful and comparable top-1 accuracy metric.
    Evaluation follows MindEye1 [7]; no discussion of pool size effects.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of VoxelFormer: Parameter-Efficient Multi-Subject Visual Decoding from fMRI." pith.science (2026). https://pith.science/paper/WRXYSN53

@misc{pith2026250909015,
  author       = {Pith},
  title        = {Pith review of: VoxelFormer: Parameter-Efficient Multi-Subject Visual Decoding from fMRI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WRXYSN53}},
  note         = {Machine review of arXiv:2509.09015}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Recent advances in fMRI-based visual decoding have enabled compelling reconstructions of perceived images. However, most approaches rely on subject-specific training, limiting scalability and practical deployment. We introduce \textbf{VoxelFormer}, a lightweight transformer architecture that enables multi-subject training for visual decoding from fMRI. VoxelFormer integrates a Token Merging Transformer (ToMer) for efficient voxel compression and a query-driven Q-Former that produces fixed-size neural representations aligned with the CLIP image embedding space. Evaluated on the 7T Natural Scenes Dataset, VoxelFormer achieves competitive retrieval performance on subjects included during training with significantly fewer parameters than existing methods. These results highlight token merging and query-based transformers as promising strategies for parameter-efficient neural decoding.

Figures

Figures reproduced from arXiv: 2509.09015 by Chenqian Le, Kushagra Yadav, Nikasadat Emami, Xujin "Chris" Liu, Xupeng Chen, Yao Wang, Yilin Zhao.

Figure 1
Figure 1. Figure 1: Overview of the proposed VoxelFormer pipeline for cross-subject fMRI-to-image decoding. Multi-subject fMRI volumes are first encoded using a token-merging encoder (ToMer) with coordinate-based positional embeddings (PE) to generate compact neural latents. These latents interact with a small set of learnable queries within a Query-Former (Q-Former), composed of repeated cross-attention, self-attention, and … view at source ↗
Figure 2
Figure 2. Figure 2: ToMer Encoder Architecture The ToMer encoder processes input fMRI data (BN, where B is batch size and N is the number of voxels in the visual cortex) by first applying a Tokenizer Layer and Positional Embedding (PE). The tokenized features are then passed through an attention block, producing latent representations (BNC) and an attention matrix (BNN). A Token Merging operation reduces the number of tokens … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 6 linked inside Pith

  1. [1]

    Using goal-driven deep learning models to understand sensory cortex,

    D. L. K. Yamins and J. J. DiCarlo, “Using goal-driven deep learning models to understand sensory cortex,”Nature Neuroscience, vol. 19, no. 3, pp. 356–365, Mar. 2016, publisher: Nature Publishing Group. [Online]. Available: https://www.nature.com/articles/nn.4244

    2016

  2. [2]

    Machine Learning for Neural Decoding,

    J. I. Glaser, A. S. Benjamin, R. H. Chowdhury, M. G. Perich, L. E. Miller, and K. P. Kording, “Machine Learning for Neural Decoding,” eNeuro, vol. 7, no. 4, Jul. 2020, publisher: Society for Neuroscience Section: Research Article: Methods/New Tools. [Online]. Available: https://www.eneuro.org/content/7/4/ENEURO.0506-19.2020

    2020

  3. [3]

    Encoding and decoding in fMRI,

    T. Naselaris, K. N. Kay, S. Nishimoto, and J. L. Gallant, “Encoding and decoding in fMRI,”NeuroImage, vol. 56, no. 2, pp. 400–410, May 2011

    2011

  4. [4]

    Learning Transferable Visual Models From Natural Language Supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning Transferable Visual Models From Natural Language Supervision,” Feb. 2021, arXiv:2103.00020 [cs]. [Online]. Available: http://arxiv.org/abs/2103.00020

    Pith/arXiv arXiv 2021

  5. [5]

    A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence,

    E. J. Allen, G. St-Yves, Y . Wu, J. L. Breedlove, J. S. Prince, L. T. Dowdle, M. Nau, B. Caron, F. Pestilli, I. Charest, J. B. Hutchinson, T. Naselaris, and K. N. Kay, “A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence,”Nature Neuroscience, vol. 25, no. 1, pp. 116–126, 2021. [Online]. Available: https://www.nature.com/...

    2021

  6. [6]

    Deep image re- construction from human brain activity,

    G. Shen, T. Horikawa, K. Majima, and Y . Kamitani, “Deep image re- construction from human brain activity,”PLoS computational biology, vol. 15, no. 1, p. e1006633, 2019

    2019

  7. [7]

    Reconstructing the mind’s eye: fmri-to- image with contrastive learning and diffusion priors,

    P. Scotti, S. Banerjee, A. Goode, P. Shabalin, A. Nguyen, A. Cohen, A. Dempster, C. Verlinde, E. Yundler, S. M. Weisberg, K. A. Norman, and A. Abraham, “Reconstructing the mind’s eye: fmri-to- image with contrastive learning and diffusion priors,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 36, 2023, arXiv:2305.18274. [Online]. Ava...

    Pith/arXiv arXiv 2023

  8. [8]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” inProceedings of the 38th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Ed...

    2021

  9. [9]

    Mindeye2: Shared-subject models enable fmri-to- image with 1 hour of data,

    P. Scotti, R. Tripathy, J. Torrico, E. Kneeland, Y . Chen, R. Narang, P. Santhirasegaran, T. Xu, T. Naselaris, K. A. Norman, and A. Abraham, “Mindeye2: Shared-subject models enable fmri-to- image with 1 hour of data,” inProceedings of the 41st International Conference on Machine Learning (ICML), 2024, arXiv:2403.11207. [Online]. Available: https://arxiv.o...

    Pith/arXiv arXiv 2024

  10. [10]

    MindBridge: A Cross- Subject Brain Decoding Framework,

    S. Wang, S. Liu, Z. Tan, and X. Wang, “MindBridge: A Cross- Subject Brain Decoding Framework,” Apr. 2024, arXiv:2404.07850 [cs]. [Online]. Available: http://arxiv.org/abs/2404.07850

    Pith/arXiv arXiv 2024

  11. [11]

    Token merging: Your ViT but faster,

    D. Bolya, C.-Y . Fu, X. Dai, P. Zhang, C. Feichtenhofer, and J. Hoff- man, “Token merging: Your ViT but faster,” inInternational Confer- ence on Learning Representations, 2023

    2023

  12. [12]

    Tokens-to-token vit: Training vision transformers from scratch on imagenet,

    L. Yuan, Y . Chen, T. Wang, W. Yu, Y . Shi, J. Zhang, F. E. H. Tay, J. Feng, and S. Yan, “Tokens-to-token vit: Training vision transformers from scratch on imagenet,” inICCV, 2021

    2021

  13. [13]

    Perceiver: General Perception with Iterative Attention,

    A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira, “Perceiver: General Perception with Iterative Attention,” inProceedings of the 38th International Conference on Machine Learning. PMLR, Jul. 2021, pp. 4651–4664, iSSN: 2640-3498. [Online]. Available: https://proceedings.mlr.press/v139/jaegle21a.html

    2021

  14. [14]

    Blip-2: Bootstrapping language-image pre-training for unified vision-language understanding and generation,

    J. Li, R. Li, C. Xiao, C. Fang, and J. Lu, “Blip-2: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” inICML, 2023

    2023

  15. [15]

    Microsoft COCO: Common Objects in Context,

    T.-Y . Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Doll ´ar, “Microsoft COCO: Common Objects in Context,” Feb. 2015, arXiv:1405.0312 [cs]. [Online]. Available: http://arxiv.org/abs/1405.0312

    Pith/arXiv arXiv 2015

  16. [16]

    Implicit neural representations with periodic activation functions,

    V . Sitzmann, J. N. Martel, A. W. Bergman, D. B. Lindell, and G. Wetzstein, “Implicit neural representations with periodic activation functions,” inarXiv, 2020

    2020

  17. [17]

    BLIP: Bootstrapping language- image pre-training for unified vision-language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. Hoi, “BLIP: Bootstrapping language- image pre-training for unified vision-language understanding and generation,” inProceedings of the 39th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, Eds., vol. 162. PMLR...

    2022

  18. [18]

    Natural scene reconstruction from fmri signals using generative latent diffusion,

    F. Ozcelik and R. VanRullen, “Natural scene reconstruction from fmri signals using generative latent diffusion,” 2023. [Online]. Available: https://arxiv.org/abs/2303.05334

    Pith/arXiv arXiv 2023

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.