Pith. sign in

REVIEW 2 major objections 6 minor 60 references

Simply swapping standard CLIP for adversarially robust CLIP variants improves fMRI brain decoding and neural alignment without changing the decoder.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 04:23 UTC pith:6ZIODKJR

load-bearing objection Clean controlled swap of CLIP targets to FARE/TeCoA improves fMRI decoding and alignment; the causal story about non-robust features is motivational, not proven. the 2 major comments →

arxiv 2607.03165 v1 pith:6ZIODKJR submitted 2026-07-03 cs.CV

Rethinking Brain Decoding with CLIP: The Role of Adversarial Robustness

Byeongseo Bok , Futa Waseda , Jun Liu , Isao Echizen This is my paper
classification cs.CV
keywords brain decodingfMRICLIPadversarial robustnessneural alignmentrepresentational similarityzero-shot classificationattribution maps
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Brain decoding often maps fMRI activity to CLIP embeddings so that brain signals can be linked to images or text. Standard CLIP, trained only for image–text matching, may rely on non-robust or shortcut features that brains do not encode well, limiting how cleanly fMRI can be aligned to it. This paper keeps the linear fMRI decoder, data, and training fixed and changes only the target embedding to publicly available adversarially robust CLIP variants. Across the large NSD dataset and the zero-shot GOD dataset, those robust targets raise image-retrieval accuracy, improve zero-shot text classification from fMRI, and produce higher Pearson correlation, cosine similarity, and representational similarity scores. Attribution maps further show that the robust models use spatially different image features from ordinary CLIP. The result is that the choice of target representation itself is a controllable lever for better brain–model alignment, and adversarial robustness is a practical criterion for choosing that representation.

Core claim

When the fMRI decoder, preprocessing, architecture, and training procedure are held fixed, replacing the target representation with adversarially trained CLIP variants (FARE and TeCoA at two perturbation strengths) consistently improves fMRI–image retrieval accuracy, zero-shot fMRI–text classification, and multiple alignment metrics relative to standard CLIP on both the NSD and GOD datasets. Attribution analysis shows low spatial and rank agreement with standard CLIP, indicating that adversarial robustness reorganizes which visual features support the alignment.

What carries the argument

Controlled target-representation swap: a linear decoder maps fMRI voxels to the 768-dimensional embedding of either standard CLIP or an adversarially robust CLIP (FARE or TeCoA), trained with the same InfoNCE or ridge objective; only the frozen target embedding changes.

Load-bearing premise

The gains come specifically because adversarial training removes non-robust features that human fMRI does not encode, rather than from other uncontrolled differences among the public model checkpoints.

What would settle it

Train matched CLIP variants that differ only in adversarial robustness (same data, same optimization except the adversarial objective) and check whether the non-robust controls erase the retrieval and RSA gains on the same NSD and GOD splits.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper investigates whether adversarially robust CLIP vision embeddings improve fMRI brain decoding relative to standard CLIP. Holding the linear fMRI decoder, data preprocessing, and training procedure fixed, the authors replace only the target representation with public robust CLIP variants (FARE-2/4, TeCoA-2/4) and evaluate on NSD and GOD. Across InfoNCE and ridge objectives they report higher fMRI–image retrieval accuracy, stronger decoder-mediated and representation-level alignment (Pearson r, cosine similarity, RSA; CKA in the appendix), and better zero-shot fMRI–text classification on GOD. An occlusion-based attribution analysis further shows low top-k IoU and moderate/variable Spearman agreement between standard CLIP and robust variants, together with more scale-dependent structure in attribution maps. The authors conclude that target-representation choice matters and that adversarial robustness is a useful selection criterion for brain decoding.

Significance. If the empirical pattern holds, the result is practically useful for a large and active literature that already treats CLIP embeddings as the default target for fMRI decoding: a simple checkpoint swap yields consistent multi-metric, multi-subject, multi-dataset gains without architectural redesign of the decoder or generator. The controlled design (fixed decoder and training; only the target changes; both contrastive and ridge; zero-shot GOD) is a clear strength and makes the performance/alignment claim easy to re-use and re-test. Attribution maps provide additional evidence of representational reorganization rather than a mere scalar accuracy bump. The causal story that gains come specifically from suppression of non-robust/shortcut features is not isolated from other checkpoint differences, but the paper’s Discussion already treats that interpretation cautiously; the narrower empirical claim remains valuable even if the precise mechanism stays open.

major comments (2)
  1. The title, abstract, and Introduction frame adversarial robustness (suppression of non-robust/shortcut features) as the operative property, but the experiments only compare public FARE/TeCoA checkpoints to standard CLIP. Those checkpoints also differ in fine-tuning data, optimization trajectory, and residual zero-shot capacity. Without at least one control that holds architecture fixed while varying robustness independently (e.g., non-adversarial fine-tunes of the same ViT-L/14, or intermediate checkpoints along the adversarial training path), the load-bearing causal claim that robustness per se drives the gains remains motivational rather than demonstrated. The Discussion already softens this; the main text and title should either add such a control or consistently state the result as “robust CLIP checkpoints improve decoding” rather than “adversarial robustness improves decoding.”
  2. Relatedly, the manuscript does not report the pure vision-side retrieval / zero-shot classification accuracy of CLIP vs. FARE/TeCoA on the same NSD and GOD image sets (independent of fMRI). If robust variants are simply stronger or better-calibrated image embeddings for these stimuli, part of the decoder gains could reflect target quality rather than neural alignment. A short table of image–image and image–text baselines for each checkpoint would substantially strengthen (or qualify) the neural-alignment interpretation of Figures 2–4 and Tables 1–4.
minor comments (6)
  1. Figure 1’s “+13%” and RSA numbers are useful but should be explicitly tied to the corresponding table/figure (FARE-4 Top-1 under contrastive on NSD; RSA means in Table 1) so readers can verify the takeaway panel.
  2. §2.2.2 / Eqs. (1)–(2): state the temperature τ value used in the main experiments in the main text (it appears only in Appendix A.1 as 0.03), since InfoNCE temperature is a free parameter of the training objective.
  3. §3.1 notes metric-dependent ranking of TeCoA vs. FARE (alignment vs. retrieval). A brief sentence on why text-supervised vs. unsupervised adversarial fine-tuning might produce this split would help readers interpret the two families.
  4. Attribution §4.1: the multi-scale occlusion hyperparameters (σ ∈ {3,5,9,17,33,65}, stride 3) are only in Appendix B.1; a one-line pointer in the main text would aid reproducibility.
  5. Table 3 GOD Top-1 for FARE-4 under ridge (0.06) is below CLIP (0.08) while higher-k recovers; flag this non-monotonicity briefly in the main text rather than only in the figure caption.
  6. Minor typos / style: “sfMRI” in Figure 1; “T eCoA” spacing in several figure labels; “double dipping” citation is fine but the NSD test-set rearrangement (§A.2.1) could be stated more clearly for readers unfamiliar with Algonauts vs. original NSD splits.

Circularity Check

0 steps flagged

No significant circularity: controlled empirical swap of target representations evaluated on held-out retrieval, zero-shot, and alignment metrics.

full rationale

The paper's load-bearing claim is that fixing the linear fMRI decoder, data, preprocessing, architecture, and training objective while substituting only the target embedding (standard CLIP vs. public FARE/TeCoA checkpoints) yields higher Pearson r, cosine similarity, RSA, top-k retrieval accuracy, and zero-shot fMRI-text classification on the NSD test split and the GOD zero-shot split. These quantities are measured against independent held-out image/fMRI pairs and non-overlapping semantic categories; none is obtained by fitting a free parameter that is then re-reported as the result. RSA is computed directly between fMRI RDMs and model RDMs; retrieval and classification use cosine similarity of decoder outputs to held-out embeddings. Attribution maps are post-hoc occlusion analyses comparing spatial patterns, not inputs to the performance claims. Hyperparameters (InfoNCE temperature, ridge λ, voxel threshold) are fixed or grid-searched on validation and do not define the outcome by construction. Citations to Ilyas, Tsipras, etc., supply motivational background on non-robust features but are not used as uniqueness theorems or self-referential premises that force the empirical numbers. The design is therefore an ordinary controlled experiment, not a circular derivation.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The work is empirical; it inherits standard fMRI preprocessing and public robust-CLIP checkpoints rather than introducing new physical entities or free constants that define the result. Load-bearing premises are domain assumptions about what adversarial training does to features and that linear InfoNCE/ridge decoding is a fair probe of representation quality.

free parameters (4)
  • InfoNCE temperature τ = 0.03
    Set to 0.03 following prior contrastive decoding work; affects the sharpness of the contrastive objective used for all models equally.
  • ridge regularization λ = grid-searched per model
    Chosen by grid search on validation performance; model-specific but selected independently of the final test claims.
  • voxel noise-ceiling threshold = 8.0
    Fixed at 8.0 for NSD voxel selection following prior pipeline; affects which voxels enter the decoder for every target equally.
  • adversarial ε of target models = 2/255 and 4/255
    ε ∈ {2/255, 4/255} inherited from the public FARE/TeCoA checkpoints; not re-fit by the authors but defines the ‘robust’ condition being tested.
axioms (4)
  • domain assumption Adversarial training suppresses non-robust features that are less aligned with human perception and brain activity.
    Invoked in Introduction and §2.1 via citations to Ilyas et al., Tsipras et al., Geirhos et al.; not re-proven here.
  • domain assumption A linear decoder trained with InfoNCE (or ridge) is a sufficient and fair probe of how well a fixed target representation space aligns with fMRI responses.
    Stated in §2.2.2 and used throughout; architecture and objective held fixed across targets.
  • domain assumption NSD and GOD preprocessing (denoised betas, noise-ceiling voxel selection, ROI visual cortex for GOD) yield valid neural representations of the viewed stimuli.
    Follows prior pipelines (§2.2.1, Appendix A.2); standard in the field.
  • domain assumption Occlusion-based multi-scale attribution on decoder–encoder cosine similarity reflects the image features each decoder relies on for alignment.
    Adapted from Palazzo et al. in §4.1; used to interpret feature reorganization.

pith-pipeline@v1.1.0-grok45 · 23387 in / 3037 out tokens · 30273 ms · 2026-07-12T04:23:53.580236+00:00 · methodology

0 comments
read the original abstract

Brain decoding aims to uncover neural mechanisms by inferring stimulus-related representations from brain signals. In fMRI studies, this is typically achieved by mapping fMRI responses to the latent representations of computational models. Recently, CLIP has become a popular choice for brain decoding due to its rich vision--language embedding space. However, aligning fMRI signals with CLIP representations remains challenging. As CLIP is not explicitly optimized for neural alignment, its representations may capture statistically predictive cues that are only partially reflected in brain activity, limiting decoding performance. In this paper, we investigate whether adversarially robust representations improve neural decoding with CLIP. Adversarial training suppresses non-robust features and promotes more stable, perceptually structured representations, which may better align with brain activity. We evaluate this by fixing the fMRI decoder and varying only the target representation (standard CLIP vs. robust variants) on fMRI-image retrieval and zero-shot classification tasks across NSD and GOD datasets. Empirical results show that this simple change consistently improves task performance and yields stronger alignment across multiple metrics. Attribution analysis further reveals consistently low agreement between standard CLIP and its robust variants, suggesting that adversarial robustness reorganizes feature importance in the visual representation. These findings suggest that the choice of target representation influences neural decoding performance and that adversarial robustness may serve as a useful criterion for brain decoding.

Figures

Figures reproduced from arXiv: 2607.03165 by Byeongseo Bok, Futa Waseda, Isao Echizen, Jun Liu.

Figure 1
Figure 1. Figure 1: A summary of our work: changing target representation to an adversarially robust one [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of alignment metrics between different models on the NSD dataset. All the [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: fMRI-image retrieval accuracy of different models. Robust CLIP improves retrieval both [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Top-k fMRI-text zero-shot classification on the GOD dataset. Error bars indicate variability [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Attribution maps of 10 randomly selected images from the test dataset. From left to [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: (a) Mean of Normalized Entropy measured by stride. (b) Standard Deviation of Normalized [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: (a) Average Top-k IoU with CLIP (b) Average Spearman Correlation with CLIP. Both plots [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Statistics of the scale-dependent Gini index [ [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Attribution maps of 10 selected images from the test dataset by high distributional dif [PITH_FULL_IMAGE:figures/full_fig_p020_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: (Left) Attribution maps of 5 randomly selected images from the test dataset. FARE-2 and [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

60 extracted references · 3 linked inside Pith

  1. [1]

    A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence.Nature neuroscience, 25(1):116–126, 2022

    Emily J Allen, Ghislain St-Yves, Yihan Wu, Jesse L Breedlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence.Nature neuroscience, 25(1):116–126, 2022

    2022

  2. [2]

    Concise explanations of neural networks using adversarial training, 2020

    Prasad Chalasani, Jiefeng Chen, Amrita Roy Chowdhury, Somesh Jha, and Xi Wu. Concise explanations of neural networks using adversarial training, 2020

    2020

  3. [3]

    Bridging the gap between brain and machine in interpreting visual semantics: Towards self-adaptive brain-to-text decoding

    Jiaxuan Chen, Yu Qi, Yueming Wang, and Gang Pan. Bridging the gap between brain and machine in interpreting visual semantics: Towards self-adaptive brain-to-text decoding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21938– 21948, 2025

    2025

  4. [4]

    On the robustness of large multimodal models against image adversarial attacks

    Xuanming Cui, Alejandro Aparcedo, Young Kyun Jang, and Ser-Nam Lim. On the robustness of large multimodal models against image adversarial attacks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24625–24634, 2024

    2024

  5. [5]

    Simulating a primary visual cortex at the front of cnns improves robustness to image perturbations.Advances in Neural Information Processing Systems, 33:13073–13087, 2020

    Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David Cox, and James J DiCarlo. Simulating a primary visual cortex at the front of cnns improves robustness to image perturbations.Advances in Neural Information Processing Systems, 33:13073–13087, 2020

    2020

  6. [6]

    Neural mechanisms of selective visual attention.Annual review of neuroscience, 18(1):193–222, 1995

    Robert Desimone, John Duncan, et al. Neural mechanisms of selective visual attention.Annual review of neuroscience, 18(1):193–222, 1995

    1995

  7. [7]

    Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto

    Guneet S. Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification, 2020

    2020

  8. [8]

    Finding shared decodable concepts and their negations in the brain

    Cory Daniel Efird, Alex Murphy, Joel Zylberberg, and Alona Fyshe. Finding shared decodable concepts and their negations in the brain. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  9. [9]

    Adversarial robustness as a prior for learned representations, 2019

    Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Adversarial robustness as a prior for learned representations, 2019

    2019

  10. [10]

    On the connec- tion between adversarial robustness and saliency map interpretability, 2019

    Christian Etmann, Sebastian Lunz, Peter Maass, and Carola-Bibiane Schönlieb. On the connec- tion between adversarial robustness and saliency map interpretability, 2019

    2019

  11. [11]

    An extended study of human-like behavior under adversarial training, 2023

    Paul Gavrikov, Janis Keuper, and Margret Keuper. An extended study of human-like behavior under adversarial training, 2023

    2023

  12. [12]

    Shortcut learning in deep neural networks.Nature Machine Intelligence, 2(11):665–673, 2020

    Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks.Nature Machine Intelligence, 2(11):665–673, 2020

    2020

  13. [13]

    Wichmann, and Wieland Brendel

    Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision, 2021

    2021

  14. [14]

    A. T. Gifford, B. Lahner, S. Saba-Sadiya, M. G. Vilas, A. Lascelles, A. Oliva, K. Kay, G. Roig, and R. M. Cichy. The algonauts project 2023 challenge: How the human brain makes sense of natural scenes, 2023

    2023

  15. [15]

    Boosting the visual interpretability of clip via adversarial fine-tuning

    Shizhan Gong, LEI Haoyu, Qi Dou, and Farzan Farnia. Boosting the visual interpretability of clip via adversarial fine-tuning. InThe Thirteenth International Conference on Learning Representations, 2025. 10

    2025

  16. [16]

    Neuroclips: Towards high-fidelity and smooth fmri-to-video reconstruction.Advances in Neural Information Processing Systems, 37:51655–51683, 2024

    Zixuan Gong, Guangyin Bao, Qi Zhang, Zhongwei Wan, Duoqian Miao, Shoujin Wang, Lei Zhu, Changwei Wang, Rongtao Xu, Liang Hu, et al. Neuroclips: Towards high-fidelity and smooth fmri-to-video reconstruction.Advances in Neural Information Processing Systems, 37:51655–51683, 2024

    2024

  17. [17]

    Explaining and harnessing adversar- ial examples.arXiv preprint arXiv:1412.6572, 2014

    Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples.arXiv preprint arXiv:1412.6572, 2014

    Pith/arXiv arXiv 2014

  18. [18]

    Gifford, Martina G

    Manshan Guo, Bhavin Choksi, Sari Sadiya, Alessandro T. Gifford, Martina G. Vilas, Ra- doslaw M. Cichy, and Gemma Roig. Limited but consistent gains in adversarial robustness by co-training object recognition models with human eeg, 2024

    2024

  19. [19]

    Generic decoding of seen and imagined objects using hierarchical visual features.Nature communications, 8(1):15037, 2017

    Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features.Nature communications, 8(1):15037, 2017

    2017

  20. [20]

    Hurley and Scott T

    Niall P. Hurley and Scott T. Rickard. Comparing measures of sparsity, 2009

    2009

  21. [21]

    Adversarial examples are not bugs, they are features, 2019

    Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features, 2019

    2019

  22. [22]

    Bridging adversarial robustness and gradient interpretability, 2019

    Beomsu Kim, Junghoon Seo, and Taegyun Jeon. Bridging adversarial robustness and gradient interpretability, 2019

    2019

  23. [23]

    Nikolaus Kriegeskorte and Pamela K. Douglas. Interpreting encoding and decoding models, 2019

    2019

  24. [24]

    Representational similarity analysis-connecting the branches of systems neuroscience.Frontiers in systems neuroscience, 2:249, 2008

    Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity analysis-connecting the branches of systems neuroscience.Frontiers in systems neuroscience, 2:249, 2008

    2008

  25. [25]

    Circu- lar analysis in systems neuroscience: the dangers of double dipping.Nature neuroscience, 12(5):535–540, 2009

    Nikolaus Kriegeskorte, W Kyle Simmons, Patrick SF Bellgowan, and Chris I Baker. Circu- lar analysis in systems neuroscience: the dangers of double dipping.Nature neuroscience, 12(5):535–540, 2009

    2009

  26. [26]

    Imagenet classification with deep convolutional neural networks.Advances in neural information processing systems, 25, 2012

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks.Advances in neural information processing systems, 25, 2012

    2012

  27. [27]

    Adversarial examples are not real features

    Ang Li, Yifei Wang, Yiwen Guo, and Yisen Wang. Adversarial examples are not real features. Advances in Neural Information Processing Systems, 36:17222–17237, 2023

    2023

  28. [28]

    Neuraldiffuser: Neuroscience-inspired diffusion guidance for fmri visual reconstruction.IEEE Transactions on Image Processing, 34:552–565, 2025

    Haoyu Li, Hao Wu, and Badong Chen. Neuraldiffuser: Neuroscience-inspired diffusion guidance for fmri visual reconstruction.IEEE Transactions on Image Processing, 34:552–565, 2025

    2025

  29. [29]

    A closer look at the explainability of contrastive language-image pre-training, 2024

    Yi Li, Hualiang Wang, Yiqun Duan, Jiheng Zhang, and Xiaomeng Li. A closer look at the explainability of contrastive language-image pre-training, 2024

    2024

  30. [30]

    Walker, Erick Cobos, Taliah Muhammad, Jacob Reimer, Matthias Bethge, Fabian H

    Zhe Li, Wieland Brendel, Edgar Y . Walker, Erick Cobos, Taliah Muhammad, Jacob Reimer, Matthias Bethge, Fabian H. Sinz, Xaq Pitkow, and Andreas S. Tolias. Learning from brains how to regularize machines, 2019

    2019

  31. [31]

    Mind reader: Reconstructing complex images from brain activities, 2022

    Sikun Lin, Thomas Sprague, and Ambuj K Singh. Mind reader: Reconstructing complex images from brain activities, 2022

    2022

  32. [32]

    Lawrence Zitnick, and Piotr Dollár

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015

    2015

  33. [33]

    A survey on fmri-based brain decoding for reconstructing multimodal stimuli, 2025

    Pengyu Liu, Guohua Dong, Dan Guo, Kun Li, Fengling Li, Xun Yang, Meng Wang, and Xiaomin Ying. A survey on fmri-based brain decoding for reconstructing multimodal stimuli, 2025

    2025

  34. [34]

    Brainclip: Brain representation via clip for generic natural visual stimulus decoding.IEEE Transactions on Medical Imaging, 44(10):3962–3972, 2025

    Yongqiang Ma, Yulong Liu, Liangjun Chen, Guibo Zhu, Badong Chen, and Nanning Zheng. Brainclip: Brain representation via clip for generic natural visual stimulus decoding.IEEE Transactions on Medical Imaging, 44(10):3962–3972, 2025. 11

    2025

  35. [35]

    Towards deep learning models resistant to adversarial attacks, 2019

    Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks, 2019

    2019

  36. [36]

    Understanding zero-shot adversarial robustness for large-scale models.arXiv preprint arXiv:2212.07016, 2022

    Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl V ondrick. Understanding zero-shot adversarial robustness for large-scale models.arXiv preprint arXiv:2212.07016, 2022

    Pith/arXiv arXiv 2022

  37. [37]

    Neuroai for ai safety, 2025

    Patrick Mineault, Niccolò Zanichelli, Joanne Zichen Peng, Anton Arkhipov, Eli Bingham, Julian Jara-Ettinger, Emily Mackevicius, Adam Marblestone, Marcelo Mattar, Andrew Payne, Sophia Sanborn, Karen Schroeder, Zenna Tavares, Andreas Tolias, and Anthony Zador. Neuroai for ai safety, 2025

    2025

  38. [38]

    Visual image reconstruction from human brain activity using a combination of multiscale local image decoders.Neuron, 60(5):915–929, 2008

    Yoichi Miyawaki, Hajime Uchida, Okito Yamashita, Masa-aki Sato, Yusuke Morito, Hiroki C Tanabe, Norihiro Sadato, and Yukiyasu Kamitani. Visual image reconstruction from human brain activity using a combination of multiscale local image decoders.Neuron, 60(5):915–929, 2008

    2008

  39. [39]

    Brain hierarchy score: Which deep neural networks are hierarchically brain-like?IScience, 24(9), 2021

    Soma Nonaka, Kei Majima, Shuntaro C Aoki, and Yukiyasu Kamitani. Brain hierarchy score: Which deep neural networks are hierarchically brain-like?IScience, 24(9), 2021

    2021

  40. [40]

    Decoding brain representations by multimodal learning of neural activity and visual features.IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(11):3833– 3849, 2020

    Simone Palazzo, Concetto Spampinato, Isaak Kavasidis, Daniela Giordano, Joseph Schmidt, and Mubarak Shah. Decoding brain representations by multimodal learning of neural activity and visual features.IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(11):3833– 3849, 2020

    2020

  41. [41]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021

    2021

  42. [42]

    Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients

    Andrew Ross and Finale Doshi-Velez. Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients. InProceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  43. [43]

    Robust clip: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models.arXiv preprint arXiv:2402.12336, 2024

    Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust clip: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models.arXiv preprint arXiv:2402.12336, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Ethan Cohen, Aidan J

    Paul S. Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Ethan Cohen, Aidan J. Dempster, Nathalie Verlinde, Elad Yundler, David Weisberg, Kenneth A. Norman, and Tanishq Mathew Abraham. Reconstructing the mind’s eye: fmri-to-image with contrastive learning and diffusion priors, 2023

    2023

  45. [45]

    Deep image recon- struction from human brain activity.PLoS computational biology, 15(1):e1006633, 2019

    Guohua Shen, Tomoyasu Horikawa, Kei Majima, and Yukiyasu Kamitani. Deep image recon- struction from human brain activity.PLoS computational biology, 15(1):e1006633, 2019

    2019

  46. [46]

    Spurious reconstruction from brain activity.Neural Networks, 190:107515, 2025

    Ken Shirakawa, Yoshihiro Nagano, Misato Tanaka, Shuntaro C Aoki, Yusuke Muraki, Kei Majima, and Yukiyasu Kamitani. Spurious reconstruction from brain activity.Neural Networks, 190:107515, 2025

    2025

  47. [47]

    Spatial-frequency channels, shape bias, and adversarial robustness.Advances in neural information processing systems, 36:4137–4149, 2023

    Ajay Subramanian, Elena Sizikova, Najib Majaj, and Denis Pelli. Spatial-frequency channels, shape bias, and adversarial robustness.Advances in neural information processing systems, 36:4137–4149, 2023

    2023

  48. [48]

    High-resolution image reconstruction with latent diffusion models from human brain activity

    Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14453–14463, 2023

    2023

  49. [49]

    Robustness may be at odds with accuracy, 2019

    Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy, 2019

    2019

  50. [50]

    Brain-aligning of semantic vectors improves neural decoding of visual stimuli.Communications Biology, 2026

    Shirin Vafaei, Ryohei Fukuma, Takufumi Yanagisawa, Huixiang Yang, Satoru Oshino, Naoki Tani, Hui Ming Khoo, Hidenori Sugano, Yasushi Iimura, Hiroharu Suzuki, et al. Brain-aligning of semantic vectors improves neural decoding of visual stimuli.Communications Biology, 2026. 12

    2026

  51. [51]

    Representation learning with contrastive predictive coding, 2019

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2019

    2019

  52. [52]

    Subtle adversarial image manipulations influence both human and machine perception.Nature communications, 14(1):4933, 2023

    Vijay Veerabadran, Josh Goldman, Shreya Shankar, Brian Cheung, Nicolas Papernot, Alexey Kurakin, Ian Goodfellow, Jonathon Shlens, Jascha Sohl-Dickstein, Michael C Mozer, et al. Subtle adversarial image manipulations influence both human and machine perception.Nature communications, 14(1):4933, 2023

    2023

  53. [53]

    A sober look at the robustness of clips to spurious features, 2024

    Qizhou Wang, Yong Lin, Yongqiang Chen, Ludwig Schmidt, Bo Han, and Tong Zhang. A sober look at the robustness of clips to spurious features, 2024

    2024

  54. [54]

    Mindbridge: A cross-subject brain decoding framework

    Shizun Wang, Songhua Liu, Zhenxiong Tan, and Xinchao Wang. Mindbridge: A cross-subject brain decoding framework. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11333–11342, 2024

    2024

  55. [55]

    Representation, alignment, and generation: A comprehensive survey of foundation models for non-invasive brain decoding

    Yifan Wang, Shaonan Wang, Yunhao Zhang, Changde Du, Cunhang Fan, Dongyang Li, Hong- peng Zhou, Hongyu Zhang, Jixing Li, Quanying Liu, et al. Representation, alignment, and generation: A comprehensive survey of foundation models for non-invasive brain decoding. bioRxiv, pages 2025–11, 2025

    2025

  56. [56]

    Bridging the vision-brain gap with an uncertainty-aware blur prior, 2025

    Haitao Wu, Qing Li, Changqing Zhang, Zhen He, and Xiaomin Ying. Bridging the vision-brain gap with an uncertainty-aware blur prior, 2025

    2025

  57. [57]

    Prince, John A

    Muquan Yu, Mu Nan, Hossein Adeli, Jacob S. Prince, John A. Pyles, Leila Wehbe, Margaret M. Henderson, Michael J. Tarr, and Andrew F. Luo. Meta-learning an in-context transformer model of human higher visual cortex, 2025

    2025

  58. [58]

    Causaladv: Adversarial robustness through the lens of causality, 2022

    Yonggang Zhang, Mingming Gong, Tongliang Liu, Gang Niu, Xinmei Tian, Bo Han, Bernhard Schölkopf, and Kun Zhang. Causaladv: Adversarial robustness through the lens of causality, 2022. A Experiment Details A.1 Training Scheme Implementation Details.We provide additional implementation details for training the linear brain decoder. Our training protocol foll...

    2022

  59. [59]

    For this setting, subjects 1, 2, 5, and 7 have full fMRI trials (3 repeated)

    introduced a data split to make sure 3 repeated betas for the test dataset, we arranged the test set back to the original NSD test set [1, 34], which contains subject-shared 982 image stimuli and corresponding fMRI betas trials to test the consistency of the results on the same stimuli. For this setting, subjects 1, 2, 5, and 7 have full fMRI trials (3 re...

  60. [60]

    All the attribution maps are generated from Subject 01. Input CLIP FARE-2 FARE-4 CLIP - FARE-2 CLIP - FARE-4 Input CLIP T eCoA-2 T eCoA-4 CLIP - T eCoA-2 CLIP - T eCoA-4 Figure 10: (Left) Attribution maps of 5 randomly selected images from the test dataset. FARE-2 and FARE-4 were selected for the comparison between different adversarial training strengths...