Pith. sign in

REVIEW 3 major objections 5 minor 76 references

Sparse Autoencoders Bridge The Deep Learning Model and The Brain

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Sparse autoencoder units trained without brain data match human visual cortex fMRI responses, with cosine similarity up to 0.76.

desk verdict Genuinely new SAE-to-fMRI alignment method whose headline 0.76 similarity lacks any null baseline, so the core claim overreaches but the tool is worth referee time. read the letter →

arxiv 2506.11123 v1 pith:W4KJJDZD submitted 2025-06-10 q-bio.NC cs.CV

Ziming Mao , Jia Xu , Zeqi Zheng , Haofang Zheng , Dabing Sheng , Yaochu Jin , Guoyuan Yang This is my paper
classification q-bio.NCcs.CV
keywords sparseautoencodersmodel-brainalignmentventralvisualpathwayfMRIresponsescortexrepresentationalsimilarityvoxeldictionaryinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that sparse autoencoders trained purely on vision-model activations, with no brain data, produce units whose activation patterns track voxel-level fMRI responses in the human ventral visual pathway when both are elicited by the same natural images; the maximum cosine similarity reaches 0.76. On this basis it builds a voxel dictionary that assigns each voxel its most similar SAE feature, showing that SAE units preserve the functional organization of known brain regions and display selectivity consistent with those regions. It then maps each model layer to cortical areas and uses the brain as a spatial template to trace how visual information transforms across layers. If correct, this yields a direct, downstream-task-free bridge between deep networks and the human visual cortex, using features learned without supervision from brain responses.

What carries the argument

The central object is a sparse autoencoder trained per layer with tied encoder-decoder weights, reconstructing activations as sparse positive combinations of interpretable feature units: $Z = \mathrm{ReLU}(WA+b)$, with loss $\|\hat{A}-A\|_2^2 + \alpha\|Z\|_1$. The load-bearing comparison is the cosine similarity between z-scored SAE unit activations and z-scored voxel fMRI responses across the shared image set, $M^{(j)}_{\mathrm{simi}} = \cos(B^{\top}/\|B\|_2, Z^{(j)}_{\mathrm{zscore}}/\|Z^{(j)}_{\mathrm{zscore}}\|_2)$, normalized over images. This similarity matrix drives two constructions: a voxel dictionary $D^{(j)}[x] = W[\arg\max_y M_{x,y}]$ assigning each voxel its most similar feature, and a voxel-wise layer alignment $C[x] = \arg\max_j \max_y M^{(j)}_{x,y}$. Projecting dictionary activations $S^{(j)} = A^{(j)}D^{(j)}$ onto the cortex lets the brain act as a spatial template for tracing information flow through the model.

What would settle it

A permutation test that shuffles the voxel-unit correspondence, or replaces trained SAE features with random features, and recomputes the maximum cosine similarities; if random features still yield maxima around 0.7, the claimed strong correspondence would reflect shared stimulus statistics rather than learned representations.

Watch

Extended reading notes

Core claim

Training layer-wise sparse autoencoders on model activations from natural images yields features whose z-scored activation patterns correlate strongly with z-scored fMRI responses of individual voxels, with cosine similarity up to 0.76. The most similar SAE feature per voxel forms a 'voxel dictionary' that preserves ROI functional structure, and the best-matching layer per voxel reconstructs the classic hierarchy, with early layers aligning to early visual cortex and deeper layers to higher-level visual areas. For CLIP ViT-B/16, the voxel-dictionary activation path suggests low-level features are transformed into abstract semantic information in early layers and later reconstructed alongside high-level information. The paper claims this constitutes a direct, downstream-task-free alignment between deep neural network representations and the human visual cortex.

Load-bearing premise

The load-bearing premise is that cosine similarity between activation profiles in response to the same images measures functional correspondence; the paper reports maxima and averages without a null distribution, so chance alignment due to shared image statistics is not ruled out.

Editorial extensions

If this is right

  • SAE features learned from images alone can be matched to individual fMRI voxels without any supervised brain encoder, providing a training-free voxel-wise alignment procedure.
  • The voxel dictionary inherits known ROI structure, so model features can be probed for face, body, scene, word, and food selectivity in a data-driven way.
  • Layer-to-cortex mapping reproduces the ventral-stream hierarchy, with early layers near V1 and later layers near high-level visual areas, across several architectures and training objectives.
  • For CLIP ViT-B/16, early layers convert low-level information into abstract semantic content while later layers re-integrate low-level details, a pattern the paper links to the residual stream.
  • The framework extends to any vision model with per-layer activations, making it a general tool for comparing model internals against human visual cortex.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported 0.76 maximum is selected across thousands of voxel-unit-layer comparisons, and the paper gives no permutation or null baseline; a natural test is whether randomly initialized SAEs or shuffled voxel labels produce similar maxima driven by shared stimulus statistics.
  • If the alignment survives such nulls, the method becomes a zero-shot cortical mapper: any new model layer could be localized on the cortex without training a brain encoder, as long as paired image-fMRI data are available.
  • The ROI-consistent selectivity of SAE units could be turned into a generative stimulus-search tool, using units tied to face, scene, or word regions to synthesize images and probe cortical selectivity directly.
  • Applying the same framework to recurrent or multi-scale architectures might reveal whether the bidirectional low- and high-level information flow observed in CLIP ViT-B/16 is specific to transformer residual streams or a general property of deep visual processing.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces SAE-BrainMap, a framework that trains sparse autoencoders (SAEs) on the layer activations of several vision models (ViT-B/16 CLIP, ImageNet, MAE, DiNOv2, SAM, ResNet50 CLIP) without any brain data, and then correlates the SAE unit activation profiles with voxel-level fMRI responses from the Natural Scenes Dataset using cosine similarity. The authors report a maximum similarity of 0.76, construct a voxel dictionary by assigning to each voxel the most similar SAE unit, and use this dictionary to argue that SAE units preserve the functional structure of visual ROIs, to establish a layer-wise brain-model alignment, and to visualize model information processing on the cortical surface. The main claims are that SAE features provide a direct, downstream-task-free bridge between deep network representations and the human ventral visual pathway.

Significance. If the central similarity result is statistically validated, the paper offers a conceptually attractive approach: features extracted by SAEs trained purely on model activations, with no supervised brain-encoding objective, might align voxel-wise with cortical fMRI responses. The work is also broad in scope, covering multiple architectures, public data, and several analyses, and the SAEs themselves are trained without brain data, which avoids the most obvious form of circularity. However, the key quantitative claims currently lack the statistical baselines needed to distinguish genuine structure from shared stimulus statistics and selection effects. The paper is therefore potentially interesting and publishable in a revised form, but the headline 'strong activation correspondence' is not yet supported by the evidence as presented.

major comments (3)
  1. [Section 3.2, Eq. (4); Table 1] The headline maximum similarity of 0.76 is the maximum over subjects, voxels, SAE units, layers, and models, and Table 1 reports no null distribution, p-values, or error bars for either the maximum or the per-voxel average (~0.26). Because natural images share low-level statistics (luminance, edges, spatial frequency), positive cosine similarities between model features and voxel responses are expected even with no true model-brain correspondence, and the maximum of an enormous search is especially uninterpretable. The paper's own NeurIPS checklist states that statistical significance is illustrated in Tables 1-3, but those tables contain no error bars or significance tests. The authors should provide a permutation or surrogate baseline (e.g., shuffling image labels, using units from untrained models, or an analytic null for the maximum of many correlations) and report selection-corrected statistics; without this, the abstract's 'strong activation correspondence' and 'direct bridge' overstate what Eq. (4) demonstrates.
  2. [Section 4.3, Eq. (6); Table 2] The claim that SAE units preserve the functional structure of ROIs is validated by computing an RSA score between the voxel dictionary and 'brain encoder weights' that were themselves trained to predict fMRI responses from the same model activations (Appendix C). Since the voxel dictionary is constructed by an argmax over the same similarity matrix used throughout the paper, and since both the dictionary and the ground-truth encoder weights derive from the same model activation statistics, the reported RSA values (highest 0.516) may reflect shared ancestry rather than an independent correspondence between SAE features and brain function. The authors should validate the dictionary against held-out fMRI data, or against an independently defined functional atlas, and should also report a null distribution for the RSA scores.
  3. [Section 4.4, Eq. (5); Table 3] The brain-model layer alignment C is obtained by taking an argmax over layers of the per-layer maximum over SAE units, and the 'slope' metric is computed on this selected alignment without any null model. The similarities to FactorTopy and Max R2 are presented without confidence intervals, and the text states that the results are 'well correlated with previous methods' without a statistical comparison. The authors should report bootstrap intervals across subjects and voxels, and compare the slope and alignment similarity against a permutation baseline in which layer labels are shuffled, before claiming a 'clear and consistent hierarchical correspondence' between model layers and the human visual cortex.
minor comments (5)
  1. [Throughout] There are numerous typos and grammatical errors that should be corrected: 'Relate Work' (Section 2), 'constrainted' (Introduction), 'dosen't' (checklist item 3), 'V oxels' (multiple occurrences), 'V oxel Dictionary' (figure captions), and inconsistent capitalization throughout.
  2. [Section 3.1, Eqs. (1)-(4)] The notation is inconsistent: Eq. (1) defines Z with a ReLU nonlinearity, but Eq. (4) computes Z_zscore = Zscore(W A + b) without the ReLU. The definition of the norm in Eq. (4) is also ambiguous (Frobenius norm vs. column-wise norms), and the matrix dimensions in the cosine expression should be stated explicitly.
  3. [Section 4.3, Figure 5 and Figure 4] The text refers to 'Figure 4' for the unit selectivity visualization and for the voxel dictionary / brain encoder similarity matrix, but the relevant figures appear to be Figures 4 and 5 in the paper; the references should be corrected and the figures should be cross-checked.
  4. [Section 4.4, Table 3] The text says 'Table 2 presents the cosine similarity between our alignment results and two voxel-wise mapping methods,' but the table referenced is Table 3. The similarity columns are also not clearly defined: the reader should be told whether the values are correlations over voxel labels, IoU-like scores, or something else.
  5. [Section 5, Limitations] The limitations section mentions only the fixed SAE hyperparameters. It does not mention the lack of statistical baselines, the selection over many comparisons, or the fact that the brain-encoder validation is model-dependent. These are important limitations that should be acknowledged.

Circularity Check

2 steps flagged · score 6.0 of 10

Partial circularity: the ROI-preservation claim and the 'ground-truth' brain-encoder RSA validation reuse the same similarity/fitted data; the headline 0.76 SAE-fMRI similarity itself is not circular.

  1. self definitional [Section 3.2 Equation 6 and Section 4.3]
    "D(j)[x] = W[argmax_y M_{x,y}], x in {0,1,...,v} (Eq. 6). ... 'By computing the representation correlation matrix based on pairwise cosine similarities, we find that the voxel dictionary preserves functional separability across ROIs and shares a high RSA score with the ground truth (highest: 0.516, see Table 2).'"

    The voxel dictionary is defined by assigning to each voxel the SAE unit whose activation profile is most similar to that voxel's fMRI profile, using the same cosine-similarity matrix M computed from B and Z. Voxels inside a predefined ROI share similar fMRI response profiles by functional definition, so the argmax assignment tends to select the same or similar SAE dictionary rows for voxels in the same ROI. The reported pairwise similarity structure of D, and hence the 'preserved functional separability across ROIs,' is inherited from the construction input B rather than being an independent confirmation that SAE units discovered the brain's ROI organization.

  2. fitted input called prediction [Table 2 caption / Section 4.3 / Appendix C]
    "'RSA Similarity Score between the brain encoder weights of the last layer and the voxel dictionary. This indicates that SAE units maintain a strong structural correspondence with the brain encoder weights, which are regarded as the ground truth representation of cortical functional structure.' Appendix C: 'training a linear layer with equation: A(j)W + b = B'."

    The 'ground truth' brain encoder weights are themselves fitted by linear regression to predict the same voxel fMRI responses B from the same model activations. The voxel dictionary is built from M, the voxel-unit similarity matrix computed from the same B and model activations. Both sides of the RSA comparison therefore share their target data and are aligned to B; the high RSA score is a consistency check between two quantities fitted to the same signal, not validation against an independent ground truth. Naming the fitted encoder weights 'ground truth' makes the claimed structural correspondence partially circular.

full rationale

The paper's central alignment step is not circular: layer-wise SAEs are trained on ImageNet activations (Equations 1-3) without any brain data, so the reported cosine similarities between SAE unit activation profiles and voxel fMRI profiles (Equation 4) are a genuine out-of-sample comparison. The core 0.76 maximum similarity is not forced by construction, though it lacks a null baseline. However, two load-bearing validation claims reduce to their own inputs. First, the voxel dictionary D(j)[x] = W[argmax_y M_{x,y}] is defined by the very similarity matrix M that already encodes each voxel's fMRI profile; voxels in the same predefined ROI share fMRI response structure, so the dictionary's within-ROI pairwise similarity (the 'preserved functional separability' in Section 4.3) is a restatement of the construction rather than an independent discovery. Second, the RSA validation in Table 2 treats a linear brain encoder fitted to the same B (Appendix C: A(j)W + b = B) as 'ground truth'; both the dictionary and the encoder weights are derived from the same model-brain association data, so their high RSA is a shared-input consistency check, not external confirmation. The lack of a permutation null for the maximum 0.76 is a statistical-significance concern rather than a circularity, and is not scored here. Overall, the 'direct bridge' similarity claim is non-circular, but the ROI-preservation and ground-truth correspondence claims are partially circular, giving a score of 6.

Assumptions & free parameters 4 free parameters · 6 assumptions · 1 invented entities

The central quantitative claims depend on hand-selected SAE hyperparameters (R, alpha, training schedule), on the assumption that cosine similarity is a meaningful correspondence measure, and on a voxel dictionary whose validation is partly tied to a fitted brain encoder. No new physical entities are introduced; the voxel dictionary is a methodological construct.

free parameters (4)
  • SAE expansion factor R = 16
    Sets the SAE hidden dimension k = R*d for every layer and model. Chosen by hand, not tuned against brain data; the paper's limitation section notes the effect of this choice is unexplored.
  • SAE sparsity coefficient alpha = 0.00086
    Taken from the prior SAE paper [18] and kept fixed for all models and layers; it controls the L1 penalty in Equation 3 and therefore the feature dictionary the whole alignment depends on.
  • SAE training schedule = 3 epochs, lr 5e-5, AdamW
    Hand-selected training configuration; the limitation section explicitly says epochs and training dataset effects were not studied.
  • brain encoder linear weights = per model, per layer, per subject
    Used as the 'ground truth' for voxel dictionary RSA validation (Section 4.3). These are fitted to fMRI data with a linear map, so any claim they validate is partly circular.
assumptions (6)
  • domain assumption SAE features are interpretable, monosemantic units that capture the model's functional processing.
    Inherited from the SAE interpretability literature [18,19] and used as motivation in Section 2 and the abstract.
  • domain assumption fMRI z-scored voxel responses are stable, meaningful measures of cortical function.
    The paper relies on NSD beta values z-scored across runs (Section 4.1), which is a standard but assumed preprocessing.
  • domain assumption Cosine similarity between per-image activation vectors is a valid measure of brain-model correspondence.
    Equation 4 is the core alignment step; no null model, calibration, or permutation test is provided to justify the scale of these similarities.
  • domain assumption The CLS token (or averaged patch tokens for SAM) is a sufficient model representation for voxel-level alignment.
    SAEs are trained only on this token (Appendix G); if the token discards spatial or low-level information, the bridge would miss those cortical responses.
  • ad hoc to paper Brain encoder weights constitute ground truth brain function.
    Section 4.3 states this directly; the weights are trained linear maps from model activations to fMRI, so treating them as ground truth builds a fitted model into the validation.
  • ad hoc to paper Voxel dictionary argmax assignment preserves ROI functional structure.
    Equation 6 defines the dictionary by maximizing similarity; using the same matrix to demonstrate ROI preservation (Section 4.3) is partly constructional.
invented entities (1)
  • Voxel dictionary D^(j)
    purpose: Assigns each cortical voxel the SAE feature with the most similar activation pattern, then projects model activations onto the cortical surface (Equations 6 and 7).
    It is a new construct defined entirely by the similarity matrix. No external benchmark or independent measurement validates the assignment; the RSA check uses a brain encoder fitted from the same model features.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sparse Autoencoders Bridge The Deep Learning Model and The Brain." pith.science (2026). https://pith.science/paper/W4KJJDZD

@misc{pith2026250611123,
  author       = {Pith},
  title        = {Pith review of: Sparse Autoencoders Bridge The Deep Learning Model and The Brain},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W4KJJDZD}},
  note         = {Machine review of arXiv:2506.11123}
}
abstract

We present SAE-BrainMap, a novel framework that directly aligns deep learning visual model representations with voxel-level fMRI responses using sparse autoencoders (SAEs). First, we train layer-wise SAEs on model activations and compute the correlations between SAE unit activations and cortical fMRI signals elicited by the same natural image stimuli with cosine similarity, revealing strong activation correspondence (maximum similarity up to 0.76). Depending on this alignment, we construct a voxel dictionary by optimally assigning the most similar SAE feature to each voxel, demonstrating that SAE units preserve the functional structure of predefined regions of interest (ROIs) and exhibit ROI-consistent selectivity. Finally, we establish fine-grained hierarchical mapping between model layers and the human ventral visual pathway, also by projecting voxel dictionary activations onto individual cortical surfaces, we visualize the dynamic transformation of the visual information in deep learning models. It is found that ViT-B/16$_{CLIP}$ tends to utilize low-level information to generate high-level semantic information in the early layers and reconstructs the low-dimension information later. Our results establish a direct, downstream-task-free bridge between deep neural networks and human visual cortex, offering new insights into model interpretability.

Figures

Figures reproduced from arXiv: 2506.11123 by the authors.

Figure 1
Figure 1. Architecture of SAE-BrainMap. First, our framework first trains Sparse Autoencoders for each model layer. Second, it computes cosine similarity matrix between SAE units activations and cortical fMRI responses. Finally, it interprets deep learning model with brain based on the cosine similarity matrix. voxel-encoding RSA (veRSA) [7, 9], further improves alignment by linearly reweighting DNN features to predict voxel-… view at source ↗
Figure 2
Figure 2. Visualizing S5 selective ROIs Ventral Visual Pathway Brain processes visual infor￾mation in a hierarchical manner [1, 35, 36, 37]. Visual information starts from the early visual cortex V1, and progresses through V2, V3, and hV4. Voxels along this pathway exhibit increasingly larger receptive fields [38] and transit from encoding low-level visual features such as edges and curvature [39, 40] to processing more compl… view at source ↗
Figure 3
Figure 3. Brain-SAEs Activation Similarity on S5. Left: voxel-wise maximum similarity score across models. Right: difference in similarity scores between models and the voxel-wise maximum Subject All Model ViT-B/16CLIP ImageNet MAE DiNOv2 SAM ResNet50CLIP Max Avg Max Avg Max Avg Max Avg Max Avg Max Avg Max Avg S1 0.762 0.259 0.618 0.249 0.503 0.220 0.605 0.239 0.484 0.231 0.421 0.212 0.644 0.244 S2 0.716 0.265 0.573 0.254 0.4… view at source ↗
Figures from the paper (37 more)
Figure 4
Figure 4. Figure 4: Representation Similarity Matrix between the brain encoder weights of the last layer of ViT-B/16CLIP, the voxel dictionary, and their mutual similarity. The voxel dictionary preserves the brain’s functional structure and share a high RSA score with brain encoder weight…
Figure 5
Figure 5. Figure 5: Visualization of Unit Selectivity and Functionality. Top: Top-5 activated images for units with highest similarity to voxels in each ROI on ViT-B/16CLIP, demonstrating consistent selectivity for faces, bodies, scenes, words, and food. Bottom: Cortical visualization of …
Figure 6
Figure 6. Figure 6: Brain-Model Layer Alignment. Top: Layer alignment for S1 based on SAE-BrainMap, FactorTopy [2] and Max R2 [14], we visualize five models with attention structure and have 12 layers. Bottom: SAE-BrainMap layer alignment on S5. ViT-B/16CLIP ImageNet MAE DiNOv2 SAM S1 S5 …
Figure 7
Figure 7. Figure 7: Top: Layer wise Voxel Dictionary average activation for ViT-B/16CLIP, S5. This reveals the model forms low-level information first, then generates high-level information while several low-level information disappeared in the middle layers and was reconstructed in the l…
Figure 8
Figure 8. Figure 8: The CKA result for ViT-B/16CLIP and ResNet50CLIP on top left corner. We com￾pare two models’ Model-Brain Layer align￾ment similarity on the top right corner, and visualize the Layer 5 of ResNet50CLIP and Layer 9 of ViT-B/16CLIP on the bottom. We compare the CKA matrix …
Figure 9
Figure 9. Figure 9: Brain-SAEs Activation Similarity on S1-4. Left: voxel-wise maximum similarity score across models. Right: difference in similarity scores between models and the voxel-wise maximum. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Brain-SAEs Activation Similarity on S5-8. Left: voxel-wise maximum similarity score across models. Right: difference in similarity scores between models and the voxel-wise maximum. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Brain-Model Layer Alignment. Layer alignment for S1 and S2 based on SAE-BrainMap, FactorTopy [2] and Max R2 [14], we visualize five models with attention structure and have 12 layers. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Brain-Model Layer Alignment. Layer alignment for S3 and S4 based on SAE-BrainMap, FactorTopy [2] and Max R2 [14], we visualize five models with attention structure and have 12 layers. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Brain-Model Layer Alignment. Layer alignment for S5 and S6 based on SAE-BrainMap, FactorTopy [2] and Max R2 [14], we visualize five models with attention structure and have 12 layers. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 14
Figure 14. Figure 14: Brain-Model Layer Alignment. Layer alignment for S7 and S8 based on SAE-BrainMap, FactorTopy [2] and Max R2 [14], we visualize five models with attention structure and have 12 layers. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]
Figure 15
Figure 15. Figure 15: Representation Similarity Matrix between the brain encoder weights of the last layer of voxel dictionary, and their mutual similarity on Subject 5. The model from top to bottom is ResNet50CLIP, DiNOv2, ImageNet, MAE, SAM and ViT-B/16CLIP 22 [PITH_FULL_IMAGE:figures/f…
Figure 16
Figure 16. Figure 16: Units Selectivity for S5 The top-5 images whose "CLS" token maximize activate the last layer’s units that selected by the brain ROIs. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_16.png]
Figure 17
Figure 17. Figure 17: Units Selectivity for S1 The top-5 images whose "CLS" token maximize activate the last layer’s units that selected by the brain ROIs. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_17.png]
Figure 18
Figure 18. Figure 18: Units Selectivity for S2 The top-5 images whose "CLS" token maximize activate the last layer’s units that selected by the brain ROIs. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_18.png]
Figure 19
Figure 19. Figure 19: Units Selectivity for S3 The top-5 images whose "CLS" token maximize activate the last layer’s units that selected by the brain ROIs. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_19.png]
Figure 20
Figure 20. Figure 20: Units Selectivity for S4 The top-5 images whose "CLS" token maximize activate the last layer’s units that selected by the brain ROIs. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_20.png]
Figure 21
Figure 21. Figure 21: Units Selectivity for S6 The top-5 images whose "CLS" token maximize activate the last layer’s units that selected by the brain ROIs. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_21.png]
Figure 22
Figure 22. Figure 22: Units Selectivity for S7 The top-5 images whose "CLS" token maximize activate the last layer’s units that selected by the brain ROIs. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_22.png]
Figure 23
Figure 23. Figure 23: Units Selectivity for S8 The top-5 images whose "CLS" token maximize activate the last layer’s units that selected by the brain ROIs. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_23.png]
Figure 24
Figure 24. Figure 24: Representation Similarity Matrix on S1 between the brain encoder weights of the last layer of voxel dictionary, and their mutual similarity. The model from top to bottom is ResNet50CLIP, DiNOv2, ImageNet, MAE, SAM and ViT-B/16CLIP 31 [PITH_FULL_IMAGE:figures/full_fig…
Figure 25
Figure 25. Figure 25: Representation Similarity Matrix on S2 between the brain encoder weights of the last layer of voxel dictionary, and their mutual similarity. The model from top to bottom is ResNet50CLIP, DiNOv2, ImageNet, MAE, SAM and ViT-B/16CLIP 32 [PITH_FULL_IMAGE:figures/full_fig…
Figure 26
Figure 26. Figure 26: Representation Similarity Matrix on S3 between the brain encoder weights of the last layer of voxel dictionary, and their mutual similarity. The model from top to bottom is ResNet50CLIP, DiNOv2, ImageNet, MAE, SAM and ViT-B/16CLIP 33 [PITH_FULL_IMAGE:figures/full_fig…
Figure 27
Figure 27. Figure 27: Representation Similarity Matrix on S4 between the brain encoder weights of the last layer of voxel dictionary, and their mutual similarity. The model from top to bottom is ResNet50CLIP, DiNOv2, ImageNet, MAE, SAM and ViT-B/16CLIP 34 [PITH_FULL_IMAGE:figures/full_fig…
Figure 28
Figure 28. Figure 28: Representation Similarity Matrix on S6 between the brain encoder weights of the last layer of voxel dictionary, and their mutual similarity. The model from top to bottom is ResNet50CLIP, DiNOv2, ImageNet, MAE, SAM and ViT-B/16CLIP 35 [PITH_FULL_IMAGE:figures/full_fig…
Figure 29
Figure 29. Figure 29: Representation Similarity Matrix on S7 between the brain encoder weights of the last layer of voxel dictionary, and their mutual similarity. The model from top to bottom is ResNet50CLIP, DiNOv2, ImageNet, MAE, SAM and ViT-B/16CLIP 36 [PITH_FULL_IMAGE:figures/full_fig…
Figure 30
Figure 30. Figure 30: Representation Similarity Matrix on S8 between the brain encoder weights of the last layer of voxel dictionary, and their mutual similarity. The model from top to bottom is ResNet50CLIP, DiNOv2, ImageNet, MAE, SAM and ViT-B/16CLIP 37 [PITH_FULL_IMAGE:figures/full_fig…
Figure 31
Figure 31. Figure 31: Layer wise Voxel Dictionary average activation for ImageNet, DiNOv2, MAE, SAM for [PITH_FULL_IMAGE:figures/full_fig_p038_31.png]
Figure 32
Figure 32. Figure 32: Layer wise Voxel Dictionary average activation for ImageNet, DiNOv2, MAE, SAM for [PITH_FULL_IMAGE:figures/full_fig_p039_32.png]
Figure 33
Figure 33. Figure 33: Layer wise Voxel Dictionary average activation for ImageNet, DiNOv2, MAE, SAM for [PITH_FULL_IMAGE:figures/full_fig_p040_33.png]
Figure 34
Figure 34. Figure 34: Layer wise Voxel Dictionary average activation for ImageNet, DiNOv2, MAE, SAM for [PITH_FULL_IMAGE:figures/full_fig_p041_34.png]
Figure 35
Figure 35. Figure 35: Layer wise Voxel Dictionary average activation for ImageNet, DiNOv2, MAE, SAM for [PITH_FULL_IMAGE:figures/full_fig_p042_35.png]
Figure 36
Figure 36. Figure 36: Layer wise Voxel Dictionary average activation for ImageNet, DiNOv2, MAE, SAM for [PITH_FULL_IMAGE:figures/full_fig_p043_36.png]
Figure 37
Figure 37. Figure 37: Layer wise Voxel Dictionary average activation for ImageNet, DiNOv2, MAE, SAM for [PITH_FULL_IMAGE:figures/full_fig_p044_37.png]
Figure 38
Figure 38. Figure 38: Layer wise Voxel Dictionary average activation for ImageNet, DiNOv2, MAE, SAM for [PITH_FULL_IMAGE:figures/full_fig_p045_38.png]
Figure 39
Figure 39. Figure 39: Layer wise Voxel Dictionary average activation for ViT-B/16 [PITH_FULL_IMAGE:figures/full_fig_p046_39.png]
Figure 40
Figure 40. Figure 40: Layer wise Voxel Dictionary average activation for ViT-B/16 [PITH_FULL_IMAGE:figures/full_fig_p047_40.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

76 extracted references · 63 canonical work pages

  1. [1]

    DiCarlo and David D

    James J. DiCarlo and David D. Cox. Untangling invariant object recognition. Trends in Cognitive Sciences, 11(8):333–341, August 2007. ISSN 13646613. doi: 10.1016/j.tics.2007.06.010

    doi:10.1016/j.tics.2007.06.010 2007

  2. [2]

    Brain Decodes Deep Nets, March 2024

    Huzheng Yang, James Gee, and Jianbo Shi. Brain Decodes Deep Nets, March 2024

    work page 2024

  3. [3]

    Coggan, Luke A

    David D. Coggan, Luke A. Allen, Oliver R. H. Farrar, et al. Differences in selectivity to natural images in early visual areas (v1–v3). Scientific Reports, 7:2444, 2017. doi: 10.1038/s41598-017-02569-4

    work page doi:10.1038/s41598-017-02569-4 2017

  4. [4]

    CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networks, June 2023

    Tuomas Oikarinen and Tsui-Wei Weng. CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networks, June 2023

    work page 2023

  5. [5]

    Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation Engineering: A Top-Down Approach to A...

    work page 2023

  6. [6]

    BrainACTIV: Identifying visuo-semantic properties driving cortical selectivity using diffusion-based image manipulation, October 2024

    Diego García Cerdas, Christina Sartzetaki, Magnus Petersen, Gemma Roig, Pascal Mettes, and Iris Groen. BrainACTIV: Identifying visuo-semantic properties driving cortical selectivity using diffusion-based image manipulation, October 2024

    work page 2024

  7. [8]

    Limits to visual representational correspondence between convolutional neural networks and the human brain

    Yaoda Xu and Maryam Vaziri-Pashkam. Limits to visual representational correspondence between convolutional neural networks and the human brain. Nature Communications, 12(1):2065, April 2021. ISSN 2041-1723. doi: 10.1038/s41467-021-22244-7

    work page doi:10.1038/s41467-021-22244-7 2021

  8. [9]

    Fixed versus mixed RSA: Explaining visual representations by fixed and mixed feature sets from shallow and deep computational models

    Seyed-Mahdi Khaligh-Razavi, Linda Henriksson, Kendrick Kay, and Nikolaus Kriegeskorte. Fixed versus mixed RSA: Explaining visual representations by fixed and mixed feature sets from shallow and deep computational models. Journal of Mathematical Psychology, 76:184–197, February 2017. ISSN 0022-2496. doi: 10.1016/j.jmp.2016.10.007

    work page doi:10.1016/j.jmp.2016.10.007 2017

Show all 76 references
  1. [10]

    Neural tuning and representational geometry

    Nikolaus Kriegeskorte and Xue-Xin Wei. Neural tuning and representational geometry. Nature Re- views Neuroscience, 22(11):703–718, November 2021. ISSN 1471-003X, 1471-0048. doi: 10.1038/ s41583-021-00502-3

    2021

  2. [12]

    Luo, Margaret M

    Andrew F. Luo, Margaret M. Henderson, Michael J. Tarr, and Leila Wehbe. BrainSCUBA: Fine-Grained Natural Language Captions of Visual Cortex Selectivity, May 2024

    2024

  3. [13]

    Luo, Jacob Yeung, Rushikesh Zawar, Shaurya Dewan, Margaret M

    Andrew F. Luo, Jacob Yeung, Rushikesh Zawar, Shaurya Dewan, Margaret M. Henderson, Leila Wehbe, and Michael J. Tarr. Brain Mapping with Dense Features: Grounding Cortical Semantic Selectivity in Natural Images With Vision Transformers, October 2024

    2024

  4. [14]

    Wang, Kendrick Kay, Thomas Naselaris, Michael J

    Aria Y . Wang, Kendrick Kay, Thomas Naselaris, Michael J. Tarr, and Leila Wehbe. Better models of human high-level visual cortex emerge from natural language supervision with a large and diverse dataset. Nature Machine Intelligence, 5(12):1415–1426, November 2023. ISSN 2522-58...

    2023

  5. [15]

    Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Models

    Andrew F Luo, Margaret M Henderson, Leila Wehbe, and Michael J Tarr. Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Models. 10

  6. [16]

    Do Vision Transformers See Like Convolutional Neural Networks? In Advances in Neural Information Processing Systems, volume 34, pages 12116–12128

    Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do Vision Transformers See Like Convolutional Neural Networks? In Advances in Neural Information Processing Systems, volume 34, pages 12116–12128. Curran Associates, Inc., 2021

    2021

  7. [17]

    CLIP-MSM: A Multi-Semantic Mapping Brain Representation for Human High-Level Visual Cortex

    Guoyuan Yang, Mufan Xue, Ziming Mao, Haofang Zheng, Jia Xu, Dabin Sheng, Ruotian Sun, Ruoqi Yang, and Xuesong Li. CLIP-MSM: A Multi-Semantic Mapping Brain Representation for Human High-Level Visual Cortex. Proceedings of the AAAI Conference on Artificial Intelligence, 39(9):91...

  8. [18]

    Sparse Autoencoders Find Highly Interpretable Features in Language Models, October 2023

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse Autoencoders Find Highly Interpretable Features in Language Models, October 2023

    2023

  9. [19]

    Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller

    Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models, March 2025

    2025

  10. [20]

    Allen, Ghislain St-Yves, Yihan Wu, Jesse L

    Emily J. Allen, Ghislain St-Yves, Yihan Wu, Jesse L. Breedlove, Jacob S. Prince, Logan T. Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, J. Benjamin Hutchinson, Thomas Naselaris, and Kendrick Kay. A massive 7T fMRI dataset to bridge cognitive neuroscience and ...

    2022 doi

  11. [21]

    Sparse Autoencoders Do Not Find Canonical Units of Analysis, February 2025

    Patrick Leask, Bart Bussmann, Michael Pearce, Joseph Bloom, Curt Tigges, Noura Al Moubayed, Lee Sharkey, and Neel Nanda. Sparse Autoencoders Do Not Find Canonical Units of Analysis, February 2025

    2025

  12. [22]

    Scaling and evaluating sparse autoencoders, June 2024

    Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders, June 2024

    2024

  13. [23]

    Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders, August 2024

    Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders, August 2024

    2024

  14. [24]

    Prince, Matthew Kowal, Victor Boutin, Isabel Papadimitriou, Binxu Wang, Martin Wattenberg, Demba Ba, and Talia Konkle

    Thomas Fel, Ekdeep Singh Lubana, Jacob S. Prince, Matthew Kowal, Victor Boutin, Isabel Papadimitriou, Binxu Wang, Martin Wattenberg, Demba Ba, and Talia Konkle. Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models, February 2025

    2025

  15. [25]

    Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment, February 2025

    Harrish Thasarathan, Julian Forsyth, Thomas Fel, Matthew Kowal, and Konstantinos Derpanis. Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment, February 2025

    2025

  16. [26]

    Interpreting CLIP with Hierarchical Sparse Autoencoders, February 2025

    Vladimir Zaigrajew, Hubert Baniecki, and Przemyslaw Biecek. Interpreting CLIP with Hierarchical Sparse Autoencoders, February 2025

    2025

  17. [27]

    Sparse Autoencoder as a Zero-Shot Classifier for Concept Erasing in Text-to-Image Diffusion Models, March 2025

    Zhihua Tian, Sirun Nan, Ming Xu, Shengfang Zhai, Wenjie Qu, Jian Liu, Kui Ren, Ruoxi Jia, and Jiaheng Zhang. Sparse Autoencoder as a Zero-Shot Classifier for Concept Erasing in Text-to-Image Diffusion Models, March 2025

    2025

  18. [28]

    The Missing Curve Detectors of InceptionV1: Applying Sparse Autoencoders to InceptionV1 Early Vision, September 2024

    Liv Gorton. The Missing Curve Detectors of InceptionV1: Applying Sparse Autoencoders to InceptionV1 Early Vision, September 2024

    2024

  19. [29]

    Toy Models of Superposition, September 2022

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy Models of Superposi...

    2022

  20. [30]

    [Interim research report] Taking features out of superposition with sparse autoencoders

    Lee Sharkey, Dan Braun, and beren. [Interim research report] Taking features out of superposition with sparse autoencoders. December 2022

    2022

  21. [31]

    Olshausen, and Yann LeCun

    Zeyu Yun, Yubei Chen, Bruno A. Olshausen, and Yann LeCun. Transformer visualization via dictionary learning: Contextualized embedding as a linear superposition of transformer factors, April 2023

    2023

  22. [32]

    A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models, March 2025

    Dong Shu, Xuansheng Wu, Haiyan Zhao, Daking Rai, Ziyu Yao, Ninghao Liu, and Mengnan Du. A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models, March 2025

    2025

  23. [34]

    A Convolutional Neural Network Interpretable Framework for Human Ventral Visual Pathway Representation

    Mufan Xue, Xinyu Wu, Jinlong Li, Xuesong Li, and Guoyuan Yang. A Convolutional Neural Network Interpretable Framework for Human Ventral Visual Pathway Representation. Proceedings of the AAAI Conference on Artificial Intelligence, 38(6):6413–6421, March 2024. ISSN 2374-3468. do...

    2024 doi

  24. [35]

    Goodale, John Paul Meenan, Heinrich H

    Melvyn A. Goodale, John Paul Meenan, Heinrich H. Bülthoff, David A. Nicolle, Kelly J. Murphy, and Carolynn I. Racicot. Separate neural pathways for the visual analysis of object shape in perception and prehension. Current Biology, 4(7):604–610, July 1994. ISSN 0960-9822. doi: ...

    1994 doi

  25. [36]

    Kalanit Grill-Spector and Kevin S. Weiner. The functional architecture of the ventral temporal cortex and its role in categorization. Nature Reviews Neuroscience, 15(8):536–548, August 2014. ISSN 1471-0048. doi: 10.1038/nrn3747

    2014 doi

  26. [37]

    Selectivity for the shape, size, and orientation of objects for grasping in neurons of monkey parietal area aip

    Akira Murata, Vittorio Gallese, Giuseppe Luppino, Masakazu Kaseda, and Hideo Sakata. Selectivity for the shape, size, and orientation of objects for grasping in neurons of monkey parietal area aip. Journal of Neurophysiology, 83(5):2580–2601, 2000. doi: 10.1152/jn.2000.83.5.2580

    doi

  27. [38]

    Dumoulin and Brian A

    Serge O. Dumoulin and Brian A. Wandell. Population receptive field estimates in human visual cortex. NeuroImage, 39(2):647–660, January 2008. ISSN 1053-8119. doi: 10.1016/j.neuroimage.2007.09.034

    2008 doi

  28. [39]

    D. H. Hubel and T. N. Wiesel. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. The Journal of Physiology, 160(1):106–154.2, January 1962. ISSN 0022-3751

    1962

  29. [40]

    Demb, Valerio Mante, David J

    Matteo Carandini, Jonathan B. Demb, Valerio Mante, David J. Tolhurst, Yang Dan, Bruno A. Olshausen, Jack L. Gallant, and Nicole C. Rust. Do we know what the early visual system does? Journal of Neuroscience, 25(46):10577–10597, 2005. ISSN 0270-6474. doi: 10.1523/JNEUROSCI.3726-05.2005

    2005 doi

  30. [41]

    Levitt, David C

    Jonathan B. Levitt, David C. Kiper, and J. Anthony Movshon. Receptive fields and functional architecture of macaque v2. Journal of Neurophysiology, 71(6):2517–2542, June 1994. doi: 10.1152/jn.1994.71.6.2517

    1994 doi

  31. [42]

    Ziemba, David J

    Jeremy Freeman, Corey M. Ziemba, David J. Heeger, Eero P. Simoncelli, and J. Anthony Movshon. A functional and perceptual signature of the second visual area in primates. Nature Neuroscience, 16(7): 974–981, July 2013. ISSN 1546-1726. doi: 10.1038/nn.3402

    2013 doi

  32. [43]

    Nancy Kanwisher, Josh McDermott, and Marvin M. Chun. The fusiform face area: A module in human extrastriate cortex specialized for face perception. Journal of Neuroscience, 17(11):4302–4311, 1997. ISSN 0270-6474. doi: 10.1523/JNEUROSCI.17-11-04302.1997

    1997

  33. [44]

    Epstein and N

    R. Epstein and N. Kanwisher. A cortical representation of the local visual environment. Nature, 392: 598–601, 1998. doi: 10.1038/33402

    1998 doi

  34. [45]

    P. E. Downing, Y . Jiang, M. Shuman, and N. Kanwisher. A cortical area selective for visual processing of the human body. Science, 293(5539):2470–2473, September 2001. doi: 10.1126/science.1063414

    2001 doi

  35. [46]

    The visual word form area: spatial and temporal characterization of an initial stage of reading in normal subjects and posterior split-brain patients

    Laurent Cohen, Stanislas Dehaene, Lionel Naccache, Stéphane Lehéricy, Ghislaine Dehaene-Lambertz, Marie-Anne Hénaff, and Franck Michel. The visual word form area: spatial and temporal characterization of an initial stage of reading in normal subjects and posterior split-brain ...

    doi

  36. [47]

    Apurva Ratan Murty, and Nancy Kanwisher

    Meenakshi Khosla, N. Apurva Ratan Murty, and Nancy Kanwisher. A highly selective response to food in human visual cortex revealed by hypothesis-free voxel decomposition. Current Biology, 32(19): 4159–4171.e9, 2022. ISSN 0960-9822. doi: 10.1016/j.cub.2022.08.009

    2022 doi

  37. [48]

    Prince, Kendrick N

    Colin Conwell, Jacob S. Prince, Kendrick N. Kay, George A. Alvarez, and Talia Konkle. A large-scale examination of inductive biases shaping high-level visual representation in brains and machines. Nature Communications, 15(1):9383, October 2024. ISSN 2041-1723. doi: 10.1038/s4...

    2024 doi

  38. [49]

    Bandettini

    Nikolaus Kriegeskorte, Marieke Mur, and Peter A. Bandettini. Representational similarity analysis - connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2, November 2008. ISSN 1662-5137. doi: 10.3389/neuro.06.004.2008

    2008 doi

  39. [50]

    Lawrence Zitnick, and Piotr Dollár

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft COCO: Common Objects in Context, February 2015

    2015

  40. [51]

    Gao, Alexander G

    James S. Gao, Alexander G. Huth, Mark D. Lescroart, and Jack L. Gallant. Pycortex: An interactive surface visualizer for fMRI. Frontiers in Neuroinformatics, 9, September 2015. ISSN 1662-5196. doi: 10.3389/fninf.2015.00023

    2015

  41. [52]

    ImageNet: A Large-Scale Hierarchical Image Database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. 12

  42. [53]

    Learning Transferable Visual Models From Natural Language Supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of th...

    2021

  43. [54]

    Deep Residual Learning for Image Recognition, December 2015

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition, December 2015

    2015

  44. [55]

    Masked Autoencoders Are Scalable Vision Learners, December 2021

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked Autoencoders Are Scalable Vision Learners, December 2021

    2021

  45. [56]

    DINOv2: Learning Robust Visual Features without Supervision, February 2024

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabb...

    2024

  46. [57]

    Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment Anything, April 2023

    2023

  47. [58]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, June 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at...

    2021

  48. [59]

    PyTorch: An Imperative Style, High-Performance Deep Learning Library, December 2019

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fan...

    2019

  49. [60]

    Network Dissection: Quantify- ing Interpretability of Deep Visual Representations, April 2017

    David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network Dissection: Quantify- ing Interpretability of Deep Visual Representations, April 2017

    2017

  50. [61]

    Similarity of Neural Network Representations Revisited

    Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of Neural Network Representations Revisited. In Proceedings of the 36th International Conference on Machine Learning, pages 3519–3529. PMLR, May 2019. 13 A Broader impact Our work introduces the si...

    2019

  51. [63]

    CLS" token for all models except SAM. We take the average of all patches in SAM’s target layer output as the

    introduces CKA method, a kind of representation analysis method similar to Representa- tion Similarities to explore two deep learning models’ activation similarity. For model A and B, Let Kij = k(xi, xj), L ij = l(yi, yj), where k, lare kernel functions and xi and yi are the i...

  52. [64]

    Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: We describe the contributions in the last paragraph of introduction 1. Guidelines: • The answer NA means that the abst...

  53. [65]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitations in Section 5. Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the paper ...

  54. [66]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] 48 Justification: The article dosen’t include theoretical results. Guidelines: • The answer NA means that th...

  55. [67]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  56. [68]

    • Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details

    Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? 49 Answer: [Yes] Justification: We provide code in suppl...

  57. [69]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: Dataset splits are mentioned ...

  58. [70]

    Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We illustrate the statistical information in Table...

  59. [71]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We provide computer res...

  60. [72]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: We follow the NeurIPS Code of Ethics strictly. Guidelines: • The answer NA mean...

  61. [73]

    Guidelines: • The answer NA means that there is no societal impact of the work performed

    Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: We describe the alignment between deep learning model and brain, provide an insight rather than particular...

  62. [74]

    Guidelines: • The answer NA means that the paper poses no such risks

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our work poses ...

  63. [75]

    Guidelines: • The answer NA means that the paper does not use existing assets

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We respect the l...

  64. [76]

    Guidelines: • The answer NA means that the paper does not release new assets

    New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: Our paper does not release new assets. Guidelines: • The answer NA means that the paper does not release new assets. ...

  65. [77]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  66. [78]

    All ethical regulations relevant to human research participants were followed

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

  67. [79]

    Answer: [NA] Justification: The core method in the research does not involve LLMs

    Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the ...

    2025

  68. [2025]

    doi: 10.1609/aaai.v39i9.32994

    ISSN 2374-3468. doi: 10.1609/aaai.v39i9.32994

    doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.