Pith. sign in

REVIEW 4 major objections 5 minor 26 references

Perception Activator: An intuitive and portable framework for brain cognitive exploration

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that injecting fMRI tokens into a vision model's multi-scale features via cross-attention reveals category and coarse spatial cues in brain activity, improving detection recall while leaving mask precision flat.

desk verdict The task setup is genuinely new, but the abstract overstates the numbers and the paper never confirms that the visual perceptron was trained without seeing the NSD test images. read the letter →

arxiv 2507.02311 v1 pith:2A7SBQ6W submitted 2025-07-03 cs.CV

Le Xu , Qi Zhang , Qixian Zhang , Hongyun Zhang , Duoqian Miao , Cairong Zhao This is my paper · ORCID
classification cs.CV
keywords fMRItoimagedecodingmulti-modalinterventionexperimentobjectdetectioninstancesegmentationcross-attentionLoRANaturalScenesDataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that fMRI signals recorded while a person views natural images contain multi-object semantic cues and coarse spatial localization information that a computer vision model can exploit. The authors build two modules: an fMRI decoder that maps voxel activity into a CLIP-aligned token space, and a visual perceptron—a Mask R-CNN with a cross-attention block inserted at every feature-pyramid level—that fuses those tokens into image features for object detection and instance segmentation. On the Natural Scenes Dataset they report that the injected fMRI signal raises most detection metrics (AP 20.6 to 20.9, AP50 35.5 to 36.7, AR 29.2 to 30.1) and slightly lifts segmentation recall, while mask AP falls from 19.3 to 18.8; they interpret this as fMRI supplying category and coarse 'where' cues without fine-boundary detail. If the claim holds, brain activity becomes a usable auxiliary cue for vision tasks and a way to probe how the brain represents multiple objects at once.

What carries the argument

The carrying mechanism is a shared cross-attention fusion block with a learnable temperature, inserted after every feature-pyramid scale of Mask R-CNN. At each scale, image features $F^{(l)}$ are linearly projected into queries $Q^{(l)}$; the pretrained fMRI token embeddings $Z$ are projected into keys $K$ and values $V$; attention weights $\operatorname{Softmax}(Q^{(l)}K^{\top} / (\tau \sqrt{d_k}))$ let each image location retrieve cues from all fMRI tokens; a residual connection and LayerNorm produce the fused feature map that feeds RoIAlign and the detection and mask heads. The fMRI decoder is pretrained with three losses—CLIP contrastive retrieval with MixCo augmentation, low-level VAE-space L1 reconstruction, and a diffusion-prior alignment loss—and is kept frozen during downstream fusion, with LoRA used only as a lightweight adapter. LoRA, the low-rank decomposition $\Delta W = BA$, is what makes the intervention portable: it changes the fusion behavior without rewriting the pretrained fMRI representations.

What would settle it

Retrain the visual perceptron on the NSD training images only, with the test set excluded, and re-run the with-fMRI and without-fMRI comparison on the 982 test stimuli; if detection AP and AR gains disappear or reverse, the central claim fails. A cheaper check is to count image overlap between Section 4.2's training set and the NSD test list: any overlap at all invalidates the comparison.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that fMRI representations can serve as an intervention condition that changes downstream vision-task behavior in a systematic way. With fMRI tokens injected through cross-attention, Mask R-CNN's detection improves across most metrics—AP 20.6→20.9, AP50 35.5→36.7, APL 47.5→49.5, AR 29.2→30.1—with the largest gains in recall and in medium/large objects; instance segmentation gains only in recall (AR100 26.2→26.9) while AP declines from 19.3 to 18.8. The authors read this pattern as showing that fMRI carries coarse object-level semantic and spatial cues: enough to help the model find objects, not enough to refine masks or boxes. They support this with attention heatmaps showing category-specific activation patterns across fMRI tokens, and with feature-difference maps concentrating changes on object centers and salient regions.

Load-bearing premise

The load-bearing assumption is that the visual perceptron never saw the NSD test stimuli during training; the paper says it was trained on all visual stimulus images watched by Subject 1 without stating that the 982 test images were held out, and if they were not, the reported gains could be memorization rather than evidence about fMRI.

Editorial extensions

If this is right

  • Injected fMRI improves detection recall more than precision, so the practical value of the signal is as a semantic 'where to look' prior rather than as a box refiner.
  • The cross-attention ablation, detection AP dropping from 20.9 to 14.6 and segmentation AP from 18.8 to 13.6 without it, shows that how the fMRI tokens are fused matters as much as having the pretrained fMRI features.
  • The LoRA rank ablation shows stable performance across ranks 2 through 32, so the fusion is insensitive to the adapter capacity setting.
  • The per-category gains on small hand-held objects like scissors and remote controls suggest fMRI carries context about semantically related objects, not just the dominant category.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A clean way to extend the claim is to train the visual perceptron strictly on the NSD training images and rerun the comparison; if the with-fMRI gains vanish on held-out images, the reported effect is memorization, not a property of fMRI.
  • Because the positive effect appears mostly in recall, a precision-side postprocessing step such as confidence calibration or stricter non-maximum suppression might convert the fMRI-driven recall into larger AP gains than the paper reports.
  • The same cross-attention injection could be tested in detection architectures without feature pyramids or RoIAlign to see whether the coarse spatial cue is a general property of fMRI or tied to two-stage Mask R-CNN designs.
  • The category-specific voxel attention patterns suggest an untested use: treating the cross-attention map as an unsupervised region-of-interest tool to identify which voxel clusters encode a given object class.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a framework called Perception Activator that uses fMRI representations from the Natural Scenes Dataset (NSD) as conditional inputs to a Mask R-CNN object detection and instance segmentation pipeline. An fMRI decoder is pretrained to map voxels into CLIP and Stable Diffusion embedding spaces, and the resulting representation is injected into multi-scale FPN features through cross-attention with a learnable temperature. The authors compare detection and segmentation metrics with and without fMRI injection, report per-category visualizations and attention heatmaps, and conclude that fMRI carries rich multi-object semantic cues and coarse spatial localization information. The central empirical claim is that fMRI injection improves downstream detection and segmentation accuracy.

Significance. If the central claim were sound, the framework would provide a useful, portable probe for studying what information fMRI signals contribute to vision models, and the use of instance-level tasks (detection and segmentation) is a reasonable complement to the usual reconstruction-based decoding literature. The paper also has strengths: it uses a public large-scale dataset (NSD), provides ablations of the pretraining strategy and the cross-attention fusion module, includes LoRA-based lightweight adaptation, and offers qualitative heatmap and feature-difference visualizations. However, the empirical core is not currently reliable: the reported numbers in Table 1 do not support the abstract's claimed gains, the training/evaluation split is not stated clearly enough to rule out test-set leakage, and the fMRI representation is a supervised projection of the same stimulus images, which confounds the claimed neural interpretation.

major comments (4)
  1. [Section 4.2 / Section 4.1] The training protocol for the visual perceptron is not sufficient to establish generalization. Section 4.2 states that the visual perceptron was trained on 'all the visual stimulus images watched by subject 1' for 20 epochs, while Section 4.1 defines a test set of 982 visual stimuli and 2770 fMRI trials for subject 1. Because in the NSD protocol subject 1 viewed both the train and test stimuli, the wording 'all the visual stimulus images watched by subject 1' plausibly includes the 982 test images. The paper must state explicitly that the 982 test stimuli and their corresponding fMRI trials were excluded from all training, including pretraining of the fMRI decoder and training of the detection/segmentation heads. If those images were not held out, the comparison in Table 1 is evaluated on training inputs and does not measure generalization, invalidating the abstract's claim that fMRI signals improve downstream detection and segmentation.
  2. [Abstract / Introduction / Table 1] The claimed improvements are not supported by Table 1. The abstract and Section 1 state that incorporating fMRI yields a 7% improvement in detection accuracy and a 4% improvement in segmentation accuracy, but Table 1 shows detection AP increasing from 20.6 to 20.9 (+0.3), AP50 from 35.5 to 36.7 (+1.2), and AP75 decreasing from 23.1 to 20.9 (-2.2); segmentation AP decreases from 19.3 to 18.8 (-0.5), AP50 increases by only 0.1, and AP75 decreases. The paper should either reconcile these numbers with a precise definition of the reported percentages (for example, relative versus absolute changes and which metric is used) or correct the abstract and introduction. As written, the headline claim contradicts the paper's own main table.
  3. [Section 3.1, Eq. (5)-(7)] The interpretation that the injected signal reflects neural content is confounded by the fMRI decoder's supervision. The decoder is pretrained to map fMRI voxels to CLIP image embeddings (via L_prior and L_BiMixCo) and to Stable Diffusion VAE latents (via L_lowlevel), using the same NSD stimulus images. The representation Z injected into the visual perceptron is therefore a learned prediction of image-derived embeddings, not a raw neural signal. Gains from injecting Z could be driven by the decoder's ability to reconstruct image semantics from the stimulus set, rather than by fMRI-specific information. A control experiment with shuffled or permuted fMRI voxel labels, or with a decoder trained on image embeddings alone without fMRI input, is needed to attribute the observed effects to neural content.
  4. [Section 5.1 / Table 1] The paper's interpretation that fMRI is beneficial for both detection and segmentation is internally inconsistent. Segmentation AP falls from 19.3 to 18.8, and detection AP75 falls from 23.1 to 20.9, while only some recall-oriented metrics (AR100, AR_M, AR_L) improve. Moreover, no error bars, confidence intervals, or significance tests are reported, so differences as small as +0.3 AP cannot be distinguished from training noise. The manuscript should report multiple runs or statistical significance, and the discussion in Section 5.1 should be revised to acknowledge that the instance-segmentation result is not an improvement in AP.
minor comments (5)
  1. [General] There are several typographical errors: '0ur' appears in Table 1 and Table 2, 'T able' appears before Table 1 and Table 2, and the text under Fig. 5 contains 'fMRI characterization' with an extra quotation mark.
  2. [Appendix B] The LoRA rank ablation text says 'As shown in Table 2' but the rank results are in Table 3; the reference should be corrected.
  3. [Section 5.1] The sentence 'AR by 1.8% for medium and large objects' is imprecise: Table 1 shows AR_M improving by 1.8 and AR_L by 0.4, but the text groups medium and large together despite their different magnitudes.
  4. [Section 5.3] The category-differentiation experiment and the per-category mAP charts in Fig. 3 and Fig. 8 are presented as supporting 'improvement in most categories', but no quantitative counts or statistical tests are given for how many of the 80 COCO categories improve or decline.
  5. [Section 6] The limitation section acknowledges that the findings are not yet used to improve reconstruction models, but it does not mention the unresolved train/test split ambiguity or the small magnitude of the reported gains; these should be listed as limitations.

Circularity Check

2 steps flagged · score 6.0 of 10

The injected fMRI representation is built by fitting a decoder to CLIP/SD embeddings of the same NSD images, and the VP training protocol as stated includes all images watched by the subject, so the reported gains do not independently confirm that fMRI contains generalizable semantic/spatial cues.

  1. fitted input called prediction [Section 3.1, Eqs. (5)-(7); Section 5.1 and Abstract]
    "Llowlevel = 1/|B| Σ |zi − ẑi|. Lprior = 1/|B| Σ ||gi − ĝi||2^2. LfMRI = Lprior + α1Llowlevel + α2LBiMixCo. ... Our results demonstrate that incorporating fMRI signals enhances the accuracy of downstream detection and segmentation, confirming that fMRI contains rich multi-object semantic cues and coarse spatial localization information."

    The injected signal is not raw fMRI; it is Z = ε(M), the output of a decoder pretrained so that Z approximates the CLIP and Stable-Diffusion embeddings (g_i, z_i) of the stimulus image itself (Eqs. 5-7). Any semantic cues in Z were placed there by the pretraining objective, fitted on the same NSD images used downstream. Observing that injecting this CLIP-aligned Z helps a detector therefore does not independently confirm that the neural signal contains those cues; it confirms that a representation trained to predict image semantics is semantically useful. The claimed 'confirmation' reduces to a check of the decoder's own training objective rather than an independent discovery about fMRI content.

  2. fitted input called prediction [Section 4.1-4.2 and Table 1]
    "Within the dataset, each subject’s train set comprises 8859 distinct visual stimuli and 24980 fMRI ... The test set contains 982 visual stimuli and 2770 fMRI. ... During the training stage of the visual perceptron, we trained all the visual stimulus images watched by subject 1 for 20 epochs."

    In NSD, subject 1 watched both the 8859 train stimuli and the 982 test stimuli. As written, 'all the visual stimulus images watched by subject 1' includes the test set, and the paper never states that the 982 test images were excluded from VP training. Table 1 then reports AP/AR for 'our' model versus the baseline. If those metrics are computed on the same images used to fit the VP (and the fMRI decoder), the reported numbers are in-sample fits, not predictions on unseen stimuli. The abstract's conclusion that fMRI 'enhances the accuracy of downstream detection and segmentation' and contains generalizable semantic/spatial cues is therefore not supported by an out-of-sample comparison; the model could have memorized the CLIP-aligned representation of each tested image.

full rationale

The paper's derivation chain is: pretrain an fMRI decoder with Eq. 7, inject the resulting representation Z into Mask R-CNN via cross-attention, compare with a no-fMRI baseline, and conclude that fMRI contains rich multi-object semantic and coarse spatial cues. Two steps undermine this as an independent derivation. First, Eq. 7 explicitly trains Z to match the CLIP and Stable-Diffusion embeddings of the same NSD images, so the semantic content injected downstream is by construction the images' CLIP/SD semantics; the downstream gain does not independently prove that the neural signal carries those cues. The baseline comparison is an external benchmark and the decoder could in principle fail, so this is partial, not total, circularity. Second, Section 4.2 states the visual perceptron was trained on 'all the visual stimulus images watched by subject 1'; NSD's test set of 982 images was also watched by subject 1 and is not stated to be excluded, so Table 1's numbers may be in-sample fits. If so, the reported 'prediction' is not out-of-sample. The paper's self-citations ([13,14,23]) are not load-bearing, and there is no imported uniqueness theorem. There is also an internal mismatch between the abstract/intro's claimed 7%/4% improvements and Table 1's detection AP +0.3 and segmentation AP -0.5; that is a non-circular validity concern but further weakens the central claim.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical entities are introduced. The central claim depends on pretraining and evaluation choices: loss weights, LoRA rank, the assumption that the fMRI decoder is brain-derived rather than image-derived, the assumption of a clean train/test split, and the assumption that the baseline is capacity-matched.

free parameters (4)
  • Loss weight alpha_1 for low-level VAE loss = not reported
    Introduced in Eq. (7); hand-set during fMRI decoder pretraining, affects representation quality.
  • Loss weight alpha_2 for BiMixCo contrastive loss = not reported
    Introduced in Eq. (7); hand-set during fMRI decoder pretraining, affects representation quality.
  • Cross-attention learnable temperature tau = learned
    Equation (10); learned during visual perceptron training, controls injection strength.
  • LoRA rank r = 16
    Appendix B ablates ranks 2 to 32 and selects 16; choosing by test performance makes downstream metrics partly data-dependent.
assumptions (4)
  • domain assumption The fMRI decoder pretrained with CLIP and SD objectives on NSD yields representations whose semantic content originates from neural activity.
    Section 3.1 aligns voxel embeddings to image embeddings of the same NSD images, so the decoder could inject image-derived rather than brain-derived content.
  • domain assumption The visual perceptron training in Section 4.2 does not include NSD test stimuli despite the phrase "all the visual stimulus images watched by subject 1".
    The paper never states that test images are excluded from VP training, and if they are included the reported gains are invalid.
  • domain assumption A Mask R-CNN baseline with no cross-attention or LoRA parameters is a fair control for capacity.
    The fMRI variant adds parameters, so any gain could come from added capacity rather than fMRI content.
  • standard math Standard mathematical operations of cross-attention, LayerNorm, LoRA, and Mask R-CNN losses are valid.
    Used without proof; these are standard components in the literature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Perception Activator: An intuitive and portable framework for brain cognitive exploration." pith.science (2026). https://pith.science/paper/2A7SBQ6W

@misc{pith2026250702311,
  author       = {Pith},
  title        = {Pith review of: Perception Activator: An intuitive and portable framework for brain cognitive exploration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2A7SBQ6W}},
  note         = {Machine review of arXiv:2507.02311}
}
read the original abstract

Recent advances in brain-vision decoding have driven significant progress, reconstructing with high fidelity perceived visual stimuli from neural activity, e.g., functional magnetic resonance imaging (fMRI), in the human visual cortex. Most existing methods decode the brain signal using a two-level strategy, i.e., pixel-level and semantic-level. However, these methods rely heavily on low-level pixel alignment yet lack sufficient and fine-grained semantic alignment, resulting in obvious reconstruction distortions of multiple semantic objects. To better understand the brain's visual perception patterns and how current decoding models process semantic objects, we have developed an experimental framework that uses fMRI representations as intervention conditions. By injecting these representations into multi-scale image features via cross-attention, we compare both downstream performance and intermediate feature changes on object detection and instance segmentation tasks with and without fMRI information. Our results demonstrate that incorporating fMRI signals enhances the accuracy of downstream detection and segmentation, confirming that fMRI contains rich multi-object semantic cues and coarse spatial localization information-elements that current models have yet to fully exploit or integrate.

Figures

Figures reproduced from arXiv: 2507.02311 by the authors.

Figure 1
Figure 1. a) The commonly used two-level decoding method. It can be seen that the low-level pixel-level information only plays an auxiliary role in the final step of recon￾struction, and the high-level and low-level information are not efficiently fused. b) A comparison between the visual stimulus image and the reconstructed image. It can be seen that the distortion problem of the existing framework occurs on multiple targets… view at source ↗
Figure 2
Figure 2. Overview of our framework, consisting of two main components: fMRI Decoder and Visual Perceptron. The Downstream Network can be adapted to different down￾stream tasks. 2.2 Instance-level Tasks In the research of computer vision, Object Detection and instance segmentation respectively undertake roles at different levels: Object detection aims to identify the categories and approximate positions of each object in the … view at source ↗
Figure 3
Figure 3. mAP point map presented by category (IoU>50). Our model has been improved in most categories. AR by 1.8% for medium and large objects—indicating that brain activity pro￾vides stronger cues for larger targets. Overall, the improvement in AR exceeds that in AP, suggesting that the injected fMRI information chiefly contributes high-level semantic and coarse spatial signals: it helps the model “find” more objects but ha… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Multi-layer average attention weight map. Here, the visual stimulus image with the main categories of cat and person is selected [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Visualization of the differences before and after feature map fusion and fMRI characterization.”Raw” represents the unfused feature map, ”fused” represents the fused one, and ”diff” is the difference after fusion minus that before fusion, representing the number of lay…
Figure 6
Figure 6. Figure 6: Visualization of the initial layer weights of fMRI. a) is the weight after pre￾training, and b) is the weight without pre-training. model exhibits a sparser activation pattern in its first layer, with most channels suppressed, whereas the no pre-trained model activates…
Figure 7
Figure 7. Figure 7: Instance segmentation result, including different types and different target quantities. C Instance segmentation results. In this section, we present the instance segmentation results of the model under various types of targets. It can be seen from [PITH_FULL_IMAGE:fi…
Figure 8
Figure 8. Figure 8: mAP point map presented by category (IoU>50). Our model has been improved in most categories [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Multi-layer average attention weight maps. The selected categories from top to bottom are aircraft, giraffes, and motorcycles in sequence [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 12 canonical work pages

  1. [1]

    https://doi.org/https://doi.org/10.1016/j.neubiorev.2004.10.009

    Fingelkurts, A.A., Fingelkurts, A.A., K¨ ahk¨ onen, S.: Functional connectivity in the brain—is it an elusive concept? Neuroscience & Biobehavioral Reviews 28(8), 827– 836 (2005). https://doi.org/https://doi.org/10.1016/j.neubiorev.2004.10.009

    work page doi:10.1016/j.neubiorev.2004.10.009 2005

  2. [2]

    IEEE Transactions on Pattern Analysis and Machine Intelligence 45(9), 10760–10777 (2023)

    Du, C., Fu, K., Li, J., He, H.: Decoding visual neural representations by multimodal learning of brain-visual-linguistic features. IEEE Transactions on Pattern Analysis and Machine Intelligence 45(9), 10760–10777 (2023). https://doi.org/10.1109/TPAMI.2023.3263181

    work page arXiv 2023

  3. [3]

    In: Proceed- ings of the 31st ACM International Conference on Multimedia

    Nie, X., Hu, B., Gao, X., Li, L., Zhang, X., Xiao, B.: Bmi-net: A brain-inspired multimodal interaction network for image aesthetic assessment. In: Proceed- ings of the 31st ACM International Conference on Multimedia. p. 5514–5522. MM ’23, Association for Computing Machinery, New York, NY, USA (2023). https://doi.org/10.1145/3581783.3611996

    arXiv 2023

  4. [4]

    IEEE Transactions on Pattern Analysis and Machine Intelligence 43(11), 3833–3849 (2021)

    Palazzo, S., Spampinato, C., Kavasidis, I., Giordano, D., Schmidt, J., Shah, M.: Decoding brain representations by multimodal learning of neural activity and vi- sual features. IEEE Transactions on Pattern Analysis and Machine Intelligence 43(11), 3833–3849 (2021). https://doi.org/10.1109/TPAMI.2020.2995909

    work page arXiv 2021

  5. [5]

    In: Hampson, M

    Linden, D.: Section 3 - introduction. In: Hampson, M. (ed.) fMRI Neurofeedback, pp. 161–169. Academic Press (2021). https://doi.org/https://doi.org/10.1016/B978-0-12-822421-2.00008-9

    work page doi:10.1016/b978-0-12-822421-2.00008-9 2021

  6. [6]

    NeuroImage 56(2), 400–410 (2011)

    Naselaris, T., Kay, K.N., Nishimoto, S., Gallant, J.L.: Encod- ing and decoding in fmri. NeuroImage 56(2), 400–410 (2011). https://doi.org/https://doi.org/10.1016/j.neuroimage.2010.07.073, multivari- ate Decoding and Brain Reading

    doi:10.1016/j.neuroimage.2010.07.073 2011

  7. [7]

    In: Proceedings of the 31st International Conference on Neural Information Processing Systems

    Parthasarathy, N., Batty, E., Falcon, W., Rutten, T., Rajpal, M., Chichilnisky, E.J., Paninski, L.: Neural networks for efficient bayesian decoding of natural images from retinal neurons. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. p. 6437–6448. NIPS’17, Curran Associates Inc., Red Hook, NY, USA (2017)

    work page 2017

  8. [8]

    Nature neuroscience 25(1), 116– 126 (2022)

    Allen, E.J., St-Yves, G., Wu, Y., Breedlove, J.L., Prince, J.S., Dowdle, L.T., Nau, M., Caron, B., Pestilli, F., Charest, I., et al.: A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience 25(1), 116– 126 (2022)

    work page 2022

Show all 26 references
  1. [9]

    Horikawa, T., Kamitani, Y.: Generic decoding of seen and imagined objects using hierarchical visual features (2016), https://arxiv.org/abs/1510.06479

    2016 arXiv

  2. [10]

    In: Koyejo, S., Mohamed, S., Agar- wal, A., Belgrave, D., Cho, K., Oh, A

    Lin, S., Sprague, T., Singh, A.K.: Mind reader: Reconstructing com- plex images from brain activities. In: Koyejo, S., Mohamed, S., Agar- wal, A., Belgrave, D., Cho, K., Oh, A. (eds.) Advances in Neural Infor- mation Processing Systems. vol. 35, pp. 29624–29636. Curran Associa...

    2022

  3. [11]

    Liu, Y., Ma, Y., Zhou, W., Zhu, G., Zheng, N.: Brainclip: Bridging brain and visual-linguistic representation via clip for generic natural visual stimulus decoding (2023), https://arxiv.org/abs/2302.12971

    2023 arXiv

  4. [12]

    Ozcelik, F., VanRullen, R.: Natural scene reconstruction from fmri signals using generative latent diffusion (2023), https://arxiv.org/abs/2303.05334

    2023 arXiv

  5. [13]

    Gong, Z., Zhang, Q., Bao, G., Zhu, L., Liu, K., Hu, L., Miao, D., Zhang, Y.: Lite- mind: Towards efficient and robust brain representation network (2024), https: //arxiv.org/abs/2312.03781 14 Xu et al

    2024 arXiv

  6. [14]

    Bao, G., Zhang, Q., Gong, Z., Zhou, J., Fan, W., Yi, K., Naseem, U., Hu, L., Miao, D.: Wills aligner: Multi-subject collaborative brain visual decoding (2024), https://arxiv.org/abs/2404.13282

    2024 arXiv

  7. [15]

    Chen, Z., Qing, J., Xiang, T., Yue, W.L., Zhou, J.H.: Seeing beyond the brain: Con- ditional diffusion model with sparse masked modeling for vision decoding (2023), https://arxiv.org/abs/2211.06956

    2023 arXiv

  8. [16]

    MM ’23, Association for Computing Machinery, New York, NY, USA (2023), https://doi.org/10.1145/3581783.3613832

    Lu, Y., Du, C., Zhou, Q., Wang, D., He, H.: Minddiffuser: Controlled image re- construction from human brain activity with semantic and structural diffusion. MM ’23, Association for Computing Machinery, New York, NY, USA (2023), https://doi.org/10.1145/3581783.3613832

    2023

  9. [17]

    Advances in Neu- ral Information Processing Systems 36, 24705–24728 (2023)

    Scotti, P., Banerjee, A., Goode, J., Shabalin, S., Nguyen, A., Dempster, A., Ver- linde, N., Yundler, E., Weisberg, D., Norman, K., et al.: Reconstructing the mind’s eye: fmri-to-image with contrastive learning and diffusion priors. Advances in Neu- ral Information Processing ...

    2023

  10. [18]

    In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Takagi, Y., Nishimoto, S.: High-resolution image reconstruction with latent dif- fusion models from human brain activity. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 14453–14463 (2023). https://doi.org/10.1109/CVPR52729.2023.01389

    2023

  11. [19]

    In: Meila, M., Zhang, T

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th Int...

    2021

  12. [20]

    In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 10674–10685 (2022). https://doi.org/10.1109/CVPR52688.2022.01042

    2022

  13. [21]

    Guo, Y., Chen, Y., Ma, Z.: Neuroclip: Neuromorphic data understanding by clip and snn (2023), https://arxiv.org/abs/2306.12073

    2023 arXiv

  14. [22]

    ArXiv abs/2010.06300 (2020), https://api.semanticscholar

    Kim, S., Lee, G., Bae, S., Yun, S.: Mixco: Mix-up contrastive learning for visual representation. ArXiv abs/2010.06300 (2020), https://api.semanticscholar. org/CorpusID:222310572

    2020 arXiv

  15. [23]

    Gong, Z., Zhang, Q., Bao, G., Zhu, L., Liu, K., Hu, L., Miao, D.: Mindtuner: Cross- subject visual decoding with visual fingerprint and semantic correction (2024), https://arxiv.org/abs/2404.12630

    2024 arXiv

  16. [24]

    Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text- conditional image generation with clip latents (2022), https://arxiv.org/abs/ 2204.06125

    2022 arXiv

  17. [25]

    org/abs/2106.09685

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models (2021), https://arxiv. org/abs/2106.09685

    2021 arXiv

  18. [26]

    (14) - pi:predicted class-score vector for proposal i;p∗ i : one-hot ground-truth

    Mai, W., Zhang, Z.: Unibrain: Unify image reconstruction and captioning all in one diffusion model from human brain activity (2023), https://arxiv.org/abs/ 2308.07428 Perception Activator 15 A Downstream Loss Ldet = 1 Rels RX i=1 Lels(pi, ˆpi) + λ Rreg RX i=1 [p∗ i > 0]Lreg(ti...

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.