FreqSelect: Frequency-Aware fMRI-to-Image Reconstruction

Junliang Ye; Lei Wang; Md Zakir Hossain

REVIEW 4 major objections 5 minor 39 references

FreqSelect: Frequency-Aware fMRI-to-Image Reconstruction

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FreqSelect filters fMRI images by frequency band to boost reconstruction

desk verdict A plausible frequency-gating idea whose central claim of consistent improvement is contradicted by the paper's own Table 1. read the letter →

arxiv 2505.12552 v2 pith:C7XFO5HD submitted 2025-05-18 eess.IV cs.AIcs.CVcs.LG

Junliang Ye , Lei Wang , Md Zakir Hossain This is my paper · ORCID

classification eess.IVcs.AIcs.CVcs.LG

keywords fMRI-to-imagereconstructionspatialfrequencybandfilteringnaturalscenesdatasetverydeepvariationalautoencoderlatentdiffusionbraindecodinggating

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes FreqSelect, a lightweight module that splits an input image into radial spatial-frequency bands and learns a single scalar weight per band before the image is encoded into the latent space of a frozen very deep variational autoencoder. The weights are trained jointly with a ridge regressor that predicts VDV-AE latents from fMRI activity, so the module learns to pass low frequencies that carry global structure and to suppress high frequencies dominated by noise. The authors claim this consistently improves fMRI-to-image reconstruction on the Natural Scenes Dataset compared with the Brain-Diffuser pipeline, most clearly on SSIM and CLIP scores at sixteen bands, and that the learned pass-through rates offer neuro-scientific insight into how visual frequencies are represented in the brain.

What carries the argument

The central mechanism is a radial band-pass decomposition of the 64x64 input image into N concentric frequency bands using binary Fourier-domain masks, followed by Eq. (5), a weighted average of the inverse-DFT band-limited images with sigmoid-activated scalar weights alpha_i = sigma(w_i). The weighted fused image is passed through a frozen VDV-AE encoder, and the weights and a ridge regressor are trained end-to-end to minimize latent-space mean squared error.

What would settle it

Run FreqSelect with N=16 on all four subjects (sub1, sub2, sub5, sub7) with multiple random seeds, report per-subject and per-seed metric distributions with error bars, and check whether the SSIM and CLIP gains over Brain-Diffuser replicate and whether the improvement is consistent across more than a single metric subset. A simpler direct test: replace the learned scalar weights with a fixed low-pass filter that passes only the lowest band, and compare its SSIM and CLIP scores; if the fixed filter matches FreqSelect, the learned weights add nothing.

Watch

Extended reading notes

Core claim

FreqSelect's central claim is that a fixed, learned set of radial frequency-band weights, applied to the stimulus image before VDV-AE encoding, improves fMRI-to-image reconstruction by acting as a content-aware gate that suppresses noise-dominated high frequencies while preserving informative low frequencies. The authors report that at N=16 the method outperforms Brain-Diffuser on SSIM (0.2961 vs 0.293) and CLIP score (92.65% vs 92.5%), while remaining competitive on other metrics. They also claim generalization across subjects and interpretable learned pass-through rates, with the model consistently assigning the highest weight to the lowest frequency band.

Load-bearing premise

The method assumes that a single global set of learned frequency-band weights, fixed after training and shared across all images and subjects, is sufficient to realize the claimed content-aware, dynamic gating and to yield consistent improvements across metrics and subjects.

Editorial extensions

If this is right

If the central claim holds, FreqSelect becomes a cheap plug-in improvement for VDV-AE-diffusion fMRI decoders, requiring only N scalar parameters and no retraining of the generative backbones.
The learned pass-through rates would provide a data-driven map of which spatial-frequency bands carry decodable neural information, potentially connecting model training dynamics to known band-pass tuning in early visual cortex.
The architecture suggests that suppressing high-frequency input components can reduce overfitting to fMRI noise, which may transfer to other neuroimaging modalities with frequency-dependent SNR.
The finding that N=8 hurts performance while N=16 helps indicates that band granularity is a critical design parameter, motivating non-uniform or data-driven frequency partitions.
The reported gains on SSIM and CLIP but losses on several other metrics imply a trade-off between structural fidelity and pixel-level or feature-level accuracy that practitioners would need to weigh.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's own numbers show the 'consistent improvement' claim is overstated: at N=16 FreqSelect beats Brain-Diffuser on only SSIM and CLIP, and loses on PixCorr, AlexNet(2/5), Inception, EfficientNet-B, and SwAV on subject 1.
Because the weights in Eq. (5) are scalars shared across all images and subjects, the 'content-aware' and 'dynamic' description in the abstract does not match the implementation; a per-image or per-subject gate would be needed to test whether adaptivity is what drives the gain.
A testable extension is to fix the frequency partition but replace scalar weights with image-dependent predictions (e.g., a small network on the FFT magnitude), then compare against the fixed-weight version to see whether 'content-aware' gating adds anything beyond a static low-pass filter.
The reported evaluation is on subject 1 only without error bars, so cross-subject generalizability, including the claim of 'generalizes across subjects,' is not empirically established in the quantitative results.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the FreqSelect module is a sensible, easy-to-explain idea, and the paper is clearly written. But the headline claim—'consistently improves reconstruction quality across both low- and high-level metrics'—doesn't survive its own Table 1. At N=16, it beats Brain-Diffuser on SSIM (0.2961 vs 0.293) and CLIP (92.65% vs 92.5%), and loses on the other six metrics. At N=4 and N=8 it's worse across the board. That's not a consistent improvement; it's a marginal gain on two metrics at one setting.

What's new here is applying learnable radial frequency-band weighting to fMRI-to-image reconstruction. The method itself is simple: decompose the image into N bands, learn scalar per-band weights, apply a weighted sum before the VDV AE encoder. The integration with the Brain-Diffuser pipeline is clean, and the visualizations of the learned weights (Figure 4) are a nice touch—low frequencies get passed, high frequencies get suppressed, which is exactly what you'd expect from fMRI's low-pass character. The related work section is thorough and honest about prior frequency-gating methods in vision. The paper also openly discusses the N=8 failure and future work, which I appreciate.

Soft spots, in order of severity. First, the central claim is overstated. The abstract and conclusion say 'consistently improves' but the numbers don't support it. The authors do walk it back in Section 3.3 to 'several metrics,' which is accurate but conflicts with the abstract. Second, the quantitative evaluation is on subject 1 only, with no error bars or significance tests. That's not enough to support any claim of improvement, given the known run-to-run variability in this pipeline. Third, the module is described as 'content-aware' and 'dynamic,' but Eq. (5) uses global scalar weights with no input dependence. It's a static learned filter. If the authors mean 'adaptive' in the sense of trained on data, that's fine, but 'content-aware' is misleading. Fourth, the interpretability analysis—reading the learned weights as neuroscientific evidence—is partially circular, since the weights were optimized to make latents predictable from fMRI, not to explain the brain. It's suggestive, not evidence. Code is not provided, which makes all of this harder to check.

The idea still has some merit. A frequency-gating pre-filter is cheap and might help in other VDV AE pipelines, and the weight visualization offers a concrete target for future work on non-uniform bands. But as a paper, the empirical support is too thin and the claims too broad. It deserves serious refereeing, because the idea is plausible and the writing is competent, but it needs major revision: factual reframing of what FreqSelect does and doesn't improve, multi-subject results with error bars, and a reining-in of the interpretability claims. If the authors can show even a reliable small gain in SSIM/CLIP across subjects, that's a modest but citable contribution.

Referee Report

4 major / 5 minor

Summary. The paper introduces FreqSelect, a learnable frequency-band weighting module inserted before a frozen VDV AE encoder in a two-stage fMRI-to-image reconstruction pipeline. Each input image is decomposed into N radial frequency bands via DFT masks; scalar weights α_i = σ(w_i) are learned jointly with a ridge regressor that predicts VDV AE latents from fMRI, and the filtered image is a normalized weighted sum of band-limited images. Stage 2 uses the Versatile Diffusion pipeline for refinement. Experiments on the Natural Scenes Dataset (four subjects for qualitative results, subject 1 for the quantitative table) compare N = 4, 8, 16 against Brain-Diffuser and report reconstructions, learned frequency weights, and failure cases.

Significance. If it worked as advertised, FreqSelect would be a simple, cheap plug-in that improves Brain-Diffuser and offers interpretable frequency profiles. The paper is clearly written and the modular design is easy to understand, and the appendix honestly lists limitations and future controls. However, the central quantitative claim is not supported by the authors' own Table 1, and the module is not input-dependent despite being described as content-aware. With a single-subject evaluation and no error bars, the evidence is insufficient to establish the claimed improvements.

major comments (4)

[§3.3, Table 1] The central claim of the abstract and conclusion is that FreqSelect 'consistently improves reconstruction quality across both low- and high-level metrics,' but Table 1 contradicts this. With N = 16, FreqSelect beats Brain-Diffuser on only two of eight metrics (SSIM 0.2961 vs 0.293; CLIP 92.65% vs 92.5%) and loses on PixCorr (0.2734 vs 0.304), AlexNet(2), AlexNet(5), Inception, EfficientNet-B, and SwAV. With N = 4 and N = 8, every metric is worse than Brain-Diffuser. Moreover, results are reported for subject 1 only, without error bars or significance tests, so even the two improvements are not established as reliable. Section 3.3 itself acknowledges only 'several metrics' at N = 16, which is inconsistent with the abstract's 'consistently.'
[§2.1, Eq. (5)] The abstract and Section 1 describe FreqSelect as 'dynamic' and a 'content-aware gate' that 'dynamically adjusts emphasis' based on relevance, but Eq. (5) defines α_i = σ(w_i), scalar parameters that do not depend on the input image x or on the fMRI signal. After training, the same fixed frequency weighting is applied to every image and every subject. If the intended contribution is input-dependent gating, the implementation does not realize it; at most the weights adapt during training, not at inference. This mismatch between the advertised mechanism and the actual formulation needs to be resolved.
[§3.3, Table 2; §3.2, Figure 6] The abstract claims that 'our method generalizes across subjects and scenes,' but the only quantitative comparison against Brain-Diffuser is for subject 1, as stated in Section 3.3. Figure 6 provides qualitative examples for four subjects, but no subject-wise metrics or statistics are reported. Consequently, cross-subject generalization is not demonstrated by the evidence presented.
[§3.2, §4, Appendix A.1] The 'neuroscientific insight' contribution is overstated. The learned frequency weights are optimized to minimize the latent-space MSE between VDV AE latents of filtered images and fMRI predictions; they are therefore a property of the frozen VDV AE encoder, the ridge regressor, and the training objective. Interpreting these weights as 'interpretable evidence about how the brain represents visual information at different spatial scales' requires additional control analyses (e.g., comparing with random features or with a different encoder), which are not provided. This should be reframed as a stimulus-preprocessing analysis rather than direct evidence of cortical frequency tuning.

minor comments (5)

[References / §3.3] Reference [13] is cited for the claim that Lin et al. first used the Natural Scenes Dataset for fMRI-to-image reconstruction with a StyleGAN2 generator, but the bibliography entry for [13] is a CVPR paper titled 'Deep frequency filtering for domain generalization' and appears unrelated to fMRI decoding. The reported 78.2% Inception value needs a correct source.
[§3.1] The 'Inception Score' is described as 'gauging both diversity and classifiability,' but the metric used in this reconstruction benchmark is typically the correlation of Inception feature activations between reconstruction and ground truth; please clarify the exact computation.
[§3.2, Figure 4] The text says the highest-frequency bands 'often reach negative values,' but Eq. (5) defines α_i = σ(w_i) ∈ (0,1); either the figure plots w_i rather than α_i, or the description is inconsistent. Please label the ordinate and clarify.
[§3.3, Table 2] The sentence before Table 2 says 'Table 2 presents qualitative comparisons,' but it is actually quantitative; also the caption says 'Best scores are in bold' but no bold entries are visible in the text version.
[General] There are scattered typos, e.g., 'we applied the General ROI mask' with a lowercase initial, 'Evaluated on the Natural Scenes dataset' with inconsistent capitalization, and 'effNet-B' in the caption.

Circularity Check

1 steps flagged · score 4.0 of 10

Partial circularity: the claimed neuroscientific insight is the fitted frequency gate restated as brain evidence; the reconstruction comparison itself is an independent held-out test.

fitted input called prediction [Abstract; Section 2.1 Eq. (5); Section 2.2 Eq. (8); Section 3.2 Figure 4 discussion]
"By dynamically emphasizing frequencies that are most predictive of brain activity and suppressing those that are uninformative, FreqSelect acts as a content-aware gate between image features and natural data. ... Beyond performance gains, the learned frequency-selection patterns offer interpretable insights into how different visual frequencies are represented in the brain."

The only coupling between frequency bands and brain data in the method is the Stage-1 objective (Eq. 8), which minimizes ||z_true - z_pred||^2 where z_true is the VDV AE latent of the FreqSelect-filtered image and z_pred is the ridge-regression prediction from fMRI. The weights alpha_i = sigma(w_i) in Eq. (5) are optimized to minimize this objective. Therefore the later statement that the learned patterns show 'how different visual frequencies are represented in the brain' restates the fitted filter values rather than providing an independent measurement or prediction. The paper gives no external neural-frequency validation for the learned weights; 'most predictive of brain activity' is, by construction, the training objective that produced alpha.

full rationale

The reconstruction improvement claim is not circular: FreqSelect and the ridge regressor are trained on the training split, and the reported metrics are computed on held-out test images with a frozen VDV AE and frozen diffusion model. The baseline numbers for Brain-Diffuser are taken from the external paper [19]; there is no load-bearing self-citation chain, no imported uniqueness theorem, and no ansatz smuggled in via self-citation. The absence of error bars, the single-subject Table 1, and the fact that N=16 wins only 2 of 8 metrics are serious support/correctness problems, not circularity. The one genuinely circular element is the secondary 'neuroscientific insight' contribution: Figure 4 and Section A.2 read the learned alpha_i as evidence about brain frequency representation, but alpha_i was optimized solely to make VDV AE latents predictable from fMRI (Eq. 8), so the 'insight' is an interpretation of the fitted weights without independent neural validation. The paper also describes the gate as 'content-aware' and 'dynamic' while Eq. (5) uses input-independent global scalars; that inconsistency is a correctness/scope mismatch, not a circular derivation, and the authors concede in A.3 that current gates are global and radial.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The main fitted quantities are the per-band scalar weights and the hand-chosen band count N. The frozen encoder and frequency-band decomposition are borrowed but unverified assumptions. No new physical entities are introduced.

free parameters (2)

Frequency band weights w_i = learned; see Figure 4(b) for pass-through rates
One scalar per frequency band (N=4, 8, 16), trained jointly with the regression; these determine which frequencies are retained.
Number of bands N = 4, 8, 16; best N=16 selected
Chosen by hand; N=8 leads to strong degradation, so the selection of N=16 is post hoc.

assumptions (3)

domain assumption The pretrained VDV AE encoder, frozen and trained on ImageNet, provides a latent space that is linearly decodable from fMRI signals.
Section 2.2 adopts the exact VDV AE configuration from Brain-Diffuser; the entire method depends on this transfer.
domain assumption Radial frequency bands defined by circular binary masks are an appropriate decomposition for natural images and fMRI noise.
Section 2.1 assumes that spatial-frequency bands capture the signal/noise distinction relevant to brain decoding.
ad hoc to paper The weighted sum with sigmoid and normalization (Eq. 5) is a sufficient and stable gating mechanism.
No theoretical justification is given; at N=8 the mechanism is unstable.

how reviews work

0 comments

Cite this review

Pith. "Pith review of FreqSelect: Frequency-Aware fMRI-to-Image Reconstruction." pith.science (2026). https://pith.science/paper/C7XFO5HD

@misc{pith2026250512552,
  author       = {Pith},
  title        = {Pith review of: FreqSelect: Frequency-Aware fMRI-to-Image Reconstruction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C7XFO5HD}},
  note         = {Machine review of arXiv:2505.12552}
}

read the original abstract

Reconstructing natural images from functional magnetic resonance imaging (fMRI) data remains a core challenge in natural decoding due to the mismatch between the richness of visual stimuli and the noisy, low resolution nature of fMRI signals. While recent two-stage models, combining deep variational autoencoders (VAEs) with diffusion models, have advanced this task, they treat all spatial-frequency components of the input equally. This uniform treatment forces the model to extract meaning features and suppress irrelevant noise simultaneously, limiting its effectiveness. We introduce FreqSelect, a lightweight, adaptive module that selectively filters spatial-frequency bands before encoding. By dynamically emphasizing frequencies that are most predictive of brain activity and suppressing those that are uninformative, FreqSelect acts as a content-aware gate between image features and natural data. It integrates seamlessly into standard very deep VAE-diffusion pipelines and requires no additional supervision. Evaluated on the Natural Scenes dataset, FreqSelect consistently improves reconstruction quality across both low- and high-level metrics. Beyond performance gains, the learned frequency-selection patterns offer interpretable insights into how different visual frequencies are represented in the brain. Our method generalizes across subjects and scenes, and holds promise for extension to other neuroimaging modalities, offering a principled approach to enhancing both decoding accuracy and neuroscientific interpretability.

Figures

Figures reproduced from arXiv: 2505.12552 by the authors.

**Figure 2.** Overview of our fMRI-to-image reconstruction pipeline with FreqSelect (Stage [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Stage 2: A latent diffusion model refines the VDVAE reconstruction using predicted CLIP features from fMRI, injecting high-level semantics and structure, for highlevel image reconstruction. While the VDVAE encoder produces a coherent low-level layout from fMRI, it is limited in capturing high-level semantic content and photorealistic textures. To address this shortcoming and ensure consistency with baseline met… view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: (a) Training dynamics of adaptive frequency band weights for [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Comparison of fMRI reconstructions from Brain-Diffuser [ [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: Example fMRI reconstructions using our FreqSelect module. The first column [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 7.** Figure 7: Failure cases from fMRI reconstructions using FreqSelect. Characterization of reconstruction failures. Reconstruction errors reveal systematic challenges in capturing fine-grained semantic details despite faithful low-frequency structure recovery [PITH_FULL_IMAGE:fig…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 24 canonical work pages

[1]

Allen, Ghislain St-Yves, Yihan Wu, Jesse L

Emily J. Allen, Ghislain St-Yves, Yihan Wu, Jesse L. Breedlove, Jacob S. Prince, Lo- gan T. Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, J.B. Hutchin- son, Thomas Naselaris, and Kendrick Kay. A massive 7t fmri dataset to bridge cog- nitive neuroscience and artificial intelligence. Nature Neuroscience, 25(1):116–126,

work page
[2]

S. Baillet. Magnetoencephalography for brain electrophysiology and imaging. Nature Neuroscience, 2017

work page 2017
[3]

Unsupervised learning of visual features by contrasting cluster assign- ments

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Ar- mand Joulin. Unsupervised learning of visual features by contrasting cluster assign- ments. In Advances in Neural Information Processing Systems , volume 33, pages 9912–9924, 2020

work page 2020
[5]

Very deep vaes generalize autoregressive models and can outperform them on images

Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images. In International Conference on Learning Representations (ICLR) ,

work page
[6]

De Valois, David G

Russell L. De Valois, David G. Albrecht, and Lars G. Thorell. Spatial frequency selec- tivity of cells in macaque visual cortex. Vision Research, 22(5):545–559, 1982. doi: 10.1016/0042-6989(82)90113-4

doi:10.1016/0042-6989(82)90113-4 1982
[7]

Generative multimodal decoding: Reconstructing images and text from human fmri

Matteo Ferrante, Tommaso Boccato, Furkan Ozcelik, Rufin VanRullen, and Nicola Toschi. Generative multimodal decoding: Reconstructing images and text from human fmri. In Deep Generative Models for Health (DGM4H) Workshop, NeurIPS 2023 , De- cember 2023. URL https://dgm4h.github.io/NeurIPS2023/Poster3. pdf. Poster; Submission Number: 3

work page 2023
[8]

Zijin Gu, Keith Jamison, Amy Kuceyeski, and Mert R. Sabuncu. Decoding natural image stimuli from fmri data with a surface-based convolutional network. In Medical Imaging with Deep Learning , volume 227, pages 107–118. PMLR, 2023

work page 2023
[9]

Linda Henriksson, Niina SalminenVaparanta, Heikki Railo, and Simo Vanni. Retino- topic maps, spatial tuning, and locations of human visual areas in surface coordinates 12 YE, W ANG, HOSSAIN: FREQUENCY -AW ARE FMRI-TO-IMAGE RECONSTRUCTION characterized with multifocal and blocked fmri designs.PLoS ONE, 7(5):e36859, 2012. doi: 10.1371/journal.pone.0036859

work page doi:10.1371/journal.pone.0036859 2012

Show all 39 references

[10]

Jie Huang and et al. Li. Adaptive frequency filters as efficient global token mixers. In ICCV, 2023. arXiv:2307.14008

2023 arXiv
[11]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, volume 25, pages 1097–1105, 2012

2012
[12]

Gradient-based learn- ing applied to document recognition

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learn- ing applied to document recognition. Proceedings of the IEEE , 86(11):2278–2324,
[13]

Shiqi Lin, Zhipeng Huang, and et al. Wang. Deep frequency filtering for domain gen- eralization. CVPR, pages XXXX–XXXX, 2023

2023
[14]

Mind- diffuser: Controlled image reconstruction from human brain activity with semantic and structural diffusion

Yizhuo Lu, Changde Du, Qiongyi Zhou, Dianpeng Wang, and Huiguang He. Mind- diffuser: Controlled image reconstruction from human brain activity with semantic and structural diffusion. In Proceedings of the 31st ACM International Conference on Multimedia (MM ’23) , pages 5899–59...

2023
[15]

Mannion, Daniel J

Damien J. Mannion, Daniel J. Kersten, and Cheryl A. Olman. Spatial frequency tuning in human retinotopic visual areas. Journal of Vision, 15(6):14, 2015. doi: 10.1167/15. 6.14

2015 doi
[16]

Tanabe, Norihiro Sadato, and Yukiyasu Kamitani

Yoichi Miyawaki, Hajime Uchida, Okito Yamashita, Masa-aki Sato, Yusuke Morito, Hiroki C. Tanabe, Norihiro Sadato, and Yukiyasu Kamitani. Visual image reconstruc- tion from human brain activity using a combination of multiscale local image decoders. Neuron, 60(5):915–929, 2008....

2008 doi
[17]

M. M. Murray, C. M. Michel, and ... Eeg mapping of neural responses to visual stimuli. NeuroImage, 2005

2005
[18]

Prenger, Kendrick N

Thomas Naselaris, Ryan J. Prenger, Kendrick N. Kay, Michael D. Oliver, and Jack L. Gallant. Bayesian reconstruction of natural images from human brain activity. Neuron, 63(6):902–915, 2009. doi: 10.1016/j.neuron.2009.08.003

2009 doi
[19]

Natural scene reconstruction from fmri signals using generative latent diffusion, 2023

Furkan Ozcelik and Rufin VanRullen. Natural scene reconstruction from fmri signals using generative latent diffusion, 2023. Scientific Reports 13:15666 (2023). DOI:10.1038/s41598-023-42891-8. Supplied as additional material brain-diffuser.pdf

2023 doi
[20]

Note on regression and inheritance in the case of two parents

Karl Pearson. Note on regression and inheritance in the case of two parents. Proceed- ings of the Royal Society of London , 58:240–242, 1895
[21]

Learning transferable visual models from natural lan- guage supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sand- hini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural lan- guage supervision. arXiv preprint a...

2021 arXiv
[22]

On the spectral bias of neural networks

Nadav Rahaman, Dev Arpit, Aristide Baratin, Yan Acosta, Yannis Lin, ..., and Yoshua Bengio. On the spectral bias of neural networks. In Proceedings of the 36th Interna- tional Conference on Machine Learning (ICML) , pages 5301–5310, 2019

2019
[23]

High-resolution image synthesis with latent diffusion models

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Om- mer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10684–10695, June 2022

2022
[24]

U-net: Convolutional networks for biomedical image segmentation

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. CoRR, abs/1505.04597, 2015. URL http:// arxiv.org/abs/1505.04597

2015 arXiv
[25]

Pintea, Jan C

Nikhil Saldanha, Silvia L. Pintea, Jan C. van Gemert, and Nergis Tomen. Frequency learning for structured cnn filters with gaussian fractional derivatives. BMVC, 2021. arXiv:2111.06660

2021 arXiv
[26]

Improved techniques for training GANs

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In Advances in Neural Information Processing Systems, volume 29, pages 2234–2242, 2016

2016
[27]

Laion-5b: An open large-scale dataset for train- ing next generation image-text models, 2022

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wight- man, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. L...

2022 arXiv
[28]

Deep im- age reconstruction from human brain activity

Guohua Shen, Tomoyasu Horikawa, Kei Majima, and Yukiyasu Kamitani. Deep im- age reconstruction from human brain activity. PLOS Computational Biology , 15(1): e1006633, 2019. doi: 10.1371/journal.pcbi.1006633

2019 doi
[29]

K. D. Singh, A. T. Smith, and M. W. Greenlee. Spatio-temporal frequency and direction sensitivities of human visual areas measured using fmri. NeuroImage, 12:550–564,
[30]

Fourier convolution block with global receptive field for mri reconstruction

Haozhong Sun, Yuze Li, Zhongsen Li, Runyu Yang, Ziming Xu, Jiaqi Dou, Haikun Qi, and Huijun Chen. Fourier convolution block with global receptive field for mri reconstruction. Medical Image Analysis, 85:102349, 2024. doi: 10.1016/j.media.2024. 103349

2024 doi
[31]

High-resolution image reconstruction with latent dif- fusion models from human brain activity

Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent dif- fusion models from human brain activity. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14453–14463, 2023

2023
[32]

Efficientnet: Rethinking model scaling for convolutional neural networks

Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pages 6105–6114, 2019. doi: 10.5555/3327144.3327196

2019
[33]

Neural discrete repre- sentation learning

Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete repre- sentation learning. In Advances in Neural Information Processing Systems (NeurIPS) 30, pages 6306–6315, 2017. 14 YE, W ANG, HOSSAIN: FREQUENCY -AW ARE FMRI-TO-IMAGE RECONSTRUCTION

2017
[34]

Bovik, Hamid R

Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004. doi: 10.1109/TIP.2003.819861

2004
[35]

Learning in the frequency domain

Kai Xu, Minghai Qin, Fei Sun, Yuhao Wang, Yen-Kuang Chen, and Fengbo Ren. Learning in the frequency domain. In CVPR, 2020. doi: 10.1109/CVPR42600.2020. 01062

2020
[36]

Versa- tile diffusion: Text, images and variations all in one diffusion model

Xingqian Xu, Zhangyang Wang, Gong Zhang, Kai Wang, and Humphrey Shi. Versa- tile diffusion: Text, images and variations all in one diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 7754– 7765, October 2023. Open access v...

2023
[1998]

doi: 10.1109/5.726791

doi
[2000]

doi: 10.1006/nimg.2000.0649

2000
[2021]

OpenReview: https://openreview.net/forum?id=RLRXCV6DbEJ, code: https://github.com/openai/vdvae
[2022]

doi: 10.1038/s41593-021-00962-x

doi

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Allen, Ghislain St-Yves, Yihan Wu, Jesse L

Emily J. Allen, Ghislain St-Yves, Yihan Wu, Jesse L. Breedlove, Jacob S. Prince, Lo- gan T. Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, J.B. Hutchin- son, Thomas Naselaris, and Kendrick Kay. A massive 7t fmri dataset to bridge cog- nitive neuroscience and artificial intelligence. Nature Neuroscience, 25(1):116–126,

work page

[2] [2]

S. Baillet. Magnetoencephalography for brain electrophysiology and imaging. Nature Neuroscience, 2017

work page 2017

[3] [3]

Unsupervised learning of visual features by contrasting cluster assign- ments

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Ar- mand Joulin. Unsupervised learning of visual features by contrasting cluster assign- ments. In Advances in Neural Information Processing Systems , volume 33, pages 9912–9924, 2020

work page 2020

[4] [5]

Very deep vaes generalize autoregressive models and can outperform them on images

Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images. In International Conference on Learning Representations (ICLR) ,

work page

[5] [6]

De Valois, David G

Russell L. De Valois, David G. Albrecht, and Lars G. Thorell. Spatial frequency selec- tivity of cells in macaque visual cortex. Vision Research, 22(5):545–559, 1982. doi: 10.1016/0042-6989(82)90113-4

doi:10.1016/0042-6989(82)90113-4 1982

[6] [7]

Generative multimodal decoding: Reconstructing images and text from human fmri

Matteo Ferrante, Tommaso Boccato, Furkan Ozcelik, Rufin VanRullen, and Nicola Toschi. Generative multimodal decoding: Reconstructing images and text from human fmri. In Deep Generative Models for Health (DGM4H) Workshop, NeurIPS 2023 , De- cember 2023. URL https://dgm4h.github.io/NeurIPS2023/Poster3. pdf. Poster; Submission Number: 3

work page 2023

[7] [8]

Zijin Gu, Keith Jamison, Amy Kuceyeski, and Mert R. Sabuncu. Decoding natural image stimuli from fmri data with a surface-based convolutional network. In Medical Imaging with Deep Learning , volume 227, pages 107–118. PMLR, 2023

work page 2023

[8] [9]

Linda Henriksson, Niina SalminenVaparanta, Heikki Railo, and Simo Vanni. Retino- topic maps, spatial tuning, and locations of human visual areas in surface coordinates 12 YE, W ANG, HOSSAIN: FREQUENCY -AW ARE FMRI-TO-IMAGE RECONSTRUCTION characterized with multifocal and blocked fmri designs.PLoS ONE, 7(5):e36859, 2012. doi: 10.1371/journal.pone.0036859

work page doi:10.1371/journal.pone.0036859 2012