REVIEW 4 major objections 5 minor 35 references
Improving Brain-to-Image Reconstruction via Fine-Grained Text Bridging
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Fine-grained text decoded from fMRI signals and fused into the semantic conditioning of existing diffusion models improves brain-to-image reconstruction beyond what coarse image captions allow.
desk verdict A plausible and useful text-bridging idea for fMRI-to-image reconstruction, but the main co-training reward L3 is never fully defined, and the evaluation is too thin to support all the headline claims. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the fine-grained text bridge: a detail-enhancement stage, an fMRI-to-text decoder, and a semantic fusion step. Detail enhancement uses LLaVA with a prompt asking for object attributes and spatial relations to regenerate captions that name objects the original NSD captions omit. The decoder maps each subject's voxels through a per-subject linear layer into a prefix embedding, interacts it with a learnable constant embedding in a 32-layer Transformer, and feeds a prefix of length 2l to GPT-2. Training first uses cross-entropy, then adds policy-gradient terms alpha L1 + beta L2 + gamma L3 from the REINFORCE algorithm, where each Li is a reward-weighted log-likelihood; the rewards are object accuracy (Jaccard on extracted nouns), text-image CLIP cosine similarity, and image-image CLIP cosine similarity. At reconstruction time, the CLIP text embedding of the decoded fine-grained text is fused by weighted average with the original high-level semantic embedding, and this fused embedding replaces the semantic condition in the diffusion model while the low-level reconstruction branch is left untouched.
What would settle it
Take a held-out set of NSD test images, have human annotators write detailed captions, and compare them with LLaVA's generated captions; if the object-accuracy reward's noun set disagrees with human annotations on a substantial fraction of images, or if retraining with human-verified captions removes the reported reconstruction gains, the central claim that fine-grained text decoding drives the improvement would be falsified.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that the semantic bottleneck in brain-to-image reconstruction is not the diffusion generator but the granularity of the text that conditions it. The authors show that replacing coarse original NSD captions with detail-enhanced captions produced by a large vision-language model improves reconstruction, and that a unified brain-to-text model can decode such fine-grained text from fMRI when trained with cross-entropy followed by REINFORCE over three non-differentiable rewards: Jaccard object accuracy J(A,B)=|A cap B|/|A cup B| on extracted nouns, CLIP text-image cosine similarity, and CLIP image-image cosine similarity. Fusing the decoded text's CLIP embedding into the high-level semantic conditioning of LDM, BrainDiffuser, or MindEye yields consistent improvements across their evaluation metrics, with the largest gains on LDM, the only one of the three that conditions solely on text.
Load-bearing premise
The whole pipeline assumes that the detail-enhanced captions produced by the vision-language model are accurate ground truth about the image, since both the brain-to-text training target and the object-accuracy reward treat those captions as correct; if the model hallucinates objects, the decoded text and measured gains may reflect caption artifacts rather than better decoding of brain activity.
Editorial extensions
If this is right
- Any existing fMRI-to-image method with a text-based semantic condition can be augmented with decoded fine-grained text without changing its low-level reconstruction branch.
- Methods whose high-level condition is purely textual benefit most, while methods that already mix text and image semantic conditions show smaller but still positive gains.
- The three reward signals are complementary: object accuracy and text-image similarity push semantic alignment, while image-image similarity helps preserve a balance with low-level fidelity.
- A single unified multi-subject brain-to-text model can decode fine-grained descriptions from fMRI, with per-subject linear layers handling voxel-shape differences.
- Decoded fine-grained text can restore details that earlier reconstructions omit, such as objects that were present in the stimulus but absent from the reconstructed image.
Reading between the lines
- Beyond the reported NSD experiments, the same text-bridge should transfer to any fMRI-to-image dataset whose stimuli can be captioned, because the brain-to-text model is trained on whatever captions are supplied and the fusion step is method-agnostic.
- The authors acknowledge that vision-language models hallucinate but do not quantify how often; a natural extension is to filter or downweight training captions whose object nouns disagree with a vision detector, which could remove caption artifacts and possibly enlarge the real gains.
- The result suggests that brain-to-text decoding quality, not diffusion generation, is the current bottleneck for semantic reconstruction; if so, investing in better language decoders should pay off directly in image metrics.
- A further implication is that reconstruction evaluation should include human-verified caption accuracy, since the measured gains may be sensitive to the quality of the LLaVA-generated ground truth.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes FgB2I, a three-stage pipeline for fMRI-to-image reconstruction. First, LLaVA is used to generate detail-enhanced captions for the visual stimuli, replacing the sparse original captions. Second, a unified brain-to-text model is trained on fMRI signals to decode fine-grained text descriptions; training combines cross-entropy loss with a REINFORCE-style policy gradient using three reward metrics: object accuracy (Jaccard similarity of extracted nouns), text-image CLIP semantic similarity, and image-image CLIP semantic similarity. Third, the decoded text embeddings are fused into the high-level semantic conditioning of three existing reconstruction methods (LDM, BrainDiffuser, MindEye). Experiments on NSD (subjects 1, 2, 5, 7) report improvements on several reconstruction metrics, and ablations on LDM are used to attribute gains to individual reward terms.
Significance. If the reported results are robust, the paper offers a practical and minimally invasive way to augment existing fMRI-to-image decoders with finer-grained semantic control, and the detail-enhancement stage is a plausible and reusable contribution. The paper's strengths include integration across three distinct reconstruction pipelines, evaluation on the standard NSD benchmark with multiple metrics, and explicit identification of caption granularity as a bottleneck in semantic reconstruction. The main weaknesses are experimental: the tables report only means with no variance or significance information, the core ablation is on one subject, and the image-image reward (L3) is never concretely defined. In addition, two of the three reward signals are CLIP-based while CLIP accuracy is a reported evaluation metric, which creates a partial circularity that is not addressed. These issues are load-bearing for the central claim that fine-grained text bridging reliably improves reconstruction, and they should be fixed before acceptance.
major comments (4)
- [Method, Fine-Grained Text Descriptions Decoding via Reinforced Co-Training, Eq. (4) and Fig. 4]
- [Method, Fine-Grained Text Descriptions Decoding via Reinforced Co-Training; Results, Main Results (Table 2)]
- [Results, Main Results and Ablation Analysis (Tables 1, 2, 3)]
- [Results, Details Enhancement Results]
minor comments (5)
- [Method, Text-Bridged Brian-to-Image Reconstruction]
- [Figure 4]
- [Eq. (1)]
- [Tables 1 and 2]
- [Experimental Settings]
Circularity Check
No significant circularity found; the reported gains are held-out generalization results rather than by-construction reductions.
full rationale
FgB2I's derivation chain is not circular. The brain-to-text model is trained on training-set fMRI with LLaVA-generated captions as targets and three policy-gradient rewards, then evaluated on the 982 held-out common test images; no test-time reward is computed, so the reported CLIP and accuracy gains are empirical generalization claims that could fail (indeed Table 1 shows some metric decreases, e.g., BrainDiffuser+DE Alex(2) at 93.6% versus 94.2%). The CLIP-based L2 reward does not make the CLIP evaluation metric circular for the same reason: at test time the text is sampled from the fMRI-conditioned policy without access to the ground-truth image or reward. The only self-citation (Xia, Yin, and Li 2024) appears in a related-work sentence on attention and memory and is not load-bearing. Two non-circular risks should be flagged: L3 is underspecified (Fig. 4 says "decoded image", but the paper never defines how that image is produced during text-decoder training), making Table 3's L3 attribution irreproducible; and the paper itself concedes "LVLMs inevitably hallucinate, producing inaccuracies that can sometimes limit the effectiveness of detail enhancement," so the pseudo-label targets are noisy. Neither risk is a by-construction equivalence between inputs and outputs.
Assumptions & free parameters
free parameters (5)
- reward trade-off weights alpha, beta, gamma =
0.01 each
- fMRI prefix length l =
10
- maximum generated text length =
77
- abstract noun filter list for object accuracy =
excludes words such as 'image', 'photo'
- LLaVA prompt for detail enhancement =
hand-written instruction quoted in Method
assumptions (6)
- domain assumption fMRI signals from the 'nsdgeneral' template contain sufficient fine-grained visual semantic information to decode detailed object and attribute descriptions.
- domain assumption CLIP embeddings are a valid measure of semantic similarity between text, images, and reconstructions.
- domain assumption LLaVA-1.5-7B-generated captions are accurate and complete enough to serve as ground-truth text for training and reward computation.
- standard math REINFORCE policy gradient provides a valid unbiased gradient for the non-differentiable reward objectives.
- domain assumption fMRI responses of subjects 1, 2, 5, and 7 and the shared test set are representative for evaluating the method.
- domain assumption The high-level CLIP conditioning in LDM, BrainDiffuser, and MindEye can be replaced or fused with text embeddings without breaking these pipelines.
Cite this review
Pith. "Pith review of Improving Brain-to-Image Reconstruction via Fine-Grained Text Bridging." pith.science (2026). https://pith.science/paper/6ZEEL23H
@misc{pith2026250522150,
author = {Pith},
title = {Pith review of: Improving Brain-to-Image Reconstruction via Fine-Grained Text Bridging},
year = {2026},
howpublished = {\url{https://pith.science/paper/6ZEEL23H}},
note = {Machine review of arXiv:2505.22150}
}
read the original abstract
Brain-to-Image reconstruction aims to recover visual stimuli perceived by humans from brain activity. However, the reconstructed visual stimuli often missing details and semantic inconsistencies, which may be attributed to insufficient semantic information. To address this issue, we propose an approach named Fine-grained Brain-to-Image reconstruction (FgB2I), which employs fine-grained text as bridge to improve image reconstruction. FgB2I comprises three key stages: detail enhancement, decoding fine-grained text descriptions, and text-bridged brain-to-image reconstruction. In the detail-enhancement stage, we leverage large vision-language models to generate fine-grained captions for visual stimuli and experimentally validate its importance. We propose three reward metrics (object accuracy, text-image semantic similarity, and image-image semantic similarity) to guide the language model in decoding fine-grained text descriptions from fMRI signals. The fine-grained text descriptions can be integrated into existing reconstruction methods to achieve fine-grained Brain-to-Image reconstruction.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline " cite write " FUNCTION editor.postfix editor num.names #1 > "( )" "( )" if FUNCTION editor.trans.postfix editor num.names #1 > "( )" "( )" if FUNCTION trans.postfix translator num.names #1 > "( )" "( )" if FUNCTION authors.editors.reflist.apa5 'field := 'dot := field num.names 'numnames := numnames 'format.num.names := format.num.names na...
-
[2]
allen2022massive APACrefauthors Allen, E J. , St-Yves, G. , Wu, Y. , Breedlove, J L. , Prince, J S. , Dowdle, L T. others APACrefauthors \ 2022 . A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence . Nature neuroscience 25 1 116--126
work page 2022
-
[3]
caron2020unsupervised APACrefauthors Caron, M. , Misra, I. , Mairal, J. , Goyal, P. , Bojanowski, P. \ Joulin, A. APACrefauthors \ 2020 . Unsupervised learning of visual features by contrasting cluster assignments Unsupervised learning of visual features by contrasting cluster assignments . Advances in neural information processing systems 33 9912--9924
work page 2020
-
[4]
chen2023rethinking APACrefauthors Chen, J. , Qi, Y. \ Pan, G. APACrefauthors \ 2023 . Rethinking Visual Reconstruction: Experience-Based Content Completion Guided by Visual Cues Rethinking visual reconstruction: Experience-based content completion guided by visual cues . 202 4856--4866 . APACrefURL https://proceedings.mlr.press/v202/chen23v.html APACrefURL
work page 2023
-
[5]
child2020very APACrefauthors Child, R. APACrefauthors \ 2020 . Very deep vaes generalize autoregressive models and can outperform them on images Very deep vaes generalize autoregressive models and can outperform them on images . arXiv preprint arXiv:2011.10650
arXiv 2020
-
[6]
child2021deep APACrefauthors Child, R. APACrefauthors \ 2021 . Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images Very deep vaes generalize autoregressive models and can outperform them on images . 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. 9th international co...
work page 2021
-
[7]
chun2011memory APACrefauthors Chun, M M. \ Johnson, M K. APACrefauthors \ 2011 . Memory: Enduring traces of perceptual and reflective attention Memory: Enduring traces of perceptual and reflective attention . Neuron 72 4 520--535
work page 2011
-
[8]
glover2011overview APACrefauthors Glover, G H. APACrefauthors \ 2011 . Overview of functional magnetic resonance imaging Overview of functional magnetic resonance imaging . Neurosurgery Clinics 22 2 133--139
work page 2011
Show all 35 references
-
[9]
\ Milner, A D
goodale1992separate APACrefauthors Goodale, M A. \ Milner, A D. APACrefauthors \ 1992 . Separate visual pathways for perception and action Separate visual pathways for perception and action . Trends in neurosciences 15 1 20--25
1992
-
[10]
, Schwartz, J H
kandel2000principles APACrefauthors Kandel, E R. , Schwartz, J H. , Jessell, T M. , Siegelbaum, S. , Hudspeth, A J. , Mack, S. \ . APACrefauthors \ 2000 . Principles of neural science Principles of neural science \ ( 4). McGraw-hill New York
2000
-
[11]
, Sutskever, I
krizhevsky2012imagenet APACrefauthors Krizhevsky, A. , Sutskever, I. \ Hinton, G E. APACrefauthors \ 2012 . Imagenet classification with deep convolutional neural networks Imagenet classification with deep convolutional neural networks . ( 25)
2012
-
[12]
\ Liang, P
li2021prefix APACrefauthors Li, X L. \ Liang, P. APACrefauthors \ 2021 . Prefix-tuning: Optimizing continuous prompts for generation Prefix-tuning: Optimizing continuous prompts for generation . arXiv preprint arXiv:2101.00190
2021 arXiv
- [13]
-
[14]
\ Hutter, F
loshchilov2019decoupled APACrefauthors Loshchilov, I. \ Hutter, F. APACrefauthors \ 2019 . Decoupled Weight Decay Regularization Decoupled weight decay regularization . 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. 7t...
2019
-
[15]
lu2023minddiffuser APACrefauthors Lu, Y. , Du, C. , Zhou, Q. , Wang, D. \ He, H. APACrefauthors \ 2023 . MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion Minddiffuser: Controlled image reconstruction from human brai...
2023
-
[16]
\ Ford, M A
luck1998role APACrefauthors Luck, S J. \ Ford, M A. APACrefauthors \ 1998 . On the role of selective attention in visual perception On the role of selective attention in visual perception . Proceedings of the National Academy of Sciences 95 3 825--830
1998
-
[17]
, Kavukcuoglu, K
DBLP:journals/nature/MnihKSRVBGRFOPB15 APACrefauthors Mnih, V. , Kavukcuoglu, K. , Silver, D. , Rusu, A A. , Veness, J. , Bellemare, M G. Hassabis, D. APACrefauthors \ 2015 . Human-level control through deep reinforcement learning Human-level control through deep reinforcement...
2015 doi
-
[18]
, Singthongchai, J
niwattanakul2013using APACrefauthors Niwattanakul, S. , Singthongchai, J. , Naenudorn, E. \ Wanapu, S. APACrefauthors \ 2013 . Using of Jaccard coefficient for keywords similarity Using of jaccard coefficient for keywords similarity . Proceedings of the international multiconf...
2013
-
[19]
\ VanRullen, R
ozcelik2303brain APACrefauthors Ozcelik, F. \ VanRullen, R. APACrefauthors \ 2023 . Natural scene reconstruction from fMRI signals using generative latent diffusion Natural scene reconstruction from fmri signals using generative latent diffusion . Scientific Reports 13 1 15666
2023
-
[20]
, Huth, A G
popham2021visual APACrefauthors Popham, S F. , Huth, A G. , Bilenko, N Y. , Deniz, F. , Gao, J S. , Nunez-Elizalde, A O. \ Gallant, J L. APACrefauthors \ 2021 . Visual and linguistic semantic representations are aligned at the border of human visual cortex Visual and linguisti...
2021
-
[21]
, Kim, J W
radford2021learning APACrefauthors Radford, A. , Kim, J W. , Hallacy, C. , Ramesh, A. , Goh, G. , Agarwal, S. others APACrefauthors \ 2021 . Learning transferable visual models from natural language supervision Learning transferable visual models from natural language supervis...
2021
-
[22]
radford2019language APACrefauthors Radford, A. , Wu, J. , Child, R. , Luan, D. , Amodei, D. , Sutskever, I. \ . APACrefauthors \ 2019 . Language models are unsupervised multitask learners Language models are unsupervised multitask learners . OpenAI blog 1 8 9
2019
-
[23]
, Banerjee, A
scotti2023reconstructing APACrefauthors Scotti, P S. , Banerjee, A. , Goode, J. , Shabalin, S. , Nguyen, A. , Cohen, E. Abraham, T M. APACrefauthors \ 2023 . Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors Reconstructing the mind's e...
-
[24]
, McAllester, D
sutton1999policy APACrefauthors Sutton, R S. , McAllester, D. , Singh, S. \ Mansour, Y. APACrefauthors \ 1999 . Policy gradient methods for reinforcement learning with function approximation Policy gradient methods for reinforcement learning with function approximation . Advan...
1999
-
[25]
, Vanhoucke, V
szegedy2016rethinking APACrefauthors Szegedy, C. , Vanhoucke, V. , Ioffe, S. , Shlens, J. \ Wojna, Z. APACrefauthors \ 2016 . Rethinking the Inception Architecture for Computer Vision Rethinking the inception architecture for computer vision . 2016 IEEE Conference on Computer ...
2016
-
[26]
\ Nishimoto, S
takagi2023high APACrefauthors Takagi, Y. \ Nishimoto, S. APACrefauthors \ 2023 . High-resolution image reconstruction with latent diffusion models from human brain activity High-resolution image reconstruction with latent diffusion models from human brain activity . IEEE/CVF C...
2023
-
[27]
\ Le, Q V
tan1905efficientnet APACrefauthors Tan, M. \ Le, Q V. APACrefauthors \ 2019 . EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks Efficientnet: Rethinking model scaling for convolutional neural networks . K. Chaudhuri\ R. Salakhutdinov\ ( ), Proceedings of...
2019
-
[28]
, Vinyals, O
van2017neural APACrefauthors Van Den Oord, A. , Vinyals, O. \ . APACrefauthors \ 2017 . Neural discrete representation learning Neural discrete representation learning . Advances in neural information processing systems 30
2017
-
[29]
, Bovik, A C
wang2004image APACrefauthors Wang, Z. , Bovik, A C. , Sheikh, H R. \ Simoncelli, E P. APACrefauthors \ 2004 . Image quality assessment: from error visibility to structural similarity Image quality assessment: from error visibility to structural similarity . IEEE Trans. Image P...
2004
-
[30]
wang2022open APACrefauthors Wang, Z. \ Ji, H. APACrefauthors \ 2022 . Open Vocabulary Electroencephalography-to-Text Decoding and Zero-Shot Sentiment Classification Open vocabulary electroencephalography-to-text decoding and zero-shot sentiment classification . Thirty-Sixth AA...
2022
-
[31]
, Zhao, S
xi2023unicorn APACrefauthors Xi, N. , Zhao, S. , Wang, H. , Liu, C. , Qin, B. \ Liu, T. APACrefauthors \ 2023 . UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human language Unicorn: Unified cognitive signal reconstruction bridging cognitive si...
2023 arXiv
-
[32]
, Yin, C
xia2024decoding APACrefauthors Xia, R. , Yin, C. \ Li, P. APACrefauthors \ 2024 . Decoding the Echoes of Vision from fMRI: Memory Disentangling for Past Semantic Information Decoding the echoes of vision from fmri: Memory disentangling for past semantic information . Proceedin...
2024
-
[33]
, Wang, Z
xu2023versatile APACrefauthors Xu, X. , Wang, Z. , Zhang, G. , Wang, K. \ Shi, H. APACrefauthors \ 2023 . Versatile diffusion: Text, images and variations all in one diffusion model Versatile diffusion: Text, images and variations all in one diffusion model . Proceedings of th...
2023
-
[34]
yin2024language APACrefauthors Yin, C. , Ye, Z. \ Li, P. APACrefauthors \ 2024 . Language Reconstruction with Brain Predictive Coding from fMRI Data Language reconstruction with brain predictive coding from fmri data . arXiv preprint arXiv:2405.11597
2024 arXiv
-
[35]
, Han, K
zhang2020connecting APACrefauthors Zhang, Y. , Han, K. , Worth, R. \ Liu, Z. APACrefauthors \ 2020 . Connecting concepts in the brain by mapping cortical representations of semantic relations Connecting concepts in the brain by mapping cortical representations of semantic rela...
2020
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.