REVIEW 4 major objections 6 minor 35 references
Predicting Human Brain States with Transformer
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read A transformer trained on 30 resting-state fMRI time points can predict the next seven brain states, roughly 5.04 seconds, with MSE below 0.15.
desk verdict Compact transformer predicts 5.04s of rs-fMRI, but the missing persistence/AR baseline undercuts the 'learned dynamics' claim for now. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is a transformer encoder-decoder treated as an autoregressive time-series forecaster. Each token is one brain state, a 379-dimensional vector of mean regional BOLD signals defined by the multi-modal parcellation atlas, and sine/cosine positional encodings inject temporal order. The encoder reads 30 consecutive tokens; the decoder receives the encoder output plus the last true token and emits a single next state, trained with mean squared error. Self-attention is what allows the model to weight relationships among distant past states, and the paper's case that this matters rests on the contrast between low error on ordered sequences and high error on shuffled ones.
What would settle it
Compute a persistence forecast on the same held-out HCP subjects, setting each predicted state equal to the previous true or predicted state, and measure the MSE over the first seven steps; if that baseline reaches or beats the transformer's 0.15, the short-horizon accuracy is explained by signal smoothness rather than by the model's learned dynamics.
Extended reading notes
Core claim
The central claim is that a transformer, trained as an autoregressive model on regional resting-state fMRI, learns enough of the brain's temporal dynamics to predict the immediate future: a single-step MSE of 0.0013, the first seven multi-step predictions below 0.15, and a shuffled-order control error of 0.97 with p < $10^{-10}$. The paper interprets the shuffled control as evidence that sequential order is indispensable; otherwise a model with the same inputs but random order would perform similarly. For long-range synthesis, prediction error grows as predicted states are fed back into the model, but the group-average functional connectivity matrix computed from generated time series correlates with the true group-average matrix at 0.50 to 0.60, indicating that the model captures the population-level organization of functional networks.
Load-bearing premise
The claim rests on the assumption that the low prediction error comes from learned brain dynamics rather than from the temporal smoothness of bandpass-filtered fMRI, and the paper reports no comparison against repeating the last observed state or a linear autoregressive model.
Editorial extensions
If this is right
- A 21.6-second resting-state fMRI segment may suffice to generate roughly five seconds of plausible future brain activity, shortening the scan time needed for applications that depend on a short predicted window.
- Long autoregressive generation, despite accumulating error, preserves the group-average functional connectome, so the model captures shared functional organization across individuals.
- The 700-fold jump in error when input frames are shuffled implies temporal ordering is essential to the model's performance, not a side effect of static regional relationships.
- Error accumulation limits reliable prediction to about seven time points, so further work on reducing cascading errors is required before longer forecasts could be used in practice.
Reading between the lines
- The paper does not compare against a persistence baseline (x_{t+1} = x_t) or a linear autoregressive model; until such baselines are shown to do worse, the unique contribution of the transformer to short-horizon prediction remains an open question.
- Because the model operates on z-scored regional signals, its reported MSE values are relative to unit-variance time series; comparing forecast errors across studies would require reporting errors in raw BOLD units or per-region variances.
- The same encoder-decoder design could be tested on task-evoked fMRI and on clinical cohorts, where the forecasting horizon and functional-connectivity fidelity would need to be weighed against scan time in concrete diagnostic tasks.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes a transformer-based autoregressive model for predicting resting-state fMRI brain states. The model takes 30 consecutive time points (21.6 s at TR=0.72 s) of 379 regional BOLD signals from the HCP dataset and predicts the next time point; iterative application generates a synthetic time series of up to 1150 time points. The authors report a single-step MSE of 0.0013 (versus 0.97 under a shuffled-order control), MSE below 0.15 for the first seven multi-step predictions (5.04 s), and similarity between the functional-connectivity matrices computed from true and predicted data (spatial correlation 0.50–0.60). Ten-fold subject-level cross-validation on 1003 HCP subjects is used, the window size and epoch count were chosen in a preliminary experiment, and the code is publicly available.
Significance. If the claims hold, the paper would demonstrate a generative model that produces plausible future brain states from a short fMRI segment, with potential applications in scan-time reduction and in generating synthetic resting-state data. The evaluation has genuine strengths: held-out subjects at the cross-validation level, a shuffled-order control as a sanity check, and public code that supports reproducibility. The Markov-chain interpretation of error accumulation is clearly stated. However, the central claim that the transformer learns brain dynamics rather than exploiting the strong temporal autocorrelation of the BOLD signal is not yet supported, because trivial baselines (persistence, AR(1)) are absent and the 'accurate up to 5.04 s' criterion is defined post hoc. The functional-connectivity result is also not discriminating between temporal dynamics and mere spatial covariance reproduction. These gaps make the significance conditional on the requested experiments.
major comments (4)
- [§3.2, §3.3 (with §2.1)] The central claim that the transformer 'learned and leveraged the temporal dependencies' and can accurately predict brain states up to 5.04 s is not established by the current evaluation, which uses only a shuffled-order control. The bandpass filter (0.01–0.1 Hz) applied at Section 2.1, combined with TR=0.72 s, induces strong temporal autocorrelation in the regional BOLD series, so a persistence forecast (x̂_{t+1}=x_t) or a per-region AR(1) model may plausibly reproduce the reported single-step MSE of 0.0013 and the low multi-step MSE over the first seven time points (Fig. 3b). The shuffled-input MSE of 0.97 is close to 1.0—the MSE of predicting the temporal mean of z-scored data—so the 700-fold difference mainly shows that order carries information, not that the transformer extracts dynamics beyond simple extrapolation. Please add persistence and AR(p) baselines evaluated on the same held-out subjects with the same metrics (MSE, correlation, and FC analysis), and report whether the transformer's errors are significantly lower.
- [§3.3] The paper defines the accuracy window as 'the first seven predicted time points (within the time of 5.04s) have relatively low MSE (< 0.15)'. This threshold is chosen after observing the results (Fig. 3b), and no pre-specified criterion, noise ceiling, or baseline-calibrated value is given. The 'accurate up to 5.04 s' conclusion is therefore post hoc. Please justify the threshold a priori, or calibrate it against the trivial baselines above, and report the distribution (mean ± SD across subjects) of the MSE at each of the first seven horizons rather than only the group average.
- [§3.3] The comparison with BrainLM ('their 20-time-point MSE is 0.568, larger than ours') is not controlled: the authors' model uses a 30-time-point input, whereas the BrainLM evaluation cited in the Introduction uses a longer input (180 time points), and the preprocessing, parcellation, and data splits differ. Under these conditions the comparison does not support the claim of superiority. Either re-implement BrainLM on the same preprocessing and folds, or present the comparison only as indicative and state explicitly which factors could explain the difference.
- [§3.4] The functional-connectivity analysis does not distinguish learned temporal dynamics from reproduced spatial covariance. The group-average FC pattern in Fig. 4b is determined by the cross-regional covariance of the signals; any generator that draws from a multivariate distribution with the empirical covariance (for instance, a null model that temporally shuffles each region's series independently) would produce a similar FC similarity (r ≈ 0.5–0.6 in Fig. 4e) without capturing sequential structure. Please compare the FC of the transformer output against that of a spatial-covariance null model and of the persistence baseline, and report whether the similarity is significantly higher.
minor comments (6)
- [Fig. 3 caption] The caption contains grammatical slips (e.g., 'which is the MSE of the 51st time points'); the entire manuscript would benefit from careful copy-editing for typos and phrasing.
- [Fig. 2] Model selection in Fig. 2 is based on single runs; no error bars, standard deviations across subjects, or multiple random seeds are reported, so the choice of window size 30 and 20 epochs is not shown to be robust.
- [§2.2] The decoder input description is ambiguous: 'the last time point of the encoder input combined with the encoder output' should be specified precisely, including tensor shapes and whether the decoder uses causal masking during the iterative autoregressive generation described in Section 2.4.
- [§2.3] It is unclear whether the MSE values in Sections 3.2 and 3.3 are averaged over the ten cross-validation folds or produced by a single final model; the manuscript should state this explicitly.
- [§3.4] In the text 'with MSE from 0.35 to 0.45 and correlation from 0.50 to 0.60', the quantity labeled MSE is the mean absolute difference between FC matrices shown in Fig. 4e; using 'MSE' here conflates two different error metrics and should be corrected.
- [§1] The claim that short-segment prediction 'can largely shorten the scanning time' is not directly demonstrated; a sentence acknowledging that the clinical benefit is currently speculative would be more accurate.
Circularity Check
No significant circularity: the transformer is trained and evaluated on disjoint subject splits, and the reported brain-state predictions are true out-of-sample forecasts.
full rationale
The paper's claim chain is empirical rather than derivational: a transformer is trained to minimize MSE on rs-fMRI sequences from 90% of HCP subjects and evaluated on held-out subjects (Section 2.3: 'among the 1003 subjects, 90% of subjects ... were used to train the model, while the remaining 10% of subjects ... were used for validation'). The reported single-step MSE of 0.0013 and multi-step errors are computed on unseen-subject data, not on the training set, so the numbers are not fits renamed as predictions. The window size and epoch count are selected on a separate forty-subject preliminary set before final training, and the evaluation protocol is fixed before reading the test MSE; no parameter is fitted to the test quantities. The shuffle control (Section 3.2) is an internal consistency check rather than a load-bearing circular step. The absence of a persistence or linear autoregressive baseline is a genuine risk to the strength of the interpretation, because bandpass-filtered BOLD time series are strongly autocorrelated, but that is a missing-control/interpretation weakness, not a reduction of the prediction to its input by construction or by self-citation. No step in the paper defines a predicted quantity in terms of the same quantity, renames a known result, or imports a uniqueness conclusion from the authors' own prior work; the cited prior work (BrainLM, transformer architectures) is external and not used to justify the authors' own results.
Assumptions & free parameters
free parameters (2)
- window_size =
30
- num_epochs =
10 (full data)
assumptions (4)
- domain assumption Resting-state fMRI signals after bandpass filtering and z-scoring are sufficiently stationary and autocorrelated to be predictable.
- domain assumption Mean squared error on z-scored signals is an appropriate measure of prediction accuracy.
- domain assumption The 379-region MMP parcellation captures the relevant brain states.
- ad hoc to paper The transformer architecture can model temporal dependencies in fMRI better than simpler linear models.
Cite this review
Pith. "Pith review of Predicting Human Brain States with Transformer." pith.science (2026). https://pith.science/paper/XWYSRLFS
@misc{pith2026241219814,
author = {Pith},
title = {Pith review of: Predicting Human Brain States with Transformer},
year = {2026},
howpublished = {\url{https://pith.science/paper/XWYSRLFS}},
note = {Machine review of arXiv:2412.19814}
}
read the original abstract
The human brain is a complex and highly dynamic system, and our current knowledge of its functional mechanism is still very limited. Fortunately, with functional magnetic resonance imaging (fMRI), we can observe blood oxygen level-dependent (BOLD) changes, reflecting neural activity, to infer brain states and dynamics. In this paper, we ask the question of whether the brain states rep-resented by the regional brain fMRI can be predicted. Due to the success of self-attention and the transformer architecture in sequential auto-regression problems (e.g., language modelling or music generation), we explore the possi-bility of the use of transformers to predict human brain resting states based on the large-scale high-quality fMRI data from the human connectome project (HCP). Current results have shown that our model can accurately predict the brain states up to 5.04s with the previous 21.6s. Furthermore, even though the prediction error accumulates for the prediction of a longer time period, the gen-erated fMRI brain states reflect the architecture of functional connectome. These promising initial results demonstrate the possibility of developing gen-erative models for fMRI data using self-attention that learns the functional or-ganization of the human brain. Our code is available at: https://github.com/syf0122/brain_state_pred.
Reference graph
Works this paper leans on
-
[1]
Frontiers in human neuroscience, 2009: p
Herculano-Houzel, S., The human brain in numbers: a linearly scaled -up primate brain. Frontiers in human neuroscience, 2009: p. 31
work page 2009
-
[2]
Shine, J.M., et al., The dynamics of functional brain networks: integrated network states during cognitive task performance. Neuron, 2016. 92(2): p. 544-554
work page 2016
-
[3]
Barch, D.M., et al., Function in the human connectome: task -fMRI and individual differences in behavior. Neuroimage, 2013. 80: p. 169-189
work page 2013
-
[4]
Dennis, E.L. and P.M. Thompson, Functional brain connectivity using fMRI in aging and Alzheimer’s disease. Neuropsychology review, 2014. 24: p. 49-62
work page 2014
-
[5]
Lupton, M.K., et al., A prospective cohort study of prodromal Alzheimer’s disease: prospective imaging study of ageing: genes, brain and behaviour (PISA). NeuroImage: Clinical, 2021. 29: p. 102527
work page 2021
-
[6]
Yu, Q., et al., Assessing dynamic brain graphs of time-varying connectivity in fMRI data: application to healthy controls and patients with schizophrenia. Neuroimage, 2015. 107: p. 345-355
work page 2015
-
[7]
Rashid, B., et al., Classification of schizophrenia and bipolar patients using static and dynamic resting-state fMRI brain connectivity. Neuroimage, 2016. 134: p. 645-657
work page 2016
-
[8]
IEEE Transactions on Neural Networks and Learning Systems, 2022
Hou, Y., et al., GCNs-net: a graph convolutional neural network approach for decoding time-resolved eeg motor imagery signals. IEEE Transactions on Neural Networks and Learning Systems, 2022
work page 2022
Show all 35 references
-
[9]
Frontiers in Bioengineering and Biotechnology, 2022
Hou, Y., et al., Deep feature mining via the attention-based bidirectional long short term memory graph convolutional neural network for human motor imagery recognition. Frontiers in Bioengineering and Biotechnology, 2022. 9: p. 706229
2022
-
[10]
IEEE transactions on biomedical engineering, 2012
Deng, F., et al., FMRI signal analysis using empirical mean curve decomposition. IEEE transactions on biomedical engineering, 2012. 60(1): p. 42-54
2012
-
[11]
IEEE Transactions on Biomedical Engineering, 2014
Lv, J., et al., Holistic atlases of functional networks and interactions reveal reciprocal organizational architecture of cortical function. IEEE Transactions on Biomedical Engineering, 2014. 62(4): p. 1120-1131
2014
-
[12]
Medical image analysis, 2015
Lv, J., et al., Sparse representation of whole -brain fMRI signals for identification of functional networks. Medical image analysis, 2015. 20(1): p. 112-134
2015
-
[13]
Latent source mining in FMRI data via deep neural network
Huang, H., et al. Latent source mining in FMRI data via deep neural network. in 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI). 2016. IEEE
2016
-
[14]
IEEE transactions on biomedical engineering, 2019
Dong, Q., et al., Modeling hierarchical brain networks via volumetric sparse deep belief network. IEEE transactions on biomedical engineering, 2019. 67(6): p. 1739-1748
2019
-
[15]
Medical Image Analysis, 2023
Zhang, S., et al., An explainable deep learning framework for characterizing and interpreting human brain states. Medical Image Analysis, 2023. 83: p. 102665
2023
-
[16]
Annals of Neurology: Official Journal of the American Neurological Association and the Child Neurology Society, 2006
Waites, A.B., et al., Functional connectivity networks are disrupted in left temporal lobe epilepsy. Annals of Neurology: Official Journal of the American Neurological Association and the Child Neurology Society, 2006. 59(2): p. 335-343
2006
-
[17]
Von Cramon, and R.I
Bubic, A., D.Y. Von Cramon, and R.I. Schubotz, Prediction, cognition and the brain. Frontiers in human neuroscience, 2010: p. 25
2010
-
[18]
Advances in neural information processing systems, 2017
Vaswani, A., et al., Attention is all you need. Advances in neural information processing systems, 2017. 30
2017
-
[19]
arXiv preprint arXiv:1809.04281, 2018
Huang, C.-Z.A., et al., Music transformer. arXiv preprint arXiv:1809.04281, 2018. 11
2018 arXiv
-
[20]
Advances in Neural Information Processing Systems, 2020
Stiennon, N., et al., Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 2020. 33: p. 3008-3021
2020
-
[21]
Swin transformer: Hierarchical vision transformer using shifted windows
Liu, Z., et al. Swin transformer: Hierarchical vision transformer using shifted windows. in Proceedings of the IEEE/CVF international conference on computer vision. 2021
2021
-
[22]
arXiv preprint arXiv:2010.11929, 2020
Dosovitskiy, A., et al., An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020
2010 arXiv
- [23]
-
[24]
Advances in Neural Information Processing Systems, 2024
Kim, P., et al., SwiFT: Swin 4D fMRI Transformer. Advances in Neural Information Processing Systems, 2024. 36
2024
-
[25]
Self-Supervised Transformers for fMRI representation
Malkiel, I., et al. Self-Supervised Transformers for fMRI representation . in International Conference on Medical Imaging with Deep Learning. 2022. PMLR
2022
-
[26]
bioRxiv, 2023: p
Ortega Caro, J., et al., BrainLM: A foundation model for brain activity recordings. bioRxiv, 2023: p. 2023.09. 12.557460
2023
-
[27]
arXiv preprint arXiv:2001.08317, 2020
Wu, N., et al., Deep transformer models for time series forecasting: The influenza prevalence case. arXiv preprint arXiv:2001.08317, 2020
2001 arXiv
-
[28]
Neuroimage, 2013
Van Essen, D.C., et al., The WU -Minn human connectome project: an overview. Neuroimage, 2013. 80: p. 62-79
2013
-
[29]
Neuroimage, 2013
Glasser, M.F., et al., The minimal preprocessing pipelines for the Human Connectome Project. Neuroimage, 2013. 80: p. 105-124
2013
-
[30]
Meyerand, and R.M
Molloy, E.K., M.E. Meyerand, and R.M. Birn, The influence of spatial resolution and smoothing on the detectability of resting-state and task fMRI. Neuroimage, 2014. 86: p. 221- 230
2014
-
[31]
Scientific reports, 2017
Lin, P., et al., Dynamic default mode network across different brain states. Scientific reports, 2017. 7(1): p. 46088
2017
-
[32]
Glasser, M.F., et al., A multi -modal parcellation of human cerebral cortex. Nature,
-
[33]
Kingma, D.P. and J. Ba, Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014
2014 arXiv
-
[34]
2022, River Publishers
Agbinya, J.I., Markov chain and its applications, in Applied Data Analytics-Principles and Applications. 2022, River Publishers. p. 1-15
2022
-
[2016]
536(7615): p. 171-178
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.