Pith. sign in

REVIEW 4 major objections 6 minor 35 references

Predicting Human Brain States with Transformer

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Pith's one-line read A transformer trained on 30 resting-state fMRI time points can predict the next seven brain states, roughly 5.04 seconds, with MSE below 0.15.

desk verdict Compact transformer predicts 5.04s of rs-fMRI, but the missing persistence/AR baseline undercuts the 'learned dynamics' claim for now. read the letter →

arxiv 2412.19814 v1 pith:XWYSRLFS submitted 2024-12-11 q-bio.NC cs.AIcs.LG

classification q-bio.NCcs.AIcs.LG
keywords fMRIbrainstatestransformerpredictionresting-statefunctionalconnectivityautoregressivemodelHumanConnectomeProject
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether future brain states, represented by the mean blood-oxygen-level-dependent (BOLD) activity of 379 regions at each fMRI time point, can be predicted from a short history. It trains a transformer on resting-state fMRI from the Human Connectome Project and reports that, given 30 time points (21.6 seconds), the model predicts the next brain state with mean squared error (MSE) 0.0013 and the next seven states (5.04 seconds) with error below 0.15. Shuffling the input order raises the single-step error more than 700-fold, which the authors read as evidence that temporal structure, not static features, drives the prediction. Over longer horizons the error accumulates, yet the fully generated time series still reproduces the group-level functional connectivity of the brain. If these results hold, short fMRI scans could be extended synthetically and generative models of brain dynamics may become practical.

What carries the argument

The load-bearing object is a transformer encoder-decoder treated as an autoregressive time-series forecaster. Each token is one brain state, a 379-dimensional vector of mean regional BOLD signals defined by the multi-modal parcellation atlas, and sine/cosine positional encodings inject temporal order. The encoder reads 30 consecutive tokens; the decoder receives the encoder output plus the last true token and emits a single next state, trained with mean squared error. Self-attention is what allows the model to weight relationships among distant past states, and the paper's case that this matters rests on the contrast between low error on ordered sequences and high error on shuffled ones.

What would settle it

Compute a persistence forecast on the same held-out HCP subjects, setting each predicted state equal to the previous true or predicted state, and measure the MSE over the first seven steps; if that baseline reaches or beats the transformer's 0.15, the short-horizon accuracy is explained by signal smoothness rather than by the model's learned dynamics.

Watch

Extended reading notes

Core claim

The central claim is that a transformer, trained as an autoregressive model on regional resting-state fMRI, learns enough of the brain's temporal dynamics to predict the immediate future: a single-step MSE of 0.0013, the first seven multi-step predictions below 0.15, and a shuffled-order control error of 0.97 with p < $10^{-10}$. The paper interprets the shuffled control as evidence that sequential order is indispensable; otherwise a model with the same inputs but random order would perform similarly. For long-range synthesis, prediction error grows as predicted states are fed back into the model, but the group-average functional connectivity matrix computed from generated time series correlates with the true group-average matrix at 0.50 to 0.60, indicating that the model captures the population-level organization of functional networks.

Load-bearing premise

The claim rests on the assumption that the low prediction error comes from learned brain dynamics rather than from the temporal smoothness of bandpass-filtered fMRI, and the paper reports no comparison against repeating the last observed state or a linear autoregressive model.

Editorial extensions

If this is right

  • A 21.6-second resting-state fMRI segment may suffice to generate roughly five seconds of plausible future brain activity, shortening the scan time needed for applications that depend on a short predicted window.
  • Long autoregressive generation, despite accumulating error, preserves the group-average functional connectome, so the model captures shared functional organization across individuals.
  • The 700-fold jump in error when input frames are shuffled implies temporal ordering is essential to the model's performance, not a side effect of static regional relationships.
  • Error accumulation limits reliable prediction to about seven time points, so further work on reducing cascading errors is required before longer forecasts could be used in practice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not compare against a persistence baseline (x_{t+1} = x_t) or a linear autoregressive model; until such baselines are shown to do worse, the unique contribution of the transformer to short-horizon prediction remains an open question.
  • Because the model operates on z-scored regional signals, its reported MSE values are relative to unit-variance time series; comparing forecast errors across studies would require reporting errors in raw BOLD units or per-region variances.
  • The same encoder-decoder design could be tested on task-evoked fMRI and on clinical cohorts, where the forecasting horizon and functional-connectivity fidelity would need to be weighed against scan time in concrete diagnostic tasks.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript proposes a transformer-based autoregressive model for predicting resting-state fMRI brain states. The model takes 30 consecutive time points (21.6 s at TR=0.72 s) of 379 regional BOLD signals from the HCP dataset and predicts the next time point; iterative application generates a synthetic time series of up to 1150 time points. The authors report a single-step MSE of 0.0013 (versus 0.97 under a shuffled-order control), MSE below 0.15 for the first seven multi-step predictions (5.04 s), and similarity between the functional-connectivity matrices computed from true and predicted data (spatial correlation 0.50–0.60). Ten-fold subject-level cross-validation on 1003 HCP subjects is used, the window size and epoch count were chosen in a preliminary experiment, and the code is publicly available.

Significance. If the claims hold, the paper would demonstrate a generative model that produces plausible future brain states from a short fMRI segment, with potential applications in scan-time reduction and in generating synthetic resting-state data. The evaluation has genuine strengths: held-out subjects at the cross-validation level, a shuffled-order control as a sanity check, and public code that supports reproducibility. The Markov-chain interpretation of error accumulation is clearly stated. However, the central claim that the transformer learns brain dynamics rather than exploiting the strong temporal autocorrelation of the BOLD signal is not yet supported, because trivial baselines (persistence, AR(1)) are absent and the 'accurate up to 5.04 s' criterion is defined post hoc. The functional-connectivity result is also not discriminating between temporal dynamics and mere spatial covariance reproduction. These gaps make the significance conditional on the requested experiments.

major comments (4)
  1. [§3.2, §3.3 (with §2.1)] The central claim that the transformer 'learned and leveraged the temporal dependencies' and can accurately predict brain states up to 5.04 s is not established by the current evaluation, which uses only a shuffled-order control. The bandpass filter (0.01–0.1 Hz) applied at Section 2.1, combined with TR=0.72 s, induces strong temporal autocorrelation in the regional BOLD series, so a persistence forecast (x̂_{t+1}=x_t) or a per-region AR(1) model may plausibly reproduce the reported single-step MSE of 0.0013 and the low multi-step MSE over the first seven time points (Fig. 3b). The shuffled-input MSE of 0.97 is close to 1.0—the MSE of predicting the temporal mean of z-scored data—so the 700-fold difference mainly shows that order carries information, not that the transformer extracts dynamics beyond simple extrapolation. Please add persistence and AR(p) baselines evaluated on the same held-out subjects with the same metrics (MSE, correlation, and FC analysis), and report whether the transformer's errors are significantly lower.
  2. [§3.3] The paper defines the accuracy window as 'the first seven predicted time points (within the time of 5.04s) have relatively low MSE (< 0.15)'. This threshold is chosen after observing the results (Fig. 3b), and no pre-specified criterion, noise ceiling, or baseline-calibrated value is given. The 'accurate up to 5.04 s' conclusion is therefore post hoc. Please justify the threshold a priori, or calibrate it against the trivial baselines above, and report the distribution (mean ± SD across subjects) of the MSE at each of the first seven horizons rather than only the group average.
  3. [§3.3] The comparison with BrainLM ('their 20-time-point MSE is 0.568, larger than ours') is not controlled: the authors' model uses a 30-time-point input, whereas the BrainLM evaluation cited in the Introduction uses a longer input (180 time points), and the preprocessing, parcellation, and data splits differ. Under these conditions the comparison does not support the claim of superiority. Either re-implement BrainLM on the same preprocessing and folds, or present the comparison only as indicative and state explicitly which factors could explain the difference.
  4. [§3.4] The functional-connectivity analysis does not distinguish learned temporal dynamics from reproduced spatial covariance. The group-average FC pattern in Fig. 4b is determined by the cross-regional covariance of the signals; any generator that draws from a multivariate distribution with the empirical covariance (for instance, a null model that temporally shuffles each region's series independently) would produce a similar FC similarity (r ≈ 0.5–0.6 in Fig. 4e) without capturing sequential structure. Please compare the FC of the transformer output against that of a spatial-covariance null model and of the persistence baseline, and report whether the similarity is significantly higher.
minor comments (6)
  1. [Fig. 3 caption] The caption contains grammatical slips (e.g., 'which is the MSE of the 51st time points'); the entire manuscript would benefit from careful copy-editing for typos and phrasing.
  2. [Fig. 2] Model selection in Fig. 2 is based on single runs; no error bars, standard deviations across subjects, or multiple random seeds are reported, so the choice of window size 30 and 20 epochs is not shown to be robust.
  3. [§2.2] The decoder input description is ambiguous: 'the last time point of the encoder input combined with the encoder output' should be specified precisely, including tensor shapes and whether the decoder uses causal masking during the iterative autoregressive generation described in Section 2.4.
  4. [§2.3] It is unclear whether the MSE values in Sections 3.2 and 3.3 are averaged over the ten cross-validation folds or produced by a single final model; the manuscript should state this explicitly.
  5. [§3.4] In the text 'with MSE from 0.35 to 0.45 and correlation from 0.50 to 0.60', the quantity labeled MSE is the mean absolute difference between FC matrices shown in Fig. 4e; using 'MSE' here conflates two different error metrics and should be corrected.
  6. [§1] The claim that short-segment prediction 'can largely shorten the scanning time' is not directly demonstrated; a sentence acknowledging that the clinical benefit is currently speculative would be more accurate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the transformer is trained and evaluated on disjoint subject splits, and the reported brain-state predictions are true out-of-sample forecasts.

full rationale

The paper's claim chain is empirical rather than derivational: a transformer is trained to minimize MSE on rs-fMRI sequences from 90% of HCP subjects and evaluated on held-out subjects (Section 2.3: 'among the 1003 subjects, 90% of subjects ... were used to train the model, while the remaining 10% of subjects ... were used for validation'). The reported single-step MSE of 0.0013 and multi-step errors are computed on unseen-subject data, not on the training set, so the numbers are not fits renamed as predictions. The window size and epoch count are selected on a separate forty-subject preliminary set before final training, and the evaluation protocol is fixed before reading the test MSE; no parameter is fitted to the test quantities. The shuffle control (Section 3.2) is an internal consistency check rather than a load-bearing circular step. The absence of a persistence or linear autoregressive baseline is a genuine risk to the strength of the interpretation, because bandpass-filtered BOLD time series are strongly autocorrelated, but that is a missing-control/interpretation weakness, not a reduction of the prediction to its input by construction or by self-citation. No step in the paper defines a predicted quantity in terms of the same quantity, renames a known result, or imports a uniqueness conclusion from the authors' own prior work; the cited prior work (BrainLM, transformer architectures) is external and not used to justify the authors' own results.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The model relies on several domain assumptions about fMRI preprocessing and the suitability of MSE, but introduces no new theoretical entities. The free parameters are hyperparameters chosen via validation, with window size and epochs directly affecting the reported performance.

free parameters (2)
  • window_size = 30
    Selected from 5,10,15,20,30,40,50 based on lowest single-step and time-series MSE on a 40-subject validation set.
  • num_epochs = 10 (full data)
    Set to 20 in preliminary tests, then changed to 10 for full training because losses stabilized after 6 epochs.
assumptions (4)
  • domain assumption Resting-state fMRI signals after bandpass filtering and z-scoring are sufficiently stationary and autocorrelated to be predictable.
    The entire prediction task assumes that BOLD time series contain a learnable temporal structure; this is a domain assumption about fMRI data.
  • domain assumption Mean squared error on z-scored signals is an appropriate measure of prediction accuracy.
    The paper evaluates performance using MSE; this assumes MSE reflects meaningful prediction quality.
  • domain assumption The 379-region MMP parcellation captures the relevant brain states.
    The model predicts regional means; this assumes regional parcellation preserves the dynamics of interest.
  • ad hoc to paper The transformer architecture can model temporal dependencies in fMRI better than simpler linear models.
    This is the hypothesis under test, but it is used as an assumption when interpreting results without a baseline comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Predicting Human Brain States with Transformer." pith.science (2026). https://pith.science/paper/XWYSRLFS

@misc{pith2026241219814,
  author       = {Pith},
  title        = {Pith review of: Predicting Human Brain States with Transformer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XWYSRLFS}},
  note         = {Machine review of arXiv:2412.19814}
}
read the original abstract

The human brain is a complex and highly dynamic system, and our current knowledge of its functional mechanism is still very limited. Fortunately, with functional magnetic resonance imaging (fMRI), we can observe blood oxygen level-dependent (BOLD) changes, reflecting neural activity, to infer brain states and dynamics. In this paper, we ask the question of whether the brain states rep-resented by the regional brain fMRI can be predicted. Due to the success of self-attention and the transformer architecture in sequential auto-regression problems (e.g., language modelling or music generation), we explore the possi-bility of the use of transformers to predict human brain resting states based on the large-scale high-quality fMRI data from the human connectome project (HCP). Current results have shown that our model can accurately predict the brain states up to 5.04s with the previous 21.6s. Furthermore, even though the prediction error accumulates for the prediction of a longer time period, the gen-erated fMRI brain states reflect the architecture of functional connectome. These promising initial results demonstrate the possibility of developing gen-erative models for fMRI data using self-attention that learns the functional or-ganization of the human brain. Our code is available at: https://github.com/syf0122/brain_state_pred.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 29 canonical work pages

  1. [1]

    Frontiers in human neuroscience, 2009: p

    Herculano-Houzel, S., The human brain in numbers: a linearly scaled -up primate brain. Frontiers in human neuroscience, 2009: p. 31

  2. [2]

    Neuron, 2016

    Shine, J.M., et al., The dynamics of functional brain networks: integrated network states during cognitive task performance. Neuron, 2016. 92(2): p. 544-554

  3. [3]

    Neuroimage, 2013

    Barch, D.M., et al., Function in the human connectome: task -fMRI and individual differences in behavior. Neuroimage, 2013. 80: p. 169-189

  4. [4]

    Dennis, E.L. and P.M. Thompson, Functional brain connectivity using fMRI in aging and Alzheimer’s disease. Neuropsychology review, 2014. 24: p. 49-62

  5. [5]

    NeuroImage: Clinical, 2021

    Lupton, M.K., et al., A prospective cohort study of prodromal Alzheimer’s disease: prospective imaging study of ageing: genes, brain and behaviour (PISA). NeuroImage: Clinical, 2021. 29: p. 102527

  6. [6]

    Neuroimage, 2015

    Yu, Q., et al., Assessing dynamic brain graphs of time-varying connectivity in fMRI data: application to healthy controls and patients with schizophrenia. Neuroimage, 2015. 107: p. 345-355

  7. [7]

    Neuroimage, 2016

    Rashid, B., et al., Classification of schizophrenia and bipolar patients using static and dynamic resting-state fMRI brain connectivity. Neuroimage, 2016. 134: p. 645-657

  8. [8]

    IEEE Transactions on Neural Networks and Learning Systems, 2022

    Hou, Y., et al., GCNs-net: a graph convolutional neural network approach for decoding time-resolved eeg motor imagery signals. IEEE Transactions on Neural Networks and Learning Systems, 2022

Show all 35 references
  1. [9]

    Frontiers in Bioengineering and Biotechnology, 2022

    Hou, Y., et al., Deep feature mining via the attention-based bidirectional long short term memory graph convolutional neural network for human motor imagery recognition. Frontiers in Bioengineering and Biotechnology, 2022. 9: p. 706229

  2. [10]

    IEEE transactions on biomedical engineering, 2012

    Deng, F., et al., FMRI signal analysis using empirical mean curve decomposition. IEEE transactions on biomedical engineering, 2012. 60(1): p. 42-54

  3. [11]

    IEEE Transactions on Biomedical Engineering, 2014

    Lv, J., et al., Holistic atlases of functional networks and interactions reveal reciprocal organizational architecture of cortical function. IEEE Transactions on Biomedical Engineering, 2014. 62(4): p. 1120-1131

  4. [12]

    Medical image analysis, 2015

    Lv, J., et al., Sparse representation of whole -brain fMRI signals for identification of functional networks. Medical image analysis, 2015. 20(1): p. 112-134

  5. [13]

    Latent source mining in FMRI data via deep neural network

    Huang, H., et al. Latent source mining in FMRI data via deep neural network. in 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI). 2016. IEEE

  6. [14]

    IEEE transactions on biomedical engineering, 2019

    Dong, Q., et al., Modeling hierarchical brain networks via volumetric sparse deep belief network. IEEE transactions on biomedical engineering, 2019. 67(6): p. 1739-1748

  7. [15]

    Medical Image Analysis, 2023

    Zhang, S., et al., An explainable deep learning framework for characterizing and interpreting human brain states. Medical Image Analysis, 2023. 83: p. 102665

  8. [16]

    Annals of Neurology: Official Journal of the American Neurological Association and the Child Neurology Society, 2006

    Waites, A.B., et al., Functional connectivity networks are disrupted in left temporal lobe epilepsy. Annals of Neurology: Official Journal of the American Neurological Association and the Child Neurology Society, 2006. 59(2): p. 335-343

  9. [17]

    Von Cramon, and R.I

    Bubic, A., D.Y. Von Cramon, and R.I. Schubotz, Prediction, cognition and the brain. Frontiers in human neuroscience, 2010: p. 25

  10. [18]

    Advances in neural information processing systems, 2017

    Vaswani, A., et al., Attention is all you need. Advances in neural information processing systems, 2017. 30

  11. [19]

    arXiv preprint arXiv:1809.04281, 2018

    Huang, C.-Z.A., et al., Music transformer. arXiv preprint arXiv:1809.04281, 2018. 11

  12. [20]

    Advances in Neural Information Processing Systems, 2020

    Stiennon, N., et al., Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 2020. 33: p. 3008-3021

  13. [21]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Liu, Z., et al. Swin transformer: Hierarchical vision transformer using shifted windows. in Proceedings of the IEEE/CVF international conference on computer vision. 2021

  14. [22]

    arXiv preprint arXiv:2010.11929, 2020

    Dosovitskiy, A., et al., An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

  15. [23]

    ArXiv, 2023

    OpenAI, GPT-4 Technical Report. ArXiv, 2023. abs/2303.08774

  16. [24]

    Advances in Neural Information Processing Systems, 2024

    Kim, P., et al., SwiFT: Swin 4D fMRI Transformer. Advances in Neural Information Processing Systems, 2024. 36

  17. [25]

    Self-Supervised Transformers for fMRI representation

    Malkiel, I., et al. Self-Supervised Transformers for fMRI representation . in International Conference on Medical Imaging with Deep Learning. 2022. PMLR

  18. [26]

    bioRxiv, 2023: p

    Ortega Caro, J., et al., BrainLM: A foundation model for brain activity recordings. bioRxiv, 2023: p. 2023.09. 12.557460

  19. [27]

    arXiv preprint arXiv:2001.08317, 2020

    Wu, N., et al., Deep transformer models for time series forecasting: The influenza prevalence case. arXiv preprint arXiv:2001.08317, 2020

  20. [28]

    Neuroimage, 2013

    Van Essen, D.C., et al., The WU -Minn human connectome project: an overview. Neuroimage, 2013. 80: p. 62-79

  21. [29]

    Neuroimage, 2013

    Glasser, M.F., et al., The minimal preprocessing pipelines for the Human Connectome Project. Neuroimage, 2013. 80: p. 105-124

  22. [30]

    Meyerand, and R.M

    Molloy, E.K., M.E. Meyerand, and R.M. Birn, The influence of spatial resolution and smoothing on the detectability of resting-state and task fMRI. Neuroimage, 2014. 86: p. 221- 230

  23. [31]

    Scientific reports, 2017

    Lin, P., et al., Dynamic default mode network across different brain states. Scientific reports, 2017. 7(1): p. 46088

  24. [32]

    Glasser, M.F., et al., A multi -modal parcellation of human cerebral cortex. Nature,

  25. [33]

    Kingma, D.P. and J. Ba, Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

  26. [34]

    2022, River Publishers

    Agbinya, J.I., Markov chain and its applications, in Applied Data Analytics-Principles and Applications. 2022, River Publishers. p. 1-15

  27. [2016]

    536(7615): p. 171-178

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.