{"id":"9a944727-aab1-49da-b610-e51b0bcb45e8","arxiv_id":"2608.07287","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Using frozen NeuroSTORM embeddings of resting-state fMRI, a logistic regression classifier distinguishes pediatric headache from healthy controls with AUROC 0.82, outperforming functional-connectivity features (AUROC 0.67).","lead":"This paper tests whether a pre-trained AI model for brain scans can tell apart children with headache from healthy controls using resting-state fMRI, and compares it with traditional brain-connectivity features. The AI embedding approach reports higher accuracy (AUROC 0.82 vs 0.67) than connectivity matrices, but the abstract overstates the method and the confidence intervals are misleading.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract's 'fine-tuned' claim is unsubstantiated; reported CIs reflect only solver stochasticity, not subject-level sampling variability, leaving the central AUROC advantage statistically unquantified.","rationale":"The paper's stated contribution is a proof-of-concept that NeuroSTORM embeddings transfer to pediatric headache classification and outperform FC-based models. For that claim to be scientifically grounded, the performance comparison must be valid at the level of independent sampling units. The paper explicitly states the repeated LOO runs differ only by random seed: 'This LOO procedure is repeated 100 times with different random seeds to capture uncertainty in the model's predicted probabilities.' With 110 participants and 189 correlated visits, the effective sample size is 110, not 189. The bootstrap procedure as described does not resample participants, so the extremely narrow CIs (e.g., 0.82–0.82) measure only solver convergence, not statistical confidence in the model's generalization. This is a standard and well-recognized issue in small-sample ML evaluation. The paper even exposes the problem by reporting near-zero CI widths across all models: GB AUROC 0.40 [0.38, 0.43] also has narrow intervals despite high variance in SHAP feature importance, which is inconsistent with genuine sampling uncertainty. Second, the fine-tuning discrepancy is concrete and verifiable: the abstract and introduction claim fine-tuning, but Section 3.4 says 'we focus on logistic regression with L1 regularization for the embedding-based representation', and Appendix A.3 shows the fine-tuning experiment was exploratory and performed worse (0.74 vs 0.82). The central reported result therefore uses frozen embeddings, and the abstract misdescribes the method. I agree with the reader that the fine-tuning wording needs correction; however, that is a reporting error rather than the load-bearing issue. The participant-level validity of the confidence intervals is the deeper concern, because without it the central quantitative claim (0.82 > 0.67) lacks a statistical foundation. The proposed concrete test—participant-level bootstrap or permutation—directly addresses that gap. I also note the paper does include some useful controls: a demographic-only model (AUROC 0.61) and sex-stratified results, which partially mitigate confounding concerns, and the code is released. Those points support keeping the verdict at CONDITIONAL rather than REJECT: the paper is a plausible proof-of-concept, but the abstract must be corrected and the primary comparison needs subject-level uncertainty quantification before the claimed advantage is accepted.","tokens_in":16601,"tokens_out":2079,"duration_ms":15927,"concrete_test":"Re-run the NeuroSTORM LR L1 and FC LR L1 pipelines with subject-level resampling: (1) draw 10,000 bootstrap samples of the 110 participants (with replacement), computing the LOO AUROC within each resample, or (2) perform a permutation test shuffling the headache label at the participant level. Report the resulting 95% CI or permutation p-value for the difference in AUROC. If the 95% CI for the 0.82 vs 0.67 difference includes 0, then the headline comparison is not supported. Separately, correct the abstract to state that the main result used a frozen encoder with linear probing.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Two concerns determine whether the central claim holds. First, the abstract states NeuroSTORM was 'fine-tuned' for headache classification, but the main results use a frozen encoder: Appendix A.3 shows exploratory fine-tuning achieved AUROC 0.74 ± 0.04, underperforming the frozen-embedding LR L1 (AUROC 0.82). The reported 0.82 result therefore does not match the abstract's description. More load-bearing is the inference problem: the 95% CIs in Table 2 (0.82 [0.82, 0.82]) are computed by LOO validation repeated 100 times with different seeds, and therefore capture only stochasticity of the solver and model initialization, not variability due to which participants are sampled. With 110 participants, 79 contributing two correlated visits, subject-level resampling is the relevant variability for generalization. The FC-matrix AUROC gap (0.82 vs 0.67) could be substantially smaller or even non-significant under a proper participant-level bootstrap or permutation test. The reader's verdict correctly identifies both issues; the fine-tuning misstatement is a reporting error, while the CI validity problem directly undermines the statistical support for the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a machine-learning study classifying pediatric headache from resting-state fMRI. The authors compare two representations: functional-connectivity matrices (1,128 features from 48 cortical ROIs) and 288-dimensional embeddings extracted from the pre-trained NeuroSTORM foundation model. Using 189 scans from 110 participants (79 with two visits), they evaluate binary headache-versus-healthy classification with participant-level leave-one-out cross-validation, repeated 100 times with different random seeds. They report AUROC 0.82 (95% CI 0.82–0.82) and AUPRC 0.93 for the NeuroSTORM embedding with L1-logistic regression, versus AUROC 0.67 for FC-based L1-logistic regression. They also report a three-class subtype model (healthy, chronic migraine, non-chronic headache) with macro-AUROC 0.69. The abstract describes the model as fine-tuned, but the main experiments use frozen embeddings; Appendix A.3 describes exploratory fine-tuning that underperformed the frozen-embedding model. The paper concludes that pre-trained fMRI embeddings transfer to pediatric headache classification without explicit FC features.","tokens_in":16847,"tokens_out":6923,"duration_ms":63466,"significance":"If the result holds, the demonstration that a pre-trained fMRI foundation model, applied as a frozen encoder, outperforms conventional FC-based classifiers in a small pediatric sample is a useful proof-of-concept with practical implications for limited-data clinical ML. The study has clear strengths: it uses an external pre-trained model (avoiding circularity), excludes all visits of a held-out participant from training (correct participant-level LOO), benchmarks against a demographic model, and reports stratified performance by sex. The code is released. However, the primary statistical evidence is undermined by the seed-only 'bootstrap' confidence intervals, which do not reflect participant-level sampling variability; the central claim of superiority over FC models is therefore not yet statistically quantified. The abstract's fine-tuning statement is also inaccurate.","major_comments":[{"comment":"The abstract states that NeuroSTORM was 'fine-tuned' for the headache-classification tasks, but the main results in Table 2 and the pipeline in Figure 1 use the frozen pre-trained encoder with L1-logistic regression on 288-dimensional embeddings. Appendix A.3 reports that the exploratory fine-tuning experiment achieved AUROC 0.74 ± 0.04, below the frozen-embedding result of 0.82. The abstract and main text should be corrected so that the primary method is described as frozen-embedding feature extraction with a linear head, and fine-tuning as a separate exploratory result.","section":"Abstract; Appendix A.3"},{"comment":"The 95% confidence intervals in Table 2 are derived by repeating leave-one-out cross-validation 100 times with different random seeds, as described in Section 3. This captures only solver and initialization stochasticity, not participant-level sampling variability, and it ignores that 79 of 110 participants contribute two correlated visits. The intervals (e.g., AUROC 0.82 [0.82, 0.82] and 0.67 [0.67, 0.67]) are therefore not valid confidence intervals for generalization performance. A participant-level bootstrap or permutation test should be provided to support the claim that NeuroSTORM outperforms FC-based models; without it, the headline 0.82 versus 0.67 gap is statistically unquantified.","section":"Section 3 (LOO bootstrap), Table 2"}],"minor_comments":[{"comment":"Replace 'bootstrapped 95% confidence intervals' with 'variation over 100 random seeds' unless a participant-level bootstrap is implemented, because the current procedure does not resample participants.","section":"Table 2, Figure 4 caption"},{"comment":"There are typos: 'compreensive' should be 'comprehensive' and 'an compreensive' should be 'a comprehensive' in Section 6; 'hedaches' should be 'headaches' in Section 5.","section":"Section 6 (Limitations), Section 5 (Discussion)"},{"comment":"The Figure 1 caption contains a stray phrase, 'Xiang Li, Cheng Wang, Yu Jiang et al. structuralfunctional', that appears to be a leftover artifact; remove or clarify it.","section":"Figure 1 caption"},{"comment":"The statement that the IRB 'waived the requirement for informed consent' is unusual for a pediatric study; clarify whether this was a waiver of written documentation, whether parental permission and child assent were obtained, and how these processes were approved.","section":"Section 3.1"},{"comment":"The demographic model bins age as '18 months–3 years' and '3–5 years', but the cohort is aged 8–22; specify that such low-age bins are empty or choose bins that match the actual age range.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable MLHC submission: the clinical question is relevant, the use of an external foundation model is timely, and the code release is a plus. My main concern is that the headline comparison lacks valid uncertainty quantification; this is fixable with participant-level resampling and should be required before publication. The abstract mismatch is also a serious reporting issue that can be resolved by careful rewording."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about arXiv:2608.07287. First, the abstract says they fine-tuned NeuroSTORM, but the main results come from a frozen encoder plus L1 logistic regression. The exploratory fine-tuning in Appendix A.3 actually did worse (AUROC 0.74 vs 0.82). That mismatch needs fixing. Second, the confidence intervals in Table 2 are not participant-level. Repeating leave-one-out 100 times with different seeds only varies the solver; it does not resample the 110 participants or account for the 79 who contribute two correlated visits. So the 0.82 [0.82, 0.82] interval is an artifact of the evaluation procedure, not a measure of generalization uncertainty.\n\nWhat is genuinely useful: it is a clean head-to-head of a pre-trained fMRI foundation model against standard FC features in a pediatric headache cohort, a population where this has not been done before. They report a demographic model and sex-stratified results, which is more than many similar papers do. They also discuss limitations openly, and the code is on GitHub. The point estimate gap (0.82 vs 0.67) is large enough that the approach is worth a closer look.\n\nThe soft spots beyond the CI issue: the subtype results are modest (macro-AUROC 0.69), and the paper is honest about that. The AUPRC numbers are inflated by the 74% prevalence, so AUROC is the more informative metric. Also, they pick the best model from the same LOO procedure, which can optimistically bias the comparison, though they limited tuning. These are minor relative to the CI problem.\n\nBottom line: this is a serious proof-of-concept that deserves referee time, but the authors need to redo the uncertainty quantification with a participant-level cluster bootstrap or permutation test, and correct the abstract. If the gap survives, it is a nice result. For now, treat the 0.82 as a point estimate. I would bring it to a reading group as a case study in how not to compute confidence intervals, and I would not rely on it as evidence until the reanalysis.","headline":"A useful proof-of-concept undermined by a misreported method and invalid confidence intervals; worth reviewing but not yet citable as evidence.","tokens_in":17360,"tokens_out":3110,"would_cite":false,"duration_ms":30155,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pre-trained fMRI encoder classifies pediatric headache with AUROC 0.82 vs 0.67 for connectivity models; fine-tuning only hurt.","keywords":["pediatric headache","resting-state fMRI","foundation model","NeuroSTORM","functional connectivity","machine learning","chronic migraine","transfer learning"],"falsifier":"Run the identical binary pipeline with participant-level resampling, for example leave-one-subject-out (excluding both visits of the held-out participant) or a cluster bootstrap that resamples the 110 participants; if NeuroSTORM's AUROC 95% confidence interval then contains 0.67, the claimed advantage over functional connectivity is not established.","tokens_in":1716,"feed_emoji":"🧠","tokens_out":2493,"duration_ms":77638,"temperature":0.7,"pith_summary":"This paper tries to show that a pre-trained fMRI foundation model can classify pediatric headache from resting-state brain scans without hand-built connectivity features. Using NeuroSTORM, a Mamba-based encoder pre-trained on over 28 million fMRI frames, with the model frozen and an L1-regularized logistic regression on its 288-dimensional embeddings, the authors report an AUROC of 0.82 and an AUPRC of 0.93 for headache versus healthy controls, compared to AUROC 0.67 for functional-connectivity matrix models. Chronic migraine was the separable subtype (AUROC 0.67 in the multiclass setting), while other headache types were not reliably distinguished. Notably, the abstract credits fine-tuning, but the paper's own Appendix A.3 reports that fine-tuning decreased performance to AUROC 0.74, so the headline results come from the frozen encoder. If the results hold, they suggest pre-trained neuroimaging models are a practical starting point for small clinical datasets where site-specific data cannot support training high-capacity models from scratch.","feed_headline":"fMRI foundation model tops connectivity maps for pediatric headache","feed_subtitle":"Frozen NeuroSTORM embeddings hit AUROC 0.82 on 189 pediatric scans; FC-based models reach 0.67.","key_machinery":"The load-bearing object is NeuroSTORM, a four-stage Mamba (state-space) encoder for 4D fMRI pre-trained on more than 28 million fMRI frames and then used as a frozen feature extractor; global pooling of its output gives a 288-dimensional embedding per scan, which feeds an L1-regularized logistic regression. The comparison pipeline uses standard fMRI preprocessing, a 48-region cortical atlas to form pairwise correlation matrices, and the same shallow classifiers on those connectivity features. The mechanism that carries the argument is transfer: a simple linear head performs well on the pre-trained embeddings, which the paper attributes to the foundation model capturing resting-state dynamics that generalize to the headache task.","core_discovery":"The central claim is that the embeddings of a pre-trained fMRI foundation model transfer to a pediatric headache classification task better than the standard functional-connectivity representation. In the authors' leave-one-out evaluation on 189 scans from 110 participants, L1-logistic regression on NeuroSTORM embeddings distinguishes any headache from healthy controls with AUROC 0.82 and AUPRC 0.93; the same classifier family trained on 1,128 functional-connectivity features reaches AUROC 0.67. Multiclass discrimination (healthy, chronic migraine, non-chronic headache) yields macro-AUROC 0.69, driven mostly by chronic migraine versus healthy controls. The paper therefore positions the foundation-model embedding as the effective feature representation under limited-data conditions, and it reports that task-specific fine-tuning of the encoder did not help.","pith_inferences":["Since the reported confidence intervals come from reseeding the solver 100 times, not from resampling participants, a participant-level resampling (cluster bootstrap or leave-one-subject-out) would likely widen the intervals substantially because 79 of 110 participants contribute two correlated visits; the 0.82-versus-0.67 gap may lose statistical significance.","A label-permutation test, or a control analysis that matches motion and acquisition wave, would clarify whether the embeddings encode headache biology or scan-level nuisances that happen to separate the groups.","A testable extension is to train only on first-visit scans and predict second-visit labels for the 79 returning participants; stable within-individual predictions would strengthen the clinical interpretation that the embeddings track an enduring headache-related state.","The discrepancy between the abstract's fine-tuning attribution and the paper's own Appendix A.3 should be corrected in revision; the frozen-embedding pipeline is what the data actually support."],"forward_implications":["If the result is correct, pre-trained fMRI encoders can act as off-the-shelf feature extractors for small clinical cohorts, sidestepping functional-connectivity feature engineering and high-dimensional overfitting.","Chronic migraine appears to carry a separable rs-fMRI signature in this cohort, while non-chronic headache subtypes (post-viral headache, NDPH, post-traumatic headache) are not cleanly separable at this sample size.","The paper's fine-tuning results imply that task-specific adaptation of a foundation model on small pediatric data is not yet beneficial; larger pediatric-specific pretraining or fine-tuning data would be needed to move subtype performance.","The demographic-only model's AUROC of 0.61 bounds what can be attributed to age and sex, so the embedding model's additional margin suggests added predictive value from brain activity, assuming the evaluation is valid."],"supporting_citations":[{"why":"Supplies NeuroSTORM, the pre-trained Mamba-based fMRI foundation model used as the frozen encoder.","marker":"Wang et al., 2026"},{"why":"Supplies the preprocessing pipeline applied to all fMRI data before building both the connectivity and embedding representations.","marker":"Whitfield-Gabrieli and Nieto-Castanon, 2012"},{"why":"Prior migraine classification from functional-connectivity matrices in adults; the benchmark this work contrasts against.","marker":"Chong et al., 2017"},{"why":"Pediatric post-traumatic headache machine learning study whose cohort and feature-reduction approach frame the functional-connectivity baseline.","marker":"Holmes et al., 2022"},{"why":"Introduces Mamba, the selective state-space architecture underlying NeuroSTORM's encoder.","marker":"Gu and Dao, 2024"},{"why":"Documents the sample-size problem in brain-wide association studies, motivating the foundation-model transfer route.","marker":"Marek et al., 2022"},{"why":"Defines the ICHD-3 diagnostic criteria used to assign the headache-subtype labels the models are trained to predict.","marker":"IHS, 2018"}],"fun_headline_variants":["fMRI foundation model outperforms connectivity for child headaches","NeuroSTORM embeddings top FC features in pediatric headache AUROC","Transfer learning from fMRI foundation model aids pediatric headache","Pre-trained fMRI embeddings classify pediatric headache better than FC","fMRI foundation model hits AUROC 0.82 vs 0.67 for pediatric headache"],"cache_read_input_tokens":19584,"weakest_assumption_plain":"The reported margin assumes the 189 scans can be treated as independent draws, but 79 of the 110 participants supply two visits each, so subject-level variability is not captured by the paper's seed-based confidence intervals.","fun_headline_variants_meta":{"raw":{"variants":["fMRI foundation model outperforms connectivity for child headaches","NeuroSTORM embeddings top FC features in pediatric headache AUROC","Transfer learning from fMRI foundation model aids pediatric headache","Pre-trained fMRI embeddings classify pediatric headache better than FC","fMRI foundation model hits AUROC 0.82 vs 0.67 for pediatric headache"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000388,"raw_usage":{"total_tokens":2113,"prompt_tokens":1076,"completion_tokens":1037,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":692,"completion_tokens_details":{"reasoning_tokens":951}},"tokens_in":692,"tokens_out":1037,"duration_ms":7782,"temperature":1.0,"reasoning_tokens":951,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T10:59:44.734224+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical binary pipeline with participant-level resampling, for example leave-one-subject-out (excluding both visits of the held-out participant) or a cluster bootstrap that resamples the 110 participants; if NeuroSTORM's AUROC 95% confidence interval then contains 0.67, the claimed advantage over functional connectivity is not established.","supporting_citations":[{"cited_title":"Conn: A Functional Connectivity Toolbox for Correlated and Anticorrelated Brain Networks , volume =","cited_arxiv_id":null,"evidence_quote":"Supplies the preprocessing pipeline applied to all fMRI data before building both the connectivity and embedding representations."}],"review_version":1}