{"id":"50ab3ed3-c20a-451e-a7ab-a387eb89d50a","arxiv_id":"2501.00378","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"STARFormer uses eigenvector centrality to reorder fMRI brain regions and a variable-window transformer to reach reported state-of-the-art accuracy on ASD and ADHD, but its cross-validation protocol may be contaminated by test data.","lead":"STARFormer, a new transformer architecture, sorts brain regions by a connectivity importance score and learns spatial and temporal patterns from fMRI to classify autism and ADHD. The paper reports top accuracy on two public datasets, but the evaluation may leak test-set information into the model's input ordering.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"EC-based ROI ordering is computed on 10% of the full dataset before the 10-fold split, so test-fold subjects can shape the input token order; as written, the SOTA comparison leaks test information and is unsupported.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the 10% of subjects used to compute the average EC-based ROI ordering is not shown to be independent of the test folds. This is the correct central issue. The paper's protocol, as written, computes a global ROI reordering from a random 10% of the full dataset and then applies that ordering in a 10-fold cross-validation over the same dataset, with no statement that the 10% is restricted to training data or recomputed per fold. Since the ordering is a data-dependent input transformation, using test subjects to determine it is a form of transductive leakage that can inflate the reported state-of-the-art numbers. This concern is load-bearing for the central claim because the claim rests entirely on the validity of the 10-fold cross-validation comparison. The secondary issue of selecting the best window configuration from the full results without nested validation reinforces the need for rejection as written, but the ROI-ordering leakage is sufficient by itself. The architecture may still be worth exploring, and the concern is empirically testable by rerunning the pipeline with per-fold, training-only EC computation. No additional concerns need to be raised; the reader's verdict of REJECT is appropriate.","tokens_in":19211,"tokens_out":6969,"duration_ms":69376,"concrete_test":"Re-run the ABIDE-I Schaefer 10-fold experiment with the EC-based ROI ordering recomputed inside each training fold, using only the training subjects of that fold (drawing the same 10% fraction from the training portion, excluding validation and test) and then applying that fold-specific ordering to the validation and test sets. Compare the resulting mean accuracy and AUC with Table 1's STARFormer row (Acc 77.57±3.70). If the accuracy drops materially or the SOTA claim reverses, the published result is inflated by test-set leakage; if the results match, the concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 states that 'we randomly selected 10% of patient samples from the dataset as input into the ROI spatial structure analysis module,' and then describes a 10-fold cross-validation with a non-overlapping training/validation/test split (80/10/10). The paper never states that the 10% EC subset is drawn from the training portion of each fold, nor that the ROI ordering is recomputed per fold. Equations (7) and (10) show that the average EC vector computed from this 10% subset is used to produce a single global Sort that is then applied to every subject's fMRI time series, including test-fold subjects. This is a classic cross-validation violation: the input representation (the order of ROI tokens) is fit using data that includes test subjects. Even if the 10% subset and the test set are disjoint by chance, the preprocessing is still fit on a mixture of training and test data, which can inflate accuracy by adapting the model's input structure to the test distribution. If 'patient samples' means only the disorder group, the ordering is additionally class-conditional, making the shortcut more severe. This alone invalidates the central claim of state-of-the-art performance in Tables 1-2. The additional selection of the best window configuration from Table 4 on the full results without nested validation is a secondary but related protocol weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes STARFormer, a transformer architecture for classifying brain disorders (ASD and ADHD) from resting-state fMRI time series. The method consists of three modules: an ROI spatial structure analysis module that computes a Granger-causality effective-connectivity matrix, derives eigenvector-centrality (EC) scores, and reorders ROI tokens within seven functional networks; a temporal feature reorganization module that segments the time series into variable-size window tokens with cross-window attention; and a spatio-temporal feature fusion module with parallel temporal and spatial transformer branches. The authors report experiments on ABIDE-I and ADHD-200 with Schaefer and AAL atlases, claiming state-of-the-art accuracy, precision, recall, and AUC relative to seven baselines, alongside ablation studies and ROI interpretability analyses. The manuscript includes a public code link and detailed pseudocode for the proposed pipeline.","tokens_in":19473,"tokens_out":3675,"duration_ms":38185,"significance":"If the reported performance were trustworthy, STARFormer would be a meaningful advance for fMRI-based computer-aided diagnosis: it combines a biologically grounded spatial reordering (effective connectivity plus EC) with a windowed temporal transformer, and the authors provide a public implementation and extensive comparisons on two widely used datasets. The interpretability analysis identifying disorder-relevant ROIs is also a useful addition. However, the paper's central empirical claim is currently unsupported because the experimental protocol leaks information from the test set into the model's input representation. The significance of the work therefore hinges entirely on whether the authors can re-establish the state-of-the-art claim under a leakage-free protocol; as written, the results cannot be taken at face value.","major_comments":[{"comment":"The EC-based ROI ordering is computed on 10% of patient samples randomly selected from the full dataset (\"We randomly selected 10% of patient samples from the dataset as input into the ROI spatial structure analysis module\") and is then applied via Eq. (10) to every subject's fMRI time series before the 10-fold cross-validation split is performed. The paper never states that this 10% subset is drawn from the training portion of each fold or that the ordering is recomputed per fold. Consequently, the ROI token order used for test-fold subjects is derived from a preprocessing step that has access to those subjects' time series (and, since only \"patient samples\" are used, to their labels). This is a classical cross-validation violation: the input representation is fit on a mixture of training and test data, which can inflate all reported metrics. The state-of-the-art claims in Tables 1 and 2, the ablation results in Table 3, and the interpretability analyses in Figs. 12-13 are therefore unsupported. The authors must either (a) compute the ROI ordering using only the training portion of each fold and recompute it per fold, or (b) use a nested cross-validation where the ordering is fit on the inner training data only, and then re-report all results.","section":"Section 5.3, Table 4"},{"comment":"The variable window configuration {16, 8, 4, 4, 8, 16} is selected based on the full test-set results on both datasets, and that same configuration is then used for the final reported model and the comparison tables. No independent validation set or nested cross-validation is used for this hyperparameter choice. This constitutes selection on the test data and introduces an additional optimistic bias into the reported accuracies and confidence intervals. The authors should either fix the window schedule a priori, or select it via an inner validation loop, and then re-evaluate on held-out test folds.","section":"Section 5.3, Table 4"}],"minor_comments":[{"comment":"Equation (7) uses the index i for both the ROI and the patient summation: \\bar{p}_i = (1/N) \\sum_{i=1}^N p_i is formally incorrect. Use a separate patient index, e.g., \\bar{p}_i = (1/N) \\sum_{k=1}^N p_i^{(k)}.","section":"Section 3.1, Eq. (7)"},{"comment":"The text states that STARFormer \"achieves optimal performance in each metric\" and then immediately notes an exception for precision on ADHD-200; this is contradictory. Please rephrase to state that it achieves optimal or near-optimal performance in most metrics, or specify the exact exceptions.","section":"Section 5.1"},{"comment":"The Limitation section discusses graph encoding, phenotypic data, and semi-supervised learning, but it does not mention the most serious methodological risk: the use of a global 10% subset for ROI ordering before cross-validation. This omission should be corrected so that the limitations reflect the actual protocol concerns raised in this report.","section":"Section 6.2"},{"comment":"There are several typographical and formatting issues: \"Temperal\" in Algorithm 1, \"Siminarly\" in Section 6.1, and the title rendering \"STARF ORMER\" in the header. These do not affect the science but should be cleaned up.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript describes a potentially interesting architecture, but the central empirical claim is compromised by a test-set leakage in the ROI-ordering step. I have recommended major revision rather than outright rejection because the flaw is fixable: the authors can recompute the ROI ordering inside each training fold and re-report all tables. However, given that the current protocol also involves selecting the variable-window configuration on the test set, the corrected results could differ substantially. If, after a proper re-analysis, the method no longer achieves state-of-the-art performance, the paper should not be accepted. I would also suggest the editor ask the authors to confirm that the GitHub repository contains the exact code used for the reported experiments, since reproducibility is a key claim of the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read STARFormer. The architecture is a reasonable mash-up of established ideas—EC-based ROI reordering inside Yeo networks, a BolT-like variable-window temporal branch, and a parallel spatial branch—and the paper is clearly written. The ablation is thoughtful, especially the random-permutation control (R vs E) which shows the EC ordering matters. Code and data are public. That is the good news.\n\nThe bad news is that the central performance claim is not supported as written. Section 4.2 says the EC ordering is computed on 10% of the dataset before the 10-fold split. There is no statement that this 10% is excluded from the test folds, and equations (7) and (10) apply one global sort to every subject's time series, including test subjects. That is a textbook leak: the input representation is fit on a mixture of training and test data. If the '10% of patient samples' means only the disorder group, the ordering is class-conditional, which is worse. The ablation's R-vs-E gap (roughly 18 points) shows how much the ordering can matter, which is exactly why this leak can inflate the reported accuracy. Table 4's window schedule is then selected on the full test results without nested validation, a secondary but related problem. The gains over the best baseline are about 2–3 points in accuracy; without the leak, the margin could easily vanish.\n\nI don't think this is a desk reject. The architecture is sensible and the bug is fixable: recompute the EC ordering per fold on the training portion only, and use nested CV for the window schedule. If the results survive that, the paper has a decent chance of being useful. As written, the SOTA claim is unsupported, so I'd treat this as 'major revision required,' not 'reject forever.' A serious referee should see it, precisely because the flaw is subtle and the fix is tractable.","headline":"Sensible architecture, but a cross-validation leak in the EC-ordering step and window selection without nested validation sink the SOTA claim as written.","tokens_in":20007,"tokens_out":3242,"would_cite":false,"duration_ms":30964,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"STARFormer claims that reordering fMRI brain regions by eigenvector centrality and reading them through variable-window attention improves ASD and ADHD classification, reporting 77.57% and 74.12% accuracy on two public datasets.","keywords":["fMRI","brain disorder diagnosis","autism spectrum disorder","ADHD","eigenvector centrality","spatio-temporal transformer","Granger causality","functional connectivity"],"falsifier":"Rerun the 10-fold protocol with the EC ordering recomputed inside each training fold from training subjects only, and compare accuracy to 77.57% (ABIDE-I) and 74.12% (ADHD-200); a meaningful drop would show that test signals leaked into the spatial ordering and inflated the headline numbers.","tokens_in":18986,"feed_emoji":"🧠","tokens_out":7012,"duration_ms":64780,"temperature":0.7,"pith_summary":"This paper claims that a transformer which first reorders brain regions by their importance in the functional network, then reads the fMRI time series through variable-size windows, classifies autism and ADHD more accurately than existing methods. On the ABIDE-I and ADHD-200 datasets it reports accuracies of 77.57% and 74.12% with the Schaefer atlas, above all nine comparison methods. The reason the ordering matters, the authors argue, is that standard transformers lack spatial inductive bias, so ranking ROIs by eigenvector centrality injects spatial structure that the attention mechanism can use. If correct, the architecture provides a more accurate and more interpretable automated screening tool for brain disorders, and its modular design could transfer to other neurological conditions.","feed_headline":"STARFormer beats 9 fMRI baselines by reordering brain regions","feed_subtitle":"Eigenvector-centrality ranking plus variable-window attention lifts accuracy to 77.6% on ABIDE-I and 74.1% on ADHD-200.","key_machinery":"The central object is the EC-ordered ROI sequence produced by the ROI spatial structure analysis module: eigenvector centrality ranks each brain region by the importance of the regions it connects to, and regions are sorted within their seven functional networks to keep related areas together. This ranked sequence $S'$ becomes the input to a dual-branch transformer in which the temporal branch segments the series into 16, then 8, then 4 window tokens, merges back, and applies cross-window attention with a learnable positional bias, while the spatial branch treats each time point as a token and applies self-attention across ROIs. The work of this machinery is to give the transformer an explicit spatial order and a multiscale temporal view without the full $O(m^2n)$ cost of global self-attention.","core_discovery":"STARFormer establishes that jointly modeling spatial and temporal structure of BOLD signals improves fMRI-based diagnosis of ASD and ADHD. Spatial structure is imposed by estimating Granger-causal effective connectivity among ROIs, computing eigenvector centrality, averaging it over a subset of patients, and sorting ROIs within each of seven functional networks by that score. The reordered time series is processed by a temporal branch using variable window tokens with cross-window attention and by a spatial branch using self-attention over time-point tokens; the branches are concatenated and fed to an MLP classifier. The paper reports the highest accuracy among compared methods, 77.57% on ABIDE-I and 74.12% on ADHD-200 with the Schaefer atlas, and its ablation shows that randomly permuting the ROI order drops ABIDE-I accuracy to 59.18%, evidence that the EC-based ordering is the load-bearing component.","pith_inferences":["A test the paper does not report: recompute the EC ordering per training fold from training subjects only, and re-run the 10-fold evaluation; if accuracy drops, the current 10% whole-dataset drawing leaked test information into the ordering.","Because the EC ordering is derived from Granger causality on a fixed 128-sample crop, its stability across scanning sites, TRs, and sequence lengths is untested; a cross-site generalization check would clarify whether the ranking is a disease biomarker or an artifact of preprocessing.","The authors commit to node centrality rather than graph topology; a natural extension is to feed the EC-ordered sequence into a graph encoder so that edge structure, not just node order, contributes to classification.","The reported top-5% ROI lists could be used as prospective hypotheses: an independent-cohort replication would show whether the attention ranking generalizes beyond ABIDE-I and ADHD-200."],"forward_implications":["Reported accuracy reaches 77.57% on ABIDE-I and 74.12% on ADHD-200 with the 400-ROI Schaefer atlas, above all nine comparison methods, with Wilcoxon signed-rank $p \\le 0.05$.","Ablation evidence attributes the gain to the EC-based ROI order: random permutation of the ROI sequence drops ABIDE-I accuracy to 59.18%.","The variable-window schedule $\\{16,8,4,4,8,16\\}$ with extended window size $w/2$ is reported as the best configuration, indicating a trade-off between local and global temporal context.","Cross-window attention reduces the attention cost from $O(m^2 n)$ to $O(4m^2 n/g)$, a factor of $g/4$.","Attention-based ROI importance analysis surfaces top-5% regions consistent with known ASD and ADHD findings, supporting clinical interpretability."],"supporting_citations":[{"why":"Supplies eigenvector centrality as the method for ranking node importance in fMRI networks.","marker":"[17]"},{"why":"Provides the Granger-causality formulation used to build the effective-connectivity matrix.","marker":"[36]"},{"why":"Defines the seven functional networks used to group ROIs before sorting.","marker":"[37]"},{"why":"Contributes the windowed-attention design and positional bias adopted in the temporal branch.","marker":"[38]"},{"why":"Provides the ABIDE-I dataset used for ASD classification experiments.","marker":"[41]"},{"why":"Provides the ADHD-200 dataset used for ADHD classification experiments.","marker":"[42]"},{"why":"Supplies the 400-ROI Schaefer parcellation used in the main experiments.","marker":"[44]"},{"why":"Supplies the 116-ROI AAL parcellation used for robustness comparison.","marker":"[45]"},{"why":"Baseline BolT transformer that STARFormer extends and compares against on fMRI time series.","marker":"[29]"}],"fun_headline_variants":["Reordering brain regions by EC lifts fMRI-based ASD/ADHD classification","STARFormer's EC-based ROI order outperforms 9 baselines in fMRI diagnosis","STARFormer scores 77.6% on ABIDE-I by reorganizing ROI time series","Eigenvector centrality ordering of ROIs drives STARFormer's top fMRI results"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 10% of patients used to compute the average ROI ordering are separate from the test folds; the paper draws that 10% from the whole dataset before the 10-fold split and never says the subset is restricted to training data.","fun_headline_variants_meta":{"raw":{"variants":["Reordering brain regions by EC lifts fMRI-based ASD/ADHD classification","STARFormer's EC-based ROI order outperforms 9 baselines in fMRI diagnosis","STARFormer scores 77.6% on ABIDE-I by reorganizing ROI time series","Eigenvector centrality ordering of ROIs drives STARFormer's top fMRI results"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000952,"raw_usage":{"total_tokens":4086,"prompt_tokens":994,"completion_tokens":3092,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":3004}},"tokens_in":610,"tokens_out":3092,"duration_ms":21754,"temperature":1.0,"reasoning_tokens":3004,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:52:44.326864+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the 10-fold protocol with the EC ordering recomputed inside each training fold from training subjects only, and compare accuracy to 77.57% (ABIDE-I) and 74.12% (ADHD-200); a meaningful drop would show that test signals leaked into the spatial ordering and inflated the headline numbers.","supporting_citations":[{"cited_title":"Eigenvector centrality mapping for analyzing connectivity patterns in fmri data of the human brain","cited_arxiv_id":null,"evidence_quote":"Supplies eigenvector centrality as the method for ranking node importance in fMRI networks."},{"cited_title":"Granger causality: A review and recent advances","cited_arxiv_id":null,"evidence_quote":"Provides the Granger-causality formulation used to build the effective-connectivity matrix."},{"cited_title":"The organization of the human cerebral cortex estimated by intrinsic functional connectivity","cited_arxiv_id":null,"evidence_quote":"Defines the seven functional networks used to group ROIs before sorting."},{"cited_title":"Swin trans- former: Hierarchical vision transformer using shifted windows","cited_arxiv_id":null,"evidence_quote":"Contributes the windowed-attention design and positional bias adopted in the temporal branch."},{"cited_title":"The neuro bureau preprocessing initiative: open sharing of preprocessed neuroimaging data and derivatives.Frontiers in Neuroinformatics, 7(27):5,","cited_arxiv_id":null,"evidence_quote":"Provides the ABIDE-I dataset used for ASD classification experiments."},{"cited_title":"The neuro bureau ADHD-200 preprocessed repository","cited_arxiv_id":null,"evidence_quote":"Provides the ADHD-200 dataset used for ADHD classification experiments."}],"review_version":1}