{"id":"bf82551f-30fd-4476-af57-5f33086eb019","arxiv_id":"2506.04745","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Avalanche length and activation differences between rest and motor imagery predicted one-session-ahead BCI success in 20 healthy users, with best-case reported accuracy of 91 percent after ROI selection.","lead":"Researchers tested whether brain activity cascades called neuronal avalanches can predict who will learn to control a brain-computer interface. In 20 people across four training sessions, changes in avalanche features between rest and motor imagery correlated with performance and predicted next-session success with reported accuracies up to 91 percent.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed 88–91% one-session-ahead accuracy is not supported because ROI selection (§2.4) and parameter selection (§3.2) are performed on the full dataset, including held-out sessions and the outcome to be predicted, before leave-one-out cross-validation; a nested evaluation is required.","rationale":"The paper's stated contribution is a predictive tool; the only quantitative evidence for it is the LSVC/LSVR accuracy. The strongest claim therefore stands or falls on whether the reported accuracy reflects genuine out-of-sample generalization. The described pipeline makes this insecure: every outcome-relevant choice (which of the 10 avalanche parameter pairs to keep, which 68-ROI mask to use) is made before LOO and uses the very sessions and scores that are later used as test labels. The LOO loop only assesses the final trained SVM, so it cannot detect this optimism. The session-shuffle control is also uninformative about this particular leak because it keeps the selected parameters and ROIs fixed. I found no independent support that would offset this: no code release, no external validation, and the permutation ANOVA before ROI selection showed no global task or session effects (Section 3.1), making it more likely that the later significant effects and high accuracy arise from the multi-step selection. This is not an accusation of intent; it is a description of the evaluation protocol as written. The concrete nested test above would settle the matter. If the nested accuracy remains high, the paper should be accepted with the claim; if it drops, the appropriate outcome is rejection or major revision. Given the reader already identified this issue and recommended REJECT, I do not adjust the verdict.","tokens_in":16748,"tokens_out":3988,"duration_ms":49735,"concrete_test":"Run a nested LOO evaluation for the S4 prediction: for each held-out subject j, (1) select avalanche parameters among Table 1 candidates using repeated-measures correlations computed only on the remaining 19 subjects and only sessions S1–S3; (2) recompute the §2.4 ROI t-tests and two-way ANOVA using only those training subjects and S1–S3; (3) recompute Δavalanche_length and Δactivations for subject j's S1–S3 with the chosen mask; (4) train LSVR/LSVC on the 19 training subjects and predict subject j's S4 success. Report accuracy and compare with 91%. If accuracy falls to near chance (or to the session-shuffle level), the headline accuracy is an artifact of outcome-dependent selection; if it remains at ~90%, the central claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Central claim: Δavalanche features from sessions S1–S3 predict S4 BCI success at 88–91%. For this claim to hold, the predictive pipeline must not use information about the held-out target session or target BCI scores when choosing features or spatial masks. The paper violates this in two places. §2.4 selects ROIs by per-subject paired t-tests (MI vs. Rest) computed independently for each session and a two-way ANOVA across subjects; this uses all four sessions and all 20 subjects, including the test subject, before the LOO split. The same ROI mask is then used to recompute features for prediction, so the held-out subject's own condition contrast and session-specific variability have influenced the feature space. §2.5 / §3.2 restrict the predictive models to parameter pairs whose repeated-measures correlation with BCI score is significant; these correlations are computed over all subjects and sessions, including the target score being predicted, and no correction is made for the 10 candidate pairs. LOO-CV then evaluates only the final model, not the selection process; this is a classic selection-on-the-outcome leak. The random-session-shuffle control does not remove this leak because the same outcome-selected parameters and ROI mask are reused. If the test subject's target-session data are removed from both selection steps, the reported 91% may drop to chance; the paper as written does not demonstrate generalization.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using neuronal avalanche features (avalanche length and activation count) extracted from EEG during motor-imagery BCI training as biomarkers of learning and as predictors of one-session-ahead BCI performance. Using data from 20 healthy participants across four sessions, the authors report significant task-condition and learning effects, repeated-measures correlations between session-to-session changes in these features and BCI scores, and longitudinal SVR/SVC models that predict future BCI success with up to 91% accuracy after region-of-interest (ROI) selection.","tokens_in":17038,"tokens_out":3749,"duration_ms":43975,"significance":"If the predictive results were valid, this would be a practically useful contribution to personalized BCI training, addressing the 'BCI inefficiency' problem with a novel dynamical feature set. The study has several strengths: a longitudinal design, the use of repeated-measures correlation, explicit comparison against non-longitudinal baselines, a session-shuffling control, and a detailed description of the avalanche-detection pipeline. The core limitation is methodological: the feature, parameter, and ROI selection steps are performed on the full dataset—including the target sessions and target BCI scores—before the leave-one-out evaluation, so the reported accuracy figures do not demonstrate out-of-sample generalization. The central claim is therefore not supported as written, though it could in principle be tested with a properly nested cross-validation scheme.","major_comments":[{"comment":"The ROI selection procedure uses within-subject paired t-tests computed independently for each session and a two-way ANOVA across all subjects, sessions, and conditions; this includes the target session and the held-out subject before the LOO split. Because the features are then recomputed on the selected ROIs, the held-out subject's own condition contrast and session-specific variability influence the feature space. The 91% LSVC accuracy reported in §3.4 therefore does not reflect generalization to new subjects; the ROI mask must be selected inside each training fold.","section":"§2.4, §3.4"},{"comment":"The predictive models are restricted to parameter combinations showing significant repeated-measures correlations with BCI scores computed over all subjects and sessions (e.g., z-threshold μ+3σ, minimum duration 12). This is a selection-on-the-outcome leak: the target scores are used to choose the features, and no adjustment is made for the 10 candidate parameter pairs. The LOO-CV in §3.3 and §3.4 evaluates only the final model, not the selection process, so the reported accuracy is optimistically biased.","section":"§2.5, §3.2"},{"comment":"The session-shuffling control does not remove the selection leak because it reuses the outcome-selected parameters and the outcome-selected ROI mask. Moreover, after ROI selection the shuffled accuracy is 84%, only 7 points below the reported 91%, which suggests that the longitudinal contribution to classification accuracy is modest and that the control cannot support the claim that the prediction generalizes to unseen subjects.","section":"§3.3, §3.4"},{"comment":"The permutation-based global ANOVA found no significant effects for any parameter combination before ROI selection, yet the paper emphasizes significant local effects (session-4 Wilcoxon tests, MI Friedman tests) without correcting for the multiple comparisons across the 10 parameter pairs and the multiple tests performed. The evidence for the claimed 'significant training and task-condition effects' is therefore weaker than the current presentation suggests.","section":"§3.1, §2.3.1"}],"minor_comments":[{"comment":"The formula for weighted mean activations contains a typo ('Weigthed') and the fraction format is garbled; please rewrite Equation (1) in standard notation.","section":"§2.2.2"},{"comment":"There are two figures numbered 'Figure 5' (one in §3.3 and one in §3.4); renumber them sequentially and ensure all in-text references match.","section":"Figures"},{"comment":"The normalization baseline 'maximum activation observed during the first Rest session' is a reasonable choice, but the sentence claiming that mean/median/minimum would give equivalent results should be supported with evidence or removed, as it is not self-evident.","section":"§2.4"},{"comment":"Reference [8] cites 'Hardman et al. (1997)' but the text attributes it to 'Nan et al., 2012'; please correct the citation-text mismatch.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses an interesting question and the avalanche-based feature framework is worth pursuing, but the central predictive claim is compromised by selection leakage outside the cross-validation loop. The paper would need a re-analysis with nested leave-one-subject-out cross-validation, where parameter selection and ROI selection are performed only on training folds. Given that this is methodologically fixable within the scope of a revision, I recommend major revision rather than outright rejection, but the current version should not be accepted without the corrected analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe new thing here is using the longitudinal change in two neuronal-avalanche features (mean avalanche length and weighted activation count, computed as Rest minus MI) to forecast one session ahead whether a BCI user will control the device. That's a reasonable and practically motivated idea, and the study has real infrastructure: 20 naive subjects, four EEG sessions, source reconstruction, and longitudinal SVR/SVC models compared against non-longitudinal baselines and a session-shuffle control.\n\nThe problem is that the headline accuracy numbers (88-91%) are not supported as estimates of generalization, because the feature-selection pipeline sees the outcome before the cross-validation split. The ROI mask is selected via per-subject t-tests and a two-way ANOVA on the full dataset, including the held-out subject and the target session; and the parameter combinations are selected on the basis of repeated-measures correlations with BCI scores computed across all subjects and sessions. Leave-one-out CV then evaluates only the final model, not the selection process. That's outcome-dependent selection, and it inflates apparent accuracy. The session-shuffle control does not remove the leak, because the same outcome-selected parameters and ROI mask are reused.\n\nThere are also smaller issues that add up. The text reports 88% accuracy first, then 91% after ROI selection, and 3 false negatives out of 20 subjects cannot give 91% accuracy. The global permutation ANOVA found no significant effects before ROI selection, which makes the post hoc nature of the results more salient. Multiple comparisons over ten parameter combinations are not corrected. And the paper has editing problems: duplicated figure numbers, typos, and inconsistent 'best' parameter settings across sections.\n\nThat said, the idea deserves a serious referee. The core flaw is fixable in principle: a nested cross-validation that selects ROIs and parameters inside each training fold, corrected multiple testing, and full reporting of all tested combinations. If the predictive effect survives that, it would be a useful result. As written, the evidence doesn't support the claim.\n\nI'd send it to review, but with the expectation of major revision and a re-analysis. It's also a useful reading-group example of selection leakage in longitudinal prediction.\n\nBest, [Your name]","headline":"Worth serious attention, but the 88–91% one-session-ahead accuracy is not backed by the analysis as reported because feature selection leaks outcome information into the cross-validation folds.","tokens_in":17619,"tokens_out":3565,"would_cite":false,"duration_ms":40322,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that avalanche-like EEG cascades can predict who will master a motor-imagery BCI in the next session, with accuracy up to 91%.","keywords":["neuronal avalanches","motor imagery BCI","EEG","BCI inefficiency","longitudinal prediction","support vector regression","support vector classification","brain-computer interface"],"falsifier":"Re-run the full pipeline with strictly nested cross-validation: inside each training fold, choose the z-threshold, minimum duration, and ROI set using only that fold's data or only the first three sessions, then predict the held-out session; if accuracy falls toward the shuffled-session control (84%) or the standard SVC baseline (59%), the selection step is inflating the 91% figure.","tokens_in":16533,"feed_emoji":"🧠","tokens_out":7974,"duration_ms":84686,"temperature":0.7,"pith_summary":"This paper argues that the brain's avalanche-like cascades, brief bursts of neural activity that spread across cortical regions, can reveal session by session whether someone is learning to control a motor-imagery brain-computer interface. Analysing EEG from 20 healthy participants across four BCI training sessions, the authors extract two avalanche features, duration and activation count, and use the difference between rest and motor imagery in these features as learning markers. These differences grow with training, track BCI performance, and feed longitudinal support-vector models that predict next-session success, reaching 88% classification accuracy on the full cortex and 91% after restricting the data to selected regions of interest. If the result holds, fixed-length BCI training schedules could be replaced by protocols that adapt to each user's measured learning trajectory.","feed_headline":"Brain avalanches predict BCI success, 91% accuracy","feed_subtitle":"EEG avalanche features reveal who will control the interface next session, enabling personalized training.","key_machinery":"The load-bearing objects are neuronal avalanches: sequences in which source-reconstructed EEG activity crosses a z-threshold in at least one of 68 cortical regions and continues until all regions fall silent. Detection depends on two parameters, the z-threshold and a minimum avalanche duration, and the paper restricts analysis to the ten parameter pairs that yield avalanches with physiologically plausible durations. From detected avalanches the authors compute mean avalanche length and a length-weighted activation count per trial, average these per session and condition, and form the rest-minus-imagery differences Δavalanche length and Δactivations. Repeated-measures correlation selects which parameter pairs and features track BCI scores, and the predictive stage relies on Longitudinal Support Vector Regression and Classification, which model each subject's sessions as a temporally weighted combination and are validated with leave-one-subject-out cross-validation.","core_discovery":"The central claim is that neuronal avalanche dynamics in source-reconstructed EEG form a longitudinal biomarker of BCI learning. Over four sessions, motor-imagery avalanches grow longer and more numerous relative to rest, and the per-session difference between rest and motor imagery in both features correlates positively with BCI scores. Those same difference features, used as inputs to Longitudinal Support Vector Regression and Classification models that encode session order through a learned temporal trend, predict one-session-ahead performance: classification accuracy reaches 88% without ROI selection and 91% with a selected set of cortical regions, against 41% and 59% for standard non-longitudinal baselines. The authors interpret this as evidence that avalanche-based features capture both task-state separation and the progression of individual learning.","pith_inferences":["If replicated, the approach suggests a practical stopping rule: after each session, refit the longitudinal model from accumulated Δ-features and continue training only until it predicts above-chance control, which could cut both frustration and cost for fast learners.","The bimodal distribution of motor-imagery avalanche lengths in the final session hints that responders and non-responders form distinct subgroups; clustering on Δ-feature trajectories could identify users who need a different task or feedback modality before they fail.","A natural extension the paper does not test is using avalanche transition matrices, the propagation probabilities between regions, as additional inputs, since those may predict not only success but also which brain networks a given user is recruiting.","The central numbers are likely sensitive to the choices of z-threshold, minimum duration, and ROI set; an external multi-session dataset with pre-registered parameters would clarify whether 91% reflects true generalization or selection within this sample."],"forward_implications":["BCI success one session ahead can be forecast from the previous sessions' rest-versus-imagery avalanche differences, so training does not need to be fixed in advance.","Because task-condition and learning effects emerge mainly in the fourth session, at least three completed sessions are needed before the model can give a reliable prediction.","Restricting features to a small set of task-relevant regions of interest improves accuracy (from 88% to 91%) and reduces computing cost, which matters for real-time BCI use.","The ordering of sessions carries information: shuffling session order lowers accuracy, so the model is reading genuine change over time rather than stable individual traits alone.","Using the 57% chance-level threshold, the classifier can flag users who are unlikely to gain control next session, allowing trainers to intervene or extend practice."],"supporting_citations":[{"why":"Defines neuronal avalanches in cortical circuits, supplying the phenomenon the features are built from.","marker":"[23]"},{"why":"Provides the avalanche detection and whole-brain EEG analysis approach the feature extraction follows.","marker":"[25]"},{"why":"Earlier demonstration that avalanche-derived measures distinguish motor imagery from rest and support BCI analysis, the direct antecedent.","marker":"[27]"},{"why":"Source of the experimental protocol and EEG dataset analysed here.","marker":"[29]"},{"why":"Repeated-measures correlation method used to link Δfeatures to BCI scores.","marker":"[33]"},{"why":"Defines the 57% chance-level threshold used to label successful BCI control.","marker":"[34]"},{"why":"Longitudinal SVR framework adapted for the regression model.","marker":"[37]"},{"why":"Longitudinal SVC framework adapted for the classifier.","marker":"[38]"}],"fun_headline_variants":["Brain avalanches forecast BCI success at 91%","Avalanche dynamics predict next-session BCI control","EEG avalanches reveal who will master BCIs","Neuronal avalanches guide personalized BCI training","Avalanche features hit 91% accuracy for BCI prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported 88–91% accuracy assumes that selecting the avalanche-detection parameters and the regions of interest after seeing the target session's BCI scores does not leak information into the prediction; if it does, the model's performance on a genuinely new user would be lower.","fun_headline_variants_meta":{"raw":{"variants":["Brain avalanches forecast BCI success at 91%","Avalanche dynamics predict next-session BCI control","EEG avalanches reveal who will master BCIs","Neuronal avalanches guide personalized BCI training","Avalanche features hit 91% accuracy for BCI prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000453,"raw_usage":{"total_tokens":2269,"prompt_tokens":924,"completion_tokens":1345,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":1264}},"tokens_in":540,"tokens_out":1345,"duration_ms":11532,"temperature":1.0,"reasoning_tokens":1264,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:34:49.460566+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the full pipeline with strictly nested cross-validation: inside each training fold, choose the z-threshold, minimum duration, and ROI set using only that fold's data or only the first three sessions, then predict the held-out session; if accuracy falls toward the shuffled-session control (84%) or the standard SVC baseline (59%), the selection step is inflating the 91% figure.","supporting_citations":[],"review_version":1}