{"id":"a5b45488-0f60-466d-a6ac-b576c1902168","arxiv_id":"2411.18392","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Across 4800 EEG deep learning trainings, a minimal filtering pipeline without artifact handling achieved the best average balanced accuracy, while raw data ranked last.","lead":"This study compares four EEG preprocessing pipelines, from raw data to heavy artifact removal, across six classification tasks and four deep learning models. Minimal filtering without artifact removal performed best on average, while raw data consistently ranked last.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Shared learning rate across pipelines is the key confound; Filt's top rank may reflect LR compatibility, not preprocessing superiority, and the paper does not test per-pipeline tuning.","rationale":"The reader's weakest assumption and my load-bearing concern coincide: a single learning rate per (task, model) pair, held constant across preprocessing pipelines, can confound the comparison. This is not an accusation of sloppy work; the paper is transparent about the choice, and the code and data are released. But the central claim is explicitly comparative ('minimal preprocessing without artifact handling is superior'), and the statistical evidence for that claim comes from models trained under a hyperparameter that is assumed, without support, to be equally appropriate for all pipelines. The proposed test is feasible with the existing repository and would directly settle whether the observed Filt-first ranking is an artifact of LR selection or a genuine property of the preprocessing. The manuscript's own Discussion already concedes that the improvement could stem from 'the sensitivity of the architecture to EEG preprocessing, or other reasons', so the concern is consistent with the authors' framing. I would therefore keep the CONDITIONAL verdict; this is UNCHANGED relative to the reader's assessment.","tokens_in":20226,"tokens_out":6697,"duration_ms":67190,"concrete_test":"Using the released eegprepro code, rerun the DeepConvNet experiments for Eye, MMI, and Parkinson with a per-pipeline learning-rate selection on the validation sets, using the same 13-value grid and the same selection rule described in Section II-C3; then recompute the average ranks and Friedman statistic. DeepConvNet is the only architecture with a significant inter-task Friedman test, so it is the decisive case. If Filt remains first in these cells, the shared-LR protocol is not the driver; if Filt's rank drops, the headline comparison is confounded and should be re-reported with per-pipeline tuned learning rates.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section II-C3 fixes one learning rate per (task, model) pair and applies it to all four pipelines (Table III). Preprocessing changes the amplitude distribution, spectral content, and artifact composition of the input, so the optimal Adam step size is not in general invariant across Raw, Filt, ICA, and ICA+ASR. The discrete grid is coarse, and the selection rule ('evaluating the performance of a subset of models on the validation set') is not reported per pipeline. With a single shared LR, the pairwise Wilcoxon tests and the Friedman/Nemenyi ranks supporting the claim that Filt is superior measure the combination of preprocessing and LR compatibility, not preprocessing alone. If, for example, ICA or ICA+ASR require a smaller LR to converge, the observed Filt advantage could shrink or reverse under per-pipeline tuning. The manuscript itself notes that inter-task significance is reached only for DeepConvNet; the LR confound applies to all four architectures and all six tasks, making it the main threat to the central comparative claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical benchmark of four EEG preprocessing pipelines (Raw, Filt, ICA, ICA+ASR) across six classification tasks and four deep learning architectures, totaling 4800 training runs. Using a nested leave-N-subject-out cross-validation scheme, the authors measure balanced accuracy and analyze pipeline differences with pairwise Wilcoxon tests and Friedman/Nemenyi tests. The central claim is that minimal filtering without artifact handling ranks first on average, while raw data ranks last, and that richer artifact-removal pipelines do not consistently help deep learning models.","tokens_in":20370,"tokens_out":4973,"duration_ms":44968,"significance":"If the central claim holds, the paper would provide useful practical guidance for the EEG deep learning community, where preprocessing choices are often made by habit rather than evidence. The study has notable strengths: subject-level splitting, multiple architectures and tasks, balanced accuracy, Holm-corrected pairwise comparisons, a public code repository, and a reproducible seeding procedure. The authors also transparently report that inter-task significance is reached only for DeepConvNet, and they discuss low statistical power. However, the comparison is compromised by a shared learning rate across pipelines and by a repeated-measures Wilcoxon analysis that treats nested CV folds as independent; these issues affect the strength of the central comparative claim.","major_comments":[{"comment":"The learning rate is chosen once per (task, model) pair and kept equal across all four pipelines. Because preprocessing changes the spectral content, artifact composition, and effective input distribution of the data, the optimal Adam step size is not expected to be invariant across Raw, Filt, ICA, and ICA+ASR, even after per-channel z-score normalization. If the shared learning rate happens to align best with Filt, the observed ranking may reflect learning-rate compatibility rather than preprocessing superiority. The paper does not report per-pipeline learning rate tuning or any sensitivity analysis. This is the main threat to the conclusion that minimal preprocessing is intrinsically better, and it should be addressed either by tuning learning rates per pipeline or by demonstrating that the ranking is stable under plausible per-pipeline learning rates.","section":"II-C3, Table III"},{"comment":"The Wilcoxon signed-rank tests compare 50 models per task, but these observations are not independent: for each outer fold, the same test subjects are reused across the five inner folds, creating clustering by outer fold. Treating the 50 models as independent repeats likely inflates significance levels, undermining the intra-task significance claims that appear in the abstract and in Section III. The manuscript cites Dietterich's warning about k-fold t-tests but does not apply the same logic to the Wilcoxon tests. A more appropriate analysis would average the five inner-fold results within each outer fold (giving 10 paired observations per task) or use a cluster-robust / mixed-effects approach that accounts for the nested structure.","section":"II-D, Figure 2 Panels A-D (I)"},{"comment":"The central conclusion that 'a minimal preprocessing without artifact handling is superior' is based on average ranks over only six tasks, and the Friedman/Nemenyi inter-task analysis is significant only for DeepConvNet (chi2_F = 9.0, p = 0.029). The authors acknowledge the low power of the Friedman test, but the general statement in the conclusion goes beyond what the statistical evidence supports. I recommend either softening the conclusion to a descriptive trend, or providing additional evidence such as task-level effect sizes, confidence intervals for average ranks, or a sensitivity analysis that increases the number of task-level replicates (e.g., split tasks into multiple dataset blocks) to make the inter-task claim statistically testable.","section":"IV, V"}],"minor_comments":[{"comment":"The text states that learning rates were searched on a discrete grid of 13 values, but only 9 values are listed, and the listed order is not monotonic (1.0e-5 appears between 2.5e-4 and 7.5e-5). Please correct the grid and the count.","section":"II-C3"},{"comment":"The learning rate selection criterion ('evaluating the performance of a subset of models on the validation set in terms of consistency') is vague. Please specify which subset of models was used and how median and rank consistency were combined into a final choice.","section":"II-C3"},{"comment":"There are several typos in the statistical terminology, including 'Friedmann' for Friedman and 'Mann-Whiteny' for Mann-Whitney. Please correct these for consistency.","section":"II-D and throughout"}],"recommendation":"major_revision","confidential_remarks":"The shared learning rate confound is the key issue. If the authors can add a per-pipeline learning rate sensitivity analysis or reframe the claims as conditional on a fixed hyperparameter choice, the paper could be suitable for publication. The nested-fold dependence in the Wilcoxon tests also needs to be addressed before the intra-task significance statements can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a useful paper, and the empirical core is better than the abstract. The authors ran 4800 trainings across six tasks and four standard architectures, using subject-level nested CV, and they ship code and data. That alone puts it ahead of most EEG-DL benchmarking work. Their supplementary demonstration that sample-level splitting inflates accuracy is a nice, citable warning. The main result—that raw data typically underperforms and that a minimal filter-only pipeline ranks first on average—is coherent with prior work and is presented with more nuance in the discussion than in the abstract.\n\nThe soft spots are real, though. The biggest is the shared learning rate. They select one LR per (task, model) pair and apply it to all four pipelines. Preprocessing changes input scale and spectral content, so the optimal Adam step size can easily differ across Raw/Filt/ICA/ICA+ASR. With a coarse grid and no per-pipeline selection, the ranking may partly reflect which pipeline is most forgiving of that particular LR. The authors don't report whether they checked this. That's a load-bearing confound for the headline claim.\n\nSecond, the repeated-measures Wilcoxon tests treat the 50 N-LNSO models per pipeline as independent. They aren't: the five inner folds within each outer fold all score the same test set. Some correction (e.g., per-outer-fold aggregation or a nested bootstrap) would make the pairwise asterisks trustworthy.\n\nThird, the inter-task Friedman test is significant only for DeepConvNet. For the other three architectures, Filt's top rank is a trend, not a significant effect. The authors admit this, so it's not hidden, but the abstract's phrasing is careful while the conclusion's wording slides toward a general claim.\n\nNone of this sinks the paper. It is a solid empirical contribution for anyone deciding how much to preprocess EEG before training a network. The subject-level splitting, the diversity of tasks/models, and the open code make it worth engaging with seriously. I'd send it to review, but with the LR issue as a required revision: either per-pipeline tuning or at least a sensitivity check showing the ranking is stable across reasonable LR choices. The stats also need tightening. This is a revise-and-resubmit, not a reject.","headline":"Useful EEG-DL preprocessing benchmark with open code and carefully chosen subject-level splits, but the shared learning rate across pipelines and weak inter-task significance make the 'minimal preprocessing wins' claim a trend rather than a proven rule.","tokens_in":20949,"tokens_out":3010,"would_cite":true,"duration_ms":28339,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a minimal, filter-only EEG preprocessing pipeline outperforms both raw data and heavier artifact-removal pipelines on average across six tasks and four deep learning architectures.","keywords":["EEG","deep learning","preprocessing pipelines","artifact removal","balanced accuracy","subject-level cross-validation","raw EEG","minimal filtering"],"falsifier":"Take a single task, re-tune the learning rate separately for each of the four pipelines on the validation set, and re-run the Nested Leave-N-Subject-Out comparison; if the filter-only pipeline no longer ranks first on average, the minimal-preprocessing advantage is an artefact of the shared hyperparameter rather than a property of the preprocessing itself.","tokens_in":19983,"feed_emoji":"🧠","tokens_out":5308,"duration_ms":47867,"temperature":0.7,"pith_summary":"This paper asks whether deep learning on EEG benefits from heavy preprocessing and answers with a qualified no. Across six classification tasks (eye state, motor imagery, Parkinson's, Alzheimer's, sleep deprivation, and first-episode psychosis), four common neural architectures, and 4,800 subject-independent trainings, raw data consistently finished last in average ranking, while a minimal pipeline that only filters and normalises the signal finished first on average for every architecture. Richer pipelines that add automatic artifact removal, such as independent-component rejection with and without artifact subspace reconstruction, did not systematically beat the simple filter. The authors read this as evidence that EEG artifacts are not pure noise for deep networks and may carry task-relevant information, and they propose minimal preprocessing as a default guideline while noting that inter-task significance appeared only for the deepest model tested.","feed_headline":"Minimal EEG filtering beats raw data and heavy artifact removal","feed_subtitle":"Across six tasks and four networks, a light filter-only pipeline ranks first on average in 4,800 trainings.","key_machinery":"The comparison device is a ranked statistical evaluation of four preprocessing pipelines treated as interventions: Raw (no preprocessing beyond channel removal and resampling), Filt (band-pass filtering, re-referencing, and normalisation), ICA (Filt plus independent-component analysis with automatic component rejection), and ICA+ASR (ICA plus artifact subspace reconstruction and bad-channel correction). Performance is measured with a Nested Leave-N-Subject-Out cross-validation, which splits subjects rather than individual windows and yields an unbiased distribution of balanced-accuracy scores for each pipeline on each task. The central identity is the Friedman average rank across the six tasks: the paper's conclusion rests on the consistency with which the minimal pipeline is ranked first and raw data are ranked last, with Nemenyi's critical difference used to visualise which pipelines are statistically separable.","core_discovery":"On the paper's own terms, the central discovery is a consistent ordering of EEG preprocessing strategies for deep learning: raw data underperform, a minimal filter-only pipeline performs best on average, and adding artifact-handling steps does not reliably help. Across all four analysed architectures, the minimal pipeline ranked first in average Friedman rank (average ranks between 1.17 and 2.00), and raw data ranked last in every case (average ranks between 2.83 and 3.33). The inter-task Friedman test reached statistical significance only for the largest architecture, DeepConvNet ($\\chi^2_F = 9.0$, $p = 0.029$); the other architectures showed the same ranking without reaching significance. The authors interpret this trend as evidence that artifacts may carry discriminative information for deep networks, so aggressively removing them can be counterproductive.","pith_inferences":["The learning rate for each task-and-model pair was selected once and shared across all four pipelines; if the optimal learning rate differs by pipeline, the ranking could reflect a hyperparameter mismatch rather than a genuine preprocessing advantage, so a per-pipeline learning-rate search is the most direct test of the paper's conclusion.","If artifacts genuinely help deep networks, the paper's trend implies that artifact-bearing or artifact-injected training data could act as a form of augmentation, a testable extension the authors do not pursue.","For clinical tasks such as Parkinson's and Alzheimer's classification, the results make it plausible that models are using artifact-related differences between patient and control groups rather than purely neural signals, which would warrant saliency or out-of-distribution audits before clinical use.","The paper's lack of inter-task significance for three of four architectures suggests the main barrier is statistical power; a larger multi-task benchmark with per-pipeline hyperparameter tuning would settle whether the ordering is universal."],"forward_implications":["Researchers can expect raw EEG to underperform even modest filtering when training deep networks on new subjects, since the raw pipeline ranked last on average for every architecture in this design.","For the six tasks studied, automatic artifact-handling pipelines do not systematically improve balanced accuracy over simple filtering, so the extra computation, expert supervision, and risk of discarding signal may not be worthwhile unless a specific task shows a gain.","The paper's guidance is that minimal preprocessing, roughly a 1–45 Hz band-pass filter plus channel-wise z-score normalisation, is a reasonable default for EEG deep learning, with artifact removal reserved for cases where task-specific evaluation justifies it.","The authors expect that increasing the number of tasks beyond six would shrink the critical difference and make the 'minimal filter first' ordering statistically significant for the other architectures as well."],"supporting_citations":[{"why":"Documents how EEG deep learning researchers commonly preprocess data and supplies the reporting checklist the study follows.","marker":"[14]"},{"why":"The previous single-task study on EEG preprocessing for deep learning that this work extends to multiple tasks and architectures.","marker":"[19]"},{"why":"Supplies the automatic preprocessing implementation used to construct all four pipelines.","marker":"[12]"},{"why":"Provides the EEGLAB framework on which the preprocessing and artifact-handling steps are built.","marker":"[31]"},{"why":"Supplies the automatic independent-component classifier used in the ICA pipeline's component rejection.","marker":"[33]"},{"why":"Supplies the artifact subspace reconstruction algorithm used in the ICA+ASR pipeline.","marker":"[34]"},{"why":"Defines one of the four neural architectures evaluated, EEGNet.","marker":"[42]"},{"why":"Defines the ShallowConvNet and DeepConvNet architectures evaluated in the study.","marker":"[43]"},{"why":"Defines the FBCNet architecture evaluated in the study.","marker":"[44]"},{"why":"Supplies the Friedman rank test used for inter-task statistical comparison of the pipelines.","marker":"[52]"}],"fun_headline_variants":["For EEG deep learning, minimal filtering outperforms heavy pipelines","Raw EEG data loses to a light filter in deep learning benchmarks","Less preprocessing wins: minimal filtering beats artifact removal in EEG DL","Minimal EEG filter beats raw and artifact removal pipelines in DL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The learning rate for each task-and-model pair is tuned once on the validation set and then used for all four pipelines, so if different preprocessing levels need different learning rates to perform at their best, the pipeline ranking could be an artefact of the shared hyperparameter rather than a true property of the preprocessing.","fun_headline_variants_meta":{"raw":{"variants":["For EEG deep learning, minimal filtering outperforms heavy pipelines","Raw EEG data loses to a light filter in deep learning benchmarks","Less preprocessing wins: minimal filtering beats artifact removal in EEG DL","Minimal EEG filter beats raw and artifact removal pipelines in DL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000774,"raw_usage":{"total_tokens":3434,"prompt_tokens":960,"completion_tokens":2474,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":2404}},"tokens_in":576,"tokens_out":2474,"duration_ms":17232,"temperature":1.0,"reasoning_tokens":2404,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:14:16.592286+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a single task, re-tune the learning rate separately for each of the four pipelines on the validation set, and re-run the Nested Leave-N-Subject-Out comparison; if the filter-only pipeline no longer ranks first on average, the minimal-preprocessing advantage is an artefact of the shared hyperparameter rather than a property of the preprocessing itself.","supporting_citations":[{"cited_title":"Deep learning-based electroencephalography analysis: a systematic review,","cited_arxiv_id":null,"evidence_quote":"Documents how EEG deep learning researchers commonly preprocess data and supplies the reporting checklist the study follows."},{"cited_title":"On EEG preprocessing role in deep learning effectiveness for mental workload classification,","cited_arxiv_id":null,"evidence_quote":"The previous single-task study on EEG preprocessing for deep learning that this work extends to multiple tasks and architectures."},{"cited_title":"BID- SAlign: a library for automatic merging and preprocessing of multiple EEG repositories,","cited_arxiv_id":null,"evidence_quote":"Supplies the automatic preprocessing implementation used to construct all four pipelines."},{"cited_title":"ICLabel: An automated electroencephalographic independent component classifier, dataset, and website,","cited_arxiv_id":null,"evidence_quote":"Supplies the automatic independent-component classifier used in the ICA pipeline's component rejection."},{"cited_title":"Real-time neuroimaging and cognitive monitoring using wearable dry EEG,","cited_arxiv_id":null,"evidence_quote":"Supplies the artifact subspace reconstruction algorithm used in the ICA+ASR pipeline."},{"cited_title":"Deep learning with convolutional neural networks for EEG decoding and visualization,","cited_arxiv_id":null,"evidence_quote":"Defines the ShallowConvNet and DeepConvNet architectures evaluated in the study."},{"cited_title":"A multi- view CNN with novel variance layer for motor imagery brain computer interface,","cited_arxiv_id":null,"evidence_quote":"Defines the FBCNet architecture evaluated in the study."}],"review_version":1}