{"id":"ccb00f61-8e14-4792-9053-b05fe6093b39","arxiv_id":"2607.21654","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A CNN trained on 250×250 EEG topographic images labels ICA components as brain activity or artifact with 89.45% reported accuracy and about 7,200× faster than manual review.","lead":"A team trained a computer-vision model to automatically label brain versus artifact signals in EEG recordings, replacing a manual 30-minute expert task. They report 89.45% accuracy and a 7,200-fold speedup, but the paper lacks code and direct comparison to existing tools.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 89.45% accuracy is unreconciled with Section IV's 87.5% mean and lacks class-wise metrics; without a clear test protocol, the central claim is unverifiable.","rationale":"The reader's weakest assumption focused on the sufficiency of single heatmap images and label reliability. My concern is complementary: regardless of whether the image format is sufficient, the reported performance metrics are internally inconsistent and under-specified, making the central claim unverifiable as written. The abstract's 89.45% does not appear in the results section, and the absence of class-wise metrics leaves open the possibility that accuracy is driven by class imbalance. This is a reporting/verification gap rather than evidence of invalidity, so CONDITIONAL remains appropriate: the authors should provide the missing evaluation details and reconcile the numbers. I partially agree with the reader because the label reliability and image-information issues are real, but I identify the evaluation protocol as the most immediately load-bearing concern for the central claim.","tokens_in":8133,"tokens_out":4015,"duration_ms":36622,"concrete_test":"Obtain the authors' test-set predictions and ground-truth labels (or reproduce with released code/data). Compute the confusion matrix, per-class precision/recall, balanced accuracy, and overall accuracy; verify which number (87.5% or 89.45%) corresponds to held-out test performance. Also require the class distribution and confirm no overlap between training data and expert-validated test images. If balanced accuracy is substantially below 89% or the 89.45% figure cannot be reproduced, the central claim is unsupported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central accuracy claim is not supported by the reported evaluation. The abstract states 89.45% accuracy, but Section IV reports a mean accuracy of 87.5% and a lowest accuracy of 82.36%, with no explanation of the discrepancy. The evaluation also omits class distribution, numerical confusion-matrix values, and per-class precision/recall/F1 (Fig. 10 is image-only). If the 5,000 IC images are class-imbalanced—e.g., a large majority are brain components—a trivial majority-class classifier could score near 89% accuracy without meaningful discrimination. The phrase '50 experiments with 5000 inputs' is ambiguous, and the described feedback path between manual expert validation and the dataset risks optimistic bias if validation images influence training. Without a clearly defined train/test split, reconciliation of the accuracy figures, and class-aware metrics, the headline performance cannot be independently assessed, so the 7200-fold speedup claim also lacks a verifiable accuracy anchor.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents a CNN-based tool that classifies EEG ICA components from 250×250 topographic heatmap images, targeting automated rejection of non-brain activity in clinical EEG workflows. The authors use 5,000 IC images from 128-channel EEG recordings collected at Cincinnati Children's Hospital Medical Center, report a 7200-fold processing-time reduction, and claim a headline accuracy of 89.45% in the abstract, while Section IV reports a mean accuracy of 87.5% and a lowest accuracy of 82.36%. The paper describes the architecture (four convolutional blocks, dense layers, ReLU activation), the data transformation pipeline, and a brief error analysis.","tokens_in":8327,"tokens_out":3233,"duration_ms":29646,"significance":"If the performance claim were supported by a rigorous, well-documented evaluation, this work would make a practical contribution to automated ICA rejection, with clear value for large-scale EEG studies and near-real-time clinical applications. The use of real clinical data, a concrete CNN architecture, and an explicit attempt at error analysis are strengths. However, the central accuracy and speedup claims are not currently verifiable: the reported accuracy figures are inconsistent, the evaluation protocol is under-specified, class-level performance is missing, and the feedback loop in Fig. 8 risks making evaluation labels non-independent. These issues are load-bearing because the entire contribution rests on the claimed accuracy and speed advantage over manual labeling.","major_comments":[{"comment":"The abstract reports an accuracy of 89.45%, while Section IV reports a mean accuracy of 87.5% and a lowest accuracy of 82.36%, with no reconciliation. This inconsistency directly affects the central claim. Please specify the exact test set size, how the 89.45% figure is computed, and provide confidence intervals or per-experiment variability across the 50 experiments.","section":"Abstract and Section IV"},{"comment":"The evaluation protocol is too vague. '50 experiments with 5000 inputs' does not describe how many unique subjects, how train/validation/test splits were formed, or whether any test data were used for model selection. Fig. 8 shows an 'Expert Validation' feedback path into the dataset; if validation images influence labels or training data, test labels may not be independent. Provide an explicit data-flow statement with disjoint train/validation/test partitions and clarify the role of the feedback loop.","section":"Section IV-A and Fig. 8"},{"comment":"No class distribution or class-wise metrics are reported. If the 5,000 IC images are imbalanced, an accuracy near 89% could be achieved by a trivial majority-class classifier. The confusion matrix in Fig. 10 is image-only; provide numerical counts. Report per-class precision, recall, F1, and clarify whether the task is binary (target vs non-target) or multi-class as in ICLabel.","section":"Section IV and Fig. 10"},{"comment":"The 7200-fold speedup claim lacks a defined comparison. What is the baseline manual time, and was it measured in this study or taken from the literature? Is the comparison per IC, per recording, or per pipeline? The speedup is a central practical claim and needs the same evaluation rigor as the accuracy metric.","section":"Section IV"}],"minor_comments":[{"comment":"The paragraph describing the PSD calculation and frequency bands is duplicated verbatim, likely an editorial error.","section":"Section III"},{"comment":"The 'Binary Step' activation is written as a piecewise linear function with parameter α, which does not match the usual binary step definition. Please correct or clarify.","section":"Eq. (3)"},{"comment":"The index terms include 'Brian' instead of 'Brain'.","section":"Index Terms"},{"comment":"The phrase 'satiric linear statistical methods' is unclear; likely 'standard' or 'classic' is intended.","section":"Section IV-B"},{"comment":"'50 experiments with 5000 inputs, addition to the automatic cross-validation' should read 'in addition to.' Also, 'perdition' should be 'prediction.'","section":"Section IV-A"},{"comment":"The statistical test referred to with p<0.05 is not described. Specify the test, sample size, and whether multiple comparisons were controlled.","section":"Section IV-B"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central claims are not yet supported by the reported evaluation. The issues are addressable in a revision if the authors provide a clear train/validation/test protocol, reconcile the accuracy figures, and report class-aware metrics with confidence intervals. The feedback loop in Fig. 8 also needs to be clarified to rule out label leakage. In current form, I would not accept; the novelty is moderate but the evaluation rigor is the main barrier."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Take this as an internal note. The paper is an incremental but plausible extension of the authors' FLAIRS work: they train a CNN to classify ICA component images as brain or artifact, using 5000 expert-labeled topographic heatmaps from a real clinical dataset. That dataset, with age and diagnosis breakdowns, is the most solid part. The idea is not new—ICLabel already does automated IC classification—but the specific image-based approach and the reported speedup are useful if they hold up.\n\nWhat the paper does well: it uses a real clinical sample (59 subjects), describes the architecture clearly enough to reproduce, does an error analysis, and explicitly lists limitations including dataset diversity and hardware bias. The 8% precision gain and 12% false-negative reduction over their own prior model are plausible incremental improvements.\n\nThe soft spots are in the evaluation. The abstract says 89.45% accuracy, but the results section reports a mean accuracy of 87.5% and a lowest accuracy of 82.36%, with no reconciliation. The confusion matrix is only shown as an image; there are no numbers, no per-class precision/recall/F1, and no class distribution. If the dataset is imbalanced (e.g., mostly brain components), a trivial classifier could produce numbers in that range. The phrase '50 experiments with 5000 inputs' is ambiguous, and the feedback path from expert validation back into the dataset in Fig. 8 risks optimistic bias if validation images are reused in training. The 7200x speedup is also ill-defined—what is the baseline? Manual expert time? That number is thrown out without protocol.\n\nThe paper is honest about its limitations, but the central claim is not verifiable as written. The authors also compare only to their own prior work; a direct comparison to ICLabel is essential, since that is the tool clinicians would actually use.\n\nBottom line: this deserves a serious referee, not a desk reject. It addresses a real bottleneck, uses real data, and the method is coherent. But it needs major revision: reconcile the accuracy figures, provide class-wise metrics and a clear train/test split, release code or at least the confusion matrix numbers, and add an ICLabel comparison. I'd send it out with that expectation. I would not cite it in its current form.","headline":"Useful incremental CNN-based ICA labeling on real clinical EEG data, but the reported accuracy is internally inconsistent and the evaluation is under-specified; worth refereeing with a request for major revision.","tokens_in":8862,"tokens_out":3226,"would_cite":false,"duration_ms":24615,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a convolutional neural network reading standard topographic heatmap images of EEG independent components can automate the expert task of rejecting non-brain activity, cutting processing time by a factor of 7,200 while","keywords":["EEG","independent component analysis","ICA rejection","convolutional neural network","topographic heatmap","artifact removal","brain activity labeling","automated classification"],"falsifier":"Run the trained model on new ICA components recorded with a different electrode layout and compare its labels with consensus labels from two independent expert reviewers; if agreement drops toward chance or no longer reaches the reported accuracy, the visual-heatmap premise fails.","tokens_in":8004,"feed_emoji":"🧠","tokens_out":6733,"duration_ms":62295,"temperature":0.7,"pith_summary":"The paper wants to show that the most labor-intensive step in cleaning EEG recordings — deciding which independent source signals (components) are genuine brain activity and which are artifacts — can be performed automatically by a computer-vision model. The model looks at a single 250×250 topographic heatmap of each component and labels it as brain or artifact. Trained on 5,000 clinical images validated by experts, the model reports 89.45% accuracy, with a mean test accuracy of 87.5% and a 7,200-fold reduction in the time a neurologist would spend on the task. If this holds beyond the training data, it makes large-scale EEG studies feasible and brings near-real-time automated artifact rejection within reach. The paper also positions the deliberately simple network as a step toward low-power hardware implementation.","feed_headline":"CNN labels brain signals 7200 times faster than neurologists","feed_subtitle":"A computer-vision tool reads EEG topographic maps and matches expert ICA labels with 89.45% accuracy.","key_machinery":"The mechanism is a convolutional neural network that takes a 250×250 topographic heatmap of an ICA component and outputs a brain/artifact label. Heatmaps are generated from power spectral density computed across six frequency bands, from delta through high gamma, over 128 scalp channels. The network consists of four convolutional blocks with batch normalization and scaling, followed by a flatten layer and two dense layers; ReLU activation was selected after comparing accuracy and power consumption. This architecture is what demonstrates that spatial pattern recognition alone, without handcrafted features, can reproduce expert labeling decisions.","core_discovery":"The discovery is that a purely visual representation of an independent component — the topographic heatmap of its scalp distribution — carries enough information for a convolutional network to separate brain activity from artifacts with roughly 89% accuracy. The authors demonstrate this by training a four-block convolutional network on 5,000 expert-labeled heatmap images and testing it with cross-validation plus expert re-inspection. The result implies that expert ICA labeling, normally a subjective 30-minute-per-recording task, can be reduced to a fast automated classification that slots into existing EEG analysis workflows.","pith_inferences":["Inference: the reported 7,200-fold speedup compares automated classification against human review time only; the full pipeline still requires ICA decomposition and heatmap rendering, so end-to-end savings will be smaller.","Inference: because the network sees only static scalp topographies, artifacts defined mainly by temporal or spectral dynamics may be missed; a head-to-head test against a temporal/spectral classifier would reveal this.","Inference: whether the model generalizes across electrode caps, montages, and clinical populations is untested; a multi-site label-agreement study would settle it.","Inference: a practical product could combine this first-pass visual classifier with a second-pass temporal check, using the CNN where it is strongest and catching its blind spots."],"forward_implications":["Per-recording ICA labeling time can drop from roughly 30 minutes of expert review to near-instant automated classification.","Large-scale EEG studies that would be impractical with manual labeling become feasible.","Automated rejection can operate in near real time, supporting brain-computer interfaces and clinical monitoring.","The model can be inserted into established EEG analysis pipelines without altering data collection or hardware.","The deliberately simple CNN keeps open the path to an embedded, low-power implementation for bedside use."],"fun_headline_variants":["Vision AI labels EEG artifacts 7200x faster than experts","CNN reads EEG heatmaps, cuts labeling time by 7200x","EEG artifact classification by vision AI: 89.45% accuracy, 7200x faster","Automated ICA labeling: CNN matches experts at 89.45% and 7200x speed"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The claim rests on the assumption that a single 250×250 topographic heatmap of one separated EEG source component carries enough distinctive visual information to tell artifacts from brain activity, and that the 5,000 expert labels used for training are accurate and representative.","fun_headline_variants_meta":{"raw":{"variants":["Vision AI labels EEG artifacts 7200x faster than experts","CNN reads EEG heatmaps, cuts labeling time by 7200x","EEG artifact classification by vision AI: 89.45% accuracy, 7200x faster","Automated ICA labeling: CNN matches experts at 89.45% and 7200x speed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000509,"raw_usage":{"total_tokens":2280,"prompt_tokens":671,"completion_tokens":1609,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":415,"completion_tokens_details":{"reasoning_tokens":1520}},"tokens_in":415,"tokens_out":1609,"duration_ms":11243,"temperature":1.0,"reasoning_tokens":1520,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T09:41:43.874371+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained model on new ICA components recorded with a different electrode layout and compare its labels with consensus labels from two independent expert reviewers; if agreement drops toward chance or no longer reaches the reported accuracy, the visual-heatmap premise fails.","supporting_citations":[],"review_version":1}