{"id":"7a8a0195-e733-4b4f-8dc0-d2998ec2caed","arxiv_id":"2507.02510","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Combining direct STFT input, increased STFT overlap, and balanced batching, TFOC-Net reports cross-subject motor imagery accuracies of 67.60%, 65.96%, and 80.22% on BCI Competition IV datasets IV-1, IV-2A, and IV-2B.","lead":"This paper reports a deep learning pipeline, TFOC-Net, that classifies motor imagery EEG in new users by feeding short-time Fourier transform spectrograms directly into a VGG-style convolutional network. The authors report accuracy improvements on three standard BCI datasets and introduce a small newly recorded EEG dataset.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Model choices (STFT overlap, balanced batching, early stopping) appear selected on LOSO test folds, so claimed gains and SOTA accuracies may be optimistically biased.","rationale":"The reader's weakest assumption is exactly the load-bearing concern I identify: hyperparameters and design choices (direct vs image-based STFT, 50% overlap, balanced batching, early stopping) appear to be selected using the LOSO test folds. This is not an ad hominem observation; the paper's own ablation narrative in Sections 3.1 and 3.2 presents the progression of choices with their effect on LOSO accuracy, and no separate validation set is described. Since the final reported numbers are the best among the variants tried on the test subjects, the central claim that these modifications cause the improvement is not fully supported. The concern is addressable: a nested or subject-disjoint validation procedure would settle whether the reported accuracies survive honest model selection. I therefore keep the verdict CONDITIONAL/UNCHANGED rather than escalating to rejection. The other issues noted by the reader (unavailable code/dataset, p=0.062 called significant, weak IV-1 comparison) are secondary; the selection-on-test-folds issue is the one that most directly threatens the headline numbers.","tokens_in":15843,"tokens_out":2946,"duration_ms":36386,"concrete_test":"Implement a nested LOSO evaluation: for each left-out subject, split the remaining subjects into train/validation, choose the STFT overlap (1-sample vs 50%), batching mode (standard vs balanced), and early-stopping epoch using only the validation split, then evaluate once on the left-out subject. If the resulting mean accuracies fall materially below 67.60% on IV-1, 65.96% on IV-2A, and 80.22% on IV-2B, the reported benchmark is partly explained by selection on test folds; if they match, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central causal claim—that direct STFT input, 50% overlap, and balanced batching each improve cross-subject MI accuracy—rests on comparisons made on the same LOSO test folds later used to report the headline numbers. In Sections 3.1 and 3.2 and Figures 3–5, the paper progresses from image-based STFT to direct STFT, then to increased overlap, and finally balanced batching, selecting the variant with the highest mean LOSO accuracy on each of IV-1, IV-2A, and IV-2B. The manuscript does not describe a validation split or nested selection procedure; Section 2.6 only states that test-subject data were excluded from training, which does not prevent test-fold information from leaking into model choice. Section 2.5 also says the best checkpoint is saved by 'validation accuracy,' but no validation set is defined. Under this protocol, the 67.60%, 65.96%, and 80.22% figures are maxima over variants evaluated on the test subjects, so both the per-modification gains and the SOTA comparison can be inflated by selection bias, especially with only 5 or 9 subjects. The absence of released code and data prevents checking whether the choices were fixed a priori.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TFOC-Net, a VGG-based CNN that classifies STFT-transformed EEG directly (rather than converting STFT outputs to images), using increased STFT overlap and a balanced batching strategy for calibration-free cross-subject motor imagery. It reports mean leave-one-subject-out accuracies of 67.60% on BCI Competition IV Dataset 1, 65.96% on Dataset 2A, and 80.22% on Dataset 2B, claims state-of-the-art performance, and additionally contributes a newly recorded EEG dataset and a time-segment analysis of the MI period.","tokens_in":16074,"tokens_out":6798,"duration_ms":76871,"significance":"The paper is a clearly written engineering study with a plausible architecture and an unusually broad evaluation across three public benchmarks plus a new dataset. The explicit calibration-free evaluation protocol and the systematic time-segment analysis are useful contributions. If the reported accuracies are unbiased, TFOC-Net would be a competitive calibration-free baseline. However, the central evaluation protocol currently permits hyperparameter choices to be made on the same test folds used for reporting, and at least one key statistical statement is incorrect. These issues must be resolved before the performance claims can be accepted as state-of-the-art.","major_comments":[{"comment":"The final pipeline—direct STFT classification, optimized (50%) overlap, and balanced batching—is selected by comparing mean LOSO accuracies across variants on the three benchmark datasets (Figures 3–5). No validation split or nested selection procedure is described: Section 2.6 states only that test-subject data were excluded from training, and Section 2.5 says the best checkpoint is saved by 'validation accuracy' without defining a validation set. This means the reported headline numbers (67.60%, 65.96%, and 80.22%) may be maxima over variants evaluated on the test folds, which is an optimistic bias, especially with only five or nine subjects. Please either (a) describe a validation-based selection protocol performed inside the training folds, (b) show that the overlap and batching choices were fixed a priori before any test-fold accuracy was inspected, or (c) report selection-bias-corrected accuracies (e.g., nested LOSO or a held-out test split). Without this, the per-modification gains and the state-of-the-art comparison are not reliable.","section":"§2.5, §2.6, §3.1–3.2, Figures 3–5"},{"comment":"The text describes the IV-1 improvement over WGAN-CNN as 'reaching a significance enhancement (p = 0.062)'. Since p = 0.062 > 0.05, this is not statistically significant at the conventional 5% level, and the wording must be corrected. Moreover, with n = 5 subjects, the Wilcoxon signed-rank test has very limited power; please report the exact test version, effect sizes, and per-subject differences, and temper the conclusion that this improvement 'reinforces the robustness and generalizability' of the method on IV-1.","section":"§3.6"},{"comment":"The blanket claim of outperforming state-of-the-art is not supported for IV-2A: TFOC-Net's mean accuracy (65.96%) is below Liang et al. (67.42%), and the argument that the difference is not significant and that TFOC-Net wins in 5 of 9 subjects does not establish superiority. In addition, the only IV-1 comparator is the authors' own WGAN-CNN, so the 'state-of-the-art' label for IV-1 rests on a single self-comparison. Please qualify the state-of-the-art claims per dataset, include independent baselines for IV-1, and clarify how the p-values in Table 5 were computed when the comparison methods do not report per-subject accuracies.","section":"§3.6, Table 5"}],"minor_comments":[{"comment":"The STFT window size and overlap parameters are not fully specified in the methodology; the overlap is first described as 'minimal (1 sample)' versus 50% only in Section 4. Please provide exact preprocessing parameters (window, hop, FFT points, channel handling) so that the experiments are reproducible.","section":"§2.4, §4"},{"comment":"Section 2.6 says a Wilcoxon rank-sum test will be used, while Section 3.1 reports Wilcoxon signed-rank tests for paired LOSO comparisons; please clarify which test was applied and why.","section":"§2.6, §3.1"},{"comment":"Figures 3–5 are tables rather than figures, the section numbering jumps from 3.2 to 3.4 (Section 3.3 is missing), and the standard-deviation formatting is inconsistent across Tables 2–5; please fix the layout and numbering.","section":"§3.1–3.6"},{"comment":"The abstract and introduction promise public release of code, trained models, and a new dataset, but no repository URL or data availability statement appears in the manuscript; please add these or explicitly state the current availability status.","section":"Abstract, §2.3.4, §5"},{"comment":"The recorded-dataset results in Figure 6 are reported without standard deviations or per-condition significance tests despite large trial-count differences across subjects (40–150 per session); please report these details.","section":"§3.5"},{"comment":"The claim that this is 'the first and only study' validated on four datasets is not substantiated by the cited literature and should be rephrased.","section":"§4"}],"recommendation":"major_revision","confidential_remarks":"The main evaluation issue is central: the headline accuracies appear to be selected on the same test folds used for reporting. If the authors can provide evidence of a fixed a priori protocol or a proper nested validation, the paper could become acceptable after the statistical and state-of-the-art claims are corrected. There is also a notable over-reliance on the authors' own WGAN-CNN as the sole IV-1 baseline, which should be addressed with independent comparators."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The main thing you should know: TFOC-Net is a plausible, clearly described empirical pipeline—direct STFT input, tuned overlap, balanced batching, VGG-style CNN—and the four-dataset evaluation (three public benchmarks plus a new recorded set) is genuinely more thorough than most MI papers. The time-window analysis is a nice extra. If the numbers were trustworthy, 80.22% on IV-2B would be a useful benchmark for calibration-free decoding.\n\nBut the numbers are probably optimistic. The paper selects the STFT overlap and the balanced-batching variant by comparing LOSO test accuracies in Figures 3–5; no separate validation set or nested selection is described. Early stopping is said to use \"validation accuracy,\" yet no validation split is defined. So the reported gains and final accuracies are maxima over variants evaluated on the test subjects, with only 5–9 subjects. That's a real selection-bias problem, not a nitpick. The claim that each modification helps is also weakened because the comparisons are made on the same folds later used for the headline numbers.\n\nOther soft spots: p=0.062 is called a \"significance enhancement\" in Section 3.6, which is wrong. The IV-1 comparison only includes their own WGAN-CNN, so \"state-of-the-art\" there is unsupported. The code and recorded dataset are promised but no link or repository identifier is provided. And \"first and only study\" with four datasets is an overclaim that invites scrutiny.\n\nWhat's good: the architecture and preprocessing are specified in enough detail to reimplement, the use of public benchmarks is standard, and the general idea—feed raw STFT magnitude to a CNN rather than converting to images—is a legitimate, potentially useful tweak that fits the existing literature. The authors also explicitly exclude test-subject data from training, which is correct for calibration-free evaluation.\n\nWho this is for: anyone working on subject-independent MI decoding or BCI calibration-free pipelines. The paper is worth a serious referee, but it needs major revision: add a proper hold-out validation split for hyperparameter choices, report nested selection or fix the choices a priori, release the code and data, and fix the statistical language. I would send it to peer review with a clear request to address the selection-bias issue before it can be accepted.","headline":"A useful but overclaimed empirical study: the method is simple and the four-dataset evaluation is valuable, yet the headline accuracies are likely inflated because key hyperparameters were selected on the LOSO test folds.","tokens_in":16629,"tokens_out":1617,"would_cite":false,"duration_ms":20620,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TFOC-Net claims benchmark-level cross-subject motor-imagery accuracy by classifying raw STFT spectra directly, raising STFT overlap to 50%, and balancing training batches across subjects.","keywords":["EEG","motor imagery","cross-subject classification","short-time Fourier transform","convolutional neural network","calibration-free BCI","balanced batching","VGG"],"falsifier":"Run the identical TFOC-Net pipeline with nested leave-one-out evaluation, selecting the STFT overlap, batching mode, and early-stopping epoch on an inner validation fold exclusive of the test subject; if the mean accuracies fall materially below 80.22% on IV-2B, 67.60% on IV-1, and 65.96% on IV-2A, the reported benchmark depends on test-set information. An independent reproduction that fixes overlap at 50% and balanced batching a priori can check whether per-subject accuracies in Figures 3–5 are replicated.","tokens_in":15617,"feed_emoji":"🧠","tokens_out":8677,"duration_ms":82349,"temperature":0.7,"pith_summary":"This paper claims that a convolutional neural network fed directly with the time-frequency representation of EEG signals can classify motor imagery in people never seen during training, reaching mean accuracies of 67.60% on BCI Competition IV Dataset 1, 65.96% on Dataset 2A, and 80.22% on Dataset 2B. The authors attribute the gains to three pipeline choices: using the raw short-time Fourier transform (STFT) magnitude instead of converting it to a small image, increasing the STFT window overlap to 50%, and forcing every training batch to contain trials from all training subjects. If these results hold, they strengthen the case for calibration-free brain-computer interfaces that work on a new user immediately. The paper also reports that shorter motor-imagery windows, especially the first three seconds or the middle two seconds, often approach the accuracy of the full four-second window, which could make BCIs faster and less tiring to use.","feed_headline":"EEG neural net hits 80% accuracy with no per-user calibration","feed_subtitle":"Raw STFT spectra plus balanced batches beat prior subject-independent motor-imagery decoders.","key_machinery":"The central objects are the STFT magnitude of each EEG trial, treated as a single-channel image whose height is the frequency bins and whose width is the time points concatenated across the three channels, and a VGG-based CNN with stacked 3x3 convolutions, dropout, and MaxNorm weight constraints. Three choices carry the argument: direct classification of the raw STFT rather than a down-sampled 32x32 image; increasing the STFT overlap from minimal to 50%, which densifies the time-frequency grid and acts as a regularizer; and balanced batching, defined as each training batch containing data from all training subjects. The mechanism is that these choices preserve temporal detail, smooth the input representation, and force the network to see inter-subject variation at every optimization step, improving generalization to held-out subjects.","core_discovery":"The central discovery is that a VGG-style CNN trained on the magnitude of the STFT of three EEG channels (C3, Cz, C4) with 50% window overlap outperforms previously published cross-subject motor-imagery classifiers on two of the three benchmark datasets and does not significantly lose on the third. On BCI Competition IV Dataset 2B, mean accuracy rises from 70.55% with the authors' previous WGAN-based pipeline to 80.22%; on IV-1 it rises from 63.7% to 67.60%; on IV-2A it rises modestly from 65.27% to 65.96%, where one published method (67.42%) is not significantly beaten. The approach is validated on a newly recorded five-subject dataset, achieving 63.54% mean accuracy, with each modification—direct STFT classification, increased overlap, and balanced batching—contributing incrementally. Evaluation is leave-one-subject-out, and the authors state that no test-subject data were used during training.","pith_inferences":["If the absence of test-subject information in hyperparameter selection is confirmed through a nested validation design, the results suggest that simple preprocessing and batching choices can close much of the gap to more complex transfer-learning and few-shot methods, so future comparisons should control for this factor.","The segment analysis implies an accuracy-versus-trial-length trade-off that could be used to optimize BCI communication rate; a natural next experiment is to combine the best single segment with the same pipeline and measure throughput.","Because the method uses only three motor-cortex channels, it is a candidate for consumer-grade wearable EEG; a testable extension is to evaluate the same pipeline on dry-electrode recordings to quantify the penalty of noisier signals.","Balanced batching's consistent benefit points to inter-subject variance as a key optimization obstacle, suggesting that other variance-reduction techniques, such as stratified sampling or contrastive objectives, may yield similar or larger gains."],"forward_implications":["A calibration-free motor-imagery BCI becomes practical at usable accuracies: a new user can be classified immediately without a recording session for personal calibration.","Because only three channels (C3, Cz, C4) are used, the approach can run on low-density, low-cost EEG headsets rather than full-cap arrays.","The time-window analysis indicates that trials can be shortened to about three seconds, or to the middle two seconds, with modest accuracy loss, enabling more responsive BCIs.","Increasing STFT overlap and balancing batches across subjects are simple, model-agnostic changes that improve cross-subject generalization and could transfer to other EEG classification pipelines.","The public release of the code and the newly recorded five-subject EEG dataset gives the community a directly comparable resource for future cross-subject work."],"supporting_citations":[{"why":"The authors' previous VGG/WGAN approach, used as the base model and main internal comparison across all datasets.","marker":"[18]"},{"why":"Liang et al.'s EISATC-Fusion, the only listed method with a higher IV-2A mean (67.42%) that TFOC-Net does not significantly beat.","marker":"[38]"},{"why":"Zhu et al.'s training-free separated-channel CNN, defining the calibration-free IV-2B benchmark at 64%.","marker":"[7]"},{"why":"Roy et al.'s continuous inter-subject decoding result on IV-2B (67.78%) that TFOC-Net exceeds.","marker":"[10]"},{"why":"Dolzhikova et al.'s ensemble CNN, providing the IV-2A and IV-2B cross-subject baselines.","marker":"[6]"},{"why":"An et al.'s few-shot dual-attention relation network, a calibration-free IV-2B benchmark at 67.93%.","marker":"[12]"},{"why":"BCI Competition IV Dataset 1, one of the three benchmark evaluation sets.","marker":"[23]"},{"why":"BCI Competition IV Dataset 2A, the second benchmark evaluation set.","marker":"[25]"},{"why":"BCI Competition IV Dataset 2B, the third benchmark evaluation set and the one with the largest reported gain.","marker":"[26]"}],"fun_headline_variants":["STFT-based CNN boosts cross-subject EEG motor imagery accuracy","No calibration needed: STFT CNN hits 80% on motor imagery","Calibration-free motor imagery: STFT-CNN reaches 80% accuracy","Cross-subject EEG: STFT-CNN gains up to 10 points without calibration","STFT transforms EEG: cross-subject MI accuracy jumps to 80%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracies are trustworthy only if the preprocessing and training choices—particularly the 50% STFT overlap, the balanced-batching scheme, and early stopping—were fixed without reference to the labels of the held-out subjects; the paper does not describe a separate validation set used to select them.","fun_headline_variants_meta":{"raw":{"variants":["STFT-based CNN boosts cross-subject EEG motor imagery accuracy","No calibration needed: STFT CNN hits 80% on motor imagery","Calibration-free motor imagery: STFT-CNN reaches 80% accuracy","Cross-subject EEG: STFT-CNN gains up to 10 points without calibration","STFT transforms EEG: cross-subject MI accuracy jumps to 80%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00087,"raw_usage":{"total_tokens":3807,"prompt_tokens":1021,"completion_tokens":2786,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":637,"completion_tokens_details":{"reasoning_tokens":2687}},"tokens_in":637,"tokens_out":2786,"duration_ms":20806,"temperature":1.0,"reasoning_tokens":2687,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:27:36.282989+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical TFOC-Net pipeline with nested leave-one-out evaluation, selecting the STFT overlap, batching mode, and early-stopping epoch on an inner validation fold exclusive of the test subject; if the mean accuracies fall materially below 80.22% on IV-2B, 67.60% on IV-1, and 65.96% on IV-2A, the reported benchmark depends on test-set information. An independent reproduction that fixes overlap at 50% and balanced batching a priori can check whether per-subject accuracies in Figures 3–5 are replicated.","supporting_citations":[{"cited_title":"Separated channel convolutional neural network to realize the training free motor imagery BCI systems,","cited_arxiv_id":null,"evidence_quote":"Zhu et al.'s training-free separated-channel CNN, defining the calibration-free IV-2B benchmark at 64%."},{"cited_title":"Deep Learning Based Inter - subject Continuous Decoding of Motor Imagery for Practical Brain -Computer Interfaces,","cited_arxiv_id":null,"evidence_quote":"Roy et al.'s continuous inter-subject decoding result on IV-2B (67.78%) that TFOC-Net exceeds."},{"cited_title":"BCI Competition IV","cited_arxiv_id":null,"evidence_quote":"BCI Competition IV Dataset 1, one of the three benchmark evaluation sets."},{"cited_title":"BCI Competition 2008-Graz data set A Experimental paradigm,","cited_arxiv_id":null,"evidence_quote":"BCI Competition IV Dataset 2A, the second benchmark evaluation set."},{"cited_title":"‘BCI competition 2008 –Graz data set B’ ,","cited_arxiv_id":null,"evidence_quote":"BCI Competition IV Dataset 2B, the third benchmark evaluation set and the one with the largest reported gain."}],"review_version":1}