{"id":"f4eea686-fa9a-4cff-99d9-5e306cfd2f8d","arxiv_id":"2412.17824","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"An ensemble of five logistic regression models with MRMR feature selection reaches 81.13% accuracy for classifying four inner speech words from EEG, but the evaluation leaks test information into feature selection.","lead":"This paper reports 81% accuracy in classifying four imagined Spanish words from surface EEG, using a subject-specific machine learning pipeline on the public Thinking Out Loud dataset. The result is presented as a strong improvement over prior work, but the evaluation methodology likely overestimates performance.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 81.13% accuracy claim is likely inflated because MRMR feature selection and the K=590 feature count are chosen on the full subject data before 10-fold CV, leaking test information into model selection.","rationale":"The reader's weakest assumption correctly identifies the central flaw: feature selection and hyperparameter selection are performed on the full subject data before cross-validation, so the test folds influence the chosen features, the feature count K, and the model family. This is not a minor methodological preference; it directly determines the magnitude of the reported 81.13% accuracy. The paper provides no nested CV, no separate validation set, and no code to check reproducibility, so the reader's REJECT verdict is appropriate. My independent read of the full text confirms the same leakage path: Section 2.6.1 applies MRMR to each subject's complete data; Section 3.1 selects K=590 using accuracy curves that include test-fold predictions; Section 2.8's 10-fold CV only splits after this selection. The ensemble construction is also underspecified, but the leakage alone is sufficient to invalidate the central claim. The proposed concrete test—a proper nested CV re-run with identical features and splits—settles whether the result survives. If it does survive, the paper would need only a reporting revision; if it does not, the headline claim is unsupported. Given the current evidence, the verdict should remain REJECT.","tokens_in":13375,"tokens_out":3147,"duration_ms":34329,"concrete_test":"Re-run the exact pipeline with all preprocessing and feature extraction unchanged, but move MRMR feature ranking and K selection inside the cross-validation loop. For each of the 10 folds per subject, apply MRMR only to the 9 training folds, select K by inner cross-validation on those training folds (or fix K on a separate held-out split), train the five-LR ensemble on the training folds, and evaluate on the held-out fold. If the resulting average accuracy and F1 fall materially below 81.13% (e.g., below the ~77.9% single-LR result or near chance for weak subjects), the reported headline number is inflated by selection leakage and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 3.2, Table 2) is that the ensemble of five logistic regression models achieves 81.13% accuracy and 81.12% F1 on inner-speech classification. This number is load-bearing: the paper's contribution and its comparison to prior work depend on it being a valid out-of-sample estimate. The evaluation protocol does not support that estimate. Section 2.6.1 states that 'MRMR is applied to the data of each subject separately', and Section 3.1 describes varying K from 20 to 610 and selecting K=590 based on the resulting accuracy/F1 curves (Figure 10). Both steps use all trials of each subject, including the trials that later serve as test folds in the 10-fold CV described in Section 2.8. The features are therefore ranked, and the feature count is tuned, using label information from the held-out folds. In addition, the choice of the final ensemble model ('stacking the five best logistic regression models') is made after comparing Table 2 accuracies produced on the same CV splits, so model selection also uses test-fold information. The reported 81.13% is thus an optimistically biased, in-sample-tuned estimate. The true generalization performance of the pipeline, if re-evaluated with proper nested cross-validation, could be substantially lower; the DL results near or below chance (1D CNN 23.73%, 2D CNN 31.83%) reinforce the concern that the raw data do not trivially support such high accuracy. The central claim as stated is not supported by the presented evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a subject-specific machine learning pipeline for inner speech recognition from surface EEG using the publicly available Thinking Out Loud dataset. The authors extract 191 features per channel, compare eight feature selection techniques, evaluate six classical ML and three deep learning models, and propose an ensemble of five logistic regression models, reporting an overall accuracy of 81.13% and F1 of 81.12%. The central claim is that this subject-specific approach with MRMR feature selection surpasses prior work on this dataset.","tokens_in":13642,"tokens_out":3432,"duration_ms":35578,"significance":"If the reported evaluation were valid, the paper would provide a strong empirical demonstration that classical ML with MRMR feature selection can achieve high accuracy in four-class inner speech decoding from surface EEG, which would be a meaningful advance for BCI applications. The study is also notable for its systematic comparison of eight feature selection methods, six classifiers, and three deep learning architectures on a public dataset. However, the validity of the central accuracy claim is compromised by a leakage in the evaluation protocol, as detailed below.","major_comments":[{"comment":"The MRMR feature selection is applied to the full data of each subject before the 10-fold cross-validation split, and the feature count K is selected by varying it from 20 to 610 and choosing K=590 based on the accuracy/F1 curves obtained on the same data. Because the test folds' label information is used to select the features and to tune K, the reported 81.13% accuracy is an optimistically biased estimate of generalization performance. The authors need to perform feature selection and K selection inside each training fold (nested cross-validation) or use a separate held-out validation set, and then report the resulting performance.","section":"§2.6.1 and §3.1"},{"comment":"The ensemble model selection also leaks test information: the choice to stack five LR models rather than the SVM+ETC+LR ensemble is made by comparing the Table 2 accuracies, which are themselves computed on the same cross-validation folds used for the final evaluation. This additional model-selection step compounds the bias. Furthermore, the manuscript does not specify how the five LR models are combined (e.g., majority voting, averaging, or a meta-classifier), which makes the ensemble result irreproducible.","section":"§3.2"},{"comment":"The deep learning results are at or near chance (1D CNN 23.73%, 2D CNN Resnet18 31.83%) while the classical ML models report 75–81% accuracy for the same four-class problem. The authors attribute this gap to small sample size, but this explanation is not fully convincing given that the same data are used. This large discrepancy reinforces the concern that the ML accuracy is inflated by the feature-selection and model-selection leakage. A properly nested evaluation is needed to determine whether the high accuracy persists when no test-fold information is used.","section":"Table 2 and §3.2"}],"minor_comments":[{"comment":"There is a typo: \"It it to be noted\" should read \"It is to be noted\".","section":"§2.7"},{"comment":"The feature selection method name is inconsistently written as \"ANOV A\" in Table 1; it should be \"ANOVA\".","section":"§3.1"},{"comment":"There are typos: \"consfusion matrix\" should be \"confusion matrix\" and \"chnnels\" should be \"channels\".","section":"§3.2 and Figure 13 caption"},{"comment":"The numeric values for the 1D and 2D CNN results (23.732, 31.826, 28.004) use an inconsistent decimal separator; they should be written as 23.73, 31.83, and 28.00 for consistency with the rest of the table.","section":"Table 2"},{"comment":"The phrase \"the five best logistic regression models\" is ambiguous: it is unclear what distinguishes these five LR models (e.g., different feature subsets, different hyperparameters, or different random seeds). The authors should clarify the construction of the ensemble.","section":"§3.2"},{"comment":"The statement says the processed dataset can be made available upon request, but the original dataset is publicly available; the authors should clarify what exactly would be provided beyond the public data.","section":"Data Availability Statement"}],"recommendation":"reject","confidential_remarks":"The central empirical claim of the paper is invalidated by a clear evaluation leakage: MRMR feature selection and hyperparameter K are selected using the full subject data before cross-validation, and the ensemble model is chosen based on the same cross-validation accuracies. This is not a minor methodological quibble but a direct threat to the reported 81.13% accuracy. While it is in principle possible to re-run the experiments with a proper nested evaluation, the current manuscript's conclusions and its comparison to prior work are unsupported as written. I therefore recommend rejection, though the authors could potentially resubmit a corrected version if the unbiased results still show competitive performance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the headline 81% accuracy is not credible as an out-of-sample number. The evaluation leaks test information into feature selection and hyperparameter choice: MRMR is run on each subject's full data (Section 2.6.1), then K is varied from 20 to 610 on the same data and K=590 picked from the resulting curves (Section 3.1). The 10-fold CV happens after that, so the folds have already seen the feature ranking and the feature count. The 81.13% is a fitted maximum, not an independent estimate. The reader's critique is correct, and the stress-test confirms it.\n\nWhat the paper does well: it is a systematic benchmark on a public dataset, with eight feature selection methods and six ML models, subject-specific topographical analysis, and honest reporting of the DL failure near chance. The VMD artifact removal is a reasonable preprocessing choice, and the observation that topographies are subject-specific motivates the subject-wise design. That part is useful for anyone working on the Thinking Out Loud dataset.\n\nSoft spots beyond the leakage: the ensemble of five logistic regression models is underspecified—how the five models differ and how stacking is implemented is not described. No error bars or per-fold variance are reported. The comparison to prior work (EEGNet) is not apples-to-apples because the pipelines differ; calling the margin \"astonishing\" is editorializing. The conclusion already admits limited generalizability, which is honest.\n\nThe central argument—that this pipeline gives a strong benchmark—does not hold up with the current evaluation, but the flaw is fixable. A nested CV or a fully within-fold MRMR and K selection would give a trustworthy number. If the redo gives, say, 70% instead of 81%, the contribution shrinks to a competent but incremental benchmark, which would still be worth a short report for the dataset.\n\nWho is this for: researchers building inner-speech BCIs and looking for a feature-selection/classifier comparison on this dataset. It deserves a serious referee, but the referee should require a redo of the evaluation and a clearer description of the ensemble. I'd lean toward \"major revision\" rather than outright reject, because the dataset work and the systematic comparison have value once the leak is fixed.","headline":"The 81% accuracy claim is likely inflated by test-set leakage in feature selection and K tuning; the paper is a competent but incremental benchmark that needs a redo before its main number can be trusted.","tokens_in":14236,"tokens_out":2519,"would_cite":false,"duration_ms":26949,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Stacking five logistic regression models after per-subject MRMR feature selection classifies four inner-speech words from 128-channel surface EEG with 81.13% accuracy and 81.12% F1, the paper reports.","keywords":["Inner Speech","Brain-Computer Interface","EEG","Machine Learning","Classification","Ensemble Model","MRMR","Logistic Regression"],"falsifier":"Re-run the subject-specific pipeline with feature selection and the choice of K=590 performed entirely inside each training fold, keeping the same five-logistic-regression ensemble; if the resulting held-out accuracy falls substantially below 81.13% or no longer beats the single logistic regression baseline, the paper's headline claim is not supported.","tokens_in":13145,"feed_emoji":"🧠","tokens_out":7117,"duration_ms":68665,"temperature":0.7,"pith_summary":"This paper aims to establish that inner speech can be decoded from non-invasive scalp EEG when each subject is modelled separately. Using the public Thinking Out Loud dataset, the authors extract 191 time-, frequency-, and time-frequency-domain features from each of 128 channels, reduce the resulting 24,448 features per subject with Minimum Redundancy Maximum Relevance (MRMR), and classify four Spanish inner-speech words with six machine learning classifiers. Their central result is that an ensemble formed by stacking five logistic regression models reaches an overall accuracy of 81.13% and an F1 score of 81.12%, the best result they report for this dataset. If correct, this would show that a computationally light, subject-specific classical machine learning pipeline is a viable route for inner speech brain-computer interfaces.","feed_headline":"Ensemble model classifies four inner-speech words from EEG at 81%","feed_subtitle":"Subject-specific feature selection plus five stacked logistic regression models beats deep learning on a public EEG benchmark.","key_machinery":"The central mechanism is the pairing of subject-specific MRMR feature selection with a stacked ensemble of five logistic regression classifiers. MRMR (Minimum Redundancy Maximum Relevance) scores each feature by the ratio of its relevance to the target word to its redundancy with previously selected features, allowing the authors to cut a 24,448-dimensional feature space down to K=590 features per subject. The paper reports that this reduction, together with the averaging effect of five logistic regression models, raises overall accuracy from 77.89% for a single logistic regression model to 81.13% for the ensemble.","core_discovery":"On the paper's own terms, the discovery is that per-subject feature selection plus a stacked ensemble of simple linear classifiers solves the four-word inner speech decoding task on this dataset. The authors report that the ensemble of five logistic regression models outperforms every individual classifier they tested, including support vector machines and extra trees, and outperforms the previous deep learning work on the Thinking Out Loud dataset by a large margin. They also report that seven of the ten subjects reach above 80% accuracy, with subject 08 reaching 93.00%, while three subjects (03, 04, and 05) remain at or below about 72%, which they explain by overlapping topographical brain activations for the four words in those subjects. The authors interpret the subject-dependent results as evidence that inner speech production is neurally idiosyncratic and therefore needs subject-specific modelling.","pith_inferences":["A leak-free replication that selects features and the K=590 count inside each training fold is the natural next experiment; the meaningful question is whether the stacked ensemble still beats the single logistic regression and the EEGNet baseline under that protocol.","The large subject-to-subject spread (93.00% for subject 08 versus 55.00% for subject 04) suggests that individual neural discriminability, not just the algorithm, determines performance, so future work could try to predict which subjects will benefit most from this pipeline.","A cross-lingual or cross-session test with different words would show whether the feature set captures general inner-speech correlates or just word-specific templates in this Spanish cohort."],"forward_implications":["Subject-specific modelling is a more reliable starting point than a one-size-fits-all model when decoding inner speech from EEG, because the same word activates different scalp regions in different subjects.","MRMR-style feature selection can compress tens of thousands of EEG features to a few hundred without degrading, and in this paper improving, classification accuracy.","On small single-subject EEG datasets, a pipeline of engineered features plus classical classifiers can outperform deep neural networks that need more training data.","Combining several logistic regression models into an ensemble is a cheap way to gain around three percentage points of accuracy over the single-model baseline."],"supporting_citations":[{"why":"supplies the Thinking Out Loud dataset, including its 128-channel EEG recordings and the four Spanish inner-speech words used in all experiments.","marker":"[23]"},{"why":"provides the previous deep learning result on this dataset that the proposed ensemble is compared against and claims to surpass.","marker":"[57]"},{"why":"is the source of the MRMR feature selection algorithm used to reduce the 24,448 features to 590 per subject.","marker":"[49]"},{"why":"demonstrates stacked ensemble models on EEG classification, the design pattern behind the proposed ensemble of five logistic regression models.","marker":"[54]"},{"why":"supplies the variational mode decomposition method used to remove low-frequency motion artifacts before feature extraction.","marker":"[30]"},{"why":"provides the overfitting rationale for why the large 24,448-feature representation requires feature selection.","marker":"[41]"}],"fun_headline_variants":["Stacked logistic models beat deep learning on EEG inner speech","EEG inner speech: five stacked logistic models reach 81%","Per-subject feature selection boosts inner speech EEG decoding","Ensemble of simple models tops deep learning for inner speech","Logistic regression stack decodes four inner speech words at 81%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported result assumes that choosing which and how many features to keep using the same subject data that later measures accuracy does not leak information from the test trials into the model.","fun_headline_variants_meta":{"raw":{"variants":["Stacked logistic models beat deep learning on EEG inner speech","EEG inner speech: five stacked logistic models reach 81%","Per-subject feature selection boosts inner speech EEG decoding","Ensemble of simple models tops deep learning for inner speech","Logistic regression stack decodes four inner speech words at 81%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000187,"raw_usage":{"total_tokens":1365,"prompt_tokens":1021,"completion_tokens":344,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":637,"completion_tokens_details":{"reasoning_tokens":259}},"tokens_in":637,"tokens_out":344,"duration_ms":3531,"temperature":1.0,"reasoning_tokens":259,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:25:53.543771+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the subject-specific pipeline with feature selection and the choice of K=590 performed entirely inside each training fold, keeping the same five-logistic-regression ensemble; if the resulting held-out accuracy falls substantially below 81.13% or no longer beats the single logistic regression baseline, the paper's headline claim is not supported.","supporting_citations":[{"cited_title":"Emotion recognition from EEG using higher order crossings,","cited_arxiv_id":null,"evidence_quote":"supplies the Thinking Out Loud dataset, including its 128-channel EEG recordings and the four Spanish inner-speech words used in all experiments."},{"cited_title":"Predicting breast cancer from risk factors using SVM and extra -trees-based feature selection method,","cited_arxiv_id":null,"evidence_quote":"provides the previous deep learning result on this dataset that the proposed ensemble is compared against and claims to surpass."},{"cited_title":"Machine learning -based classification of healthy and impaired gaits using 3D -GRF signals,","cited_arxiv_id":null,"evidence_quote":"is the source of the MRMR feature selection algorithm used to reduce the 24,448 features to 590 per subject."},{"cited_title":"Performance assessment of artificial neural network using chi -square and backward elimination feature selection methods for landslide susceptibility analysis,","cited_arxiv_id":null,"evidence_quote":"demonstrates stacked ensemble models on EEG classification, the design pattern behind the proposed ensemble of five logistic regression models."},{"cited_title":"Classification of V owels from Imagined Speech with Convolutional Neural Networks,","cited_arxiv_id":null,"evidence_quote":"supplies the variational mode decomposition method used to remove low-frequency motion artifacts before feature extraction."},{"cited_title":"Variational mode decomposition,","cited_arxiv_id":null,"evidence_quote":"provides the overfitting rationale for why the large 24,448-feature representation requires feature selection."}],"review_version":1}