{"id":"8dbea2f5-ae3c-4baf-9fbf-3db96ded626a","arxiv_id":"1908.04784","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":13,"one_line_summary":"An evolutionary-optimised multilayer perceptron achieves accuracy close to a tuned LSTM on two small EEG classification tasks at lower training cost, and outperforms it on a ten-class digit-imagery task.","lead":"This paper combines evolutionary algorithms for EEG feature selection and neural network design, and compares the resulting classifier against a tuned LSTM on three small brain-signal datasets. The authors report that the evolutionary MLP nearly matches the LSTM on attention and emotion tasks while being much cheaper to train, and exceeds it on a digit-imagery task.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation leaks test information: feature and topology selection occur before 10-fold CV, so the reported accuracies are optimistically biased and may not support the DEvo-vs-LSTM comparison.","rationale":"The reader's weakest assumption identifies the feature-selection-before-CV leak, which is the same underlying selection-bias problem I consider most load-bearing. I broaden it slightly by noting that the evolutionary MLP topology search also appears to use the full dataset and that the final reported accuracies equal the best fitness values from that search, making the published numbers in-sample estimates rather than unbiased generalization estimates. This concern directly threatens the central comparative claim because the bias may affect the MLP and LSTM differently, and the small subject pools and single-subject MBD dataset amplify the risk. The reader's CONDITIONAL verdict already flags this as addressable, and my analysis does not change the verdict; it reinforces the need for a nested evaluation protocol before the accuracy comparison can be trusted. I therefore recommend UNCHANGED.","tokens_in":18609,"tokens_out":4057,"duration_ms":40465,"concrete_test":"Re-run the complete pipeline with nested cross-validation: outer 10-fold CV; inside each training fold, run the evolutionary feature selection and the MLP topology search using only that fold's training portion (with an inner validation split for model selection), then evaluate the selected model on the outer test fold. Report unbiased accuracies for DEvo MLP, LSTM, and both boosted variants, and compare them with Table 8. If the MindBigData DEvo MLP advantage shrinks toward chance or the Mental/Emotional State closeness to LSTM disappears, the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that an evolutionary MLP approaches or beats an LSTM—rests on the accuracies in Table 8. Those numbers appear to be produced by a protocol in which both feature selection and MLP topology optimization use the entire dataset before the reported 10-fold cross-validation is applied. Section 4.1 states that the evolutionary attribute search computes information gain on the full dataset to select features, and Section 4.2 reports that MLP topology is evolved with classification accuracy as the fitness. The 'global best accuracy' values in Tables 4–6 exactly match the final DEvo MLP entries in Table 8 (79.81%, 96.11%, 27.07%). This implies the fitness used during evolution is the same accuracy later reported as the final result, meaning the outer test folds contributed to feature selection and hyperparameter selection. A proper nested cross-validation would keep feature selection and model selection strictly inside each training fold; otherwise the reported numbers are selection-bias-inflated estimates, not generalization accuracies. The comparison between DEvo MLP and LSTM could change under an unbiased protocol, especially because the feature set was chosen by univariate information gain, which may favor the MLP's ability to exploit those particular features, and because the MBD dataset has very few samples (one subject, 30 seconds per class, 40 selected features) where overfitting is a serious risk.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a 'Deep Evolutionary' (DEvo) pipeline for EEG classification: an evolutionary algorithm selects discriminative features from an initial set of 2550 statistical EEG features, and a second evolutionary search optimises the topology of a Multilayer Perceptron. The authors compare the resulting MLP with a manually tuned LSTM and with AdaBoosted versions of both, on three datasets: a mental-state (attention) dataset from four subjects, an emotional-state dataset from two subjects, and the MindBigData digit-guessing dataset from one subject. Reported results are that an AdaBoosted LSTM achieves 84.44%, 97.06%, and 9.94% accuracy on the three problems, while the DEvo-optimised MLP achieves 79.81%, 96.11%, and 27.07%, and an AdaBoosted DEvo MLP achieves 31.35% on the digit task, with substantially lower training times.","tokens_in":18945,"tokens_out":7787,"duration_ms":72699,"significance":"The empirical question addressed—whether evolutionary feature and architecture search can make a lightweight MLP competitive with a deep recurrent model on small EEG datasets—is of genuine interest, and the authors are to be credited for making their datasets publicly available and for repeating each evolutionary run three times. The speciation mechanism to avoid premature convergence is a useful practical detail. However, the current evaluation protocol does not establish the claimed result: feature selection and topology selection are carried out before the reported cross-validation, so the headline accuracies are likely optimistically biased. The significance of the contribution is therefore conditional on a corrected nested evaluation; if the advantage of the DEvo MLP survives that correction, the paper would be a solid empirical contribution.","major_comments":[{"comment":"Feature selection is performed on the full dataset before the reported 10-fold cross-validation. Section 4.1 states that the evolutionary attribute search uses information gain computed on the entire dataset and that the resulting attribute subset is taken forward while the original datasets are discarded; Section 3.3 describes the MLP as evaluated with 10-fold cross-validation, but no step places feature selection inside the cross-validation loop. Consequently, the held-out folds have already influenced which features are used, so the Table 8 accuracies are not unbiased estimates of generalization. The DEvo-versus-LSTM comparison should be re-run with feature selection nested inside each training fold, or with an outer cross-validation loop that performs selection and topology search on the training part only.","section":"3.3, 4.1, Table 8"},{"comment":"The final 'global best accuracy' values in Tables 4–6 are identical to the DEvo MLP entries in Table 8 (79.81%, 96.11%, and 27.07%). This exact correspondence indicates that the number reported as the final result is the same accuracy that was used as the evolutionary fitness for model selection. Even if the fitness accuracy itself came from a 10-fold CV on the selected feature sets, using those same fold accuracies to choose the topology and then reporting them as the final accuracy incurs selection bias. The authors should either report a fresh evaluation on folds untouched by the evolutionary search or implement nested CV for the topology search.","section":"Tables 4–6 vs Table 8"},{"comment":"The evaluation also ignores the temporal structure of the EEG windows. Section 3.2 creates sliding windows with 0.5s overlap, and Section 3.3 describes 10-fold CV without saying that folds are grouped by subject or by non-overlapping segments. A random split of overlapping windows places near-duplicate time segments in both training and test folds, and for the first two datasets the same subjects also appear in both parts; both effects inflate the measured accuracies. The CV protocol should be based on subject-independent, non-overlapping folds.","section":"3.2, 3.3"},{"comment":"With only four, two, and one subjects in the three experiments, the paper reports no standard deviations, confidence intervals, or significance tests for the Table 8 accuracies. The abstract's 'significantly higher' and 'significantly quicker' are therefore unsupported by the evidence as presented. Per-fold results, variance over the three evolutionary runs, and appropriate statistical comparisons are needed to support the central claims.","section":"3.1, 4.4, Abstract"}],"minor_comments":[{"comment":"The arithmetic mean formula is missing the 1/N factor and is typeset incorrectly; it should be μ = (1/N) Σᵢ xᵢ.","section":"3.2, Eq. (10)"},{"comment":"The left-hand side of Eq. (4) should be iₜ, not oₜ, to match the input gate and its use in Eq. (6).","section":"2.4.2, Eq. (4)"},{"comment":"The caption reads 'on the two optimised datasets', but the table contains three datasets; the caption should be corrected.","section":"Table 8 caption"},{"comment":"The paper uses 'Adaptive Boosted', 'Adaptive Boosting', and 'AdaBoosted' interchangeably; a single consistent term should be used.","section":"Throughout"},{"comment":"Section 3.2 reports 2147 features per measure, while the abstract and Section 4.1 say 2550; these counts should be reconciled.","section":"3.2, 4.1"}],"recommendation":"major_revision","confidential_remarks":"The exact matching of fitness values and final reported accuracies (Tables 4–6 versus Table 8) is the most concrete symptom of the leakage problem; the editor should ask the authors to address this explicitly in the revision. The paper is otherwise self-contained and does not raise citation concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this paper combines evolutionary feature selection with evolutionary MLP topology search for three small EEG classification tasks. The most interesting result is on the MindBigData digit task, where an AdaBoosted DEvo MLP reaches 31.35% accuracy versus 9.94% for the boosted LSTM, at a fraction of the training cost. That specific comparison is new and worth knowing about.\n\nWhat the paper does well: the pipeline is clearly described, the datasets are public, training times are reported, and the authors are upfront about the small subject pools and the single-subject MBD data. They even note the possible EMG contamination in the mental-state data. That kind of honesty counts.\n\nThe soft spot is load-bearing. Feature selection is done by computing information gain on the full dataset before any cross-validation, and MLP topology is evolved using accuracy as fitness on what looks like the full dataset too. The global best accuracy from evolution (Tables 4-6) is exactly what is reported as the final model accuracy in Table 8 (79.81, 96.11, 27.07). So the test folds effectively participated in choosing the features and the network architecture. That makes the reported accuracies optimistically biased. A nested cross-validation would keep feature selection and model selection inside each training fold, and the DEvo-vs-LSTM comparison could change under that protocol. The absence of error bars or significance tests matters more than usual given the tiny subject counts: four subjects for mental state, two for emotional state, one for MBD.\n\nMinor comments: 'deep' is a stretch for a network with at most three hidden layers, and the manual LSTM tuning vs evolutionary MLP search is not an apples-to-apples comparison (the authors acknowledge this).\n\nWho it's for: people working on low-cost EEG classification and evolutionary neural architecture search. It's a practical, incremental contribution, not a breakthrough.\n\nIf this lands on my desk, I'd send it to reviewers. The idea is coherent, the method is reproducible from public data, and the potential cost-accuracy trade-off is practically relevant. But I'd expect major revision, specifically a redo of the evaluation with proper nested CV and variance estimates. The central claim may survive, but it isn't supported by the evaluation as it stands.","headline":"Useful integration of evolutionary feature selection and MLP topology search for EEG, but the evaluation leaks test information into model selection, so the headline accuracies are optimistic.","tokens_in":19477,"tokens_out":2461,"would_cite":false,"duration_ms":23735,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An evolutionary EEG pipeline matches a deep LSTM on mental states, beats it on digit guessing, and trains in a tenth of the time.","keywords":["EEG classification","evolutionary algorithm","feature selection","multilayer perceptron","LSTM","AdaBoost","brain-machine interface","MindBigData digits"],"falsifier":"Re-run the three experiments with feature selection nested inside the cross-validation loop—selecting features anew on the training folds only—and compare the final accuracies with Table 8; if the digit-task accuracy drops from 31.35% toward the 10% chance level, the selection step was carrying the result.","tokens_in":18432,"feed_emoji":"🧠","tokens_out":9324,"duration_ms":89496,"temperature":0.7,"pith_summary":"The paper argues that bioinspired optimisation can replace expensive deep learning in EEG-based brain-machine interfaces. It tests a pipeline that evolves both the feature set and the neural-network topology for classifying attention states, emotional states, and imagined digits from four-electrode EEG. The evolved multilayer perceptron comes within a few points of a boosted LSTM on the first two tasks, beats it by a wide margin on the digit task (31.35% versus 9.94%), and trains in roughly a tenth of the time. If the pipeline is sound, cheap consumer EEG hardware plus evolutionary optimisation could provide practical real-time mental-state classification without GPU-scale deep learning.","feed_headline":"Evolved EEG classifier beats deep LSTM on digit guessing","feed_subtitle":"It also matches LSTM accuracy on attention and emotion while training in about a tenth of the time.","key_machinery":"The load-bearing mechanism is the DEvo pipeline: first, an evolutionary algorithm (population 20, 20 generations, tournament breeding) searches subsets of 2550 statistical EEG features, using information gain as fitness; this cuts the datasets to 99 features for attention, 500 for emotion, and 40 for imagined digits. Second, a second evolutionary search optimises the MLP topology, searching up to three hidden layers and up to 100 neurons per layer, with 10-fold cross-validated accuracy as fitness and a speciation scheme—three species, mating restricted within a species, and a 5% chance an offspring changes species—to prevent premature convergence. The LSTM comparator is manually tuned over 25 to 125 units, and AdaBoost is applied to both model types.","core_discovery":"The paper's central claim is that a fully bioinspired pipeline—evolutionary selection of EEG features followed by evolutionary search over multilayer perceptron (MLP) topologies, optionally boosted with AdaBoost—can match or beat a manually tuned LSTM on brain-signal classification while training in roughly a tenth of the time. In the final benchmark, the AdaBoosted LSTM is the most accurate model on the attention and emotion datasets (84.44% and 97.06%), but the plain evolved MLP is close (79.81% and 96.11%), and on the ten-class imagined-digit dataset the AdaBoosted evolved MLP reaches 31.35% while the AdaBoosted LSTM reaches only 9.94%. The authors interpret this as evidence that the expensive deep model is not needed once feature selection and network topology are themselves optimised by evolution; the topology search repeatedly converged on a single hidden layer, mirroring the manual LSTM finding that depth one worked best. The evolved MLP trains in seconds to tens of seconds on the datasets used, which the authors connect to real-time brain-machine interaction.","pith_inferences":["The same two-stage recipe—evolve a feature subset, then evolve a shallow network topology—could be transferred to other high-dimensional biosignals such as EMG or ECG, where windowed statistical features are the standard representation.","For brain-machine interaction, the sub-minute training times suggest adaptive per-user retraining, where the classifier is re-evolved as the user's signal distribution drifts; the paper mentions real-time training but does not develop this adaptive loop.","If the 31.35% digit-guessing accuracy survives replication across many subjects, it would indicate that imagined numerals are partially decodable from four frontal and temporal electrodes; the paper itself cautions that its single-subject digit data makes transfer learning unlikely, so multi-subject replication is the decisive next experiment.","The strong performance of single-hidden-layer networks across both the evolved MLP and the manually tuned LSTM suggests that for windowed statistical features, feature quality matters more than network depth, a hypothesis that could be tested directly by fixing the evolved features and varying depth."],"forward_implications":["The reported accuracies imply that a four-electrode EEG headband can support three-class mental-state and emotion classification with a shallow optimised MLP, since the evolutionary search repeatedly converged on a single hidden layer.","Because the DEvo MLP trains in seconds to tens of seconds while the LSTM takes minutes, per-user retraining in real time becomes practical, which the authors connect to applications such as grading meditation sessions or hands-free interaction.","The results imply that AdaBoost is not uniformly helpful: it lifted the DEvo MLP on the ten-class digit task (from 27.07% to 31.35%) but left the already-strong LSTM accuracy essentially unchanged or slightly lower, so boosting decisions should be made per model and per task.","The evolutionary feature selector's chosen subset sizes (99, 500, and 40 from 2550 features) imply that the amount of useful statistical structure varies sharply across EEG tasks, and that dataset-specific selection matters more than a fixed feature set."],"supporting_citations":[{"why":"Supplies the emotional-state EEG dataset and recording setup used in the second experiment.","marker":"[3]"},{"why":"Supplies the mental-state EEG dataset and the statistical feature extraction approach the paper builds on.","marker":"[17]"},{"why":"The prior evolutionary optimisation of fully connected ANN topology that the MLP search directly extends.","marker":"[39]"},{"why":"Defines the LSTM block structure used as the deep-learning comparator.","marker":"[2]"},{"why":"Introduces AdaBoost, the ensemble method applied to both the MLP and LSTM.","marker":"[49]"},{"why":"Provides the logarithmic covariance model that generates part of the initial 2550-feature set.","marker":"[51]"},{"why":"Supplies tournament breeding selection used in the evolutionary attribute search.","marker":"[53]"}],"fun_headline_variants":["Evolutionary MLP matches LSTM on EEG, wins on digits, 10x faster","Bioinspired MLP tops LSTM on imagined numbers, trains in seconds","EEG digit guessing: evolved MLP beats LSTM by 3x","Deep LSTM not needed: evolved MLP wins on brain digits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results rest on the assumption that the evolutionary feature selection of Section 4.1, which computes information gain on the whole dataset before the 10-fold cross-validation described in Section 3.3, does not leak test-fold information into the selected features; if it does, every reported accuracy is optimistic.","fun_headline_variants_meta":{"raw":{"variants":["Evolutionary MLP matches LSTM on EEG, wins on digits, 10x faster","Bioinspired MLP tops LSTM on imagined numbers, trains in seconds","EEG digit guessing: evolved MLP beats LSTM by 3x","Deep LSTM not needed: evolved MLP wins on brain digits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000673,"raw_usage":{"total_tokens":3145,"prompt_tokens":1108,"completion_tokens":2037,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":724,"completion_tokens_details":{"reasoning_tokens":1953}},"tokens_in":724,"tokens_out":2037,"duration_ms":14381,"temperature":1.0,"reasoning_tokens":1953,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:33:15.812929+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the three experiments with feature selection nested inside the cross-validation loop—selecting features anew on the training folds only—and compare the final accuracies with Table 8; if the digit-task accuracy drops from 31.35% toward the 10% chance level, the selection step was carrying the result.","supporting_citations":[{"cited_title":"Evolutionary optimisation of fully connected artificial neural network topology,","cited_arxiv_id":null,"evidence_quote":"The prior evolutionary optimisation of fully connected ANN topology that the MLP search directly extends."},{"cited_title":"A decision-theoretic generaliza- tionofon-linelearningandanapplicationtoboosting,","cited_arxiv_id":null,"evidence_quote":"Introduces AdaBoost, the ensemble method applied to both the MLP and LSTM."},{"cited_title":"The matrix- logarithmic covariance model,","cited_arxiv_id":null,"evidence_quote":"Provides the logarithmic covariance model that generates part of the initial 2550-feature set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies tournament breeding selection used in the evolutionary attribute search."}],"review_version":1}