{"id":"58d580d1-6475-40e3-a9a4-a3954eacef51","arxiv_id":"2411.13217","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"An EEG classifier using channel-pair log-energy features and a bi-LSTM reports 98.66% accuracy for speech versus music, 61.59% for four genres, and 96.96% for musical taste, but the random split of overlapping trials inflates the results.","lead":"This paper uses brain-wave (EEG) recordings from six listeners to classify whether they are hearing speech or music, which of four music genres they hear, and whether they like the song, using energy differences between EEG channels and a bidirectional LSTM network.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Trial-level random split with 50%-overlapping windows lets the bi-LSTM memorize stimulus and subject identity; the reported accuracies are not evidence of generalization.","rationale":"The reader's weakest_assumption is exactly the load-bearing point: a trial-level random split of overlapping windows from a small stimulus set cannot support generalization claims. I checked the manuscript's protocol statement in Section 4 and the data description in Section 2.3: 6 subjects, 20 songs, 150 sentences, 400-ms/1-s trials with 50% overlap, and a random 50/50 split. The feature construction in Section 3.1 computes energy over the whole trial window, so overlapping train/test trials share raw samples; even non-overlapping trials from the same song or sentence are assigned randomly to both partitions, allowing stimulus and subject memorization. This is not a disagreement with consensus; it is an internal validity failure of the evaluation. The paper provides no code, data, or baselines, but the decisive issue is independent of those: averaging over 10 random splits does not remove the leakage. The concrete check I propose is a disjoint-stimulus and leave-one-subject-out reanalysis. If accuracies collapse, the abstract's 'satisfactory performance' is unsupported and the manuscript should be rejected; if they persist, the concern is refuted and the paper would need a revised evaluation description. Given the current evidence, I agree with the reader's REJECT and do not change the verdict.","tokens_in":11622,"tokens_out":6574,"duration_ms":74612,"concrete_test":"Re-run the three tasks with a disjoint-stimulus protocol: for each fold, hold out all trials from one entire song (music) or one entire sentence (voice) from all subjects, train the same E(n)/E'(n) bi-LSTM on the remaining stimuli, and test on the held-out stimulus. As a second disjointness check, run leave-one-subject-out (train on other subjects' trials, test on the held-out subject's trials). If music/voice accuracy falls from 98-99% toward 50%, genre accuracy from 61% toward 25%, or taste accuracy from 96% toward chance, the random trial-level split is the source of the reported performance. Report per-fold and mean accuracy for both variants.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the random 50/50 split of trials into training and test measures generalization. Section 4 states exactly this protocol, and every headline number (98.66% music/voice, 61.59% genre, 96.96% taste) rests on it. The data come from only 20 songs and 150 sentences recorded from 6 subjects, cut into 400-ms or 1-s windows with 50% overlap. Because the energy features in Section 3.1 are computed over the whole trial window, overlapping training and test trials share half their raw samples and are nearly identical inputs. Even non-overlapping trials from the same song or sentence are randomly assigned to both partitions, so the network can memorize stimulus identity or subject identity instead of learning the intended auditory category. This also contradicts the Section 4 claim of an 'inter-subject scenario,' since every subject contributes to both training and test. With only 20 songs and 6 subjects, near-perfect binary accuracies and >90% taste accuracies are exactly what stimulus or subject memorization would produce. If this protocol is invalid, the abstract's central claim of satisfactory performance is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes an EEG-based classification scheme that characterizes each trial by a 61x61 matrix of pairwise log-energy differences between EEG channels (E(n)) and a temporal-difference variant (E'(n)), and feeds these matrices into a bidirectional LSTM for three tasks: binary music/voice classification, 4-class musical genre classification, and binary/multi-class classification of musical taste. The data come from two experiments with 6 subjects, using 20 music fragments and 150 spoken sentences, segmented into 400-ms or 1-s trials with 50% overlap. Reported accuracies reach 98.66% for music/voice (and up to 99.97% for the E'(n) variant), 61.59% for 4-class genre, and 96.96% for binary musical taste. The central claim is that these results demonstrate satisfactory performance of the proposed scheme. The paper also includes a discussion of energy topoplots and comparisons across trial durations and feature variants.","tokens_in":11846,"tokens_out":2872,"duration_ms":32435,"significance":"If the reported accuracies were valid, the paper would be a useful contribution to EEG-based auditory stimulus classification, particularly the energy-difference feature matrix and the bi-LSTM architecture. The dataset description and the feature extraction procedure are reasonably detailed, and the authors report multiple experimental variations. However, the evaluation protocol is invalid for the claims made: random splitting of overlapping trials from a small number of stimuli and subjects allows the network to memorize stimulus and subject identity rather than learn generalizable auditory categories. Because every headline accuracy in the abstract and Section 4 rests on this protocol, the central claim is unsupported. The paper does not provide machine-checked proofs, reproducible code, or baseline comparisons that would partially compensate for the methodological issue.","major_comments":[{"comment":"The evaluation protocol is invalid for measuring generalization. The manuscript states: 'The dataset used in each test is divided randomly into two subsets, one subset with 50% of the trials to train the network, and the other 50% of the trials are used to test the performance.' Because trials are 400-ms or 1-s windows with 50% overlap (Section 3.1), overlapping trials from the same recording share half of their raw samples. A random split places adjacent, nearly identical trials in both training and test sets, so the test accuracy reflects memorization of the training trials rather than classification of new brain responses. This flaw affects all results in Tables 6, 7, and 8 and the abstract's headline accuracies.","section":"Section 4"},{"comment":"The claim of an 'inter-subject scenario' is contradicted by the actual split. The random trial-level split pools all trials from all 6 subjects and divides them randomly, so every subject contributes to both training and test partitions. The model can therefore learn subject-specific EEG characteristics and use them to classify test trials from the same subjects. A true inter-subject evaluation would require training on some subjects and testing on held-out subjects; with only 6 subjects, such an evaluation would have only a few folds and would likely yield far lower accuracy. As it stands, the test procedure is not inter-subject and the phrase is misleading.","section":"Section 4"},{"comment":"There is also stimulus-level leakage. The music/voice task uses only 20 distinct music fragments (5 per genre) and 150 distinct sentences (30 per language), and all trials derived from a single stimulus share the same class label. The random split assigns trials from the same song or sentence to both training and test, so the network can memorize the identity of the specific audio excerpt rather than learning the abstract class 'music' or 'voice.' This is especially severe for the binary music/voice task, where 98.66% accuracy can be trivially achieved by recognizing the 20 songs against the 150 sentences. The same reasoning applies to the musical taste labels, which are tied to individual songs.","section":"Section 4.1"},{"comment":"No baseline or chance-level comparison is provided. The paper reports accuracies but does not compare against a simple classifier on the same features, a random classifier, or a cross-validation scheme with non-overlapping trials. Given the leakage described above, the reader cannot determine whether the proposed bi-LSTM adds any value beyond stimulus memorization. Additionally, the 10 repeated runs are averaged without reporting standard deviations, confidence intervals, or statistical significance tests, so even the internal variability of the reported numbers is unknown.","section":"Section 4"}],"minor_comments":[{"comment":"The variable D is used in Eqs. (6) and (7) before it is defined as equal to 1; please define D before its first use.","section":"Section 3.1"},{"comment":"The integration limits in Eq. (2) appear as '—1' and are likely a typographical artifact; they should presumably be -∞ to ∞ or the full frequency range. Please correct the notation.","section":"Section 3.1"},{"comment":"There is a typo: 'acuiqre' should be 'acquire.'","section":"Abstract"},{"comment":"The ballad and reggaeton tables list 5 songs each, but the four genre tables overall show 5 excerpts per genre; however, the text in Section 2.2.1 says 'Five fragments of different songs from each musical genre are played,' which is consistent. Please verify that the total count of 20 fragments matches the trial counts in Table 4.","section":"Appendix"},{"comment":"The phrase 'inter-subject scenario' in Section 4 should be removed or redefined, since the actual protocol is intra-subject with random trial splitting. If a subject-independent evaluation is intended, a leave-one-subject-out or subject-disjoint split should be used and clearly described.","section":"General"}],"recommendation":"reject","confidential_remarks":"The paper appears to have been published already in Neural Computing and Applications (DOI 10.1007/s00521-023-09061-3), so this review is evidently post-publication. The methodological flaw is load-bearing: every reported accuracy is contaminated by data leakage through overlapping trials and shared stimuli. In my view, the only way to salvage the claims is to re-run the evaluation with non-overlapping trials and a subject-disjoint split, which would likely require new data or a fundamentally different experimental design. Given the current dataset size (6 subjects, 20 songs, 150 sentences), even a corrected evaluation would have limited statistical power, so I cannot recommend acceptance or major revision as a path forward for this manuscript in its present form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline you should take from this paper is that the reported accuracies are not evidence of generalization. Section 4 describes a random 50/50 split of trials into train and test, and the trials are 400 ms windows with 50% overlap from just 20 songs and 150 sentences recorded from 6 subjects. The network can memorize stimulus and subject identity instead of learning the intended categories. The near-perfect music/voice results and the 96.96% taste result are consistent with that memorization, not with a genuine classifier.\n\nWhat is actually new: the feature matrix of pairwise log-energy differences between all 61 channels is a clean, reproducible construction, and the paper reports a new EEG dataset with clearly defined tasks. The authors are transparent about their protocol, which is what allows the flaw to be spotted quickly. The genre classification accuracy of 61.59% is modest and honestly reported.\n\nWhere the paper falls down: the evaluation protocol. The split is not inter-subject, despite the claim in Section 4; each subject appears in both training and test. Adjacent overlapping trials share raw samples. All trials from one song share the same label. Under those conditions, the model can simply learn to recognize the recording or the person. The paper lacks any control: no leave-one-subject-out or leave-one-song-out cross-validation, no comparison to a simple baseline on the same features, no permutation test against chance. All of that would be needed to support the claims. The small number of subjects and stimuli makes the memorization problem worse.\n\nThe feature construction might be of interest to people working on EEG-based audio classification, but the data are not openly available and the performance claims should not be cited. This is a well-written paper with a load-bearing methodological error. If it crossed a desk as a fresh submission, I would desk reject it; the failure is visible from the protocol description alone. For a reading group, it is a useful example of why trial-level random splits on overlapping windows are invalid for single-trial EEG classification.\n\nRecommendation: do not cite the performance numbers. Treat it as a data collection and feature-engineering note at most.","headline":"The reported accuracies are inflated by a trial-level random split of overlapping windows; the model memorizes stimulus and subject identity, so the central claims are unsupported.","tokens_in":12356,"tokens_out":2889,"would_cite":false,"duration_ms":30870,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims relative EEG channel energies read by a bidirectional LSTM label 400-millisecond trials as music or speech at 98.66% accuracy, distinguish four genres at 61.59%, and detect musical taste at 96.96%.","keywords":["EEG","bi-LSTM","energy-based features","music versus speech classification","musical genre classification","musical taste","brain-computer interface","auditory stimuli"],"falsifier":"Re-run the pipeline while holding out entire subjects, or using only non-overlapping trials from different songs, and compare the accuracy to the reported 98.66%, 61.59%, and 96.96%. If accuracy on held-out subjects or held-out stimuli falls to near chance, the central claim that the energy relations generalize across people and audio would be unsupported.","tokens_in":11428,"feed_emoji":"🧠","tokens_out":9350,"duration_ms":83636,"temperature":0.7,"pith_summary":"This paper sets out to show that what a person is listening to can be read from brief EEG recordings: pairwise differences in per-channel signal energy, arranged in a matrix and processed by a bidirectional long short-term memory network, are enough to label single 400-millisecond trials as music or speech, to tell among four music genres, and to tell whether the listener likes the song. On their own recordings from six participants, the scheme reaches 98.66% accuracy for the music-versus-speech task and up to 99.97% when the temporal evolution of energy differences is included. Accuracy for four-way genre classification reaches 61.59%, and binary like/dislike classification reaches 96.96%. The authors present these numbers as evidence that relative energy across brain areas carries consistent information about auditory content and preference, which would open low-cost EEG-based applications in music recommendation and brain-computer interaction.","feed_headline":"Brain-wave energy maps separate music from speech at 99.97%","feed_subtitle":"The approach labels 400-millisecond EEG trials, pointing toward real-time music recommendation and brain-computer interfaces.","key_machinery":"The central object is the channel-energy-difference matrix E(n): a 61-by-61 matrix whose entry at row i and column j is the difference in dB between the energy of channel i and the energy of channel j during trial n, so each row describes one channel's energy relative to every other channel and the diagonal is zero. A second matrix E'(n) approximates the discrete derivative of E(n) between consecutive trials, capturing how relative channel energies evolve over time. These matrices are fed as sequences into a bidirectional LSTM layer with 20 hidden units, followed by a fully connected layer and softmax, which lets the network use both past and future context in deciding the class of each trial.","core_discovery":"At the paper's core is the claim that a matrix of all pairwise differences in per-channel EEG energy over short trials, read as a sequence by a bi-LSTM, is a sufficient characterization of the brain's reaction to auditory content. For each trial, the energy of each of the 61 channels is computed through the Fourier transform and converted to decibels, and the 61-by-61 matrix E(n) stores every channel-minus-channel difference; the companion matrix E'(n) does the same for the change in these differences between consecutive trials. The bi-LSTM with 20 hidden units then consumes these matrices as time series. The best distinguishing power comes from E'(n) for voice-versus-music (99.97%) and for genre (61.59%), while the plain 400 ms E(n) matrix gives the best taste classification (96.96%). The authors interpret the overall pattern as evidence that energy relations between brain areas change measurably with the kind of sound and with the listener's preference.","pith_inferences":["A stricter evaluation not reported in the paper would hold out entire subjects or entire songs rather than randomly splitting overlapping trials; until that is done, the reported accuracies should be read as upper bounds on how well the features generalize to new listeners and new audio.","Because E(n) is antisymmetric with a zero diagonal, the 61-by-61 matrix is highly redundant; a natural testable extension is to select a small subset of channel pairs and measure how accuracy degrades, which would also make the approach more portable.","The network was trained for only five epochs with twenty hidden units, so the feature construction is likely doing most of the work; one could test whether a simpler classifier such as a linear model on the same matrices recovers most of the accuracy, which would separate the contribution of the features from that of the recurrent architecture.","A further experiment could separate musical taste from song familiarity and genre preference, since each subject heard a different set of songs and the like/dislike label may be entangled with how well the song is known."],"forward_implications":["Real-time EEG-based audio tagging becomes plausible: a 400 ms window is short enough to label what a user is hearing while it happens, without waiting for a longer stimulus.","Since the energy-difference feature is computed with an FFT and pairwise subtraction, it is cheap enough for consumer EEG hardware and needs no per-channel supervised feature learning.","Decoding like-versus-dislike at 96.96% from single-trial responses suggests a neural correlate of musical preference that recommendation systems could exploit without explicit user ratings.","The improvement from E'(n) over E(n) in the genre task indicates that the temporal change of relative channel energies carries information that static energy maps miss, guiding where to look for better auditory EEG features."],"supporting_citations":[{"why":"Supplies the 64-channel EEG recording system and electrode layout from which the 61 channel signals are taken.","marker":"[22]"},{"why":"Justifies the choice of 400 ms and 1 s trial durations by linking them to the P300 event-related potential.","marker":"[24]"},{"why":"Provides the fast Fourier transform used to compute each channel's energy from the time-domain EEG trial.","marker":"[25]"},{"why":"Defines the long short-term memory cell that the bidirectional LSTM classifier is built from.","marker":"[27]"},{"why":"Previously applied a bi-LSTM architecture to EEG error detection, which this paper extends to auditory stimulus classification.","marker":"[30]"},{"why":"Supplies the classical and metal music excerpts used as stimuli in the genre and taste classification experiments.","marker":"[33]"}],"fun_headline_variants":["EEG energy deltas yield 99.97% separation of speech and music","Bi-LSTM on EEG energy maps separates speech from music at 99.97%","Pairwise EEG energy differences classify music vs speech at 99.97%","EEG energy matrix plus bi-LSTM: 99.97% for speech-music discrimination"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a random 50/50 split of overlapping 400-millisecond trials into training and test sets measures generalization to new listening; because the trials come from only six subjects and a small set of repeated songs and sentences, this premise is what the reported accuracies rest on.","fun_headline_variants_meta":{"raw":{"variants":["EEG energy deltas yield 99.97% separation of speech and music","Bi-LSTM on EEG energy maps separates speech from music at 99.97%","Pairwise EEG energy differences classify music vs speech at 99.97%","EEG energy matrix plus bi-LSTM: 99.97% for speech-music discrimination"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000877,"raw_usage":{"total_tokens":3802,"prompt_tokens":964,"completion_tokens":2838,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":2748}},"tokens_in":580,"tokens_out":2838,"duration_ms":18113,"temperature":1.0,"reasoning_tokens":2748,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:40:51.365995+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the pipeline while holding out entire subjects, or using only non-overlapping trials from different songs, and compare the accuracy to the reported 98.66%, 61.59%, and 96.96%. If accuracy on held-out subjects or held-out stimuli falls to near chance, the central claim that the energy relations generalize across people and audio would be unsupported.","supporting_citations":[{"cited_title":"BrainVision, accessed 27 July","cited_arxiv_id":null,"evidence_quote":"Supplies the 64-channel EEG recording system and electrode layout from which the 61 channel signals are taken."},{"cited_title":"Springer, London","cited_arxiv_id":null,"evidence_quote":"Justifies the choice of 400 ms and 1 s trial durations by linking them to the P300 event-related potential."},{"cited_title":"Neural Comput 9(8):1735-1780","cited_arxiv_id":null,"evidence_quote":"Defines the long short-term memory cell that the bidirectional LSTM classifier is built from."},{"cited_title":"Biomed Signal Process Control 78:103885","cited_arxiv_id":null,"evidence_quote":"Previously applied a bi-LSTM architecture to EEG error detection, which this paper extends to auditory stimulus classification."},{"cited_title":"In: Proceedings of the 2nd inter­ national symposium on music information retrieval, Indiana, vol","cited_arxiv_id":null,"evidence_quote":"Supplies the classical and metal music excerpts used as stimuli in the genre and taste classification experiments."}],"review_version":1}