{"id":"e46658d3-4dda-4bef-8d45-07de9b74855e","arxiv_id":"2411.11308","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A multimodal speech-text-EEG match-mismatch model shows that semantic cues outperform acoustic cues in dichotic listening, while acoustic and semantic cues perform similarly in natural listening.","lead":"Researchers built a deep-learning model, STEM3, that reads EEG brain signals together with the audio and text of what someone is hearing, and then checks whether the brain response matches the speech. The method lets them compare how the brain uses sound versus meaning during normal listening and during a difficult two-ear dichotic listening task.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dichotic text advantage may be a subject-identity shortcut: attended side is fixed per subject, so MM accuracy need not reflect time-resolved semantic encoding.","rationale":"The reader's CONDITIONAL verdict is appropriate, but the most load-bearing issue is sharper than the ones listed. Trial selection, missing code, and underfitted baselines are real reproducibility concerns; the more decisive problem is that the dichotic MM comparison can be solved by a subject-identity shortcut, and the text modality makes that shortcut considerably easier than the envelope modality. This threatens both the absolute accuracy values and the semantic-vs-acoustic gap. The paper has genuine strengths: public datasets, a clear MM framing, and internally consistent STEM3 results across folds, which support conditional acceptance rather than rejection. I partially agree with the reader's weakest-assumption note about word2vec as a semantic proxy, but the missing control for fixed attended side is more fundamental. The proposed epoch-swap test is a concrete, feasible check that would settle whether the headline dichotomy result actually reflects neural encoding of semantics.","tokens_in":15394,"tokens_out":14369,"duration_ms":162134,"concrete_test":"Run an inference-time within-subject epoch swap on the dichotic test folds: for each test EEG segment, replace it with an EEG segment from another trial of the same subject while keeping the text/audio pair fixed, then recompute λ=0 and λ=1 MM accuracy. If text accuracy remains near 82% or the ~4.5-point text-over-envelope gap persists, the model is exploiting subject-level attended-side identity rather than per-segment EEG-stimulus correspondence, and the central semantic-dominance claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In the dichotic dataset, each participant is assigned to attend either the left or right ear for all 30 trials (Section II-A), so the attended story and speaker are constant within a subject. In the MM formulation (Section II-D), the positive pair is always the attended story segment and the negative pair is the unattended story segment. A model can therefore solve the task by recognizing the subject's fixed attended book/speaker from the EEG's subject-specific signature, without performing time-resolved stimulus-response matching. This is a known shortcut in auditory attention decoding when attention side is confounded with subject identity. The text branch makes the shortcut especially easy: the two narratives have distinct word identities, while the envelopes of two male voices are more confusable. The reported 82.00% (w2v) vs 77.45% (envelope) in Table I, interpreted in Section IV-C as stronger semantic encoding in EEG, may instead reflect how readily each feature encodes the constant attended-side label. The paper does not control for this (no leave-subject-out or within-subject epoch-shuffle experiments), and the Limitations section (IV-F) does not list it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a match-mismatch (MM) classification framework, STEM3, to compare how acoustic (speech envelope) and semantic (word2vec text embeddings) features of speech are encoded in EEG, under both natural and dichotic listening. The authors report that in natural listening (Dataset Small) acoustic and semantic cues yield similar MM accuracy (93.63% vs 93.24%), that in dichotic listening semantic cues outperform acoustic cues (82.00% vs 77.45%), that a multimodal combination reaches 83.76% on the dichotic task, that word boundary information significantly helps, and that a right-ear advantage is observed. The proposed model is compared with several prior MM models, and the authors claim large relative improvements.","tokens_in":15566,"tokens_out":6080,"duration_ms":62251,"significance":"If the findings are robust, the study offers a useful quantitative benchmark for comparing the neural encoding of acoustic and semantic speech features, and it extends the match-mismatch paradigm to dichotic listening, a relevant direction for auditory attention decoding. Strengths of the paper are its use of public datasets, a clearly specified model architecture, and several ablation studies (word boundaries, brain regions, loss functions). However, the central dichotic-listening claim (semantic over acoustic advantage) is threatened by a subject-identity confound, because attention side is fixed per subject and the MM task can be solved by recognizing the subject's constant attended story rather than by time-resolved stimulus-response matching. The behavioral trial-selection step and the baseline comparisons further weaken the evidence. These issues need to be addressed before the specific conclusions can be considered reliable.","major_comments":[{"comment":"In the dichotic dataset each participant is assigned to attend either the left or right ear for all 30 trials, so the attended story, the unattended story, and the speaker identities are constant within each subject. The MM task (Section II-D) defines the positive pair as the attended story segment and the negative pair as the unattended story segment. Because the training and test folds are split by trials rather than by subjects (Section II-H, 'Training and Evaluation Setup'), the model can exploit a subject-specific signature (e.g., overall EEG characteristics, or the speaker identity of the attended story) to classify correctly without performing time-resolved matching between the stimulus and the EEG. The reported semantic advantage (82.00% vs 77.45% in Table I) may therefore reflect that the two stories' word2vec text representations are more separable than the two male speakers' envelopes, rather than a stronger neural encoding of semantics. The paper does not provide any control experiment, such as leave-subject-out evaluation or a within-subject epoch-shuffle test, and the Limitations section (IV-F) does not list this confound. This is load-bearing for the abstract claim (c) that semantic cues are significantly better in dichotic listening, so it must be addressed.","section":"II-A, II-D, IV-C, Table I"},{"comment":"The trial-selection procedure removes 35% of dichotic trials based on behavioral comprehension thresholds (attended score >60% and unattended score <40%). Because this selection is applied before any train/test split, the reported accuracies are for a subset of trials that are behaviorally 'clean', which can inflate MM accuracy and may also differentially affect the acoustic and semantic conditions. More importantly, the selection uses information from the test trials themselves (their post-hoc comprehension scores), which is not available in a real-world auditory attention decoding setting; the results therefore generalize only to trials where the listener demonstrably attended as instructed. The paper should report results on the full trial set, or at least justify why the selected subset is the appropriate evaluation population.","section":"II-H, III-B"},{"comment":"The claimed relative improvements over baselines are not consistent with the numbers in Table I. For example, for the Natural Dataset - Small, the acoustic STEM3 result of 93.63% versus the baseline [12] of 65.23% corresponds to a relative improvement of (93.63-65.23)/65.23 = 43.5%, not the reported 79%; for the Dichotic Dataset, (77.45-62.00)/62.00 = 24.9%, not 50.73%. The reader cannot reproduce the stated improvements from the reported numbers. Additionally, the Wang et al. [17] results on Natural Dataset - Small (36.33%) and Natural Dataset - Large (50.33%) are at or below chance (50%), which is suspicious and suggests that the comparison protocol may not be matched (e.g., different segment durations, trial selection, or model inputs). The paper should clarify the exact definition of 'relative improvement' and ensure that baselines are evaluated under the same train/test scheme and input features.","section":"Table I, Section III-B"},{"comment":"The choice of the similarity function and the hyperparameter λ (modality mixing weight) is made using the dichotic listening data (Figure 4), and the best configuration (Sim1 with λ=0.5) is then used for all subsequent experiments, including the final reported accuracies. If this selection is performed on the same data that is later used for evaluation, the reported results on the dichotic dataset are partly the result of tuning on the test set. The authors should state whether the λ and similarity-function selection was done on a separate validation set or within each cross-validation fold, and if it was not, they should acknowledge the potential optimistic bias.","section":"III-A, II-I, Figure 4"}],"minor_comments":[{"comment":"The description of the trial-selection figure (Figure 3) says 'incorrectly responding to the unattended side', but the selection criterion in the text is a score of 40% or less rather than 'incorrect'. Please make the wording consistent.","section":"II-H"},{"comment":"The 'Without Word Boundary' ablation is described only as not using word boundaries, but it is unclear what temporal pooling scheme is used instead (e.g., fixed-length pooling, no pooling, or frame-level classification). Since word2vec features are inherently word-level, it is also unclear what 'without word boundary' means for the text branch. Please specify the exact architecture used in this ablation.","section":"II-G, Table II"},{"comment":"The sentence 'In natural listening condition (large), we find the acoustic features to be significantly better than textual features' should be rephrased as 'we find that the acoustic features...' for clarity. Also, the discussion should note that the natural-listening conclusion of 'similar' levels is based only on Dataset Small, since the large dataset shows a significant acoustic advantage.","section":"IV-B"},{"comment":"The word2vec embeddings are described as capturing semantics of words, but the paper does not discuss that these embeddings are pretrained on Google News and may not reflect the specific narrative context of the audiobook stories. This is a model choice that should be acknowledged when interpreting 'semantic cues'.","section":"II-F"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an interesting and timely problem, and the use of public datasets is commendable. However, the central dichotic-listening claim is currently not well supported because of the subject-identity confound and the trial-selection procedure. I would urge the editor to require the authors to add control experiments (e.g., leave-subject-out evaluation, within-subject shuffled labels, or a 'subject-only' baseline) and to report results with and without trial selection. If these controls indicate that the semantic advantage disappears, the paper's main contribution would be substantially weakened. The baseline comparison issues also need to be reconciled, since the reader cannot verify the claimed relative improvements from the reported numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, this paper ships a genuinely useful idea: a match-mismatch model that pools EEG and acoustic features at word boundaries and combines them with text embeddings, evaluated on public natural-listening and dichotic datasets. The natural-listening part is largely convincing — acoustic and semantic cues give similar accuracy there, and the word-boundary ablation is clean. Second, the headline dichotic result — that text beats envelope (82.0 vs 77.5, joint 83.8) — is probably an artifact of task design. In the dichotic dataset each subject always attends to the same ear/story across all 30 trials. So the attended story is a constant per subject, and the positive/negative pairs reduce to 'which of two fixed stories was this subject listening to?' The model can get high accuracy by recognizing subject identity from EEG, without any time-resolved stimulus-response matching. The text branch makes this even easier, because the word2vec embeddings are identical for each trial of a subject while the envelope differs somewhat. No leave-subject-out, no within-subject epoch-shuffle, and the limitations section does not mention this. That is a load-bearing flaw for the paper's main claim.\n\nThere is other good stuff. The trial-selection issue (dropping 35% based on comprehension) is a legitimate concern but less fatal. The acoustic-vs-semantic comparison is also entangled because the single model is trained with random λ and then evaluated at λ=0 or 1; independent single-modality training would be cleaner. Baselines are under-specified, especially the very low numbers attributed to Wang et al. And no code is released.\n\nWho is this for? The framework and word-boundary results are of interest to anyone working on EEG-based attention decoding or neural speech tracking. The natural-listening analysis may survive a reanalysis. But the dichotic story needs a proper control experiment before it should be published as evidence of semantic dominance.\n\nRecommendation: send it to peer review, but the authors must add leave-subject-out or within-subject epoch-shuffle controls and independently trained modality branches. If the dichotic advantage vanishes under those controls, the paper still has value as a modeling contribution, but the semantic-dominance conclusion is currently unsupported.","headline":"The unified MM framework is promising and the natural-listening results are solid, but the dichotic semantic-dominance claim is compromised by a subject-identity confound the authors do not control.","tokens_in":16153,"tokens_out":3778,"would_cite":false,"duration_ms":34854,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that in dichotic listening, EEG matches the attended story's text (82%) better than its sound envelope (77.45%), and the joint model reaches 83.76%.","keywords":["EEG","match-mismatch classification","dichotic listening","auditory attention decoding","speech envelope","word2vec","word boundaries","multimodal deep learning"],"falsifier":"Replace each word's word2vec vector in the dichotic listening pipeline with a random vector drawn from the same embedding distribution, keeping word boundaries and word order unchanged. If match-mismatch accuracy stays near 82%, the semantic advantage is an artifact of word-level alignment; if it drops toward the envelope's 77%, the EEG is genuinely tracking word meaning.","tokens_in":15148,"feed_emoji":"🧠","tokens_out":5008,"duration_ms":46230,"temperature":0.7,"pith_summary":"The paper asks how much of what the brain encodes while listening to speech is acoustic (the sound envelope) versus semantic (the meaning of words), and it answers by building a match-mismatch classifier: given an EEG segment and a speech segment, decide whether the speech evoked that brain response. In natural, single-stream listening, both cue types support the match decision about equally (93.63% vs 93.24% accuracy). In dichotic listening with a different story in each ear, the semantic stream wins: 82.00% for word2vec text features versus 77.45% for the acoustic envelope, and a model combining both reaches 83.76%. The paper also shows that word-boundary information significantly improves all conditions, supporting the idea that speech perception is segmented word by word, and that right-ear attention is decoded better than left-ear attention.","feed_headline":"Semantics beat acoustics in the brain's cocktail-party response","feed_subtitle":"In competing-talker listening, EEG matches the attended text 82% of the time vs 77% for sound; combining both hits 84%.","key_machinery":"The load-bearing object is STEM3, a three-branch sequence model. One branch encodes the speech envelope (a common proxy for acoustic neural tracking), one encodes the text of the speech using word2vec embeddings, and one encodes EEG; all three are pooled to word level using forced-alignment word boundaries, passed through LSTM or transformer layers, and compared via the negative L1 distance between stimulus and EEG embeddings. A single joint model is trained with a modality weight lambda that is randomly set to text-only, speech-only, or both, allowing the same model to produce all three accuracy numbers. The word-boundary average pooling is what lets the model align the acoustic and neural time courses at the rate of words rather than raw samples.","core_discovery":"The central discovery is that in a demanding two-talker (dichotic) listening situation, EEG signals carry a stronger signature of the attended story's text than of its acoustic envelope, while in ordinary natural listening the two are comparable. The paper establishes this through a unified match-mismatch model (STEM3) with separate acoustic, semantic, and EEG branches, and reports that the semantic branch reaches 82.00% match-mismatch accuracy versus 77.45% for the envelope branch; the multimodal branch reaches 83.76%. It further finds that accurate word boundaries matter for all modalities (random boundaries do not help), and that subjects attending to the right ear are decoded more accurately than those attending to the left ear, consistent with a right-ear advantage. The authors interpret these results as evidence that the brain prioritizes semantic content over low-level acoustics under competing-talker conditions.","pith_inferences":["Because word2vec embeddings encode word identity and co-occurrence statistics as well as meaning, the semantic advantage in dichotic listening may partly reflect lexical or positional alignment; a control with meaning-scrambled word vectors would separate true semantic encoding from word-level timing.","The trial selection removed 35% of dichotic trials based on comprehension scores, so the reported accuracies may overstate performance for inattentive or distracted listeners; testing on the excluded trials would quantify that gap.","The same three-branch architecture could be extended to other acoustic features (pitch, phonetic features) and to contextual embeddings (e.g., transformer-based), which would map a fuller hierarchy of what EEG tracks in competing speech."],"forward_implications":["Under dichotic listening, the match-mismatch accuracy of the multimodal model (83.76%) can serve as a segment-level estimate of how well the attended stream can be identified from EEG alone, a relevant signal for neuro-steered hearing aids.","Since random word boundaries do not reproduce the benefit of true boundaries, any practical decoder should use accurate word segmentation, not just fixed windows.","In natural listening, acoustic and semantic cues are interchangeable for this task, so models that drop one modality may still match performance in easy conditions.","The right-ear advantage appears not only in behavior but also in the EEG match-mismatch accuracy, which could make attention decoders more reliable when the attended ear is the right ear."],"supporting_citations":[{"why":"Supplies both the natural and dichotic speech-EEG datasets used for the main experiments.","marker":"[4]"},{"why":"The sentence-level LSTM baseline whose natural-listening results STEM3 improves on.","marker":"[12]"},{"why":"The LSTM baseline whose dichotic-listening results STEM3 improves on.","marker":"[42]"},{"why":"The multi-head attention+GRU baseline compared in Tables I and IV.","marker":"[21]"},{"why":"The self-supervised/contextual-text baseline that motivates the text branch comparison.","marker":"[17]"},{"why":"Prior work establishing word-boundary segmentation for EEG match-mismatch that STEM3 builds upon.","marker":"[22]"},{"why":"Provides the word2vec embeddings used as the semantic stimulus features.","marker":"[47]"},{"why":"SparrKULee, the large natural-listening dataset used for cross-subject evaluation.","marker":"[39]"},{"why":"Classic reference for the right-ear advantage used to interpret the ear-split results.","marker":"[26]"}],"fun_headline_variants":["In two-talker scenes, EEG matches text better than sound","Semantic cues dominate acoustic cues in dichotic listening","Brain's EEG shows semantic edge over acoustics in cocktail party","When listening in a crowd, brain tracks meaning over sound"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that word2vec embeddings of the attended story's transcript capture the semantic cues the EEG is actually encoding, and that the unattended story's embeddings serve as a genuinely mismatched semantic stimulus; if word2vec is not a faithful stand-in for the brain's semantics, the measured semantic advantage could be an artifact of word-level timing rather than meaning.","fun_headline_variants_meta":{"raw":{"variants":["In two-talker scenes, EEG matches text better than sound","Semantic cues dominate acoustic cues in dichotic listening","Brain's EEG shows semantic edge over acoustics in cocktail party","When listening in a crowd, brain tracks meaning over sound"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00022,"raw_usage":{"total_tokens":1469,"prompt_tokens":989,"completion_tokens":480,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":412}},"tokens_in":605,"tokens_out":480,"duration_ms":5613,"temperature":1.0,"reasoning_tokens":412,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:38:57.796703+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace each word's word2vec vector in the dichotic listening pipeline with a random vector drawn from the same embedding distribution, keeping word boundaries and word order unchanged. If match-mismatch accuracy stays near 82%, the semantic advantage is an artifact of word-level alignment; if it drops toward the envelope's 77%, the EEG is genuinely tracking word meaning.","supporting_citations":[{"cited_title":"Electrophysiological correlates of semantic dissimilarity reflect the comprehension of natural, narrative speech,","cited_arxiv_id":null,"evidence_quote":"Supplies both the natural and dichotic speech-EEG datasets used for the main experiments."},{"cited_title":"An LSTM based architecture to relate speech stimulus to EEG,","cited_arxiv_id":null,"evidence_quote":"The sentence-level LSTM baseline whose natural-listening results STEM3 improves on."},{"cited_title":"Extracting different levels of speech information from EEG using an LSTM-based model,","cited_arxiv_id":null,"evidence_quote":"The LSTM baseline whose dichotic-listening results STEM3 improves on."},{"cited_title":"Multi- head attention and GRU for improved match-mismatch classification of speech stimulus and EEG response,","cited_arxiv_id":null,"evidence_quote":"The multi-head attention+GRU baseline compared in Tables I and IV."},{"cited_title":"Self-supervised speech representation and contextual text embedding for match-mismatch classification with EEG recording,","cited_arxiv_id":null,"evidence_quote":"The self-supervised/contextual-text baseline that motivates the text branch comparison."},{"cited_title":"Enhancing the EEG speech match mismatch tasks with word boundaries,","cited_arxiv_id":null,"evidence_quote":"Prior work establishing word-boundary segmentation for EEG match-mismatch that STEM3 builds upon."},{"cited_title":"SparrKULee: A speech-evoked auditory response reposi- tory of the KU Leuven, containing EEG of 85 participants,","cited_arxiv_id":null,"evidence_quote":"SparrKULee, the large natural-listening dataset used for cross-subject evaluation."},{"cited_title":"The development of the right ear advantage in dichotic listening with focused attention,","cited_arxiv_id":null,"evidence_quote":"Classic reference for the right-ear advantage used to interpret the ear-split results."}],"review_version":1}