{"id":"4c812a11-aa9f-4b3c-94a7-a5a60dd180bb","arxiv_id":"2501.04844","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A multi-task EEG decoder with an auxiliary phoneme predictor decodes listened speech waveforms and phoneme sequences in parallel and reports improvements over prior single-task models.","lead":"This paper adds an auxiliary phoneme-prediction branch to an EEG-to-speech decoder, letting the same model output both heard speech waveforms and phoneme text at once. On the N400 dataset, the joint model reports substantially better speech reconstruction and better phoneme accuracy than the baselines it compares to.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Speech gains in Table II may stem from the added branch/objective rather than phoneme supervision; a random-target CTC control is needed before the enhancement claim is accepted.","rationale":"The reader's weakest assumption is the most load-bearing condition for the abstract's claim. If speech gains are reproduced by a control branch with random or meaningless targets, then the contribution attributed to simultaneous phoneme prediction collapses to a generic multi-task regularization effect. The paper's own conformer comparison provides some indirect evidence against a simple capacity explanation, because the smallest branch gives the best speech numbers, but it leaves open the possibility that the magnitude or presence of a secondary gradient signal, not its linguistic content, is what matters. I therefore agree with the reader. The phoneme-evaluation issues are secondary here because both models are scored under the same teacher-forced protocol and the two clean test splits still favor CB-2. I would not downgrade to reject: the control experiment is straightforward, the source code is public, and the paper's conclusions are conditional on it. Keeping the verdict at CONDITIONAL with the random-target CTC ablation as a required revision is the honest disposition.","tokens_in":8329,"tokens_out":6322,"duration_ms":68230,"concrete_test":"Run one ablation for the best speech configuration, CB-0: keep the architecture and Eq. (3) exactly as in Section II-B, but replace the target phoneme sequence z in the CTC loss with randomly permuted phoneme sequences (shuffle the target sequences across utterances within each batch, preserving length distribution), with α = 0.3 and the same number of training iterations and seeds. If the MCD/Mel-Corr of this random-target model matches CB-0 in Table II within 95% confidence intervals, the Table II improvement over FESDE cannot be attributed to phoneme supervision; if random targets produce no gain while real phoneme targets do, the central claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim asserts that the auxiliary phoneme predictor improves listened speech decoding. In the proposed training, the EEG module is optimized by L = L_EEG + αL_CTC (Eq. 3), while the speech module receives EEG embeddings through a gradient stop; consequently any speech gain must arise from changes in the EEG embeddings. Section III-B compares the full OURS configurations (CB-0/1/2) with FESDE in Table II, but each OURS configuration differs from FESDE in two confounded ways: it adds a parallel auxiliary branch (LSTM/attention or conformer blocks) and it adds αL_CTC to the EEG-module objective. No ablation trains FESDE with an equally sized auxiliary branch whose targets carry no phoneme information, nor does any control match the extra capacity and gradient scale without CTC. The CB-0 versus CB-2 trade-off shows added capacity alone is not monotonically beneficial, but it does not rule out that any secondary gradient signal, rather than phoneme content specifically, regularizes EEG encoder training. Because the paper's title and first contribution claim phoneme prediction is the enhancer, this missing control is load-bearing for the primary claim, not a minor robustness caveat.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-task framework for listened-speech decoding from EEG, built on the authors' previous FESDE model. An auxiliary phoneme predictor (CB-0/1/2 variants) is attached to the EEG embeddings and trained with a CTC loss, while a gradient stop isolates the speech module. The paper reports that all three configurations outperform FESDE on speech reconstruction metrics (Table II) and that the best configuration outperforms an adapted EEG2TEXT baseline on phoneme-sequence top-k accuracy (Table III). It also presents a phoneme-group analysis of consonant and vowel categories. The central claim is that the auxiliary phoneme-prediction task enhances the primary speech-decoding task, enabling simultaneous speech and text output.","tokens_in":8540,"tokens_out":5617,"duration_ms":57559,"significance":"If the enhancement claim is established, the paper would show that a single EEG encoder can serve both speech waveform and phoneme decoding, turning a multi-modal output into a benefit rather than a trade-off. The strengths include the release of source code and speech samples, the use of a public dataset, consistent and large improvements in speech metrics, and a detailed phoneme-group analysis. However, the current experiments do not separate the effect of phoneme supervision from the added capacity and regularization of the auxiliary branch, and the phoneme evaluation is partly conditioned on true previous phonemes and includes a test condition with seen target sequences. These gaps prevent the significance of the reported gains from being fully supported. The phoneme-group analysis is a useful descriptive contribution that partially offsets these concerns.","major_comments":[{"comment":"The attribution of the speech-decoding gains to phoneme supervision is confounded. OURS differs from FESDE by simultaneously (i) adding a parallel phoneme branch with extra parameters and (ii) adding alpha*L_CTC to the EEG-module objective in Eq. (3). Because a gradient stop prevents the speech module from influencing the EEG module, any speech gain must arise from changed EEG embeddings, but those embeddings are changed by the combined effect of the extra capacity and the CTC loss. No ablation trains FESDE with an equal-capacity auxiliary branch whose targets carry no phoneme information (e.g., randomized or permuted phoneme labels), nor a speech-only model with matched parameter count and training iterations. The CB-0 vs. CB-2 comparison shows that added capacity alone is not monotonically beneficial, but it does not rule out that any secondary gradient signal, rather than phoneme content specifically, regularizes the EEG encoder. This control is load-bearing for the paper's second contribution and should be added before the enhancement claim is accepted.","section":"Section III-B, Eq. (3), Table II"},{"comment":"The reported phoneme metrics are conditional top-k accuracies computed given previous true phonemes (teacher forcing). This does not measure full phoneme-sequence decoding from EEG alone, because errors at earlier positions are not propagated; a model that exploits phonotactic regularities can score well even if it fails at the first phoneme. In addition, the 'unseen subject' test set shares target phoneme sequences with the training set, a fact the authors acknowledge when excluding it from Fig. 2c, yet the high numbers for that condition are still presented in Table III without a marker or caveat. The authors should report an unconditional or full-sequence metric (e.g., phoneme error rate with a language-model-free decoding protocol) and clearly separate the seen-phoneme condition from generalization claims.","section":"Section IV-B, Table III, Fig. 2c"}],"minor_comments":[{"comment":"The statement that 'the proposed approach also outperforms previous methods in both modalities' is stronger than the data: in Table III, CB-1 is below EEG2TEXT-PT on the unseen-subject test set (51.07 vs. 53.38 top-1), and only CB-2 consistently outperforms the baseline. Please qualify the claim to the best configuration or specify the configuration being claimed.","section":"Abstract and Section IV"},{"comment":"The notation for the CTC loss is inconsistent: Eq. (1) defines L_CTC, while Eq. (3) writes L_ctc. In addition, the variables z and \\hat{z} in Eq. (1) are not explicitly defined in the text; please define the target and predicted phoneme sequences at that point.","section":"Section II-B and III-B"},{"comment":"There is a typo: 'ouputs' should be 'outputs'. Also, the sentence 'The EEG encoder takes preprocessed EEG signals as input and ouputs EEG embeddings' repeats the description of the EEG encoder; consider tightening the paragraph.","section":"Section II-A1"},{"comment":"The data-split description refers only to [16]; the number of sentences per held-out test set, the specific subject identifiers, and the degree of phoneme-sequence overlap between train and test are not specified. These details are needed to interpret Table III, especially for the unseen-subject condition.","section":"Section III-A"},{"comment":"The caption of Figure 2 contains typos ('T enseness', 'T op k Accuracy'), and the y-axis of panel (c) is labeled as a fraction (0.3-0.7) while the text calls it 'Top-3 accuracy (%)'. Please align the units and labels.","section":"Section IV-B and Figure 2"},{"comment":"The hyperparameter alpha = 0.3 is chosen heuristically and no sensitivity analysis is provided. Since the central claim concerns the effect of the auxiliary task, a small sweep around 0.3 (e.g., 0.1, 0.5) would increase confidence that the reported results are not tied to this specific value.","section":"Eq. (3) and Section III-B"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is transparent about building on the authors' own FESDE framework, and the reuse of that pipeline is cited. The main risk is that the central attribution of speech gains to phoneme supervision is not yet proven; this is fixable with the ablations described in the major comments. I do not see grounds for rejection, but the current evidence supports only a more modest claim about multi-task training with an auxiliary phoneme branch."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on 2501.04844. The headline: the authors take their own FESDE EEG-to-speech model, bolt on a CTC phoneme predictor that shares the EEG encoder, and report that speech reconstruction gets much better (Mel-Corr roughly doubles). That's a genuinely interesting result, and to my knowledge no one has shown parallel waveform-plus-phoneme decoding from EEG in one model. The code and samples are public, the writing is clear, and the gains on speech are consistent across all three test splits with confidence intervals. The phoneme-group analysis is a nice extra, and the stop-gradient between the EEG and speech modules is a sensible design choice.\n\nThe soft spots are real though. First, the central attribution isn't proven. The comparison is against FESDE, but the new model differs in two confounded ways: it adds an auxiliary branch with its own parameters, and it adds the CTC loss to the EEG objective. Without a control that uses a matched-capacity auxiliary branch with random phoneme targets (or some other non-phoneme task), you can't say the improvement comes from phoneme supervision rather than from regularization induced by any secondary gradient. The stress-test note has this right; this is load-bearing for the main claim, not a minor caveat.\n\nSecond, the abstract overstates. 'Outperforms previous methods in both modalities' is only true for the CB-2 config on the two clean test sets; CB-0 and CB-1 are actually worse than EEG2TEXT on unseen speech and unseen-both in Table III. And the 'unseen subject' set is contaminated because the target phoneme sequences appear in training, as the authors themselves note. That win shouldn't be headline material.\n\nThird, the phoneme metric is teacher-forced top-k accuracy given previous phonemes. It's a defensible choice without an external LM, but it measures conditional decoding, not full-sequence decoding. The authors are transparent about this, so I'd call it minor.\n\nOverall: the architecture is a reasonable contribution, the speech findings are plausible, and the paper deserves a serious referee. But the enhancement claim needs the random-target ablation, and the abstract needs to be reined in. I'd send it to review with that expectation.","headline":"A promising multi-task EEG-to-speech/phone decoding result whose central attribution claim is not yet proven; needs a control ablation and a more precise abstract.","tokens_in":9075,"tokens_out":2786,"would_cite":true,"duration_ms":26340,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding an auxiliary phoneme-sequence predictor to an EEG-to-speech model lets one system decode listened speech as both audio and text in parallel, and improves both outputs.","keywords":["EEG speech decoding","phoneme sequence prediction","parallel decoding","brain-computer interface","multi-task learning","listened speech","CTC loss","audio reconstruction"],"falsifier":"Train a control model identical to CB-0 but with the phoneme predictor fed randomly shuffled phoneme targets, or with a same-capacity branch trained only on EEG reconstruction; if mel-spectrogram correlation stays close to the reported 22–32% range, the gains are not caused by phoneme supervision and the central claim collapses. A matched-capacity speech-only baseline with no phoneme branch would also settle the attribution.","tokens_in":8145,"feed_emoji":"🧠","tokens_out":4910,"duration_ms":41015,"temperature":0.7,"pith_summary":"This paper tries to show that a brain-computer interface can decode listened speech from EEG in two modalities at once—reconstructed audio waveforms and text-like phoneme sequences—and that asking the model to predict phonemes as a side task actually makes the audio reconstruction better. On the N400 dataset with held-out speakers and sentences, the proposed model roughly doubles the mel-spectrogram correlation of the decoded speech compared with the previous end-to-end system, while also beating a text-decoding baseline on phoneme top-1 accuracy in the harder test conditions. The authors interpret this as evidence that phoneme-level supervision sharpens the EEG embeddings in a way that benefits the primary speech reconstruction task. If true, the result matters for BCI communication because a single model could deliver both audible and textual output without chaining separate decoders.","feed_headline":"Phoneme branch doubles EEG speech correlation, beats text baselines","feed_subtitle":"A parallel phoneme branch lifts reconstructed waveform accuracy and beats prior text-decoding baselines on the same EEG data.","key_machinery":"The load-bearing object is the auxiliary phoneme predictor: a conformer-block encoder followed by a single-layer LSTM with attention, fed the same EEG embeddings that the speech module uses, and trained with connectionist temporal classification (CTC) loss. Its total objective is $L = L_{EEG} + \\alpha L_{CTC}$ with $\\alpha = 0.3$, so the EEG module is pushed to preserve phoneme-discriminative structure in its embeddings. A gradient stop between the EEG module and the speech module means the speech branch does not influence the EEG encoder's training signal; the phoneme loss is what changes the shared embedding geometry relative to the previous system.","core_discovery":"The central claim is that parallel phoneme sequence prediction is not just an add-on output but a training signal that improves listened speech decoding from EEG. Concretely, the paper's configurations CB-0, CB-1, and CB-2, which share EEG and speech modules with the FESDE baseline but add a phoneme branch trained with CTC loss, reduce mel-cepstral distortion from about 11.7–11.8 dB to 9.6–11.3 dB and raise mel-spectrogram correlation from roughly 13–14% to 22–32% across the three held-out test sets. On phoneme decoding, the two-conformer-block configuration beats the adapted EEG2TEXT baseline on top-1 accuracy on all three test sets, with the largest margin on the unseen-both set. The paper also reports a trade-off: more conformer blocks improve phoneme decoding while slightly hurting speech reconstruction, yet even the strongest phoneme configuration keeps speech reconstruction above the previous baseline.","pith_inferences":["If the phoneme branch is genuinely doing the work, a natural testable extension is to use richer phoneme targets, such as those with stress or duration marks, or to add a language model over phonemes; the paper's own results suggest gains would concentrate on consonants, where the modality trade-off is largest.","The reported trade-off across conformer depth suggests that the auxiliary task's parameter count, not just the task itself, may be a confound; matched-capacity ablations with random or shuffled phoneme targets would separate phoneme supervision from extra parameters.","The high top-1 accuracy on the unseen-subject test set likely reflects sequence memorization, since the target phoneme sequences are already seen during training; future subject-independent evaluation should hold out sentences as well.","The same parallel architecture could be tried on imagined or attempted speech, where phoneme targets are less well time-aligned and CTC's many-to-one flexibility may behave differently."],"forward_implications":["A single EEG-to-speech model can output both reconstructed audio and phoneme text simultaneously, removing the need to chain a speech synthesizer or recognizer after a single-modality decoder.","Phoneme-level supervision improves waveform reconstruction enough to roughly double mel-spectrogram correlation over the previous end-to-end baseline.","The choice of phoneme predictor capacity trades off the two modalities: one LSTM layer gives the best audio, while two conformer blocks give the best phoneme accuracy.","The phoneme-level analysis shows articulation classes differ in decodability, with nasal, dental, and voiced consonants easier in audio and front, close, and lax vowels easier in text.","The approach is currently limited to listened speech perception, and the authors point to imagined or attempted speech as the natural next step."],"supporting_citations":[{"why":"Supplies the FESDE baseline and the EEG/speech module architecture that the proposed framework extends.","marker":"[16]"},{"why":"Supplies the EEG2TEXT text-decoding baseline that the phoneme branch is compared against.","marker":"[8]"},{"why":"Provides the N400 EEG dataset and the listened-speech experimental paradigm.","marker":"[24]"},{"why":"Provides the CTC loss used to train the phoneme predictor.","marker":"[23]"},{"why":"Supplies the conformer blocks used in the phoneme predictor.","marker":"[22]"},{"why":"Supplies the WaveNet residual blocks used in the speech encoder.","marker":"[14]"},{"why":"Supplies HiFi-GAN V1 as the speech decoder for waveform generation.","marker":"[19]"},{"why":"Supplies the structured state space (S4) block used in the EEG encoder.","marker":"[17]"}],"fun_headline_variants":["Phoneme branch lifts EEG speech on both tracks","Parallel phoneme prediction boosts EEG speech decoding","EEG speech improves with a parallel phoneme branch","Side phoneme task lifts EEG speech waveform and text"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper credits the phoneme-prediction task for the speech gains, but it never runs a control with the same extra capacity and no phoneme information, so the improvement could come from the extra parameters rather than the phoneme supervision.","fun_headline_variants_meta":{"raw":{"variants":["Phoneme branch lifts EEG speech on both tracks","Parallel phoneme prediction boosts EEG speech decoding","EEG speech improves with a parallel phoneme branch","Side phoneme task lifts EEG speech waveform and text"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000757,"raw_usage":{"total_tokens":3355,"prompt_tokens":924,"completion_tokens":2431,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":2370}},"tokens_in":540,"tokens_out":2431,"duration_ms":15919,"temperature":1.0,"reasoning_tokens":2370,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:23:52.402790+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a control model identical to CB-0 but with the phoneme predictor fed randomly shuffled phoneme targets, or with a same-capacity branch trained only on EEG reconstruction; if mel-spectrogram correlation stays close to the reported 22–32% range, the gains are not caused by phoneme supervision and the central claim collapses. A matched-capacity speech-only baseline with no phoneme branch would also settle the attribution.","supporting_citations":[{"cited_title":"Toward fully-end-to-end listened speech decoding from eeg signals,","cited_arxiv_id":null,"evidence_quote":"Supplies the FESDE baseline and the EEG/speech module architecture that the proposed framework extends."},{"cited_title":"Evoking the n400 event-related potential (erp) component using a publicly available novel set of sentences with semantically incongruent or congruent eggplants (endings),","cited_arxiv_id":null,"evidence_quote":"Provides the N400 EEG dataset and the listened-speech experimental paradigm."},{"cited_title":"Conformer: Convolution- augmented Transformer for Speech Recognition,","cited_arxiv_id":null,"evidence_quote":"Supplies the conformer blocks used in the phoneme predictor."},{"cited_title":"WaveNet: A Generative Model for Raw Audio,","cited_arxiv_id":null,"evidence_quote":"Supplies the WaveNet residual blocks used in the speech encoder."},{"cited_title":"Efficiently modeling long sequences with structured state spaces,","cited_arxiv_id":null,"evidence_quote":"Supplies the structured state space (S4) block used in the EEG encoder."}],"review_version":1}