Pith. sign in

REVIEW 4 major objections 5 minor 24 references

State-of-the-art Speech Recognition using EEG and Towards Decoding of Speech Spectrum From EEG

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper reports that EEG recorded while people speak or listen in noise can drive continuous end-to-end speech recognition on a small English vocabulary, and that an LSTM regression model decodes speech spectrum features (MFCCs) from…

desk verdict Interesting preliminary work on EEG-based speech recognition and MFCC decoding, but the headline decoding numbers cannot be trusted until the train/test split and baselines are clarified. read the letter →

arxiv 1908.05743 v5 pith:4EZGF73W submitted 2019-08-14 eess.AS cs.SD

Gautam Krishna , Yan Han , Co Tran , Mason Carnahan , Ahmed H Tewfik This is my paper · ORCID
classification eess.AScs.SD
keywords EEGspeechrecognitionend-to-endASRCTCattentionmodelRNNtransducerMFCCdecodingLSTMregressionsynthesisfrom
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that non-invasive EEG signals carry enough information about spoken and heard speech to support both recognition and spectrum reconstruction on a small English vocabulary. Using end-to-end ASR models (CTC, attention, RNN transducer), the authors report continuous noisy speech recognition for three to nine sentences, with the attention model reaching 0% word error rate on the three-sentence spoken corpus. They also train LSTM-regression, GAN, and WGAN models to map EEG to MFCC features, and report that LSTM regression gives the lowest RMSE and mel cepstral distortion on held-out test data. If these results hold, EEG could become a non-invasive input for speech-restoration interfaces, avoiding the surgery required by ECoG-based decoders.

What carries the argument

The machinery is a pipeline from EEG to text or spectrum. Three EEG feature sets are extracted at 100 Hz — the set used in earlier work (RMS, zero-crossing rate, moving-window average, kurtosis, power spectral entropy), a short-time-Fourier/wavelet-entropy set, and a neuroscience-style band-entropy/Hurst/fractal-dimension set — then reduced with kernel PCA and augmented with delta and delta-delta features. Recognition runs through three end-to-end ASR architectures: CTC, attention-based encoder-decoder, and RNN transducer. Spectrum decoding runs through a two-layer LSTM regression model and through GAN/WGAN generators that take real EEG and output MFCC vectors; the LSTM regression model carries the argument because it produces the reported low test errors, while the adversarial models are presented as a less stable alternative.

What would settle it

Re-run the MFCC-decoding experiments with a leave-one-subject-out or leave-one-sentence-out split: if the LSTM regression's normalized RMSE and MCD rise to the level of the GAN/WGAN results (or higher), the low reported errors are artifacts of train/test overlap rather than evidence of generalizable speech-spectrum decoding from EEG.

Watch

Extended reading notes

Core claim

The central claim is that EEG recorded in parallel with speech (and with listening) can be mapped directly to text and to acoustic features. For recognition, the paper uses character-level CTC and RNN-transducer models and a word-level attention encoder-decoder, all trained on EEG feature sets after KPCA dimensionality reduction; on the smallest corpus of three sentences the attention model's test WER is 0%, rising as vocabulary grows, and the CTC model yields CER values from about 27% to 73% depending on condition and corpus size. For spectrum decoding, an LSTM regression model predicts 13-dimensional MFCC sequences from EEG, with normalized RMSE around 0.027-0.126 and MCD around 1.3-5.7, while GAN and WGAN models perform substantially worse and are harder to train. The paper frames these as feasibility results: EEG supports continuous noisy speech recognition across different experimental conditions and provides preliminary, not production-ready, evidence for speech synthesis from EEG.

Load-bearing premise

The speech-spectrum decoding results depend on the assumption that the 90/10 train/test split keeps the same utterance from the same speaker out of the training set; the paper does not state that the split is subject-independent.

Editorial extensions

If this is right

  • If the recognition results generalize beyond the small corpus, EEG could be used as a non-invasive input modality for voice-activated systems and speech-restoration devices, avoiding ECoG surgery.
  • The reported performance ranking — attention and CTC outperform RNN transducer, and LSTM regression outperforms GAN/WGAN — gives future EEG-based systems a concrete architecture baseline to start from.
  • Since error rates rise with vocabulary size on all models, larger multi-speaker EEG corpora and external language models would be needed before real-world continuous EEG speech recognition is practical.
  • The predicted MFCC features can in principle be converted to audible speech with Griffin-Lim reconstruction, making a complete silent-speech interface from EEG a testable next step.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The MFCC-decoding numbers should be read with caution until a subject-independent split is tested: because every subject repeats the same sentences, a random 90/10 split can place repetitions of the same sentence in both training and test, which would make the low RMSE/MCD values partly a memorization result.
  • A stronger test of the recognition claim would use held-out sentences (leave-one-sentence-out) as well as held-out speakers; the reported 0% WER on three sentences likely depends on the closed vocabulary of nine sentences.
  • If the decoding pipeline does transfer across subjects, the natural next comparison is to the invasive ECoG decoder that inspired this work, since EEG's non-invasiveness would be its main practical advantage.
  • The observation that a simple LSTM regression beats adversarial models suggests that for EEG-to-spectrum problems, reconstruction loss may be the right objective and GAN training instability is a cost without evident benefit in this data regime.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper claims two main results: continuous noisy speech recognition from EEG using end-to-end ASR models (CTC, attention, RNN transducer) across three databases and three EEG feature sets, and decoding of speech spectrum (MFCC) from EEG using LSTM regression, GAN, and WGAN models. The authors report low WER/CER for small sentence corpora (e.g., 0% WER for 3 sentences in some settings) and very low normalized RMSE and MCD for MFCC prediction (e.g., normalized RMSE 0.0291 in Table 8). The paper concludes that EEG can support both recognition and synthesis of speech over a limited vocabulary.

Significance. If the claims are valid, this would be a notable step toward non-invasive EEG-based speech communication, with implications for assistive technology and brain-computer interfaces. The paper is commendable for exploring multiple experimental conditions (spoken, listen, concatenated), three feature sets, and several model families, and for providing preliminary synthesis results. However, the central feasibility claim is not currently supported by the evidence as presented: the decoding experiments lack a specified train/test split unit, no baselines or significance tests are provided, and the ASR experiments have similar split ambiguities. These issues are load-bearing because they affect whether the reported metrics reflect true generalization. Therefore, the current significance is conditional on a substantially improved validation.

major comments (4)
  1. [Sections 6.1 and 6.2, Tables 8-11] The train/test split for the MFCC decoding experiments is not specified at the level of frames, trials, sentences, or subjects. In Sections 6.1 and 6.2 the paper states only that 90% of the data is used for training and 10% for testing, without defining the unit of the split. Since each subject repeats the same nine (or thirty) sentences, a random frame-level split can place nearly identical EEG and MFCC frames in both training and test sets. Given the strong autocorrelation of these signals, this would inflate the decoding metrics and undermine the claim of spectrum decoding. Please provide the split unit and results for a subject-independent and utterance-independent split.
  2. [Section 8, Tables 1-7] The ASR experiments for CTC and attention models use an 80/10/10 train/validation/test split, but the split unit is not stated. Only the RNN transducer section (Section 8) explicitly separates subjects. If the CTC/attention splits are random at the utterance or frame level, then the same sentence spoken or listened to by the same subject can appear in both training and test sets, artificially lowering WER and CER. The paper must clarify the split unit and provide results for a subject-independent split, or justify why the current split is appropriate.
  3. [Tables 8-11 and Tables 1-7] No baselines or statistical significance tests are reported anywhere. For the decoding experiments, there is no comparison to a trivial predictor such as the mean MFCC or the previous-frame MFCC, so the low normalized RMSE and MCD values in Tables 8-11 cannot be evaluated. For the ASR experiments, there is no chance-level WER/CER or confidence intervals, making it impossible to judge whether the results are meaningful. The authors should add baselines and significance tests, or explicitly justify their absence.
  4. [Section 8 and Tables 1-7] The paper claims 'state-of-the-art speech recognition using EEG' in the title and abstract, but it does not compare the reported WER/CER with the prior results in references [1] and [2] under matched conditions. The only stated comparison is internal among the authors' own feature sets and models. Without a direct comparison to existing EEG-based ASR results, the 'state-of-the-art' claim is unsupported. Please provide quantitative comparisons with the same evaluation protocols.
minor comments (5)
  1. [Section 5] The procedure for choosing the KPCA dimension using the explained-variance plot from PCA is conceptually questionable, since KPCA does not share the same explained-variance interpretation as PCA. Please clarify the justification or use a more standard model-selection approach (e.g., validation-set performance).
  2. [Section 6] The text says 'we used three evaluation metrics: RMSE, Normalized RMSE and Mel cepstral distortion (MCD)' but does not define how the average values in Tables 8-11 are computed across test set samples. Please specify the averaging procedure.
  3. [Section 10] The paper mentions that the databases are planned for release, but they are not currently available. Given that the split ambiguity is a central issue, providing data or detailed subject/utterance indices for the splits would greatly help verification.
  4. [Throughout] Several figures (Figures 3-5, 6-7, 8-10, etc.) are not referenced in the main text, making it unclear where the reader should look for supporting evidence. Please add appropriate references.
  5. [Section 4] The feature extraction text states 'for this set, EEG feature dimension was 31(channels) × 5 or 155' and similar lines, but the notation is inconsistent with the later statement that final dimensions include delta and delta-delta features. Please clarify the dimensions before and after taking differentials.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the reported results are held-out test predictions and self-citations are methodological imports, not load-bearing derivations.

full rationale

I walked the derivation chain from EEG feature extraction through ASR and MFCC decoding. The paper's central claims are empirical: CTC/attention/RNN-transducer models map EEG features to text, and LSTM/GAN/WGAN models map EEG features to MFCC features. The reported test metrics (CER, WER, RMSE, normalized RMSE, MCD) are computed on held-out test splits, not on training fits. The self-citations to the authors' prior work ([1], [2]) are used only to import feature definitions, preprocessing steps, KPCA kernel choice, and the attention mechanism; none of these imports is itself derived from the target results of this paper, and none functions as a uniqueness theorem or a forced ansatz that would make the present conclusions equivalent to the cited inputs. The KPCA dimension for feature set 2 is chosen using a development set, which is a hyperparameter-selection procedure rather than a fitted parameter renamed as a prediction. The normalization of RMSE by the test-set range is a reporting convention and does not inject the target value into the model. The unspecified 90/10 split in Sections 6.1 and 6.2 is a potential data-leakage and generalization-validity concern, but leakage is not circularity: even with leakage, the evaluation is not the same as setting the prediction equal to a fitted input by construction. No equation or self-citation chain reduces the claimed decoding or recognition results to their own inputs. Therefore the paper shows no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper depends on standard ML assumptions, several domain assumptions about EEG data quality and synchronization, and a few data-driven hyperparameter choices (KPCA dimensions). No new theoretical entities are introduced.

free parameters (4)
  • KPCA dimension for EEG feature set 1 = 30
    Chosen from cumulative explained variance plot (Figure 8); data-driven.
  • KPCA dimension for EEG feature set 2 = 50
    Chosen using a development set because the explained variance plot was ambiguous (Figure 10).
  • Number of training epochs per model = 800 (CTC), 150 (Attention), 200 (RNN-T), 500 (GAN/WGAN)
    Chosen by observing loss convergence; no early stopping.
  • Number of sentences used for evaluation = 3, 5, 7, 9
    The corpus contains only 9 sentences; these subsets are used to show scaling.
assumptions (4)
  • standard math CTC loss, attention mechanism, RNN-T, and GAN/WGAN training are taken as given from cited prior work.
    These are standard model components the paper relies on without re-deriving.
  • domain assumption The EEG preprocessing (band-pass filter, ICA artifact removal) preserves speech-relevant neural signals and removes artifacts.
    Preprocessing details are given in Section 4, but the assumption that the remaining signal is informative for ASR or MFCC decoding is not validated independently.
  • domain assumption MFCC features capture enough speech information for reconstruction and are suitable as the decoding target.
    MFCC is a standard speech representation, but the paper does not justify its completeness for speech synthesis from EEG.
  • domain assumption The recorded EEG is time-locked to the corresponding audio (spoken or listened).
    The synchronization between EEG and audio is implied in the experimental design but not explicitly verified or reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of State-of-the-art Speech Recognition using EEG and Towards Decoding of Speech Spectrum From EEG." pith.science (2026). https://pith.science/paper/4EZGF73W

@misc{pith2026190805743,
  author       = {Pith},
  title        = {Pith review of: State-of-the-art Speech Recognition using EEG and Towards Decoding of Speech Spectrum From EEG},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4EZGF73W}},
  note         = {Machine review of arXiv:1908.05743}
}
read the original abstract

In this paper we first demonstrate continuous noisy speech recognition using electroencephalography (EEG) signals on English vocabulary using different types of state of the art end-to-end automatic speech recognition (ASR) models, we further provide results obtained using EEG data recorded under different experimental conditions. We finally demonstrate decoding of speech spectrum from EEG signals using a long short term memory (LSTM) based regression model and Generative Adversarial Network (GAN) based model. Our results demonstrate the feasibility of using EEG signals for continuous noisy speech recognition under different experimental conditions and we provide preliminary results for synthesis of speech from EEG features.

Figures

Figures reproduced from arXiv: 1908.05743 by the authors.

Figure 1
Figure 1. Generator in our GAN model [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Discriminator in our GAN model Our generator model consists of two layers of LSTM with 128 hidden units in each layer followed by a time distributed dense layer with 13 hidden units. During training, real listen EEG features say with dimension 30 (when used with EEG feature set 1) from training set are fed into the generator model and the generator outputs a vector of dimension 13, which can be considered as fake li… view at source ↗
Figure 3
Figure 3. EEG channel locations for the cap used in our experiments [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (24 more)
Figure 4
Figure 4. Figure 4: Three dimensional view of the EEG sensor locations for the cap used in our experiments [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Three dimensional view of the EEG sensor locations for the cap used in our experiments [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Attention model training loss convergence [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: CTC model training loss convergence [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Cumulative explained variance plot for EEG feature set 1 [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Cumulative explained variance plot for EEG feature set 3 [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Cumulative explained variance plot for EEG feature set 2 [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Training loss for WGAN discriminator model [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Training loss for WGAN generator model [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Training loss for GAN generator model [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Training loss for GAN discriminator model [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 15
Figure 15. Figure 15: Training loss for LSTM regression model [PITH_FULL_IMAGE:figures/full_fig_p017_15.png]
Figure 16
Figure 16. Figure 16: Test time result for WGAN model with feature set 1 EEG [PITH_FULL_IMAGE:figures/full_fig_p017_16.png]
Figure 17
Figure 17. Figure 17: Test time result for GAN model with feature set 1 EEG [PITH_FULL_IMAGE:figures/full_fig_p017_17.png]
Figure 18
Figure 18. Figure 18: Test time result for LSTM regression model with feature set 1 EEG [PITH_FULL_IMAGE:figures/full_fig_p018_18.png]
Figure 19
Figure 19. Figure 19: Test time result for WGAN model with feature set 2 EEG [PITH_FULL_IMAGE:figures/full_fig_p018_19.png]
Figure 20
Figure 20. Figure 20: Test time result for GAN model with feature set 2 EEG [PITH_FULL_IMAGE:figures/full_fig_p018_20.png]
Figure 21
Figure 21. Figure 21: Test time result for LSTM regression model with feature set 2 EEG [PITH_FULL_IMAGE:figures/full_fig_p019_21.png]
Figure 22
Figure 22. Figure 22: Test time result for WGAN model with feature set 3 EEG [PITH_FULL_IMAGE:figures/full_fig_p019_22.png]
Figure 23
Figure 23. Figure 23: Test time result for GAN model with feature set 3 EEG [PITH_FULL_IMAGE:figures/full_fig_p019_23.png]
Figure 24
Figure 24. Figure 24: Test time result for LSTM regression model with feature set 3 EEG [PITH_FULL_IMAGE:figures/full_fig_p020_24.png]
Figure 25
Figure 25. Figure 25: Test time result for WGAN model for predicting spoken MFCC from spoken EEG [PITH_FULL_IMAGE:figures/full_fig_p020_25.png]
Figure 26
Figure 26. Figure 26: Test time result for GAN model for predicting spoken MFCC from spoken EEG [PITH_FULL_IMAGE:figures/full_fig_p020_26.png]
Figure 27
Figure 27. Figure 27: Test time result for LSTM regression model for predicting spoken MFCC from spoken [PITH_FULL_IMAGE:figures/full_fig_p021_27.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 19 canonical work pages

  1. [1]

    Krishna, G., C. Tran, J. Yu, et al. Speech recognition with no speech or with noisy speech. In Acoustics, Speech and Signal Processing (ICASSP), 2019 IEEE International Conference on . IEEE, 2019

    work page 2019

  2. [2]

    Krishna, G., C. Tran, M. Carnahan, et al. Advancing speech recognition with no speech or with noisy speech. In 2019 27th European Signal Processing Conference (EUSIPCO) . IEEE, 2019

    work page 2019

  3. [3]

    Anumanchipalli, G. K., J. Chartier, E. F. Chang. Speech synthesis from neural decoding of spoken sentences. Nature, 568(7753):493, 2019

    work page 2019

  4. [4]

    Salari, E

    Ramsey, N., E. Salari, E. Aarnoutse, et al. Decoding spoken phonemes from sensorimotor cortex with high-density ecog grids. Neuroimage, 2017

    work page 2017

  5. [5]

    Zhao, S., F. Rudzicz. Classifying phonological categories in imagined and articulated speech. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 992–996. IEEE, 2015

    work page 2015

  6. [6]

    Yang, X., K. Wang, S. A. Shamma. Auditory representations of acoustic signals. Tech. rep., 1991

    work page 1991

  7. [7]

    Mesgarani, N., S. Shamma. Speech processing with a cortical representation of audio. In Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on , pages 5872–5875. IEEE, 2011

    work page 2011

  8. [8]

    Schmidhuber

    Hochreiter, S., J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997

    work page 1997

Show all 24 references
  1. [9]

    Pouget-Abadie, M

    Goodfellow, I., J. Pouget-Abadie, M. Mirza, et al. Generative adversarial nets. In Advances in neural information processing systems, pages 2672–2680. 2014

    2014

  2. [10]

    Chintala, L

    Arjovsky, M., S. Chintala, L. Bottou. Wasserstein generative adversarial networks. In Interna- tional Conference on Machine Learning , pages 214–223. 2017

    2017

  3. [11]

    Fernández, F

    Graves, A., S. Fernández, F. Gomez, et al. Connectionist temporal classification: labelling unseg- mented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369–376. ACM, 2006. 10

    2006

  4. [12]

    Graves, A., N. Jaitly. Towards end-to-end speech recognition with recurrent neural networks. In International Conference on Machine Learning , pages 1764–1772. 2014

    2014

  5. [13]

    Van Merriënboer, C

    Cho, K., B. Van Merriënboer, C. Gulcehre, et al. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014

    2014 arXiv

  6. [14]

    Chorowski, J. K., D. Bahdanau, D. Serdyuk, et al. Attention-based models for speech recognition. In Advances in neural information processing systems , pages 577–585. 2015

    2015

  7. [15]

    Bahdanau, D., K. Cho, Y . Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014

    2014 arXiv

  8. [16]

    Sequence transduction with recurrent neural networks

    Graves, A. Sequence transduction with recurrent neural networks. arXiv preprint arXiv:1211.3711, 2012

    2012 arXiv

  9. [17]

    Mohamed, G

    Graves, A., A.-r. Mohamed, G. Hinton. Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing , pages 6645–6649. IEEE, 2013

    2013

  10. [18]

    Gulcehre, K

    Chung, J., C. Gulcehre, K. Cho, et al. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014

    2014 arXiv

  11. [19]

    Kingma, D. P., J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  12. [20]

    Williams, R. J., D. Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270–280, 1989

    1989

  13. [21]

    Toutios, V

    Narayanan, S., A. Toutios, V . Ramanarayanan, et al. Real-time magnetic resonance imaging and electromagnetic articulography database for speech production research (tc). The Journal of the Acoustical Society of America , 136(3):1307–1311, 2014

    2014

  14. [22]

    Delorme, A., S. Makeig. Eeglab: an open source toolbox for analysis of single-trial eeg dynamics including independent component analysis. Journal of neuroscience methods , 134(1):9–21, 2004

    2004

  15. [23]

    Schölkopf, A

    Mika, S., B. Schölkopf, A. J. Smola, et al. Kernel pca and de-noising in feature spaces. In Advances in neural information processing systems , pages 536–542. 1999

    1999

  16. [24]

    Griffin, D., J. Lim. Signal estimation from modified short-time fourier transform. IEEE Transactions on Acoustics, Speech, and Signal Processing , 32(2):236–243, 1984. A Additional Figures and Tables Tables 12, 13 and 14 shows the test time results for RNN transducer model for d...

    1984

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.