Pith. sign in

REVIEW 5 major objections 6 minor 26 references

ETS: Open Vocabulary Electroencephalography-To-Text Decoding and Sentiment Classification

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EEG plus eye-tracking, fed through a CNN-Transformer encoder and a pretrained BART decoder, decodes open-vocabulary sentences at BLEU-1 43.39 and zero-shot sentiment at 68.18 F1, surpassing the prior BART baseline.

desk verdict Incremental but useful extension of Wang & Ji; headline SOTA claims don't hold up because the baseline wasn't re-run on the same data. read the letter →

arxiv 2506.14783 v1 pith:POHGWX4C submitted 2025-05-26 cs.LG cs.CLcs.HC

Mohamed Masry , Mohamed Amen , Mohamed Elzyat , Mohamed Hamed , Norhan Magdy , Maram Khaled This is my paper · ORCID
classification cs.LGcs.CLcs.HC
keywords EEG-to-textdecodingopenvocabularysentimentclassificationeye-trackingzero-shotlearningZuCodatasetBARTlanguagemodelbrain-computerinterface
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces ETS, a framework that turns non-invasive EEG recorded during natural reading, together with synchronized eye-tracking, into open-vocabulary text and sentence-level sentiment labels. The central claim is that a per-word multimodal encoder followed by a pretrained BART decoder generates unseen sentences with BLEU-1 43.39, BLEU-2 30.36, BLEU-3 23.67, BLEU-4 20.22, and ROUGE-1 F1 36.66, all above the prior BART-based baseline. The same decoded text, passed through a pretrained text sentiment classifier, reaches 68.18 F1 on ternary sentiment without any direct EEG-sentiment training, outperforming direct EEG-to-sentiment models. If these numbers hold under matched evaluation, they would mark a step from small closed-vocabulary EEG classification toward practical open-vocabulary brain-to-text interfaces.

What carries the argument

The load-bearing component is the per-word multimodal tensor built from 8 spectral power bands across 105 time points plus 3 eye-tracking fixation features (first fixation duration, total reading time, gaze duration), which lets the CNN-Transformer encoder align noisy EEG to discrete tokens. The decoder is a pretrained BART language model that maps the encoded sequence to open-vocabulary text, and the zero-shot sentiment arm reuses the decoded text with a separate text-trained BART sentiment classifier, so no EEG-sentiment pairs are needed.

What would settle it

Re-run the reference [1] baseline on ETS's exact 80/10/10 sentence-level split and preprocessing pipeline, then recompute BLEU-1 through BLEU-4 and ROUGE-1; if the baseline's numbers move substantially, the claimed margins (e.g., +13.42 BLEU-4) are evaluation artifacts rather than decoding improvements.

Watch

Extended reading notes

Core claim

ETS is an end-to-end sequence-to-sequence model: each word's EEG segment is bandpass-filtered into theta, alpha, beta, and gamma bands and combined with three eye-tracking fixation features into a tensor; a CNN extracts spatial-frequency patterns, adapter modules project the features into a Transformer encoder, and a pretrained BART decoder generates the sentence. On the full ZuCo suite spanning 30 participants and four reading tasks, the paper reports BLEU-1 43.39, BLEU-2 30.36, BLEU-3 23.67, BLEU-4 20.22, and ROUGE-1 F1 36.66, exceeding the reference [1] baseline by margins that grow with n-gram order. A T5 decoder variant reaches BLEU-1 48.15 but loses on higher-order n-grams. For sentiment, the paper's two-stage pipeline first decodes EEG to text, then applies a pretrained BART sentiment classifier, reaching 68.18 F1 on the SR v1.0 ternary test set, far above the direct EEG-to-sentiment classifier; text-only classifiers on ground-truth sentences remain the upper bound.

Load-bearing premise

The reported gains over the baseline assume the baseline's published numbers were produced on the same data split and preprocessing as ETS, since the baseline was not re-run under the ETS protocol.

Editorial extensions

If this is right

  • If the reported numbers hold, open-vocabulary EEG-to-text decoding becomes feasible with non-invasive recording during reading, at least for the ZuCo task distribution.
  • The growing relative gain with n-gram order (from +8.2% BLEU-1 to +197.4% BLEU-4) suggests the model captures multi-word structure, not just word identity.
  • The zero-shot sentiment result implies intermediate text is a stronger carrier of affect than raw EEG features, so progress in decoding quality should directly improve downstream semantic tasks.
  • The T5-vs-BART comparison indicates the choice of pretrained decoder changes the error profile: T5 improves early-token alignment while BART holds the advantage on longer, structured sentences.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A matched re-run of the reference [1] baseline under ETS's exact split and preprocessing could shrink or erase the reported gains, because the baseline was not re-run under the same protocol.
  • The reliance on eye-tracking fixations to segment words ties the method to reading paradigms; extending to listening or imagined speech would require a different alignment signal, so the open-vocabulary claim is scoped to read text.
  • One could test whether the sentiment gain is driven by decoding fidelity or by the strength of the text classifier: feeding partially corrupted or length-controlled decoded sentences to the same BART sentiment model would quantify how much sentiment accuracy depends on exact word choices.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes ETS, a framework for open-vocabulary EEG-to-text decoding and zero-shot sentiment classification. It combines word-level EEG spectral features with eye-tracking fixation features, encodes them with a CNN-Transformer dual-stream encoder, and decodes with a pretrained BART or T5 model. The authors evaluate on the ZuCo v1.0/v2.0 datasets, reporting BLEU-1 of 43.39 and ROUGE-1 F1 of 36.66, which they present as surpassing the Wang & Ji baseline, and a zero-shot EEG-to-text-to-sentiment pipeline that reaches 68.18 F1 on ternary sentiment classification. The central claim is that integrating eye-tracking with pretrained language models improves both decoding and sentiment analysis from non-invasive EEG.

Significance. If the reported gains are reproducible, the work would be a useful empirical advance: it incorporates additional eye-tracking features (first fixation duration, total reading time) beyond the gaze duration used in prior work, demonstrates that a single generative pipeline can serve both text decoding and zero-shot sentiment classification, and provides concrete generation examples alongside both BART and T5 decoders. These strengths, however, are not yet sufficient because the headline comparison to the Wang & Ji baseline is not controlled, all results are point estimates without variance or significance tests, and there are inconsistencies between the claimed and reported improvements.

major comments (5)
  1. [§4.2, Table 2] The headline claim that ETS surpasses the Wang & Ji baseline is not supported because the baseline numbers are taken from Ref. [1] rather than re-run under the ETS protocol. Section 4.1/Table 1 shows ETS is trained on ZuCo v1.0 + v2.0 (30 participants, four reading tasks) with an 80/10/10 sentence-level split, while the Wang & Ji baseline was evaluated in a different experimental setting (single version, likely different subjects and preprocessing). The reported BLEU/ROUGE deltas (+3.29 BLEU-1, +14.66 ROUGE-1 F1) could therefore reflect added data, additional eye-tracking features, or different metric computation rather than the proposed architecture. To make the central claim credible, the authors must re-run the baseline under the exact same split, preprocessing, and evaluation code, or clearly justify comparability.
  2. [§1 and §4.2] The abstract and introduction claim '+15% BLEU-1 / ROUGE-1' over prior baselines, but Table 2 reports a BLEU-1 improvement of +3.29 absolute (+8.2% relative) and ROUGE-1 F1 improvement of +6.56 absolute (+29.8% relative). The claimed +15% does not match either of these numbers, and it is unclear whether it refers to a different metric or a different subset. This inconsistency suggests the headline numbers were not carefully cross-checked, and it should be corrected with a precise statement of which metrics and which relative/absolute improvements are reported.
  3. [§4.2, Tables 2 and 3] All results are reported as single point estimates without variance, confidence intervals, or significance tests. EEG-to-text performance is known to vary strongly across subjects, sessions, and random seeds, and the differences between ETS and the baseline (e.g., +3.29 BLEU-1) may be within noise. The authors should report mean and standard deviation over at least five random seeds, and ideally a paired bootstrap or permutation test for the headline comparison. This is necessary to establish that the architecture, rather than chance, explains the reported gains.
  4. [§4.3, Table 5] The zero-shot sentiment claim is also not rigorously evaluated. The direct EEG-to-sentiment baselines ('EEG Encoder+BERT/RoBERTa') appear to be trained only on SR v1.0, while the ETS decoder used in the PS rows was trained on all four task subsets, giving it additional data. Moreover, no variance or significance testing is reported for any of the F1 values in Table 5, and some numbers (e.g., ETS+Roberta F1 36.30 vs. EEG Encoder+Roberta 37.14) are very close. The paper should report controlled comparisons with matched training data and appropriate error bars.
  5. [§4.2, Table 3] The model selection between BART and T5 appears to be based on the test set: the text states T5 outperforms BART on BLEU-1 and BLEU-2, while BART is chosen for the main result because it is 'competitive, particularly for longer and more structured sentence generation' (as shown in Table 3). If the decoder choice is made after inspecting test metrics, the reported results are optimistically biased. The authors should specify a validation-based selection criterion or report both models with confidence intervals.
minor comments (6)
  1. [§3.1] Equation (1) contains a notation error: 'p(st ∈ v ∖ E, s<t)' should be 'p(s_t ∈ V | E, s_<t)', and the product and loss function should use consistent conditional probability notation.
  2. [Abstract] The sentence 'Previous studies have achieved high accuracy on small-closed vocabularies, but it still struggles on open vocabularies' has a subject-verb agreement error ('it' should be 'they').
  3. [Table 5] The abbreviations C, PS, and RS in the 'Test on' column are defined only in the caption; please also introduce them in the main text before the table is referenced.
  4. [References] References [8] and [9] lack publication years and venues; the BLEU and ROUGE citations should be completed for the camera-ready version.
  5. [Figure 1] Figure 1 is referenced in the text but the figure content is not rendered in the manuscript; ensure the figure is included with proper subfigure labels (a) and (b).
  6. [Table 4] The generation examples in Table 4 show hallucinated numbers and ungrammatical output; a short discussion of these failure modes would strengthen the limitations analysis.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ETS is an empirical pipeline whose outputs are evaluated against external benchmarks; no fitted constant is relabeled as a prediction.

full rationale

I walked the derivation chain. ETS is trained end-to-end on paired EEG-text data (Section 3.1) with a standard cross-entropy objective; BLEU/ROUGE scores in Table 2 are computed on an unseen 10% sentence-level split, and the Wang & Ji baseline is an external published result, not a fitted parameter of ETS. The zero-shot sentiment pipeline (Section 3.2) first decodes EEG into text with the trained decoder and then applies an externally pretrained text sentiment classifier (Socher et al., ref. [11]); no direct EEG-sentiment labels are used in training, so the pipeline is not defined in terms of the labels it predicts. The closest weakness is experimental rather than circular: the Wang & Ji baseline is not re-run on ETS's broader ZuCo v1.0+v2.0 preprocessing and split, and the choice between BART and T5 seems test-informed. These are validity threats, not reductions of the claim to its inputs. No equation or module makes the target metric an input by construction, and the paper does not rely on self-citations or imported uniqueness theorems. Hence no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No explicit scalar constants are fitted by hand in the reported formulas; all learned parameters are standard neural network weights. The main hand-chosen inputs are frequency bands and eye-tracking metrics, which are standard design choices rather than tuned values reported in the paper. The central assumptions are about data comparability and the validity of the baseline comparison.

assumptions (4)
  • domain assumption ZuCo EEG and eye-tracking signals are synchronized and word-level aligned as provided.
    The method builds word-level EEG tensors from eye-tracking fixations; if synchronization or word boundaries are wrong, the input features are invalid. This enters in Section 3.1.
  • ad hoc to paper The Wang and Ji baseline numbers in Table 2 were computed under comparable preprocessing and test split.
    The paper does not re-run the baseline; it borrows numbers from the cited paper while training ETS on a larger multi-version dataset. If the splits differ, the claimed gains are invalid. This enters in Section 4.2 and Table 2.
  • domain assumption The pretrained text sentiment classifier used in the zero-shot pipeline generalizes to noisy decoded text.
    The pipeline assumes a classifier trained on clean text gives meaningful labels for imperfect EEG-decoded sentences; no calibration or robustness analysis is reported. This enters in Section 3.2 and Section 4.3.
  • domain assumption Test sentences are entirely unseen and no sentence appears in both training and test folds across subjects.
    The dataset was split at sentence level with an 80/10/10 split, but cross-subject duplicates are not analyzed. If the same sentence appears in train and test from different subjects, the test set is contaminated. This enters in Section 4.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ETS: Open Vocabulary Electroencephalography-To-Text Decoding and Sentiment Classification." pith.science (2026). https://pith.science/paper/POHGWX4C

@misc{pith2026250614783,
  author       = {Pith},
  title        = {Pith review of: ETS: Open Vocabulary Electroencephalography-To-Text Decoding and Sentiment Classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/POHGWX4C}},
  note         = {Machine review of arXiv:2506.14783}
}
read the original abstract

Decoding natural language from brain activity using non-invasive electroencephalography (EEG) remains a significant challenge in neuroscience and machine learning, particularly for open-vocabulary scenarios where traditional methods struggle with noise and variability. Previous studies have achieved high accuracy on small-closed vocabularies, but it still struggles on open vocabularies. In this study, we propose ETS, a framework that integrates EEG with synchronized eye-tracking data to address two critical tasks: (1) open-vocabulary text generation and (2) sentiment classification of perceived language. Our model achieves a superior performance on BLEU and Rouge score for EEG-To-Text decoding and up to 10% F1 score on EEG-based ternary sentiment classification, which significantly outperforms supervised baselines. Furthermore, we show that our proposed model can handle data from various subjects and sources, showing great potential for high performance open vocabulary eeg-to-text system.

Figures

Figures reproduced from arXiv: 2506.14783 by the authors.

Figure 1
Figure 1. B shows the main architecture, by first decoding EEG into text, the system leverages the representational power of large language models for sentiment analysis without ever seeing labeled EEG–sentiment pairs. This modular strategy was first demonstrated in the open￾vocabulary EEG-to-text and zero-shot sentiment work of Wang & Ji, who showed that intermediate text yields far better sentiment predictions than direct E… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 18 canonical work pages

  1. [1]

    Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classification

    Z. Wang and H. Ji, “Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classification, ” Dec. 2021, [Online]. Available: http://arxiv.org/abs/2112.02690

    work page Pith review arXiv 2021

  2. [2]

    DeWave: Discrete EEG Waves Encoding for Brain Dynamics to Text Translation,

    Y . Duan, J. Zhou, Z. Wang, Y .-K. Wang, and C.-T. Lin, “DeWave: Discrete EEG Waves Encoding for Brain Dynamics to Text Translation, ” Sep. 2023, [Online]. Available: http://arxiv.org/abs/2309.14030

    arXiv 2023

  3. [3]

    BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension,

    M. Lewis et al., “BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension, ” Oct. 2019, [Online]. Available: http://arxiv.org/abs/1910.13461

    arXiv 2019

  4. [4]

    High-performance brain-to-text communication via handwriting,

    F . R. Willett, D. T. Avansino, L. R. Hochberg, J. M. Henderson, and K. V . Shenoy, “High-performance brain-to-text communication via handwriting, ” Nature, vol. 593, no. 7858, pp. 249–254, May 2021, doi: 10.1038/s41586-021-03506-2

    doi:10.1038/s41586-021-03506-2 2021

  5. [5]

    Data descriptor: ZuCo, a simultaneous EEG and eye-tracking resource for natural sentence reading,

    N. Hollenstein, J. Rotsztejn, M. Troendle, A. Pedroni, C. Zhang, and N. Langer, “Data descriptor: ZuCo, a simultaneous EEG and eye-tracking resource for natural sentence reading, ” Sci Data, vol. 5, 2018, doi: 10.1038/sdata.2018.291

    work page doi:10.1038/sdata.2018.291 2018

  6. [6]

    Attention Is All You Need

    A. Vaswani et al., “Attention Is All You Need. ”

    work page

  7. [7]

    ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation,

    N. Hollenstein, M. Troendle, C. Zhang, and N. Langer, “ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation, ” 2020. [Online]. Available: https://osf.io/q3zws/

    work page 2020

  8. [8]

    BLEU: a Method for Automatic Evaluation of Machine Translation

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: a Method for Automatic Evaluation of Machine Translation. ”

    work page

Show all 26 references
  1. [9]

    ROUGE: A Package for Automatic Evaluation of Summaries

    C.-Y . Lin, “ROUGE: A Package for Automatic Evaluation of Summaries. ”

  2. [10]

    Prediction of advertisement preference by fusing EEG response and sentiment analysis,

    H. Gauba, P . Kumar, P . P . Roy, P . Singh, D. P . Dogra, and B. Raman, “Prediction of advertisement preference by fusing EEG response and sentiment analysis, ” Neural Networks, vol. 92, pp. 77–88, Aug. 2017, doi: 10.1016/j.neunet.2017.01.013

    2017 doi

  3. [11]

    Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank,

    R. Socher et al., “Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank, ” Association for Computational Linguistics. [Online]. Available: http://nlp.stanford.edu/

  4. [12]

    Neuroprosthesis for Decoding Speech in a Paralyzed Person with Anarthria,

    D. A. Moses et al., “Neuroprosthesis for Decoding Speech in a Paralyzed Person with Anarthria, ” New England Journal of Medicine, vol. 385, no. 3, pp. 217–227, Jul. 2021, doi: 10.1056/nejmoa2027540

    2021 doi

  5. [13]

    Editorial: Brain-Computer Interfaces and Augmented/Virtual Reality,

    F . Putze et al., “Editorial: Brain-Computer Interfaces and Augmented/Virtual Reality, ” May 12, 2020, Frontiers Media S.A. doi: 10.3389/fnhum.2020.00144

    2020

  6. [14]

    Multi-scale order recurrence quantification analysis of EEG signals evoked by manual acupuncture in healthy subjects,

    G. Yi et al., “Multi-scale order recurrence quantification analysis of EEG signals evoked by manual acupuncture in healthy subjects, ” Cogn Neurodyn, vol. 7, no. 1, pp. 79–88, 2013, doi: 10.1007/s11571- 012-9221-1

    2013 doi

  7. [15]

    EEGNet: A Compact Convolutional Network for EEG-based Brain-Computer Interfaces,

    V . J. Lawhern, A. J. Solon, N. R. Waytowich, S. M. Gordon, C. P . Hung, and B. J. Lance, “EEGNet: A Compact Convolutional Network for EEG-based Brain-Computer Interfaces, ” Nov. 2016, doi: 10.1088/1741-2552/aace8c

    2016 doi

  8. [16]

    Parameter-Efficient Transfer Learning for NLP ,

    N. Houlsby et al., “Parameter-Efficient Transfer Learning for NLP , ” Feb. 2019, [Online]. Available: http://arxiv.org/abs/1902.00751

    2019 arXiv

  9. [17]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    J. Devlin, M.-W. Chang, K. Lee, K. T. Google, and A. I. Language, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. ” [Online]. Available: https://github.com/tensorflow/tensor2tensor

  10. [18]

    Spatial and temporal resolutions of EEG: Is it really black and white? A scalp current density view,

    B. Burle, L. Spieser, C. Roger, L. Casini, T. Hasbroucq, and F . Vidal, “Spatial and temporal resolutions of EEG: Is it really black and white? A scalp current density view, ” International Journal of Psychophysiology, vol. 97, no. 3, pp. 210–220, Sep. 2015, doi: 10.1016/j.ijp...

    2015 doi

  11. [19]

    Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,

    C. Raffel et al., “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, ”

  12. [20]

    RoBERTa: A Robustly Optimized BERT Pretraining Approach,

    Y . Liu et al., “RoBERTa: A Robustly Optimized BERT Pretraining Approach, ” 2019. [Online]. Available: https://github.com/pytorch/fairseq

    2019

  13. [21]

    Target speaker detection with concealed EEG around the ear,

    B. Mirkovic, M. G. Bleichner, M. De Vos, and S. Debener, “Target speaker detection with concealed EEG around the ear, ” Front Neurosci, vol. 10, no. JUL, 2016, doi: 10.3389/fnins.2016.00349

    2016

  14. [22]

    Subject identification from electroencephalogram (EEG) signals during imagined speech,

    K. Brigham and B. V . K. V . Kumar, “Subject identification from electroencephalogram (EEG) signals during imagined speech, ” in 2010 Fourth IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS), IEEE, 2010, pp. 1–8

    2010

  15. [23]

    Reading Task Classification Using EEG and Eye-Tracking Data,

    N. Hollenstein et al., “Reading Task Classification Using EEG and Eye-Tracking Data, ” Dec. 2021, [Online]. Available: http://arxiv.org/abs/2112.06310

    2021 arXiv

  16. [24]

    DEAP: A Database for Emotion Analysis using Physiological Signals

    S. Koelstra et al., “DEAP: A Database for Emotion Analysis using Physiological Signals. ” [Online]. Available: http://www.eecs.qmul.ac.uk/mmv/datasets/deap/

  17. [25]

    Thinking out loud, an open- access EEG-based BCI dataset for inner speech recognition,

    N. Nieto, V . Peterson, H. L. Rufiner, J. E. Kamienkowski, and R. Spies, “Thinking out loud, an open- access EEG-based BCI dataset for inner speech recognition, ” Sci Data, vol. 9, no. 1, Dec. 2022, doi: 10.1038/s41597-022-01147-2

    2022 doi

  18. [2020]

    Available: http://jmlr.org/papers/v21/20-074.html

    [Online]. Available: http://jmlr.org/papers/v21/20-074.html

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.