Pith. sign in

REVIEW 4 major objections 4 minor 15 references

Is EEG-to-Text Feasible in Real-World Scenarios? An In-Depth Analysis Using a Neuropsychology-Inspired Benchmark

T0 review · 4 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Existing EEG-to-text results fail because benchmarks ignore EEG instability, not because brain signals lack language; a repeated-recall benchmark shows teacher-forcing-free decoding is feasible.

desk verdict A real dataset and a fair critique of ZuCo, but the feasibility claim is carried by an embedding-correlation metric and a closed-vocabulary design that can't bear the weight. read the letter →

arxiv 2607.18749 v1 pith:KYMWPUFY submitted 2026-07-21 cs.LG cs.CLcs.ETq-bio.NC

Zihan Zhang (1) , Yu Bao (1 , 2) , Xiao Ding (1) , Tianyi Jiang (3) , Kai Xiong (4) ((1) Research Center for Social Computing , Interactive Robotics , Harbin Institute of Technology
show 4 more authors
(2) Shanghai Innovation Institute (3) State Key Laboratory for Novel Software Technology Nanjing University (4) Zhongguancun Laboratory)
This is my paper
classification cs.LGcs.CLcs.ETq-bio.NC
keywords EEG-to-textbrain-computerinterfaceinnerspeechEEGinstabilityteacher-forcingevaluationembeddingalignmentbenchmarkdatasetscalinglaws
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper attempts to settle whether non-invasive EEG signals contain decodable linguistic information—a question with direct bearing on building speech prostheses that avoid brain surgery. It argues that the field's doubts come from benchmark design rather than biology: existing natural-reading benchmarks ignore EEG's instability, forcing models into teacher-forced evaluation where random noise scores as well as real brain data. To fix this, the authors build COFETT, an eight-day corpus in which the same participants repeatedly imagine the same sentences, and evaluate models by embedding alignment rather than token-level next-word prediction under gold conditioning. Their experiments show clear separation between models and from noise controls under teacher-forcing-free evaluation, and their scaling analysis projects that roughly 700 repetitions per sentence would bring EEG-to-text near machine-translation-level alignment. If correct, this makes EEG-to-text a viable direction and shifts the central issue from whether neural signal exists to how benchmarks measure it.

What carries the argument

The load-bearing construction is COFETT's repeated-recall paradigm: the same participant is asked to silently recall verbatim the same set of sentences on four or sixteen separate occasions across eight days, so that time-varying noise is implicitly averaged and the invariant signal associated with a sentence is exposed. Evaluation replaces BLEU and teacher-forcing with a vector-alignment protocol: EEG embeddings are trained against text embeddings and scored by Pearson correlation, with training and test sets drawn from different sessions of the same participant. Two label-free similarity measures (MINDFUL and CCA) and an LLM-style scaling-law analysis support the claim that the remaining s

What would settle it

Train a model on all COFETT training sessions for a participant, then test on imagined renditions of sentences that never appeared in training and were recorded on a different day; if the embedding-alignment score drops to the noise-input baseline, then the apparent linguistic decoding is actually session or participant identity rather than language content.

Watch

Extended reading notes

Core claim

The paper's central claim is that the enduring doubt about EEG-to-text is an artifact of benchmark design: natural-reading corpora such as ZuCo contain no within-participant repetitions, so models cannot overcome EEG instability and are effectively evaluated on the pretrained language model's priors. Replacing the EEG input with random noise yields statistically indistinguishable scores under teacher-forcing on those benchmarks. COFETT fixes this by recording the same participants silently recalling the same sentences verbatim across eight days—four or sixteen repetitions per sentence—and by replacing token-level teacher-forced scoring with Pearson correlation between EEG-derived embeddings

Load-bearing premise

The load-bearing premise, acknowledged in the paper's Section 4.3 similarity table, is that the small difference between repeated renditions of the same imagined sentence and renditions of different sentences—after accounting for participant and recording day—is a genuine linguistic signal that training can amplify; if that residual is mostly session noise, COFETT measures participant identity rather than language.

Editorial extensions

If this is right

  • Existing EEG-to-text results measured on natural-reading corpora under teacher-forcing should not be read as evidence of neural decoding; noise substitution produces statistically indistinguishable scores.
  • Teacher-forcing-free evaluation becomes possible when a benchmark contains within-participant repetitions and grades semantic embedding alignment rather than token matches.
  • Training and test data must come from the same semantic domain; adding sentences outside the test domain does not improve alignment.
  • Practical EEG-to-text at current signal quality would require roughly 40 times more paired data—on the order of 700 repetitions per sentence—making data collection the field's next bottleneck.
  • The COFETT protocol gives the field a way to distinguish models that genuinely use EEG from models that only exploit the pretrained language model's prior.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editor's inference: the paper's own similarity table shows that matching sentence identity adds only a small effect once participant and time are controlled; a decisive test is whether a model trained on COFETT holds its alignment on entirely unseen sentences from the same participant on a later day—if not, the benchmark may reward session-specific signatures rather than linguistic content.
  • Editor's inference: the ~700-repetition projection assumes the current embedding-alignment ceiling and roughly linear scaling; a plausible alternative path is to combine repeated-recall collection with models designed to factor out session drift, which could reduce the data multiplier.
  • Editor's inference: because only two participants completed the protocol, cross-participant value is unestablished; a useful extension is to test whether models pretrained on one participant's repeated sessions transfer to a second participant's single-session data.
  • Editor's inference: the vector-alignment metric measures semantic closeness in a fixed embedding space, not readable text; demonstrating usable communication would require a follow-on study showing that high-alignment models produce intelligible free-running sentences, a step the paper's own Limitations section leaves open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper argues that existing EEG-to-text benchmarks (e.g., ZuCo) neglect EEG instability, leading to teacher-forcing reliance and failure to distinguish real EEG from noise. It introduces COFETT, a benchmark with within-participant repeated inner-speech recordings across 8 days, and an evaluation protocol based on Pearson correlation between EEG and text embeddings. The authors report that on COFETT, embedding correlation separates EEG from noise controls, while token-level metrics do not, and that scaling-law analysis suggests ~700 repetitions per sentence are needed for machine-translation-like performance. The central conclusion is that EEG contains linguistically decodable information and that COFETT enables teacher-forcing-free evaluation.

Significance. If substantiated, the paper would make a valuable contribution by diagnosing a real limitation of current EEG2Text benchmarks and by releasing a dataset designed to counter EEG instability. The explicit noise controls (random EEG at train and test) and the decision to release the corpus are strengths. However, the feasibility claim is not supported by the paper's own metrics: the only separating metric is the training objective, the sentence-identity effect is negligible, and the scaling-law analysis is closed-vocabulary. The work is better viewed as a dataset/benchmark proposal with promising controls, but its central scientific claim requires substantial additional evidence.

major comments (4)
  1. [Section 3.2 / Table 2] The only metric that separates EEG from noise on COFETT is Emb., which is the same Pearson-correlation objective used for training (Fig. 4). Token-level metrics without teacher forcing remain at noise level (e.g., EEGPT BLEU-1 16.23 vs noise 16.04; T5 15.70 vs 15.50). Reporting the training loss as the evaluation metric is circular; the separation could reflect memorization of stable non-linguistic traces (length, timing) rather than linguistic decoding. An independent metric (e.g., held-out sentence classification or perplexity on generated text) is needed.
  2. [Section 4.3 / Table 3] The paper's own CCA/MINDFUL analysis shows that after conditioning on participant and time, sentence identity contributes almost nothing: same-day same-text CCA 21.74 vs same-day different-text 21.48; cross-day 18.27 vs 17.70. This undercuts the premise that COFETT isolates sentence-specific linguistic signal. The conclusion in Section 4.3 that 'EEG contains linguistically decodable structure... decoding is feasible' is a leap from a weak effect that is dominated by participant/time identity. The embedding separation in Table 2 could be driven by these non-linguistic factors.
  3. [Section 4.4] The scaling-law experiment is closed-vocabulary: D_test_2 contains repetitions 15 and 16 of every sentence in M, while D_train_2 contains j repetitions of a subset M'. Whenever |M'| approaches or exceeds the size of M, the model can memorize sentence-specific traces (e.g., recall-phase duration t = 0.4(x+1) s, trial position) without learning language. The 'in-/out-of-domain' explanation for the flattening at M' > M is an untested assumption; no held-out generalization to new sentences is reported. The extrapolation to ~700 repetitions is a self-fit without reported parameters, confidence intervals, or alternative models; it is not a falsifiable prediction.
  4. [Appendix A] The dataset contains only two participants (S1, S2) after screening. With n=2, the benchmark's reliability, the across-method ranking, and the scaling-law predictions cannot be generalized. The paper should report within-participant stability and at least discuss the population-level validity of a two-participant benchmark.
minor comments (4)
  1. [Table 2 caption] The labels 'w/o rep' and 'w/ rep' are confusing; specify that 'w/ rep' is the full COFETT protocol and 'w/o rep' is the ablation with unique sentences.
  2. [Figure 5] Figure 5 is referenced but not included in the main text; if it is in the supplement, say so, and add axes labels, units, and uncertainty estimates/error bars.
  3. [Section 3.2] The 'Emb.' metric is not formally defined in the main text. State how EEG embeddings are extracted, which text embedding model is used, and whether the same encoder is used across all baselines.
  4. [Section 4.3] 'Monte Carlo sampling' is mentioned but no details are given on the number of pairs, seeds, or statistical significance tests; provide confidence intervals for the CCA/MINDFUL means.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central feasibility claim is supported by independent noise controls and held-out embedding alignment; self-citations are not load-bearing.

full rationale

The paper's claimed derivation—that COFETT enables teacher-forcing-free evaluation and that EEG contains decodable linguistic structure—does not reduce by construction to its inputs. The evaluation protocol trains a model to output text embeddings and measures held-out Pearson correlation with LaBSE embeddings; test sentences/repetitions are not the same samples used for fitting, and a random-noise control at both train and test collapses the embedding correlation (Table 1), providing an independent baseline that separates EEG-driven performance from language-prior artifacts. The statistical evidence in Table 3 is an empirical measurement of a small sentence-identity effect, not a definitional restatement of the conclusion; the leap from 'small effect' to 'decoding feasible' is an inferential overreach about effect size, not circularity. The scaling-law extrapolation to ~700 repetitions is a projection from the authors' own fitted trend, but it is not used as an input to the feasibility conclusion and no fitted parameter is renamed as a prediction. Self-citations (Chisco, Zhang et al. 2024) supply stimulus materials and a paradigm cue, but the same design is also anchored to external references (Proix et al., Nieto et al., Tian & Poeppel), so the self-citation is not load-bearing. The closed-vocabulary design (D_train_2 ⊂ M, D_test_2 = repetitions 15–16 of all M) raises a possible confound of length/memorization, but that is a validity limitation of the benchmark, not an equation-level circularity. No step in the paper exhibits a specific reduction of a claimed prediction to a fitted value or a definitional identity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No new physical or theoretical entities are introduced; COFETT is a new dataset, which is not a postulated entity. The central parameters are design choices (number of repetitions, sentence set size, 4-character recall-error threshold), but the only fitted quantity used in the argument is the unreported scaling-law slope.

free parameters (1)
  • scaling-law slope (repetitions to alignment) = not reported
    Section 4.5 extrapolates to ~700 repetitions from a 'linear trend inferred above'; slope and intercept are not given, so the projection is an unreported fit.
assumptions (4)
  • domain assumption The word-by-word recall phase produces inner speech that encodes the sentence's linguistic content
    Section 3.1: 'The word-by-word recall phase... represents the Inner Speech.' If recall EEG reflects generic memory/attention rather than sentence content, the benchmark does not isolate language.
  • domain assumption Pearson correlation between a trained EEG encoder's output and LaBSE text embeddings is a valid proxy for EEG-to-text decoding quality
    Section 3.2 replaces BLEU with embedding correlation; the feasibility conclusion depends on this proxy. The paper's own Table 2 shows token-level BLEU does not separate EEG from noise on COFETT.
  • domain assumption Two participants selected by lowest ANT scores after meditation training are sufficient to establish feasibility
    Appendix A selects the 2 best of 7 volunteers after IBMT training; any conclusion about general EEG2Text feasibility is conditioned on this unrepresentative sample.
  • ad hoc to paper Failure to transfer to sentences outside the training sentence set is due to semantic-domain mismatch rather than closed-vocabulary memorization
    Section 4.4 attributes the plateau when M' > M to in-/out-of-domain effects, but the test set consists of held-out repetitions of the same sentences, so the result is equally explained by closed-set sentence identification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Is EEG-to-Text Feasible in Real-World Scenarios? An In-Depth Analysis Using a Neuropsychology-Inspired Benchmark." pith.science (2026). https://pith.science/paper/KYMWPUFY

@misc{pith2026260718749,
  author       = {Pith},
  title        = {Pith review of: Is EEG-to-Text Feasible in Real-World Scenarios? An In-Depth Analysis Using a Neuropsychology-Inspired Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KYMWPUFY}},
  note         = {Machine review of arXiv:2607.18749}
}
read the original abstract

Translating brain signals into text could restore communication for people with severe paralysis, yet practically usable systems to date rely on invasive electrocorticography (ECoG). Electroencephalography (EEG) offers a non-invasive alternative, and EEG-to-text (EEG2Text) has been widely explored. Interestingly, however, EEG2Text models generally rely on teacher-forcing evaluation; without it, they fail to generate meaningful decoding. This reliance prevents EEG2Text from being applied in real-world, non-academic settings. This has fueled numerous debates about whether EEG2Text is a meaningful direction, by extension, and whether EEG truly contains decodable linguistic information. Here, using a neuropsychology-informed paradigm, we find that existing EEG2Text benchmarks have neglected EEG instability, a flaw that has confounded inference and sparked debate. Our experiments furnish key evidence for the feasibility of teacher-forcing-free EEG2Text decoding. Accordingly, we assemble the Corpus OF Eeg-To-Text (COFETT) using a 128-channel high-density EEG cap, providing a benchmark dedicated to evaluating EEG2Text models. In comparisons with multiple existing benchmarks, COFETT achieves SOTA ability to distinguish among model performances and enables robust, teacher-forcing-free evaluation, thereby opening a path toward practical EEG2Text applications. COFETT is open sourced in https://github.com/baoyudu/COFETT.

Figures

Figures reproduced from arXiv: 2607.18749 by the authors.

Figure 1
Figure 1. EEG instability in EEG2Text. The EEG elicited by imagining the same sentence on day 1 ver￾sus day 2 exhibits different components and features. rectly decode Inner Speech, enabling near-real￾time transformation of cortical signals into verbal output with high accuracy (Willett et al., 2023; Card et al., 2024). However, the requirement for craniotomy and implanted electrodes restricts use to patients who cannot under… view at source ↗
Figure 2
Figure 2. The full experimental procedure for each participant ( [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Comparison of supervision methods for EEG2Text models. spired by previous psycholinguistic studies on imagined speech, where reading guides the imag￾ination process. The imagined speech is consid￾ered to be embedded within the participant’s In￾ner Speech (Zhang et al., 2024; Nieto et al., 2022). Moreover, the reading phase activates brain re￾gions associated with language processing, such as the left temporal lobe a… view at source ↗
Figures from the paper (5 more)
Figure 3
Figure 3. Figure 3: The EEG dataset obtained from the experi [PITH_FULL_IMAGE:figures/full_fig_p004_3.png]
Figure 5
Figure 5. Figure 5: Alignment performance at different data scales. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Embedding distances of the same sentence [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Main experimental setup. I EEG Data Preprocessing Pipeline Raw EEG data were initially recorded at a sam￾pling rate of 1,000 Hz and stored in the original .edf format. Subsequently, the data were downsam￾pled to 500 Hz to facilitate processing. The PREP algorithm was t…
Figure 8
Figure 8. Figure 8: EEG preprocessing pipeline (part (f) cites previous work( [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 2 linked inside Pith

  1. [1]

    Removing line noise without committing to a specific filtering strategy

  2. [2]

    Robust referencing of the signal relative to an estimate of the “true” average reference

  3. [3]

    Detection and interpolation of bad channels relative to this reference

  4. [4]

    Filtering: Inthisstep, weapplieda50Hznotch filter to remove power-line noise and a zero-phase high-pass finite impulse response (FIR) filter with a1Hzcutoff

    Retaining sufficient information to allow users to re-reference or undo interpolation of specific channels. Filtering: Inthisstep, weapplieda50Hznotch filter to remove power-line noise and a zero-phase high-pass finite impulse response (FIR) filter with a1Hzcutoff. Nolow-passfilteringwasperformed, as explained in the ‘Resampling’ section. Autoreject(Jas e...

    2017

  5. [5]

    Today’s dinner tasted great

    has been provided as follows. To ensure the dataset is suitable for training BCIs for everyday use, the textual materials were designed to encompass a broad range of daily lan- guage. To achieve this, Chisco manually selected expressions from the Chinese social media plat- form Weibo, as well as public datasets ROCstory (Mostafazadeh et al., 2016) and Dai...

    2016

  6. [6]

    If the standard deviation of any component is zero (indicating a constant), set the corre- lation coefficient to zero

    Compute Pearson Correlation Coefficients Description: For each of the first three pro- jected components, calculate the Pearson cor- relation coefficient betweenC1full and C2full. If the standard deviation of any component is zero (indicating a constant), set the corre- lation coefficient to zero. Mathematical Representation: For each component j = 1, 2, 3...

    2017

  7. [10]

    MathematicalRepresentation: Let T bethe number of time steps or samples

    Covariance Matrix Computation Description: Compute the covariance matri- ces of the input data matricesX1 and X2. MathematicalRepresentation: Let T bethe number of time steps or samples. The covari- ance matrices are calculated as: R11 = 1 T X1 · X1T R12 = 1 T X1 · X2T R22 = 1 T X2 · X2T

  8. [11]

    Mathematical Representation: M = (R11 + R22)−1(R12 + RT 12) Solve the eigenvalue equation: Mvi = λivi where λi are the eigenvalues andvi are the corresponding eigenvectors

    Eigenvalue Problem Solving Description: Constructmatrix M andfindits eigenvalues and eigenvectors. Mathematical Representation: M = (R11 + R22)−1(R12 + RT 12) Solve the eigenvalue equation: Mvi = λivi where λi are the eigenvalues andvi are the corresponding eigenvectors

Show all 15 references
  1. [12]

    Mathematical Representation: Let λ1 ≥ λ2 ≥

    Extract and Sort Eigenvalues and Eigen- vectors Description: Sorttheeigenvaluesindescend- ing order and reorder the eigenvectorsaccord- ingly to prioritize the most significant compo- nents. Mathematical Representation: Let λ1 ≥ λ2 ≥ . . . ≥ λK be the sorted eigenval- ues, and...

  2. [13]

    Mathematical Representation: W = [v1,v2,v3]

    Extract Principal Components Description: Select the top three eigenvec- tors corresponding to the largest eigenvalues to form the projection matrixW. Mathematical Representation: W = [v1,v2,v3]

  3. [14]

    Mathematical Representation: C1full = W T X1 C2full = W T X2 Here, C1fulland C2fullrepresenttheprojected data in the new feature space

    Projection into New Feature Space Description: Project the original data matri- ces X1and X2into the new feature space de- fined by the projection matrixW. Mathematical Representation: C1full = W T X1 C2full = W T X2 Here, C1fulland C2fullrepresenttheprojected data in the new ...

  4. [2006]

    Progress in Brain Research, 159:275–295

    High-frequency gamma oscillations and human brain mapping with electrocorticography . Progress in Brain Research, 159:275–295. Jacek P Dmochowski, Paul Sajda, Joao Dias, and Lu- cas C Parra. 2012. Correlated components of ongo- ing eeg point to emotionally laden attention–a po...

    2012

  5. [2014]

    neuroimage, 86:446–460

    Mnesoftwareforprocessingmegandeegdata. neuroimage, 86:446–460. Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, and 1 others. 2020. Scaling laws for autoregressive gen- erative modeling. ...

    2020 arXiv

  6. [2018]

    Nora Hollenstein, Marius Troendle, Ce Zhang, and Nicolas Langer

    Zuco, a simultaneous eeg and eye-tracking re- source for natural sentence reading.Scientific data, 5(1):1–13. Nora Hollenstein, Marius Troendle, Ce Zhang, and Nicolas Langer. 2019. Zuco 2.0: A dataset of physi- ological recordings during natural reading and anno- tation. arXiv...

    2019 arXiv

  7. [2024]

    Communications Biol- ogy, 7(1):1363

    Measuring instability in chronic human intra- cortical neural recordings towards stable, long-term brain-computer interfaces. Communications Biol- ogy, 7(1):1363. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Pe...

    2020

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.