Neuro2Semantic: A Transfer Learning Framework for Semantic Reconstruction of Continuous Language from Human Intracranial EEG

Ashesh Mehta; Gavin Mischler; Nima Mesgarani; Richard Antonello; Siavash Shams; Stephan Bickel

REVIEW 4 major objections 5 minor 39 references

Neuro2Semantic: A Transfer Learning Framework for Semantic Reconstruction of Continuous Language from Human Intracranial EEG

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Neuro2Semantic claims that perceived speech semantics can be decoded from intracranial EEG by aligning neural signals with pre-trained text embeddings and then inverting those embeddings into natural language, with strong results from…

desk verdict A plausible iEEG-to-text framework with a novel two-phase design, undercut by a headline 'outperforms' claim that rests on sentence-level t-tests ignoring subject/story clustering. read the letter →

arxiv 2506.00381 v1 pith:GE7DBMLW submitted 2025-05-31 cs.CL eess.ASeess.SP

Siavash Shams , Richard Antonello , Gavin Mischler , Stephan Bickel , Ashesh Mehta , Nima Mesgarani This is my paper

classification cs.CLeess.ASeess.SP

keywords braindecodingsemanticreconstructionintracranialEEGtransferlearningtextembeddingalignmentVec2Textnaturallanguagegeneration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Neuro2Semantic sets out to show that continuous, open-vocabulary text can be reconstructed from intracranial EEG (iEEG) recordings of perceived speech by reusing pretrained language models instead of training a decoder from scratch. The proposed pipeline first trains an LSTM adapter to map high-frequency neural activity envelopes into the embedding space of a pretrained text encoder, then fine-tunes a pretrained embedding-to-text corrector so the aligned neural embeddings can be read out as natural sentences. The authors report that 30 minutes of listening data suffices, that the model outperforms a recent continuous-decoding baseline on semantic accuracy, and that it generalizes to entirely held-out stories without additional training. They present the study as exploratory because the dataset contains three epilepsy patients, and they treat the alignment phase as the load-bearing component, supported by ablations in which removing either phase lowers performance.

What carries the argument

The carrying mechanism is the aligned embedding space produced by the adapter. An LSTM reads iEEG high-gamma envelopes and outputs fixed-dimensional embeddings, trained with a weighted combination of contrastive and triplet margin losses against target sentence embeddings. With the adapter frozen, the pretrained Vec2Text corrector — an encoder-decoder transformer that inverts text embeddings by iteratively updating a hypothesis text $x^{(t)}$ to maximize $\cos(\hat{e}(x), e)$ — is fine-tuned to read out natural language from the neural-aligned embeddings. Only one refinement step is used during evaluation, so the corrector's generalization depends on how close the adapter's outputs sit to the pretrained embedding manifold.

What would settle it

Measure the cosine similarity between adapter output embeddings and their target sentence embeddings on held-out stories. If those similarities are no higher than similarity to randomly chosen text embeddings, or if the fine-tuned corrector's single refinement step cannot move a hypothesis text closer to the target on out-of-domain sentences, the transfer claim fails and reconstruction scores should fall to near-random levels.

Watch

Extended reading notes

Core claim

The central claim is that semantic decoding of language from iEEG decomposes into two transfer steps: align neural signals to a pretrained text embedding space, then invert that embedding space back into text. After training the frozen LSTM adapter with a weighted contrastive-plus-triplet loss against the pretrained sentence embeddings, the authors fine-tune the Vec2Text corrector for two epochs with a single refinement step and report sentence-level BERTScore of $0.195 \pm 0.128$ and BLEU of $0.079 \pm 0.062$ across six cross-validation stories, above the random control and above the baseline on semantic quality. The same configuration retains an out-of-domain advantage when entire stories are held out, and both data and electrode scaling curves rise roughly linearly. The paper's interpretation is that the method captures broad semantic structure rather than memorizing training sentences, and that neither the adapter alone nor the corrector alone reproduces the full-model result.

Load-bearing premise

The load-bearing premise is that after only two epochs of fine-tuning, the pretrained Vec2Text corrector can invert embeddings produced by the frozen LSTM adapter from held-out and out-of-domain stories, which requires those neural embeddings to fall close enough to the distribution of text embeddings the corrector was originally trained on.

Editorial extensions

If this is right

With roughly 30 minutes of per-subject listening data, continuous semantic decoding from iEEG becomes practical in clinical recording sessions rather than requiring many hours of data.
The decoder is not limited to a fixed vocabulary or candidate set, so the generated text is open-vocabulary natural language.
Holding out entire stories still yields above-baseline semantic reconstruction, indicating that the learned mapping transfers to novel semantic content.
Reconstruction quality improves roughly linearly with both training data and electrode coverage, so denser cortical sampling and larger datasets should directly improve decoding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the adapter is agnostic to recording modality, the same align-then-invert recipe could be retrained on scalp EEG, MEG, or fMRI features; a direct test would keep the corrector fixed and train only a new adapter on the new input features.
The fact that two fine-tuning epochs and one refinement step suffice suggests alignment quality, not corrector capacity, is the current bottleneck, so future gains may come from better alignment losses rather than larger language models.
The large variance across random electrode subsets implies that electrode placement matters more than raw channel count; an extension would learn to select or weight electrodes, which could improve decoding with fixed coverage.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing to know about this paper is that the framework is real, but the headline comparison is statistically fragile. The authors build a two-phase iEEG semantic decoder: an LSTM adapter aligns high-gamma envelope features to ada-002 text embeddings with a contrastive plus triplet loss, then a fine-tuned Vec2Text corrector inverts those embeddings into free text. As far as I can tell, no prior iEEG paper does that combination, and the ablations show both phases contribute. Code is released, the method is described in enough detail to reproduce, and the scaling experiments give a useful sense of data and electrode dependence.

The soft spots are real but not fatal. The main one is the statistics behind 'outperforms the baseline.' The comparison in Table 1 and Figure 2A uses paired t-tests over pooled sentence-level BERTScore/BLEU pairs across three subjects and six stories. Sentences from the same story are not independent, so those p-values are likely anti-conservative. The paper reports no per-subject or per-story means, no mixed-effects model, and no clustered resampling. Since the central claim is explicitly comparative, the unit of analysis should be subject or story, not individual sentences. The stress-test note is correct on this.

Second, the baseline is an adapted fMRI Bayesian decoder from Tang et al., not an iEEG-specific method. That makes the comparison weak: the bar is low. A proper iEEG decoder baseline, even a simple regression to embeddings, would be more convincing. Third, the hyperparameters (alpha, temperature, learning rate) were chosen by coordinate descent, and the paper doesn't say whether that tuning was done on the held-out test stories or only on validation folds. That matters for the zero-shot claims. Fourth, the neural data aren't released, so independent verification is limited to code and the authors' own recordings.

The distribution-shift concern about the pre-trained corrector is real but minor: the paper is upfront that fine-tuning adapts the corrector to neural-aligned embeddings, and the out-of-domain results suggest it works reasonably, though without stronger statistics I wouldn't over-read them.

Overall, this is a serious, honest paper. The authors call it exploratory, and that's about right. It deserves a proper peer review: the method is novel, the problem is important, and the weaknesses are fixable. I'd send it out and ask for clustered statistics, a stronger baseline, and clarification of the tuning procedure. I'd probably cite it if I were working on iEEG decoders.

Referee Report

4 major / 5 minor

Summary. The paper introduces Neuro2Semantic, a two-phase transfer-learning framework for decoding perceived-speech semantics from intracranial EEG. Phase 1 trains an LSTM adapter to map high-gamma envelope segments to text-embedding-ada-002 embeddings via a weighted contrastive and triplet loss; Phase 2 freezes the adapter and fine-tunes a Vec2Text corrector to generate continuous text from the aligned neural embeddings. The model is evaluated on three epilepsy patients listening to six podcast-like conversations (about 30 minutes of speech), using BERTScore and BLEU against a version of the Tang et al. (2023) fMRI decoder adapted to iEEG. The authors report ablations, zero-shot out-of-domain results, data- and electrode-scaling curves, and state in the discussion that the small sample size and clinical population limit generalizability, explicitly presenting the study as exploratory. Code and trained models are released.

Significance. If the comparative claims held, Neuro2Semantic would be a useful demonstration that pre-trained text-embedding inversion can be applied to iEEG with very limited training data, enabling open-vocabulary, continuous text generation. The two-phase design, the out-of-domain test, the ablations, and the public code release are genuine strengths. I do not see a circularity problem: the method is a supervised mapping from neural activity to pre-trained text embeddings. However, the evidence as presented is not yet at the level claimed in the abstract: the baseline is not shown to be state of the art for iEEG, the statistical tests ignore sentence-level clustering within subjects and stories, and the absolute BERTScore and BLEU values are low. The paper's practical significance is therefore conditional on a clustered re-analysis and more cautious interpretation of the reported effect sizes.

major comments (4)

[Section 3.2, Table 1, Figure 2A] The central 'outperforms' claim is supported only by paired t-tests computed over pooled sentence-level BERTScore and BLEU values. Because the three subjects each contribute many sentences drawn from the same six stories, the independence assumption of the t-test is violated, and the caption of Table 1 states that the asterisks mark improvements over the random control, not over the baseline. A paired t-test over thousands of correlated sentence pairs can yield p < 0.05 even when no consistent per-subject or per-story improvement exists. The manuscript reports no subject-level or story-level means, no mixed-effects model with random intercepts for subject and story, and no clustered bootstrap. I ask that the authors re-run the comparison with subject or story as the unit of analysis (or a mixed-effects model), report the corresponding effect sizes, and show that the headline result survives that analysis.
[Section 2.3, Section 3.2] The baseline is the fMRI Bayesian decoding method of Tang et al. (2023), adapted to iEEG by using high-gamma features and shorter finite impulse response delays, and Section 3.2 refers to it as 'a recent state-of-the-art method.' The paper does not establish that this adapted method is state of the art for iEEG semantic decoding, and it does not compare against any iEEG-specific decoder, such as Makin et al. [23] or other intracranial speech decoders cited in the introduction. Since the abstract's comparative claim depends on this baseline, the authors should either add an appropriate iEEG baseline or explicitly reframe the contribution as showing an advantage over this particular adapted fMRI method, and remove or qualify the 'state-of-the-art' language.
[Abstract, Section 3.2, Table 1] The abstract and text describe BERTScore 0.195 and BLEU 0.079 as 'strong performance,' but these are low in absolute terms, and the only reference point is the random control (BERTScore -0.245). No comparison with previously reported iEEG decoding scores on the same metrics, no human evaluation, and no measure of inter-subject reliability is provided. The practical significance of the 30-minute low-data claim is therefore not established by the reported numbers. Please add external context for these metrics or soften the 'strong performance' characterization to match the evidence.
[Section 2.1.2, Section 3.1, Figure 2B] Section 2.1.2 describes an iterative corrector that refines text hypotheses until cosine similarity converges, but Section 3.1 states that the corrector used only one refinement step and two epochs of fine-tuning; the manuscript should reconcile this discrepancy. More importantly, the zero-shot out-of-domain claim in Section 3.3 assumes that the LSTM adapter maps novel neural inputs into the pre-trained text-embedding distribution well enough for the Vec2Text corrector to invert them, but the paper reports no diagnostic of this distributional match. I ask for quantitative evidence, such as cosine similarity between predicted and true text embeddings on held-out stories, per-story scores with error bars, or a comparison of embedding distributions, before accepting the generalization claim; Figure 2B currently shows a bar plot without significance markers or error bars.

minor comments (5)

[Section 3.1] The leave-one-out cross-validation description is ambiguous: 'the last trial of each story was left out for testing' is followed by 'the held-out trial from each story served as the test set for that split,' and it is unclear whether the six folds correspond to trials or whole stories. Please clarify the fold structure and the relation between the training set and test set for each fold.
[Section 2.1.1, Section 2.3] Notation is underspecified: the dimension of the text and neural embeddings, the definitions of R and S in the baseline likelihood p(R|S), and the number of finite impulse response delays used in the adapted baseline are not given.
[Figure 2D, Figure 2E] The scaling plots are described as showing 'significant performance improvements,' but no statistical test across the five runs or across training-data percentages is reported; the axes in the bottom-right panel also lack a clear metric label.
[Section 3.1, Eq. (1)] The alignment loss is described as a weighted combination of a CLIP contrastive loss and a triplet margin loss, but the connection between the CLIP reference [28] and this specific formulation is not explained, and the sentence 'batch-level similarity optimization' is vague.
[Figure 2A] The 'random control' is not defined in the main text; please state how the random baseline was constructed (for example, shuffled neural-to-text pairings) so that the negative BERTScore can be interpreted correctly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the neural-to-text pipeline is a supervised mapping evaluated on held-out trials and stories; self-citations are not load-bearing.

full rationale

The claimed derivation chain is a two-phase supervised mapping: iEEG high-gamma features are passed through an LSTM adapter trained with a contrastive plus triplet loss to align neural embeddings to a pre-trained text embedding space, and a pre-trained Vec2Text corrector is fine-tuned to invert those aligned embeddings into text. The adapter is trained against text embeddings of the training sentences, and the corrector is trained to reconstruct the text of those sentences from the adapter outputs. Evaluation is then performed on held-out data: the last trial of each story in the main comparison and entirely held-out stories in the zero-shot out-of-domain experiment. Because the test sentences are not used to fit the adapter or the corrector, the reported BERTScore and BLEU values are genuine predictions rather than quantities forced by construction. No equation in the paper defines a target in terms of the output it is claimed to predict; the objective in Eq. (1) is a standard alignment loss, and Eq. (2)-(3) describe the pre-existing Vec2Text inversion procedure. The only self-citations are the electrode selection criterion ('same selection criteria as [29]') and the baseline encoding model ('following Mischler et al. [29]'); neither is invoked as a theorem that forbids alternatives, and neither supplies the framework's mapping from neural data to text. The t-test clustering concern raised in the skeptic headline is a statistical robustness issue, not circularity, and the paper itself notes the small sample size as a limitation. Overall, the central claim is a supervised empirical comparison and is self-contained relative to its inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard domain assumptions about the information content of high-gamma iEEG, the invertibility of text embeddings, and the transferability of a trained adapter, plus several tuned hyperparameters (alpha, tau, refinement steps, learning rate). No new physical entities are introduced.

free parameters (4)

alpha = 0.25
Weight balancing contrastive and triplet losses in Eq. 1; optimized by coordinate descent (Sec. 3.1).
temperature tau = 0.1
Temperature for CLIP-style contrastive loss, optimized by coordinate descent (Sec. 3.1).
corrector refinement steps = 1
Number of iterative refinement steps in Vec2Text during fine-tuning (Sec. 3.1).
adapter learning rate = 1.3e-3
Adam learning rate for LSTM adapter training (Sec. 3.1).

assumptions (4)

domain assumption High-gamma band (70-150 Hz) envelope of iEEG, downsampled to 100 Hz, carries sufficient semantic information about perceived speech.
Standard in auditory neuroscience; the paper computes this envelope and uses it as input without validation on this dataset (Sec. 2.2).
domain assumption text-embedding-ada-002 embeddings capture sentence meaning in a space that is alignable with neural activity.
The paper relies on the pre-trained embedding model to provide the target semantic representation (Sec. 2.1.1).
domain assumption Vec2Text can recover fluent text from an embedding with a single refinement step after only two epochs of fine-tuning.
The corrector is assumed to invert neural-aligned embeddings sufficiently for semantic evaluation (Sec. 2.1.2 and 3.1).
ad hoc to paper The electrode selection criteria from Mischler et al. apply to this dataset and do not bias the decoding results.
The paper says 'only responsive electrodes were used (same selection criteria as [29])' without specifying the criteria (Sec. 2.2).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Neuro2Semantic: A Transfer Learning Framework for Semantic Reconstruction of Continuous Language from Human Intracranial EEG." pith.science (2026). https://pith.science/paper/GE7DBMLW

@misc{pith2026250600381,
  author       = {Pith},
  title        = {Pith review of: Neuro2Semantic: A Transfer Learning Framework for Semantic Reconstruction of Continuous Language from Human Intracranial EEG},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GE7DBMLW}},
  note         = {Machine review of arXiv:2506.00381}
}

read the original abstract

Decoding continuous language from neural signals remains a significant challenge in the intersection of neuroscience and artificial intelligence. We introduce Neuro2Semantic, a novel framework that reconstructs the semantic content of perceived speech from intracranial EEG (iEEG) recordings. Our approach consists of two phases: first, an LSTM-based adapter aligns neural signals with pre-trained text embeddings; second, a corrector module generates continuous, natural text directly from these aligned embeddings. This flexible method overcomes the limitations of previous decoding approaches and enables unconstrained text generation. Neuro2Semantic achieves strong performance with as little as 30 minutes of neural data, outperforming a recent state-of-the-art method in low-data settings. These results highlight the potential for practical applications in brain-computer interfaces and neural decoding technologies.

Figures

Figures reproduced from arXiv: 2506.00381 by the authors.

**Figure 1.** Neuro2Semantic architecture and training methodology. Training is split into 2 phases. In Phase 1, an adapter module is trained to output a neural embedding that is aligned with a fixed sentence embedding. In Phase 2, a corrector module is trained to read out the neural embedding as continuous language. To achieve effective alignment between the neural embeddings generated by the LSTM adapter and the corresponding… view at source ↗

**Figure 2.** Performance comparison between Neuro2Semantic, baseline, and random control. The BLEU and BERTScore gains correspond to a tangible boost in semantic accuracy (A) Boxplots of BERTScore (left) and BLEU Score (right) comparing the performance of Neuro2Semantic, the baseline model [6], and a random control. Significance is indicated with a star based on a paired t-test (p < 0.05). (B) Out-of-domain performance for each… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 32 canonical work pages

[23]

Brain decoding: to- ward real-time reconstruction of visual perception,

Y . Benchetrit, H. Banville, and J.-R. King, “Brain decoding: to- ward real-time reconstruction of visual perception,” inThe Twelfth International Conference on Learning Representations, 2024

work page 2024
[1]

These models aim to reconstruct stimuli or intentions based on measured neural activity [1]

Introduction Recent advances at the intersection of artificial intelligence (AI) and neuroscience have enabled powerful new modeling capabil- ities, particularly in the development of neural decoding mod- els. These models aim to reconstruct stimuli or intentions based on measured neural activity [1]. Decoding models have been explored across various neur...

work page 2025
[2]

Methods 2.1. Neuro2Semantic The proposed Neuro2Semantic framework, illustrated in Figure 1, is designed to map neural signals to their corresponding se- mantic content through a two-phase training process. In the first phase, an LSTM adapter processes the neural data and aligns it with text embeddings obtained from a pre-trained text em- bedding model ( t...

work page Pith review arXiv 2025
[3]

Experimental Setup Training Procedure

Experiments and Results 3.1. Experimental Setup Training Procedure. We trained the model using a leave-one- out cross-validation approach, where the last trial of each story was left out for testing. Each trial was split into sentences, with the corresponding neural data segment from when the sentence was spoken used for training. This setup prevented any...

work page
[4]

Discussion Neuro2Semantic demonstrates significant advances in neural language decoding through its novel two-phase architecture and efficient data utilization. Unlike classification-based ap- proaches [23] or retrieval-oriented frameworks [7], our model directly aligns iEEG signals with semantic embeddings, en- abling unconstrained text generation withou...

work page
[5]

With just 30 minutes of data, it out- performs existing methods and demonstrates strong zero-shot generalization and unconstrained text generation

Conclusion We introduce Neuro2Semantic, a transfer learning framework that decodes continuous language from neural signals using pre- trained text embeddings. With just 30 minutes of data, it out- performs existing methods and demonstrates strong zero-shot generalization and unconstrained text generation. The approach scales with larger datasets and limit...

work page
[6]

Semantic reconstruc- tion of continuous language from non-invasive brain recordings,

J. Tang, A. LeBel, S. Jain, and A. G. Huth, “Semantic reconstruc- tion of continuous language from non-invasive brain recordings,” Nature Neuroscience, pp. 1–9, 2023

work page 2023
[7]

Interpreting encoding and decoding models,

N. Kriegeskorte and P. K. Douglas, “Interpreting encoding and decoding models,” Current opinion in neurobiology, vol. 55, pp. 167–179, 2019

work page 2019

Show all 39 references

[8]

Progress in speech decoding from the electrocor- ticogram,

S. Chakrabarti, H. M. Sandberg, J. S. Brumberg, and D. J. Krusienski, “Progress in speech decoding from the electrocor- ticogram,” Biomedical Engineering Letters , vol. 5, pp. 10–21, 2015

2015
[9]

Towards reconstructing intelligible speech from the human auditory cortex,

H. Akbari, B. Khalighinejad, J. L. Herrero, A. D. Mehta, and N. Mesgarani, “Towards reconstructing intelligible speech from the human auditory cortex,” Scientific reports, vol. 9, no. 1, p. 874, 2019

2019
[10]

Brainbert: Self-supervised repre- sentation learning for intracranial recordings,

C. Wang, V . Subramaniam, A. U. Yaari, G. Kreiman, B. Katz, I. Cases, and A. Barbu, “Brainbert: Self-supervised repre- sentation learning for intracranial recordings,” arXiv preprint arXiv:2302.14367, 2023

2023 arXiv
[11]

En- coding and decoding in fmri,

T. Naselaris, K. N. Kay, S. Nishimoto, and J. L. Gallant, “En- coding and decoding in fmri,” Neuroimage, vol. 56, no. 2, pp. 400–410, 2011

2011
[12]

Toward fully-end-to-end listened speech decoding from eeg signals,

J. Lee, A. Kommineni, T. Feng, K. Avramidis, X. Shi, S. R. Kadiri, and S. Narayanan, “Toward fully-end-to-end listened speech decoding from eeg signals,” in Interspeech 2024, 2024, pp. 1500–1504

2024
[13]

Decoding speech from non-invasive brain recordings,

A. D ´efossez, C. Caucheteux, J. Rapin, O. Kabeli, and J.-R. King, “Decoding speech from non-invasive brain recordings,” arXiv preprint arXiv:2208.12266, 2022

2022 arXiv
[14]

Se- mantic reconstruction of continuous language from meg signals,

B. Wang, X. Xu, L. Zhang, B. Xiao, X. Wu, and J. Chen, “Se- mantic reconstruction of continuous language from meg signals,” in ICASSP 2024-2024 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP) . IEEE, 2024, pp. 2190–2194

2024
[15]

Open vocabulary electroencephalography-to- text decoding and zero-shot sentiment classification,

Z. Wang and H. Ji, “Open vocabulary electroencephalography-to- text decoding and zero-shot sentiment classification,” in Proceed- ings of the AAAI Conference on Artificial Intelligence , vol. 36, no. 5, 2022, pp. 5350–5358

2022
[16]

Eeg2text: Open vocabulary eeg-to-text decoding with eeg pre-training and multi-view transformer,

H. Liu, D. Hajialigol, B. Antony, A. Han, and X. Wang, “Eeg2text: Open vocabulary eeg-to-text decoding with eeg pre-training and multi-view transformer,” arXiv preprint arXiv:2405.02165, 2024

2024 arXiv
[17]

Neu- ral2speech: A transfer learning framework for neural-driven speech reconstruction,

J. Li, C. Guo, L. Fu, L. Fan, E. F. Chang, and Y . Li, “Neu- ral2speech: A transfer learning framework for neural-driven speech reconstruction,” in ICASSP 2024-2024 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 2200–2204

2024
[18]

A high-performance speech neuroprosthesis,

F. R. Willett, E. M. Kunz, C. Fan, D. T. Avansino, G. H. Wilson, E. Y . Choi, F. Kamdar, M. F. Glasser, L. R. Hochberg, S. Druck- mann et al., “A high-performance speech neuroprosthesis,” Na- ture, vol. 620, no. 7976, pp. 1031–1036, 2023

2023
[19]

Noninvasive brain-computer inter- face: decoding arm movement kinematics and motor control,

N. Robinson and A. Vinod, “Noninvasive brain-computer inter- face: decoding arm movement kinematics and motor control,” IEEE Systems, Man, and Cybernetics Magazine, vol. 2, no. 4, pp. 4–16, 2016

2016
[20]

High performance communication by people with paralysis using an intracortical brain-computer interface,

C. Pandarinath, P. Nuyujukian, C. H. Blabe, B. L. Sorice, J. Saab, F. R. Willett, L. R. Hochberg, K. V . Shenoy, and J. M. Hender- son, “High performance communication by people with paralysis using an intracortical brain-computer interface,” elife, vol. 6, p. e18554, 2017

2017
[21]

Generalized decoding for pixel, im- age, and language,

X. Zou, Z.-Y . Dou, J. Yang, Z. Gan, L. Li, C. Li, X. Dai, H. Behl, J. Wang, L. Yuan et al. , “Generalized decoding for pixel, im- age, and language,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 15 116– 15 127

2023
[22]

Dream: Visual decoding from reversing human visual system,

W. Xia, R. de Charette, C. Oztireli, and J.-H. Xue, “Dream: Visual decoding from reversing human visual system,” inProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024, pp. 8226–8235

2024
[24]

Second, after aligning the neural embeddings, we fine-tune a pre-trained text reconstruction model [27] to extract coherent text from the neural-aligned embeddings

adapter to align neural data with a pre-trained text embed- ding space [25, 26] using a contrastive loss function. Second, after aligning the neural embeddings, we fine-tune a pre-trained text reconstruction model [27] to extract coherent text from the neural-aligned embedding...
[25]

A high-performance neuroprosthesis for speech decoding and avatar control,

S. L. Metzger, K. T. Littlejohn, A. B. Silva, D. A. Moses, M. P. Seaton, R. Wang, M. E. Dougherty, J. R. Liu, P. Wu, M. A. Berger et al., “A high-performance neuroprosthesis for speech decoding and avatar control,” Nature, vol. 620, no. 7976, pp. 1037–1046, 2023

2023
[26]

Natural speech reveals the semantic maps that tile human cerebral cortex,

A. G. Huth, W. A. De Heer, T. L. Griffiths, F. E. Theunissen, and J. L. Gallant, “Natural speech reveals the semantic maps that tile human cerebral cortex,” Nature, vol. 532, no. 7600, pp. 453–458, 2016

2016
[27]

Neural decoding of semantic concepts: A systematic literature review,

M. Ryb ´aˇr and I. Daly, “Neural decoding of semantic concepts: A systematic literature review,” Journal of Neural Engineering , vol. 19, no. 2, p. 021002, 2022

2022
[28]

Decoding imagined and spo- ken phrases from non-invasive neural (meg) signals,

D. Dash, P. Ferrari, and J. Wang, “Decoding imagined and spo- ken phrases from non-invasive neural (meg) signals,”Frontiers in neuroscience, vol. 14, p. 290, 2020

2020
[29]

Machine translation of cortical activity to text with an encoder–decoder framework,

J. G. Makin, D. A. Moses, and E. F. Chang, “Machine translation of cortical activity to text with an encoder–decoder framework,” Nature neuroscience, vol. 23, no. 4, pp. 575–582, 2020

2020
[30]

Long short-term memory,

S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Comput., vol. 9, no. 8, p. 1735–1780, Nov. 1997

1997
[31]

Exploring the Limits of Trans- fer Learning with a Unified Text-to-Text Transformer,

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the Limits of Trans- fer Learning with a Unified Text-to-Text Transformer,”Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020

2020
[32]

Openai api: Embeddings,

OpenAI, “Openai api: Embeddings,” https://platform.openai. com/docs/guides/embeddings, 2022

2022
[33]

Text Em- beddings Reveal (Almost) As Much As Text,

J. Morris, V . Kuleshov, V . Shmatikov, and A. Rush, “Text Em- beddings Reveal (Almost) As Much As Text,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali, Eds. Singa- pore: Association for Computationa...

2023
[34]

Learning Transferable Visual Models From Natural Language Supervision,

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agar- wal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning Transferable Visual Models From Natural Language Supervision,” in Proceedings of the 38th International Conference on Machine...

2021
[35]

Contextual feature extraction hierarchies converge in large lan- guage models and the brain,

G. Mischler, Y . A. Li, S. Bickel, A. D. Mehta, and N. Mesgarani, “Contextual feature extraction hierarchies converge in large lan- guage models and the brain,” Nature Machine Intelligence , pp. 1–11, 2024

2024
[36]

Comparison of time–frequency responses and the event-related potential to auditory speech stim- uli in human cortex,

E. Edwards, M. Soltani, W. Kim, S. S. Dalal, S. S. Nagarajan, M. S. Berger, and R. T. Knight, “Comparison of time–frequency responses and the event-related potential to auditory speech stim- uli in human cortex,” Journal of neurophysiology, vol. 102, no. 1, pp. 377–386, 2009

2009
[37]

Adam: A method for stochastic op- timization,

D. P. Kingma and J. Ba, “Adam: A method for stochastic op- timization,” in 3rd International Conference on Learning Repre- sentations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Con- ference Track Proceedings, Y . Bengio and Y . LeCun, Eds., 2015

2015
[38]

Bleu: a method for automatic evaluation of machine translation,

K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311–318

2002
[39]

BERTScore: Evaluating Text Generation with BERT,

T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi, “BERTScore: Evaluating Text Generation with BERT,” Apr. 2020

2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [23]

Brain decoding: to- ward real-time reconstruction of visual perception,

Y . Benchetrit, H. Banville, and J.-R. King, “Brain decoding: to- ward real-time reconstruction of visual perception,” inThe Twelfth International Conference on Learning Representations, 2024

work page 2024

[2] [1]

These models aim to reconstruct stimuli or intentions based on measured neural activity [1]

Introduction Recent advances at the intersection of artificial intelligence (AI) and neuroscience have enabled powerful new modeling capabil- ities, particularly in the development of neural decoding mod- els. These models aim to reconstruct stimuli or intentions based on measured neural activity [1]. Decoding models have been explored across various neur...

work page 2025

[3] [2]

Methods 2.1. Neuro2Semantic The proposed Neuro2Semantic framework, illustrated in Figure 1, is designed to map neural signals to their corresponding se- mantic content through a two-phase training process. In the first phase, an LSTM adapter processes the neural data and aligns it with text embeddings obtained from a pre-trained text em- bedding model ( t...

work page Pith review arXiv 2025

[4] [3]

Experimental Setup Training Procedure

Experiments and Results 3.1. Experimental Setup Training Procedure. We trained the model using a leave-one- out cross-validation approach, where the last trial of each story was left out for testing. Each trial was split into sentences, with the corresponding neural data segment from when the sentence was spoken used for training. This setup prevented any...

work page

[5] [4]

Discussion Neuro2Semantic demonstrates significant advances in neural language decoding through its novel two-phase architecture and efficient data utilization. Unlike classification-based ap- proaches [23] or retrieval-oriented frameworks [7], our model directly aligns iEEG signals with semantic embeddings, en- abling unconstrained text generation withou...

work page

[6] [5]

With just 30 minutes of data, it out- performs existing methods and demonstrates strong zero-shot generalization and unconstrained text generation

Conclusion We introduce Neuro2Semantic, a transfer learning framework that decodes continuous language from neural signals using pre- trained text embeddings. With just 30 minutes of data, it out- performs existing methods and demonstrates strong zero-shot generalization and unconstrained text generation. The approach scales with larger datasets and limit...

work page

[7] [6]

Semantic reconstruc- tion of continuous language from non-invasive brain recordings,

J. Tang, A. LeBel, S. Jain, and A. G. Huth, “Semantic reconstruc- tion of continuous language from non-invasive brain recordings,” Nature Neuroscience, pp. 1–9, 2023

work page 2023

[8] [7]

Interpreting encoding and decoding models,

N. Kriegeskorte and P. K. Douglas, “Interpreting encoding and decoding models,” Current opinion in neurobiology, vol. 55, pp. 167–179, 2019

work page 2019