Teaching Wav2Vec2 the Language of the Brain

Eilon Vaadia; Florian M\"uller; Leon Hermann; Peter Chin; Sarel Cohen; Tobias Fiedler; Tobias Friedrich

REVIEW 3 major objections 5 minor 8 references

Teaching Wav2Vec2 the Language of the Brain

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Pre-trained speech model Wav2Vec2 transfers to brain decoding: replacing its audio front end with a GRU extractor and fine-tuning cuts word error from 66.21% to 30.97%.

desk verdict A solid, well-designed transfer-learning experiment with a credible central claim that needs stronger statistical reporting and a more defensible test split before the headline numbers can be taken at face value. read the letter →

arxiv 2501.09459 v1 pith:RC4DS2MJ submitted 2025-01-16 cs.LG

Tobias Fiedler , Leon Hermann , Florian Müller , Sarel Cohen , Peter Chin , Tobias Friedrich , Eilon Vaadia This is my paper · ORCID

classification cs.LG

keywords brain-computerinterfacespeechdecodingWav2Vec2transferlearningself-supervisedpretrainingneuralspikingactivityCTClossfine-tuning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Brain-computer interfaces that decode attempted speech from neural recordings are limited by small datasets, while audio speech recognition has abundant data and large pre-trained models. This paper claims that the two tasks share enough structure for transfer: replacing Wav2Vec2's audio feature extractor with an untrained GRU-based extractor and fine-tuning the pre-trained weights end-to-end lowers character error rate from 39.01% to 18.54% and word error rate from 66.21% to 30.97% on the same 45 architectures. If the claim is right, it gives small-sample neural decoding a practical route to transformer-level performance, which matters for making speech BCIs viable for paralyzed patients.

What carries the argument

The central mechanism is Wav2Vec2's transformer encoder plus character head, pre-trained on 960 hours of audio, with its CNN audio feature extractor replaced by an untrained Brain Feature Extractor (BFE) built from a stacked bidirectional GRU and fully-connected projection layers. The BFE is fed day-specific linear input layers and rolling z-scored spike features, and it is trained to emit vectors that play the role of Wav2Vec2's 'speech units' so the pre-trained transformer can contextualize them. The whole pipeline is optimized with CTC loss, which aligns unsegmented neural sequences to character transcripts without frame-level labels; during full fine-tuning the Wav2Vec2 modules use a much smaller learning rate that only ramps up after the BFE output starts stabilizing, protecting the pre-trained weights from early random gradients.

What would settle it

Retrain the 45 architectures in all three setups on the full training data with several random seeds and evaluate on an independent holdout such as a later recording block or the official competition test set; the transfer claim fails if pre-trained fine-tuning does not consistently beat from-scratch training by a large margin.

Watch

Extended reading notes

Core claim

The paper's central discovery is that knowledge learned by Wav2Vec2 on audio speech does transfer to micro-electrode recordings of attempted speech, as long as the audio front end is replaced by a trainable Brain Feature Extractor (BFE) and the whole model is fine-tuned end-to-end. Across 45 BFE architectures, full fine-tuning of pre-trained weights produced the best run, with a word error rate of 30.97% and a character error rate of 18.54%, beating the best from-scratch run by about 20 percentage points CER and the best frozen-Wav2Vec2 run by about 16 points. In 40 of 45 architectures the fine-tuned pre-trained model beat the best run of both alternative setups, and the authors read the consistent ordering full-fine-tuning > frozen > from-scratch as evidence that the pre-trained weights contribute real linguistic knowledge rather than merely a convenient initialization.

Load-bearing premise

The reported gains rest on a hand-made test split that takes the first recording block of each day as test data and on a single training run per configuration; if first blocks are unrepresentative of later blocks or the runs are noisy, the transfer advantage could shrink or disappear.

Editorial extensions

If this is right

Pre-trained speech models become a strong default starting point for neural speech-decoding pipelines, reducing the amount of labeled brain data needed to train a usable decoder.
The same BFE-replacement recipe extends to newer speech encoders: the paper's exploratory full fine-tuning with a Wav2Vec2Conformer reached 15.72% CER and 26.67% WER.
Because full fine-tuning beats freezing, the benefit comes from adapting the pre-trained transformer and character head to brain-derived features, not just from reusing the feature extractor.
Frozen Wav2Vec2's better-than-random performance in some architectures shows that certain BFE designs already map neural spiking activity into a representation the pre-trained transformer can interpret without any gradient update to Wav2Vec2.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: the 44-of-45 architecture sweep suggests that BFE architecture choice matters more in the transfer setting than in from-scratch training, so future work should search BFE designs jointly with the transfer setup rather than fixing the GRU family.
Beyond the paper: a decisive control would replace pre-trained Wav2Vec2 with a randomly initialized transformer of the same size, isolating whether the gain comes from speech-specific pre-training or simply from a large well-tuned architecture.
Beyond the paper: the observed shrinkage in separation between brain and audio features after the transformer suggests a quantitative representation-distance metric could be used to select BFE architectures without running full decoding experiments.
Beyond the paper: if the transfer effect survives independent evaluation, it would strengthen the case for cross-modal transfer in other small-data neural decoding settings such as electrocorticography or EEG-based speech decoding.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should read this one if you care about transfer learning for BCI speech decoding. The central claim is that a pre-trained Wav2Vec2 checkpoint, fine-tuned end-to-end through an untrained GRU-based Brain Feature Extractor, transfers to single-patient brain-to-text decoding significantly better than training from scratch. The evidence for that qualitative claim is genuinely convincing: 44 of 45 architectures favor full fine-tuning over the same architecture in the other two setups, and the paired Wilcoxon tests give p-values below 1e-14. The design is honest, the pre-trained checkpoint is external, and the comparison of pre-trained versus random initialization under identical training is a real experiment, not a circular setup.

The main soft spots are statistical and evaluative rather than conceptual. Each configuration is trained once, so Table I's headline numbers are single-run best-checkpoint estimates with no variance. The 20.46-point gap is a comparison of two noisy best-runs over 45 correlated configurations; the gap could easily be smaller or larger with seeds. The test set is built by taking the first block of each day from the training partition (Section III-A), a non-random split whose representativeness is unverified. Since all three setups share that split, the relative comparison is partly protected, but the absolute numbers and any comparison with prior work rest on it. I also caught an internal inconsistency: the abstract reports a Frozen Wav2Vec2 CER of 32.46%, while Table I reports 34.46%; the ranking is unaffected, but the discrepancy should be fixed. And the Frozen-versus-Scratch contrast is only marginal (p = 0.0562), so calling that difference "significant" in the discussion is too strong.

The novelty is moderate but real: adapting a pre-trained audio speech model to brain recordings by swapping the audio feature extractor for a GRU is a new application, and the systematic comparison across 45 architectures is a useful method-level contribution. The citation pattern to Willett, Benster, and Wu is appropriate. The paper is not a rival to current state-of-the-art WER numbers, and it does not claim to be.

This paper deserves a serious referee. The core claim is well-founded and important enough for the BCI community, but the statistical reporting, the hand-made test split, and the CER inconsistency need to be addressed before the headline numbers can be trusted. I would send it to peer review with a request for multiple seeds, a more principled split or cross-validated evaluation, and corrected reporting.

Referee Report

3 major / 5 minor

Summary. The paper asks whether a speech-recognition transformer pretrained on audio, Wav2Vec2, can transfer knowledge to decoding continuous speech from neuronal spiking activity. The authors replace Wav2Vec2's audio feature extractor with an untrained GRU-based Brain Feature Extractor (BFE), then compare three setups—full fine-tuning of the pretrained model, freezing the pretrained model, and training the same architecture from scratch—across 45 BFE architectures on a single-patient brain-to-text dataset. The central reported result is that full fine-tuning achieves the best CER of 18.54% and WER of 30.97%, versus 39.01% CER / 66.21% WER from scratch and 34.46% CER / 53.29% WER with a frozen pretrained model. The paper also reports that in 44 of 45 architecture-matched comparisons full fine-tuning wins, and one-sided Wilcoxon tests are highly significant (p < 1e-14) against both alternatives. The authors conclude that knowledge transfer from audio speech recognition to brain decoding is possible and significantly improves performance for identical architectures.

Significance. If the result holds, it is a useful demonstration that large pretrained audio speech models can be repurposed for invasive BCI decoding despite the domain gap between audio and brain recordings, which matters because BCI training data are scarce. The study has several genuine strengths: the pretrained checkpoint is an external artifact trained on audio; the comparison is architecture-matched with 45 BFE variants; the from-scratch and frozen conditions provide meaningful controls; the paired design and Wilcoxon tests directly support the ranking claim; and source code is provided. I also see no circularity problem: no equation reduces to a fitted parameter, and the transfer claim is not reframed as a definition. However, the evaluation rests on a hand-constructed first-block-of-each-day test split and on single runs per configuration, so the magnitude of the reported improvement and any comparison to prior work are less certain than the qualitative transfer claim.

major comments (3)

[III-A] The test set is constructed by taking the first block of each day of the training set, a non-random partition. This assumes that first recording blocks are representative of later blocks; electrode settling, neural drift, or changing recording conditions could make first blocks systematically different. Because all three setups share the same test set, the relative comparison is partly protected, but the absolute CER/WER values and the 20.46-point transfer gap in Table I depend entirely on this split, and the paper later compares numerics to prior work such as Willett et al.'s 23.8% WER. The authors should justify the representativeness of first blocks or report results under alternative splits (e.g., random split or last-block-per-day), and state how sensitive the headline gap is to the split rule.
[IV-A, Table I] Each configuration is trained once, and the reported headline numbers are the best over 45 BFE architectures. The 20.46-point difference between the best full-fine-tuning run and the best from-scratch run is therefore a difference of two single-run point estimates, selected as best over correlated architecture grids, with no variance estimate. The paired Wilcoxon tests and the 44/45 architecture-matched comparisons support the broad ranking, but they do not quantify the uncertainty of the best-run advantage. I ask the authors to run at least a few seeds per configuration (or otherwise provide resampling-based confidence intervals) and to report mean and standard deviation, so that the headline transfer magnitude is not a winner's-maximum over single runs.
[Abstract and Table I] The abstract reports the Frozen Wav2Vec2 CER as 32.46%, while Table I reports 34.46% and Section I also states 32.46%. This internal inconsistency in a headline number should be corrected and the correct value verified across all occurrences before publication. The error does not change the ranking, but it signals that the reported numbers need a careful audit.

minor comments (5)

[II] The related work section contains a typo: "Willet" should be "Willett" (e.g., "Willet et al. implemented a GRU model").
[III-A] The sentence "The work of the brain-to-text researchers suggested that area 44 contained little information for speech classification so we also excluded that data from training" is unclear: the two arrays in area 44 are described earlier, but it is not stated explicitly whether all area 44 channels were excluded and how this interacts with the per-day fully connected layer.
[IV-B] The t-SNE visualizations in Figure 4 do not report the perplexity or number of iterations used; adding these parameters would make the latent analysis reproducible.
[IV-A] The phrase "Frozen Wav2Vec2 training runs having a CER better than a random predictor" would benefit from a definition of the random predictor (e.g., predicting the most frequent character, or uniform random guessing) so that the claim is testable.
[IV-A, Table I] For the Wav2Vec2Conformer row, the LM-assisted WER (26.81%) is slightly worse than the non-LM WER (26.67%); a sentence explaining why LM decoding can hurt in this case would prevent confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central transfer claim is an empirical outcome of matched pretrained-vs-from-scratch trials on an external Wav2Vec2 checkpoint, not a reduction to its own inputs.

full rationale

The paper's central claim is that Wav2Vec2 pretraining on audio transfers to brain decoding. This is tested by running identical training pipelines for 45 BFE architectures under three initializations: full fine-tuning from the external facebook/wav2vec2-base-960h checkpoint, frozen pretrained weights, and random initialization. The pretrained checkpoint is an external artifact trained on audio; the comparison condition (from scratch) uses the same brain data and architecture and therefore does not define the outcome by construction. The test partition is the first block of each day of the supplied training set (Section III-A), which raises validity concerns about representativeness and single-seed variance, but it is not used to fit any parameter that the reported CER/WER values are derived from. Model selection is performed on a separate validation set (the dataset's given test set) before evaluating on the constructed test set, and the paper reports Wilcoxon tests comparing the full distributions of 45 runs, not only the best runs. No equation in the paper reduces to a fitted input, no load-bearing self-citation or imported uniqueness theorem is used, and the Wav2Vec2 architecture/checkpoint are externally provided. The only noteworthy issues—non-random split, no variance estimate for the headline best run, and the 32.46% vs 34.46% CER inconsistency between the abstract and Table I—are correctness/reporting risks, not circularity. Score 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The central claim rests on hand-chosen training choices and inherited preprocessing rather than on fitted parameters that encode the answer. The main free selections are the 45-architecture grid (with best-of-45 reporting), learning-rate schedules, and the self-made test split, all disclosed. The BFE is a new architectural component with an empirical handle through held-out test results. Preprocessing and the area-44 exclusion are inherited from Willett et al. without re-validation. No invented physical entities or fitted constants are introduced.

free parameters (5)

BFE architecture grid = GRU depths 1/3/5; hidden sizes 256/512/1024; FC [], [128], [256], [512], [512,128]
45 hand-chosen architectures; headline numbers are the best run selected from this grid on the validation set.
General learning rate = 1e-4
Hand-chosen; equal across the three setups, so it does not drive the comparison, but shapes all absolute results.
Wav2Vec2 module learning rate schedule = 1e-5, linearly ramping from 0 between epochs 7 and 10
Hand-designed to prevent destroying pre-trained weights while BFE outputs are still random (Section III-C).
Early stopping patience / max epochs = 10 epochs / 100 epochs
Hand-chosen; the best validation checkpoint is selected per run, adding an epoch-level model selection.
Test-set construction rule = first block of each day of the training partition
A hand-chosen split that defines every reported CER/WER and reduces training sentences from 8,800 to 7,920.

assumptions (5)

standard math Connectionist Temporal Classification (CTC) loss is an appropriate objective for aligning brain-derived features to character sequences.
The whole pipeline optimizes CTC over Wav2Vec2 LM-head logits (Section III-C); if CTC alignment is unsuitable for neural features, the reported results would change.
domain assumption Willett et al. preprocessing (z-scoring, per-day fully connected layer, rolling feature adaptation) transfers validly to this setting.
Inherited from prior work without independent validation here (Sections III-A and III-B).
domain assumption The 45 BFE configurations can be treated as paired observations in the Wilcoxon signed-rank tests.
The test pools across architectures as if exchangeable, but architectures share hyperparameters, so the effective number of independent units is lower than 45.
domain assumption Area 44 electrodes add little speech information, justifying their exclusion.
Taken from the brain-to-text researchers' suggestion, not re-verified in this study (Section III-A).
domain assumption The pre-trained Wav2Vec2 transformer can meaningfully process out-of-distribution brain-derived features.
Latent analysis shows brain and audio features are strongly separated before contextualization, yet the frozen transformer still produces usable logits (Section IV-B).

invented entities (1)

Brain Feature Extractor (BFE) independent evidence
purpose: Bidirectional GRU plus fully connected projection that converts preprocessed spike-count and spike-power features into the representation expected by the pre-trained Wav2Vec2 transformer.
Validated by held-out test performance (frozen setup reaches 34.46% CER, better than random), so it has an empirical handle. It is an architectural component, not a physical entity, so it has no external falsifiable prediction beyond the reported task performance.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Teaching Wav2Vec2 the Language of the Brain." pith.science (2026). https://pith.science/paper/RC4DS2MJ

@misc{pith2026250109459,
  author       = {Pith},
  title        = {Pith review of: Teaching Wav2Vec2 the Language of the Brain},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RC4DS2MJ}},
  note         = {Machine review of arXiv:2501.09459}
}

read the original abstract

The decoding of continuously spoken speech from neuronal activity has the potential to become an important clinical solution for paralyzed patients. Deep Learning Brain Computer Interfaces (BCIs) have recently successfully mapped neuronal activity to text contents in subjects who attempted to formulate speech. However, only small BCI datasets are available. In contrast, labeled data and pre-trained models for the closely related task of speech recognition from audio are widely available. One such model is Wav2Vec2 which has been trained in a self-supervised fashion to create meaningful representations of speech audio data. In this study, we show that patterns learned by Wav2Vec2 are transferable to brain data. Specifically, we replace its audio feature extractor with an untrained Brain Feature Extractor (BFE) model. We then execute full fine-tuning with pre-trained weights for Wav2Vec2, training ''from scratch'' without pre-trained weights as well as freezing a pre-trained Wav2Vec2 and training only the BFE each for 45 different BFE architectures. Across these experiments, the best run is from full fine-tuning with pre-trained weights, achieving a Character Error Rate (CER) of 18.54\%, outperforming the best training from scratch run by 20.46\% and that of frozen Wav2Vec2 training by 15.92\% percentage points. These results indicate that knowledge transfer from audio speech recognition to brain decoding is possible and significantly improves brain decoding performance for the same architectures. Related source code is available at https://github.com/tfiedlerdev/Wav2Vec2ForBrain.

Figures

Figures reproduced from arXiv: 2501.09459 by the authors.

**Figure 2.** Conceptual depiction of experiment setups. Setup 1 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 4.** Features extracted by BFE (blue) and pre-trained [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

8 extracted references · 2 canonical work pages

[1]

A high-performance speech neuroprosthesis,

F. R. Willett, E. M. Kunz, C. Fan, D. T. Avansino, G. H. Wilson, E. Y . Choi, F. Kamdar, M. F. Glasser, L. R. Hochberg, S. Druckmann, K. V . Shenoy, and J. M. Henderson, “A high-performance speech neuroprosthesis,” Nature, vol. 620, no. 7976, pp. 1031–1036, Aug 2023. [Online]. Available: https://doi.org/10.1038/s41586-023-06377-x

doi:10.1038/s41586-023-06377-x 2023
[2]

Learning phrase representations using rnn encoder-decoder for statistical machine translation,

K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y . Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” 2014

2014
[3]

A cross-modal approach to silent speech with llm-enhanced recognition,

T. Benster, G. Wilson, R. Elisha, F. R. Willett, and S. Druckmann, “A cross-modal approach to silent speech with llm-enhanced recognition,” arXiv preprint arXiv:2403.05583 , 2024

arXiv 2024
[4]

Attention is all you need,

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems , I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017. [Online]. Available: https://...

work page 2017
[5]

wav2vec 2.0: A framework for self-supervised learning of speech representations,

A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” 2020. [Online]. Available: https://arxiv.org/abs/2006.11477

arXiv 2020
[6]

On decoder-only architecture for speech- to-text and large language model integration,

J. Wu, Y . Gaur, Z. Chen, L. Zhou, Y . Zhu, T. Wang, J. Li, S. Liu, B. Ren, L. Liu, and Y . Wu, “On decoder-only architecture for speech- to-text and large language model integration,” in 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2023, pp. 1–8

work page 2023
[7]

Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,

A. Graves, S. Fern ´andez, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proceedings of the 23rd International Conference on Machine Learning , ser. ICML ’06. New York, NY , USA: Association for Computing Machinery, 2006, p. 369–376. [Online]. Available: https:...

arXiv 2006
[8]

Mamba: Linear-time sequence modeling with selective state spaces,

A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” 2023

2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

A high-performance speech neuroprosthesis,

F. R. Willett, E. M. Kunz, C. Fan, D. T. Avansino, G. H. Wilson, E. Y . Choi, F. Kamdar, M. F. Glasser, L. R. Hochberg, S. Druckmann, K. V . Shenoy, and J. M. Henderson, “A high-performance speech neuroprosthesis,” Nature, vol. 620, no. 7976, pp. 1031–1036, Aug 2023. [Online]. Available: https://doi.org/10.1038/s41586-023-06377-x

doi:10.1038/s41586-023-06377-x 2023

[2] [2]

Learning phrase representations using rnn encoder-decoder for statistical machine translation,

K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y . Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” 2014

2014

[3] [3]

A cross-modal approach to silent speech with llm-enhanced recognition,

T. Benster, G. Wilson, R. Elisha, F. R. Willett, and S. Druckmann, “A cross-modal approach to silent speech with llm-enhanced recognition,” arXiv preprint arXiv:2403.05583 , 2024

arXiv 2024

[4] [4]

Attention is all you need,

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems , I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017. [Online]. Available: https://...

work page 2017

[5] [5]

wav2vec 2.0: A framework for self-supervised learning of speech representations,

A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” 2020. [Online]. Available: https://arxiv.org/abs/2006.11477

arXiv 2020

[6] [6]

On decoder-only architecture for speech- to-text and large language model integration,

J. Wu, Y . Gaur, Z. Chen, L. Zhou, Y . Zhu, T. Wang, J. Li, S. Liu, B. Ren, L. Liu, and Y . Wu, “On decoder-only architecture for speech- to-text and large language model integration,” in 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2023, pp. 1–8

work page 2023

[7] [7]

Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,

A. Graves, S. Fern ´andez, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proceedings of the 23rd International Conference on Machine Learning , ser. ICML ’06. New York, NY , USA: Association for Computing Machinery, 2006, p. 369–376. [Online]. Available: https:...

arXiv 2006

[8] [8]

Mamba: Linear-time sequence modeling with selective state spaces,

A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” 2023

2023