{"id":"10993c82-d9d9-4f9a-9a96-06d9ffabc066","arxiv_id":"2505.13446","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"The first non-invasive brain-to-text system exceeds chance baselines on six metrics by combining LLM-based rescoring, out-of-vocabulary in-filling, and selective pooling of MEG/EEG datasets.","lead":"A non-invasive brain-to-text pipeline, reading MEG and EEG signals from people listening to audiobooks, beats random-chance baselines across six metrics for the first time, with BLEU scores 1.4 to 2.6 times higher than prior attempts. The method combines word-level brain decoding with LLM rescoring, out-of-vocabulary in-filling, and selective pooling of datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The noise-input control does not rule out LLM memorization of the public-domain stimulus; a text-only baseline is needed to attribute the real-vs-noise gap to brain data.","rationale":"The reader's weakest assumption identifies exactly the load-bearing concern: the validity of the noise-input control depends on the LLM not memorizing the public-domain stimulus. This is not a manufactured objection; the paper's own Appendix L prompts the LLM to complete and in-fill the very passages that constitute the test set, and the stimuli are Sherlock Holmes stories, which are almost certainly in the training corpora of Llama and Claude. The paper does not include a text-only baseline or a memorization test, and Appendix J does not list this limitation. This concern does not by itself disprove the central claim, because the real-vs-noise gap may survive such a test, but it is the point where the argument is least secure. The reader's CONDITIONAL verdict already reflects this uncertainty, so no verdict change is warranted; however, the condition should explicitly include the text-only control proposed above.","tokens_in":22154,"tokens_out":6686,"duration_ms":68117,"concrete_test":"Run the full beam+fill and beam+IC-fill decoding pipelines on a text-only control in which the neural word-classifier logits are replaced by uniform random draws from the retrieval vocabulary, preserving the exact prompts, in-filling protocol, and evaluation metrics of Table 3. If this no-signal control achieves BLEU statistically indistinguishable from the real-input beam+fill BLEU, the above-chance result is an LLM prior artifact rather than evidence of brain decoding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the noise-input control in Table 3, but that control does not eliminate a serious confound: the LLM rescorer/in-filler (Llama 3.2-1B and Claude 3.7 Sonnet) likely has the public-domain Sherlock Holmes stimulus in its training data. Appendix L shows the LLM is explicitly prompted to complete and in-fill the exact predicted sequences from the decoded test passages. If the MEG-based word classifier produces even a few correct anchor words from a memorized passage, the LLM can regenerate the surrounding narrative from memory, whereas noise inputs provide no such anchors. The observed real-vs-noise gap (BLEU .25 vs .15, Table 3) therefore conflates genuine neural information with memorization-amplified reconstruction, and the paper provides no text-only/no-signal baseline and no training-data overlap analysis. Appendix J discusses decoding performance but omits this risk. Until the LLM prior is controlled independently of the brain-derived signal, the claim that the decoded text carries above-chance, brain-derived information is not fully secured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a non-invasive brain-to-text (B2T) pipeline that combines MEG/EEG word classification with LLM-based beam rescoring and predictive in-filling, together with a selective dataset-pooling strategy. On the LibriBrain MEG dataset of a subject listening to Sherlock Holmes audiobooks, the method reports BLEU 0.25 with real inputs versus 0.15 for noise inputs and 0.07 for random word selection, and claims to be the first non-invasive B2T method to surpass all critical chance baselines across standard metrics. Additional claims include a 2.1--2.3x improvement in word classification accuracy from dataset pooling and an open-vocabulary capability via an XGBoost OOV detector (AUROC 0.88). The manuscript includes detailed ablations, random baselines, session-independent train/test splits to avoid nonsense correlations, and extensive appendices describing prompts, hyperparameters, and decoding examples.","tokens_in":22365,"tokens_out":9701,"duration_ms":93582,"significance":"If the central claim holds, this is a meaningful milestone for non-invasive BCI research: it would demonstrate that word-level neural decoding combined with language-model priors can exceed no-signal baselines, and that heterogenous MEG/EEG datasets can be pooled to improve performance. The paper is unusually careful on several fronts: it provides noise-input and random-selection baselines, uses session-independent splits, reports standard errors over seeds, and explicitly addresses the teacher-forcing failure mode identified by Jo et al. The dataset-pooling analysis, with standalone performance as a quality measure, is a potentially useful contribution in its own right. However, the strength of the headline attribution claim is tempered by the LLM memorization risk and by cross-protocol comparisons in the main table.","major_comments":[{"comment":"The noise-input control is a good baseline, but it does not fully address the possibility that the performance gap is amplified by LLM memorization of the public-domain Sherlock Holmes stimulus. Llama 3.2-1B and Claude 3.7 Sonnet are likely to have seen this widely reproduced text during training, and Appendix L shows that the LLM is explicitly prompted to complete and in-fill the exact predicted sequences, including <UNK> positions. Even though the real-vs-noise comparison in Table 3 holds the LLM constant, a few correct anchor words from the brain decoder could trigger the LLM to reconstruct the surrounding narrative from memory, thereby inflating BLEU and other n-gram metrics. To support the statement in Section 4.1 that \"the results are attributable to decoding brain data,\" please add (i) a text-only baseline that runs the same beam search and in-filling with no informative neural input (e.g., an encoder whose outputs are replaced by a constant or by an uninformative distribution), and (ii) an analysis of whether the test stories appear in the training corpora of the two LLMs. The concern is partially mitigated by the observation that in-filling improves noise and real inputs by similar absolute amounts (noise +0.05, real +0.03 BLEU) and by the non-verbatim decoded examples in Appendix M, but the suggested experiments would make the attribution claim rigorous.","section":"Section 4.1, Table 3, Appendix L"},{"comment":"The headline claim that BLEU is raised by 1.4--2.6x over prior work rests on cross-protocol comparisons that mix fMRI, EEG, and MEG systems with different stimuli, vocabularies, and evaluation protocols. For example, Tang et al. is an fMRI semantic reconstruction system rather than a word-level decoder, so a direct BLEU comparison is not apples-to-apples, and the \"delta to random\" values are computed relative to each method's own random baseline, which is not a common scale. Please restrict the state-of-the-art claim to the matched MEG comparisons in Table 2, or clearly qualify the cross-protocol comparison in the abstract and in Section 4.1. Without this qualification, the quantitative \"up to 2.6x\" claim overstates the evidence.","section":"Table 1 and Abstract"}],"minor_comments":[{"comment":"The abstract and Section 4.1 state p<<.001 for the comparisons in Table 3, but no test statistics or p-values are reported anywhere in the manuscript. Please report the actual p-values and state the test used (e.g., one-tailed or two-tailed Welch's t-test).","section":"Section 4.1 / Appendix G"},{"comment":"The \"Ours (best)\" row in Table 1 appears to combine the best metric from different model configurations (e.g., greedy for some metrics, beam+fill for others). Please state explicitly which configuration produced each entry, or report a single configuration consistently.","section":"Section 4.3 / Table 1"},{"comment":"The reported correlation r=.95, p=.048 between standalone performance and improvement from joint training appears to be based on a very small number of datasets. Please state the number of points, the correlation type (Pearson or Spearman), and discuss the fragility of this result.","section":"Figure 3A"},{"comment":"The in-context transcription uses Claude 3.7 Sonnet through an API with 4096 thinking tokens. This introduces non-determinism and reproducibility concerns. Please provide the exact model version, inference temperature, and date of access, or consider using an open-weight model with fixed checkpoints.","section":"Appendix G, LLM APIs"},{"comment":"The toy example for <UNK> insertion is useful, but the surrounding text contains a minor typo: \"we include or exclude <UNK>\" should read \"we include or exclude the <UNK> token.\"","section":"Appendix F"}],"recommendation":"major_revision","confidential_remarks":"This is a strong and timely paper with unusually careful baselining. The main risk is that reviewers will focus on the LLM memorization issue; a text-only baseline and a training-data overlap analysis would substantially strengthen the attribution claim. I would also encourage the authors to soften the cross-protocol SOTA language in the abstract and Table 1, since the matched MEG comparisons in Table 2 are already convincing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is the first non-invasive B2T paper I've seen that runs the right controls, and the real-vs-noise gap survives; but the headline attribution to brain data is not yet secured because the LLM prior may be doing more of the work than the numbers suggest. Deserves a serious referee, not acceptance as-is.\n\nWhat's actually new: the OOV in-filling strategy and the quality-based selective pooling rule. The rest—contrastive word classification, transformer context encoding, LLM beam-search rescoring—is known machinery applied to a new setting. That's fine; the paper says so. The genuinely good part is the experimental discipline. Session-based splits address nonsense correlations, two random baselines are defined and used, and the external Gwilliams evaluation gives some independent grounding. Appendix F even handles the WER alignment issue with <UNK> tokens and random-fill baselines. That is real care, and it puts this paper above most of the non-invasive decoding literature.\n\nThe stress-test concern holds up. The stimulus is a public-domain Sherlock Holmes audiobook, and Appendix L shows the LLM is asked to complete and in-fill exactly these predicted sequences. Llama 3.2-1B and Claude 3.7 Sonnet have almost certainly seen this text. The noise-input control from Table 3 is a good control, but it is not a text-only baseline: with noise inputs the LLM gets no anchors; with real inputs it gets a few correct words and can regrow the story from memory. The observed gap (BLEU .25 vs .15) therefore conflates brain-derived signal with memorization-amplified reconstruction. The paper does not report a no-signal/text-only condition or any training-data overlap analysis. This is a load-bearing gap: the central claim is that decoded text carries above-chance brain-derived information.\n\nTwo smaller issues. Table 1's 'Ours best' cherry-picks across decoding variants and mixes unmatched fMRI/EEG/MEG protocols; less important than the LLM issue but should be tied to a fixed pipeline. And no code or data are released; for a claim this strong, that matters. The citation pattern is otherwise honest—prior MEG/EEG decoding work is credited, and the controls from Jo et al. are central.\n\nWho this is for: people working on speech decoding BCIs and anyone who evaluates non-invasive decoding claims. The paper sets a validation standard—random selection and noise-input controls—that the field should adopt.\n\nRecommendation: send it to peer review. The work is serious, the flaws are specific and fixable, and the fix (text-only baseline, memorization check, fixed pipeline for Table 1) is clear. I'd want to see that before accepting, but this deserves referee time, not a desk reject.","headline":"Careful controls make the above-chance claim plausible, but the missing text-only baseline leaves the LLM-memorization confound unresolved; worth peer review with that control required.","tokens_in":22911,"tokens_out":3443,"would_cite":true,"duration_ms":33378,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that non-invasive brain recordings can be decoded into text at levels that beat all chance baselines, for the first time.","keywords":["brain-to-text","non-invasive BCI","MEG decoding","speech decoding","LLM rescoring","dataset pooling","chance baselines"],"falsifier":"Take the exact beam search and in-filling recipe but feed it the stimulus text with no brain data, letting the language model decode from its own priors; if that text-only BLEU approaches the real-input value (0.25) instead of the noise-input value (0.15), the claimed brain contribution is largely language-model memory. A cleaner version is to run the full model on a newly written story that could not be in any language model's training data and check whether the real-versus-noise gap survives.","tokens_in":21930,"feed_emoji":"🧠","tokens_out":11971,"duration_ms":113501,"temperature":0.7,"pith_summary":"This paper claims that heard speech can be transcribed from non-invasive MEG recordings at a level that beats both random word selection and noise-input baselines on every standard metric. If correct, it removes a key obstacle to brain-computer interfaces that restore communication without surgery, since prior non-invasive attempts had never convincingly passed those baselines. The route is to turn a closed-vocabulary word classifier into a sentence decoder: a language model rescores beam-search candidates, a detector finds words outside the vocabulary, and an in-filling language model fills them using context. On the LibriBrain test set the best configuration reaches BLEU 0.25 against 0.07 for random selection, and selective pooling of datasets more than doubles word classification accuracy.","feed_headline":"First non-invasive brain-to-text passes every chance baseline","feed_subtitle":"A word classifier plus LLM rescoring lifts BLEU to 2.6x prior work, opening a non-surgical path to speech brain-computer interfaces.","key_machinery":"The load-bearing mechanism is a beam search that scores each partial sentence as the sum of the neural word probability $\\log P_{\\textrm{model}}(w_i\\mid x)$ and a weighted language-model prior $\\lambda \\log P_{\\textrm{LLM}}(w_{1:i})$, turning a single-word classifier into a sequence decoder. Out-of-vocabulary words are detected by a gradient-boosted classifier fed with softmax-distribution statistics (the paper reports 88% AUROC) and are then filled in by a language model using surrounding context; a second in-context variant prompts the language model to both reselect from the top-five word probabilities and fill unknown positions. Cross-dataset training is made possible by a spatial-attention projection that maps heterogeneous sensor arrays into one latent space, with dataset selection guided by each dataset's standalone accuracy.","core_discovery":"The central claim is that a non-invasive electrophysiological brain-to-text system can exceed chance for the first time. The system decodes words from MEG segments aligned to word onsets, scores candidate sentences with a language-model beam search, and fills out-of-vocabulary positions from context; against a random-word baseline it reports BLEU 0.25 versus 0.07, with every metric significantly above both random selection and noise inputs ($p<0.001$). The paper also claims that pooling datasets by using standalone performance as a quality measure improves word classification accuracy by 2.1–2.3× and that these methods surpass all previously reported non-invasive approaches, with BLEU improved up to 2.6× over prior work. The claim is explicitly about heard speech with word-onset alignment, not inner or attempted speech.","pith_inferences":["The paper never reports a text-only baseline, so how much of the 0.25-versus-0.15 gap is language-model predictability rather than brain signal remains an open test.","If the result survives that test, the same rescoring-and-in-filling pipeline is a natural candidate for attempted or inner speech once word onsets can be obtained, though the paper does not claim this.","The pooling rule suggests a practical selection criterion for any future dataset: pool with a dataset only when its standalone accuracy is high, and verify with a deliberately low-quality dataset that no improvement follows.","For assistive communication, exact transcription may be the wrong bar; even at high word-error rates the decoded text appears to retain semantic structure, so a metric focused on usable meaning may be more clinically relevant."],"forward_implications":["Any future non-invasive B2T claim will need to be validated against both random-word and noise-input baselines; this paper's ablation table establishes that bar.","Word-level classifiers that already exist can be upgraded to sequence decoders through language-model rescoring, so progress in word decoding translates directly to text decoding.","Out-of-vocabulary words need not be retrained into the classifier: an uncertainty-based detector plus language-model in-filling expands the usable vocabulary.","Datasets can be combined by ranking them on standalone quality, so new recordings can be added to the training pool without hand-designed harmonisation.","The scaling behavior reported in the paper implies that current MEG datasets are still data-limited and that collecting more data should keep improving accuracy."],"supporting_citations":[{"why":"The word-level classification model that this work extends; supplies the transformer context encoder and contrastive training setup.","marker":"[9]"},{"why":"The chance-baseline critique that showed prior EEG-to-text results collapsed under noise-input controls; sets the baselines the paper must beat.","marker":"[15]"},{"why":"Provides the spatial-attention sensor projection and the contrastive speech-MEG matching objective used in the signal encoder.","marker":"[8]"},{"why":"The 50-hour MEG corpus used as the main training and test dataset.","marker":"[35]"},{"why":"The auxiliary MEG narrative dataset pooled with LibriBrain to demonstrate the 2.1–2.3× accuracy gains.","marker":"[36]"},{"why":"The prior fMRI semantic decoding benchmark that the paper compares against and improves upon on most metrics.","marker":"[21]"},{"why":"A prior MEG brain-to-text model whose reported random baseline and results are used as comparison points.","marker":"[14]"},{"why":"The language model whose log-probabilities are added to the beam search score for rescoring.","marker":"[30]"},{"why":"The instruction-following language model used for in-context in-filling and transcription.","marker":"[33]"}],"fun_headline_variants":["Non-invasive brain-to-text beats chance for first time","Brain-to-text from MEG tops random baseline, no surgery","LLM rescoring unlocks non-invasive brain-to-text 2.6x","First non-invasive speech BCI surpasses chance metrics","Word classifier plus LLM rescoring beats chance in B2T"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The noise-input control is only valid if the language models used for rescoring and in-filling cannot themselves predict the test passages, which are public-domain Sherlock Holmes stories that are likely present in the models' training data.","fun_headline_variants_meta":{"raw":{"variants":["Non-invasive brain-to-text beats chance for first time","Brain-to-text from MEG tops random baseline, no surgery","LLM rescoring unlocks non-invasive brain-to-text 2.6x","First non-invasive speech BCI surpasses chance metrics","Word classifier plus LLM rescoring beats chance in B2T"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1322,"prompt_tokens":948,"completion_tokens":374,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":288}},"tokens_in":564,"tokens_out":374,"duration_ms":3630,"temperature":1.0,"reasoning_tokens":288,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:14:20.411519+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the exact beam search and in-filling recipe but feed it the stimulus text with no brain data, letting the language model decode from its own priors; if that text-only BLEU approaches the real-input value (0.25) instead of the noise-input value (0.15), the claimed brain contribution is largely language-model memory. A cleaner version is to run the full model on a newly written story that could not be in any language model's training data and check whether the real-versus-noise gap survives.","supporting_citations":[{"cited_title":"Are EEG- to-text models working? The Fourth International Workshop on Human Brain and Artificial Intelligence at IJCAI, 2024","cited_arxiv_id":null,"evidence_quote":"The chance-baseline critique that showed prior EEG-to-text results collapsed under noise-input controls; sets the baselines the paper must beat."},{"cited_title":"LibriBrain: Over 50 hours of within-subject MEG to improve speech decoding methods at scale","cited_arxiv_id":null,"evidence_quote":"The 50-hour MEG corpus used as the main training and test dataset."},{"cited_title":"A 10-hour within-participant magnetoencephalography narrative dataset to test models of language com- prehension","cited_arxiv_id":null,"evidence_quote":"The auxiliary MEG narrative dataset pooled with LibriBrain to demonstrate the 2.1–2.3× accuracy gains."},{"cited_title":"Semantic reconstruction of continuous language from non-invasive brain recordings","cited_arxiv_id":null,"evidence_quote":"The prior fMRI semantic decoding benchmark that the paper compares against and improves upon on most metrics."},{"cited_title":"meta-llama/Llama-3.2-1B","cited_arxiv_id":null,"evidence_quote":"The language model whose log-probabilities are added to the beam search score for rescoring."},{"cited_title":"Claude 3.7 Sonnet and Claude Code","cited_arxiv_id":null,"evidence_quote":"The instruction-following language model used for in-context in-filling and transcription."}],"review_version":1}