{"id":"a7985043-40d5-4c85-bfc2-cf51a874e968","arxiv_id":"2501.03246","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Text embeddings from CLIP and GPT-2 predict MEG responses to spoken stories better than audio features, with text effects strongest over frontal sensors and audio effects over lateral temporal sensors.","lead":"This paper compares AI systems that predict brain responses to spoken stories, recorded with MEG in eight listeners. Text-based models (CLIP, GPT-2) predicted frontal brain activity better than audio-based models, but the text models received more context, so the headline result needs a controlled follow-up.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Text-vs-audio superiority is confounded: text inputs include 20 words of past context while audio inputs are limited to the 3-second window, so the frontal text advantage may reflect information asymmetry rather than distinct neural pathways.","rationale":"The reader's weakest assumption identifies exactly the load-bearing concern: the text and audio pipelines are not informationally comparable because text inputs include 20 preceding words while audio inputs are only the 3-second stimulus. This directly threatens the paper's highest-level conclusion, not a secondary detail. The paper has independent support in the form of public code and data, held-out ridge regression, and permutation-based nulls, so the issue is not lack of evidence that the models predict MEG above chance. The issue is that the comparative claim in the abstract cannot be evaluated without matching the information available to each model. The reader's CONDITIONAL verdict already reflects this; my read does not change that verdict. The additional concerns about sensor-level Broca attribution and p-value resolution are real but secondary; the context asymmetry is the primary reason the headline claim should not be accepted as currently stated.","tokens_in":8589,"tokens_out":3148,"duration_ms":30602,"concrete_test":"Run an information-matched control: repeat the audio encoding pipeline using wav2vec2 (or TFD) embeddings computed over a window that includes the same 20 preceding words of audio context as the text condition, and also run the text pipeline restricted to the 5 words inside the 3-second window. Then recompute sensor-wise Pearson correlations and compare frontal 8-30 Hz values. If the text advantage disappears or reverses under matched context, the central claim is unsupported; if it survives, the confound is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that text embeddings outperform audio embeddings and that CLIP/GPT-2 preferentially engage frontal language regions (Broca's area) in the 8-30 Hz range. For this comparison to be interpretable, the two input representations must be informationally comparable. They are not. Section 3.3 constructs each text sequence from 25 words: the onset word, the 5 following words that correspond to the 3-second MEG window, and 20 preceding words of past context. Section 3.2 computes audio features only from the 3-second stimulus window, with no preceding context. Thus wav2vec2/TFD and CLIP/GPT-2 differ not only in modality but also in the amount of available context. The larger frontal Pearson correlation for text models could simply reflect the extra 20 words of linguistic context, which may help predict slow (8-30 Hz) frontal activity, rather than a fundamental auditory-versus-linguistic pathway difference. Moreover, the PC values in Table 1 are point estimates with no direct model-versus-model significance test; only each model is tested against a permutation null. So the headline claim rests on an assumption that has not been checked and that the methods as described violate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MEG encoding models for spoken language, comparing audio-based features (STFT time-frequency decompositions and wav2vec2 embeddings) with text-based features (CLIP and GPT-2 embeddings) in predicting MEG time-frequency responses. Using the MEG-MASC dataset with 8 subjects, the authors report that text-based models achieve higher Pearson correlation than audio-based models, and that the two families of features engage different spatial patterns (temporal versus frontal), particularly in an 8–30 Hz range. The evaluation uses held-out data and permutation-based significance testing against zero correlation. The manuscript's central claim is that text feature spaces carry more information about frontal language-region MEG activity than audio features, implying distinct neural pathways for auditory and linguistic processing.","tokens_in":8758,"tokens_out":2733,"duration_ms":26991,"significance":"If the central comparison were valid, this would be a useful contribution to MEG-based language encoding, extending prior fMRI work to a modality with higher temporal resolution. The paper's strengths include subject-wise held-out evaluation, permutation-based null distributions, and publicly available data and code. However, the headline claim that text embeddings outperform audio embeddings is not currently supported because the two input pipelines are informationally asymmetric, and no direct model-versus-model significance test is provided. The spatial and frequency-band conclusions rest on visual inspection of topographies and post-hoc band selection rather than quantitative contrasts. These issues are load-bearing for the paper's main conclusions, so the result is presently conditional rather than established.","major_comments":[{"comment":"The text-versus-audio comparison is confounded by unequal context. Text sequences are constructed from 25 words: the onset word, the 5 following words that correspond to the 3-second MEG window, and 20 preceding words of past context (Section 3.3). Audio features, by contrast, are computed only from the 3-second stimulus window with no preceding context (Section 3.2). Thus the higher Pearson correlation for CLIP and GPT-2 could reflect the additional 20 words of linguistic context rather than a fundamental advantage of textual representations. To support the claim that text outperforms audio, the authors should provide audio models with matched context (e.g., audio from the preceding words) or ablate the past-context component of the text models.","section":"Sections 3.2–3.3 and Table 1"},{"comment":"The paper does not test whether text and audio models differ significantly from each other; the permutation test only compares each model's predictions against a null of zero correlation. The abstract's statement that 'the text-to-MEG model outperforms the audio-based model' requires a paired statistical comparison (e.g., bootstrap or permutation of model labels across subjects and sensors) with appropriate correction for multiple comparisons. Without such a test, the observed difference in mean PC could be within sampling variability, especially given the large standard deviations reported in Table 1.","section":"Section 3.5 and Table 1"},{"comment":"The claims that textual embeddings 'primarily engage the frontal cortex, particularly Broca's area' and that this is strongest in the 8–30 Hz range are based on visual inspection of topomaps and aggregate correlation values rather than on a quantitative region-of-interest or source-localization analysis. The 8–30 Hz band appears to be identified post hoc after examining multiple frequency bands, and no statistical contrast is provided to show that frontal correlations are significantly greater for text than for audio, or that the 8–30 Hz effect is significantly stronger than in other bands. This limits the support for the proposed distinct-neural-pathway interpretation.","section":"Abstract, Section 5.1, and Figures 2–3"}],"minor_comments":[{"comment":"The text refers to a 0.5–30 Hz bandpass filter but later calls the full spectrum '0–30 Hz'; the discrepancy in the lower bound should be clarified.","section":"Section 3.1"},{"comment":"The ridge regression objective is written with 'min' but the norm notation is inconsistent; the equation should be typeset carefully to distinguish the loss being minimized from the model prediction.","section":"Section 3.4"},{"comment":"The R² values are extremely small (on the order of 10^-4); the paper would benefit from an explicit statement of whether these are sensor-averaged values and how they relate to the Pearson correlation values, since both are reported.","section":"Table 1"},{"comment":"The permutation test uses 100 repetitions, which is a relatively small number for stable empirical P-values at the 0.05 level; the authors should report the resolution of the null distribution or use more permutations.","section":"Section 3.5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of the journal and addresses a topic of interest. However, the central claim rests on a confounded comparison and missing model-to-model statistics. The issues are fixable with a re-analysis (e.g., adding context-matched audio or ablating text context) and a direct significance test, so I would not reject outright, but I would not accept in the current form. I would encourage the editor to request the matched-context analysis before further consideration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a real held-out MEG encoding study with public code and data, and the four-way feature comparison (TFD, wav2vec2, CLIP, GPT-2) on MEG-MASC is new in that combination. The paper is not broken, but the headline claim—text embeddings beat audio embeddings and preferentially engage Broca’s area—doesn’t survive close reading. The text pipeline gets 25 words (20 preceding context + onset + 5 following), while the audio pipeline only gets the 3-second stimulus window. So the text models have strictly more information. That alone could explain the higher frontal Pearson correlation. No model-vs-model significance test is run; everything is tested against a permutation null, so we only know each model beats chance. The reported p-values like 5.2e-3, 3.2e-3, and 2.9e-3 are below what 100 permutations can resolve (minimum ~0.01), so the precision is illusory.\n\nWhat the paper does well: evaluation is genuinely held-out, ridge regression with cross-validated lambda, permutation null, and the code/data links are in place. Effect sizes are tiny (PC 2–6e-3, R2 ~1e-4), which is typical for sensor-level MEG encoding, but the topographic patterns are visually consistent across frequency bands. The descriptive observation that audio features track lateral temporal regions while text features track frontal sites is plausible, though naming Broca’s area from sensor-level topographies is overreach.\n\nTwo smaller issues: the Discussion (Section 5.1) says the Text model shows “slightly reduced intensity compared to the Audio model” and later says the “direct audio model shows more intense correlations in frontal areas,” which contradicts Table 1 where CLIP/GPT-2 have higher global PC. And the abstract’s 8–30 Hz emphasis looks like post-hoc selection of the band where the contrast is strongest.\n\nBottom line: the central comparative claim is confounded and the statistical support is thinner than reported. But these are fixable. A matched-context audio condition, pairwise model-vs-model significance tests, and source-level analysis would make this a useful paper. It deserves a serious referee to push for those revisions, though in current form I would not cite the main claim. It would be a good reading-group example of how feature-input asymmetries can drive encoding results.","headline":"Held-out MEG encoding comparison with real code/data, but the text-beats-audio headline rests on a context-length confound and no model-vs-model significance test.","tokens_in":9409,"tokens_out":3443,"would_cite":false,"duration_ms":29377,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Text embeddings predict frontal MEG activity better than audio features, revealing separate neural routes for sound and meaning.","keywords":["MEG encoding","auditory perception","language comprehension","ridge regression","wav2vec2","CLIP","GPT-2","neural pathways"],"falsifier":"A concrete experiment: build an audio-to-MEG model that also receives the 20-second preceding audio context (or all 25 words rendered as speech) and compare its Pearson correlation to the text model on the same MEG windows; if the audio model then matches or exceeds the text model, the claim of separate pathways is falsified.","tokens_in":1419,"feed_emoji":"🧠","tokens_out":3955,"duration_ms":47485,"temperature":0.7,"pith_summary":"This paper asks whether the brain's response to spoken stories is better predicted by the acoustic properties of the speech or by the linguistic content of the words. Using MEG recordings from eight subjects listening to naturalistic stories, the authors build two families of encoding models: audio-to-MEG models fed with time-frequency decompositions or wav2vec2 speech features, and text-to-MEG models fed with CLIP or GPT-2 word embeddings. Both families predict neural activity significantly above chance, but the text-based models achieve higher Pearson correlations with the observed MEG signals. The spatial patterns also diverge: audio features track lateral temporal regions tied to auditory processing, while text features engage frontal cortex, including Broca's area, particularly in the 8–30 Hz range. If correct, this supports a functional separation between sensory auditory processing and higher-order linguistic encoding, and it demonstrates that MEG encoding models can quantify that separation at millisecond resolution.","feed_headline":"Text beats audio at predicting brain's language response","feed_subtitle":"MEG encoding shows word meanings drive frontal activity while sound drives temporal regions.","key_machinery":"The central objects are two encoding pipelines that both terminate in a ridge regression layer mapping stimulus embeddings to MEG time-frequency decompositions. The audio pipeline uses either short-time Fourier transform time-frequency decompositions or wav2vec2 latent representations from the 3-second speech window. The text pipeline uses CLIP or GPT-2 embeddings of a 25-word sequence (20 preceding words, the onset word, and 5 following words) to predict the same MEG window. The comparison metric is the Pearson correlation between real and predicted flattened spectrograms, evaluated per sensor and frequency band, with statistical significance assessed by permutation-based null distributions and Z-scores. The load-bearing comparison is the difference in predictive accuracy between the two feature families, together with the topographical separation of where each family's predictions correlate with the data.","core_discovery":"The central claim is that in a direct comparison on the same MEG dataset, linguistic embeddings carry more predictive information about brain responses to spoken language than acoustic embeddings do, and that the two feature types map onto anatomically distinct networks. Concretely, the text-to-MEG models (CLIP and GPT-2) outperform the audio-to-MEG models (TFD and wav2vec2) on global Pearson Correlation averaged over all sensors and frequency bands, with the largest advantage in the alpha and beta bands. Spatially, the audio models concentrate their highest correlations in lateral temporal areas, consistent with primary and associative auditory processing, whereas the text models extend strong correlations into frontal regions, especially left-lateralized language areas such as Broca's area. The paper interprets this as evidence that auditory stimuli travel through direct sensory pathways while linguistic information is encoded by networks that integrate meaning and cognitive control, and that these pathways are distinguishable with MEG at the level of single-word temporal windows.","pith_inferences":["A direct test the authors did not run would be to match the information available to both pipelines—for example, feeding the audio model a 3-second window preceded by 20 seconds of audio context, or feeding the text model only the 5 words inside the window—to see whether the text advantage shrinks or disappears when context length is equalized.","The same dataset could be used to ask whether a multimodal model that concatenates audio and text embeddings improves prediction over either alone; the paper's topography suggests such a model might show additive temporal and frontal contributions.","The stronger text-model performance in Broca's area could be probed further by using lesion or clinical data from patients with frontal language damage, testing whether the text-to-MEG correlation drops selectively.","The authors suggest future work with larger audio or text pre-trained models; a concrete extension would be to test whether hierarchical transformers that encode longer discourse (beyond 25 words) shift predictive power toward prefrontal or parietal regions associated with discourse-level integration."],"forward_implications":["If text embeddings outperform audio features for frontal MEG prediction, then word-level linguistic content is a stronger driver of frontal language-region activity than the acoustic waveform alone, at least in the 8–30 Hz range.","The topographical split (temporal for audio, frontal for text) offers a quantitative, MEG-based signature that could separate sensory speech processing from semantic integration without needing invasive recordings.","The ridge-regression encoding framework on naturalistic story listening can be extended to other transformer embeddings, potentially tracking how model size or training objective shifts the predicted neural topography.","The significant Z-scores (ranging up to about 8.1 for GPT-2) indicate the observed correlations are not due to chance, supporting the use of MEG encoding as a reliable assay for language-comprehension studies.","The frequency-band analysis suggests that the text advantage is not uniform: it is strongest in alpha and beta bands, implying that semantic encoding is carried by faster oscillatory activity over the frontal cortex."],"supporting_citations":[{"why":"Supplies the MEG-MASC dataset of naturalistic spoken stories used for all encoding experiments.","marker":"[Gwilliams et al., 2023]"},{"why":"Establishes the word-context MEG encoding paradigm and the ridge-regression evaluation procedure that this paper adapts.","marker":"[Oota et al., 2023]"},{"why":"Provides the wav2vec2 speech representation used as one of the two audio feature families.","marker":"[Baevski et al., 2020]"},{"why":"Provides the CLIP text encoder whose embeddings are used in the text-to-MEG pipeline.","marker":"[Radford et al., 2021]"},{"why":"Provides the GPT-2 text encoder, the other text feature family, which achieves the highest global Pearson correlation.","marker":"[Radford et al., 2019]"},{"why":"Demonstrates that wav2vec2 latents can be aligned with MEG via contrastive learning, motivating the use of speech latents for brain-signal prediction.","marker":"[Défossez et al., 2023]"}],"fun_headline_variants":["Linguistic embeddings beat acoustic in MEG","Text wins over audio in predicting brain speech","Meaning tops sound in MEG language encoding","Frontal text beats temporal audio in MEG","MEG shows text outperforms audio for language"],"cache_read_input_tokens":11392,"weakest_assumption_plain":"The load-bearing premise is that the text and audio pipelines are informationally comparable, so the higher text-model correlation can be attributed to linguistic content rather than to the text models having access to more context (20 preceding and 5 following words vs. just the 3-second audio window).","fun_headline_variants_meta":{"raw":{"variants":["Linguistic embeddings beat acoustic in MEG","Text wins over audio in predicting brain speech","Meaning tops sound in MEG language encoding","Frontal text beats temporal audio in MEG","MEG shows text outperforms audio for language"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000368,"raw_usage":{"total_tokens":2010,"prompt_tokens":1017,"completion_tokens":993,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":633,"completion_tokens_details":{"reasoning_tokens":924}},"tokens_in":633,"tokens_out":993,"duration_ms":9352,"temperature":1.0,"reasoning_tokens":924,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:46:04.063707+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete experiment: build an audio-to-MEG model that also receives the 20-second preceding audio context (or all 25 words rendered as speech) and compare its Pearson correlation to the text model on the same MEG windows; if the audio model then matches or exceeds the text model, the claim of separate pathways is falsified.","supporting_citations":[{"cited_title":"Introducing meg-masc a high-quality magneto-encephalography dataset for evaluating natural speech processing","cited_arxiv_id":null,"evidence_quote":"Supplies the MEG-MASC dataset of naturalistic spoken stories used for all encoding experiments."},{"cited_title":"Meg encoding using word context semantics in listening stories","cited_arxiv_id":null,"evidence_quote":"Establishes the word-context MEG encoding paradigm and the ridge-regression evaluation procedure that this paper adapts."}],"review_version":1}