Pith. sign in

REVIEW 1 major objections 6 minor 1 cited by

LibriBrain: Over 50 Hours of Within-Subject MEG to Improve Speech Decoding Methods at Scale

T0 review · 1 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new open dataset records over 50 hours of MEG from one person listening to Sherlock Holmes audiobooks, making it the largest single-subject MEG speech corpus to date and providing reproducible benchmarks for speech detection, phoneme…

desk verdict A solid, genuinely useful dataset release; the 50-hour claim holds up, and the forced-alignment label validation is the one real caveat. read the letter →

arxiv 2506.02098 v1 pith:S63VMP73 submitted 2025-06-02 cs.LG

Miran Özdogan , Gilad Landau , Gereon Elvers , Dulhan Jayalath , Pratik Somaiya , Francesco Mantegna , Mark Woolrich , Oiwi Parker Jones This is my paper
classification cs.LG
keywords MEGspeechdecodingsingle-subjectdatasetbrain-computerinterfacephonemeclassificationworddetectionwithin-subjectscaling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces LibriBrain, an open magnetoencephalography dataset of over 50 hours of recordings from a single participant listening to audiobooks, and claims it is the largest single-subject MEG dataset for speech decoding to date—about five times larger than the next comparable resource and tens of times larger than most. The authors' central purpose is to give machine-learning researchers a deep, within-subject resource that supports scaling studies in non-invasive speech decoding, together with standard train/validation/test splits, a Python data loader, and reproducible baselines. The baselines show that decoders trained on the data beat chance on speech detection, phoneme classification, and word classification, and that performance grows roughly logarithmically as training data increases. If the central claim is right, the field gains a public benchmark that removes a major bottleneck: most non-invasive speech datasets give only one to two hours per subject, which is too shallow to train strong decoders.

What carries the argument

The carrying object is the dataset's within-subject depth: 95 recording sessions from one participant, the same narrator and genre throughout, 306 sensor channels downsampled to 250 Hz with 4 ms samples, and event files giving word and phoneme onsets produced by an automatic forced aligner followed by extensive manual correction. This depth lets a single model train on 51.57 hours of data without cross-subject variability, and the fixed splits—with validation and test sessions recorded on a separate day from all training data—make the benchmark resistant to session-level information leakage. A second mechanism is phoneme averaging: the paper shows that averaging repeated MEG windows for the same phoneme before classification raises balanced accuracy above 60% at 100 repetitions, which simulates a realistic brain-computer interface usage pattern.

What would settle it

Take a held-out subset of LibriBrain audio, have a trained phonetician mark phoneme boundaries by hand on a few thousand tokens, and compare those boundaries to the released event files; if the median absolute onset disagreement is comparable to or larger than the mean phoneme duration (about 80 ms) or is systematically different across phoneme classes, then the phoneme classification benchmark is measuring alignment artifacts rather than neural speech decoding.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a single participant can be recorded for 52.32 hours of naturalistic listening and that the resulting brain responses support supervised decoding of speech presence, phoneme identity, and word identity, with performance improving roughly logarithmically as training data increases. The paper reports 306-channel MEG recordings from one native English speaker listening to seven Sherlock Holmes audiobooks read by a single narrator, yielding 466,230 word events and over 1.5 million phoneme events aligned to the audio. On the word classification task, the replicated state-of-the-art model reaches top-10 balanced accuracy of 0.3621 on LibriBrain versus 0.3261 on the next-largest within-subject dataset at matched data volume, and the scaling curves show LibriBrain continuing to improve where the comparison dataset plateaus. The dataset, splits, loaders, and baselines are released so that these results can be reproduced and extended.

Load-bearing premise

The phoneme and word labels come from an automatic forced aligner corrected by hand, and the paper only checks that longer linguistic units receive longer duration annotations, not that the onset times are accurate enough for 39-way phoneme classification at the millisecond scale.

Editorial extensions

If this is right

  • Speech detection, phoneme classification, and word classification all beat chance on the held-out sessions, so the dataset demonstrably carries decodable speech information at the single-trial level.
  • Decoding performance rises roughly logarithmically as training data grows from 5 to 50 hours, so further within-subject scaling is the most direct route to better non-invasive decoders.
  • Averaging repeated MEG windows of the same phoneme pushes balanced accuracy above 60% at 100 repetitions, which simulates a practical BCI setting where users repeat inputs.
  • The fixed train/validation/test splits, with validation and test recorded on a separate day, give the community a leakage-resistant standard for comparing methods.
  • Word classification on LibriBrain outperforms the next-largest dataset at matched training volumes, supporting the claim that depth, not breadth, drives decoding progress.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the logarithmic scaling law continues past 50 hours, collecting more hours from the same participant should yield further gains; a natural extension is to fit the scaling curve and predict performance at 100 hours before collecting them.
  • Because the data come from one narrator and one genre, the paper does not establish how models trained on LibriBrain transfer to other voices or speaking styles; a testable extension is fine-tuning on a second narrator and measuring the drop.
  • The manual correction effort suggests label quality above typical forced alignment, but the paper never quantifies onset error; an independent hand-labelled subset would settle whether phoneme baselines reflect neural signals or alignment noise.
  • The separate-day holdout design means the benchmark includes day-to-day variability; this makes the baselines conservative, and models that explicitly handle session effects might do better than the reported numbers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 6 minor

Summary. The paper introduces LibriBrain, a single-subject MEG dataset of over 50 hours of recordings from one participant listening to Sherlock Holmes audiobooks, together with word- and phoneme-level annotations, a Python API, standardized train/validation/test splits, and baseline results for speech detection, phoneme classification, and word classification. The headline claim is that LibriBrain is the largest single-subject MEG dataset for speech decoding, approximately 5 times larger than the next comparable dataset (Armeni et al., 2022) and 50 times larger than most others. The experiments show statistically significant decoding above random baselines and performance that improves with training data volume, and the word classification results are reproduced with permission from a companion preprint.

Significance. If the resource claim holds, LibriBrain is a valuable community asset: it provides unprecedented per-subject MEG depth for naturalistic speech, enabling scaling studies and more reliable non-invasive speech decoding benchmarks. The paper's strengths include public data and code release, a documented preprocessing pipeline, standard splits to prevent leakage, exact permutation tests against baselines, and explicit reporting of computational requirements. The scaling results, while preliminary, provide a useful reference point. The main caveat—that phoneme/word labels rest on forced alignment validated only by a duration-monotonicity check—does not undermine the central dataset-size claim but does affect the strength of the benchmark contributions.

major comments (1)
  1. [Appendix B.5, Table 8] The validation of the forced-alignment annotations consists solely of a duration-monotonicity check: longer linguistic units receive longer durations. Because the phoneme and word classification baselines in Sections 4.2 and 4.3 depend on millisecond-scale onset accuracy, and because the text states that manual correction was applied, the paper should provide quantitative validation of boundary accuracy (for example, agreement between Gentle and manual boundaries on a held-out subset, or a comparison with an independent aligner). Without such validation, the benchmark results cannot be fully interpreted, although this does not affect the central claim about dataset size.
minor comments (6)
  1. [Section 3.2] The stated average of 5421.28 words per session does not match Table 2's totals: 466,230 words across 93 sessions gives about 5013 words per session. Please clarify the denominator or correct the figure.
  2. [Section 3.2, Table 2, Appendix B.6] The session counts 95, 93, and 91 appear in different places. These are reconcilable (95 total recorded, 2 held out for competitions, leaving 93 released with 91 train + 1 validation + 1 test), but the text should state this relationship explicitly to avoid confusion.
  3. [Figure 3] The scaling plots for speech detection and phoneme classification use only three training volumes (5, 10, and 50 hours). The 'approximately logarithmic' claim should be qualified as suggestive, or additional intermediate training volumes should be included to support a stronger scaling-law statement.
  4. [Table 2, Table 6] Table 2 totals 52.32 released hours while Table 6 lists total audiobook hours as 53:02:41. The difference is presumably the two competition holdout sessions, but the paper should state this explicitly so readers do not treat the two totals as inconsistent.
  5. [Section 4.2, Table 4] The Macro F1 score is at chance level (0.0253 vs 0.0258). The text explains that this is due to the power-law phoneme distribution, but the table or caption should mark this clearly so the result is not misread as a failure of the model.
  6. [Throughout] Occasional typos and formatting issues should be corrected: 'cannon' for 'canon' in Appendix B.1, 'V owel' in the Figure 2 caption, 'learnign' in Appendix B.6, and inconsistent use of 'V AD' vs 'VAD'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central resource claim is independently supported by the paper's own recorded-duration statistics, and the benchmark comparisons are run against an external dataset.

full rationale

The paper's central claim is the release of a new dataset, LibriBrain, with over 50 hours of within-subject MEG recordings, described as 5x larger than the next comparable dataset. This claim is supported directly by the paper's own collection statistics (Table 1: 52 total hours and 52.32 hours/subject for LibriBrain vs 10.0 hours/subject for Armeni et al. 2022; Table 2: 3,139 minutes / 52.32 hours across train, validation, and test; Table 6: 53:02:41 total audiobook hours). These are measured properties of the released resource, not quantities derived from a model or fitted to the claim, so the size claim is self-contained and externally checkable by downloading the data. The word classification results in Table 5 and Figure 3c are reproduced with permission from Jayalath et al. (2025a), a companion paper with overlapping authors; however, the comparison is made against the independent Armeni et al. (2022) dataset, and the paper explicitly reports that the LibriBrain model outperforms the Armeni-trained model (0.3621 vs 0.3261 top-10 balanced accuracy). This is an external benchmark comparison, not a prediction forced by construction. The scaling curves in Figure 3 are descriptive summaries of the authors' own experiments and are not presented as predictions derived from the data. The only notable caveat is the validation of forced-alignment labels, which relies on duration monotonicity (Table 8) rather than onset-level accuracy; but label accuracy is a data-quality limitation affecting downstream benchmark results, not a circular dependency in the derivation of the paper's claims. The paper also discusses limitations, including single-subject design, listening-only paradigm, and focused language content, and does not invoke a uniqueness theorem or a self-citation to justify its core contribution. No load-bearing step reduces to its own input, and no prediction is equivalent to a fitted parameter by construction. Therefore the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities and makes no parametric derivation. Its central claims rest on data quality, annotation accuracy, and the assumption that single-subject MEG responses contain decodable speech information. No free parameters are fitted to produce the central claims.

assumptions (4)
  • domain assumption MEG recordings from one listener contain information sufficient for the three decoding tasks above chance.
    Section 4 treats the significant baselines as validation that the dataset carries decodable speech information; this is the empirical premise of the resource's usefulness.
  • domain assumption The forced-alignment based phoneme and word annotations are accurate enough for supervised classification.
    Section B.5 describes forced alignment plus manual correction, with validation limited to duration-length monotonicity (Table 8), not gold-standard timing accuracy.
  • domain assumption Brain responses are stable enough across 95 sessions for train/validation/test splits to be meaningful.
    Section 3.2 and B.2 describe data collected over many sessions on different days; the benchmark setup assumes within-subject cross-session consistency.
  • domain assumption Test and validation sessions from Sherlock1 chapters 11 and 12 are representative of the training distribution.
    Section B.6 designates these as standard splits; if these chapters are atypical, the baselines would misestimate generalization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LibriBrain: Over 50 Hours of Within-Subject MEG to Improve Speech Decoding Methods at Scale." pith.science (2026). https://pith.science/paper/S63VMP73

@misc{pith2026250602098,
  author       = {Pith},
  title        = {Pith review of: LibriBrain: Over 50 Hours of Within-Subject MEG to Improve Speech Decoding Methods at Scale},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S63VMP73}},
  note         = {Machine review of arXiv:2506.02098}
}
abstract

LibriBrain represents the largest single-subject MEG dataset to date for speech decoding, with over 50 hours of recordings -- 5$\times$ larger than the next comparable dataset and 50$\times$ larger than most. This unprecedented `depth' of within-subject data enables exploration of neural representations at a scale previously unavailable with non-invasive methods. LibriBrain comprises high-quality MEG recordings together with detailed annotations from a single participant listening to naturalistic spoken English, covering nearly the full Sherlock Holmes canon. Designed to support advances in neural decoding, LibriBrain comes with a Python library for streamlined integration with deep learning frameworks, standard data splits for reproducibility, and baseline results for three foundational decoding tasks: speech detection, phoneme classification, and word classification. Baseline experiments demonstrate that increasing training data yields substantial improvements in decoding performance, highlighting the value of scaling up deep, within-subject datasets. By releasing this dataset, we aim to empower the research community to advance speech decoding methodologies and accelerate the development of safe, effective clinical brain-computer interfaces.

Figures

Figures reproduced from arXiv: 2506.02098 by the authors.

Figure 1
Figure 1. Comparison of within-subject data volume in non-invasive speech datasets. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the LibriBrain dataset. (A) Illustration of the MEG scanner and sensor [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Impact of training data volume on model performance across tasks. Subsets were randomly [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Minimal pre-processing example for one representative recording session. Power spectral [PITH_FULL_IMAGE:figures/full_fig_p019_4.png]
Figure 5
Figure 5. Figure 5: Convolutional Neural Network for Phoneme Classification [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]
Figure 6
Figure 6. Figure 6: Per-phoneme analysis of our model’s performance. (A) Distribution of training samples [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]
Figure 7
Figure 7. Figure 7: Effect of sample averaging on phoneme classification performance. Accuracy improves [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: Confusion Matrix for the Phoneme Classification Task averaging over 10 samples for each [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: Differences in amplitude, phase, and power between speech and non-speech segments. [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 10
Figure 10. Figure 10: Top-10 model predictions for the word classification task. For illustration, four examples [PITH_FULL_IMAGE:figures/full_fig_p026_10.png]
Figure 11
Figure 11. Figure 11: Impact of word frequency in the training set on classification model performance. Words [PITH_FULL_IMAGE:figures/full_fig_p026_11.png]
Figure 12
Figure 12. Figure 12: Effect of Part-of-Speech (POS) category on word classification model performance. (A) [PITH_FULL_IMAGE:figures/full_fig_p027_12.png]
Figure 13
Figure 13. Figure 13: Top Function Words Cloud [PITH_FULL_IMAGE:figures/full_fig_p027_13.png]
Figure 14
Figure 14. Figure 14: Top Content Words Cloud. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_14.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The 2025 PNPL Competition: Speech Detection and Phoneme Classification in the LibriBrain Dataset

    cs.LG 2025-06 conditional novelty 6.0 of 10

    The 2025 PNPL competition presents over 50 hours of within-subject MEG data, defines speech detection and phoneme classification benchmarks with F1-macro scoring, and reports reference baselines of 68.04% and 60.39%.

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The claims made in the abstract and introduction accurately reflect the proper- ties of the dataset as described in Section 3, the experimental findings detailed in Section 4, and the limitations di...

    work page

  2. [2]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] 30 Justification: We discuss limitations in Section 5.2. Ethical and privacy considerations are discussed in Appendix A.1. Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the paper has limitatio...

    work page

  3. [3]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: We do not present theoretical results. Guidelines: • The answer NA means that the paper does not include theoretical results. • All the theorems, formulas, and proofs in the...

    work page

  4. [4]

    We also provide simple access to the dataset ensuring that our results are easily reproducible

    Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: Aside from providing the c...

    work page

  5. [5]

    The link to these is available in Section 4

    Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We provide execution-ready scripts to recreate the training runs of the experi- mental results. The link to...

    work page

  6. [6]

    We describe data splits in Section B.6

    Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: We detail information necessary to understand the main results in Section 4. We describe data splits in Secti...

    work page

  7. [7]

    For the phoneme classification and speech detection experiments we report statistical significance compared to a naive baseline as measured by an exact permutation test

    Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We report error bars that represent standard error across multiple seeds in Section 4. For the phoneme classification and speech d...

    work page

  8. [8]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] 33 Justification: We provide details on computer resources need to reproduce the results in Appendix E.3, D.3, and F.2...

    work page

Show all 16 references
  1. [9]

    The NeurIPS Code of Ethics was reviewed and adhered to

    Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: We discuss ethical considerations at length in A. The NeurIPS Code of Ethics wa...

  2. [10]

    Guidelines: • The answer NA means that there is no societal impact of the work performed

    Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We provide an extensive discussion of potential positive and negative societal impacts in Section 5, and ...

  3. [11]

    However, we are intentionally releasing them under a CC BY-NC license to retain the ability to prevent potential unethical commercial use in the future

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: There are no im...

  4. [12]

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The baseline wor...

    2022

  5. [13]

    Guidelines: • The answer NA means that the paper does not release new assets

    New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We provide extensive documentation alongside our code and dataset on GitHub and Hugging Face including information o...

  6. [14]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  7. [15]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

  8. [16]

    Answer: [NA] Justification: This research does not involve LLMs as any important, original, or non- standard components

    Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the ...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.