Pith. sign in

REVIEW 4 major objections 6 minor 23 references

Loquacious Set: 25,000 Hours of Transcribed and Diverse English Speech Recognition Data for Research and Commercial Use

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read A 25,000-hour English speech corpus built from six open datasets gives researchers and industry a common, commercially usable benchmark.

desk verdict Useful curated 25k-hour commercial ASR dataset with sensible splits and recipes, but the headline benchmark claim rests on unaddressed train/eval overlap between Libriheavy/LibriSpeech and YODAS en003. read the letter →

arxiv 2505.21578 v1 pith:JD75TWRR submitted 2025-05-27 cs.CL cs.SDeess.AS

classification cs.CLcs.SDeess.AS
keywords automaticspeechrecognitionEnglishcorpuscommercial-usedatasetcurationbenchmarkConformerworderrorratenormalizedtranscriptions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the speech community needs a successor to LibriSpeech—large enough for modern models, diverse in accents and recording conditions, and licensed so both academics and industry can use it—and that no existing dataset satisfies all those requirements. It introduces the Loquacious Set, a 25,000-hour English corpus assembled and curated from six permissively licensed sources, with normalized transcriptions, three nested training subsets (250, 2,500, and 25,000 hours), a clean subset, and dedicated development and test sets. The authors report that conformer models trained with the released recipes reach word error rates that are competitive with, or better than, much larger systems trained on undocumented data. If the claims hold, the corpus gives both academic and industrial researchers a shared, reproducible evaluation ground for English speech recognition.

What carries the argument

The load-bearing object is the curated corpus itself: six permissively licensed source collections are combined into nested training splits (small, medium, large, clean) and two evaluation sets, with all audio resampled to 16 kHz, segmented to 1–40 seconds (3–40 seconds for YODAS), and filtered by language and duration. Transcription normalization is the main curation step: numerals are converted to words, sentences containing non-English characters or more than four symbols are removed, punctuation and YouTube-style descriptors are stripped, and the remaining text is upper-cased and restricted to the 26 Latin letters plus the apostrophe. This machinery makes the dataset reproducible and gives every sample a standard audio-label pair, which is what allows a single training recipe to be applied across heterogeneous sources.

What would settle it

Compute the overlap at the segment, speaker, and book level between the training splits and the evaluation splits: check whether any Libriheavy training segment shares an audiobook or speaker with LibriSpeech test-other, and whether any YODAS training segment matches the YODAS-based dev/test segments. If overlap exists, rerun the reported word error rate table on a strictly disjoint split; a large increase in word error rate would show that the published numbers partly reflect in-domain test material.

Watch

Extended reading notes

Core claim

The central claim is that the Loquacious Set is the first large-scale English speech recognition corpus to combine a permissive commercial license, acoustic and linguistic diversity, normalized transcripts, defined training subsets, and dedicated evaluation sets. It pools audio from six existing datasets—Common Voice, VoxPopuli, Libriheavy, People's Speech, YODAS, and LibriSpeech—and applies a five-step text-normalization pipeline, duration filtering, and language identification, yielding 25,000 hours of curated speech in about 9.4 million segments with an average duration of 9.6 seconds. The paper reports that conformer models trained on this corpus achieve a 4.6% word error rate on the LibriSpeech test-other set with a 480M-parameter model, beating a widely used 1.5B-parameter open speech recognition system that scores 5.1%; the same model reaches 6.9% on the VoxPopuli test set and 12.0% on the Common Voice test set. The smaller subsets are designed so that 250-hour and 2,500-hour training runs can fit on four GPUs, making the benchmark accessible to groups without large compute budgets.

Load-bearing premise

The reported benchmark results assume that the audio in the evaluation sets is not drawn from the same books, speakers, or segments as the training data: the Libriheavy training data and the LibriSpeech test-other set both derive from the same public-domain audiobook collection, and the YODAS-based dev/test segments come from a source subset that is also used in training, and the paper does not state that overlapping content was removed.

Editorial extensions

If this is right

  • Groups with four GPUs can train competitive models on the 250-hour or 2,500-hour subsets, while the full 25,000-hour set is designed for larger laboratories.
  • Researchers in industry can legally use the corpus for commercial speech recognition development, closing the licensing gap that blocks industrial use of several existing large datasets.
  • Models trained on the Loquacious Set beat a widely used 1.5B-parameter open speech recognition system on LibriSpeech test-other (4.6% versus 5.1% word error rate) and on VoxPopuli test (6.9% versus 7.5%) with fewer parameters, according to the paper's measurements.
  • Adding related but not identical data helps cross-domain performance: a 100M model trained on the corpus gets a 15.9% word error rate on the Common Voice test set versus 19.5% when trained on Common Voice alone.
  • The clean 13,000-hour subset matches the full 25,000-hour model on LibriSpeech test-other (6.0% word error rate with a 250M model), suggesting a cheaper compute path for clean-speech research.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step is to publish the full per-utterance language-identification decisions and the exact filtering statistics for each source; that would let other builders reproduce the curation and quantify how much of YODAS and People's Speech survives normalization.
  • The same curation recipe—normalization, duration bounds, language filtering, and nested subsets—could be ported to other permissively licensed multilingual corpora, though the language-identification step would need per-language validation.
  • Versioning will determine whether the Loquacious Set becomes a lasting benchmark: since Common Voice releases change over time, freezing a snapshot and releasing hashes of audio segments would preserve comparability across future work.
  • A stricter evaluation protocol would exclude any audiobook or YODAS source that appears in training; such a split would test generalization to unseen speakers and books more cleanly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces the Loquacious Set, a curated English ASR corpus of roughly 25,000 hours built from six existing datasets (VoxPopuli, CommonVoice, People's Speech, YODAS, Libriheavy, and LibriSpeech), with text normalization, audio filtering, and defined small/medium/large/clean training splits plus dev/test sets. It ships SpeechBrain recipes and reports conformer encoder-decoder WERs on LibriSpeech test-other, VoxPopuli, CommonVoice, and its own dev/test, claiming state-of-the-art results that beat Whisper models at comparable or smaller parameter counts.

Significance. If the held-out evaluation is confirmed and the normalization quality is demonstrated, the dataset would be a valuable common resource for academic and industrial ASR: it has a permissive licensing profile, defined subsets of different sizes, and covers diverse acoustic conditions. The release of code and recipes is a concrete reproducibility strength. The main claims, however, depend on the evaluation sets being genuinely out-of-domain, which is not currently established.

major comments (4)
  1. [3.1, Table 1] The paper does not address training/evaluation overlap. The large split includes 11,000 hours of Libriheavy, which is derived from LibriVox audiobooks, while the evaluation sets include LibriSpeech test-other (also LibriVox-derived); moreover, the YODAS dev/test segments are extracted from en003, which is among the YODAS subsets used to form the 6,100-hour training portion. The manuscript never states whether book-, speaker-, or segment-level decontamination was performed. Please state the overlap-removal procedure and report statistics on remaining overlap, or explicitly justify why overlap is absent. Without this, the reported WERs in Table 2 on LibriSpeech test-other and the Loquacious dev/test are not established as held-out, which weakens the comparison with Whisper and the claim of state-of-the-art performance.
  2. [3.1] The selection rule for the large split is given only as 'up to a certain threshold to reach 25,000 hours of speech.' The per-source caps and any quality thresholds should be stated explicitly (e.g., a table of the exact hours taken from each source and the random seed). While the code may contain these details, the paper should provide them for reproducibility, especially because Table 1 reports 25,150 hours rather than 25,000.
  3. [3.2] The normalization process is described but its resulting transcription quality is not quantified. The paper claims 'normalised transcriptions with few errors' but provides no measurement of residual errors after normalization (e.g., character error rate against a held-out reference subset, or the number of manually verified samples for YODAS and People's Speech). Since unreliable transcriptions are a stated limitation of the source corpora, please report a concrete accuracy figure for the normalized text, including for the manually verified YODAS dev/test segments.
  4. [4.2] The Whisper comparison contains a factual error: the text states 'our 100M models obtains 5.7%' on LibriSpeech test-other, but Table 2 shows that the 250M model achieves 5.7%, while the 100M model achieves 6.0%. Please correct the parameter count in the comparison (and adjust the following sentence about '3 times more parameters' accordingly). This correction is needed to support the claim that the Loquacious recipes beat Whisper at smaller model sizes.
minor comments (6)
  1. [4.1] The phrase 'height of them are used' should be 'eight of them are used'.
  2. [3.1] The sentence 'most of the samples below this threshold correspond are just noises' contains a grammatical error; it should read 'correspond to noise' or 'are just noise.'
  3. [Throughout] The dataset name appears as 'CommonV oice' in several places due to a LaTeX artifact; please correct to 'CommonVoice.'
  4. [Table 1, 3.1] Table 1 gives the large split as 25,150 hours and the clean split as 13,150 hours, while the text in Section 3.1 says 25,000 and 13,000 hours; please harmonize these numbers.
  5. [3.1] The text mentions 'YoutuBe videos' in the Acoustic conditions paragraph; this should be 'YouTube videos.'
  6. [4.2] The literature numbers for Whisper (e.g., 6.7% on LibriSpeech test-other for Small.en) are cited without a reference; please add a source for these numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the paper's claims are empirical dataset construction and reproducible ASR benchmarks; the evaluation-overlap concern is a data-hygiene issue, not a circularity step.

full rationale

The Loquacious Set paper contains no derivation chain in which an output quantity is defined in terms of the quantity it claims to predict. The central outputs are (i) a curated 25,000-hour corpus assembled by subsampling six existing datasets and (ii) measured WERs from SpeechBrain conformer recipes. These are empirical measurements, not predictions derived from a fitted parameter. The SpeechBrain citation [14] is a toolkit reference whose authors overlap with the present paper, but the experimental hyperparameters and table of results are fully specified in the paper, so the citation is not load-bearing for the WER claims. The language-identification filtering of YODAS uses a published SpeechBrain model as an external tool; this is a data-cleaning step, not an argument whose conclusion is assumed. The strongest integrity concern is that Loquacious dev/test are drawn from YODAS en003 while the large training split includes YODAS audio, and that Libriheavy training and LibriSpeech test-other share the LibriVox domain; without an explicit decontamination statement the reported WERs could be optimistic. However, this is potential train/evaluation leakage, not circularity: the WER numbers are not equivalent by construction to the training data, and the paper does not claim a derivation that reduces to its inputs. No fitted input is renamed as a prediction, no uniqueness theorem is imported, and no ansatz is smuggled in via citation. Under the stated criteria, the appropriate finding is no significant circularity (score 0).

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical entities are introduced. The central resource depends on hand-set curation thresholds and unstated disjointness assumptions between training and evaluation audio.

free parameters (3)
  • Large split size caps = Libriheavy 11,000h, YODAS 6,100h, People's Speech 5,900h, CommonVoice 1,600h, VoxPopuli 550h
    The paper says large contains data 'up to a certain threshold' to reach 25,000 hours, but does not justify the per-source caps; these are hand-set choices.
  • YODAS minimum segment duration = 3 seconds
    Introduced ad hoc because the paper says most YODAS samples below this threshold are noises, while other sources use a 1-second lower bound.
  • Symbol removal threshold = more than 4 symbols triggers removal
    Hand-chosen cutoff in the text normalization pipeline; sentences with more than four symbols are removed as uncertain.
assumptions (4)
  • domain assumption Source datasets' licenses permit commercial combination
    The licensing claim for all six sources is asserted from Table 1; the paper does not independently verify the provenance of every YouTube audio file in YODAS.
  • domain assumption SpeechBrain VoxLingua107 language ID correctly identifies English in YODAS
    The 9,200-hour English estimate depends on this model; only 'a few random samples' were manually checked.
  • domain assumption Libriheavy training data do not overlap LibriSpeech test-other
    Both come from LibriVox audiobooks, but the paper never states that overlapping books or speakers were removed; this is essential for the external LibriSpeech WER comparisons.
  • domain assumption YODAS dev and test segments are disjoint from the large training split
    Not stated in Section 3.1; if the 1.5-hour dev and test segments come from en003, which is also a training source, contamination is possible.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Loquacious Set: 25,000 Hours of Transcribed and Diverse English Speech Recognition Data for Research and Commercial Use." pith.science (2026). https://pith.science/paper/JD75TWRR

@misc{pith2026250521578,
  author       = {Pith},
  title        = {Pith review of: Loquacious Set: 25,000 Hours of Transcribed and Diverse English Speech Recognition Data for Research and Commercial Use},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JD75TWRR}},
  note         = {Machine review of arXiv:2505.21578}
}
read the original abstract

Automatic speech recognition (ASR) research is driven by the availability of common datasets between industrial researchers and academics, encouraging comparisons and evaluations. LibriSpeech, despite its long success as an ASR benchmark, is now limited by its size and focus on clean, read speech, leading to near-zero word error rates. More recent datasets, including MOSEL, YODAS, Gigaspeech, OWSM, Libriheavy or People's Speech suffer from major limitations including licenses that researchers in the industry cannot use, unreliable transcriptions, incorrect audio data, or the lack of evaluation sets. This work presents the Loquacious Set, a 25,000-hour curated collection of commercially usable English speech. Featuring hundreds of thousands of speakers with diverse accents and a wide range of speech types (read, spontaneous, talks, clean, noisy), the Loquacious Set is designed to work for academics and researchers in the industry to build ASR systems in real-world scenarios.

Figures

Figures reproduced from arXiv: 2505.21578 by the authors.

Figure 1
Figure 1. Durations of the 9.6M samples of the large training split of the Loquacious Set. The average is 9.6 seconds. symbols are removed as the pronunciation may again be too uncertain. The fifth step relies on the nltk library to remove any remaining YouTube descriptors like [explosion] or [light music]. Finally, all words are upper-cased and checked to only contain the 26 Latin letters or the apostrophe. Due to imperfect … view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 13 canonical work pages

  1. [1]

    open-source

    Introduction The success of deep learning in automatic speech recognition (ASR) hinges on access to standardised datasets. Corpus like the Wall Street Journal (WSJ) [1] and TIMIT [2] were crucial for early ASR research, but their limited size and price rapidly became limiting issues. The advent of LibriSpeech [3], with its open CC-BY license and 1,000 hou...

  2. [2]

    Loquacious Set: 25,000 Hours of Transcribed and Diverse English Speech Recognition Data for Research and Commercial Use

    Large Scale English ASR Datasets The Loquacious Set is not the first attempt to create a standard- ised large-scale ASR dataset for the English language. This sec- tion discusses others relevant initiatives. The CommonV oice dataset [13] is based on crowd- sourcing, and can theoretically leverage millions of users recording each a few samples on their dev...

  3. [3]

    dev-other

    The Loquacious Set This section describes the data selection and corpus structure of the Loquacious Set (section 3.1) as well as the process used to normalise the transcriptions (section 3.2). License considerations. The Loquacious dataset uses existing data and does not impose any new licensing restrictions. Each component of the original datasets keeps ...

  4. [4]

    Speech Recognition Experiments This section details the experimental protocol (section 4.1) as well as the speech recognition results (section 4.2) ob- tained with the Loquacious Set on various benchmarking speech recognition datasets. 4.1. Experimental protocol Models are trained on the three subsets of the Loquacious Set, hence containing either 250, 25...

  5. [5]

    The dataset is easy to reproduce thanks to the SpeechBrain re- leased source code and can be loaded in a single line of code

    Conclusion The Loquacious Set offers 25,000 hours of normalised tran- scribed and diverse English speech recognition data available for academics, researchers in the industry and commercial use. The dataset is easy to reproduce thanks to the SpeechBrain re- leased source code and can be loaded in a single line of code. It introduces a new common ground fo...

  6. [6]

    Mosel: 950,000 hours of speech data for open-source speech foundation model train- ing on eu languages,

    M. Gaido, S. Papi, L. Bentivogli, A. Brutti, M. Cettolo, R. Gretter, M. Matassoni, M. Nabih, and M. Negri, “Mosel: 950,000 hours of speech data for open-source speech foundation model train- ing on eu languages,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , 2024, pp. 13 934–13 947

  7. [7]

    The design for the wall street journal- based csr corpus,

    D. B. Paul and J. Baker, “The design for the wall street journal- based csr corpus,” in Speech and Natural Language: Proceed- ings of a Workshop Held at Harriman, New York, February 23-26, 1992, 1992

  8. [8]

    Darpa timit acoustic-phonetic continous speech corpus cd-rom. nist speech disc 1-1.1,

    J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, and D. S. Pallett, “Darpa timit acoustic-phonetic continous speech corpus cd-rom. nist speech disc 1-1.1,”NASA STI/Recon technical report n, vol. 93, p. 27403, 1993

Show all 23 references
  1. [9]

    Lib- rispeech: an asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: an asr corpus based on public domain audio books,” in 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210

  2. [10]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning . PMLR, 2023, pp. 28 492–28 518

  3. [11]

    Conformer-1: Robust asr via large-scale semisupervised boot- strapping,

    K. Zhang, L. Chkhetiani, F. M. Ramirez, Y . Khare, A. Vanzo, M. Liang, S. R. Martin, G. Oexle, R. Bousbib, T. Peyash et al., “Conformer-1: Robust asr via large-scale semisupervised boot- strapping,” arXiv preprint arXiv:2404.07341, 2024

  4. [12]

    Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio,

    G. Chen, S. Chai, G. Wang, J. Du, W.-Q. Zhang, C. Weng, D. Su, D. Povey, J. Trmal, J. Zhang et al., “Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio,” arXiv preprint arXiv:2106.06909, 2021

  5. [13]

    Yodas: Youtube-oriented dataset for audio and speech,

    X. Li, S. Takamichi, T. Saeki, W. Chen, S. Shiota, and S. Watan- abe, “Yodas: Youtube-oriented dataset for audio and speech,” in 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2023, pp. 1–8

  6. [14]

    Libriheavy: a 50,000 hours asr corpus with punc- tuation casing and context,

    W. Kang, X. Yang, Z. Yao, F. Kuang, Y . Yang, L. Guo, L. Lin, and D. Povey, “Libriheavy: a 50,000 hours asr corpus with punc- tuation casing and context,” in ICASSP 2024-2024 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 1...

  7. [15]

    People’s Speech [9] and YODAS [7] are the most recent attempts at overcoming the read versus spontaneous speech issue at large scale

    (MLS) is very similar to Libriheavy as it comes from the same source but is multilingual. People’s Speech [9] and YODAS [7] are the most recent attempts at overcoming the read versus spontaneous speech issue at large scale. The People’s Speech provides 30,000 hours of mostly A...

  8. [16]

    The people’s speech: A large-scale diverse english speech recognition dataset for commercial usage,

    D. Galvez, G. Diamos, J. Ciro, J. F. Cer ´on, K. Achorn, A. Gopi, D. Kanter, M. Lam, M. Mazumder, and V . J. Reddi, “The people’s speech: A large-scale diverse english speech recognition dataset for commercial usage,” arXiv preprint arXiv:2111.09344, 2021

  9. [17]

    Speechstew: Simply mix all available speech recognition data to train one large neural network,

    W. Chan, D. Park, C. Lee, Y . Zhang, Q. Le, and M. Norouzi, “Speechstew: Simply mix all available speech recognition data to train one large neural network,”arXiv preprint arXiv:2104.02133, 2021

  10. [18]

    Reproducing whisper-style training using an open-source toolkit and publicly available data,

    Y . Peng, J. Tian, B. Yan, D. Berrebbi, X. Chang, X. Li, J. Shi, S. Arora, W. Chen, R. Sharma et al., “Reproducing whisper-style training using an open-source toolkit and publicly available data,” in 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEE...

  11. [19]

    Common voice: A massively-multilingual speech corpus,

    R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,”arXiv preprint arXiv:1912.06670, 2019

  12. [20]

    Open-source conversational ai with speechbrain 1.0,

    M. Ravanelli, T. Parcollet, A. Moumen, S. de Langen, C. Subakan, P. Plantinga, Y . Wang, P. Mousavi, L. D. Libera, A. Ploujnikov, F. Paissan, D. Borra, S. Zaiem, Z. Zhao, S. Zhang, G. Karakasidis, S.-L. Yeh, P. Champion, A. Rouhe, R. Braun, F. Mai, J. Zuluaga- Gomez, S. M. Mou...

  13. [21]

    Mls: A large-scale multilingual dataset for speech research,

    V . Pratap, Q. Xu, A. Sriram, G. Synnaeve, and R. Collobert, “Mls: A large-scale multilingual dataset for speech research,” arXiv preprint arXiv:2012.03411, 2020

  14. [22]

    V oxpopuli: A large-scale multilingual speech corpus for representation learn- ing, semi-supervised learning and interpretation,

    C. Wang, M. Riviere, A. Lee, A. Wu, C. Talnikar, D. Haz- iza, M. Williamson, J. Pino, and E. Dupoux, “V oxpopuli: A large-scale multilingual speech corpus for representation learn- ing, semi-supervised learning and interpretation,” in Proceedings of the 59th Annual Meeting of ...

  15. [23]

    Joint ctc-attention based end-to-end speech recognition using multi-task learning,

    S. Kim, T. Hori, and S. Watanabe, “Joint ctc-attention based end-to-end speech recognition using multi-task learning,” in2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2017, pp. 4835–4839

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.