Pith. sign in

REVIEW 3 major objections 4 minor 4 cited by

A fully public 1M-hour dataset trains speech recognizers that match Whisper's zero-shot word error rates at every scale.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

An open 1M-hour English speech dataset plus Whisper-architecture models trained on it match Whisper's word error rates on short and long-form benchmarks.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection OLMoASR is a serious open-data Whisper reproduction with real artifacts and controlled ablations; the main open risk is eval-set contamination that the paper doesn't rule out. the 3 major comments →

arxiv 2508.20869 v1 pith:QS3L25QI submitted 2025-08-28 cs.SD cs.CLcs.LGeess.AS

OLMoASR: Open Models and Data for Training Robust Speech Recognition Models

Huong Ngo , Matt Deitke , Martijn Bartelds , Sarah Pratt , Josh Gardner , Matt Jordan , Ludwig Schmidt This is my paper
classification cs.SD cs.CLcs.LGeess.AS
keywords speech recognitiondata curationweb-scale datasetzero-shot generalizationWhisper architectureword error rateopen modelsweak supervision
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that Whisper-class zero-shot speech recognition does not require a proprietary training set. It builds OLMoASR-Pool, 3M hours of web-scraped English audio with transcripts, then applies text-heuristic filters to get a 1M-hour curated set, OLMoASR-Mix. Training Whisper-architecture models on this mix yields word error rates within roughly 0.4 to 1.0 percentage points of Whisper's English-only models across five sizes, including 12.8% versus 12.4% short-form and 11.0% versus 10.5% long-form at the medium scale. The controlled comparisons isolate data as the variable: ablations show the curation layer, especially removing repeating transcript lines, is what closes most of the gap, and the web-scale mix beats academic corpora. If the result holds, fully open data can reproduce the generalization that was thought to depend on Whisper's closed 680K-hour corpus.

Core claim

On the paper's terms, the discovery is that a fully open pipeline—a 3M-hour web-scraped pool, a curation stack, and the Whisper architecture itself—is sufficient to reproduce Whisper's zero-shot robustness. OLMoASR models match Whisper's English-only models from 39M to 1.55B parameters on 14 short-form and 7 long-form benchmarks, with the medium model at 12.8% and 11.0% WER versus Whisper-medium.en's 12.4% and 10.5%. The paper attributes the result to dataset design rather than architecture: keeping the Whisper model fixed and changing only the data, curated OLMoASR-Mix outperforms the unfiltered subset, outperforms the same models trained on academic data or on YODAS, and shows positive eff

What carries the argument

The load-bearing mechanism is the curation pipeline: spoken-language and text-language alignment using VoxLingua107 and pycld2, text heuristics that flag machine-generated transcripts (mostly-uppercase casing, exactly repeating lines, and a manual-versus-machine transcript WER filter), and transcript-level fuzzy deduplication via 5-gram minhash. The second mechanism is the controlled evaluation design: the Whisper encoder-decoder architecture, tokenizer, and decoding setup are held fixed, so every WER difference across ablations is attributed to the data. The repeating-lines filter is singled out as the largest single lever, a 14.5-point WER improvement on short-form evaluation.

Load-bearing premise

The comparison is only honest if the public evaluation sets are not substantially present in the web-scraped training pool; the paper documents transcript-level decontamination only for TED-LIUM3 and uses a 75% similarity dedup, so lightly edited evaluation audio could still be in the mix.

What would settle it

Search the audio and transcripts of LibriSpeech, CommonVoice, CHiME-6, AMI, CORAAL, and the long-form sets against OLMoASR-Pool using audio fingerprinting and relaxed transcript matching; any substantial overlap would lower the reported zero-shot WERs, and retraining on a strictly decontaminated subset would show a bigger gap to Whisper than the 0.4 to 1.0 point spread.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Whisper-level zero-shot ASR can be reached with public data alone; a closed 680K-hour corpus is not a necessary ingredient.
  • The same curation recipe should transfer to other model scales and to other weakly supervised speech tasks, since it is data-side rather than architecture-side.
  • Web-scale curated data is a robustness intervention: OLMoASR-Mix-trained models beat academic-data models on out-of-distribution sets such as CHiME-6 and CORAAL.
  • Releasing the IDs of the pool and mix allows third parties to audit contamination, bias, privacy, and fairness in a way Whisper's closed data does not.
  • Increasing data beyond moderate amounts gives diminishing returns for a small model, with the full 1M-hour set supplying an extra short-form gain.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the no-contamination premise holds, the parity result implies Whisper's edge is mostly its weakly supervised web-scale data, not architecture, which would strengthen the case for open weak-supervision efforts in other modalities.
  • A natural next audit is audio-level rather than transcript-level overlap detection: the paper only reports decontamination for TED-LIUM3, so a fingerprint search of LibriSpeech, CommonVoice, and the long-form sets against OLMoASR-Pool would test the zero-shot reading directly.
  • The heuristics are proxies for machine-generated transcripts; learned quality scorers or ASR-confidence estimates could push the filtered fraction beyond 1M hours and improve WER further.
  • The English-only scope leaves open whether the same pipeline would close the gap on multilingual Whisper, where language coverage rather than transcript quality may dominate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces OLMoASR-Pool, a roughly 3M-hour web-scraped English audio-transcript corpus, and a text-heuristic curation pipeline that yields the 1M-hour OLMoASR-Mix training set. Using the Whisper architecture/tokenizer with FlashAttention and a controlled training recipe, the authors train OLMoASR models from 39M to 1.55B parameters and evaluate them zero-shot on 14 short-form and 7 long-form English benchmarks. The central claim is that OLMoASR achieves WER comparable to OpenAI's Whisper across scales (e.g., medium.en: 12.8 vs 12.4 short-form, 11.0 vs 10.5 long-form), that curation is essential for this result, and that open data/models/code are released.

Significance. If the central comparison holds, this is a landmark open-data contribution: it would provide the first fully open reproduction of Whisper-class zero-shot ASR at multiple scales, with a public 1M-hour training corpus, released model weights, and evaluation code. The paper also makes a credible data-centric case: controlled ablations in Section 5 isolate data composition and show large WER gains from filtering. The release of data IDs and the explicit discussion of responsible use are additional strengths. The credibility of the headline claim, however, rests on contamination control and statistical support that the manuscript currently does not provide.

major comments (3)
  1. [Appendix C / §4.1] The decontamination step is documented only for TED-LIUM3: "we decontaminate the evaluation datasets of TED-LIUM3 against our training corpus" and the preceding sentence describes the n-gram search in the singular. The 21 benchmark sets in Tables 4 and 5 are asserted to be "not used for training," but no explicit contamination check is reported against LibriSpeech, CommonVoice, WSJ, Switchboard, CHiME-6, AMI, CORAAL, or the long-form sets. Because OLMoASR-Pool is a 3M-hour web scrape, and several of these corpora are derived from widely reposted audio (e.g., LibriSpeech from LibriVox audiobooks), a small overlap would close the 0.4–1.0 WER gaps on which the "comparable to Whisper" claim rests. The Fuzzy minhash dedup described in Appendix C (5-grams, 75% Jaccard) is a dedup step, not an eval-set decontamination check, and cannot catch lightly edited or partial transcripts. This is a miss
  2. [§2.2.2, Tables 1–3 vs Tables 4–5] The curation pipeline is selected on the evaluation target. Section 2.2.2 states that filter choices and the manual-machine comparison thresholds (0.5 document-level, 0.7 segment-level) were determined "through experiments" using "average performance across 14 short-form speech recognition datasets" — the same 14 datasets that appear in Table 4. Consequently, the reported zero-shot numbers are not an unbiased estimate of generalization; the pipeline has been tuned to the very benchmark average used as the headline comparison, while Whisper's thresholds were not tuned on these sets. This is not equation-level circularity, but it is selection-on-evaluation. Please evaluate the pipeline using a held-out validation split that is disjoint from the 14 short-form and 7 long-form benchmark sets, or use a nested-selection procedure, and report both the selected and unselected results.
  3. [Tables 4 and 5; Figures 1 and 3] All model configurations are reported from a single training run with no seeds, confidence intervals, or significance tests. The headline claim of "comparable" performance is based on average gaps of 0.4–1.0 WER, which are near the plausible run-to-run noise for ASR training at these scales; individual cells show much larger differences (e.g., AMI-SDM tiny.en: 55.4 vs 50.3; CallHome large.en: 16.5 vs 15.8). Without variance estimates, paired significance tests, or at least multiple seeds for the main configurations, the comparison is not statistically supported. Please report at least a small number of seeds for the main models (or bootstrap confidence intervals from test segments) to quantify the uncertainty in the averages.
minor comments (4)
  1. [Abstract] The phrase "Whisper's largest English-only model Whisper-medium.en" is imprecise: Whisper's English-only variants include medium.en, but the Whisper family has larger multilingual large models. Suggest rewording to "the largest English-only Whisper variant considered here."
  2. [Appendix C] Please specify the exact n-gram size used in the decontamination search, whether the search is run only on transcripts (as opposed to audio), and how the 286 contaminated transcripts were distributed across evaluation sets. The current text says only "n-grams of size 10" in the TED-LIUM3 sentence, which is ambiguous about scope.
  3. [§2.2.2] The threshold description is internally inconsistent: one sentence says "omitting pairs where the score is lower than a specific threshold," while the next paragraph and Table 3 say pairs with WER above the threshold are removed. Please correct the directionality and state the convention clearly.
  4. [Tables 1–3] The column header "Data hours" is followed by raw numbers without units; clarify whether the values are total hours or hours after filtering, and whether "percent remaining" is computed by hours or by segment count. Figure 2's "percentages are relative to most recent filtered subset" could also be stated more explicitly.

Circularity Check

1 steps flagged

Partially circular: the curation filters were selected on the same 14 short-form benchmarks that later serve as the headline short-form results; no other circularity found.

specific steps
  1. fitted input called prediction [Section 2.2.2, Tables 1-3; Section 4.2, Figure 3/Table 4]
    "All experiments are performed on the OLMOASR-tiny.en model and compared to a baseline that has only been trained on data filtered with the audio-text language alignment filter. ... Short-form WER refers to the average performance across 14 short-form speech recognition datasets. ... Through experiments, we determined thresholds of 0.5 for document-level and 0.7 for segment-level filtering."

    The curation decisions (casing filter, repeating-line removal, manual-machine comparison, thresholds 0.5/0.7, and the final filter combination) were chosen by measuring average short-form WER on the same 14 benchmarks that later appear as the headline short-form results in Section 4.2 (Table 4, Figure 3). Tables 1-3 report the improvement of each filter using this exact metric, and Section 4.2 then presents the same metric as evidence that 'OLMOASR achieves comparable average performance to Whisper.' Thus the short-form WER headline is not an independent prediction of the pipeline's value: the pipeline was fit to that evaluation metric. The Whisper comparison is external and the final numbers are measurements, so the circularity is partial rather than a definitional equivalence.

full rationale

This is an empirical dataset/model paper, not a formal derivation, so no equation-level circularity is present. The main circularity concern is that the curation pipeline was validated on the same 14 short-form benchmarks that later serve as the headline short-form results: the filter choices in Section 2.2 were selected by minimizing average short-form WER on those benchmarks, and Section 4.2 then reports the same benchmark average as evidence of parity with Whisper. This is a test-set selection/overfitting issue rather than a definitional equivalence: the final WER numbers are real measurements and the Whisper comparison is external, so the claim is not forced by construction. Self-citations (DataComp-LM, OLMo) are used only as methodological examples and are not load-bearing. Separately, Appendix C states that decontamination was performed only against TED-LIUM3, so the zero-shot/'held-out' interpretation of the other 20 evaluation benchmarks is not established; this is a missing control for the stated zero-shot claim, not a circularity. Overall, one significant but partial circularity: score 4.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 0 invented entities

The paper's contribution is an empirical artifact (dataset, pipeline, models), so few explicit axioms are needed; the listed axioms are the load-bearing external facts and modeling choices. The fitted quantities are the curation thresholds and filter-selection decisions, all tuned against the average short-form WER of OLMoASR-tiny.en on the 14 evaluation benchmarks. No invented entities are postulated; the dataset and models are concrete artifacts.

free parameters (3)
  • document-level manual-machine comparison threshold = 0.5 WER
    Section 2.2.2: 'Through experiments, we determined thresholds of 0.5 for document-level and 0.7 for segment-level filtering.' Chosen by short-form WER of OLMoASR-tiny.en on the 14 eval benchmarks; removing 54.8% of data improves WER by 16.5%.
  • segment-level manual-machine comparison threshold = 0.7 WER
    Section 2.2.2; the finer-grained alignment filter threshold was selected by the same eval-driven procedure on the 14 short-form benchmarks.
  • filter combination (repeats + upper-case, excluding lower-case) = repeating-lines removal + upper-case removal
    Table 2: the combined casing and repeats filter scored 0.7% WER worse when lower-case removal was included (22.6 vs 21.9), so the final combination was chosen on eval WER.
axioms (6)
  • domain assumption Whisper was trained on 680K hours of weakly labeled data, and the second and third Whisper versions used comparable weakly labeled data scale
    Introduction and Section 2.1 use this undisclosed figure from Radford et al. 2023 to claim OLMoASR-Mix 'surpasses the scale of data used to train the initial Whisper models' and 'matches the amount used to train the second and third versions'. Whisper's data remains closed, so this scale match is not independently checkable.
  • domain assumption VoxLingua107 (audio) and pycld2 (text) top-1 language predictions correctly identify English-only pairs
    Section 2.2.1; mislabeled non-English or mixed-language pairs pass into the training set, and English audio tagged as non-English is dropped, biasing the language distribution.
  • domain assumption Manual-versus-machine transcript WER reliably separates faithful from unfaithful or misaligned audio-text pairs
    Section 2.2.2: 'Although automatic transcripts are less precise, they reliably capture speech utterances.' Pairs with WER above 0.5/0.7 are presumed low-quality; the filter also removes hard-but-faithful pairs that a weak ASR fails on, shaping the difficulty distribution of the training data.
  • domain assumption The reconstructed Whisper training loop matches Whisper's actual recipe closely enough for a fair comparison
    Section 2.1: 'we construct a training loop and data processing pipeline to the best of our abilities to match what Whisper used.' Whisper's code and data are unavailable; the paper validates the loop only by monitoring training and validation loss curves.
  • domain assumption Evaluation sets are not contaminated by the training pool
    Appendix C documents n-gram decontamination for TED-LIUM3 only; no contamination check is described for LibriSpeech, CommonVoice, CHiME-6, AMI, CORAAL, WSJ, Switchboard, VoxPopuli, Fleurs, or Artie, even though the pool is web-scraped public audio where these corpora or their source material may appear.
  • standard math FineWeb minhash deduplication parameters (5-grams, 112 hashes, 14 buckets, 75% Jaccard target) are adequate for ASR transcript dedup
    Appendix C adapts parameters from Penedo et al. 2024b without ASR-specific validation; the 75% Jaccard target is coarse and may miss near-duplicates.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of OLMoASR: Open Models and Data for Training Robust Speech Recognition Models." pith.science (2026). https://pith.science/paper/QS3L25QI

@misc{pith2026250820869,
  author       = {Pith},
  title        = {Pith review of: OLMoASR: Open Models and Data for Training Robust Speech Recognition Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QS3L25QI}},
  note         = {Machine review of arXiv:2508.20869}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Improvements in training data scale and quality have led to significant advances, yet its influence in speech recognition remains underexplored. In this paper, we present a large-scale dataset, OLMoASR-Pool, and series of models, OLMoASR, to study and develop robust zero-shot speech recognition models. Beginning from OLMoASR-Pool, a collection of 3M hours of English audio and 17M transcripts, we design text heuristic filters to remove low-quality or mistranscribed data. Our curation pipeline produces a new dataset containing 1M hours of high-quality audio-transcript pairs, which we call OLMoASR-Mix. We use OLMoASR-Mix to train the OLMoASR-Mix suite of models, ranging from 39M (tiny.en) to 1.5B (large.en) parameters. Across all model scales, OLMoASR achieves comparable average performance to OpenAI's Whisper on short and long-form speech recognition benchmarks. Notably, OLMoASR-medium.en attains a 12.8\% and 11.0\% word error rate (WER) that is on par with Whisper's largest English-only model Whisper-medium.en's 12.4\% and 10.5\% WER for short and long-form recognition respectively (at equivalent parameter count). OLMoASR-Pool, OLMoASR models, and filtering, training and evaluation code will be made publicly available to further research on robust speech processing.

Figures

Figures reproduced from arXiv: 2508.20869 by Huong Ngo, Josh Gardner, Ludwig Schmidt, Martijn Bartelds, Matt Deitke, Matt Jordan, Sarah Pratt.

Figure 1
Figure 1. Figure 1: Performance on LibriSpeech.test-clean (left) and average performance across 14 short [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Construction of OLMOASR-MIX from OLMOASR-POOL. Segmentation reduces OLMOASR-POOL from 3M to 2.4M hours. Percentages are relative to most recent filtered sub￾set, based on the number of hours or segments. tems. Recent work has illustrated inferior performance from training on automatic transcripts for speech recognition (Li et al., 2023) or a mix of human and machine-labeled data on translation systems (Fer… view at source ↗
Figure 3
Figure 3. Figure 3: Average performance across 14 short-form speech recognition benchmarks (left) and [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: We plot 11 supervised models trained on LibriSpeech without any robustness interventions [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: We plot the average performance of OLMOASR-74M on 14 short-form and 7 long-form evaluation sets, while varying the total data trained on. The fraction of OLMOASR-MIX used for training is based on number of hours. 5.2 RESULTS FROM TRAINING ON ACADEMIC DATASETS 100 200 300 400 500 Total data seen (in K hours) 15 16 17 18 19 Average WER Short-form Speech Recognition Trained on OWSM-Eng Trained on OLMoASR-Mix … view at source ↗
Figure 6
Figure 6. Figure 6: Short-form speech recognition performance of OLM [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: We plot 3 OLMOASR trained on OWSM-Eng data without any robustness interventions and demonstrate their WER on a reference test set (LibriSpeech test-clean) and the average WER across 2 out-of-distribution evaluation sets (CHiME-6, CORAAL). We also plot the performance of zero-shot OLMOASR models to compare to the former. 6 RELATED WORK Large-scale English ASR Datasets English ASR datasets have grown dramati… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. DataComp-VLM: Improved Open Datasets for Vision-Language Models

    cs.CV 2026-06 conditional novelty 8.0

    DataComp-VLM benchmark shows instruction-heavy data mixing outperforms filtering for VLM training, with DCVLM-Baseline achieving 63.6% on 33 tasks for 8B models (+5.4pp over FineVision).

  2. Phonemes vs. Projectors: An Investigation of Speech-Language Interfaces for LLM-based ASR

    eess.AS 2026-04 unverdicted novelty 7.0

    Phoneme-based interfaces match or surpass projector-based ones for LLM ASR, especially in low-resource languages, and a BPE-phoneme hybrid offers additional improvements.

  3. DataComp-VLM: Improved Open Datasets for Vision-Language Models

    cs.CV 2026-06 unverdicted novelty 6.0

    DataComp-VLM benchmark shows instruction-heavy data mixtures outperform caption-heavy ones for VLM training, with DCVLM-Baseline reaching 63.6% on 33 tasks using 200B tokens, +5.4pp over FineVision.

  4. Raon-OpenTTS: Open Models and Data for Robust Text-to-Speech

    eess.AS 2026-05 unverdicted novelty 5.0

    Raon-OpenTTS provides an open 510K-hour curated speech dataset and DiT-based TTS models up to 1B parameters that achieve competitive WER and speaker similarity on benchmarks versus closed models trained on millions of hours.

Reference graph

Works this paper leans on

46 extracted references · 10 canonical work pages · cited by 3 Pith papers · 2 internal anchors

  1. [1]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  2. [2]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  3. [3]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    work page

  4. [4]

    wav2vec 2.0: A framework for self-supervised learning of speech representations

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp.\ 12449--12460. Curran Associates, Inc., 2020. URL https://proceedings...

    work page 2020

  5. [5]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  6. [6]

    SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network

    William Chan, Daniel Park, Chris Lee, Yu Zhang, Quoc Le, and Mohammad Norouzi. Speechstew: Simply mix all available speech recognition data to train one large neural network, 2021. URL https://arxiv.org/abs/2104.02133

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  7. [7]

    Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio

    Guoguo Chen, Shuzhou Chai, Guan-Bo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, Mingjie Jin, Sanjeev Khudanpur, Shinji Watanabe, Shuaijiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongqing Wang, Zhao You, and Zhiyong Yan. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. I...

    doi:10.21437/interspeech.2021-1965 2021

  8. [8]

    Owls: Scaling laws for multilingual speech recognition and translation models, 2025

    William Chen, Jinchuan Tian, Yifan Peng, Brian Yan, Chao-Han Huck Yang, and Shinji Watanabe. Owls: Scaling laws for multilingual speech recognition and translation models, 2025. URL https://arxiv.org/abs/2502.10373

    Pith/arXiv arXiv 2025

  9. [9]

    Reproducible scaling laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 2818--2829, 2023

    work page 2023

  10. [10]

    Seamless Communication, Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, Christopher Klaiber, Pengwei Li, Daniel Licht, Jean Maillard, Alice Rakotoarison, Kaushik Ram Sadagopan, Guillaume Wenzek, Ethan Ye, Bapi Akula, Peng-Jen Chen, Naji El Hachem, B...

    Pith/arXiv arXiv 2023

  11. [11]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. URL https://arxiv.org/abs/2205.14135

    Pith/arXiv arXiv 2022

  12. [12]

    Scaling Laws for Multilingual Neural Machine Translation

    Patrick Fernandes, Behrooz Ghorbani, Xavier Garcia, Markus Freitag, and Orhan Firat. Scaling laws for multilingual neural machine translation, 2023. URL https://arxiv.org/abs/2302.09650

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  13. [13]

    Datacomp: In search of the next generation of multimodal datasets

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36: 0 27092--27112, 2023

    work page 2023

  14. [14]

    The people’s speech: A large-scale diverse english speech recognition dataset for commercial usage

    Daniel Galvez, Greg Diamos, Juan Torres, Keith Achorn, Juan Cer\' o n, Anjali Gopi, David Kanter, Max Lam, Mark Mazumder, and Vijay Janapa Reddi. The people’s speech: A large-scale diverse english speech recognition dataset for commercial usage. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Dataset...

    work page 2021

  15. [15]

    The pile: An 800gb dataset of diverse text for language modeling, 2020

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020. URL https://arxiv.org/abs/2101.00027

    Pith/arXiv arXiv 2020

  16. [16]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Efficient multimodal learning from data-centric perspective

    Muyang He, Yexin Liu, Boya Wu, Jianhao Yuan, Yueze Wang, Tiejun Huang, and Bo Zhao. Efficient multimodal learning from data-centric perspective. arXiv preprint arXiv:2402.11530, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Datacomp-lm: In search of the next generation of training sets for language models

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al. Datacomp-lm: In search of the next generation of training sets for language models. Advances in Neural Information Processing Systems, 37: 0 14200--14282, 2024

    2024

  19. [19]

    Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yu Hsieh, Dhruba Ghosh, Josh Gardn...

    Pith/arXiv arXiv 2025

  20. [21]

    Rethinking evaluation in asr: Are our models robust enough? In Interspeech 2021, pp.\ 311--315, 2021

    Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Paden Tomasello, Jacob Kahn, Gilad Avidov, Ronan Collobert, and Gabriel Synnaeve. Rethinking evaluation in asr: Are our models robust enough? In Interspeech 2021, pp.\ 311--315, 2021. doi:10.21437/Interspeech.2021-1758

    doi:10.21437/interspeech.2021-1758 2021

  21. [22]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  22. [23]

    Llm360: Towards fully transparent open-source llms

    Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, et al. Llm360: Towards fully transparent open-source llms. arXiv preprint arXiv:2312.06550, 2023

    Pith/arXiv arXiv 2023

  23. [24]

    2 olmo 2 furious

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656, 2024

    Pith/arXiv arXiv 2024

  24. [25]

    Librispeech: An asr corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 5206--5210, 2015. doi:10.1109/ICASSP.2015.7178964

    arXiv 2015

  25. [26]

    The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only

    Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116, 2023

    Pith/arXiv arXiv 2023

  26. [27]

    The fineweb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydl \' c ek, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37: 0 30811--30849, 2024 a

    work page 2024

  27. [28]

    The fineweb datasets: Decanting the web for the finest text data at scale, 2024 b

    Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024 b . URL https://arxiv.org/abs/2406.17557

    Pith/arXiv arXiv 2024

  28. [29]

    Reproducing whisper-style training using an open-source toolkit and publicly available data

    Yifan Peng, Jinchuan Tian, Brian Yan, Dan Berrebbi, Xuankai Chang, Xinjian Li, Jiatong Shi, Siddhant Arora, William Chen, Roshan Sharma, Wangyou Zhang, Yui Sudo, Muhammad Shakeel, Jee-Weon Jung, Soumi Maiti, and Shinji Watanabe. Reproducing whisper-style training using an open-source toolkit and publicly available data. In 2023 IEEE Automatic Speech Recog...

    arXiv 2023

  29. [30]

    Owsm-ctc: An open encoder-only speech foundation model for speech recognition, translation, and language identification, 2024 a

    Yifan Peng, Yui Sudo, Muhammad Shakeel, and Shinji Watanabe. Owsm-ctc: An open encoder-only speech foundation model for speech recognition, translation, and language identification, 2024 a . URL https://arxiv.org/abs/2402.12654

    Pith/arXiv arXiv 2024

  30. [31]

    Owsm v3.1: Better and faster open whisper-style speech models based on e-branchformer

    Yifan Peng, Jinchuan Tian, William Chen, Siddhant Arora, Brian Yan, Yui Sudo, Muhammad Shakeel, Kwanghee Choi, Jiatong Shi, Xuankai Chang, Jee weon Jung, and Shinji Watanabe. Owsm v3.1: Better and faster open whisper-style speech models based on e-branchformer. In Interspeech 2024, pp.\ 352--356, 2024 b . doi:10.21437/Interspeech.2024-1194

    doi:10.21437/interspeech.2024-1194 2024

  31. [32]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.\ 8748--8763. PmLR, 2021

    2021

  32. [33]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceed...

    work page 2023

  33. [34]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. URL https://arxiv.org/abs/1910.10683

    Pith/arXiv arXiv 2023

  34. [35]

    Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A

    Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander...

    Pith/arXiv arXiv 2024

  35. [37]

    Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset, 2025

    Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset, 2025. URL https://arxiv.org/abs/2412.02595

    Pith/arXiv arXiv 2025

  36. [38]

    Measuring robustness to natural distribution shifts in image classification, 2020

    Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification, 2020. URL https://arxiv.org/abs/2007.00644

    Pith/arXiv arXiv 2020

  37. [39]

    On the effects of heterogeneous data sources on speech-to-text foundation models

    Jinchuan Tian, Yifan Peng, William Chen, Kwanghee Choi, Karen Livescu, and Shinji Watanabe. On the effects of heterogeneous data sources on speech-to-text foundation models. In Interspeech 2024, pp.\ 3959--3963, 2024. doi:10.21437/Interspeech.2024-1938

    work page doi:10.21437/interspeech.2024-1938 2024

  38. [40]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023 a

    Pith/arXiv arXiv 2023

  39. [41]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023 b

    Pith/arXiv arXiv 2023

  40. [42]

    Voxlingua107: A dataset for spoken language recognition

    Jörgen Valk and Tanel Alumäe. Voxlingua107: A dataset for spoken language recognition. In 2021 IEEE Spoken Language Technology Workshop (SLT), pp.\ 652--658, 2021. doi:10.1109/SLT48900.2021.9383459

    arXiv 2021

  41. [43]

    Redpajama: an open dataset for training large language models, 2024

    Maurice Weber, Daniel Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. Redpajama: an open dataset for training large language models, 2024. URL https://arxiv....

    Pith/arXiv arXiv 2024

  42. [44]

    Organize the web: Constructing domains enhances pre-training data curation

    Alexander Wettig, Kyle Lo, Sewon Min, Hannaneh Hajishirzi, Danqi Chen, and Luca Soldaini. Organize the web: Constructing domains enhances pre-training data curation. arXiv preprint arXiv:2502.10341, 2025

    Pith/arXiv arXiv 2025

  43. [45]

    Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302, 2024

    Pith/arXiv arXiv 2024

  44. [46]

    Yu Zhang, Daniel S. Park, Wei Han, James Qin, Anmol Gulati, Joel Shor, Aren Jansen, Yuanzhong Xu, Yanping Huang, Shibo Wang, Zongwei Zhou, Bo Li, Min Ma, William Chan, Jiahui Yu, Yongqiang Wang, Liangliang Cao, Khe Chai Sim, Bhuvana Ramabhadran, Tara N. Sainath, Françoise Beaufays, Zhifeng Chen, Quoc V. Le, Chung-Cheng Chiu, Ruoming Pang, and Yonghui Wu. ...

    arXiv 2022

  45. [47]

    Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, Zhong Meng, Ke Hu, Andrew Rosenberg, Rohit Prabhavalkar, Daniel S. Park, Parisa Haghani, Jason Riesa, Ginger Perng, Hagen Soltau, Trevor Strohman, Bhuvana Ramabhadran, Tara Sainath, Pedro Moreno, Chung-Cheng Chiu, Johan Schalkwyk, François...

    Pith/arXiv arXiv 2023

  46. [48]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.