Pith. sign in

REVIEW 3 major objections 5 minor 20 references

FAMA: The First Large-Scale Open-Science Speech Foundation Model for English and Italian

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FAMA claims that a speech foundation model trained only on open-source data can match Whisper on English and Italian ASR while running up to eight times faster, backed by released code, data, and weights.

desk verdict FAMA is a genuinely useful open-science speech resource, but the evaluation split ambiguity must be resolved before the headline numbers can be trusted. read the letter →

arxiv 2505.22759 v2 pith:WWB52JRO submitted 2025-05-28 cs.CL cs.AIcs.SD

Sara Papi , Marco Gaido , Luisa Bentivogli , Alessio Brutti , Mauro Cettolo , Roberto Gretter , Marco Matassoni , Mohamed Nabih
show 1 more author
Matteo Negri
This is my paper
classification cs.CLcs.AIcs.SD
keywords speechfoundationmodelopenscienceautomaticrecognitiontranslationEnglishItalianpseudo-labelingConformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces FAMA, a family of two open-science speech foundation models for English and Italian, trained entirely on more than 150,000 hours of speech data released under open-source-compliant licenses. The claim is that FAMA achieves automatic speech recognition performance competitive with Whisper while running up to eight times faster, and that it outperforms the open-source OWSM system on both ASR and speech translation. The paper also contributes a new dataset of roughly 16,000 hours of cleaned and pseudo-labeled speech, plus automatic translations for over 130,000 additional hours. The aim is to show that fully transparent, legally reusable speech models can be built at scale without sacrificing practical quality.

What carries the argument

The load-bearing design is an encoder-decoder where a Conformer encoder (24 layers in the medium model) is paired with a Transformer decoder half as deep (12 layers), so each autoregressive step is cheaper while the encoder retains most of the representational capacity. Training combines a label-smoothed cross-entropy loss with two CTC losses, one on an intermediate encoder layer and one on the final encoder output, and proceeds in two stages: ASR-only pre-training followed by joint ASR and speech-translation training with balanced task sampling. A new 16,000-hour dataset is built from permissively licensed YouTube audio, cleaned and segmented with Silero VAD and SHAS, and pseudo-labeled with Whisper large-v3 for transcripts and MADLAD-400 for translations.

What would settle it

Inspect the released FAMA training data and check whether any audio segments from the exact evaluation test splits of CommonVoice, MLS, VoxPopuli, CoVoST2, and FLEURS appear in it; if any do, the reported WER and COMET figures would be inflated.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that a large-scale speech foundation model does not need proprietary training data or closed code: FAMA is trained exclusively on open-source-compliant audio, with all data, code, and weights released, and still reaches near-Whisper accuracy on English and Italian ASR while being markedly faster. The authors report that FAMA medium outperforms Whisper medium on English and Italian ASR, approaches Whisper large-v3, and improves over OWSM v3.1 medium by up to 0.152 COMET on speech translation. The speed gain is attributed to a Conformer encoder that is twice as deep as the decoder, cutting per-pass autoregressive cost, and to the absence of fixed-length padding.

Load-bearing premise

The reported ASR and ST scores assume that the evaluation test splits of CommonVoice, MLS, VoxPopuli, CoVoST2, and FLEURS were excluded from the training data, but the paper never explicitly states this exclusion.

Editorial extensions

If this is right

  • If the results hold, open-science speech models can be competitive with closed ones for well-resourced languages, making speech technology reproducible and legally reusable.
  • The shallower-decoder architecture offers a simple recipe for faster inference that other speech model families could adopt.
  • The released training artifacts enable direct contamination checks and fair benchmarking of future models.
  • The two-stage training with balanced ASR and ST sampling avoids catastrophic forgetting while adding a new task, as shown by perplexity curves.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension is whether the same two-stage, balanced-sampling recipe transfers to additional EU languages using the MOSEL dataset, which would broaden the open-science speech ecosystem beyond English and Italian.
  • The speed advantage likely grows on longer audio because the decoder's shallowness reduces per-token generation cost; the paper's xRTF numbers on CommonVoice could be complemented by measurements on longer-form speech.
  • The authors' reliance on Whisper for pseudo-labels means the open-data pipeline inherits Whisper's transcription errors; comparing FAMA trained on gold versus pseudo-labeled subsets could quantify this ceiling.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces FAMA, a family of two encoder-decoder speech foundation models (small and medium) trained on more than 150k hours of open-source-compliant speech data for English and Italian. It also contributes a new pseudo-labeled corpus of about 16k hours and automatic translations for over 130k hours. The authors report ASR (WER) results on CommonVoice, MLS, and VoxPopuli and speech translation (COMET) results on CoVoST2 and FLEURS, comparing against Whisper, SeamlessM4T, and OWSM, and they report inference speedups of up to 8x. All code, data, and model weights are released under open licenses.

Significance. If the empirical claims hold, FAMA is a valuable open-science contribution: it is the first large-scale speech foundation model trained exclusively on OS-compliant data for two languages, and it demonstrates that such models can approach Whisper-level ASR quality at higher speed while outperforming OWSM. The release of code, training data, and models is a concrete strength that the community can build on. However, the central 'competitive performance' claim rests entirely on Table 3, and the manuscript currently does not establish that the evaluation splits were excluded from the training data. This is a load-bearing omission that must be resolved before the headline results can be accepted.

major comments (3)
  1. [§2.3 and Tables 1-3] The manuscript never states that the evaluation splits of CommonVoice, MLS, VoxPopuli, CoVoST2, and FLEURS were excluded from the training sets listed in Tables 1 and 2. Since those datasets appear as training sources and §2.3 says evaluation is conducted on the same benchmarks, the WER and COMET numbers in Table 3 may be optimistically biased by train/test overlap. This is load-bearing because the claim that FAMA is competitive with Whisper and outperforms OWSM depends on those numbers. Please state explicitly which split identifiers (e.g., CommonVoice test, MLS test-other, VoxPopuli test, CoVoST2 test, FLEURS test) were used, confirm that no audio segment or transcript from any evaluation set appears in training or in the automatically labeled YouTube-Commons data, and report overlap statistics or a re-evaluation on disjoint held-out subsets.
  2. [§3.2 and Table 3] All reported ASR and ST results come from single runs with no error bars, confidence intervals, or significance tests. Several conclusions are based on small WER differences, such as the 0.8 WER improvement of FAMA-medium over FAMA-small and the gains over OWSM. Without variance estimates or at least paired significance tests, it is unclear whether these differences are reliable. Please provide multiple seeds or bootstrap/paired significance results for at least the main comparisons.
  3. [§2.4 and Table 3] The comparison with OWSM uses a beam size of 3 for OWSM while all other models are decoded with beam size 5, explicitly because of out-of-memory issues. Since OWSM is the main open-science baseline and the paper claims consistent improvements over it, this uncontrolled decoding setup could bias the comparison. Please either report OWSM results with beam 5 under memory-efficient settings or demonstrate that beam size does not alter the conclusions.
minor comments (5)
  1. [Introduction] The sentence 'with up to 4.2 WER and 0.152 COMET improvement on average across languages compared to OWSM' is not directly traceable to Table 3; please specify which model configuration and language yield the 4.2 WER improvement.
  2. [§2.1, Table 1] The column header '#hours Labelen it' is confusing; consider splitting the label information into separate 'en' and 'it' label columns or rephrasing the header.
  3. [§2.3] The statement that 'CommonVoice also serving as the validation set for both en and it' should specify whether this is the official validation split and that this split is disjoint from the reported CommonVoice test results.
  4. [§3.3] The phrase 'spanning in the range 2, 4, 8, and 16' is ungrammatical; please rephrase. Also, define xRTF at first use.
  5. [Throughout] There are several typographical and wording issues, including 'themself' in the introduction, 'Y ouTube' in §2.1, and the missing article in 'the learning rate (lrS1) scheduler adopted' in §2.3.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: FAMA's ASR/ST results are empirical evaluations against external gold benchmarks; no derivation reduces to its inputs by construction. The train/eval overlap ambiguity is an evaluation-hygiene risk, not circularity.

full rationale

The paper makes no first-principles derivation or fitted-parameter-as-prediction claim. Its central claims—competitive WER/COMET and higher throughput—are empirical results from training Conformer-transformer models on open datasets and testing on public gold benchmarks. Hyperparameters (pASR, lrS2) are selected from validation perplexity curves and then fixed for the final models; this is standard model selection, not a fitted input renamed as a prediction. Self-citations (MOSEL data, FBK-fairseq Conformer fix, SFM+LLM survey) supply data, software, and motivation but do not constitute the proof of the headline result; none is a uniqueness theorem or an ansatz that forces the outcome. The only self-referential concern is that Table 1 lists CommonVoice, MLS, VoxPopuli, CoVoST2, and FLEURS as training sources while Section 2.3 evaluates on those same benchmarks, and the text does not explicitly state that the official test splits were excluded from training. If the splits overlapped, the reported numbers would be optimistically biased; however, that is a data-contamination/experimental-design question, not a circular derivation. The paper's own framing—releasing all artifacts to 'avoid data contamination issues'—indicates awareness, but the split-exclusion statement is absent. This ambiguity does not satisfy the standard for flagging circularity, which requires exhibiting a specific reduction of the result to its inputs by definition or by a self-citation chain.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central performance claims rest on several unverified domain assumptions: the accuracy of the teacher models used for pseudo-labeling, the legal/OS status of the source data, and the (unstated) disjointness of training and evaluation splits. The hyperparameters chosen by hand (loss weights, sampling probability, learning rates, CTC weight, filtering thresholds) are all free parameters that affect the reported numbers.

free parameters (5)
  • Loss weights lambda1, lambda2, lambda3 = 5.0, 1.0, 2.0
    Chosen by hand in Section 2.3 to combine CE, source CTC, and target CTC losses; directly shape the trained model.
  • ASR sampling probability pASR = 0.5
    Selected based on validation perplexity in Section 3.1; balances ASR and ST objectives and controls catastrophic forgetting.
  • Second-stage learning rate lrS2 = 1e-4 (small), 1e-5 (medium)
    Chosen in Section 3.1 to avoid ASR perplexity degradation; higher values caused 3-4 WER drops.
  • CTC rescoring weight = 0.2
    Used at inference for joint CTC rescoring reported in Table 3.
  • ST length ratio thresholds rmin/rmax = 0.75/1.45 (en-it), 0.65/1.35 (it-en)
    Used to filter machine-translated training pairs; based on source-target character length distributions.
assumptions (5)
  • domain assumption Whisper large-v3 transcripts are accurate enough to serve as training labels for 16,028 hours of YouTube-Commons speech.
    Section 2.1 states that automatic transcripts were created with Whisper large-v3 and then used as training labels; no quality check is reported.
  • domain assumption MADLAD-400 3B-MT translations are accurate enough to serve as ST training targets for over 130k hours.
    Section 2.1 states all speech transcripts were automatically translated with MADLAD-400 and used to train ST; no quality check is reported.
  • domain assumption The evaluation splits of CommonVoice, MLS, VoxPopuli, CoVoST2, and FLEURS are disjoint from the corresponding training splits.
    Section 2.1 lists these datasets in the training set while Section 2.3 evaluates on them, without stating that test portions were excluded.
  • domain assumption YouTube-Commons audio is OS-compliant under CC-BY 4.0 and legally usable for training and release.
    Section 2.1 cites the YouTube-Commons dataset license but provides no independent verification that all audio is CC-BY.
  • domain assumption The hardware and batching conditions in Section 3.3 are representative of practical deployment throughput.
    xRTF is measured on a single A40 with maximum batch sizes per model, which may favor the smaller FAMA models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FAMA: The First Large-Scale Open-Science Speech Foundation Model for English and Italian." pith.science (2026). https://pith.science/paper/WWB52JRO

@misc{pith2026250522759,
  author       = {Pith},
  title        = {Pith review of: FAMA: The First Large-Scale Open-Science Speech Foundation Model for English and Italian},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WWB52JRO}},
  note         = {Machine review of arXiv:2505.22759}
}
read the original abstract

The development of speech foundation models (SFMs) like Whisper and SeamlessM4T has significantly advanced the field of speech processing. However, their closed nature--with inaccessible training data and code--poses major reproducibility and fair evaluation challenges. While other domains have made substantial progress toward open science by developing fully transparent models trained on open-source (OS) code and data, similar efforts in speech remain limited. To fill this gap, we introduce FAMA, the first family of open science SFMs for English and Italian, trained on 150k+ hours of OS speech data. Moreover, we present a new dataset containing 16k hours of cleaned and pseudo-labeled speech for both languages. Results show that FAMA achieves competitive performance compared to existing SFMs while being up to 8 times faster. All artifacts, including code, datasets, and models, are released under OS-compliant licenses, promoting openness in speech technology research.

Figures

Figures reproduced from arXiv: 2505.22759 by the authors.

Figure 1
Figure 1. Average ASR and ST perplexity (ppl) on both English and Italian up to 500k steps of the [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

20 extracted references · 14 canonical work pages

  1. [3]

    In Findings of the Association for Computational Linguistics: ACL 2023 , pages 3676–3687, Toronto, Canada

    Non-repeatable experiments and non-reproducible results: The reproducibility crisis in human evaluation in NLP. In Findings of the Association for Computational Linguistics: ACL 2023 , pages 3676–3687, Toronto, Canada. Associa- tion for Computational Linguistics. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Halla...

    work page 2023

  2. [5]

    In 2022 IEEE Spoken Language T echnology Workshop (SL T), pages 798–805

    Fleurs: Few-shot learning evaluation of universal representa- tions of speech. In 2022 IEEE Spoken Language T echnology Workshop (SL T), pages 798–805. Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Y ang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping

    work page 2022

  3. [6]

    arXiv preprint arXiv:2409.11402

    Nvlm: Open frontier-class multimodal llms. arXiv preprint arXiv:2409.11402 . Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Y ang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al

    arXiv

  4. [7]

    arXiv preprint arXiv:2409.17146

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146 . Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Y ang, and Ge Li

    arXiv

  5. [8]

    In Findings of the Association for Computational Linguistics: ACL 2024 , pages 12039–12050, Bangkok, Thailand

    Generaliza- tion or memorization: Data contamination and trustworthy evaluation for large language models. In Findings of the Association for Computational Linguistics: ACL 2024 , pages 12039–12050, Bangkok, Thailand. Association for Computational Linguistics. FAMA: The First Large-Scale Open-Science Speech Foundation Model for English and Italian 12 Marc...

    work page 2024

  6. [13]

    In Interspeech 2024, pages 352–356

    Owsm v3.1: Better and faster open whisper-style speech models based on e-branchformer. In Interspeech 2024, pages 352–356. Yifan Peng, Jinchuan Tian, Brian Y an, Dan Berrebbi, Xuankai Chang, Xinjian Li, Jiatong Shi, Siddhant Arora, William Chen, Roshan Sharma, Wangyou Zhang, Yui Sudo, Muhammad Shakeel, Jee-Weon Jung, Soumi Maiti, and Shinji Watanabe

    work page 2024

  7. [14]

    In 2023 IEEE Automatic Speech Recognition and Under- standing Workshop (ASRU), pages 1–8

    Reproducing whisper-style training using an open- source toolkit and publicly available data. In 2023 IEEE Automatic Speech Recognition and Under- standing Workshop (ASRU), pages 1–8. Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert

    work page 2023

  8. [15]

    MLS: A Large-Scale Multilingual Dataset for Speech Research. In Proc. Interspeech 2020 , pages 2757–

    work page 2020

Show all 20 references
  1. [16]

    In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online

    COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, Dav...

    2020

  2. [17]

    In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 46–52, Miami, Florida, USA

    OpenOmni: A collaborative open source tool for building future-ready multimodal conversational agents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 46–52, Miami, Florida, USA. Association for Computati...

    2024

  3. [18]

    In Interspeech 2022 , pages 106–110

    Shas: Approaching optimal segmentation for end-to-end speech translation. In Interspeech 2022 , pages 106–110. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin

    2022

  4. [19]

    In Proceedings of the 2020 Conference of the Asian Chapter of the Association for Computational Linguistics (AACL): System Demonstrations

    fairseq S2T: Fast speech-to-text modeling with fairseq. In Proceedings of the 2020 Conference of the Asian Chapter of the Association for Computational Linguistics (AACL): System Demonstrations . Changhan Wang, Anne Wu, Jiatao Gu, and Juan Pino. 2021b. CoVoST 2 and Massively M...

    2020

  5. [20]

    arXiv preprint arXiv:2403.13784

    The model openness framework: Promoting completeness and openness for reproducibility, transparency, and usability in artificial intelligence. arXiv preprint arXiv:2403.13784. Brian Y an, Siddharth Dalmia, Y osuke Higuchi, Graham Neubig, Florian Metze, Alan W Black, and Shinji...

    arXiv

  6. [2015]

    In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 5206–5210

    Librispeech: An ASR corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 5206–5210. Sara Papi, Marco Gaido, Andrea Pilzer, and Matteo Negri

    2015

  7. [2018]

    In Proceedings of the 2018 Conference on Em- pirical Methods in Natural Language Processing: System Demonstrations , pages 66–71, Brussels, Belgium

    SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Em- pirical Methods in Natural Language Processing: System Demonstrations , pages 66–71, Brussels, Belgium. Association for C...

    2018

  8. [2019]

    SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition. In Proc. Interspeech 2019 , pages 2613–2617. Yifan Peng, Jinchuan Tian, William Chen, Siddhant Arora, Brian Y an, Yui Sudo, Muhammad Shakeel, Kwanghee Choi, Jiatong Shi, Xuankai Chang, Jee weon Ju...

    2019

  9. [2020]

    Conformer: Convolution-augmented Transformer for Speech Recognition. In Proc. Interspeech 2020 , pages 5036–5040. Sudipta Kar, Giuseppe Castellucci, Simone Filice, Shervin Malmasi, and Oleg Rokhlenko

    2020

  10. [2022]

    arXiv preprint arXiv:2211.05100

    Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 . Henry Chesbrough

    arXiv

  11. [2023]

    arXiv preprint arXiv:2308.11596

    Seamlessm4t: Massively multilingual & multimodal machine translation. arXiv preprint arXiv:2308.11596 . Anya Belz, Craig Thomson, Ehud Reiter, and Simon Mille

    arXiv

  12. [2024]

    arXiv preprint arXiv:2402.01945

    A case study on filtering for end-to-end speech translation. arXiv preprint arXiv:2402.01945 . R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F . M. Tyers, and G. Weber

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.