Pith. sign in

REVIEW 3 major objections 4 minor 14 references

This paper claims that published French benchmark scores from FLEURS and CommonVoice do not predict how well 24 pretrained speech recognizers handle spontaneous Québec French, and that a domain-matched benchmark built from public inquiry he

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

On a new Quebec French benchmark, standard multilingual ASR benchmarks poorly predict real-world performance; the best open model reaches 8% WER while typical models sit near 14%.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Careful QF ASR benchmark with a plausible domain-gap claim, but the headline cross-benchmark comparison rests on unpublished external numbers and a single fragile rank reversal. the 3 major comments →

arxiv 2508.21193 v1 pith:KCOCLIC5 submitted 2025-08-28 eess.AS

Benchmarking Large Pretrained Multilingual Models on Qu\'ebec French Speech Recognition

Coralie Serrand , Gilles Boulianne , Amira Morsli This is my paper
classification eess.AS
keywords speech recognitionQuébec Frenchbenchmarkingword error ratemultilingual pretrained modelsspontaneous speechsemantic similaritydomain-matched evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that published word-error rates on clean, read French benchmarks such as FLEURS and CommonVoice do not transfer to spontaneous Québec French as spoken in public inquiry hearings. It builds an evaluation pipeline over the CommissionsQC corpus, tests 24 pretrained models without fine-tuning, and finds large rank reversals: models that look worst on CommonVoice can be best on Québec French, and vice versa. The practical point is that practitioners serving Québec French need a domain-matched benchmark, not global-French leaderboards, when choosing a model. The paper also reports speed and semantic fidelity, so the comparison covers cost and meaning retention, not just word errors.

Core claim

This paper claims that performance on CommissionsQC—a 782-hour corpus of spontaneous Québec French from public inquiry hearings—is the relevant yardstick for Québec French automatic speech recognition, and that on that yardstick the published French results for FLEURS and CommonVoice do not predict model quality. Across 24 pretrained models, the best public result is whisper-large-v3-turbo at 8.2% word error rate with a real-time factor of 6%, essentially tied with a from-scratch model trained on the same corpus; French-fine-tuned variants of Whisper score worse than their multilingual base versions. The paper reports rank reversals, including whisper-large-v3 ranking first on CommissionsQC

What carries the argument

The load-bearing object is CommissionsQC, a Québec French speech corpus built from two public inquiry hearings, with 782 hours of training and 27 hours of development and test speech in spontaneous, conversation-style audio with reverberation and background noise. Around it, the paper builds a single evaluation pipeline that applies the same data preparation, transcription, text normalization (a French-aware 'basic' normalizer that splits compound words and converts numbers), and scoring to every model, producing word and character error rates, real-time factor, and Bert F1 semantic similarity. The pipeline is what makes the cross-model and cross-benchmark comparisons interpretable; the corp

Load-bearing premise

The non-predictivity claim depends on the published FLEURS and CommonVoice French error rates being comparable to each other and to this paper's measurements, but those numbers are taken from model cards and literature rather than recomputed inside the paper's own pipeline.

What would settle it

Re-run the nine overlapping models on the French subsets of FLEURS and CommonVoice with the same normalization and scoring code used for CommissionsQC; if their rankings then match the CommissionsQC ranking, the central non-predictivity claim would be disproved. A second check: transcribe CommissionsQC audio after noise reduction; if errors drop to benchmark-like levels, the gap is largely acoustic rather than dialectal.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • The best publicly available model, whisper-large-v3-turbo, reaches 8.2% word error rate at 6% real-time factor on CommissionsQC, matching a from-scratch model trained on 780 hours of Québec French.
  • French-fine-tuned variants can underperform their multilingual base versions on regional French, so 'fine-tuned for French' is not automatically better for Québec French.
  • FLEURS and CommonVoice leaderboard ranks should not be used to select a speech recognizer for spontaneous Québec French; domain-matched evaluation changes the ordering.
  • Cloud regional French services land in the middle of the pack and run much slower than open-weight local models, so the best accuracy-speed tradeoff in this benchmark is open-source.
  • Multimodal models are highly sensitive to prompt phrasing for this task, with observed differences of about 10% absolute word error rate across prompts.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same rank-reversal pattern is likely to appear for other regional and low-resource varieties of major languages, because the confounds of dialect, spontaneous style, and room acoustics are not unique to Québec French.
  • Editorial inference: the paper's imported benchmark numbers make the specific ordering in its cross-benchmark figure less settled than the qualitative non-predictivity claim; an apples-to-apples recomputation could shift some ranks while leaving the overall conclusion intact.
  • Editorial inference: the authors' stated next step of fine-tuning on CommissionsQC is the direct test of how much of the domain gap is learnable; the from-scratch result suggests a Whisper-size fine-tune could plausibly go below today's 8% word error rate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper evaluates 24 publicly available large pretrained multilingual speech recognition models on the CommissionsQC corpus of spontaneous Québec French speech. Using a uniform evaluation pipeline, it reports WER, CER, real-time factor, and BERTScore, and compares its measurements against published FLEURS and CommonVoice French WER numbers for nine models. The central claim is that published benchmark results are not good predictors of performance on CommissionsQC, illustrated by rank reversals and large WER gaps. The paper also reports gender-balanced analysis, normalization sensitivity, and speed/accuracy trade-offs, and compares against two models trained from scratch on the in-domain data.

Significance. If the non-predictivity result holds, it is a useful caution for practitioners selecting speech models for regional varieties, where standard benchmarks may be misleading. The paper's strengths are its unified evaluation pipeline, careful normalization analysis (Section 3.7), gender breakdown (Table 6), candid limitations, and comparison of many model families. However, the load-bearing cross-benchmark comparison in Section 3.6/Figure 3 depends on external published numbers that are not enumerated, and the ranking claims lack statistical support. These issues are fixable and do not undermine the value of the dataset evaluation itself, but they must be addressed before the headline claim can be fully credited.

major comments (3)
  1. [Section 3.6, Figure 3] The central non-predictivity claim relies on published FLEURS/CommonVoice WERs for nine models, but these numbers are not enumerated anywhere. The paper must provide a table listing, for each model, the benchmark WER, the exact source (model card/paper), model checkpoint/version, benchmark split, and text normalization used. This is not a minor omission: Section 3.7 shows that switching from the paper's basic normalization to the whisper normalizer changes WER for whisper-large-v3 by 1.2% absolute, and several models in Table 3 are within 1-2% of each other. A stale or mismatched CommonVoice point for whisper-large-v3 could invert the headline rank-reversal ('worst on CV, best on CommissionsQC'), undermining the abstract's claim that published results 'are not good predictors.'
  2. [Table 3, Section 3] The rankings in Table 3 have no error bars, confidence intervals, or significance tests. The top two models differ by 0.2% absolute WER (8.2 vs 8.4), and several models are separated by <1%. Since the test sets are fixed (262K words), bootstrap confidence intervals or paired significance tests are feasible and would support the ranking claims. As written, the reader cannot distinguish true differences from noise, which weakens the practical model-selection guidance that is a stated contribution.
  3. [Section 3.5] Prompts for multimodal models were selected by best performance on the Bast development set, whereas speech-only models received no equivalent prompt tuning. This breaks comparability: Phi-4 and gemini-2.0 may be overfit to the development set, and the speech-only models are at a systematic disadvantage in Table 3. The Limitations section acknowledges the subjectivity, but the main results treat multimodal and speech-only models as directly comparable. Please report the recommended-prompt results as the primary comparison, or provide a sensitivity analysis showing that the ranking is robust to prompt choice.
minor comments (4)
  1. [Section 2.4] Typo: 'Levenstein' should be 'Levenshtein'.
  2. [Section 3.7 / Figure 3] The statement that the WER gap in Figure 3 is 'probably underestimated' assumes published results used whisper normalization. Since the normalizers for the external FLEURS/CommonVoice numbers are not documented, please soften this or state the assumption explicitly per model.
  3. [Table 2] whisper-tiny has a '-' for parameters; complete the entry for consistency. Also note cloud models' RTF includes waiting for processing power (Section 3.2), so the RTF column mixes local and cloud conditions; a footnote should flag which values are contaminated.
  4. [General] No code or pipeline release is mentioned. Given the goal of a reproducible benchmark, even a small release of the evaluation scripts (with synthetic or sample audio) would strengthen the paper.

Circularity Check

0 steps flagged

No significant circularity: empirical benchmark with external comparisons.

full rationale

This is an empirical evaluation paper, not a derivation. The authors measure WER, CER, RTF, and BERTScore of 24 pretrained models on their own CommissionsQC corpus and compare these measured values with published French FLEURS/CommonVoice numbers. The central claim—that published benchmark results do not predict CommissionsQC performance—is an observed cross-benchmark comparison, not the output of a fitted model. No parameter is fitted to a subset of data and then renamed a prediction. The only self-referential element is the citation of the authors' own CommissionsQC dataset (Serrand et al., 2025) as the evaluation data; this is a data resource, not a load-bearing argument whose conclusion is contained in its premise. The paper's Section 3.7 even acknowledges the normalization sensitivity of the comparison, which is a correctness/reproducibility caveat rather than circularity. The external FLEURS/CommonVoice numbers are taken from model cards and literature, but using external observations as comparators does not make the claim circular; at worst it introduces measurement heterogeneity, which the paper partially discloses. Overall, the derivation chain is independent of the conclusions, and the paper is self-contained against external benchmarks. Score 1 reflects the negligible presence of a self-citation (the dataset) that is not load-bearing.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

No fitted parameters enter the central claim; the benchmark is observational. The main assumptions are about data quality, comparability of published benchmark numbers, and the validity of BERTScore as a semantic metric.

axioms (5)
  • domain assumption CommissionsQC reference transcripts and speaker splits are accurate and free of leakage
    The benchmark's WER numbers are computed against these transcripts; if transcripts or splits are wrong, all model rankings are affected. See Section 2.1, relying on Serrand et al. 2025.
  • domain assumption Published FLEURS and CommonVoice French results used in Figure 3 are obtained under comparable evaluation conditions
    The non-predictivity claim is a cross-benchmark comparison, so the external published numbers must be trustworthy and comparable. See Section 3.6 and Figure 3.
  • domain assumption BERTScore with multilingual BERT is a valid proxy for semantic accuracy in Quebec French
    Used to compute Bert F1; if the embedding model is poor for QF, semantic conclusions weaken. See Section 2.4.
  • domain assumption Results generalize from two public inquiry hearings to Quebec French at large
    Authors themselves note informal registers, youth or rural speech, telephone conversations, and everyday dialogue are not represented. See Section 5 Limitations.
  • domain assumption No significant data contamination of CommissionsQC in model pretraining
    Authors state they verified recordings are no longer publicly accessible but cannot entirely rule out indirect exposure. See Section 5 Limitations.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking Large Pretrained Multilingual Models on Qu\'ebec French Speech Recognition." pith.science (2026). https://pith.science/paper/KCOCLIC5

@misc{pith2026250821193,
  author       = {Pith},
  title        = {Pith review of: Benchmarking Large Pretrained Multilingual Models on Qu\'ebec French Speech Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KCOCLIC5}},
  note         = {Machine review of arXiv:2508.21193}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We evaluate the performance of large pretrained multilingual speech recognition models on a regional variety of French spoken in Qu\'ebec, Canada, in terms of speed, word error rate and semantic accuracy. To this end we build a benchmark and evaluation pipeline based on the CommissionsQc datasets, a corpus of spontaneous conversations recorded during public inquiries recently held in Qu\'ebec. Published results for these models on well-known benchmarks such as FLEURS or CommonVoice are not good predictors of the performance we observe on CommissionsQC. Our results should be of interest for practitioners interested in building speech applications for realistic conditions or regional language varieties.

Figures

Figures reproduced from arXiv: 2508.21193 by Amira Morsli, Coralie Serrand, Gilles Boulianne.

Figure 1
Figure 1. Figure 1: Evaluation pipeline. The input to the pipeline is a dataset, a model and a normalizer. Once these are loaded, the data preparation and transcription steps are executed. The resulting transcribed text is normalized to com￾pensate for differences in orthographic variants, as described in section 2.4, before computing eval￾uation metrics such as word error rate. The only model-dependent elements are the data … view at source ↗
Figure 2
Figure 2. Figure 2: Word error rate as a function of RTF for the tested models. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance on french FLEURS, french CommonVoice and CommissionsQC (averaged over Bast and Charb development and test sets), sorted in order of increasing error rate on FLEURS. Normalization %WER BERT F1 basic 8.17 95.88 whisper 9.39 95.15 [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 8 canonical work pages · 1 internal anchor

  1. [1]

    Benchmarking Large Pretrained Multilingual Models on Qu\'ebec French Speech Recognition

    Introduction The availability of large multilingual datasets such as MLS (Multilingual LibriSpeech (Pratap et al., 2020), VoxPopuli (Wang et al., 2021), and Common Voice (Ardila et al., 2020) have been instrumental in the development of large pretrained speech recog- nition models. However, these datasets include only the main varieties of English and oth...

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  2. [2]

    Experiments For back-to-back comparison, all experiments are performed using the same pipeline implementation. All evaluations are done with pretrained models as available, without retraining or fine-tuning, ex- cept for two models trained from scratch to estab- lish a baseline for the performance that can be attained when in-domain data is available in s...

    work page 2025

  3. [3]

    cloud" in Table 2 are avail- able through online transcription services. In all cases we used the

    Results Performance evaluation results appear in Table 3, as aggregated over the four sets of Bast and Charb development and test sets, comprising a total 262K words. More detailed results are provided in Ap- pendix A. The best performing model, in terms of error rate as well as Bert F1 score, is espnet_transformer, a model trained from scratch on the Bas...

    work page 2025

  4. [4]

    Our Figure 2: Word error rate as a function of RTF for the tested models

    Conclusion We evaluated 24 pretrained multilingual speech recognition models on CommissionsQC, a Québec French benchmark of public inquiries, in terms of speed, word error rate and semantic accuracy. Our Figure 2: Word error rate as a function of RTF for the tested models. Figure 3: Performance on french FLEURS, french CommonVoice and CommissionsQC (avera...

    work page

  5. [5]

    A key constraint is the difficulty of isolating spe- cific factors that influence model performance

    Limitations This study has several limitations related to both the nature of the dataset and the evaluation setup. A key constraint is the difficulty of isolating spe- cific factors that influence model performance. The CommissionsQC corpus combines multiple chal- lenging dimensions—including a regional variety of French, spontaneous speech, and realistic...

    work page

  6. [6]

    Ethical considerations This work raises ethical concerns primarily related to the nature of the audio data and the absence of explicit speaker consent. Although the recordings in CommissionsQC were originally captured during public hearings and are part of the public record, the individuals recorded did not provide informed consent for their voices to be ...

    work page

  7. [7]

    Bibliographical References Abderrahman Abouelenin, Atabak Ashfaq, Atkin- son, Adam, et al. 2025. Phi-4-Mini Tech- nical Report: Compact yet Powerful Multi- modal Language Models via Mixture-of-LoRAs. ArXiv:2503.01743 [cs.CL]. Alëna Aksënova, Zhehuai Chen, Chung-Cheng Chiu, Daan van Esch, Pavel Golik, et al

    Pith/arXiv arXiv 2025

  8. [11]

    Robust Speech Recognition via Large- Scale Weak Supervision. In Proc. ICML, pages 28492–28518. Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, et al. 2021. SpeechBrain: A General-Purpose Speech Toolkit. ArXiv:2106.04624 [eess]. Dima Rekesh, Nithin Rao Koluguri, Samuel Kri- man, Somshubra Majumdar, Vahid Noroozi, et al

    Pith/arXiv arXiv 2021

  9. [12]

    Fast Conformer with Linearly Scalable At- tention for Efficient Speech Recognition. In Proc. ASRU, pages 1–8. Andrew Rosenberg, Yu Zhang, Bhuvana Ramab- hadran, Y e Jia, Pedro Moreno, Y onghui Wu, and Zelin Wu. 2019. Speech Recognition with Aug- mented Synthesized Speech. In Proc. ASRU, pages 996–1002, SG, Singapore. IEEE. Anthony Rousseau, Gilles Boulian...

    work page 2019

  10. [2016]

    ArXiv:1508.07909

    Neural Machine Translation of Rare Words with Subword Units. ArXiv:1508.07909. Coralie Serrand, Amira Morsli, and Gilles Bou- lianne. 2025. CommissionsQC: a Québec French speech corpus for automatic speech recognition. In Proc. Interspeech, pages 3918– 3922. Jimmy Tobin, Qisheng Li, Subhashini Venu- gopalan, Katie Seaver, Richard Cave, and Katrin Tomanek....

    Pith/arXiv arXiv 2025

  11. [2021]

    VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation. In Proc. ACL, pages 993–1003. Shinji Watanabe, Florian Boyer, Xuankai Chang, Pengcheng Guo, Tomoki Hayashi, et al. 2020. The 2020 ESPnet update: new features, broad- ened applications, performance improvements, and future plans. In...

    Pith/arXiv arXiv 2020

  12. [2022]

    ArXiv:2205.08014 [eess]

    Accented Speech Recognition: Bench- marking, Pre-training, and Diverse Data. ArXiv:2205.08014 [eess]. Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, et al. 2020. Common voice: A massively-multilingual speech corpus. In Proc. LREC, pages 4218–4222. Sourav Banerjee, Ayushi Agarwal, and Promila Ghosh. 2024. High-precision medica...

    Pith/arXiv arXiv 2020

  13. [2023]

    Journal of Machine Learning Re- search, 25(97):1–52

    Scaling Speech Technology to 1,000+ Languages. Journal of Machine Learning Re- search, 25(97):1–52. Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. 2020. MLS: A large-scale multilingual dataset for speech research. In Proc. Interspeech, pages 2757–2761. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine McLe...

    work page 2020

  14. [2024]

    ArXiv:2309.05472 [cs]

    LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self- supervised Representations of French Speech. ArXiv:2309.05472 [cs]. Daniel S. Park, William Chan, Yu Zhang, Chung- Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. 2019. SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition. ArXiv:1904.08779. Vine...

    Pith/arXiv arXiv 2019

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.