On a new Quebec French benchmark, standard multilingual ASR benchmarks poorly predict real-world performance; the best open model reaches 8% WER while typical models sit near 14%.
Benchmarking Large Pretrained Multilingual Models on Qu\'ebec French Speech Recognition
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
We evaluate the performance of large pretrained multilingual speech recognition models on a regional variety of French spoken in Qu\'ebec, Canada, in terms of speed, word error rate and semantic accuracy. To this end we build a benchmark and evaluation pipeline based on the CommissionsQc datasets, a corpus of spontaneous conversations recorded during public inquiries recently held in Qu\'ebec. Published results for these models on well-known benchmarks such as FLEURS or CommonVoice are not good predictors of the performance we observe on CommissionsQC. Our results should be of interest for practitioners interested in building speech applications for realistic conditions or regional language varieties.
fields
eess.AS 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Benchmarking Large Pretrained Multilingual Models on Qu\'ebec French Speech Recognition
On a new Quebec French benchmark, standard multilingual ASR benchmarks poorly predict real-world performance; the best open model reaches 8% WER while typical models sit near 14%.