Pith. sign in

REVIEW 1 cited by

WER we are and WER we think we are

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2010.03432 v1 pith:QBCTKPH2 submitted 2020-10-07 cs.CL cs.LGcs.SDeess.AS

Piotr Szymański , Piotr Żelasko , Mikolaj Morzy , Adrian Szymczak , Marzena Ży{l}a-Hoppe , Joanna Banaszczak , Lukasz Augustyniak , Jan Mizgajski
show 1 more author
Yishay Carmiel
This is my paper
classification cs.CLcs.LGcs.SDeess.AS
keywords systemsbenchmarkdatasetsreal-lifespeechwersachievedannotations
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Natural language processing of conversational speech requires the availability of high-quality transcripts. In this paper, we express our skepticism towards the recent reports of very low Word Error Rates (WERs) achieved by modern Automatic Speech Recognition (ASR) systems on benchmark datasets. We outline several problems with popular benchmarks and compare three state-of-the-art commercial ASR systems on an internal dataset of real-life spontaneous human conversations and HUB'05 public benchmark. We show that WERs are significantly higher than the best reported results. We formulate a set of guidelines which may aid in the creation of real-life, multi-domain datasets with high quality annotations for training and testing of robust ASR systems.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PSRB: A Comprehensive Benchmark for Evaluating Persian ASR Systems

    eess.AS 2025-05 conditional novelty 6.0 of 10

    PSRB, a 10.4-hour Persian benchmark built from 3,372 clips and 756 speakers, evaluates ten ASR models and introduces SW-WER, showing that systems are far weaker on regional accents, children's speech, and informal aud...

Pith tools