pith. sign in

arxiv: 2606.21237 · v1 · pith:UPQEQKCKnew · submitted 2026-06-19 · 💻 cs.CL · cs.SD

OpenWER: Improving Cross-Lingual ASR Evaluation and Enabling Token-Based Accuracy Metrics

Korbinian Kuhn , Gottfried Zimmermann This is my paper

Pith reviewed 2026-06-26 14:37 UTC · model grok-4.3

classification 💻 cs.CL cs.SD
keywords automatic speech recognitionword error ratecross-lingual evaluationmultilingual ASRtext normalizationcompound word detectionLevenshtein alignmentevaluation metrics
0
0 comments X

The pith

OpenWER applies language-specific normalisation and compound word detection to produce more reliable Word Error Rate scores across languages.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Automatic speech recognition models now handle many languages at once, yet standard evaluation with Word Error Rate often fails to account for language-specific text features like compound words. The paper presents OpenWER, an open-source tool that normalizes reference and hypothesis texts according to each language's conventions before alignment. This produces lower WER values than common libraries, with absolute reductions reaching 25 percent in tests on 52 languages. The same alignment also supports token-level metrics and metadata attachment for finer-grained accuracy analysis. These changes matter for fair comparisons between models that serve different languages and for more trustworthy results on low-resource languages.

Core claim

OpenWER is an open-source implementation that improves WER robustness through language-specific normalisation and compound word detection. A token-based Levenshtein alignment preserves complementary metrics and allows metadata embedding for granular accuracy scores. Analysis of 52 languages shows absolute WER reductions of up to 25% compared to common libraries.

What carries the argument

OpenWER, the implementation that performs language-specific normalisation plus compound word detection before applying token-based Levenshtein alignment to compute WER.

If this is right

  • Evaluations of multilingual ASR models become more consistent across languages.
  • Low-resource languages receive more accurate performance estimates than before.
  • Researchers can attach per-token metadata to produce additional accuracy measures alongside WER.
  • Cross-lingual model comparisons rest on a more uniform metric foundation.
  • Standard libraries may need updates to match the normalisation rules introduced here.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Adoption of OpenWER could gradually shift ASR benchmark reporting toward language-aware normalisation as a default practice.
  • The token-level alignment approach might be reused for other sequence metrics such as character error rate in future tools.
  • Model developers could incorporate similar normalisation steps during training to reduce the gap between training and evaluation conditions.
  • Benchmark organizers might add compound-word handling as a required preprocessing step in shared tasks.

Load-bearing premise

The reported WER reductions result specifically from the language-specific normalisation and compound word detection rather than from differences in tokenization, data preprocessing, or baseline library configurations.

What would settle it

Re-run the 52-language evaluation suite with OpenWER while forcing identical tokenization and preprocessing steps as the baseline libraries; if the 25 percent reductions disappear, the central claim does not hold.

Figures

Figures reproduced from arXiv: 2606.21237 by Gottfried Zimmermann, Korbinian Kuhn.

Figure 1
Figure 1. Figure 1: Overview of OpenWER components: Input can be raw text or ASR-generated word lists. Basic tokenisation is built in, but can be replaced by an NLP-tokeniser. General or language-specific normalisation can be applied to all input and tokenisation methods. Hypothesis and reference are aligned using an extended Levenshtein distance algorithm that handles punctuation tokens and compound words. After alignment, t… view at source ↗
Figure 2
Figure 2. Figure 2: Word Error Rate comparison on Common Voice 17 [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
read the original abstract

Advances in deep learning and end-to-end Automatic Speech Recognition (ASR) have enabled robust multilingual models, but evaluation metrics remain limited in assessing accuracy. Efforts to improve or replace the common metric Word Error Rate (WER) often focus on English, leaving evaluations for low-resource languages under-explored and hindering fair cross-lingual comparisons. We present OpenWER, an open-source implementation that improves WER robustness through language-specific normalisation and compound word detection. A token-based Levenshtein alignment preserves complementary metrics and allows metadata embedding for granular accuracy scores. Our analysis of 52 languages shows absolute WER reductions of up to 25% compared to common libraries. OpenWER contributes to fairness in ASR research by increasing the reliability of WER across diverse languages and enabling more comprehensive accuracy evaluations.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces OpenWER, an open-source library for computing Word Error Rate (WER) in multilingual ASR. It adds language-specific text normalisation and compound-word detection to improve robustness, implements a token-based Levenshtein alignment that retains complementary token-level metrics and supports metadata embedding, and reports absolute WER reductions of up to 25 % across 52 languages relative to common libraries.

Significance. If the reported reductions are shown to arise specifically from the normalisation and compound-word components under controlled conditions, the work would strengthen the reliability of cross-lingual ASR evaluation and support more granular accuracy analyses. The open-source release and preservation of token-level metrics are concrete strengths that could be adopted by the community.

major comments (2)
  1. [Abstract and §4] Abstract and §4 (Experiments): the headline claim of up to 25 % absolute WER reduction across 52 languages is load-bearing, yet the manuscript supplies no explicit statement that reference-text cleaning, tokenisation, and alignment rules were held identical between OpenWER and the compared libraries. Because WER is known to be sensitive to exactly these preprocessing choices, the attribution of the delta to the new normalisation features cannot be evaluated from the current description.
  2. [§4] §4: no table or figure reports per-language WER values, baseline library names and versions, or any measure of statistical significance or variance across runs. Without these data the cross-lingual claim remains untestable.
minor comments (2)
  1. [§3.2] §3.2: the description of the token-based alignment would benefit from a small worked example showing how metadata is embedded and how the resulting token-level scores are aggregated back to WER.
  2. [References] References: several standard multilingual ASR evaluation papers (e.g., on Common Voice or FLEURS) are not cited when discussing cross-lingual challenges.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. We address each major point below and agree to revisions that improve clarity and testability of the results.

read point-by-point responses

  1. Referee: [Abstract and §4] Abstract and §4 (Experiments): the headline claim of up to 25 % absolute WER reduction across 52 languages is load-bearing, yet the manuscript supplies no explicit statement that reference-text cleaning, tokenisation, and alignment rules were held identical between OpenWER and the compared libraries. Because WER is known to be sensitive to exactly these preprocessing choices, the attribution of the delta to the new normalisation features cannot be evaluated from the current description.

    Authors: We agree that an explicit statement is required. All experiments used identical reference transcripts across libraries; the reported deltas arise solely from OpenWER's language-specific normalisation, compound-word detection, and token-based alignment versus the standard rules in the baseline libraries. We will revise §4 to state this control explicitly so that attribution to the new components can be evaluated. revision: yes

  2. Referee: [§4] §4: no table or figure reports per-language WER values, baseline library names and versions, or any measure of statistical significance or variance across runs. Without these data the cross-lingual claim remains untestable.

    Authors: We agree that per-language values, library versions, and distributional information would strengthen testability. We will add a supplementary table (referenced from §4) listing per-language WERs for OpenWER and each baseline, together with the exact library names and versions. Because WER computation is deterministic for fixed references and hypotheses, run-to-run variance does not apply; we will instead report the distribution and range of improvements across the 52 languages to support the 'up to 25 %' claim. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical implementation with no derivation chain

full rationale

The paper describes an open-source engineering implementation of a WER metric with language-specific normalisation and compound-word detection. The central claim consists of empirical WER reductions observed across 52 languages when compared to common libraries. No equations, first-principles derivations, fitted parameters, or predictions are present. No self-citations or ansatzes are invoked to justify any load-bearing step. The result is therefore self-contained as a reported implementation outcome rather than a derived quantity that reduces to its own inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No mathematical model or derivation is present; the work consists of software rules for text normalization and alignment.

pith-pipeline@v0.9.1-grok · 5657 in / 1007 out tokens · 24805 ms · 2026-06-26T14:37:43.947924+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 2 canonical work pages · 1 internal anchor

  1. [1]

    OpenWER: Improving Cross-Lingual ASR Evaluation and Enabling Token-Based Accuracy Metrics

    Introduction Large-scale deep learning techniques [1, 2, 3] and transformer- based Automatic Speech Recognition (ASR) [4, 5, 6, 7] have led to robust multilingual models that support a wide range of languages [8, 9]. Evaluating their transcription accuracy using universal metrics is essential for unbiased and fair performance comparisons across models and...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  2. [2]

    Figure 1 il- lustrates its modular design, with interchangeable elements for input formats, tokenisation methods, language-specific normal- isation, and evaluation metrics

    OpenWER This section outlines the components of OpenWER. Figure 1 il- lustrates its modular design, with interchangeable elements for input formats, tokenisation methods, language-specific normal- isation, and evaluation metrics. 2.1. Input The computation of the edit distance requires a reference and hypothesis input. These inputs can be raw text or a st...

    2024

  3. [3]

    Algorithm Robustness Compound word detection and punctuation tokens can alter the edit path in Levenshtein distance calculations, affecting WER scores

    Evaluation 3.1. Algorithm Robustness Compound word detection and punctuation tokens can alter the edit path in Levenshtein distance calculations, affecting WER scores. To assess the robustness of OpenWER, we compare its results with those of the widely used JiWER library. Without any text normalisation, case-sensitivity, and dis- abled compound word detec...

  4. [4]

    WER has been criticised for its limitations, but alternative metrics have yet to see wider adoption

    Conclusion Objective metrics are essential to measure the transcription ac- curacy of ASR models across languages and to identify per- formance gaps, particularly in low-resource languages. WER has been criticised for its limitations, but alternative metrics have yet to see wider adoption. OpenWER addresses these challenges and improves WER’s reliability ...

  5. [5]

    Stiftung Innovation in der Hochschullehre

    Acknowledgements This work was conducted as part of the SHUFFLE Project and funded by “Stiftung Innovation in der Hochschullehre”

  6. [6]

    Semi-supervised learning with ladder networks,

    A. Rasmus, H. Valpola, M. Honkala, M. Berglund, and T. Raiko, “Semi-supervised learning with ladder networks,” inConference on Neural Information Processing Systems (NeurIPS) 2015. Cur- ran Associates, Inc., Dec 2015, pp. 3546—-3554

    2015

  7. [7]

    wav2vec 2.0: A framework for self-supervised learning of speech representa- tions,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representa- tions,” inConference on Neural Information Processing Systems (NeurIPS) 2020. virtual: Curran Associates, Inc., Dec 2020, pp. 12 449–12 460

    2020

  8. [8]

    Unsupervised training and directed manual transcription for lvcsr,

    K. Yu, M. Gales, L. Wang, and P. C. Woodland, “Unsupervised training and directed manual transcription for lvcsr,”Speech Com- munication, vol. 52, no. 7, pp. 652–663, Jul 2010

    2010

  9. [9]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” inConference on Neural Information Processing Systems (NeurIPS) 2017. Long Beach, California, USA: Curran Asso- ciates, Inc., Dec 2017

    2017

  10. [10]

    Attention-based models for speech recognition,

    J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y . Ben- gio, “Attention-based models for speech recognition,” inConfer- ence on Neural Information Processing Systems (NeurIPS) 2015. Montreal, Canada: Curran Associates, Inc., Dec 2015

    2015

  11. [11]

    Conformer: Convolution-augmented transformer for speech recognition,

    A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” in INTERSPEECH 2020. virtual: ISCA, Oct 2020, pp. 5036–5040

    2020

  12. [12]

    Branchformer: Parallel MLP-attention architectures to capture local and global context for speech recognition and understanding,

    Y . Peng, S. Dalmia, I. Lane, and S. Watanabe, “Branchformer: Parallel MLP-attention architectures to capture local and global context for speech recognition and understanding,” inInterna- tional Conference on Machine Learning (ICML) 2022. Balti- more, Maryland, USA: PMLR, Jul 2022, pp. 17 627–17 643

    2022

  13. [13]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. Mcleavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” inInternational Conference on Machine Learning (ICML) 2023. Honolulu, Hawaii, USA: PMLR, Jul 2023, pp. 28 492–28 518

    2023

  14. [14]

    Seamless: Multilingual expressive and streaming speech translation,

    L. Barrault, Y .-A. Chung, M. C. Meglioli, D. Dale, N. Dong, M. Duppenthaler, P.-A. Duquenne, B. Ellis, H. Elsahar, J. Haa- heimet al., “Seamless: Multilingual expressive and streaming speech translation,”arXiv preprint arXiv:2312.05187, 2023

    work page arXiv 2023

  15. [15]

    Binary codes capable of correcting deletions, insertions, and reversals,

    V . I. Levenshtein, “Binary codes capable of correcting deletions, insertions, and reversals,” inSoviet physics doklady, vol. 10, no. 8. Soviet Union, 1966, pp. 707–710

    1966

  16. [16]

    Is word error rate a good indicator for spoken language understanding accuracy,

    Y .-Y . Wang, A. Acero, and C. Chelba, “Is word error rate a good indicator for spoken language understanding accuracy,” in IEEE Automatic Speech Recognition and Understanding Work- shop (ASRU) 2003. Saint Thomas, Virgin Islands, USA: IEEE, Nov 2003, pp. 577–582

    2003

  17. [17]

    Predicting human perceived accuracy of asr systems

    T. Mishra, A. Ljolje, and M. Gilbert, “Predicting human perceived accuracy of asr systems.” inINTERSPEECH 2011. Florence, Italy: ISCA, Aug 2011, pp. 1945–1948

    2011

  18. [18]

    Automatic human utility evaluation of asr systems: Does wer really predict performance?

    B. Favre, K. Cheung, S. Kazemian, A. Lee, Y . Liu, C. Munteanu, A. Nenkova, D. Ochei, G. Penn, S. Tratz, C. V oss, and F. Zeller, “Automatic human utility evaluation of asr systems: Does wer really predict performance?” inINTERSPEECH 2013. Lyon, France: ISCA, Aug 2013, pp. 3463–3467

    2013

  19. [19]

    Methods for evaluation of imperfect captioning tools by deaf or hard-of-hearing users at different reading literacy levels,

    L. Berke, S. Kafle, and M. Huenerfauth, “Methods for evaluation of imperfect captioning tools by deaf or hard-of-hearing users at different reading literacy levels,” inACM Conference on Human Factors in Computing Systems (CHI) 2018. ACM, Apr 2018, pp. 1–12

    2018

  20. [20]

    Effect of speech recognition er- rors on text understandability for people who are deaf or hard of hearing,

    S. Kafle and M. Huenerfauth, “Effect of speech recognition er- rors on text understandability for people who are deaf or hard of hearing,” inSpeech and Language Processing for Assistive Tech- nologies (SPLAT) 2016. San Francisco, California, USA: ISCA, Sep 2016, pp. 20–25

    2016

  21. [21]

    Mod- eling word importance in conversational transcripts: Toward im- proved live captioning for deaf and hard of hearing viewers,

    A. A. Amin, S. Hassan, M. Huenerfauth, and C. O. Alm, “Mod- eling word importance in conversational transcripts: Toward im- proved live captioning for deaf and hard of hearing viewers,” in International Web for All Conference (W4A) 2023. ACM, Apr 2023, pp. 79–83

    2023

  22. [22]

    Qualita- tive evaluation of language model rescoring in automatic speech recognition,

    T. B. Roux, M. Rouvier, J. Wottawa, and R. Dufour, “Qualita- tive evaluation of language model rescoring in automatic speech recognition,” inINTERSPEECH 2022. ISCA, Sep 2022, pp. 3968–3972

    2022

  23. [23]

    Caption accuracy met- rics project,

    T. Apone, M. Brooks, and T. O’Connell, “Caption accuracy met- rics project,” Boston, Massachusetts, USA, Sep 2011

    2011

  24. [24]

    Evaluating the usability of auto- matically generated captions for people who are deaf or hard of hearing,

    S. Kafle and M. Huenerfauth, “Evaluating the usability of auto- matically generated captions for people who are deaf or hard of hearing,” inACM SIGACCESS Conference on Computers and Ac- cessibility (ASSETS) 2017. ACM, Oct 2017, pp. 165–174

    2017

  25. [25]

    Predicting the understandability of imperfect english cap- tions for people who are deaf or hard of hearing,

    ——, “Predicting the understandability of imperfect english cap- tions for people who are deaf or hard of hearing,”ACM Transac- tions on Accessible Computing, vol. 12, no. 2, Jun 2019

    2019

  26. [26]

    Com- paring the accuracy of ace and wer caption metrics when applied to live television captioning,

    T. Wells, D. Christoffels, C. V ogler, and R. Kushalnagar, “Com- paring the accuracy of ace and wer caption metrics when applied to live television captioning,” inJoint International Conference on Digital Inclusion, Assistive Technology & Accessibility (ICCHP- AAATE 2022). Springer, Jul 2022, pp. 522–528

    2022

  27. [27]

    How users experience closed captions on live television: Quality metrics remain a challenge,

    M. Arroyo Chavez, M. Feanny, M. Seita, B. Thompson, K. Delk, S. Officer, A. Glasser, R. Kushalnagar, and C. V ogler, “How users experience closed captions on live television: Quality metrics remain a challenge,” inACM Conference on Human Factors in Computing Systems (CHI) 2024. ACM, May 2024

    2024

  28. [28]

    From wer and ril to mer and wil: improved evaluation measures for connected speech recogni- tion,

    A. Morris, V . Maier, and P. Green, “From wer and ril to mer and wil: improved evaluation measures for connected speech recogni- tion,” inInternational Conference on Spoken Language Process- ing (ICSLP) 2004. Jeju Island, Korea: ISCA, Oct 2004, pp. 2765–2768

    2004

  29. [29]

    Racial disparities in automated speech recognition,

    A. Koenecke, A. Nam, E. Lake, J. Nudell, M. Quartey, Z. Menge- sha, C. Toups, J. R. Rickford, D. Jurafsky, and S. Goel, “Racial disparities in automated speech recognition,”National Academy of Sciences, vol. 117, no. 14, pp. 7684–7689, Apr 2020

    2020

  30. [30]

    Perspectives of deaf and hard of hearing viewers of captions,

    J. Butler, “Perspectives of deaf and hard of hearing viewers of captions,”American Annals of the Deaf, vol. 163, no. 5, pp. 534– 553, Feb 2019

    2019

  31. [31]

    Librispeech-pc: Benchmark for evaluation of punctuation and capitalization capabilities of end-to-end asr mod- els,

    A. Meister, M. Novikov, N. Karpov, E. Bakhturina, V . Lavrukhin, and B. Ginsburg, “Librispeech-pc: Benchmark for evaluation of punctuation and capitalization capabilities of end-to-end asr mod- els,” inIEEE Automatic Speech Recognition and Understanding Workshop (ASRU) 2023. IEEE, Dec 2023, pp. 1–7

    2023

  32. [32]

    Beyond levenshtein: Leveraging multiple algorithms for robust word error rate com- putations and granular error classifications,

    K. Kuhn, V . Kersken, and G. Zimmermann, “Beyond levenshtein: Leveraging multiple algorithms for robust word error rate com- putations and granular error classifications,” inINTERSPEECH

  33. [33]

    4543–4547

    Kos, Greece: ISCA, Sep 2024, pp. 4543–4547

    2024

  34. [34]

    A technique for computer detection and correction of spelling errors,

    F. J. Damerau, “A technique for computer detection and correction of spelling errors,”Communications of the ACM, vol. 7, no. 3, pp. 171–176, Mar 1964

    1964

  35. [35]

    String comparator metrics and enhanced decision rules in the fellegi-sunter model of record linkage,

    W. E. Winkler, “String comparator metrics and enhanced decision rules in the fellegi-sunter model of record linkage,”Proceedings of the Section on Survey Research Methods, pp. 354–359, 1990

    1990

  36. [36]

    Common voice: A massively-multilingual speech corpus,

    R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” in Language Resources and Evaluation Conference (LREC) 2020. European Language Resources Association, May 2020, pp. 4211– 4215

    2020