A Paradigm for Interpreting Metrics and Identifying Critical Errors in Automatic Speech Recognition

Jane Wottawa; Mickael Rouvier; Richard Dufour; Thibault Ba\~neras-Roux

arxiv: 2605.03671 · v1 · submitted 2026-05-05 · 💻 cs.CL

A Paradigm for Interpreting Metrics and Identifying Critical Errors in Automatic Speech Recognition

Thibault Ba\~neras-Roux , Mickael Rouvier , Jane Wottawa , Richard Dufour This is my paper

Pith reviewed 2026-05-07 16:33 UTC · model grok-4.3

classification 💻 cs.CL

keywords automatic speech recognitionword error ratecharacter error rateminimum edit distancehuman perceptiontranscription evaluationmetric interpretationerror severity

0 comments

The pith

Incorporating any speech metric into a Minimum Edit Distance produces an interpretable error rate that aligns with human perception of transcription mistakes.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes a paradigm that embeds a chosen evaluation metric into a Minimum Edit Distance calculation to create a score equivalent to traditional error rates. This approach makes metric outputs easier to interpret while better reflecting how humans judge the seriousness of errors in automatic speech transcriptions. It addresses the known weaknesses of WER and CER by linking errors directly to perceptual severity. The method also opens a way to study which transcription mistakes matter most from a human viewpoint.

Core claim

The central claim is that any chosen metric can be incorporated into a Minimum Edit Distance framework to yield an equivalent error rate, called minED. This minED parallels transcription errors with human perception and enables an original analysis of error severity from that perspective.

What carries the argument

The Minimum Edit Distance (minED) created by incorporating a chosen metric, which acts as the interpretable equivalent to standard error rates while approximating human judgment.

If this is right

Any existing metric can be converted into an error-rate equivalent that is directly comparable across systems.
Errors in transcriptions can be ranked and studied according to their human-perceived impact rather than just count.
Critical errors can be identified more reliably for applications that require human-like evaluation.
The paradigm allows direct study of how linguistic and semantic information influences perceived error severity.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

This conversion could support training ASR models with objectives that optimize for human-aligned scores instead of raw WER.
The approach might extend to other sequence tasks where interpretability of evaluation metrics is needed.
Real-time systems could use minED to flag and prioritize correction of the most perceptually damaging errors.

Load-bearing premise

That embedding any chosen metric into the minED framework will preserve its approximation to human perception without introducing inconsistencies or losing the metric's original properties.

What would settle it

A controlled comparison where minED scores from multiple metrics fail to rank transcription errors by severity in the same order as direct human ratings, or where minED shows no improvement in correlation with human judgments over the raw metric.

Figures

Figures reproduced from arXiv: 2605.03671 by Jane Wottawa, Mickael Rouvier, Richard Dufour, Thibault Ba\~neras-Roux.

**Figure 1.** Figure 1: ) consists in a cosine similarity distance between the reference and the hypothesis using embeddings obtained at sentence level. BERTScore [9], applied on various natural language processing (NLP) tasks [10, 11], computes a similarity score for each token in the candidate sentence with each token in the reference sentence using contextual embeddings view at source ↗

**Figure 2.** Figure 2: Computed graph of each possible modification to an error-free hypothesis with the minWED paradigm. Each edge correspond to a corrected error. Given the reference, we have three word errors, each one of a different type: 1 substitution, 1 insertion, 1 deletion. The metric is based on a lower-is-better rule. The token ϵ correspond to deletions. for hypothesis and reference. The hypothesis corresponds to the … view at source ↗

**Figure 3.** Figure 3: , correcting the substitution cook/book will improve the metric performance of 0.1 no matter if an/a was corrected while in view at source ↗

**Figure 4.** Figure 4: Example of impact of correction on inconsistent metric. Metric is based on a lower-is-better rule. might produce more errors. For example, the hypothesis “she worked on state of the art systems” presents 1 substitution and 3 insertions according to the following reference “she worked on state-of-the-art systems”. Correcting the hypothesis might lead to the following: “she worked on state-of-the-art of the… view at source ↗

**Figure 5.** Figure 5: Distribution of SemDist gains for each POS tag corrected. Error bars represent the average gain and the standard deviation. All SemDist values have been multiplied by 100. 4.2. Threshold impact In view at source ↗

read the original abstract

The most commonly used metrics for evaluating automatic speech transcriptions, namely Word Error Rate (WER) and Character Error Rate (CER), have been heavily criticized for their poor correlation to human perception and their inability to take into account linguistic and semantic information. While metric-based embeddings, seeking to approximate human perception, have been proposed, their scores remain difficult to interpret, unlike WER and CER. In this article, we overcome this problem by proposing a paradigm that consists in incorporating a chosen metric into it in order to obtain an equivalent of the error rate: a Minimum Edit Distance (minED). This approach parallels transcription errors with their human perception, also allowing an original study of the severity of these errors from a human perspective.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The minED incorporation idea tries to make perception-based metrics as readable as WER, but the abstract leaves the actual construction too vague to verify if it works or preserves the right properties.

read the letter

The paper's main move is to take any metric that aims to track human perception of transcription errors and fold it into a minimum edit distance so the output looks like a familiar error rate. That framing is the piece that feels new compared to prior embedding work, and it does a clean job of naming the two problems at once: WER and CER ignore semantics, while embedding scores are hard to read back as counts of mistakes. The authors also note that the same setup could let people look at which kinds of errors feel more severe to listeners, which is a reasonable direction to explore. What the work does well is stay focused on a practical pain point in ASR evaluation and sketch a route that keeps the interpretability of edit distances while trying to import human-aligned scoring. The soft spot is exactly the one the stress-test flags. The abstract says the metric is 'incorporated' but gives no definition of how that happens, whether the metric changes substitution costs, alignment costs, or the whole objective, or whether the result remains a true metric. Without that step spelled out, there is no way to check if the minED still approximates human perception or whether it breaks when the input metric is non-metric or high-dimensional. The full text is referenced but the description stays at the same high level, so the central claim cannot be checked for consistency or for any actual validation against human ratings. This paper is aimed at people who work on ASR evaluation metrics and want something more interpretable than current embedding scores. A reader already thinking about human-aligned evaluation might pick up the direction, but anyone looking for a ready-to-use method or reproducible results will find it thin. I would send it to peer review because the underlying problem is real and the proposed direction is coherent enough to deserve referee time, provided the authors supply the missing construction, a small worked example, and some check that the output still correlates with perception.

Referee Report

2 major / 1 minor

Summary. The manuscript proposes a paradigm for evaluating automatic speech recognition (ASR) transcriptions by incorporating a chosen metric into a Minimum Edit Distance (minED) framework. This is intended to yield an interpretable error-rate equivalent that aligns transcription errors with human perception, overcoming limitations of WER and CER such as poor correlation with human judgment and lack of linguistic/semantic awareness, while also enabling analysis of error severity from a human perspective.

Significance. If the incorporation mechanism can be rigorously defined and shown to preserve metric properties while approximating human perception, the work could provide a useful bridge between embedding-based metrics and traditional interpretable error rates, potentially improving ASR evaluation practices.

major comments (2)

[Abstract] Abstract and introduction: the central construction—how a chosen metric is 'incorporated' into minED to obtain an equivalent error rate—is not defined. It is unclear whether the metric replaces substitution costs, defines alignment costs, or is used in another way; whether the resulting minED satisfies metric axioms (symmetry, triangle inequality); or how optimization over alignments is performed. Without this, the claim that minED parallels errors with human perception cannot be evaluated.
[Abstract] The manuscript provides no derivation, validation data, or implementation details showing that the minED output preserves the original metric's semantics or approximates human perception without introducing inconsistencies (e.g., when the input metric is non-metric or high-dimensional). This is load-bearing for the equivalence claim.

minor comments (1)

[Abstract] The abstract uses 'it' ambiguously when referring to the paradigm; clarify the referent.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for their insightful comments on our manuscript. We address the concerns regarding the definition of the paradigm and the supporting details below, and we will revise the manuscript to improve clarity and provide additional information.

read point-by-point responses

Referee: [Abstract] Abstract and introduction: the central construction—how a chosen metric is 'incorporated' into minED to obtain an equivalent error rate—is not defined. It is unclear whether the metric replaces substitution costs, defines alignment costs, or is used in another way; whether the resulting minED satisfies metric axioms (symmetry, triangle inequality); or how optimization over alignments is performed. Without this, the claim that minED parallels errors with human perception cannot be evaluated.

Authors: The manuscript does define the incorporation in the body of the paper, but we agree that the abstract and introduction are too brief. The paradigm incorporates the metric by using it as the substitution cost in the edit distance calculation, with the minED found through standard dynamic programming optimization over possible alignments. We will revise the abstract and introduction to include this explicit description. We will also add a discussion on the metric properties, noting that symmetry and triangle inequality are preserved if the original metric satisfies them. revision: yes
Referee: [Abstract] The manuscript provides no derivation, validation data, or implementation details showing that the minED output preserves the original metric's semantics or approximates human perception without introducing inconsistencies (e.g., when the input metric is non-metric or high-dimensional). This is load-bearing for the equivalence claim.

Authors: We will add a derivation in the revised manuscript to show preservation of semantics. Implementation details, including how to handle the optimization, will be provided. For validation, the paper includes an analysis of error severity from a human perspective using the paradigm, but we acknowledge the lack of large-scale human judgment correlation studies. We will clarify this scope and discuss potential inconsistencies for non-metric cases. revision: partial

standing simulated objections not resolved

Large-scale empirical validation data against human perception, as the work is focused on proposing the paradigm rather than extensive benchmarking.

Circularity Check

0 steps flagged

No circularity detected; minED paradigm is self-contained proposal

full rationale

The paper's central proposal is to incorporate an arbitrary chosen metric into a Minimum Edit Distance (minED) construction to yield an interpretable error-rate equivalent. No equations, fitted parameters, or reductions are exhibited in the provided text that would make the output equivalent to its inputs by construction. The approach is described as building directly on existing metrics (WER, CER) and prior metric-based embeddings without self-referential fitting, self-citation load-bearing premises, or ansatz smuggling. The derivation chain therefore remains independent and does not reduce to a renaming or self-definition of the target result.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review provides no explicit free parameters, axioms, or invented entities; the paradigm is described at conceptual level without mathematical details or new postulated constructs.

pith-pipeline@v0.9.0 · 5427 in / 1082 out tokens · 55651 ms · 2026-05-07T16:33:43.044874+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Reference graph

Works this paper leans on

27 extracted references · 27 canonical work pages · 1 internal anchor

[1]

Evaluating an ASR system typically involves a comparison between manual (reference) and automatic (hypothesis) tran- scriptions using a chosen metric

Introduction Although Automatic Speech Recognition (ASR) performance greatly improved with the recent progress in machine learning and the massive increase in data used for model training, tran- scription errors are still present, their proportion depending on the context in which these systems are used. Evaluating an ASR system typically involves a compa...

work page
[2]

A Paradigm for Interpreting Metrics and Identifying Critical Errors in Automatic Speech Recognition

Dataset with human perception annotations The HATS dataset1 [12] is an open-access corpus for French, in- tended to evaluate the correlation between ASR evaluation met- rics and human perception from the reader’s perspective. It was created using the REPERE corpus [13], containing audio and manually written transcripts of radio and television broadcast in...

work page internal anchor Pith review Pith/arXiv arXiv 2026
[3]

accept- able

A Paradigm for Metric Interpretation The purpose of this paradigm is to provide interpretability for metrics that have scores that are difficult to comprehend. This consists in calculating the minimum number of modifications to be applied to the hypothesis so that it is sufficiently close to the reference regarding its human perception. Following this ide...

work page
[4]

she worked on state of the art systems

Analysis 4.1. Linguistic analysis Using the minWED paradigm, each error in the hypothesis corresponds either to a word in the reference (substituted or deleted) or to an insertion (i.e.word only present in the hypoth- esis). Using a state-of-the-art part-of-speech (POS) tagger for French [21], we propose to linguistically associate a morpho- syntactic cla...

work page
[5]

Our study indicates that the minED approach presents a more comprehensible strategy for evaluat- ing Automatic Speech Recognition (ASR) systems

Conclusions and perspectives We have proposed a paradigm (minED) allowing both to make Automatic Speech Recognition (ASR) metrics interpretable but also to highlight critical transcription errors from the point of view of human perception. Our study indicates that the minED approach presents a more comprehensible strategy for evaluat- ing Automatic Speech...

work page
[6]

The use of MinWED and MinED metrics lead to a decrease in correlation with human perception

Limitations Although the proposed paradigm allows metrics to be inter- pretable, there are limitations to consider. The use of MinWED and MinED metrics lead to a decrease in correlation with human perception. Depending on the threshold, this decrease may ren- der the use of metrics other than WER irrelevant. Secondly, as modern metrics are not always cons...

work page
[7]

Automatic hu- man utility evaluation of ASR systems: Does WER really predict performance?

B. Favre, K. Cheung, S. Kazemian, A. Lee, Y . Liu, C. Munteanu, A. Nenkova, D. Ochei, G. Penn, S. Tratzet al., “Automatic hu- man utility evaluation of ASR systems: Does WER really predict performance?” inINTERSPEECH, 2013, pp. 3463–3467

work page 2013
[8]

Phonetically-oriented word error align- ment for speech recognition error analysis in speech translation,

N. Ruiz and M. Federico, “Phonetically-oriented word error align- ment for speech recognition error analysis in speech translation,” in2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU). IEEE, 2015, pp. 296–302

work page 2015
[9]

Evaluating the usability of auto- matically generated captions for people who are deaf or hard of hearing,

S. Kafle and M. Huenerfauth, “Evaluating the usability of auto- matically generated captions for people who are deaf or hard of hearing,” inProceedings of the 19th International ACM SIGAC- CESS Conference on Computers and Accessibility, 2017, pp. 165– 174

work page 2017
[10]

Meaning Error Rate: ASR domain-specific metric framework,

L. Gordeeva, V . Ershov, O. Gulyaev, and I. Kuralenok, “Meaning Error Rate: ASR domain-specific metric framework,” inProceed- ings of the 27th ACM SIGKDD Conference on Knowledge Dis- covery & Data Mining, 2021, pp. 458–466

work page 2021
[11]

Qual- itative Evaluation of Language Model Rescoring in Automatic Speech Recognition,

T. Ba ˜neras-Roux, M. Rouvier, J. Wottawa, and R. Dufour, “Qual- itative Evaluation of Language Model Rescoring in Automatic Speech Recognition,” inInterspeech 2022, 2022

work page 2022
[12]

Learning word vectors for 157 languages,

´E. Grave, P. Bojanowski, P. Gupta, A. Joulin, and T. Mikolov, “Learning word vectors for 157 languages,” inProceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018

work page 2018
[13]

Enriching word vectors with subword information,

P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, “Enriching word vectors with subword information,”Transactions of the as- sociation for computational linguistics, vol. 5, pp. 135–146, 2017

work page 2017
[14]

Semantic Distance: A New Metric for ASR Per- formance Analysis Towards Spoken Language Understanding,

S. Kim, A. Arora, D. Le, C.-F. Yeh, C. Fuegen, O. Kalinli, and M. L. Seltzer, “Semantic Distance: A New Metric for ASR Per- formance Analysis Towards Spoken Language Understanding,” in Proc. Interspeech 2021, 2021, pp. 1977–1981

work page 2021
[15]

Bertscore: Evaluating text generation with bert,

T. Zhang*, V . Kishore*, F. Wu*, K. Q. Weinberger, and Y . Artzi, “Bertscore: Evaluating text generation with bert,” inInternational Conference on Learning Representations, 2020. [Online]. Available: https://openreview.net/forum?id=SkeHuCVFDr

work page 2020
[16]

Ap- plying bert to document retrieval with birch,

Z. A. Yilmaz, S. Wang, W. Yang, H. Zhang, and J. Lin, “Ap- plying bert to document retrieval with birch,” inProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP): System Demonstra- tions, 2019, pp. 19–24

work page 2019
[17]

A fine-grained analysis of bertscore,

M. Hanna and O. Bojar, “A fine-grained analysis of bertscore,” inProceedings of the Sixth Conference on Machine Translation, 2021, pp. 507–517

work page 2021
[18]

HATS: An open dataset integrating human percep- tion applied to the evaluation of Automatic Speech Recognition metrics,

Anonymous, “HATS: An open dataset integrating human percep- tion applied to the evaluation of Automatic Speech Recognition metrics,” 2023

work page 2023
[19]

The repere corpus: a multimodal corpus for person recognition,

A. Giraudel, M. Carr ´e, V . Mapelli, J. Kahn, O. Galibert, and L. Quintard, “The repere corpus: a multimodal corpus for person recognition,” inInternational Conference on Language Resources and Evaluation (LREC), 2012, pp. 1102–1107

work page 2012
[20]

Evaluating User Perception of Speech Recognition System Quality with Semantic Distance Metric,

S. Kim, D. Le, W. Zheng, T. Singh, A. Arora, X. Zhai, C. Fuegen, O. Kalinli, and M. Seltzer, “Evaluating User Perception of Speech Recognition System Quality with Semantic Distance Metric,” in Proc. Interspeech 2022, 2022, pp. 3978–3982

work page 2022
[21]

SpeechBrain: A general- purpose speech toolkit

M. Ravanelli, T. Parcollet, P. Plantinga, A. Rouhe, S. Cornell, L. Lugosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong, J.-C. Chou, S.-L. Yeh, S.-W. Fu, C.-F. Liao, E. Rastorgueva, F. Grondin, W. Aris, H. Na, Y . Gao, R. D. Mori, and Y . Ben- gio, “SpeechBrain: A general-purpose speech toolkit,” 2021, arXiv:2106.04624

work page arXiv 2021
[22]

The Kaldi speech recognition toolkit,

D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y . Qian, P. Schwarzet al., “The Kaldi speech recognition toolkit,” inIEEE 2011 workshop on automatic speech recognition and understanding, no. CONF. IEEE Signal Processing Society, 2011

work page 2011
[23]

How reliable are annotations via crowd- sourcing: a study about inter-annotator agreement for multi-label image annotation,

S. Nowak and S. R ¨uger, “How reliable are annotations via crowd- sourcing: a study about inter-annotator agreement for multi-label image annotation,” inProceedings of the international conference on Multimedia information retrieval, 2010, pp. 557–566

work page 2010
[24]

Sentence-BERT: Sentence Em- beddings using Siamese BERT-Networks,

N. Reimers and I. Gurevych, “Sentence-BERT: Sentence Em- beddings using Siamese BERT-Networks,” inProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019, pp. 3982–3992

work page 2019
[25]

CamemBERT: a Tasty French Language Model,

L. Martin, B. Muller, P. J. O. Su ´arez, Y . Dupont, L. Romary, ´E. V . De La Clergerie, D. Seddah, and B. Sagot, “CamemBERT: a Tasty French Language Model,” inProceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, 2020, pp. 7203–7219

work page 2020
[26]

Bert: Pre- training of deep bidirectional transformers for language under- standing,

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre- training of deep bidirectional transformers for language under- standing,” inProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 4171–4186

work page 2019
[27]

Antilles: An open french linguistically enriched part-of-speech corpus,

Y . Labrak and R. Dufour, “Antilles: An open french linguistically enriched part-of-speech corpus,” inText, Speech, and Dialogue: 25th International Conference, TSD 2022, Brno, Czech Republic, September 6–9, 2022, Proceedings. Springer, 2022, pp. 28–38

work page 2022

[1] [1]

Evaluating an ASR system typically involves a comparison between manual (reference) and automatic (hypothesis) tran- scriptions using a chosen metric

Introduction Although Automatic Speech Recognition (ASR) performance greatly improved with the recent progress in machine learning and the massive increase in data used for model training, tran- scription errors are still present, their proportion depending on the context in which these systems are used. Evaluating an ASR system typically involves a compa...

work page

[2] [2]

A Paradigm for Interpreting Metrics and Identifying Critical Errors in Automatic Speech Recognition

Dataset with human perception annotations The HATS dataset1 [12] is an open-access corpus for French, in- tended to evaluate the correlation between ASR evaluation met- rics and human perception from the reader’s perspective. It was created using the REPERE corpus [13], containing audio and manually written transcripts of radio and television broadcast in...

work page internal anchor Pith review Pith/arXiv arXiv 2026

[3] [3]

accept- able

A Paradigm for Metric Interpretation The purpose of this paradigm is to provide interpretability for metrics that have scores that are difficult to comprehend. This consists in calculating the minimum number of modifications to be applied to the hypothesis so that it is sufficiently close to the reference regarding its human perception. Following this ide...

work page

[4] [4]

she worked on state of the art systems

Analysis 4.1. Linguistic analysis Using the minWED paradigm, each error in the hypothesis corresponds either to a word in the reference (substituted or deleted) or to an insertion (i.e.word only present in the hypoth- esis). Using a state-of-the-art part-of-speech (POS) tagger for French [21], we propose to linguistically associate a morpho- syntactic cla...

work page

[5] [5]

Our study indicates that the minED approach presents a more comprehensible strategy for evaluat- ing Automatic Speech Recognition (ASR) systems

Conclusions and perspectives We have proposed a paradigm (minED) allowing both to make Automatic Speech Recognition (ASR) metrics interpretable but also to highlight critical transcription errors from the point of view of human perception. Our study indicates that the minED approach presents a more comprehensible strategy for evaluat- ing Automatic Speech...

work page

[6] [6]

The use of MinWED and MinED metrics lead to a decrease in correlation with human perception

Limitations Although the proposed paradigm allows metrics to be inter- pretable, there are limitations to consider. The use of MinWED and MinED metrics lead to a decrease in correlation with human perception. Depending on the threshold, this decrease may ren- der the use of metrics other than WER irrelevant. Secondly, as modern metrics are not always cons...

work page

[7] [7]

Automatic hu- man utility evaluation of ASR systems: Does WER really predict performance?

B. Favre, K. Cheung, S. Kazemian, A. Lee, Y . Liu, C. Munteanu, A. Nenkova, D. Ochei, G. Penn, S. Tratzet al., “Automatic hu- man utility evaluation of ASR systems: Does WER really predict performance?” inINTERSPEECH, 2013, pp. 3463–3467

work page 2013

[8] [8]

Phonetically-oriented word error align- ment for speech recognition error analysis in speech translation,

N. Ruiz and M. Federico, “Phonetically-oriented word error align- ment for speech recognition error analysis in speech translation,” in2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU). IEEE, 2015, pp. 296–302

work page 2015

[9] [9]

Evaluating the usability of auto- matically generated captions for people who are deaf or hard of hearing,

S. Kafle and M. Huenerfauth, “Evaluating the usability of auto- matically generated captions for people who are deaf or hard of hearing,” inProceedings of the 19th International ACM SIGAC- CESS Conference on Computers and Accessibility, 2017, pp. 165– 174

work page 2017

[10] [10]

Meaning Error Rate: ASR domain-specific metric framework,

L. Gordeeva, V . Ershov, O. Gulyaev, and I. Kuralenok, “Meaning Error Rate: ASR domain-specific metric framework,” inProceed- ings of the 27th ACM SIGKDD Conference on Knowledge Dis- covery & Data Mining, 2021, pp. 458–466

work page 2021

[11] [11]

Qual- itative Evaluation of Language Model Rescoring in Automatic Speech Recognition,

T. Ba ˜neras-Roux, M. Rouvier, J. Wottawa, and R. Dufour, “Qual- itative Evaluation of Language Model Rescoring in Automatic Speech Recognition,” inInterspeech 2022, 2022

work page 2022

[12] [12]

Learning word vectors for 157 languages,

´E. Grave, P. Bojanowski, P. Gupta, A. Joulin, and T. Mikolov, “Learning word vectors for 157 languages,” inProceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018

work page 2018

[13] [13]

Enriching word vectors with subword information,

P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, “Enriching word vectors with subword information,”Transactions of the as- sociation for computational linguistics, vol. 5, pp. 135–146, 2017

work page 2017

[14] [14]

Semantic Distance: A New Metric for ASR Per- formance Analysis Towards Spoken Language Understanding,

S. Kim, A. Arora, D. Le, C.-F. Yeh, C. Fuegen, O. Kalinli, and M. L. Seltzer, “Semantic Distance: A New Metric for ASR Per- formance Analysis Towards Spoken Language Understanding,” in Proc. Interspeech 2021, 2021, pp. 1977–1981

work page 2021

[15] [15]

Bertscore: Evaluating text generation with bert,

T. Zhang*, V . Kishore*, F. Wu*, K. Q. Weinberger, and Y . Artzi, “Bertscore: Evaluating text generation with bert,” inInternational Conference on Learning Representations, 2020. [Online]. Available: https://openreview.net/forum?id=SkeHuCVFDr

work page 2020

[16] [16]

Ap- plying bert to document retrieval with birch,

Z. A. Yilmaz, S. Wang, W. Yang, H. Zhang, and J. Lin, “Ap- plying bert to document retrieval with birch,” inProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP): System Demonstra- tions, 2019, pp. 19–24

work page 2019

[17] [17]

A fine-grained analysis of bertscore,

M. Hanna and O. Bojar, “A fine-grained analysis of bertscore,” inProceedings of the Sixth Conference on Machine Translation, 2021, pp. 507–517

work page 2021

[18] [18]

HATS: An open dataset integrating human percep- tion applied to the evaluation of Automatic Speech Recognition metrics,

Anonymous, “HATS: An open dataset integrating human percep- tion applied to the evaluation of Automatic Speech Recognition metrics,” 2023

work page 2023

[19] [19]

The repere corpus: a multimodal corpus for person recognition,

A. Giraudel, M. Carr ´e, V . Mapelli, J. Kahn, O. Galibert, and L. Quintard, “The repere corpus: a multimodal corpus for person recognition,” inInternational Conference on Language Resources and Evaluation (LREC), 2012, pp. 1102–1107

work page 2012

[20] [20]

Evaluating User Perception of Speech Recognition System Quality with Semantic Distance Metric,

S. Kim, D. Le, W. Zheng, T. Singh, A. Arora, X. Zhai, C. Fuegen, O. Kalinli, and M. Seltzer, “Evaluating User Perception of Speech Recognition System Quality with Semantic Distance Metric,” in Proc. Interspeech 2022, 2022, pp. 3978–3982

work page 2022

[21] [21]

SpeechBrain: A general- purpose speech toolkit

M. Ravanelli, T. Parcollet, P. Plantinga, A. Rouhe, S. Cornell, L. Lugosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong, J.-C. Chou, S.-L. Yeh, S.-W. Fu, C.-F. Liao, E. Rastorgueva, F. Grondin, W. Aris, H. Na, Y . Gao, R. D. Mori, and Y . Ben- gio, “SpeechBrain: A general-purpose speech toolkit,” 2021, arXiv:2106.04624

work page arXiv 2021

[22] [22]

The Kaldi speech recognition toolkit,

D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y . Qian, P. Schwarzet al., “The Kaldi speech recognition toolkit,” inIEEE 2011 workshop on automatic speech recognition and understanding, no. CONF. IEEE Signal Processing Society, 2011

work page 2011

[23] [23]

How reliable are annotations via crowd- sourcing: a study about inter-annotator agreement for multi-label image annotation,

S. Nowak and S. R ¨uger, “How reliable are annotations via crowd- sourcing: a study about inter-annotator agreement for multi-label image annotation,” inProceedings of the international conference on Multimedia information retrieval, 2010, pp. 557–566

work page 2010

[24] [24]

Sentence-BERT: Sentence Em- beddings using Siamese BERT-Networks,

N. Reimers and I. Gurevych, “Sentence-BERT: Sentence Em- beddings using Siamese BERT-Networks,” inProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019, pp. 3982–3992

work page 2019

[25] [25]

CamemBERT: a Tasty French Language Model,

L. Martin, B. Muller, P. J. O. Su ´arez, Y . Dupont, L. Romary, ´E. V . De La Clergerie, D. Seddah, and B. Sagot, “CamemBERT: a Tasty French Language Model,” inProceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, 2020, pp. 7203–7219

work page 2020

[26] [26]

Bert: Pre- training of deep bidirectional transformers for language under- standing,

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre- training of deep bidirectional transformers for language under- standing,” inProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 4171–4186

work page 2019

[27] [27]

Antilles: An open french linguistically enriched part-of-speech corpus,

Y . Labrak and R. Dufour, “Antilles: An open french linguistically enriched part-of-speech corpus,” inText, Speech, and Dialogue: 25th International Conference, TSD 2022, Brno, Czech Republic, September 6–9, 2022, Proceedings. Springer, 2022, pp. 28–38

work page 2022