Pith. sign in

REVIEW 4 major objections 5 minor 65 references

PMF-CEC: Phoneme-augmented Multimodal Fusion for Context-aware ASR Error Correction with Error-specific Selective Decoding

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A phoneme-augmented correction model reduces ASR rare-word errors on homophones by up to 12.6%.

desk verdict Solid incremental ASR postprocessor; needs direct comparison with PATCorrect/PGCC and a G2P error analysis to fully support the novelty claim. read the letter →

arxiv 2506.11064 v1 pith:E4TGU4ZO submitted 2025-05-31 eess.AS cs.AIcs.CLcs.SD

Jiajun He , Tomoki Toda This is my paper
classification eess.AScs.AIcs.CLcs.SD
keywords automaticspeechrecognitionphonemeinformationmultimodalfusionerrordetectioncontext-awarecorrectionrarewordlisthomophonedisambiguationretentionprobabilitymechanism
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a post-processing ASR error correction model can resolve homophone errors—rare words that sound like common ones—by adding a phoneme channel to text-based correction, and that filtering low-confidence edits prevents the error detector from introducing new mistakes. It reports that this two-part system, PMF-CEC, lowers biased word error rate on rare words by 8.39% to 12.56% and overall WER by 4.25% to 14.46% relative to its text-only predecessor ED-CEC across five datasets, while staying fast enough for near-real-time use. A sympathetic reader would care because it offers a way to improve rare-word recognition in existing ASR systems without retraining the recognizer, and it outperforms large language model-based correction on speed and on stability as biasing lists grow.

What carries the argument

The central object is the phoneme-augmented multimodal fusion (PMF) module: a cross-attention layer that treats the BERT text representation $E^{(I)}$ as the query and the XPhoneBERT phoneme representation $E^{(P_s)}$ as both key and value, then adds the result to the text representation. This injection of pronunciation information directly into the error-detection and correction modules is what allows homophone disambiguation. The second load-bearing mechanism is the retention probability mechanism (RPM): at inference, each editing operation's softmax confidence is compared against a 0.5 threshold, and low-confidence edits are reverted to the original transcript, counteracting overdetection in the error detection module. A context decoder with a learned <no-context> dummy token decides whether to copy from the rare word list or generate a new token.

What would settle it

Construct a test set of near-homophone rare words (for example, 'tsavo' versus 'savo') and deliberately corrupt or remove the phoneme sequences; if B-WER still drops by the same amount relative to ED-CEC, the phoneme channel is not carrying the claimed advantage. A second check is to vary the RPM threshold on a dataset where overdetection is already minimal: if the B-WER gain over ED-CEC remains, then the fusion alone, not the RPM, is responsible for the improvement.

Watch

Extended reading notes

Core claim

PMF-CEC claims that feeding the same ASR transcript twice—once as BERT token embeddings and once as XPhoneBERT phoneme embeddings, then aligning them with cross-attention—lets the correction model tell apart words that are pronounced alike but spelled differently, such as 'savo' versus 'tsavo'. On top of this, a retention probability mechanism scores each predicted edit (Keep, Delete, or Change) and only applies edits whose confidence exceeds a threshold of 0.5, which the paper shows reduces overcorrection and raises detection accuracy. The paper's experiments position PMF-CEC as outperforming ED-CEC and other contextual biasing methods on all five datasets, matching or beating LLM-based decoders on biased word error rate while running in tens of milliseconds and staying stable with biasing lists up to 3000 words.

Load-bearing premise

The phoneme sequences produced by the Text2PhonemeSequence tool must be accurate enough that the cross-attention fusion can distinguish homophones; the paper provides no analysis of how G2P errors or noisy phoneme-to-token alignment affect the results.

Editorial extensions

If this is right

  • Any ASR system that emits a 1-best transcript can use PMF-CEC without retraining the recognizer, because correction happens as postprocessing.
  • Relative to ED-CEC, PMF-CEC reduces biased WER by 8.39% to 12.56% and overall WER by 4.25% to 14.46% across ATIS, SNIPS, LibriSpeech, DATA2, and PRLVS.
  • The retention probability mechanism raises error detection accuracy from 95.73% to 97.40% on LibriSpeech test-clean while lowering WER, indicating less overcorrection.
  • Inference stays at 27.21 to 38.12 ms per utterance, making the method 2.4 to 5.3 times faster than the autoregressive SC BART baseline.
  • PMF-CEC can be stacked with LLM-based decoders: combining it with SLAM-ASR lowers WER from 5.75% to 4.66%, and combining it with Whispering LLaMA lowers WER from 15.61% to 13.14%.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If phoneme sequences were generated by a tool that explicitly preserves homophone distinctions, such as stress or syllable boundaries, the B-WER gain might extend to languages with deeper orthography; the paper does not test this.
  • The same cross-attention fusion could be applied to other token-level post-editing tasks over ASR output, such as named-entity normalization or punctuation restoration, but the paper only evaluates full-sentence correction.
  • A controlled stress test with synthetic homophone pairs whose phoneme sequences differ by exactly one phone would isolate whether the gain comes from the phoneme signal itself or from the RPM thresholding; the paper does not separate these contributions.
  • Because the method requires no ASR retraining, its value should be greatest in deployed systems where the biasing list changes frequently, a scenario the paper only partially covers with its large-list experiments.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents PMF-CEC, an ASR post-processing method that extends the authors' earlier ED-CEC system by adding phoneme-augmented multimodal fusion and a retention probability mechanism (RPM). Phoneme sequences, generated by an external G2P tool, are encoded with XPhoneBERT and fused with BERT text representations through cross-attention and residual addition (Eqs. 2-3). The AED module is supplemented with an inference-time confidence threshold that keeps the original editing operation when the predicted operation's confidence is low. Experiments on five datasets (ATIS, SNIPS, LibriSpeech, DATA2, PRLVS) report WER, B-WER, and inference-time comparisons against several AEC baselines, contextual biasing methods, and LLM-based ASR/AEC systems, along with ablations, few-shot generalization, and domain adaptation studies.

Significance. If the reported results hold, the paper offers a practically attractive lightweight post-processing method for correcting homophone errors on rare words, with measured speed advantages over autoregressive and LLM-based alternatives. The evaluation is broad: five datasets, multiple ASR backends (LAS, Kaldi, WeNet, ESPnet, SpeechBrain, Whisper), and quantitative inference-time measurements. The paper also includes useful ablations (Table XI), few-shot generalization (Table XII), and limited-data domain adaptation (Table XIII) analyses. The main contribution, phoneme-augmented multimodal fusion, is clearly the right direction for the stated problem, and the RPM is a simple, sensible fix for the overdetection issue.

major comments (4)
  1. [Section III-B and Eq. (2)] The phoneme-augmented fusion, which is the central novelty of the paper, depends entirely on the correctness of the Text2PhonemeSequence G2P tool, yet the paper provides no quantitative evaluation of G2P accuracy. This is particularly important for the target rare words and proper nouns, where G2P errors would corrupt both the source transcript phonemes and the context-item phonemes used in the cross-attention of Eq. (2). The single illustrative example in Fig. 8 cannot establish reliability. Please report G2P accuracy on the rare-word lists, or provide an ablation with oracle phoneme sequences, or present a manual error analysis on homophone-confusable items.
  2. [Section V-A, Table VI] The core claim that PMF-CEC improves over ED-CEC is supported only by single point estimates. Several differences are small, for example on LibriSpeech test-clean WER 2.52% vs. 2.71% and U-WER 1.12% vs. 1.25%. Without confidence intervals, bootstrap significance tests, or multiple training seeds, it is not possible to assess whether these gains are reliable. Please add utterance-level paired tests or bootstrap resampling across test sets, and report variance across seeds if training is stochastic.
  3. [Section V-A and Table I] The paper does not directly compare against the two most closely related phoneme-aware methods, PATCorrect [17] and PGCC [24], even though Table I lists PGCC as a method that handles homophones and the abstract claims superiority over 'other contextual biasing methods'. Since PGCC and PATCorrect are precisely the baselines that target homophone correction, the central claim requires at least one direct experimental comparison on a common dataset, or a clear statement of why such a comparison is infeasible, together with a narrowed claim.
  4. [Section V-D, Table XI] The text describing the RPM ablation is internally inconsistent with the table. It states that removing the RPM increased WER and that 'the accuracy of editing operations in the AED module rose from 95.73% to 97.40%', but Table XI shows the full model has 97.40% accuracy and the w/o RPM variant has 95.73%. Please correct this reversal. In addition, the RPM threshold is a free parameter (set to 0.5 in Section IV-A) and no sensitivity analysis is provided; a threshold sweep on at least one validation set would strengthen the claim that the mechanism, rather than a carefully chosen value, is responsible for the improvement.
minor comments (5)
  1. [Section V-B heading] The heading reads 'PMF-ECE' but should read 'PMF-CEC'.
  2. [Section V-H, Fig. 7] The sentence 'where "alum" is misrecognized as "erlangen"' is confusing: in the example, the original ASR output is 'alum', ED-CEC produces 'algebra', and the ground truth is 'erlangen'. Please rephrase to state that 'alum' should have been corrected to 'erlangen' but ED-CEC instead selected 'algebra'.
  3. [Table VII] The speed ratio convention is ambiguous: the row 'vs ConstDecodertrans 0.8×' could be misread as PMF-CEC being faster, whereas the times show PMF-CEC is about 1.5 times slower than ConstDecoder trans. Please define the ratio as baseline-time / PMF-time and state in the text that values below 1 indicate PMF-CEC is slower.
  4. [Section V-E] The text says the rare-word-list-size experiments are 'based on the results shown in Table VIII', but the actual results are in Fig. 6. Please correct the cross-reference.
  5. [Section V-C, Table IX] The comparison with Whispering LLaMA is not head-to-head: WL M uses 5-best hypotheses and Whisper-large audio features, while PMF-CEC is applied to Whisper 1-best output. The text acknowledges this, but the table should state the input configuration clearly in a footnote to avoid over-interpretation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: reported gains are empirical comparisons against external baselines; self-citations are not load-bearing.

full rationale

The paper's central claims are empirical rather than derivational: PMF-CEC is evaluated on five public datasets against external baselines (SC BART, ConstDecoder, TCPGen, GPT-2, SLAM-ASR, MaLa-ASR, Whispering LLaMA) and against the authors' prior ED-CEC system, which is a published, independently testable baseline rather than a citation that supplies the evidence. The RPM threshold is tuned on validation data, not fitted to the test quantities it is claimed to improve. The phoneme inputs come from an external G2P tool, and the paper's acknowledged limitation that phoneme information is injected only at the postprocessing stage (Section V-I) is a scope limitation, not a circular step. The absence of a quantitative G2P error analysis is a robustness/correctness concern for the claimed homophone improvements, but it does not make the evaluation equivalent to its inputs by construction. Self-citations to ED-CEC and related prior work are normal predecessor references; they are not used to forbid alternatives or to justify a uniqueness claim. Accordingly, no step in the paper reduces to a fitted input, a renamed known result, or a self-citation chain, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or symbolic entities. It relies on pretrained encoders, a text-to-phoneme tool, and two hand-set hyperparameters. The central claims are empirical, so the ledger captures the main modeling assumptions and tuned values rather than derivation-level postulates.

free parameters (3)
  • RPM confidence threshold = 0.5
    Set following empirical validation (Section IV-A and V-D); controls whether low-confidence edit operations are retained and directly affects reported WER and AED accuracy.
  • loss weight gamma = 3
    Tuned on validation data (Section IV-A); balances the detection loss and correction loss and could shift which errors the model prioritizes.
  • GPT-2 rescoring LM weight = not reported
    Optimized on each validation set for the GPT-2 baseline (Section V-B); affects the baseline B-WER numbers that PMF-CEC is compared against.
assumptions (4)
  • domain assumption Phoneme sequences produced by Text2PhonemeSequence are correct and complete enough to distinguish homophones.
    The central contribution depends on phoneme input quality; the paper does not evaluate G2P errors or coverage on rare proper nouns (Section III-B).
  • domain assumption Pretrained BERT and XPhoneBERT representations remain informative after finetuning on the ASR error correction task.
    The model initializes text and phoneme encoders from these pretrained checkpoints and relies on their transfer (Section IV-A).
  • domain assumption LCS alignment between the ASR transcript and ground truth yields correct KEEP, DELETE, and CHANGE edit labels.
    Supervision for the AED module is generated via LCS alignment (Section III-B); incorrect alignments would teach the model poor edit decisions.
  • domain assumption Rare word lists extracted from slides and reference transcripts approximate real-world biasing lists.
    Evaluation relies on the simulated and OCR-derived rare word lists (Section IV-C); deviations from deployment conditions could change the measured gains.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PMF-CEC: Phoneme-augmented Multimodal Fusion for Context-aware ASR Error Correction with Error-specific Selective Decoding." pith.science (2026). https://pith.science/paper/E4TGU4ZO

@misc{pith2026250611064,
  author       = {Pith},
  title        = {Pith review of: PMF-CEC: Phoneme-augmented Multimodal Fusion for Context-aware ASR Error Correction with Error-specific Selective Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E4TGU4ZO}},
  note         = {Machine review of arXiv:2506.11064}
}
read the original abstract

End-to-end automatic speech recognition (ASR) models often struggle to accurately recognize rare words. Previously, we introduced an ASR postprocessing method called error detection and context-aware error correction (ED-CEC), which leverages contextual information such as named entities and technical terms to improve the accuracy of ASR transcripts. Although ED-CEC achieves a notable success in correcting rare words, its accuracy remains low when dealing with rare words that have similar pronunciations but different spellings. To address this issue, we proposed a phoneme-augmented multimodal fusion method for context-aware error correction (PMF-CEC) method on the basis of ED-CEC, which allowed for better differentiation between target rare words and homophones. Additionally, we observed that the previous ASR error detection module suffers from overdetection. To mitigate this, we introduced a retention probability mechanism to filter out editing operations with confidence scores below a set threshold, preserving the original operation to improve error detection accuracy. Experiments conducted on five datasets demonstrated that our proposed PMF-CEC maintains reasonable inference speed while further reducing the biased word error rate compared with ED-CEC, showing a stronger advantage in correcting homophones. Moreover, our method outperforms other contextual biasing methods, and remains valuable compared with LLM-based methods in terms of faster inference and better robustness under large biasing lists.

Figures

Figures reproduced from arXiv: 2506.11064 by the authors.

Figure 1
Figure 1. Illustration of the differences between our previous ED-CEC method [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overall architecture of the proposed PMF-CEC model. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Illustration of the PMF module. E. ASR Error Detection (AED) Module The AED module takes the multimodal representation E(M) as input to predict editing labels for each token in the original ASR transcript, as shown in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Illustration of the context decoder. Context Decoder. The illustration of the context decoder is shown in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Process for creating the rare word list: (a) Simulating real-world tasks using the LibriSpeech dataset, following the method described in [37], which [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: WER results of Librispeech test sets with varying rare word list sizes. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Examples of correcting the rare words “maier” and “erlangen” in the PRLVS dataset. We present slides with the rare words, the original ASR [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: An example from the DATA2 test set illustrating the challenge of rare [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

65 extracted references · 56 canonical work pages

  1. [17]

    PATCorrect: Non-autoregressive phoneme-augmented transformer for ASR error correction,

    Z. Zhang, Z. Wang, R. Kamma, S. Eswaran, and N. Sadagopan, “PATCorrect: Non-autoregressive phoneme-augmented transformer for ASR error correction,” inProc. Interspeech, 2023, pp. 3904–3908

    work page 2023

  2. [24]

    Pronunciation guided copy and correction model for ASR error correction,

    L. Dong, W. Wang, Z. Yu, Y . Huang, J. Guo, and G. Zhou, “Pronunciation guided copy and correction model for ASR error correction,”International Journal of Machine Learning and Cybernetics, pp. 1–13, 2024

    work page 2024

  3. [1]

    Robust speech recog- nition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recog- nition via large-scale weak supervision,” inProc. ICML, 2023, pp. 28 492–28 518

    work page 2023

  4. [2]

    Improving neural biasing for contextual speech recognition by early context injection and text perturbation,

    R. Huang, M. Yarmohammadi, S. Khudanpur, and D. Povey, “Improving neural biasing for contextual speech recognition by early context injection and text perturbation,” inProc. Interspeech, 2024, pp. 752–756

    work page 2024

  5. [3]

    Multi-modal video sum- marization based on two-stage fusion of audio, visual, and recognized text information,

    Z. Yang, J. He, and T. Toda, “Multi-modal video sum- marization based on two-stage fusion of audio, visual, and recognized text information,” inProc. APSIPA ASC, 2024, pp. 1–6

    work page 2024

  6. [4]

    In-context learning for few-shot nested named entity recognition,

    M. Zhang, B. Wang, H. Fei, and M. Zhang, “In-context learning for few-shot nested named entity recognition,” inProc. ICASSP, 2024, pp. 10 026–10 030

    work page 2024

  7. [5]

    MF-AED-AEC: Speech emotion recognition by leveraging multimodal fusion, ASR error detection, and ASR error correction,

    J. He, X. Shi, X. Li, and T. Toda, “MF-AED-AEC: Speech emotion recognition by leveraging multimodal fusion, ASR error detection, and ASR error correction,” inProc. ICASSP, 2024, pp. 11 066–11 070. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 15

    work page 2024

  8. [6]

    Tree-constrained pointer generator with graph neural network encodings for contextual speech recognition,

    G. Sun, C. Zhang, and P. C. Woodland, “Tree-constrained pointer generator with graph neural network encodings for contextual speech recognition,” inProc. Interspeech, 2022, pp. 2043–2047

    work page 2022

Show all 65 references
  1. [7]

    MaLa-ASR: Multimedia-assisted LLM-based ASR,

    G. Yang, Z. Ma, F. Yu, Z. Gao, S. Zhang, and X. Chen, “MaLa-ASR: Multimedia-assisted LLM-based ASR,” in Proc. Interspeech, 2024, pp. 2405–2409

    2024

  2. [8]

    Contextual speech recognition in end- to-end neural network systems using beam search

    I. Williams, A. Kannan, P. S. Aleksic, D. Rybach, and T. N. Sainath, “Contextual speech recognition in end- to-end neural network systems using beam search.” in Interspeech, 2018, pp. 2227–2231

    2018

  3. [9]

    Deep shallow fusion for RNN-T personalization,

    D. Le, G. Keren, J. Chan, J. Mahadeokar, C. Fuegen, and M. L. Seltzer, “Deep shallow fusion for RNN-T personalization,” inProc. SLT, 2021, pp. 251–257

    2021

  4. [10]

    Fac- torized neural transducer for efficient language model adaptation,

    X. Chen, Z. Meng, S. Parthasarathy, and J. Li, “Fac- torized neural transducer for efficient language model adaptation,” inProc. ICASSP, 2022, pp. 8132–8136

    2022

  5. [11]

    Tree-constrained pointer generator for end-to-end contextual speech recog- nition,

    G. Sun, C. Zhang, and P. C. Woodland, “Tree-constrained pointer generator for end-to-end contextual speech recog- nition,” inProc. ASRU, 2021, pp. 780–787

    2021

  6. [12]

    Graph neural networks for contextual ASR with the tree-constrained pointer generator,

    G. Sun, C. Zhang, and P. C. Woodland, “Graph neural networks for contextual ASR with the tree-constrained pointer generator,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, pp. 2407–2417, 2024

    2024

  7. [13]

    Deep context: end-to-end contextual speech recognition,

    G. Pundak, T. N. Sainath, R. Prabhavalkar, A. Kan- nan, and D. Zhao, “Deep context: end-to-end contextual speech recognition,” inProc. SLT, 2018, pp. 418–425

    2018

  8. [14]

    In- stant one-shot word-learning for context-specific neu- ral sequence-to-sequence speech recognition,

    C. Huber, J. Hussain, S. St ¨uker, and A. Waibel, “In- stant one-shot word-learning for context-specific neu- ral sequence-to-sequence speech recognition,” inProc. ASRU, 2021, pp. 1–7

    2021

  9. [15]

    Seed-ASR: Understanding diverse speech and contexts with LLM- based speech recognition,

    Y . Bai, J. Chen, J. Chen, W. Chen, Z. Chen, C. Ding, L. Dong, Q. Dong, Y . Du, K. Gaoet al., “Seed-ASR: Understanding diverse speech and contexts with LLM- based speech recognition,” inarXiv:2407.04675, 2024

    2024 arXiv

  10. [16]

    An embarrassingly simple approach for LLM with strong ASR capacity,

    Z. Ma, G. Yang, Y . Yang, Z. Gao, J. Wang, Z. Du, F. Yu, Q. Chen, S. Zheng, S. Zhanget al., “An embarrassingly simple approach for LLM with strong ASR capacity,” in arXiv:2402.08846, 2024

    2024 arXiv

  11. [18]

    Towards contextual spelling correction for customization of end-to-end speech recognition systems,

    X. Wang, Y . Liu, J. Li, V . Miljanic, S. Zhao, and H. Khalil, “Towards contextual spelling correction for customization of end-to-end speech recognition systems,” IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, vol. 30, pp. 3089–3097, 2022

    2022

  12. [19]

    ASR error correction with constrained decoding on operation prediction,

    J. Yang, R. Li, and W. Peng, “ASR error correction with constrained decoding on operation prediction,” inProc. Interspeech, 2022, p. 3874–3878

    2022

  13. [20]

    Crossmodal ASR error correction with discrete speech units,

    Y . Li, P. Chen, P. Bell, and C. Lai, “Crossmodal ASR error correction with discrete speech units,” inProc. SLT, 2024, pp. 1–8

    2024

  14. [21]

    ED-CEC: Improving rare word recognition using ASR postprocessing based on error detection and context-aware error correction,

    J. He, Z. Yang, and T. Toda, “ED-CEC: Improving rare word recognition using ASR postprocessing based on error detection and context-aware error correction,” in Proc. ASRU, 2023, pp. 1–6

    2023

  15. [22]

    ASR error correction with dual-channel self-supervised learning,

    F. Zhang, M. Tu, S. Liu, and J. Yan, “ASR error correction with dual-channel self-supervised learning,” in Proc. ICASSP, 2022, pp. 7282–7286

    2022

  16. [23]

    Enhancing recognition of rare words in ASR through error detection and context- aware error correction,

    J. He, Z. Yang, and T. Toda, “Enhancing recognition of rare words in ASR through error detection and context- aware error correction,”IEICE Tech. Rep., vol. 123, no. 292, pp. 13–18, 2023

    2023

  17. [25]

    Hyporadise: An open baseline for generative speech recognition with large language models,

    C. Chen, Y . Hu, C.-H. H. Yang, S. M. Siniscalchi, P.-Y . Chen, and E.-S. Chng, “Hyporadise: An open baseline for generative speech recognition with large language models,” inProc. NeurIPS, 2023, pp. 31 665–31 688

    2023

  18. [26]

    Generative speech recognition error correction with large language models and task-activating prompting,

    C.-H. H. Yang, Y . Gu, Y .-C. Liu, S. Ghosh, I. Bulyko, and A. Stolcke, “Generative speech recognition error correction with large language models and task-activating prompting,” inProc. ASRU, 2023, pp. 1–8

    2023

  19. [27]

    Large language model based generative error correction: A challenge and baselines for speech recognition, speaker tagging, and emotion recognition,

    C.-H. H. Yang, T. Park, Y . Gong, Y . Li, Z. Chen, Y .- T. Lin, C. Chen, Y . Hu, K. Dhawan, P. ˙Zelaskoet al., “Large language model based generative error correction: A challenge and baselines for speech recognition, speaker tagging, and emotion recognition,” inProc. SLT, 202...

    2024

  20. [28]

    Large language models are efficient learners of noise-robust speech recognition,

    Y . Hu, C. Chen, C.-H. H. Yang, R. Li, C. Zhang, P.-Y . Chen, and E. Chng, “Large language models are efficient learners of noise-robust speech recognition,” inProc. ICLR, 2024, pp. 1–25

    2024

  21. [29]

    Revise, reason, and recognize: LLM-based emotion recognition via emotion-specific prompts and ASR error correction,

    Y . Li, Y . Gong, C.-H. H. Yang, P. Bell, and C. Lai, “Revise, reason, and recognize: LLM-based emotion recognition via emotion-specific prompts and ASR error correction,” inProc. ICASSP, 2025, pp. 1–5

    2025

  22. [30]

    ASR error cor- rection using large language models,

    R. Ma, M. Qian, M. Gales, and K. Knill, “ASR error cor- rection using large language models,”IEEE Transactions on Audio, Speech and Language Processing, vol. 33, pp. 1389–1401, 2025

    2025

  23. [31]

    Bringing contextual information to Google speech recognition

    P. S. Aleksic, M. Ghodsi, A. H. Michaely, C. Al- lauzen, K. B. Hall, B. Roark, D. Rybach, and P. J. Moreno, “Bringing contextual information to Google speech recognition.” inProc. Interspeech, 2015, pp. 468– 472

    2015

  24. [32]

    Personalized speech recognition on mobile devices,

    I. McGraw, R. Prabhavalkar, R. Alvarez, M. G. Arenas, K. Rao, D. Rybach, O. Alsharif, H. Sak, A. Gruenstein, F. Beaufayset al., “Personalized speech recognition on mobile devices,” inProc. ICASSP, 2016, pp. 5955–5959

    2016

  25. [33]

    Shallow-fusion end-to-end contex- tual biasing

    D. Zhao, T. N. Sainath, D. Rybach, P. Rondon, D. Bhatia, B. Li, and R. Pang, “Shallow-fusion end-to-end contex- tual biasing.” inProc. Interspeech, 2019, pp. 1418–1422

    2019

  26. [34]

    Im- proving ASR contextual biasing with guided attention,

    J. Tang, K. Kim, S. Shon, F. Wu, and P. Sridhar, “Im- proving ASR contextual biasing with guided attention,” inProc. ICASSP, 2024, pp. 12 096–12 100

    2024

  27. [35]

    NAM+: Towards scalable end-to-end contextual biasing for adaptive ASR,

    T. Munkhdalai, Z. Wu, G. Pundak, K. C. Sim, J. Li, P. Rondon, and T. N. Sainath, “NAM+: Towards scalable end-to-end contextual biasing for adaptive ASR,” inProc. SLT, 2023, pp. 190–196

    2023

  28. [36]

    Robust JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 16 acoustic and semantic contextual biasing in neural trans- ducers for speech recognition,

    X. Fu, K. M. Sathyendra, A. Gandhe, J. Liu, G. P. Strimel, R. McGowan, and A. Mouchtaris, “Robust JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 16 acoustic and semantic contextual biasing in neural trans- ducers for speech recognition,” inProc. ICASSP, 2023, pp. 1–5

    2021

  29. [37]

    Contextualized streaming end-to-end speech recognition with trie-based deep biasing and shallow fusion,

    D. Le, M. Jain, G. Keren, S. Kim, Y . Shi, J. Mahadeokar, J. Chan, Y . Shangguan, C. Fuegen, O. Kalinliet al., “Contextualized streaming end-to-end speech recognition with trie-based deep biasing and shallow fusion,” inProc. Interspeech, 2021, pp. 1772–1776

    2021

  30. [38]

    Joint grapheme and phoneme embeddings for contextual end-to-end ASR,

    Z. Chen, M. Jain, Y . Wang, M. L. Seltzer, and C. Fuegen, “Joint grapheme and phoneme embeddings for contextual end-to-end ASR,” inProc. Interspeech, 2019, pp. 3490– 3494

    2019

  31. [39]

    Phoebe: Pronunciation-aware contextualization for end-to-end speech recognition,

    A. Bruguier, R. Prabhavalkar, G. Pundak, and T. N. Sainath, “Phoebe: Pronunciation-aware contextualization for end-to-end speech recognition,” inProc. ICASSP, 2019, pp. 6171–6175

    2019

  32. [40]

    Context-aware transformer transducer for speech recognition,

    F.-J. Chang, J. Liu, M. Radfar, A. Mouchtaris, M. Omol- ogo, A. Rastrow, and S. Kunzmann, “Context-aware transformer transducer for speech recognition,” inProc. ASRU, 2021, pp. 503–510

    2021

  33. [41]

    Contextual adapters for personalized speech recognition in neural transducers,

    K. M. Sathyendra, T. Muniyappa, F.-J. Chang, J. Liu, J. Su, G. P. Strimel, A. Mouchtaris, and S. Kunzmann, “Contextual adapters for personalized speech recognition in neural transducers,” inProc. ICASSP, 2022, pp. 8537– 8541

    2022

  34. [42]

    Contextualized automatic speech recog- nition with attention-based bias phrase boosted beam search,

    Y . Sudo, M. Shakeel, Y . Fukumoto, Y . Peng, and S. Watanabe, “Contextualized automatic speech recog- nition with attention-based bias phrase boosted beam search,” inProc. ICASSP, 2024, pp. 10 896–10 900

    2024

  35. [43]

    LCB-Net: Long- context biasing for audio-visual speech recognition,

    F. Yu, H. Wang, X. Shi, and S. Zhang, “LCB-Net: Long- context biasing for audio-visual speech recognition,” in Proc. ICASSP, 2024, pp. 10 621–10 625

    2024

  36. [44]

    A light-weight contextual spelling correction model for customizing transducer-based speech recognition systems,

    X. Wang, Y . Liu, S. Zhao, and J. Li, “A light-weight contextual spelling correction model for customizing transducer-based speech recognition systems,” inProc. Interspeech, 2021, pp. 1982–1986

    2021

  37. [45]

    Contextual modeling for document-level ASR error correction,

    J. Jiang, X. Yin, X. Wan, W. Peng, R. Li, J. Yang, and Y . Zhou, “Contextual modeling for document-level ASR error correction,” inProc. LREC-COLING, 2024, pp. 3855–3867

    2024

  38. [46]

    Google’s neural machine translation system: Bridging the gap between human and machine transla- tion,

    Y . Wu, M. Schuster, Z. Chen, Q. V . Le, M. Norouzi, W. Macherey, M. Krikun, Y . Cao, Q. Gao, K. Macherey et al., “Google’s neural machine translation system: Bridging the gap between human and machine transla- tion,”arXiv:1609.08144, 2016

    2016 arXiv

  39. [47]

    BERT: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,”Proc. NAACL, pp. 4171– 4186, 2019

    2019

  40. [48]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” inProc. NIPS, 2017, pp. 5998–6008

    2017

  41. [49]

    XPhoneBERT: A pre-trained multilingual model for phoneme representations for text-to-speech,

    L. The Nguyen, T. Pham, and D. Q. Nguyen, “XPhoneBERT: A pre-trained multilingual model for phoneme representations for text-to-speech,” inProc. Interspeech, 2023, pp. 5506–5510

    2023

  42. [50]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” inProc. ICLR, 2015, pp. 1–11

    2015

  43. [51]

    Phoneme- BERT: Joint language modelling of phoneme sequence and ASR transcript,

    M. N. Sundararaman, A. Kumar, and J. Vepa, “Phoneme- BERT: Joint language modelling of phoneme sequence and ASR transcript,” inProc. Interspeech, 2021, pp. 3236–3240

    2021

  44. [52]

    Listen, attend and spell: A neural net- work for large vocabulary conversational speech recog- nition,

    W. Chanet al., “Listen, attend and spell: A neural net- work for large vocabulary conversational speech recog- nition,” inProc. ICASSP, 2016, pp. 4960–4964

    2016

  45. [53]

    Learning ASR-robust contextualized embeddings for spoken language under- standing,

    C.-W. Huang and Y .-N. Chen, “Learning ASR-robust contextualized embeddings for spoken language under- standing,” inProc. ICASSP, 2020, pp. 8009–8013

    2020

  46. [54]

    The Kaldi speech recognition toolkit,

    D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glem- bek, N. Goel, M. Hannemann, P. Motlicek, Y . Qian, P. Schwarzet al., “The Kaldi speech recognition toolkit,” inProc. ASRU, 2011

    2011

  47. [55]

    LibriSpeech: An ASR corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “LibriSpeech: An ASR corpus based on public domain audio books,” inProc. ICASSP, 2015, pp. 5206–5210

    2015

  48. [56]

    WeNet: Pro- duction oriented streaming and non-streaming end-to-end speech recognition toolkit,

    Z. Yao, D. Wu, X. Wang, B. Zhang, F. Yu, C. Yang, Z. Peng, X. Chen, L. Xie, and X. Lei, “WeNet: Pro- duction oriented streaming and non-streaming end-to-end speech recognition toolkit,” inProc. Interspeech, 2021, pp. 4054–4058

    2021

  49. [57]

    End-to-end named entity recognition from english speech,

    H. Yadav, S. Ghosh, Y . Yu, and R. R. Shah, “End-to-end named entity recognition from english speech,” inProc. Interspeech, 2020, pp. 4268–4272

    2020

  50. [58]

    ESPnet: End-to-end speech processing toolkit,

    S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y . Unno, N. E. Y . Soplin, J. Heymann, M. Wiesner, N. Chenet al., “ESPnet: End-to-end speech processing toolkit,” inProc. Interspeech, 2018, pp. 2207–2211

    2018

  51. [59]

    Multimodal corpus analysis of autoblog 2020: lecture videos in machine learning,

    A. Hernandez and S. H. Yang, “Multimodal corpus analysis of autoblog 2020: lecture videos in machine learning,” inProc. SPECOM, 2021, pp. 262–270

    2020

  52. [60]

    SpeechBrain: A general-purpose speech toolkit,

    M. Ravanelli, T. Parcollet, P. Plantinga, A. Rouhe, S. Cor- nell, L. Lugosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhonget al., “SpeechBrain: A general-purpose speech toolkit,”arXiv:2106.04624, 2021

    2021 arXiv

  53. [61]

    BART based semantic correction for Mandarin auto- matic speech recognition system,

    Y . Zhao, X. Yang, J. Wang, Y . Gao, C. Yan, and Y . Zhou, “BART based semantic correction for Mandarin auto- matic speech recognition system,” inProc. Interspeech, 2021, pp. 2017–2021

    2021

  54. [62]

    Pre-trained summarization distillation,

    S. Shleifer and A. M. Rush, “Pre-trained summarization distillation,”arXiv:2010.13002, 2020

    2010 arXiv

  55. [63]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskeveret al., “Language models are unsupervised multitask learners,”OpenAI blog, 1(8), 9., 2019

    2019

  56. [64]

    Whispering LLaMa: A cross-modal generative error correction framework for speech recognition,

    S. Radhakrishnan, C.-H. H. Yang, S. A. Khan, R. Kumar, N. A. Kiani, D. Gomez-Cabrero, and J. N. Tegner, “Whispering LLaMa: A cross-modal generative error correction framework for speech recognition,” inProc. EMNLP, 2023, pp. 10 007–10 016

    2023

  57. [65]

    LLaMa: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azharet al., “LLaMa: Open and efficient foundation language models,” inarXiv:2302.13971, 2023

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.