OWSM-Biasing: Contextualizing Open Whisper-Style Speech Models for Automatic Speech Recognition with Dynamic Vocabulary

Atsushi Kojima; Lianbo Liu; Tomoya Mizumoto; Yui Sudo; Yusuke Fujita

REVIEW 4 major objections 6 minor 47 references

OWSM-Biasing: Contextualizing Open Whisper-Style Speech Models for Automatic Speech Recognition with Dynamic Vocabulary

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a frozen speech foundation model can gain contextual biasing through dynamic vocabulary extensions, improving biasing word error rate by 11.6 points while cutting decoding cost.

desk verdict A practical, credible integration of dynamic-vocabulary biasing into OWSM v3.1; main weakness is unmeasured biasing-encoder cost behind the RTF claim. read the letter →

arxiv 2506.09448 v1 pith:K4G2FQQF submitted 2025-06-11 cs.SD cs.CLeess.AS

Yui Sudo , Yusuke Fujita , Atsushi Kojima , Tomoya Mizumoto , Lianbo Liu This is my paper

classification cs.SDcs.CLeess.AS

keywords contextualbiasingdynamicvocabularyspeechfoundationmodelsautomaticrecognitionrarewordreal-timefactorOWSMv3.1Libri

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a large pretrained speech recognizer can gain contextual biasing without being retrained: freeze the recognizer, attach a small trainable module that expands the output vocabulary with one token for each word on an editable list, and keep the pretrained weights fixed. The payoff would be that rare and domain-specific words, which speech foundation models often miss, can be recognized accurately even when only a small amount of domain audio is available. On LibriSpeech 100 test-clean the authors report the biasing word error rate falling from 15.5 to 3.9 (an 11.6-point improvement), overall word error rate improving from 3.9 to 3.0, and decoding becoming 7.5% faster. They attribute the speed gain to shorter output sequences: a biasing word is emitted as one token instead of several subword tokens.

What carries the argument

The load-bearing piece is the dynamic-vocabulary biasing module set: a small Transformer biasing encoder with mean pooling produces a learned representation for every word in the biasing list; an extended embedding layer routes dynamic tokens to those representations; and an extended output layer adds dot-product similarity scores for the dynamic tokens to the original vocabulary logits before the softmax. Training only these modules while freezing OWSM v3.1 preserves the foundation model's knowledge. The representations can be cached while the list is unchanged, and emitting a whole word as one token shortens the decoder's output, which is what produces the reported real-time-factor reduction.

What would settle it

Run the model on LibriSpeech 100 test-clean with a biasing list that is rebuilt from scratch for every utterance, and compare the real-time factor with the frozen OWSM v3.1 baseline under the same beam size and hardware; if the real-time factor is not lower, the reported 7.5% speed advantage is conditional on list stability rather than a general property of the method.

Watch

Extended reading notes

Core claim

The central discovery is that dynamic-vocabulary contextual biasing can be layered onto OWSM v3.1 with all pretrained parameters frozen, and that this combination beats both training a biasing model from scratch and fine-tuning the whole OWSM for biasing. The method adds a biasing encoder that converts the word list into embeddings, an extended embedding layer that lets the decoder consume those dynamic tokens, and an extended output layer that scores them alongside the static vocabulary. Because the biasing words are generated as single tokens, decoding needs fewer iterations, and because the static vocabulary path is untouched, the recognizer's general quality is preserved. The gain persists as the list grows from 100 to 2,000 words, and the authors report that prompt-based biasing, by contrast, gives little benefit on the base model while the dynamic vocabulary method improves both base and medium versions.

Load-bearing premise

The computational-efficiency claim rests on the assumption that the biasing list changes infrequently, so the biasing encoder only runs once and its word representations can be cached; the paper excludes that encoder from the speed measurement for this reason.

Editorial extensions

If this is right

A speech foundation model can be adapted to a new vocabulary with a small amount of labeled audio, because only the biasing modules need training.
Words that never appeared in training data become reachable, not just words seen a few times.
Decoding can get faster even with extra modules, because a long word collapses to one output token and the number of decoding iterations drops.
Freezing the backbone avoids catastrophic forgetting, so general ASR quality is not traded away for rare-word accuracy.
The same extension should transfer to other open Whisper-style speech models whose encoder and decoder remain unchanged.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural stress test the paper does not run is a per-utterance biasing list, which defeats the embedding cache; in that setting the reported speed advantage would likely shrink or disappear even if the accuracy gains remain.
The speed benefit depends on word length: biasing words that are already a single subword token yield no decoding-length saving, so lexicons dominated by short words would see less real-time-factor gain.
The same recipe is most credible as a low-cost production upgrade for domains such as names, places, and product terms, where the list is stable and the target words are precisely the ones the frozen model misses; a test with such an out-of-domain list would confirm the LibriSpeech result transfers.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The headline: this is a solid engineering paper that shows dynamic-vocabulary biasing can be bolted onto OWSM v3.1 with frozen weights and get real gains. The B-WER improvement on test-clean is large (15.5 to 3.9 at N=100) and the overall WER also improves from 3.9 to 3.0. Those results are consistent across list sizes and both model sizes, which makes me believe the effect is real, not an artifact. The claimed novelty is narrow—the dynamic vocabulary method already exists [34]—but this is the first application to OWSM, and the small architectural tweak (repositioning the linear layer) is sensible and clearly described. The comparison against prompt-based biasing is useful: the proposed method works for the base model where prompting doesn't, which is a meaningful practical point.

Soft spots: the paper reports single runs without confidence intervals or significance tests. For a result this cheap to run (LibriSpeech-100, 100-200 epochs on small models), that's an easy fix and I'd want it before trusting the numbers. The biasing weight mu=0.3 is never swept, so we don't know how sensitive the B-WER gain is. There's also a textual ambiguity in 3.3: it says the OWSM encoder and decoder are frozen, but then says 'we train the embedding and output layers of OWSM.' Those can't both be true as written; presumably they mean the extended biasing modules, but the wording needs fixing. The RTF claim has a bigger caveat: the biasing encoder is excluded from measurement, justified by caching V unless the list changes. If the list is static or slowly changing, fine, but the paper gives no data on update frequency and no measurement of what the 6-layer biasing encoder costs when you do need to re-encode. That could easily wipe out the 7.5% RTF gain. This doesn't break the central WER contribution, but it makes the complexity claim conditional.

On the citation pattern: [34] is the method this builds on, and it's properly credited. I don't see any missing prior work that matters.

Bottom line: the central claim holds. This is a practical recipe for ASR practitioners who want domain vocabulary support in an SFM without retraining. It deserves a serious referee. I'd ask for error bars, a mu sweep, a cleaned-up description of what's frozen, and either a measurement of the biasing encoder latency or a clear deployment assumption about list update frequency.

Referee Report

4 major / 6 minor

Summary. The paper proposes OWSM-Biasing, a method that integrates the dynamic-vocabulary contextual biasing approach of [34] into OWSM v3.1 while freezing all OWSM parameters. Only a compact biasing encoder and extended embedding/output layers are trained. Experiments on LibriSpeech 100 with OWSM v3.1 base and medium show substantial reductions in biasing word error rate (B-WER) and modest overall WER improvements over the non-biasing OWSM baseline, with a claimed 7.5% real-time factor (RTF) reduction. The paper also compares against scratch-trained dynamic vocabulary, fully fine-tuned OWSM, and prompt-based biasing.

Significance. If the reported results are robust, the paper offers a practical recipe for contextualizing speech foundation models in data-scarce domains without sacrificing the generalization knowledge encoded in the frozen model. The architecture is simple and compatible with existing OWSM checkpoints, and the comparison across base and medium scales, multiple biasing-list sizes, and against a strong fine-tuned baseline is useful. The explicit computational-cost analysis is a strength, although the RTF result rests on a caching assumption that needs to be made quantitative.

major comments (4)

[Section 4.2, Table 2] The headline claim of a 7.5% RTF reduction is not supported as stated because the biasing encoder is excluded from the RTF measurement. Section 4.1 justifies this exclusion by asserting that the biasing-list embedding V can be cached 'unless the biasing list is updated, which is infrequently (e.g., not on a per-utterance basis).' This is an unverified assumption, and it is particularly consequential for a method whose purpose is dynamic vocabulary: in many realistic deployment settings the list changes across users, domains, or utterances. The biasing encoder is a 6-layer Transformer with 15.5M–27.4M parameters, and the paper reports no measurement of its encoding latency, no amortized-cost analysis as a function of list-update frequency, and no evidence for the 'infrequently updated' premise. Please either measure the biasing-encoder latency and report RTF under specific update schedules (e.g., per utterance, per hour, per domain), or explicitly restrict the RTF claim to the static-list scenario and describe the regime in which the 7.5% figure is valid.
[Section 4.2, Table 2] The text states that fully fine-tuning OWSM for dynamic-vocabulary biasing 'suffers from catastrophic forgetting, potentially degrading overall WER compared to OWSM v3.1 (A1 vs. A3).' On the test-clean N=100 row of Table 2, however, A3 achieves WER 3.6 versus A1's 3.9, so the comparison in that row contradicts the stated degradation. The degradation does appear at N=1000 (4.0 vs. 3.9), so the claim should be qualified to specific list sizes or supported with additional evidence. As written, the narrative overstates the fine-tuning baseline's weakness.
[Section 3.3 and Section 4.1] All experiments use a single biasing weight mu=0.3, with no sensitivity analysis. Since mu directly controls the trade-off between over-biasing and under-biasing and hence the balance between B-WER and overall WER, the reported improvements could be specific to this one value. Please provide a sweep over mu for at least one configuration (e.g., OWSM-Biasing base on test-clean with N=100 and N=1000) to show that the qualitative conclusions are robust.
[Section 4.5 and Table 2] The results are reported as single-run point estimates without confidence intervals or significance testing. Some of the headline margins are small: the overall WER improvement is 0.9 point on test-clean (3.9 to 3.0), and the RTF reduction is 7.5%. For a paper that emphasizes comparisons across methods and list sizes, reporting only one run makes it hard to assess whether the smaller differences are meaningful. Please provide multiple runs, confidence intervals, or a significance test for at least the main comparisons in Tables 2 and 4.

minor comments (6)

[Section 4.1] The phrase 'with a beam size of 3, with a beam size of 3' is duplicated; please remove the repetition.
[Table 1] There are two 'Total params' rows, and the 'Trainable rate' values are inconsistent: for the base model, 15.5M/74.5M is 20.8%, while 15.5M/59.0M is 26.3%; for the medium model, 27.4M/928.0M is 3.0%. Please clarify what denominator is intended and remove the duplicate row.
[Table 4] The caption contains the typo 'comaprison'; it should read 'comparison'.
[Figure 1] The caption says 'Red and blue block represent trainable and frozen modules'; the singular 'block' should be 'blocks'.
[Equation (10)] The notation Linear(V^T) is ambiguous: please specify whether the linear layer is applied along the token dimension and define the output dimension, so that the dot product in Eq. (10) is dimensionally clear.
[Section 4.1] The paper uses B-WER following [33] but does not restate the definition. Since B-WER is central to the evaluation, please give the formula or a precise definition in the experimental setup.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported gains are empirical evaluation results, and the disclosed RTF caching caveat is a limitation, not a circular step.

full rationale

The paper's central claims are empirical outcomes, not derivations whose conclusions are equivalent to their inputs. The architecture is given explicitly in Eqs. (6)-(11), so the dynamic-vocabulary mechanism is not imported as an unexamined black box; although it originates in the authors' prior work [34], the paper re-derives it and trains the scratch baseline A2 under the same setup, making the citation an independently verifiable starting point rather than a load-bearing self-citation. The biasing weight mu is an inference hyperparameter fixed at 0.3, not a fitted parameter renamed as a prediction, and no equation defines WER or B-WER in terms of mu, the biasing list, or OWSM's own outputs. The one caveat is the RTF measurement in Section 4.1, which excludes the biasing encoder on the stated assumption that V can be cached when the biasing list is infrequently updated; this is a transparent limitation about deployment conditions, not a circularity, because the reported 7.5% RTF reduction is an arithmetic consequence of measured decoding iterations and the disclosed exclusion. If the biasing list changed per utterance the RTF claim could fail, but that is a validity risk, not a derivation that reduces to itself. Accordingly, no circular step is present.

Assumptions & free parameters 1 free parameters · 4 assumptions · 1 invented entities

The paper introduces no new physical entities and derives no new theory. Its principal assumptions are the stability of the biasing list, the compatibility of the frozen OWSM core with attached dynamic-vocabulary modules, and the validity of combining static and dynamic scores with a single softmax. The only explicit free parameter affecting the headline result is the biasing weight mu.

free parameters (1)

biasing weight mu = 0.3
Interpolates static and dynamic token scores at inference (Section 3.3). No validation criterion or sensitivity sweep is reported, so the headline B-WER gains depend on this chosen value.

assumptions (4)

domain assumption OWSM v3.1's pretrained encoder and decoder encode useful knowledge that can be leveraged for rare words without updating the core parameters.
The method freezes OWSM and relies on its representations to make dynamic-token prediction accurate (Section 3.3).
domain assumption The dynamic-vocabulary method of reference [34] remains effective when its embedding and output extensions are attached to frozen OWSM components.
The paper combines [34] with OWSM and reports gains, but the compatibility is an empirical assumption rather than a derived property.
domain assumption The biasing list B is available at inference and changes infrequently, so the biasing encoder embeddings V can be precomputed.
The RTF measurement excludes the biasing encoder on this basis (Section 4.1). If the list changes per utterance, the RTF benefit may not hold.
domain assumption Static vocabulary scores and dynamic vocabulary scores can be combined in a single softmax without additional calibration.
Equation (11) concatenates alpha_s and alpha_b and applies softmax, assuming the two score sets are compatible enough to share one probability distribution.

invented entities (1)

Dynamic vocabulary tokens <b_n>, for example <alligator> independent evidence
purpose: Represent each biasing word as a single output token so the decoder can emit the whole word in one step, replacing multiple subword tokens.
The mechanism originates in reference [34], which reports its own experiments. This paper adds OWSM-specific integration and RTF evidence. It is a modeling construct, not a new physical entity, and it is not pulled from a hat solely for this paper.

how reviews work

0 comments

Cite this review

Pith. "Pith review of OWSM-Biasing: Contextualizing Open Whisper-Style Speech Models for Automatic Speech Recognition with Dynamic Vocabulary." pith.science (2026). https://pith.science/paper/K4G2FQQF

@misc{pith2026250609448,
  author       = {Pith},
  title        = {Pith review of: OWSM-Biasing: Contextualizing Open Whisper-Style Speech Models for Automatic Speech Recognition with Dynamic Vocabulary},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K4G2FQQF}},
  note         = {Machine review of arXiv:2506.09448}
}

read the original abstract

Speech foundation models (SFMs), such as Open Whisper-Style Speech Models (OWSM), are trained on massive datasets to achieve accurate automatic speech recognition. However, even SFMs struggle to accurately recognize rare and unseen words. While contextual biasing (CB) is a promising approach to improve recognition of such words, most CB methods are trained from scratch, resulting in lower performance than SFMs due to the lack of pre-trained knowledge. This paper integrates an existing CB method with OWSM v3.1 while freezing its pre-trained parameters. By leveraging the knowledge embedded in SFMs, the proposed method enables effective CB while preserving the advantages of SFMs, even with a small dataset. Experimental results show that the proposed method improves the biasing word error rate (B-WER) by 11.6 points, resulting in a 0.9 point improvement in the overall WER while reducing the real-time factor by 7.5% compared to the non-biasing baseline on the LibriSpeech 100 test-clean set.

Figures

Figures reproduced from arXiv: 2506.09448 by the authors.

**Figure 1.** shows the overall architecture of the proposed method, which integrates OWSM v3.1 with the dynamic vocabularybased CB method [34]. Since this method does not change the core structures of the encoder and decoder, only the biasing modules ( [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** The biasing modules of the dynamic vocabulary. Red and gray blocks show trainable and non-trainable components. 3.3. Training and decoding To preserve the advantages of OWSM v3.1, we freeze the parameters of the OWSM encoder and decoder as shown in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Performance comparison on rare and unseen words. 4.3. Analysis on rare and unseen words Figure 3a illustrates the error rates for rare and unseen words that occur less than 100 times in the training data. The red and blue lines represent the non-biasing baseline and the proposed method with a biasing list size of N = 100, respectively. The baseline struggles to accurately recognize these words, especially those tha… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: compares the proposed method with the prompt-based CB method [7] for both OWSM v3.1 base and medium models. For reference, we also include the results of SpeechLMbased methods (C1, C2) [13, 17]. While OWSM v3.1 medium (0.9B parameters) improves the B-WER by prompting…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 40 canonical work pages

[34]

Phoneme-aware encoding for prefix-tree-based contextual ASR,

H. Futami, E. Tsunoo, Y . Kashiwagi, H. Ogawa, S. Arora, and S. Watanabe, “Phoneme-aware encoding for prefix-tree-based contextual ASR,” inProc. ICASSP, 2024

work page 2024
[1]

These models are trained on mas- sive datasets, allowing them to generalize well across multiple domains and achieve high recognition accuracy

Introduction Speech foundation models (SFMs) [1–7], such as OpenAI’s Whisper [1] and Open Whisper-Style Speech Models (OWSM) [2], have demonstrated remarkable performance in automatic speech recognition (ASR). These models are trained on mas- sive datasets, allowing them to generalize well across multiple domains and achieve high recognition accuracy. Des...

arXiv 2025
[2]

OWSM v3.1 This section provides an overview of OWSM v3.1 [7], which is integrated into the dynamic vocabulary-based CB method [34] in Section 3. 2.1. OWSM encoder OWSM v3.1 adopts the E-Branchformer [38] for the encoder. The OWSM encoder consists of stacked E-Branchformer lay- ers, which utilize parallel branches to capture both local and global features....

work page
[3]

all”, “ig

Integration with dynamic vocabulary Figure 1 shows the overall architecture of the proposed method, which integrates OWSM v3.1 with the dynamic vocabulary- based CB method [34]. Since this method does not change the core structures of the encoder and decoder, only the bias- ing modules (Figure 2) are added to OWSM v3.1 while freezing the pre-trained param...

work page
[4]

We train the embed- ding and output layers of OWSM with vocabulary sizeKof 5,000

Figure 2 illustrates the trainable components in the biasing modules, where red and gray blocks represent the trainable and non-trainable components, respectively. We train the embed- ding and output layers of OWSM with vocabulary sizeKof 5,000. The trainable modules primarily consist of lightweight components, such as linear and embedding layers. Althoug...

work page
[5]

the” and “and

Experiment We conduct several experiments to verify the effectiveness of the proposed method. 4.1. Experimental setup Table 1 shows the model configurations of the proposed method, which consists of OWSM v3.1 and the biasing mod- ules. OWSM v3.1 employs the E-Branchformer [38] and Trans- former as the encoder and decoder, respectively. We use both the bas...

work page
[6]

Conclusion This paper integrates the dynamic vocabulary-based CB method with OWSM v3.1, freezing the pre-trained parameters to enable effective CB even with a small dataset, while preserving the advantages of SFMs. The proposed method improves the B- WER by 11.6 points, resulting in a 0.9 point improvement in the overall WER while reducing the RTF by 7.5%...

work page
[7]

Ro- bust speech recognition via large-scale weak supervision

R. Alec, K. Jong W., X. T., B. G., M. Christine, and S. Ilya, “Ro- bust speech recognition via large-scale weak supervision.” 2022

work page 2022

Show all 47 references

[8]

Repro- ducing Whisper-style training using an open-source toolkit and publicly available data,

Y . Peng, J. Tian, B. Yan, D. Berrebbi, X. Changet al., “Repro- ducing Whisper-style training using an open-source toolkit and publicly available data,” inProc. ASRU, 2023, pp. 1–8

2023
[9]

Google USM: Scaling automatic speech recognition beyond 100 lan- guages,

Y . Zhang, W. Han, J. Qin, Y . Wang, A. Bapnaet al., “Google USM: Scaling automatic speech recognition beyond 100 lan- guages,”arXiv preprint arXiv:2303.01037, 2023

2023 arXiv
[10]

Scal- ing speech technology to 1,000+ languages,

V . Pratap, A. Tjandra, B. Shi, P. Tomasello, A. Babuet al., “Scal- ing speech technology to 1,000+ languages,”Journal of Machine Learning Research, vol. 25, no. 97, pp. 1–52, 2024

2024
[11]

Less is more: Accurate speech recognition & translation without web-scale data,

K. C. Puvvada, P. ˙Zelasko, H. Huang, O. Hrinchuk, N. R. Koluguri et al., “Less is more: Accurate speech recognition & translation without web-scale data,” inProc. Interspeech, 2024

2024
[12]

OWSM-CTC: An open encoder-only speech foundation model for speech recog- nition, translation, and language identification,

Y . Peng, Y . Sudo, M. Shakeel, and S. Watanabe, “OWSM-CTC: An open encoder-only speech foundation model for speech recog- nition, translation, and language identification,” inProc. ACL, 2024, pp. 10 192–10 209

2024
[13]

OWSM v3.1: Better and faster open Whisper-style speech models based on e- branchformer,

Y . Peng, J. Tian, W. Chen, S. Arora, B. Yanet al., “OWSM v3.1: Better and faster open Whisper-style speech models based on e- branchformer,” inProc. Interspeech, 2024, pp. 352–356

2024
[14]

CB-Whisper: Contex- tual biasing Whisper using open-vocabulary keyword-spotting,

Y . Li, Y . Li, M. Zhang, C. Su, J. Yuet al., “CB-Whisper: Contex- tual biasing Whisper using open-vocabulary keyword-spotting,” inProc. LREC-COLING, 2024, pp. 2941–2946

2024
[15]

Can contex- tual biasing remain effective with Whisper and GPT-2?

G. Sun, X. Zheng, C. Zhang, and P. C. Woodland, “Can contex- tual biasing remain effective with Whisper and GPT-2?” inProc. Interspeech, 2023, pp. 1289–1293

2023
[16]

Adding user feedback to enhance CB-Whisper,

R. Monteiro, “Adding user feedback to enhance CB-Whisper,” in Interspeech 2024, 2024, pp. 347–351

2024
[17]

Contextual biasing to improve domain- specific custom vocabulary audio transcription without explicit fine-tuning of Whisper model,

V . Lall and Y . Liu, “Contextual biasing to improve domain- specific custom vocabulary audio transcription without explicit fine-tuning of Whisper model,” in2024 7th International Con- ference on Machine Learning and Natural Language Processing (MLNLP), 2024, pp. 1–6

2024
[18]

A multitask train- ing approach to enhance Whisper with open-vocabulary keyword spotting,

Y . Li, M. Zhang, C. Su, Y . Li, X. Qiaoet al., “A multitask train- ing approach to enhance Whisper with open-vocabulary keyword spotting,” inProc. Interspeech, 2024, pp. 1260–1264

2024
[19]

Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models,

Y . Chu, J. Xu, X. Zhou, Q. Yang, S. Zhanget al., “Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models,”arXiv preprint arXiv:2311.07919, 2023

2023 arXiv
[20]

SALMONN: Towards generic hearing abilities for large language models,

C. Tang, W. Yu, G. Sun, X. Chen, T. Tanet al., “SALMONN: Towards generic hearing abilities for large language models,” in Proc. ICLR, 2024

2024
[21]

An embarrass- ingly simple approach for LLM with strong ASR capacity,

Z. Ma, G. Yang, Y . Yang, Z. Gao, J. Wanget al., “An embarrass- ingly simple approach for LLM with strong ASR capacity,”arXiv preprint arXiv:2402.08846, 2024

2024 arXiv
[22]

V oiceTextBlender: Augmenting large language models with speech capabilities via single-stage joint speech-text supervised fine-tuning,

Y . Peng, K. C. Puvvada, Z. Chen, P. Zelasko, H. Huanget al., “V oiceTextBlender: Augmenting large language models with speech capabilities via single-stage joint speech-text supervised fine-tuning,”arXiv preprint arXiv:2410.17485, 2024

arXiv
[23]

Contextual biasing speech recognition in speech-enhanced large language model,

X. Gong, A. Lv, Z. Wang, and Y . Qian, “Contextual biasing speech recognition in speech-enhanced large language model,” in Proc. Interspeech, 2024, pp. 257–261

2024
[24]

Harnessing the zero- shot power of instruction-tuned large language model in end-to- end speech recognition,

Y . Higuchi, T. Ogawa, and T. Kobayashi, “Harnessing the zero- shot power of instruction-tuned large language model in end-to- end speech recognition,”arXiv preprint arXiv:2309.10524, 2023

2023 arXiv
[25]

Deep context: End-to-end contextual speech recogni- tion,

G. Pundak, T. N. Sainath, R. Prabhavalkar, A. Kannan, and D. Zhao, “Deep context: End-to-end contextual speech recogni- tion,” inProc. SLT, 2018, pp. 418–425

2018
[26]

Contextual RNN-T for open domain asr,

M. Jain, G. Keren, J. Mahadeokar, and Y . Saraf, “Contextual RNN-T for open domain asr,” inProc. Interspeech, 2020, pp. 11– 15

2020
[27]

Instant one-shot word-learning for context-specific neural sequence-to-sequence speech recognition,

C. Huber, J. Hussain, S. St ¨uker, and A. Waibel, “Instant one-shot word-learning for context-specific neural sequence-to-sequence speech recognition,” inProc. ASRU, 2021, pp. 1–7

2021
[28]

Retraining-free customized ASR for enharmonic words based on a named-entity-aware model and phoneme similarity estimation,

Y . Sudo, K. Hata, and K. Nakadai, “Retraining-free customized ASR for enharmonic words based on a named-entity-aware model and phoneme similarity estimation,” inProc. Interspeech, 2023, pp. 3312–3316

2023
[29]

Contex- tualized End-to-End Speech Recognition with Contextual Phrase Prediction Network,

K. Huang, A. Zhang, Z. Yang, P. Guo, B. Muet al., “Contex- tualized End-to-End Speech Recognition with Contextual Phrase Prediction Network,” inProc. Interspeech, 2023, pp. 4933–4937

2023
[30]

Copyne: Better contextual asr by copying named entities,

S. Zhou, Z. Li, Y . Hong, M. Zhang, Z. Wang, and B. Huai, “Copyne: Better contextual asr by copying named entities,”arXiv preprint arXiv:2305.12839, 2023

2023 arXiv
[31]

Lib- rispeech: an ASR corpus based on public domain audio books,

V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: an ASR corpus based on public domain audio books,” inProc. ICASSP, 2015, pp. 5206–5210

2015
[32]

Contextualized end-to-end automatic speech recognition with intermediate bias- ing loss,

M. Shakeel, Y . Sudo, Y . Peng, and S. Watanabe, “Contextualized end-to-end automatic speech recognition with intermediate bias- ing loss,” inProc. Interspeech, 2024, pp. 3909–3913

2024
[33]

Corpus of spontaneous Japanese: Its design and evaluation,

K. Maekawa, “Corpus of spontaneous Japanese: Its design and evaluation,” inISCA & IEEE Workshop on Spontaneous Speech Processing and Recognition, 2003

2003
[35]

Interbiasing: Boost unseen word recognition through biasing intermediate predictions,

Y . Nakagome and M. Hentschel, “Interbiasing: Boost unseen word recognition through biasing intermediate predictions,” in Proc. Interspeech, 2024, pp. 207–211

2024
[36]

PromptASR for contextualized ASR with controllable style,

X. Yang, W. Kang, Z. Yao, Y . Yang, L. Guoet al., “PromptASR for contextualized ASR with controllable style,” inProc. ICASSP, 2024, pp. 10 536–10 540

2024
[37]

Contextualized automatic speech recognition with attention- based bias phrase boosted beam search,

Y . Sudo, M. Shakeel, Y . Fukumoto, Y . Peng, and S. Watan- abe, “Contextualized automatic speech recognition with attention- based bias phrase boosted beam search,” inProc. ICASSP, 2024, pp. 10 896–10 900

2024
[38]

Improving large-scale deep biasing with phoneme features and text-only data in streaming transducer,

J. Qiu, L. Huang, B. Li, J. Zhang, L. Lu, and Z. Ma, “Improving large-scale deep biasing with phoneme features and text-only data in streaming transducer,” inProc. ASRU, 2023, pp. 1–8

2023
[39]

Contextualized streaming end-to-end speech recognition with trie-based deep biasing and shallow fusion,

D. Le, M. Jain, G. Keren, S. Kimet al., “Contextualized streaming end-to-end speech recognition with trie-based deep biasing and shallow fusion,” inProc. Interspeech, 2021, pp. 1772–1776

2021
[40]

Contextualized automatic speech recognition with dynamic vo- cabulary,

Y . Sudo, Y . Fukumoto, M. Shakeel, Y . Peng, and S. Watanabe, “Contextualized automatic speech recognition with dynamic vo- cabulary,” inProc. SLT, 2024, pp. 78–85

2024
[41]

Hy- brid ctc/attention architecture for end-to-end speech recognition,

S. Watanabe, T. Hori, S. Kim, J. R. Hershey, and T. Hayashi, “Hy- brid ctc/attention architecture for end-to-end speech recognition,” IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1240–1253, 2017

2017
[42]

Time- synchronous one-pass beam search for parallel online and offline transducers with dynamic block training,

Y . Sudo, M. Shakeel, Y . Peng, and S. Watanabe, “Time- synchronous one-pass beam search for parallel online and offline transducers with dynamic block training,” inProc. Interspeech, 2023, pp. 4479–4483

2023
[43]

Joint beam search integrating ctc, attention, and transducer decoders,

Y . Sudo, M. Shakeel, Y . Fukumoto, B. Yan, J. Shi, Y . Peng, and S. Watanabe, “Joint beam search integrating ctc, attention, and transducer decoders,”IEEE Transactions on Audio, Speech and Language Processing, vol. 33, pp. 598–612, 2025

2025
[44]

E-Branchformer: Branchformer with enhanced merging for speech recognition,

K. Kim, F. Wu, Y . Penget al., “E-Branchformer: Branchformer with enhanced merging for speech recognition,” inProc. SLT, 2023, pp. 84–91

2023
[45]

Attention is all you need,

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” inProc. NeurIPS, 2017, pp. 5998–6008

2017
[46]

Adam: A method for stochastic opti- mization,

D. P. Kingma and J. Ba, “Adam: A method for stochastic opti- mization,” inProc. ICLR, 2015

2015
[47]

ESPnet: End-to-end speech processing toolkit,

S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitobaet al., “ESPnet: End-to-end speech processing toolkit,” inProc. Inter- speech, 2018, pp. 2207–2211

2018

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [34]

Phoneme-aware encoding for prefix-tree-based contextual ASR,

H. Futami, E. Tsunoo, Y . Kashiwagi, H. Ogawa, S. Arora, and S. Watanabe, “Phoneme-aware encoding for prefix-tree-based contextual ASR,” inProc. ICASSP, 2024

work page 2024

[2] [1]

These models are trained on mas- sive datasets, allowing them to generalize well across multiple domains and achieve high recognition accuracy

Introduction Speech foundation models (SFMs) [1–7], such as OpenAI’s Whisper [1] and Open Whisper-Style Speech Models (OWSM) [2], have demonstrated remarkable performance in automatic speech recognition (ASR). These models are trained on mas- sive datasets, allowing them to generalize well across multiple domains and achieve high recognition accuracy. Des...

arXiv 2025

[3] [2]

OWSM v3.1 This section provides an overview of OWSM v3.1 [7], which is integrated into the dynamic vocabulary-based CB method [34] in Section 3. 2.1. OWSM encoder OWSM v3.1 adopts the E-Branchformer [38] for the encoder. The OWSM encoder consists of stacked E-Branchformer lay- ers, which utilize parallel branches to capture both local and global features....

work page

[4] [3]

all”, “ig

Integration with dynamic vocabulary Figure 1 shows the overall architecture of the proposed method, which integrates OWSM v3.1 with the dynamic vocabulary- based CB method [34]. Since this method does not change the core structures of the encoder and decoder, only the bias- ing modules (Figure 2) are added to OWSM v3.1 while freezing the pre-trained param...

work page

[5] [4]

We train the embed- ding and output layers of OWSM with vocabulary sizeKof 5,000

Figure 2 illustrates the trainable components in the biasing modules, where red and gray blocks represent the trainable and non-trainable components, respectively. We train the embed- ding and output layers of OWSM with vocabulary sizeKof 5,000. The trainable modules primarily consist of lightweight components, such as linear and embedding layers. Althoug...

work page

[6] [5]

the” and “and

Experiment We conduct several experiments to verify the effectiveness of the proposed method. 4.1. Experimental setup Table 1 shows the model configurations of the proposed method, which consists of OWSM v3.1 and the biasing mod- ules. OWSM v3.1 employs the E-Branchformer [38] and Trans- former as the encoder and decoder, respectively. We use both the bas...

work page

[7] [6]

Conclusion This paper integrates the dynamic vocabulary-based CB method with OWSM v3.1, freezing the pre-trained parameters to enable effective CB even with a small dataset, while preserving the advantages of SFMs. The proposed method improves the B- WER by 11.6 points, resulting in a 0.9 point improvement in the overall WER while reducing the RTF by 7.5%...

work page

[8] [7]

Ro- bust speech recognition via large-scale weak supervision

R. Alec, K. Jong W., X. T., B. G., M. Christine, and S. Ilya, “Ro- bust speech recognition via large-scale weak supervision.” 2022

work page 2022