REVIEW 5 major objections 5 minor 21 references
Adapting Whisper for Streaming Speech Recognition via Two-Pass Decoding
T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read With adequate fine-tuning data, Whisper can be adapted into a capable streaming ASR model, reaching 17.30% WER on post-2023 earnings calls in real time on CPUs.
desk verdict A credible engineering recipe for streaming Whisper on CPU, with a real gem in the hybrid tokenizer ablation—but the missing positional-embedding detail for long audio is exactly what a reviewer must pin down. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the Unified Two-pass (U2) structure: one encoder feeding two decoders, where a Connectionist Temporal Classification (CTC) decoder trained with causal attention masks generates streaming top-k partial hypotheses through prefix beam search, and the original attention decoder produces the final transcript by rescoring those hypotheses at endpoints. The key adaptation is the hybrid tokenizer: the CTC branch predicts over only the first 8,000 tokens of Whisper's byte-pair-encoding vocabulary (covering digits, letters, and common subwords), while the attention decoder keeps the full token space, with a retokenization step converting CTC output strings into the full space, including Whisper-specific prompt tokens, before rescoring. This division of labor gives the from-scratch CTC branch a smaller, better-covered target space for streaming prediction, while preserving the pretrained decoder's linguistic and formatting knowledge for the final rerank.
What would settle it
Measure the oracle top-k WER on the earnings test set: the best possible WER among the hypotheses presented to the attention decoder for rescoring. If this oracle WER is close to the CTC-only WER, the second pass contributes almost nothing and the streaming branch alone is responsible for accuracy; if a systematic test set of rare financial terms shows that the 8,000-token CTC vocabulary cannot represent the correct words at all, the central claim that fine-tuned Whisper becomes a capable streaming model would fail on exactly the domain it targets.
Extended reading notes
Core claim
The central discovery is that Whisper's pretrained encoder remains usable as a streaming encoder if a from-scratch CTC decoder is trained on top under causal attention masks, with the original decoder kept as a rescoring component rather than being used for streaming generation. The paper demonstrates this by fine-tuning Whisper Medium with a hybrid CTC-attention loss and random chunk-size sampling (0.1–1.0 seconds during training), then running streaming inference with prefix beam search in the CTC branch and a single batched rescoring pass in the attention decoder. On the earnings test set, the fine-tuned streaming model reaches 17.30% WER with 5,800 hours of training data, outperforming the same architecture trained from scratch (20.59%) and beating a pseudo-streaming Whisper baseline at small chunk sizes on both earnings and LibriSpeech test-clean. The paper further shows that a hybrid tokenizer—8,000 tokens for the CTC branch, full vocabulary for the attention decoder—consistently improves WER, with the largest gains in low-resource settings (e.g., 21.09% vs 23.51% at 725 hours) and diminishing returns at full data scale (17.30% vs 17.51% at 5,800 hours).
Load-bearing premise
The load-bearing premise is that the streaming CTC branch keeps the correct transcript inside its top-k candidate list throughout the stream; if the right hypothesis is pruned early, the attention decoder's rescoring cannot revive it, so the whole two-pass advantage rests on the from-scratch CTC branch's hypothesis quality.
Editorial extensions
If this is right
- A pretrained offline ASR model can be repurposed into a streaming production system by fine-tuning, meaning teams with in-domain audio can avoid building a streaming architecture from scratch.
- The hybrid tokenizer provides a data-scaling recipe: when in-domain data is limited, shrink the streaming branch's token space; as more data accumulates, the full token space becomes viable and the advantage of shrinking fades.
- Chunk size and maximum delay act as accuracy-latency dials on a single checkpoint, letting the same deployment trade WER (16.65% at 1500 ms chunks) against computational cost (RTF 0.34 at a 20 s maximum delay).
- Since rescoring improves WER only modestly over the CTC branch alone (e.g., 17.60% to 17.30% at 1000 ms chunks), most of the streaming accuracy is carried by the CTC partial branch, and future work should focus on improving its hypothesis quality.
Reading between the lines
- A testable extension is whether the residual WER gap between U2 Whisper and fine-tuned non-streaming Whisper closes with more than 5,800 hours of in-domain data, or whether the causal mask and reduced CTC vocabulary impose a hard ceiling; the paper's data-scaling table suggests the gap is shrinking but does not establish its limit.
- The formatting errors at small chunk sizes (e.g., '$1.3 million' becoming '1.3 million dollars') point to a possible architectural variant the paper does not explore: a delayed-streaming right-context window that lets the CTC branch see a few hundred milliseconds of future audio, which could recover formatting without giving up full streaming.
- The paper compares against a pseudo-streaming Whisper baseline that wins at large chunk sizes, suggesting a practical decision rule: for applications with >1.5-second acceptable delay and no fine-tuning budget, the offline baseline may be preferable; U2's advantage is specifically in low-latency partial transcript regimes.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper adapts OpenAI Whisper for streaming ASR by converting it into a Unified Two-pass (U2) structure: a causally-masked encoder with a lightweight CTC decoder produces streaming partial hypotheses via prefix beam search, and the original Whisper attention decoder rescors the top-k partial hypotheses at endpoint detection. A hybrid tokenizer is proposed in which the CTC branch operates on only the first 8,000 Whisper tokens, while the attention decoder keeps the full token set. The authors fine-tune Whisper Medium on 725–5,800 hours of internal earnings-call data and evaluate on a held-out post-2023 earnings test set and LibriSpeech, reporting WER, RTF, and latency. The main claims are that, with sufficient in-domain fine-tuning data, Whisper can become a capable streaming ASR model running in real time on CPUs, and that the hybrid tokenizer improves generalization, particularly in low-resource settings.
Significance. If the results hold, this is a practically valuable recipe for turning a large pretrained ASR encoder-decoder into a streaming model without losing much accuracy, and the hybrid-tokenizer idea is a useful contribution for domain-specific fine-tuning. The time-based train/test split (test calls after 2023) is methodologically sound, and the open-source WeNet implementation improves reproducibility. The reported CPU RTF below 1 for a 769M-parameter model is notable. However, the manuscript currently lacks several load-bearing details: it does not explain how Whisper's 1500-position absolute positional embeddings are extended or reset for long-form streaming, it provides no error bars or significance tests for the key comparisons, and it omits the loss weight and training hyperparameters needed to reproduce the central experiments.
major comments (5)
- [§2.2, §3] The manuscript does not state how Whisper's learned absolute positional embeddings (1500 positions, corresponding to 30 seconds) are handled when streaming audio longer than 30 seconds. The fine-tuning clips are 5–20 seconds long, so the model never sees positions beyond 1500 during training, yet the earnings test set consists of approximately 7-minute calls. If positions are reset per chunk, the KV-cache reuse described in §2.2 becomes ill-defined; if positions continue past 1500, the model extrapolates beyond its pretrained embedding range. This architectural detail is essential to the long-form streaming claim and must be clarified.
- [Table 1, §4.1] The hybrid-tokenizer versus single-tokenizer WER differences are reported without error bars or significance tests. At 5,800 hours the gap is only 0.21% absolute (17.51% vs. 17.30%), which may be within run-to-run variance. Because the paper's advertised contribution includes the claim that the hybrid tokenizer 'consistently performs better,' the authors should provide confidence intervals (e.g., bootstrap over the 83 test calls) or multiple training runs to support this claim.
- [§4.2] The oracle WER of the top-10 CTC prefix-beam hypotheses (or the best WER among the top-6 rescoring candidates) is not reported. Since the two-pass scheme's final accuracy is bounded by whether the correct hypothesis survives the streaming CTC branch, the paper needs to show the oracle WER together with the selected WER. This is particularly important given the authors' own concession in §4.2 that if the correct hypothesis is pruned early during CTC prefix beam search, subsequent rescoring cannot fix it.
- [Equation (1), §4.1] The hybrid loss weight alpha is never reported, and the paper does not state the learning rate, optimizer, batch size, or other training hyperparameters. The final training stage is described only as 'apply a hybrid loss until validation WER stops improving for three consecutive epochs.' Without alpha and a training schedule, the central experiments cannot be reproduced or compared with other fine-tuning recipes.
- [§2.3] The hybrid tokenizer description is ambiguous. The text first says the CTC token space is restricted to the first 8,000 tokens of the Whisper tokenizer, then states that 'we use these 8,000 tokens to form the CTC prediction targets via SentencePiece.' Whisper's tokenizer is a BPE tokenizer; it is unclear whether a new SentencePiece model with 8,000 tokens is trained or existing Whisper BPE subwords are reused. This should be clarified, since the tokenization scheme is central to the proposed method.
minor comments (5)
- [§4.3, Figures 2–4] The quantitative comparisons against UFAL Whisper appear only in figures, without numerical values in the text or tables. Please provide the corresponding numbers in a table or supplementary material so the claimed crossover behavior can be verified.
- [§4.2] The sentence 'The quantized model results in less than 0.3% absolute WER degradation' is not supported by a displayed result; either include a table or remove the claim.
- [§2.1, §4.2] Training chunk sizes are sampled in [0.1, 1.0] seconds, but Table 2 and Figure 2 evaluate chunk sizes of 1.5 and 6 seconds. The paper should comment on extrapolation to out-of-range chunk sizes.
- [§4.3] The statement that UFAL does not achieve real-time speed on CPUs even with 8-bit quantization lacks RTF numbers; please report them.
- [§2.2] The endpoint detection is only described as 'after 0.5 seconds of silence or when the max delay constraint is reached'; please clarify whether this is a VAD or a fixed segmenter, and what happens if max delay is reached mid-word.
Circularity Check
No significant circularity: the central streaming-adaptation claim is supported by held-out evaluation and external LibriSpeech benchmarks; self-citations to U2/WeNet are not load-bearing.
full rationale
The paper's core claim is empirical: fine-tuning Whisper with a U2-style CTC/attention structure yields streaming partial transcripts and competitive WER. This is tested on a held-out post-2023 earnings set (83 samples, 10 hours) and on LibriSpeech test splits after training on pre-2023 earnings data or LibriSpeech's standard training partition, so the evaluation is not forced by construction. The hybrid-tokenizer comparison is an ablation over training-set size rather than a fitted parameter renamed as a prediction. The U2 architecture and WeNet toolkit are cited from prior work by overlapping authors, but the citations are not used as a uniqueness argument or as a substitute for evidence; the implementation is open-source and the paper's own ablations carry the argument. The conceded limitation that early CTC pruning cannot be repaired by rescoring (Section 4.2) is an honest constraint, not circularity. The missing positional-embedding extension for audio beyond 30 seconds is a completeness/correctness risk, not a circular step, because no evaluation result is defined in terms of that mechanism. Overall the derivation chain is self-contained against external benchmarks, so no circular step is exhibited.
Assumptions & free parameters
free parameters (7)
- CTC/attention loss weight alpha
- CTC token vocabulary size =
8,000
- CTC prefix beam size =
10
- Rescoring candidate count =
6
- Maximum delay =
12 s (default)
- Training chunk-size sampling range =
0.1 to 1.0 s
- Quantization bits =
8
assumptions (5)
- domain assumption CTC training with implicit alignment yields accurate streaming partial transcripts.
- domain assumption A causal attention mask during training preserves enough encoder accuracy for streaming inference.
- domain assumption The internal earnings-call dataset is representative of the production target and its post-2023 test set is a fair generalization probe.
- domain assumption Pretrained Whisper parameters provide a good initialization for the fine-tuned streaming model.
- domain assumption No language model or additional text resources are needed for rescoring.
Cite this review
Pith. "Pith review of Adapting Whisper for Streaming Speech Recognition via Two-Pass Decoding." pith.science (2026). https://pith.science/paper/7HDUMZ3J
@misc{pith2026250612154,
author = {Pith},
title = {Pith review of: Adapting Whisper for Streaming Speech Recognition via Two-Pass Decoding},
year = {2026},
howpublished = {\url{https://pith.science/paper/7HDUMZ3J}},
note = {Machine review of arXiv:2506.12154}
}
read the original abstract
OpenAI Whisper is a family of robust Automatic Speech Recognition (ASR) models trained on 680,000 hours of audio. However, its encoder-decoder architecture, trained with a sequence-to-sequence objective, lacks native support for streaming ASR. In this paper, we fine-tune Whisper for streaming ASR using the WeNet toolkit by adopting a Unified Two-pass (U2) structure. We introduce an additional Connectionist Temporal Classification (CTC) decoder trained with causal attention masks to generate streaming partial transcripts, while the original Whisper decoder reranks these partial outputs. Our experiments on LibriSpeech and an earnings call dataset demonstrate that, with adequate fine-tuning data, Whisper can be adapted into a capable streaming ASR model. We also introduce a hybrid tokenizer approach, which uses a smaller token space for the CTC decoder while retaining Whisper's original token space for the attention decoder, resulting in improved data efficiency and generalization.
Figures
Reference graph
Works this paper leans on
-
[1]
Whisper [1], released by OpenAI, exemplifies this trend
Introduction Large-scale training has provided substantial improvements in both accuracy and robustness for speech recognition models. Whisper [1], released by OpenAI, exemplifies this trend. It was trained on 680,000 hours of audio and achieves high per- formance across diverse public benchmarks. However, due to its non-causal design, it inherently lacks...
-
[2]
Adapting Whisper for Streaming Speech Recognition via Two-Pass Decoding
Methods 2.1. Adapting Whisper to a U2 model U2 ASR models [6] aim to provide a unified architecture for both non-streaming and streaming ASR. The U2 model com- prises an encoder, a CTC decoder, and an attention decoder. Figure 1 illustrates how we adapt Whisper using the U2 struc- ture. During training, both the CTC and attention decoders learn to generat...
work page Pith review arXiv 2025
-
[3]
Datasets Although we include experiments on LibriSpeech [15] for com- pleteness and comparison with other approaches, our primary focus is on an internally curated dataset of earnings calls. We also considered Earnings-22 [16], but excluded it due to its lim- ited size, which is insufficient for training required by the ex- periments. The internal earning...
work page 2023
-
[4]
Results To assess how our proposed approach scales with data, we fine- tune the Whisper Medium model using subsets of the earnings call training data. The subsets consist of 725, 1,450, 2,900, and 5,800 hours of audio, respectively, and we evaluate both the single-tokenizer and hybrid-tokenizer approaches. We first train with causal attention masks, focus...
work page 1935
-
[5]
Conclusion We have presented a method for adapting Whisper into a streaming ASR model using the U2 architecture, achieving per- formance comparable to the original Whisper setup. We intro- duced a hybrid tokenizer to improve generalization, especially when fine-tuning is performed with limited data. Our experi- ments highlight the data-scaling behavior of...
-
[6]
Robust speech recognition via large-scale weak su- pervision,
A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak su- pervision,” inProc. of the 40th International Conference on Ma- chine Learning, 2023, pp. 28 492–28 518
work page 2023
-
[7]
Turning whisper into real- time transcription system,
D. Mach ´aˇcek, R. Dabre, and O. Bojar, “Turning whisper into real- time transcription system,” inProc. of the 13th International Joint Conference on Natural Language Processing and the 3rd Confer- ence of the Asia-Pacific Chapter of the Association for Computa- tional Linguistics: System Demonstrations, 2023, pp. 17–24
work page 2023
-
[8]
Simul-whisper: Attention-guided streaming whisper with truncation detection,
H. Wang, G. Hu, G. Lin, W.-Q. Zhang, and J. Li, “Simul-whisper: Attention-guided streaming whisper with truncation detection,” in Proc. INTERSPEECH 2024 – 25 th Annual Conference of the In- ternational Speech Communication Association, 2024, pp. 4483– 4487
work page 2024
Show all 21 references
-
[9]
Con- nectionist temporal classification: labelling unsegmented se- quence data with recurrent neural networks,
A. Graves, S. Fern ´andez, F. Gomez, and J. Schmidhuber, “Con- nectionist temporal classification: labelling unsegmented se- quence data with recurrent neural networks,” inProc. of the 23rd International Conference on Machine Learning, 2006, pp. 369– 376
2006
-
[10]
Sequence transduction with recurrent neural net- works,
A. Graves, “Sequence transduction with recurrent neural net- works,” inProc. ICML Representation Learning Workshop, 2012
2012
-
[11]
Unified streaming and non-streaming two- pass end-to-end model for speech recognition,
B. Zhang, D. Wu, Z. Yao, X. Wang, F. Yu, C. Yang, L. Guo, Y . Hu, L. Xie, and X. Lei, “Unified streaming and non-streaming two- pass end-to-end model for speech recognition,”arXiv preprint arXiv:2012.05481, 2020
2012 arXiv
-
[12]
Wenet: Production oriented streaming and non-streaming end-to-end speech recognition toolkit,
Z. Yao, X. W. Di Wu, B. Zhang, F. Yu, C. Yang, Z. Peng, X. Chen, L. Xie, and X. Lei, “Wenet: Production oriented streaming and non-streaming end-to-end speech recognition toolkit,” inProc. INTERSPEECH 2021 – 22 nd Annual Conference of the Interna- tional Speech Communication A...
2021
-
[13]
Wenet 2.0: More productive end- to-end speech recognition toolkit,
B. Zhang, D. Wu, Z. Peng, X. Song, Z. Yao, H. Lv, L. Xie, C. Yang, F. Pan, and J. Niu, “Wenet 2.0: More productive end- to-end speech recognition toolkit,” inProc. INTERSPEECH 2022 – 23rd Annual Conference of the International Speech Communi- cation Association, Incheon, Korea...
2022
-
[14]
Neural machine translation of rare words with subword units,
R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” inProceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), K. Erk and N. A. Smith, Eds. Berlin, Germany: Association for...
2016
-
[15]
Hy- brid ctc/attention architecture for end-to-end speech recognition,
S. Watanabe, T. Hori, S. Kim, J. R. Hershey, and T. Hayashi, “Hy- brid ctc/attention architecture for end-to-end speech recognition,” IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1240–1253, 2017
2017
-
[16]
A new algorithm for data compression,
P. Gage, “A new algorithm for data compression,”The C Users Journal, vol. 12, no. 2, pp. 23–38, 1994
1994
-
[17]
Language models are unsupervised multitask learn- ers,
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learn- ers,”OpenAI blog, vol. 1, no. 9, 2019
2019
-
[18]
Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,
T. Kudo and J. Richardson, “Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,” 2018. [Online]. Available: https: //arxiv.org/abs/1808.06226
2018 arXiv
-
[19]
PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation,
J. Ansel, E. Yang, H. He, and et al., “PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation,” in29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS ’24)....
2024
-
[20]
Lib- rispeech: an asr corpus based on public domain audio books,
V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: an asr corpus based on public domain audio books,” in2015 IEEE International Conference on Acoustics, Speech and Signal processing (ICASSP). IEEE, 2015, pp. 5206–5210
2015
-
[21]
Earnings-22: A practical benchmark for accents in the wild,
M. Del Rio, P. Ha, Q. McNamara, C. Miller, and S. Chandra, “Earnings-22: A practical benchmark for accents in the wild,” arXiv preprint arXiv:2203.15591, 2022
2022 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.