Pith. sign in

REVIEW 4 major objections 5 minor 76 references

WhisperFlow: speech foundation models in real time

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read WhisperFlow claims to transcribe live speech on ARM laptops in about half a second, using a trained 0.5-second 'hush word' to stop Whisper cleanly.

desk verdict Clever hush-word design and honest systems engineering; the main gap is untested short-buffer start behavior plus missing variance and released code. read the letter →

arxiv 2412.11272 v2 pith:2PBSSA5G submitted 2024-12-15 cs.SD eess.AS

Rongxiang Wang , Zhiming Xu , Felix Xiaozhu Lin This is my paper · ORCID
classification cs.SDeess.AS
keywords streamingspeechrecognitionWhisperhushwordbeampruningCPU/GPUpipeliningon-deviceinferenceerrorratelatency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

WhisperFlow claims that speech foundation models like Whisper can run streaming speech recognition on ordinary client devices, not just in the cloud, by cutting both the compute per input and the latency of each decoding round. The system rests on three optimizations: a 0.5-second learned audio segment, the "hush word," appended to input to stop the model without hallucination and replace the expensive fixed 30-second padding; a beam-pruning scheme that reuses earlier decoding results to shrink the beam search; and a CPU/GPU pipeline that decodes on the CPU while the GPU encodes the next round. On commodity ARM devices, the paper reports 1.6x-4.7x lower per-word latency, down to 0.5 second, with word error rates close to non-streaming Whisper. If true, this would make live captioning, meeting transcription, and on-device speech understanding practical without a server.

What carries the argument

The load-bearing object is the hush word: a 0.5-second, model-specific audio segment trained with the model frozen so that appending it to a voice input produces the normal transcript followed by the end-of-transcript token, instead of the hallucination that short padding causes. It cuts encoding input length from 30 seconds to roughly the utterance length. Two system mechanisms carry the rest of the speedup. Beam pruning uses the previous round's output as a reference, aligning it by search, matching ordinary tokens while skipping punctuation and timestamps, and keeping beam width 1 when the top token matches; mismatch triggers fallback to the full beam. CPU/GPU pipelining maps encoding, prompt prefill, and DTW timestamping to the GPU and opportunistically runs decoding on the CPU, with thread counts chosen by offline profiling. The paper also relies on LocalAgreement-2, materializing a transcript segment only after two consecutive rounds produce it.

What would settle it

Run Whisper medium with the trained hush word on a held-out corpus of noisy or accented short utterances and compare WER against both 30-second padding and no padding; if the error rate approaches the no-padding baseline or hallucinated text appears, the hush-word mechanism does not generalize as claimed.

Watch

Extended reading notes

Core claim

The paper's central claim is that the main obstacles to streaming Whisper are not fundamental—fixed-length padding, redundant beam search, and idle CPU resources—and each can be removed. A short, trained "hush word" appended to the raw audio lets the model end transcription cleanly, avoiding the 30-second padding that dominates encoding cost. Beam pruning aligns the current decoding round with the previous round's transcript and keeps the beam at width 1 whenever tokens match, falling back to the full beam only on mismatch. Offloading decoding to the CPU, with offline-tuned thread pools, lets encoding and decoding of successive rounds overlap. Together these changes reduce per-word latency by 1.6x-4.7x to as low as 0.5 second, while WER stays within a few percent of the non-streaming "gold" transcript.

Load-bearing premise

The load-bearing premise is that the single 0.5-second hush word, trained on LibriSpeech, will stop hallucination for any input it is appended to; if it fails on noisy audio, strong accents, or very short utterances, the encoding savings and the reported word-error rates both collapse.

Editorial extensions

If this is right

  • On an M2 Max, Whisper medium can maintain per-word latency close to 1 second, while Whisper base on an entry-level MacBook Air runs at about 1.2 seconds per word drawing 7 W total.
  • The hush word reduces encoding GFLOPS about 3x compared to 30-second padding while keeping WER close to default, and it beats zero or white-noise padding of 1-5 seconds on most tested datasets.
  • Beam pruning keeps beam width reduced in 53.8%-68.6% of decoding rounds, bringing average beam size from 5 to 2.26-2.85.
  • Shorter step lengths lower per-word latency but also lower accuracy once the model sees too little audio context; below roughly 0.25 seconds accuracy degrades significantly.
  • The CPU/GPU pipeline is sensitive to thread allocation; a poor allocation can underperform GPU-only execution, and pipeline mode raises power draw even as it cuts latency.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: The hush word's success suggests the same trick could be trained for any encoder-decoder speech model that pads inputs, potentially eliminating fixed-length padding beyond Whisper itself.
  • Inference: Because hush word robustness is only demonstrated on LibriSpeech-trained and tested domains, a natural stress test is noise, accents, and very short utterances; failure there would bound the latency gain to clean speech.
  • Inference: Beam pruning plus hush word could combine with speculative decoding, since the reference-alignment machinery is complementary to draft-model verification.
  • Inference: The 7 W figure suggests that with a hush word and pruned beam, the dominant remaining cost may be model size; a distilled or quantized Whisper could push latency below the perception threshold on weaker devices.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents WhisperFlow, an on-device streaming speech processing system built on top of Whisper.cpp. It combines three optimizations: a learned 0.5-second 'hush word' appended to audio inputs to avoid the 30-second padding that Whisper would otherwise require; a beam-pruning scheme that reuses previous decoding rounds as references after aligning sliding-window audio buffers; and a CPU/GPU pipeline with offline-profiled thread allocation. The system is evaluated on Whisper base/small/medium across Apple M2-series devices and an Orange Pi, using TED-LIUM3 long-form as the main streaming benchmark and LibriSpeech, TED-LIUM3 short-form, and FLEURS for the hush word. The authors report per-word latency reductions of 1.6x-4.7x relative to a re-implemented Whisper-Streaming baseline, WER within a few points of non-streaming Whisper, and roughly 7 W power draw on an entry-level MacBook Air.

Significance. If the claims hold, this is a useful systems contribution: it repurposes an adversarial audio segment into a benign padding replacement, it is the first work I am aware of to map encoder and decoder stages of a speech foundation model to CPU and GPU in a streaming pipeline, and the evaluation covers a broad set of models, platforms, datasets, and ablations. The hush-word idea is falsifiable and could generalize to other encoder-decoder speech models, and the beam-pruning/pipelining design is directly relevant to low-latency on-device ASR. The strengths are the breadth of the ablation study and the inclusion of power traces, which are rare in this literature. The main risks are that the hush word is evaluated only on moderately long utterances and is partly a tuned artifact, and that the empirical comparisons lack variance information, so the quantitative headline numbers should be treated with caution until reproduced.

major comments (4)
  1. [6.1, 4.1.2, 5] The hush word is never evaluated on the short audio buffers that occur at stream start in best-effort mode. Section 6.1 restricts the hush-word benchmark to audio samples of 5-25 seconds, while Section 5 sets the step length to 0.01 s, so early processing rounds begin with near-empty buffers that grow gradually; for a buffer shorter than about 1 second, the fixed 0.5 s hush word is a substantial fraction of the input. Section 4.1.2 itself reports that longer hush words 'introduce unique hallucinations' on short audio, indicating a length-dependent failure mode. If the hush word fails on short or noisy input, both the encoding-GFLOPS savings shown in Figure 10 and the claimed negligible accuracy degradation could be lost, and LocalAgreement-2 could turn repeated errors into confirmed transcript. Please measure start-of-stream behavior on short utterances (e.g., 0.5-3 s) and report both WER and per-word latency for the first rounds.
  2. [4.1.2, Figure 11] The hush word is a fitted vector: it is trained on LibriSpeech audio with the expected transcript as the target, and its 0.5-second length appears to be selected by overall WER. The LibriSpeech test-clean and test-other points in Figure 11 therefore evaluate a tuned artifact on data from the same distribution used for training, which makes the comparison against zero/noise padding on those datasets partly circular. The TED-LIUM3 and FLEURS results are more convincing, but the paper does not report how the length sweep was separated from test data. Please clarify the validation procedure, report the length sweep on a held-out set, and add robustness tests (noise, accents, very short inputs) before claiming a generally applicable 'negligible accuracy degradation'.
  3. [6.2, Figure 8, Abstract] The abstract's 'negligible accuracy degradation' is contradicted by the paper's own results. Figure 8 shows that Ours-pipeline incurs noticeable degradation for the small model, and Section 6.2 states that step lengths below 0.25 s degrade accuracy significantly due to lack of audio context. The claim should be qualified to the operating region actually measured, and the paper should report WER at matched step lengths for Ours, Ours-pipeline, and Whisper-S rather than only selected step-length points. This matters because the headline latency reduction is partly obtained by running at shorter step lengths, which is exactly the regime where accuracy degrades.
  4. [6.1-6.3] Every latency and WER point in the evaluation appears to be a single measurement with no variance, confidence interval, or significance test; the power traces in Figure 9 also represent one run. Given that the headline differences are 1.6x-4.7x in latency and 0.2%-2.4% in WER, and that the paper itself shows sensitivity to thread allocation (Figure 12), repeated trials with error bars or statistical tests are needed to establish that the observed improvements are not due to measurement noise. Please report the number of runs, standard deviations, and the significance of the main end-to-end comparisons.
minor comments (5)
  1. [3.2] The word 'exemplfied' in the LocalAgreement-2 sentence is a typo for 'exemplified'.
  2. [1.3] The paper states 'We will make our code publicly available' but provides no artifact or link; for a systems paper whose main claims are empirical, a release link or an explicit availability statement in the evaluation section would improve reproducibility.
  3. [7, References] References [49] and [50] appear to be the same paper (Controlling Whisper) and should be merged, while the related-work text cites it separately from Muting Whisper, which is confusing.
  4. [4.1.2] Hush word training cost is nontrivial (48-72 hours on A100/A40 for the three models) yet this cost is not mentioned in the resource/energy discussion in Section 6.2; a short sentence acknowledging the one-time training cost would make the deployment trade-off complete.
  5. [6.1] The dataset name 'Fleurs' should be standardized to 'FLEURS' in the body text; the caption of Figure 11 uses 'Fleurs EN' while the text uses 'FLEURS'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: WhisperFlow's latency and accuracy claims are measured end-to-end against external baselines; the trained hush word is evaluated on held-out datasets and the only self-citation is peripheral.

full rationale

WhisperFlow's three optimizations are evaluated by direct end-to-end measurement, not derived from the assumptions that define them. The hush word is a trained parameter optimized on LibriSpeech to reproduce the original transcript; reporting WER on LibriSpeech test-clean/other is a conventional held-out generalization measurement, and the same trained vector is also tested on TED-LIUM3 and FLEURS, which are outside the training distribution. The 0.5-second length selection is a hyperparameter choice; the paper does not state that it was tuned on the reported test sets, and the GFLOPS savings are a direct consequence of input length rather than of transcript matching. Beam pruning and CPU/GPU pipelining are system optimizations whose latency and accuracy effects are measured end-to-end against Whisper-S, an external system reimplemented in C++ for fairness, and against the non-streaming 'gold' Whisper. The only self-citation, [67] (Wang and Lin, MobiCom '24), appears as related-work context for speculative decoding and pilot inference and is not load-bearing for any central claim. The skeptical concern that hush-word behavior on sub-3-second streaming-start buffers is untested is a robustness and coverage limitation, not a circular derivation; the paper itself notes in Section 4.1.2 that longer hush words hallucinate on short audio, which is an acknowledged empirical limitation rather than a hidden assumption. No equation or fitted parameter is exhibited that reduces a claimed prediction to its own input by construction.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The system's gains rest on several hand-tuned or fitted parameters: the learned hush word vector and its length, the step length, and the thread allocation. The background assumptions are empirical: the hush word generalizes, LocalAgreement-2 preserves accuracy, and CPU/GPU decoding speeds are comparable on Apple silicon. These are not proven, only demonstrated on the tested platforms.

free parameters (4)
  • hush word audio segment = 0.5 s (8000 samples) learned vector per model
    Trained on LibriSpeech to make frozen Whisper output the transcript and stop; length selected by WER on evaluation data.
  • hush word length = 0.5 s
    Chosen empirically: longer hush words lower training loss but introduce hallucinations on short audio.
  • step length = 0.01 s on Apple devices (best effort)
    Hand-set to force back-to-back processing and minimize per-word latency; shorter steps degrade accuracy.
  • thread allocation = e.g., C6:G5 on M2 Max (varies by platform/model)
    Selected by offline profiling of all splits on a 4-minute sample to minimize per-word latency.
assumptions (5)
  • domain assumption Whisper models require fixed 30s input; shorter padded inputs cause hallucination.
    Motivates the hush word; based on issue [1] and Table 1 measurements.
  • domain assumption A single hush word trained on LibriSpeech generalizes to other domains and utterances for that model.
    Demonstrated on TED-LIUM3 and FLEURS, but not guaranteed by construction.
  • domain assumption Consecutive sliding-window audio buffers produce sufficiently alignable transcript references for beam pruning.
    Required for beam pruning; paper uses search-based alignment and a fallback on mismatch.
  • domain assumption CPU and GPU have comparable decoding speed on target platforms.
    Table 2 shows 1.1x-1.2x CPU/GPU gap on M2 Pro; acknowledged not to hold on Orange Pi.
  • domain assumption LocalAgreement-2 confirmation over two rounds yields near non-streaming accuracy.
    Used to materialize transcripts; inherited from Whisper-Streaming [33].

how reviews work

0 comments
Cite this review

Pith. "Pith review of WhisperFlow: speech foundation models in real time." pith.science (2026). https://pith.science/paper/2PBSSA5G

@misc{pith2026241211272,
  author       = {Pith},
  title        = {Pith review of: WhisperFlow: speech foundation models in real time},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2PBSSA5G}},
  note         = {Machine review of arXiv:2412.11272}
}
read the original abstract

Speech foundation models, such as OpenAI's Whisper, become the state of the art in speech understanding due to their strong accuracy and generalizability. Yet, their applications are mostly limited to processing pre-recorded speech, whereas processing of streaming speech, in particular doing it efficiently, remains rudimentary. Behind this inefficiency are multiple fundamental reasons: (1) speech foundation models are trained to process long, fixed-length voice inputs (often 30 seconds); (2) encoding each voice input requires encoding as many as 1,500 tokens with tens of transformer layers; (3) decoding each output entails an irregular, complex beam search. As such, streaming speech processing on resource-constrained client devices is more expensive than other AI tasks, e.g., text generation. To this end, we present a novel framework, WhisperFlow, which embodies both model and system optimizations. (1) Hush word as a short, learnable audio segment; appended to a voice input, a hush word gracefully stops the speech model from processing more input without hallucination; (2) Beam pruning, which aligns streaming audio buffers over time and reuses results from earlier decoding rounds, therefore significantly accelerating decoding; and (3) CPU/GPU pipelining, which not only maps to the encoding/decoding stages dynamically, but also tunes to an optimal resource ratio, respecting the encoding/decoding speed that varies across voice inputs, models, and hardware. We test WhisperFlow on commodity ARM platforms with 4-12 CPU cores and 10-30 GPU cores. It reduces per-word latency by 1.6x-4.7x to as low as 0.5 second, while seeing negligible accuracy degradation. On an entry-level MacBook Air, WhisperFlow can keep the per-word latency around 1 second, with the whole device drawing only 7 Watts in total.

Figures

Figures reproduced from arXiv: 2412.11272 by the authors.

Figure 1
Figure 1. WhisperFlow running on Apple M2, which is evalu￾ated in Section 6 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. A comparison of approaches [34, 47] for running a speech foundation model over streaming input computer system continuously transcribes the speech into uttered words in real-time [23, 29, 35]. Such a task, known as streaming speech processing (SSP), is at the core of various applications in￾cluding transcribing daily conversations [71], live-stream transcrip￾tions [52], online meetings [64], medical documentation [8… view at source ↗
Figure 3
Figure 3. The WhisperFlow overview. Online operations (1) Processing rounds. Our system continuously ingests streaming audio into the audio buffer. Periodically, the sys￾tem takes a snapshot of the current audio buffer, invokes the model for inference, and produces output tokens to its transcript buffer. Such periodic inferences are referred to as “processing rounds” as shown in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: The hush word is a learnable audio segment de [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Our beam pruning reduces redundant beam search [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Our CPU/GPU pipelining opportunistically offloads [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Compared to Whisper-S, Ours-pipeline reduces per￾word latency by 1.6×–4.7×, across three models and three devices. Both systems run with best effort to achieve their lowest possible latencies, respectively. to represent the state-of-the-art SSP system without foundatio…
Figure 9
Figure 9. Figure 9: Power traces for processing 4-minute audio on M2 [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: An ablation study of our system, showing that all [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 12
Figure 12. Figure 12: The CPU/GPU pipelining requires careful alloca [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 37 canonical work pages

  1. [1]

    Accessed: 2024-11-3

    Faster streaming support issue 137 of whisper.cpp, 2024. Accessed: 2024-11-3. URL: https://github.com/ggerganov/whisper.cpp/issues/137

    work page 2024

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Did you hear that? adversarial examples against automatic speech recognition

    Moustafa Alzantot, Bharathan Balaji, and Mani Srivastava. Did you hear that? adversarial examples against automatic speech recognition. arXiv preprint arXiv:1801.00554, 2018

    arXiv 2018

  4. [4]

    Apple macbook air tech specs, 2024

    Apple. Apple macbook air tech specs, 2024. Accessed: 2024-10-30. URL: https: //www.apple.com/macbook-air/specs/

    work page 2024

  5. [5]

    Neural machine trans- lation by jointly learning to align and translate

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine trans- lation by jointly learning to align and translate. CoRR, abs/1409.0473, 2014. URL: https://api.semanticscholar.org/CorpusID:11212020

    arXiv 2014

  6. [6]

    J. Baker. The dragon system–an overview. IEEE Transactions on Acoustics, Speech, and Signal Processing, 23(1):24–29, 1975. doi:10.1109/TASSP.1975.1162650

    work page arXiv 1975

  7. [7]

    A mathematical theory of adaptive control processes

    Richard Bellman and Robert Kalaba. A mathematical theory of adaptive control processes. Proceedings of the National Academy of Sciences , 45(8):1288–1290, 1959

    1959

  8. [8]

    Speech recognition for clinical documentation from 1990 to 2018: a systematic review

    Suzanne V Blackley, Jessica Huynh, Liqin Wang, Zfania Korach, and Li Zhou. Speech recognition for clinical documentation from 1990 to 2018: a systematic review. Journal of the american medical informatics association , 26(4):324–338, 2019

    work page 1990

Show all 76 references
  1. [9]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey W...

    1901

  2. [10]

    Audio adversarial examples: Targeted attacks on speech-to-text

    Nicholas Carlini and David Wagner. Audio adversarial examples: Targeted attacks on speech-to-text. In 2018 IEEE security and privacy workshops (SPW) , pages 1–7. IEEE, 2018

    2018

  3. [11]

    https://github.com/corsix/amx, 2022

    Peter Cawley et al. https://github.com/corsix/amx, 2022. Accessed: 2024-10-30. URL: https://github.com/corsix/amx

    2022

  4. [12]

    In 29th USENIX Security Symposium (USENIX Security 20) , pages 2667–2684, 2020

    Yuxuan Chen, Xuejing Yuan, Jiangshan Zhang, Yue Zhao, Shengzhi Zhang, Kai Chen, and XiaoFeng Wang.{Devil’s} whisper: A general approach for physical adversarial attacks against commercial black-box speech recognition devices. In 29th USENIX Security Symposium (USENIX Security ...

    2020

  5. [13]

    Fleurs: Few-shot learning evaluation of universal representations of speech

    Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. Fleurs: Few-shot learning evaluation of universal representations of speech. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 798–805. IEEE, 2023

    2022

  6. [14]

    Automatic recognition of spoken digits

    Ken H Davis, R Biddulph, and Stephen Balashek. Automatic recognition of spoken digits. The Journal of the Acoustical Society of America , 24(6):637–642, 1952

    1952

  7. [15]

    Bert: Pre- training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lang...

    2019

  8. [16]

    Speculative decoding for 2x faster whisper inference, 2023

    Sanchit Gandhi. Speculative decoding for 2x faster whisper inference, 2023. Accessed: 2024-7-21. URL: https://huggingface .co/blog/whisper-speculative- decoding

    2023

  9. [17]

    ggerganov/llama.cpp, 2022

    Georgi Gerganov. ggerganov/llama.cpp, 2022. Accessed: 2024-10-30. URL: https://github.com/ggerganov/llama.cpp

    2022

  10. [18]

    ggerganov/whisper.cpp, 2022

    Georgi Gerganov. ggerganov/whisper.cpp, 2022. Accessed: 2024-10-30. URL: https://github.com/ggerganov/whisper.cpp

    2022

  11. [19]

    Sequence transduction with recurrent neural networks

    Alex Graves. Sequence transduction with recurrent neural networks. arXiv preprint arXiv:1211.3711, 2012

    2012 arXiv

  12. [20]

    Conformer: Convolution-augmented transformer for speech recognition

    Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmented transformer for speech recognition. arXiv preprint arXiv:2005.08100, 2020

    2005 arXiv

  13. [21]

    MLX: Efficient and flexible machine learning on apple silicon, 2023

    Awni Hannun, Jagrit Digani, Angelos Katharopoulos, and Ronan Collobert. MLX: Efficient and flexible machine learning on apple silicon, 2023. URL: https:// 12 github.com/ml-explore

    2023

  14. [22]

    Speech understanding systems: Summary of results of the five-year research effort, 1976

    P Hayes-Roth, M Fox, G Gill, DJ Mostow, and R Reddy. Speech understanding systems: Summary of results of the five-year research effort, 1976

    1976

  15. [23]

    Streaming end-to-end speech recognition for mobile devices

    Yanzhang He, Tara N Sainath, Rohit Prabhavalkar, Ian McGraw, Raziel Alvarez, Ding Zhao, David Rybach, Anjuli Kannan, Yonghui Wu, Ruoming Pang, et al. Streaming end-to-end speech recognition for mobile devices. InICASSP 2019-2019 IEEE International Conference on Acoustics, Spee...

    2019

  16. [24]

    Ted-lium 3: Twice as much data and corpus repartition for experiments on speaker adaptation

    François Hernandez, Vincent Nguyen, Sahar Ghannay, Natalia Tomashenko, and Yannick Esteve. Ted-lium 3: Twice as much data and corpus repartition for experiments on speaker adaptation. In Speech and Computer: 20th International Conference, SPECOM 2018, Leipzig, Germany, Septemb...

    2018

  17. [25]

    Advances in joint ctc-attention based end-to-end speech recognition with a deep cnn encoder and rnn-lm

    Takaaki Hori, Shinji Watanabe, Yu Zhang, and William Chan. Advances in joint ctc-attention based end-to-end speech recognition with a deep cnn encoder and rnn-lm. arXiv preprint arXiv:1706.02737, 2017

    2017 arXiv

  18. [26]

    Swapadvisor: Pushing deep learning beyond the gpu memory limit via smart swapping

    Chien-Chin Huang, Gu Jin, and Jinyang Li. Swapadvisor: Pushing deep learning beyond the gpu memory limit via smart swapping. In Proceedings of the Twenty- Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’20, page ...

    2020

  19. [27]

    Deepum: Tensor migration and prefetching in unified memory

    Jaehoon Jung, Jinpyo Kim, and Jaejin Lee. Deepum: Tensor migration and prefetching in unified memory. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operat- ing Systems, Volume 2, ASPLOS 2023, page 207–221, New Yo...

    2023

  20. [28]

    Speech and language processing, 2000

    Daniel Jurafsky. Speech and language processing, 2000

    2000

  21. [29]

    Large- scale multilingual speech recognition with a streaming end-to-end model

    Anjuli Kannan, Arindrima Datta, Tara N Sainath, Eugene Weinstein, Bhuvana Ramabhadran, Yonghui Wu, Ankur Bapna, Zhifeng Chen, and Seungji Lee. Large- scale multilingual speech recognition with a streaming end-to-end model. arXiv preprint arXiv:1909.05330, 2019

    1909 arXiv

  22. [30]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  23. [31]

    Convolution-augmented parameter-efficient fine-tuning for speech recognition

    Kwangyoun Kim, Suwon Shon, Yi-Te Hsu, Prashant Sridhar, Karen Livescu, and Shinji Watanabe. Convolution-augmented parameter-efficient fine-tuning for speech recognition. In Proc. Interspeech 2024, pages 2830–2834, 2024

    2024

  24. [32]

    Mahoney, Amir Gholami, and Kurt Keutzer

    Sehoon Kim, Karttikeya Mangalam, Suhong Moon, John Canny, Jitendra Malik, Michael W. Mahoney, Amir Gholami, and Kurt Keutzer. Speculative decoding with big little decoder. In Advances in Neural Information Processing Systems . Curran Associates, Inc., 2023. URL: https://arxiv....

    2023 arXiv

  25. [33]

    Low-latency sequence-to- sequence speech recognition and translation by partial hypothesis selection

    Danni Liu, Gerasimos Spanakis, and Jan Niehues. Low-latency sequence-to- sequence speech recognition and translation by partial hypothesis selection. In Interspeech, 2020. URL: https://api.semanticscholar.org/CorpusID:218862974

    2020

  26. [34]

    Turning whisper into real-time transcription system

    Dominik Macháček, Raj Dabre, and Ondřej Bojar. Turning whisper into real-time transcription system. arXiv preprint arXiv:2307.14743, 2023

    2023 arXiv

  27. [35]

    Streaming automatic speech recog- nition with the transformer model

    Niko Moritz, Takaaki Hori, and Jonathan Le. Streaming automatic speech recog- nition with the transformer model. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 6074–6078. IEEE, 2020

    2020

  28. [36]

    Universal adversarial perturbations for speech recognition systems

    Paarth Neekhara, Shehzeen Hussain, Prakhar Pandey, Shlomo Dubnov, Julian McAuley, and Farinaz Koushanfar. Universal adversarial perturbations for speech recognition systems. arXiv preprint arXiv:1905.03828, 2019

    1905 arXiv

  29. [37]

    There is more than one kind of robustness: Fooling whisper with adversarial examples

    Raphael Olivier and Bhiksha Raj. There is more than one kind of robustness: Fooling whisper with adversarial examples. arXiv preprint arXiv:2210.17316, 2022

    2022 arXiv

  30. [38]

    Train- ing language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Train- ing language models to follow instructions with human feedback. Advances in neural information processing systems , 35:2...

    2022

  31. [39]

    Lib- rispeech: an asr corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Lib- rispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP) , pages 5206–5210. IEEE, 2015

    2015

  32. [40]

    Splitwise: Efficient generative llm inference using phase splitting

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pages 118–132...

    2024

  33. [41]

    Branchformer: Parallel mlp-attention architectures to capture local and global context for speech recognition and understanding

    Yifan Peng, Siddharth Dalmia, Ian Lane, and Shinji Watanabe. Branchformer: Parallel mlp-attention architectures to capture local and global context for speech recognition and understanding. In International Conference on Machine Learning , pages 17627–17643. PMLR, 2022

    2022

  34. [42]

    Owsm-ctc: An open encoder-only speech foundation model for speech recognition, translation, and language identification

    Yifan Peng, Yui Sudo, Muhammad Shakeel, and Shinji Watanabe. Owsm-ctc: An open encoder-only speech foundation model for speech recognition, translation, and language identification. arXiv preprint arXiv:2402.12654, 2024

    2024 arXiv

  35. [43]

    Yifan Peng, Jinchuan Tian, William Chen, Siddhant Arora, Brian Yan, Yui Sudo, Muhammad Shakeel, Kwanghee Choi, Jiatong Shi, Xuankai Chang, et al. Owsm v3. 1: Better and faster open whisper-style speech models based on e-branchformer. arXiv preprint arXiv:2401.16658, 2024

    2024 arXiv

  36. [44]

    Reproducing whisper-style training using an open-source toolkit and publicly available data

    Yifan Peng, Jinchuan Tian, Brian Yan, Dan Berrebbi, Xuankai Chang, Xinjian Li, Jiatong Shi, Siddhant Arora, William Chen, Roshan Sharma, et al. Reproducing whisper-style training using an open-source toolkit and publicly available data. In 2023 IEEE Automatic Speech Recognitio...

    2023

  37. [45]

    Speech percep- tion at the interface of neurobiology and linguistics

    David Poeppel, William J Idsardi, and Virginie Van Wassenhove. Speech percep- tion at the interface of neurobiology and linguistics. Philosophical Transactions of the Royal Society B: Biological Sciences , 363(1493):1071–1086, 2008

    2008

  38. [46]

    Improving language understanding by generative pre-training

    Alec Radford. Improving language understanding by generative pre-training. 2018

    2018

  39. [47]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning , pages 28492–28518. PMLR, 2023

    2023

  40. [48]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019

    2019

  41. [50]

    Controlling whisper: Universal acoustic adversarial attacks to control speech foundation models

    Vyas Raina and Mark Gales. Controlling whisper: Universal acoustic adversarial attacks to control speech foundation models. arXiv preprint arXiv:2407.04482 , 2024

    2024 arXiv

  42. [51]

    Muting whisper: A universal acoustic adversarial attack on speech foundation models

    Vyas Raina, Rao Ma, Charles McGhee, Kate Knill, and Mark Gales. Muting whisper: A universal acoustic adversarial attack on speech foundation models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natura...

    2024 doi

  43. [52]

    Paul Robinson, and Bradley S

    Rohit Ranchal, Teresa Taber-Doughty, Yiren Guo, Keith Bain, Heather Martin, J. Paul Robinson, and Bradley S. Duerstock. Using speech recognition for real- time captioning and lecture transcription in the classroom. IEEE Transactions on Learning Technologies, 6(4):299–311, 2013...

    2013 doi

  44. [53]

    Exploring architectures, data and units for streaming end-to-end speech recognition with rnn-transducer

    Kanishka Rao, Haşim Sak, and Rohit Prabhavalkar. Exploring architectures, data and units for streaming end-to-end speech recognition with rnn-transducer. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) , pages 193–199. IEEE, 2017

    2017

  45. [54]

    ZeRO-Offload: De- mocratizing Billion-Scale model training

    Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. ZeRO-Offload: De- mocratizing Billion-Scale model training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21) , pages 551–564. USENIX Associati...

    2021

  46. [55]

    Enhancing the ted-lium corpus with selected data for language modeling and more ted talks

    Anthony Rousseau, Paul Deléglise, Yannick Esteve, et al. Enhancing the ted-lium corpus with selected data for language modeling and more ted talks. In LREC, pages 3935–3939, 2014

    2014

  47. [56]

    Adversarial attacks against automatic speech recognition systems via psychoacoustic hiding

    Lea Schönherr, Katharina Siobhan Kohls, Steffen Zeiler, Thorsten Holz, and Dorothea Kolossa. Adversarial attacks against automatic speech recognition systems via psychoacoustic hiding. ArXiv, abs/1808.05665, 2018. URL: https: //api.semanticscholar.org/CorpusID:52040758

    2018 arXiv

  48. [57]

    Review of speech-to-text recognition technology for enhancing learning

    Rustam Shadiev, Wu-Yuin Hwang, Nian-Shing Chen, and Yueh-Min Huang. Review of speech-to-text recognition technology for enhancing learning. Journal of Educational Technology & Society , 17(4):65–84, 2014

    2014

  49. [58]

    Dissecting user-perceived latency of on-device e2e speech recognition

    Yuan Shangguan, Rohit Prabhavalkar, Hang Su, Jay Mahadeokar, Yangyang Shi, Jiatong Zhou, Chunyang Wu, Duc Le, Ozlem Kalinli, Christian Fuegen, et al. Dissecting user-perceived latency of on-device e2e speech recognition. arXiv preprint arXiv:2104.02207, 2021

    2021 arXiv

  50. [59]

    Powerinfer: Fast large lan- guage model serving with a consumer-grade gpu

    Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. Powerinfer: Fast large lan- guage model serving with a consumer-grade gpu. arXiv preprint arXiv:2312.12456, 2023

    2023 arXiv

  51. [61]

    Instantaneous grammatical error correction with shallow aggressive decoding

    Xin Sun, Tao Ge, Furu Wei, and Houfeng Wang. Instantaneous grammatical error correction with shallow aggressive decoding. arXiv preprint arXiv:2106.04970, 2021

    2021 arXiv

  52. [62]

    Goodfellow, and Rob Fergus

    Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In Yoshua Bengio and Yann LeCun, editors,2nd International Conference on Learning Representations, ICLR 2014, Banff, AB,...

    2014 arXiv

  53. [63]

    Streaming trans- former asr with blockwise synchronous beam search

    Emiru Tsunoo, Yosuke Kashiwagi, and Shinji Watanabe. Streaming trans- former asr with blockwise synchronous beam search. In 2021 IEEE Spo- ken Language Technology Workshop (SLT) , pages 22–29, 2021. doi:10.1109/ SLT48900.2021.9383517

    2021

  54. [64]

    The calo meeting speech recognition and understanding system

    Gokhan Tur, Andreas Stolcke, Lynn Voss, John Dowding, Benoît Favre, Raquel Fernández, Matthew Frampton, Michael Frandsen, Clint Frederickson, Martin 13 Graciarena, et al. The calo meeting speech recognition and understanding system. In 2008 IEEE Spoken Language Technology Work...

    2008

  55. [65]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  56. [66]

    Simul- whisper: Attention-guided streaming whisper with truncation detection

    Haoyu Wang, Guoqiang Hu, Guodong Lin, Wei-Qiang Zhang, and Jian Li. Simul- whisper: Attention-guided streaming whisper with truncation detection. arXiv preprint arXiv:2406.10052, 2024

    2024 arXiv

  57. [67]

    Turbocharge speech understanding with pilot inference

    Rongxiang Wang and Felix Xiaozhu Lin. Turbocharge speech understanding with pilot inference. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking , ACM MobiCom ’24, page 1299–1313, New York, NY, USA, 2024. Association for Computing Mach...

    2024

  58. [68]

    ESPnet: End-to- end speech processing toolkit

    Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai. ESPnet: End-to- end speech processing toolkit. In Proceedings of Inters...

  59. [69]

    Loongserve: Efficiently serving long-context large language models with elastic sequence parallelism

    Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. Loongserve: Efficiently serving long-context large language models with elastic sequence parallelism. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles , SOSP ’24, page 64...

  60. [70]

    Lossless speedup of au- toregressive translation with generalized aggressive decoding

    Heming Xia, Tao Ge, Furu Wei, and Zhifang Sui. Lossless speedup of au- toregressive translation with generalized aggressive decoding. arXiv preprint arXiv:2203.16487, 2022

    2022 arXiv

  61. [71]

    Toward human parity in con- versational speech recognition

    Wayne Xiong, Jasha Droppo, Xuedong Huang, Frank Seide, Michael L Seltzer, Andreas Stolcke, Dong Yu, and Geoffrey Zweig. Toward human parity in con- versational speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 25(12):2410–2423, 2017

    2017

  62. [72]

    Empowering 1000 tokens/second on-device llm prefilling with mllm-npu

    Daliang Xu, Hao Zhang, Liming Yang, Ruiqi Liu, Gang Huang, Mengwei Xu, and Xuanzhe Liu. Empowering 1000 tokens/second on-device llm prefilling with mllm-npu. arXiv preprint arXiv:2407.05858, 2024

    2024 arXiv

  63. [73]

    Inference with reference: Lossless acceleration of large language models

    Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furu Wei. Inference with reference: Lossless acceleration of large language models. arXiv preprint arXiv:2304.04487, 2023

    2023 arXiv

  64. [74]

    Transformer transducer: A streamable speech recog- nition model with transformer encoders and rnn-t loss

    Qian Zhang, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott, Stephen Koo, and Shankar Kumar. Transformer transducer: A streamable speech recog- nition model with transformer encoders and rnn-t loss. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech ...

    2020

  65. [75]

    Black-box adversarial attacks on commercial speech platforms with minimal information

    Baolin Zheng, Peipei Jiang, Qian Wang, Qi Li, Chao Shen, Cong Wang, Yunjie Ge, Qingyang Teng, and Shenyi Zhang. Black-box adversarial attacks on commercial speech platforms with minimal information. In Proceedings of the 2021 ACM SIGSAC conference on computer and communication...

    2021

  66. [76]

    In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , pages 193–210, 2024

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang.{DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24...

    2024

  67. [2018]

    URL: http://dx.doi.org/10.21437/Interspeech.2018-1456, doi:10.21437/ Interspeech.2018-1456

    2018 doi

  68. [2024]

    doi:10.1145/3694715.3695948

    Association for Computing Machinery. doi:10.1145/3694715.3695948

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.