A two-stage replay-based post-training method corrects ASR timestamp drift across non-speech gaps while preserving recognition far better than ordinary timestamp fine-tuning.
Distil-whisper: Robust knowledge distillation via large-scale pseudo labelling
8 Pith papers cite this work. Polarity classification is still indexing.
8
Pith papers citing it
abstract
As the size of pre-trained speech recognition models increases, running these large models in low-latency or resource-constrained environments becomes challenging. In this work, we leverage pseudo-labelling to assemble a large-scale open-source dataset which we use to distill the Whisper model into a smaller variant, called Distil-Whisper. Using a simple word error rate (WER) heuristic, we select only the highest quality pseudo-labels for training. The distilled model is 5.8 times faster with 51% fewer parameters, while performing to within 1% WER on out-of-distribution test data in a zero-shot transfer setting. Distil-Whisper maintains the robustness of the Whisper model to difficult acoustic conditions, while being less prone to hallucination errors on long-form audio. Distil-Whisper is designed to be paired with Whisper for speculative decoding, yielding a 2 times speed-up while mathematically ensuring the same outputs as the original model. To facilitate further research in this domain, we make our training code, inference code and models publicly accessible.
citation-role summary
method 1
citation-polarity summary
years
2026 8roles
method 1polarities
use method 1representative citing papers
Murmur matches single-pass long-context ASR accuracy on AMI-IHM while cutting latency 4.2x by tuning chunk size and using intra-chunk attention sparsity via KV eviction.
A 241M multi-task student trained with suffix identity, VAR loss, and a decoupled Q2L head matches or beats most VLMs and safety APIs on grounded sensitive scene graphs at 7.6× lower latency.
CANDLE applies CTC alignment to Arabic character deduplication, achieving 5.37% sentence error rate on clean text and up to 12.8% tokenizer fertility reduction.
Presents a layer- and point-wise projection mapping for manifold-based logit distillation combined with LoRA to enable low-parameter student training with reported WER gains.
On-policy distillation from a Qwen-ASR teacher improves a 0.6B Ark-ASR model over supervised fine-tuning and a same-scale baseline on four of five ASR benchmarks using 100k hours of speech.
Modern ASR models with noisy training and language models correlate better with human WER for speech enhancement evaluation than simpler models, yet their robustness makes them less suitable for purely acoustic assessments.
citing papers explorer
-
REDDIT: Correcting Model-Generated Timestamp Drift in ASR without Forgetting via Replay-Based Distribution Editing
A two-stage replay-based post-training method corrects ASR timestamp drift across non-speech gaps while preserving recognition far better than ordinary timestamp fine-tuning.
-
MURMUR: An Efficient Inference System for Long-Form ASR
Murmur matches single-pass long-context ASR accuracy on AMI-IHM while cutting latency 4.2x by tuning chunk size and using intra-chunk attention sparsity via KV eviction.
-
SenBen: Sensitive Scene Graphs for Explainable Content Moderation
A 241M multi-task student trained with suffix identity, VAR loss, and a decoupled Q2L head matches or beats most VLMs and safety APIs on grounded sensitive scene graphs at 7.6× lower latency.
-
CANDLE: CTC-based Arabic Noisy-character Deduplication using a Lightweight Encoder
CANDLE applies CTC alignment to Arabic character deduplication, achieving 5.37% sentence error rate on clean text and up to 12.8% tokenizer fertility reduction.
-
Logit Distillation on Manifolds: Mapping by Learning
Presents a layer- and point-wise projection mapping for manifold-based logit distillation combined with LoRA to enable low-parameter student training with reported WER gains.
-
Data-Efficient On-Policy Distillation for Automatic Speech Recognition
On-policy distillation from a Qwen-ASR teacher improves a 0.6B Ark-ASR model over supervised fine-tuning and a same-scale baseline on four of five ASR benchmarks using 100k hours of speech.
-
Too Good to Be True: A Study on Modern Automatic Speech Recognition for the Evaluation of Speech Enhancement
Modern ASR models with noisy training and language models correlate better with human WER for speech enhancement evaluation than simpler models, yet their robustness makes them less suitable for purely acoustic assessments.
- Best-of-$N$ TTS Evaluation is Confounded by ASR Family Alignment