Pith. sign in

REVIEW 4 major objections 7 minor 39 references

This paper claims a two-phase 'adaptive self-knowledge distillation' yields a Whisper-derived model that runs 5x faster and beats its teacher's word error rate on in-domain benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 11:55 UTC pith:YYKZM5CD

load-bearing objection The method is a reasonable distillation schedule, but the paper's headline result—1.07% WER improvement over the teacher with 5x speedup—does not survive contact with its own held-out numbers. the 4 major comments →

arxiv 2601.19919 v2 pith:YYKZM5CD submitted 2026-01-08 cs.CL cs.AIcs.SD

ASKD-Whisper: Adaptive Self-knowledge Distillation for Efficient and Low-Latency Automatic Speech Recognition

Junseok Lee , Nahun Kim , Sangyong Lee , Chang-Jae Chun This is my paper
classification cs.CL cs.AIcs.SD
keywords knowledge distillationautomatic speech recognitionmodel compressionself-knowledge distillationadaptive curriculumWhisperword error rateinference latency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to fix a known failure of knowledge distillation for speech recognition: students that strictly mimic a large teacher inherit the teacher's blind spots and over-confident errors, hurting generalization. It proposes a two-phase curriculum called ASKD: first warm up with full teacher supervision, then linearly decay the teacher's influence until it drops below a threshold, then switch to self-knowledge distillation where the student learns from its own previous predictions with an increasing weight. Applied to Whisper, this produces FastWhisper-large, which the paper reports runs five times faster than the teacher and averages a 1.07% lower word error rate on five benchmarks—though those benchmarks overlap with training data.

Core claim

On the paper's own terms, the central claim is that a two-stage distillation curriculum—warm-up, then linearly decaying teacher KL-divergence weight, then self-knowledge distillation with gradually increasing soft-label weight—yields a compact speech recognizer that is both faster and more accurate than one trained with static knowledge distillation. Specifically, FastWhisper-large (740M parameters, using a frozen Whisper-large-v3 encoder and a three-layer Transformer decoder) reaches an average word error rate of 6.37% across LibriSpeech test-clean/test-other, TED-LIUM, AMI, and Earnings-22, versus 7.44% for Whisper-large-v3, while running at roughly 132 ms per utterance versus 659 ms, a fa

What carries the argument

The core mechanism is the adaptive weighting of two losses. Adaptive knowledge distillation (AKD) uses L_AKD = α_AKD^e * KL(P_student, P_teacher), where α_AKD starts at 1, stays fixed during a two-epoch warm-up, then decreases linearly with epoch e. When α_AKD crosses the threshold λ=0.5, training switches to self-knowledge distillation (SKD), where the loss is cross-entropy between the student's distribution and a mixture of the one-hot hard label y and the teacher distribution P_T from the previous epoch: L_SKD = CE((1-α_SKD^e)y + α_SKD^e P_T, P_S), with α_SKD growing from 0.8·e/E_t. The switch lets the student first inherit the teacher's rich representations, then progressively rely on it

Load-bearing premise

The claim that FastWhisper-large outperforms its teacher rests on an average over benchmarks that are also used in training; on the two held-out datasets the teacher still has the lower word error rate.

What would settle it

The paper's own held-out results (Table 3) show FastWhisper-large with WER 9.59 on VoxPopuli and 10.2 on GigaSpeech versus 9.54 and 10.0 for Whisper-large-v3; if those sets are accepted as the generalization test, the claim of beating the teacher is already contradicted.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A student trained with ASKD beats its teacher on the average of the five in-domain benchmarks (6.37% vs 7.44% WER) and runs 5x faster.
  • ASKD outperforms both static KD with pseudo-labeling and AKD alone on LibriSpeech and Earnings-22, showing the two-phase design is doing the work.
  • The same parameter count (0.74B) with one-thirteenth the training audio yields 1.23% lower average WER than distil-Whisper-large-v3.
  • On held-out GigaSpeech and VoxPopuli, FastWhisper-large stays within 0.2% WER of the teacher, which the paper interprets as strong generalization rather than outperformance.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 'outperforms the teacher' claim is based on an average over five benchmarks that overlap with training data; on the two truly held-out sets the teacher still has the lower word error rate, so the generalization part of the claim is weaker than the abstract suggests.
  • The paper's conclusion quotes a 0.97% average WER improvement while the abstract and Table 2 imply 1.07%; this numerical inconsistency should be reconciled.
  • Since the encoder is a frozen pre-trained Whisper encoder, the latency gain comes almost entirely from the shallow three-layer decoder; an ablation with a randomly initialized encoder would isolate how much of the accuracy retention is attributable to ASKD versus the strong encoder features.
  • The self-distillation phase uses the previous epoch's teacher distribution as a soft label with an increasing mixing weight, which is a form of temporal ensembling; using an exponential moving average of the student's own predictions across epochs might stabilize the soft labels further, an extension the paper does not test.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes Adaptive Self-Knowledge Distillation (ASKD), a two-stage training curriculum for ASR model compression. In the first stage, the student is trained with a teacher-distillation loss whose weight is linearly decayed after a warm-up period (adaptive KD); once the weight falls below a threshold, training switches to self-knowledge distillation using the student's own previous-epoch predictions as soft targets. This is instantiated as FastWhisper-small and FastWhisper-large, which use frozen Whisper encoders with lightweight custom decoders. The authors train on 1,634 hours of LibriSpeech, TED-LIUM, LJSpeech, Earnings-22, and AMI, and report that FastWhisper-large achieves a 1.07-point lower average WER than Whisper-large-v3 across five evaluation sets, with a 5x inference speedup. They claim this establishes a new state-of-the-art for generalizable model compression.

Significance. The core idea of ASKD—progressively reducing teacher dependence and then injecting self-distillation—is a plausible and reasonably motivated extension of existing KD and self-KD work. The latency result (5x faster than Whisper-large-v3) is concrete and practically useful, and Table 1 shows a consistent in-domain improvement of ASKD over standard KD+PL on the small model. However, the paper ships no code, model, or training hyperparameters, reports no error bars or significance tests, and, most importantly, its central generalization claim is contradicted by the paper's own held-out numbers. If re-scoped as a compression method with near-parity on unseen domains, the contribution would be modest but real; as written, the claimed 'generalizable state-of-the-art' is not supported.

major comments (4)
  1. [§3.4, Table 2 vs. §3.1, Table 3] The headline 1.07-point WER advantage of FastWhisper-large over Whisper-large-v3 is computed over five test sets (LibriSpeech test-clean/test-other, TED-LIUM, AMI, Earnings-22), all of which are listed in §3.1 as training corpora. This is an in-domain fit, not evidence of generalization. The only held-out evaluation, Table 3, shows FastWhisper-large worse than Whisper-large-v3 on both VoxPopuli (9.59 vs. 9.54) and GigaSpeech (10.2 vs. 10.0). The abstract's claim that ASKD 'outperforms its teacher' and establishes 'a new state-of-the-art for generalizable model compression' is therefore unsupported by the data presented. The authors must either supply genuinely held-out benchmarks with confidence intervals or substantially weaken the claim.
  2. [§3.6, Fig. 2, Eq. (2), Algorithm 1] The ablation in Fig. 2 selects the minimum alpha_AKD value of 0.5 using LibriSpeech test-clean, which is the same set included in the headline Table 2 average—this makes the reported improvement partly circular. More seriously, the schedule in Eq. (2) with alpha_initial_AKD=1, Ew=2, Et=10 reaches alpha_AKD=0.2 at e=10, not 0.5, and Algorithm 1 contains no clamping or floor to enforce a 'minimum' alpha_AKD. If 0.5 is meant to be a floor, the implementation is missing; if it is only the value at which the switch to self-KD occurs, then lambda=0.5 already defines that switch. The paper needs to specify the exact schedule actually used.
  3. [§3.3, Table 1] The comparison between ASKD-trained FastWhisper-small and 'distill-Whisper-small' is confounded: the two models have different architectures (FastWhisper-small uses a frozen Whisper-small encoder with a three-layer decoder, while distill-Whisper-small is a standard distilled Whisper variant), different parameter counts, and different training data. Table 1 therefore does not isolate the effect of ASKD. A proper control would train the same FastWhisper-small architecture with standard KD+PL, SKD alone, AKD alone, and ASKD under identical data and hyperparameters.
  4. [§3.5, Table 3] The text states that WER differences of less than 0.2% on unseen datasets 'indicat[e] strong generalization.' However, both differences are negative relative to the teacher (VoxPopuli: +0.05; GigaSpeech: +0.2), so the consistent direction contradicts 'strong generalization.' Without confidence intervals or significance tests, this result should be described as near-parity or slight degradation, not generalization improvement.
minor comments (7)
  1. [Title/Abstract] The arXiv title is 'ASKD-Whisper' while the paper title and Index Terms use 'FastWhisper.' Please align the title and abstract with the model name used throughout.
  2. [Abstract vs. Conclusion] The abstract reports a 1.07% lower WER, while the conclusion reports 0.97%. These numbers should be reconciled.
  3. [Eq. (2) and Algorithm 1] The epoch indexing is ambiguous: e is initialized at 0 in Algorithm 1, but Eq. (2) subtracts Ew from e. Please clarify whether epochs are 0-indexed or 1-indexed.
  4. [Algorithm 1] The temperature tau is listed as an input and used in the AKD phase but not in the SKD phase (line 13 applies softmax without dividing by tau). If tau is intended only for AKD, state so; otherwise correct the pseudocode.
  5. [Training details] No optimizer, learning rate, batch size, or decoding hyperparameters are reported. Since Algorithm 1 includes a learning rate eta but never gives its value, the experiments are not reproducible from the text.
  6. [Table 2] The column headers 'Size/h' and 'Params/B' are unclear. Please define them explicitly (training hours and billions of parameters, presumably).
  7. [Table 3] The relative latency row reports '659 m/s' and '132 m/s'; these should be milliseconds per sample or per utterance, not meters/second. Please fix the units.

Circularity Check

1 steps flagged

No equation-level circularity in the ASKD losses, but the headline 1.07% WER advantage is partly a fitted-input result: the min-αAKD hyperparameter is tuned on LibriSpeech test-clean and that same set is included in the headline average, while the only held-out sets show the teacher is slightly better.

specific steps
  1. fitted input called prediction [Section 3.6 (Fig. 2) and Table 2]
    "To evaluate αAKD, FastWhisper-small was used and WER was measured on the Librispeech test-clean. The results confirmed that dynamically lowering the αAKD value was more effective in improving the self-training capacity of the student model compared to using a fixed αAKD. Furthermore, setting the minimum αAKD to 0.5 resulted in the most effective knowledge distillation performance."

    The minimum αAKD is selected by minimizing WER on LibriSpeech test-clean. That same LibriSpeech test-clean column is then included in the Table 2 five-set average that produces the claimed 1.07% average WER advantage over Whisper-large-v3. Thus the headline advantage is partly fitted to the evaluation set rather than independently predicted. The paper's own held-out sets in Table 3 show FastWhisper-large worse than the teacher on both GigaSpeech and VoxPopuli (9.59 vs 9.54; 10.2 vs 10.0), so the 'generalizable' claim cannot be separated from this in-sample tuning.

full rationale

The ASKD method itself is not circular at the equation level: Eq. (1)-(4) define a decaying-KL distillation loss plus a self-distillation loss with explicit schedules, none of which is defined in terms of the reported WER. There is also no load-bearing self-citation chain; the cited AdaKD and SKD works are standard external references. The only genuine circular/leaky step is the hyperparameter selection: Section 3.6 tunes min αAKD on LibriSpeech test-clean, and Table 2's average includes that same test set, so the central '1.07% lower than teacher' number is partly an in-sample fitted result rather than a clean prediction. Additionally, the rest of Table 2 is drawn from training-domain corpora listed in Section 3.1, making the 'generalizable model compression' headline an in-domain evaluation; the only true held-out benchmark, Table 3, shows the teacher winning on both sets. These are serious evaluation-validity concerns, but they do not make the core distillation derivation circular, so the score is moderate rather than extreme.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 0 invented entities

The method contributes no new mathematical derivation; its behavior is governed by hand-set schedule parameters and a frozen pretrained encoder. The most consequential free parameter (minimum alpha_AKD / switch threshold) was selected on an evaluation set, and the generalization claim assumes in-domain benchmarks are representative of held-out performance.

free parameters (7)
  • initial KD weight alpha_initial_AKD = 1.0
    Set by hand in Section 2.2; controls how much teacher distribution is copied in the warm-up phase.
  • initial self-KD weight alpha_initial_SKD = 0.8
    Set by hand in Section 2.2; caps blend of previous-epoch soft targets at 80%.
  • KD/self-KD switch threshold lambda = 0.5
    Chosen in Section 2.2; no derivation, and it interacts with the alpha_AKD schedule.
  • minimum alpha_AKD = 0.5
    Selected by a WER sweep on LibriSpeech test-clean (Section 3.6, Fig. 2), i.e., tuned on an evaluation benchmark.
  • warm-up epochs E_w = 2
    Hand-set in Algorithm 1; period of constant teacher weight before decay.
  • total epochs E_t = 10
    Hand-set in Algorithm 1; short schedule chosen without stated justification.
  • temperature tau = unspecified
    Appears in Algorithm 1 but no value is reported; changes the softness of KL targets.
axioms (5)
  • domain assumption Frozen Whisper encoder features are sufficient for the target ASR task; only the decoder needs training.
    Section 3.2 freezes theta_E; if the encoder features do not transfer to the target domains, all reported gains fail.
  • domain assumption KL divergence between student and teacher output distributions is a valid supervision signal at every token.
    Eq. (1) uses L_KL without analysis of token-level calibration or alignment.
  • domain assumption Self-distillation from previous-epoch soft labels acts as a regularizer that prevents overfitting.
    Section 2.2 and Eq. (3) rely on [22,23]; no new theoretical support is provided.
  • ad hoc to paper The linear decay and growth schedules in Eqs. (2) and (4) are a reasonable curriculum.
    No derivation is given; effectiveness is only demonstrated by the experiment selected post hoc.
  • ad hoc to paper Benchmarks whose domains appear in the training corpus are acceptable for measuring generalization.
    Tables 1-2 report WER on LibriSpeech, TED-LIUM, Earnings-22, and AMI despite their training counterparts being listed in Section 3.1.

pith-pipeline@v1.3.0-alltime-deepseek · 7477 in / 16340 out tokens · 160886 ms · 2026-08-03T11:55:00.841412+00:00 · methodology

0 comments
read the original abstract

Knowledge distillation (KD) is one of the most effective paradigms for compressing large-scale foundation models into deployable architectures. In the context of Automatic Speech Recognition (ASR), previous studies have predominantly focused on forcing the student model to strictly mimic the predictive distribution of a massive teacher model. However, this static dependency often presents an inherent trade-off: while the student rapidly acquires basic linguistic representations, it simultaneously inherits the teacher's domain-specific blind spots and over-confident hallucinations, leading to a severe decline in out-of-distribution generalization capacity. To effectively mitigate this issue, we propose Adaptive Self-Knowledge Distillation (ASKD), a dynamic curriculum framework. ASKD systematically decays the dependency on the teacher's distribution as training progresses-thereby unlocking the student's independent reasoning capacity-and subsequently employs a self-knowledge distillation phase to act as a structural regularizer. By applying ASKD, we distill the massive Whisper architecture into a compact variant, ASKD-Whisper. In our comprehensive evaluations across diverse acoustic domains, ASKD-Whisper not only achieves a 5x speedup in inference latency but also outperforms its teacher model by yielding a 1.07% lower word error rate (WER). These results demonstrate that ASKD effectively prevents teacher-induced overfitting and establishes a new state-of-the-art for generalizable model compression.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 14 linked inside Pith

  1. [1]

    In addition, with the rapid advancement of hardware, deep learning (DL)-based ASR is gaining more attention [3]

    INTRODUCTION Automatic speech recognition (ASR) is becoming in- creasingly important as its applications expand to vari- ous domains, such as automatic transcription of meetings, human-to-robot communication emergency response cen- ters, and search engines [1, 2]. In addition, with the rapid advancement of hardware, deep learning (DL)-based ASR is gaining...

  2. [2]

    However, KD methods that heavily emphasize model compression fail to ensure gen- eralization capacity

    KD has been extensively studied with a primary fo- cus on model compression. However, KD methods that heavily emphasize model compression fail to ensure gen- eralization capacity. Furthermore, excessive reliance on the teacher model, as observed in pseudo-labeling (PL), can negatively impact the training process. To this end, our study presents a novel ap...

    Pith/arXiv arXiv 2026

  3. [3]

    EXPERIMENT RESUL TS 3.1. Data Description In our experiments, we used five speech recognition datasets for training and evaluation: a total of 1,620 hours of audio-text paired datasets were constructed from 960 hours of LibriSpeech [24], 453 hours of TED-LIUM release3 [25], 24 hours of LJSpeech [26], 105 hours Earnings-22 [27], 78 hours of AMI Meeting Corp...

  4. [4]

    METHODOLOGY The flow-chart of ASKD proposed in this paper is shown in Fig. 1. In the case of ASKD, training is performed in two steps. Firstly, the self-training capacity of the stude nt model is improved through AKD. Secondly, the general- ization capacity of the student model trained with AKD was further improved through SKD. Mel Spectrogram Fig. 1 . Flo...

  5. [5]

    Pushing the limits of semi-supervised learning for automatic speech recognitio n,

    Y u Zhang, James Qin, Daniel S Park, Wei Han, Chung-Cheng C hiu, Ruoming Pang, Quoc V Le, and Y onghui Wu, “Pushing the limits of semi-supervised learning for automatic speech recognitio n,” arXiv preprint arXiv:2010.10504, 2020

    Pith/arXiv arXiv 2010

  6. [6]

    • In our experiments, FastWhisper achieved lower infer- ence latency than the original Whisper model while ensuring robust ASR performance across evaluation datasets

    encoder, FastWhisper effectively extracts audio fea- tures, enabling efficient learning [15] even with limited data. • In our experiments, FastWhisper achieved lower infer- ence latency than the original Whisper model while ensuring robust ASR performance across evaluation datasets

  7. [7]

    The use of ASKD and the Whisper encoder for training resulted in a 0.97% lower WER score compared to the teacher model, Whisper, despite the smaller dataset and fewer parameters

    CONCLUSION In this paper, we propose ASKD to address the low gen- eralization capacity of student model in standard KD, enhancing both self-training and generalization capacity . The use of ASKD and the Whisper encoder for training resulted in a 0.97% lower WER score compared to the teacher model, Whisper, despite the smaller dataset and fewer parameters....

  8. [8]

    Quantifying bias in automatic speech recogn ition,

    Siyuan Feng, Olya Kudina, Bence Mark Halpern, and Odette Scharenborg, “Quantifying bias in automatic speech recogn ition,” arXiv preprint arXiv:2103.15122, 2021

    Pith/arXiv arXiv 2021

  9. [9]

    Far-field au- tomatic speech recognition,

    Reinhold Haeb-Umbach, Jahn Heymann, Lukas Drude, Shinj i Watanabe, Marc Delcroix, and Tomohiro Nakatani, “Far-field au- tomatic speech recognition,” Proc. of the IEEE , vol. 109, no. 2, pp. 124–148, 2021

    2021

  10. [10]

    Almost unsupervised text to speech and automatic spee ch recognition,

    Yi Ren, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Y an Liu, “Almost unsupervised text to speech and automatic spee ch recognition,” in Proc. 36th International Conference on Machine Learning, 09–15 Jun 2019, vol. 97, pp. 5410–5419

    2019

  11. [11]

    wav2vec 2.0: A framework for self-supervised learni ng of speech representations,

    Alexei Baevski, Y uhao Zhou, Abdelrahman Mohamed, and Michael Auli, “wav2vec 2.0: A framework for self-supervised learni ng of speech representations,” in Advances in Neural Information Pro- cessing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Bal- can, and H. Lin, Eds., 2020, vol. 33, pp. 12449–12460

    2020

  12. [12]

    Robust speech recognit ion via large-scale weak supervision,

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Chris - tine McLeavey, and Ilya Sutskever, “Robust speech recognit ion via large-scale weak supervision,” in International conference on ma- chine learning, 2023, pp. 28492–28518

    2023

  13. [13]

    Beyond neural scaling laws: beating power l aw scaling via data pruning,

    Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya G anguli, and Ari Morcos, “Beyond neural scaling laws: beating power l aw scaling via data pruning,” in Advances in Neural Information Pro- cessing Systems, 2022, vol. 35, pp. 19523–19536

    2022

  14. [14]

    A better and faster end-to-end mo del for streaming asr,

    Bo Li, Anmol Gulati, Jiahui Y u, Tara N. Sainath, Chung- Cheng Chiu, Arun Narayanan, Shuo-Yiin Chang, Ruoming Pang, Y anzhang He, James Qin, Wei Han, Qiao Liang, Y u Zhang, Trevor Strohman, and Y onghui Wu, “A better and faster end-to-end mo del for streaming asr,” in Proc. ICASSP. IEEE, 2021, pp. 5634–5638

    2021

  15. [15]

    Shrinkml: End-t o- end asr model compression using reinforcement learning,

    Lukasz Dudziak, Mohamed S Abdelfattah, Ravichander Vip perla, Stefanos Laskaridis, and Nicholas D Lane, “Shrinkml: End-t o- end asr model compression using reinforcement learning,” arXiv preprint arXiv:1907.03540, 2019

    Pith/arXiv arXiv 1907

  16. [16]

    Iterative compression of end-to-end asr model usin g au- toml,

    Abhinav Mehrotra, Lukasz Dudziak, Jinsu Y eo, Y oung-yo on Lee, Ravichander Vipperla, Mohamed S Abdelfattah, Sourav Bhat- tacharya, Samin Ishtiaq, Alberto Gil CP Ramos, SangJeong Le e, et al., “Iterative compression of end-to-end asr model usin g au- toml,” arXiv preprint arXiv:2008.02897, 2020

    Pith/arXiv arXiv 2008

  17. [17]

    Multi-stage progressive compression o f con- former transducer for on-device speech recognition.,

    Jash Rathod, Nauman Dawalatabad, Shatrughan Singh, an d Dhananjaya Gowda, “Multi-stage progressive compression o f con- former transducer for on-device speech recognition.,” in Inter- speech, 2022, pp. 1691–1695

    2022

  18. [18]

    Distilling the knowledge in a neural network,

    Geoffrey Hinton, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015

    Pith/arXiv arXiv 2015

  19. [19]

    Knowledge distillation: A survey,

    Jianping Gou, Baosheng Y u, Stephen J Maybank, and Dache ng Tao, “Knowledge distillation: A survey,” International Journal of Com- puter Vision, vol. 129, no. 6, pp. 1789–1819, 2021

    2021

  20. [20]

    Distil hubert: Speech representation learning by layer-wise distillatio n of hidden- unit bert,

    Heng-Jui Chang, Shu-wen Y ang, and Hung-yi Lee, “Distil hubert: Speech representation learning by layer-wise distillatio n of hidden- unit bert,” in Proc. ICASSP. IEEE, 2022, pp. 7087–7091

    2022

  21. [21]

    Distil- whisper: Robust knowledge distillation via large-scale ps eudo la- belling,

    Sanchit Gandhi, Patrick von Platen, and Alexander M Rus h, “Distil- whisper: Robust knowledge distillation via large-scale ps eudo la- belling,” arXiv preprint arXiv:2311.00430, 2023

    Pith/arXiv arXiv 2023

  22. [22]

    On informatio n and sufficiency,

    Solomon Kullback and Richard A Leibler, “On informatio n and sufficiency,” The annals of mathematical statistics , vol. 22, no. 1, pp. 79–86, 1951

    1951

  23. [23]

    Fithubert: Going thinner and deeper for kno wl- edge distillation of speech self-supervised learning,

    Y eonghyeon Lee, Kangwook Jang, Jahyun Goo, Y oungmoon J ung, and Hoirin Kim, “Fithubert: Going thinner and deeper for kno wl- edge distillation of speech self-supervised learning,” arXiv preprint arXiv:2207.00555, 2022

    Pith/arXiv arXiv 2022

  24. [24]

    Masked token similarity transfer for compressing transfo rmer- based asr models,

    Euntae Choi, Y oushin Lim, Byeong-Y eol Kim, Hyung Y ong K im, Hanbin Lee, Y unkyu Lim, Seung Woo Y u, and Sungjoo Y oo, “Masked token similarity transfer for compressing transfo rmer- based asr models,” in Proc. ICASSP. IEEE, 2023, pp. 1–5

    2023

  25. [25]

    Does knowledge distillation re ally work?,

    Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Al exander A Alemi, and Andrew G Wilson, “Does knowledge distillation re ally work?,” in Advances in Neural Information Processing Systems , 2021, vol. 34, pp. 6906–6919

    2021

  26. [26]

    Know ledge from the dark side: Entropy-reweighted knowledge distilla tion for balanced knowledge transfer,

    Chi-Ping Su, Ching-Hsun Tseng, and Shin-Jye Lee, “Know ledge from the dark side: Entropy-reweighted knowledge distilla tion for balanced knowledge transfer,” arXiv preprint arXiv:2311.13621 , 2023

    Pith/arXiv arXiv 2023

  27. [27]

    Adakd: Dynamic knowledge distillation of asr models using adaptive loss weighting,

    Shreyan Ganguly, Roshan Nayak, Rakshith Rao, Ujan Deb, and Prathosh AP , “Adakd: Dynamic knowledge distillation of asr models using adaptive loss weighting,” arXiv preprint arXiv:2405.08019, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Self-knowledge distillation with progressive refi nement of targets,

    Kyungyul Kim, ByeongMoon Ji, Doyoung Y oon, and Sangheu m Hwang, “Self-knowledge distillation with progressive refi nement of targets,” in Proc. of the IEEE/CVF international conference on computer vision, 2021, pp. 6567–6576

    2021

  29. [29]

    A theore tical anal- ysis of soft-label vs hard-label training in neural network s,

    Saptarshi Mandal, Xiaojun Lin, and R Srikant, “A theore tical anal- ysis of soft-label vs hard-label training in neural network s,” arXiv preprint arXiv:2412.09579, 2024

    Pith/arXiv arXiv 2024

  30. [30]

    Librispeech: an asr corpus based on public domain a udio books,

    V assil Panayotov, Guoguo Chen, Daniel Povey, and Sanje ev Khu- danpur, “Librispeech: an asr corpus based on public domain a udio books,” in Proc. ICASSP. IEEE, 2015, pp. 5206–5210

    2015

  31. [31]

    Ted-lium 3: Twice as much data and corpus repartition for experiments on speaker adap tation,

    Franc ¸ois Hernandez, Vincent Nguyen, Sahar Ghannay, N atalia Tomashenko, and Y annick Esteve, “Ted-lium 3: Twice as much data and corpus repartition for experiments on speaker adap tation,” in Speech and Computer: 20th International Conference, SPECO M 2018, Leipzig, Germany, September 18–22, 2018, Proceeding s 20 . Springer, 2018, pp. 198–208

    2018

  32. [32]

    The lj speech dataset,

    Keith Ito and Linda Johnson, “The lj speech dataset,” https://keithito.com/LJ-Speech-Dataset/ , 2017

    2017

  33. [33]

    Earnings-22: A practical benchmark for ac cents in the wild,

    Miguel Del Rio, Peter Ha, Quinten McNamara, Corey Mille r, and Shipra Chandra, “Earnings-22: A practical benchmark for ac cents in the wild,” arXiv preprint arXiv:2203.15591, 2022

    Pith/arXiv arXiv 2022

  34. [34]

    Unleashing the killer corpus: experie nces in creating the multi-everything ami meeting corpus,

    Jean Carletta, “Unleashing the killer corpus: experie nces in creating the multi-everything ami meeting corpus,” Language Resources and Evaluation, vol. 41, pp. 181–190, 2007

    2007

  35. [35]

    Gigaspeech: An evolving, multi-domai n asr corpus with 10,000 hours of transcribed audio,

    Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei- Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, et al., “Gigaspeech: An evolving, multi-domai n asr corpus with 10,000 hours of transcribed audio,” arXiv preprint arXiv:2106.06909, 2021

    Pith/arXiv arXiv 2021

  36. [36]

    V oxpopuli: A large-scale multilingual spe ech corpus for representation learning, semi-supervised learning and in- terpretation,

    Changhan Wang, Morgane Rivi` ere, Ann Lee, Anne Wu, Chai tanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Em - manuel Dupoux, “V oxpopuli: A large-scale multilingual spe ech corpus for representation learning, semi-supervised learning and in- terpretation,” in ACL 2021-59th Annual Meeting of the Association for Computational Linguistics, 2021

    2021

  37. [37]

    Glu variants improve transformer,

    Noam Shazeer, “Glu variants improve transformer,” arXiv preprint arXiv:2002.05202, 2020

    Pith/arXiv arXiv 2002

  38. [38]

    Nvidia nemo canary model pushes the frontier of speech recognition and translation,

    Krishna Puvvada, Piotr Zelasko, He Huang, (Steve) Olek sii Hrinchuk, Nithin Koluguri, Somshubra Majumdar, Elena Ras- torgueva, Kunal Dhawan, Zhehuai Chen, Vitaly Lavrukhin, Jagadeesh Balam, and Boris G, “Nvidia nemo canary model pushes the frontier of speech recognition and translation, ” https://nvidia.github.io/NeMo/blogs/2024/, 2024

    2024

  39. [39]

    C risper- whisper: Accurate timestamps on verbatim speech transcrip tions,

    Mario Zusag, Laurin Wagner, and Bernhad Thallinger, “C risper- whisper: Accurate timestamps on verbatim speech transcrip tions,” in Interspeech 2024, 2024, pp. 1265–1269

    2024