{"id":"48a8a879-ac5c-4a61-b0cc-6571ce6d80ff","arxiv_id":"2601.19919","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"A distillation method that decays teacher loss then applies self-distillation yields a Whisper-derived ASR model with 5x lower latency and slightly better average WER only on in-domain noisy datasets.","lead":"This paper proposes ASKD, a two-phase distillation recipe: lean on a Whisper teacher first, then switch the student to learning from its own outputs. The resulting FastWhisper runs about 5x faster, but the claim that it beats Whisper's accuracy holds only on benchmarks that overlap its training data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed 1.07% WER advantage over teacher is computed only on training-domain benchmarks; the two held-out sets in Table 3 show FastWhisper-large is slightly worse, so the 'generalizable SOTA' claim is unsupported.","rationale":"The reader's verdict is REJECT because the headline result is an artifact of in-domain evaluation. My independent reading confirms this is the most load-bearing weakness. For the central claim to hold, the student must be at least as good as the teacher on data not used in training; otherwise 'generalizable model compression' is just 'compression that overfits training distribution.' Table 3 is the only evidence that speaks to generalization, and it shows the student losing on both unseen corpora. The absolute gaps (0.05 and 0.20 WER) are plausibly within noise, but that cuts both ways: the in-domain gains (0.1-1.2 points) may also be noise, and no confidence intervals or seeds are reported. The tuning of λ and min αAKD on LibriSpeech test-clean (Fig. 2) means even the in-domain numbers are not a clean evaluation. I therefore do not see a different, more severe flaw; the reader's concern is the load-bearing one. The paper does contribute a plausible curriculum (ASKD) and a large speedup, but the abstract overstates what the evidence supports. A leave-one-domain-out experiment or a held-out-only average would settle the matter; the arithmetic on Table 3 already points to failure. Verdict should remain REJECT.","tokens_in":7758,"tokens_out":7783,"duration_ms":79592,"concrete_test":"Recompute the claimed '1.07% lower WER' using only held-out evaluations: average FastWhisper-large WER on GigaSpeech and VoxPopuli (Table 3) is 9.895, versus Whisper-large-v3's 9.77. If this calculation is confirmed, the teacher is better on unseen domains, so the abstract's generalization claim and 'new SOTA for generalizable compression' are not supported. This single arithmetic check decides whether the headline should be scoped to in-domain benchmarks.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that ASKD-trained FastWhisper-large 'outperforms its teacher' by 1.07% WER and establishes generalizable model compression—rests entirely on Table 2. But every row in Table 2 is measured on Test-clean/Test-other (LibriSpeech), TED-LIUM, AMI, and Earnings-22, and §3.1 lists LibriSpeech, TED-LIUM, Earnings-22, and AMI (plus LJSpeech) as training corpora. Thus the headline 6.37 vs 7.44 average is an in-domain fit, not a generalization result. The paper's own held-out evaluation, Table 3 (GigaSpeech, VoxPopuli), shows FastWhisper-large worse than Whisper-large-v3 on both sets: 9.59 vs 9.54 and 10.2 vs 10.0; averaged, the teacher wins 9.77 vs 9.895. Section 3.5 spins this deficit as 'strong generalization' because the absolute differences are <0.2%, but the direction is consistently negative. With no error bars, significance tests, or released code, and with λ/αAKD tuned on LibriSpeech test-clean (Fig. 2), the 'outperforms teacher / new SOTA for generalizable compression' claim is unsupported. The method may still be a useful compression trick, but the paper's headline requires held-out evidence it does not provide.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Adaptive Self-Knowledge Distillation (ASKD), a two-stage training curriculum for ASR model compression. In the first stage, the student is trained with a teacher-distillation loss whose weight is linearly decayed after a warm-up period (adaptive KD); once the weight falls below a threshold, training switches to self-knowledge distillation using the student's own previous-epoch predictions as soft targets. This is instantiated as FastWhisper-small and FastWhisper-large, which use frozen Whisper encoders with lightweight custom decoders. The authors train on 1,634 hours of LibriSpeech, TED-LIUM, LJSpeech, Earnings-22, and AMI, and report that FastWhisper-large achieves a 1.07-point lower average WER than Whisper-large-v3 across five evaluation sets, with a 5x inference speedup. They claim this establishes a new state-of-the-art for generalizable model compression.","tokens_in":8200,"tokens_out":8596,"duration_ms":93375,"significance":"The core idea of ASKD—progressively reducing teacher dependence and then injecting self-distillation—is a plausible and reasonably motivated extension of existing KD and self-KD work. The latency result (5x faster than Whisper-large-v3) is concrete and practically useful, and Table 1 shows a consistent in-domain improvement of ASKD over standard KD+PL on the small model. However, the paper ships no code, model, or training hyperparameters, reports no error bars or significance tests, and, most importantly, its central generalization claim is contradicted by the paper's own held-out numbers. If re-scoped as a compression method with near-parity on unseen domains, the contribution would be modest but real; as written, the claimed 'generalizable state-of-the-art' is not supported.","major_comments":[{"comment":"The headline 1.07-point WER advantage of FastWhisper-large over Whisper-large-v3 is computed over five test sets (LibriSpeech test-clean/test-other, TED-LIUM, AMI, Earnings-22), all of which are listed in §3.1 as training corpora. This is an in-domain fit, not evidence of generalization. The only held-out evaluation, Table 3, shows FastWhisper-large worse than Whisper-large-v3 on both VoxPopuli (9.59 vs. 9.54) and GigaSpeech (10.2 vs. 10.0). The abstract's claim that ASKD 'outperforms its teacher' and establishes 'a new state-of-the-art for generalizable model compression' is therefore unsupported by the data presented. The authors must either supply genuinely held-out benchmarks with confidence intervals or substantially weaken the claim.","section":"§3.4, Table 2 vs. §3.1, Table 3"},{"comment":"The ablation in Fig. 2 selects the minimum alpha_AKD value of 0.5 using LibriSpeech test-clean, which is the same set included in the headline Table 2 average—this makes the reported improvement partly circular. More seriously, the schedule in Eq. (2) with alpha_initial_AKD=1, Ew=2, Et=10 reaches alpha_AKD=0.2 at e=10, not 0.5, and Algorithm 1 contains no clamping or floor to enforce a 'minimum' alpha_AKD. If 0.5 is meant to be a floor, the implementation is missing; if it is only the value at which the switch to self-KD occurs, then lambda=0.5 already defines that switch. The paper needs to specify the exact schedule actually used.","section":"§3.6, Fig. 2, Eq. (2), Algorithm 1"},{"comment":"The comparison between ASKD-trained FastWhisper-small and 'distill-Whisper-small' is confounded: the two models have different architectures (FastWhisper-small uses a frozen Whisper-small encoder with a three-layer decoder, while distill-Whisper-small is a standard distilled Whisper variant), different parameter counts, and different training data. Table 1 therefore does not isolate the effect of ASKD. A proper control would train the same FastWhisper-small architecture with standard KD+PL, SKD alone, AKD alone, and ASKD under identical data and hyperparameters.","section":"§3.3, Table 1"},{"comment":"The text states that WER differences of less than 0.2% on unseen datasets 'indicat[e] strong generalization.' However, both differences are negative relative to the teacher (VoxPopuli: +0.05; GigaSpeech: +0.2), so the consistent direction contradicts 'strong generalization.' Without confidence intervals or significance tests, this result should be described as near-parity or slight degradation, not generalization improvement.","section":"§3.5, Table 3"}],"minor_comments":[{"comment":"The arXiv title is 'ASKD-Whisper' while the paper title and Index Terms use 'FastWhisper.' Please align the title and abstract with the model name used throughout.","section":"Title/Abstract"},{"comment":"The abstract reports a 1.07% lower WER, while the conclusion reports 0.97%. These numbers should be reconciled.","section":"Abstract vs. Conclusion"},{"comment":"The epoch indexing is ambiguous: e is initialized at 0 in Algorithm 1, but Eq. (2) subtracts Ew from e. Please clarify whether epochs are 0-indexed or 1-indexed.","section":"Eq. (2) and Algorithm 1"},{"comment":"The temperature tau is listed as an input and used in the AKD phase but not in the SKD phase (line 13 applies softmax without dividing by tau). If tau is intended only for AKD, state so; otherwise correct the pseudocode.","section":"Algorithm 1"},{"comment":"No optimizer, learning rate, batch size, or decoding hyperparameters are reported. Since Algorithm 1 includes a learning rate eta but never gives its value, the experiments are not reproducible from the text.","section":"Training details"},{"comment":"The column headers 'Size/h' and 'Params/B' are unclear. Please define them explicitly (training hours and billions of parameters, presumably).","section":"Table 2"},{"comment":"The relative latency row reports '659 m/s' and '132 m/s'; these should be milliseconds per sample or per utterance, not meters/second. Please fix the units.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper's central empirical claim is not supported by its own held-out results, and the evaluation design has a circular element (tuning on a test set that later appears in the headline average). However, the method is not fundamentally broken and could be salvageable if the claims are reframed as a compression technique with near-parity generalization and 5x speedup, and if the schedule/ablation issues are fixed. I therefore recommend major revision rather than outright rejection, but the authors must address the held-out discrepancy and the alpha_AKD schedule inconsistency before this paper is publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: ASKD is a plausible curriculum. You start with standard KD, linearly decay the teacher weight, and once it crosses a threshold you switch to self-distillation with targets from the previous epoch. The components are known, but the specific two-phase schedule applied to a Whisper encoder plus small decoder is not something I've seen in the cited papers. The 5x speedup is credible because the decoder is three layers; no one should dispute the latency part.\n\nThe problem is the accuracy claim. Table 2's average of 6.37 vs 7.44 looks good, but every row comes from a corpus used in training (LibriSpeech, TED-LIUM, AMI, Earnings-22). FastWhisper-large is actually worse than Whisper-large-v3 on Test-clean (2.34 vs 2.01) and Test-other (4.52 vs 3.91). The only genuinely unseen sets are in Table 3: VoxPopuli and GigaSpeech, where FastWhisper-large is worse on both, 9.59 vs 9.54 and 10.2 vs 10.0. So the \"outperforms the teacher\" claim is an artifact of averaging in-domain benchmarks, and the \"strong generalization\" sentence in Section 3.5 spins a consistent negative direction as if it were neutral. On top of that, the minimum alpha was tuned on LibriSpeech test-clean (Fig. 2), and that same test-clean appears in Table 2. No error bars, no significance tests, no code. The abstract's 1.07% also disagrees with the conclusion's 0.97%, which is minor but sloppy.\n\nI don't think the method is bad. It follows from existing work on adaptive KD and self-distillation, and Table 1 shows it beating a standard KD+PL baseline on FastWhisper-small. But the contribution here is incremental, and the evaluation overclaims. If the authors re-ran Table 2 on unseen domains only, or presented the in-domain/out-of-domain split honestly, the paper would be a decent workshop contribution.\n\nI'd send this to peer review because a referee can sort out the claim from the evidence, and the method deserves a fair test. I wouldn't cite it in my own work yet, and I'd want to see code and held-out numbers first.","headline":"The method is a reasonable distillation schedule, but the paper's headline result—1.07% WER improvement over the teacher with 5x speedup—does not survive contact with its own held-out numbers.","tokens_in":8662,"tokens_out":3154,"would_cite":false,"duration_ms":32574,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims a two-phase 'adaptive self-knowledge distillation' yields a Whisper-derived model that runs 5x faster and beats its teacher's word error rate on in-domain benchmarks.","keywords":["knowledge distillation","automatic speech recognition","model compression","self-knowledge distillation","adaptive curriculum","Whisper","word error rate","inference latency"],"falsifier":"The paper's own held-out results (Table 3) show FastWhisper-large with WER 9.59 on VoxPopuli and 10.2 on GigaSpeech versus 9.54 and 10.0 for Whisper-large-v3; if those sets are accepted as the generalization test, the claim of beating the teacher is already contradicted.","tokens_in":7650,"feed_emoji":"⚡","tokens_out":6472,"duration_ms":60441,"temperature":0.7,"pith_summary":"The paper tries to fix a known failure of knowledge distillation for speech recognition: students that strictly mimic a large teacher inherit the teacher's blind spots and over-confident errors, hurting generalization. It proposes a two-phase curriculum called ASKD: first warm up with full teacher supervision, then linearly decay the teacher's influence until it drops below a threshold, then switch to self-knowledge distillation where the student learns from its own previous predictions with an increasing weight. Applied to Whisper, this produces FastWhisper-large, which the paper reports runs five times faster than the teacher and averages a 1.07% lower word error rate on five benchmarks—though those benchmarks overlap with training data.","feed_headline":"5x faster speech model nearly matches Whisper on new audio","feed_subtitle":"Adaptive distillation beats its teacher on familiar benchmarks, holds within 0.2% WER on held-out sets.","key_machinery":"The core mechanism is the adaptive weighting of two losses. Adaptive knowledge distillation (AKD) uses L_AKD = α_AKD^e * KL(P_student, P_teacher), where α_AKD starts at 1, stays fixed during a two-epoch warm-up, then decreases linearly with epoch e. When α_AKD crosses the threshold λ=0.5, training switches to self-knowledge distillation (SKD), where the loss is cross-entropy between the student's distribution and a mixture of the one-hot hard label y and the teacher distribution P_T from the previous epoch: L_SKD = CE((1-α_SKD^e)y + α_SKD^e P_T, P_S), with α_SKD growing from 0.8·e/E_t. The switch lets the student first inherit the teacher's rich representations, then progressively rely on it","core_discovery":"On the paper's own terms, the central claim is that a two-stage distillation curriculum—warm-up, then linearly decaying teacher KL-divergence weight, then self-knowledge distillation with gradually increasing soft-label weight—yields a compact speech recognizer that is both faster and more accurate than one trained with static knowledge distillation. Specifically, FastWhisper-large (740M parameters, using a frozen Whisper-large-v3 encoder and a three-layer Transformer decoder) reaches an average word error rate of 6.37% across LibriSpeech test-clean/test-other, TED-LIUM, AMI, and Earnings-22, versus 7.44% for Whisper-large-v3, while running at roughly 132 ms per utterance versus 659 ms, a fa","pith_inferences":["The 'outperforms the teacher' claim is based on an average over five benchmarks that overlap with training data; on the two truly held-out sets the teacher still has the lower word error rate, so the generalization part of the claim is weaker than the abstract suggests.","The paper's conclusion quotes a 0.97% average WER improvement while the abstract and Table 2 imply 1.07%; this numerical inconsistency should be reconciled.","Since the encoder is a frozen pre-trained Whisper encoder, the latency gain comes almost entirely from the shallow three-layer decoder; an ablation with a randomly initialized encoder would isolate how much of the accuracy retention is attributable to ASKD versus the strong encoder features.","The self-distillation phase uses the previous epoch's teacher distribution as a soft label with an increasing mixing weight, which is a form of temporal ensembling; using an exponential moving average of the student's own predictions across epochs might stabilize the soft labels further, an extension the paper does not test."],"forward_implications":["A student trained with ASKD beats its teacher on the average of the five in-domain benchmarks (6.37% vs 7.44% WER) and runs 5x faster.","ASKD outperforms both static KD with pseudo-labeling and AKD alone on LibriSpeech and Earnings-22, showing the two-phase design is doing the work.","The same parameter count (0.74B) with one-thirteenth the training audio yields 1.23% lower average WER than distil-Whisper-large-v3.","On held-out GigaSpeech and VoxPopuli, FastWhisper-large stays within 0.2% WER of the teacher, which the paper interprets as strong generalization rather than outperformance."],"fun_headline_variants":["Adaptive distillation makes compact Whisper 5x faster, more precise","Self-distillation lets smaller speech model outdo Whisper","5x speedup and lower WER: adaptive distillation beats static KD","Training curriculum shrinks Whisper, improves accuracy 1%","New distillation approach yields faster, better speech recognition"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The claim that FastWhisper-large outperforms its teacher rests on an average over benchmarks that are also used in training; on the two held-out datasets the teacher still has the lower word error rate.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive distillation makes compact Whisper 5x faster, more precise","Self-distillation lets smaller speech model outdo Whisper","5x speedup and lower WER: adaptive distillation beats static KD","Training curriculum shrinks Whisper, improves accuracy 1%","New distillation approach yields faster, better speech recognition"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000711,"raw_usage":{"total_tokens":3065,"prompt_tokens":801,"completion_tokens":2264,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":2178}},"tokens_in":545,"tokens_out":2264,"duration_ms":15728,"temperature":1.0,"reasoning_tokens":2178,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T11:55:00.841412+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"The paper's own held-out results (Table 3) show FastWhisper-large with WER 9.59 on VoxPopuli and 10.2 on GigaSpeech versus 9.54 and 10.0 for Whisper-large-v3; if those sets are accepted as the generalization test, the claim of beating the teacher is already contradicted.","supporting_citations":[],"review_version":1}