{"id":"37d72cf2-463c-4136-890f-5dd33f035836","arxiv_id":"2509.09987","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Whisper's cross-attention heads can be filtered, without any training, to produce word-level timestamps more accurate than prior approaches at tolerances of 20-100 ms.","lead":"The paper shows that Whisper's decoder has attention heads that act as an internal word aligner, and that teacher-forcing it with characters then filtering those heads yields word timestamps accurate at 20-100 ms. This could let developers skip separate alignment models and use Whisper itself for time-aligned transcripts.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported advantage may be inflated: top-10 head count is selected on test data and AMI margins are within noise, so the unsupervised claim is not yet established.","rationale":"The reader's verdict is CONDITIONAL and flags test-set tuning of the head count as one of four weaknesses. My stress-test agrees that this is the most load-bearing issue: the paper's headline depends on F1 numbers that are selected on the test data, and the margin on AMI is too small to interpret without error bars. The reader's weakest_assumption focused on the ℓ2 heuristic's ability to select alignment heads; that is related, but I view the test-set model selection and lack of confidence intervals as the more direct threat to the central claim. The concrete test I propose would resolve whether the reported advantage is an artifact of tuning or a genuine property of the method. The verdict remains CONDITIONAL because these issues are addressable with a validation split and error bars, not fatal to the underlying idea.","tokens_in":10838,"tokens_out":7745,"duration_ms":99263,"concrete_test":"Split each dataset into a validation set and a test set (e.g., use TIMIT core test, LibriSpeech dev-clean for validation and test-clean for test, and a held-out subset of AMI eval). Select H on the validation set by maximizing F1 over {1,5,10,15,20}; then freeze that H and report F1 on the test set for the proposed method and all baselines, with bootstrap 95% confidence intervals across utterances. Also report the result with a fixed, pre-specified H=10. If the validation-chosen H differs from 10, or if the test-set AMI margin over CrisperWhisper/WhisperX falls inside the confidence interval, the central claim of consistent superiority is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—that the proposed training-free method beats prior aligners at 20–100 ms tolerance—rests on F1 scores in Tables II–IV. These scores all use the top-10 attention heads (H=10). The paper does not explain how H=10 was chosen: Table V searches H∈{1,5,10,20,all} on TIMIT and AMI test sets, and §IV-D explicitly sweeps H∈{5,10,15,20} on test data for Canary-1B. Choosing H by test-set F1 is a form of supervision on the evaluation labels, so the reported numbers are the result of model selection, not a purely unsupervised pipeline. This matters because the method's only tunable parameter is H; a different H would likely give lower F1. Furthermore, the advantage over strong baselines is tiny on AMI (65.3 vs. 64.9 for CrisperWhisper in Table IV, and 61.9 vs. 63.5 for WhisperX in Table III without CrisperWhisper transcripts) and no confidence intervals are reported. If the H=10 choice is not stable on a held-out validation set, or if the AMI margins are within utterance-level noise, the headline 'more accurate than prior work under a stricter tolerance' may not reproduce. The heuristic's inability to identify the oracle head as the top-1 (0.0% in Table V) compounds this: the method depends on averaging many heads, so the exact choice of H is especially load-bearing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether Whisper's decoder cross-attention heads implicitly encode word-level alignments. It identifies 'oracle' heads using ground-truth boundaries, showing that they produce alignments close to MFA. It then proposes an unsupervised head-filtering criterion based on the sum of row and column ℓ2 norms (§II-B), and combines it with teacher forcing Whisper with characters instead of wordpieces (§II-A), followed by DTW post-processing. Experiments on TIMIT, LibriSpeech, and AMI report F1 scores at 20–100 ms tolerance, comparing against Whisper's default heads, WhisperX, CrisperWhisper, a gradient-norm baseline, and MFA. The paper claims that this training-free method outperforms prior work under stricter tolerances, and that the findings transfer to Whisper large-v2 and Canary-1B.","tokens_in":11169,"tokens_out":5580,"duration_ms":56302,"significance":"If validated, the central finding—that Whisper contains attention heads that serve as accurate word aligners and can be selected without training—would be practically valuable, enabling training-free word timestamps from a widely used ASR model. The oracle-head analysis and the character-teacher-forcing technique are useful contributions. However, the current evidence is weakened by test-set model selection of the number of heads H, an internal inconsistency between Tables II and III, and small or negative margins on AMI. The 'characters are better' claim also does not hold for Canary-1B oracle heads. The core idea is promising, but the evaluation needs to be made rigorous before the headline claims can be accepted.","major_comments":[{"comment":"The number of retained heads H is selected on test data. §IV-D states: 'we sweep across {5, 10, 15, 20} heads to select and report the best result.' Table V similarly searches H∈{1,5,10,20,all} on TIMIT and AMI. Since H is the only tunable parameter and F1 varies substantially (TIMIT 80.7→75.0 from top-10 to top-20; AMI 61.9→58.6), the reported F1 values reflect test-set model selection, not a purely unsupervised pipeline. Please choose H on a validation split or provide evidence that results are stable across reasonable H.","section":"§IV-D, Table V"},{"comment":"There is an internal inconsistency for the same configuration. Table II lists 'Whisper oracle character' on AMI as 75.2 (50 ms) and 83.1 (100 ms). Table III lists 'Whisper oracle heads character' on AMI as 75.2 (50 ms) and 79.8 (100 ms). These should be identical. The discrepancy in the 100 ms value suggests a data-processing error or a typo in at least one table. This undermines confidence in the numerical values.","section":"Table II vs Table III"},{"comment":"The abstract and §IV-C1 claim that 'using characters produces finer and more accurate alignments than using wordpieces.' This is contradicted by the Canary-1B oracle row in Table VII: with oracle heads, wordpieces give 64.8 while characters give 63.8. The paper's statement in §IV-D that the findings 'transfer to Canary-1B' is therefore not fully correct. The claim should be qualified to Whisper models or an explanation should be provided for the reversed result.","section":"§IV-C1, Table VII"},{"comment":"No confidence intervals, error bars, or significance tests are reported, and the claimed advantage over prior work is not consistent across datasets. On AMI at 50 ms, WhisperX achieves 63.5 vs. the proposed 61.9 (Table III), and the margin over CrisperWhisper is only 65.3 vs. 64.9 (Table IV). Given typical utterance-level variability, these differences may be within noise. The headline 'more accurate than prior work under a stricter tolerance' should be restricted to the datasets and tolerances where the margin is meaningful, or supported with statistical measures.","section":"§III-C, Tables III–IV"},{"comment":"The paper's own Table V shows that the oracle head is never the top-1 head (0.0%) and is contained in the top-10 heads in only 77.1% (TIMIT) and 82.5% (AMI) of utterances. Thus the method's success relies on averaging many heads, not on reliably selecting the oracle head. This makes the choice of H particularly load-bearing and amplifies the test-set selection concern in the first comment. The authors should analyze sensitivity to H more thoroughly, e.g., via validation curves or aggregated results over H ranges.","section":"§IV-C2, Table V"}],"minor_comments":[{"comment":"Equation (2) begins with ', a'k = Dec(...)', which appears to be a typo; the dummy variable is not explicitly defined, and the notation y'_{0:k-1} should be clarified.","section":"§II-A, Eq. (2)"},{"comment":"The evaluation protocol of Strgar and Harwath [43] is referenced but not described. For reproducibility, please specify how true positives are matched, how word identities are handled, and whether multiple matches are allowed.","section":"§III-C"},{"comment":"The baselines 'Whisper averaging upper half layers' and 'Whisper fixed heads (commit dd985ac)' are not fully specified. Please state which layers are averaged and which fixed heads are used, or provide the exact commit/config.","section":"§IV-B"},{"comment":"The legend uses 'Whisper (averaged)' and 'Whisper (fixed)', which do not match the row labels in Table II. Align the terminology for clarity.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The central finding is interesting, but the evaluation needs tightening. The test-set sweep of H and the Table II/III inconsistency are issues that can probably be fixed with additional experiments and corrections. The 'characters are better' claim should also be reconciled with the Canary-1B oracle result. I recommend major revision rather than rejection, as the core 'Whisper has an internal word aligner' finding is plausible and valuable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's the short version: the paper's central discovery is real and worth knowing about. Whisper's decoder cross-attention contains heads that track word boundaries surprisingly well, and re-running the decoder with character-level teacher forcing makes those maps finer and more monotonic. That is a genuine empirical finding, and the norm-based head filter (row + column ℓ2) is a simple, reasonable proxy for \"alignment-like\" that beats entropy and coverage in their Table VI. The comparisons are reproduced against WhisperX and CrisperWhisper, code is released, and the oracle-head analysis is honest: they show that the oracle head is essentially never the top-1 filtered head, and the method works by averaging over the top-10.\n\nWhere the paper is soft: the headline F1 numbers all use H=10, and H is selected on the test set. Table V sweeps H on TIMIT/AMI; §IV-D openly sweeps {5,10,15,20} on test data for Canary-1B and reports the best. That makes the reported numbers a model-selection artifact, not a purely unsupervised pipeline. The effect size is also thin on AMI (65.3 vs 64.9 vs CrisperWhisper) with no error bars, so that comparison is uninterpretable. The claim that characters always help is contradicted by their own Table VII for Canary-1B oracle heads (63.8 vs 64.8). Minor: citation [40] doesn't match the claim about TIMIT training-set transcriptions; that looks like a wrong reference. None of this sinks the main finding — the oracle analysis and the qualitative figures show the attention maps really do encode alignments — but the paper should report validation-set choice of H, confidence intervals or at least per-utterance variance, and soften the \"always better\" character claim.\n\nWho's it for: anyone extracting word timestamps from Whisper, and people studying what attention heads encode in seq2seq ASR. It deserves a serious referee; the weaknesses are fixable with extra experiments and honest reporting. I'd engage with it.","headline":"A genuinely useful finding — Whisper's decoder has alignment-like attention heads and character teacher forcing sharpens them — but the headline numbers are tuned on test data, so the margin over baselines is not yet nailed down.","tokens_in":11668,"tokens_out":1848,"would_cite":true,"duration_ms":20045,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that Whisper's decoder contains attention heads that align words to audio, and that a training-free, character-based head filter extracts timestamps more accurately than prior training-based approaches at strict tolerances.","keywords":["word alignment","attention heads","Whisper","teacher forcing","character-level tokenization","unsupervised alignment","dynamic time warping","speech recognition"],"falsifier":"On a held-out corpus with manual word boundaries, compute the F1 of the top-10 norm-filtered character heads; if it falls to the level of simply averaging all heads, the filtering claim fails. More sharply, if the correlation between the norm score and per-head F1 disappears on utterances with multiple plausible alignments, the heuristic is not discovering alignment heads but exploiting a confound.","tokens_in":10699,"feed_emoji":"🎙️","tokens_out":5726,"duration_ms":60992,"temperature":0.7,"pith_summary":"Whisper was trained to transcribe speech, not to timestamp it, yet the paper finds that some of its decoder attention heads produce word alignments that are almost as accurate as those of a dedicated forced aligner. The authors call these oracle heads, show they are visually and quantitatively distinct from other heads, and identify them per utterance without any fixed architectural position. They also show that teacher-forcing Whisper with characters instead of wordpieces produces finer, more monotonic attention maps. Based on those findings, they propose a fully unsupervised pipeline—score each head by the sum of its row and column ℓ2 norms, keep the top ten, average, and run dynamic time warping—that outperforms previous training-based aligners at 20–100 ms tolerances. If true, accurate word timestamps can be obtained from Whisper with no extra training and no external aligner.","feed_headline":"Whisper's attention heads already align words to audio","feed_subtitle":"A character-level teacher-forcing trick plus head filtering beats prior aligners at 20–100 ms tolerances.","key_machinery":"The method rests on cross-attention maps from Whisper's decoder, teacher forcing with character-level tokenizations, and the filtering score s(A) = sum of row-wise ℓ2 norms plus column-wise ℓ2 norms (which acts like a Rényi-2 entropy concentration measure on both axes). The selected maps are averaged, column-normalized, and fed to dynamic time warping (DTW) to derive word boundaries.","core_discovery":"The central claim is that, even though Whisper is not trained to produce timestamps, some of its cross-attention heads do encode word-to-audio alignment. The authors call the single best head for an utterance the oracle head and show that oracle heads are close to a standard forced aligner in F1 at 50 ms. They then show that replacing wordpieces with characters under teacher forcing makes the attention maps finer and more monotonic, and that a simple unsupervised score—the sum of row and column ℓ2 norms of the attention map—picks heads that are alignment-like often enough. Keeping the top ten such heads, averaging them, and running DTW yields word timestamps that beat prior training-based ap","pith_inferences":["A possible follow-up is a per-utterance head selector that learns to predict which heads will be oracle-quality, since the paper's own results show the top-1 norm head essentially never matches the oracle; closing that gap would approach the oracle ceiling.","Character-level teacher forcing may generalize beyond Whisper: any attention-based encoder-decoder with a coarse subword vocabulary could inherit finer alignment maps by decoding with characters during inference.","The oracle heads' distribution shifts when switching from wordpieces to characters, suggesting alignment information is not tied to a fixed architectural location but emerges from the interaction of token granularity and attention dynamics; probing with phonetic labels could test whether these heads track acoustic boundaries directly.","Because the filtering score is unsupervised, it could be applied without ground-truth alignments to any sequence-to-sequence ASR decoder; reimplementing it on a non-Whisper model would test whether the phenomenon is Whisper-specific."],"forward_implications":["Whisper itself can serve as a word aligner, so an external alignment model is unnecessary for timestamped transcription.","Characters are a better token granularity than wordpieces for attention-based alignment, even in a model not trained on characters.","Per-utterance dynamic head selection outperforms fixed head sets, so alignment heads are not uniformly distributed in the architecture.","The method transfers to larger Whisper variants and other encoder-decoder ASR models, with encoder frame rate setting a resolution limit.","Strict tolerances (below 100 ms) are the regime where these differences appear; loose 200 ms tolerances hide the advantage."],"fun_headline_variants":["Whisper's hidden heads align words without training","Unsupervised trick extracts word timestamps from Whisper","Whisper's cross-attention already does word alignment","Char forcing + head filtering: Whisper’s free aligner"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The claim rests on the norm score ranking alignment-like heads highly enough that averaging the top ten makes up for the fact that the single best-scoring head is almost never the true oracle head, and on character teacher forcing producing attention maps that remain faithful to alignment even though Whisper was not trained on characters.","fun_headline_variants_meta":{"raw":{"variants":["Whisper's hidden heads align words without training","Unsupervised trick extracts word timestamps from Whisper","Whisper's cross-attention already does word alignment","Char forcing + head filtering: Whisper’s free aligner"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0003,"raw_usage":{"total_tokens":1530,"prompt_tokens":664,"completion_tokens":866,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":408,"completion_tokens_details":{"reasoning_tokens":798}},"tokens_in":408,"tokens_out":866,"duration_ms":9492,"temperature":1.0,"reasoning_tokens":798,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T18:20:46.740784+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out corpus with manual word boundaries, compute the F1 of the top-10 norm-filtered character heads; if it falls to the level of simply averaging all heads, the filtering claim fails. More sharply, if the correlation between the norm score and per-head F1 disappears on utterances with multiple plausible alignments, the heuristic is not discovering alignment heads but exploiting a confound.","supporting_citations":[],"review_version":1}