{"id":"210b1c86-a562-425b-8ca6-7314b6695ef1","arxiv_id":"2412.05589","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SQ-Whisper injects trainable speaker queries into Whisper's encoder and decoder, beating prior target-speaker ASR systems with new state-of-the-art WERs on Libri2Mix and WSJ0-2Mix.","lead":"This paper adapts Whisper, a single-speaker speech recognition model, to recognize one chosen speaker's words in overlapping two-person audio. The approach sets new error-rate records on standard two-speaker benchmarks and shows a general way to extend speech foundation models to cocktail-party scenarios.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline SOTA comparisons vs TS-HuBERT may be confounded by Whisper's pretraining overlap with the LibriSpeech/WSJ0 source data; the paper provides no control for this, and the abstract's 15% figure is not the TS-HuBERT comparison.","rationale":"The reader's weakest assumption was enrollment reliability, citing Section VI-F's matched-versus-mismatched experiment. That is a real limitation, but it is inherent to the TS-ASR task definition: target-speaker ASR requires an enrollment and assumes the target is present. The paper discloses the failure mode clearly, so I do not see it as the most load-bearing threat to the central claim. The more consequential threat is that the headline comparisons against TS-HuBERT are not controlled for the very different pretraining of Whisper versus WavLM/HuBERT. Whisper is a supervised ASR foundation model likely trained on the same open corpora from which Libri2Mix and WSJ0-2Mix are built; if so, the reported SOTA numbers may reflect memorized or highly familiar single-speaker transcriptions, not the proposed SQ-Former mechanism. The paper's internal TSE-Whisper baselines (same Whisper backbone) mitigate this, and the SQ-Whisper gains over TSE-Whisper (20.1 vs 28.3 full; 23.2 vs 25.6 LoRA) are strong evidence that the adaptation method itself helps. Still, the abstract specifically claims superiority over TS-HuBERT, and that comparison is not clean. I also note the abstract's 15% relative improvement is not reproducible from the TS-HuBERT row; the correct value is about 19% for the full model. This does not weaken the method, but it is an internal inconsistency that must be fixed. Overall, the empirical work is solid enough for conditional acceptance, with the requested overlap analysis and abstract correction.","tokens_in":16946,"tokens_out":12770,"duration_ms":129317,"concrete_test":"Check Whisper's pretraining overlap with the source utterances: run the frozen medium Whisper model in zero-shot single-speaker ASR on the clean LibriSpeech and WSJ0 utterances that form the target side of Libri2Mix/WSJ0-2Mix Test, and compare WER against a held-out set of the same speakers' utterances not plausibly in Whisper's training data. If WER on the mixture-source utterances is substantially lower, pretraining contamination is confirmed; then re-run the SQ-Whisper versus TSE-Whisper comparison on a TS-ASR test set assembled from a post-2022 corpus outside Whisper's pretraining, and check whether the relative gain over TSE-Whisper persists.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SQ-Whisper beats TS-HuBERT by up to 15% relative WER on Libri2Mix and 10% on WSJ0-2Mix, establishing new SOTA. For this claim to be evidence about the adaptation method, the comparison must be controlled. It is not fully controlled: SQ-Whisper starts from Whisper medium, a 764M-parameter supervised model trained on 680k hours of labeled web audio, while the TS-HuBERT and WavLM baselines are self-supervised models not trained to transcribe LibriSpeech or WSJ0. Libri2Mix is constructed by mixing LibriSpeech utterances, and WSJ0-2Mix from WSJ0; these are public corpora that may well overlap Whisper's pretraining data, LibriSpeech especially. The paper gives no analysis of this overlap, so the reported gains (Table II: 24.8 -> 20.1; Table VI: 6.1 -> 5.5) could partly reflect Whisper's prior exposure to the source utterances or speakers rather than the SQ-Former's target-speaker adaptation. The internal TSE-Whisper comparisons in Tables I-II provide partial control and are reassuring, but the abstract's headline comparison to TS-HuBERT remains confounded. Additionally, the abstract's '15%' relative reduction on Libri2Mix does not match the TS-HuBERT row: from Table II, (24.8-20.1)/24.8 = 19%; the 15% value is the PIT-Transformer comparison (23.5 -> 20.1), a different baseline. This should be corrected in the abstract.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SQ-Whisper, an adaptation of the Whisper speech foundation model to target-speaker ASR (TS-ASR). The method inserts a Speaker-Querying Transformer (SQ-Former) between Whisper's convolutional front-end and transformer encoder; a set of trainable queries cross-attends to the mixture representation conditioned on a target-speaker enrollment, producing speaker prompts that are appended to both encoder and decoder inputs. A speaker contrastive loss is added to make the prompts discriminative. Experiments on Libri2Mix, WSJ0-2Mix, and AMI compare SQ-Whisper with TSE-Whisper, TS-HuBERT, WavLM, and other baselines, and the paper reports state-of-the-art WERs of 14.6% on Libri2Mix and 4.4% on WSJ0-2Mix when using data augmentation. The paper also integrates and ablates four TSE modules and evaluates LoRA-based parameter-efficient fine-tuning.","tokens_in":17381,"tokens_out":7360,"duration_ms":59591,"significance":"If the results are reproducible, the contribution is valuable: it offers a principled way to extend a supervised speech foundation model to overlapped-speech recognition, shows consistent gains over a Whisper-based TSE baseline across three datasets, and releases code. The internal comparisons between TSE-Whisper and SQ-Whisper provide solid evidence for the adaptation method itself. The headline comparisons to TS-HuBERT are partially confounded by Whisper's pretraining data overlap, which tempers the strength of the state-of-the-art claims, but the underlying technical idea is sound and the paper is well structured.","major_comments":[{"comment":"The abstract states that SQ-Whisper yields 'up to 15% relative reduction in WER' compared with TS-HuBERT, but Table II shows TS-HuBERT at 24.8 and SQ-Whisper (Full) at 20.1, which is a 19% relative reduction; the 15% figure in Section VI-B corresponds to the comparison against PIT-Transformer (23.5 to 20.1) and should be attributed to that baseline, not to TS-HuBERT.","section":"Abstract / Section VI-B"},{"comment":"Whisper medium is a 764M-parameter supervised ASR model trained on 680k hours of labeled web audio, which very likely includes LibriSpeech and possibly WSJ0, the source corpora of Libri2Mix and WSJ0-2Mix. The TS-HuBERT and WavLM baselines are self-supervised and were not trained on transcripts of these corpora, so the reported gains over TS-HuBERT (19% on Libri2Mix, 10% on WSJ0-2Mix) may partly reflect the base model's prior exposure to the test data rather than the SQ-Former. The internal TSE-Whisper comparisons are controlled and reassuring, but the abstract's headline claims should either be qualified or supported by a control experiment using a Whisper-style model that has no LibriSpeech/WSJ0 exposure (e.g., an OWSM model).","section":"Section II-A / Tables II and VI"},{"comment":"The robustness analysis shows that with mismatched enrollment, WER degrades from 20.1% to 71.8% on the Libri2Mix Test set, which is worse than the vanilla Whisper baseline (54.3%). This failure mode is a central limitation of the method and should be clearly stated in the abstract and conclusion; the current framing of the method as robustly 'eliminating interfering speakers' is misleading without this qualifier.","section":"Section VI-F / Table V"},{"comment":"All reported WERs come from a single training run without error bars or significance tests, yet the text repeatedly uses the word 'significant' (e.g., Sections VI-B and VII). Since some of the headline differences are small (e.g., 23.2 vs. 24.8 on Libri2Mix, 22.3 vs. 21.2 on AMI), the authors should report variance across seeds or perform a statistical test to substantiate these claims.","section":"Tables I-VII"}],"minor_comments":[{"comment":"The temperature kappa in Eq. (14) is never given a value in the hyperparameter description; please report it.","section":"Section V-B"},{"comment":"The parameter count for TSE-Whisper (Full) Add is 762.98M in Table I but 762.85M in Table II; please make these consistent.","section":"Tables I and II"},{"comment":"The statement that LoRA fine-tuned models outperform fully fine-tuned models 'across all adaptation methods' is contradicted by the Cat row in Table I (30.2 full vs. 31.2 LoRA on Test).","section":"Section VI-A"},{"comment":"The captions and diagrams contain typos: 'Conv1D + GLUE' should be 'Conv1D + GELU', and 'Framwork' should be 'Framework'.","section":"Figures 2 and 3"},{"comment":"'liner projection' should be 'linear projection'.","section":"Section IV-D"},{"comment":"Please clarify whether the 'Separation + Whisper' baseline in Tables I-II uses a separation model trained on the same Libri2Mix training data; if so, the comparison should be contextualized in the text because the separation model may be trained on the same test conditions.","section":"Section V-A"}],"recommendation":"major_revision","confidential_remarks":"The main barrier to acceptance is the abstract's inaccurate 15%/TS-HuBERT framing and the unresolved pretraining-overlap confound. The technical contribution is sound and the code release is a strength. I would be willing to reconsider if the authors add a control experiment or substantially temper the state-of-the-art claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SQ-Whisper is a genuine empirical step forward for target-speaker ASR. The new piece is the SQ-Former: a small set of learnable query vectors that attend to enrollment audio and mixture features, trained with a speaker contrastive loss, producing speaker prompts inserted into both encoder and decoder of Whisper. The paper does the comparison work you'd want: it tries four TSE adapters on Whisper, then shows SQ-Whisper beats the best TSE-Whisper by a wide margin, and it reports ablations on query count, contrastive loss, and prompt placement. The mismatched-enrollment experiment (Table V) is honest and useful: with a wrong enrollment the system collapses to 71.8% WER, which tells you the method depends on reliable enrollment.\n\nThe main soft spot is the headline comparison. The abstract says 'up to 15%' relative reduction over TS-HuBERT on Libri2Mix, but the table shows 19% (24.8→20.1); the 15% figure is actually the improvement over the PIT-Transformer baseline (23.5→20.1). That mismatch should be fixed. The larger caveat is that the comparison to TS-HuBERT is not controlled for pretraining data. Whisper was trained on 680k hours of web audio that likely includes LibriSpeech and WSJ0 utterances or speakers, while TS-HuBERT and WavLM were self-supervised and never saw these transcriptions. So part of the gain over TS-HuBERT could be Whisper's prior exposure rather than the SQ-Former. The internal comparison against TSE-Whisper (same Whisper base) is clean and still shows big gains, so the mechanism does something real; but the paper should either report overlap analysis or soften the claim that the method is the source of the full SOTA improvement. The AMI experiments use oracle human-annotated segmentation, which is a strong assumption; and there are no error bars or multiple seeds anywhere.\n\nNone of this changes my overall read: the method is new, reproducible (code and recipes are released), and the ablation story is unusually complete for this area. The paper deserves a serious referee and, after the abstract fix and a discussion of the pretraining-overlap confound, conditional acceptance.","headline":"Genuinely new speaker-query adaptation for Whisper with solid ablations, but the headline 15% claim is mis-stated and the comparison to TS-HuBERT may be inflated by pretraining overlap.","tokens_in":17892,"tokens_out":2090,"would_cite":true,"duration_ms":18632,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SQ-Whisper adapts Whisper to transcribe one voice from overlapping speech.","keywords":["target-speaker ASR","speech foundation model","Whisper","speaker prompt","SQ-Former","overlapped speech","speaker contrastive learning","parameter-efficient fine-tuning"],"falsifier":"Hold out a subset of speakers during training and test with enrollment clips from those unseen speakers; if SQ-Whisper still separates them well, the model generalizes beyond memorized voices, but if WER degrades sharply or the speaker prompts stop separating, the claimed mechanism fails.","tokens_in":16780,"feed_emoji":"🎙️","tokens_out":5476,"duration_ms":47977,"temperature":0.7,"pith_summary":"The paper asks whether a single-speaker speech foundation model can be adapted to target-speaker ASR: transcribing one chosen voice from a mixture of overlapping talkers. It answers yes, with a new module called SQ-Former that turns a few trainable query vectors into speaker prompts conditioned on both the mixture and a short enrollment clip of the target speaker. On Libri2Mix the full model reduces word error rate to 20.1%, and with added training data and speed perturbation reaches 14.6%; on WSJ0-2Mix the LoRA-tuned model reaches 4.4%. These results beat the previous TS-HuBERT baseline and a separation-plus-Whisper pipeline, suggesting that the learned prompts let Whisper separate speakers internally rather than requiring an external separation front end.","feed_headline":"SQ-Whisper cuts overlapped-speech errors by up to 15%","feed_subtitle":"A querying module steers Whisper to one voice, beating purpose-built systems and reaching 4.4% word error on WSJ0-2Mix.","key_machinery":"The central object is the SQ-Former, a small Transformer adaptor with 16 trainable query vectors. In each block, the queries first attend to enrolled target-speaker features, then cross-attend to the Whisper encoder's mixture representation; the output is a fixed-length speaker prompt appended to the encoder input and inserted between special tokens in the decoder. An accompanying speaker contrastive loss pulls a prompt toward the enrollment of the same speaker and away from other speakers in the batch, which the authors show is worth roughly 5% absolute WER.","core_discovery":"The paper's central claim is that Whisper, trained only on single-speaker audio, can be steered to recognize overlapped speech by injecting a learned speaker prompt into both the encoder and the decoder. The prompt is produced by an SQ-Former module that lets trainable queries attend to the enrollment speech to absorb the target voice, then cross-attends to the mixture to find the parts of the acoustic representation belonging to that voice. A speaker contrastive loss makes the resulting prompts cluster by speaker identity, and the paper reports up to 15% and 10% relative WER reductions over TS-HuBERT on Libri2Mix and WSJ0-2Mix respectively. Prompting both the encoder and the decoder gives further gains, showing the speaker prompt is useful at both the acoustic and the linguistic decoding stages.","pith_inferences":["The authors do not test whether the SQ-Former transfers to self-supervised encoders such as HuBERT; a testable extension is to attach it to such a model and check whether the query-search mechanism works on representations learned without ASR supervision.","The mismatched-enrollment collapse suggests the model may use the prompt as a hard gate rather than a soft bias; if true, training with deliberately mismatched enrollments could improve robustness.","Because the contrastive loss currently uses only in-batch negatives, scaling to much larger batches or explicit speaker banks could make the prompts more discriminative and reduce the enrollment sensitivity.","The prompt that separates speakers for ASR could also be probed for speaker verification or diarization, although the paper does not evaluate those tasks."],"forward_implications":["With matched enrollment, SQ-Whisper outperforms the prior TS-HuBERT baseline by up to 15% relative WER on Libri2Mix and 10% on WSJ0-2Mix.","Adding speed perturbation and the larger Train-360 set yields state-of-the-art WERs of 14.6% on Libri2Mix Test and 4.4% on WSJ0-2Mix Test.","The method works with LoRA, freezing most of Whisper's weights: LoRA SQ-Whisper with 40.76M trainable parameters beats TS-HuBERT with 105.18M trainable parameters.","The encoder-side and decoder-side prompts are complementary; removing the contrastive loss or using only one prompt degrades performance.","On the AMI meeting corpus, SQ-Whisper reaches 22.0% WER, close to an SOT system pre-trained on 900k hours of multi-speaker data despite being adapted from single-speaker Whisper.","Because SQ-Former is a generic Transformer adaptor, the authors argue the method can be transplanted to other Transformer-based speech foundation models.","The learned prompts appear to encode speaker identity rather than just acoustic content: with mismatched enrollment the model collapses to a fixed output, so enrollment reliability is the key deployment risk.","A testable extension is to increase the number of in-batch negative speakers during contrastive training to see whether the mismatched-enrollment failure rate drops."],"supporting_citations":[{"why":"The Whisper model is the single-speaker foundation model that the paper adapts for TS-ASR.","marker":"[11]"},{"why":"The Querying Transformer in BLIP-2 supplies the query-based cross-attention idea behind the SQ-Former.","marker":"[9]"},{"why":"The TSE module for speaker embeddings is the baseline adaptation method that the paper integrates into TSE-Whisper.","marker":"[27]"},{"why":"TS-HuBERT is the main prior TS-ASR system whose reported WERs are used as the comparison baseline.","marker":"[28]"},{"why":"Prompt-tuned Whisper is the prior supervised-model approach to TS-ASR that SQ-Whisper extends and improves upon.","marker":"[29]"},{"why":"LoRA is the parameter-efficient fine-tuning technique used in half of the SQ-Whisper experiments.","marker":"[46]"},{"why":"LibriMix is one of the two main simulated mixture datasets used for evaluation.","marker":"[47]"},{"why":"WSJ0-2Mix is the second simulated mixture dataset used for evaluation.","marker":"[48]"},{"why":"The AMI meeting corpus supplies the real-world overlapping-speech evaluation.","marker":"[49]"}],"fun_headline_variants":["SQ-Whisper: speaker queries cut overlapped-speech WER by 15%","Whisper steered to one voice: 15% WER drop on overlapping speech","Speaker-query Whisper tops TS-HuBERT, hits 4.4% WER on WSJ0-2Mix","SQ-Whisper adapts Whisper to overlapped audio, beats robust baselines","Target-speaker ASR via querying Whisper: up to 15% WER improvement"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a clean enrollment utterance of the target speaker is available at test time and that this speaker is actually present in the mixture; when enrollment is mismatched, the reported word error rate jumps from about 20% to nearly 72%.","fun_headline_variants_meta":{"raw":{"variants":["SQ-Whisper: speaker queries cut overlapped-speech WER by 15%","Whisper steered to one voice: 15% WER drop on overlapping speech","Speaker-query Whisper tops TS-HuBERT, hits 4.4% WER on WSJ0-2Mix","SQ-Whisper adapts Whisper to overlapped audio, beats robust baselines","Target-speaker ASR via querying Whisper: up to 15% WER improvement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001078,"raw_usage":{"total_tokens":4559,"prompt_tokens":1039,"completion_tokens":3520,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":655,"completion_tokens_details":{"reasoning_tokens":3404}},"tokens_in":655,"tokens_out":3520,"duration_ms":22168,"temperature":1.0,"reasoning_tokens":3404,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:33:09.077402+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out a subset of speakers during training and test with enrollment clips from those unseen speakers; if SQ-Whisper still separates them well, the model generalizes beyond memorized voices, but if WER degrades sharply or the speaker prompts stop separating, the claimed mechanism fails.","supporting_citations":[{"cited_title":"BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,","cited_arxiv_id":null,"evidence_quote":"The Querying Transformer in BLIP-2 supplies the query-based cross-attention idea behind the SQ-Former."},{"cited_title":"Adapting self- supervised models to multi-talker speech recognition using speaker embeddings,","cited_arxiv_id":null,"evidence_quote":"The TSE module for speaker embeddings is the baseline adaptation method that the paper integrates into TSE-Whisper."},{"cited_title":"Weakly-supervised speech pre-training: A case study on target speech recognition,","cited_arxiv_id":null,"evidence_quote":"TS-HuBERT is the main prior TS-ASR system whose reported WERs are used as the comparison baseline."},{"cited_title":"Extending Whisper with prompt tuning to target-speaker ASR,","cited_arxiv_id":null,"evidence_quote":"Prompt-tuned Whisper is the prior supervised-model approach to TS-ASR that SQ-Whisper extends and improves upon."},{"cited_title":"LoRA: Low-rank adaptation of large language models,","cited_arxiv_id":null,"evidence_quote":"LoRA is the parameter-efficient fine-tuning technique used in half of the SQ-Whisper experiments."},{"cited_title":"Deep clustering: Discriminative embeddings for segmentation and separation,","cited_arxiv_id":null,"evidence_quote":"WSJ0-2Mix is the second simulated mixture dataset used for evaluation."},{"cited_title":"The AMI meeting corpus: A pre-announcement,","cited_arxiv_id":null,"evidence_quote":"The AMI meeting corpus supplies the real-world overlapping-speech evaluation."}],"review_version":1}