{"id":"18baa430-7db9-4af3-aadc-f915f14be84d","arxiv_id":"2506.13414","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A modular two-speaker ASR pipeline combining DiariZen diarization and DiCoW target-speaker Whisper achieves 16.75% micro-average tcpWER/CER and second place in the MLC-SLM challenge Task 2.","lead":"This paper describes a two-speaker speech recognition system that combines a diarization model (DiariZen) with a Whisper variant conditioned on speaker activity (DiCoW), and reports second place in the MLC-SLM multilingual challenge. The authors also find labeling errors in the challenge training data and show that a voice-activity detector partially fixes the resulting problems.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The decisive weakness is Section 4.3's untested proxy: the final test-set gain from Silero VAD (28.6% to 17.4%) assumes the test-like development set faithfully reproduces the muted test protocol, yet the energy threshold and VAD weight 0.8 are neither specified nor ablated.","rationale":"The reader's weakest assumption identifies the same load-bearing point: the VAD mitigation is validated only on a custom 'test-like' development set with an unspecified construction rule, and the VAD weight 0.8 is hand-set without ablation. This is exactly where the paper's most striking numerical result lives, and it is where the evidence is thinnest. The central challenge-rank claim (second place, 16.75% micro-average tcpWER/CER) is an official outcome and is not undermined by this concern; what is undermined is the paper's secondary but emphasized claim that labeling-inconsistency mitigation is what drives the improvement. The paper is candid that DER gains are misleading, which is a point in its favor, but that candor transfers the evidential weight to the downstream tcpWER numbers, making the proxy and the VAD weight even more load-bearing. Releasing model weights and recipes is real support for reproducibility, but it does not settle whether the hand-set weight and the proxy construction would generalize. Since the reader already assigned a CONDITIONAL verdict for essentially this reason, no verdict adjustment is needed; the recommended test is a VAD-weight sweep on explicitly varied versions of the test-like development set, which would convert the concern into a documented fact.","tokens_in":8761,"tokens_out":4367,"duration_ms":44856,"concrete_test":"Regenerate the test-like development set with explicit, varied energy thresholds (e.g., mute unannotated segments above the 50th, 75th, and 90th percentile of RMS energy) and sweep the Silero VAD weight over 0.0, 0.4, 0.8, and 1.0 while fixing every other pipeline component. Report the tcpWER/CER curve for each threshold. If 0.8 is not the minimum, or if the optimal weight shifts with the threshold, the reported 28.6% to 17.4% test gain is not robustly established and the transfer assumption fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline mitigation result, that adding Silero VAD improves test tcpWER/CER from 28.6% to 17.4%, rests on two unvalidated premises. In Section 4.3, the authors build a 'test-like' development set by muting unannotated segments with 'noticeable energy' but do not specify the energy threshold, even though they assert that the actual test set mutes all unannotated segments. If the test protocol differs from this proxy, the FTVAD system's advantage may be an artifact of matching the proxy rather than a robust property. The Silero VAD weight of 0.8 is introduced with no ablation or tuning curve, so nothing demonstrates that this hand-set value transfers from the proxy to the true test distribution. Table 3 makes this more acute: FTVAD raises DER on the test-like development set relative to FT (16.2% vs. 12.4%), so the only evidence for the VAD benefit is the downstream tcpWER on a single synthetic development split plus one test evaluation. The component-wise comparison in Table 2 also confounds the diarization change with the DiCoW/turbo/long-form inference change, but the decisive unvalidated quantity is the VAD-specific gain, since the paper's secondary contribution and a large part of the final-system improvement depend on it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper describes the BUT system for the MLC-SLM Challenge Task 2: a two-speaker ASR pipeline combining DiariZen (a Pyannote-based diarization pipeline with WavLM/Conformer EEND) with DiCoW (a diarization-conditioned Whisper variant). The authors report that DiariZen outperforms the Pyannote baseline in both out-of-domain and fine-tuned settings on the development data, that DiCoW retains multilingual ASR ability despite English-only target-speaker fine-tuning, and that the final system reaches a micro-average tcpWER/CER of 16.75% and ranks second in the challenge. They also document labeling inconsistencies in the MLC-SLM training/development data (unannotated speech and silence labeled as speech) and propose a Silero VAD-based mitigation, reporting an improvement on the test set from 28.6% to 17.4%.","tokens_in":8973,"tokens_out":5473,"duration_ms":47778,"significance":"If validated, the paper's contribution is a competitive, non-LLM modular system for multilingual two-speaker transcription, plus a useful analysis of diarization training under loosely annotated data. The release of the DiCoW and DiariZen models is a concrete strength, as is the authors' explicit caution (Section 4.1) that DER gains may reflect learning to mimic inconsistent annotations rather than improved diarization. The main concerns are the unexplained discrepancy between the two reported test-set numbers (16.75% vs. 17.4%) and the lack of validation of the key parameters in the VAD mitigation, which is the basis of the system's largest reported improvement.","major_comments":[{"comment":"The abstract states the final system achieves a micro-average tcpWER/CER of 16.75%, but Section 4.3 reports that the VAD-based approach improves the test set from 28.6% to 17.4%. The relationship between these two test-set numbers is not explained; the paper must clarify which configuration yields 16.75% (e.g., whether 17.4% is an intermediate result, a different evaluation protocol, or a subset) and why they differ.","section":"Section 4.3 (test-set results)"},{"comment":"The 'test-like' development set is constructed by muting unannotated segments with 'noticeable energy,' but the energy threshold is not specified, and no evidence is provided that this proxy reproduces the official test protocol's 'cleaner' annotation. Since the test-set improvement (28.6% to 17.4%) is the main evidence for the VAD mitigation, the paper should specify the threshold and, ideally, validate the proxy against the actual test protocol or an additional held-out set.","section":"Section 4.3 (test-like development set)"},{"comment":"The Silero VAD weight of 0.8 is introduced without an ablation or tuning curve. This is particularly important because Table 3 shows that FT+VAD worsens DER on the test-like development set (16.2% vs. 12.4% for FT), and the benefit only appears in the downstream tcpWER/CER. To support the transferability of this hand-set weight from the proxy to the test distribution, the paper should report sensitivity to this weight.","section":"Section 4.3 (Silero VAD weight)"},{"comment":"The 'Real diar' comparison confounds two changes: the diarization module (DiariZen vs. Pyannote) and the ASR/inference module (DiCoW with long-form decoding vs. Whisper large-v3 with chunked decoding). Consequently, the reported gains cannot be attributed to the diarization replacement alone. A decomposition (e.g., Pyannote segmentation with DiCoW, or DiariZen segmentation with baseline Whisper) or an explicit statement of this confound is needed.","section":"Section 4.2, Table 2"}],"minor_comments":[{"comment":"The language label 'Inidian English' should be 'Indian English.'","section":"Table 1"},{"comment":"The caution about DER gains being potentially misleading is welcome, but the abstract should also mention this caveat, since it currently states the fine-tuned DiariZen 'continues to outperform the fine-tuned Pyannote baseline' without noting that this may reflect annotation mimicry.","section":"Section 4.1"},{"comment":"The systems labeled 'FT+V AD' contain a stray space in the 'V AD' acronym; it should be 'FT+VAD' throughout.","section":"Table 3"},{"comment":"The caption and text do not specify which columns correspond to the four system variants; a short legend or description would improve readability.","section":"Figure 3"},{"comment":"The sentence 'We use a challenge text normalization procedure to the reference transcripts' is slightly ambiguous; it should say 'apply ... to' rather than 'to.'","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a challenge system description with useful reproducible artifacts. The authors should check whether the 16.75% vs. 17.4% discrepancy is a typo, since the headline ranking claim depends on it. I recommend asking for the missing ablations and proxy validation, but I do not see grounds for rejection if those are supplied. The self-citations are reasonable given the system builds on the authors' prior work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is not a new architecture paper: DiCoW and DiariZen are prior work from the same group, and the combination is expected. Second, the genuinely interesting contribution is in Section 4.3: the authors show that MLC-SLM training labels contain missing speech and false speech intervals, that fine-tuning the diarizer learns those inconsistencies, and that an auxiliary VAD fixes much of the damage on the official test set, taking tcpWER/CER from 28.6% to 17.4%. That analysis is worth a careful read.\n\nCredit where due: the authors are upfront that DER improvements may be misleading (Section 4.1), they release model weights, and they give enough detail to reproduce the pipeline. The multilingual results are useful evidence that diarization-conditioned Whisper retains multilingual ability after English-only fine-tuning.\n\nSoft spots. The stress-test note lands: the VAD benefit is established through a \"test-like\" development set where \"unannotated segments with noticeable energy are muted\" — the energy threshold is not specified — and the Silero VAD weight of 0.8 appears without an ablation or tuning curve. Because FT+VAD actually raises DER on the test-like split (16.2% vs 12.4%), the entire case for the VAD rests on downstream tcpWER on one proxy plus one test evaluation. In a challenge ranking that is acceptable evidence, but it is not a controlled finding. Second, Table 2's full-system comparison confounds the diarization swap with the Whisper turbo / long-form inference change; for a system-description paper that is tolerable, but it weakens component-level attribution. Third, the test-set result has no error bars; that is normal for a challenge, just do not over-rotate on the exact 16.75%.\n\nWho is this for? Researchers working on multi-talker ASR or diarization-conditioned ASR, and anyone preparing for MLC-SLM-style evaluations. The labeling-inconsistency discussion is also useful for challenge organizers.\n\nRecommendation: this deserves peer review. It is honest, reproducible enough, and the VAD finding is interesting even if the proxy needs better documentation. I would ask the authors to specify how the test-like set was constructed and to show a short ablation of the VAD weight, but I would accept the paper contingent on that.","headline":"A competent challenge write-up whose real value is the labeling-inconsistency analysis and VAD mitigation, but the VAD case rests on an under-documented proxy and a single test evaluation.","tokens_in":9623,"tokens_out":2713,"would_cite":true,"duration_ms":25534,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that a Whisper ASR model conditioned on frame-level diarization probabilities, rather than speaker embeddings, transcribes two-speaker conversations in 15 languages and ranks second in the MLC-SLM challenge with a 16.75%…","keywords":["diarization-conditioned ASR","target-speaker ASR","speaker diarization","Whisper","WavLM","voice activity detection","multilingual ASR","MLC-SLM challenge"],"falsifier":"Re-run DiCoW+DiariZen on the official MLC-SLM test set with the Silero VAD weight set to 0.5, 0.7, 0.9, and 1.0 while keeping everything else fixed; if 0.8 is not near-optimal, or if the gap between fine-tuned-with-VAD and fine-tuned-only narrows below the reported 28.6%-to-17.4% margin, the mitigation claim fails. Independently, have human annotators label a sample of the test set and compute DER for the fine-tuned DiariZen with and without VAD; a large false-alarm rate on human-verified silence would confirm the label-inconsistency story, while low false alarms would contradict it.","tokens_in":8507,"feed_emoji":"🎙️","tokens_out":7629,"duration_ms":63839,"temperature":0.7,"pith_summary":"This paper describes a complete two-speaker ASR pipeline for the MLC-SLM challenge: DiariZen, a WavLM-and-Conformer diarization pipeline built on Pyannote, feeds speaker-activity probabilities to DiCoW, a Whisper variant whose encoder applies frame-dependent affine transformations conditioned on four speaking states. The authors claim this combination generalizes across 15 languages without domain adaptation, keeps Whisper's multilingual ability after English-only fine-tuning, and reaches a micro-average tcpWER/CER of 16.75 percent, ranking second in Task 2. The paper also argues that the challenge training data contains labeling inconsistencies, skipped speech and silence marked as speech, that poison diarization fine-tuning, and that adding a Silero voice-activity detector with weight 0.8 improves the test-set tcpWER from 28.6 percent to 17.4 percent. A sympathetic reader would care because the result suggests modular, non-LLM systems can compete on multilingual conversational ASR, and because the labeling-mismatch analysis offers a cautionary lesson for any diarization fine-tuning on loosely annotated data.","feed_headline":"Diarization-conditioned Whisper places 2nd in 15-language ASR challenge","feed_subtitle":"Whisper variant that reads speaker-activity masks beats the baseline across 15 languages.","key_machinery":"The load-bearing mechanism is the Silence-Target-NonTarget-Overlap (STNO) mask: from diarization probabilities $d(s,t)$ the system forms four per-frame probabilities $p_S, p_T, p_N, p_O$ that sum to one, and each Whisper encoder layer applies four affine transformations $W_l z_t + b_l$, one per state, blended by those probabilities via the FDDT (frame-level diarization-dependent transformation). This carries the conditioning signal from diarization into the ASR encoder without speaker embeddings or enrollment audio. On the diarization side, DiariZen's local EEND module combines a WavLM backbone with a Conformer and a powerset classification head, then clusters speaker embeddings across chunks. The third piece is the auxiliary Silero VAD merged into the diarization probabilities with weight 0.8, which suppresses the false speech predictions the fine-tuned model learned from inconsistently labeled silence.","core_discovery":"The central claim is that frame-level diarization masks are a sufficient conditioning signal for target-speaker ASR: DiCoW replaces speaker embeddings with per-frame probabilities of silence, target speech, non-target speech, and overlap, and applies these as a weighted blend of four learned affine transforms in every Whisper encoder layer. Because the transforms are initialized to the identity, the pretrained Whisper's behavior is preserved at the start of fine-tuning, which the authors say explains why the model keeps its multilingual performance even though the fine-tuning data is English-only. On the MLC-SLM Task 2 evaluation, the full system with fine-tuned DiariZen segmentation achieves 16.75% micro-average tcpWER/CER, second place, and DiariZen alone beats the Pyannote baseline on both out-of-domain and fine-tuned conditions. A secondary claim is that apparent DER improvements from fine-tuning can be artifacts: the fine-tuned diarizer learns to mimic the inconsistent training labels, and a VAD-based speech/silence correction improves the test tcpWER from 28.6% to 17.4% even though it makes DER look worse on the dev set.","pith_inferences":["The STNO conditioning idea generalizes: the same four-state mask could be fed to a speech LLM decoder in place of Whisper's encoder, which the paper names as future work, and could extend to more than two speakers by partitioning the overlap state.","The VAD-weight transfer is the fragile link: 0.8 was chosen without a documented sweep on the modified dev set, so the 17.4% test number should be read as conditional on that hand-set value holding on the true test protocol.","The label-inconsistency analysis implies that other challenge participants who fine-tuned diarization on the same data likely absorbed the same bias, so part of the ranking gap may reflect who corrected for annotation artifacts rather than intrinsic diarization quality.","A testable extension: recompute the modified 'test-like' dev set with different energy thresholds and check whether the FT+VAD advantage over FT is stable; if it flips, the proxy is not faithful enough to guide system choices."],"forward_implications":["A two-speaker ASR system can be built modularly from diarization and a conditioned Whisper, with no LLM rescoring, and still place second in a 15-language challenge.","Target-speaker ASR does not require speaker embeddings or enrollment: frame-level activity probabilities carry enough speaker identity information for Whisper to separate two speakers.","Fine-tuning Whisper's encoder with identity-initialized affine transforms on English data does not erase multilingual recognition, so diarization-conditioned adaptation is a viable route to low-resource language support.","Diarization fine-tuning on loosely annotated data can hurt downstream ASR by teaching the model to reproduce missing-speech and silence-as-speech labels; auxiliary VAD post-processing is a cheap corrective.","Because the test protocol mutes unannotated segments, diarization systems should be evaluated on a test-like protocol with muted unannotated audio, not on the raw dev set, to predict real performance."],"supporting_citations":[{"why":"Supplies the WavLM Large self-supervised backbone that DiariZen's frame-level model is built on and later structurally pruned.","marker":"[1]"},{"why":"Defines Whisper, the pretrained ASR model whose encoder DiCoW modifies and whose multilingual capabilities the paper argues are preserved.","marker":"[5]"},{"why":"Defines DiCoW and its FDDT training recipe (CTC preheat, FDDT preheat, full fine-tuning), which this paper follows without changing core hyperparameters.","marker":"[24]"},{"why":"Defines DiariZen, the WavLM+Conformer EEND pipeline with chunking, embeddings, and clustering that replaces the Pyannote baseline.","marker":"[26]"},{"why":"Is the Pyannote diarization pipeline that DiariZen builds on and that serves as the baseline in most DER and tcpWER comparisons.","marker":"[27]"},{"why":"Provides the powerset multi-class cross-entropy loss used to train the two-speaker local EEND classification head.","marker":"[28]"},{"why":"Provides the Silero voice activity detector used at weight 0.8 to merge speech/silence probabilities, yielding the test-set tcpWER improvement from 28.6% to 17.4%.","marker":"[44]"}],"fun_headline_variants":["Diarization-conditioned Whisper takes 2nd in multilingual ASR","Mask-conditioned Whisper beats baseline across 15 languages","Frame-level diarization masks boost Whisper to 2nd place","Second in MLC-SLM: whisper with speaker-activity masks","DiCoW: identity transforms keep Whisper multilingual after fine-tune"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that the VAD mitigation improves the test set from 28.6% to 17.4% rests on two premises: that the test set follows a cleaner protocol in which all unannotated segments are truly muted, and that the authors' modified 'test-like' development set, where unannotated segments with noticeable energy are muted using an unspecified energy threshold, faithfully reproduces that protocol, so the hand-set Silero VAD weight of 0.8 transfers.","fun_headline_variants_meta":{"raw":{"variants":["Diarization-conditioned Whisper takes 2nd in multilingual ASR","Mask-conditioned Whisper beats baseline across 15 languages","Frame-level diarization masks boost Whisper to 2nd place","Second in MLC-SLM: whisper with speaker-activity masks","DiCoW: identity transforms keep Whisper multilingual after fine-tune"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000589,"raw_usage":{"total_tokens":2798,"prompt_tokens":1013,"completion_tokens":1785,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":1702}},"tokens_in":629,"tokens_out":1785,"duration_ms":12499,"temperature":1.0,"reasoning_tokens":1702,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:01:37.882720+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run DiCoW+DiariZen on the official MLC-SLM test set with the Silero VAD weight set to 0.5, 0.7, 0.9, and 1.0 while keeping everything else fixed; if 0.8 is not near-optimal, or if the gap between fine-tuned-with-VAD and fine-tuned-only narrows below the reported 28.6%-to-17.4% margin, the mitigation claim fails. Independently, have human annotators label a sample of the test set and compute DER for the fine-tuned DiariZen with and without VAD; a large false-alarm rate on human-verified silence would confirm the label-inconsistency story, while low false alarms would contradict it.","supporting_citations":[{"cited_title":"These models achieve remarkable accuracy by leveraging mas- sive training data [7, 8] and scaling up model parameters [9]","cited_arxiv_id":null,"evidence_quote":"Supplies the WavLM Large self-supervised backbone that DiariZen's frame-level model is built on and later structurally pruned."},{"cited_title":"Our system placed second in the MLC-SLM Challenge, showing strong performance across diverse lan- guages","cited_arxiv_id":null,"evidence_quote":"Defines Whisper, the pretrained ASR model whose encoder DiCoW modifies and whose multilingual capabilities the paper argues are preserved."},{"cited_title":"Permutation invariant training of deep models for speaker-independent multi-talker speech separation,","cited_arxiv_id":null,"evidence_quote":"Defines DiCoW and its FDDT training recipe (CTC preheat, FDDT preheat, full fine-tuning), which this paper follows without changing core hyperparameters."},{"cited_title":"Conformer-based target-speaker automatic speech recognition for single-channel audio,","cited_arxiv_id":null,"evidence_quote":"Defines DiariZen, the WavLM+Conformer EEND pipeline with chunking, embeddings, and clustering that replaces the Pyannote baseline."},{"cited_title":"Adapting self-supervised models to multi-talker speech recognition using speaker embeddings,","cited_arxiv_id":null,"evidence_quote":"Is the Pyannote diarization pipeline that DiariZen builds on and that serves as the baseline in most DER and tcpWER comparisons."},{"cited_title":"Empowering Whisper as a joint multi-talker and target-talker speech recognition system,","cited_arxiv_id":null,"evidence_quote":"Provides the powerset multi-class cross-entropy loss used to train the two-speaker local EEND classification head."}],"review_version":2}