{"id":"1297f443-7f66-4606-a1e9-3e23bb638ec3","arxiv_id":"2411.13849","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A single sequence-to-sequence network with detection and representation decoders achieves state-of-the-art online and offline speaker diarization on DIHARD-II and DIHARD-III.","lead":"This paper introduces S2SND, a neural framework that performs speaker diarization (who spoke when) both online and offline in one model, detecting unknown speakers with a pseudo-speaker embedding and learning speaker representations inside the network. It reports state-of-the-art diarization error rates on the DIHARD-II and DIHARD-III benchmarks, suggesting a practical step toward real-time meeting transcription without a separate voice activity detector.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim rests on selecting the best of 24 configurations on the evaluation set, so the reported DERs may be optimistic; a validation-based selection is needed before the headline numbers can be accepted.","rationale":"The paper's central claim is an empirical SOTA claim. The strongest evidence is Table II, which sweeps 24 configurations and highlights the minimum DER on the eval set. No pre-registered or validation-based selection of the configuration is described; only thresholds are tuned on dev. Since the same eval set is used for selection and reporting, the best-of-24 number has positive expected bias. The spread in Table II (e.g., 24.41% vs 27.79% for online DIHARD-II Medium) means this bias can be several DER points, comparable to the claimed margin over prior SOTA. The reader's identified weakest assumption (one new speaker per block) is real but does not threaten the reported numbers as directly; it affects generalization to other data. Therefore the most load-bearing concern is the selection methodology. The concrete test is to select the configuration on the dev splits and evaluate once on eval; if the resulting DERs no longer beat the prior systems, the SOTA claim fails. This is a clean, actionable check, so the verdict remains CONDITIONAL as the reader recommended.","tokens_in":25405,"tokens_out":6331,"duration_ms":63470,"concrete_test":"Force a pre-registered selection rule: use dev39/dev51 to choose the configuration (model size, training strategy, chunk length, right-context length) for each dataset and metric mode, then evaluate that single configuration on DIHARD-II/III eval and recompute DERs. If the validated configuration's DERs are no longer below the best prior rows in Tables III/IV, or rise by more than the ~1-3 DER-point spread seen within Table II, the headline SOTA claim is attributable to test-set selection. A supplementary bootstrap of the 24 eval scores can quantify the expected maximum bias.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Abstract, Tables III and IV) is that S2SND achieves new state-of-the-art DERs on DIHARD-II and DIHARD-III. The empirical basis is Table II, which evaluates 24 configurations (S1-S24) on the evaluation sets, varying model size, simulation corpus, training strategy, chunk length, and right-context length. The gray-highlighted 'best' rows are selected by taking the minimum DER over these runs on the same test set used to report the headline numbers. The text only states that thresholds tau_1 and tau_2 are tuned on the validation split (Sec. IV-C2); there is no indication that the configuration itself was selected on dev39/dev51. Because the same eval examples determine both model selection and performance reporting, the reported DERs are optimistically biased. The spread across configurations is material: on DIHARD-II online, the Medium model DER ranges from 24.41% (S22) to 27.79% (S13); offline ranges from 21.95% (S21) to 24.09% (S17). The gap between the selected best and the configuration a validation-based selection would pick could be several DER points, which is the same order as the claimed improvement over prior SOTA. Therefore, the central SOTA claim is not established unless configuration choice is independent of the eval set.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes S2SND, a sequence-to-sequence neural diarization framework that performs online and offline speaker diarization with a single model. It augments a TSVAD-style encoder-decoder with two decoders: a detection decoder that predicts voice activities from partially given speaker embeddings, and a representation decoder that extracts speaker embeddings from predicted voice activities. A pseudo-speaker embedding is used to detect and enroll one unknown speaker per block, and a buffer stores the extracted embeddings for subsequent blocks. The model is trained with a masked-speaker prediction objective and an ArcFace loss on the representation branch, plus a knowledge-distillation strategy from a larger teacher extractor. Experiments on DIHARD-II and DIHARD-III report new state-of-the-art DERs: online without oracle VAD 24.41%/17.12%, offline without oracle VAD 21.95%/15.13%, with additional gains when oracle VAD post-processing is applied. The paper also reports speaker-counting accuracy and computing efficiency for the proposed models.","tokens_in":25713,"tokens_out":6112,"duration_ms":57780,"significance":"If the reported results are robust, the paper makes a significant contribution: it unifies online and offline diarization in a clustering-free, PIT-free architecture with automatic speaker enrollment, a property that previous TSVAD/EEND hybrid systems lack. The masked-speaker prediction and target-voice embedding extraction are novel and reasonably motivated, and the paper provides a detailed ablation study in Table II as well as comparisons on two standard benchmarks. The strengths include a clear description of the architecture, training objectives, and inference protocol, as well as a pseudocode specification of the online algorithm. However, the headline SOTA claim currently rests on model selection performed on the evaluation sets, which can materially inflate the reported gains, and there is a train/test mismatch in the representation decoder's inputs. These issues must be addressed before the central claim can be accepted.","major_comments":[{"comment":"The headline SOTA numbers are selected on the evaluation sets themselves. Table II reports DERs for 24 configurations (S1-S24) on the DIHARD-II and DIHARD-III evaluation sets, and the text says 'We select the lowest online and offline DERs for each model size on DIHARD-II and DIHARD-III datasets as the representative results.' The configuration selection (model size, simulation corpus, training strategy, chunk length, right-context length) is therefore not independent of the test data. The only explicitly validation-tuned hyperparameters are the thresholds tau_1 and tau_2 (Sec. IV-C2). Because the spread across configurations is material (e.g., Medium online DIHARD-II ranges from 24.41% in S22 to 27.79% in S17; offline from 21.95% in S21 to 24.09% in S17), the gap between the evaluation-selected best and a validation-selected configuration could be several DER points, which is the same order as the claimed improvement over prior SOTA. Please repeat configuration selection on the validation splits (dev39/dev51) and report both the validation-selected configuration and its evaluation-set DER, or report all configurations on the validation sets so the reader can assess the optimism.","section":"Sec. V-A, Table II, Tables III and IV"},{"comment":"The representation decoder is trained with oracle voice activities as auxiliary queries but deployed with predicted voice activities. In Sec. III-B2 the representation decoder 'utilizes extractor output X as feature embeddings and voice activities Y as auxiliary queries', where Y is ground truth, and the ArcFace loss in Eq. (4) is computed under this oracle-query condition. During inference (Sec. III-C2), the same decoder receives the predicted activities Y_hat as queries. The paper does not describe any mechanism to close this train/test gap, such as scheduled sampling, query dropout/noising, or a loss that also evaluates the representation branch on predicted activities. This exposure bias is particularly concerning for the pseudo-speaker channel, whose predicted activity is used both to enroll new speakers and to extract their embeddings, so errors can accumulate blockwise. Please either train the representation decoder with the detection decoder's outputs (with teacher forcing), or provide a validation-set analysis quantifying the effect of this mismatch.","section":"Sec. III-B2, Eq. (4), Sec. III-C2"},{"comment":"The online pipeline relies on the assumption that at most one new speaker appears in a small block shift, because there is only a single pseudo-speaker embedding slot. If two or more unseen speakers first appear in the same block, only one can be enrolled in that block; the others are delayed until later blocks and may be missed if their speech is short. This is a stated limitation, but the paper does not quantify how often the assumption is violated in the DIHARD-II/III evaluation sets, nor how violations affect DER. Please report the distribution of the number of new speakers per block on these evaluation sets and, if possible, compare with a variant that uses multiple pseudo-speaker slots or that delays enrollment decisions by one block, so that the robustness of the SOTA claim to this assumption can be assessed.","section":"Abstract, Sec. III-C2, Sec. III-C3"}],"minor_comments":[{"comment":"The definition of the weight function W(hat y) uses the notation t' not in Overlap, but the set Overlap is not formally defined. Please specify how overlapping frames are determined from the predicted voice activities of all speakers.","section":"Sec. III-C2"},{"comment":"The loop 'for n in range(1, len(S))' appears to use the embedding dimension S as the loop bound; the intended bound is likely the number of enrolled speaker slots (len(spk_list) or N). Please correct the pseudocode.","section":"Algorithm 1"},{"comment":"The text states that tau_1 and tau_2 are determined by grid search on the validation set, but it does not report the grid range or the resulting values. Please include these values for reproducibility.","section":"Sec. IV-C2"},{"comment":"The table caption says the lowest DERs of each model size are highlighted, but the selection is done on the evaluation set. In addition to the suggested validation-based selection, it would be helpful to mark the validation-best configuration separately in the table.","section":"Table II"},{"comment":"The sentence 'our best online DER (24.41%) is even lower than the previous best offline system (24.64%)' mixes online and offline conditions; the comparison is only valid under the no-oracle-VAD setting and with the selected configuration. Please clarify this phrasing.","section":"Sec. V-B"},{"comment":"There is a typo: 'represnets' should be 'represents' in the sentence describing the pseudo-speaker output.","section":"Sec. III-C2"},{"comment":"The offline rescoring mechanism is described as re-decoding the audio with the final speaker-embedding buffer, but the paper does not specify whether the rescoring pass uses the same block-wise chunking with right-context or a full-sequence decode. Please clarify the inference setup for the offline mode.","section":"Sec. III-C3"}],"recommendation":"major_revision","confidential_remarks":"The paper's technical core is sound and the reported gains are plausible, but the evaluation-selected configurations in Table II currently prevent the SOTA claim from being accepted without further evidence. The train/test mismatch in the representation decoder is also a genuine modeling gap. Both are fixable within the scope of a revision. I would encourage the authors to release the trained models or at least the validation-selected configuration results, as the community will likely want to reproduce the online inference behavior. The self-citations to Seq2Seq-TSVAD and MIMO-TSVAD are appropriate and do not create circularity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on diarization: the dual-decoder S2SND design is genuinely new, and the paper is clearly written. But the headline SOTA claims should not be quoted as-is. The best-of-24 configuration appears to be selected on the eval set, not on validation, so the reported DERs are likely optimistic. That is fixable and does not sink the method, but it needs to be settled before the numbers are used.\n\nWhat is actually good: (1) the masked speaker prediction with a learnable pseudo-speaker embedding is a clean way to detect unknown speakers without an external VAD; (2) the representation decoder, trained jointly with ArcFace on a learnable embedding matrix, makes embedding extraction part of the network rather than a bolt-on; (3) the experiments are extensive, covering model size, simulation corpus, distillation, and latency trade-offs, and the DIHARD comparisons are plausible. The speaker-counting analysis is a nice addition. The self-citations are relevant, not padding.\n\nThe soft spots are real but proportionate. The biggest one: Table II reports 24 configurations, and the test-set DER is used to pick the 'best' ones that then appear in Tables III and IV. Only tau_1 and tau_2 are tuned on validation. The spread across configurations is material (e.g., DIHARD-II online Medium ranges from 24.41% to 27.79%), so some of the claimed margin over prior work may be selection noise. Re-doing the configuration choice on the dev split would make the SOTA claim trustworthy. Second, the representation decoder is trained with ground-truth voice activities but deployed with predicted ones; that teacher-forcing gap is common but should be acknowledged and ideally measured. Third, the 'up to one new speaker per block' assumption is explicit but limits the online setting; it probably holds for most of DIHARD, but the failure cases are not analyzed. Finally, no code or data are released, so the results are not independently reproducible.\n\nThis is a solid empirical paper with a genuine architectural contribution. The central SOTA claim is not yet established, but the method itself deserves serious engagement. For a diarization researcher this is a useful read, and the selection issue is a fixable flaw rather than a fatal one. I would send it to peer review, but the reviewers should insist on validation-based configuration selection and a clear statement of the teacher-forcing mismatch before publication.","headline":"Novel dual-decoder diarization architecture with strong but inflated SOTA numbers due to eval-set model selection.","tokens_in":26219,"tokens_out":2273,"would_cite":true,"duration_ms":24002,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single sequence-to-sequence network performs speaker diarization online and offline, discovering new speakers without a prior enrollment stage.","keywords":["speaker diarization","online speaker diarization","sequence-to-sequence","target-speaker voice activity detection","speaker embedding","masked speaker prediction","DIHARD","blockwise inference"],"falsifier":"Take a test set of conversations where, by design, two previously unenrolled speakers begin speaking within the same chunk while all other conditions match DIHARD-III, and measure the online DER of S2SND; a large error spike on those recordings would confirm that the single-pseudo-speaker slot is the bottleneck. A simpler probe: run the model online on DIHARD-III but force the block shift to align with moments of simultaneous speaker onsets and compare DER against the reported 17.12%.","tokens_in":25210,"feed_emoji":"🎙️","tokens_out":4758,"duration_ms":37354,"temperature":0.7,"pith_summary":"This paper argues that speaker diarization—deciding who spoke when in a conversation—can be done by a single sequence-to-sequence network that both detects previously unseen speakers and learns their voice embeddings on the fly. The proposed S2SND framework processes audio block by block, maintaining a buffer of speaker embeddings from earlier blocks, and uses a learnable pseudo-speaker slot to catch any new voice that appears. Because it never needs a separate clustering step, permutation-invariant training, or pre-extracted enrollment from a prior diarization system, the same model can run online with low latency and then re-score the whole recording offline. On the DIHARD-II and DIHARD-III benchmarks the authors report new best diarization error rates in both modes without oracle voice activity detection: 24.41% and 17.12% online, 21.95% and 15.13% offline. If correct, this would collapse the usual separation between online and offline diarization systems.","feed_headline":"Single model sets new marks for live and offline speaker diarization","feed_subtitle":"On DIHARD-II and DIHARD-III it beats prior systems without oracle voice activity detection.","key_machinery":"The core machinery is a pair of inverse sequence-to-sequence decoders with a shared encoder and a learnable speaker-embedding matrix. The detection decoder takes speaker embeddings (or the pseudo-speaker embedding for unknown voices) as auxiliary queries and predicts binary voice activities; the representation decoder takes those predicted voice activities as auxiliary queries and extracts speaker embeddings, trained by an ArcFace loss against the same embedding matrix. Masked speaker prediction during training randomly erases one speaker's embedding so the model learns to associate missing voices with the pseudo-speaker embedding, and blockwise inference with a weighted speaker-embedding buffer propagates identities across long recordings. This design avoids unsupervised clustering and permutation-invariant training while keeping speaker identities consistent across blocks.","core_discovery":"The central claim is that speaker detection and speaker representation can be jointly learned in one sequence-to-sequence architecture, so that a diarization system no longer needs an external enrollment or clustering stage. Two decoders share one encoder: a detection decoder maps speaker embeddings to voice activities, and a representation decoder maps voice activities back to speaker embeddings, with a learnable embedding matrix and ArcFace loss tying the two spaces together. During training, randomly masking one speaker's embedding teaches the model to route unknown speakers through a fixed pseudo-speaker embedding; during inference, a single pseudo-speaker slot lets the model enroll at most one new speaker per block shift, after which the buffer update feeds the next block. When the audio ends, the accumulated embeddings re-decode the whole signal for offline use. The paper reports that this design surpasses previous online and offline systems on the DIHARD-II and DIHARD-III evaluation sets.","pith_inferences":["If the one-new-speaker-per-block assumption fails in a recording—two previously unseen speakers first talk in the same chunk—the second speaker can only be enrolled later and may be missed entirely if their turn is short; a natural extension is to add multiple pseudo-speaker slots or a delayed enrollment pass.","The same inverse-decoder trick (voice activities as queries to extract embeddings) could transfer to other tasks that need to track an open set of identities, such as audio-visual diarization or meeting transcription with name enrollment.","Because the embedding space is learned jointly with detection rather than frozen from an external speaker-verification model, the framework's gains may depend on the diversity of the simulation corpus; the paper's distillation results suggest scale matters when the model is large enough to use it."],"forward_implications":["The same S2SND model, without any architectural change, serves as both an online streamer (blockwise, low latency) and an offline system (full re-scoring), so one deployment covers both use cases.","New best published DERs on DIHARD-II are reported: 24.41% online and 21.95% offline without oracle VAD, beating prior systems that sometimes rely on oracle VAD.","On DIHARD-III the model reaches 17.12% online and 15.13% offline, approaching the previous best offline systems while running online.","Speaker counting accuracy on DIHARD-III reaches 79.54% without oracle VAD, the highest among the systems the paper compares against.","The masked-speaker training recipe combined with knowledge distillation from a larger teacher extractor is what unlocks the gains for the medium model with the large VoxBlink2 corpus."],"supporting_citations":[{"why":"Provides the sequence-to-sequence TSVAD architecture that S2SND extends with detection and representation decoders.","marker":"[16]"},{"why":"EEND-GLA is the strongest online and offline EEND-based baseline the paper compares against on both DIHARD sets.","marker":"[27]"},{"why":"First online TSVAD framework; S2SND removes its reliance on an external VAD for new-speaker detection.","marker":"[29]"},{"why":"OTS-VAD is the online TSVAD baseline on DIHARD-III that S2SND outperforms.","marker":"[30]"},{"why":"Masked language modeling inspires the masked speaker prediction used to train the pseudo-speaker slot.","marker":"[72]"},{"why":"ArcFace loss supervises the representation decoder's extracted embeddings and shapes the joint embedding space.","marker":"[73]"},{"why":"VoxCeleb2 corpus supplies the training identities used for the pretraining-based S2SND models.","marker":"[74]"},{"why":"VoxBlink2 corpus supplies the large speaker identities used with knowledge distillation.","marker":"[75]"},{"why":"DIHARD-II evaluation set and protocol are the benchmark for the reported online and offline DERs.","marker":"[77]"},{"why":"DIHARD-III evaluation set and protocol are the benchmark for the reported online and offline DERs.","marker":"[78]"}],"fun_headline_variants":["One model learns to detect and represent speakers for diarization","Joint speaker detection and embedding in a single seq2seq network","End-to-end diarization without enrollment beats DIHARD baselines","Seq2seq diarization learns detection and representation together"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The online pipeline assumes that at most one new speaker can appear in any small block shift, because only a single pseudo-speaker embedding is available per block; if two unseen speakers start speaking in the same chunk, only one can be enrolled in that block and the other may be missed.","fun_headline_variants_meta":{"raw":{"variants":["One model learns to detect and represent speakers for diarization","Joint speaker detection and embedding in a single seq2seq network","End-to-end diarization without enrollment beats DIHARD baselines","Seq2seq diarization learns detection and representation together"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000505,"raw_usage":{"total_tokens":2488,"prompt_tokens":994,"completion_tokens":1494,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":1422}},"tokens_in":610,"tokens_out":1494,"duration_ms":10907,"temperature":1.0,"reasoning_tokens":1422,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:48:38.135608+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a test set of conversations where, by design, two previously unenrolled speakers begin speaking within the same chunk while all other conditions match DIHARD-III, and measure the online DER of S2SND; a large error spike on those recordings would confirm that the single-pseudo-speaker slot is the bottleneck. A simpler probe: run the model online on DIHARD-III but force the block shift to align with moments of simultaneous speaker onsets and compare DER against the reported 17.12%.","supporting_citations":[{"cited_title":"Target-speaker voice activity detection via sequence-to-sequence prediction,","cited_arxiv_id":null,"evidence_quote":"Provides the sequence-to-sequence TSVAD architecture that S2SND extends with detection and representation decoders."},{"cited_title":"Online neural diarization of unlimited numbers of speakers using global and local attractors,","cited_arxiv_id":null,"evidence_quote":"EEND-GLA is the strongest online and offline EEND-based baseline the paper compares against on both DIHARD sets."},{"cited_title":"Online target speaker voice activity detection for speaker diarization,","cited_arxiv_id":null,"evidence_quote":"First online TSVAD framework; S2SND removes its reliance on an external VAD for new-speaker detection."},{"cited_title":"Online neural speaker diarization with target speaker tracking,","cited_arxiv_id":null,"evidence_quote":"OTS-VAD is the online TSVAD baseline on DIHARD-III that S2SND outperforms."},{"cited_title":"BERT: pre-training of deep bidirectional transformers for language understanding,","cited_arxiv_id":null,"evidence_quote":"Masked language modeling inspires the masked speaker prediction used to train the pseudo-speaker slot."},{"cited_title":"V oxceleb2: Deep speaker recognition,","cited_arxiv_id":null,"evidence_quote":"VoxCeleb2 corpus supplies the training identities used for the pretraining-based S2SND models."},{"cited_title":"V oxblink2: A 100k+ speaker recognition corpus and the open-set speaker-identification benchmark,","cited_arxiv_id":null,"evidence_quote":"VoxBlink2 corpus supplies the large speaker identities used with knowledge distillation."},{"cited_title":"The third dihard diarization challenge,","cited_arxiv_id":null,"evidence_quote":"DIHARD-III evaluation set and protocol are the benchmark for the reported online and offline DERs."}],"review_version":1}