Pith. sign in

REVIEW 3 major objections 7 minor 6 cited by

Sequence-to-Sequence Neural Diarization with Automatic Speaker Detection and Representation

T0 review · 3 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read A single sequence-to-sequence network performs speaker diarization online and offline, discovering new speakers without a prior enrollment stage.

desk verdict Novel dual-decoder diarization architecture with strong but inflated SOTA numbers due to eval-set model selection. read the letter →

arxiv 2411.13849 v2 pith:BCVXU3JH submitted 2024-11-21 eess.AS

classification eess.AS
keywords speakerdiarizationonlinesequence-to-sequencetarget-speakervoiceactivitydetectionembeddingmaskedpredictionDIHARDblockwiseinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that speaker diarization—deciding who spoke when in a conversation—can be done by a single sequence-to-sequence network that both detects previously unseen speakers and learns their voice embeddings on the fly. The proposed S2SND framework processes audio block by block, maintaining a buffer of speaker embeddings from earlier blocks, and uses a learnable pseudo-speaker slot to catch any new voice that appears. Because it never needs a separate clustering step, permutation-invariant training, or pre-extracted enrollment from a prior diarization system, the same model can run online with low latency and then re-score the whole recording offline. On the DIHARD-II and DIHARD-III benchmarks the authors report new best diarization error rates in both modes without oracle voice activity detection: 24.41% and 17.12% online, 21.95% and 15.13% offline. If correct, this would collapse the usual separation between online and offline diarization systems.

What carries the argument

The core machinery is a pair of inverse sequence-to-sequence decoders with a shared encoder and a learnable speaker-embedding matrix. The detection decoder takes speaker embeddings (or the pseudo-speaker embedding for unknown voices) as auxiliary queries and predicts binary voice activities; the representation decoder takes those predicted voice activities as auxiliary queries and extracts speaker embeddings, trained by an ArcFace loss against the same embedding matrix. Masked speaker prediction during training randomly erases one speaker's embedding so the model learns to associate missing voices with the pseudo-speaker embedding, and blockwise inference with a weighted speaker-embedding buffer propagates identities across long recordings. This design avoids unsupervised clustering and permutation-invariant training while keeping speaker identities consistent across blocks.

What would settle it

Take a test set of conversations where, by design, two previously unenrolled speakers begin speaking within the same chunk while all other conditions match DIHARD-III, and measure the online DER of S2SND; a large error spike on those recordings would confirm that the single-pseudo-speaker slot is the bottleneck. A simpler probe: run the model online on DIHARD-III but force the block shift to align with moments of simultaneous speaker onsets and compare DER against the reported 17.12%.

Watch

Extended reading notes

Core claim

The central claim is that speaker detection and speaker representation can be jointly learned in one sequence-to-sequence architecture, so that a diarization system no longer needs an external enrollment or clustering stage. Two decoders share one encoder: a detection decoder maps speaker embeddings to voice activities, and a representation decoder maps voice activities back to speaker embeddings, with a learnable embedding matrix and ArcFace loss tying the two spaces together. During training, randomly masking one speaker's embedding teaches the model to route unknown speakers through a fixed pseudo-speaker embedding; during inference, a single pseudo-speaker slot lets the model enroll at most one new speaker per block shift, after which the buffer update feeds the next block. When the audio ends, the accumulated embeddings re-decode the whole signal for offline use. The paper reports that this design surpasses previous online and offline systems on the DIHARD-II and DIHARD-III evaluation sets.

Load-bearing premise

The online pipeline assumes that at most one new speaker can appear in any small block shift, because only a single pseudo-speaker embedding is available per block; if two unseen speakers start speaking in the same chunk, only one can be enrolled in that block and the other may be missed.

Editorial extensions

If this is right

  • The same S2SND model, without any architectural change, serves as both an online streamer (blockwise, low latency) and an offline system (full re-scoring), so one deployment covers both use cases.
  • New best published DERs on DIHARD-II are reported: 24.41% online and 21.95% offline without oracle VAD, beating prior systems that sometimes rely on oracle VAD.
  • On DIHARD-III the model reaches 17.12% online and 15.13% offline, approaching the previous best offline systems while running online.
  • Speaker counting accuracy on DIHARD-III reaches 79.54% without oracle VAD, the highest among the systems the paper compares against.
  • The masked-speaker training recipe combined with knowledge distillation from a larger teacher extractor is what unlocks the gains for the medium model with the large VoxBlink2 corpus.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the one-new-speaker-per-block assumption fails in a recording—two previously unseen speakers first talk in the same chunk—the second speaker can only be enrolled later and may be missed entirely if their turn is short; a natural extension is to add multiple pseudo-speaker slots or a delayed enrollment pass.
  • The same inverse-decoder trick (voice activities as queries to extract embeddings) could transfer to other tasks that need to track an open set of identities, such as audio-visual diarization or meeting transcription with name enrollment.
  • Because the embedding space is learned jointly with detection rather than frozen from an external speaker-verification model, the framework's gains may depend on the diversity of the simulation corpus; the paper's distillation results suggest scale matters when the model is large enough to use it.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes S2SND, a sequence-to-sequence neural diarization framework that performs online and offline speaker diarization with a single model. It augments a TSVAD-style encoder-decoder with two decoders: a detection decoder that predicts voice activities from partially given speaker embeddings, and a representation decoder that extracts speaker embeddings from predicted voice activities. A pseudo-speaker embedding is used to detect and enroll one unknown speaker per block, and a buffer stores the extracted embeddings for subsequent blocks. The model is trained with a masked-speaker prediction objective and an ArcFace loss on the representation branch, plus a knowledge-distillation strategy from a larger teacher extractor. Experiments on DIHARD-II and DIHARD-III report new state-of-the-art DERs: online without oracle VAD 24.41%/17.12%, offline without oracle VAD 21.95%/15.13%, with additional gains when oracle VAD post-processing is applied. The paper also reports speaker-counting accuracy and computing efficiency for the proposed models.

Significance. If the reported results are robust, the paper makes a significant contribution: it unifies online and offline diarization in a clustering-free, PIT-free architecture with automatic speaker enrollment, a property that previous TSVAD/EEND hybrid systems lack. The masked-speaker prediction and target-voice embedding extraction are novel and reasonably motivated, and the paper provides a detailed ablation study in Table II as well as comparisons on two standard benchmarks. The strengths include a clear description of the architecture, training objectives, and inference protocol, as well as a pseudocode specification of the online algorithm. However, the headline SOTA claim currently rests on model selection performed on the evaluation sets, which can materially inflate the reported gains, and there is a train/test mismatch in the representation decoder's inputs. These issues must be addressed before the central claim can be accepted.

major comments (3)
  1. [Sec. V-A, Table II, Tables III and IV] The headline SOTA numbers are selected on the evaluation sets themselves. Table II reports DERs for 24 configurations (S1-S24) on the DIHARD-II and DIHARD-III evaluation sets, and the text says 'We select the lowest online and offline DERs for each model size on DIHARD-II and DIHARD-III datasets as the representative results.' The configuration selection (model size, simulation corpus, training strategy, chunk length, right-context length) is therefore not independent of the test data. The only explicitly validation-tuned hyperparameters are the thresholds tau_1 and tau_2 (Sec. IV-C2). Because the spread across configurations is material (e.g., Medium online DIHARD-II ranges from 24.41% in S22 to 27.79% in S17; offline from 21.95% in S21 to 24.09% in S17), the gap between the evaluation-selected best and a validation-selected configuration could be several DER points, which is the same order as the claimed improvement over prior SOTA. Please repeat configuration selection on the validation splits (dev39/dev51) and report both the validation-selected configuration and its evaluation-set DER, or report all configurations on the validation sets so the reader can assess the optimism.
  2. [Sec. III-B2, Eq. (4), Sec. III-C2] The representation decoder is trained with oracle voice activities as auxiliary queries but deployed with predicted voice activities. In Sec. III-B2 the representation decoder 'utilizes extractor output X as feature embeddings and voice activities Y as auxiliary queries', where Y is ground truth, and the ArcFace loss in Eq. (4) is computed under this oracle-query condition. During inference (Sec. III-C2), the same decoder receives the predicted activities Y_hat as queries. The paper does not describe any mechanism to close this train/test gap, such as scheduled sampling, query dropout/noising, or a loss that also evaluates the representation branch on predicted activities. This exposure bias is particularly concerning for the pseudo-speaker channel, whose predicted activity is used both to enroll new speakers and to extract their embeddings, so errors can accumulate blockwise. Please either train the representation decoder with the detection decoder's outputs (with teacher forcing), or provide a validation-set analysis quantifying the effect of this mismatch.
  3. [Abstract, Sec. III-C2, Sec. III-C3] The online pipeline relies on the assumption that at most one new speaker appears in a small block shift, because there is only a single pseudo-speaker embedding slot. If two or more unseen speakers first appear in the same block, only one can be enrolled in that block; the others are delayed until later blocks and may be missed if their speech is short. This is a stated limitation, but the paper does not quantify how often the assumption is violated in the DIHARD-II/III evaluation sets, nor how violations affect DER. Please report the distribution of the number of new speakers per block on these evaluation sets and, if possible, compare with a variant that uses multiple pseudo-speaker slots or that delays enrollment decisions by one block, so that the robustness of the SOTA claim to this assumption can be assessed.
minor comments (7)
  1. [Sec. III-C2] The definition of the weight function W(hat y) uses the notation t' not in Overlap, but the set Overlap is not formally defined. Please specify how overlapping frames are determined from the predicted voice activities of all speakers.
  2. [Algorithm 1] The loop 'for n in range(1, len(S))' appears to use the embedding dimension S as the loop bound; the intended bound is likely the number of enrolled speaker slots (len(spk_list) or N). Please correct the pseudocode.
  3. [Sec. IV-C2] The text states that tau_1 and tau_2 are determined by grid search on the validation set, but it does not report the grid range or the resulting values. Please include these values for reproducibility.
  4. [Table II] The table caption says the lowest DERs of each model size are highlighted, but the selection is done on the evaluation set. In addition to the suggested validation-based selection, it would be helpful to mark the validation-best configuration separately in the table.
  5. [Sec. V-B] The sentence 'our best online DER (24.41%) is even lower than the previous best offline system (24.64%)' mixes online and offline conditions; the comparison is only valid under the no-oracle-VAD setting and with the selected configuration. Please clarify this phrasing.
  6. [Sec. III-C2] There is a typo: 'represnets' should be 'represents' in the sentence describing the pseudo-speaker output.
  7. [Sec. III-C3] The offline rescoring mechanism is described as re-decoding the audio with the final speaker-embedding buffer, but the paper does not specify whether the rescoring pass uses the same block-wise chunking with right-context or a full-sequence decode. Please clarify the inference setup for the offline mode.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the S2SND claims are empirical benchmark results with external comparisons; self-citations are architectural lineage and dataset sources, not load-bearing derivations.

full rationale

The central claim is an empirical SOTA benchmark result, not a derivation from an assumed target. The S2SND architecture is an extension of the authors' own Seq2Seq-TSVAD, but that lineage is not load-bearing support for the reported DER numbers: the numbers come from training on simulated data, adapting on DIHARD dev splits, grid-searching thresholds tau_1 and tau_2 on dev39/dev51, and comparing against external systems on DIHARD-II/III evaluation sets. The detection/representation decoder loop is an architectural feedback mechanism, not a logical definitional equivalence. Self-citations to [16], [55], and [75] provide architecture details, training-strategy choices, and a speech corpus, but none are used to forbid alternatives or to define the target metric into existence. The notable methodological caveat is that the 'best' configurations in Table II appear to be selected on the evaluation sets themselves, which is a test-set selection bias—an experimental-design/correctness risk, not a circularity. Because this concern concerns evaluation protocol rather than the derivation reducing to its own inputs, the circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 2 invented entities

The model's performance depends on a series of domain assumptions and tuning choices: the one-new-speaker-per-block assumption, the discriminability of voice activity for embedding extraction, the transferability of simulated training data, and the hand-set speaker capacity and buffer thresholds. No fundamental physical or mathematical axioms are introduced beyond standard neural network training assumptions.

free parameters (5)
  • tau_1 = grid-searched per dataset (exact value not reported)
    Threshold for the pseudo-speaker embedding weight; determines whether an unknown speaker is enrolled in the buffer (Sec IV-C2).
  • tau_2 = grid-searched per dataset (exact value not reported)
    Threshold for enrolled-speaker embedding weight; low-quality embeddings are discarded below it (Sec IV-C2).
  • speaker capacity N = 30
    Preset maximum number of speakers the model can handle simultaneously; chosen to cover most datasets (Sec IV-C1).
  • chunk and right-context lengths = 0.48s/0.64s chunk, 0/0.16s right context; best per dataset selected
    These sliding-window sizes trade latency against DER, and the best combination is picked from the evaluated grid for each benchmark (Table II).
  • ArcFace scale alpha and margin m = alpha=32, m=0.2
    Taken from prior speaker recognition practice; affect embedding discriminability but are not tuned for this task (Sec IV-B1).
assumptions (4)
  • domain assumption At most one new speaker may appear in a small block shift.
    The inference design allocates a single pseudo-speaker slot per block, so the online detector can enroll at most one unknown speaker per chunk (abstract and Sec III-C).
  • domain assumption Voice activity patterns are sufficiently discriminative to extract speaker embeddings, since single-speaker speech usually occupies most of the conversation.
    The representation decoder uses predicted voice activities as queries; the paper argues this works because non-overlapped speech dominates real data (Sec III-B2).
  • domain assumption The learnable embedding matrix E_all provides valid supervised targets for the representation decoder.
    Training uses rows of E_all as both the detection decoder input and the ArcFace targets; at inference E_all is discarded, assuming the learned space generalizes to unseen speakers (Sec III-B).
  • domain assumption Simulated mixtures of 1-3 speakers from VoxCeleb2/VoxBlink2 transfer to real DIHARD domains.
    The model is trained on on-the-fly simulated data and then adapted to DIHARD dev sets; good eval results are taken as evidence the simulation matches real conditions (Sec IV-A).
invented entities (2)
  • pseudo-speaker embedding e_pse independent evidence
    purpose: A learnable placeholder embedding that represents any unknown or unenrolled speaker in the detection decoder, enabling discovery of new speakers without prior enrollment.
    Its behavior is tested by the DIHARD results and the speaker-counting confusion matrices (Fig 7), which show the model can detect unseen speakers, though the embedding itself is a model parameter.
  • non-speech embedding e_non
    purpose: A learnable padding embedding used to fill unused speaker slots so that the decoder sees a fixed-capacity input.
    It is an internal padding token with no independent falsifiable handle; its effect is only observable through the overall diarization quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sequence-to-Sequence Neural Diarization with Automatic Speaker Detection and Representation." pith.science (2026). https://pith.science/paper/BCVXU3JH

@misc{pith2026241113849,
  author       = {Pith},
  title        = {Pith review of: Sequence-to-Sequence Neural Diarization with Automatic Speaker Detection and Representation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BCVXU3JH}},
  note         = {Machine review of arXiv:2411.13849}
}
read the original abstract

This paper proposes a novel Sequence-to-Sequence Neural Diarization (S2SND) framework to perform online and offline speaker diarization. It is developed from the sequence-to-sequence architecture of our previous target-speaker voice activity detection system and then evolves into a new diarization paradigm by addressing two critical problems. 1) Speaker Detection: The proposed approach can utilize partially given speaker embeddings to discover the unknown speaker and predict the target voice activities in the audio signal. It does not require a prior diarization system for speaker enrollment in advance. 2) Speaker Representation: The proposed approach can adopt the predicted voice activities as reference information to extract speaker embeddings from the audio signal simultaneously. The representation space of speaker embedding is jointly learned within the whole diarization network without using an extra speaker embedding model. During inference, the S2SND framework can process long audio recordings blockwise. The detection module utilizes the previously obtained speaker-embedding buffer to predict both enrolled and unknown speakers' voice activities for each coming audio block. Next, the speaker-embedding buffer is updated according to the predictions of the representation module. Assuming that up to one new speaker may appear in a small block shift, our model iteratively predicts the results of each block and extracts target embeddings for the subsequent blocks until the signal ends. Finally, the last speaker-embedding buffer can re-score the entire audio, achieving highly accurate diarization performance as an offline system. Experimental results show that ...

Figures

Figures reproduced from arXiv: 2411.13849 by the authors.

Figure 1
Figure 1. Overview of our speaker diarization frameworks from offline-only to online/offline scenarios: (a) Previous Sequence-to-Sequence Target-Speaker [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The Sequence-to-Sequence Neural Diarization (S2SND) framework. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The structure of the modified Speaker-wise Decoder. For clarity, the [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The inference diagram of the Sequence-to-Sequence Neural Diarization (S2SND) framework. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Updating strategy of the speaker-embedding buffer. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Illustration of the training strategy based on knowledge distillation. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Confusion matrices for speaker counting on the DIHARD-III evaluation set. The Pyannote.audio v3.1, VBx, and DiaPer results are provided by their [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Streaming Sortformer: Speaker Cache-Based Online Speaker Diarization with Arrival-Time Ordering

    eess.AS 2025-07 conditional novelty 6.0 of 10

    A streaming Sortformer with an arrival-ordered speaker cache achieves lower diarization error than prior online systems on DIHARD III and CALLHOME, even at 0.32 second latency.

  2. Diarization-Aware Multi-Speaker Automatic Speech Recognition via Large Language Models

    eess.AS 2025-06 conditional novelty 6.0 of 10

    An LLM conditioned on speaker embeddings and utterance time boundaries jointly transcribes and timestamps overlapping multi-speaker speech.

  3. Dissecting the Segmentation Model of End-to-End Diarization with Vector Clustering

    cs.SD 2025-06 conditional novelty 5.0 of 10

    A controlled 120-configuration study of EEND-VC speaker diarization finds finetuned WavLM encoders, Conformer/Mamba decoders, and longer chunks give the biggest gains, with the best system reaching state-of-the-art on...

  4. The Multimodal Information Based Speech Processing (MISP) 2025 Challenge: Audio-Visual Diarization and Recognition

    cs.SD 2025-05 conditional novelty 5.0 of 10

    The top MISP 2025 systems achieve DER 8.09%, CER 9.48%, and cpCER 11.56%, far outperforming the provided audio-visual baselines.

  5. The DKU System for Multi-Speaker Automatic Speech Recognition in MLC-SLM Challenge

    eess.AS 2025-07 conditional novelty 4.0 of 10

    A challenge system combining speaker diarization, speaker embeddings, and a Qwen2.5 LLM adapter architecture reports 18.08% tcpWER on multilingual multi-speaker ASR, far below the 60.39% baseline.

  6. Multi-Channel Sequence-to-Sequence Neural Diarization: Experimental Results for The MISP 2025 Challenge

    eess.AS 2025-05 conditional novelty 4.0 of 10

    Extending S2SND with a channel-attention module for multi-channel audio achieves an 8.09% diarization error rate, first place in the MISP 2025 speaker diarization task.

Reference graph

Works this paper leans on

90 extracted references · 63 canonical work pages · cited by 6 Pith papers

  1. [1]

    A review of speaker diarization: Recent advances with deep learning,

    T. J. Park, N. Kanda, D. Dimitriadis, K. J. Han, S. Watanabe, and S. Narayanan, “A review of speaker diarization: Recent advances with deep learning,”Computer Speech & Language, vol. 72, p. 101317, 2022

  2. [2]

    Joint speaker counting, speech recognition, and speaker identification for overlapped speech of any number of speakers,

    N. Kanda, Y . Gaur, X. Wang, Z. Meng, Z. Chen, T. Zhou, and T. Yoshioka, “Joint speaker counting, speech recognition, and speaker identification for overlapped speech of any number of speakers,” inProc. INTERSPEECH, 2020, pp. 36–40

  3. [3]

    Unsupervised methods for speaker diarization: An integrated and iterative approach,

    S. H. Shum, N. Dehak, R. Dehak, and J. R. Glass, “Unsupervised methods for speaker diarization: An integrated and iterative approach,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 21, no. 10, pp. 2015–2028, 2013

  4. [4]

    A study of the cosine distance-based mean shift for telephone speech diarization,

    M. Senoussaoui, P. Kenny, T. Stafylakis, and P. Dumouchel, “A study of the cosine distance-based mean shift for telephone speech diarization,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 22, no. 1, pp. 217–227, 2014

  5. [5]

    Speaker diarization with plda i-vector scoring and unsupervised calibration,

    G. Sell and D. Garcia-Romero, “Speaker diarization with plda i-vector scoring and unsupervised calibration,” inProc. SLT, 2014, pp. 413–417

  6. [6]

    Speaker diarization with lstm,

    Q. Wang, C. Downey, L. Wan, P. A. Mansfield, and I. L. Moreno, “Speaker diarization with lstm,” inProc. ICASSP, 2018, pp. 5239–5243

  7. [7]

    Lstm based similarity measurement with spectral clustering for speaker diarization,

    Q. Lin, R. Yin, M. Li, H. Bredin, and C. Barras, “Lstm based similarity measurement with spectral clustering for speaker diarization,” inProc. INTERSPEECH, 2019, pp. 366–370

  8. [8]

    Bayesian hmm clustering of x-vector sequences (vbx) in speaker diarization: Theory, implemen- tation and analysis on standard tasks,

    F. Landini, J. Profant, M. Diez, and L. Burget, “Bayesian hmm clustering of x-vector sequences (vbx) in speaker diarization: Theory, implemen- tation and analysis on standard tasks,”Computer Speech & Language, vol. 71, p. 101254, 2022

Show all 90 references
  1. [9]

    Similarity measurement of segment-level speaker embeddings in speaker diarization,

    W. Wang, Q. Lin, D. Cai, and M. Li, “Similarity measurement of segment-level speaker embeddings in speaker diarization,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 2645–2658, 2022

  2. [10]

    End-to-end neural speaker diarization with permutation-free objec- tives,

    Y . Fujita, N. Kanda, S. Horiguchi, K. Nagamatsu, and S. Watanabe, “End-to-end neural speaker diarization with permutation-free objec- tives,” inProc. INTERSPEECH, 2019, pp. 4300–4304

  3. [11]

    End-to-end neural speaker diarization with self-attention,

    Y . Fujita, N. Kanda, S. Horiguchi, Y . Xue, K. Nagamatsu, and S. Watan- abe, “End-to-end neural speaker diarization with self-attention,” inProc. ASRU, 2019, pp. 296–303

  4. [12]

    End- to-end speaker diarization for an unknown number of speakers with encoder-decoder based attractors,

    S. Horiguchi, Y . Fujita, S. Watanabe, Y . Xue, and K. Nagamatsu, “End- to-end speaker diarization for an unknown number of speakers with encoder-decoder based attractors,” inProc. INTERSPEECH, 2020, pp. 269–273

  5. [13]

    Encoder- decoder based attractors for end-to-end neural diarization,

    S. Horiguchi, Y . Fujita, S. Watanabe, Y . Xue, and P. Garc ´ıa, “Encoder- decoder based attractors for end-to-end neural diarization,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 1493–1507, 2022

  6. [14]

    Multitalker speech separation with utterance-level permutation invariant training of deep recurrent neural networks,

    M. Kolbæk, D. Yu, Z.-H. Tan, and J. Jensen, “Multitalker speech separation with utterance-level permutation invariant training of deep recurrent neural networks,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 10, pp. 1901–1913, 2017

  7. [15]

    Target-speaker voice activity detection: A novel approach for multi-speaker diarization in a dinner party scenario,

    I. Medennikov, M. Korenevsky, T. Prisyach, Y . Khokhlov, M. Ko- renevskaya, I. Sorokin, T. Timofeeva, A. Mitrofanov, A. Andrusenko, I. Podluzhny, A. Laptev, and A. Romanenko, “Target-speaker voice activity detection: A novel approach for multi-speaker diarization in a dinner p...

  8. [16]

    Target-speaker voice activity detection via sequence-to-sequence prediction,

    M. Cheng, W. Wang, Y . Zhang, X. Qin, and M. Li, “Target-speaker voice activity detection via sequence-to-sequence prediction,” inProc. ICASSP, 2023, pp. 1–5

  9. [17]

    Ustc-nelslip system description for dihard-iii challenge,

    Y . Wang, M. He, S. Niu, L. Sun, T. Gao, X. Fang, J. Pan, J. Du, and C.- H. Lee, “Ustc-nelslip system description for dihard-iii challenge,”arXiv preprint arXiv:2103.10661, 2021

  10. [18]

    The dku-dukeece-lenovo system for the diarization task of the 2021 voxceleb speaker recognition challenge,

    W. Wang, D. Cai, Q. Lin, L. Yang, J. Wang, J. Wang, and M. Li, “The dku-dukeece-lenovo system for the diarization task of the 2021 voxceleb speaker recognition challenge,”arXiv preprint arXiv:2109.02002, 2021

  11. [19]

    The dku-dukeece diarization system for the voxceleb speaker recognition challenge 2022,

    W. Wang, X. Qin, M. Cheng, Y . Zhang, K. Wang, and M. Li, “The dku-dukeece diarization system for the voxceleb speaker recognition challenge 2022,”arXiv preprint arXiv:2210.01677, 2022

  12. [20]

    The dku-msxf diarization system for the voxceleb speaker recognition challenge 2023,

    M. Cheng, W. Wang, X. Qin, Y . Lin, N. Jiang, G. Zhao, and M. Li, “The dku-msxf diarization system for the voxceleb speaker recognition challenge 2023,” inProc. NCMMSC, J. Jia, Z. Ling, X. Chen, Y . Li, and Z. Zhang, Eds. Springer Nature Singapore, 2024, pp. 330–337

  13. [21]

    The voxceleb speaker recognition chal- lenge: A retrospective,

    J. Huh, J. S. Chung, A. Nagrani, A. Brown, J.-w. Jung, D. Garcia- Romero, and A. Zisserman, “The voxceleb speaker recognition chal- lenge: A retrospective,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 3850–3866, 2024

  14. [22]

    Developing on-line speaker diarization system,

    D. Dimitriadis and P. Fousek, “Developing on-line speaker diarization system,” inProc. INTERSPEECH, 2017, pp. 2739–2743

  15. [23]

    Low-latency online speaker diarization with graph-based label genera- tion,

    Y . Zhang, Q. Lin, W. Wang, L. Yang, X. Wang, J. Wang, and M. Li, “Low-latency online speaker diarization with graph-based label genera- tion,” inProc. Odyssey, 2022, pp. 162–169

  16. [24]

    Bw-eda-eend: streaming end-to-end neural speaker diarization for a variable number of speakers,

    E. Han, C. Lee, and A. Stolcke, “Bw-eda-eend: streaming end-to-end neural speaker diarization for a variable number of speakers,” inProc. ICASSP, 2021, pp. 7193–7197

  17. [25]

    Online end-to-end neural diarization with speaker-tracing buffer,

    Y . Xue, S. Horiguchi, Y . Fujita, S. Watanabe, P. Garc ´ıa, and K. Naga- matsu, “Online end-to-end neural diarization with speaker-tracing buffer,” inProc. SLT, 2021, pp. 841–848

  18. [26]

    Online streaming end-to-end neural diariza- tion handling overlapping speech and flexible numbers of speakers,

    Y . Xue, S. Horiguchi, Y . Fujita, Y . Takashima, S. Watanabe, L. P. G. Perera, and K. Nagamatsu, “Online streaming end-to-end neural diariza- tion handling overlapping speech and flexible numbers of speakers,” in Proc. INTERSPEECH, 2021, pp. 3116–3120

  19. [27]

    Online neural diarization of unlimited numbers of speakers using global and local attractors,

    S. Horiguchi, S. Watanabe, P. Garc ´ıa, Y . Takashima, and Y . Kawaguchi, “Online neural diarization of unlimited numbers of speakers using global and local attractors,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 706–720, 2023

  20. [28]

    Frame-wise streaming end-to-end speaker diarization with non-autoregressive self-attention-based attractors,

    D. Liang, N. Shao, and X. Li, “Frame-wise streaming end-to-end speaker diarization with non-autoregressive self-attention-based attractors,” in Proc. ICASSP, 2024, pp. 10 521–10 525

  21. [29]

    Online target speaker voice activity detection for speaker diarization,

    W. Wang, M. Li, and Q. Lin, “Online target speaker voice activity detection for speaker diarization,” inProc. INTERSPEECH, 2022, pp. 1441–1445

  22. [30]

    Online neural speaker diarization with target speaker tracking,

    W. Wang and M. Li, “Online neural speaker diarization with target speaker tracking,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024

  23. [31]

    Temporal modeling using dilated convolution and gating for voice-activity-detection,

    S.-Y . Chang, B. Li, G. Simko, T. N. Sainath, A. Tripathi, A. van den Oord, and O. Vinyals, “Temporal modeling using dilated convolution and gating for voice-activity-detection,” inProc. ICASSP, 2018, pp. 5549– 5553

  24. [32]

    Convolutional neural network for speaker change detection in telephone speaker diarization system,

    M. Hr ´uz and Z. Zaj´ıc, “Convolutional neural network for speaker change detection in telephone speaker diarization system,” inProc. ICASSP, 2017, pp. 4945–4949

  25. [33]

    Diarization is hard: Some experiences and lessons learned for the jhu team in the inaugural dihard challenge,

    G. Sell, D. Snyder, A. McCree, D. Garcia-Romero, J. Villalba, M. Ma- ciejewski, V . Manohar, N. Dehak, D. Povey, S. Watanabe, and S. Khu- danpur, “Diarization is hard: Some experiences and lessons learned for the jhu team in the inaugural dihard challenge,” inProc. INTERSPEECH...

  26. [34]

    Front- end factor analysis for speaker verification,

    N. Dehak, P. J. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet, “Front- end factor analysis for speaker verification,”IEEE Transactions on Audio, Speech, and Language Processing, vol. 19, no. 4, pp. 788–798, 2011. 15

  27. [35]

    X-vectors: Robust dnn embeddings for speaker recognition,

    D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, “X-vectors: Robust dnn embeddings for speaker recognition,” inProc. ICASSP, 2018, pp. 5329–5333

  28. [36]

    But system for the second dihard speech diarization challenge,

    F. Landini, S. Wang, M. Diez, L. Burget, P. Mat ˇejka, K. ˇZmol´ıkov´a, L. Moˇsner, A. Silnova, O. Plchot, O. Novotn´y, H. Zeinali, and J. Rohdin, “But system for the second dihard speech diarization challenge,” inProc. ICASSP, 2020, pp. 6529–6533

  29. [37]

    End-to-end speaker segmentation for overlap-aware resegmentation,

    H. Bredin and A. Laurent, “End-to-end speaker segmentation for overlap-aware resegmentation,” inProc. INTERSPEECH, 2021, pp. 3111–3115

  30. [38]

    End-to-end speaker diarization conditioned on speech activity and overlap detection,

    Y . Takashima, Y . Fujita, S. Watanabe, S. Horiguchi, P. Garc ´ıa, and K. Nagamatsu, “End-to-end speaker diarization conditioned on speech activity and overlap detection,” inProc. SLT, 2021, pp. 849–856

  31. [39]

    Integrating end-to-end neural and clustering-based diarization: Getting the best of both worlds,

    K. Kinoshita, M. Delcroix, and N. Tawara, “Integrating end-to-end neural and clustering-based diarization: Getting the best of both worlds,” inProc. ICASSP, 2021, pp. 7198–7202

  32. [40]

    Advances in integration of end-to-end neural and clustering-based diarization for real conversational speech,

    ——, “Advances in integration of end-to-end neural and clustering-based diarization for real conversational speech,” inProc. INTERSPEECH, 2021, pp. 3565–3569

  33. [41]

    Tight integration of neural- and clustering-based diarization through deep unfolding of infinite gaussian mixture model,

    K. Kinoshita, M. Delcroix, and T. Iwata, “Tight integration of neural- and clustering-based diarization through deep unfolding of infinite gaussian mixture model,” inProc. ICASSP, 2022, pp. 8382–8386

  34. [42]

    Towards neural diarization for unlimited numbers of speakers using global and local attractors,

    S. Horiguchi, S. Watanabe, P. Garc ´ıa, Y . Xue, Y . Takashima, and Y . Kawaguchi, “Towards neural diarization for unlimited numbers of speakers using global and local attractors,” inProc. ASRU, 2021, pp. 98–105

  35. [43]

    End-to-end neu- ral speaker diarization with an iterative refinement of non-autoregressive attention-based attractors,

    M. Rybicka, J. Villalba, N. Dehak, and K. Kowalczyk, “End-to-end neu- ral speaker diarization with an iterative refinement of non-autoregressive attention-based attractors,” inProc. INTERSPEECH, 2022, pp. 5090– 5094

  36. [44]

    Neural diarization with non-autoregressive intermediate attractors,

    Y . Fujita, T. Komatsu, R. Scheibler, Y . Kida, and T. Ogawa, “Neural diarization with non-autoregressive intermediate attractors,” inProc. ICASSP, 2023, pp. 1–5

  37. [45]

    Diaper: End- to-end neural diarization with perceiver-based attractors,

    F. Landini, M. Diez, T. Stafylakis, and L. Burget, “Diaper: End- to-end neural diarization with perceiver-based attractors,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 3450–3465, 2024

  38. [46]

    Improving transformer-based end-to-end speaker diarization by assigning auxiliary losses to attention heads,

    Y .-R. Jeoung, J.-Y . Yang, J.-H. Choi, and J.-H. Chang, “Improving transformer-based end-to-end speaker diarization by assigning auxiliary losses to attention heads,” inProc. ICASSP, 2023, pp. 1–5

  39. [47]

    Improving neural diarization through speaker attribute attractors and local dependency modeling,

    D. Palzer, M. Maciejewski, and E. Fosler-Lussier, “Improving neural diarization through speaker attribute attractors and local dependency modeling,” inProc. ICASSP, 2024, pp. 11 911–11 915

  40. [48]

    Self-supervised speaker diarization,

    Y . Dissen, F. Kreuk, and J. Keshet, “Self-supervised speaker diarization,” inProc. INTERSPEECH, 2022, pp. 4013–4017

  41. [49]

    Semi-supervised training with pseudo-labeling for end- to-end neural diarization,

    Y . Takashima, Y . Fujita, S. Horiguchi, S. Watanabe, L. P. G. Perera, and K. Nagamatsu, “Semi-supervised training with pseudo-labeling for end- to-end neural diarization,” inProc. INTERSPEECH, 2021, pp. 3096– 3100

  42. [50]

    Target- speaker voice activity detection with improved i-vector estimation for unknown number of speaker,

    M. He, D. Raj, Z. Huang, J. Du, Z. Chen, and S. Watanabe, “Target- speaker voice activity detection with improved i-vector estimation for unknown number of speaker,” inProc. INTERSPEECH, 2021, pp. 3555– 3559

  43. [51]

    Multi-target extractor and detector for unknown-number speaker diarization,

    C.-Y . Cheng, H.-S. Lee, Y . Tsao, and H.-M. Wang, “Multi-target extractor and detector for unknown-number speaker diarization,”IEEE Signal Processing Letters, vol. 30, pp. 638–642, 2023

  44. [52]

    Target speaker voice activity detection with transformers and its integration with end- to-end neural diarization,

    D. Wang, X. Xiao, N. Kanda, T. Yoshioka, and J. Wu, “Target speaker voice activity detection with transformers and its integration with end- to-end neural diarization,” inProc. ICASSP, 2023, pp. 1–5

  45. [53]

    Cross-channel attention-based target speaker voice activity detection: Experimental results for the m2met challenge,

    W. Wang, X. Qin, and M. Li, “Cross-channel attention-based target speaker voice activity detection: Experimental results for the m2met challenge,” inProc. ICASSP, 2022, pp. 9171–9175

  46. [54]

    The whu-alibaba audio-visual speaker diarization system for the misp 2022 challenge,

    M. Cheng, H. Wang, Z. Wang, Q. Fu, and M. Li, “The whu-alibaba audio-visual speaker diarization system for the misp 2022 challenge,” inProc. ICASSP, 2023, pp. 1–2

  47. [55]

    Multi-input multi-output target-speaker voice activity detection for unified, flexible, and robust audio-visual speaker diarization,

    M. Cheng and M. Li, “Multi-input multi-output target-speaker voice activity detection for unified, flexible, and robust audio-visual speaker diarization,”arXiv preprint arXiv:2401.08052, 2024

  48. [56]

    Target speech diarization with multimodal prompts,

    Y . Jiang, R. Tao, Z. Chen, Y . Qian, and H. Li, “Target speech diarization with multimodal prompts,”arXiv preprint arXiv:2406.07198, 2024

  49. [57]

    Joint inference of speaker diarization and asr with multi-stage information sharing,

    W. Wang, D. Cai, M. Cheng, and M. Li, “Joint inference of speaker diarization and asr with multi-stage information sharing,” inProc. ICASSP, 2024, pp. 11 011–11 015

  50. [58]

    Flow-tsvad: Target- speaker voice activity detection via latent flow matching,

    Z. Chen, B. Han, S. Wang, Y . Jiang, and Y . Qian, “Flow-tsvad: Target- speaker voice activity detection via latent flow matching,”arXiv preprint arXiv:2409.04859, 2024

  51. [59]

    Guided speaker embedding,

    S. Horiguchi, T. Moriya, A. Ando, T. Ashihara, H. Sato, N. Tawara, and M. Delcroix, “Guided speaker embedding,” inProc. ICASSP, 2025, pp. 1–5

  52. [60]

    Ansd-ma-mse: Adaptive neural speaker diarization using memory-aware multi-speaker embed- ding,

    M.-K. He, J. Du, Q.-F. Liu, and C.-H. Lee, “Ansd-ma-mse: Adaptive neural speaker diarization using memory-aware multi-speaker embed- ding,”IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, vol. 31, pp. 1561–1573, 2023

  53. [61]

    Fully supervised speaker diarization,

    A. Zhang, Q. Wang, Z. Zhu, J. Paisley, and C. Wang, “Fully supervised speaker diarization,” inProc. ICASSP, 2019, pp. 6301–6305

  54. [62]

    Supervised online diarization with sample mean loss for multi-domain data,

    E. Fini and A. Brutti, “Supervised online diarization with sample mean loss for multi-domain data,” inProc. ICASSP, 2020, pp. 7134–7138

  55. [63]

    Overlap-aware low-latency online speaker diarization based on end-to-end local seg- mentation,

    J. M. Coria, H. Bredin, S. Ghannay, and S. Rosset, “Overlap-aware low-latency online speaker diarization based on end-to-end local seg- mentation,” inProc. ASRU, 2021, pp. 1139–1146

  56. [64]

    Probabilistic back-ends for online speaker recognition and clustering,

    A. Sholokhov, N. Kuzmin, K. A. Lee, and E. S. Chng, “Probabilistic back-ends for online speaker recognition and clustering,” inProc. ICASSP, 2023, pp. 1–5

  57. [65]

    Interrelate training and clustering for online speaker diarization,

    Y . Chen, G. Cheng, R. Yang, P. Zhang, and Y . Yan, “Interrelate training and clustering for online speaker diarization,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 1352–1364, 2024

  58. [66]

    Transformer-XL: Attentive language models beyond a fixed-length context,

    Z. Dai, Z. Yang, Y . Yang, J. Carbonell, Q. Le, and R. Salakhutdinov, “Transformer-XL: Attentive language models beyond a fixed-length context,” inProc. ACL. Association for Computational Linguistics, 2019, pp. 2978–2988

  59. [67]

    Enhancing low-latency speaker diarization with spatial dictionary learning,

    W. Chen, T. T. Anh, X. Zhong, and E. S. Chng, “Enhancing low-latency speaker diarization with spatial dictionary learning,” inProc. ICASSP, 2024, pp. 11 371–11 375

  60. [68]

    Overview of speaker modeling and its applications: From the lens of deep speaker representation learning,

    S. Wang, Z. Chen, K. A. Lee, Y . Qian, and H. Li, “Overview of speaker modeling and its applications: From the lens of deep speaker representation learning,”arXiv preprint arXiv:2407.15188, 2024

  61. [69]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProc. CVPR, 2016

  62. [70]

    Conformer: Convolution- augmented transformer for speech recognition,

    A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu, and R. Pang, “Conformer: Convolution- augmented transformer for speech recognition,” inProc. INTER- SPEECH, 2020, pp. 5036–5040

  63. [71]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” inProc. NeurIPS, vol. 30, 2017

  64. [72]

    BERT: pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” inProc. NAACL, 2019, pp. 4171–4186

  65. [73]

    Arcface: Additive angular margin loss for deep face recognition,

    J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “Arcface: Additive angular margin loss for deep face recognition,” inProc. CVPR, 2019

  66. [74]

    V oxceleb2: Deep speaker recognition,

    J. S. Chung, A. Nagrani, and A. Zisserman, “V oxceleb2: Deep speaker recognition,” inProc. INTERSPEECH, 2018, pp. 1086–1090

  67. [75]

    V oxblink2: A 100k+ speaker recognition corpus and the open-set speaker-identification benchmark,

    Y . Lin, M. Cheng, F. Zhang, Y . Gao, S. Zhang, and M. Li, “V oxblink2: A 100k+ speaker recognition corpus and the open-set speaker-identification benchmark,” inProc. INTERSPEECH, 2024, pp. 4263–4267

  68. [76]

    Funasr: A fundamental end-to-end speech recognition toolkit,

    Z. Gao, Z. Li, J. Wang, H. Luo, X. Shi, M. Chen, Y . Li, L. Zuo, Z. Du, and S. Zhang, “Funasr: A fundamental end-to-end speech recognition toolkit,” inProc. INTERSPEECH, 2023, pp. 1593–1597

  69. [77]

    The second dihard diarization challenge: Dataset, task, and baselines,

    N. Ryant, K. Church, C. Cieri, A. Cristia, J. Du, S. Ganapathy, and M. Liberman, “The second dihard diarization challenge: Dataset, task, and baselines,” inProc. INTERSPEECH, 2019, pp. 978–982

  70. [78]

    The third dihard diarization challenge,

    N. Ryant, P. Singh, V . Krishnamohan, R. Varma, K. Church, C. Cieri, J. Du, S. Ganapathy, and M. Liberman, “The third dihard diarization challenge,” inProc. INTERSPEECH, 2021, pp. 3570–3574

  71. [79]

    V oxceleb: A large- scale speaker identification dataset,

    A. Nagrani, J. S. Chung, and A. Zisserman, “V oxceleb: A large- scale speaker identification dataset,” inProc. INTERSPEECH, 2017, pp. 2616–2620

  72. [80]

    Distilling the knowledge in a neural network,

    G. Hinton, “Distilling the knowledge in a neural network,”arXiv preprint arXiv:1503.02531, 2015

  73. [81]

    Musan: A music, speech, and noise corpus,

    D. Snyder, G. Chen, and D. Povey, “Musan: A music, speech, and noise corpus,”arXiv preprint arXiv:1510.08484, 2015

  74. [82]

    A study on data augmentation of reverberant speech for robust speech recognition,

    T. Ko, V . Peddinti, D. Povey, M. L. Seltzer, and S. Khudanpur, “A study on data augmentation of reverberant speech for robust speech recognition,” inProc. ICASSP, 2017, pp. 5220–5224

  75. [83]

    Decoupled weight decay regularization,

    I. Loshchilov, “Decoupled weight decay regularization,”arXiv preprint arXiv:1711.05101, 2017

  76. [84]

    Online speaker diarization with core samples selection,

    Y . Yue, J. Du, M.-K. He, Y . Yeung, and R. Wang, “Online speaker diarization with core samples selection,” inProc. INTERSPEECH, 2022, pp. 1466–1470

  77. [85]

    Absolute decision corrupts absolutely: Conservative online speaker diarisation,

    Y . Kwon, H.-S. Heo, B.-J. Lee, Y . J. Kim, and J.-W. Jung, “Absolute decision corrupts absolutely: Conservative online speaker diarisation,” inProc. ICASSP, 2023, pp. 1–5. 16

  78. [86]

    End-to-end speaker diarization as post-processing,

    S. Horiguchi, P. Garc ´ıa, Y . Fujita, S. Watanabe, and K. Nagamatsu, “End-to-end speaker diarization as post-processing,” inProc. ICASSP, 2021, pp. 7188–7192

  79. [87]

    Attention-based encoder- decoder end-to-end neural diarization with embedding enhancer,

    Z. Chen, B. Han, S. Wang, and Y . Qian, “Attention-based encoder- decoder end-to-end neural diarization with embedding enhancer,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 1636–1649, 2024

  80. [88]

    Powerset multi-class cross entropy loss for neural speaker diarization,

    A. Plaquet and H. Bredin, “Powerset multi-class cross entropy loss for neural speaker diarization,” inProc. INTERSPEECH, 2023, pp. 3222– 3226

  81. [89]

    Eend-m2f: Masked- attention mask transformers for speaker diarization,

    M. H ¨ark¨onen, S. J. Broughton, and L. Samarakoon, “Eend-m2f: Masked- attention mask transformers for speaker diarization,” inProc. INTER- SPEECH, 2024, pp. 37–41

  82. [90]

    The hitachi- jhu dihard iii system: Competitive end-to-end neural diarization and x-vector clustering systems combined by dover-lap,

    S. Horiguchi, N. Yalta, P. Garcia, Y . Takashima, Y . Xue, D. Raj, Z. Huang, Y . Fujita, S. Watanabe, and S. Khudanpur, “The hitachi- jhu dihard iii system: Competitive end-to-end neural diarization and x-vector clustering systems combined by dover-lap,”arXiv preprint arXiv:21...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.