Pith. sign in

REVIEW 3 major objections 5 minor 27 references

Cross-attention and Self-attention for Audio-visual Speaker Diarization in MISP-Meeting Challenge

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read CASA-Net fuses cross-attention and self-attention to cut audio-visual speaker diarization error on the MISP 2025 evaluation set to 8.18%, a 47.3% relative improvement over the 15.52% baseline.

desk verdict Solid challenge-system writeup with a clean ablation, but the Oracle-VAD ambiguity undermines the headline 8.18% vs baseline comparison until clarified. read the letter →

arxiv 2506.02621 v1 pith:OREBVW7C submitted 2025-06-03 cs.SD

classification cs.SD
keywords audio-visualspeakerdiarizationMISP2025challengecross-attentionself-attentionCASA-Netpseudo-labelrefinementMixupaugmentationerrorrate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper describes a system for Task 1 of the MISP 2025 Challenge, where the goal is audio-visual speaker diarization: deciding who spoke when in multi-speaker meetings using both audio and lip-image video. The authors claim that their CASA-Net architecture, which uses cross-attention to align audio and visual frame embeddings and self-attention to model temporal context, reaches an 8.18% diarization error rate (DER) on the evaluation set, a 47.3% relative improvement over the official 15.52% baseline. They further claim this result comes from combining CASA-Net with an ECAPA-TDNN x-vector speaker embedding, pseudo-label refinement and retraining, Mixup augmentation, and median-filter and overlap-averaging post-processing. A sympathetic reading is that the paper is trying to show that explicit cross-modal attention is a practical improvement over simple concatenation for audio-visual diarization, not just a theoretical option.

What carries the argument

The central object is CASA-Net, a two-stage attention fusion block. In the cross-attention stage, concatenated audio and i-vector/x-vector embeddings $F_a$ serve as key and value while visual lip embeddings $F_v$ serve as query, and symmetrically in the other direction, producing $F_{a\to v}$ and $F_{v\to a}$ via scaled dot-product attention; the self-attention stage then runs over the concatenation of the two cross-attended streams to capture temporal context across frames. This attention-based fusion replaces the plain concatenation used in the baseline decoder.

What would settle it

Disable the Oracle VAD, replace it with a separately trained voice-activity detector, and recompute DER on the MISP 2025 development and evaluation sets; if the error rises substantially above 8.18% or approaches the 15.52% baseline, the headline improvement depends on oracle speech-activity information. Also rerun the official baseline with the same Oracle VAD enabled to confirm both systems are compared under identical conditions.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that a cross-attention and self-attention fusion network, applied to visual voice-activity-detection embeddings, audio filterbank features, and x-vector speaker embeddings, reliably improves diarization timestamps in noisy overlapped meetings. The reported numbers support this: the full system achieves 7.35% DER on development and 8.18% on evaluation; ablations show removing CASA-Net raises evaluation DER to 17.04%, removing pseudo-label refinement to 10.41%, removing Mixup to 9.71%, and removing post-processing to 9.03%. The authors interpret this as evidence that each component contributes, with the attention fusion providing the largest single gain.

Load-bearing premise

The load-bearing premise is that the reported 8.18% diarization error rate is a fair measure of an autonomous system; the evaluation permitted use of a perfect speech-activity detector (Oracle VAD), and the paper does not quantify how much of the gain depends on it.

Editorial extensions

If this is right

  • If the reported numbers hold, attention-based fusion is a direct replacement for concatenation in end-to-end audio-visual diarization pipelines, with a large accuracy gain on the MISP 2025 task.
  • The pseudo-label refinement loop means that a system can use its own visual-VAD-derived timestamps to retrain the frame-level network, reducing train-test mismatch in speaker-embedding features.
  • Mixup on lip images and speaker embeddings, together with median filtering and overlap averaging on predictions, are presented as cheap additions that each yield roughly 0.6 to 0.9 percentage points of absolute DER improvement.
  • The paper's strongest claim is that the complete system matches the state-of-the-art performance on the leaderboard at 8.18% DER; if accepted, it sets a new target for subsequent audio-visual speaker diarization challenge entries.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the evaluation protocol allowed use of an Oracle VAD, and the paper does not report a fully automatic DER; the 8.18% number may include a component that an autonomous system would need to estimate.
  • Beyond the paper: the ablation table compares the full system against the official baseline, but the system differs from the baseline in several ways (visual encoder, audio encoder, x-vector vs. i-vector, training data, post-processing). The 47.3% improvement should therefore be read as the gain of the whole package, not of CASA-Net alone.
  • Beyond the paper: a natural extension would be to test the same cross-attention and self-attention fusion on audio-only diarization or on target-speaker voice activity detection, to see whether the gains transfer beyond the MISP corpus.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper describes the authors' system for Task 1 of the MISP 2025 Challenge, which performs audio-visual speaker diarization (AVSD). The proposed system builds on their prior CASA-Net architecture, using cross-attention between audio features (FBank plus an ECAPA-TDNN x-vector embedding) and visual features (V-VAD lip-region embeddings), followed by self-attention, and then a decoder that predicts per-speaker frame labels. Training includes pseudo-label refinement and retraining, Mixup augmentation, and post-processing with median filtering and overlap averaging. On the MISP 2025 development and evaluation sets, the system reports DER of 7.35% and 8.18%, respectively, compared with the official baseline DER of 15.52% on the evaluation set, a 47.3% relative improvement. A component-wise ablation in Table 1 attributes improvements to post-processing, Mixup, pseudo-label refinement, ECAPA-TDNN, and CASA-Net.

Significance. If the reported evaluation result is obtained under a fully specified and fair protocol, the paper presents a substantial practical improvement over the MISP 2025 baseline and a useful validation of cross-attention/self-attention fusion for AVSD. The ablation study is systematic and covers the main components of the system, which is a strength. However, the paper does not provide code, checkpoints, or full architectural details, and it leaves a central evaluation ambiguity concerning the use of Oracle VAD. The lack of error bars or statistical significance testing further tempers the strength of the component-level claims. On balance, the result is plausible and potentially valuable, but the current presentation does not yet support the headline claim unambiguously.

major comments (3)
  1. [§4.2, Table 1] The manuscript never states whether the reported 8.18% DER, or any row in Table 1, uses the permitted Oracle VAD. The evaluation paragraph in §4.2 says only that "the competition provided access to the Oracle V AD, which participants were permitted to utilize." Since Oracle VAD supplies ground-truth speech activity, it can remove false alarms and misses that an automatic system would have to detect. The headline comparison of 8.18% versus the official baseline 15.52% is therefore ambiguous. Please report a fully automatic DER (without Oracle VAD) and, if Oracle VAD is used anywhere in the evaluation or post-processing, quantify its contribution. Also state explicitly whether the official baseline is equally oracle-assisted.
  2. [§5, Table 1] All ablation differences (0.59% to 0.85% absolute DER) are obtained from a single pass over one evaluation set, with no error bars, confidence intervals, or significance tests. Meeting-level variability in DER is typically nontrivial, so the claim that the "progressive performance degradation" validates the hierarchical design is stronger than the evidence supports. Please add bootstrap confidence intervals or per-meeting standard deviations, or at least acknowledge the limited statistical power of comparisons made on a single evaluation set.
  3. [§3.1, Eq. (1)-(2)] The cross-attention equations are under-specified: the symbols Q_v, K_a, V_a, Q_a, K_v, V_v are not defined, and the paper does not specify the linear projection layers, number of attention heads, feature dimensions, or how the outputs of the multi-head operations are combined. The self-attention module's depth, residual structure, and the way its output is fed to the decoder are also omitted. This makes the central architectural contribution difficult to reproduce independently.
minor comments (5)
  1. [Abstract and §1] The abstract says the paper "introduce[s] CASA-Net," while §1 says "Using CASA-Net [11]," where [11] is the authors' own prior paper. Please clarify what is new in this submission relative to [11] and adjust the abstract accordingly.
  2. [§4] The opening sentence of §4, "The overall audio-visual speaker diarization system based on CASA-Net is shown in Figure 2," repeats the opening sentence of §3 verbatim. This appears to be an editing artifact and should be corrected.
  3. [Eq. (1)-(2), Eq. (3)] Define the scaling dimension d in the attention equations and use a symbol other than T for transpose to avoid confusion with the frame index T. In Eq. (3), the total duration is typeset as T OT AL; please fix the spacing and specify whether speaker error is computed only on overlapping speech as in the DIHARD protocol.
  4. [§4.2] The terms "V-VAD" and "Oracle VAD" are used in close proximity and are easy to confuse. Please define both explicitly (learned visual voice activity detection versus ground-truth speech activity) and state how each is used in training, pseudo-label generation, and evaluation.
  5. [Table 1] The "Official Baseline" row has no Development Set entry. Please provide the baseline development DER or explain why it is unavailable, so readers can interpret the development-set improvements.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports an empirical system with ablations against the challenge baseline, and its self-citation of the earlier CASA-Net paper is not load-bearing.

full rationale

This is an empirical challenge-system paper rather than a derivation. The headline result, an 8.18% DER on the MISP 2025 evaluation set, is measured against the official baseline and supported by an ablation study in Table 1. No equation in the paper defines a prediction in terms of the quantity it is claimed to predict, and no fitted parameter is renamed as a prediction. The authors cite their own prior CASA-Net work [11] as the architectural basis, but that is an independently published earlier paper, and the current paper's contribution is evaluated by removing components (CASA-Net, ECAPA-TDNN, pseudo-label refinement, Mixup, post-processing) and observing performance changes. Pseudo-label refinement does iterate on the model's own outputs, but this is a training strategy, not a proof mechanism or a prediction that reduces to its input by construction. The use of Oracle VAD is a potential evaluation-validity concern for the headline DER, but it is not a circularity: the paper does not claim to derive the DER from the Oracle VAD, and the omission of a fully automatic DER is a reporting weakness, not a self-referential reduction. Therefore, no circular step can be exhibited with a quote and a specific reduction, and the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The system is empirical, so its free parameters are experimental hyperparameters rather than derived constants. The key assumptions are domain-level premises about visual speech cues, the permissibility of Oracle VAD, and the integrity of challenge scoring. No new physical or conceptual entities are introduced.

free parameters (4)
  • Learning rate = 1e-4
    Used for both V-VAD and CASA-Net training with Adam; chosen by hand and affects final DER.
  • Block size and overlap = 8 seconds with 4-second overlap
    Defines the temporal segmentation for training and evaluation; influences the overlap averaging post-processing.
  • Pseudo-label threshold = not specified in paper
    The conversion of model probabilities to timestamp pseudo-labels must rely on a threshold that is not described; this directly controls the retraining loop.
  • Median filter window size = not specified in paper
    Post-processing parameter for outlier removal; the paper attributes a 0.85% DER drop to this module without giving the exact configuration.
assumptions (4)
  • domain assumption Visual lip movements are reliable indicators of speaker activity
    The V-VAD encoding in Section 2.1 assumes lip images contain speech activity cues that survive noise and overlap better than audio alone.
  • domain assumption Oracle VAD use is allowed and does not distort the comparison to the baseline
    Section 4.2 states the oracle is permitted, but the paper does not analyze its effect, so the reported DER may depend on this premise.
  • domain assumption The MISP 2025 challenge ground truth and scoring are correct
    The headline DER is a single number produced by the external challenge evaluation; the paper takes it as ground truth.
  • domain assumption Attention mechanisms can capture the cross-modal alignment needed for fusion
    CASA-Net's design (Section 3.1) assumes cross-attention and self-attention are sufficient for audio-visual fusion and temporal context, without a formal guarantee.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cross-attention and Self-attention for Audio-visual Speaker Diarization in MISP-Meeting Challenge." pith.science (2026). https://pith.science/paper/OREBVW7C

@misc{pith2026250602621,
  author       = {Pith},
  title        = {Pith review of: Cross-attention and Self-attention for Audio-visual Speaker Diarization in MISP-Meeting Challenge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OREBVW7C}},
  note         = {Machine review of arXiv:2506.02621}
}
read the original abstract

This paper presents the system developed for Task 1 of the Multi-modal Information-based Speech Processing (MISP) 2025 Challenge. We introduce CASA-Net, an embedding fusion method designed for end-to-end audio-visual speaker diarization (AVSD) systems. CASA-Net incorporates a cross-attention (CA) module to effectively capture cross-modal interactions in audio-visual signals and employs a self-attention (SA) module to learn contextual relationships among audio-visual frames. To further enhance performance, we adopt a training strategy that integrates pseudo-label refinement and retraining, improving the accuracy of timestamp predictions. Additionally, median filtering and overlap averaging are applied as post-processing techniques to eliminate outliers and smooth prediction labels. Our system achieved a diarization error rate (DER) of 8.18% on the evaluation set, representing a relative improvement of 47.3% over the baseline DER of 15.52%.

Figures

Figures reproduced from arXiv: 2506.02621 by the authors.

Figure 1
Figure 1. The baseline of MISP 2025 Challenge encoder, formulated as XV ∈ R T ×W×H×N , where T denotes the number of frames, and W, H represent the image width and height, respectively, while N is the number of speakers. The encoder extracts visual embeddings EV ∈ R T ×DV ×N , where DV is the embedding dimension. 2.2. Audio temporal encoder The audio temporal encoder processes FBank features through a four-layer convolutional… view at source ↗
Figure 2
Figure 2. Framework of the audio-visual speaker diarization system based on CASA-Net. 𝐼 𝐸𝐴 𝐸𝑉 𝑄2 𝐾1 𝑉1 𝑄1 𝐾2 𝑉2 CA CA SA CASA Decoder Multiple Speaker Probabilities SPK 1 SPK 2 ··· SPK N (a) The framework of CASA network. Multi-Head Attention Add & Norm Q K 𝐹𝑣/𝐹𝑎 𝐹𝑎/𝐹𝑣 V Concat (𝐹𝑎→𝑣, 𝐹𝑣→𝑎 ) Multi-Head Attention Add & Norm Q K V (b) Structure of CA and SA. (b-1) Cross-Attention (CA) (b-2) Self-Attention (SA) [PITH_FULL_IMAGE… view at source ↗
Figure 3
Figure 3. The framework of CASA network and the structure of CA and SA from an initial log file generated using V-VAD for the first￾round training. The system then computes the loss based on the predicted results and ground-truth labels. Through post￾processing, timestamp pseudo-labels are generated and con￾verted into a new log file, replacing the initial log file. This process is iteratively refined over multiple training r… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 20 canonical work pages

  1. [1]

    who spoke when

    Introduction Audio-visual speaker diarization (A VSD) is a technology that integrates both audio and visual signals to determine “who spoke when” [1] in multi-speaker conversational scenarios. It has been widely applied in areas such as remote video confer- encing summaries [2] and audiovisual speech transcription [3]. A VSD has evolved from traditional a...

  2. [2]

    Cross-attention and Self-attention for Audio-visual Speaker Diarization in MISP-Meeting Challenge

    Baseline works The MISP 2025 Challenge employs a baseline model [7] that integrates several key components: visual and audio temporal encoders, an i-vector extractor, and a decoder. The overall ar- chitecture is depicted in Figure 1. The following subsections provide detailed descriptions of each module. 2.1. Visual temporal encoder The visual temporal en...

  3. [3]

    System description The overall audio-visual speaker diarization system based on CASA-Net is shown in Figure 2. Compared to the baseline, the main improvements include modifications to the visual en- coder, the introduction of the CASA-Net architecture, and the incorporation of an x-vector-based speaker embedding model as both the audio encoder and speaker...

  4. [4]

    The following section details the experimental process

    Datasets and experimental setup The overall audio-visual speaker diarization system based on CASA-Net is shown in Figure 2. The following section details the experimental process. 4.1. Data preparation Our primary dataset is the MISP 2025 Challenge corpus, which comprises a 119-hour training set (each meeting lasting 2 hours), a 3-hour development set (ea...

  5. [5]

    As systematically sum- marized in Table 1, the experimental results reveal critical in- sights about the structure design of our proposed system

    Experimental results To quantitatively evaluate the contribution of each proposed component, we conduct a comprehensive ablation study under the DIHARD-III evaluation protocol. As systematically sum- marized in Table 1, the experimental results reveal critical in- sights about the structure design of our proposed system. Our final integrated system achiev...

  6. [6]

    Our system employs a CASA-Net based architecture to effectively address the temporal misalignment between audio and video streams

    Conclusion In this paper, we propose an audio-visual speaker diarization system for Task 1 of the MISP 2025 Challenge. Our system employs a CASA-Net based architecture to effectively address the temporal misalignment between audio and video streams. By integrating cross-attention and self-attention modules, we achieve seamless audio-visual feature fusion ...

  7. [7]

    Acknowledgements This work was supported in part by the National Natural Science Foundation of China under Grants 62371407 and 62276220, and the Innovation of Policing Science and Tech- nology, Fujian province (Grant number: 2024Y0068)

  8. [8]

    Lip-reading with densely connected temporal convolutional networks,

    P. Ma, Y . Wang, J. Shen, S. Petridis, and M. Pantic, “Lip-reading with densely connected temporal convolutional networks,” inPro- ceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2021, pp. 2857–2866

Show all 27 references
  1. [9]

    A review of speaker diarization: Recent advances with deep learning,

    T. J. Park, N. Kanda, D. Dimitriadis, K. J. Han, S. Watanabe, and S. Narayanan, “A review of speaker diarization: Recent advances with deep learning,” Computer Speech & Language , vol. 72, p. 101317, 2022

  2. [10]

    Third dihard challenge evaluation plan,

    N. Ryant, K. Church, C. Cieri, J. Du, S. Ganapathy, and M. Liber- man, “Third dihard challenge evaluation plan,” arXiv preprint arXiv:2006.05815, 2020

  3. [11]

    End-to-end neural speaker diarization with self- attention,

    Y . Fujita, N. Kanda, S. Horiguchi, Y . Xue, K. Nagamatsu, and S. Watanabe, “End-to-end neural speaker diarization with self- attention,” in 2019 IEEE Automatic Speech Recognition and Un- derstanding Workshop (ASRU). IEEE, 2019, pp. 296–303

  4. [12]

    Target-speaker voice activity de- tection: a novel approach for multi-speaker diarization in a dinner party scenario,

    I. Medennikov, M. Korenevsky, T. Prisyach, Y . Khokhlov, M. Ko- renevskaya, I. Sorokin, T. Timofeeva, A. Mitrofanov, A. An- drusenko, I. Podluzhny et al., “Target-speaker voice activity de- tection: a novel approach for multi-speaker diarization in a dinner party scenario,” ar...

  5. [13]

    Quantitative as- sociation of vocal-tract and facial behavior,

    H. Yehia, P. Rubin, and E. Vatikiotis-Bateson, “Quantitative as- sociation of vocal-tract and facial behavior,”Speech Communica- tion, vol. 26, no. 1-2, pp. 23–43, 1998

  6. [14]

    Hearing lips and seeing voices,

    H. McGurk and J. MacDonald, “Hearing lips and seeing voices,” Nature, vol. 264, no. 5588, pp. 746–748, 1976

  7. [15]

    End-to-end audio-visual neu- ral speaker diarization,

    M.-K. He, J. Du, and C.-H. Lee, “End-to-end audio-visual neu- ral speaker diarization,” Proc. Interspeech 2022, pp. 1461–1465, 2022

  8. [16]

    Two-dimensional digital signal processing ii. trans- forms and median filters

    T. S. Huang, “Two-dimensional digital signal processing ii. trans- forms and median filters.” in Two-dimensional digital signal pro- cessing II. Transforms and median filters. Springer, 1981

  9. [17]

    Effi- cient video transformers with spatial-temporal token selection,

    J. Wang, X. Yang, H. Li, L. Liu, Z. Wu, and Y .-G. Jiang, “Effi- cient video transformers with spatial-temporal token selection,” in Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXV . Springer, 2022, pp. 69–86

  10. [18]

    for dereverberation of multi-channel audio. To leverage spatial information from multiple channels, the Adaptive Beam- forming algorithm from the Kaldi toolkit [19] is applied to merge multi-channel far-field audio into a single-channel sig- nal. The augmented audio dataset co...

  11. [19]

    Spatial-temporal transformer for 3d point cloud sequences,

    Y . Wei, H. Liu, T. Xie, Q. Ke, and Y . Guo, “Spatial-temporal transformer for 3d point cloud sequences,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vi- sion, 2022, pp. 1171–1180

  12. [20]

    Casa-net: Cross- attention and self-attention for end-to-end audio-visual speaker di- arization,

    H. Zhou, T. Li, J. Wang, L. Li, and Q. Hong, “Casa-net: Cross- attention and self-attention for end-to-end audio-visual speaker di- arization,” in 2023 Asia Pacific Signal and Information Process- ing Association Annual Summit and Conference (APSIPA ASC) . IEEE, 2023, pp. 102–106

  13. [21]

    Cn-celeb: a challenging chinese speaker recognition dataset,

    Y . Fan, J. Kang, L. Li, K. Li, H. Chen, S. Cheng, P. Zhang, Z. Zhou, Y . Cai, and D. Wang, “Cn-celeb: a challenging chinese speaker recognition dataset,” in ICASSP 2020-2020 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 7604–7608

  14. [22]

    Ecapa- tdnn: Emphasized channel attention, propagation and ag- gregation in tdnn based speaker verification,

    B. Desplanques, J. Thienpondt, and K. Demuynck, “Ecapa- tdnn: Emphasized channel attention, propagation and ag- gregation in tdnn based speaker verification,” arXiv preprint arXiv:2005.07143, 2020

  15. [23]

    mixup: Beyond empirical risk minimization,

    H. Zhang, “mixup: Beyond empirical risk minimization,” arXiv preprint arXiv:1710.09412, 2017

  16. [24]

    Augment on manifold: Mixup regularization with umap,

    Y . El-Laham, E. Fons, D. Daudert, and S. Vyetrenko, “Augment on manifold: Mixup regularization with umap,” in ICASSP 2024- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 7040–7044

  17. [25]

    A quan- titative comparison of overlapping and non-overlapping sliding windows for human activity recognition using inertial sensors,

    A. Dehghani, O. Sarbishei, T. Glatard, and E. Shihab, “A quan- titative comparison of overlapping and non-overlapping sliding windows for human activity recognition using inertial sensors,” Sensors, vol. 19, no. 22, p. 5026, 2019

  18. [26]

    Nara-wpe: A python package for weighted prediction error dereverberation in numpy and tensorflow for online and offline processing,

    L. Drude, J. Heymann, C. Boeddeker, and R. Haeb-Umbach, “Nara-wpe: A python package for weighted prediction error dereverberation in numpy and tensorflow for online and offline processing,” in Speech Communication; 13th ITG-Symposium . VDE, 2018, pp. 1–5

  19. [27]

    The kaldi speech recognition toolkit,

    D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y . Qian, P. Schwarzet al., “The kaldi speech recognition toolkit,” inIEEE 2011 workshop on automatic speech recognition and understanding. IEEE Signal Processing Society, 2011

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.