REVIEW 3 major objections 5 minor 27 references
Cross-attention and Self-attention for Audio-visual Speaker Diarization in MISP-Meeting Challenge
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read CASA-Net fuses cross-attention and self-attention to cut audio-visual speaker diarization error on the MISP 2025 evaluation set to 8.18%, a 47.3% relative improvement over the 15.52% baseline.
desk verdict Solid challenge-system writeup with a clean ablation, but the Oracle-VAD ambiguity undermines the headline 8.18% vs baseline comparison until clarified. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is CASA-Net, a two-stage attention fusion block. In the cross-attention stage, concatenated audio and i-vector/x-vector embeddings $F_a$ serve as key and value while visual lip embeddings $F_v$ serve as query, and symmetrically in the other direction, producing $F_{a\to v}$ and $F_{v\to a}$ via scaled dot-product attention; the self-attention stage then runs over the concatenation of the two cross-attended streams to capture temporal context across frames. This attention-based fusion replaces the plain concatenation used in the baseline decoder.
What would settle it
Disable the Oracle VAD, replace it with a separately trained voice-activity detector, and recompute DER on the MISP 2025 development and evaluation sets; if the error rises substantially above 8.18% or approaches the 15.52% baseline, the headline improvement depends on oracle speech-activity information. Also rerun the official baseline with the same Oracle VAD enabled to confirm both systems are compared under identical conditions.
Extended reading notes
Core claim
On its own terms, the paper's central discovery is that a cross-attention and self-attention fusion network, applied to visual voice-activity-detection embeddings, audio filterbank features, and x-vector speaker embeddings, reliably improves diarization timestamps in noisy overlapped meetings. The reported numbers support this: the full system achieves 7.35% DER on development and 8.18% on evaluation; ablations show removing CASA-Net raises evaluation DER to 17.04%, removing pseudo-label refinement to 10.41%, removing Mixup to 9.71%, and removing post-processing to 9.03%. The authors interpret this as evidence that each component contributes, with the attention fusion providing the largest single gain.
Load-bearing premise
The load-bearing premise is that the reported 8.18% diarization error rate is a fair measure of an autonomous system; the evaluation permitted use of a perfect speech-activity detector (Oracle VAD), and the paper does not quantify how much of the gain depends on it.
Editorial extensions
If this is right
- If the reported numbers hold, attention-based fusion is a direct replacement for concatenation in end-to-end audio-visual diarization pipelines, with a large accuracy gain on the MISP 2025 task.
- The pseudo-label refinement loop means that a system can use its own visual-VAD-derived timestamps to retrain the frame-level network, reducing train-test mismatch in speaker-embedding features.
- Mixup on lip images and speaker embeddings, together with median filtering and overlap averaging on predictions, are presented as cheap additions that each yield roughly 0.6 to 0.9 percentage points of absolute DER improvement.
- The paper's strongest claim is that the complete system matches the state-of-the-art performance on the leaderboard at 8.18% DER; if accepted, it sets a new target for subsequent audio-visual speaker diarization challenge entries.
Reading between the lines
- Beyond the paper: the evaluation protocol allowed use of an Oracle VAD, and the paper does not report a fully automatic DER; the 8.18% number may include a component that an autonomous system would need to estimate.
- Beyond the paper: the ablation table compares the full system against the official baseline, but the system differs from the baseline in several ways (visual encoder, audio encoder, x-vector vs. i-vector, training data, post-processing). The 47.3% improvement should therefore be read as the gain of the whole package, not of CASA-Net alone.
- Beyond the paper: a natural extension would be to test the same cross-attention and self-attention fusion on audio-only diarization or on target-speaker voice activity detection, to see whether the gains transfer beyond the MISP corpus.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes the authors' system for Task 1 of the MISP 2025 Challenge, which performs audio-visual speaker diarization (AVSD). The proposed system builds on their prior CASA-Net architecture, using cross-attention between audio features (FBank plus an ECAPA-TDNN x-vector embedding) and visual features (V-VAD lip-region embeddings), followed by self-attention, and then a decoder that predicts per-speaker frame labels. Training includes pseudo-label refinement and retraining, Mixup augmentation, and post-processing with median filtering and overlap averaging. On the MISP 2025 development and evaluation sets, the system reports DER of 7.35% and 8.18%, respectively, compared with the official baseline DER of 15.52% on the evaluation set, a 47.3% relative improvement. A component-wise ablation in Table 1 attributes improvements to post-processing, Mixup, pseudo-label refinement, ECAPA-TDNN, and CASA-Net.
Significance. If the reported evaluation result is obtained under a fully specified and fair protocol, the paper presents a substantial practical improvement over the MISP 2025 baseline and a useful validation of cross-attention/self-attention fusion for AVSD. The ablation study is systematic and covers the main components of the system, which is a strength. However, the paper does not provide code, checkpoints, or full architectural details, and it leaves a central evaluation ambiguity concerning the use of Oracle VAD. The lack of error bars or statistical significance testing further tempers the strength of the component-level claims. On balance, the result is plausible and potentially valuable, but the current presentation does not yet support the headline claim unambiguously.
major comments (3)
- [§4.2, Table 1] The manuscript never states whether the reported 8.18% DER, or any row in Table 1, uses the permitted Oracle VAD. The evaluation paragraph in §4.2 says only that "the competition provided access to the Oracle V AD, which participants were permitted to utilize." Since Oracle VAD supplies ground-truth speech activity, it can remove false alarms and misses that an automatic system would have to detect. The headline comparison of 8.18% versus the official baseline 15.52% is therefore ambiguous. Please report a fully automatic DER (without Oracle VAD) and, if Oracle VAD is used anywhere in the evaluation or post-processing, quantify its contribution. Also state explicitly whether the official baseline is equally oracle-assisted.
- [§5, Table 1] All ablation differences (0.59% to 0.85% absolute DER) are obtained from a single pass over one evaluation set, with no error bars, confidence intervals, or significance tests. Meeting-level variability in DER is typically nontrivial, so the claim that the "progressive performance degradation" validates the hierarchical design is stronger than the evidence supports. Please add bootstrap confidence intervals or per-meeting standard deviations, or at least acknowledge the limited statistical power of comparisons made on a single evaluation set.
- [§3.1, Eq. (1)-(2)] The cross-attention equations are under-specified: the symbols Q_v, K_a, V_a, Q_a, K_v, V_v are not defined, and the paper does not specify the linear projection layers, number of attention heads, feature dimensions, or how the outputs of the multi-head operations are combined. The self-attention module's depth, residual structure, and the way its output is fed to the decoder are also omitted. This makes the central architectural contribution difficult to reproduce independently.
minor comments (5)
- [Abstract and §1] The abstract says the paper "introduce[s] CASA-Net," while §1 says "Using CASA-Net [11]," where [11] is the authors' own prior paper. Please clarify what is new in this submission relative to [11] and adjust the abstract accordingly.
- [§4] The opening sentence of §4, "The overall audio-visual speaker diarization system based on CASA-Net is shown in Figure 2," repeats the opening sentence of §3 verbatim. This appears to be an editing artifact and should be corrected.
- [Eq. (1)-(2), Eq. (3)] Define the scaling dimension d in the attention equations and use a symbol other than T for transpose to avoid confusion with the frame index T. In Eq. (3), the total duration is typeset as T OT AL; please fix the spacing and specify whether speaker error is computed only on overlapping speech as in the DIHARD protocol.
- [§4.2] The terms "V-VAD" and "Oracle VAD" are used in close proximity and are easy to confuse. Please define both explicitly (learned visual voice activity detection versus ground-truth speech activity) and state how each is used in training, pseudo-label generation, and evaluation.
- [Table 1] The "Official Baseline" row has no Development Set entry. Please provide the baseline development DER or explain why it is unavailable, so readers can interpret the development-set improvements.
Circularity Check
No significant circularity: the paper reports an empirical system with ablations against the challenge baseline, and its self-citation of the earlier CASA-Net paper is not load-bearing.
full rationale
This is an empirical challenge-system paper rather than a derivation. The headline result, an 8.18% DER on the MISP 2025 evaluation set, is measured against the official baseline and supported by an ablation study in Table 1. No equation in the paper defines a prediction in terms of the quantity it is claimed to predict, and no fitted parameter is renamed as a prediction. The authors cite their own prior CASA-Net work [11] as the architectural basis, but that is an independently published earlier paper, and the current paper's contribution is evaluated by removing components (CASA-Net, ECAPA-TDNN, pseudo-label refinement, Mixup, post-processing) and observing performance changes. Pseudo-label refinement does iterate on the model's own outputs, but this is a training strategy, not a proof mechanism or a prediction that reduces to its input by construction. The use of Oracle VAD is a potential evaluation-validity concern for the headline DER, but it is not a circularity: the paper does not claim to derive the DER from the Oracle VAD, and the omission of a fully automatic DER is a reporting weakness, not a self-referential reduction. Therefore, no circular step can be exhibited with a quote and a specific reduction, and the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (4)
- Learning rate =
1e-4
- Block size and overlap =
8 seconds with 4-second overlap
- Pseudo-label threshold =
not specified in paper
- Median filter window size =
not specified in paper
assumptions (4)
- domain assumption Visual lip movements are reliable indicators of speaker activity
- domain assumption Oracle VAD use is allowed and does not distort the comparison to the baseline
- domain assumption The MISP 2025 challenge ground truth and scoring are correct
- domain assumption Attention mechanisms can capture the cross-modal alignment needed for fusion
Cite this review
Pith. "Pith review of Cross-attention and Self-attention for Audio-visual Speaker Diarization in MISP-Meeting Challenge." pith.science (2026). https://pith.science/paper/OREBVW7C
@misc{pith2026250602621,
author = {Pith},
title = {Pith review of: Cross-attention and Self-attention for Audio-visual Speaker Diarization in MISP-Meeting Challenge},
year = {2026},
howpublished = {\url{https://pith.science/paper/OREBVW7C}},
note = {Machine review of arXiv:2506.02621}
}
read the original abstract
This paper presents the system developed for Task 1 of the Multi-modal Information-based Speech Processing (MISP) 2025 Challenge. We introduce CASA-Net, an embedding fusion method designed for end-to-end audio-visual speaker diarization (AVSD) systems. CASA-Net incorporates a cross-attention (CA) module to effectively capture cross-modal interactions in audio-visual signals and employs a self-attention (SA) module to learn contextual relationships among audio-visual frames. To further enhance performance, we adopt a training strategy that integrates pseudo-label refinement and retraining, improving the accuracy of timestamp predictions. Additionally, median filtering and overlap averaging are applied as post-processing techniques to eliminate outliers and smooth prediction labels. Our system achieved a diarization error rate (DER) of 8.18% on the evaluation set, representing a relative improvement of 47.3% over the baseline DER of 15.52%.
Figures
Reference graph
Works this paper leans on
-
[1]
Introduction Audio-visual speaker diarization (A VSD) is a technology that integrates both audio and visual signals to determine “who spoke when” [1] in multi-speaker conversational scenarios. It has been widely applied in areas such as remote video confer- encing summaries [2] and audiovisual speech transcription [3]. A VSD has evolved from traditional a...
work page 2025
-
[2]
Cross-attention and Self-attention for Audio-visual Speaker Diarization in MISP-Meeting Challenge
Baseline works The MISP 2025 Challenge employs a baseline model [7] that integrates several key components: visual and audio temporal encoders, an i-vector extractor, and a decoder. The overall ar- chitecture is depicted in Figure 1. The following subsections provide detailed descriptions of each module. 2.1. Visual temporal encoder The visual temporal en...
work page Pith review arXiv 2025
-
[3]
System description The overall audio-visual speaker diarization system based on CASA-Net is shown in Figure 2. Compared to the baseline, the main improvements include modifications to the visual en- coder, the introduction of the CASA-Net architecture, and the incorporation of an x-vector-based speaker embedding model as both the audio encoder and speaker...
-
[4]
The following section details the experimental process
Datasets and experimental setup The overall audio-visual speaker diarization system based on CASA-Net is shown in Figure 2. The following section details the experimental process. 4.1. Data preparation Our primary dataset is the MISP 2025 Challenge corpus, which comprises a 119-hour training set (each meeting lasting 2 hours), a 3-hour development set (ea...
work page 2025
-
[5]
Experimental results To quantitatively evaluate the contribution of each proposed component, we conduct a comprehensive ablation study under the DIHARD-III evaluation protocol. As systematically sum- marized in Table 1, the experimental results reveal critical in- sights about the structure design of our proposed system. Our final integrated system achiev...
-
[6]
Conclusion In this paper, we propose an audio-visual speaker diarization system for Task 1 of the MISP 2025 Challenge. Our system employs a CASA-Net based architecture to effectively address the temporal misalignment between audio and video streams. By integrating cross-attention and self-attention modules, we achieve seamless audio-visual feature fusion ...
work page 2025
-
[7]
Acknowledgements This work was supported in part by the National Natural Science Foundation of China under Grants 62371407 and 62276220, and the Innovation of Policing Science and Tech- nology, Fujian province (Grant number: 2024Y0068)
-
[8]
Lip-reading with densely connected temporal convolutional networks,
P. Ma, Y . Wang, J. Shen, S. Petridis, and M. Pantic, “Lip-reading with densely connected temporal convolutional networks,” inPro- ceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2021, pp. 2857–2866
work page 2021
Show all 27 references
-
[9]
A review of speaker diarization: Recent advances with deep learning,
T. J. Park, N. Kanda, D. Dimitriadis, K. J. Han, S. Watanabe, and S. Narayanan, “A review of speaker diarization: Recent advances with deep learning,” Computer Speech & Language , vol. 72, p. 101317, 2022
2022
-
[10]
Third dihard challenge evaluation plan,
N. Ryant, K. Church, C. Cieri, J. Du, S. Ganapathy, and M. Liber- man, “Third dihard challenge evaluation plan,” arXiv preprint arXiv:2006.05815, 2020
2006 arXiv
-
[11]
End-to-end neural speaker diarization with self- attention,
Y . Fujita, N. Kanda, S. Horiguchi, Y . Xue, K. Nagamatsu, and S. Watanabe, “End-to-end neural speaker diarization with self- attention,” in 2019 IEEE Automatic Speech Recognition and Un- derstanding Workshop (ASRU). IEEE, 2019, pp. 296–303
2019
-
[12]
Target-speaker voice activity de- tection: a novel approach for multi-speaker diarization in a dinner party scenario,
I. Medennikov, M. Korenevsky, T. Prisyach, Y . Khokhlov, M. Ko- renevskaya, I. Sorokin, T. Timofeeva, A. Mitrofanov, A. An- drusenko, I. Podluzhny et al., “Target-speaker voice activity de- tection: a novel approach for multi-speaker diarization in a dinner party scenario,” ar...
2005 arXiv
-
[13]
Quantitative as- sociation of vocal-tract and facial behavior,
H. Yehia, P. Rubin, and E. Vatikiotis-Bateson, “Quantitative as- sociation of vocal-tract and facial behavior,”Speech Communica- tion, vol. 26, no. 1-2, pp. 23–43, 1998
1998
-
[14]
Hearing lips and seeing voices,
H. McGurk and J. MacDonald, “Hearing lips and seeing voices,” Nature, vol. 264, no. 5588, pp. 746–748, 1976
1976
-
[15]
End-to-end audio-visual neu- ral speaker diarization,
M.-K. He, J. Du, and C.-H. Lee, “End-to-end audio-visual neu- ral speaker diarization,” Proc. Interspeech 2022, pp. 1461–1465, 2022
2022
-
[16]
Two-dimensional digital signal processing ii. trans- forms and median filters
T. S. Huang, “Two-dimensional digital signal processing ii. trans- forms and median filters.” in Two-dimensional digital signal pro- cessing II. Transforms and median filters. Springer, 1981
1981
-
[17]
Effi- cient video transformers with spatial-temporal token selection,
J. Wang, X. Yang, H. Li, L. Liu, Z. Wu, and Y .-G. Jiang, “Effi- cient video transformers with spatial-temporal token selection,” in Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXV . Springer, 2022, pp. 69–86
2022
-
[18]
for dereverberation of multi-channel audio. To leverage spatial information from multiple channels, the Adaptive Beam- forming algorithm from the Kaldi toolkit [19] is applied to merge multi-channel far-field audio into a single-channel sig- nal. The augmented audio dataset co...
-
[19]
Spatial-temporal transformer for 3d point cloud sequences,
Y . Wei, H. Liu, T. Xie, Q. Ke, and Y . Guo, “Spatial-temporal transformer for 3d point cloud sequences,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vi- sion, 2022, pp. 1171–1180
2022
-
[20]
Casa-net: Cross- attention and self-attention for end-to-end audio-visual speaker di- arization,
H. Zhou, T. Li, J. Wang, L. Li, and Q. Hong, “Casa-net: Cross- attention and self-attention for end-to-end audio-visual speaker di- arization,” in 2023 Asia Pacific Signal and Information Process- ing Association Annual Summit and Conference (APSIPA ASC) . IEEE, 2023, pp. 102–106
2023
-
[21]
Cn-celeb: a challenging chinese speaker recognition dataset,
Y . Fan, J. Kang, L. Li, K. Li, H. Chen, S. Cheng, P. Zhang, Z. Zhou, Y . Cai, and D. Wang, “Cn-celeb: a challenging chinese speaker recognition dataset,” in ICASSP 2020-2020 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 7604–7608
2020
-
[22]
Ecapa- tdnn: Emphasized channel attention, propagation and ag- gregation in tdnn based speaker verification,
B. Desplanques, J. Thienpondt, and K. Demuynck, “Ecapa- tdnn: Emphasized channel attention, propagation and ag- gregation in tdnn based speaker verification,” arXiv preprint arXiv:2005.07143, 2020
2005 arXiv
-
[23]
mixup: Beyond empirical risk minimization,
H. Zhang, “mixup: Beyond empirical risk minimization,” arXiv preprint arXiv:1710.09412, 2017
2017 arXiv
-
[24]
Augment on manifold: Mixup regularization with umap,
Y . El-Laham, E. Fons, D. Daudert, and S. Vyetrenko, “Augment on manifold: Mixup regularization with umap,” in ICASSP 2024- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 7040–7044
2024
-
[25]
A quan- titative comparison of overlapping and non-overlapping sliding windows for human activity recognition using inertial sensors,
A. Dehghani, O. Sarbishei, T. Glatard, and E. Shihab, “A quan- titative comparison of overlapping and non-overlapping sliding windows for human activity recognition using inertial sensors,” Sensors, vol. 19, no. 22, p. 5026, 2019
2019
-
[26]
Nara-wpe: A python package for weighted prediction error dereverberation in numpy and tensorflow for online and offline processing,
L. Drude, J. Heymann, C. Boeddeker, and R. Haeb-Umbach, “Nara-wpe: A python package for weighted prediction error dereverberation in numpy and tensorflow for online and offline processing,” in Speech Communication; 13th ITG-Symposium . VDE, 2018, pp. 1–5
2018
-
[27]
The kaldi speech recognition toolkit,
D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y . Qian, P. Schwarzet al., “The kaldi speech recognition toolkit,” inIEEE 2011 workshop on automatic speech recognition and understanding. IEEE Signal Processing Society, 2011
2011
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.